I've blogged before about
using Buildbot to build our application server.
One problem with it is that the build (and testing) process can be memory
intensive, which can sometimes exceed the memory that we have available in our
Kubernetes cluster. I could add another worker node, but that would be a
waste, since we do builds infrequently.
Fortunately, the Buildbot developers have already built a solution to this:
A latent buildslave is a virtual server that is provisioned on-demand. That
means that when a build isn't active, then we don't have to pay for an extra
server to be active; we only have to pay for the compute time that we actually
need for builds (plus a bit of storage space).
I chose to use AWS EC2 as the basis of our buildslave. Buildbot also supports
OpenStack, so I could have just used
DreamCompute, which we already use
for our Kubernetes cluster, but with AWS EC2, we can take advantage of
and save even more money if we needed to. In any event, the setup would have
been pretty much the same.
Setting up a latent buildslave on AWS is pretty straightforward. First, create
an EC2 instance in order to build a base image for the buildslave. I started
with a Debian image. Then, install any necessary software for the buildslave.
For us, that included the buildslave software itself (Debian package
buildbot-slave), git, tinc, npm, and Docker. Most of our build process happens
inside of Docker containers, so we don't need anything else. We use tinc to
build a virtual network with our Kubernetes cluster, so that we can push Docker
images to our own private Docker repository.
After installing the necessary software, we need to configure it. It's
configured just like a normal buildslave would be configured: I configured
tinc, added an ssh key so that it could check out our source code, configured
Docker so that it could push to our repository, and of course configured the
Buildbot slave itself. Once it's configured, I cleaned up the image a bit
(truncated logs, cleared bash history, etc.), and then took a snapshot in the
AWS control panel, giving it a name so that it would show up as an AMI.
Finally, I added the latent buildslave in our Buildbot master configuration,
giving it the name of the AMI that was created. Once set up, it ran pretty
much as expected. I pushed out a change, Buildbot master created a new EC2
instance, built our application server, pushed and deployed it to our
Kubernetes cluster, and after a short delay (to make sure there are no other
builds), deleted the EC2 instance. In all, the EC2 instance ran for about 20
minutes. Timings will vary, of course, but it will run for less than an hour.
If we were paying full price for a t2.micro instance in us-east-1, each build
would cost just over 1 cent. We also need to add in the storage cost for the
AMI which, given that I started with an 8GB image, will cost us at most 80
cents per month (since EBS snapshots don't store empty blocks, it should be
less than that). We probably average about two builds a month, giving us an
average monthly cost of at most 83 cents, which is not too bad.