May 14, 2019

How secure is emoji-based verification?

11:46 -0400

In February, Riot 1.0 was released, which featured a brand new verification system for device keys. The Android and iOS versions of Riot also recently included the new verification system. Key verification is an important part of ensuring that encrypted messages are only read by the people whom you want to be able to read. With emoji verification, instead of comparing long incomprehensible strings of characters, as in previous versions of Riot, key verification can now be done by comparing seven emoji. The seven emoji are chosen from a pool of 64, chosen to be distinguishable from each other and describable with few words, with the goal that people using different platforms with different emoji renderings should still be able to verify each other. (Riot also supports using a short sequence of decimal numbers when verifying against other Matrix clients that are unable to show emoji.)

The astute reader will note that seven emoji chosen from a pool of 64 (repetitions are allowed) gives you a total of 647 or 242 total possibilities. But surely 42 bits (despite 42 being the answer to the ultimate question of life, the universe, and everything) isn't enough to verify a single encryption key, let alone two!? What's going on here?

The short answer is that it isn't simply verifying the encryption keys directly. Rather, the devices involved are negotiating a shared secret, and the emoji are used to verify the shared secret negotiation. The shared secret is then used by the devices to verify their device keys by way of a Message Authentication Code (MAC). You'll note that if you begin a key verification in Riot, cancel it, and then begin a new verification, then you'll see that the emoji are different each time. This is because the devices negotiate a new shared secret each time you do a verification.

So are 42 bits secure enough to verify the shared secret? Let's look at how it works in more detail.

Basic Diffie-Hellman

Suppose that Alice and Bob wish to verify their keys, and Mallory wishes to attack the verification process so that she can trick Alice and Bob into verifying her own keys. Let's first consider what happens if Alice and Bob simply perform an (Elliptic-Curve) Diffie-Hellman. Elliptic-Curve Diffie-Hellman, once the elliptic curve is established, simply consists of one message in each direction: Alice and Bob each generate a key pair, and send each other their public keys. They use their own private key and the other person's public key, do some math, and end up arriving at the same shared secret. An eavesdropper cannot discover the shared secret, as they only know the public keys, which is not enough to calculate the shared secret. But if Mallory can intercept and alter the communication between Alice and Bob, then she can replace their public keys with her own public key: when Alice sends Bob, Bob instead receives Mallory's public key, and similarly when Bob sends Alice his public key. Alice and Bob then complete the calculations using Mallory's public key, and rather than having a shared secret between the two of them, they each have a shared secret with Mallory. If they do not verify that the shared secret matches, then they will not know that Mallory has attacked their secret sharing.

Alice and Bob perform a Diffie-Hellman
Mallory intercepts Alice and Bob's exchange

Diffie-Hellman with a verified shared secret

In order to detect Mallory's attack on their communication, Alice and Bob try to verify that they have ended up with the same shared secret. They don't want to verify the entire secret, as that would involve comparing a long string, and verifying the secret directly would expose the shared secret to an eavesdropper. So instead, they take some sort of hash of the secret and compare some number n of bits from the hash in some way, say through the use of an authentication string generated based on those n bits. Now Mallory can't just sit in the middle and just send any key to Alice and Bob; she needs to make sure that when they compare the bits from the hash, they'll get the same result, even though the shared secrets are different. If the Diffie-Hellman is done in a way where one person (say Alice) sends her public key first, and then Bob sends his public key after he receives Alice's public key, then Mallory will need to send some key to Alice, and then she will need to try, on average, about 2n-1 different keys to send to Bob before she finds one that has the same n bits of the hash. However, Mallory is able to do this before sending anything to Bob, so if n is small enough, Alice and Bob won't notice anything other than maybe a slight delay. If n is increased, it may increase the delay introduced by Mallory's attack so that it is noticeable, but then it means that Alice and Bob need to compare more bits, making it more work for them to verify since the authentication string that they compare will be longer.

Alice and Bob verify the shared secret; Mallory tries to get around this verification

Diffie-Hellman with a verified shared secret and hash commitment

At this point, we borrow an idea from ZRTP (by Phil Zimmerman (the creator of PGP), and others). Rather than just having Alice and Bob send each other their public keys, the exchange begins with Bob sending Alice a hash of his public key (and some other information). This is called a hash commitment. Alice then sends Bob her public key, and finally Bob sends Alice his public key. By having Bob send a hash of his key first (and by using a strong hash), this prevents Mallory from being able try different keys like she was able to before, because she now needs to find a key that not only results in a collision in the authentication string, but also has the same hash as the hash that Bob sent.

Bob sends Alice a hash commitment before any public keys are exchanged

Let's look at what Mallory would need to do to attack the exchange, step by step. Bob sends Alice a hash of his public key. Mallory intercepts the transmission and can alter it or not; we'll take a look later on at whether she wants to modify it or not, and if so, how she would want to alter it. Alice receives the (possibly modified) hash from Bob, and sends Bob her public key. Again, Mallory intercepts the transmission. Mallory needs Bob to see a public key that she has a private key for, so that she can calculate a shared secret with Bob, so she creates a new key pair and sends the public key to Bob instead of Alice's public key. Now Bob sends his public key to Alice. Again, Mallory intercepts the transmission, and now needs to create a new key pair and send the public key to Alice. However, the public key that she sends must satisfy two criteria: first of all, the authentication string generated by that key pair with Alice's key pair must match the authentication string generated by the key that she send to Bob with Bob's key pair. And secondly, the public key must hash to the same value as the hash that Alice received initially. If Mallory did not modify the hash that Bob sent, then she is stuck trying to find a key that has the same hash, which if the hash is good, should be nearly impossible. So what Mallory should have done was to replace the hash that Bob sent with the hash of a key that she already knows. If she does a birthday attack, she may end up with more than one key with the same hash, but again, if the hash is good, this should be nearly impossible, so we will assume that she only has one key for that hash. Now Mallory has solved the problem of making her key match the hash that Alice received, but not the problem of making the authentication strings match. Just by chance, Mallory may have selected a key that produces the same authentication string on both sides; this has a 1 in 2n chance of happening. If the key that she has already selected doesn't produce the same authentication string, then Mallory is stuck, as she's back to the point of trying to find another key that satisfies the two criteria. So no matter how much computing power Mallory has (short of being able break the hashing algorithm), Mallory only ever gets one guess to successfully attack the secret exchange, and that guess has a 1 in 2n chance of being correct. So by sending a hash first, the authentication string can be much shorter while still giving good security (hence the name Short Authentication String).

Mallory tries to attack an exchange that uses a hash commitment

So given that Mallory has a 1 in 2n chance of successfully attacking a key exchange, how good is that? As noted earlier, by verifying 7 emoji, we're verifying 42 bits, so Mallory has a 1 in 242=4,398,046,511,104 (4 trillion) chance of success, which is quite low for a single attempted attack. What if Mallory attempts multiple attacks at the same time? For example, if I did a key verification with every single person in the world (currently estimated by one site as 7.7 billion people), then how many attacks might be successful? Since each of the attacks is independent, this is known in probability as a Binomial distribution, which due to the numbers involved, is approximated using a Poisson distribution with rate μ=7,700,000,0004,398,046,411,104≈0.00175. For a Poisson distribution with rate μ, the probability of no successes is equal to e, so in this case, the probability that Mallory will have no successes is approximately e-0.00175≈0.998. In other words, if I verify keys with every person on earth and Mallory tries to attack every one of the verifications, there is a 99.8% chance that none of her attempts will be successful.

Now, this is not necessarily the only way that Mallory can attack the key verification process, but this is the most interesting part of the verification method. We have included measures to protect against other possible attacks and we believe that it is secure. We encourage people to review the details of the verification process, and let us know of any potential issues. While the verification process has not been audited yet, we do plan on having it audited in the future, along with other parts of Matrix's end-to-end encryption.

0 Comments
January 4, 2019

Happy Birthday, XMPP!

14:29 -0500

Today is the 20th anniversary of XMPP, also known as Jabber. I don't remember much about how I first heard about Jabber, but it was likely through the Slashdot post. I've been running my own personal XMPP server for a while (I believe I started with ejabberd, though now it runs prosody), and I've written some XMPP-related software.

Nowadays, I work full-time on Matrix, which you could say is a competitor to XMPP. However, I think that both projects would benefit from co-operation, and I think that a little friendly competition is helpful. At the end of the day though, I'm hoping that an open, decentralized, secure communications protocol will become commonplace, whether it be XMPP or Matrix, rather than having the majority of people on multiple proprietary walled gardens. However, with XMPP and Matrix both having features for interoperability with other networks (through transports in XMPP, and through Application Services in Matrix), I think that it's likely that we'll end up with XMPP and Matrix co-existing in a federation.

So congratulations to the XMPP community on the past 20 years, and I hope that the XMPP and Matrix communities can work together to make our shared dream a reality.

0 Comments
May 7, 2018

A New Vector for my Career

10:19 -0400

In three weeks, I will be joining the team at New Vector, working with Matrix, an open communications protocol. It's exciting to be working full time on Free/Open Source software again (I used to work for a Moodle partner). Matrix itself is pretty exciting, with features such as federation (the ability to host your own server and communicate with anyone else using Matrix), bridging together different communication networks, and end-to-end encryption.

My tasks at New Vector will be quite varied. At some point I will be working on bridges, but to start with, I'll probably be helping out with some of the more pressing tasks such as spec wrangling (both documenting missing parts of the spec, and working with the community on spec improvements), doing some work on Dendrite, and helping out with some of the outstanding end-to-end encryption UX work.

I've been doing some Matrix-related things in my spare time, and I've been enjoying it, both working with the technology and interacting with the community. But my free time has been quite limited, so I'm really looking forward to being able to work on Matrix full-time starting in a few weeks.

0 Comments
September 10, 2017

An introduction to end-to-end encryption in Matrix and Riot

21:10 -0400

Disclaimer

End-to-end encryption in Matrix and in Riot are in Beta, and may be subject to change.

I have made every effort to ensure the accuracy of the information in this post, but this should not be viewed as an official guide to end-to-end encryption in Matrix or Riot.

Introduction

End-to-end encryption is one of the main features of the Matrix communications protocol and of Riot, a glossy client for Matrix. This post provides a high-level overview of what end-to-end encryption is, how it works in Matrix and Riot, and how to use it. It is intended to be understandable to people who are starting with little to no knowledge of encryption, while still being as accurate as possible, and the goal is to help people get a better understanding of end-to-end encryption in Matrix so that they can use it more securely and effectively.

What is end-to-end encryption?

Encryption is a way of ensuring that unauthorized people cannot view information that is not intended for them. Encryption takes the information and, using an encryption key, scrambles the information in a such a way that it cannot be read without the corresponding decryption key. (In some encryption systems, the encryption and decryption keys are the same, whereas in others, they are different.)

In some communication systems that involve a server, the connection between each user and the server is encrypted so that anyone who taps into that connection cannot read any messages. By default, all communication in Matrix is encrypted in this way. However, this still allows messages to be read by server administrators, or anyone who manages to gain access to the server.

End-to-end encryption (sometimes abbreviated as e2e encryption, or simply e2e or e2ee) means that messages are encrypted by the sender in such a way that only the people you are communicating with can read it — none of the servers in between can read the message.

Why do I need end-to-end encryption?

Whether it's our credit card or banking details, health records, corporate strategy, or even plans for a surprise party, we all have things that we would prefer not to be made public. End-to-end encryption helps maintain your privacy.

Using end-to-end encryption even for messages that don't need to be secret also helps increase the security of messages that do need to be secret, as it prevents someone from determining which messages have sensitive information and which ones don't.

Are all conversations in Matrix end-to-end encrypted?

End-to-end encryption can be enabled on each room individually. While encryption is still in beta, all rooms are unencrypted by default. Once encryption is out of beta, then private rooms will be encrypted by default.

If you have sufficient privileges (normally moderator or admin permissions) in a room, you can go to the room settings and enable encryption. Note that once encryption is enabled in a room, it cannot be disabled again.

Riot indicates encrypted rooms with a locked icon next to the message input box, and unencrypted rooms with an unlocked icon.

Why won't all rooms be encrypted?

There are several reasons why some rooms will not be encrypted even after encryption is out of beta. In brief, some of the reasons are that encryption interferes with certain types of integrations (including the bots and bridges hosted by matrix.org), encryption prevents people from reading messages sent before they joined the room (which is useful for some rooms such as rooms used as support forums), encryption can slow down sending messages (which should not be noticeable in small rooms, but could be quite significant in large rooms), and encryption is of questionable value in a room that anyone can join and read.

What's the deal with all these devices?

Matrix encrypts messages to devices rather than to users. This allows for greater flexibility and privacy. For example, if your phone gets stolen, then you can tell your contacts to blacklist your phone, and whoever has your phone will not be able to decrypt any future conversations, without affecting any of your other devices.

Why does Riot complain about "unknown devices" when I send a message in an encrypted chat?

When you try to communicate with someone, Riot will fetch the list of that person's devices from the server, including an encryption key for each device that can be used to encrypt messages so that they can be read on that device. However, Riot has no way of determining whether that the key is legitimate or if it was planted or altered by someone trying to snoop in on your conversations, so it warns you when it encounters a device that it hasn't seen before.

Riot allows you the option to send messages even to devices that you haven't verified, or to verify the key to tell Riot that it is trusted, or to blacklist the device to tell Riot that it should never encrypt messages to that device.

How do I verify devices?

Note that the current device verification process is only temporary and in the future will be replaced by something that's easier to use.

In order to verify someone's device, you need to have some reasonably secure way to communicate with them. It doesn't have to be secret (if someone listens in on the key verification process, it won't make it any less secure), but it has to be something that won't allow someone else to be able to impersonate you or the device's owner. For example, if you know the device owner's voice, you can phone them, or even start a video call with them in Riot. You can also verify someone's devices if you meet them in person.

When you're ready to verify someone's devices, you can click on their avatar in any conversation that you have with them, and Riot will show you a list of their devices. Find the device that you want to verify, and click the "Verify" button under it. This will show the device's name, ID and key.

The other person will then have to go to their user settings on the device that you want to verify, and find the device key there. You can then compare the keys, and if they match, then you can click the button saying so, and their device is now verified.

Repeat this for all of their devices that you want to verify.

This may seem like a lot of work, and it is, but there are plans to improve this in the future, before end-to-end encryption leaves Beta. For example, in the future your devices may be able to vouch for each other so that others will only have to verify one of your devices.

How does encryption work in Matrix?

Conceptually, when you first send a message in an encrypted room, your Riot client generates a random key to encrypt your message, sends the encrypted message to the server, and then sends the decryption key to all the devices in the room that should be allowed to decrypt the message. Of course, the decryption key is sent encrypted (based on¹ the device's unique key, which you verified above) so that it cannot be intercepted. The recipient then fetches the message decryption key and the encrypted message and decrypts the message.

In order to avoid having to re-send decryption keys to every device for every message you send, Matrix's encryption system includes a method for generating a new key based on an old key. So for the next message you send, your Riot client will use that method on your previous encryption key to generate a new key, and the recipients will use the same method and generate the same key, so that when you send a message encrypted using the new key, the recipients can decrypt the message without any extra key exchange. The new key will only need to be sent to any new devices that showed up in between when the first message was sent and when the second message was sent.

Riot will occasionally start from scratch, generating a new random key and sending it to all the devices in the room. This happens, for example, whenever someone leaves a room, after you have sent a certain number of messages, or after a certain amount of time.

As a result of how encryption is done in Matrix, there are several encryption and decryption keys being used. The main ones that you may need to be aware of are the device keys and the message decryption keys. The message decryption keys allow you to decrypt encrypted messages, and device keys allow you to send the decryption keys securely to other devices. Device keys are unique to each device and cannot be copied from one device to another, whereas decryption keys may be sent from one device to another, or exported from one device and imported to another, in order to allow you to read older messages.

¹ The decryption key is not encrypted directly with the device's key, but uses a more complicated method to improve security.

Help! I can't read some encrypted messages!

There are a few main possible reasons for not being able to decrypt a message.

The first possible reason is that you were not a member of the chat when the message was sent. In this case, it is by design that you cannot decrypt the message; decryption keys for messages are only sent to the users that are in the room when the message was sent.

Another possible reason is that your device was not registered at the time the message was sent. When a message is sent, the sender only sends the decryption key to devices that it knows about; when you log into a new device, that device has not yet received the decryption key for the message, and so cannot decrypt the message. (Note that when you log out and log in again, your new session is considered a new device from Riot's perspective.) There are two ways around this. One way is to export the decryption keys from another device that is able to decrypt the message, and import the keys into the new device. Another way is to verify your new device with another device: When Riot encounters a message that it cannot decrypt, it will ask your other devices for the decryption keys for that message. If you have verified that device from your other devices, then they will send the decryption key to your new device. Recent versions of Riot may automatically prompt you to verify new devices.

The final reason that you might not be able to decrypt a message is that you have encountered a bug. If you are interested in the technical details, you can see the tracking issue for encryption bugs, but the short story is that developers are aware of most (if not all) of the bugs and are working on fixing them. Some bugs can be worked around by the sender clearing Riot's cache and reloading (in their user settings), or by leaving a room and rejoining. Other bugs can only by worked around by logging out and logging back in. However, note that this will create a new device that will need to be re-verified by others, and you will probably want to export your decryption keys before logging out and import them after you log back in so that you can read old messages.

When will encryption be out of beta?

Before encryption is out of beta, the developers need to fix some of the remaining bugs that prevent people from decrypting messages that they should be able to decrypt, and to make the device verification process more usable. It is difficult to estimate when this work will be completed as the developers are working on other issues as well.

0 Comments
July 14, 2017

Matrix community roundup

08:34 -0400

Last week, the Matrix team put out a call to arms for the community to support Matrix financially, and the community response has been great. Response in online forums such as reddit and Hacker News has been extremely positive, and in the first 24 hours, the community pledged well over $1000/month through Patreon and Liberapay. Although it has slowed down since then, it is now over $1700/month, getting close to the first goal of supporting one developer working half time on Matrix.

While last week was the first time that the Matrix community was able to contribute hard dollars towards the development of Matrix, the community has been supporting Matrix in other ways for quite a while. Since I started following Matrix a little less than a year ago, I've seen that the Matrix community has been quite active testing, filing and triaging bugs, contributing code to core projects, writing bots and bridges, providing support in the Matrix rooms, and more.

Here's a quick roundup of some of the new things that the community has come up with since the Matrix Holiday Special. I have undoubtedly missed some projects, so apologies in advance for all those that I've missed.

Voyager

TravisR has been experimenting a lot with Matrix, and one of the unique things he has come up with is a bot that maps out how Matrix rooms are related to each other by noting when one room is mentioned in another room. The results are then mapped in a massive graph.

Linux distributions

Gentoo: PureTryOut has created an overlay for Gentoo for installing some Matrix-related software.

Debian: Synapse and the Matrix plugin for Pidgin have been packaged for Debian and are included in Debian unstable, and other Matrix-related software has been packaged and submitted for inclusion. Myself and others have also been working on forming a Matrix packaging team.

Integrations

One of Matrix's main features is the ability to bridge with different networks, and while the core Matrix team has had their hands full maintaining the IRC, Gitter, and Slack bridges, the community has been writing bridges to other networks.

Puppeting bridges: The Matrix Hacks group has added quite a few new bridges this year so far. Looking at their GitHub account, they have bridges for Hangouts, Slack, Skype, Signal, and GroupMe, in addition to the previously-announced iMessage bridge. Discussion and support for these bridges take place in #matrix-puppet-bridge:matrix.org.

email: Two email integrations have been written, which work in different ways. Max's bridge allows email users to participate in Matrix rooms while TravisR's bot sends messages to rooms when it receives an email.

Discord: Half-Shot has also written a Discord bridge.

Clients

Riot: While the core Riot team was busy working on creating an improved experience for new users, the community implemented some of Riot Web/Desktop's most requested features, resulting in the release of Riot Web/Desktop 0.10 in which all of the major new features were initiated by the community. I think that's an achievement that the community can be very proud of, as well as the core Riot team, for being able to foster such an active community.

Nheko: As good as Riot is, it isn't for everyone. Development on other clients such as Quaternion has continued, but a new one, Nheko, has been started recently which already seems quite promising.

Matrix Recorder: Although Matrix keeps all history on the server, some people want to keep their own copy of history. Alex created Matrix Recorder, which saves history to a local SQLite database. Matrix Recorder even supports saving history from end-to-end encrypted rooms.

e2e crypto: While end-to-end cryptography is still in beta in Riot, some brave souls have been experimenting with it in other clients. In addition to Matrix Recorder's support of e2e rooms as mentioned above, penguin42 has done work on adding e2e to the Matrix Pidgin plugin and davidar has added e2e support to his Hubot adapter.

SHA2017 badge: One of the most intriguing projects is the badge for the SHA2017 camp, which reportedly contains a Matrix client. I don't know what they're using Matrix for, so I hope they do a write up at some point.

Documentation/Talks

The Matrix community has been busy writing documentation and blog posts, and doing talks about Matrix. Coffee has been collecting helpful information about Matrix into a machine-readable knowledge base. Some guides for Riot have been written, including usage basics by muppeth, maxigaz's guide, and an introduction from an IRC perspective. CryptoAUSTRALIA recently had a workshop for setting up Synapse and Riot, and published a tutorial online. And PureTryOut did a Matrix talk at the Dutch Linux User's Group a few months ago. There have certainly been other people from the community doing talks about Matrix that I am not aware of.

Server list

A federated communications protocol is less valuable if users can't find servers to join. Since there is no official list yet, Alex set up a list of Matrix servers. Though to call it just a list of servers is an understatement. It includes statistics on each server such as uptime, response times from various locations, and SSL test scores, so that users can make a more informed choice of servers. If you are running a Matrix server, whether public or private, please consider submitting your server to the list to improve visibility for your server and to strengthen the federation.

GSoC

Matrix was again accepted as a mentoring organization for Google Summer of Code and has three students. Two of them are working on iOS-related projects, and since I don't have an iOS device, I haven't been following their progress. However, Michael (a.k.a t3chguy), in addition to improving Riot, has been working on creating a search engine-friendly view of public rooms, which will be helpful for Matrix rooms that are used as support forums.


Matrix would not be what it is today without the support of the community, and I'm looking forward to seeing what the community will develop in the future. Last week, the community was invited to contribute financially towards Matrix's development. But for those who are unable to contribute in this way but still want to support Matrix, or for those who have pledged money but still want to do more, hopefully this list gives some ideas for how you can help out, either by supporting an existing project or starting your own.

Addendum (July 19, 2017)

Some projects that I missed:

Max has written an alternative Identity Server implementation called mxisd. Identity Servers haven't been getting as much attention as homeservers, application services, or clients, so it's great that someone has been working on an alternative implementation.

TravisR has also been working on an Dimension, alternative implementation of Riot's integration manager.

0 Comments
April 1, 2017

An alternate transport for the Matrix Client-Server API

00:00 -0400

Matrix is an open communications protocol that has many great features. However, one flaw that it has is that the baseline specification is based on long-polling HTTP requests, which is not very efficient. In order to address this deficiency, I've created a spec that presents an alternative transport for the Matrix Client-Server API that uses a protocol that was designed for real-time communications instead of using HTTP.

0 Comments
March 13, 2017

The latest additions to my init.el

11:00 -0400

Inspired by xkcd (but using Alt-mousewheel):

(global-set-key (kbd "<M-mouse-5>") 'undo)
(global-set-key (kbd "<M-mouse-4>") 'redo)

And, since I sometimes need to paste from an HTTP request into a buffer:

(defun insert-from-url (url)
  (interactive "MURL: ")
  (let ((url-request-method "GET")
        (dest (current-buffer))
        (src (url-retrieve-synchronously url)))
    (set-buffer src)
    (goto-char (point-min))
    (search-forward "\n\n")
    (set-buffer dest)
    (insert-buffer-substring src (match-end 0))))
0 Comments
January 25, 2017

On transparency

21:01 -0500

I've written briefly before about the value of companies being open and transparent. Back then, I wrote that the way that companies react when things go wrong is a good way to differentiate between them. No matter what company you deal with, things will go wrong at one point or another. Some companies try to avoid responsibility, or only tell you that something has happened if you ask them. Others companies are much more open about what happened.

Matrix.org (and the associated Riot.im) is an example of a team that falls into the latter category. And last night's incident is a good example. Their post-mortem blog post is a great example for others to follow. It gives a detailed timeline of what happened and why the outage occurred. And it finishes off with steps that they will take to prevent future incidents.

Kudos to the Matrix.org team for their transparency.

0 Comments
December 1, 2016

Let's Encrypt for Kubernetes

21:08 -0500

A while ago, I blogged about automatic Let's Encrypt certificate renewal with nginx. Since then, I've also set up renewal in our Kubernetes cluster.

Like with nginx, I'm using acme-tiny to do the renewals. For Kubernetes, I created a Docker image. It reads the Let's Encrypt secret key from /etc/acme-tiny/secrets/account.key, and CSR files from /etc/acme-tiny/csrs/{name}.csr. In Kubernetes, these can be set up by mounting a Secrets store and a ConfigMap, respectively. It also reads the current certificates from /etc/acme-tiny/certs/{name}, which should also be set up by mounting a ConfigMap (called certificates), since that is where the container will put the new certificates.

Starting an acme-tiny pod will start an nginx server to store the .well-known directory for the Acme challenge. Running /opt/acme-tiny-utils/renew in the pod will renew the certificate if it will expire within 20 days (running it with the -f option will disable the check). Of course, we want the renewal to be automated, so we want to set up a sort of cron task. Kubernetes has cron jobs since 1.4, but at the time I was setting this up, we were still on 1.3. Kubernetes also does cron jobs by creating a new pod, whereas the way I want this to work is to run a program in an existing pod (though it could be set up to work the other way too). So I have another cron Docker image, which I have set up to run

kubectl exec `kubectl get pods --namespace=lb -l role=acme -o name | cut -d / -f 2` --namespace=lb ./renew sbscalculus.com

every day. That command finds the acme-tiny pod and runs the renew command, telling it to renew the sbscalculus.com certificate.

Now in order for the Acme challenge to work, HTTP requests to /.well-known/acme-challenge/ get redirected to acme-tiny rather than to the regular pods serving those services. Our services are behind our HAProxy image. So I have a 0acmetiny entry (the 0 causes it to be sorted before all other entries) in the services ConfigMap for HAProxy that reads:

    {
      "namespace": "lb",
      "selector": {
        "role": "acme"
      },
      "hostnames": ["^.*"],
      "path": "^/\\.well-known/acme-challenge/.*$",
      "ports": [80]
    }

This causes HAProxy to all the Acme challeges to the acme-tiny pod, while leaving all the other requests alone.

And that's how we have our certificates automatically renewed from Let's Encrypt.

0 Comments
September 6, 2016

Buildbot latent build slaves

12:28 -0400

I've blogged before about using Buildbot to build our application server. One problem with it is that the build (and testing) process can be memory intensive, which can sometimes exceed the memory that we have available in our Kubernetes cluster. I could add another worker node, but that would be a waste, since we do builds infrequently.

Fortunately, the Buildbot developers have already built a solution to this: latent buildslaves. A latent buildslave is a virtual server that is provisioned on-demand. That means that when a build isn't active, then we don't have to pay for an extra server to be active; we only have to pay for the compute time that we actually need for builds (plus a bit of storage space).

I chose to use AWS EC2 as the basis of our buildslave. Buildbot also supports OpenStack, so I could have just used DreamCompute, which we already use for our Kubernetes cluster, but with AWS EC2, we can take advantage of spot instances and save even more money if we needed to. In any event, the setup would have been pretty much the same.

Setting up a latent buildslave on AWS is pretty straightforward. First, create an EC2 instance in order to build a base image for the buildslave. I started with a Debian image. Then, install any necessary software for the buildslave. For us, that included the buildslave software itself (Debian package buildbot-slave), git, tinc, npm, and Docker. Most of our build process happens inside of Docker containers, so we don't need anything else. We use tinc to build a virtual network with our Kubernetes cluster, so that we can push Docker images to our own private Docker repository.

After installing the necessary software, we need to configure it. It's configured just like a normal buildslave would be configured: I configured tinc, added an ssh key so that it could check out our source code, configured Docker so that it could push to our repository, and of course configured the Buildbot slave itself. Once it's configured, I cleaned up the image a bit (truncated logs, cleared bash history, etc.), and then took a snapshot in the AWS control panel, giving it a name so that it would show up as an AMI.

Finally, I added the latent buildslave in our Buildbot master configuration, giving it the name of the AMI that was created. Once set up, it ran pretty much as expected. I pushed out a change, Buildbot master created a new EC2 instance, built our application server, pushed and deployed it to our Kubernetes cluster, and after a short delay (to make sure there are no other builds), deleted the EC2 instance. In all, the EC2 instance ran for about 20 minutes. Timings will vary, of course, but it will run for less than an hour. If we were paying full price for a t2.micro instance in us-east-1, each build would cost just over 1 cent. We also need to add in the storage cost for the AMI which, given that I started with an 8GB image, will cost us at most 80 cents per month (since EBS snapshots don't store empty blocks, it should be less than that). We probably average about two builds a month, giving us an average monthly cost of at most 83 cents, which is not too bad.

0 Comments