July 14, 2017

Matrix community roundup

08:34 -0400

Last week, the Matrix team put out a call to arms for the community to support Matrix financially, and the community response has been great. Response in online forums such as reddit and Hacker News has been extremely positive, and in the first 24 hours, the community pledged well over $1000/month through Patreon and Liberapay. Although it has slowed down since then, it is now over $1700/month, getting close to the first goal of supporting one developer working half time on Matrix.

While last week was the first time that the Matrix community was able to contribute hard dollars towards the development of Matrix, the community has been supporting Matrix in other ways for quite a while. Since I started following Matrix a little less than a year ago, I've seen that the Matrix community has been quite active testing, filing and triaging bugs, contributing code to core projects, writing bots and bridges, providing support in the Matrix rooms, and more.

Here's a quick roundup of some of the new things that the community has come up with since the Matrix Holiday Special. I have undoubtedly missed some projects, so apologies in advance for all those that I've missed.


TravisR has been experimenting a lot with Matrix, and one of the unique things he has come up with is a bot that maps out how Matrix rooms are related to each other by noting when one room is mentioned in another room. The results are then mapped in a massive graph.

Linux distributions

Gentoo: PureTryOut has created an overlay for Gentoo for installing some Matrix-related software.

Debian: Synapse and the Matrix plugin for Pidgin have been packaged for Debian and are included in Debian unstable, and other Matrix-related software has been packaged and submitted for inclusion. Myself and others have also been working on forming a Matrix packaging team.


One of Matrix's main features is the ability to bridge with different networks, and while the core Matrix team has had their hands full maintaining the IRC, Gitter, and Slack bridges, the community has been writing bridges to other networks.

Puppeting bridges: The Matrix Hacks group has added quite a few new bridges this year so far. Looking at their GitHub account, they have bridges for Hangouts, Slack, Skype, Signal, and GroupMe, in addition to the previously-announced iMessage bridge. Discussion and support for these bridges take place in #matrix-puppet-bridge:matrix.org.

email: Two email integrations have been written, which work in different ways. Max's bridge allows email users to participate in Matrix rooms while TravisR's bot sends messages to rooms when it receives an email.

Discord: Half-Shot has also written a Discord bridge.


Riot: While the core Riot team was busy working on creating an improved experience for new users, the community implemented some of Riot Web/Desktop's most requested features, resulting in the release of Riot Web/Desktop 0.10 in which all of the major new features were initiated by the community. I think that's an achievement that the community can be very proud of, as well as the core Riot team, for being able to foster such an active community.

Nheko: As good as Riot is, it isn't for everyone. Development on other clients such as Quaternion has continued, but a new one, Nheko, has been started recently which already seems quite promising.

Matrix Recorder: Although Matrix keeps all history on the server, some people want to keep their own copy of history. Alex created Matrix Recorder, which saves history to a local SQLite database. Matrix Recorder even supports saving history from end-to-end encrypted rooms.

e2e crypto: While end-to-end cryptography is still in beta in Riot, some brave souls have been experimenting with it in other clients. In addition to Matrix Recorder's support of e2e rooms as mentioned above, penguin42 has done work on adding e2e to the Matrix Pidgin plugin and davidar has added e2e support to his Hubot adapter.

SHA2017 badge: One of the most intriguing projects is the badge for the SHA2017 camp, which reportedly contains a Matrix client. I don't know what they're using Matrix for, so I hope they do a write up at some point.


The Matrix community has been busy writing documentation and blog posts, and doing talks about Matrix. Coffee has been collecting helpful information about Matrix into a machine-readable knowledge base. Some guides for Riot have been written, including usage basics by muppeth, maxigaz's guide, and an introduction from an IRC perspective. CryptoAUSTRALIA recently had a workshop for setting up Synapse and Riot, and published a tutorial online. And PureTryOut did a Matrix talk at the Dutch Linux User's Group a few months ago. There have certainly been other people from the community doing talks about Matrix that I am not aware of.

Server list

A federated communications protocol is less valuable if users can't find servers to join. Since there is no official list yet, Alex set up a list of Matrix servers. Though to call it just a list of servers is an understatement. It includes statistics on each server such as uptime, response times from various locations, and SSL test scores, so that users can make a more informed choice of servers. If you are running a Matrix server, whether public or private, please consider submitting your server to the list to improve visibility for your server and to strengthen the federation.


Matrix was again accepted as a mentoring organization for Google Summer of Code and has three students. Two of them are working on iOS-related projects, and since I don't have an iOS device, I haven't been following their progress. However, Michael (a.k.a t3chguy), in addition to improving Riot, has been working on creating a search engine-friendly view of public rooms, which will be helpful for Matrix rooms that are used as support forums.

Matrix would not be what it is today without the support of the community, and I'm looking forward to seeing what the community will develop in the future. Last week, the community was invited to contribute financially towards Matrix's development. But for those who are unable to contribute in this way but still want to support Matrix, or for those who have pledged money but still want to do more, hopefully this list gives some ideas for how you can help out, either by supporting an existing project or starting your own.

Addendum (July 19, 2017)

Some projects that I missed:

Max has written an alternative Identity Server implementation called mxisd. Identity Servers haven't been getting as much attention as homeservers, application services, or clients, so it's great that someone has been working on an alternative implementation.

TravisR has also been working on an Dimension, alternative implementation of Riot's integration manager.

April 1, 2017

An alternate transport for the Matrix Client-Server API

00:00 -0400

Matrix is an open communications protocol that has many great features. However, one flaw that it has is that the baseline specification is based on long-polling HTTP requests, which is not very efficient. In order to address this deficiency, I've created a spec that presents an alternative transport for the Matrix Client-Server API that uses a protocol that was designed for real-time communications instead of using HTTP.

March 13, 2017

The latest additions to my init.el

11:00 -0400

Inspired by xkcd (but using Alt-mousewheel):

(global-set-key (kbd "<M-mouse-5>") 'undo)
(global-set-key (kbd "<M-mouse-4>") 'redo)

And, since I sometimes need to paste from an HTTP request into a buffer:

(defun insert-from-url (url)
  (interactive "MURL: ")
  (let ((url-request-method "GET")
        (dest (current-buffer))
        (src (url-retrieve-synchronously url)))
    (set-buffer src)
    (goto-char (point-min))
    (search-forward "\n\n")
    (set-buffer dest)
    (insert-buffer-substring src (match-end 0))))
January 25, 2017

On transparency

21:01 -0500

I've written briefly before about the value of companies being open and transparent. Back then, I wrote that the way that companies react when things go wrong is a good way to differentiate between them. No matter what company you deal with, things will go wrong at one point or another. Some companies try to avoid responsibility, or only tell you that something has happened if you ask them. Others companies are much more open about what happened.

Matrix.org (and the associated Riot.im) is an example of a team that falls into the latter category. And last night's incident is a good example. Their post-mortem blog post is a great example for others to follow. It gives a detailed timeline of what happened and why the outage occurred. And it finishes off with steps that they will take to prevent future incidents.

Kudos to the Matrix.org team for their transparency.

December 1, 2016

Let's Encrypt for Kubernetes

21:08 -0500

A while ago, I blogged about automatic Let's Encrypt certificate renewal with nginx. Since then, I've also set up renewal in our Kubernetes cluster.

Like with nginx, I'm using acme-tiny to do the renewals. For Kubernetes, I created a Docker image. It reads the Let's Encrypt secret key from /etc/acme-tiny/secrets/account.key, and CSR files from /etc/acme-tiny/csrs/{name}.csr. In Kubernetes, these can be set up by mounting a Secrets store and a ConfigMap, respectively. It also reads the current certificates from /etc/acme-tiny/certs/{name}, which should also be set up by mounting a ConfigMap (called certificates), since that is where the container will put the new certificates.

Starting an acme-tiny pod will start an nginx server to store the .well-known directory for the Acme challenge. Running /opt/acme-tiny-utils/renew in the pod will renew the certificate if it will expire within 20 days (running it with the -f option will disable the check). Of course, we want the renewal to be automated, so we want to set up a sort of cron task. Kubernetes has cron jobs since 1.4, but at the time I was setting this up, we were still on 1.3. Kubernetes also does cron jobs by creating a new pod, whereas the way I want this to work is to run a program in an existing pod (though it could be set up to work the other way too). So I have another cron Docker image, which I have set up to run

kubectl exec `kubectl get pods --namespace=lb -l role=acme -o name | cut -d / -f 2` --namespace=lb ./renew sbscalculus.com

every day. That command finds the acme-tiny pod and runs the renew command, telling it to renew the sbscalculus.com certificate.

Now in order for the Acme challenge to work, HTTP requests to /.well-known/acme-challenge/ get redirected to acme-tiny rather than to the regular pods serving those services. Our services are behind our HAProxy image. So I have a 0acmetiny entry (the 0 causes it to be sorted before all other entries) in the services ConfigMap for HAProxy that reads:

      "namespace": "lb",
      "selector": {
        "role": "acme"
      "hostnames": ["^.*"],
      "path": "^/\\.well-known/acme-challenge/.*$",
      "ports": [80]

This causes HAProxy to all the Acme challeges to the acme-tiny pod, while leaving all the other requests alone.

And that's how we have our certificates automatically renewed from Let's Encrypt.

September 6, 2016

Buildbot latent build slaves

12:28 -0400

I've blogged before about using Buildbot to build our application server. One problem with it is that the build (and testing) process can be memory intensive, which can sometimes exceed the memory that we have available in our Kubernetes cluster. I could add another worker node, but that would be a waste, since we do builds infrequently.

Fortunately, the Buildbot developers have already built a solution to this: latent buildslaves. A latent buildslave is a virtual server that is provisioned on-demand. That means that when a build isn't active, then we don't have to pay for an extra server to be active; we only have to pay for the compute time that we actually need for builds (plus a bit of storage space).

I chose to use AWS EC2 as the basis of our buildslave. Buildbot also supports OpenStack, so I could have just used DreamCompute, which we already use for our Kubernetes cluster, but with AWS EC2, we can take advantage of spot instances and save even more money if we needed to. In any event, the setup would have been pretty much the same.

Setting up a latent buildslave on AWS is pretty straightforward. First, create an EC2 instance in order to build a base image for the buildslave. I started with a Debian image. Then, install any necessary software for the buildslave. For us, that included the buildslave software itself (Debian package buildbot-slave), git, tinc, npm, and Docker. Most of our build process happens inside of Docker containers, so we don't need anything else. We use tinc to build a virtual network with our Kubernetes cluster, so that we can push Docker images to our own private Docker repository.

After installing the necessary software, we need to configure it. It's configured just like a normal buildslave would be configured: I configured tinc, added an ssh key so that it could check out our source code, configured Docker so that it could push to our repository, and of course configured the Buildbot slave itself. Once it's configured, I cleaned up the image a bit (truncated logs, cleared bash history, etc.), and then took a snapshot in the AWS control panel, giving it a name so that it would show up as an AMI.

Finally, I added the latent buildslave in our Buildbot master configuration, giving it the name of the AMI that was created. Once set up, it ran pretty much as expected. I pushed out a change, Buildbot master created a new EC2 instance, built our application server, pushed and deployed it to our Kubernetes cluster, and after a short delay (to make sure there are no other builds), deleted the EC2 instance. In all, the EC2 instance ran for about 20 minutes. Timings will vary, of course, but it will run for less than an hour. If we were paying full price for a t2.micro instance in us-east-1, each build would cost just over 1 cent. We also need to add in the storage cost for the AMI which, given that I started with an 8GB image, will cost us at most 80 cents per month (since EBS snapshots don't store empty blocks, it should be less than that). We probably average about two builds a month, giving us an average monthly cost of at most 83 cents, which is not too bad.

June 22, 2016

Load balancing Kubernetes pods

10:10 -0400

At work, we recently switched from Tutum (now Docker Cloud) to Kubernetes. Part of that work was building up a load balancer. Kubernetes has built-in load balancing capabilities, but it only works with Google Compute or AWS, which we are not using. It also requires a public IP address for each service, which usually means extra (unnecessary) costs.

Having previously worked with Tutum's HAProxy image, I figured I could do the same thing with Kubernetes. A quick web search didn't turn up any existing project, so I quickly wrote my own. Basically, we have HAProxy handling all incoming HTTP(S) connections and passing them off to different services based on the Host header. There's a watcher that watches Kubernetes such as new/deleted pods for relevant changes and updates the HAProxy configuration so that it always sends requests to the right place. I also improved the setup by adding in a Varnish cache for some of our services. Here's how it all works.

We have two sets of pods: an set of HAProxy pods and a set of Varnish pods. Each pod has a Python process that watches etcd for Kubernetes changes, updates the appropriate (HAProxy or Varnish) configuration, and tells HAProxy/Varnish about the new configuration. Why do we watch etcd instead of using the Kubernetes API directly? Because, as far as I can tell, in the Kubernetes API, you can only watch one type of object (either pods, configmaps, secrets, etc.) for changes, whereas we need to watch multiple types at once, so dealing with the Kubernetes API means that we would need to make multiple simultaneous API requests, which would just make things more complicated.

Unlike Tutum's HAProxy image, which only allows you to change certain settings using environment variables, our entire configuration template is configurable using Jinja2 templates. This gives us a lot more flexibility, including being able to plug in Varnish fairly easily without having to make any code changes to the HAProxy configurator. Also, configuration variables for services are stored in their own ConfigMap, rather than as environment variables in the target pods which allows us to make configuration changes without restarting the pods.

When combining HAProxy and Varnish, one question to ask is how to arrange them: HAProxy in front of Varnish, or Varnish in front of HAProxy? We are using a setup similar to the one recommended in the HAProxy blog. In that setup, HAProxy handles all requests and passes non-cacheable requests directly to the backend servers. Cacheable requests are, of course, passed to Varnish. If Varnish has a cache miss, then it passes the request back to HAProxy, which then hands off the request to the backend server. As the article points out, in the event of a cache miss, there's a lot of requests, but cache misses should be very infrequent since Varnish only sees cacheable content. One main difference between the setup we have and the one in the article is that in the article, HAProxy listens on two IP addresses: one for requests coming from the public, and one for requests coming from Varnish. In our setup, we don't have two IP addresses for HAProxy to use. Instead, Varnish adds a request header that indicates that the request is coming from it, and HAProxy checks for that header.

At first, I set the Python process as the pod's command (the pod's PID 1), but ran into a slight issue. HAProxy reloads its configuration by, well, not reloading its configuration; it starts a new set of processes with the new configuration, which means that we ended up with a lot of zombie processes. To fix this, I could have changed the Python process to reap the zombies, but it was easier to just use Yelp's dumb-init instead.

We have the HAProxy pods managed as a DaemonSet, so it's running on every node, and the pods are set to use host networking for better performance. HAProxy itself is small enough that, at least with our current traffic, it doesn't affect the nodes much, so it isn't a problem for us right now to run it on every node. If we get enough traffic that it does make a difference, we can dedicate a node to it without much problem. One thing about this setup is that, even though it uses Kubernetes, HAProxy and Varnish don't need to be managed by Kubernetes. It just needs to be able to talk to etcd. So if we ever need a dedicated load balancer, we can spin up a node (or nodes) that just runs HAProxy and/or Varnish, say, using a DaemonSet and nodeSelector. Varnish is managed as a normal Kubernetes deployment and uses the normal container networking, so there's a bit of overhead there, but is fine for now. Again, if we have more concerns about performance, we can change our configuration easily enough.

It all seems to be working fairly well so far. There are some configuration tweaks that I'll have to go make, and there's one strange issue where Varnish doesn't like one of our services and just returns an empty response. But other than that, Varnish and HAProxy are just doing what they're supposed to do.

All the code is available on GitHub (HAProxy, Varnish).

June 9, 2016

Kubernetes vs Docker Cloud

09:42 -0400

Note: this is not a comprehensive comparison of Kubernetes and Docker Cloud. It is just based on my own experiences. I am also using Tutum and Docker Cloud more or less interchangeably, since Tutum became Docker Cloud.

At work, we used to use Tutum for orchestrating our Docker containers for our Calculus practice problems site. While it was in beta, Tutum was free, but Tutum has now become Docker cloud and costs about $15 per month per managed node per month, on top of server costs. Although we got three free nodes since we were Tutum beta testers, we still felt the pricing was a bit steep, since the management costs would be more than the hosting costs. Even more so since we would have needed more private Docker repositories than what would have been included.

So I started looking for self-hosted alternatives. The one I settle on was Kubernetes, which originated from Google. Obviously, if you go self-hosted, you need to have enough system administration knowledge to do it, whereas with Docker Cloud, you don't need to know anything about system administration. It's also a bit more time consuming to set up — it took me about a week to set up Kubernetes (though most of that time was scripting the process so that we could do it again more quickly next time), whereas with Tutum, it took less than a day to get up and running.

Kubernetes will require at least one server for itself — if you want to ensure high availability, you'll want to run multiple masters. We're running on top of CoreOS, and a 512MB node seems a bit tight for the master for our setup. A 1GB node was big enough that, although they recommend not to, I allowed the master to schedule running other pods.

Kubernetes seems to have a large-ish overhead on the worker nodes (a.k.a. minions). Running top, the system processes take up at least 200MB, which means that on a 512MB node, you'd only have about 300MB to run your own pods unless you have swap space. I have no idea what the overhead on a Tutum/Docker cloud node was, since I didn't have access to check.

Previously, under Tutum, we were running on 5*512MB nodes, each of which had 512MB swap space. Currently, we're running on 3*1GB worker nodes plus 1*1GB master node (which also serves as a worker), no swap. (We'll probably need to add another worker in the near future (or maybe another combined master/worker) though under Tutum, we would have probably needed another node with the changes that I'm planning anyways.) Since we also moved from DigitalOcean to DreamHost (affiliate link) and their new DreamCompute service (which just came out of Beta as we were looking into self-hosting), our new setup ended up costing $1 less per month.

Under Tutum, the only way to pass in configuration (other than baking it into your Docker image, or unless you run your own configuration server) is through environment variables. With Kubernetes, you have more options, such as ConfigMaps and Secrets. That gives you more flexibility and allows (depending on your setup) on changing configuration on-the-fly. For example, I created an auto-updating HAProxy configuration that allows you to specify a configuration template via a ConfigMap. When you update the ConfigMap, HAProxy gets immediately reconfigured with almost no downtime. This is in contrast to the Tutum equivalent, in which a configuration change (via environment variables) would require a restart and hence more downtime.

The other configuration methods also allows the configuration to be more decoupled. For example, with Tutum's HAProxy, the configuration for a service such as virtual host names are specified using the target container's environment variables, which means that if you want to change the set of virtual hosts or the SSL certificate, you would need to restart your application containers. Since our application server takes a little while to restart, we want to avoid having to do that. On the other hand, if the configuration were set in HAProxy's environment, then it would be lost to other services that might want to use it (such as monitoring software that would might use the HTTP_CHECK variable). With a ConfigMap, however, the configuration does not need to belong to one side or the other; it can stand on its own, and so it doesn't interfere with the application container, and can be accessed by other pods.

Kubernetes can be all configured using YAML (or JSON) files, which means that everything can be version controlled. Under Tutum, things are primarily configured via the web interface, though they do have a command-line tool that you could use as well. However, the command-line tool uses a different syntax for creating versus updating, whereas with Kubernetes, you can just "kubectl apply -f", so even if you use the Tutum CLI and keep a script under version control for creating your services, it's easier to forget to change your script after you've changed a service.

There are a few things that Tutum does that Kubernetes doesn't do. For example, Tutum has built-in node management (if you use AWS, DigitalOcean, or one of the other providers that it is made to work with), whereas with Kubernetes, you're responsible for setting up your own nodes. Though there are apparently tools built on top of Kubernetes that do similar things, I never really looked into them, since we currently don't need to bring up/take down nodes very frequently. Tutum also has more deployment strategies (such as "emptiest node" and "high availability"), which was not that important for us, but might be more important for others.

Based on my experience so far, Kubernetes seems to be a better fit for us. For people who are unable/unwilling to administer their own servers, Docker Cloud would definitely be the better choice, and starting with Tutum definitely gave me time to look around in the Docker ecosystem before diving into a self-hosted solution.

April 29, 2016

Let's encrypt errata

10:06 -0400

Back in February, I posted about Automatic Let's Encrypt certificates on nginx. One of the scripts had a problem in that it downloaded the Let's Encrypt X1 intermediate certificate. Let's Encrypt recently switched to using their X3 intermidiate, which means that Firefox was unable to reach sites using the generated certificates, and Chrome/IE/Safari needed to make an extra download to verify the certificate.

Of course, instead of just changing the script to download the X3 certificate, it's best to automatically download the right certificate. So I whipped up a quick Python script, cert-chain-resolver-py (inspired by the Go version) that checks a certificate and downloads the other certificates in the chain.

I've updated my original blog post. The changed script is /usr/local/sbin/letsencrypt-renew, and of course you'll need to install cert-chain-resolver-py (the script expects it to be in /opt/cert-chain-resolver-py).

April 8, 2016

Antagonistic Co-operation

08:56 -0400

This article was originally written for our housing co-operative's newsletter. Even though it was written in the context of a housing co-operative, I think the idea is useful in other contexts as well.


The word "antagonistic" and its relatives generally have negative connotations. Nobody likes to be antagonized. In literature, the antagonist in a story works against the protagonist or main character, so we do not like to see the antagonist succeed. However, antagonism can be essential in some cases. Many of our muscles come in what are called "antagonistic pairs," without which you would not be able to move. Muscles can only pull (by contracting) and relax; muscles cannot push. If you only had biceps, you would only be able to bend your arm; you also need your triceps in order to be able to straighten your arm. Your basic movements rely on muscles that oppose each other, yet work together to allow you to walk, lift, or write.

But sometimes our muscles do not work as they should. If you have ever experienced a cramp, you know how painful this can be sometimes. A cramp happens when a muscle suddenly tightens and will not loosen. Many cases of back pain are also due to muscles that fail to relax as they should. Some people require regular massage therapy due to pain caused by tight muscles.

As a co-operative, we should strive to operate like a well functioning body. As members of our co-operative, we all have different opinions and priorities, and we pull our co-operative in different directions. Some people may be more focused on providing activities for our children, and some are more focused on helping our elders adapt to new challenges. Some people prefer to be frugal, while others may wish to spend money to improve the quality of life here. Some people value a strict adherence to our bylaws, while others adopt a more "live and let live" attitude. Each of these views is welcome in our co-operative, and we should celebrate our differences. Indeed, without different opinions pulling us in different directions, our co-operative would be as lifeless as a skeleton with no muscles.

But in order for our co-operative to get anywhere, we must be willing, not just to pull in the direction that we want to go, but also to sometimes let go when others are pulling in a different direction. Sometimes we must allow other members to go ahead with their opinions and priorities without getting in their way.

Unlike our bodies, however, our co-operative does not have a central "brain" coordinating our actions, telling us when to pull and when to let go. Instead we must, as a co-operative, come to an agreement among ourselves. We must communicate with each other, and come to understand the perspectives of other members. Then we can decide when each member should have an opportunity to pull so that we do not prevent our co-op from moving forward by pulling in opposite directions at the same time.

We often see people with opposing viewpoints as adversaries. But while we may be antagonistic, we can still be co-operative.


This article may be copied under the terms of the Creative Commons Attribution-ShareAlike license.