August 28, 2015
15:50 -0400
Hubert Chathi: congratulations to @phoenixframework.org on releasing 1.0. Looking forward to trying out out, someday... # #
0 Comments
August 26, 2015

Limiting concurrency in Node.js with promises

21:41 -0400

The nice thing about Node.js is its asynchronous execution model, which means that it can handle many requests very quickly. The flip side of this is that it can also generate many requests very quickly, which is fine if they can then be handled quickly, and not so good when they can't. For one application that I'm working on, some of the work gets offloaded to an external process; a new process is created for each request. (Unfortunately, that's the architecture that I'm stuck with for now.) And when doing a batch operation, Node.js will happily spawn hundreds of processes at once, without caring that doing so will cause everything on my workstation to slow to a crawl.

Limiting concurrency in Node.js has been written about elsewhere, but I'd like to share my promise-based version of this solution. In particular, this was built for the bluebird flavour of promises.

Suppose that we have a function f that performs some task, and returns a promise that is fulfilled when that task in completed. We want to ensure that we don't have too many instances of f running at the same time.

We need to keep track of how many instances are currently running, we need a queue of instances when we've reached our limit, and of course we need to define what our limit is.

var queue = [];
var numRunning = 0;
var max = 10

Our queue will just contain functions that, when called, will call f with the appropriate arguments, as well as perform the record keeping necessary for calling f. So to process the queue, we just check whether we are below our run limit, check whether the queue is non-empty, and run the function at the front of the queue.

function runnext()
{
    numRunning--;
    if (numRunning <= max && queue.length)
    {
        queue.shift()();
    }
}

Now we create a wrapper function f1 that will limit the concurrency of f. We will call f with the same arguments that f1 is called with. If we have already reached our limit, we queue the request; otherwile, we run f immediately. When we run f, whether it is immediately, or in the future, we must first increment our counter. After f is done, we process the next element in the queue. We must process the queue whether f succeeds or not, and we don't want to change the resolution of f's promise, so we tack a finally onto the promise returned by f.

function f1 ()
{
    var args = Array.prototype.slice.call(arguments);
    return new Promise(function (resolve, reject) {
        function run() {
            numRunning++;
            resolve(f.apply(undefined, args)
                    .finally(runnext));
        }
        if (numRunning > max)
        {
            queue.push(run);
        }
        else
        {
            run();
        }
    });
}

Of course, if you need to do this a lot, you may want to wrap this all up in a higher-order function. For example:

function limit(f, max)
{
    var queue = [];
    var numRunning = 0;

    function runnext()
    {
        numRunning--;
        if (numRunning <= max && queue.length)
        {
            queue.shift()();
        }
    }

    return function ()
    {
        var args = Array.prototype.slice.call(arguments);
        return new Promise(function (resolve, reject) {
            function run() {
                numRunning++;
                resolve(f.apply(undefined, args)
                        .finally(runnext));
            }
            if (numRunning > max)
            {
                queue.push(run);
            }
            else
            {
                run();
            }
        });
    };
}

This would be used as:

f = limit(f, 10);

which would replace f with a new function that is equivalent to f, except that only 10 instances will be running at a time.

0 Comments
August 19, 2015
11:05 -0400
Hubert Chathi: # second tooth came out Sunday
0 Comments
July 6, 2015
12:52 -0400
Hubert Chathi: Congratulations to @moodle.com for providing free # hosting
0 Comments
May 24, 2015

First steps in CI/CD with Buildbot

15:59 -0400

At work, I've been looking into Continuous Integration and Continuous Delivery/Deployment. So far, our procedures have been mostly manual, which means that some things take longer than necessary, and sometimes things get missed. The more that can be automated, the less developer time has to be spent on mundane tasks, and the less brain power needed to remember all the steps.

There are many CI solutions out there, and after investigating a bunch of them, I settled on using Buildbot for a few reasons:

  • it can manage multiple codebases for the same project, unlike many of the simpler CI tools. This is important since the back end for the next iteration of our product is based on plugins that live in individual git repositories.
  • it is lightweight enough to run on our low-powered VPS.
  • it has a flexible configuration language (its configuration file is Python code) and is easily extendable.

Right now, we're in development mode for our product, and I want to make sure that our development test site is always running the latest available code. That means combining plugins together, running unit tests, and if everything checks out, deploying. Eventually, my hope is to be able to tag a branch and have our production site update automatically.

The setup

Our code has one main tree, with plugins each in their own directory within a special plugins directory. The development test site should track the master branch on the main tree and all relevant plugins. For ease of deployment (especially for new development environments), we want to use git submodules to pull in all the relevant plugins. However, the master branch will be the basis of all deployments, which may use different plugins, or different versions of plugins, and so should not have any plugins specified in itself. Instead, we have one branch for each deployed version, which includes as submodules the plugins that are used for that build.

The builds

Managing git submodules can be a bit of a pain. Especially since we're not developing on the branch that actually contains the submodules, managing them would require switching branches, pulling the correct versions of the submodules, and pushing.

The first step in automation, then, is to automatically update the deployment branch whenever either a plugin or the main tree are updated. Buildbot has a list of the plugins that are used in a deployment branch, along with the branch that it follows. Each plugin is associated with a repository, and we use the codebase setting in Buildbot to keep the plugins separate. We then have a scheduler listen on the appropriate codebases, and triggers a build whenever any pushes are made. A Buildbot slave then checks out the latest version of the deployment branch, merges in the changes to the main tree and the submodules, and then pushes out a new version of the deployment branch.

Naturally, pushes to plugins and to the main tree are generally grouped. For example, changes in one plugin may require, or enable, changes in other plugins. We don't want a separate commit in our deployment branch for each change in each plugin, so we take advantage of Buildbot's ability to merge changes. We also wait for the git repositories to be stable for two minutes before running the build, to make sure that all the changes are caught. This reduces the number of commits we have in our deployment branch, making things a bit cleaner.

When Buildbot pushes out a new version of the deployment branch, this in turn triggers another build in Buildbot. Buildbot checks out the full sources, including submodules, installs the required node modules, installs configuration files for testing, and then runs the unit tests. If the tests all pass, then this triggers yet another build.

The final build checks out the latest sources into the web application directory for the test site, and then notifies the web server (currently using Passenger) to restart the web application.

Next steps

This setup seems to be working fairly well so far, but the setup isn't complete yet. Being a first attempt, I'm sure there are many improvements that can be made to the setup, both in terms of flexibility and performance. Especially since we only have one site being updated, the configuration works fine for now, but can probably be made more general to make it easier to deploy multiple sites.

One major issue in the current setup, though, is the lack of notifications. Currently, in order to check the state of the build, I need to view Buildbot's web UI, which is inconvenient. Buildbot has email notification built in, but I just haven't had the chance to set it up yet. When I do set it up, I will likely set it to notify on any failure, as well as whenever a deployment is made.

I'd also like to get XMPP notifications, which isn't built into Buildbot, and so is something that I would have to write myself. Buildbot is based on Twisted, which has an XMPP module built in, so it should be doable. I think the XMPP module is a bit outdated, but we don't need any fancy functionality, so hopefully it will work well enough.

I'm looking into using Docker for deployments once we're ready to push this project to production, so I'll need to look into creating build steps for Docker. The VPS that we're currently using for Buildbot is OpenVZ-based, and so does not support Docker, so we'd need to put a Buildbot slave on a Docker-capable host for building and testing the Docker images, or even use a Docker container as a Buildbot slave, which would be even better.

There's probably a lot that can be done to improve the output in the UI too. For example, when the unit tests are run, it only reports whether the tests passed or failed. It should be possible to create a custom build step that will report how many tests failed.

Assessment

Although Buildbot seems like the best fit for our setup, it isn't perfect. The main thing that I'd like is better project support. Buildbot allows you to set projects on change sets, but I'd like to be able to set projects on builds as well, in order to filter by projects in the waterfall view.

All in all, Buildbot seems like a worthwhile tool that is flexible, yet easy enough to configure. It's no-nonsense and just does what it claims to do. The documentation is well done, and for simple projects, you should be able to just dive right in without any issues. For more complex projects, it's helpful to understand what's going on before charging right in. Of course, I just charged right in without understanding certain concepts, so I had to redo some stuff to make it work better, but the fact that I was able to actually get it to work in the first place, even doing it the wrong way, gives some indication to its power.

0 Comments
May 5, 2015
11:52 -0400
Hubert Chathi: Let kids read what they want.
0 Comments
April 30, 2015
14:16 -0400
Hubert Chathi: Never thought I'd see the day when the NDP are ahead in Alberta
Via: reddit
0 Comments
April 26, 2015
10:33 -0400
Hubert Chathi: There's a new sheriff in town. #
0 Comments
April 15, 2015

Switching to nginx

09:17 -0400

I think that I've been running lighttpd for almost as long as I've had a VPS, but I've recently decided to switch to nginx.

The main reason that I've decided to switch is that lighttpd no longer seems to be actively developed. They still do bug fix releases, but aside from that, development seems to have been stalled. They have been working on their 1.5 branch for years, without marking it as stable. In fact, they even started working on a 2.0 branch without first releasing 1.5, which was a warning sign that development was losing focus.

Nginx has some weirdnesses and unexpected design decisions, though. For example

One feature that I will miss from lighttpd is its ability to automatically split SCRIPT_NAME and PATH_INFO based on what files are actually on the filesystem. I depend on that feature in my own CMS, which means I'll have to implement it myself, which is slightly inconvenient, but not too big of a deal.

I slightly prefer the lighttpd configuration file format, but that could be just a matter of what I'm used to.

Switching to nginx means that I'll be able to try out Passenger, which seems like a very interesting application server.

I've already switched my dev machine. Next I'll switch our home server, and once I have the CMS changes done, I'll switch my VPS.

0 Comments
March 29, 2015

New album: 2015-03 (20 pictures)

15:28 -0400
[untitled] [untitled] [untitled] [untitled] (16 more...)
View entire album.

Photos of Gareth from March 2015.

0 Comments
February 19, 2015
11:04 -0500
Hubert Chathi: This is how you destroy a brand. After using and recommending ThinkPads for over a decade, my next laptop may not be one.
0 Comments
February 13, 2015
10:28 -0500
Hubert Chathi: First, @LinkedIn.com kills its RSS feeds, and now it restricts its API access. It's almost like LinkedIn hates its users.
Via: reddit
0 Comments
January 1, 2015
13:08 -0500
Hubert Chathi: is annoyed that USB video capture devices aren't just standard UVC devices
0 Comments
December 18, 2014
20:09 -0500
Hubert Chathi: What do you mean it isn't Friday yet?
0 Comments
December 16, 2014
22:10 -0500
Hubert Chathi: Farewell Dr. Dobbs :(
Via: reddit
0 Comments
December 13, 2014
09:51 -0500
Hubert Chathi: # loves The Magic Schoolbus
0 Comments
November 19, 2014
17:16 -0500
Hubert Chathi: Regarding the recent # GR, I pretty much agree with rra's take on the results.
0 Comments
November 17, 2014
14:30 -0500
Hubert Chathi: is making a rope net to add to the kids' jungle gym. Lots of tying...
0 Comments