whalebuilder: building packages using Docker

The problem

or: why not build build packages in your working environment?

• different packages may have conflicting build requirements
• make sure your build-depends are correct
• some software compiles differently if you have certain packages installed
• build for a different release than what you have

The solution

• build packages in a minimal, isolated environment (e.g. another machine, VM, chroot)
• how?
  • sbuild
  • pbuilder/cowbuilder
  • whalebuilder
  • travis.debian.net

Why whalebuilder?

How whalebuilder is different

• union filesystems (no need for copying)
• creates re-usable images with build-dependencies installed
• Docker gives us some extra features for free
  • no network access
  • checks for filesystem changes outside of build tree (e.g. temporary files that aren't cleaned up)

Docker basics

very high-level overview

• images: like a base filesystem
  • has layers (like an onion or an ogre)
• container: a running instance of an image
  • isolated from the rest of the system
    • more isolated than a chroot, less isolated than a VM

How whalebuilder works

• start with a "base" system image
  • create your own, or download pre-built image
• create a new image with only build-dependencies
• build package
• copy result out of Docker

Basic usage: creating a base image

whalebuilder create [--debootstrap] [-r dist] <imagename>

• Docker images that you don't build yourself might not be trustworthy, so consider using --debootstrap
  • "fun" story: --debootstrap option broke recently, but I fixed it (I hope) in 0.5

Basic usage: building a package

whalebuilder build <baseimage> <dscfile>

e.g.

docker images | grep whalebuilder
whalebuilder build whalebuilder-base-debootstrap:sid \
  noweb_2.11b-11.dsc

or

whalebuilder build --no-install-depends \
  whalebuilder_build/noweb:2.11b-11
  noweb_2.11b-11.dsc

Tips & Tricks

non-base base images

• if you maintain many packages with similar build-dependencies (e.g. many Qt packages)
• create a directory with this Dockerfile:

FROM whalebuilder-base:sid # your normal base image

RUN apt-get update \
  && apt-get install -y --no-install-recommends qtbase5-dev ... \
  && apt-get clean

• run:

docker build -t whalebuilder-qt:sid .

• and use whalebuilder-qt:sid as your base image when building

incrementally creating a build-dependency image

• packaging new software
• aren't sure the exact dependencies
• e.g. upstream just says "it needs Qt 5"
  • qtbase5-dev, qtdeclarative5-dev? qt5keychain-dev? libqt5webkit5-dev?

adding non-Debian sources/packages

• suppose that you're building a few new packages (e.g. a program and an associated library), not yet in Debian, and one new package (the program) build-depends on another new package (the library)
• install extra packages individually with --deb option (v0.5)

whalebuilder build --deb some_deb_file.deb <imagename> <dscfile>

refresh dependency image

• e.g. when a build-dependency gets updated (e.g. library transitions)
• with --pull (if using a pre-built base image)
• with --no-cache (v0.5)

Why not whalebuilder

• only works where Docker works (amd64, arm?)
• doesn't work with remote Docker
  • but you can run whalebuilder on a remote host
• might not be scriptable (I make no guarantees)
• others?

The future (help welcome)

• specify dpkg-buildpackage arguments
• use working directory instead of having to build source package
  • e.g. you don't have dependencies required to build the source
• automatically run lintian, debsign
• git-buildpackage, etc. support

Questions?

slides at: https://www.uhoreg.ca/documents/debconf17-whalebuilder