whalebuilder: building packages using Docker

Hubert Chathi

DebConf 17 — August 11, 2017

The problem

or: why not build build packages in your working environment?

  • different packages may have conflicting build requirements
  • make sure your build-depends are correct
  • some software compiles differently if you have certain packages installed
  • build for a different release than what you have

The solution

  • build packages in a minimal, isolated environment (e.g. another machine, VM, chroot)
  • how?
    • sbuild
    • pbuilder/cowbuilder
    • whalebuilder
    • travis.debian.net

Why whalebuilder?

compiling.png

(xkcd #303)

setting up the build environment in pbuilder/cowbuilder takes a long time

  • copying chroot
  • installing build-dependencies
    • e.g. noweb build-depends on TeX Live, which is large, but once all the build-dependencies are installed, the actual package build is fairly quick (~ 2mins in my old laptop with HDD)
    • I need to fix a bug, which may involve trial and error, and don't want to wait for TeX Live to install over and over

How whalebuilder is different

  • union filesystems (no need for copying)
  • creates re-usable images with build-dependencies installed
  • Docker gives us some extra features for free
    • no network access
    • checks for filesystem changes outside of build tree (e.g. temporary files that aren't cleaned up)

Docker basics

very high-level overview

  • images: like a base filesystem
    • has layers (like an onion or an ogre)
  • container: a running instance of an image
    • isolated from the rest of the system
      • more isolated than a chroot, less isolated than a VM

How whalebuilder works

  • start with a "base" system image
    • create your own, or download pre-built image
  • create a new image with only build-dependencies
  • build package
  • copy result out of Docker

Basic usage: creating a base image

whalebuilder create [--debootstrap] [-r dist] <imagename>
  • Docker images that you don't build yourself might not be trustworthy, so consider using --debootstrap
    • "fun" story: --debootstrap option broke recently, but I fixed it (I hope) in 0.5
  • if the base image is out of date, create a new one or
whalebuilder update <imagename>
  • but rebuild occasionally
    • or else your base image will grow (check with docker history)
    • so that you don't accumulate obsolete packages

I also provide pre-built base images (whalebuilder/debian)

  • auto-built, based on Docker's Debian images
  • Ubuntu images to come
  • but again, you may want to be wary of images that you don't build yourself

Basic usage: building a package

whalebuilder build <baseimage> <dscfile>

e.g. 

docker images | grep whalebuilder
whalebuilder build whalebuilder-base-debootstrap:sid \
  noweb_2.11b-11.dsc

or 

whalebuilder build --no-install-depends \
  whalebuilder_build/noweb:2.11b-11
  noweb_2.11b-11.dsc

Failing build

  • when a build fails, whalebuilder does not remove the container
  • can inspect it with
    • docker export
    • docker commit
    • others? Help from Docker gurus?

Tips & Tricks

non-base base images

  • if you maintain many packages with similar build-dependencies (e.g. many Qt packages)
  • create a directory with this Dockerfile:
FROM whalebuilder-base:sid # your normal base image

RUN apt-get update \
  && apt-get install -y --no-install-recommends qtbase5-dev ... \
  && apt-get clean
  • run:
docker build -t whalebuilder-qt:sid .
  • and use whalebuilder-qt:sid as your base image when building

incrementally creating a build-dependency image

  • packaging new software
  • aren't sure the exact dependencies
  • e.g. upstream just says "it needs Qt 5"
    • qtbase5-dev, qtdeclarative5-dev? qt5keychain-dev? libqt5webkit5-dev?

  • start with a base guess 

    Build-Depends: ..., qtbase5-dev, libqt5webkit5-dev
  • try to build with whalebuilder
  • get an error message, and figure out what additional package(s) is/are needed
  • update Build-Depends
  • use old dependency image as base image
whalebuilder build whalebuilder_build/nheko:0.0_2bgit20170607.82b48d9-1 <dscfile>
  • repeat as needed

adding non-Debian sources/packages

  • suppose that you're building a few new packages (e.g. a program and an associated library), not yet in Debian, and one new package (the program) build-depends on another new package (the library)
  • install extra packages individually with --deb option (v0.5)
whalebuilder build --deb some_deb_file.deb <imagename> <dscfile>
  • suppose that you're building for not-Debian, and you need packages from another apt repository
  • use the --hook option to add commands to Dockerfile
# note: all one line
whalebuilder build --hook
  'RUN echo "deb http://ftp.debian.org/debian jessie-backports main non-free contrib"
     > /etc/apt/sources.list.d/backports.list
   && echo "deb http://matrix.org/packages/debian/ jessie main"
     > /etc/apt/sources.list.d/synapse.list
   && apt-get update
   && apt-get install -y --no-install-recommends curl ca-certificates
   && curl https://matrix.org/packages/debian/repo-key.asc | apt-key add -
   && apt-get remove -y curl ca-certificates && apt-get autoremove'
   <imagename> <dscfile>
  • TODO: make that friendlier

refresh dependency image

  • e.g. when a build-dependency gets updated (e.g. library transitions)
  • with --pull (if using a pre-built base image)
  • with --no-cache (v0.5)

Why not whalebuilder

  • only works where Docker works (amd64, arm?)
  • doesn't work with remote Docker
    • but you can run whalebuilder on a remote host
  • might not be scriptable (I make no guarantees)
  • others?

The future (help welcome)

  • specify dpkg-buildpackage arguments
  • use working directory instead of having to build source package
    • e.g. you don't have dependencies required to build the source
  • automatically run lintian, debsign
  • git-buildpackage, etc. support
  • cross-compiling
  • build on non-Debian OSes
  • support eatmydata, ccache for faster builds
  • get whalebuilder into a stable Debian release!

Questions?

slides at: https://www.uhoreg.ca/documents/debconf17-whalebuilder