Declaratively Provision Docker Images Using Nix

I like Docker. If you don’t understand why, read the 3.5k word epic that I wrote about it at InfoQ. In this post I’ll assume you’ve read my InfoQ article, or are at least somewhat familiar with Docker and its features. Here’s two features that I care about in particular:

After playing with Docker a while and deploying some apps with it, one thing that I feel could some help is the provisioning aspect of it: how do get your application and its dependencies into a container image?

The standard way of provisioning a Docker image is using a Dockerfile, which is basically a simple imperative script that builds up an image from a base image step by step. A typical Dockerfile looks starts like this:

FROM ubuntu
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y openssh-server python curl

Every command that you run is committed, resulting in an aufs layer. This can be helpful, because Docker can now do basic caching. For instance, if your build fails at the last line, and you fix it and rerun the build, it can use the image resulting from the first succeeding lines and start from there. However, a few problems follow from this approach:

  1. Can those first initial lines really be cached, or may their result be dependent on the time of being run? Answer: yes, running these lines tomorrow may yield different results than running them today, but Docker will naively assume they will always result in the same thing.
  2. AuFS can only handle a few dozen layers, if your Dockerfile has too many commands the build will simply fail. So, you better make them count. The result is ugly stuff like this.
  3. There’s very little support for reuse. There’s no include files or configuration language. The only form of reuse is using base images where you create a base image with software common to all other images, and then you use FROM to base future images on. Yet, you’re still limited by (2), the layers all add up.

There are other tools out that you can use to provision a Docker container, like Chef. But I find these tools rather heavy weight and I really don’t want to add extra weight to my container by including a deployment tool.

At the same time I was playing with Docker, I also did a fair bit of deployment work with Nix. For instance, the developer site we launched, as well as the load-balanced REPL servers are all deployed using Nix and NixOps onto EC2 machines running NixOS. The developer site, releases site and soon the main website all run on separate wordpress installs that in principle share a lot of parts (plug-ins, themes). Nix makes it really easy to implement this reuse. It’s really a joy to work with. Ask my colleagues: every day that I used Nix I praised it in our company chatroom. It’s that cool, once you get over the initial learning curve.

However, NixOps and NixOS are pretty all-or-nothing solutions. To use it you need machines that run NixOS, which can run on “real” hardware, EC2 and Hetzner, but most other cloud providers (e.g. DigitalOcean, which I really like — especially its prices) don’t support it and may not for a while, or ever. Therefore, at this time it’s difficult to deploy applications onto random cloud providers using Nix technology. Its portability is limited.

So, for the past weeks I’ve been thinking: how can the portability of Docker and the general provisioning awesomeness of Nix be combined?

But let’s first take a brief step back and reiterate why Nix is and why I like it (and you will too, once you invest some time in learning it).

What Nix brings to the table

Nix is a relatively new package manager for Unix systems, it’s not specific to Linux, it works on any Unix system, in principle. This tweet sums it up pretty succinctly:

Tweet

With Nix:

While Nix itself is “just” a package manager, there are tools built on top of it, including the NixOS Linux distribution called. Based on a single Nix configuration file, Nix can derive and entire system, which can be deployed locally, or remotely via NixOps.

In NixOS, all services run using systemd, kernels are deployed and a bunch of utility processes are running at all times. As a result, a minimal NixOS closure quickly becomes hundreds of megabytes big and too heavy-weight for a Docker container.

Regardless, NixOS configurations are kind of nice and clean and would make a great way of provisioning Docker images as well. For instance, here’s how to run a simple Apache server serving static files from ./www directory:

{ config, pkgs, ... }:
{
  services.httpd = {
    enable = true;
    documentRoot = ./www;
    adminAddr = "zef.hemel@logicblox.com";
  };
}

The ./www there refers to the path ./www local to the system configuration file. When the system configuration is built, the contents of ./www is automatically copied into the Nix store and becomes part of the dependencies of the system configuration. So this idea of “first I copy all my web files to /var/www, and then I point Apache to it” goes away.

To bring this awesomeness to Docker, I’ve been hacking on a project called nix-docker, which allows you to quickly and efficiently build Docker images using NixOS modules. In fact, the example I just gave can be built into a Docker image just great.

Introducing nix-docker

Rather than using systemd to run services inside the container (which is tricky to get to work inside of a Docker container and has a slew of dependencies of its own), I opted for using supervisord, which appears to be the de-facto standard for running multiple services at once in a Docker container. A simple application running a node.js server can be defined as follows:

{ config, pkgs, ... }:
{
  supervisord.services.nodeApp = {
    command = "${pkgs.nodejs}/bin/node ${./app}/server.js";
  };

  docker.ports = [ 8080 ];
}

This assumes you have an app/ directory in the same directory as the configuration file with a server.js in it that runs a server on port 8080. The docker.ports configuration ensures that port 8080 is exposed to the outside world (the equivalent of EXPOSE in a Dockerfile).

Now let’s say you named this file configuration.nix. You can now build it into a Docker image as follows:

# nix-docker -b -t zefhemel/myapp configuration.nix

We’ll get to the -b at the very end, the -t option is used to name the image (zefhemel/myapp in this case) and configuration.nix is the file name of the config to build.

And that’s it. In case you need Redis for your application, that can be enabled easily, because there’s a reusable Redis module already available:

{ config, pkgs, ... }:
{
  supervisord.services.nodeApp = {
    command = "${pkgs.nodejs}/bin/node ${./app}/server.js";
  };

  services.redis.enable = true;

  docker.ports = [ 8080 ];
}

The first thing nix-docker will do is build the system configuration and all its dependencies, in this case including:

and various other things. Note that building may sound heavier than it is. Nix can fetch prebuilt binaries for most packages, and only ever rebuilds something if it’s both not already available in the local nix store, or downloadable from a binary cache. So generally the process when iterating is very quick.

Once the build completes, it copies the closure to a bare busybox-based Docker image, sets some meta information (like the exposed ports, volumes etc.) and it’s done. You can now push your image to a Docker registry and run it anywhere where Docker runs.

To reduce the per-image size even more, it’s possible to use base images, nix-docker is clever enough to check what /nix/store paths are available in the base image already and not to copy those again for the new image thereby greatly reducing image sizes. For instance, to base it on my zefhemel/base-nix image:

# nix-docker -b -t zefhemel/myapp --from zefhemel/base-nix configuration.nix

And the resulting image will be much smaller, because it won’t have to copy many of the common things (like supervisord etc.).

One more thing

While nix-docker is a pragmatic solution, many Nix hackers won’t like the way I just described it to work. They’ll say “hey, Nix already has perfect support for isolated installations of software, why would you need to copy all that stuff into a container and ship it around?” And they’d be kind of right. A more Nix-native thing to do would be to ship a Nix closure for the application you want to run to the server where you want to run it, run that in an essentially empty Docker container where you bind-mount the host’s Nix store into the container. And guess what…

nix-docker supports this too!

This feature enables a second way of distributing Docker containers without the use of Docker registries. All it requires is to have Nix installed on the host machine.

To use this feature, simply leave out the -b option (note: in this mode you can even use nix-docker without having Docker installed on the building machine):

$ nix-docker -t my-app configuration.nix

This will not build a Docker container. Instead, it’ll build a Nix package that you can ship to the target server via SSH (using nix-copy-closure) and then run the docker-run script that it comes with. What this script will do is build a very minimal Docker image on-demand containing only some meta data (like EXPOSE and VOLUME, RUN commands in a Dockerfile) and mounting in the host’s Nix store into the container via -v /nix/store:/nix/store.

There’s a few reasons you would do this:

  1. This allows you to avoid Docker registries altogether. It enables you to distribute a Docker application to any server via SSH.
  2. It’s more disk-space efficient. No Nix store paths are every duplicated on the same machine.
  3. Build times are much faster, since Nix builds are fully incremental, only things that have not been build before will be built.

The only drawback is that Nix (not NixOS) needs to be installed on the target machine, thereby reducing portability somewhat. In practice I often use this mode during development and then use the -b option to build a “proper” Docker image when all is set up right.

How to try it out

For convenience, the nix-docker repo includes a Vagrantfile. Which makes it easy to get started if you have Vagrant installed, and also allows you to play when you’re on Mac OS X or even Windows. Vagrant will automatically install Docker, Nix and nix-docker so you can get started immediately. The repo contains some sample configurations for you to try.

To learn more, have a look at the code and learn more about Nix in general. In the end, it’s just Nix all the way down. In fact, the only part of nix-docker that’s not written in Nix is the 85 line nix-docker Python script that ties everything together.

Current state

It’s still early days for nix-docker. Currently I’ve built and enabled only a few modules including:

Many more existing NixOS modules should work but have to be tested first.

Update: Here’s a post describing how to provision a Docker container with Wordpress using nix-docker.