The clever peepz over at Moby are clearly hard at work revolutionising the way we work with containers and images. I have recently been interested in the BuildKit project.
BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. buildkit
What is BuildKit?
In a nutshell BuildKit is an engine which interprets a graph of instructions into a target container image format. With the ability to target multiple export formats (e.g. OCI or docker), support multiple frontends (e.g. Dockerfile) and provide all sorts of wonderful features such as efficient layer caching and operation parallelization.
BuildKit is separate from Docker and only requires a container runtime to facilitate the execution of operations to create image layers. The currently supported runtime's are
containerd and plain old runc.The BuildKit project itself consists of two key components. The first being the build daemon
buildkitd. The second being a cli tool called buildctl which is used to facilitate command line communication with the buildkitd instance.The BuildKit cli
buildctl is configured to receive a graph based IR (intermediate representation) of operations. This IR is called llb which stands for low-level builder. This format is defined in the BuildKit project using protobuf. The buildctl command itself expects an llb to be marshalled in this format on STDIN. Currently there are some handy Go libraries for describing and marshalling an llb to an io.Writer. However, since the format is protobuf it will be really easy to port llb generating code to other languages. As a die hard gopher myself though, I am fortunate enough to be able to start hacking on this right away.It is the
llb medium which facilitates BuildKit's ability to have multiple frontends. There currently exists an experimental frontend for Dockerfile. I believe the intent of this project is to replace the current docker builder baked into the docker command.The BuildKit Github project contains an
examples folder with lots of Go binaries which spit protobuf encoded llb instructions which can be piped into buildctl, examined and then executed. I am going to walk through how this can be achieved on a mac.Walkthrough
To start with here is my setup:
Mac OSX Version 10.11.6 (15G1004)
go version
go version go1.9 darwin/amd64
Docker For Mac Version 17.12.0-ce-rc4-mac44 (21438)
docker version
Client: Version: 17.12.0-ce-rc4 API version: 1.35 Go version: go1.9.2 Git commit: 6a2c058 Built: Wed Dec 20 15:53:52 2017 OS/Arch: darwin/amd64 Server: Engine: Version: 17.12.0-ce-rc4 API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: 6a2c058 Built: Wed Dec 20 15:59:49 2017 OS/Arch: linux/amd64 Experimental: true
make --version
GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-apple-darwin11.3.0
jq --version
jq-1.5
Step one is to fetch the
buildkit project itself. I am going to be building it from source for this walkthrough.go get github.com/moby/buildkit
Step two is to get into the BuildKit project itself, so that we can build it and play with the examples.
cd $GOPATH/src/github.com/moby/buildkit
Step three is to build the
buildctl tool I described earlier. For this walkthrough I will not be building the daemon for mac, but instead I will run the daemon as a docker container in a later step.
This make command will build a target binary into the local ./bin directory, which is why we will refer to the binary as ./bin/buildctl-darwin from here on out.make bin/buildctl-darwin
Next we will run
buildkitd as a container. For this we can fetch tonistiigi's handy image, run it as a privileged container and expose its 1234 port on the host.
This has been taken directly from the Buildkit README.docker run -d --privileged -p 1234:1234 tonistiigi/buildkit --addr tcp://0.0.0.0:1234
export BUILDKIT_HOST=tcp://0.0.0.0:1234
buildctl build --help
Now we can start playing with BuildKit. For the purpose of this walkthrough, we will use
examples/buildkit0. The aim of this example is to build buildkit itself along with runc and containerd.
To help us understand this, we are going to use the buildctl debug dump-llb command to convert the llb protobuf into a more readable JSON array of layer operations. Then we will pipe the JSON through jq to prettify.go run examples/buildkit0/buildkit.go | ./bin/buildctl-darwin debug dump-llb | jq '.'
Here is a snippet of the JSON output from
dump-llb://...
{
"Op": {
"inputs": [
{
"digest": "sha256:6c494995583024a1d0aa139a485de80e0fc08420db8f3fff743d7c8661db2038",
"index": 1
}
],
"Op": {
"Exec": {
"meta": {
"args": [
"ls",
"-l",
"/bin"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"cwd": "/"
},
"mounts": [
{
"input": 0,
"dest": "/",
"output": 0
}
]
}
}
},
"Digest": "sha256:9a5e639ad73c2329313a7809e60f57a2bd4cfff20e6094dd8ad8d42f69cae397",
"OpMetadata": {}
}
//...
This is the penultimate operation in the sequence. Looking at the generating code itself in
examples/buildkit0/buildkit.go you will find the above operation is created by the line:out := bk.Run(llb.Shlex("ls -l /bin")) // debug output
Now lets build this example!
go run examples/buildkit0/buildkit.go | ./bin/buildctl-darwin build
INFO[0000] tracing logs to /var/folders/9k/c2skyh6565q8005str8r1b9w0000gn/T/buildctl453683322
[+] Building 59.3s (19/19) FINISHED
=> CACHED docker-image://docker.io/library/golang:1.9-alpine 0.0s
=> => resolve docker.io/library/golang:1.9-alpine 0.5s
=> apk add --no-cache g++ linux-headers 11.1s
=> CACHED docker-image://docker.io/library/alpine:latest 0.0s
=> => resolve docker.io/library/alpine:latest 1.8s
=> apk add --no-cache git make 3.5s
=> git clone https://github.com/opencontainers/runc.git /go/src/github.com/opencontainers/runc 3.3s
=> git clone https://github.com/moby/buildkit.git /go/src/github.com/moby/buildkit 18.3s
=> apk add --no-cache btrfs-progs-dev 5.8s
=> git checkout -q 74a17296470088de3805e138d3d87c62e613dfc4 0.4s
=> go build -o /usr/bin/runc ./ 3.4s
=> git clone https://github.com/containerd/containerd.git /go/src/github.com/containerd/containerd 12.5s
=> git checkout -q v1.0.0 0.6s
=> go build -o /bin/buildctl ./cmd/buildctl 11.7s
=> go build -o /bin/buildkitd ./cmd/buildkitd 19.7s
=> make bin/containerd 24.7s
=> cp -a /src/bin/buildctl /dest/bin/ 0.4s
=> cp -a /src/usr/bin/runc /dest/bin/ 0.5s
=> cp -a /src/go/src/github.com/containerd/containerd/bin/containerd /dest/bin/ 0.4s
=> cp -a /src/bin/buildkitd /dest/bin/ 0.4s
=> ls -l /bin 0.4s
You should see some nice interactive output, similar to what is pasted above. However, you may be thinking at this point; where is this image? It is actually currently only inside the
buildkitd build cache. So lets take a look at the cache../bin/buildctl-darwin du
ID RECLAIMABLE SIZE LAST ACCESSED
sha256:e67ac2c9401cbc0394d10 true 348.83MB
omrrlstueubebyt7ecbw6gwrf true 151.84MB
mxido1mek1s7o3bbrkjecgtxp true 151.84MB
... truncated
Now to get these layers into something tangible you will need to use an exporter. To do so, use the
--exporter flag on the build call. Thanks to the build cache the subsequent calls will be super quick.go run examples/buildkit0/buildkit.go | ./bin/buildctl-darwin build --exporter=docker --exporter-opt name=buildkit0 | docker load
For this example I have used the
docker exporter which spits out the tarball of the image on STDOUT, which is suitable for piping into docker load. The output of this command looks something like this:INFO[0000] tracing logs to /var/folders/9k/c2skyh6565q8005str8r1b9w0000gn/T/buildctl725843021
[+] Building 2.6s (20/20) FINISHED
=> CACHED docker-image://docker.io/library/alpine:latest 0.0s
=> => resolve docker.io/library/alpine:latest 1.9s
=> CACHED docker-image://docker.io/library/golang:1.9-alpine 0.0s
=> => resolve docker.io/library/golang:1.9-alpine 1.9s
=> CACHED apk add --no-cache g++ linux-headers 0.0s
=> CACHED apk add --no-cache git make 0.0s
=> CACHED git clone https://github.com/moby/buildkit.git /go/src/github.com/moby/buildkit 0.0s
=> CACHED go build -o /bin/buildctl ./cmd/buildctl 0.0s
=> CACHED cp -a /src/bin/buildctl /dest/bin/ 0.0s
=> CACHED git clone https://github.com/opencontainers/runc.git /go/src/github.com/opencontainers/runc 0.0s
=> CACHED git checkout -q 74a17296470088de3805e138d3d87c62e613dfc4 0.0s
=> CACHED go build -o /usr/bin/runc ./ 0.0s
=> CACHED cp -a /src/usr/bin/runc /dest/bin/ 0.0s
=> CACHED apk add --no-cache btrfs-progs-dev 0.0s
=> CACHED git clone https://github.com/containerd/containerd.git /go/src/github.com/containerd/containerd 0.0s
=> CACHED git checkout -q v1.0.0 0.0s
=> CACHED make bin/containerd 0.0s
=> CACHED cp -a /src/go/src/github.com/containerd/containerd/bin/containerd /dest/bin/ 0.0s
=> CACHED go build -o /bin/buildkitd ./cmd/buildkitd 0.0s
=> CACHED cp -a /src/bin/buildkitd /dest/bin/ 0.0s
=> CACHED ls -l /bin 0.0s
=> exporting to oci image format 2.6s
=> => exporting layers 2.1s
=> => exporting config sha256:efa327e9aeeadffdc162168b0c0b40990eeb577a1002ef1484de51dd6cdde7f4 0.0s
=> => exporting manifest sha256:6793e4492fd46b94cf80405cabacd8f15e0e936a311a24219ccefce49329250e 0.0s
=> => sending tarball 0.5s
4557c2a02b80: Loading layer [==================================================>] 4.922MB/4.922MB
12fd2a20e004: Loading layer [==================================================>] 2.848MB/2.848MB
8855e1274f0c: Loading layer [==================================================>] 6.725MB/6.725MB
57a8e2e4ca07: Loading layer [==================================================>] 9.422MB/9.422MB
e14e118cf849: Loading layer [==================================================>] 108B/108B
Loaded image: buildkit0:latest
The majority of the time spent was exporting the image thanks to the build cache. There is even a handy output to show the cache hits, much like
docker build.Now to be certain it all worked as expected, check your docker instance's image database.
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
...
buildkit0 latest efa327e9aeea 292 years ago 86.7MB
You may be a little concerned how the image got created 292 years ago. However, you will find that this is the product the date pretty-printer being supplied with the epoch.
It appears that at the time of writing this blog post, the docker exporter doesn't set the
created_at metadata on the docker image format.If this project interests you, I would recommend you start hacking on it.