Launching Consul cluster with firebuild and other news

The goal has been reached, I have a Consul cluster running

Some two months ago, when I started the Firecracker journey, I set myself a goal to run en etcd cluster in Firecracker microVMs. Many lines of code later, after tackling the problem the hard way, there’s an outcome.

Okay, it’s not etcd but rather HashiCorp Consul.

Here’s how a 3 node Consul cluster is launched with firebuild:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Start Consul 1 with IP 192.168.127.10:
sudo firebuild run \
    --profile=standard \
    --from=combust-labs/consul:1.9.4 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8 \
    --ip-address=192.168.127.10 \
    --name=consul1 \
    --daemonize \
    -- agent -server -client 0.0.0.0 -bootstrap-expect 3 -data-dir /consul/data -retry-join 192.168.127.11 -retry-join 192.168.127.12 -node consul1

# Start Consul 2 with IP 192.168.127.11:
sudo firebuild run \
    --profile=standard \
    --from=combust-labs/consul:1.9.4 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8 \
    --ip-address=192.168.127.11 \
    --name=consul2 \
    --daemonize \
    -- agent -server -client 0.0.0.0 -bootstrap-expect 3 -data-dir /consul/data -retry-join 192.168.127.10 -retry-join 192.168.127.12 -node consul2

# Start Consul 3 with IP 192.168.127.10:
sudo firebuild run \
    --profile=standard \
    --from=combust-labs/consul:1.9.4 \
    --cni-network-name=machines \
    --vmlinux-id=vmlinux-v5.8 \
    --ip-address=192.168.127.12 \
    --name=consul3 \
    --daemonize \
    -- agent -server -client 0.0.0.0 -bootstrap-expect 3 -data-dir /consul/data -retry-join 192.168.127.10 -retry-join 192.168.127.11 -node consul3

After a couple of seconds required to boot the VMs, start the services and reconcile the cluster, the cluster can be queries for status:

1
$ curl http://192.168.127.12:8500/v1/status/leader

The output will be similar to:

1
"192.168.127.10:8300"

What about the DNS? Consul nodes were launched with with -node flags. The cluster uses the default 8600 DNS port and default data center name of dc1.

Hence, we can query for a specific node:

1
$ dig @192.168.127.10 -p 8600 consul1.node.dc1.consul

The response looks similar to:

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @192.168.127.10 -p 8600 consul1.node.dc1.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13511
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul1.node.dc1.consul.	IN	A

;; ANSWER SECTION:
consul1.node.dc1.consul. 0	IN	A	192.168.127.10

;; ADDITIONAL SECTION:
consul1.node.dc1.consul. 0	IN	TXT	"consul-network-segment="

;; Query time: 1 msec
;; SERVER: 192.168.127.10#8600(192.168.127.10)
;; WHEN: Wed Apr 14 21:31:00 UTC 2021
;; MSG SIZE  rcvd: 104

§in other news

§documentation is online

There is a new firebuild documentation website online. It’s not complete but it’s a step forward from maintaining a huge readme file in a GitHub repository. The documentation is available here[1].

§Docker image based builds

why not both

Apart from that, there was a bit more work down in the trenches. From the very beginning, it was possible to create a root file system from a Dockerfile. But sometimes a Dockerfile is not sufficient, especially in case of a multi-stage build where the actual build happens outside of a Dockerfile.

An example of this is the Jaeger tracing Dockerfile with binary artifacts built in a separate make step.

It is now possible to build a root file system directly from a Docker image. Here’s an example of building Jaeger 1.22:

1
2
3
4
5
6
7
8
sudo firebuild rootfs \
    --profile=standard \
    --docker-image=jaegertracing/all-in-one:1.22 \
    --docker-image-base=alpine:3.13 \
    --cni-network-name=machine-builds \
    --vmlinux-id=vmlinux-v5.8 \
    --mem=512 \
    --tag=combust-labs/jaeger-all-in-one:1.22

Right now, more details only in the huge readme[2]. Documentation will be updated in the following weeks.

§vminit rootfs bootstrap now with MMDS

The next big update is the vminit rootfs MMDS based bootstrap. Until now, the rootfs was build via an SSH connection. This has been replaced with MMDS based bootstrap and the SSH code has been removed from firebuild. More about that in this article[3].

§vms can be named

As seen in the Consul cluster example at the top, the VMs can be named so it is much easier to refer to them later. No need to hunt the random VM ID anymore. Just start the VM with --name=unique1 and this is possible:

1
2
3
VMIP=$(sudo firebuild inspect \
    --profile=standard \
    --vmm-id=unique1 | jq '.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IP' -r)

By the way, the metadata JSON uses camel case for easier integration with tools like jq.

There are caveats related to names. The name can be maximum 20 characters long and only alphanumeric characters are allowed.

§test coverage

There have been many tests written and the coverage, especially for builds and resource discovery, has been increased significantly.

§what’s coming

The next big research areas over the coming weeks, in no particular order:

  • expose guest ports via command line flags on the host through iptables integration
  • run command to have support for adding files to the VM on boot
  • additional volumes for long lived data persistence
  • service discovery integration; first step is to integrate with Consul service catalog and DNS - this will open the door for launching more complex infrastructures with firebuild

That’s it for today, thank you for reading!


  1. firebuild documentation

  2. firebuild readme: build directly from a Docker image

  3. firebuild rootfs - gRPC with mTLS