Firecracker is recently making rounds on the internet as this relatively new, awesome technology for running lightweight VMs.
As something coming from AWS and powering AWS Lambda, my original perception was that it’s not easy to set up and use. However, this write from Julia Evans proved me wrong. So, as I have recently picked up a used Dell R720 with decent amount of RAM and CPUs, it was time to take these two for a spin together.
Sipping the first coffee this gloomy Nürburgring weather morning, the thought of putting Vault on Firecracker seemed somewhat amusing, … and the day was gone.
I’m going to show you how I’ve done it. Do I like Firecracker? It’s been only one day but yes, it’s pretty neat and easy to use.
§Environment
Clean HWE Ubuntu 18.04.5 Server installation with the password-less sudoer user.
§Intro
A Firecracker VM requires a Linux Linux Kernel and a root file system. Most of the people who talk about Firecracker use example hello-vmlinux and a root file system from AWS. I wanted to learn how to build these myself so I went with building my own 5.8 kernel and a root file system extracted from the default official HashiCorp Vault Docker image. The steps:
- install all the software required to build Linux kernel and install Docker
- setup directory structure to work with
- install Firecracker
- get, configure and build Linux kernel
- extract Vault file system from a running container
- run Vault Firecracker microVM
§Dependencies
To avoid multiple apt-get updates, add Docker repository first:
1
2
3
4
5
|
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
|
Next, install all the software required to compile the kernel:
1
2
3
4
5
6
7
8
9
|
sudo apt-get update
sudo apt-get install \
bison \
build-essential \
flex \
git \
libelf-dev \
libncurses5-dev \
libssl-dev -y
|
followed by Docker dependencies:
1
2
3
4
5
6
|
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common -y
|
and finally, Docker itself:
1
2
3
4
5
6
|
sudo apt-get install \
docker-ce \
docker-ce-cli \
containerd.io -y
sudo groupadd docker # this may report that the group already exists
sudo usermod -aG docker $USER
|
§Setup the directory structure
To have everything in one place, create the following directories:
1
2
3
|
sudo mkdir -p /firecracker
sudo chown -R ${USER}:${USER} /firecracker
mkdir -p /firecracker/{configs,filesystems,kernels,linux.git,releases}
|
configs: this is where we will put the VM config JSON filesfilesystems: this is where the extracted file systems will residekernels: this directory contains pre-built Linux kernelslinux.git: Linux sources go herereleases: Firecracker releases will be installed here
§Install Firecracker
This is as simple as downloading a pre-built binary release from GitHub and putting it on the PATH. I wanted to have versioning available for ease of upgrading in the future so I’ve built a shell program to manage this for me. You can find the program in this repository on GitHub. Long story, short:
- download
install-firecracker.sh program and put it in /firecracker directory; technically does not matter where but the program assumes the /firecracker/... directory structure from the previous step
chmod +x /firecracker/install-firecracker.sh- run:
sudo /firecracker/install-firecracker.sh
This will download the latest Firecracker release, install the release in /firecracker/releases directory and create /usr/bin/firecracker-<version> and /usr/bin/jailer-<version> links. You’ll also get /usr/bin/firecracker and /usr/bin/jailer links pointing to the version links. You can now run Firecracker:
§Get Linux Kernel
I’m going to use 5.8 kernel. Do I need one? Not sure but why not.
1
|
export KERNEL_VERSION=v5.8
|
Clone sources from GitHub:
1
2
3
|
cd /firecracker/linux.git
git clone https://github.com/torvalds/linux.git .
git checkout ${KERNEL_VERSION}
|
So, I’m not really fluent at this and…
This is the first time I’m building the kernel but fortunately one of the the Firecracker getting started documents points to a recommended kernel config. What’s less fortunate, the document talks about kernel v4.20 and the config is for v4.14.174. I’ve downloaded the file and placed it in /firecracker/linux.git/.config anyway but when I tried building the v5.8 kernel, make was insisting on recreating the config and asked me a lot of questions about what I want.
I don’t know what I want and I don’t know if make took the values from the old .config so I basically held Enter down for a bit. make moved my .config to .config.old and gave me a new .config file for v5.8 kernel. I took the new generated file and compared it with the original v4.14.174 config.
These files have few thousand lines so I wrote a program in Golang which loads both versions and compares the values.
With a flag, it allows bringing non-existing values from the good config to the new one. You can find this program in the kernel-configs directory of this repository[]. To get the v5.8 config, I basically executed:
1
2
3
4
|
go run ./compare-configs.go \
--good-config=./4.14.174.config \
--new-config=./5.8.config \
--bring-old-non-existing
|
Next, I’ve replaced the /firecracker/linux.git/.config file contents with the result of my compare configs program and re-run kernel build. This time, it did not insist on recreating anything and used my config, definitely. Happy times!
§Build the kernel
You probably want to change the value of -j to something like 4 or 8. The build will take a bit more time.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
time make vmlinux -j32
...
LD vmlinux.o
MODPOST vmlinux.symvers
MODINFO modules.builtin.modinfo
GEN modules.builtin
LD .tmp_vmlinux.kallsyms1
KSYM .tmp_vmlinux.kallsyms1.o
LD .tmp_vmlinux.kallsyms2
KSYM .tmp_vmlinux.kallsyms2.o
LD vmlinux
SORTTAB vmlinux
SYSMAP System.map
real 0m54.052s
user 23m51.313s
sys 2m35.287s
|
After the build is complete, copy the vmlinux binary to the /firecracker/kernels directory:
1
|
mv ./vmlinux /firecracker/kernels/vmlinux-${KERNEL_VERSION}
|
§Build the file system
Fun part starts here. The article from Julia Evans talks about getting an init system installed in the container. I have naively tried simply extracting the file system and running it bluntly without doing any additional configuration but Firecracker was complaining about not having /sbin/openrc available. Makes sense, Docker images generally don’t need an init system. So I had to find a method to get one in.
Fortunately, Vault Docker image is built from Alpine Linux and the Creating a rootfs Image instructions show how to add an init system to a file system of an Alpine based container.
We’ve building a file system for Vault:
Create a file system file and format it as ext4:
1
2
3
|
rm /firecracker/filesystems/vault-root.ext4
dd if=/dev/zero of=/firecracker/filesystems/vault-root.ext4 bs=1M count=500
mkfs.ext4 /firecracker/filesystems/vault-root.ext4
|
I have no idea how much space I needed. 50 megs wasn’t enough so I gave it 500 instead.
Create a mount directory and mount the file system file:
1
2
3
|
mkdir -p /firecracker/filesystems/mnt-${FS}
sudo mount /firecracker/filesystems/${FS}-root.ext4 \
/firecracker/filesystems/mnt-${FS}
|
Now, run the Vault container and fetch the container ID, attach file system mount directory to the container:
1
|
export CONTAINER_ID=$(docker run -t --rm -v /firecracker/filesystems/mnt-${FS}:/export-rootfs -d vault:latest)
|
Get the shell in the container:
1
|
docker exec -ti ${CONTAINER_ID} /bin/sh
|
Now, in the container shell, execute these commands:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
# install the init system and some extra tools:
apk add openrc
apk add util-linux
# set up a login terminal on the serial console (ttyS0):
ln -s agetty /etc/init.d/agetty.ttyS0
echo ttyS0 > /etc/securetty
rc-update add agetty.ttyS0 default
# Make sure special file systems are mounted on boot:
rc-update add devfs boot
rc-update add procfs boot
rc-update add sysfs boot
# EXTRA: I had to add these lines:
# --------------------------------
# 1. enable local services:
# https://wiki.gentoo.org/wiki//etc/local.d
rc-update add local default
# 2. create a local service to Start Vault dev server on system boot:
echo "#!/bin/sh" >> /etc/local.d/HashiCorpVault.start
echo "/usr/local/bin/docker-entrypoint.sh server -dev && reboot || reboot" >> /etc/local.d/HashiCorpVault.start
# 3. make it executable:
chmod +x /etc/local.d/HashiCorpVault.start
# 4. For convenience, enable output from local service so I can see errors:
echo rc_verbose=yes > /etc/conf.d/local
# 5. make sure I also have /home and /vault directories in my exported file system
for d in home vault; do tar c "/$d" | tar x -C /export-rootfs; done
# EXTRA / end
# Then, copy the newly configured system to the rootfs image:
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /export-rootfs; done
for dir in dev proc run sys var; do mkdir /export-rootfs/${dir}; done
# All done, exit docker shell
exit
|
A few words about the HashiCorpVault.start service file. One method to shut the Firecracker VM gracefully down is to call reboot from inside of the VM. This is because Firecracker exits on CPU reset, more info here. Hence the command:
1
|
/usr/local/bin/docker-entrypoint.sh server -dev && reboot || reboot
|
will start the Vault server using the regular docker-entrypoint.sh from the original image.
The && reboot part will ensure the VM stops automatically after Vault exits gracefully. The || reboot part will stop the VM if Vault does not start for whatever reason.
This saves a hassle of doing ps -a and sudo kill <pid> dance when things go south.
You can now stop the container and unmount the file system:
1
2
|
docker stop ${CONTAINER_ID}
sudo umount /firecracker/filesystems/mnt-${FS}
|
§Launch Vault on Firecracker
Firecracker VMs can, by design, only use Linux tap devices for networking. There are tools that create ad-hoc devices using CNI plugins but I went with the method from Julia Evans. So, this part is directly lifted from Julia Evans.
Prepare kernel boot args:
1
2
3
4
5
|
export MASK_LONG="255.255.255.252"
export FC_IP="169.254.0.21"
export TAP_IP="169.254.0.22"
export KERNEL_BOOT_ARGS="ro console=ttyS0 noapic reboot=k panic=1 pci=off nomodules random.trust_cpu=on"
export KERNEL_BOOT_ARGS="${KERNEL_BOOT_ARGS} ip=${FC_IP}::${TAP_IP}:${MASK_LONG}::eth0:off"
|
Setup tap network interface:
1
2
3
4
5
6
7
8
9
10
|
export TAP_DEV="fc-88-tap0"
export MASK_SHORT="/30"
export FC_MAC="02:FC:00:00:00:05"
sudo ip link del "$TAP_DEV" 2> /dev/null || true
sudo ip tuntap add dev "$TAP_DEV" mode tap
sudo sysctl -w net.ipv4.conf.${TAP_DEV}.proxy_arp=1 > /dev/null
sudo sysctl -w net.ipv6.conf.${TAP_DEV}.disable_ipv6=1 > /dev/null
sudo ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
sudo ip link set dev "$TAP_DEV" up
|
Write the VM config:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
cat <<EOF > /firecracker/configs/vault-config.json
{
"boot-source": {
"kernel_image_path": "/firecracker/kernels/vmlinux-${KERNEL_VERSION}",
"boot_args": "$KERNEL_BOOT_ARGS"
},
"drives": [
{
"drive_id": "rootfs",
"path_on_host": "/firecracker/filesystems/${FS}-root.ext4",
"is_root_device": true,
"is_read_only": false
}
],
"network-interfaces": [
{
"iface_id": "eth0",
"guest_mac": "${FC_MAC}",
"host_dev_name": "${TAP_DEV}"
}
],
"machine-config": {
"vcpu_count": 1,
"mem_size_mib": 128,
"ht_enabled": false
}
}
EOF
|
And launch:
1
|
firecracker --no-api --config-file /firecracker/configs/vault-config.json
|
The result will be similar to:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
[ 0.000000] Linux version 5.8.0 (radek@r720sas) (gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, GNU ld (GNU Binutils for Ubuntu) 2.30) #4 SMP Sat Feb 6 18:50:30 UTC 2021
[ 0.000000] Command line: ro console=ttyS0 noapic reboot=k panic=1 pci=off nomodules random.trust_cpu=on ip=169.254.0.21::169.254.0.22:255.255.255.252::eth0:off root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007ffffff] usable
...
* Mounting persistent storage (pstore) filesystem ... [ ok ]
* Starting local ... * Executing "/etc/local.d/HashiCorpVault.start" ...==> Vault server configuration:
Api Address: http://0.0.0.0:8200
Cgo: disabled
Cluster Address: https://0.0.0.0:8201
Go Version: go1.15.7
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: inmem
Version: Vault v1.6.2
Version Sha: be65a227ef2e80f8588b3b13584b5c0d9238c1d7
==> Vault server started! Log data will stream in below:
2021-02-06T21:13:22.006Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2021-02-06T21:13:22.007Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
2021-02-06T21:13:22.020Z [INFO] core: security barrier not initialized
2021-02-06T21:13:22.020Z [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1
...
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variable:
$ export VAULT_ADDR='http://0.0.0.0:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: YqKuQlzPiMUQXphehp0M7DAvsqImqNJrvJqAn/R0nyc=
Root Token: s.F4erraVTvHnx3oU4Ac8zwaCP
Development mode should NOT be used in production installations!
|
Try it out by opening another terminal on the same machine and running:
1
2
|
curl http://169.254.0.21:8200/sys/health
{"errors":[]}
|
It’s alive.
You can stop the VM by simply pressing CTRL+C in the terminal window where Vault is running. The VM will shut gracefully down:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
^C==> Vault shutdown triggered
2021-02-06T21:13:33.863Z [INFO] core: marked as sealed
2021-02-06T21:13:33.866Z [INFO] core: pre-seal teardown starting
2021-02-06T21:13:33.869Z [INFO] rollback: stopping rollback manager
2021-02-06T21:13:33.872Z [INFO] core: pre-seal teardown complete
2021-02-06T21:13:33.873Z [INFO] core: stopping cluster listeners
2021-02-06T21:13:33.874Z [INFO] core.cluster-listener: forwarding rpc listeners stopped
2021-02-06T21:13:34.076Z [INFO] core.cluster-listener: rpc listeners successfully shut down
2021-02-06T21:13:34.082Z [INFO] core: cluster listeners successfully shut down
2021-02-06T21:13:34.085Z [INFO] core: vault is sealed
[ ok ]
[ ok ]
* Stopping local ... [ ok ]
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
[ 15.877742] Unregister pv shared memory for cpu 0
[ 15.880260] reboot: Restarting system
[ 15.881627] reboot: machine restart
radek@r720:/firecracker$
|
That’s all for today.