Currently, when a
rootfs is built, the guest is started with an SSH server and the bootstrap process executes via an SSH connection. I don’t like this and want to replace the SSH method with an MMDS based solution. MMDS is already present in the
firebuild run command.
run uses the
vminit component from
firebuild-mmds. When the guest starts, the
vminit guest service connects to the MMDS endpoint, downloads the metadata and configures the VM. This is pretty similar to cloud-init but I don’t want cloud-init at this stage. Writing a cloud-init provider in Python is a bit of a head scratcher, can be done but maybe some other time.
rootfs bootstrap is pretty similar to
run in the sense that it also starts a guest VM. There is no reason why it should not work the same way. Bye SSH, welcome MMDS, easy peasy. Not so… The big difference between
rootfsrequires access to
COPYresources present in the Docker artifact
Multiple approaches are possible. Firecracker supports
vsock devices. The guest can connect to the host and vice-versa, even without a network interface. Very nice but
vsock is pretty low level and since the guest requires at least egress—Dockerfiles are full of package installation and pulling random stuff from the Internet—there was really no point going that way.
An alternative is a host service which the guest can connect to and fetch whatever is needed. I originally wanted a HTTP service but since there is a need of bi-directional communication without much protocol overhead, gRPC seems to be a better fit.
§kiss, keep it simply secure
I opted for the following:
firebuildwill start a bootstrap only gRPC server, one per
firebuildwill put the bootstrap endpoint in MMDS
- the guest will connect via
vminitto MMDS and discover the bootstrap endpoint
- the guest will connect to the gRPC service via
vminit bootstrap, download commands and resources and execute the bootstrap
What I wanted was that every connection is always TLS protected, even when the operator would not configure TLS for the bootstrap process. In fact, mutual TLS is preferred so I made a decision to never allow a non-TLS connection or insecure certificates.
- the CA chain and client certificate will be delivered via MMDS metadata
§the solution, embedded CA
No insecure certificates imply a certificate authority being available. I’ve written about certificate authorities before. Deploying something like Vault is not really difficult but during testing and development, considering the requirements, adds some friction.
I don’t like managing development dependency certificate files. I mean, I’ve done it but it’s always a bit messy. It requires extra tools, documentation and there are those pesky extra steps to follow in the readme, or
make steps to execute.
firebuild is written in Golang which has an awesome first class support for anything TLS/PKI/x509 related. Turns out a mini CA is less than 300 lines of code!
firebuild will use an embedded certificate authority. It’s lightweight and does only bare minimum to look like a CA but support a short-lived
rootfs build process. If
server cert and
server key are not provided, it does the following:
- on start, generate the root CA certificate
- optionally, when configured, generate an intermediate CA
- generate a server
*tls.Configwith a newly generated server certificate
- generate a client
*tls.Configwith a newly generated client certificate
- automatically configures the certificate and the client
*tls.Configto fulfill the gRPC server name requirement
Here’s how to use it:
With key sizes of
2048 bits, it takes a reasonable time to start, the overhead isn’t significant. Considering that
rootfs build is not very time sensitive, this seems pretty okay. Of course, there will be an option to use an already deployed CA instead of this one.