firebuild rootfs - gRPC with mTLS

§the problem

Currently, when a rootfs is built, the guest is started with an SSH server and the bootstrap process executes via an SSH connection. I don’t like this and want to replace the SSH method with an MMDS based solution. MMDS is already present in the firebuild run command.

run uses the vminit component from firebuild-mmds. When the guest starts, the vminit guest service connects to the MMDS endpoint, downloads the metadata and configures the VM. This is pretty similar to cloud-init but I don’t want cloud-init at this stage. Writing a cloud-init provider in Python is a bit of a head scratcher, can be done but maybe some other time.

rootfs bootstrap is pretty similar to run in the sense that it also starts a guest VM. There is no reason why it should not work the same way. Bye SSH, welcome MMDS, easy peasy. Not so… The big difference between run and rootfs is:

rootfs requires access to RUN commands and ADD / COPY resources present in the Docker artifact

Multiple approaches are possible. Firecracker supports vsock devices. The guest can connect to the host and vice-versa, even without a network interface. Very nice but vsock is pretty low level and since the guest requires at least egress—Dockerfiles are full of package installation and pulling random stuff from the Internet—there was really no point going that way.

An alternative is a host service which the guest can connect to and fetch whatever is needed. I originally wanted a HTTP service but since there is a need of bi-directional communication without much protocol overhead, gRPC seems to be a better fit.

§kiss, keep it simply secure

I opted for the following:

• firebuild will start a bootstrap only gRPC server, one per rootfs command run
• firebuild will put the bootstrap endpoint in MMDS
• the guest will connect via vminit to MMDS and discover the bootstrap endpoint
• the guest will connect to the gRPC service via vminit bootstrap, download commands and resources and execute the bootstrap

What I wanted was that every connection is always TLS protected, even when the operator would not configure TLS for the bootstrap process. In fact, mutual TLS is preferred so I made a decision to never allow a non-TLS connection or insecure certificates.

• the CA chain and client certificate will be delivered via MMDS metadata

§the solution, embedded CA

No insecure certificates imply a certificate authority being available. I’ve written about certificate authorities before^[1]. Deploying something like Vault is not really difficult but during testing and development, considering the requirements, adds some friction.

I don’t like managing development dependency certificate files. I mean, I’ve done it but it’s always a bit messy. It requires extra tools, documentation and there are those pesky extra steps to follow in the readme, or make steps to execute.

firebuild is written in Golang which has an awesome first class support for anything TLS/PKI/x509 related. Turns out a mini CA is less than 300 lines of code!

firebuild will use an embedded certificate authority. It’s lightweight and does only bare minimum to look like a CA but support a short-lived rootfs build process. If cacert, server cert and server key are not provided, it does the following:

• on start, generate the root CA certificate
• optionally, when configured, generate an intermediate CA
• generate a server *tls.Config with a newly generated server certificate
• generate a client *tls.Config with a newly generated client certificate
• automatically configures the certificate and the client *tls.Config to fulfill the gRPC server name requirement

Here’s how to use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

package main

import (
    "github.com/combust-labs/firebuild-embedded-ca/ca"
    "github.com/hashicorp/go-hclog"
	"google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
)

func main() {

    logger := hclog.Default()

    grpcServiceName := "grpc-service-name"

    grpcServerOptions := []grpc.ServerOption{}

    embeddedCA, embeddedCAErr := ca.NewDefaultEmbeddedCAWithLogger(&ca.EmbeddedCAConfig{
        Addresses: []string{grpcServiceName},
        KeySize:   4096,
    }, logger.Named("embdedded-ca"))
    if embeddedCAErr != nil {
        panic(embeddedCAErr)
    }

    serverTLSConfig, tlsConfigErr := embeddedCA.NewServerCertTLSConfig()
    if tlsConfigErr != nil {
        panic(tlsConfigErr)
    }

    clientTLSConfig, err := embeddedCA.NewClientCertTLSConfig(grpcServiceName)
    if err != nil {
        panic(embeddedCAErr)
    }

    grpcServerOptions = append(grpcServerOptions, grpc.Creds(credentials.NewTLS(serverTLSConfig)))

    listener, listenerErr := net.Listen("tcp", "127.0.0.1:0")
    if listenerErr != nil {
        panic(listenerErr)
    }

    grpcServer = grpc.NewServer(grpcServerOptions...)
    ///proto.Register...(grpcServer, ...)

    chanErr := make(chan struct{})
    go func() {
        if err := s.srv.Serve(listener); err != nil {
            logger.Error("failed grpc serve", "reason", err)
            close(chanErr)
        }
    }()

    grpcConn, _ := grpc.Dial(listener.Addr().String(),
		grpc.WithTransportCredentials(credentials.NewTLS(clientTLSConfig)))
    
    // ...

}

With key sizes of 2048 bits, it takes a reasonable time to start, the overhead isn’t significant. Considering that rootfs build is not very time sensitive, this seems pretty okay. Of course, there will be an option to use an already deployed CA instead of this one.

The embedded CA is available on GitHub^[2] under an Apache 2 license.