Multi-tenant Vault PKI with custom root PEM bundles

In the previous article^[1], I have investigated modern PKI software alternatives. One of the options on the list was HashiCorp Vault. The natural next step is to set up a Vault PKI.

This article documents setting up an imaginary multi-tenant Vault PKI with custom PEM bundles generated with OpenSSL. The steps the following:

create a root CA with OpenSSL
create intermediate CAs for imaginary clients with OpenSSL
using HashiCorp Vault in development mode:
- import custom bundle with root and intermediate certificates
- configure Vault roles
- issue a certificate

The method for generating the root and intermediate CAs comes from OpenSSL Certificate Authority guide written by Jamie Nguyen^[2]. I’m including the scripts and the configuration in this article for reference.

§The result

As an outcome of this article, the reader will be able to create a new root CA and add new intermediate CAs to existing root CAs with a single command. This command will also prepare the Vault PEM bundle. With the Vault bundle and four more shell commands, the user will have a ready to use certificate with a CA chain. The process will be:

run init-intermediate.sh <ca-name> <client-id> to have an intermediate CA
enable PKI for the new client with vault shell command
import a PEM bundle with vault shell command
add a permission with vault shell command
issue a certificate with vault shell command

§Create a root CA

I’m going to start by creating an environment file which will contain a location for the CA and distinguished name settings. As the root of the CA, I’m going to use ~/.ca directory.

1

mkdir -p ~/.ca

Now, I will create the environment file, ~/.ca/.article-ca:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


tee ~/.ca/.article-ca >/dev/null <<EOF
# DN defaults:
export CA_DN_DEFAULT_COUNTRY_CODE=DE
export CA_DN_DEFAULT_STATE_OR_PROVINCE="NRW"
export CA_DN_DEFAULT_LOCALITY="Herzogenrath"
export CA_DN_DEFAULT_ORG="klarrio.com"
export CA_DN_DEFAULT_ORG_UNIT=gmbh
export CA_DN_DEFAULT_EMAIL_ADDRESS="radek.gruchalski@klarrio.com"
export CA_DN_DEFAULT_COMMON_NAME="${CA_DN_DEFAULT_ORG_UNIT}.${CA_DN_DEFAULT_ORG}"
# Certificate settings:
export DEFAULT_BITS=2048
export DEFAULT_CA_DAYS=365
export DEFAULT_DAYS=90
export NS_CLIENT_CERT_COMMENT="OpenSSL Generated Server Certificate"
export NS_SERVER_CERT_COMMENT="OpenSSL Generated Server Certificate"
EOF

The next step is to create a shell program responsible for writing the CA root configuration. This, and the following intermediate, are by far the two longest bits of code in this article.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165


#!/usr/bin/env bash
set -eu

CA_NAME=${1}

if [ -f ${HOME}/.ca/.${CA_NAME} ]; then
    source ${HOME}/.ca/.${CA_NAME}
else
    echo ${HOME}/.ca/.${CA_NAME} not found
    exit 1
fi

CADIR=${HOME}/.ca/${CA_NAME}
mkdir -p ${CADIR} && cd ${CADIR}

# init subdirectories, index.txt and serial files:
mkdir certs crl newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial

# write the root CA config:
tee openssl.cnf >/dev/null <<EOF
# OpenSSL root CA configuration file.

[ ca ]
# man ca
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = ${CADIR}
certs             = \$dir/certs
crl_dir           = \$dir/crl
new_certs_dir     = \$dir/newcerts
database          = \$dir/index.txt
serial            = \$dir/serial
RANDFILE          = \$dir/private/.rand

# The root key and root certificate.
private_key       = \$dir/private/ca.key.pem
certificate       = \$dir/certs/ca.cert.pem

# For certificate revocation lists.
crlnumber         = \$dir/crlnumber
crl               = \$dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = ${DEFAULT_DAYS}
preserve          = no
policy            = policy_strict

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of 'man ca'.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the 'ca' man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the 'req' tool ('man req').
default_bits        = ${DEFAULT_BITS}
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = ${CA_DN_DEFAULT_COUNTRY_CODE}
stateOrProvinceName_default     = "${CA_DN_DEFAULT_STATE_OR_PROVINCE}"
localityName_default            = "${CA_DN_DEFAULT_LOCALITY}"
0.organizationName_default      = "${CA_DN_DEFAULT_ORG}"
organizationalUnitName_default  = "${CA_DN_DEFAULT_ORG_UNIT}"
emailAddress_default            = "${CA_DN_DEFAULT_EMAIL_ADDRESS}"

[ v3_ca ]
# Extensions for a typical CA ('man x509v3_config').
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA ('man x509v3_config').
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates ('man x509v3_config').
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "${NS_CLIENT_CERT_COMMENT}"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates ('man x509v3_config').
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "${NS_SERVER_CERT_COMMENT}"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs ('man x509v3_config').
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates ('man ocsp').
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSignings
EOF

# generate root CA:
openssl genrsa -aes256 -out ${CADIR}/private/ca.key.pem 4096
chmod 400 ${CADIR}/private/ca.key.pem
openssl req -config ${CADIR}/openssl.cnf \
      -key ${CADIR}/private/ca.key.pem \
      -new -x509 -days ${DEFAULT_CA_DAYS} -sha256 -extensions v3_ca \
      -out ${CADIR}/certs/ca.cert.pem
chmod 444 ${CADIR}/certs/ca.cert.pem
openssl x509 -noout -text -in ${CADIR}/certs/ca.cert.pem

After saving the program as ~/.ca/init-ca.sh, making it executable with chmod +x ~/.ca/init-ca.sh and running as ~/.ca/init-ca.sh article-ca, the new CA has been created in ~/.ca/article-ca. Using this method and changing the CA name given to the program as the first argument, more root CAs can be created.

The program will ask three times for the CA private key passphrase: to create the key, to verify and to use the key. Next, it will ask to confirm default values for the distinguished name. When all values are confirmed and correct, the CA root certificate will be generated.

The output will be similar to:

Generating RSA private key, 4096 bit long modulus
..........................++
...........................................................................................................................................................................................++
e is 65537 (0x10001)
Enter pass phrase for /Users/rad/.ca/article-ca/private/ca.key.pem:
Verifying - Enter pass phrase for /Users/rad/.ca/article-ca/private/ca.key.pem:
Enter pass phrase for /Users/rad/.ca/article-ca/private/ca.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [DE]:
State or Province Name [NRW]:
Locality Name [Herzogenrath]:
Organization Name [klarrio.com]:
Organizational Unit Name [gmbh]:
Common Name []:
Email Address [radek.gruchalski@klarrio.com]:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 14292483172117400839 (0xc65919b8604e4d07)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=DE, ST=NRW, L=Herzogenrath, O=klarrio.com, OU=gmbh/emailAddress=radek.gruchalski@klarrio.com
        Validity
            Not Before: Sep  8 20:53:48 2020 GMT
            Not After : Sep  8 20:53:48 2021 GMT
        Subject: C=DE, ST=NRW, L=Herzogenrath, O=klarrio.com, OU=gmbh/emailAddress=radek.gruchalski@klarrio.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:ab:bf:02:ef:72:b3:ce:ac:4b:37:01:1b:57:fc:
                    ...
                    0d:84:73:28:32:e8:f1:99:64:ee:f5:b2:6f:21:7f:
                    9e:08:21
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                06:DE:59:CD:9A:21:9F:A0:B2:32:EE:B3:E6:89:11:10:9C:6E:83:0F
            X509v3 Authority Key Identifier:
                keyid:06:DE:59:CD:9A:21:9F:A0:B2:32:EE:B3:E6:89:11:10:9C:6E:83:0F

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         35:30:29:2c:39:1b:57:81:d8:95:9a:26:1f:5c:44:62:65:ca:
         ...
         fe:28:53:b1:76:63:22:cd

on disk, there should be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


[rad] ~ $ tree ~/.ca/article-ca/
/Users/rad/.ca/article-ca/
├── certs
│   └── ca.cert.pem
├── crl
├── index.txt
├── newcerts
├── openssl.cnf
├── private
│   └── ca.key.pem
└── serial

4 directories, 5 files

§Create an intermediate CA

The intermediate CA reuses the environment file of the root CA. The program to create it is very similar to the init-ca.sh.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195


#!/usr/bin/env bash
set -eu

CA_NAME=${1}
INTERMEDIATE_NAME=${2}

if [ -f ${HOME}/.ca/.${CA_NAME} ]; then
    source ${HOME}/.ca/.${CA_NAME}
else
    echo ${HOME}/.ca/.${CA_NAME} not found
    exit 1
fi

CADIR=${HOME}/.ca/${CA_NAME}
INTERMEDIATEDIR=${CADIR}/intermediate/${INTERMEDIATE_NAME}

mkdir -p ${INTERMEDIATEDIR} && cd ${INTERMEDIATEDIR}

# init subdirectories, index.txt and serial files:
mkdir certs chain crl csr newcerts private vault-bundle
chmod 700 private
touch index.txt
echo 1000 > serial
echo 1000 > crlnumber

# write the intermediate config:
tee openssl.cnf >/dev/null <<EOF
# OpenSSL intermediate CA configuration file.

[ ca ]
# 'man ca'
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = ${INTERMEDIATEDIR}
certs             = \$dir/certs
crl_dir           = \$dir/crl
new_certs_dir     = \$dir/newcerts
database          = \$dir/index.txt
serial            = \$dir/serial
RANDFILE          = \$dir/private/.rand

# The root key and root certificate.
private_key       = \$dir/private/intermediate.key.pem
certificate       = \$dir/certs/intermediate.cert.pem

# For certificate revocation lists.
crlnumber         = \$dir/crlnumber
crl               = \$dir/crl/intermediate.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = ${DEFAULT_DAYS}
preserve          = no
policy            = policy_loose

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of 'man ca'.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the 'ca' man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the 'req' tool ('man req').
default_bits        = ${DEFAULT_BITS}
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = ${CA_DN_DEFAULT_COUNTRY_CODE}
stateOrProvinceName_default     = "${CA_DN_DEFAULT_STATE_OR_PROVINCE}"
localityName_default            = "${CA_DN_DEFAULT_LOCALITY}"
0.organizationName_default      = "${CA_DN_DEFAULT_ORG}"
organizationalUnitName_default  = "${CA_DN_DEFAULT_ORG_UNIT}"
commonName_default              = "${INTERMEDIATE_NAME}.${CA_DN_DEFAULT_COMMON_NAME}"
emailAddress_default            = "${CA_DN_DEFAULT_EMAIL_ADDRESS}"

[ v3_ca ]
# Extensions for a typical CA ('man x509v3_config').
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA ('man x509v3_config').
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates ('man x509v3_config').
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "${NS_CLIENT_CERT_COMMENT}"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates ('man x509v3_config').
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "${NS_SERVER_CERT_COMMENT}"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs ('man x509v3_config').
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates ('man ocsp').
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSigning
EOF

# generate intermediate CA certificate for ${CA_NAME}/${INTERMEDIATE_NAME} in ${INTERMEDIATEDIR}..."
openssl genrsa -aes256 \
    -out ${INTERMEDIATEDIR}/private/intermediate.key.pem 4096
chmod 400 ${INTERMEDIATEDIR}/private/intermediate.key.pem
openssl req -config ${INTERMEDIATEDIR}/openssl.cnf -new -sha256 \
    -key ${INTERMEDIATEDIR}/private/intermediate.key.pem \
    -out ${INTERMEDIATEDIR}/csr/intermediate.csr.pem
openssl ca -config ${CADIR}/openssl.cnf -extensions v3_intermediate_ca \
    -days ${DEFAULT_CA_DAYS} -notext -md sha256 \
    -in ${INTERMEDIATEDIR}/csr/intermediate.csr.pem \
    -out ${INTERMEDIATEDIR}/certs/intermediate.cert.pem
chmod 444 ${INTERMEDIATEDIR}/certs/intermediate.cert.pem
openssl x509 -noout -text \
    -in ${INTERMEDIATEDIR}/certs/intermediate.cert.pem
openssl verify -CAfile ${CADIR}/certs/ca.cert.pem \
    ${INTERMEDIATEDIR}/certs/intermediate.cert.pem

# generate the chain:
cat ${INTERMEDIATEDIR}/certs/intermediate.cert.pem \
      ${CADIR}/certs/ca.cert.pem > ${INTERMEDIATEDIR}/chain/ca-chain.cert.pem
chmod 444 ${INTERMEDIATEDIR}/certs/ca-chain.cert.pem

# convert intermediate CA key to PKCS1 format required by Vault bundle
openssl rsa -in ${INTERMEDIATEDIR}/private/intermediate.key.pem \
    -out ${INTERMEDIATEDIR}/vault-bundle/intermediate-pkcs1.key.pem \
    -outform pem

# generate the actual Vault bundle:
cat ${INTERMEDIATEDIR}/chain/ca-chain.cert.pem \
    ${INTERMEDIATEDIR}/vault-bundle/intermediate-pkcs1.key.pem > \
    ${INTERMEDIATEDIR}/vault-bundle/bundle.pem

# remove the pkcs1 file:
rm ${INTERMEDIATEDIR}/vault-bundle/intermediate-pkcs1.key.pem

After saving as ~/.ca/init-intermediate.sh, making executable with chmod +x ~/.ca/init-intermediate.sh, it can be executed. I’m going to create two intermediate CAs immediately. In my case, mirroring the imaginary use case for setting up Keycloak with multiple clients^[3], I have:

ClientA ~/.ca/init-intermediate.sh article-ca client-a
ClientB ~/.ca/init-intermediate.sh article-ca client-b

In each case, the program will ask the following:

the intermediate CA key password, three times: to create the key, verify the password and use the key
verify to confirm distinguished name defaults, additionally a common name
root CA key password during intermediate signing
a couple of confirmations
once again a passphrase of the intermediate CA key for pkcs1 conversion

After these two commands are finished, the files on disk look similar to:

[rad] ~ $ tree ~/.ca/article-ca/
/Users/rad/.ca/article-ca/
├── certs
│   └── ca.cert.pem
├── crl
├── index.txt
├── index.txt.attr
├── index.txt.attr.old
├── index.txt.old
├── intermediate
│   ├── client-a
│   │   ├── certs
│   │   │   └── intermediate.cert.pem
│   │   ├── chain
│   │   │   └── ca-chain.cert.pem
│   │   ├── crl
│   │   ├── crlnumber
│   │   ├── csr
│   │   │   └── intermediate.csr.pem
│   │   ├── index.txt
│   │   ├── newcerts
│   │   ├── openssl.cnf
│   │   ├── private
│   │   │   └── intermediate.key.pem
│   │   ├── serial
│   │   └── vault-bundle
│   │       └── bundle.pem
│   └── client-b
│       ├── certs
│       │   └── intermediate.cert.pem
│       ├── chain
│       │   └── ca-chain.cert.pem
│       ├── crl
│       ├── crlnumber
│       ├── csr
│       │   └── intermediate.csr.pem
│       ├── index.txt
│       ├── newcerts
│       ├── openssl.cnf
│       ├── private
│       │   └── intermediate.key.pem
│       ├── serial
│       └── vault-bundle
│           └── bundle.pem
├── newcerts
│   ├── 1000.pem
│   └── 1001.pem
├── openssl.cnf
├── private
│   └── ca.key.pem
├── serial
└── serial.old

21 directories, 29 files

The output of the intermediate CA init, for one of the executions, is similar to:

Generating RSA private key, 4096 bit long modulus
...........................................++
.....................................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
Enter pass phrase for /Users/rad/.ca/article-ca/intermediate/client-a/private/intermediate.key.pem:
Verifying - Enter pass phrase for /Users/rad/.ca/article-ca/intermediate/client-a/private/intermediate.key.pem:
Enter pass phrase for /Users/rad/.ca/article-ca/intermediate/client-a/private/intermediate.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [DE]:
State or Province Name [NRW]:
Locality Name [Herzogenrath]:
Organization Name [klarrio.com]:
Organizational Unit Name [gmbh]:
Common Name [client-a.gmbh.klarrio.com]:
Email Address [radek.gruchalski@klarrio.com]:
Using configuration from /Users/rad/.ca/article-ca/openssl.cnf
Enter pass phrase for /Users/rad/.ca/article-ca/private/ca.key.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Sep  8 21:05:08 2020 GMT
            Not After : Sep  8 21:05:08 2021 GMT
        Subject:
            countryName               = DE
            stateOrProvinceName       = NRW
            organizationName          = klarrio.com
            organizationalUnitName    = gmbh
            commonName                = client-a.gmbh.klarrio.com
            emailAddress              = radek.gruchalski@klarrio.com
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                47:7D:8A:9E:DC:23:03:7A:AA:E4:79:A8:98:EE:40:54:01:84:1C:E8
            X509v3 Authority Key Identifier:
                keyid:06:DE:59:CD:9A:21:9F:A0:B2:32:EE:B3:E6:89:11:10:9C:6E:83:0F

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
Certificate is to be certified until Sep  8 21:05:08 2021 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=DE, ST=NRW, L=Herzogenrath, O=klarrio.com, OU=gmbh/emailAddress=radek.gruchalski@klarrio.com
        Validity
            Not Before: Sep  9 21:05:08 2020 GMT
            Not After : Sep  9 21:05:08 2021 GMT
        Subject: C=DE, ST=NRW, O=klarrio.com, OU=gmbh, CN=client-a.gmbh.klarrio.com/emailAddress=radek.gruchalski@klarrio.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:aa:c3:eb:7e:c4:2a:79:2e:bc:7b:6f:c2:61:f9:
                    ...
                    80:27:da:c6:b8:ff:20:3b:7f:9b:fd:ff:15:d0:2c:
                    e7:2b:03
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                47:7D:8A:9E:DC:23:03:7A:AA:E4:79:A8:98:EE:40:54:01:84:1C:E8
            X509v3 Authority Key Identifier:
                keyid:06:DE:59:CD:9A:21:9F:A0:B2:32:EE:B3:E6:89:11:10:9C:6E:83:0F

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         22:b5:e3:6e:a5:d6:0d:6b:30:65:a3:d9:68:27:38:2b:ea:64:
         ...
         39:ac:c3:66:88:8c:f0:eb
/Users/rad/.ca/article-ca/intermediate/client-a/certs/intermediate.cert.pem: OK
Enter pass phrase for /Users/rad/.ca/article-ca/intermediate/client-a/private/intermediate.key.pem:
writing RSA key

At this stage, I have a root CA and two intermediate CA for respective common names:

ClientA: client-a.gmbh.klarrio.com
ClientB: client-b.gmbh.klarrio.com

For every intermediate, I have a PEM bundle ready to be imported to HashiCorp Vault.

§Start Vault in development mode

I’m going to use Vault development mode, with VAULT_DEV_ROOT_TOKEN_ID hard coded and default port 8200 exposed so it can be reached from localhost.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


docker run --rm \
    -p 8200:8200 \
    -ti \
    -e 'VAULT_DEV_ROOT_TOKEN_ID=dev-token' \
    -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
    --cap-add=IPC_LOCK \
    vault
...
2020-09-08T21:08:07.855Z [INFO]  expiration: revoked lease: lease_id=auth/token/root/h128b471d9329710fbbd34dc5a4a98b4d0c7fe104799e818447a013762aed1e66
2020-09-08T21:08:07.859Z [INFO]  core: successful mount: namespace= path=secret/ type=kv
2020-09-08T21:08:07.869Z [INFO]  secrets.kv.kv_9795fd99: collecting keys to upgrade
2020-09-08T21:08:07.869Z [INFO]  secrets.kv.kv_9795fd99: done collecting keys: num_keys=1
2020-09-08T21:08:07.869Z [INFO]  secrets.kv.kv_9795fd99: upgrading keys finished

Now, in another terminal window, I am going to export the token and Vault address environment variables:

1
2


export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=dev-token

Finally, I can enable PKIs:

1
2


vault secrets enable -path=client-a pki
vault secrets enable -path=client-b pki

I can see Vault confirming:

2020-09-08T21:11:32.133Z [INFO]  core: successful mount: namespace= path=client-a/ type=pki
2020-09-08T21:11:32.914Z [INFO]  core: successful mount: namespace= path=client-b/ type=pki

§Importing the bundles

The next step is to import the PEM bundles, they were prepared during the intermediate CA creation:

1
2
3
4


[rad] ~ $ vault write client-a/config/ca pem_bundle=@${HOME}/.ca/article-ca/intermediate/client-a/vault-bundle/bundle.pem
Success! Data written to: client-a/config/ca
[rad] ~ $ vault write client-b/config/ca pem_bundle=@${HOME}/.ca/article-ca/intermediate/client-b/vault-bundle/bundle.pem
Success! Data written to: client-b/config/ca

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


[rad] ~ $ vault write client-a/roles/system-of-client-a-com \
    allow_localhost=true \
    allow_bare_domains=true \
    allowed_domains="localhost,client-a.gmbh.klarrio.com" \
    allow_subdomains=true \
    max_ttl="720h"
Success! Data written to: client-a/roles/system-of-client-a-com
[rad] ~ $ vault write client-b/roles/system-of-client-b-com \
    allow_localhost=true \
    allow_bare_domains=true \
    allowed_domains="localhost,client-b.gmbh.klarrio.com" \
    allow_subdomains=true \
    max_ttl="720h"
Success! Data written to: client-b/roles/system-of-client-b-com

§Notes to the examples above

By setting allow_bare_domains to true, I allow the user to generate a certificate for any literal domain specified in allowed_domains, so the user can issue a certificate for client-[X].gmbh.klarrio.com.
By setting allow_localhost to true, I allow issuing a certificate for localhost, useful for testing.
By setting allow_subdomains to true, I give the user the ability to issue certificates with common names that are subdomains of allowed_domains.

All the interesting options are documented in the Vault documentation^[4].

§Requesting certificates

To request a certificate, simply issue the following command:

1
2
3
4


vault write client-a/issue/system-of-client-a-com \
    common_name="become.client-a.gmbh.klarrio.com" \
    ttl="240h" \
    format=pem

To which the output is … pretty verbose …:

Key                 Value
---                 -----
ca_chain            [-----BEGIN CERTIFICATE-----
MIIF+zCCA+OgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgYQxCzAJBgNVBAYTAkRF
MQwwCgYDVQQIDANOUlcxFTATBgNVBAcMDEhlcnpvZ2VucmF0aDEUMBIGA1UECgwL
...
KhlaTNBfyYaIYXeTQgCa+ar7OcZQhKMvPv1dTOFtZQ8Rjk5+8Y0yOazDZoiM8Os=
-----END CERTIFICATE----- -----BEGIN CERTIFICATE-----
MIIF8DCCA9igAwIBAgIJAMZZGbhgTk0HMA0GCSqGSIb3DQEBCwUAMIGEMQswCQYD
...
kY3o7PdJs57rbyjVo3UWLZwQbqBkMpH3zxjKLco8lE3UscEU7Up/ZxqQF9i3Wxj1
bahjbSfvI/V5gKaVkfcp/Pe1wavMFSI0GueEzP4oU7F2YyLN
-----END CERTIFICATE-----]
certificate         -----BEGIN CERTIFICATE-----
MIIE8TCCAtmgAwIBAgIUZWaMXolFNt/ufzgUOnvu79ZCVjEwDQYJKoZIhvcNAQEL
BQAwgZMxCzAJBgNVBAYTAkRFMQwwCgYDVQQIDANOUlcxFDASBgNVBAoMC2tsYXJy
...
nJOv5KyaSz8OCKdX+JlVmU9Qoapj4EyXOZQ+LS8RBsFXrbjpjqxdd3kpiEhpcQwN
7UXijzXX25gZKwFPTof+VdAKa5M1/uU9G15KwL4S/vJwYGwL/zo799qrGdd/+plV
cOd4GA883CW0DdC8QuU2+w3HIRS6
-----END CERTIFICATE-----
expiration          1600616181
issuing_ca          -----BEGIN CERTIFICATE-----
MIIF+zCCA+OgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgYQxCzAJBgNVBAYTAkRF
MQwwCgYDVQQIDANOUlcxFTATBgNVBAcMDEhlcnpvZ2VucmF0aDEUMBIGA1UECgwL
a2xhcnJpby5jb20xDTALBgNVBAsMBGdtYmgxKzApBgkqhkiG9w0BCQEWHHJhZGVr
...
pIjI/wXZwURNn920ODhsA0v467Llw4gMTTjxOfDIFrdZ46936IMzHNOXI5b87dfU
KhlaTNBfyYaIYXeTQgCa+ar7OcZQhKMvPv1dTOFtZQ8Rjk5+8Y0yOazDZoiM8Os=
-----END CERTIFICATE-----
private_key         -----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA57Ww9OC6UzHaLAALdbotEoTf5c2qw4BufVlOB7zZh+GbX2hR
...
zcDV96GoXPEnuXHdVsfePFIdPS7IRmAEn72UW6u39mVqGJuX1/5tk76ay6cPHP/B
xtX2UAplk9bD046wNh1/PxudBnqavOFf4F5uDizYhyihbmmhS/mcxA==
-----END RSA PRIVATE KEY-----
private_key_type    rsa
serial_number       65:66:8c:5e:89:45:36:df:ee:7f:38:14:3a:7b:ee:ef:d6:42:56:31

Cool, we have the certificate, a private key and a certificate chain. The CA chain can now be copied to chain.pem file.

CA chain

Pay attention to the -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- bit. It needs to be saved as:

...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...

Copy the certificate to certificate.pem:

certificate

And the private to key.pem:

private key

A server with TLS transport can now be created using the chain.pem, certificate.pem and key.pem files.

§Why is this cool

It’s not possible to request certificates for domain names not in allowed_domains:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


[rad] ~ $ vault write client-a/issue/system-of-client-a-com \
    common_name="become.client-b.gmbh.klarrio.com" \
    ttl="240h" \
    format=pem
Error writing data to client-a/issue/system-of-client-a-com: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/client-a/issue/system-of-client-a-com
Code: 400. Errors:

* common name become.client-b.gmbh.klarrio.com not allowed by this role

It’s possible to provide alt names:

1
2
3
4
5


vault write client-b/issue/system-of-client-b-com \
    common_name="get-in-touch.client-b.gmbh.klarrio.com" \
    alt_names="you.client-b.gmbh.klarrio.com,or-you.client-b.gmbh.klarrio.com" \
    ttl="720h" \
    format=pem

Certificates generated for ClientA do not validate for ClientB and vice versa - different intermediate
It is very easy to define a policy with different criteria. For example: to allow a user of ClientB to generate a certificate for specific domain only:

1
2
3
4


[rad] ~ $ vault write client-a/roles/get-in-touch \
    allow_bare_domains=true \
    allowed_domains="get-in-touch.client-a.gmbh.klarrio.com" \
    max_ttl="1h"

so that it’s possible to:

1
2
3
4


vault write client-a/issue/get-in-touch \
    common_name="get-in-touch.client-a.gmbh.klarrio.com" \
    ttl="1h" \
    format=pem

but not:

1
2
3
4


vault write client-a/issue/get-in-touch \
    common_name="maybe.get-in-touch.client-a.gmbh.klarrio.com" \
    ttl="1h" \
    format=pem

§Conclusion

Once the initial setting up of the program to create the CAs is finished, the rest of the process is straightforward. Adding new clients (intermediates) is fairly easy. Configuring Vault is not a big job. Vault PKI is a solid choice for a multi-tenant PKI solution.

Of course, there are still some challenges left on the table. Mainly storing the root and intermediate certificates safely and properly ensuring who can access them.