CertAutoPilot OS with an HSM
CertAutoPilot encrypts its secrets with a key-encryption key (KEK). By default
the key lives in each node's secrets.env. With an HSM it is created and kept
inside the HSM, and the appliance only ever asks the HSM to encrypt and decrypt.
This page sets up a three-node cluster with an HSM. Every step is a console command on one node; the screenshots are from a real cluster.
| Step | Where | What you do |
|---|---|---|
| 1 | the HSM | create the partition or app; get the client files, the token label, the PIN |
| 2 | node 1 | install the client, form the cluster with the HSM, make a join token |
| 3 | node 2, node 3 | install the client, join |
| 4 | node 1 | check the cluster |
| 5 | node 1 | rotating the key, whenever you need it |
Before you start
- Decide now. The HSM is chosen when the cluster is formed. A cluster formed without one does not move to an HSM later: that is a new cluster plus a data re-import (changing the KEK provider).
- Supported HSMs. Thales Luna Cloud HSM and Luna Network HSM, in FIPS mode or not; Fortanix DSM. A Luna partition in FIPS mode generates the AES-GCM IV itself; CertAutoPilot settles that on its own and nothing is set for it. AWS CloudHSM is not supported.
- Every node. The client is installed on every node, before
cluster initon the first and beforecluster joinon the others. Give each node its own client identity. - Network. Each node reaches the HSM: Luna Cloud HSM
<region>.hsm.dpondemand.io:443, Luna Network HSM its NTLS port1792, Fortanix<region>.smartkey.io:443. - Appliance. Every node has its static address (
set network static …,apply,network confirm) and a synchronised clock, as for any cluster. CertAutoPilot OS 0.2.4 or later.
1. Prepare the HSM
Whatever the HSM, you end this step with three things: the client files for each node, the token label and the PIN.
- Luna Cloud HSM (DPoD)
- Luna Network HSM
- Fortanix DSM
-
In Thales Data Protection on Demand, add a Luna Cloud HSM service. Remove FIPS restrictions is your choice: CertAutoPilot works with it checked or unchecked. The region cannot be changed later.
-
On the service, Create Service Client once per node (
cap-1,cap-2,cap-3) and download each zip. Keep them as secrets: each holds that client's credentials. -
Initialize the partition once, from any Linux machine, with the
lunacminside one of the zips:$ unzip setup-cap-1.zip -d dpod && cd dpod && tar -xf cvclient-min.tar$ source ./setenv && ./bin/64/lunacmlunacm:> partition init -label cap-prod # Partition SO password, then a cloning domainlunacm:> role login -name po # the Partition SO passwordlunacm:> role init -name co # the Crypto Officer password: the PINlunacm:> role logoutlunacm:> role login -name co # if asked, 'role changepw -name co'lunacm:> exitPasswords are at least 7 characters. Keep the SO password and the cloning domain: Luna needs them to manage and back up the partition; CertAutoPilot never uses them.
| You now have | |
|---|---|
| Client files | one zip per node (setup-cap-1.zip …) |
| Token label | the partition label (cap-prod) |
| PIN | the Crypto Officer password |
On a Linux machine with the Luna Client installed:
- Make one client identity per node:
vtl createCert -n cap-1(andcap-2,cap-3). - On the Luna appliance, register each one and assign the partition:
client register -client cap-1 -ip 10.0.0.11, thenclient assignPartition -client cap-1 -partition cap-prod. - Add the appliance to the client and check:
vtl addServer -n <hsm> -c <server cert>,vtl verify. - Pack, per node,
Chrystoki.confand every file it names into one.tar.gz(or.zip): the client librarylibCryptoki2_64.so, the node's certificate and key (cap-1.pem,cap-1Key.pem) and the server'sCAFile.pem. The paths inChrystoki.confmay stay asvtlwrote them.
| You now have | |
|---|---|
| Client files | one bundle per node (cap-1-luna.tar.gz …) |
| Token label | the partition label (or the HA group's) |
| PIN | the partition's Crypto Officer password |
-
In the DSM console, add a Group (
cap-hsm) and in it an App with the PKCS#11 interface and API Key authentication. On the app's permissions for the group, turn Export off. -
Download the Linux PKCS#11 library (
fortanix_pkcs11.so) from Fortanix's PKCS#11 download page; check its SHA-256 there. -
Write
pkcs11.confwith your account's region:api_endpoint = "https://eu.smartkey.io"
| You now have | |
|---|---|
| Client files | fortanix_pkcs11.so and pkcs11.conf (the same for every node) |
| Token label | Fortanix Token |
| PIN | the app's API key (DSM → Apps → the app → View API key details) |
2. Node 1 — install the client and form the cluster
Copy the client files to the node, from your workstation:
- Luna Cloud HSM (DPoD)
- Luna Network HSM
- Fortanix DSM
$ scp setup-cap-1.zip admin@10.0.0.11:
$ scp cap-1-luna.tar.gz admin@10.0.0.11:
$ scp fortanix_pkcs11.so pkcs11.conf admin@10.0.0.11:
Install the client. The command prints the module path; the cluster wizard offers it as its default.
- Luna Cloud HSM (DPoD)
- Luna Network HSM
- Fortanix DSM
cap> import pkcs11-client /home/admin/cap-1-luna.tar.gz
The same output as for Luna Cloud HSM, with luna in place of dpod.
cap> import pkcs11-module /home/admin/fortanix_pkcs11.so
cap> import pkcs11-config /home/admin/pkcs11.conf
Check that the node reaches the HSM. diagnose hsm loads the module the
way the backend will and lists the tokens it sees; your token label must be
among them.
Form the cluster with the HSM. In setup cluster answer init, the
virtual address (or Enter for none), y for the HSM, Enter for the module
path, then the token label and the PIN (not shown as you type).
The cluster forms in the background, in a minute or two. While it forms, the
HSM creates the first key, certautopilot-kek-v1. show cluster shows the
result:
Make a join token for the next node. Each token is for one node:
3. Node 2 and node 3 — install the client and join
On node 2, with node 2's own client files:
-
Copy and install the client exactly as on node 1 (
scp, thenimport …), and rundiagnose hsm: the token label must be listed. -
Join with the token from node 1. The node takes the HSM settings and the PIN from the leader; there is nothing more to type:
-
Check that the node logs in to the HSM with the cluster's PIN:
On node 3: cluster token new on node 1 again, then the same three steps
with node 3's client files.
4. Check the cluster
On node 1, show cluster lists the three members:
diagnose hsm on any node shows that node's login to the HSM.
5. Rotating the key
Run these on node 1 (the leader). Each one covers every node.
Create the next key. kek add 2 creates certautopilot-kek-v2 inside the
HSM, restarts every backend and waits until each one has loaded it:
Re-encrypt the data from version 1 to version 2:
Follow it with kek status until version 2 is active and version 1 holds
no record:
IV source says who generates the AES-GCM IV of each wrapped key: host
(CertAutoPilot) or hsm. A Luna partition in FIPS mode shows hsm.
Retire the old key once you no longer need to restore a backup made before the rotation:
- Each node switches its writes to the new key within about 30 seconds of
the rotation's end. If
kek remove 1reports records still on version 1, runkek resweep 1, thenkek remove 1again. kek removedoes not delete the key object inside the HSM. Delete it with the HSM's own tools once no backup you may restore still needs it.
When something goes wrong
| What you see | What it means | What to do |
|---|---|---|
cluster join: this node has no PKCS#11 module at … | The client is not installed on this node. Nothing was changed. | Step 1 of node 2/3 on this node, then cluster join again. |
diagnose hsm: C_Initialize: 0x… | The client cannot read its configuration or cannot reach the HSM. | Re-import the client files; check the node's route and firewall to the HSM. |
diagnose hsm: no token labelled '…' | The label the cluster was formed with is not on the HSM. | Luna: the partition label. Fortanix: Fortanix Token. |
diagnose hsm: login with the cluster PIN: 0xa0 | Wrong PIN. | Luna: the Crypto Officer password. Fortanix: the app's API key. |
diagnose hsm: CKM_AES_GCM NOT offered | The HSM cannot do what CertAutoPilot needs. | Use a supported HSM. |
setup cluster finished, but show cluster stays empty | Forming the cluster failed; it was undone. | logs agent names the reason; usually the label, the PIN or the HSM's reachability. Fix it, then setup cluster again. |
cluster join: the token was revoked / has expired | The token is no longer valid. | cluster token new on node 1, then join with the new token. |
kek remove: version 1 is still referenced by N record(s) | Records written while the nodes were switching to the new key. | kek resweep 1, then kek remove 1. |
Reference
Placing a client by hand
The import commands are a convenience. From shell (sudo), a client can
be placed anywhere when three rules hold:
- The backend runs as
certautopilotand must read the files:root:certautopilot 0640, libraries0750, every directory on the way traversable. - The client's variables go in
/etc/cap-pkcs11/env, oneVAR=valueper line (Luna:ChrystokiConfigurationPath=<the directory of Chrystoki.conf>). The backend, the cluster agent and the console'skekcommands read it. - What the client writes goes under
/var/lib/cap-pkcs11, the one HSM directory the backend may write (Luna:MutexFolder).
A Luna Cloud HSM client placed by hand under /opt/luna:
$ sudo mkdir -p /opt/luna && cd /opt/luna
$ sudo python3 -m zipfile -e ~/setup-cap-1.zip pkg
$ sudo tar -xf pkg/cvclient-min.tar && sudo cp pkg/Chrystoki.conf pkg/*.pem . && sudo rm -rf pkg
$ sudo bash -c 'source ./setenv'
$ sudo sed -i 's|^\( *MutexFolder *=\).*|\1 /var/lib/cap-pkcs11/luna;|' Chrystoki.conf
$ sudo install -d -m 2770 -o root -g certautopilot /var/lib/cap-pkcs11/luna
$ sudo chown -R root:certautopilot /opt/luna && sudo chmod -R g+rX,o-rwx /opt/luna
$ echo ChrystokiConfigurationPath=/opt/luna | sudo tee /etc/cap-pkcs11/env
The module path is then /opt/luna/libs/64/libCryptoki2.so; type it in
setup cluster. diagnose hsm finds the client through the variable. The
application's CLI run from the shell (sudo cap kek …) does not load the
variables; use the console's kek commands.
Fortanix: the API key inside pkcs11.conf
Instead of entering the API key as the PIN, add api_key = "…" to
pkcs11.conf before importing it and give file:///etc/fortanix/pkcs11.conf
as the PIN. The key then lives only in that file, and replacing it is one
import pkcs11-config per node. The file does not travel with a join: import
it on every node.
Answer files and the OVA
An answer file (cluster.encryption in /etc/cap/setup.yaml.example) or the
OVA's cap.cluster.kek_provider is applied at first boot, before a client can
be imported; only the image's SoftHSM module (a lab) is there then. For a
vendor HSM, deploy with cluster role none and follow this page from step 2.
Console commands and their standalone equivalents
| Standalone / PKCS#11 pages | CertAutoPilot OS console |
|---|---|
install the vendor client, copy the .so | import pkcs11-module <file> / import pkcs11-client <file> |
write the client config, chown, chmod | import pkcs11-config <file> (Fortanix) / part of import pkcs11-client (Luna) |
pkcs11-tool -L, vtl verify | diagnose hsm |
get.sh … --kek-provider=pkcs11 … | setup cluster → init → HSM y |
certautopilot kek pkcs11-init --version=N + a restart per node | kek add N |
certautopilot kek verify --target=N | kek verify N |
certautopilot kek rotate --from-version=A --to-version=B | kek rotate A B |
certautopilot kek status | kek status / show kek |
certautopilot kek remove --version=N | kek remove N |