Skip to main content

CertAutoPilot OS with an HSM

CertAutoPilot encrypts its secrets with a key-encryption key (KEK). By default the key lives in each node's secrets.env. With an HSM it is created and kept inside the HSM, and the appliance only ever asks the HSM to encrypt and decrypt.

This page sets up a three-node cluster with an HSM. Every step is a console command on one node; the screenshots are from a real cluster.

StepWhereWhat you do
1the HSMcreate the partition or app; get the client files, the token label, the PIN
2node 1install the client, form the cluster with the HSM, make a join token
3node 2, node 3install the client, join
4node 1check the cluster
5node 1rotating the key, whenever you need it

Before you start​

  • Decide now. The HSM is chosen when the cluster is formed. A cluster formed without one does not move to an HSM later: that is a new cluster plus a data re-import (changing the KEK provider).
  • Supported HSMs. Thales Luna Cloud HSM and Luna Network HSM, in FIPS mode or not; Fortanix DSM. A Luna partition in FIPS mode generates the AES-GCM IV itself; CertAutoPilot settles that on its own and nothing is set for it. AWS CloudHSM is not supported.
  • Every node. The client is installed on every node, before cluster init on the first and before cluster join on the others. Give each node its own client identity.
  • Network. Each node reaches the HSM: Luna Cloud HSM <region>.hsm.dpondemand.io:443, Luna Network HSM its NTLS port 1792, Fortanix <region>.smartkey.io:443.
  • Appliance. Every node has its static address (set network static …, apply, network confirm) and a synchronised clock, as for any cluster. CertAutoPilot OS 0.2.4 or later.

1. Prepare the HSM​

Whatever the HSM, you end this step with three things: the client files for each node, the token label and the PIN.

  1. In Thales Data Protection on Demand, add a Luna Cloud HSM service. Remove FIPS restrictions is your choice: CertAutoPilot works with it checked or unchecked. The region cannot be changed later.

  2. On the service, Create Service Client once per node (cap-1, cap-2, cap-3) and download each zip. Keep them as secrets: each holds that client's credentials.

  3. Initialize the partition once, from any Linux machine, with the lunacm inside one of the zips:

    $ unzip setup-cap-1.zip -d dpod && cd dpod && tar -xf cvclient-min.tar
    $ source ./setenv && ./bin/64/lunacm
    lunacm:> partition init -label cap-prod # Partition SO password, then a cloning domain
    lunacm:> role login -name po # the Partition SO password
    lunacm:> role init -name co # the Crypto Officer password: the PIN
    lunacm:> role logout
    lunacm:> role login -name co # if asked, 'role changepw -name co'
    lunacm:> exit

    Passwords are at least 7 characters. Keep the SO password and the cloning domain: Luna needs them to manage and back up the partition; CertAutoPilot never uses them.

You now have
Client filesone zip per node (setup-cap-1.zip …)
Token labelthe partition label (cap-prod)
PINthe Crypto Officer password

2. Node 1 — install the client and form the cluster​

Copy the client files to the node, from your workstation:

$ scp setup-cap-1.zip admin@10.0.0.11:

Install the client. The command prints the module path; the cluster wizard offers it as its default.

import pkcs11-client on node 1

Check that the node reaches the HSM. diagnose hsm loads the module the way the backend will and lists the tokens it sees; your token label must be among them.

diagnose hsm before the cluster exists

Form the cluster with the HSM. In setup cluster answer init, the virtual address (or Enter for none), y for the HSM, Enter for the module path, then the token label and the PIN (not shown as you type).

setup cluster with the HSM

The cluster forms in the background, in a minute or two. While it forms, the HSM creates the first key, certautopilot-kek-v1. show cluster shows the result:

show cluster on the new leader

Make a join token for the next node. Each token is for one node:

cluster token new

3. Node 2 and node 3 — install the client and join​

On node 2, with node 2's own client files:

  1. Copy and install the client exactly as on node 1 (scp, then import …), and run diagnose hsm: the token label must be listed.

    import pkcs11-client on node 2

  2. Join with the token from node 1. The node takes the HSM settings and the PIN from the leader; there is nothing more to type:

    cluster join on node 2

  3. Check that the node logs in to the HSM with the cluster's PIN:

    diagnose hsm on a member

On node 3: cluster token new on node 1 again, then the same three steps with node 3's client files.

4. Check the cluster​

On node 1, show cluster lists the three members:

show cluster with three members

diagnose hsm on any node shows that node's login to the HSM.

5. Rotating the key​

Run these on node 1 (the leader). Each one covers every node.

Create the next key. kek add 2 creates certautopilot-kek-v2 inside the HSM, restarts every backend and waits until each one has loaded it:

kek add 2

Re-encrypt the data from version 1 to version 2:

kek rotate 1 2

Follow it with kek status until version 2 is active and version 1 holds no record:

kek status after the rotation

IV source says who generates the AES-GCM IV of each wrapped key: host (CertAutoPilot) or hsm. A Luna partition in FIPS mode shows hsm.

Retire the old key once you no longer need to restore a backup made before the rotation:

kek remove 1

  • Each node switches its writes to the new key within about 30 seconds of the rotation's end. If kek remove 1 reports records still on version 1, run kek resweep 1, then kek remove 1 again.
  • kek remove does not delete the key object inside the HSM. Delete it with the HSM's own tools once no backup you may restore still needs it.

When something goes wrong​

What you seeWhat it meansWhat to do
cluster join: this node has no PKCS#11 module at …The client is not installed on this node. Nothing was changed.Step 1 of node 2/3 on this node, then cluster join again.
diagnose hsm: C_Initialize: 0x…The client cannot read its configuration or cannot reach the HSM.Re-import the client files; check the node's route and firewall to the HSM.
diagnose hsm: no token labelled '…'The label the cluster was formed with is not on the HSM.Luna: the partition label. Fortanix: Fortanix Token.
diagnose hsm: login with the cluster PIN: 0xa0Wrong PIN.Luna: the Crypto Officer password. Fortanix: the app's API key.
diagnose hsm: CKM_AES_GCM NOT offeredThe HSM cannot do what CertAutoPilot needs.Use a supported HSM.
setup cluster finished, but show cluster stays emptyForming the cluster failed; it was undone.logs agent names the reason; usually the label, the PIN or the HSM's reachability. Fix it, then setup cluster again.
cluster join: the token was revoked / has expiredThe token is no longer valid.cluster token new on node 1, then join with the new token.
kek remove: version 1 is still referenced by N record(s)Records written while the nodes were switching to the new key.kek resweep 1, then kek remove 1.

Reference​

Placing a client by hand

The import commands are a convenience. From shell (sudo), a client can be placed anywhere when three rules hold:

  1. The backend runs as certautopilot and must read the files: root:certautopilot 0640, libraries 0750, every directory on the way traversable.
  2. The client's variables go in /etc/cap-pkcs11/env, one VAR=value per line (Luna: ChrystokiConfigurationPath=<the directory of Chrystoki.conf>). The backend, the cluster agent and the console's kek commands read it.
  3. What the client writes goes under /var/lib/cap-pkcs11, the one HSM directory the backend may write (Luna: MutexFolder).

A Luna Cloud HSM client placed by hand under /opt/luna:

$ sudo mkdir -p /opt/luna && cd /opt/luna
$ sudo python3 -m zipfile -e ~/setup-cap-1.zip pkg
$ sudo tar -xf pkg/cvclient-min.tar && sudo cp pkg/Chrystoki.conf pkg/*.pem . && sudo rm -rf pkg
$ sudo bash -c 'source ./setenv'
$ sudo sed -i 's|^\( *MutexFolder *=\).*|\1 /var/lib/cap-pkcs11/luna;|' Chrystoki.conf
$ sudo install -d -m 2770 -o root -g certautopilot /var/lib/cap-pkcs11/luna
$ sudo chown -R root:certautopilot /opt/luna && sudo chmod -R g+rX,o-rwx /opt/luna
$ echo ChrystokiConfigurationPath=/opt/luna | sudo tee /etc/cap-pkcs11/env

The module path is then /opt/luna/libs/64/libCryptoki2.so; type it in setup cluster. diagnose hsm finds the client through the variable. The application's CLI run from the shell (sudo cap kek …) does not load the variables; use the console's kek commands.

Fortanix: the API key inside pkcs11.conf

Instead of entering the API key as the PIN, add api_key = "…" to pkcs11.conf before importing it and give file:///etc/fortanix/pkcs11.conf as the PIN. The key then lives only in that file, and replacing it is one import pkcs11-config per node. The file does not travel with a join: import it on every node.

Answer files and the OVA

An answer file (cluster.encryption in /etc/cap/setup.yaml.example) or the OVA's cap.cluster.kek_provider is applied at first boot, before a client can be imported; only the image's SoftHSM module (a lab) is there then. For a vendor HSM, deploy with cluster role none and follow this page from step 2.

Console commands and their standalone equivalents
Standalone / PKCS#11 pagesCertAutoPilot OS console
install the vendor client, copy the .soimport pkcs11-module <file> / import pkcs11-client <file>
write the client config, chown, chmodimport pkcs11-config <file> (Fortanix) / part of import pkcs11-client (Luna)
pkcs11-tool -L, vtl verifydiagnose hsm
get.sh … --kek-provider=pkcs11 …setup cluster → init → HSM y
certautopilot kek pkcs11-init --version=N + a restart per nodekek add N
certautopilot kek verify --target=Nkek verify N
certautopilot kek rotate --from-version=A --to-version=Bkek rotate A B
certautopilot kek statuskek status / show kek
certautopilot kek remove --version=Nkek remove N