Skip to main content

PKCS#11 setup

The pkcs11 provider stores the KEK inside a Hardware Security Module. CertAutoPilot never sees key material — only opaque HSM handles and the wrapped DEK blobs. This page walks through preparing a token, running the capability probe, and completing the install-lock.

On the CertAutoPilot OS appliance

The appliance has no shell: the vendor library and its configuration are imported on the console (import pkcs11-module, import pkcs11-config, import pkcs11-client; diagnose hsm checks the result), the HSM is chosen in setup cluster, and kek add|verify|rotate|remove replace the certautopilot kek … commands and the restarts. See CertAutoPilot OS with an HSM.

Multi-VM? Use an HA HSM.

SoftHSM2 is single-host only — the token lives as files under /var/lib/softhsm/tokens. Two VMs sharing one MongoDB but running independent SoftHSM2 instances will silently produce different key material under the same label, breaking every cross-VM envelope decrypt. A proper multi-VM pkcs11 deployment requires a network-reachable or HA HSM (AWS CloudHSM cluster, Thales Luna HA group, Azure Dedicated HSM, or a network-bridged YubiHSM2). See Multi-VM → PKCS#11 provider for the install sequence and rotation in a cluster for what changes in that setup.

Where the pieces come from​

CertAutoPilot talks to your HSM through a vendor-provided PKCS#11 client library — a .so file the HSM manufacturer ships. We do not build, bundle, or distribute this library; it is the HSM driver for your specific hardware, identical to how a printer vendor ships its own CUPS driver. The split between "what we ship" and "what the vendor ships" matters because the vendor library carries all the network / authentication / HA failover logic and has to match your HSM exactly.

ComponentWho provides itWhere it lives
certautopilot binaryCertAutoPilot release (tarball / Helm)/usr/local/bin/certautopilot
PKCS#11 client library (.so)HSM vendor — download from their portal, install their packageVaries per vendor (see table below)
HSM network config (IP, TLS certs, HA group)You, using the vendor's own CLI tools (vtl for Luna, configure for CloudHSM, etc.) — these edit a vendor-specific config file that the .so reads on dlopenVendor config file (see table below); never in config.yaml
config.yaml encryption.pkcs11 blockYou — at minimum module_path (path to the .so), token_label and pin_env (env var name). token_serial or slot_number can select the token instead of token_label; key_label_prefix, max_sessions, pool_wait_timeout and use_gcm_iv_from_hsm are also available/etc/certautopilot/config.yaml

Vendor library + config locations​

HSMLibrary pathNetwork config fileWho populates network config
SoftHSM2 (dev only)/usr/lib/softhsm/libsofthsm2.so (or /opt/homebrew/lib/softhsm/... on macOS)/etc/softhsm/softhsm2.conf (file-based, no network)Package default
AWS CloudHSM/opt/cloudhsm/lib/libcloudhsm_pkcs11.so/opt/cloudhsm/etc/cloudhsm_client.cfgsudo /opt/cloudhsm/bin/configure -a <cluster-ENI-IP> — writes the cluster endpoint into the cfg file
Thales (SafeNet) Luna Network HSM/usr/safenet/lunaclient/lib/libCryptoki2_64.so/etc/Chrystoki.confsudo vtl addServer -n <hsm-ip> -c <server-ca-cert> — adds HSM IP / cert / port to Chrystoki.conf
Azure Dedicated HSMSame Luna client as above (Azure uses Luna under the hood)/etc/Chrystoki.confSame vtl addServer flow against the Azure-provisioned HSM
Fortanix DSM/opt/fortanix/pkcs11/fortanix_pkcs11.so (5.x; libsdkms-pkcs11.so before 5.0)/etc/fortanix/pkcs11.confYou write api_endpoint (and optionally api_key) per the Fortanix walkthrough

In other words: CertAutoPilot's config.yaml has no HSM IP, TLS cert, HA group, or port fields. PKCS#11 abstraction's whole point is that those details live in the vendor's own config — we just dlopen the library and ask for wrap/unwrap. If you're looking at config.yaml wondering where to put 10.0.1.100, the answer is "not here; run vtl addServer (or your vendor's equivalent) first."

Multi-VM rule of thumb

Because the network config lives in the vendor's file, every VM must have an identical vendor config — the same HSM IP(s), the same client cert, the same HA group. Usually operators standardise this in their config-management layer (copy the vendor config file to every host after setup). See Multi-VM → PKCS#11 provider.

Architecture summary​

CertAutoPilot backend HSM (PKCS#11 module)
┌─────────────────────┐ dlopen ┌──────────────────────┐
│ PKCS11Provider │ ───────────► │ libpkcs11.so │
│ (crypto11) │ session │ ├── token │
│ Wrap(dek) │ ◄──────────► │ │ └── AES key V1 │
│ Unwrap(blob, v) │ CKM_AES_ │ │ └── AES key V2 │
│ │ GCM │ └── … │
└─────────────────────┘ └──────────────────────┘

Required HSM mechanisms​

MechanismPurpose
CKM_AES_KEY_GENGenerate the AES-256 KEK at kek pkcs11-init.
CKM_AES_GCMEvery Wrap / Unwrap call.

Both are standard PKCS#11 v2.40 mechanisms. Any FIPS-approved HSM from the last decade supports them. We still probe at pkcs11-init time because vendor firmware regressions exist.

Prerequisites​

  • A PKCS#11-compliant HSM with an initialised token.
  • The vendor's PKCS#11 client library on the CertAutoPilot host (e.g. /usr/lib/softhsm/libsofthsm2.so for SoftHSM2, /opt/cloudhsm/lib/libcloudhsm_pkcs11.so for AWS CloudHSM, /usr/safenet/lunaclient/lib/libCryptoki2_64.so for Thales Luna).
  • A crypto user PIN for the token.
  • A fresh MongoDB (no prior env-provider install) — see provider migration.

The capability probe​

When kek pkcs11-init has to create a key, it first runs a probe against a throwaway key:

  1. Reachability — the module loads and the token opens with the PIN. The token's mechanism list is not read; a missing CKM_AES_GCM shows up as a round-trip failure in step 3.
  2. Key generation — a throwaway AES-256 key.
  3. AEAD round-trip — encrypts a random DEK, decrypts it, compares. It is tried with an IV from CertAutoPilot first; if that does not work, the token is opened again and the round-trip is made with an IV the HSM generates. The probe prints which one worked (IV: supplied by CertAutoPilot (12 bytes) or IV: generated by the HSM (16 bytes)); the service settles it the same way at every start.
  4. Cleanup — deletes the throwaway key.

--probe-only checks the HSM and changes nothing. It needs no database and runs whatever encryption.provider says, so it can check an HSM before anything is set up:

cap kek pkcs11-init --probe-only # the four steps above on a throwaway key
cap kek pkcs11-init --probe-only --version=1 # the existing key labelled certautopilot-kek-v1

With --version, the existing key is found, an AES-GCM round-trip is made with it, and its attributes are printed (key type, length, whether it is stored in the token, sensitive, extractable, allowed to encrypt and decrypt). An extractable key, or one not allowed to both encrypt and decrypt, is printed as a warning — set that in your HSM's key policy.

What pkcs11-init does for a version​

kek pkcs11-init --version=N looks at the keystore row of version N and at the keys labelled <prefix>N in the token:

Keystore rowKey in the tokenResult
presentyesChecks that the key is the one the version was created with (its key check value); generates nothing. A row from before 1.7.3 gets its check value now.
removed—Says so and exits 0 (an installer re-running --version=1 on a rotated cluster).
presentnoRefused: wrong token, or the key was lost. Nothing is generated.
absentyesRegisters the key. When stored data was sealed under version N, the key must open it first.
absentnoGenerates an AES-256 key and registers it — refused when stored data of version N exists (a new key cannot open it).

Several keys under one label, or a key that exists only in a session (CKA_TOKEN false), are refused. Every refusal is the last line the command prints. kek pkcs11-register follows the same table but never generates a key.

One-command install (standalone)​

The bootstrap script wraps probe + init + install into a single command. Run it on the target host (or from a jump host over SSH); get.sh downloads the pinned tarball, verifies its SHA256, and runs the install pipeline. The bootstrap is the only supported entry point.

Pick one of the two PIN-ingress modes:

# Inline PIN (one-liner; argv-visible during install only,
# persists afterward only in /etc/certautopilot/secrets.env mode 0600):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
| sudo bash -s -- --version=<latest> --mongo=local \
--kek-provider=pkcs11 \
--pkcs11-module=/usr/lib/softhsm/libsofthsm2.so \
--pkcs11-token-label=certautopilot-prod \
--pkcs11-pin='<hsm-user-pin>'
# PIN from a file only you can read (production-grade; never in argv or
# history). mktemp makes a fresh file — a fixed /tmp name could already
# exist, owned by someone else:
PIN_FILE=$(mktemp)
printf '%s' "$HSM_PIN" > "$PIN_FILE"
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
| sudo bash -s -- --version=<latest> --mongo=local \
--kek-provider=pkcs11 \
--pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
--pkcs11-token-label=certautopilot \
--pkcs11-pin-file="$PIN_FILE"
shred -u "$PIN_FILE"

Between MongoDB bootstrap and systemd start, the installer:

  1. Runs the HSM probe against a throwaway key.
  2. Generates the real V1 AES-256 key with label certautopilot-kek-v1 (or your configured prefix).
  3. Writes the kek_install lock into MongoDB as {provider: "pkcs11"}.
  4. Inserts keystore metadata (key label → version mapping).

After this, every future backend start reads the lock, confirms the config-declared provider matches, and loads the HSM session pool. A probe failure aborts the install before any persistent state is committed.

PIN handling​

  • Runtime (service): The PIN lives in /etc/certautopilot/secrets.env (mode 0600, owned by the certautopilot service user). Systemd loads it via EnvironmentFile= on every start — reboots and systemctl restart pick it up automatically.
  • Install-time ingress: three ways, in order of preference: --pkcs11-pin-file=<path> (file must be mode 0600 or 0400) > --pkcs11-pin=<value> (argv-visible during install only) > env var named by --pkcs11-pin-env= (default CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN). The first two are recommended for curl | sudo bash pipelines because sudo typically drops arbitrary env vars.
  • Kubernetes: add the PIN as a key in the rendered Secret; the Deployment references it via envFrom.
  • CloudHSM: the PIN format is <username>:<password> — crypto-user credentials, not a simple password.
  • Backup: treat secrets.env like any other root-level credential. Include it in your backup procedure; multi-VM deployments use --secrets-from=<path> on the second+ host to seed the same file.

Admin-created keys (pkcs11-register)​

If your HSM workflow requires keys to be created by an administrator outside CertAutoPilot (key ceremony, vendor policy), create the AES-256 key with label certautopilot-kek-v1 (or your configured prefix), then register it:

# With pkcs11-tool (OpenSC)
pkcs11-tool --module $MODULE --token-label $TOKEN --login --pin $PIN \
--keygen --key-type AES:32 \
--label certautopilot-kek-v1 --id 01 \
--usage-decrypt --sensitive

# Register with CertAutoPilot (prints the key's attributes, runs a round-trip)
cap kek pkcs11-register --version=1

pkcs11-register does not modify the HSM — it inserts the keystore row (with the key check value) and writes the install lock. It follows the same table as pkcs11-init, except that it never generates a key. It refuses a key that exists only in a session; other attributes (extractable, not allowed to encrypt and decrypt, not AES-256) are printed as warnings.

Rotation on PKCS#11​

Same shape as env rotation:

  1. cap kek pkcs11-init --version=2 — probes + creates V2 key inside the HSM.
  2. Restart each node so it loads V2 from the HSM alongside V1.
  3. cap kek verify --target=2 (checks target is loaded on every live node; current need not match yet — that's what rotation will flip).
  4. cap kek rotate --from-version=1 --to-version=2.
  5. AUTOMATIC — each node's heartbeat tick (~30 s) picks up V2 as the new active from the keystore. A forced restart is optional if you need the switch to be instant.
  6. cap kek remove --version=1 when your backup window closes. It refuses if any record still references V1, if any encrypted data is invisible to the reference count, or if any live node still reports V1 as its current version. Note the result is the removed state, not retired: V1's key material is then ignored even if it is still on the host, and cap kek reinstate --version=1 is what undoes that.

Full runbook: KEK rotation.

While the service runs​

  • Lost sessions reconnect by themselves. When the HSM ends the process's sessions or logs it out (a token restart, an HA failover, another client's logout), the next operation opens the token again and is retried once. When the library itself was shut down, the old context is closed first so the library is loaded again. Calls that fail together share one reconnect. A failed reconnect is not retried at once: the next attempt waits 1 s, doubling to 30 s; in that wait encryption answers "the HSM is not reachable now" without contacting it.
  • A refused PIN stops the retries. When the token refuses the PIN on a reconnect, the process does not log in again until it is restarted — repeated failed logins can lock the user PIN.
  • Timeouts. An HSM call that does not answer within 30 s is abandoned (and counts as a lost HSM). The session pool opens at most max_sessions sessions (default 32) and waits at most pool_wait_timeout (default 30 s) for a free one; when every session stays busy that long the operation fails with "resource pool timed out" — raise max_sessions if your HSM allows more sessions per application.
  • Health check. Every health_interval (default 60 s, 0 turns it off) the process makes one round trip with the current key and opens every loaded version's key check value — one HSM operation each, on its own goroutine. The result is in kek verify's cluster table (HSM column, with the error of a failing node), in the certautopilot_kek_healthy metric (1 / 0) and certautopilot_kek_reconnects, and in the log once when an outage starts and once when it ends. /readyz does not include it.

Key check value​

Each keystore row carries a short value sealed by its version's key. Every process opens it before using the key it finds under the version's label — at start, at every reconnect and in the health check. A node pointed at another token, or a key re-created under the same label, therefore stops with "the key under this label is not the key this version was created with" instead of starting and writing data no other node can read. With several keys under one label, the one that opens the value is used and the others are warned about.

Rows created before 1.7.3 get their value the first time a node starts: the first node stores it, every other node must open it.

kek status shows, when the HSM cannot be opened for the registry, a KEY column per version (verified, WRONG KEY, missing, unknown). kek reinstate refuses a version whose key is missing, appears more than once, or does not open its check value.

Troubleshooting​

"AES-GCM works with neither IV source"​

The token does not encrypt and decrypt with AES-GCM, whoever supplies the IV. Run pkcs11-tool --module $MODULE -M | grep -i gcm to see whether it offers CKM_AES_GCM. If not, the HSM is incompatible — see PKCS#11 vendors.

"PIN env var X is empty"​

The env var is not exported in the process environment. For standalone: check /etc/certautopilot/secrets.env, or run the command through cap, which loads it. For K8s: check the rendered Secret has the key.

"key v1 (label=…) not found in token"​

The label prefix in the configuration does not match the key in your HSM, or the process opened another token. Adjust encryption.pkcs11.key_label_prefix (the version number is appended directly: prefix certautopilot-kek-v gives certautopilot-kek-v1), or check token_label. kek pkcs11-init never generates a key for a version that already has a keystore row.

"the key under this label is not the key this version was created with"​

The token holds a key under the version's label, but not the one the version was created with: another token, or a key re-created under the same label. Data written with the original key cannot be read with this one. Point the node at the token that holds the original key.

"the token refused the PIN"​

A reconnect was refused for the PIN; the process does not try again until it is restarted (failed logins can lock the PIN). Fix the PIN in secrets.env and restart the node.

"refusing to run: this database is locked for provider='env'"​

You're pointing pkcs11-init at a MongoDB that was initialised for the env provider. See provider migration for the fresh-install path.

See also​