PKCS#11 setup

The pkcs11 provider stores the KEK inside a Hardware Security Module. CertAutoPilot never sees key material — only opaque HSM handles and the wrapped DEK blobs. This page walks through preparing a token, running the capability probe, and completing the install-lock.

Multi-VM? Use an HA HSM.

SoftHSM2 is single-host only — the token lives as files under /var/lib/softhsm/tokens. Two VMs sharing one MongoDB but running independent SoftHSM2 instances will silently produce different key material under the same label, breaking every cross-VM envelope decrypt. A proper multi-VM pkcs11 deployment requires a network-reachable or HA HSM (AWS CloudHSM cluster, Thales Luna HA group, Azure Dedicated HSM, or a network-bridged YubiHSM2). See Multi-VM → PKCS#11 provider for the install sequence and rotation in a cluster for what changes in that setup.

Where the pieces come from

CertAutoPilot talks to your HSM through a vendor-provided PKCS#11 client library — a .so file the HSM manufacturer ships. We do not build, bundle, or distribute this library; it is the HSM driver for your specific hardware, identical to how a printer vendor ships its own CUPS driver. The split between "what we ship" and "what the vendor ships" matters because the vendor library carries all the network / authentication / HA failover logic and has to match your HSM exactly.

Component	Who provides it	Where it lives
`certautopilot` binary	CertAutoPilot release (tarball / Helm)	`/usr/local/bin/certautopilot`
PKCS#11 client library (`.so`)	HSM vendor — download from their portal, install their package	Varies per vendor (see table below)
HSM network config (IP, TLS certs, HA group)	You, using the vendor's own CLI tools (`vtl` for Luna, `configure` for CloudHSM, etc.) — these edit a vendor-specific config file that the `.so` reads on `dlopen`	Vendor config file (see table below); never in `config.yaml`
`config.yaml` `encryption.pkcs11` block	You — at minimum `module_path` (path to the `.so`), `token_label` and `pin_env` (env var name). `token_serial` or `slot_number` can select the token instead of `token_label`; `key_label_prefix`, `max_sessions`, `pool_wait_timeout` and `use_gcm_iv_from_hsm` are also available	`/etc/certautopilot/config.yaml`

Vendor library + config locations

HSM	Library path	Network config file	Who populates network config
SoftHSM2 (dev only)	`/usr/lib/softhsm/libsofthsm2.so` (or `/opt/homebrew/lib/softhsm/...` on macOS)	`/etc/softhsm/softhsm2.conf` (file-based, no network)	Package default
AWS CloudHSM	`/opt/cloudhsm/lib/libcloudhsm_pkcs11.so`	`/opt/cloudhsm/etc/cloudhsm_client.cfg`	`sudo /opt/cloudhsm/bin/configure -a <cluster-ENI-IP>` — writes the cluster endpoint into the cfg file
Thales (SafeNet) Luna Network HSM	`/usr/safenet/lunaclient/lib/libCryptoki2_64.so`	`/etc/Chrystoki.conf`	`sudo vtl addServer -n <hsm-ip> -c <server-ca-cert>` — adds HSM IP / cert / port to `Chrystoki.conf`
Azure Dedicated HSM	Same Luna client as above (Azure uses Luna under the hood)	`/etc/Chrystoki.conf`	Same `vtl addServer` flow against the Azure-provisioned HSM
Fortanix DSM	`/opt/fortanix/pkcs11/libsdkms-pkcs11.so` (default)	`/opt/fortanix/pkcs11/pkcs11.conf`	Fortanix UI issues an app credential; you drop it into the conf file per Fortanix docs

In other words: CertAutoPilot's config.yaml has no HSM IP, TLS cert, HA group, or port fields. PKCS#11 abstraction's whole point is that those details live in the vendor's own config — we just dlopen the library and ask for wrap/unwrap. If you're looking at config.yaml wondering where to put 10.0.1.100, the answer is "not here; run vtl addServer (or your vendor's equivalent) first."

Multi-VM rule of thumb

Because the network config lives in the vendor's file, every VM must have an identical vendor config — the same HSM IP(s), the same client cert, the same HA group. Usually operators standardise this in their config-management layer (copy the vendor config file to every host after setup). See Multi-VM → PKCS#11 provider.

Architecture summary

 CertAutoPilot backend                    HSM (PKCS#11 module)
 ┌─────────────────────┐    dlopen     ┌──────────────────────┐
 │ PKCS11Provider      │ ───────────►  │ libpkcs11.so         │
 │  (crypto11)         │    session    │  ├── token           │
 │  Wrap(dek)          │ ◄──────────►  │  │   └── AES key V1  │
 │  Unwrap(blob, v)    │    CKM_AES_   │  │   └── AES key V2  │
 │                     │    GCM        │  └── …               │
 └─────────────────────┘               └──────────────────────┘

Required HSM mechanisms

Mechanism	Purpose
`CKM_AES_KEY_GEN`	Generate the AES-256 KEK at `kek pkcs11-init`.
`CKM_AES_GCM`	Every Wrap / Unwrap call.

Both are standard PKCS#11 v2.40 mechanisms. Any FIPS-approved HSM from the last decade supports them. We still probe at pkcs11-init time because vendor firmware regressions exist.

Prerequisites

A PKCS#11-compliant HSM with an initialised token.
The vendor's PKCS#11 client library on the CertAutoPilot host (e.g. /usr/lib/softhsm/libsofthsm2.so for SoftHSM2, /opt/cloudhsm/lib/libcloudhsm_pkcs11.so for AWS CloudHSM, /usr/safenet/lunaclient/lib/libCryptoki2_64.so for Thales Luna).
A crypto user PIN for the token.
A fresh MongoDB (no prior env-provider install) — see provider migration.

The capability probe

Before committing anything to the HSM or MongoDB, kek pkcs11-init runs a four-phase probe:

Reachability — confirms the module loads and the token is usable. It does not enumerate the token's mechanism list; a missing CKM_AES_GCM surfaces as a round-trip failure in step 3 instead.
Key generation — generates a throwaway AES-256 key. It does not read CKA_SENSITIVE / CKA_EXTRACTABLE back — enforce non-extractability through your HSM's key policy and confirm it with vendor tooling.
AEAD round-trip — encrypts a random DEK, decrypts, compares.
Cleanup — deletes the throwaway key.

The --probe-only flag runs phases 1–3 against a brand-new key, deletes it, and exits without persisting anything. Use this to validate a new HSM before committing:

cap kek pkcs11-init --version=1 --probe-only

One-command install (standalone)

The bootstrap script wraps probe + init + install into a single command. Run it on the target host (or from a jump host over SSH); get.sh downloads the pinned tarball, verifies its SHA256, and runs the install pipeline. The bootstrap is the only supported entry point.

Pick one of the two PIN-ingress modes:

# Inline PIN (one-liner; argv-visible during install only,
# persists afterward only in /etc/certautopilot/secrets.env mode 0600):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=<latest> --mongo=local \
    --kek-provider=pkcs11 \
    --pkcs11-module=/usr/lib/softhsm/libsofthsm2.so \
    --pkcs11-token-label=certautopilot-prod \
    --pkcs11-pin='<hsm-user-pin>'

# PIN from a mode-0600 file (production-grade; never in argv or history):
umask 077
printf '%s' "$HSM_PIN" > /tmp/cap-pin
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=<latest> --mongo=local \
    --kek-provider=pkcs11 \
    --pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
    --pkcs11-token-label=certautopilot \
    --pkcs11-pin-file=/tmp/cap-pin
shred -u /tmp/cap-pin

Between MongoDB bootstrap and systemd start, the installer:

Runs the HSM probe against a throwaway key.
Generates the real V1 AES-256 key with label certautopilot-kek-v1 (or your configured prefix).
Writes the kek_install lock into MongoDB as {provider: "pkcs11"}.
Inserts keystore metadata (key label → version mapping).

After this, every future backend start reads the lock, confirms the config-declared provider matches, and loads the HSM session pool. A probe failure aborts the install before any persistent state is committed.

PIN handling

Runtime (service): The PIN lives in /etc/certautopilot/secrets.env (mode 0600, owned by the certautopilot service user). Systemd loads it via EnvironmentFile= on every start — reboots and systemctl restart pick it up automatically.
Install-time ingress: three ways, in order of preference: --pkcs11-pin-file=<path> (file must be mode 0600 or 0400) > --pkcs11-pin=<value> (argv-visible during install only) > env var named by --pkcs11-pin-env= (default CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN). The first two are recommended for curl | sudo bash pipelines because sudo typically drops arbitrary env vars.
Kubernetes: add the PIN as a key in the rendered Secret; the Deployment references it via envFrom.
CloudHSM: the PIN format is <username>:<password> — crypto-user credentials, not a simple password.
Backup: treat secrets.env like any other root-level credential. Include it in your backup procedure; multi-VM deployments use --secrets-from=<path> on the second+ host to seed the same file.

Admin-created keys (pkcs11-register)

If your HSM workflow requires keys to be created by an administrator outside CertAutoPilot (key ceremony, vendor policy), create the AES-256 key with label certautopilot-kek-v1 (or your configured prefix), then register it:

# With pkcs11-tool (OpenSC)
pkcs11-tool --module $MODULE --token-label $TOKEN --login --pin $PIN \
  --keygen --key-type AES:32 \
  --label certautopilot-kek-v1 --id 01 \
  --usage-decrypt --sensitive

# Register with CertAutoPilot (verifies attributes + runs round-trip)
cap kek pkcs11-register --version=1

pkcs11-register does NOT modify the HSM — only inserts the keystore metadata + writes the install lock.

Rotation on PKCS#11

Same shape as env rotation:

cap kek pkcs11-init --version=2 — probes + creates V2 key inside the HSM.
Restart each node so it loads V2 from the HSM alongside V1.
cap kek verify --target=2 (checks target is loaded on every live node; current need not match yet — that's what rotation will flip).
cap kek rotate --from-version=1 --to-version=2.
AUTOMATIC — each node's heartbeat tick (~30 s) picks up V2 as the new active from the keystore. A forced restart is optional if you need the switch to be instant.
cap kek remove --version=1 when your backup window closes. It refuses if any record still references V1, if any encrypted data is invisible to the reference count, or if any live node still reports V1 as its current version. Note the result is the removed state, not retired: V1's key material is then ignored even if it is still on the host, and cap kek reinstate --version=1 is what undoes that.

Full runbook: KEK rotation.

Troubleshooting

"HSM does not support CKM_AES_GCM"

Run pkcs11-tool --module $MODULE -M | grep -i gcm to confirm. If missing, the HSM is incompatible — see PKCS#11 vendors.

"PIN env var X is empty"

The env var is not exported in the process environment. For standalone: check /etc/certautopilot/secrets.env. For K8s: check the rendered Secret has the key.

"key 'certautopilot-kek-v1' not found in token"

The label prefix in config doesn't match the key in your HSM. Either adjust encryption.pkcs11.key_label_prefix in config, or re-run kek pkcs11-init --version=1 (only safe if no production data yet).

"refusing to run: this database is locked for provider='env'"

You're pointing pkcs11-init at a MongoDB that was initialised for the env provider. See provider migration for the fresh-install path.

Where the pieces come from​

Vendor library + config locations​

Architecture summary​

Required HSM mechanisms​

Prerequisites​

The capability probe​

One-command install (standalone)​

PIN handling​

Admin-created keys (pkcs11-register)​

Rotation on PKCS#11​

Troubleshooting​

"HSM does not support CKM_AES_GCM"​

"PIN env var X is empty"​

"key 'certautopilot-kek-v1' not found in token"​

"refusing to run: this database is locked for provider='env'"​

See also​