PKCS#11 setup

The pkcs11 provider stores the KEK inside a Hardware Security Module. CertAutoPilot never sees key material — only opaque HSM handles and the wrapped DEK blobs. This page walks through preparing a token, running the capability probe, and completing the install-lock.

Multi-VM? Use an HA HSM.

SoftHSM2 is single-host only — the token lives as files under /var/lib/softhsm/tokens. Two VMs sharing one MongoDB but running independent SoftHSM2 instances will silently produce different key material under the same label, breaking every cross-VM envelope decrypt. A proper multi-VM pkcs11 deployment requires a network-reachable or HA HSM (AWS CloudHSM cluster, Thales Luna HA group, Azure Dedicated HSM, or a network-bridged YubiHSM2). See Multi-VM → PKCS#11 topology for the install sequence and rotation in a cluster for what changes in that setup.

Where the pieces come from

CertAutoPilot talks to your HSM through a vendor-provided PKCS#11 client library — a .so file the HSM manufacturer ships. We do not build, bundle, or distribute this library; it is the HSM driver for your specific hardware, identical to how a printer vendor ships its own CUPS driver. The split between "what we ship" and "what the vendor ships" matters because the vendor library carries all the network / authentication / HA failover logic and has to match your HSM exactly.

Component	Who provides it	Where it lives
`certautopilot` binary	CertAutoPilot release (tarball / Helm)	`/usr/local/bin/certautopilot`
PKCS#11 client library (`.so`)	HSM vendor — download from their portal, install their package	Varies per vendor (see table below)
HSM network config (IP, TLS certs, HA group)	You, using the vendor's own CLI tools (`vtl` for Luna, `configure` for CloudHSM, etc.) — these edit a vendor-specific config file that the `.so` reads on `dlopen`	Vendor config file (see table below); never in `config.yaml`
`config.yaml` `encryption.pkcs11` block	You — three values: `module_path` (path to the `.so`), `token_label`, `pin_env` (env var name)	`/etc/certautopilot/config.yaml`

Vendor library + config locations

HSM	Library path	Network config file	Who populates network config
SoftHSM2 (dev only)	`/usr/lib/softhsm/libsofthsm2.so` (or `/opt/homebrew/lib/softhsm/...` on macOS)	`/etc/softhsm/softhsm2.conf` (file-based, no network)	Package default
AWS CloudHSM	`/opt/cloudhsm/lib/libcloudhsm_pkcs11.so`	`/opt/cloudhsm/etc/cloudhsm_client.cfg`	`sudo /opt/cloudhsm/bin/configure -a <cluster-ENI-IP>` — writes the cluster endpoint into the cfg file
Thales (SafeNet) Luna Network HSM	`/usr/safenet/lunaclient/lib/libCryptoki2_64.so`	`/etc/Chrystoki.conf`	`sudo vtl addServer -n <hsm-ip> -c <server-ca-cert>` — adds HSM IP / cert / port to `Chrystoki.conf`
Azure Dedicated HSM	Same Luna client as above (Azure uses Luna under the hood)	`/etc/Chrystoki.conf`	Same `vtl addServer` flow against the Azure-provisioned HSM
Fortanix DSM	`/opt/fortanix/pkcs11/libsdkms-pkcs11.so` (default)	`/opt/fortanix/pkcs11/pkcs11.conf`	Fortanix UI issues an app credential; you drop it into the conf file per Fortanix docs

In other words: CertAutoPilot's config.yaml has no HSM IP, TLS cert, HA group, or port fields. PKCS#11 abstraction's whole point is that those details live in the vendor's own config — we just dlopen the library and ask for wrap/unwrap. If you're looking at config.yaml wondering where to put 10.0.1.100, the answer is "not here; run vtl addServer (or your vendor's equivalent) first."

Multi-VM rule of thumb

Because the network config lives in the vendor's file, every VM must have an identical vendor config — the same HSM IP(s), the same client cert, the same HA group. Usually operators standardise this in their config-management layer (copy the vendor config file to every host after setup). See Multi-VM → PKCS#11 HSM topology.

Architecture summary

 CertAutoPilot backend                    HSM (PKCS#11 module)
 ┌─────────────────────┐    dlopen     ┌──────────────────────┐
 │ PKCS11Provider      │ ───────────►  │ libpkcs11.so         │
 │  (crypto11)         │    session    │  ├── token           │
 │  Wrap(dek)          │ ◄──────────►  │  │   └── AES key V1  │
 │  Unwrap(blob, v)    │    CKM_AES_   │  │   └── AES key V2  │
 │                     │    GCM        │  └── …               │
 └─────────────────────┘               └──────────────────────┘

Required HSM mechanisms

Mechanism	Purpose
`CKM_AES_KEY_GEN`	Generate the AES-256 KEK at `kek pkcs11-init`.
`CKM_AES_GCM`	Every Wrap / Unwrap call.

Both are standard PKCS#11 v2.40 mechanisms. Any FIPS-approved HSM from the last decade supports them. We still probe at pkcs11-init time because vendor firmware regressions exist.

Prerequisites

A PKCS#11-compliant HSM with an initialised token.
The vendor's PKCS#11 client library on the CertAutoPilot host (e.g. /usr/lib/softhsm/libsofthsm2.so for SoftHSM2, /opt/cloudhsm/lib/libcloudhsm_pkcs11.so for AWS CloudHSM, /usr/safenet/lunaclient/lib/libCryptoki2_64.so for Thales Luna).
A crypto user PIN for the token.
A fresh MongoDB (no prior env-provider install) — see provider migration.

The capability probe

Before committing anything to the HSM or MongoDB, kek pkcs11-init runs a four-phase probe:

Mechanism discovery — queries the token's mechanism list.
Key generation — generates a throwaway AES-256 key, verifies CKA_SENSITIVE=true and CKA_EXTRACTABLE=false attributes.
AEAD round-trip — encrypts a random DEK, decrypts, compares.
Cleanup — deletes the throwaway key.

The --probe-only flag runs phases 1–3 against a brand-new key, deletes it, and exits without persisting anything. Use this to validate a new HSM before committing:

cap kek pkcs11-init --version=1 --probe-only

One-command install (standalone)

The bootstrap script wraps probe + init + install into a single command. Run it on the target host (or from a jump host over SSH); get.sh downloads the pinned tarball, verifies its SHA256, and runs the install pipeline. The bootstrap is the only supported entry point.

Pick one of the two PIN-ingress modes:

# Inline PIN (one-liner; argv-visible during install only,
# persists afterward only in /etc/certautopilot/secrets.env mode 0600):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=1.4.0 --mongo=local \
    --kek-provider=pkcs11 \
    --pkcs11-module=/usr/lib/softhsm/libsofthsm2.so \
    --pkcs11-token-label=certautopilot-prod \
    --pkcs11-pin='<hsm-user-pin>'

# PIN from a mode-0600 file (production-grade; never in argv or history):
umask 077
printf '%s' "$HSM_PIN" > /tmp/cap-pin
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=1.4.0 --mongo=local \
    --kek-provider=pkcs11 \
    --pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
    --pkcs11-token-label=certautopilot \
    --pkcs11-pin-file=/tmp/cap-pin
shred -u /tmp/cap-pin

Between MongoDB bootstrap and systemd start, the installer:

Runs the HSM probe against a throwaway key.
Generates the real V1 AES-256 key with label certautopilot-kek-v1 (or your configured prefix).
Writes the kek_install lock into MongoDB as {provider: "pkcs11"}.
Inserts keystore metadata (key label → version mapping).

After this, every future backend start reads the lock, confirms the config-declared provider matches, and loads the HSM session pool. A probe failure aborts the install before any persistent state is committed.

PIN handling

Runtime (service): The PIN lives in /etc/certautopilot/secrets.env (mode 0600, owned by the certautopilot service user). Systemd loads it via EnvironmentFile= on every start — reboots and systemctl restart pick it up automatically.
Install-time ingress: three ways, in order of preference: --pkcs11-pin-file=<path> (file must be mode 0600) > --pkcs11-pin=<value> (argv-visible during install only) > env var named by --pkcs11-pin-env= (default CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN). The first two are recommended for curl | sudo bash pipelines because sudo typically drops arbitrary env vars.
Kubernetes: add the PIN as a key in the rendered Secret; the Deployment references it via envFrom.
CloudHSM: the PIN format is <username>:<password> — crypto-user credentials, not a simple password.
Backup: treat secrets.env like any other root-level credential. Include it in your backup procedure; multi-VM deployments use --secrets-from=<path> on the second+ host to seed the same file.

Admin-created keys (pkcs11-register)

If your HSM workflow requires keys to be created by an administrator outside CertAutoPilot (key ceremony, vendor policy), create the AES-256 key with label certautopilot-kek-v1 (or your configured prefix), then register it:

# With pkcs11-tool (OpenSC)
pkcs11-tool --module $MODULE --token-label $TOKEN --login --pin $PIN \
  --keygen --key-type AES:32 \
  --label certautopilot-kek-v1 --id 01 \
  --usage-decrypt --sensitive

# Register with CertAutoPilot (verifies attributes + runs round-trip)
cap kek pkcs11-register --version=1

pkcs11-register does NOT modify the HSM — only inserts the keystore metadata + writes the install lock.

Rotation on PKCS#11

Same shape as env rotation:

cap kek pkcs11-init --version=2 — probes + creates V2 key inside the HSM.
Restart each node so it loads V2 from the HSM alongside V1.
cap kek verify --target=2 (checks target is loaded on every live node; current need not match yet — that's what rotation will flip).
cap kek rotate --from-version=1 --to-version=2.
AUTOMATIC — each node's heartbeat tick (~30 s) picks up V2 as the new active from the keystore. A forced restart is optional if you need the switch to be instant.
cap kek remove --version=1 when your backup window closes — refuses with a clear error if any live node is still on V1.

Full runbook: KEK rotation.

Troubleshooting

"HSM does not support CKM_AES_GCM"

Run pkcs11-tool --module $MODULE -M | grep -i gcm to confirm. If missing, the HSM is incompatible — see PKCS#11 vendors.

"PIN env var X is empty"

The env var is not exported in the process environment. For standalone: check /etc/certautopilot/secrets.env. For K8s: check the rendered Secret has the key.

"key 'certautopilot-kek-v1' not found in token"

The label prefix in config doesn't match the key in your HSM. Either adjust encryption.pkcs11.key_label_prefix in config, or re-run kek pkcs11-init --version=1 (only safe if no production data yet).

"refusing to run: this database is locked for provider='env'"

You're pointing pkcs11-init at a MongoDB that was initialised for the env provider. See provider migration for the fresh-install path.

Where the pieces come from​

Vendor library + config locations​

Architecture summary​

Required HSM mechanisms​

Prerequisites​

The capability probe​

One-command install (standalone)​

PIN handling​

Admin-created keys (pkcs11-register)​

Rotation on PKCS#11​

Troubleshooting​

"HSM does not support CKM_AES_GCM"​

"PIN env var X is empty"​

"key 'certautopilot-kek-v1' not found in token"​

"refusing to run: this database is locked for provider='env'"​

See also​