Skip to main content

Changing KEK provider — fresh install required

CertAutoPilot's KEK provider (env vs pkcs11) is chosen at install time and locked immutably in the MongoDB kek_install singleton. There is no in-place migration path: changing the provider requires a fresh install against a new MongoDB. This page explains why, and walks through the migration procedure.

Why no in-place migration​

Each provider's envelope wrap/unwrap behaviour differs (key material in process memory vs. key material resident in HSM). A running system with mixed-provider state would need code to:

  • Route every Wrap to one provider and every Unwrap to both (by envelope's provider tag).
  • Hold both providers in memory simultaneously with different failure modes.
  • Track per-version source so rotation knows which provider owns what.
  • Coordinate cluster-wide rollout across provider boundaries.

We deliberately don't write that code. It's a large surface area, rarely exercised, and easy to get wrong in ways that lose data. Fresh install + data re-import is the supported path.

What you can change without migration​

  • KEK versions within the same provider — rotation (kek rotate --from-version=M --to-version=N) is designed for this.
  • Adding a new HSM key to an existing pkcs11 install for rotation (kek pkcs11-init --version=N+1).
  • Changing the HSM module path or token label — the lock tracks provider only, so this is allowed, but nothing verifies that the new token holds the same key material. Startup only checks that a key exists under each expected label and that the current version round-trips; a same-labelled but different key passes both checks and silently makes every existing envelope undecryptable. Only re-point at a token that is a genuine replica or backup of the original.

When you DO need to migrate​

  • env → pkcs11 to move KEK into an HSM for compliance.
  • pkcs11 → env (rare; might happen if an HSM is decommissioned).
  • Swapping one HSM family for another (e.g. SoftHSM2 dev → Fortanix DSM production) — technically still pkcs11 but the key material is in a different physical device, so you need a fresh DB + new keys.

Migration procedure​

1. Back up everything on the current install​

mongodump --uri=$MONGO_URI --out=/backup/pre-migration-$(date -I)
# Plus your secrets.env or K8s Secret, kept in a secure off-host
# location.

2. Export data that needs to survive​

CertAutoPilot's encrypted records (certificate private keys, DNS credentials, module secrets, etc.) are tied to the current KEK. If you need to preserve certificates:

  • Download each cert's PEM bundle via the API or UI while the service is still running. See Download.
  • Export DNS / module credentials via the same route, documenting which belongs to which project.

3. Provision a fresh MongoDB​

Different cluster, different DB name, or a clean db.dropDatabase() — whichever your operational policy allows.

4. Install with the new provider​

Standalone — one command, PIN inline or via a mode-0600 file:

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
| sudo bash -s -- --version=<latest> --mongo=local \
--kek-provider=pkcs11 \
--pkcs11-module=/opt/fortanix/pkcs11/fortanix_pkcs11.so \
--pkcs11-token-label=<label> \
--pkcs11-pin='<hsm-user-pin>'

K8s: run certautopilot kek pkcs11-init --version=1 against the new MongoDB before any Deployment starts — a Job or one-off pod with the same config, HSM library mount and PIN. It creates the HSM key, the install lock and the keystore row; without it every pod exits with provider="pkcs11" but the database has no install record. Then update values.yaml with secrets.encryption.provider: pkcs11 + pkcs11.* settings and helm install against the new namespace or cluster.

5. Re-import data​

Via the API or UI. Create projects, re-upload DNS credentials, re-issue certificates, etc. The new install wraps everything under the new provider from scratch.

6. Retire the old install​

Once the migration is verified. The old MongoDB + secrets are the only way to decrypt the old envelopes, so keep backups until you're confident the new install covers all your workloads.

Install-lock error messages​

When you run kek pkcs11-init --version=1 against a fresh MongoDB, it writes the lock as {provider: "pkcs11"}. From that moment on, any binary invocation pointing at this MongoDB with encryption.provider: env will fail fast at BuildRegistry:

KEK provider mismatch: installed="pkcs11" (since 2026-05-01T...)
but config says "env". Provider is immutable after install.

Similarly, running pkcs11-init against an env-locked MongoDB fails immediately without touching the HSM:

refusing to run: this database is locked for provider="env"
(since 2025-08-15T09:00:00Z by service). pkcs11-init cannot run
against a "env"-locked installation.

These guards prevent operators from accidentally contaminating one installation with another's data or keys — the implicit coupling of envelope tags + per-provider wrapping logic would otherwise allow silent corruption.

Rollback​

If a migration attempt fails, stop the new service, restore the old MongoDB + secrets from backup, start the old service. Rollback is "use the old DB again" — but only if you migrated to a separate database and still hold the step-1 backup. If you took the db.dropDatabase() option in step 3, there is nothing to roll back to: that destroys the only copy of the old envelopes.

Troubleshooting​

I forgot to export a cert before migration​

The old install still has its DB and secrets. Start it again against the original infra (you kept the backups in step 1), download the missing cert, then re-upload to the new install.

Can I just kek rotate to move to the HSM?​

No. kek rotate re-wraps envelopes with a new KEK version of the same provider. It cannot cross the provider boundary. See Why no in-place migration.

See also​