KEK rotation
Rotate the key-encryption-key without downtime, even across a multi-replica cluster. The fleet-aware rotation is the headline feature in 1.4.x.
Why rotate
Most compliance regimes (SOC 2, ISO 27001, PCI DSS) require periodic rotation of long-lived encryption keys. A 12-month rotation is typical; 6 months for high-assurance environments. CertAutoPilot makes it cheap so you can do it more often without operational fear.
What rotation actually changes
Reminder: every secret field is encrypted with a per-field DEK, and the DEK is wrapped by a versioned KEK. Rotation:
- You provision KEK_v(N+1) alongside KEK_vN and restart the fleet so both are loaded.
- Rotation walks each collection a page at a time. For every encrypted field it decrypts the value, generates a fresh DEK, re-encrypts the value under it, and wraps that DEK with KEK_v(N+1).
- It updates each record's
kek_versionfield in the same write. - It flips the keystore's active row to v(N+1). KEK_vN stays in place and stays loaded — retiring it is a separate, deliberate step (
kek remove) that you take once your backup-retention window has closed.
Two consequences worth planning around. Cost scales with the size of the secrets, not just the record count, because every value is genuinely re-encrypted; and each secret exists as plaintext in the worker's memory for the moment it is re-sealed. This is also the point at which AAD field binding is applied — if encryption.aad_binding is not enabled fleet-wide before you rotate, the re-seal writes those fields back unbound.
Knowing when a version is safe to retire
certautopilot kek status reports a RECORDS column — how many stored documents are still sealed under each version.
VERSION STATUS SOURCE RECORDS CREATED RETIRED REMOVED
v1 retired env 0 2026-08-01T09:12:00Z 2026-08-27T19:00:23Z -
v2 active env 2481 2026-08-27T19:00:23Z - -
The same number appears in Settings → KEK Versions, next to Loaded by. Read them together, because they answer different questions:
- Loaded by counts live processes that hold the key material. It does not change as data moves forward.
- Records counts stored documents that still need the key. This is the number that decides whether a version can be retired.
A version showing RECORDS = 0 has no documents stamped with it. Treat that as an indicator, not a clearance — certautopilot kek remove runs the real check. It refuses while stamped records remain, and also refuses in three subtler cases where a zero would be misleading:
- a document whose version stamp is missing, or disagrees with the envelopes it actually carries — run
certautopilot kek repair-versions, then rotate and retry; - a document holding encrypted data nested inside a JSON field, which rotation cannot move forward. Two things can be named here: an open approval request carrying a secret value (approve it, resubmit it, or let it expire), or syslog TLS material that has not yet moved out of the settings record — check the startup log for why the move did not complete.
- a field whose envelope is sealed under the version even though no document is stamped with it. A stamp records a document's oldest envelope, so a newer one can hide behind it; the check reads the envelopes themselves.
--force skips every one of these checks and accepts that the affected data becomes unreadable.
These guards live in kek remove. Removing a CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V* line from secrets.env yourself bypasses them completely. The service still starts — startup only verifies that the current version is present — but it logs a warning for every keystore version whose material is missing, and documents sealed under that version fail at the moment something reads them. Retire versions with kek remove, and check the startup log after any change to secrets.env. If you deleted the line by hand, the remedy is to put it back — the keystore still lists the version as retired, so there is nothing to reinstate. certautopilot kek reinstate --version=N applies only after kek remove marked a version removed, and only while its key material still exists.
Fleet-aware rotation
In a multi-replica cluster, every API and worker pod has the KEK in memory. Rotation must guarantee:
- No pod ever encrypts under v(N+1) before every pod knows v(N+1).
- No pod ever fails to decrypt because it doesn't have a KEK version some record references.
Rotation is a single operator-initiated pass, not a phased protocol — there are no phases to watch and none are reported. What protects a fleet is a preflight plus one atomic flip:
- Before it starts, the preflight requires every live process (one that has heartbeated within the last 60 seconds) to already hold the target version's key material and to be running the same KEK provider.
certautopilot kek verify --target=Nruns exactly this check. - While it runs, every process still writes under the old version. Records move one at a time; both versions stay loaded, so any process can read either.
- At the end, the keystore's active row flips to the new version in a single write. Each process picks that up on its next heartbeat (about 30 seconds) and starts sealing new data under it, without a restart.
Records written under the old version while the rotation is running may not be caught by that pass — batches page forward through each collection and do not revisit ground they have covered. Nothing is lost: those records stay readable, and kek remove refuses while any of them still reference the old version. Expect to run a second kek rotate --from-version=<old> --to-version=<new> to sweep them up before retiring the old key.
Triggering a rotation
Rotation is initiated from the CLI on a host with KEK file access. Before running, every process in the fleet must be restarted with the new KEK loaded — the rotate command's preflight refuses if the target version is not among the loaded versions. Loading it is what matters: CERTAUTOPILOT_ENCRYPTION_CURRENT_VERSION only seeds which version a fresh install starts on and is ignored once the keystore has any versions, so it does not have to be bumped before rotating. Confirm readiness first:
certautopilot kek verify --target=2
certautopilot kek rotate --to-version=2
Optional flags: --from-version=N, --batch-size, --concurrency. Pass --from-version when you are sweeping records left on an older version than the current source, or when this host is pinned by encryption.current_version_override — the default silently follows that pin. Note --concurrency is accepted but currently has no effect; how many collections run at once is set by worker.max_concurrency.
On a multi-node standalone cluster
Do not do this by hand, node by node. Every node needs the new key material in its own secrets.env and every node has to restart before the preflight will pass, and distributing key material by hand is exactly where it ends up somewhere it should not be. Use the cluster wrapper from the first node:
cap-cluster kek-rotate <new-version>
It mints the new key (or, for PKCS#11, creates it inside the HSM — no material is copied between nodes), writes it to every node's secrets.env over stdin rather than the command line, restarts each backend so the version loads, starts the rotation, and waits for it to finish. cap-cluster kek-add <N> does the distribution and restart alone if you want to stage the key ahead of the rotation; cap-cluster kek-drop <N> removes retired material fleet-wide afterwards.
On a single-node standalone install, cap kek rotate --to-version=<N> after adding the key locally is enough. On Kubernetes, add the encryption-env-kek-v{N} Secret key, roll the pods, then kubectl exec … certautopilot kek rotate --to-version=<N> --config=/config/config.yaml.
Settings → KEK Versions in the UI is a read-only monitoring surface — it shows the version table, rotation history, in-progress record counters (processed / skipped / failed), and fleet drift / process-vs-keystore mismatches. There is no "Rotate" button, no in-UI cancel, and no UI scheduling; every transition is operator-initiated via CLI.
Every record the rotation moves can afterwards be read only with the new KEK. Lose the new key material and you lose those secrets — and the loss is silent, because a fleet missing a retired key still starts normally. Sync the new KEK to your secret store before triggering the rotation, verify the sync, and keep the old key until your backup-retention window closes.
Cancel & reverse rotation
An in-progress rotation can be halted with certautopilot kek rotate --cancel. Running batches finish their current item at the next boundary and exit cleanly. Already-rewrapped envelopes stay on the new version; un-rewrapped envelopes stay on the old version — the database is left mid-rotation, which is safe to read from (both KEKs are still loaded) but should be resolved by either resuming or reversing.
Rotation is forward-only and cannot be reversed in place: kek rotate refuses any target that is not higher than the source, so a command naming the old version as the target fails with target version 1 must be greater than source version 2. There is no kek rollback.
Being left mid-rotation is not a problem to undo — both KEKs are loaded, so everything reads correctly. Resume by re-running the same rotation; it picks up the records still on the old version.
If you genuinely need to move off a key you have just rotated into — a key you no longer trust, say — provision a further version whose material you control and rotate forward into that. The version you left behind stays readable until you kek remove it.
Scheduled rotation
Rotation is currently a deliberate, operator-driven action — there is no built-in scheduler. To rotate periodically, wrap certautopilot kek rotate --to-version=<next> in your existing job runner (cron, systemd timer, Kubernetes CronJob) on the host that holds KEK file access — making sure the new KEK material is loaded fleet-wide first, since that is what the preflight checks. Note that a rotation writes no audit-log entries and is therefore not forwarded to syslog, whether started by hand or by a job runner — the operator identity is kept on the rotation record and surfaced in the UI's rotation history instead.
HSM-backed KEKs
If your KEK lives in an HSM (PKCS#11), you create the new key inside the HSM first with certautopilot kek pkcs11-init --version=N; kek rotate refuses to start if that material is not already loaded. The key never leaves the HSM. The wrap/unwrap operations all happen in HSM-land. Performance is bounded by HSM throughput; expect rotation to take longer (5–60 minutes for typical inventories).