KEK versions page
Settings → KEK Versions is the read-only monitoring surface for envelope-encryption key state. It shows which KEK versions exist, which one new envelopes are wrapped with, how many live instances have each version loaded, and the progress/history of rotations. There is no Rotate button — every transition is operator-initiated via the CLI (certautopilot kek rotate). For rotation mechanics and the runbook, see KEK rotation.
The page requires the admin role, the same as Cluster instances. It auto-refreshes every 30 seconds; a Refresh button forces a reload.
Summary card
| Field | Meaning |
|---|---|
| Provider | The KEK provider in use (env or pkcs11). |
| Current version | The version this API server wraps new envelopes with. Shown in green normally; warning-colored when it disagrees with the keystore, alongside a second red keystore: vN tag showing what the keystore actually says (see drift alerts). |
| Fleet readiness | N/M on v<current> — how many live instances are on the current version out of all live instances. Green when everyone agrees, warning-colored when instances lag. Reads no live instances when nothing has heartbeated recently, which is not the same as 0 / 0 — it means the page cannot see the fleet at all. |
| Active rotation | v<from> → v<to> when a rotation is in progress, otherwise none. After kek rotate --cancel it also carries a cancel requested tag — that tag is the confirmation the request was registered; the rotation stops at the next batch boundary. |
Versions table
One row per KEK version recorded in the keystore (kek_versions collection):
| Column | Meaning |
|---|---|
| Version | v1, v2, … |
| State | active (the wrap target), retired (loaded for decrypt but not written with), removed (marked via kek remove — the material is then ignored at startup even if it is still on the host; reversible with certautopilot kek reinstate --version=N, after which every node must be restarted so the key loads again). Note retired covers two different situations: a version superseded by a rotation, and a newly staged version you have not rotated into yet — a key added to secrets.env or the HSM is recorded as retired the first time it is seen. |
| Source | Where the key material comes from. |
| Created / Retired / Removed | Lifecycle timestamps. |
| Loaded by | count / total — how many live instances currently have this version's key material loaded, from the heartbeat roster. |
| Records | How many stored documents are stamped with this version. This answers a different question from Loaded by: that column counts machines holding the key, this one indicates whether the data still needs it. unknown means the count could not be taken — never read it as zero. |
A stamp records a document's oldest envelope, so Records = 0 is an indicator rather than a clearance: a newer envelope can hide behind an older stamp. certautopilot kek remove reads the envelopes themselves, along with values nested inside JSON documents and documents with no stamp at all, and refuses if any of them still needs the version. Treat the column as the signal to go and try, not as permission.
On the active version a non-zero count is the healthy steady state — that is where new data is written.
How to read the Loaded by colors:
- active + full load — green: healthy.
- active + partial load — warning: fleet drift, some hosts are missing the active version. Sync
secrets.env(env provider) or verify HSM reach (pkcs11) on the lagging hosts and restart them. - retired + any count — neutral: expected during the retention window after a rotation; use
kek removewhen you're ready to drop it. CheckCreatedand the version number first — a newly staged version is alsoretired, and removing that one takes away the key you are about to rotate into (kek rotatethen refuses it outright). - removed + a red count — documents are still sealed under a version the fleet no longer loads. Run
certautopilot kek reinstate --version=N, restart the fleet, then rotate those documents forward. - removed + count > 0 — warning:
kek removeran but the env var / HSM key is still present on some hosts — clean up and restart. - removed + count = 0 — green: fully drained.
Drift and mismatch alerts
Three alert banners can appear above the tables:
- Version pinned by config override (info) — the process was started with
encryption.current_version_override, which pins the wrap target and bypasses the keystore. Expected during disaster-recovery pinning; remove the override and restart to return to normal. - Keystore drift (error) — this API server wraps new envelopes with a different version than the keystore's active row. The keystore is the source of truth, and processes re-read it on every heartbeat (30 s by default) and adopt the active version without a restart, so this normally clears itself within a tick. If it persists, this host could not adopt that version — almost always because the version's key material is not loaded here; check
secrets.env/ HSM reach and the service log. Restarting alone will not fix it. The banner is per-process, so behind a load balancer it may appear for only some replicas. An intentionalencryption.current_version_overridepin also produces it, and is reported separately. - Fleet mismatch (warning) — one or more live instances report a KEK version different from the one the API server that served this page is on. There is no fleet-wide reference version, and an instance that is ahead is flagged exactly like one behind, so if the responding replica is itself the outlier every healthy instance appears mismatched — cross-check Current version against the keystore row before acting. This is not what the rotation preflight gates on: the preflight requires only that every live instance has the target version's key material loaded and runs the same provider (
certautopilot kek verify --target=N). A persistent mismatch is still worth investigating — it usually means a host cannot adopt the active version because its key material is missing. Per-host details are on the Cluster instances page.
Active rotation card
While a rotation runs, a progress card shows: processed / skipped / failed record counters, start time, the initiating operator, and a progress bar (warning-colored if any records failed). Immediately after kek rotate returns there is a brief window where the total record count is still 0 — the page shows a spinner ("Waiting for first batch count…") instead of a misleading 0% bar.
Monitor or cancel from the CLI:
certautopilot kek status
certautopilot kek rotate --cancel
Rotation history
The 10 most recent rotations — including the one in progress, which is lifted out into its own card above, so during a rotation this table shows nine. Each row carries: from → to versions, status (completed, completed_with_errors, failed, cancelled, in_progress, pending), start/completion time, duration, processed/skipped/failed counters, and who started it.
--from-version defaults to the keystore's active row, but passing it explicitly keeps the audit log unambiguous.
Checking readiness before a rotation
Before running kek rotate:
- Fleet readiness in the summary card must read
N/N— every live instance on the current version. - The fleet mismatch banner must be absent. If it lists lagging instances, fix key material on those hosts first (details per host on Settings → Cluster).
- The target version must show as Loaded by
N/Nin the versions table — every instance needs the new key loaded before rotation starts. - Confirm from the CLI with
certautopilot kek verify --target=<N>— see Fleet readiness for exit codes and laggard interpretation.