Skip to main content

Fleet readiness

A KEK rotation only works if every running process has already loaded the new version. The fleet-readiness check is what guarantees that. Each process heartbeats into MongoDB every 30 s reporting its loaded versions and current version; kek verify --target=N refuses to ready-up if any live process is missing the target key material.

Why a pre-rotation gate​

A rotation reads each envelope, unwraps with the old KEK, then rewraps with the new KEK using an explicit target-version wrap (crypto.EncryptWithVersion) so the outer doc.kek_version field and the inner envelope's kv tag are always consistent. For that rewrap to succeed on every process that may run a batch, every process must have the target key material loaded. If even one lagging process (missing VN+1) picked up a batch, it would fail at the wrap step — the rotation filter would stall at that collection.

The readiness gate also covers a subtler case: processes that never loaded the new key would continue writing new secrets with the old KEK version during rotation. Each such write produces a document tagged with the old version. It is not lost, but it is not reliably swept up by the running rotation either: batches page forward through a collection by document id and never revisit ground already covered, so a record written behind the cursor survives that pass. Keeping every node at least LOADED on the target avoids that churn; the per-node current is flipped atomically at rotation finalize (SwapActive in the keystore) and each node hot-reloads it on its next heartbeat tick without a restart.

The heartbeat mechanism​

  • Every CertAutoPilot process (API / worker / scheduler) writes a record to the process_heartbeats collection every 30 s.
  • The record carries: hostname + PID (or pod name for K8s), loaded KEK versions, current version, provider, process role, last start time.
  • A TTL index expires records 120 s after last write. So a process that crashed 2 minutes ago drops off the roster.

kek verify --target=N​

cap kek verify --target=2    # standalone hosts; in K8s: kubectl exec into a pod and run certautopilot kek verify --target=2
  • Reads the heartbeat collection.
  • Applies a 60-second freshness window — records older than 60 s are treated as stale (not as laggard). This protects against a restart in progress.
  • For each fresh record, checks: does loaded_versions contain target? Is the reported provider the same as the local CLI's? Current_version is NOT required to match the target — the keystore's active version only flips at rotation finalize, and every live node then hot-reloads it within one heartbeat tick (~30 s).
  • Exit codes:
    • 0 — READY. Every fresh process has the target version loaded and reports the same provider as the local CLI.
    • 2 — NOT READY. One or more processes lag (missing the target version, or on a different provider). The laggard list is printed on stderr.
    • 1 — error (DB unreachable, invalid target).

Interpreting the laggard list​

NOT READY for rotation to v2. Lagging instances:
  - cap-api-1/3f2a91 (pid=1234 mode=all): provider=env loaded=[1] current=1
Provision the v2 key material on the listed instances (env var or HSM init) and restart them before rotating.

The list goes to stdout, not stderr — 2>/dev/null will not filter it out, and >/dev/null discards exactly the part you wanted.

  • loaded=[1] current=1 — the process never loaded V2. Env var missing or typo on that host; HSM tokens: check the key label exists in the token.
  • loaded=[1,2] current=1 — NOT a laggard. V2 is loaded; current being V1 reflects the keystore's still-active V1, which is exactly the pre-rotation state. kek verify --target=2 passes on this row.
  • provider=env while the CLI reports provider=pkcs11 (or vice versa) — fleet is mid-provider-migration. Rotation refuses until every process matches; finish the migration first (provider migration).

Edge cases​

--local​

kek verify --local skips the fleet check entirely and only verifies the current CLI's ability to round-trip wrap/unwrap. Useful for confirming a single-host standalone has the new KEK loaded.

Newly-started pod not yet visible​

A process emits its first heartbeat immediately at startup rather than waiting a full tick, so it appears in the roster within a second of becoming healthy — no 30-second wait is needed.

What verify cannot see is a process that has not got that far: one still building its key registry, or one that failed to start at all. Readiness only speaks for processes that came up. Count the roster against the number of processes you expect before trusting a READY verdict.

Long-lived job​

Heartbeats come from the process, not from each job. A worker running a 10-minute distribution is still heartbeating in the background; verify reflects its current state.

Read-only replicas​

If you run on a MongoDB read-replica (secondaryPreferred or similar), the heartbeat writes may be slightly delayed. The 60-second freshness window absorbs normal replication lag.

Readiness during rotation​

kek rotate --from-version=M --to-version=N runs the same readiness check internally as kek verify, then starts a background job. The per-doc UpdateOne filter ({kek_version: fromVersion}) makes the rotation tolerant to mid-rotation topology changes: a process that restarts or crashes never half-rotates a document (each rewrap is atomic across all encrypted fields on a doc), and a record that is rotated is rotated completely. Records written under the old version during the run are a different matter: batches move forward through each collection by document id, so anything landing behind the cursor is missed by that pass. Stragglers are safe — both keys are loaded, and kek remove refuses while any record still references the old version — but expect to run a second kek rotate --from-version=OLD --to-version=NEW to sweep them up before retiring the old key.

If a process comes back mid-rotation with a missing target key (operator typo on restart), the documents it touches fail at the re-seal step and the rotation's failed_records counter increments. Those documents are counted and skipped — the batch advances past them and nothing retries them automatically. The rotation reaches a terminal completed_with_errors status, not a visibly stuck in_progress one, so treat that status as unfinished work rather than a successful run.

Recover by restoring the fleet state and re-running the rotation explicitly: kek rotate --from-version=OLD --to-version=NEW. Already-rotated documents are filtered out by their version tag, so the re-run only picks up what was missed. Note that --from-version becomes mandatory once the active pointer has moved — a bare kek rotate --to-version=NEW is rejected because the target is no longer higher than the source.

Post-rotation: every live node's heartbeat tick (~30 s) detects the SwapActive in the keystore and hot-reloads its in-memory current KEK. kek remove --version=OLD includes a built-in safety check that refuses while any live node still reports OLD as current — combined with the auto-reload, it becomes observably safe to retire the old version as soon as the heartbeat convergence window passes.

Programmatic access​

The heartbeat roster is exposed via the cluster-instances API (GET /api/v1/settings/cluster-instances, admin role) and rendered on the Settings → Cluster page. Each row carries loaded_kek_versions, current_kek_version, provider, last_heartbeat, and started_at. The Settings → KEK versions page adds a fleet-readiness summary (N/N instances on v<active>). For alerting, poll the API and diff against the expected target version; a future release will add a dedicated Prometheus gauge for laggard count.

Troubleshooting​

"Ghost" laggard that isn't actually running​

The heartbeat TTL is 120 s. A process that crashed should disappear within 2 minutes. If you still see one, check MongoDB cluster health — write replication to the primary may be stalling.

Verify always reports one process lagging​

Often a stuck init container or a one-off CronJob that starts with an older image. In K8s: kubectl get pods -L certautopilot.version --all-namespaces.

See also​