Cluster instances
Settings → Cluster lists every live CertAutoPilot process — useful the moment you run more than one instance (see Multi-VM deployment). Each process heartbeats into MongoDB every 30 seconds; the page joins that roster with the leader locks and KEK state. Backed by GET /api/v1/settings/cluster-instances (admin role). Auto-refreshes every 30 seconds.
The instance table
| Column | Meaning |
|---|---|
| Instance | Operator-facing instance name plus the full process ID (name/short-uuid). |
| Mode | Runtime mode the process was started with: api, worker, scheduler, or all — see Runtime modes. |
| Host / PID | Hostname and process ID. |
| Version | Binary version — spot mixed-version fleets during rolling upgrades. |
| Uptime | Since process start (hover for the start timestamp). |
| Last heartbeat | Age of the most recent heartbeat (hover for the timestamp). Live processes report within the last minute. |
| Roles | Leader badges currently held: scheduler leader, domain leader, discovery leader. Non-leaders show —. |
| KEK | The KEK version this process wraps new envelopes with, plus all loaded versions in parentheses. A warning-colored badge means the host lags the fleet's current version — sync its key material before the next rotation. See KEK versions page. |
How instances appear and disappear
- Each process upserts a heartbeat record on start and every 30 seconds after (
scheduler.heartbeat_interval, default 30s). - The page counts a process as live if its heartbeat is younger than 2× the heartbeat interval (60s by default) — the empty state and the Cutoff timestamp at the bottom reflect this.
- On graceful shutdown the process deletes its own record immediately. A crashed process drops off the live list within ~60s and its record is purged by a 120s MongoDB TTL index.
A stale instance (present but heartbeat > 1 minute old) usually means the process crashed or lost MongoDB connectivity; a healthy fleet never shows one for more than moments.
Leader election
Three components run leader-elected singletons, each with its own MongoDB lock (locks collection, application-enforced expiry via findAndModify — no TTL index):
| Lock key | Role badge |
|---|---|
certautopilot:scheduler:leader | scheduler leader |
certautopilot:domain-checker:leader | domain leader |
certautopilot:discovery-checker:leader | discovery leader |
Mechanics:
- Lock TTL is 90 seconds (
scheduler.leader_lock_ttl); the holder renews it every heartbeat tick (30s). Config validation enforcesheartbeat_interval < leader_lock_ttl. - Non-leaders retry acquisition every 30 seconds plus up to 15 seconds of random jitter.
- When the leader dies, its lock stays valid until it expires (up to 90s), then the first retrying replica takes over — worst-case failover is roughly 135 seconds (TTL + retry wait + jitter). Sweeps missed in that window run on the new leader's next cycle; nothing is lost, only delayed.
The three roles can land on different instances — that's normal. See Scheduler status & sweeps for what each leader actually runs.
Warning cards
- KEK mismatch — some live instances lag the fleet's current KEK version. Sync
secrets.env(env provider) or verify HSM reach (pkcs11) on the listed hosts and restart them before starting a rotation; the rotation preflight otherwise fails fleet-wide. - Leader locks without a live heartbeat ("orphan leaders") — a lock is active in MongoDB but no live instance matches its owner. Most often the lock was written by an older binary that predates cluster visibility, so the UI can't correlate the opaque owner to a process row. The lock expires and is re-acquired on the next cycle (typically within 90s); persistent orphans warrant checking MongoDB health.