Cluster instances
Settings → Cluster lists every live CertAutoPilot process — useful the moment you run more than one instance (see Multi-VM deployment). Each process heartbeats into MongoDB every 30 seconds; the page joins that roster with the leader locks and KEK state. Backed by GET /api/v1/settings/cluster-instances (admin role). Auto-refreshes every 30 seconds.
The instance table
| Column | Meaning |
|---|---|
| Instance | Operator-facing instance name plus the full process ID (name/short-uuid). |
| Mode | Runtime mode the process was started with: api, worker, scheduler, or all — see Runtime modes. |
| Host / PID | Hostname and process ID. |
| Version | Binary version — spot mixed-version fleets during rolling upgrades. |
| Uptime | Since process start (hover for the start timestamp). |
| Last heartbeat | Age of the most recent heartbeat (hover for the timestamp). Live processes report within the last minute. |
| Roles | Leader badges currently held: scheduler leader, domain leader, discovery leader. Non-leaders show —. |
| DB (replica-set deployments only — the column is not rendered at all for a standalone database, where the summary line above the table says everything) | The replica-set role of the database member running on the SAME machine as this instance: PRIMARY where writes are accepted, SECONDARY elsewhere, — for a standalone database or when the role cannot be determined. It is not the member this instance sends writes to — the driver routes those to the primary from every node. |
| KEK | The KEK version this process wraps new envelopes with, plus all loaded versions in parentheses. A warning-colored badge means the host lags the fleet's current version — sync its key material before the next rotation. See KEK versions page. |
Above the table a single line summarises the database itself: whether it is a standalone instance or a replica set, and for a set, which member is currently the primary. Together with the DB column it answers "where do writes go right now" without leaving the page.
How instances appear and disappear
- Each process upserts a heartbeat record on start and every 30 seconds after (
scheduler.heartbeat_interval, default 30s). - The page counts a process as live if its heartbeat is younger than 2× the heartbeat interval (60s by default) — the empty state and the Cutoff timestamp at the bottom reflect this.
- On graceful shutdown the process deletes its own record immediately. A crashed process drops off the live list within ~60s and its record is purged by a 120s MongoDB TTL index.
There is no stale row to watch for: an instance whose heartbeat falls outside the cutoff is filtered out of the table entirely. The signal is a row disappearing — a process that vanishes without a graceful shutdown crashed or lost MongoDB connectivity. Compare the row count against the number of processes you expect, and check the Cutoff timestamp at the bottom of the page.
Leader election
Three components run leader-elected singletons, each with its own MongoDB lock (locks collection, application-enforced expiry via findAndModify — no TTL index):
| Lock key | Role badge |
|---|---|
certautopilot:scheduler:leader | scheduler leader |
certautopilot:domain-checker:leader | domain leader |
certautopilot:discovery-checker:leader | discovery leader |
Mechanics:
- Lock TTL is 90 seconds (
scheduler.leader_lock_ttl); the holder renews it every heartbeat tick (30s). Config validation logs an error ifheartbeat_interval >= leader_lock_ttlbut does not block startup — the service will run, look healthy, and continuously lose leadership. Check the startup log after changing either value. - Non-leaders retry acquisition every 30 seconds plus up to 15 seconds of random jitter.
- When the leader dies, its lock stays valid until it expires (up to 90s), then the first retrying replica takes over — worst-case failover is roughly 135 seconds (TTL + retry wait + jitter). Sweeps missed in that window run on the new leader's next cycle; nothing is lost, only delayed.
The three roles can land on different instances — that's normal. See Scheduler status & sweeps for what each leader actually runs.
Warning cards
- KEK mismatch — some live instances report a different current KEK version than the API server serving this page. Sync
secrets.env(env provider) or verify HSM reach (pkcs11) on the listed hosts. This is not the rotation gate: the preflight checks that every live instance has the target version's key material loaded, which is whatcertautopilot kek verify --target=Nreports. - Cluster view is degraded — the heartbeat or leader-lock read failed (each bounded to 5 s), so the instance table and role badges are empty or incomplete. The database summary just below usually explains why; a replica set with no primary is the common cause. The rest of the page still renders.
- Database unreachable from this instance — the instance serving the page cannot reach the database. The rest of the page still renders from what it already knows, deliberately: an incident dashboard that returns an error tells you nothing about the fleet.
- Replica set has no primary — the set is up but no member accepts writes, so every write is failing. Usually an election in progress (resolves in seconds) or a lost quorum.
certautopilot statusprints the same view in a terminal and exits non-zero, so it can be wired into monitoring. - Leader locks without a live heartbeat ("orphan leaders") — a lock is active in MongoDB but no live instance matches its owner. Most often the lock was written by an older binary that predates cluster visibility, so the UI can't correlate the opaque owner to a process row. The lock expires and is re-acquired on the next cycle (typically within 90s); persistent orphans warrant checking MongoDB health.