Skip to main content

High availability

Replicate every layer: a 3-member MongoDB replica set, multi-replica API, leader-elected scheduler, and rolling upgrades that never drop in-flight ACME orders.

Availability targets​

  • Survive a single zone failure with no data loss.
  • Survive a rolling restart of any one workload type with zero failed renewal jobs.
  • API 99.95% monthly. Background jobs eventually-consistent within 5 minutes.

MongoDB replica set​

Three voting members, one per AZ. Use a managed offering (Atlas, AWS DocumentDB-with-replica-set-protocol, or self-hosted with the Mongo Operator). Connection string lists all members.

CERTAUTOPILOT_DATABASE_URI="mongodb://m0,m1,m2/certautopilot?replicaSet=rs0&readPreference=primaryPreferred&w=majority"

Reads default to primary for consistency. A replica set is optional, not required: job claims and distributed locks are single-document atomic findAndModify operations, and the one multi-document transaction (the KEK version swap during rotation) falls back to a safe sequential path on standalone MongoDB.

Self-hosted 3-VM replica set via --nodes​

For VM deployments without a managed database, the standalone installer builds the replica set itself. One command on the first host installs the whole cluster:

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=<pinned> --nodes=10.0.0.11,10.0.0.12,10.0.0.13

What you get and what to know:

  • The first three nodes each run a keyfile-authenticated MongoDB 8.0 member of the caprs replica set (on hosts whose Linux kernel is 6.19 or newer — e.g. Ubuntu 26.04 — the installer automatically uses MongoDB 8.2 instead, because 8.0 refuses to start on those kernels) (the first node carries election priority 2); nodes 4+ run CertAutoPilot without a local mongod. Every node's backend uses a multi-host connection string, so losing any single member — including the primary — keeps the application available after the ~5-15s election.
  • Open 27017/tcp between the three mongo nodes. The installer adds peer-scoped firewalld/ufw rules when a managed firewall is active and warns loudly when it is not — on cloud VMs the security-group rule is yours to add, or the replica set will not converge.
  • 2 nodes with local mongo is refused (no quorum); use 3+ nodes or --mongo=external. Growing an existing single-node install into a replica set in place is not supported — start from fresh nodes or move to an external replica set.
  • The nightly backup timer (--enable-backup) runs on the first node and dumps with --oplog for a consistent point-in-time snapshot.
  • Updates run per node: mongo-less nodes and SECONDARY members first, the first node (primary) last.
  • Checking replica-set health: cap status on any node prints the deployment mode, every member with its current state (PRIMARY / SECONDARY / UNREACHABLE) and round-trip time, and exits non-zero when the set has no writable primary — usable from scripts and monitoring. The web UI shows the same information under Settings → Cluster: a DB column on the instance table marks which machine currently hosts the PRIMARY (the database leader) and which are SECONDARY members.

API replicas​

Run at least 3. The Ingress / Load Balancer should send health checks to GET /healthz; deep readiness is on GET /readyz (verifies MongoDB reachability).

Worker replicas​

Workers scale horizontally. They claim jobs from a Mongo collection; only one worker ever runs a given job. Queue depth is exported as a Prometheus metric (and the Helm chart's optional prometheusRule alerts on it) — scale workers on that signal, manually or via your own HPA wiring.

Scheduler & leader election​

Run two scheduler pods. They acquire a lease in MongoDB; only the leader enqueues time-driven work (renewal windows, ARI refreshes, expiration and revocation checks, distribution and discovery sweeps). If the leader stops renewing its lease, the standby takes over once the lease expires (code default 90 seconds).

The Helm chart overrides this to 2 hours

scheduler.leaderLockTTL in the chart's values.yaml is 2h, not the code's 90 s — so on a Helm install, scheduler failover can take up to two hours. Lower it to 90s unless you have a reason not to.

Why not 3 schedulers?

Two is enough. The lease itself is the correctness boundary; more replicas only buys you marginal failover speed and adds cost.

Clock synchronization​

All instances must run NTP (chrony/systemd-timesyncd) and stay within roughly ±15 seconds of each other. Leader election, job leases and sweep locks compare each machine's own clock against absolute timestamps stored in MongoDB; a fast-skewed instance can steal a lease that is still held (the tightest lock tolerates ~30 seconds of skew), producing duplicate concurrent runs. This is a hard requirement for any multi-instance deployment.

Rolling upgrade​

  1. No migration job runs — there are no schema migrations; indexes are ensured at startup.
  2. API rolls under Kubernetes' default maxUnavailable: 25% — the chart ships no PodDisruptionBudget, so add your own if you need to bound evictions.
  3. Worker rolls — in-flight jobs that lose their pod are re-claimed by a peer after the job lock expires (5 minutes).
  4. Scheduler rolls last; the leader steps down so the new pod can pick up the lease.
Windows fleets: device locks serialize at batch granularity

Deploys to the same Windows host (IIS / WinRM / Exchange modules) serialize on a per-host lock held for the WHOLE child batch. When several certificates deploy to overlapping Windows hosts, keep distribution_fanout_batch_size modest (≤25) so a competing batch never waits longer than the lock window; a child that does time out waiting is automatically re-enqueued once with a 5-minute delay.

New feature fields vs mixed-version workers

Configuration written by a NEW api pod can carry fields an OLD worker binary doesn't know (e.g. a distribution target-override type introduced in the new release). The old worker silently ignores the unknown field and runs with the pre-override behavior. Two rules: complete the rollout before configuring features the new release introduced, and don't downgrade below the introducing version while such configuration exists — after a downgrade every renewal would silently execute without it.

Backup & DR​

  • Snapshot MongoDB at the storage layer or use mongodump at least daily. Store offsite.
  • Back up the KEK separately. Without it the snapshot is useless.
  • Audit log forwarding to SIEM is your second line of defence — even if the primary site is unrecoverable, the audit trail of operations survives.

DR test​

Quarterly: restore the most recent snapshot to a clean cluster, supply the KEK, then verify three things — certautopilot kek status (every active and retired row has loaded key material), POST /api/v1/audit-logs/verify (the audit hash chain), and GET /readyz. The KEK check is the one that proves the pairing actually works; a restore that boots is not evidence that the data can be decrypted. The fixture project should issue a test certificate against Let's Encrypt staging end-to-end.

See also​