Operations
The day-2 surface: leader-elected scheduler, MongoDB-backed job queue with retries and dead-letter, the cluster instances view that joins heartbeats with locks, domain tracking for registrar-side expiry, and an incident runbook for the common failures.
Scheduler
The scheduler is a leader-elected cron. Exactly one instance is active at any moment; the rest heartbeat and wait. On leader failure the lock expires and a standby takes over. Sweeps run on a configurable cadence and enqueue jobs for the worker pool.
scheduler:
interval: 1h # sweep cadence
leader_lock_ttl: 90s # lock lifetime
heartbeat_interval: 30s # leader re-asserts the lock every N
heartbeat_interval should be ≤ leader_lock_ttl / 2 so a short network blip doesn't expire the lock. Defaults satisfy this.
Leader election
- A MongoDB
findAndModifyon thelockscollection atomically acquires the lock. - The instance holding the lock runs all sweeps; standbys watch for it to expire.
- On graceful shutdown (SIGTERM), the leader releases the lock explicitly so the standby picks up immediately. On crash, wait
leader_lock_ttl. - A stale leader that can't reach MongoDB realises it lost the lock and steps down; the standby takes over. Split-brain is prevented by MongoDB's single-writer semantics on the
findAndModify.
What the scheduler does
Every cycle (the interval is Settings → General → Renewal Check Interval, default 60 minutes; scheduler.interval is only the fallback before settings load), the main scheduler leader runs, among others:
- Renewal check — find auto-renew certs whose
next_renewal_athas passed (threshold before expiry, default 30 days, or earlier when ACME ARI says so). Enqueuerenew_certificate/msca_renew_certificate. - Renewal failure safety net — put auto-renew certs stuck in
renewal_failedback on the retry ladder after a 24-hour cool-down. - Expiration check — every scheduler cycle, enqueue
certificate_expiration_check, which marks expired certificates, emitscert.expiredonce per expiry, and emitscert.expiring_soonby each certificate's renewal plan: an overdue automatic renewal, the lead times of a certificate nobody renews automatically, and the last 3 days (details). - Revocation check — once an hour, check certificates' revocation status at the CA.
- ARI refresh — for CAs that support ACME Renewal Information, refresh the suggested renewal window (up to 100 certs per cycle).
- Distribution sweep — enqueue
distribution_executefor distributions pending re-distribution, reconcile stuck fan-outs and close out stalled rollbacks. - Recovery sweeps — stuck issuances/renewals (zombie recovery), leftover DNS cleanup, orphaned phased DNS-01 workflows, stuck AD CS polls, stalled bulk actions, stale notification reservations, expired HTTP-01 tokens.
- Approval expiry — flip pending approval requests past their expiry to
expired.
Two further leader-elected components run their own loops: the domain checker (WHOIS / DNS health checks for tracked domains, every domain_check_interval_minutes) and the discovery checker (hourly: enqueue discovery_execute for sources whose next scan time has passed, and one discovery_findings_refresh and one discovery_orphan_cleanup per project with discovery sources). The full catalog with intervals is on Scheduler status & sweeps.
Job queue
Every asynchronous operation runs as a job in the MongoDB-backed queue. One handler per type. Workers claim jobs atomically, retry on failure with backoff, and land in dead-letter after N attempts.
Job types
| Type | Purpose |
|---|---|
issue_certificate / renew_certificate / revoke_certificate | ACME lifecycle. (There is no reissue_certificate job — a reissue enqueues issue_certificate.) |
validate_manual_dns / cleanup_dns | DNS-01 manual flow + post-issuance cleanup. |
msca_issue_certificate / msca_renew_certificate / msca_poll_pending | Microsoft AD CS lifecycle (enrollment over RPC or CES, renewal, pending-approval polling). |
await_dns_propagation / finalize_order | Phased DNS-01: wait for propagation without holding a worker, then finalize. |
certificate_expiration_check / domain_expiration_check / revocation_check | Scheduler-driven checks. |
distribution_execute / distribution_rollback | Module execution / rollback. |
notification.send | Channel delivery. |
discovery_execute | Network, AXFR, cloud-DNS or CT-log discovery scan. |
discovery_findings_refresh | Hourly recompute of a project's discovery findings and discovery expiry alerts. |
discovery_orphan_cleanup | Hourly (and a few minutes after a source delete) removal of discovered certificates nothing holds any more. |
kek_rotation_orchestrate / kek_rotation_collection | KEK rotation: orchestrator + per-collection re-wrap. |
States
pending → active → completed / failed / cancelled, and dead once a job has used all its attempts. A retryable failure goes back to pending with a delayed next run; failed is a terminal failure that is not retried. There is no in_progress, succeeded or dead_letter status — filtering on those returns nothing. (partial is a result status on a completed job, not a job status.)
Retry backoff (default): 30 s doubling per attempt — 30 s, 1 m, 2 m, 4 m, 8 m — with ±20 % jitter, capped at 10 minutes, up to the job's max_attempts (default 3). Certificate renewal is not retried through this queue ladder at all: it is re-driven by the scheduler on its own escalating schedule (15 min → 1 h → 6 h → 12 h → 24 h, 5 attempts). Error classification (network / io_transient / auth / validation / io_permanent) informs whether to retry — auth and validation go straight to failed without burning retries.
Queue lanes
Distribution jobs (distribution_execute, distribution_rollback) run on a dedicated distWorker. Main workers explicitly filter them out via UseRegisteredTypesAsFilter(). Benefit: a 500-target fan-out doesn't starve ACME / MSCA / notification jobs.
Browse, retry, cancel
- Jobs in the top navigation. Filter by status tab; the type filter is the
?type=URL parameter. See Jobs. - Click for detail — summary, workflow progress, and the job's logs, which stream incrementally while the job runs.
- Retry re-enqueues a failed / dead job. Project admin role required (rate-limited to 20 per minute). Retry doesn't change the payload — fix the underlying cause first (expired credential, etc.) then retry.
- Cancel moves a pending job to
cancelledimmediately; an active job gets a cancel request and stops at its next checkpoint. Project admin role required.
Idempotency & dead-letter
Job handlers are idempotent against their natural key — a second issue_certificate for the same cert while one is already in progress is dedup'd at the enqueue site. A second system run of the same distribution that meets the per-certificate/module lock is deferred to the distribution sweep (no failure, no lost version); a manual Execute fails with lock is held and is re-run; fan-out children are guarded by the aggregate state machine.
Jobs that exhaust their retry ladder transition to dead. A terminally failed or dead job emits job.failed (coalesced to one notification per job type per hour; certificate issuance, renewal, revocation, AD CS and distribution jobs report through their own cert.* events instead, and notification.send never emits it). Wire it to your ops channel. On Jobs, switch the status tab to Dead to inspect — fix the underlying cause then click Retry on the row (which re-enqueues the job and resets the attempt counter). The same Retry action is available on failed jobs.
Cluster instances
A consolidated view (Settings → Cluster) of every live CertAutoPilot process — identity, leader roles, uptime, loaded KEK versions — joined in one table so operators can answer who is doing what without touching mongosh.
Each row is one process, joined from two collections:
process_heartbeats— emitted every 30 s by every process regardless of run mode (api,worker,scheduler,all). Records liveness, PID, hostname, binary version, loaded KEK versions, andinstance_name.locks— distributed leader locks for the scheduler, domain checker, and discovery checker. Each active lock has a matching heartbeat owner; the UI stamps role badges onto that row.
Identity triple resolved at startup:
Name— operator-facing display. Resolution chain:server.instance_name→CERTAUTOPILOT_INSTANCE_NAME→POD_NAME→HOSTNAME→os.Hostname()→ literal"unknown".ID—<Name>/<short-uuid>, authoritative, used as the lock owner + heartbeat_id. The UUID suffix prevents collisions if two pods accidentally share a Name, and avoids resurrecting an old document on restart.Mode—api/worker/scheduler/all.
API: GET /api/v1/settings/cluster-instances (admin-only). Returns a list of instances with roles (e.g. scheduler_leader, domain_leader) and orphan_leaders (locks whose owner heartbeat has aged out — useful for spotting stale locks during a primary failover).
Domain tracking
A cert can be perfectly valid while the underlying domain expires out from under you. Domain tracking watches each registered domain's WHOIS expiry and fires alerts on the configured thresholds. Complementary to cert expiry — different time horizon, different responsible party.
Add a tracker: Domains → Add Domain. Enter the domain name; an initial check runs immediately. Each domain is then re-checked about once a day (sooner, from 6 hours, while lookups are failing), picked up by the domain checker every domain_check_interval_minutes.
Events: domain.expiring_soon once the registration is within 30 days of expiry (at most once a week per domain); domain.expired past the date; plus domain.dangling_dns, domain.dnssec_missing and domain.no_dmarc from the DNS health check. A failed lookup raises no event — thin-WHOIS TLDs (.ai / .gg / .io) return no expiry, which simply shows as unknown.
Lookups use registry WHOIS over TCP/43; RDAP is not used. The outbound network policy allows TCP/43 to WHOIS hosts by default.
Runbook
Playbooks for the common incidents.
Renewal failing for a specific cert
- Open the cert detail; scroll to Timeline. The
cert.renewal_failedevent carrieserror_class+attempt. - Click through to the failed job; inspect logs.
- By class:
- network — ACME directory or DNS provider unreachable.
curlfrom the worker host. - auth — DNS credential or ACME account revoked / expired. Re-issue, update credential.
- validation — DNS TXT record didn't propagate. Check zone TTL, registrar console for stale records.
- io_permanent — CA rejected the request (rate limit, policy). Read the CA's error verbatim in the log.
- network — ACME directory or DNS provider unreachable.
- Fix the root cause; click Renew on the cert to retry immediately, or wait for the next scheduler sweep.
Dead-letter jobs piling up
- Open Jobs; switch the status tab to Dead.
- Look for patterns by type — one type failing en masse = systemic (credential rotation, CA outage, MongoDB index missing).
- Resolve the common cause; click Retry on each affected row (no bulk-retry button — fix the cause once, retry the rows individually or via the API).
- For stragglers, inspect individually. Sometimes the right answer is Cancel (cert was revoked, no point retrying).
Scheduler won't elect a leader
- Check
certautopilot_scheduler_is_leaderacross pods. Should be exactly one at 1. - All zeros: no instance can write to
locks. MongoDB primary probably unreachable. - Two ones: caching glitch — wait one
leader_lock_ttl, one should drop. - Logs showing
context deadline exceededon lock renewal indicate slow MongoDB writes.
MongoDB replica-set failover
- Pods see connection errors for a few seconds during primary election.
- Driver retries; most in-flight requests recover.
- Scheduler loses its lock; standby re-acquires once the new primary accepts writes.
- Workers mid-job: current attempt may fail; next retry succeeds.
- If primary election takes longer than
leader_lock_ttl, sweeps are delayed correspondingly. Nothing to do otherwise.
Suspected KEK compromise
Treat as urgent. The KEK wraps every secret; if it leaks, every secret is at risk. Full procedure: KEK rotation.
- Generate a new KEK version (
openssl rand -hex 32). - Add to env / HSM. Restart the fleet so both old and new are loaded.
certautopilot kek verify --target=<new>— confirm all processes loaded.certautopilot kek rotate --from-version=<current> --to-version=<new>. A background job re-seals every stored secret.- Once
kek statusshows complete,kek remove --version=<old>. Drop the old key from env / delete the HSM key.
The bare binary cannot read the database credentials, so every command above fails with an unrelated Unauthorized error. Use cap kek … on a standalone host — the wrapper loads secrets.env and re-execs under sudo — or kubectl exec … certautopilot kek … --config=/config/config.yaml on Kubernetes. For a whole cluster, cap-cluster kek-rotate <new> does steps 2–4 in one command.
- Audit-log review: cross-reference admin actions in the compromise window. Consider rotating dependent secrets too (JWT secret, module credentials).
ACME provider outage
- Renewals + issuances for that CA fail with
networkerrors. - CertAutoPilot retries on the configured ladder — transient blips are absorbed.
- Extended outage (hours): create an account on a fallback CA (Sectigo / Google / ZeroSSL); reissue urgent certs manually.
- Once the CA recovers, retry the dead jobs from the outage window (there is no bulk retry — one row at a time, or via the API); renewals are re-driven by the scheduler on their own.
License expired with no replacement
- At
expthe backend enters a 7-day grace window. Cert lifecycle (issue, renew, distribute, revoke) keeps running; the UI shows an expired-banner and the license status endpoint reportsin_grace_period: true. - Licensing enforces two counts only: certificates and distribution targets. A Free instance is capped at 1 certificate and 1 target; an Enterprise licence takes its certificate cap from the licence itself (0 = unlimited) and has no target cap. Once a cap is reached, creating new certificates or targets is blocked; existing ones keep renewing and deploying.
- After the 7-day grace window nothing else closes. There are no per-feature gates — LDAP, OTP policy and syslog forwarding are available on every tier — and renewals, distribution and revocation continue regardless of licence state.
- Obtain a new license; Settings → License.
Fan-out distribution stuck
- Jobs → the Active tab, with
?type=distribution_executein the URL. - Many stuck:
distWorkerisn't running. Check worker pod health. - One stuck: inspect its log. Usually a specific target hanging on connect — TCP timeout will eventually resolve it.
- Cancel the parent distribution to mark all pending children cancelled. Retry after fixing.
Lost both MongoDB primary AND secret store
Catastrophic — you've lost the data and the ability to decrypt it.
- Restore MongoDB from the latest backup.
- Restore the secret store (
secrets.envor K8s Secret) from the same time window — KEK and envelopes must match. - Start the service.
- Any certs issued between backup time and loss are gone. Re-issue.
- Post-mortem: why are backups not synchronised? Fix the procedure.
Observability hooks
Operations alerting hangs off the metrics / events surface. The high-value alerts:
certautopilot_scheduler_is_leader— must be exactly one.certautopilot_scheduler_leader_transitions_total— incrementing frequently → MongoDB write instability or bad config ratio.certautopilot_job_queue_depth{status}— growingpendingmeans workers can't keep up.increase(certautopilot_job_total{status="dead"}[15m])— there is no dedicated dead-letter metric; alert on the rate at which jobs reach thedeadoutcome. Any non-zero increase warrants triage.job.failedevent — wire to triage channel.domain.expiring_soon/domain.expired— wire to whoever owns registrar renewals.
There is no scheduler.leader_election and no job.dead_letter event; a rule subscribed to either
will never fire. For leadership changes, alert on the certautopilot_scheduler_leader_transitions_total
metric instead.
Full metric reference: Observability.
Upgrades & backups
- Helm:
helm upgraderolling restart. Leader election handles the scheduler transition cleanly. - Standalone:
upgrade.sh. Service bounce; config preserved. - Backups: snapshot MongoDB + the secret store on the same schedule. Restore drills should rehearse the joint restore — out-of-sync KEK + envelopes is unrecoverable.