Skip to main content

Operations

The day-2 surface: leader-elected scheduler, MongoDB-backed job queue with retries and dead-letter, the cluster instances view that joins heartbeats with locks, domain tracking for registrar-side expiry, and an incident runbook for the common failures.

Scheduler​

The scheduler is a leader-elected cron. Exactly one instance is active at any moment; the rest heartbeat and wait. On leader failure the lock expires and a standby takes over. Sweeps run on a configurable cadence and enqueue jobs for the worker pool.

scheduler:
interval: 1h # sweep cadence
leader_lock_ttl: 90s # lock lifetime
heartbeat_interval: 30s # leader re-asserts the lock every N

heartbeat_interval should be ≤ leader_lock_ttl / 2 so a short network blip doesn't expire the lock. Defaults satisfy this.

Leader election​

  • A MongoDB findAndModify on the locks collection atomically acquires the lock.
  • The instance holding the lock runs all sweeps; standbys watch for it to expire.
  • On graceful shutdown (SIGTERM), the leader releases the lock explicitly so the standby picks up immediately. On crash, wait leader_lock_ttl.
  • A stale leader that can't reach MongoDB realises it lost the lock and steps down; the standby takes over. Split-brain is prevented by MongoDB's single-writer semantics on the findAndModify.

What the scheduler does​

Every cycle (the interval is Settings → General → Renewal Check Interval, default 60 minutes; scheduler.interval is only the fallback before settings load), the main scheduler leader runs, among others:

  • Renewal check — find auto-renew certs whose next_renewal_at has passed (threshold before expiry, default 30 days, or earlier when ACME ARI says so). Enqueue renew_certificate / msca_renew_certificate.
  • Renewal failure safety net — put auto-renew certs stuck in renewal_failed back on the retry ladder after a 24-hour cool-down.
  • Expiration check — every scheduler cycle, enqueue certificate_expiration_check, which marks expired certificates, emits cert.expired once per expiry, and emits cert.expiring_soon by each certificate's renewal plan: an overdue automatic renewal, the lead times of a certificate nobody renews automatically, and the last 3 days (details).
  • Revocation check — once an hour, check certificates' revocation status at the CA.
  • ARI refresh — for CAs that support ACME Renewal Information, refresh the suggested renewal window (up to 100 certs per cycle).
  • Distribution sweep — enqueue distribution_execute for distributions pending re-distribution, reconcile stuck fan-outs and close out stalled rollbacks.
  • Recovery sweeps — stuck issuances/renewals (zombie recovery), leftover DNS cleanup, orphaned phased DNS-01 workflows, stuck AD CS polls, stalled bulk actions, stale notification reservations, expired HTTP-01 tokens.
  • Approval expiry — flip pending approval requests past their expiry to expired.

Two further leader-elected components run their own loops: the domain checker (WHOIS / DNS health checks for tracked domains, every domain_check_interval_minutes) and the discovery checker (hourly: enqueue discovery_execute for sources whose next scan time has passed, and one discovery_findings_refresh and one discovery_orphan_cleanup per project with discovery sources). The full catalog with intervals is on Scheduler status & sweeps.

Job queue​

Every asynchronous operation runs as a job in the MongoDB-backed queue. One handler per type. Workers claim jobs atomically, retry on failure with backoff, and land in dead-letter after N attempts.

Job types​

TypePurpose
issue_certificate / renew_certificate / revoke_certificateACME lifecycle. (There is no reissue_certificate job — a reissue enqueues issue_certificate.)
validate_manual_dns / cleanup_dnsDNS-01 manual flow + post-issuance cleanup.
msca_issue_certificate / msca_renew_certificate / msca_poll_pendingMicrosoft AD CS lifecycle (enrollment over RPC or CES, renewal, pending-approval polling).
await_dns_propagation / finalize_orderPhased DNS-01: wait for propagation without holding a worker, then finalize.
certificate_expiration_check / domain_expiration_check / revocation_checkScheduler-driven checks.
distribution_execute / distribution_rollbackModule execution / rollback.
notification.sendChannel delivery.
discovery_executeNetwork, AXFR, cloud-DNS or CT-log discovery scan.
discovery_findings_refreshHourly recompute of a project's discovery findings and discovery expiry alerts.
discovery_orphan_cleanupHourly (and a few minutes after a source delete) removal of discovered certificates nothing holds any more.
kek_rotation_orchestrate / kek_rotation_collectionKEK rotation: orchestrator + per-collection re-wrap.

States​

pending → active → completed / failed / cancelled, and dead once a job has used all its attempts. A retryable failure goes back to pending with a delayed next run; failed is a terminal failure that is not retried. There is no in_progress, succeeded or dead_letter status — filtering on those returns nothing. (partial is a result status on a completed job, not a job status.)

Retry backoff (default): 30 s doubling per attempt — 30 s, 1 m, 2 m, 4 m, 8 m — with ±20 % jitter, capped at 10 minutes, up to the job's max_attempts (default 3). Certificate renewal is not retried through this queue ladder at all: it is re-driven by the scheduler on its own escalating schedule (15 min → 1 h → 6 h → 12 h → 24 h, 5 attempts). Error classification (network / io_transient / auth / validation / io_permanent) informs whether to retry — auth and validation go straight to failed without burning retries.

Queue lanes​

Distribution jobs (distribution_execute, distribution_rollback) run on a dedicated distWorker. Main workers explicitly filter them out via UseRegisteredTypesAsFilter(). Benefit: a 500-target fan-out doesn't starve ACME / MSCA / notification jobs.

Browse, retry, cancel​

  • Jobs in the top navigation. Filter by status tab; the type filter is the ?type= URL parameter. See Jobs.
  • Click for detail — summary, workflow progress, and the job's logs, which stream incrementally while the job runs.
  • Retry re-enqueues a failed / dead job. Project admin role required (rate-limited to 20 per minute). Retry doesn't change the payload — fix the underlying cause first (expired credential, etc.) then retry.
  • Cancel moves a pending job to cancelled immediately; an active job gets a cancel request and stops at its next checkpoint. Project admin role required.

Idempotency & dead-letter​

Job handlers are idempotent against their natural key — a second issue_certificate for the same cert while one is already in progress is dedup'd at the enqueue site. A second system run of the same distribution that meets the per-certificate/module lock is deferred to the distribution sweep (no failure, no lost version); a manual Execute fails with lock is held and is re-run; fan-out children are guarded by the aggregate state machine.

Jobs that exhaust their retry ladder transition to dead. A terminally failed or dead job emits job.failed (coalesced to one notification per job type per hour; certificate issuance, renewal, revocation, AD CS and distribution jobs report through their own cert.* events instead, and notification.send never emits it). Wire it to your ops channel. On Jobs, switch the status tab to Dead to inspect — fix the underlying cause then click Retry on the row (which re-enqueues the job and resets the attempt counter). The same Retry action is available on failed jobs.

Cluster instances​

A consolidated view (Settings → Cluster) of every live CertAutoPilot process — identity, leader roles, uptime, loaded KEK versions — joined in one table so operators can answer who is doing what without touching mongosh.

Each row is one process, joined from two collections:

  • process_heartbeats — emitted every 30 s by every process regardless of run mode (api, worker, scheduler, all). Records liveness, PID, hostname, binary version, loaded KEK versions, and instance_name.
  • locks — distributed leader locks for the scheduler, domain checker, and discovery checker. Each active lock has a matching heartbeat owner; the UI stamps role badges onto that row.

Identity triple resolved at startup:

  • Name — operator-facing display. Resolution chain: server.instance_name → CERTAUTOPILOT_INSTANCE_NAME → POD_NAME → HOSTNAME → os.Hostname() → literal "unknown".
  • ID — <Name>/<short-uuid>, authoritative, used as the lock owner + heartbeat _id. The UUID suffix prevents collisions if two pods accidentally share a Name, and avoids resurrecting an old document on restart.
  • Mode — api / worker / scheduler / all.

API: GET /api/v1/settings/cluster-instances (admin-only). Returns a list of instances with roles (e.g. scheduler_leader, domain_leader) and orphan_leaders (locks whose owner heartbeat has aged out — useful for spotting stale locks during a primary failover).

Domain tracking​

A cert can be perfectly valid while the underlying domain expires out from under you. Domain tracking watches each registered domain's WHOIS expiry and fires alerts on the configured thresholds. Complementary to cert expiry — different time horizon, different responsible party.

Add a tracker: Domains → Add Domain. Enter the domain name; an initial check runs immediately. Each domain is then re-checked about once a day (sooner, from 6 hours, while lookups are failing), picked up by the domain checker every domain_check_interval_minutes.

Events: domain.expiring_soon once the registration is within 30 days of expiry (at most once a week per domain); domain.expired past the date; plus domain.dangling_dns, domain.dnssec_missing and domain.no_dmarc from the DNS health check. A failed lookup raises no event — thin-WHOIS TLDs (.ai / .gg / .io) return no expiry, which simply shows as unknown.

Lookups use registry WHOIS over TCP/43; RDAP is not used. The outbound network policy allows TCP/43 to WHOIS hosts by default.

Runbook​

Playbooks for the common incidents.

Renewal failing for a specific cert​

  1. Open the cert detail; scroll to Timeline. The cert.renewal_failed event carries error_class + attempt.
  2. Click through to the failed job; inspect logs.
  3. By class:
    • network — ACME directory or DNS provider unreachable. curl from the worker host.
    • auth — DNS credential or ACME account revoked / expired. Re-issue, update credential.
    • validation — DNS TXT record didn't propagate. Check zone TTL, registrar console for stale records.
    • io_permanent — CA rejected the request (rate limit, policy). Read the CA's error verbatim in the log.
  4. Fix the root cause; click Renew on the cert to retry immediately, or wait for the next scheduler sweep.

Dead-letter jobs piling up​

  1. Open Jobs; switch the status tab to Dead.
  2. Look for patterns by type — one type failing en masse = systemic (credential rotation, CA outage, MongoDB index missing).
  3. Resolve the common cause; click Retry on each affected row (no bulk-retry button — fix the cause once, retry the rows individually or via the API).
  4. For stragglers, inspect individually. Sometimes the right answer is Cancel (cert was revoked, no point retrying).

Scheduler won't elect a leader​

  1. Check certautopilot_scheduler_is_leader across pods. Should be exactly one at 1.
  2. All zeros: no instance can write to locks. MongoDB primary probably unreachable.
  3. Two ones: caching glitch — wait one leader_lock_ttl, one should drop.
  4. Logs showing context deadline exceeded on lock renewal indicate slow MongoDB writes.

MongoDB replica-set failover​

  1. Pods see connection errors for a few seconds during primary election.
  2. Driver retries; most in-flight requests recover.
  3. Scheduler loses its lock; standby re-acquires once the new primary accepts writes.
  4. Workers mid-job: current attempt may fail; next retry succeeds.
  5. If primary election takes longer than leader_lock_ttl, sweeps are delayed correspondingly. Nothing to do otherwise.

Suspected KEK compromise​

Treat as urgent. The KEK wraps every secret; if it leaks, every secret is at risk. Full procedure: KEK rotation.

  1. Generate a new KEK version (openssl rand -hex 32).
  2. Add to env / HSM. Restart the fleet so both old and new are loaded.
  3. certautopilot kek verify --target=<new> — confirm all processes loaded.
  4. certautopilot kek rotate --from-version=<current> --to-version=<new>. A background job re-seals every stored secret.
  5. Once kek status shows complete, kek remove --version=<old>. Drop the old key from env / delete the HSM key.
Run these through the wrapper on a standalone install

The bare binary cannot read the database credentials, so every command above fails with an unrelated Unauthorized error. Use cap kek … on a standalone host — the wrapper loads secrets.env and re-execs under sudo — or kubectl exec … certautopilot kek … --config=/config/config.yaml on Kubernetes. For a whole cluster, cap-cluster kek-rotate <new> does steps 2–4 in one command.

  1. Audit-log review: cross-reference admin actions in the compromise window. Consider rotating dependent secrets too (JWT secret, module credentials).

ACME provider outage​

  1. Renewals + issuances for that CA fail with network errors.
  2. CertAutoPilot retries on the configured ladder — transient blips are absorbed.
  3. Extended outage (hours): create an account on a fallback CA (Sectigo / Google / ZeroSSL); reissue urgent certs manually.
  4. Once the CA recovers, retry the dead jobs from the outage window (there is no bulk retry — one row at a time, or via the API); renewals are re-driven by the scheduler on their own.

License expired with no replacement​

  1. At exp the backend enters a 7-day grace window. Cert lifecycle (issue, renew, distribute, revoke) keeps running; the UI shows an expired-banner and the license status endpoint reports in_grace_period: true.
  2. Licensing enforces two counts only: certificates and distribution targets. A Free instance is capped at 1 certificate and 1 target; an Enterprise licence takes its certificate cap from the licence itself (0 = unlimited) and has no target cap. Once a cap is reached, creating new certificates or targets is blocked; existing ones keep renewing and deploying.
  3. After the 7-day grace window nothing else closes. There are no per-feature gates — LDAP, OTP policy and syslog forwarding are available on every tier — and renewals, distribution and revocation continue regardless of licence state.
  4. Obtain a new license; Settings → License.

Fan-out distribution stuck​

  1. Jobs → the Active tab, with ?type=distribution_execute in the URL.
  2. Many stuck: distWorker isn't running. Check worker pod health.
  3. One stuck: inspect its log. Usually a specific target hanging on connect — TCP timeout will eventually resolve it.
  4. Cancel the parent distribution to mark all pending children cancelled. Retry after fixing.

Lost both MongoDB primary AND secret store​

Catastrophic — you've lost the data and the ability to decrypt it.

  1. Restore MongoDB from the latest backup.
  2. Restore the secret store (secrets.env or K8s Secret) from the same time window — KEK and envelopes must match.
  3. Start the service.
  4. Any certs issued between backup time and loss are gone. Re-issue.
  5. Post-mortem: why are backups not synchronised? Fix the procedure.

Observability hooks​

Operations alerting hangs off the metrics / events surface. The high-value alerts:

  • certautopilot_scheduler_is_leader — must be exactly one.
  • certautopilot_scheduler_leader_transitions_total — incrementing frequently → MongoDB write instability or bad config ratio.
  • certautopilot_job_queue_depth{status} — growing pending means workers can't keep up.
  • increase(certautopilot_job_total{status="dead"}[15m]) — there is no dedicated dead-letter metric; alert on the rate at which jobs reach the dead outcome. Any non-zero increase warrants triage.
  • job.failed event — wire to triage channel.
  • domain.expiring_soon / domain.expired — wire to whoever owns registrar renewals.

There is no scheduler.leader_election and no job.dead_letter event; a rule subscribed to either will never fire. For leadership changes, alert on the certautopilot_scheduler_leader_transitions_total metric instead.

Full metric reference: Observability.

Upgrades & backups​

  • Helm: helm upgrade rolling restart. Leader election handles the scheduler transition cleanly.
  • Standalone: upgrade.sh. Service bounce; config preserved.
  • Backups: snapshot MongoDB + the secret store on the same schedule. Restore drills should rehearse the joint restore — out-of-sync KEK + envelopes is unrecoverable.

See also​