Skip to main content

Three runtime modes

One binary, three roles. certautopilot serve --mode=<x> picks which role this process plays. all is the single-host default; split them across pods when you want to scale each surface independently.

The modes​

api​

HTTP listener on server.host:server.port. Handles every UI and API request. Stateless — you can run as many instances as load demands. Does not pull jobs, does not run the scheduler.

worker​

Polls the jobs collection with findAndModify to claim jobs. Two worker loops run in-process:

  • Main worker — ACME issuance, MSCA enrollment, CA-connector vendor orders, renewals, revocation, notifications, discovery, DNS cleanup.
  • distWorker — polls only the distribution lane (distribution_execute, distribution_rollback). Separate lane so large fan-outs don't starve the main queue.

A worker instance runs both loops. The lane split is automatic and cannot be narrowed per instance — there is no distribution-only worker mode (see below).

scheduler​

Leader-elected cron. Only one instance is active at a time; the rest heartbeat to the lock collection and take over on leader failure. Each cycle runs on the Renewal check interval from Settings → General (falling back to scheduler.interval, default 1 h):

  • Renewal sweep — find certs inside their renewal window, enqueue renew_certificate (or msca_renew_certificate / ca_place_order, by issuer).
  • Expiration and revocation checks — enqueue certificate_expiration_check (which emits cert.expiring_soon / cert.expired notifications) and revocation_check.
  • ARI refresh — poll ACME Renewal Info for participating CAs.
  • Distribution sweep — re-arm pending distributions, reconcile stuck fan-outs and stalled rollbacks.
  • Recovery sweeps — zombie certificates, orphaned phased DNS-01, stuck MSCA workflows and stuck CA-connector orders, DNS cleanup retries, stalled bulk actions, expired approval requests, stale notification reservations, expired HTTP-01 tokens.

Two further leader-only loops run on their own cadence: the domain expiration checker (WHOIS-based domain renewal checks) and the discovery checker (kicks off sources on their configured schedule).

all​

Runs api + worker + scheduler in the same process. Default. Fine up to a few thousand certs on a sized VM.

Why split​

  • Independent scaling. API load spikes on business hours. Workers scale with issuance + distribution volume. Schedulers only need HA (2 instances total).
  • Fault isolation. A worker stuck on an HSM-session bug doesn't take down the API path.
  • Resource shapes. API pods benefit from more replicas at lower CPU; workers from fewer replicas at higher CPU + memory.
  • Deployment boundaries. Only the API pod needs to be reachable from browsers — workers and schedulers can run in a more locked-down segment.

Recommended split​

SizeShape
Small (up to ~1k certs)1 × all
Medium (1k – 10k certs)2 × API, 2 × worker, 2 × scheduler
Large (> 10k certs, heavy distribution)3 × API, 3 × worker + 2 × dedicated distribution worker, 2 × scheduler

Two schedulers is the sweet spot: one leader + one standby. More than two buys nothing since only one is active.

Standalone (systemd) with split modes​

Multiple standalone hosts can coexist against the same MongoDB URI. Install each with --mongo=external --mongo-uri=… and edit /etc/systemd/system/certautopilot.service to add --mode=api (or worker / scheduler) to the ExecStart line. Restart the unit. Leader election just works across hosts.

Splitting modes on Kubernetes​

The bundled chart does not split modes

The Helm chart renders a single backend Deployment whose arguments are fixed at serve --mode=all, plus a frontend Deployment. There are no api: / worker: / scheduler: value keys and no extraArgs — setting them is silently ignored by Helm, leaving every pod on --mode=all. Splitting roles today means running your own manifests. See Helm chart.

Distribution lane isolation​

Distribution work is already isolated from the main queue inside every process: a dedicated worker claims only distribution_execute and distribution_rollback, and the main worker excludes them. This is automatic and not operator-selectable — there is no distribution-only deployment mode and no environment variable to request one. Tune throughput with worker.max_concurrency and DistributionSSHMaxConcurrency.

Health probes per mode​

  • api: /healthz (liveness, non-auth) and /readyz (readiness — returns 503 when the MongoDB ping fails).
  • worker and scheduler: no HTTP listener at all — a process in either mode never starts one, so /healthz and /readyz are unreachable and there is no sibling port to configure. An HTTP readiness probe on such a pod will fail permanently. Use an exec liveness probe, and read readiness from the cluster-instances view (each process heartbeats into MongoDB every 30 seconds).

Graceful shutdown​

  • API pods drain in-flight HTTP requests before exiting (SIGTERM).
  • Workers finish the current job and then exit. In-flight fan-out child jobs are picked up by surviving workers.
  • Scheduler releases its lock explicitly on exit so the standby takes over immediately rather than waiting for TTL.

See also​