Three runtime modes
One binary, three roles. certautopilot serve --mode=<x> picks which role this process plays. all is the single-host default; split them across pods when you want to scale each surface independently.
The modes
api
HTTP listener on server.host:server.port. Handles every UI and API request. Stateless — you can run as many instances as load demands. Does not pull jobs, does not run the scheduler.
worker
Polls the jobs collection with findAndModify to claim jobs. Two worker loops run in-process:
- Main worker — ACME issuance, MSCA enrollment, CA-connector vendor orders, renewals, revocation, notifications, discovery, DNS cleanup.
- distWorker — polls only the distribution lane (
distribution_execute,distribution_rollback). Separate lane so large fan-outs don't starve the main queue.
A worker instance runs both loops. The lane split is automatic and cannot be narrowed per instance — there is no distribution-only worker mode (see below).
scheduler
Leader-elected cron. Only one instance is active at a time; the rest heartbeat to the lock collection and take over on leader failure. Each cycle runs on the Renewal check interval from Settings → General (falling back to scheduler.interval, default 1 h):
- Renewal sweep — find certs inside their renewal window, enqueue
renew_certificate(ormsca_renew_certificate/ca_place_order, by issuer). - Expiration and revocation checks — enqueue
certificate_expiration_check(which emitscert.expiring_soon/cert.expirednotifications) andrevocation_check. - ARI refresh — poll ACME Renewal Info for participating CAs.
- Distribution sweep — re-arm pending distributions, reconcile stuck fan-outs and stalled rollbacks.
- Recovery sweeps — zombie certificates, orphaned phased DNS-01, stuck MSCA workflows and stuck CA-connector orders, DNS cleanup retries, stalled bulk actions, expired approval requests, stale notification reservations, expired HTTP-01 tokens.
Two further leader-only loops run on their own cadence: the domain expiration checker (WHOIS-based domain renewal checks) and the discovery checker (kicks off sources on their configured schedule).
all
Runs api + worker + scheduler in the same process. Default. Fine up to a few thousand certs on a sized VM.
Why split
- Independent scaling. API load spikes on business hours. Workers scale with issuance + distribution volume. Schedulers only need HA (2 instances total).
- Fault isolation. A worker stuck on an HSM-session bug doesn't take down the API path.
- Resource shapes. API pods benefit from more replicas at lower CPU; workers from fewer replicas at higher CPU + memory.
- Deployment boundaries. Only the API pod needs to be reachable from browsers — workers and schedulers can run in a more locked-down segment.
Recommended split
| Size | Shape |
|---|---|
| Small (up to ~1k certs) | 1 × all |
| Medium (1k – 10k certs) | 2 × API, 2 × worker, 2 × scheduler |
| Large (> 10k certs, heavy distribution) | 3 × API, 3 × worker + 2 × dedicated distribution worker, 2 × scheduler |
Two schedulers is the sweet spot: one leader + one standby. More than two buys nothing since only one is active.
Standalone (systemd) with split modes
Multiple standalone hosts can coexist against the same MongoDB URI. Install each with --mongo=external --mongo-uri=… and edit /etc/systemd/system/certautopilot.service to add --mode=api (or worker / scheduler) to the ExecStart line. Restart the unit. Leader election just works across hosts.
Splitting modes on Kubernetes
The Helm chart renders a single backend Deployment whose arguments are fixed at
serve --mode=all, plus a frontend Deployment. There are no api: / worker: / scheduler:
value keys and no extraArgs — setting them is silently ignored by Helm, leaving every pod on
--mode=all. Splitting roles today means running your own manifests. See Helm chart.
Distribution lane isolation
Distribution work is already isolated from the main queue inside every process: a dedicated
worker claims only distribution_execute and distribution_rollback, and the main worker excludes
them. This is automatic and not operator-selectable — there is no distribution-only deployment mode
and no environment variable to request one. Tune throughput with worker.max_concurrency and
DistributionSSHMaxConcurrency.
Health probes per mode
api:/healthz(liveness, non-auth) and/readyz(readiness — returns 503 when the MongoDB ping fails).workerandscheduler: no HTTP listener at all — a process in either mode never starts one, so/healthzand/readyzare unreachable and there is no sibling port to configure. An HTTP readiness probe on such a pod will fail permanently. Use anexecliveness probe, and read readiness from the cluster-instances view (each process heartbeats into MongoDB every 30 seconds).
Graceful shutdown
- API pods drain in-flight HTTP requests before exiting (SIGTERM).
- Workers finish the current job and then exit. In-flight fan-out child jobs are picked up by surviving workers.
- Scheduler releases its lock explicitly on exit so the standby takes over immediately rather than waiting for TTL.