Skip to main content

Three runtime modes

One binary, three roles. certautopilot serve --mode=<x> picks which role this process plays. all is the single-host default; split them across pods when you want to scale each surface independently.

The modes​

api​

HTTP listener on server.host:server.port. Handles every UI and API request. Stateless — you can run as many instances as load demands. Does not pull jobs, does not run the scheduler.

worker​

Polls the jobs collection with findAndModify to claim jobs. Two worker loops run in-process:

  • Main worker — ACME issuance, MSCA enrollment, renewals, revocation, notifications, discovery, DNS cleanup.
  • distWorker — polls only the distribution lane (distribution_execute, distribution_rollback). Separate lane so large fan-outs don't starve the main queue.

A worker instance runs both loops. To physically isolate distribution workers (dedicated hardware, different node affinity), run additional worker instances with targeted queue filters — see below.

scheduler​

Leader-elected cron. Only one instance is active at a time; the rest heartbeat to the lock collection and take over on leader failure. Emits scheduled jobs on the scheduler.interval cadence:

  • Renewal sweep — find certs inside their renewal window, enqueue renew_certificate.
  • Expiration sweep — emit cert.expiring_soon / cert.expired events.
  • ARI refresh — poll ACME Renewal Info for participating CAs.
  • Domain expiration sweep — WHOIS-based domain renewal checks.
  • Discovery schedules — kick off sources on their configured cadence.
  • Rate-limit counter resets.

all​

Runs api + worker + scheduler in the same process. Default. Fine up to a few thousand certs on a sized VM.

Why split​

  • Independent scaling. API load spikes on business hours. Workers scale with issuance + distribution volume. Schedulers only need HA (2 instances total).
  • Fault isolation. A worker stuck on an HSM-session bug doesn't take down the API path.
  • Resource shapes. API pods benefit from more replicas at lower CPU; workers from fewer replicas at higher CPU + memory.
  • Deployment boundaries. Only the API pod needs to be reachable from browsers — workers and schedulers can run in a more locked-down segment.

Recommended split​

SizeShape
Small (up to ~1k certs)1 × all
Medium (1k – 10k certs)2 × API, 2 × worker, 2 × scheduler
Large (> 10k certs, heavy distribution)3 × API, 3 × worker + 2 × dedicated distribution worker, 2 × scheduler

Two schedulers is the sweet spot: one leader + one standby. More than two buys nothing since only one is active.

Standalone (systemd) with split modes​

Multiple standalone hosts can coexist against the same MongoDB URI. Install each with --mongo=external --mongo-uri=… and edit /etc/systemd/system/certautopilot.service to add --mode=api (or worker / scheduler) to the ExecStart line. Restart the unit. Leader election just works across hosts.

Splitting modes on Kubernetes​

The bundled chart does not split modes

The Helm chart renders a single backend Deployment whose arguments are fixed at serve --mode=all, plus a frontend Deployment. There are no api: / worker: / scheduler: value keys and no extraArgs — setting them is silently ignored by Helm, leaving every pod on --mode=all. Splitting roles today means running your own manifests. See Helm chart.

Distribution lane isolation​

Distribution work is already isolated from the main queue inside every process: a dedicated worker claims only distribution_execute and distribution_rollback, and the main worker excludes them. This is automatic and not operator-selectable — there is no distribution-only deployment mode and no environment variable to request one. Tune throughput with worker.max_concurrency and DistributionSSHMaxConcurrency.

Health probes per mode​

  • api: /healthz (liveness, non-auth) and /readyz (readiness — returns 503 during startup).
  • worker and scheduler: no HTTP listener at all — a process in either mode never starts one, so /healthz and /readyz are unreachable and there is no sibling port to configure. An HTTP readiness probe on such a pod will fail permanently. Use an exec liveness probe, and read readiness from the cluster-instances view (each process heartbeats into MongoDB every 30 seconds).

Graceful shutdown​

  • API pods drain in-flight HTTP requests before exiting (SIGTERM).
  • Workers finish the current job and then exit. In-flight fan-out child jobs are picked up by surviving workers.
  • Scheduler releases its lock explicitly on exit so the standby takes over immediately rather than waiting for TTL.

See also​