Skip to main content

Three runtime modes

One binary, three roles. certautopilot serve --mode=<x> picks which role this process plays. all is the single-host default; split them across pods when you want to scale each surface independently.

The modes​

api​

HTTP listener on server.host:server.port. Handles every UI and API request. Stateless — you can run as many instances as load demands. Does not pull jobs, does not run the scheduler.

worker​

Polls the jobs collection with findAndModify to claim jobs. Two worker loops run in-process:

  • Main worker — ACME issuance, MSCA enrollment, renewals, revocation, notifications, discovery, DNS cleanup.
  • distWorker — polls only the distribution lane (distribution_execute, distribution_rollback). Separate lane so large fan-outs don't starve the main queue.

A worker instance runs both loops. To physically isolate distribution workers (dedicated hardware, different node affinity), run additional worker instances with targeted queue filters — see below.

scheduler​

Leader-elected cron. Only one instance is active at a time; the rest heartbeat to the lock collection and take over on leader failure. Emits scheduled jobs on the scheduler.interval cadence:

  • Renewal sweep — find certs inside their renewal window, enqueue renew_certificate.
  • Expiration sweep — emit cert.expiring_soon / cert.expired events.
  • ARI refresh — poll ACME Renewal Info for participating CAs.
  • Domain expiration sweep — WHOIS-based domain renewal checks.
  • Discovery schedules — kick off sources on their configured cadence.
  • Rate-limit counter resets.

all​

Runs api + worker + scheduler in the same process. Default. Fine up to a few thousand certs on a sized VM.

Why split​

  • Independent scaling. API load spikes on business hours. Workers scale with issuance + distribution volume. Schedulers only need HA (2 instances total).
  • Fault isolation. A worker stuck on an HSM-session bug doesn't take down the API path.
  • Resource shapes. API pods benefit from more replicas at lower CPU; workers from fewer replicas at higher CPU + memory.
  • Deployment boundaries. Only the API pod needs to be reachable from browsers — workers and schedulers can run in a more locked-down segment.

Recommended split​

SizeShape
Small (up to ~1k certs)1 × all
Medium (1k – 10k certs)2 × API, 2 × worker, 2 × scheduler
Large (> 10k certs, heavy distribution)3 × API, 3 × worker + 2 × dedicated distribution worker, 2 × scheduler

Two schedulers is the sweet spot: one leader + one standby. More than two buys nothing since only one is active.

Standalone (systemd) with split modes​

Multiple standalone hosts can coexist against the same MongoDB URI. Install each with --mongo=external --mongo-uri=… and edit /etc/systemd/system/certautopilot.service to add --mode=api (or worker / scheduler) to the ExecStart line. Restart the unit. Leader election just works across hosts.

Helm with split modes​

# values.yaml excerpt
api:
  replicaCount: 2
  extraArgs: ["--mode=api"]
worker:
  replicaCount: 2
  extraArgs: ["--mode=worker"]
scheduler:
  replicaCount: 2
  extraArgs: ["--mode=scheduler"]

The chart renders three Deployments. Each one shares the same config + secrets + MongoDB. See Helm chart.

Dedicated distribution workers​

When a single distribution fans out to hundreds of targets, you want distribution work isolated from the main queue. Run a separate worker deployment with an env var signalling distribution-only mode:

CERTAUTOPILOT_WORKER_DISTRIBUTION_ONLY=true

These pods only claim jobs from the distribution lane. Pair with a larger SSH concurrency cap (DistributionSSHMaxConcurrency) if your targets can absorb it.

Health probes per mode​

  • api: /healthz (liveness, non-auth) and /readyz (readiness — returns 503 during startup).
  • worker + scheduler: expose the same endpoints on a sibling port (configurable). K8s can gate pod readiness on them.

Graceful shutdown​

  • API pods drain in-flight HTTP requests before exiting (SIGTERM).
  • Workers finish the current job and then exit. In-flight fan-out child jobs are picked up by surviving workers.
  • Scheduler releases its lock explicitly on exit so the standby takes over immediately rather than waiting for TTL.

See also​