Job types
Every job type the workers process: what enqueues it, what it does, which worker lane runs it, and how many attempts it gets before going dead. All jobs live in the MongoDB jobs collection and are visible in the Jobs page.
How to read this page
- Triggered by —
user(an API/UI action enqueues it),scheduler(the leader-elected periodic sweep), orsystem(another job or internal flow enqueues it). - Lane —
mainis the general worker;distis the dedicated distribution worker that handles onlydistribution_executeanddistribution_rollback(the main worker excludes those two types). See queue lanes. - Attempts —
max_attemptsset at enqueue time. When a handler returns an error, the job is rescheduled with exponential backoff (30 s × 2ⁿ⁻¹, capped at 10 minutes); once attempts reachmax_attemptsthe job goesdeadand is never retried automatically.
ACME
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
issue_certificate | user (issue / approval execute) | Starts the ACME order. For automatic DNS-01 (the default) it runs the present phase (create order, publish every TXT) and hands off to await_dns_propagation — releasing the worker instead of blocking on slow DNS. Manual-DNS, HTTP validation (HTTP-01 — no propagation to wait for), and sequential/RRset providers run the whole order synchronously here. | main | 3 |
renew_certificate | user (manual renew, bulk renew) · scheduler (renewal sweep) | Same as issue_certificate for an existing certificate (with ARI replaces on renew), then swaps in the new artifact. | main | 3 |
await_dns_propagation | system (phased issue/renew hand-off) | A self-rescheduling gate: polls DNS until every challenge TXT has propagated (honouring the operator's propagation deadline, holding no worker between checks), then enqueues finalize_order. | main | 3 |
finalize_order | system (after propagation) | Accepts the challenges, finalizes the ACME order, downloads + stores the cert/key, then runs cleanup, distribution, and events. Writes the real CERT_ISSUED / CERT_RENEWED audit. | main | 3 |
validate_manual_dns | user (Validate button) · system (after challenge prep) | Checks that manually created _acme-challenge TXT records have propagated, then completes validation. | main | 3 |
cleanup_dns | system (after ACME workflow) · scheduler (cleanup sweep) | Deletes leftover _acme-challenge TXT records. | main | 3 |
revoke_certificate | user | Revokes the certificate at the ACME CA and marks it revoked. | main | 3 |
Automatic DNS-01 issuance/renewal defaults to this phased flow (present → await → finalize) so a slow-propagating zone never pins a worker; a kill-switch (Settings → General) falls back to the synchronous path. The scheduler enqueues renewals with the idempotency key renew:<cert_id>:<expires_at_unix>, so a renewal that is already pending or running is never enqueued twice for the same expiry.
Microsoft AD CS (MSCA)
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
msca_issue_certificate | user | Generates a CSR and submits it to the CA via CES (MS-WSTEP). | main | 3 |
msca_renew_certificate | user · scheduler (renewal sweep) | Re-enrolls against the same template; the renewal sweep picks this type automatically for MSCA-issued certificates. | main | 3 |
msca_poll_pending | system (self-rescheduling) | Polls the CA for a request that returned pending (CA-manager approval). Re-enqueues itself with a delay until the CA issues or denies. | main | 3 |
Distribution
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
distribution_execute | user (Execute) · system (after issue/renew) · scheduler (distribution sweep) | Pushes the certificate to the distribution's targets via the module (SSH, Kubernetes, webhook, …), runs post-distribution validation, records per-target results. | dist | 1 |
distribution_rollback | user (Rollback) · system (auto-rollback after a failed/partial run) | Re-deploys a previous retained certificate version to the distribution's targets through the module's normal deploy path. | dist | 3 |
Its payload is {distribution_id, artifact_id?}. artifact_id pins the exact version to roll back to (the UI version picker); omitted or empty means the newest eligible previous version. Auto-rollback always sets it explicitly. See Rollback.
A distribution with a maintenance window is woken by a delayed distribution_execute armed at the window's start. Exactly one such wake exists per distribution: arming a new one retires any wake left for a different window, and moving the window, switching the distribution off or unlinking it cancels the wake outright — which is why a job can appear as cancelled without anyone having cancelled it.
distribution_execute deliberately gets a single job-level attempt — retries happen inside the distribution flow instead, as per-target retry waves for failed targets whose error class is retryable (up to 3 waves). Large distributions are split into mode=batch child jobs of the same type. See fan-out execution and per-target retry.
KEK rotation
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
kek_rotation_orchestrate | user (certautopilot kek rotate CLI) · system (re-enqueued by each collection chain as it completes) | Ticks the rotation. On the first run it counts candidates and enqueues one batch per target collection; on later runs it checks for completion; on the final tick it promotes the new version to active in the keystore, retires the old one, and closes the rotation record. Expect it to appear many times during a single rotation. | main | 3 |
kek_rotation_collection | system (enqueued by the orchestrator, self-chaining) | Re-encrypts one batch of envelopes in one collection from the old KEK version to the new one, then enqueues the next batch. | main | 3 |
See the KEK rotation runbook and the kek rotate CLI reference.
Maintenance
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
certificate_expiration_check | scheduler (hourly-bucketed) | Marks expired certificates, counts expiring-soon certificates (feeds the certautopilot_certificates_expiring_soon gauge), fires expiry notifications. | main | 3 |
domain_expiration_check | scheduler · user (domain add / check now) | WHOIS lookup for tracked domains; updates registration-expiry state. | main | 3 |
revocation_check | scheduler (hourly-bucketed) | Checks OCSP/CRL revocation status of active certificates and flags revoked ones. Also sweeps each certificate's retained previous versions so a version revoked out-of-band at the CA drops out of rollback eligibility automatically. | main | 3 |
The scheduler enqueues the check jobs with hour-bucket idempotency keys (e.g. exp_check:<hour>), so restarting the scheduler within the same hour does not double-run them.
Notification
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
notification.send | system (event matched a notification rule) | Renders the template and delivers the message via the channel (email, Slack, Teams). | main | 3 |
Discovery
| Job type | Triggered by | What it does | Lane | Attempts |
|---|---|---|---|---|
discovery_execute | user (Run now) · scheduler (discovery checker) | Scans a discovery source (network/endpoint scan, or CT-log lookup for CT sources) and upserts discovered certificates and findings. | main | 1 |
CT-log sources are executed by the same discovery_execute job type — the handler dispatches to a CT executor internally; there is no separate CT job type.
Partial results
A handler can finish with some work done and some failed. Instead of failing the whole job (which would re-run the successful part), it returns the ErrJobPartial sentinel: the job is marked completed with result_status=partial. Two job types use this today:
distribution_execute— some targets succeeded, some failed.discovery_execute— some endpoints/domains scanned, some errored.
partial is a result status on a completed job, not a job status. Filtering the Jobs page by Failed will not show partial jobs — look at the result badge on completed jobs instead.
Job lifecycle recap
pending → active → completed | failed (retry) | dead | cancelled. Failed attempts below max_attempts go back to pending with backoff; a worker that dies mid-job leaves an active job whose lock expires after 5 minutes, at which point any worker reclaims it. Dead jobs can be retried manually from the Jobs page or the API.