Skip to main content

Job types

Every job type the workers process: what enqueues it, what it does, which worker lane runs it, and how many attempts it gets before going dead. All jobs live in the MongoDB jobs collection and are visible in the Jobs page.

How to read this page​

  • Triggered by — user (an API/UI action enqueues it), scheduler (the leader-elected periodic sweep), or system (another job or internal flow enqueues it).
  • Lane — main is the general worker; dist is the dedicated distribution worker that handles only distribution_execute, distribution_rollback and distribution_revalidate (the main worker excludes those types). See queue lanes.
  • Attempts — max_attempts set at enqueue time. When a handler returns an error, the job is rescheduled with exponential backoff (30 s × 2ⁿ⁻¹ with ±20 % random jitter, capped at 10 minutes — the jitter spreads out retries of jobs that failed together, e.g. the batches of one distribution); once attempts reach max_attempts the job goes dead and is never retried automatically.

ACME​

Job typeTriggered byWhat it doesLaneAttempts
issue_certificateuser (issue / approval execute)Starts the ACME order. For automatic DNS-01 (the default) it runs the present phase (create order, publish every TXT) and hands off to await_dns_propagation — releasing the worker instead of blocking on slow DNS. Manual-DNS, HTTP validation (HTTP-01 — no propagation to wait for), sequential/RRset providers (RFC 2136) and the providers that must clean up in the presenting process (Cloudflare, DigitalOcean, OVH, Plesk) run the whole order synchronously here.main3
renew_certificateuser (manual renew, bulk renew) · scheduler (renewal sweep)Same as issue_certificate for an existing certificate (with ARI replaces on renew), then swaps in the new artifact.main3
await_dns_propagationsystem (phased issue/renew hand-off)A self-rescheduling gate: polls DNS until every challenge TXT has propagated (honouring the operator's propagation deadline, holding no worker between checks), then enqueues finalize_order.main3
finalize_ordersystem (after propagation)Accepts the challenges, finalizes the ACME order, downloads + stores the cert/key, then runs cleanup, distribution, and events. Writes the real CERT_ISSUED / CERT_RENEWED audit. Gets 5 attempts rather than 3, so a transient database error after the CA has already issued does not discard the issued certificate.main5
validate_manual_dnsuser (Validate button) · system (after challenge prep)Checks that manually created _acme-challenge TXT records have propagated, then completes validation.main3
cleanup_dnssystem (after ACME workflow) · scheduler (cleanup sweep)Deletes leftover _acme-challenge TXT records.main3
revoke_certificateuserRevokes the certificate at the ACME CA and marks it revoked.main3

Automatic DNS-01 issuance/renewal defaults to this phased flow (present → await → finalize) so a slow-propagating zone never pins a worker; a kill-switch (Settings → General) falls back to the synchronous path. The scheduler enqueues renewals with the idempotency key renew:<cert_id>:<expires_at_unix>, so a renewal that is already pending or running is never enqueued twice for the same expiry.

Microsoft AD CS (MSCA)​

Job typeTriggered byWhat it doesLaneAttempts
msca_issue_certificateuserGenerates a CSR and submits it to the CA — over RPC (MS-ICPR, TCP 445) or CES (MS-WSTEP), per the connection.main3
msca_renew_certificateuser · scheduler (renewal sweep)Re-enrolls against the same template; the renewal sweep picks this type automatically for MSCA-issued certificates.main3
msca_poll_pendingsystem (self-rescheduling)Polls the CA for a request that returned pending (CA-manager approval). Re-enqueues itself with a delay until the CA issues or denies, for up to 7 days from the request. At that deadline it asks the CA once more — an approval granted at the last minute is still stored — and otherwise ends the request as expired. The workflow record is kept a further 7 days after the deadline, then deleted automatically.main3

CA connectors​

Job typeTriggered byWhat it doesLaneAttempts
ca_place_orderuser (issue / reissue / approval execute / manual or bulk renew) · scheduler (renewal sweep, operation: renew) · system (after domain validation)Loads the CA connection and the zone's DNS writer first. For a vendor that validates domains on its account (GlobalSign Atlas) it first asks the vendor which names are validated — free — and when one is not, writes the vendor's validation record, hands the wait to ca_poll_order and orders nothing yet; a new ca_place_order orders once every name is validated. Otherwise it generates or reuses the key, builds the CSR, places the vendor order, records the order id, writes the vendor's DCV record (CNAME or TXT) when the vendor asks for one and arms ca_poll_order. A failure before the order exists is retried; a failure after it is terminal for the job (a retry could place a second paid order). A pending order that lost its poll job is re-armed instead of re-ordered.main3
ca_poll_ordersystem (self-rescheduling)Waits until the validation record is visible on the configured resolvers without calling the vendor, then either asks the vendor to check the domain (account-level validation — when every name is validated it queues the ordering ca_place_order and ends) or polls the vendor with growing intervals until the certificate is issued, the vendor refuses, or the order deadline passes. Holds no worker between ticks; stores the certificate and writes the real CERT_ISSUED / CERT_RENEWED (or _FAILED) audit. A refused, mismatching or expired order fails the poll job (visible on the Jobs page and in the job metrics), after the certificate and the order state were already updated.main3
ca_cleanup_dcvsystem (after the order closes or the domains are validated) · scheduler (stuck-order sweep)Removes the validation record from the zone through the credential it was written with. After the last attempt the order is marked cleanup failed and the certificate page names the record to delete by hand.main5

Distribution​

Job typeTriggered byWhat it doesLaneAttempts
distribution_executeuser (Execute) · system (after issue/renew) · scheduler (distribution sweep)Pushes the certificate to the distribution's targets via the module (SSH, Kubernetes, webhook, …), runs post-distribution validation, records per-target results.dist1
distribution_rollbackuser (Rollback) · system (auto-rollback after a failed/partial run)Re-deploys a previous retained certificate version to the distribution's targets through the module's normal deploy path.dist3
distribution_revalidatesystem (5 minutes after a run that ended unverified)Re-runs only the post-deploy validation of the targets whose check failed — nothing is deployed or restarted — and settles the distribution: all pass ⇒ success, otherwise partial (also when its targets were deleted, or when it could not run on its last attempt). Does nothing when a newer run or certificate version replaced the one it checks. See post-distribution validation.dist3

Its payload is {distribution_id, artifact_id?}. artifact_id pins the exact version to roll back to (the UI version picker); omitted or empty means the newest eligible previous version. Auto-rollback always sets it explicitly. See Rollback.

A distribution with a maintenance window is woken by a delayed distribution_execute armed at the window's start. Exactly one such wake exists per distribution: arming a new one retires any wake left for a different window, and moving the window, switching the distribution off or unlinking it cancels the wake outright — which is why a job can appear as cancelled without anyone having cancelled it.

distribution_execute deliberately gets a single job-level attempt — retries happen inside the distribution flow instead, as per-target retry waves for failed targets whose error class is retryable (up to 3 waves). A system-started run that finds the per-certificate/module distribution lock held by another run completes without deploying (job log: another run of this distribution is in progress; deferred) and re-arms the distribution for the next distribution sweep; a manual Execute in the same situation fails with lock is held and is re-run by the operator, and a rollback retries (see fan-out execution flow). Large distributions are split into mode=batch child jobs of the same type. See fan-out execution and per-target retry.

KEK rotation​

Job typeTriggered byWhat it doesLaneAttempts
kek_rotation_orchestrateuser (certautopilot kek rotate CLI) · system (re-enqueued by each collection chain as it completes)Ticks the rotation. On the first run it counts candidates and enqueues one batch per target collection; on later runs it checks for completion; on the final tick it promotes the new version to active in the keystore, retires the old one, and closes the rotation record. Expect it to appear many times during a single rotation.main3
kek_rotation_collectionsystem (enqueued by the orchestrator, self-chaining)Re-encrypts one batch of envelopes in one collection from the old KEK version to the new one, then enqueues the next batch.main3

See the KEK rotation runbook and the kek rotate CLI reference.

Maintenance​

Job typeTriggered byWhat it doesLaneAttempts
certificate_expiration_checkscheduler (every cycle; hourly-bucketed while queued)Marks expired certificates, counts expiring-soon certificates (feeds the certautopilot_certificates_expiring_soon gauge), raises cert.expired once per expiry and cert.expiring_soon by each certificate's renewal plan (when).main3
domain_expiration_checkscheduler · user (domain add / check now)WHOIS lookup for tracked domains; updates registration-expiry state.main3
revocation_checkscheduler (hourly-bucketed)Checks OCSP/CRL revocation status of active certificates and flags revoked ones. Also sweeps each certificate's retained previous versions so a version revoked out-of-band at the CA drops out of rollback eligibility automatically.main3

The scheduler enqueues the check jobs with hour-bucket idempotency keys (e.g. exp_check:<hour>), so restarting the scheduler within the same hour does not double-run them.

Notification​

Job typeTriggered byWhat it doesLaneAttempts
notification.sendsystem (event matched a notification rule)Renders the template and delivers the message via the channel (email, Slack, Teams, generic webhook).main3

Discovery​

Job typeTriggered byWhat it doesLaneAttempts
discovery_executeuser (Run now) · scheduler (discovery checker)Scans a discovery source (network/endpoint scan, or CT-log lookup for CT sources) and upserts discovered certificates and findings.main1
discovery_findings_refreshscheduler (discovery checker, hourly)Recomputes one project's discovery findings and sends discovery.finding_opened / discovery.cert_expiring / discovery.cert_expired, without a scan. Skipped when a scan refreshed the project in the last 5 minutes.main1
discovery_orphan_cleanupscheduler (discovery checker, hourly); a discovery source delete (6 minutes later)Removes the project's discovered certificates nothing holds any more — no endpoint and no server location — and last written more than 5 minutes ago.main1 (hourly), 3 (after a delete)

CT-log sources are executed by the same discovery_execute job type — the handler dispatches to a CT executor internally; there is no separate CT job type.

Partial results​

A handler can finish with some work done and some failed. Instead of failing the whole job (which would re-run the successful part), it returns the ErrJobPartial sentinel: the job is marked completed with result_status=partial. Two job types use this today:

  • distribution_execute — some targets succeeded, some failed.
  • distribution_revalidate — the validation re-check still failed (the distribution ends partial).
  • discovery_execute — some endpoints/domains scanned, some errored.

A distribution_execute run that ends unverified (every target deployed, only the post-deploy check failed) completes without a partial result and sends no failure notification yet: the verdict — and the Distribution Failed notification when it fails again — comes from its distribution_revalidate job about five minutes later. Automation that polls the execute job should read the distribution's last_status (or wait for the re-check job) rather than treat that job's success as a verified deploy.

note

partial is a result status on a completed job, not a job status. Filtering the Jobs page by Failed will not show partial jobs — look at the result badge on completed jobs instead.

Job lifecycle recap​

pending → active → completed | failed (retry) | dead | cancelled. Failed attempts below max_attempts go back to pending with backoff; a worker that dies mid-job leaves an active job whose lock expires after 5 minutes, at which point any worker reclaims it. A worker that stops cleanly hands its running jobs back to the queue; that is not a cancel — the job runs again. A job an operator cancels while it runs ends cancelled, never retried. Dead jobs can be retried manually from the Jobs page or the API.

See also​