Job types

Every job type the workers process: what enqueues it, what it does, which worker lane runs it, and how many attempts it gets before going dead. All jobs live in the MongoDB jobs collection and are visible in the Jobs page.

How to read this page

• Triggered by — user (an API/UI action enqueues it), scheduler (the leader-elected periodic sweep), or system (another job or internal flow enqueues it).
• Lane — main is the general worker; dist is the dedicated distribution worker that handles only distribution_execute and distribution_rollback (the main worker excludes those two types). See queue lanes.
• Attempts — max_attempts set at enqueue time. When a handler returns an error, the job is rescheduled with exponential backoff (30 s × 2ⁿ⁻¹, capped at 10 minutes); once attempts reach max_attempts the job goes dead and is never retried automatically.

ACME

Job type	Triggered by	What it does	Lane	Attempts
issue_certificate	user (issue / approval execute)	Runs the ACME order workflow: DNS-01 challenge, validation, finalize, store cert + key.	main	3
renew_certificate	user (manual renew, bulk renew) · scheduler (renewal sweep)	Re-runs the ACME order for an existing certificate and swaps in the new artifact.	main	3
validate_manual_dns	user (Validate button) · system (after challenge prep)	Checks that manually created _acme-challenge TXT records have propagated, then completes validation.	main	3
cleanup_dns	system (after ACME workflow) · scheduler (cleanup sweep)	Deletes leftover _acme-challenge TXT records.	main	3
revoke_certificate	user	Revokes the certificate at the ACME CA and marks it revoked.	main	3

The scheduler enqueues renewals with the idempotency key renew:<cert_id>:<expires_at_unix>, so a renewal that is already pending or running is never enqueued twice for the same expiry.

Microsoft AD CS (MSCA)

Job type	Triggered by	What it does	Lane	Attempts
msca_issue_certificate	user	Generates a CSR and submits it to the CA via CES (MS-WSTEP).	main	3
msca_renew_certificate	user · scheduler (renewal sweep)	Re-enrolls against the same template; the renewal sweep picks this type automatically for MSCA-issued certificates.	main	3
msca_poll_pending	system (self-rescheduling)	Polls the CA for a request that returned pending (CA-manager approval). Re-enqueues itself with a delay until the CA issues or denies.	main	3

Distribution

Job type	Triggered by	What it does	Lane	Attempts
distribution_execute	user (Execute) · system (after issue/renew) · scheduler (distribution sweep)	Pushes the certificate to the distribution's targets via the module (SSH, Kubernetes, webhook, …), runs post-distribution validation, records per-target results.	dist	1
distribution_rollback	user (Rollback)	Restores the previous certificate material on targets, for modules that support rollback.	dist	3

distribution_execute deliberately gets a single job-level attempt — retries happen inside the distribution flow instead, as per-target retry waves for failed targets whose error class is retryable (up to 3 waves). Large distributions are split into mode=batch child jobs of the same type. See fan-out execution and per-target retry.

KEK rotation

Job type	Triggered by	What it does	Lane	Attempts
kek_rotation_orchestrate	user (certautopilot kek rotate CLI)	Orchestrates a KEK rotation: plans the target collections and enqueues the first collection batch.	main	3
kek_rotation_collection	system (enqueued by the orchestrator, self-chaining)	Re-encrypts one batch of envelopes in one collection from the old KEK version to the new one, then enqueues the next batch.	main	3

See the KEK rotation runbook and the kek rotate CLI reference.

Maintenance

Job type	Triggered by	What it does	Lane	Attempts
certificate_expiration_check	scheduler (hourly-bucketed)	Marks expired certificates, counts expiring-soon certificates (feeds the certautopilot_certificates_expiring_soon gauge), fires expiry notifications.	main	3
domain_expiration_check	scheduler · user (domain add / check now)	WHOIS lookup for tracked domains; updates registration-expiry state.	main	3
revocation_check	scheduler (hourly-bucketed)	Checks OCSP/CRL revocation status of active certificates and flags revoked ones.	main	3

The scheduler enqueues the check jobs with hour-bucket idempotency keys (e.g. exp_check:<hour>), so restarting the scheduler within the same hour does not double-run them.

Notification

Job type	Triggered by	What it does	Lane	Attempts
notification.send	system (event matched a notification rule)	Renders the template and delivers the message via the channel (email, Slack, Teams).	main	3

Discovery

Job type	Triggered by	What it does	Lane	Attempts
discovery_execute	user (Run now) · scheduler (discovery checker)	Scans a discovery source (network/endpoint scan, or CT-log lookup for CT sources) and upserts discovered certificates and findings.	main	1

CT-log sources are executed by the same discovery_execute job type — the handler dispatches to a CT executor internally; there is no separate CT job type.

Partial results

A handler can finish with some work done and some failed. Instead of failing the whole job (which would re-run the successful part), it returns the ErrJobPartial sentinel: the job is marked completed with result_status=partial. Two job types use this today:

• distribution_execute — some targets succeeded, some failed.
• discovery_execute — some endpoints/domains scanned, some errored.

note

partial is a result status on a completed job, not a job status. Filtering the Jobs page by Failed will not show partial jobs — look at the result badge on completed jobs instead.

Job lifecycle recap

pending → active → completed | failed (retry) | dead | cancelled. Failed attempts below max_attempts go back to pending with backoff; a worker that dies mid-job leaves an active job whose lock expires after 5 minutes, at which point any worker reclaims it. Dead jobs can be retried manually from the Jobs page or the API.

See also

• Jobs page (browse, retry, cancel)
• Job queue operations
• Distribution fan-out
• Distribution error codes
• Operations runbook