Jobs
Every asynchronous operation — issuance, renewal, distribution, discovery, notifications — runs as a job in the MongoDB-backed queue. The Jobs page is where you watch them run, read their logs live, retry the failed ones, and cancel the stuck ones.
Open it from the top navigation: Jobs. The page is project-scoped — the project selector determines which jobs you see.
The job list
The page header shows the Job Queue title with an Auto-refreshing badge (the list refetches every 10 seconds), a Go to Job ID… search box that jumps straight to a job detail page, and a Refresh button for an immediate refetch.
Status tabs
A tab bar above the table filters by lifecycle state: All · Pending · Active · Completed · Failed · Dead · Cancelled. Switching a tab resets to page 1.
There is no type dropdown in the UI — the type filter is driven by the ?type= URL parameter (and the status tab can be preselected with ?status=). Edit the URL by hand to filter, e.g. /jobs?type=distribution_execute.
Columns
| Column | Content |
|---|---|
| Type | Job type with underscores rendered as spaces. Fan-out batch children carry a small child tag. |
| Status | Coloured tag per state. A completed job whose result_status is partial shows a partial warning tag instead. |
| Certificate | Certificate name (or ID) the job belongs to, if any. |
| Triggered By | User display name, or a system trigger (renewal sweep, fan-out). |
| Attempts | Attempt counter. |
| Progress | Per-target progress bar for jobs that report totals (distribution fan-out). - otherwise. |
| Created | Enqueue timestamp. |
| Duration | completed_at − started_at, in seconds. Empty while running. |
| Error | Last error message in red; hover to read the full text, or open the detail. A job that failed an attempt and then succeeded on a retry shows no error here — the failed attempt is kept in the job's Logs. |
| Actions | Retry (failed/dead jobs) and Cancel (pending/active jobs) icon buttons. Retry is refused for a first-issuance job of a certificate that already holds a certificate (use Re-issue on the certificate), for any issuance or renewal job of a revoked certificate, and when a newer job with the same purpose is already queued or running. |
Clicking anywhere on a row opens the job detail. Pagination uses the default page size from Settings → General (UI preferences).
Job lifecycle
| State | Meaning |
|---|---|
pending | Enqueued, waiting for a worker to pick it up. |
active | A worker holds the lock and is executing it. |
completed | Finished successfully. May carry result_status: partial (some targets succeeded, some failed — shown as a partial tag). If an earlier attempt of the same job failed, that error is cleared when the job completes, so a green job never displays a stale failure; the failed attempt remains in the job's Logs. |
failed | Failed terminally — the handler reported an error that retrying cannot fix, so no further attempts run. (A retryable error instead puts the job back to pending with a delayed next run, until max_attempts is exhausted.) Requires a manual Retry. |
dead | All attempts exhausted — dead-lettered. Requires a manual Retry. |
cancelled | Cancelled before or during execution — by an operator, or by CertAutoPilot itself when the job's reason disappeared. The product cancels a queued deployment when its maintenance window is moved, when its distribution is switched off or unlinked, and when a newer wake replaces it, so a cancelled job here does not always mean somebody clicked Cancel. |
Job detail
Search a job ID or click a row to open Jobs → detail. The header shows a Back button, the job ID, and a Live tag while the job is pending/active (the page then refetches every 2 seconds).
Summary card
A bordered descriptions table with: Type, Status (plus partial tag where applicable), Attempts (current / max), Duration, Triggered By, Certificate (links to the certificate detail), Created / Started / Completed timestamps, Parent Job (fan-out children link back to the parent), Idempotency Key, and the full Error text (copyable).
Below common error patterns, the page appends a context hint automatically — e.g. DNS errors suggest checking propagation and credential permissions; SSH errors suggest reachability and firewall rules; WinRM/IIS errors suggest Windows credentials and binding configuration; timeouts suggest network connectivity.
ACME workflow progress
For issue_certificate, renew_certificate, and validate_manual_dns jobs, a Workflow Progress card renders a vertical step list: Initialized → Awaiting Validation → DNS Presenting → DNS Propagating → Validating → Finalizing → Downloading → Storing → Completed. The current step is highlighted; a failed job marks its step red.
Two extra panels appear when relevant:
- Manual DNS Challenges — while the job waits at Awaiting Validation, each pending TXT record is listed with its FQDN and a copyable value. See Manual DNS challenge.
- Presented Records — which TXT records were pushed, via which DNS provider, and when.
MSCA workflow
MSCA jobs show an MSCA Workflow card: current step, certificate template, CA request ID, poll count, renewal mode, subject summary, terminal reason (issued or an error), and start time.
CA order (CA connectors)
ca_place_order, ca_poll_order and ca_cleanup_dcv jobs show a CA Order card: vendor, order id, product, operation (issue / renew), the validation record (name, type, value) and where it was written, propagation and cleanup timestamps, poll count with the vendor's last answer code, the order deadline and the terminal reason (issued, failed, expired, superseded, cancelled). See CA connectors.
Target progress
Distribution jobs that report per-target totals render a Target Progress bar (Targets: current / total) — active while running, red on failure.
Fan-out batch jobs
Large distributions are split into batch child jobs (see Distribution). The detail page annotates these:
- A failed batch job whose failed targets are retried automatically shows an info alert: failed targets are retried as a separate retry job — the remaining attempts on this job are not used.
- Automatic retry jobs are labelled as such, with a Parent Job link back to the originating batch.
Logs viewer
The Logs card lists every log line with a level tag, the message, and a timestamp. Levels are colour-coded: INFO (blue), WARN (orange, amber row background), ERROR (red, red row background). Lines with extra detail (command output, SOAP faults, stack context) expand into a scrollable monospace block. Bracketed tags and IP:port patterns in messages are rendered bold for scanning.
While the job is active the panel shows an auto-refresh badge and streams new lines incrementally (cursor-based, every few seconds), auto-scrolling to the bottom — scroll up to pause auto-scroll, scroll back down to resume. When the job finishes, a final full refetch catches any missed lines.
Retrying a job
- Find a job in failed or dead state (list row action or the Retry button on the detail page).
- Click Retry. The job resets to
pendingwith its error cleared and re-runs on the same job document (the idempotency key is preserved).
- Requires the Admin project role or higher.
- Rate-limited to 20 retries per minute.
- Only
failedanddeadjobs are retryable — the API rejects anything else.
Cancelling a job
- Find a job in pending or active state.
- Click Cancel (red icon in the list, or the button on the detail page).
- A pending job is cancelled immediately.
- An active job gets a cancel request; the worker aborts at its next checkpoint — the UI confirms with "Job cancel requested" rather than an instant state change. The job ends cancelled even when the cancel interrupts it mid-step; it is never retried.
- Requires the Admin project role or higher.
ca_place_orderjobs cannot be retried from here (422): a retry would place a new paid order at the vendor, and a failed one may already hold an order the vendor never confirmed. Check the vendor panel, then use Reissue on the certificate, whose confirmation says whether it buys a new certificate. A failedca_poll_orderorca_cleanup_dcvjob is safe to retry.ca_cleanup_dcvjobs cannot be cancelled (the API answers 422): they remove a CA-connector vendor's validation record from your zone, and cancelling would leave it behind with nothing flagging it.
Common job types
| Type | What it does |
|---|---|
issue_certificate | Full ACME issuance (order, DNS-01 challenge, finalize, store). |
renew_certificate | ACME renewal of an existing certificate. |
revoke_certificate | Revokes a certificate at the CA. |
validate_manual_dns | Validates operator-created TXT records on the manual DNS path. |
await_dns_propagation | Phased DNS-01: re-checks, without holding a worker, until the challenge TXT records are visible, then hands over to finalize. |
finalize_order | Phased DNS-01: asks the CA to validate, finalizes the order and stores the certificate. |
cleanup_dns | Removes challenge TXT records after validation. |
msca_issue_certificate | Microsoft AD CS enrollment (RPC or CES/CEP, per connection). |
msca_renew_certificate | AD CS renewal. |
msca_poll_pending | Polls the Windows CA for a pending (manager-approval) request. |
ca_place_order | CA connector: validates the domains with the vendor when it needs that first, then places the vendor order (issue or renew) and writes the vendor's validation record into the zone when one is asked for. |
ca_poll_order | CA connector: waits for the validation record to propagate, then asks the vendor to check the domain or polls it until issued, refused or past the order deadline — without holding a worker between checks. |
ca_cleanup_dcv | CA connector: removes the validation record once it is no longer needed. |
certificate_expiration_check | Periodic sweep that flags expiring certificates. |
revocation_check | Periodic check of certificates' revocation status at the CA. |
domain_expiration_check | WHOIS + DNS health check for a tracked domain. |
distribution_execute | Delivers a certificate to targets (also the fan-out batch/retry child type). |
distribution_rollback | Rolls a distribution back by re-deploying a previous retained certificate version. |
notification.send | Delivers one notification to one channel. |
discovery_execute | Runs a discovery scan against a source. |
discovery_orphan_cleanup | Removes a project's discovered certificates nothing holds any more (hourly, and a few minutes after a discovery source is deleted). |
discovery_findings_refresh | Recomputes one project's discovery findings and sends its discovery expiry alerts, without a scan (hourly). |
Troubleshooting with job logs
- Open the failing job and read the Error row first — the inline hint often names the failing subsystem.
- Scan the logs for the last ERROR line; expand its detail block for raw output (DNS responses, SSH/WinRM output, SOAP faults).
- For ACME jobs, check which workflow step went red — a failure at DNS Propagating is a DNS problem, at Finalizing a CA problem.
- For distribution fan-out, follow the Parent Job / retry-job links to find the job that holds the per-target results.
- Fix the cause, then Retry (failed/dead) — or re-trigger from the certificate or distribution page.