Skip to main content

Jobs

Every asynchronous operation — issuance, renewal, distribution, discovery, notifications — runs as a job in the MongoDB-backed queue. The Jobs page is where you watch them run, read their logs live, retry the failed ones, and cancel the stuck ones.

Open it from the top navigation: Jobs. The page is project-scoped — the project selector determines which jobs you see.

The job list​

The page header shows the Job Queue title with an Auto-refreshing badge (the list refetches every 10 seconds), a Go to Job ID… search box that jumps straight to a job detail page, and a Refresh button for an immediate refetch.

Status tabs​

A tab bar above the table filters by lifecycle state: All · Pending · Active · Completed · Failed · Dead · Cancelled. Switching a tab resets to page 1.

note

There is no type dropdown in the UI — the type filter is driven by the ?type= URL parameter (and the status tab can be preselected with ?status=). Edit the URL by hand to filter, e.g. /jobs?type=distribution_execute.

Columns​

ColumnContent
TypeJob type with underscores rendered as spaces. Fan-out batch children carry a small child tag.
StatusColoured tag per state. A completed job whose result_status is partial shows a partial warning tag instead.
CertificateCertificate name (or ID) the job belongs to, if any.
Triggered ByUser display name, or a system trigger (renewal sweep, fan-out).
AttemptsAttempt counter.
ProgressPer-target progress bar for jobs that report totals (distribution fan-out). - otherwise.
CreatedEnqueue timestamp.
Durationcompleted_at − started_at, in seconds. Empty while running.
ErrorLast error message in red; hover to read the full text, or open the detail. A job that failed an attempt and then succeeded on a retry shows no error here — the failed attempt is kept in the job's Logs.
ActionsRetry (failed/dead jobs) and Cancel (pending/active jobs) icon buttons. Retry is refused for a first-issuance job of a certificate that already holds a certificate (use Re-issue on the certificate), for any issuance or renewal job of a revoked certificate, and when a newer job with the same purpose is already queued or running.

Clicking anywhere on a row opens the job detail. Pagination uses the default page size from Settings → General (UI preferences).

Job lifecycle​

StateMeaning
pendingEnqueued, waiting for a worker to pick it up.
activeA worker holds the lock and is executing it.
completedFinished successfully. May carry result_status: partial (some targets succeeded, some failed — shown as a partial tag). If an earlier attempt of the same job failed, that error is cleared when the job completes, so a green job never displays a stale failure; the failed attempt remains in the job's Logs.
failedFailed terminally — the handler reported an error that retrying cannot fix, so no further attempts run. (A retryable error instead puts the job back to pending with a delayed next run, until max_attempts is exhausted.) Requires a manual Retry.
deadAll attempts exhausted — dead-lettered. Requires a manual Retry.
cancelledCancelled before or during execution — by an operator, or by CertAutoPilot itself when the job's reason disappeared. The product cancels a queued deployment when its maintenance window is moved, when its distribution is switched off or unlinked, and when a newer wake replaces it, so a cancelled job here does not always mean somebody clicked Cancel.

Job detail​

Search a job ID or click a row to open Jobs → detail. The header shows a Back button, the job ID, and a Live tag while the job is pending/active (the page then refetches every 2 seconds).

Summary card​

A bordered descriptions table with: Type, Status (plus partial tag where applicable), Attempts (current / max), Duration, Triggered By, Certificate (links to the certificate detail), Created / Started / Completed timestamps, Parent Job (fan-out children link back to the parent), Idempotency Key, and the full Error text (copyable).

Below common error patterns, the page appends a context hint automatically — e.g. DNS errors suggest checking propagation and credential permissions; SSH errors suggest reachability and firewall rules; WinRM/IIS errors suggest Windows credentials and binding configuration; timeouts suggest network connectivity.

ACME workflow progress​

For issue_certificate, renew_certificate, and validate_manual_dns jobs, a Workflow Progress card renders a vertical step list: Initialized → Awaiting Validation → DNS Presenting → DNS Propagating → Validating → Finalizing → Downloading → Storing → Completed. The current step is highlighted; a failed job marks its step red.

Two extra panels appear when relevant:

  • Manual DNS Challenges — while the job waits at Awaiting Validation, each pending TXT record is listed with its FQDN and a copyable value. See Manual DNS challenge.
  • Presented Records — which TXT records were pushed, via which DNS provider, and when.

MSCA workflow​

MSCA jobs show an MSCA Workflow card: current step, certificate template, CA request ID, poll count, renewal mode, subject summary, terminal reason (issued or an error), and start time.

CA order (CA connectors)​

ca_place_order, ca_poll_order and ca_cleanup_dcv jobs show a CA Order card: vendor, order id, product, operation (issue / renew), the validation record (name, type, value) and where it was written, propagation and cleanup timestamps, poll count with the vendor's last answer code, the order deadline and the terminal reason (issued, failed, expired, superseded, cancelled). See CA connectors.

Target progress​

Distribution jobs that report per-target totals render a Target Progress bar (Targets: current / total) — active while running, red on failure.

Fan-out batch jobs​

Large distributions are split into batch child jobs (see Distribution). The detail page annotates these:

  • A failed batch job whose failed targets are retried automatically shows an info alert: failed targets are retried as a separate retry job — the remaining attempts on this job are not used.
  • Automatic retry jobs are labelled as such, with a Parent Job link back to the originating batch.

Logs viewer​

The Logs card lists every log line with a level tag, the message, and a timestamp. Levels are colour-coded: INFO (blue), WARN (orange, amber row background), ERROR (red, red row background). Lines with extra detail (command output, SOAP faults, stack context) expand into a scrollable monospace block. Bracketed tags and IP:port patterns in messages are rendered bold for scanning.

While the job is active the panel shows an auto-refresh badge and streams new lines incrementally (cursor-based, every few seconds), auto-scrolling to the bottom — scroll up to pause auto-scroll, scroll back down to resume. When the job finishes, a final full refetch catches any missed lines.

Retrying a job​

  1. Find a job in failed or dead state (list row action or the Retry button on the detail page).
  2. Click Retry. The job resets to pending with its error cleared and re-runs on the same job document (the idempotency key is preserved).
  • Requires the Admin project role or higher.
  • Rate-limited to 20 retries per minute.
  • Only failed and dead jobs are retryable — the API rejects anything else.

Cancelling a job​

  1. Find a job in pending or active state.
  2. Click Cancel (red icon in the list, or the button on the detail page).
  • A pending job is cancelled immediately.
  • An active job gets a cancel request; the worker aborts at its next checkpoint — the UI confirms with "Job cancel requested" rather than an instant state change. The job ends cancelled even when the cancel interrupts it mid-step; it is never retried.
  • Requires the Admin project role or higher.
  • ca_place_order jobs cannot be retried from here (422): a retry would place a new paid order at the vendor, and a failed one may already hold an order the vendor never confirmed. Check the vendor panel, then use Reissue on the certificate, whose confirmation says whether it buys a new certificate. A failed ca_poll_order or ca_cleanup_dcv job is safe to retry.
  • ca_cleanup_dcv jobs cannot be cancelled (the API answers 422): they remove a CA-connector vendor's validation record from your zone, and cancelling would leave it behind with nothing flagging it.

Common job types​

TypeWhat it does
issue_certificateFull ACME issuance (order, DNS-01 challenge, finalize, store).
renew_certificateACME renewal of an existing certificate.
revoke_certificateRevokes a certificate at the CA.
validate_manual_dnsValidates operator-created TXT records on the manual DNS path.
await_dns_propagationPhased DNS-01: re-checks, without holding a worker, until the challenge TXT records are visible, then hands over to finalize.
finalize_orderPhased DNS-01: asks the CA to validate, finalizes the order and stores the certificate.
cleanup_dnsRemoves challenge TXT records after validation.
msca_issue_certificateMicrosoft AD CS enrollment (RPC or CES/CEP, per connection).
msca_renew_certificateAD CS renewal.
msca_poll_pendingPolls the Windows CA for a pending (manager-approval) request.
ca_place_orderCA connector: validates the domains with the vendor when it needs that first, then places the vendor order (issue or renew) and writes the vendor's validation record into the zone when one is asked for.
ca_poll_orderCA connector: waits for the validation record to propagate, then asks the vendor to check the domain or polls it until issued, refused or past the order deadline — without holding a worker between checks.
ca_cleanup_dcvCA connector: removes the validation record once it is no longer needed.
certificate_expiration_checkPeriodic sweep that flags expiring certificates.
revocation_checkPeriodic check of certificates' revocation status at the CA.
domain_expiration_checkWHOIS + DNS health check for a tracked domain.
distribution_executeDelivers a certificate to targets (also the fan-out batch/retry child type).
distribution_rollbackRolls a distribution back by re-deploying a previous retained certificate version.
notification.sendDelivers one notification to one channel.
discovery_executeRuns a discovery scan against a source.
discovery_orphan_cleanupRemoves a project's discovered certificates nothing holds any more (hourly, and a few minutes after a discovery source is deleted).
discovery_findings_refreshRecomputes one project's discovery findings and sends its discovery expiry alerts, without a scan (hourly).

Troubleshooting with job logs​

  1. Open the failing job and read the Error row first — the inline hint often names the failing subsystem.
  2. Scan the logs for the last ERROR line; expand its detail block for raw output (DNS responses, SSH/WinRM output, SOAP faults).
  3. For ACME jobs, check which workflow step went red — a failure at DNS Propagating is a DNS problem, at Finalizing a CA problem.
  4. For distribution fan-out, follow the Parent Job / retry-job links to find the job that holds the per-target results.
  5. Fix the cause, then Retry (failed/dead) — or re-trigger from the certificate or distribution page.

See also​