Skip to main content

Scheduler status & sweeps

The scheduler is the periodic half of CertAutoPilot: it finds work and enqueues jobs; workers execute them. It runs in processes started with --mode=scheduler or --mode=all, and only one instance per component acts at a time — leadership is held via a MongoDB distributed lock (see Cluster instances for TTLs and failover timing). Sweep results are visible on Settings → System Status (how to read that page).

Sweep catalog​

Three leader-elected components and the sweeps below. License re-validation is not one of them: an online license is checked against the license server when a server process starts, at most once per 24 hours for the whole installation (the first process to claim the check does it), and whenever an admin presses Validate on the License page. Each instance's 60-second loop only re-reads the stored result from the database; it makes no license-server call.

Main Scheduler​

All main-scheduler sweeps run once per cycle; the cycle interval is Settings → General → Renewal Check Interval (renewal_check_interval_minutes, default 60, range 5–1440).

SweepWhat it does
Renewal Failure Safety NetPuts auto-renew certificates stuck in renewal_failed (and not yet expired) back on the renewal retry ladder after a 24-hour cool-down, so a multi-day CA or DNS outage cannot permanently stop auto-renewal. Runs just before the renewal check.
Renewal CheckFinds certificates due for renewal (see below) and enqueues renew_certificate / msca_renew_certificate / ca_place_order jobs (by issuer type). Not affected by the certificate limit — existing certificates are always renewed.
ARI Window RefreshRefreshes ACME Renewal Information windows for eligible certificates (up to 100 per cycle) and may move next_renewal_at earlier when the CA advises it.
Expiration CheckEnqueues a certificate_expiration_check job every cycle (the hourly key only keeps a second one from queueing while one is pending) — drives the expiry notifications, each raised once (when).
Revocation CheckEnqueues one hourly revocation status check job.
Distribution SweepEnqueues distribution jobs for certificates whose linked distributions are pending re-distribution, reconciles fan-outs stuck mid-aggregation, and closes out rollbacks whose job died without recording a result.
Stalled Bulk Action RecoveryCloses out bulk actions that have shown in progress for more than 30 minutes but can no longer complete on their own.
Zombie Certificate RecoveryResets certificates stuck in renewal_in_progress (> 45 min) or issuing (> 2 h) and re-enqueues them if auto-renew is on.
DNS Cleanup RetryRe-enqueues cleanup_dns jobs for workflows that finished issuance but left challenge TXT records behind.
Orphaned Phased DNS-01 RecoveryRecovers phased DNS-01 workflows (idle > 20 min) that no gate or finalize job is driving any more: cleans up the TXT records, resets the certificate status and closes the workflow.
Stuck MSCA RecoveryResumes AD CS pending-approval workflows (idle > 90 min) whose poll chain died, or terminates them once past their 7-day limit.
Stuck CA Order RecoveryFor CA connector orders idle > 90 min with no place/poll/cleanup job alive: re-arms the vendor poll when the certificate still waits (issuing / renewal_in_progress), otherwise closes the order as superseded and enqueues the DCV record cleanup.
Approval Request ExpiryExpires approval requests past their expiry and emits the approval-expired notification.
Stale Notification Reservation RecoveryMarks notification sends stuck as reserved for more than 15 minutes as failed, so a lost notification shows up in History.
Expired HTTP-01 Token CleanupDeletes HTTP-01 challenge tokens past their expiry (a fallback to the database's own expiry index).

Domain Checker​

SweepWhat it doesInterval
Domain WHOIS CheckFinds tracked domains due for a WHOIS expiration check and enqueues domain_expiration_check jobs in batches, rate-limited (2s between enqueues).domain_check_interval_minutes (default 60); batch size domain_check_batch_size (default 50)

Discovery Checker​

SweepWhat it doesInterval
Discovery Source ScanFinds discovery sources due for a scan and enqueues discovery_execute jobs.hourly (fixed)
Discovery Zombie RecoveryMarks discovery runs whose heartbeat is > 10 min stale (started > 30 min ago) as failed so the source can be re-scanned.hourly (fixed)
Discovery Orphan CleanupEnqueues one discovery_orphan_cleanup job per project that has discovery sources or still holds discovered certificates: discovered certificates that no endpoint and no server location holds, and that were last written more than 5 minutes ago, leave the inventory.hourly (fixed)
Discovery Findings RefreshEnqueues one discovery_findings_refresh job per project that has discovery sources: findings are recomputed and discovery expiry alerts sent without a scan. A project a scan refreshed in the last 5 minutes is skipped.hourly (fixed)

How the renewal sweep decides a certificate is due​

A certificate is picked up when all of these hold:

  • auto_renew is on,
  • status is active, renewal_pending, or expired (an expired certificate only within 30 days of its expiry — older ones need a manual renew),
  • next_renewal_at ≤ now,
  • renewal attempts haven't exhausted the maximum.

Each cycle picks up at most 500 due certificates, soonest-to-expire first; the rest follow on the next cycles.

next_renewal_at is computed at issuance/renewal as expires_at − renewal_threshold_days, where the threshold is the certificate's own Renewal Threshold if set, otherwise the global default (Settings → General → Renewal Threshold, default 30 days). If the CA supports ACME ARI, the ARI sweep can override next_renewal_at with the CA's suggested window. Full lifecycle details: Renewal.

Detection latency

The sweep runs every renewal_check_interval_minutes. With the default 60, a certificate becomes eligible up to an hour before the scheduler notices — irrelevant against a 30-day threshold, but worth remembering when testing with tiny thresholds.

Manual vs scheduled work​

The scheduler is only one producer for the job queue. Manual actions — issuing, renewing, revoking from the UI or API, bulk renew, running a distribution — enqueue jobs directly and do not wait for a sweep. The same workers execute both kinds; the scheduler's job is exclusively the periodic discovery of due work. Consequently, a dead scheduler doesn't break manual operations — it silently stops automatic renewals, which is why the System Status health badge flips to critical when no leader holds the lock.

Status endpoint​

  • GET /api/v1/settings/scheduler-status (admin) — per-component health, active leader, and each sweep's last run with duration and counters (scanned, matched, enqueued, updated, cleaned, skipped, failed).
  • GET /api/v1/settings/scheduler-status/history?component=<c>&sweep=<s>&limit=N (admin) — recent runs for one sweep (default 20, max 100; retained 7 days).

Both feed the Settings → System Status page.

See also​