Operations runbook

Symptom → diagnosis → resolution → prevention for the ten incidents you are most likely to page on. Every command and metric on this page is verified against the current release. API base URL used throughout: https://<host>/api/v1. The curl examples use cookie auth (-b cookies.txt) plus the CSRF header on mutations; a Bearer access token works equally (and skips CSRF).

For the short-form version of the most common three scenarios, see Day-2 operations. Metric names referenced below are documented in the Prometheus metrics reference.

1. Dead-letter jobs

Jobs that exhaust all retry attempts (default: 3) move to dead status. Dead jobs are terminal and will not be retried automatically. See job types for per-type attempt counts.

Symptoms

• Certificates stuck in issuing, renewal_in_progress, or renewal_pending status.
• certautopilot_job_queue_depth{status="dead"} is non-zero.
• certautopilot_job_total{status="dead"} increasing.
• Notifications for job.failed events.

Diagnosis

1. Query dead jobs via the API:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/jobs?status=dead&page=1&page_size=20' | jq .

2. Inspect a specific dead job for its error message:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/jobs/<job_id>' | \
  jq '.job | {id, type, status, error_msg, attempts, max_attempts, certificate_id, created_at}'

Key fields: error_msg (the last error before the job went dead), attempts vs max_attempts (confirms retries were exhausted), certificate_id (links back to the affected certificate).

3. Check the job's log entries for the full execution history:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/jobs/<job_id>/logs?page=1&page_size=50' | \
  jq '.logs[] | {ts, level, message}'

Look for error-level entries showing the progression across attempts. Reclaim entries indicate a previous worker died mid-execution.

4. Check Prometheus for the failure rate:

rate(certautopilot_job_total{status="dead"}[1h])

Resolution

1. Fix the root cause identified from error_msg and job logs. Common causes: ACME account deactivated or rate-limited, DNS credential expired or missing permissions, target host unreachable (distribution jobs).

2. Retry the dead job (admin project role; rate-limited to 20 requests/minute):

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/jobs/<job_id>/retry'

This resets the job to pending with next_run_at set to now; the next available worker picks it up. The same action is available in the UI — see retrying a job.

3. Verify the retried job completes by polling the job status.

Prevention

• Notification rules for job.failed events.
• Alert on certautopilot_job_queue_depth{status="dead"} > 0.
• Failed attempts back off exponentially (30 s × 2ⁿ⁻¹, capped at 10 minutes) before going dead. If root causes are transient but slow to clear, the backoff window may simply be too short — fix the dependency rather than raising attempt counts.

2. Renewal failures

Certificates are not renewing automatically as they approach expiration.

Symptoms

• Certificate status remains active past its next_renewal_at timestamp, or is renewal_failed with a non-zero renewal_attempts count.
• certautopilot_certificates_expiring_soon gauge rising.
• No renew_certificate (or msca_renew_certificate) jobs appearing in the queue.

Diagnosis

1. Verify the scheduler is running and is the leader:

certautopilot_scheduler_is_leader == 1

If this is 0 on all instances, no renewal sweeps run — see leader election issues.

2. Check the certificate's renewal settings:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/certificates/<cert_id>' | \
  jq '{auto_renew, status, expires_at, next_renewal_at, renewal_threshold_days, renewal_attempts, last_error}'

3. Validate the ACME account:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/acme-accounts/<account_id>' | \
  jq '{status, is_registered, last_validated_at}'

If status is not active or is_registered is false, re-validate:

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/acme-accounts/<account_id>/validate'

4. Test the DNS credential (needs permission to create and delete TXT records in the zone — see DNS troubleshooting):

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/dns-credentials/<cred_id>/test'

5. Verify the zone configuration — confirm the zone's dns_credential_id points to a valid, active credential:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/zones/<zone_id>' | jq '{domain, dns_credential_id}'

6. Check for existing in-flight renewal jobs. The scheduler uses idempotency keys (renew:<cert_id>:<expires_at_unix>), so a pending or active renewal blocks duplicate enqueue:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/jobs?status=pending&type=renew_certificate&certificate_id=<cert_id>' | jq .

7. Check CA rate limits for the zone:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/zones/<zone_id>/rate-limits' | jq .

Resolution

1. Fix the underlying issue (ACME account, DNS credentials, zone config, rate limits).
2. Trigger a manual renewal:

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/certificates/<cert_id>/renew'

3. Monitor the resulting job via the certificate's job list or the Jobs page.

Prevention

• Set renewal_threshold_days high enough to allow multiple retry cycles before expiration.
• Notification rules for cert.expiring_soon and cert.renewal_failed events.
• Periodically validate ACME accounts and test DNS credentials.
• Alert on certautopilot_certificates_expiring_soon. See renewal for the full renewal model.

3. Leader election issues

The scheduler uses MongoDB-backed distributed locking for leader election; only one scheduler instance runs sweeps at a time.

Symptoms

• certautopilot_scheduler_is_leader is 0 on all instances.
• No renew_certificate or certificate_expiration_check jobs being created.
• Scheduler logs show repeated "not leader, waiting to retry" messages.

Diagnosis

1. Check the leader metric across all scheduler instances — exactly one should report 1.

2. Inspect the lock document directly in MongoDB:

db.locks.findOne({ _id: "certautopilot:scheduler:leader" })

Check owner (which instance holds it) and expires_at. If expires_at is in the past but no instance acquires the lock, suspect MongoDB connectivity.

3. Check MongoDB connectivity from the scheduler pod:

kubectl exec -it <scheduler-pod> -- mongosh --eval 'db.adminCommand("ping")'

4. Check scheduler logs for leader heartbeat failed, failed to release leader lock, or acquired leader lock:

kubectl logs <scheduler-pod> --tail=100 | grep -E 'leader|heartbeat|lock'

Resolution

Stale lock (previous leader died without releasing). The lock TTL defaults to 90 s (scheduler.leader_lock_ttl); the heartbeat extends it every 30 s (scheduler.heartbeat_interval). Normally you just wait — a standby acquires the lock on its next retry (every 30 s). To force it:

db.locks.deleteOne({ _id: "certautopilot:scheduler:leader" })

MongoDB connectivity issues. Resolve the network or MongoDB availability problem; the scheduler retries acquisition every 30 seconds.

All schedulers restarted simultaneously. The first instance to acquire wins. Worst-case transition time is roughly heartbeat interval (30 s) + retry wait (30 s) + clock skew ≈ 62 s.

Rolling updates. Use maxUnavailable: 0 so the new scheduler pod starts before the old one terminates; the old pod releases the leader lock explicitly during graceful shutdown.

Prevention

• Deploy 2 scheduler replicas — see high availability.
• Alert if sum(certautopilot_scheduler_is_leader) == 0 for more than 5 minutes.
• Watch certautopilot_scheduler_leader_transitions_total — frequent transitions suggest instability.
• Give pods enough terminationGracePeriodSeconds for the shutdown hook to release the lock.

4. KEK rotation

CertAutoPilot uses envelope encryption: each sensitive field gets a random AES-256-GCM DEK, encrypted by the KEK; kek_version is tracked on every encrypted record.

Rotation is a built-in, resume-safe background job driven by the CLI — do not hand-roll re-encryption scripts. The full procedure, failure modes, and rollback story live in the KEK rotation runbook; the command surface is in the CLI reference. The short version:

# Standalone hosts: use the cap wrapper — it loads secrets.env, sudo alone does not.
# In Kubernetes, run the bare binary via kubectl exec instead.
cap kek status                       # pre-flight: versions, provider, cluster roster
cap kek verify --target=<N>          # is the whole fleet ready for version N?
cap kek rotate --to-version=<N>      # leader-elected background re-encryption
cap kek status                       # watch progress / completion
cap kek remove --version=<N-1>       # retire the old version (after heartbeat window)

Collections holding envelope-encrypted fields (what the rotation touches):

Collection	Encrypted fields
acme_accounts	encrypted_private_key, eab_hmac_key_encrypted
dns_credentials	encrypted_config
certificate_private_keys	encrypted_key
module_credentials	encrypted_secret
notification_channels	encrypted_config
users	totp_encrypted_secret (TOTP-enabled users)

To audit progress or verify completion per collection:

db.acme_accounts.aggregate([{ $group: { _id: "$kek_version", count: { $sum: 1 } } }])

warning

Never remove an old KEK version while any record or any live process still references it — kek remove enforces both checks, and --force only bypasses the record scan, not the fleet check. Keep the old KEK material until kek remove succeeds.

Prevention

• Schedule rotations per your security policy; rehearse in staging first.
• Store KEKs in a secrets manager, never in version control.
• Record which KEK versions each MongoDB backup depends on — see backup & restore.

5. Queue depth high

The job queue has accumulated a large number of pending jobs.

Symptoms

• certautopilot_job_queue_depth{status="pending"} exceeds your threshold (e.g. > 1000).
• Time from job created_at to pickup growing; certautopilot_job_duration_seconds shifting right.
• Certificates waiting longer than expected for issuance or renewal.

Diagnosis

1. Queue depth by status — the gauge is refreshed every 60 s by the worker:

certautopilot_job_queue_depth

2. Queue depth by job type in MongoDB:

db.jobs.aggregate([
  { $match: { status: "pending" } },
  { $group: { _id: "$type", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
])

3. Stuck active jobs (lock expired but not reclaimed — workers reclaim these automatically while polling; a high count means workers aren't running):

db.jobs.find({ status: "active", lock_expires_at: { $lt: new Date() } }).count()

4. Worker count and processing rate:

kubectl get pods -l app=certautopilot,mode=worker

rate(certautopilot_job_total[5m])

Resolution

1. Scale workers horizontally. Workers are stateless; each polls every 2 seconds and claims jobs atomically via findOneAndUpdate:

kubectl scale deployment certautopilot-worker --replicas=5

2. Cancel unnecessary pending jobs (duplicates, obsolete):

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/jobs/<job_id>/cancel'

3. Check for cascading job creation. Some flows create follow-up jobs (issue_certificate enqueues cleanup_dns; issuance/renewal and the scheduler sweep enqueue distribution_execute; fan-out splits into batch children). A backlog in parent jobs can cascade. Note that distribution_execute/distribution_rollback run on a dedicated worker lane — a main-queue backlog and a distribution backlog scale independently.

Prevention

• Autoscale worker deployments on queue depth; alert on certautopilot_job_queue_depth{status="pending"} > 500 (tune to your workload).
• Watch certautopilot_job_retries_total — high retry rates amplify queue depth.
• Ensure MongoDB has capacity for the jobs collection; the {status, next_run_at} index is critical for claim performance (indexes are ensured automatically at startup).

6. LDAP outage

Users with auth_source: "ldap" cannot authenticate while the LDAP server is unreachable.

Symptoms

• LDAP users receive login failures; local users log in normally.
• The LDAP test endpoint returns connection errors.

Diagnosis

1. Test LDAP connectivity from the application (owner role):

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/settings/ldap/test'

2. Review the LDAP settings (host, port, bind_dn, base_dn):

curl -s -b cookies.txt 'https://<host>/api/v1/settings/ldap' | jq .

3. Test network reachability from the pod and check application logs:

kubectl exec -it <api-pod> -- nc -zv <ldap-host> <ldap-port>
kubectl logs <api-pod> --tail=100 | grep -i ldap

Resolution

1. Local accounts still work. Users with auth_source: "local" authenticate against bcrypt hashes in MongoDB — use a local admin account for emergency access.

2. Temporary outage: wait for LDAP to recover; CertAutoPilot does not cache LDAP credentials. Users holding valid access tokens (15 m) and refresh tokens (7 d) keep working without re-authenticating.

3. Settings need updating: PUT /api/v1/settings/ldap (owner role) with the corrected host/port/bind settings, then re-test. Field reference: LDAP settings.

4. LDAP permanently decommissioned: convert LDAP users to local accounts (auth_source: "local" in the users collection), then set passwords via user management.

Prevention

• Keep at least one local owner account for break-glass access, credentials stored in a password manager or sealed secret.
• Monitor LDAP externally; use a failover/replica URI if your directory supports it.

7. Audit chain verification

The audit log is an HMAC chain: each entry's checksum is HMAC-SHA256(signingKey, prevChecksum + entryID + timestamp + action + resourceType), with the JWT secret as signing key. Any direct modification breaks the chain. Background: Audit & SIEM.

When to run

Compliance evidence, suspicion of unauthorized database modification, or routine periodic integrity checks.

Diagnosis

Run chain verification via the API (owner role, rate-limited to 1 request/minute):

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/audit-logs/verify?limit=1000' | jq .

{
  "verified": 1000,
  "intact": true,
  "breaks": [],
  "next_since": "2026-02-27T10:30:00.000000001Z"
}

For large logs, paginate with &since=<next_since> from the previous response. If intact is false, each element of breaks carries entry_id, timestamp, expected_checksum, actual_checksum; fetch the entry via GET /api/v1/audit-logs/<entry_id>.

Resolution

A break at entry N means entry N was tampered with, or entry N−1's checksum was changed (which cascades).

1. Treat it as a security incident; record the break details.
2. Investigate MongoDB access logs — the application should be the only writer to audit_logs.
3. If the break coincides with a known event (e.g. a database restore), document it; a break at the restore boundary is expected. Continue monitoring from that point forward.

Prevention

• Restrict direct write access to audit_logs; back it up with a read-only MongoDB user.
• Run verification periodically (weekly/monthly) and alert on failures.
• Changing jwt.secret breaks verification of entries written under the old secret — treat the secret as part of the audit trust anchor.

8. MongoDB recovery

Backup strategy, schedules, and full restore procedures are covered in Backup & restore — follow that page for the mechanics (mongodump/mongorestore, secret-store pairing, Kubernetes variants). This section covers what to verify after a restore, which is where incidents usually hide.

Before restoring

Stop all CertAutoPilot instances (API, worker, scheduler) so nothing writes during the restore.

Post-restore verification

1. Health endpoints — bring up one API replica and check GET /healthz (liveness) and GET /readyz (readiness).

2. Indexes — recreated automatically at startup (database.EnsureIndexes()); confirm in startup logs.

3. Decryption — the configured KEK versions must cover every kek_version present in the restored data. Trigger a decryption path to prove it:

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/acme-accounts/<account_id>/validate'

If this fails with an unknown-KEK-version error, load the KEK that matches the backup — see backup & restore troubleshooting.

4. Stale locks — locks captured in the backup may block the scheduler:

db.locks.deleteMany({ expires_at: { $lt: new Date() } })

5. Orphaned active jobs — jobs that were active at backup time carry stale locks; workers reclaim them automatically once lock_expires_at passes (5-minute lock TTL). No action needed beyond awareness:

db.jobs.find({ status: "active" }).count()

6. Scale everything back up and watch the Jobs page for the first scheduler sweep.

Prevention

• Automate backups and run periodic restore drills.
• Record the KEK versions each backup depends on, and store KEK material separately from the dump.

9. Worker graceful shutdown

How workers handle shutdown signals and interact with the Kubernetes pod lifecycle.

Symptoms

• Jobs appearing to fail or be reclaimed during deployments.
• certautopilot_job_retries_total spiking during rollouts.

Diagnosis

Check worker logs from the terminated pod:

kubectl logs <worker-pod> --previous --tail=50 | grep -E 'shutdown|graceful|reclaim'

Expected messages: shutdown initiated → in-flight job completed within deadline, or timeout exceeded → job lock released for reclaim.

The shutdown sequence

1. SIGTERM received; the worker stops polling for new jobs.
2. It waits for the in-flight job — the handler gets a background context so it can finish.
3. serve.go calls worker.Shutdown(55s) (sized as terminationGracePeriodSeconds 60 s − preStop 5 s).
4. If the job completes in time: clean shutdown, job marked normally.
5. If the timeout is exceeded, the worker releases the job lock: status back to pending, locked_by/lock_expires_at cleared, attempts decremented by one (the interrupted attempt is not counted), next_run_at set to now — another worker picks it up immediately.

If Kubernetes SIGKILLs before that completes, the lock simply expires after the 5-minute lock TTL and any worker reclaims the job. While a job runs, a heartbeat goroutine extends the lock and checks for cancellation requests, so live jobs are never reclaimed out from under a healthy worker.

Prevention

• terminationGracePeriodSeconds: 65 or higher (55 s shutdown timeout + buffer).
• Rolling updates with maxUnavailable: 0 so replacement workers are up before old ones drain.
• Long-running jobs (e.g. ACME issuance waiting on DNS propagation) are safe either way — the reclaim mechanism guarantees no work is lost, at the cost of a re-run.

10. Distribution failures

Certificate distribution to targets (SSH hosts, Kubernetes clusters, webhooks, …) is failing. Module-specific troubleshooting lives on each module's page under Distribution; this scenario is the generic triage path.

Symptoms

• Distribution last_status is failed or partial.
• certautopilot_distribution_total{status="failed"} increasing.
• distribution_execute jobs in failed or dead status; certificates with distribution_pending: true for extended periods.

Diagnosis

1. Distribution status and per-target results:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/certificates/<cert_id>/distributions' | \
  jq '.[] | {id, module_config_id, last_status, last_error, last_error_at}'

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/certificates/<cert_id>/distributions' | \
  jq '.[].last_target_results[] | {target_name, status, error, error_code, duration_ms}'

Decode error_code with the distribution error codes reference — the code's class also tells you whether fan-out already retried the target.

2. Job logs:

curl -s -b cookies.txt \
  'https://<host>/api/v1/projects/<project_id>/jobs?type=distribution_execute&certificate_id=<cert_id>&page=1&page_size=5' | \
  jq '.jobs[0] | {id, status, error_msg}'

3. Module config health & dry run (no changes made):

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/modules/<module_config_id>/health'

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/certificates/<cert_id>/distributions/<dist_id>/dry-run'

4. Target connectivity — for SSH: kubectl exec -it <worker-pod> -- nc -zv <target-host> <port>; for Kubernetes: verify the kubeconfig/in-cluster access; for webhooks: verify the endpoint returns 2xx. Remember the outbound network policy blocks link-local and cloud-metadata ranges by default.

5. Credentials — list GET /projects/<project_id>/credentials and confirm the credential the target references still exists and is valid (CREDENTIAL_MISSING in error_code is the giveaway).

Resolution

1. Fix connectivity/auth/credential issues per the error code.
2. Update the credential if needed: PATCH /projects/<project_id>/credentials/<cred_id>.
3. Re-execute:

curl -s -X POST -b cookies.txt -H "X-CSRF-Token: <csrf_token>" \
  'https://<host>/api/v1/projects/<project_id>/certificates/<cert_id>/distributions/<dist_id>/execute'

4. If the target ended up in a bad state and the module supports rollback, POST .../distributions/<dist_id>/rollback.
5. Stuck distribution_pending: the scheduler's distribution sweep re-enqueues certificates whose distribution_pending has been true for more than 5 minutes. If nothing happens, check the scheduler leader first.

Prevention

• Dry-run before enabling automatic distribution; run health checks on module configs periodically.
• Enable auto_rollback_on_failure only where rollback is safe for the module.
• Notification rules for distribution.failed; alert on rate(certautopilot_distribution_total{status="failed"}[15m]).
• Test PathSet permissions and ActionSet commands before pointing production certs at them; give Kubernetes service accounts the minimum RBAC needed for Secret writes.

Quick reference

Key metrics

Full catalogue: Prometheus metrics reference.

Metric	Alert on
certautopilot_scheduler_is_leader	sum across instances == 0 for > 5 m
certautopilot_scheduler_leader_transitions_total	> 5 in 1 h (instability)
certautopilot_job_queue_depth{status="pending"}	> 500
certautopilot_job_queue_depth{status="dead"}	> 0
certautopilot_job_retries_total	sustained rate during steady state
certautopilot_certificates_expiring_soon	> 0
certautopilot_distribution_total{status="failed"}	rate > 0
certautopilot_notification_failures_total	rate > 0
http_request_duration_seconds	p99 > 5 s

Health endpoints

Endpoint	Auth	Purpose
GET /healthz	none	Liveness probe
GET /readyz	none	Readiness probe
GET /metrics	none	Prometheus metrics (API listener only)

Key MongoDB collections

Collection	Purpose	Encrypted fields
jobs / job_logs	Job queue + execution logs	—
locks	Distributed locks (scheduler leader, KEK install)	—
audit_logs	HMAC-chained audit trail	— (HMAC checksums)
acme_accounts	ACME registrations	encrypted_private_key, eab_hmac_key_encrypted
dns_credentials	DNS provider credentials	encrypted_config
certificate_private_keys	TLS private keys	encrypted_key
module_credentials	Distribution credentials	encrypted_secret
notification_channels	Notification config	encrypted_config
users	User accounts	totp_encrypted_secret

See also

• Day-2 operations (scheduler, job queue, cluster)
• Troubleshooting (install, ACME, DNS, upgrade)
• Prometheus metrics reference
• Job types reference
• KEK rotation
• Backup & restore