Prometheus metrics
The complete list of Prometheus metrics the backend registers — name, type, labels, and meaning — plus where the /metrics endpoint lives and which process emits which metric.
Endpoint & scrape setup
The API server exposes GET /metrics (Prometheus text format) on the same listener as the REST API (default 0.0.0.0:8181, standalone installs bind 127.0.0.1:18181 behind nginx). The endpoint is unauthenticated, like /healthz and /readyz.
# /etc/prometheus/conf.d/certautopilot.yml
- job_name: certautopilot
scrape_interval: 30s
metrics_path: /metrics
static_configs:
- targets: ["cap-host.internal:8181"]
Config keys under telemetry:
telemetry:
metrics:
enabled: true # default: true
tracing:
enabled: false
endpoint: ""
sample_rate: 1.0 # default: 1.0
The /metrics endpoint is served by the API HTTP server only — processes started with --mode=worker or --mode=scheduler do not run an HTTP listener and cannot be scraped. Job, distribution, notification, discovery, and scheduler metrics are emitted in-process by whichever process does the work: with --mode=all (the default) everything lands on one /metrics endpoint, but in a split deployment (separate api/worker/scheduler processes) worker- and scheduler-side metrics are not exposed. Plan your topology accordingly if you need those series.
HTTP & API
Emitted by the API server's metrics middleware. The path label uses the route pattern (e.g. /api/v1/projects/:projectId/certificates/:id), not the raw URL, so certificate IDs do not explode cardinality.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
http_request_duration_seconds | histogram | method, path, status | HTTP request duration. Default Prometheus buckets. |
http_requests_total | counter | method, path, status | Total HTTP requests. |
auth_login_total | counter | auth_source, result | Login attempts. Registered but not currently emitted. |
auth_refresh_total | counter | result | Token refresh attempts. Registered but not currently emitted. |
db_query_duration_seconds | histogram | query | Database query duration. Registered but not currently emitted. |
Job queue
Emitted by the worker process (--mode=worker or all).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_job_duration_seconds | histogram | type, status | Job execution duration. Buckets: 1, 5, 10, 30, 60, 120, 300, 600 s. |
certautopilot_job_total | counter | type, status | Jobs processed, by job type and outcome (completed, failed, dead, cancelled). Jobs finishing with a partial result count as completed. |
certautopilot_job_queue_depth | gauge | status | Current number of jobs per status (pending, active, dead, …). Refreshed every 60 s by the worker. |
certautopilot_job_stuck_active | gauge | — | Active jobs running longer than the stuck threshold — a backstop for a handler hung past every source-level timeout. Alert on > 0 for 5m; recovery is a process restart, after which the job's lock lapses and another worker cleanly reclaims it. |
certautopilot_job_retries_total | counter | type | Retry re-schedules (failed attempts below max_attempts). |
certautopilot_job_stale_mark_skipped_total | counter | type, mark | Terminal job-status writes skipped because the job had been reclaimed by another run (the reclaiming run's outcome is authoritative). |
Certificates
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_certificates_expiring_soon | gauge | — | Certificates within the expiry threshold. Set by the certificate_expiration_check job (worker process). |
certautopilot_certificates_total | gauge | status | Certificates by status. Registered but not currently emitted. |
certautopilot_certificate_renew_total | counter | ca, status | Certificate renewals. Registered but not currently emitted. |
certautopilot_certificate_renew_duration_seconds | histogram | ca | Renewal duration. Buckets: 10–1200 s. Registered but not currently emitted. |
Distribution
Emitted by the distribution worker while executing distribution_execute jobs.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_distribution_total | counter | module_type, status | Distribution executions by module (ssh, kubernetes, webhook, …) and result (success, partial, failed). Rollback re-deploys are counted separately, not here. |
certautopilot_distribution_rollback_total | counter | module_type, status | Rollback re-deploys by module and result (success, partial, failed) — the distribution_rollback job path, kept out of certautopilot_distribution_total so normal-deploy rates stay clean. |
certautopilot_distribution_duration_seconds | histogram | module_type | Distribution execution duration. Buckets: 1–300 s. |
certautopilot_distribution_validation_total | counter | module_type, valid | Post-distribution validation outcomes (valid="true" when all endpoints passed). |
Scheduler
Emitted by the scheduler process (--mode=scheduler or all).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_scheduler_is_leader | gauge | — | 1 on the instance holding the leader lock, 0 on standbys. |
certautopilot_scheduler_leader_transitions_total | counter | — | Leader election transitions. Frequent transitions indicate instability. |
Notifications
Emitted by the worker while executing notification.send jobs.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_notification_total | counter | channel_type, status | Notification deliveries: sent, failed, or skipped (rule matched but delivery suppressed). |
certautopilot_notification_failures_total | counter | channel_type | Delivery failures by channel type (smtp, slack, teams, webhook). |
Discovery
Emitted by the worker while executing discovery_execute jobs.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_discovery_ordering_violations_total | counter | — | Successful SNI-less baseline scan results that arrived after their host:port's SNI rows, violating the SNI crawler's baseline-before-SNI ordering contract (late failed baselines are order-independent and not counted). Always 0 in normal operation; a non-zero rate is benign for data integrity (the affected rows are persisted instead of default-cert-suppressed, so a few duplicate inventory rows may appear) but signals the crawler's round ordering changed and should be investigated. |
Metrics marked "registered but not currently emitted" are declared in the telemetry registry but have no emission site in the current codebase — their series will not appear in scrape output (labelled counters and histograms only materialise on first use). They are listed for completeness and may start reporting in a future release. Some older documentation describes a broader cap_* metric family; the names on this page are the ones the binary actually registers.
External secret stores
Emitted whenever a target credential is resolved through an external secret store — on every deploy, dry-run and health check, since the value is never cached.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_secret_store_read_total | counter | store_type, outcome | Secret reads from an external store, by store type and result. |
certautopilot_secret_store_read_duration_seconds | histogram | store_type | Duration of a secret read. |
certautopilot_secret_store_retry_total | counter | store_type | Reads that needed an immediate in-store retry (transport error or 5xx). A rising rate points at store instability, which fan-out children cannot absorb. |
Useful queries & alerts
# Dead jobs accumulating
certautopilot_job_queue_depth{status="dead"} > 0
# No scheduler leader anywhere (renewals stall)
sum(certautopilot_scheduler_is_leader) == 0
# Distribution failure rate by module
rate(certautopilot_distribution_total{status="failed"}[15m])
# Job retry storm (often deploy- or dependency-related)
rate(certautopilot_job_retries_total[5m]) > 0.2
# Certificates approaching expiry
certautopilot_certificates_expiring_soon > 0