Prometheus metrics
The complete list of Prometheus metrics the backend registers — name, type, labels, and meaning — plus where the /metrics endpoint lives and which process emits which metric.
Endpoint & scrape setup
The API server exposes GET /metrics (Prometheus text format) on the same listener as the REST API (default 0.0.0.0:8181, standalone installs bind 127.0.0.1:18181 behind nginx). The endpoint is unauthenticated, like /healthz and /readyz.
# /etc/prometheus/conf.d/certautopilot.yml
- job_name: certautopilot
scrape_interval: 30s
metrics_path: /metrics
static_configs:
- targets: ["cap-host.internal:8181"]
Config keys under telemetry:
telemetry:
metrics:
enabled: true # default: true
tracing:
enabled: false
endpoint: ""
sample_rate: 1.0 # default: 1.0
The /metrics endpoint is served by the API HTTP server only — processes started with --mode=worker or --mode=scheduler do not run an HTTP listener and cannot be scraped. Job, distribution, notification, and scheduler metrics are emitted in-process by whichever process does the work: with --mode=all (the default) everything lands on one /metrics endpoint, but in a split deployment (separate api/worker/scheduler processes) worker- and scheduler-side metrics are not exposed. Plan your topology accordingly if you need those series.
HTTP & API
Emitted by the API server's metrics middleware. The path label uses the route pattern (e.g. /api/v1/projects/:projectId/certificates/:id), not the raw URL, so certificate IDs do not explode cardinality.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
http_request_duration_seconds | histogram | method, path, status | HTTP request duration. Default Prometheus buckets. |
http_requests_total | counter | method, path, status | Total HTTP requests. |
auth_login_total | counter | auth_source, result | Login attempts. Registered but not currently emitted. |
auth_refresh_total | counter | result | Token refresh attempts. Registered but not currently emitted. |
db_query_duration_seconds | histogram | query | Database query duration. Registered but not currently emitted. |
Job queue
Emitted by the worker process (--mode=worker or all).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_job_duration_seconds | histogram | type, status | Job execution duration. Buckets: 1, 5, 10, 30, 60, 120, 300, 600 s. |
certautopilot_job_total | counter | type, status | Jobs processed, by job type and outcome (completed, failed, dead, cancelled). Jobs finishing with a partial result count as completed. |
certautopilot_job_queue_depth | gauge | status | Current number of jobs per status (pending, active, dead, …). Refreshed every 60 s by the worker. |
certautopilot_job_retries_total | counter | type | Retry re-schedules (failed attempts below max_attempts). |
Certificates
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_certificates_expiring_soon | gauge | — | Certificates within the expiry threshold. Set by the certificate_expiration_check job (worker process). |
certautopilot_certificates_total | gauge | status | Certificates by status. Registered but not currently emitted. |
certautopilot_certificate_renew_total | counter | ca, status | Certificate renewals. Registered but not currently emitted. |
certautopilot_certificate_renew_duration_seconds | histogram | ca | Renewal duration. Buckets: 10–1200 s. Registered but not currently emitted. |
Distribution
Emitted by the distribution worker while executing distribution_execute jobs.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_distribution_total | counter | module_type, status | Distribution executions by module (ssh, kubernetes, webhook, …) and result (success, partial, failed). |
certautopilot_distribution_duration_seconds | histogram | module_type | Distribution execution duration. Buckets: 1–300 s. |
certautopilot_distribution_validation_total | counter | module_type, valid | Post-distribution validation outcomes (valid="true" when all endpoints passed). |
Scheduler
Emitted by the scheduler process (--mode=scheduler or all).
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_scheduler_is_leader | gauge | — | 1 on the instance holding the leader lock, 0 on standbys. |
certautopilot_scheduler_leader_transitions_total | counter | — | Leader election transitions. Frequent transitions indicate instability. |
Notifications
Emitted by the worker while executing notification.send jobs.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
certautopilot_notification_total | counter | channel_type, status | Notification deliveries: sent, failed, or skipped (rule matched but delivery suppressed). |
certautopilot_notification_failures_total | counter | channel_type | Delivery failures by channel type (email, slack, teams). |
Metrics marked "registered but not currently emitted" are declared in the telemetry registry but have no emission site in the current codebase — their series will not appear in scrape output (labelled counters and histograms only materialise on first use). They are listed for completeness and may start reporting in a future release. Some older documentation describes a broader cap_* metric family; the names on this page are the ones the binary actually registers.
Useful queries & alerts
# Dead jobs accumulating
certautopilot_job_queue_depth{status="dead"} > 0
# No scheduler leader anywhere (renewals stall)
sum(certautopilot_scheduler_is_leader) == 0
# Distribution failure rate by module
rate(certautopilot_distribution_total{status="failed"}[15m])
# Job retry storm (often deploy- or dependency-related)
rate(certautopilot_job_retries_total[5m]) > 0.2
# Certificates approaching expiry
certautopilot_certificates_expiring_soon > 0