Skip to main content

Observability

Two signal pipes: Prometheus metrics for fleet health and SLOs, and syslog forwarding for SIEM archival of every audit-relevant event. The audit log covers the chain-integrity and verification path.

Prometheus metrics​

The API process exposes /metrics in the Prometheus text format. It carries no secrets, but it describes your installation (routes, job and distribution activity, which module types and secret stores you use, login success and failure counts). Two settings control it:

  • telemetry.metrics.token (env CERTAUTOPILOT_TELEMETRY_METRICS_TOKEN): when set, /metrics answers only requests with Authorization: Bearer <token> (401 otherwise). Recommended whenever the backend port is reachable by anyone but your monitoring system.
  • telemetry.metrics.enabled: false removes the endpoint.

With no token the endpoint is open to whoever can reach the backend's port — keep that port off untrusted networks.

Scrape — Helm chart​

# values.yaml
serviceMonitor:
enabled: true
interval: 30s
bearerTokenSecret: # optional: an existing Secret holding the token
name: cap-metrics
key: token # the backend requires it, the ServiceMonitor sends it
labels:
release: prometheus # match your Prometheus operator's selector
prometheusRule:
enabled: true # ships recommended alerts (see below)

Scrape — standalone​

# /etc/prometheus/conf.d/certautopilot.yml — Prometheus (or an agent) on the CertAutoPilot host
- job_name: certautopilot
scrape_interval: 30s
static_configs:
- targets: ["127.0.0.1:18181"]
metrics_path: /metrics
# authorization: # when telemetry.metrics.token is set
# credentials_file: /etc/prometheus/cap-metrics-token

On a standalone install the backend binds loopback only (127.0.0.1:18181), and the bundled nginx does not publish /metrics (it did before 1.7.0; the upgrade — and an installer re-run — removes that one location = /metrics block from an existing site file, matching its braces so nothing nested in it is left behind. A backup copy of the site file is taken first; if nginx refuses the edited file, the backup is put back unchanged and the log names it, so you can remove the block by hand — the block is marked by its # --- Prometheus metrics (public) --- comment line). Scrape it on the host itself — a local Prometheus or a forwarding agent — or, to scrape from another host, set telemetry.metrics.token and add your own nginx location that proxies /metrics to the loopback port.

Core metrics​

The most useful series for day-to-day operations (full list with types, labels, and emission sites: Prometheus metrics reference):

http_requests_total{method,path,status} counter
http_request_duration_seconds{method,path,status} histogram

certautopilot_job_total{type,status} counter # completed|failed|dead|cancelled
certautopilot_job_duration_seconds{type,status} histogram
certautopilot_job_queue_depth{status} gauge # refreshed every 60 s
certautopilot_job_stuck_active gauge # active jobs past the stuck threshold
certautopilot_job_retries_total{type} counter
certautopilot_job_stale_mark_skipped_total counter # stale-job sweeps that declined to act

certautopilot_certificates_total{status} gauge # inventory by status
certautopilot_certificates_expiring_soon gauge # set by the expiration-check job
certautopilot_certificate_renew_total{issuer_type,status} counter
certautopilot_certificate_renew_duration_seconds{issuer_type} histogram

certautopilot_distribution_total{module_type,status} counter # success|unverified|partial|failed
certautopilot_distribution_rollback_total{module_type,status} counter # rollback re-deploys
certautopilot_distribution_duration_seconds{module_type} histogram
certautopilot_distribution_validation_total{module_type,valid} counter

certautopilot_scheduler_is_leader gauge
certautopilot_scheduler_leader_transitions_total counter

certautopilot_secret_store_read_total{store_type,outcome} counter # every credential read from an external store
certautopilot_secret_store_read_duration_seconds{store_type} histogram
certautopilot_secret_store_retry_total{store_type} counter # transient-failure second attempts

certautopilot_notification_total{channel_type,status} counter # sent|failed|skipped
certautopilot_notification_failures_total{channel_type} counter

certautopilot_discovery_ordering_violations_total counter # expected to stay 0
Worker/scheduler processes have no /metrics

Only the API HTTP server exposes /metrics. In split-mode deployments (--mode=worker / --mode=scheduler) those processes have no listener — job and scheduler metrics are emitted by whichever process does the work, so in --mode=all you see everything on the API port, while split deployments only surface metrics from the API process. Plan dashboards accordingly.

Recommended alerts​

Starting points for a standalone Prometheus rules file (adjust thresholds to taste):

ALERT JobsGoingDead
IF increase(certautopilot_job_total{status="dead"}[15m]) > 0
FOR 15m
LABELS { severity="warning" }

ALERT CertificatesExpiringSoon
IF certautopilot_certificates_expiring_soon > 0
FOR 1h
LABELS { severity="critical" }

ALERT DistributionFailures
IF increase(certautopilot_distribution_total{status="failed"}[30m]) > 0
FOR 30m
LABELS { severity="warning" }

ALERT SchedulerLeaderFlapping
IF increase(certautopilot_scheduler_leader_transitions_total[1h]) > 4
LABELS { severity="warning" }

ALERT NotificationDeliveryFailing
IF increase(certautopilot_notification_failures_total[30m]) > 0
FOR 30m
LABELS { severity="warning" }

ALERT DiscoveryOrderingViolations
IF increase(certautopilot_discovery_ordering_violations_total[1h]) > 0
LABELS { severity="warning" }

Syslog forwarding​

Every audit event — login, cert issuance, revocation, role change — forwards to an external syslog relay when enabled, in RFC 5424, CEF or LEEF format over UDP, TCP, or TCP+TLS. KEK rotation is not among them: key-lifecycle operations run from the CLI and are not written to the audit log, so an alert built on seeing them in syslog will never fire. Pair with your SIEM (Splunk, ELK / Logstash, Graylog, Datadog, Azure Sentinel) for long-term retention and cross-system correlation.

Configure: Settings → Syslog (org admin role).

  • Host + port — typical: 514 (UDP), 601 (TCP), 6514 (TCP+TLS).
  • Transport: udp / tcp / tcp+tls.
  • Format: rfc5424 (default), cef or leef.
  • Max attempts (default 10) and lock timeout (default 120 s, minimum 30) — the delivery retry controls.
  • TLS CA cert / client cert / client key (tcp+tls only) — PEM material for a private CA or mutual TLS.

Facility, app-name and hostname are not configurable: every message is emitted as facility local0, app-name certautopilot, and the process's own hostname.

Click Test after saving — a dummy syslog message confirms reachability and parses on the relay.

Message format​

<134>1 2026-04-21T14:23:45.123456789Z cap-a certautopilot - - [audit@certautopilot audit_id="..." org_id="..." action="CERT_ISSUED" actor="alice@example.com" resource_type="certificate" resource_id="..." outcome="success"]

Three things to configure a parser correctly:

  • The priority is always <134> — facility local0 (16) × 8 + severity info (6). It does not vary with the event.
  • PROCID and MSGID are always -, and there is no free-text message after the structured-data block.
  • The SD-ID is audit@certautopilot, and the seven keys are exactly audit_id, org_id, action, actor, resource_type, resource_id, outcome. action is truncated to 64 bytes and resource_id to 128, to keep a UDP datagram within safe bounds.

If your SIEM allowlists structured-data IDs, allow audit@certautopilot.

Severity in CEF and LEEF​

RFC 5424 output carries no per-event severity (see above). The cef and leef formats do, derived from the audit outcome:

OutcomeSeverity value
failure7
cancelled4
success3

Delivery reliability​

  • UDP: fire-and-forget. Lost packets are lost. Fine for best-effort feeds.
  • TCP: guaranteed delivery per RFC 5425. There is no buffer_size setting — undelivered events stay in the audit collection and are retried by the forwarder, bounded by max attempts (default 10) with escalating backoff. An event that exhausts its attempts is abandoned. Tune max_attempts and lock_timeout in the syslog settings rather than a buffer size.
  • TCP + TLS: same as TCP plus transport encryption. Required for external SIEMs across untrusted networks.

Audit logs themselves stay in MongoDB regardless of syslog delivery — syslog is a secondary export path, not the primary record. See Audit & SIEM for the chain-integrity model.

SIEM integration​

  • Splunk: use a syslog: 5424 sourcetype. The structured data becomes parsed fields. Splunk Add-on for Unix and Linux handles 5424 natively.
  • ELK / Logstash: syslog input on TCP 6514 with TLS, kv filter on the structured-data prefix.
  • Graylog: Syslog TCP or Syslog UDP input. Enable Parse structured data under input settings.

Egress & SSRF guards​

Outbound network policy (SSRF guard) blocks link-local and cloud-metadata addresses by default. The metrics scrape and syslog relay must be reachable from the backend's namespace / VPC; Kubernetes NetworkPolicies often block egress by default — add explicit rules. Allowlist private endpoints at the network-policy layer if the relay is intentionally on a private network.

Troubleshooting​

Prometheus shows the target as DOWN​

Hit /metrics directly from the scrape host — it should return text starting with # HELP. A 401 means telemetry.metrics.token is set and the scrape does not send Authorization: Bearer <token>; a 404 means telemetry.metrics.enabled is false; HTML means a proxy answered (the standalone nginx no longer publishes /metrics — scrape the loopback port on the host).

"Syslog Test" passes but nothing arrives at the SIEM​

Almost always a relay-side parser problem (the SIEM rejected the structured-data block). Capture on the relay with tcpdump -i any port 6514 (or a relay-specific debug log) and confirm the message hits it. If the message is on the wire but the SIEM dropped it, the structured-data SD-ID — audit@certautopilot — isn't allowlisted in your input config.

TCP+TLS syslog: "x509: certificate signed by unknown authority"​

The relay's CA isn't in CertAutoPilot's trust set. Paste the CA PEM into the TLS CA cert field, or add it to the system trust store on the host running the backend.

See also​