Skip to main content

Observability

Three signal pipes: Prometheus metrics for fleet health and SLOs, OpenTelemetry traces for end-to-end latency and external-API correlation, and syslog forwarding for SIEM archival of every audit-relevant event. The audit log covers the chain-integrity and verification path.

Prometheus metrics​

Every CertAutoPilot process exposes /metrics in the Prometheus text format. The endpoint is unauthenticated but carries no secrets — it returns Prometheus counters, gauges, and histograms only. If you terminate outside-world traffic on the same listener, block /metrics at the LB level or expose it on an internal listener.

Scrape — Helm chart​

# values.yaml
serviceMonitor:
  enabled: true
  interval: 30s
  labels:
    release: prometheus   # match your Prometheus operator's selector
prometheusRule:
  enabled: true           # ships recommended alerts (see below)

Scrape — standalone​

# /etc/prometheus/conf.d/certautopilot.yml
- job_name: certautopilot
  scrape_interval: 30s
  scheme: https
  tls_config:
    insecure_skip_verify: true   # the standalone default is a self-signed cert
  static_configs:
    - targets: ["cap-host.internal:443"]
  metrics_path: /metrics

On a standalone install the backend binds loopback only (127.0.0.1:18181) and is reachable solely through nginx on the public port — scraping :18181 from another host will never connect. Note /metrics is unauthenticated, so block it at the load balancer if the host is internet-facing.

Core metrics​

The most useful series for day-to-day operations (full list with types, labels, and emission sites: Prometheus metrics reference):

http_requests_total{method,path,status}                     counter
http_request_duration_seconds{method,path,status}           histogram

certautopilot_job_total{type,status}                        counter    # completed|failed|dead|cancelled
certautopilot_job_duration_seconds{type,status}             histogram
certautopilot_job_queue_depth{status}                       gauge      # refreshed every 60 s
certautopilot_job_stuck_active                              gauge      # active jobs past the stuck threshold
certautopilot_job_retries_total{type}                       counter
certautopilot_job_stale_mark_skipped_total                  counter    # stale-job sweeps that declined to act

certautopilot_certificates_total{status}                    gauge      # inventory by status
certautopilot_certificates_expiring_soon                    gauge      # set by the expiration-check job
certautopilot_certificate_renew_total{issuer_type,status}   counter
certautopilot_certificate_renew_duration_seconds{issuer_type} histogram

certautopilot_distribution_total{module_type,status}        counter    # success|partial|failed
certautopilot_distribution_rollback_total{module_type,status} counter  # rollback re-deploys
certautopilot_distribution_duration_seconds{module_type}    histogram
certautopilot_distribution_validation_total{module_type,valid} counter

certautopilot_scheduler_is_leader                           gauge
certautopilot_scheduler_leader_transitions_total            counter

certautopilot_secret_store_read_total{store_type,outcome}   counter    # every credential read from an external store
certautopilot_secret_store_read_duration_seconds{store_type} histogram
certautopilot_secret_store_retry_total{store_type}          counter    # transient-failure second attempts

certautopilot_notification_total{channel_type,status}       counter    # sent|failed|skipped
certautopilot_notification_failures_total{channel_type}     counter

certautopilot_discovery_ordering_violations_total           counter    # expected to stay 0
Worker/scheduler processes have no /metrics

Only the API HTTP server exposes /metrics. In split-mode deployments (--mode=worker / --mode=scheduler) those processes have no listener — job and scheduler metrics are emitted by whichever process does the work, so in --mode=all you see everything on the API port, while split deployments only surface metrics from the API process. Plan dashboards accordingly.

Recommended alerts​

Starting points for a standalone Prometheus rules file (adjust thresholds to taste):

ALERT JobsGoingDead
  IF increase(certautopilot_job_total{status="dead"}[15m]) > 0
  FOR 15m
  LABELS { severity="warning" }

ALERT CertificatesExpiringSoon
  IF certautopilot_certificates_expiring_soon > 0
  FOR 1h
  LABELS { severity="critical" }

ALERT DistributionFailures
  IF increase(certautopilot_distribution_total{status="failed"}[30m]) > 0
  FOR 30m
  LABELS { severity="warning" }

ALERT SchedulerLeaderFlapping
  IF increase(certautopilot_scheduler_leader_transitions_total[1h]) > 4
  LABELS { severity="warning" }

ALERT NotificationDeliveryFailing
  IF increase(certautopilot_notification_failures_total[30m]) > 0
  FOR 30m
  LABELS { severity="warning" }

ALERT DiscoveryOrderingViolations
  IF increase(certautopilot_discovery_ordering_violations_total[1h]) > 0
  LABELS { severity="warning" }

OpenTelemetry tracing​

CertAutoPilot emits OTLP traces for every HTTP request and every job execution. Spans include cert IDs, project IDs, and external-API latencies — correlate "this user's renewal failed" to "the ACME provider returned 5xx twice" in one view.

Tracing wiring is inactive in current builds

The tracing config keys exist and the OTel instrumentation is scaffolded, but the initializer is not wired into the server startup in current releases — setting telemetry.tracing.enabled has no effect yet. Treat this section as forward-looking.

Enable:

telemetry:
  tracing:
    enabled: true
    endpoint: http://otel-collector.observability:4318
    sample_rate: 0.1   # 1.0 for debug runs

sample_rate is parent-based — incoming requests with an upstream sampling decision are honoured; otherwise the probabilistic sampler kicks in at the configured rate.

Transport: OTLP/gRPC only (port 4317), always plaintext. The endpoint is a bare host:port — no scheme is parsed, and there is no OTLP/HTTP exporter and no TLS option.

What gets traced​

  • HTTP requests — method, route pattern, status, actor, org, project, child spans for DB queries / job enqueues / service calls.
  • Jobs — job type, ID, attempt number, certificate / distribution ID, child spans for external-API calls (ACME, MSCA, DNS, module target), DB writes, event emissions. Errors set the span to Error with the classification tag (network / auth / etc.).
  • Propagation — W3C Trace Context + Baggage on inbound and outbound. A webhook receiver that propagates the context back produces end-to-end traces across the system boundary.

Span attributes worth filtering on​

AttributeValues
cap.tenant.org_idopaque id
cap.tenant.project_idopaque id
cap.actor.typeuser / system
cap.cert.idopaque id
cap.job.typejob type name
cap.job.attempt1..N
cap.external.provideracme, msca, cloudflare, azurekeyvault, …
cap.error.classnetwork / auth / io_transient / io_permanent / validation

Minimal collector config​

receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }
      grpc: { endpoint: 0.0.0.0:4317 }
processors:
  batch:
exporters:
  otlp/tempo:
    endpoint: tempo.observability:4317
    tls: { insecure: true }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Syslog forwarding​

Every audit event — login, cert issuance, revocation, role change — forwards to an external syslog relay when enabled, in RFC 5424, CEF or LEEF format over UDP, TCP, or TCP+TLS. KEK rotation is not among them: key-lifecycle operations run from the CLI and are not written to the audit log, so an alert built on seeing them in syslog will never fire. Pair with your SIEM (Splunk, ELK / Logstash, Graylog, Datadog, Azure Sentinel) for long-term retention and cross-system correlation.

Configure: Settings → Syslog (org admin role).

  • Host + port — typical: 514 (UDP), 601 (TCP), 6514 (TCP+TLS).
  • Transport: udp / tcp / tcp+tls.
  • Format: rfc5424 (default), cef or leef.
  • Max attempts (default 10) and lock timeout (default 120 s, minimum 30) — the delivery retry controls.
  • TLS CA cert / client cert / client key (tcp+tls only) — PEM material for a private CA or mutual TLS.

Facility, app-name and hostname are not configurable: every message is emitted as facility local0, app-name certautopilot, and the process's own hostname.

Click Test after saving — a dummy syslog message confirms reachability and parses on the relay.

Message format​

<134>1 2026-04-21T14:23:45.123456789Z cap-a certautopilot - - [audit@certautopilot audit_id="..." org_id="..." action="CERT_ISSUED" actor="alice@example.com" resource_type="certificate" resource_id="..." outcome="success"]

Three things to configure a parser correctly:

  • The priority is always <134> — facility local0 (16) × 8 + severity info (6). It does not vary with the event.
  • PROCID and MSGID are always -, and there is no free-text message after the structured-data block.
  • The SD-ID is audit@certautopilot, and the seven keys are exactly audit_id, org_id, action, actor, resource_type, resource_id, outcome. action is truncated to 64 bytes and resource_id to 128, to keep a UDP datagram within safe bounds.

If your SIEM allowlists structured-data IDs, allow audit@certautopilot.

Severity in CEF and LEEF​

RFC 5424 output carries no per-event severity (see above). The cef and leef formats do, derived from the audit outcome:

OutcomeSeverity value
failure7
cancelled4
success3

Delivery reliability​

  • UDP: fire-and-forget. Lost packets are lost. Fine for best-effort feeds.
  • TCP: guaranteed delivery per RFC 5425. There is no buffer_size setting — undelivered events stay in the audit collection and are retried by the forwarder, bounded by max attempts (default 10) with escalating backoff. An event that exhausts its attempts is abandoned. Tune max_attempts and lock_timeout in the syslog settings rather than a buffer size.
  • TCP + TLS: same as TCP plus transport encryption. Required for external SIEMs across untrusted networks.

Audit logs themselves stay in MongoDB regardless of syslog delivery — syslog is a secondary export path, not the primary record. See Audit & SIEM for the chain-integrity model.

SIEM integration​

  • Splunk: use a syslog: 5424 sourcetype. The structured data becomes parsed fields. Splunk Add-on for Unix and Linux handles 5424 natively.
  • ELK / Logstash: syslog input on TCP 6514 with TLS, kv filter on the structured-data prefix.
  • Graylog: Syslog TCP or Syslog UDP input. Enable Parse structured data under input settings.

Egress & SSRF guards​

Outbound network policy (SSRF guard) blocks link-local and cloud-metadata addresses by default. The metrics scrape, OTLP collector, and syslog relay must be reachable from the backend's namespace / VPC; Kubernetes NetworkPolicies often block egress by default — add explicit rules. Allowlist private endpoints at the network-policy layer if the relay or collector is intentionally on a private network.

Troubleshooting​

Prometheus shows the target as DOWN​

Check that /metrics isn't blocked by your nginx / LB. Hit it directly from the scrape host — should return text starting with # HELP. If you see HTML or 401, the path is being intercepted; expose a separate internal listener.

No traces appear in the backend​

Sample rate is 0 or the endpoint is wrong. Set sample_rate: 1.0 for a debug run and tail the collector's logs for POST /v1/traces. Check that the namespace egress allows TCP to the collector port.

"Syslog Test" passes but nothing arrives at the SIEM​

Almost always a relay-side parser problem (the SIEM rejected the structured-data block). Capture on the relay with tcpdump -i any port 6514 (or a relay-specific debug log) and confirm the message hits it. If the message is on the wire but the SIEM dropped it, the structured-data SD-ID — audit@certautopilot — isn't allowlisted in your input config.

TCP+TLS syslog: "x509: certificate signed by unknown authority"​

The relay's CA isn't in CertAutoPilot's trust set. Paste the CA PEM into the TLS CA cert field, or add it to the system trust store on the host running the backend.

See also​