Syslog forwarding
Forward every audit event to a SIEM as syslog over UDP, TCP, or TCP+TLS, encoded as RFC 5424, CEF (ArcSight/Splunk), or LEEF (IBM QRadar). Delivery is queued in MongoDB and retried with exponential backoff, so a receiver outage doesn't drop events. This page is the deep dive behind the audit page's syslog summary.
What gets forwarded
Audit events only — the same entries that appear in Audit & SIEM. When the forwarder is enabled, each new audit entry is written with a syslog_status: pending marker at creation; a background forwarder picks pending entries up and ships them. Two consequences:
- The local audit record is always persisted first — syslog problems never lose or block audit writes.
- Forwarding is not retroactive: only events created while syslog is enabled are sent. Enabling it does not replay history.
Message format
Three wire formats are available via the Format setting: RFC 5424 (default), CEF (ArcSight / Splunk), and LEEF (IBM QRadar). All three carry the same audit fields and are wrapped in an RFC 5424 syslog header for transport — the difference is how the payload is encoded so your SIEM parses it natively. There is no RFC 3164 or raw-JSON option.
RFC 5424 (default)
<134>1 2026-06-25T12:34:56.789012Z cap-backend-1 certautopilot - - [audit@certautopilot audit_id="665f..." org_id="org-1" action="CERTIFICATE_ISSUED" actor="alex@example.com" resource_type="certificate" resource_id="9b2e..." outcome="success"]
| Part | Value |
|---|---|
| PRI | fixed <134> — facility 16 (local0), severity 6 (info) |
| VERSION | 1 |
| TIMESTAMP | RFC 3339 with nanoseconds, UTC |
| HOSTNAME | backend host / pod name (resolved at process start, - if unknown) |
| APP-NAME | certautopilot |
| PROCID / MSGID | - (unused) |
| STRUCTURED-DATA | one SD element, [audit@certautopilot ...] |
Structured-data keys:
| Key | Content |
|---|---|
audit_id | Audit entry ID — join back to the local log |
org_id | Organization |
action | Audit action, e.g. CERTIFICATE_ISSUED (truncated at 64 chars) |
actor | Actor name — user email, or system |
resource_type | e.g. certificate, settings |
resource_id | Resource identifier (truncated at 128 chars) |
outcome | failure for actions ending _FAILED / _DEAD, cancelled for _CANCELLED, else success |
Values are quoted with Go-style escaping (JSON-like). The high-cardinality free-text fields are per-field truncated (action at 64, resource_id at 128) to keep messages within safe UDP bounds; truncation is per value, so it can never split a key=value pair. CR, LF, and NUL are folded to spaces in every field so a value can neither split the single-line frame nor be cut short by a receiver that treats NUL as a string terminator.
CEF — ArcSight / Splunk
ArcSight Common Event Format, also parsed natively by Splunk (the CEF add-on) and most SIEMs. The CEF payload sits in the syslog MSG position:
<134>1 2026-06-25T12:34:56.789012Z cap-backend-1 certautopilot - - - CEF:0|CertAutoPilot|CertAutoPilot|1.5.36|CERTIFICATE_ISSUED|CERTIFICATE_ISSUED|3|rt=1782... externalId=665f... suser=alex@example.com src=10.0.0.5 act=CERTIFICATE_ISSUED outcome=success cs1Label=OrgId cs1=org-1 cs2Label=ResourceType cs2=certificate cs3Label=ResourceId cs3=9b2e... cs4Label=ProjectId cs4=proj-1
Header: CEF:0|Vendor|Product|Version|SignatureID|Name|Severity|Extension, where Vendor/Product are CertAutoPilot, Version is the running build, SignatureID/Name are the audit action, and Severity maps the outcome — success→3, cancelled→4, failure→7 (a low informational baseline; correlate up in the SIEM). Extension keys use the ArcSight dictionary: rt (event time, epoch ms), externalId (audit id), suser (actor), src (actor IP), act, outcome, and cs1–cs4 custom strings (org, resource type, resource id, project). Header fields escape \ and |; extension values escape \ and =.
LEEF — IBM QRadar
IBM QRadar Log Event Extended Format (LEEF 2.0), with an explicit caret (^) attribute delimiter declared in the header:
<134>1 2026-06-25T12:34:56.789012Z cap-backend-1 certautopilot - - - LEEF:2.0|CertAutoPilot|CertAutoPilot|1.5.36|CERTIFICATE_ISSUED|^|devTime=1782823 [...] ^sev=3^cat=CERTIFICATE_ISSUED^usrName=alex@example.com^src=10.0.0.5^outcome=success^auditId=665f...^orgId=org-1^projectId=proj-1^resource=certificate^resourceId=9b2e...
Header: LEEF:2.0|Vendor|Product|Version|EventID|DelimiterChar|attributes. QRadar auto-maps the normalized keys devTime (epoch milliseconds — no devTimeFormat needed, and unambiguous regardless of the collector's timezone), sev (severity 1–10, mapped from the outcome like CEF: success→3, cancelled→4, failure→7), usrName, src, and cat; the rest (outcome, auditId, orgId, projectId, resource, resourceId) are custom attributes. Values escape \, the ^ delimiter, and =.
Settings
Settings → Syslog. Viewing, saving, and testing require the admin org role. Config lives in the settings collection under syslog_config.
| Field (JSON) | UI label | Default / range | Notes |
|---|---|---|---|
enabled | Enable | off | Master switch for the forwarder. |
protocol | Protocol | — | udp, tcp, or tcp+tls. |
format | Format | rfc5424 | Payload encoding: rfc5424, cef (ArcSight/Splunk), or leef (QRadar). See Message format. |
host | Host | — | Receiver hostname / IP (also the TLS SNI + verification name). |
port | Port | 514 (1–65535) | |
max_attempts | Max Retry Attempts | 10 (1–100) | Per event, before it's abandoned as dead. |
lock_timeout | Lock Timeout (seconds) | 120 (30–3600) | How long a replica may hold an in-flight event before another reclaims it. |
ca_cert | CA Certificate | — | PEM; only shown for tcp+tls. |
client_cert | Client Certificate | — | PEM; enables mTLS. Must be paired with the key. |
client_key | Client Key | — | PEM; both-or-neither with the client cert. |
The three TLS fields are envelope-encrypted at rest, held in their own record apart from the rest of the syslog settings, and never returned to the browser — responses carry has_ca_cert / has_client_cert / has_client_key booleans, and leaving a field blank on save keeps the stored value. Because they have their own record they are covered by KEK rotation and by its safety checks like every other encrypted field.
The client certificate and key are validated as a pair against the result of the save rather than the submitted form alone, so you can replace just one of them while the other stays as stored — what is rejected is ending up with only one of the two.
This material used to live inside the syslog settings record, where saving any syslog setting re-encrypted it another time; after the first such save the certificates could no longer be read. The first start after upgrading moves it into its own record and repairs anything affected, with no action needed. If the encryption key it was originally sealed under is no longer loaded, the move cannot happen: it is reported in the startup log, nothing is discarded, and the syslog settings page refuses to overwrite the material until it is resolved.
On a multi-node cluster, upgrade every node. The move happens as soon as the first upgraded node starts, and a node still running the previous version cannot see the material in its new place — it reports no certificates and, if syslog is enabled over TLS, its own forwarding fails until it is upgraded. Nothing is lost and no action is needed beyond finishing the upgrade, but audit events that exhaust their retries on a node during that window are abandoned. If your syslog destination requires client certificates and you cannot upgrade the nodes close together, turn syslog forwarding off for the duration.
Config changes are picked up by running forwarders automatically (the config is re-read about once a minute) — no restart needed.
Transports & TLS
- udp — fire-and-forget datagrams; no delivery signal beyond the local send.
- tcp — persistent connection, 10-second connect timeout, 10-second write deadline per message; the connection is closed and re-established after a failed send.
- tcp+tls — as TCP, plus TLS with minimum version 1.2 and server-name verification against Host. A pasted CA certificate is used to verify private receivers; supplying a client cert + key turns on mutual TLS.
Delivery semantics
Queued, at-least-once, bounded:
- Each audit entry tracks its own state:
pending → sending → sent, ordeadaftermax_attemptsfailures. Dead entries stay in MongoDB with the last error for forensics — they are never silently discarded, but they are not retried further. - The forwarder polls every 2 seconds, claims up to 50 pending entries per cycle in FIFO order, and marks each
senton success. - Failures are retried with exponential backoff — 5 s, 30 s, 2 min, 10 min, then hourly (±20% jitter) — until
max_attemptsis reached. A SIEM outage therefore delays events rather than losing them (sizemax_attemptsto cover your worst outage: the default 10 attempts spans roughly 6–7 hours). - Multi-replica safe: entries are claimed with a per-pod lock, so only one replica sends a given event. If a replica dies mid-send, its claim expires after
lock_timeoutand another replica retries — in that crash window the receiver may see a duplicate (useaudit_idto de-duplicate downstream).
Test the connection
The Test Connection button (POST /settings/syslog/test) opens a dedicated one-off connection with the values currently in the form (falling back to stored TLS material where fields are blank) and sends a single test event with action="SYSLOG_TEST". Success or the transport/TLS error is reported inline; the test itself is audit-logged as SYSLOG_CONNECTION_TESTED.
A UDP "success" only means the datagram left the host — there is no acknowledgement. Confirm arrival in the SIEM. TCP/TLS tests genuinely validate connectivity and the handshake.
Troubleshooting
Nothing arrives at the SIEM
Check, in order: the Enable switch is on; the license includes syslog; the test button succeeds; and the events you expect were created after enabling (no retroactive replay). For TCP+TLS, a wrong CA or hostname mismatch fails the handshake — the test surfaces the exact TLS error.
Events arrive in bursts, minutes late
That's backoff draining after a receiver outage — entries queued as pending and are being retried on the 5 s → hourly schedule. Expected; nothing to fix once the receiver is stable.
Some events never arrived
They likely exhausted max_attempts during a long outage and are marked dead (with the last error) on the audit entries. Raise Max Retry Attempts to widen the tolerance window; dead entries are not re-sent automatically.
Duplicate events in the SIEM
A replica crashed between sending and acknowledging; the entry was reclaimed after lock_timeout and re-sent. De-duplicate on the audit_id structured-data key.