Distribution primitives

The reusable objects every distribution composes from: Path Sets (where files land), Action Sets (what runs after), Module Credentials (auth material), Project Variables (per-environment values), Validation Endpoints (post-deploy TLS fingerprint check), and the Fan-out execution model. Define each once; reuse across dozens of targets and certificates.

Who can edit these

Path Sets, Action Sets, and Project Variables are now operator-editable, gated by default through the config-change approval toggle (Settings → General → Approval Requirements, ON by default) — an operator's create/update/delete is filed as an approval request an admin approves. Module Credentials remain admin-only. Admins and owners bypass approval. See Approval workflow and Auth & RBAC.

Path sets

A path set describes where certificate files land on a target. Linux (SSH) path sets carry a POSIX path, an owner (formatted user:group), a mode (octal), and an output format. Windows (WinRM) path sets use drive-letter (C:\…) or UNC (\\server\share\…) paths; owner / mode do not apply (Windows ACLs are managed on the host).

The target_kind field selects the OS. Bind-time validation prevents cross-platform mismatches, so an SSH target cannot bind a Windows path set or vice versa.

Entry schema:

Field	Required	Purpose
source	yes	Which artifact to place: cert (leaf), chain (intermediates), fullchain, private_key, combined (cert + chain + key), pfx (server-built PKCS#12 — works on both Linux and Windows; see below).
path	yes	Absolute target path. Linux: POSIX. Windows: drive-letter or UNC.
owner	no	user or user:group — letters, digits, ., _, -, at most 32 characters per part (validated on save). Linux only — stripped on Windows. Applied when the file is written and re-checked on every run; drift on an unchanged file is corrected with a job-log warning.
mode	no	Octal perms, 3–4 digits (e.g. 0644; validated on save). Linux only. Applied when the file is written and re-checked on every run, like owner.
format	no	pem (default) or der. DER only valid for cert or private_key. Rejected for pfx.
passphrase_variable	iff source=pfx	Name of a secret-flagged project variable that supplies the PKCS#12 passphrase at distribute time. Only the variable name lives on the path set; the value never does.

Example — nginx:

[
  {"source": "fullchain",   "path": "/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.crt", "owner": "root:root", "mode": "0644"},
  {"source": "private_key", "path": "/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.key", "owner": "root:root", "mode": "0600"}
]

Path variables: paths use ${{ NAME }} placeholder substitution — not Go templates. Two kinds of variable are available:

• Reserved certificate-context variables (CAP_ prefix, always present): CAP_PRIMARY_DOMAIN, CAP_DOMAIN_SLUG (lowercased [a-z0-9-], wildcard *. stripped), CAP_CERT_TOKEN (the 8-char stable per-certificate hash), and CAP_DEPLOY_NAME (cap-<token>-<slug>). Prefer CAP_DOMAIN_SLUG over CAP_PRIMARY_DOMAIN inside a path — a wildcard * in the raw domain breaks filesystem paths, whereas the slug is path-safe.
• Project variables by name. Secret-flagged variables are rejected in paths (a rendered path lands in logs and execution detail, which would leak the secret). Certificate-context vars win over project variables on a name collision; the CAP_ prefix is reserved at project-variable creation.

A literal ${{ ... }} is escaped with a preceding backslash (\${{ NOT_A_VAR }}). An unresolved variable name fails the distribution.

PKCS#12 (.pfx) — server-built bundles

For consumers that expect a single PFX file (Windows Certificate Store via Import-PfxCertificate, IIS legacy bindings, modern Tomcat with keystoreType="PKCS12", Spring Boot server.ssl.key-store, .NET Core Kestrel), CertAutoPilot builds the bundle server-side and sends the bytes straight to the target — no openssl there.

On Windows targets the bytes travel on the remote command's standard input, never on its command line, so the key material does not appear in Windows process-creation auditing. The target checks the content it received before the file replaces anything. The encoder uses sslmate's pkcs12.Modern set (AES-256, SHA-256 HMAC).

Operator workflow:

1. Settings → Variables → Add a variable, e.g. APP_PFX_PASSPHRASE, mark Secret: yes, set the chosen passphrase. Encrypted at rest with envelope encryption.
2. Path Sets → Add. Pick the target OS, add a row with source: pfx, an absolute path, and select APP_PFX_PASSPHRASE from the passphrase-variable dropdown.
3. Bind to a target, distribute. The PFX is built once per file; the decrypted passphrase only exists inside the build call's scope.

{
  "source": "pfx",
  "path": "/opt/tomcat/conf/keystore.p12",
  "owner": "tomcat:tomcat",
  "mode": "0600",
  "passphrase_variable": "APP_PFX_PASSPHRASE"
}

PKCS#12 idempotency caveat

The encoder injects a fresh IV/salt per encode, so the bytes change every time. The hash-based skip in both SSH and WinRM modules always reports changed for pfx files, and the bound Action Set runs on every redistribute. If you need quiet redistributes, add a thumbprint check inside your Action Set (Windows: (Get-PfxCertificate ...).Thumbprint; Linux: keytool -list -keystore ... -storepass ...) and gate the restart on a stored marker.

Action sets

An action set is the list of commands the SSH or WinRM module runs on a target after file placement. Linux action sets run shell commands or a script body with sudo (run as), allowlisting, and timeouts. Windows action sets run PowerShell commands or an inline script via powershell.exe / pwsh.exe; run as and command_mode do not apply on Windows. target_kind selects the OS; bind-time validation enforces the match.

Command mode (simple)

mode: command
commands:
  - "nginx -t"
  - "systemctl reload nginx"
run_as: "root"
timeout_seconds: 30

Each command runs sequentially; non-zero on any aborts the rest and marks the target failed.

Inline-script mode (complex)

mode: script_inline
script_body: |
  #!/bin/bash
  set -euo pipefail
  nginx -t
  systemctl reload nginx
  SUBJECT=$(echo | openssl s_client -connect localhost:443 -servername "${{ CAP_PRIMARY_DOMAIN }}" 2>/dev/null | openssl x509 -subject -noout)
  echo "$SUBJECT" | grep -q "${{ CAP_PRIMARY_DOMAIN }}" || { echo "Active cert mismatch" >&2; exit 1; }
shell: "/bin/bash"
run_as: "root"
timeout_seconds: 60

Field reference

Field	Required	Purpose
mode	yes	command or script_inline — the same two values on Linux and Windows. Anything else is rejected at creation with mode must be command|script_inline.
commands	commands mode	List of shell strings.
script_body	script mode	Full script text. Template-expanded before execution.
shell	no	Shell to invoke, validated on save. Linux: an absolute path to sh, bash, dash, zsh, ksh, mksh, ash or fish (default /bin/bash); Windows: powershell.exe (default) or pwsh.exe.
command_mode	no (linux only)	exec (default) runs the command directly; shell wraps it in /bin/sh -c. Each command gets its own SSH session either way — there is no batching mode, and unrecognised values are silently treated as exec.
run_as	no (linux only)	User to execute as — the command is wrapped in sudo -u <user>, without -n. A sudoers rule that prompts for a password therefore blocks until the timeout instead of failing fast; use NOPASSWD entries. Validated on save against ^[a-z_][a-z0-9_-]{0,31}$.
allowed_commands	no	Regex allowlist, enforced in command mode only — an inline script body is uploaded and executed with no pattern check at all. The regex is matched against the whole command string and is not anchored, so a pattern like systemctl reload nginx also admits rm -rf /var/lib/x && systemctl reload nginx; anchor your patterns (^systemctl reload nginx$). An empty allowlist permits every command.
timeout_seconds	no	Per-command timeout, 0–3600 (validated on save). No default — leave it unset (or 0) and there is no per-command bound at all; only the per-target timeout (300 s) applies. Set it explicitly if a command can hang.
run_always	no	Default false. By default the action runs when a deployed file actually changes (hash-based skip) or when the action has not yet completed successfully for the certificate version currently on disk — a crash-safety net, so a run that wrote the files but died before the reload re-runs the reload next time even though the files are byte-identical. A genuine no-op still skips. Set true to run on every distribution even when nothing changed — for verification/diagnostic actions that must run each time (e.g. printing the deployed certificate). When it runs despite no change the log shows no file changed, running anyway (run_always enabled).

Variables in actions: action-set commands and script bodies use the same ${{ NAME }} substitution as paths (not Go templates — there is no .cn/.paths template context). Available names are project variables (referenced by bare uppercase name) plus the four reserved certificate-context variables ${{ CAP_PRIMARY_DOMAIN }}, ${{ CAP_DOMAIN_SLUG }}, ${{ CAP_CERT_TOKEN }}, and ${{ CAP_DEPLOY_NAME }}.

Privilege escalation on Linux wraps the command in sudo -u <user> — without -n, so sudo is free to prompt. A rule that asks for a password will therefore hang the session until the timeout rather than failing immediately. Give the target passwordless rules for the exact commands:

# /etc/sudoers.d/certautopilot
Cmnd_Alias CAP_RELOAD = /usr/sbin/nginx -t, /bin/systemctl reload nginx
capdeploy ALL=(ALL) NOPASSWD: CAP_RELOAD

Module credentials

A single store for every piece of auth material distribution modules need: SSH private keys, kubeconfigs, WinRM username/password pairs, F5 / NetScaler admin passwords, Vault tokens or AppRole pairs, Huawei AK/SK pairs, Azure service principals. All credentials live in module_credentials and are referenced by targets via immutable ID. A credential either holds its value here, sealed with envelope encryption, or references one in your own secret manager — see External secret stores.

Type	Fields	Used by
SSH private key	PEM key + optional passphrase	SSH module
SSH password	Username + password	SSH module
Kubeconfig	Inline kubeconfig YAML	Kubernetes module
IIS WinRM (iis_winrm)	{"username","password"} — the auth method (ntlm/basic/kerberos + realm/KDC) is chosen on the target; legacy blobs carrying auth_type/domain/kdc_host are still honored when the target sets none	IIS module only
WinRM password (winrm_password)	{"username","password"}	Generic WinRM module only
F5 BIG-IP password (f5bigip_password)	{"username","password"}	F5 BIG-IP only
NetScaler password (netscaler_password)	{"username","password"}	NetScaler only
Vault token / AppRole	Token, or role_id + secret_id	HashiCorp Vault
Webhook HMAC / bearer	HMAC signing secret, or bearer token	Webhook module

Each credential type is admitted for exactly the target types listed: a credential created for IIS cannot be bound to a generic WinRM target, and an F5 credential cannot be bound to a NetScaler target, even though the JSON shapes are identical. | SMTP password / PKCS#12 passphrase | Relay password; optional P12 export passphrase | SMTP module | | AWS AK/SK | Access key + secret key | AWS ACM | | Huawei AK/SK | Access key + secret key | Huawei Cloud | | Cloudflare API Token | API token (Zone → SSL and Certificates → Edit) | Cloudflare | | cPanel / WHM API Token | Username + API token (SSL install / WHM access) | cPanel / WHM | | Azure Service Principal | tenant_id + client_id + client_secret (JSON) | Azure Key Vault | | MerlinCDN token | Personal access token + org/workspace IDs | MerlinCDN | | Exchange WinRM | Username + password (auth type + domain on the target) | Microsoft Exchange | | PAN-OS credentials | JSON {"username","password"} or {"username","api_key"} — username is always required (the partial commit is scoped to it) | PAN-OS | | PAM360 API Token | AUTHTOKEN of a PAM360 API user | Secret stores (store login — not used by targets directly) |

Why separate from targets: one SSH key opens ten hosts (one credential → ten targets, one rotation point); credentials only surface to admins (viewers see target lists without the auth material); credential reads and rotations get a clean audit trail.

Rotate by editing the credential and pasting new material — every future distribution picks it up. There is no version history; in-flight jobs holding a cloned client finish with the old material. Wait for in-flight to drain before considering the old material fully revoked.

Or don't store it here at all. A credential can instead reference a secret in your own secret manager (HashiCorp Vault / OpenBao, ManageEngine PAM360), in which case CertAutoPilot keeps no value — only the reference — and reads it from the store every time it is needed. Rotating on your side then takes effect immediately, with nothing to update here. See External secret stores.

Delete is blocked if any target still references the credential, or if a secret store authenticates with it. The UI surfaces blockers; repoint or delete those first.

Per-credential allowlists exist for sensitive modules (e.g. Kubernetes allowed namespaces, allowed name prefix) — use them to prevent a single-tenant credential from being aimed at the wrong namespace by mistake.

Project variables

Key-value pairs scoped to a project, referenced from path sets, action sets, and module configurations through ${{ NAME }} placeholder substitution (not Go templates). Use them to keep per-environment differences (hostnames, ports, service names) out of reusable templates.

Plain vs sensitive:

• Plain — readable in the UI by any project operator. Use for hostnames, port numbers, service names, feature flags.
• Sensitive — value is write-only after creation. List views show <sensitive>. Create, update and delete are audited; worker reads at distribution time are not, so correlate a variable's use with the distribution job log rather than the audit trail. Use for shared tokens, secondary API keys, PKCS#12 passphrases (referenced via passphrase_variable on a pfx path-set row).

Schema:

• name — uppercase identifier, regex ^[A-Z_][A-Z0-9_]{0,127}$. Names are trimmed and uppercased on save.
• value — string.
• is_secret — boolean.
• description — free-text.

Reference by bare uppercase name inside ${{ }}:

# Path set entry
{
  "source": "fullchain",
  "path": "/etc/nginx/ssl/${{ SERVICE_NAME }}.crt"
}

# Action set command
commands:
  - "rsync /etc/nginx/ssl/ ${{ BASTION_HOST }}:/mnt/backup/"

Only path sets and action sets support project-variable substitution. Webhook headers and body do not — the webhook body template exposes certificate fields only, not project variables.

Pattern: one project per environment. The same path-set + action-set combination references the same variable names; the project switches what the names resolve to. production, staging, dr.

Sensitive value redaction is best-effort

If a sensitive value lands on a target host in plaintext (written to a file, echoed into a script), the redaction pipeline stops at that boundary. Don't put irreversible-exposure secrets into action sets that echo them.

Delete is blocked if any path set, action set, or target config still references the variable. The UI surfaces references.

Validation endpoints

A distribution that returns success from the module layer might still not be live on the endpoint — the module reload returned 0 but systemd was already running a stale copy; a Kubernetes Secret was updated but the ingress controller didn't hot-reload; an F5 profile was created but the virtual server is pinned to the old one. Post-distribution validation closes the gap: after every target completes, the backend opens an outbound TLS handshake to a configured endpoint, hashes the presented cert, and compares it to the one we just deployed. Mismatch → distribution is downgraded to partial.

Configure on the target (applies to every distribution using it) or on the distribution (overrides / supplements). Validation endpoints → Add endpoint.

Field	Required	Purpose
hostname / ip	at least one	Address of the TLS endpoint. Set either or both — when both are given the check dials the ip and presents hostname as the TLS SNI (the ingress / SNI-routed load-balancer case).
port	no	TLS port. 443 for HTTPS, 636 for LDAPS.
sni	no	SNI value. When empty it defaults to the endpoint's own hostname (empty only for an IP-only endpoint). Set explicitly when the same endpoint serves multiple certs keyed by SNI.
method	no	tls_fingerprint (default) or none (skip).
retries	no	Total attempts. Default 3. Reload takes a moment.
retry_delay_ms	no	Pause between attempts, milliseconds. Default 2000.
timeout_ms	no	Per-attempt TLS handshake timeout, milliseconds. Default 5000.

How fingerprint check works: dial the endpoint (ip if set, else hostname) on port over TCP, initiate TLS with the configured SNI, read the server cert from the handshake, compute SHA-256 of its DER bytes, compare to the SHA-256 of the cert just distributed. Match → pass; mismatch or connection error → retry then fail.

SSRF guards apply: link-local and cloud-metadata IPs are blocked; DNS must resolve to a non-blocked target. Allowlist private endpoints at the network-policy layer if needed.

Partial-failure semantics: if validation fails for any endpoint on an otherwise-successful target, the per-target result moves to partial; the distribution aggregate becomes partial if any child is partial. Each endpoint's outcome is recorded in DistTargetResult.ValidationResults.

Fan-out execution

Fan-out is off by default (DistributionFanoutThreshold = 0). Once you set a positive threshold and a distribution aims at a target group at least that large, the backend splits it into batch child jobs, schedules them on a dedicated worker lane, classifies failures, and rolls the results up atomically. Faster wall-clock, bounded blast radius, queue fairness — ACME / MSCA / notification jobs keep running on the main lane while a 500-target SSH fan-out runs on the dist lane.

Tuning settings (Settings → General → Distribution):

Setting	Default	Purpose
DistributionFanoutThreshold	0 (disabled)	Target-count floor at which fan-out engages. 0 turns fan-out off entirely — every distribution runs on the single-job path regardless of size. Set a positive value to enable it.
DistributionFanoutBatchSize	50	Targets per child job (applies only when a positive threshold is reached).
DistributionSSHMaxConcurrency	0 (module default)	Inside a batch, how many SSH sessions run in parallel. 0 uses the SSH module's built-in default (10).

Tune batch size up for fast targets (Kubernetes, webhook), down for slow ones (SSH reload on a busy box). Concurrency should stay well below your fleet's sustainable session cap.

Execution flow

1. Enqueue. The backend chunks targets into BatchSize, enqueues one distribution_execute child per batch with mode=batch, then sets FanOutTotal atomically. If any enqueue fails, the entire fan-out is aborted (no partial enqueue).
2. Schedule. A dedicated distWorker polls only for distribution_execute and distribution_rollback; main-queue workers ignore these.
3. Execute. Each batch child runs the module against every target in its slice, up to SSHMaxConcurrency in parallel.
4. Record. RecordChildResult does an atomic $inc + $addToSet per target into FanOutSucceeded / FanOutFailed / FanOutPartial / FanOutCancelled.
5. Per-target retry. Failed targets classified as network or io_transient are re-enqueued as a retry-mode child with just the retry IDs. Up to a per-target retry cap.
6. Aggregate completion. TryCompleteAggregate runs a MongoDB aggregation pipeline update with $expr + $switch that flips the distribution to success / partial / failed atomically when the children counters add up to FanOutTotal. No race.

Error classification

• network — connect refused, timeout, DNS failure. Retryable.
• io_transient — 5xx, EOF during transfer, temporary deadlock. Retryable.
• io_permanent — 4xx that won't change, target rejected the artifact. Not retryable.
• auth — credential invalid / permission denied. Not retryable.
• validation — post-distribution TLS fingerprint mismatch. Not retryable.

Aggregate status matrix

Children	Distribution status
All Succeeded == Total	success
Succeeded + Partial > 0 with any Failed	partial
Failed == Total	failed
Any Cancelled with others done	cancelled

Observe: the distribution detail page shows progress over FanOutSucceeded + FanOutFailed + FanOutPartial / FanOutTotal. Each batch child appears in Jobs with its own log stream and can be cancelled individually — a fan-out has no parent job to cancel.

Where to manage

• Settings → Distribution → Path sets — create / edit / delete path sets.
• Settings → Distribution → Action sets — create / edit / delete action sets, with a Lint variables button to pre-expand templates against a dummy cert.
• Settings → Distribution → Credentials — module credentials (admin role).
• Settings → Variables — project variables, plain or secret.
• Targets / Distributions — bind path set + action set + credential per target; configure validation endpoints inline.
• Settings → General → Distribution — fan-out thresholds and concurrency caps.

See also

• Distribution overview
• Targets & groups
• SSH · Kubernetes · IIS · WinRM
• F5 BIG-IP · NetScaler · Vault · Huawei · Azure Key Vault · Cloudflare · cPanel · MerlinCDN · Exchange · PAN-OS · Webhook · SMTP
• Approval workflow (gating Execute)