Distribution primitives
The reusable objects every distribution composes from: Path Sets (where files land), Action Sets (what runs after), Module Credentials (auth material), Project Variables (per-environment values), Validation Endpoints (post-deploy TLS fingerprint check), and the Fan-out execution model. Define each once; reuse across dozens of targets and certificates.
Path Sets, Action Sets, and Project Variables are now operator-editable, gated by default through the configuration changes rule of the project's approval policy (Settings → Approvals, all operators by default) — an operator's create/update/delete is filed as an approval request an admin approves. Module Credentials remain admin-only. Admins and owners bypass approval. See Approval workflow and Auth & RBAC.
Path sets
A path set describes where certificate files land on a target. Linux (SSH) path sets carry a POSIX path, an owner (formatted user:group), a mode (octal), and an output format. Windows (WinRM) path sets use drive-letter (C:\…) or UNC (\\server\share\…) paths; owner / mode do not apply (Windows ACLs are managed on the host).
The target_kind field selects the OS. Bind-time validation prevents cross-platform mismatches, so an SSH target cannot bind a Windows path set or vice versa.
Entry schema:
| Field | Required | Purpose |
|---|---|---|
source | yes | Which artifact to place: cert (leaf), chain (intermediates), fullchain, private_key, combined (cert + chain + key), pfx (server-built PKCS#12), jks (server-built Java KeyStore), p7b (PKCS#7 chain bundle: leaf + intermediates, no key). All work on both Linux and Windows; see below. |
path | yes | Absolute target path. Linux: POSIX. Windows: drive-letter or UNC. |
owner | no | user or user:group — letters, digits, ., _, -, at most 32 characters per part (validated on save). Linux only — stripped on Windows. Applied when the file is written and re-checked on every run; drift on an unchanged file is corrected with a job-log warning. |
mode | no | Octal perms, 3–4 digits (e.g. 0644; validated on save). Linux only. Applied when the file is written and re-checked on every run, like owner. |
format | no | pem (default) or der. DER only valid for cert, an unencrypted private_key, and p7b. Rejected for pfx and jks. |
passphrase_variable | pfx, jks: required · private_key: optional | Name of a secret-flagged project variable that supplies the password at distribute time: the PKCS#12 or keystore password, or — on a private_key row — the passphrase the key is written encrypted with. Only the variable name lives on the path set; the value never does. Refused on every other source. A jks password may only use Latin-1 characters (ç, ö, ü are fine; ş, ğ, ı, İ are not — Java could not open the keystore), otherwise the deployment fails with that reason. |
alias | no (jks only) | The keystore entry alias: 1–64 letters, digits, ., _, -. Empty = the certificate's primary domain (which can be a wildcard such as *.example.com). The keystore stores aliases in lower case, as Java's JKS does. |
Example — nginx:
[
{"source": "fullchain", "path": "/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.crt", "owner": "root:root", "mode": "0644"},
{"source": "private_key", "path": "/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.key", "owner": "root:root", "mode": "0600"}
]
Path variables: paths use ${{ NAME }} placeholder substitution — not Go templates. Two kinds of variable are available:
- Reserved certificate-context variables (
CAP_prefix, always present):CAP_PRIMARY_DOMAIN,CAP_DOMAIN_SLUG(lowercased[a-z0-9-], wildcard*.stripped),CAP_CERT_TOKEN(the 8-char stable per-certificate hash), andCAP_DEPLOY_NAME(cap-<token>-<slug>). PreferCAP_DOMAIN_SLUGoverCAP_PRIMARY_DOMAINinside a path — a wildcard*in the raw domain breaks filesystem paths, whereas the slug is path-safe. - Project variables by name. Secret-flagged variables are rejected in paths (a rendered path lands in logs and execution detail, which would leak the secret). Certificate-context vars win over project variables on a name collision; the
CAP_prefix is reserved at project-variable creation.
A Linux path may start with a variable — ${{ SITE_DIR }}/ssl/fullchain.pem with SITE_DIR = /var/www/site — because the check that it is absolute runs on the rendered path, at deploy time and in the dry run (a rendered path that is not absolute fails that target). A path that does not start with a variable must start with / when you save it, and a .. segment is refused either way.
A literal ${{ ... }} is escaped with a preceding backslash (\${{ NOT_A_VAR }}). An unresolved variable name fails the distribution.
PKCS#12 (.pfx) — server-built bundles
For consumers that expect a single PFX file (Windows Certificate Store via Import-PfxCertificate, IIS legacy bindings, modern Tomcat with keystoreType="PKCS12", Spring Boot server.ssl.key-store, .NET Core Kestrel), CertAutoPilot builds the bundle server-side and sends the bytes straight to the target — no openssl there.
On Windows targets the bytes travel on the remote command's standard input, never on its command line, so the key material does not appear in Windows process-creation auditing. The target checks the content it received before the file replaces anything. By default the file is encrypted with AES-256 (PBKDF2-HMAC-SHA256, HMAC-SHA256 MAC). Windows Server 2016 and Windows 10 before 1709 cannot open it: Import-PfxCertificate and certutil answer "The specified network password is not correct." even with the right password. For those hosts set the row's PFX encryption to Legacy ("pfx_encryption": "legacy"): the file is then encrypted with TripleDES-SHA1, which they can open. Legacy is weaker (guessing the password is cheaper), so use a long random password for that variable. A Path Set asking for Legacy is refused with 409 CLUSTER_UPGRADE_REQUIRED while a node older than 1.7.0 is running, because that node would still write AES. The IIS and Exchange modules always use TripleDES for their one-time file, so they need no setting.
Operator workflow:
- Distribution → Variables → Add a variable, e.g.
APP_PFX_PASSPHRASE, mark Secret: yes, set the chosen passphrase. Encrypted at rest with envelope encryption. - Path Sets → Add. Pick the target OS, add a row with
source: pfx, an absolute path, and selectAPP_PFX_PASSPHRASEfrom the passphrase-variable dropdown. - Bind to a target, distribute. The PFX is built once per file; the decrypted passphrase only exists inside the build call's scope.
{
"source": "pfx",
"path": "/opt/tomcat/conf/keystore.p12",
"owner": "tomcat:tomcat",
"mode": "0600",
"passphrase_variable": "APP_PFX_PASSPHRASE",
"pfx_encryption": "modern"
}
pfx_encryption is optional (modern when left out) and applies to pfx rows only; on any other source it is refused with 400.
The encoder injects a fresh IV/salt per encode, so the bytes change every time. The hash-based skip in both SSH and WinRM modules always reports changed for pfx files, and the bound Action Set runs on every redistribute. If you need quiet redistributes, add a thumbprint check inside your Action Set (Windows: (Get-PfxCertificate ...).Thumbprint; Linux: keytool -list -keystore ... -storepass ...) and gate the restart on a stored marker.
Java KeyStore (.jks) — server-built
For Java consumers that still read a JKS keystore (older Tomcat keystoreType="JKS", legacy Java applications), use source: jks. CertAutoPilot builds the keystore server-side with one private-key entry: the private key with the leaf and its intermediates as the certificate chain. The passphrase_variable value is both the keystore password and the entry (key) password. The entry is named by alias, or by the certificate's primary domain when alias is empty.
{
"source": "jks",
"path": "/opt/app/conf/keystore.jks",
"owner": "app:app",
"mode": "0600",
"passphrase_variable": "APP_KEYSTORE_PASS",
"alias": "app"
}
Like a .pfx, a .jks is different on every build (it records its creation time and is sealed with a fresh salt), so it is rewritten on every distribution and the bound Action Set runs each time.
PKCS#7 chain bundle (.p7b)
source: p7b writes a certificates-only PKCS#7 bundle: the leaf followed by its intermediates, without the private key (Windows certificate import wizards, some Java and appliance tools). format: pem (the default) writes a -----BEGIN PKCS7----- file, format: der the binary form. The bundle is built deterministically, so an unchanged certificate gives the same bytes and the file is skipped like any PEM file. Because it holds no key, it is written with the host's default permissions unless you set mode.
Encrypted private key
A private_key row with a passphrase_variable writes the key as a passphrase-protected PKCS#8 PEM (-----BEGIN ENCRYPTED PRIVATE KEY-----, AES-256-CBC with PBKDF2-HMAC-SHA256 — the scheme openssl pkcs8 -topk8 -v2 aes256 produces), for servers that are configured to read an encrypted key (nginx ssl_password_file, Apache SSLPassPhraseDialog, Java and .NET loaders). Without a passphrase_variable the key is written unencrypted, exactly as before. An encrypted key is PEM only (format: der is refused). Its salt is random, so it is rewritten on every distribution like a .pfx.
Action sets
An action set is the list of commands the SSH or WinRM module runs on a target after file placement. Linux action sets run shell commands or a script body with sudo (run as), allowlisting, and timeouts. Windows action sets run PowerShell commands or an inline script via powershell.exe / pwsh.exe; run as and command_mode do not apply on Windows. target_kind selects the OS; bind-time validation enforces the match.
Command mode (simple)
mode: command
commands:
- "nginx -t"
- "systemctl reload nginx"
run_as: "root"
timeout_seconds: 30
Each command runs sequentially; non-zero on any aborts the rest and marks the target failed.
Inline-script mode (complex)
mode: script_inline
script_body: |
#!/bin/bash
set -euo pipefail
nginx -t
systemctl reload nginx
SUBJECT=$(echo | openssl s_client -connect localhost:443 -servername "${{ CAP_PRIMARY_DOMAIN }}" 2>/dev/null | openssl x509 -subject -noout)
echo "$SUBJECT" | grep -q "${{ CAP_PRIMARY_DOMAIN }}" || { echo "Active cert mismatch" >&2; exit 1; }
shell: "/bin/bash"
run_as: "root"
timeout_seconds: 60
Field reference
| Field | Required | Purpose |
|---|---|---|
mode | yes | command or script_inline — the same two values on Linux and Windows. Anything else is rejected at creation with mode must be command|script_inline. |
commands | commands mode | List of shell strings. |
script_body | script mode | Full script text. Template-expanded before execution. |
shell | no | Shell to invoke, validated on save. Linux: an absolute path to sh, bash, dash, zsh, ksh, mksh, ash or fish (default /bin/bash); Windows: powershell.exe (default) or pwsh.exe. |
command_mode | no (linux only) | exec (default) runs the command directly; shell wraps it in /bin/sh -c. Each command gets its own SSH session either way — there is no batching mode, and unrecognised values are silently treated as exec. |
run_as | no (linux only) | User to execute as — the command is wrapped in sudo -u <user>, without -n. A sudoers rule that prompts for a password therefore blocks until the timeout instead of failing fast; use NOPASSWD entries. Validated on save against ^[a-z_][a-z0-9_-]{0,31}$. |
allowed_commands | no | Regex allowlist, enforced in command mode only — an inline script body is uploaded and executed with no pattern check at all. The regex is matched against the whole command string and is not anchored, so a pattern like systemctl reload nginx also admits rm -rf /var/lib/x && systemctl reload nginx; anchor your patterns (^systemctl reload nginx$). An empty allowlist permits every command. |
timeout_seconds | no | Per-command timeout, 0–3600 (validated on save). No default — leave it unset (or 0) and there is no per-command bound at all; only the per-target timeout (300 s) applies. Set it explicitly if a command can hang. |
run_always | no | Default false. By default the action runs when a deployed file actually changes (hash-based skip) or when the action has not yet completed successfully for the certificate version currently on disk — a crash-safety net, so a run that wrote the files but died before the reload re-runs the reload next time even though the files are byte-identical. A genuine no-op still skips. Set true to run on every distribution even when nothing changed — for verification/diagnostic actions that must run each time (e.g. printing the deployed certificate). When it runs despite no change the log shows no file changed, running anyway (run_always enabled). |
Variables in actions: action-set commands and script bodies use the same ${{ NAME }} substitution as paths (not Go templates — there is no .cn/.paths template context). Available names are project variables (referenced by bare uppercase name) plus the four reserved certificate-context variables ${{ CAP_PRIMARY_DOMAIN }}, ${{ CAP_DOMAIN_SLUG }}, ${{ CAP_CERT_TOKEN }}, and ${{ CAP_DEPLOY_NAME }}.
Privilege escalation on Linux wraps the command in sudo -u <user> — without -n, so sudo is free to prompt. A rule that asks for a password will therefore hang the session until the timeout rather than failing immediately. Give the target passwordless rules for the exact commands:
# /etc/sudoers.d/certautopilot
Cmnd_Alias CAP_RELOAD = /usr/sbin/nginx -t, /bin/systemctl reload nginx
capdeploy ALL=(ALL) NOPASSWD: CAP_RELOAD
Module credentials
A single store for every piece of auth material distribution modules need: SSH private keys, kubeconfigs, WinRM username/password pairs, F5 / NetScaler admin passwords, Vault tokens or AppRole pairs, Huawei AK/SK pairs, Azure service principals. All credentials live in module_credentials and are referenced by targets via immutable ID. A credential either holds its value here, sealed with envelope encryption, or references one in your own secret manager — see External secret stores.
| Type | Fields | Used by |
|---|---|---|
| SSH private key | PEM key + optional passphrase | SSH module |
| SSH password | Username + password | SSH module |
| Kubeconfig | Inline kubeconfig YAML | Kubernetes module |
IIS WinRM (iis_winrm) | {"username","password"} — the auth method (ntlm/basic/kerberos + realm/KDC) is chosen on the target; legacy blobs carrying auth_type/domain/kdc_host are still honored when the target sets none | IIS module only |
WinRM password (winrm_password) | {"username","password"} | Generic WinRM module only |
F5 BIG-IP password (f5bigip_password) | {"username","password"} | F5 BIG-IP only |
NetScaler password (netscaler_password) | {"username","password"} | NetScaler only |
| Vault token / AppRole | Token, or role_id + secret_id | HashiCorp Vault |
| Webhook HMAC / bearer | HMAC signing secret, or bearer token | Webhook module |
| SMTP password / PKCS#12 passphrase | Relay password; optional P12 export passphrase | SMTP module |
| AWS AK/SK | Access key + secret key | AWS ACM |
| Huawei AK/SK | Access key + secret key | Huawei Cloud |
| Cloudflare API Token | API token (Zone → SSL and Certificates → Edit) | Cloudflare |
| cPanel / WHM API Token | Username + API token (SSL install / WHM access) | cPanel / WHM |
| Azure Service Principal | tenant_id + client_id + client_secret (JSON) | Azure Key Vault |
| MerlinCDN token | Personal access token + org/workspace IDs | MerlinCDN |
| Exchange WinRM | Username + password (auth type + domain on the target) | Microsoft Exchange |
| PAN-OS credentials | JSON {"username","password"} or {"username","api_key"} — username is always required (the partial commit is scoped to it) | PAN-OS |
| FortiWeb credentials | JSON {"username","password"} — no API token exists; least privilege: sysgrp read-write + traroutegrp read | FortiWeb |
| FortiGate API token | JSON {"api_token"} — the REST API administrator's token (16–128 letters and digits), sent as Authorization: Bearer; least privilege: vpngrp read-write, plus fwgrp address / others read-write only for VIP / SSL inspection profile placements | FortiGate |
A10 credentials (a10_password) | JSON {"username","password"} — an ACOS administrator with aXAPI access; least privilege: privilege write (shared) or privilege partition-write <p> | A10 Thunder |
| Elchi credentials | JSON {"username","password"} of a dedicated service account (Admin recommended, or Editor; member of a group of the project; no OTP) | Elchi |
| PAM360 API Token | AUTHTOKEN of a PAM360 API user | Secret stores (store login — not used by targets directly) |
Each credential type is admitted for exactly the target types listed: a credential created for IIS cannot be bound to a generic WinRM target, and an F5 credential cannot be bound to a NetScaler target, even though the JSON shapes are identical.
Why separate from targets: one SSH key opens ten hosts (one credential → ten targets, one rotation point); credentials only surface to admins (viewers see target lists without the auth material); credential reads and rotations get a clean audit trail.
Rotate by editing the credential and pasting new material — every future distribution picks it up. There is no version history; in-flight jobs holding a cloned client finish with the old material. Wait for in-flight to drain before considering the old material fully revoked.
Or don't store it here at all. A credential can instead reference a secret in your own secret manager (HashiCorp Vault / OpenBao, ManageEngine PAM360), in which case CertAutoPilot keeps no value — only the reference — and reads it from the store every time it is needed. Rotating on your side then takes effect immediately, with nothing to update here. See External secret stores.
Which hosts a credential may be sent to
A stored secret is never shown again, but it is sent to whatever its target points at. So every credential is bound to the hosts it is used with (its Allowed hosts, shown on the credential's edit page):
- Pairing a credential with a host it is not bound to yet — a new target, a target's host edited, a target group's default credential, a module's per-target credential override, a secret store's address, or a live picker on an unsaved connection — adds that host when it is done by the person who entered the credential's secret, or by an organization owner. Nothing changes for the usual case where the admin who creates a credential also adds its targets.
- Anyone else is refused with 422
CREDENTIAL_HOST_NOT_ALLOWED: they could otherwise point the credential at a server of their own and read the secret it receives. To use it with a new host they enter the credential's secret again (edit the credential) — then they know it anyway — or ask the person who entered it or an organization owner. An approved change counts the approving admin too. - The list is checked again right before every use (deploy, dry run, health check, picker), whatever wrote the pairing.
- Removing a host from Allowed hosts is open to anyone who may edit the credential; adding one by hand follows the rule above.
- The credential's edit page shows Secret entered by (the username), and a refusal names that person. The target form warns before you save when the chosen credential is not bound to the target's host — and says whether saving will add it (you entered its secret, or you are an owner) or be refused.
- A credential's address-like fields inside the secret — the Kerberos KDC or domain of a WinRM-type credential — move the same way: without the password entered again, only the person who entered it or an owner (422
CREDENTIAL_SECRET_REQUIRED). - The address is everything the credential is sent through: the host (or, for SSH, the
ipthe module dials when set), a Kerberos KDC and — when no KDC is named — the realm (domain) it is found through, an Exchange server list, a Key Vault URL, a secret store's address. Weakening a bound target's certificate check follows the same rule, since it hands the secret to whoever sits on the path: 422CREDENTIAL_TLS_CHANGE_NOT_ALLOWEDfor anyone but the person who entered the secret or an owner. A weakening is any of: verification turned off or the connection made plain; chain verification (ssl_verify) swapped for a pinned fingerprint; aca_certadded beside the system roots while the chain is verified (a new trust anchor); aca_certor a pinned fingerprint replaced or removed. Turning verification on is always allowed, and so is adding a pin while verification stays on, or adding a pin orca_certto a target that verified nothing. - A credential that reads from a secret store has no secret to re-enter: pointing it at another path of the store is also for the person who set it up or an owner (422
CREDENTIAL_REF_CHANGE_NOT_ALLOWED), and never counts as entering the secret. - A new target that does not verify the server's certificate (verification off, TLS off, plaintext) is judged like turning verification off on an existing one: on a host the credential is bound to, only the person who entered the secret or an owner may save it (422
CREDENTIAL_TLS_CHANGE_NOT_ALLOWED); for IIS and Exchange an unsetuse_tlscounts as on. A secret store's own TLS switch and pinned CA follow the same rule for its auth credential. - When someone else enters the secret, the hosts a previous setter — other than the credential's creator — had paired with it are removed, and the update's answer lists them (
removed_hosts). A stored junk value plus a host of one's own must not receive the real secret the moment its owner enters it again. Hosts the creator paired stay. - A credential that reads from a secret store is bound by the person who set the store up (or an owner): nobody enters its value, and a new credential pointing at an existing path of the store must not become a way to send that value elsewhere.
- An F5
host, NetScalernsipor Kubernetesapi_server_overridemust be a plain host name or IP address with an optional numeric port — a value such asfw.corp:443@evil.example(which a URL parser would dial asevil.example) is refused at save, and the binding never reduces such a value to its host-looking part. - Credentials from before 1.7.0 are bound once, at start-up, to the hosts their targets, groups, module overrides and secret stores use; until then they are not restricted. The pass repeats every hour, so a credential an older node writes during a rolling upgrade is bound without a restart.
DNS credentials and AD CS connections hold their address and secret in one record: changing the nameserver / API URL / WinRM host / TLS checks of a DNS credential, or the CEP/CES endpoints, any RPC setting, the authentication or the user name of an RPC connection, or the TLS settings of an AD CS connection, needs the secret in the same request — unless the person who entered it or an owner makes the change (422 DNS_CREDENTIAL_SECRET_REQUIRED / MSCA_CONNECTION_CREDENTIAL_REQUIRED).
Delete is blocked if any target still references the credential, or if a secret store authenticates with it. The UI surfaces blockers; repoint or delete those first.
Per-credential allowlists exist for sensitive modules (e.g. Kubernetes allowed namespaces, allowed name prefix) — use them to prevent a single-tenant credential from being aimed at the wrong namespace by mistake.
Project variables
Key-value pairs scoped to a project, referenced from path sets and action sets through ${{ NAME }} placeholder substitution (not Go templates). Use them to keep per-environment differences (hostnames, ports, service names) out of reusable templates.
Plain vs sensitive:
- Plain — readable in the UI by any project member (viewer and up). Use for hostnames, port numbers, service names, feature flags.
- Sensitive — value is write-only after creation. List views show
<sensitive>. Create, update and delete are audited; worker reads at distribution time are not, so correlate a variable's use with the distribution job log rather than the audit trail. Use for shared tokens, secondary API keys, PKCS#12 passphrases (referenced viapassphrase_variableon a pfx path-set row).
Schema:
name— uppercase identifier, regex^[A-Z_][A-Z0-9_]{0,127}$. Names are trimmed and uppercased on save.value— string.is_secret— boolean.description— free-text.
Reference by bare uppercase name inside ${{ }}:
# Path set entry
{
"source": "fullchain",
"path": "/etc/nginx/ssl/${{ SERVICE_NAME }}.crt"
}
# Action set command
commands:
- "rsync /etc/nginx/ssl/ ${{ BASTION_HOST }}:/mnt/backup/"
Only path sets and action sets support project-variable substitution. Webhook headers and body do not — the webhook body template exposes certificate fields only, not project variables.
Pattern: one project per environment. The same path-set + action-set combination references the same variable names; the project switches what the names resolve to. production, staging, dr.
If a sensitive value lands on a target host in plaintext (written to a file, echoed into a script), the redaction pipeline stops at that boundary. Don't put irreversible-exposure secrets into action sets that echo them.
Delete is not blocked by references — nothing tracks which path sets or action sets name a variable. A path set or action set that still references a deleted variable fails the next distribution with an unresolved-variable error, so check your path sets and action sets (the action-set Lint variables button helps) before deleting.
Validation endpoints
A distribution that returns success from the module layer might still not be live on the endpoint — the module reload returned 0 but systemd was already running a stale copy; a Kubernetes Secret was updated but the ingress controller didn't hot-reload; an F5 profile was created but the virtual server is pinned to the old one. Post-distribution validation closes the gap: after every target completes, the backend opens an outbound TLS handshake to a configured endpoint, hashes the presented cert, and compares it to the one we just deployed. Mismatch → distribution is downgraded to partial.
Configure on the target (applies to every distribution using it) or on the distribution (overrides / supplements). Validation endpoints → Add endpoint.
| Field | Required | Purpose |
|---|---|---|
hostname / ip | at least one | Address of the TLS endpoint. Set either or both — when both are given the check dials the ip and presents hostname as the TLS SNI (the ingress / SNI-routed load-balancer case). |
port | no | TLS port. 443 for HTTPS, 636 for LDAPS. |
sni | no | SNI value. When empty it defaults to the endpoint's own hostname (empty only for an IP-only endpoint). Set explicitly when the same endpoint serves multiple certs keyed by SNI. |
method | no | tls_fingerprint (default) or none (skip). |
retries | no | Total attempts. Default 3. Reload takes a moment. |
retry_delay_ms | no | Pause between attempts, milliseconds. Default 2000. |
timeout_ms | no | Per-attempt TLS handshake timeout, milliseconds. Default 5000. |
How fingerprint check works: dial the endpoint (ip if set, else hostname) on port over TCP, initiate TLS with the configured SNI, read the server cert from the handshake, compute SHA-256 of its DER bytes, compare to the SHA-256 of the cert just distributed. Match → pass; mismatch or connection error → retry then fail.
SSRF guards apply: link-local and cloud-metadata IPs are blocked; DNS must resolve to a non-blocked target. Allowlist private endpoints at the network-policy layer if needed.
Partial-failure semantics: if validation fails for any endpoint on an otherwise-successful target, the per-target result moves to partial. When every target deployed and only validation failed, a normal run (not a fan-out batch, not a rollback) is first recorded as unverified and re-checked once about five minutes later — passing makes it success, failing again makes it partial (see Post-deploy validation); the fan-out aggregate becomes partial if any child is partial. Each endpoint's outcome is recorded in DistTargetResult.ValidationResults.
Fan-out execution
Fan-out is off by default (DistributionFanoutThreshold = 0). Once you set a positive threshold and a distribution aims at a target group at least that large, the backend splits it into batch child jobs, schedules them on a dedicated worker lane, classifies failures, and rolls the results up atomically. Faster wall-clock, bounded blast radius, queue fairness — ACME / MSCA / notification jobs keep running on the main lane while a 500-target SSH fan-out runs on the dist lane.
Tuning settings (Settings → General → Distribution):
| Setting | Default | Purpose |
|---|---|---|
DistributionFanoutThreshold | 0 (disabled) | Target-count floor at which fan-out engages. 0 turns fan-out off entirely — every distribution runs on the single-job path regardless of size. Set a positive value to enable it. |
DistributionFanoutBatchSize | 50 | Targets per child job (applies only when a positive threshold is reached). |
DistributionSSHMaxConcurrency | 0 (module default) | Inside a batch, how many SSH sessions run in parallel. 0 uses the SSH module's built-in default (10). |
Tune batch size up for fast targets (Kubernetes, webhook), down for slow ones (SSH reload on a busy box). Concurrency should stay well below your fleet's sustainable session cap.
Execution flow
- Enqueue. The backend chunks targets into
BatchSize, enqueues onedistribution_executechild per batch withmode=batch, then setsFanOutTotalatomically. If any enqueue fails, the entire fan-out is aborted (no partial enqueue). - Schedule. A dedicated
distWorkerpolls only fordistribution_executeanddistribution_rollback; main-queue workers ignore these. - Execute. Each batch child runs the module against every target in its slice, up to
SSHMaxConcurrencyin parallel. - Record.
RecordChildResultdoes an atomic$inc(one per child job, with an$addToSetduplicate guard on the child job id) intoFanOutSucceeded/FanOutFailed/FanOutPartial/FanOutCancelled. - Per-target retry. Failed targets classified as
networkorio_transientare re-enqueued as aretry-mode child with just the retry IDs. At most 3 retry waves per distribution. - Aggregate completion.
TryCompleteAggregateruns a MongoDB aggregation pipeline update with$expr+$switchthat flips the distribution tosuccess/partial/failed/cancelledatomically when the children counters add up toFanOutTotaland no retry child is outstanding. No race.
Error classification
network— connect refused, timeout, DNS failure. Retryable.io_transient— broken pipe, connection reset, transient SFTP/transfer errors, module-specific transient codes (e.g. 5xx, rate limits). Retryable.io_permanent— remote permission denied, no space left, target rejected the artifact. Not retryable.auth— credential invalid / permission denied. Not retryable.validation— configuration or input errors: cert/key mismatch, invalid path, missing target or variable. Not retryable. (A post-distribution TLS fingerprint mismatch is not an error class — it downgrades the target topartial, see Validation endpoints.)
Aggregate status matrix
| Children | Distribution status |
|---|---|
every child cancelled (Cancelled == Total) | cancelled |
no child succeeded (Succeeded == 0) | failed |
zero Failed, zero Partial, zero Cancelled | success |
| anything else | partial |
Observe: the distribution detail page shows progress over FanOutSucceeded + FanOutFailed + FanOutPartial / FanOutTotal. Each batch child appears in Jobs with its own log stream and can be cancelled individually — a fan-out has no parent job to cancel.
Where to manage
- Settings → Distribution → Path sets — create / edit / delete path sets.
- Settings → Distribution → Action sets — create / edit / delete action sets, with a Lint variables button to pre-expand templates against a dummy cert.
- Settings → Distribution → Credentials — module credentials (admin role).
- Distribution → Variables — project variables, plain or secret.
- Targets / Distributions — bind path set + action set + credential per target; configure validation endpoints inline.
- Settings → General → Distribution — fan-out thresholds and concurrency caps.
See also
- Distribution overview
- Targets & groups
- SSH · Kubernetes · IIS · WinRM
- F5 BIG-IP · NetScaler · Vault · Huawei · Azure Key Vault · Cloudflare · cPanel · MerlinCDN · Exchange · PAN-OS · FortiWeb · FortiGate · A10 Thunder · Elchi · Webhook · SMTP
- Approval workflow (gating Execute)
Video tutorials
Short, hands-on videos showing this feature working for real. English audio, English and Turkish subtitles. All video tutorials →