SSH module
The SSH module copies the certificate + private key + chain to a target host using SCP/SFTP, then runs post-deployment actions (typically systemctl reload nginx or equivalent). It is the most-used distribution module. Rollback re-deploys a previous retained certificate version through the same deploy path.
Overview
- Transport: SSH v2 (SFTP for writes).
- Auth: an unencrypted PEM SSH private key, or a password. The module decides which by trying to parse the credential as a key and falling back to password authentication — so a passphrase-protected key is not supported: it fails to parse and is then offered as a password, which the host rejects. Decrypt the key before storing it (CertAutoPilot envelope-encrypts it at rest). TLS client certificates are not an SSH auth method and are not supported.
- Path set: defines where each file goes (cert, key, chain) with ownership + mode.
- Action set: shell commands run after writes, typically to reload the consuming service.
- Rollback: supported — a rollback re-writes the previous version's cert/key files at the PathSet paths and re-runs the ActionSet (Rollback).
- Concurrency: the module runs up to 10 SSH targets of one distribution in parallel by default.
DistributionSSHMaxConcurrency(Settings → General) replaces that default for every SSH distribution when set (1–50;0leaves the module default) — it can raise the parallelism as well as lower it. Turning off Parallel execution (Settings → General) forces sequential execution for every module, SSH included — it wins overDistributionSSHMaxConcurrency, which is not consulted while the switch is off.
Prerequisites
Minimum access
CertAutoPilot signs in as one ordinary Linux user. File writes, chmod and chown run as that user, never through sudo. Give it exactly this:
- SSH login with public key or password (keyboard-interactive alone is not enough), and SFTP.
- Write and search (
w+x) on every directory a path set writes to. The simplest way is to make the user the owner of the certificate directory. - Only for a path-set row with
owner: log in as root (or a user withCAP_CHOWN).chownto another user needs root and runs withoutsudo; without it the file is still written and the failedchownis a warning. - Only for an action set with
run_as: aNOPASSWDsudoers rule for exactly the commands it runs, as that user (no TTY, no password is sent). Ascript_inlineaction set runssudo -u <run_as> <shell> /tmp/certautopilot.XXXXXX, and its 0700 script is readable only by the SSH user, so thererun_asmust be root. - Only for an action set or a path-set
owner: an account that may run commands (not limited toForceCommand internal-sftp), plus whatever the commands themselves need.
Not needed: root login, a sudo rule for file writes, or ALL=(ALL) sudo. An SFTP-only account is enough when no row sets an owner and no action set is bound.
# As root on the host: a dedicated user that owns the certificate directory
useradd --create-home --shell /bin/bash cap-deploy
install -d -m 700 -o cap-deploy -g cap-deploy /home/cap-deploy/.ssh
install -m 600 -o cap-deploy -g cap-deploy cap_deploy.pub /home/cap-deploy/.ssh/authorized_keys
install -d -m 755 -o cap-deploy -g cap-deploy /etc/nginx/ssl
# Only if the action set uses run_as root (command mode)
echo 'cap-deploy ALL=(root) NOPASSWD: /usr/sbin/nginx -t, /usr/bin/systemctl reload nginx' > /etc/sudoers.d/certautopilot
chmod 440 /etc/sudoers.d/certautopilot
visudo -cf /etc/sudoers.d/certautopilot
Before you start
- An SSH module credential (key or password) saved under Settings → Distribution → Credentials.
- Target hosts reachable from the backend. The backend's pod or VM must have SSH egress to every target.
- sshd with the SFTP subsystem enabled.
- A path set and an optional action set.
Create an SSH target
- Settings → Distribution → Targets → New.
- Module type: SSH.
- Hostname + port (default
22). - Username: the account on the remote host.
- Credential: pick the SSH key credential you created.
- Save → the health check opens an SSH connection (and, in the detailed check, an SFTP session) and closes it. No command is run on the host.
Host-key verification
By default the module accepts whatever host key the remote presents — anything answering on the configured address is treated as the target, and the private key is written to it. Turn on Require approval when the host key changes on the target to change that.
With it on:
- The first key seen during a deployment is remembered silently. You do nothing, and nothing is blocked — enabling it on a target that already has deployments is safe. Dry-runs and health checks never establish the pin.
- Every later connection is checked against that key. A match proceeds as before.
- A mismatch stops the deployment with
SSH_HOST_KEY_CHANGED, records the new key, and raises a review prompt on the target. Nothing is written to the host. Dry-runs and health checks are checked against the same pin and will report the target as unreachable, but only a deployment records the key and raises the prompt. - If the original host comes back and matches the pin again, the prompt clears itself. A single transient mismatch does not leave you with a review you can only resolve by approving.
- A blocked target does not drag its neighbours down: because nothing was written to it, a host-key failure alone never triggers auto-rollback, so the hosts that did receive the certificate keep it.
- Approving queues the blocked distributions for redistribution, so the deployment resumes on its own — the confirmation tells you how many will re-run. Distributions the renewal sweep would skip anyway are not counted: one left partial (some targets already had the certificate), one whose module config has automatic distribution turned off, or one that was rolled back — re-run those from the certificate's Distributions tab.
CertAutoPilot cannot tell whether the first key it sees is the right one — that is what trust-on-first-use means. If something is impersonating the host at that moment, the impostor's key is what gets remembered, and connections to the genuine host are then rejected instead. Where host identity really matters, compare the pinned fingerprint against the host itself after the first deployment:
ssh-keyscan -p <port> <host> | ssh-keygen -lf -
Use ssh-keyscan rather than reading a single key file: a host normally offers several key types (Ed25519, ECDSA, RSA) and CertAutoPilot pins whichever one the connection negotiated, so ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub may well print a fingerprint that was never used. Once a key is pinned, later connections offer that key type first, so a host that keeps its key does not start failing because it also offers others. The other types stay available as fallback — a host that legitimately dropped the pinned type still connects far enough to be reported as a reviewable change, rather than failing negotiation with nothing to approve.
Approving is an operator decision, not a machine one: CertAutoPilot cannot tell a rebuilt host from an impostor. The review dialog gives you what makes the call possible — how long the old key had been trusted and how many deployments it served, the new fingerprint in the same form ssh-keygen -lf prints, and a command to compare it against the host itself. The approval is bound to the fingerprint you were shown: if the key changes again between opening the dialog and clicking approve, the request is refused rather than silently blessing a key you never reviewed. Approval is deliberately not routed through the configuration-approval workflow, so a rebuilt server does not hold up a renewal waiting for a second person; the decision is written to the audit log with both fingerprints instead.
Turning the setting back off leaves the remembered key in place, so re-enabling it later does not blindly trust whatever answers next.
The Host key column on the targets list shows what state a target is in — Awaiting first key before anything is pinned, Pinned once one is, Changed — review when a decision is waiting. Click Pinned or Changed — review to see the fingerprint, how long it has been trusted and how many deployments it has served; that is how you compare it against the host after a first deployment. (Awaiting first key has nothing recorded yet, so it is not clickable.)
Host certificates. If your servers present an OpenSSH host certificate, CertAutoPilot records the key inside it rather than the certificate. Certificates are short-lived by design, so pinning one would report a changed host key on every routine renewal. Note this is not CA-based verification — the certificate's signature is not checked against a trusted CA; only the underlying key is remembered.
A load balancer, HA pair or round-robin DNS name presents a different host key depending on which machine answers, so verification would fail at random. Leave the setting off for those targets and rely on network-layer control instead.
Nothing changes from previous releases, and nothing warns you — which is the point of it being opt-in. Restrict SSH egress from the CertAutoPilot host to the intended targets (management VLAN, firewall allowlist, bastion) and treat DNS and ARP integrity on that path as part of your threat model, because a host-key change will not be detected.
Path set
A path set defines the files you want CertAutoPilot to write, with ownership and mode. Example for nginx:
/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.crt root:root 0644 <cert+chain>
/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.key root:root 0600 <private key>
/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.ca root:root 0644 <chain only>
owner (user or user:group — letters, digits, ., _, -; at most 32 characters per part) and mode (3–4 octal digits) are validated when the path set is saved. They are applied when a file is written and re-checked on every later run: if the mode or owner of an unchanged file drifted on the host, it is put back and a warning line is written to the job log.
Files that carry a private key default to 0600. For the private_key, combined, pfx and jks sources an omitted mode means 0600, not the server's umask (which is 0644 on most hosts and used to leave the key world-readable until someone noticed). Other sources without a mode keep the umask default as before. Set mode explicitly when the key must be group-readable, e.g. 0640.
Writes are atomic where the server allows it, and never expose the bytes early. Each file is written to a sibling <path>.captmp that is restricted to the target mode while still empty, filled, then renamed over the destination — so a reader sees the old file or the new one, never a truncated one, and a connection that drops mid-transfer leaves the existing file intact. If the destination is a symlink, the file it points to is what gets replaced, as before. A stray .captmp next to a path is the trace of a failed run and is safe to delete.
Two consequences of the rename are worth knowing. It needs write permission on the directory, and it replaces the file's inode — the new file is owned by the SSH user (unless owner is set) and does not keep ACLs, extended attributes or hard links of the old one. When the directory is not writable (the common hardening: a root-owned 0755 directory holding a pre-created key file chowned to the SSH user), or the server cannot rename over an existing file (non-OpenSSH servers without posix-rename), the module falls back to the previous behaviour — it truncates and rewrites the file in place, keeping inode and owner — and says so with a warning in the job log. That path has a moment where a reader can see an empty or partial file; grant write on the directory if you need the atomic replace. The module never deletes the live file to make room for a rename.
Existing files are re-checked. On a run where a file's content is unchanged, its mode is compared with the effective mode and corrected with a warning. Because private-key sources now default to 0600, a key that was written under an older version without an explicit mode (typically 0644) is tightened to 0600 on the next run even though nothing else changed, and the ActionSet does not run. A service that runs as a different user and could read the 0644 key keeps working until its next restart. If that key must stay group- or world-readable, set mode explicitly on the path set.
A refused chmod (for example a chrooted internal-sftp that blocks setstat) fails the file only for private-key sources; for other sources it stays the warning it always was.
Paths use ${{ NAME }} placeholder substitution — not Go templates. CAP_DOMAIN_SLUG is a reserved certificate-context variable (the primary domain lowercased to [a-z0-9-], wildcard *. stripped so it stays path-safe); the other reserved vars are CAP_PRIMARY_DOMAIN, CAP_CERT_TOKEN, and CAP_DEPLOY_NAME. You can reference project variables (Variables) the same way, but secret-flagged variables are rejected in paths. Typical practice is one path set per service template (nginx, haproxy, apache, postgres, dovecot, etc.) and reuse it across many targets.
Action set
Actions run over SSH in sequence after every file has been written. Example:
sudo nginx -t
sudo systemctl reload nginx
Use nginx -t-style preflight to fail fast before reloading. A non-zero exit status marks the target as failed and triggers auto-rollback if it's enabled for the distribution.
By default the action runs only when a written file actually changes (hash-based skip), so reloads stay idempotent on no-op redistributes. Set run_always on the action set to run it on every distribution even when nothing changed — for verification/diagnostic actions that must run each time. There is a third trigger: CertAutoPilot records, per certificate and target, the version for which the action last completed; if that marker is missing or belongs to another version (a previous run wrote the files but crashed before the reload), the action runs once even though the files already match.
run_as (^[a-z_][a-z0-9_-]{0,31}$), shell (an absolute path to sh, bash, dash, zsh, ksh, mksh, ash or fish; default /bin/bash) and timeout_seconds (0–3600) are validated when the action set is saved.
Execute a distribution
- On the cert detail page → Distribution tab → Add distribution.
- Pick the SSH module and a target or target group. The path set and action set come from the target (Settings → Distribution → Targets); a certificate can swap them per target from the distribution's Overrides drawer.
- Optionally enable validation endpoints so we confirm the cert is live post-deploy (see Validation).
- Click Dry-run first. It returns a plan, not an exact manifest: files whose hash already matches are omitted, and key-bearing rows (
private_key,combined,pfx,jks) report "would upload if changed" because the key is not loaded during a dry run. Review it as intent, then execute. - Execute. Progress shows per target; a green tick means files written + action set returned 0. Red means failure — open the target's log to see whether it was transport (
SSH_CONNECT, classnetwork— including a handshake the peer dropped and a host that did not finish within the target's time budget), rejected credentials (SSH_AUTH/CREDENTIAL_MISSING,auth), a transient file-transfer problem (SFTP_FAILED/FILE_UPLOAD,io_transient), a refusal by the host (SSH_PERMISSION,io_permanent— permission denied, missing directory, read-only filesystem; never retried), or a non-zero post-deploy action (SCRIPT_EXIT_NONZERO,io_transient— retried, not a validation failure). Deterministic pre-flight problems — a missing SSH spec, a path set or action set that no longer resolves, a bad path or variable — come back asSSH_VALIDATIONand are never retried.
Rollback
Supported via previous-version re-deploy: a rollback re-writes the previous retained certificate version's cert/key files at the PathSet paths and re-runs the ActionSet — the module's normal deploy path, fed older material. Pick the version in the rollback version picker; eligibility rules, auto-rollback, and the sticky rolled_back status are covered in Rollback.
The module no longer writes .bak backup copies before overwriting — rollback comes from CertAutoPilot's own version history, not from files on the host. Existing .bak files on hosts are inert leftovers from older versions and are safe to delete manually.
Troubleshooting
SSH_AUTH although the key works in a terminal
SSH_AUTH means the host rejected the credentials during the handshake — it is not a file-permission problem (those are SSH_PERMISSION, see below). The usual cause is a passphrase-protected key: it cannot be parsed, so it is offered as a password and refused. Remove the passphrase (ssh-keygen -p -f key -N "") and save the credential again. Otherwise check the target user, that ~/.ssh/authorized_keys holds the right public key, and — for password credentials — that sshd allows the password method (keyboard-interactive is never offered). A handshake that fails for a transport reason (the peer closes the connection, connection reset) is reported as SSH_CONNECT and retried.
SSH_PERMISSION — the host refused a file operation
Creating or writing a path-set file was refused: permission denied on the file or its directory, a missing parent directory the SSH user could not create, a path that is not a directory, or a read-only filesystem. It is not retried — give the SSH user write access to the destination directories (file writes never go through sudo), create the directory, or fix the path. Passwordless sudo is only needed for the action-set commands.
A deploy was interrupted and the service will not start
Normally this cannot happen: files are staged to a sibling <path>.captmp and renamed into place, so an interrupted transfer (per-target timeout, job cancel, worker restart) leaves the previous file intact and at most a stray .captmp next to it. The exception is the in-place fallback — taken, with a warning in the job log, when the SSH user cannot write the directory or the server cannot rename over an existing file (see Path set above). On that path the write truncates the existing file first, and an interruption can leave it truncated or empty; re-run the distribution to repair it. Putting a config test (nginx -t, haproxy -c) as the first ActionSet command catches it on the next run, and granting write on the directory removes the fallback altogether.
(The WinRM module always stages to a sibling file and renames it into place only after a clean decode.)
SSH_HOST_KEY_ALGO
Host-key negotiation failed before any key was presented: the server offers only key types CertAutoPilot does not advertise once verification is on — in practice a DSA-only server. There is nothing to approve; add a modern host key to the server, or turn verification off for that target.
SSH_HOST_KEY_CHANGED
The server presented a host key that does not match the one remembered for this target, so nothing was deployed. Open Settings → Distribution → Targets; the target carries a Changed — review tag and the page banner links straight to it. Compare the new fingerprint against the host — ssh-keyscan -p <port> <host> | ssh-keygen -lf -, run from a machine you trust — before approving. Do not read a single key file: CertAutoPilot pins whichever key type the connection negotiated, so a specific .pub file may print a fingerprint that was never used.
This error is never retried — retrying would just re-present the same key — so the deployment stays failed until you decide.
If the target does not have host-key verification enabled, a rebuilt VM (or an impostor on the same address) is accepted silently and there is nothing to review; see Host-key verification.
Action set times out
Long-running reload (systemd's default is 90 s). Raise timeout_seconds on the action set itself (it is a field on the library resource, not on the distribution) — and note that leaving it unset means no per-command bound at all, only the 300 s per-target ceiling; the action will be killed and the target marked failed if it exceeds.
See also
Multi-certificate hosts
Two mechanisms let one SSH target serve many certificates:
- Per-distribution PathSet/ActionSet — each certificate's distribution can select its own PathSet (where files land) and ActionSet (what runs afterwards) from the certificate's Distributions → Overrides drawer (
ssh { path_set_id, action_set_id }, with a*default row). A selected override must resolve — a deleted or wrong-kind resource fails the run instead of silently using the target's own libraries. - Certificate-context variables in paths — PathSet paths may contain
${{ CAP_PRIMARY_DOMAIN }},${{ CAP_DOMAIN_SLUG }},${{ CAP_CERT_TOKEN }}or${{ CAP_DEPLOY_NAME }}, rendered per certificate at distribution time:/etc/ssl/${{ CAP_DOMAIN_SLUG }}/fullchain.pem. PreferCAP_DOMAIN_SLUGoverCAP_PRIMARY_DOMAINin paths: on a wildcard certificate the primary domain is*.example.comand the literal*lands in the directory name, while the slug renders asexample-com. Project variables also render; secret variables are rejected in paths; a missing variable fails that target. Note that substitution always applies: a literal${{ ... }}in a path or command must be escaped with a backslash (\${{ ... }}). A rollback re-deploys the previous version through the same PathSet rendering.
When several certificates share a host, put ${{ CAP_CERT_TOKEN }} or ${{ CAP_DEPLOY_NAME }} in the path (or use distinct PathSets) so files never collide.
Who may choose and run it. Choosing or changing a distribution's PathSet or ActionSet in the Overrides drawer needs the Admin role, and so does running or rolling back by hand a distribution whose targets or overrides run an ActionSet — both decide which administrator-written commands run on which hosts. An Operator gets 403 ACCESS_DENIED, or an approval request for an admin when the project's approval policy gates that Operator's distribution actions. The automatic run after a renewal is not affected.
Behaviours worth knowing
- A failed
chmodon the temp file fails the run; a failedchownis a warning. The permission restriction happens before any byte of a private key is written, and a host that refuses it stops the deploy rather than leaving the key world-readable.chownfailures (and the re-check on later runs) stay warnings in the job log, not entries in the distribution table's Warnings column, so read the log after the first deploy to a new host. On every later run the mode and owner are re-checked even for unchanged files and corrected when they drifted (again logged as a warning). - Action output is truncated at 4 KB per command or per script in the job log (WinRM keeps 16 KB). Redirect verbose output to a file on the host if you need all of it. Before it is written, the output is redacted like any other log line — PEM private-key blocks, long base64 runs, bearer tokens and
key=valuecredential shapes — and additionally any run of 80+ hex byte pairs, whether joined, line-wrapped (xxd -p), space-separated (od -An -tx1,hexdump) or colon-separated (openssl rsa -text), so a command that prints the deployed key as hex leaves[REDACTED]in the log rather than the key. Not recognised: a bare P-256 scalar (openssl ec -textprints 32 bytes, the size of a SHA-256 digest) andod -xwith its offset column. Do not print key material from a script. The job log is readable by every project viewer; an ActionSet that deliberately obfuscates output can still leak, which is why ActionSet changes sit behind the configuration approval gate. pfx,jksand encryptedprivate_keyrows are rewritten on every run. A PKCS#12 or JKS build, and an encrypted key, use a fresh random salt (a JKS also records its creation time), so the hash never matches the remote file; the file is uploaded and the action set runs on every deploy of that target. Ap7brow is deterministic and is skipped when unchanged.- The action set can run without a file change. Besides
run_always, a missing or stale per-certificate action marker (a previous run wrote the files but crashed before the reload; the first run after binding an action set) makes it run once. allowed_commandspatterns are unanchored substring matches, applied incommandmode only — anchor them (^…$); inline scripts are never checked.- No per-host lock. Two distributions (two certificates) can deploy to the same host at the same time; if they write the same paths or restart the same service, they race. Use distinct paths (
${{ CAP_DOMAIN_SLUG }}) per certificate.
Video tutorials
Short, hands-on videos showing this feature working for real. English audio, English and Turkish subtitles. All video tutorials →