Skip to main content

SSH module

The SSH module copies the certificate + private key + chain to a target host using SCP/SFTP, then runs post-deployment actions (typically systemctl reload nginx or equivalent). It is the most-used distribution module. Rollback re-deploys a previous retained certificate version through the same deploy path.

Overview​

  • Transport: SSH v2 (SFTP for writes).
  • Auth: an unencrypted PEM SSH private key, or a password. The module decides which by trying to parse the credential as a key and falling back to password authentication — so a passphrase-protected key is not supported: it fails to parse and is then offered as a password, which the host rejects. Decrypt the key before storing it (CertAutoPilot envelope-encrypts it at rest). TLS client certificates are not an SSH auth method and are not supported.
  • Path set: defines where each file goes (cert, key, chain) with ownership + mode.
  • Action set: shell commands run after writes, typically to reload the consuming service.
  • Rollback: supported — a rollback re-writes the previous version's cert/key files at the PathSet paths and re-runs the ActionSet (Rollback).
  • Concurrency: the module runs up to 10 SSH targets of one distribution in parallel by default. DistributionSSHMaxConcurrency (Settings → General) replaces that default for every SSH distribution when set (1–50; 0 leaves the module default) — it can raise the parallelism as well as lower it. Turning off Parallel execution (Settings → General) forces sequential execution for every module, SSH included — it wins over DistributionSSHMaxConcurrency, which is not consulted while the switch is off.

Prerequisites​

  • An SSH module credential (key or password) saved under Settings → Distribution → Credentials.
  • Target hosts reachable from the backend. The backend's pod or VM must have SSH egress to every target.
  • A user account on each target with write permission to the cert/key directories, plus sudo (NOPASSWD) for the action set if it uses privileged commands.
  • A path set and an optional action set.

Create an SSH target​

  1. Settings → Distribution → Targets → New.
  2. Module type: SSH.
  3. Hostname + port (default 22).
  4. Username: the account on the remote host.
  5. Credential: pick the SSH key credential you created.
  6. Save → the health check opens an SSH connection (and, in the detailed check, an SFTP session) and closes it. No command is run on the host.

Host-key verification​

By default the module accepts whatever host key the remote presents — anything answering on the configured address is treated as the target, and the private key is written to it. Turn on Require approval when the host key changes on the target to change that.

With it on:

  • The first key seen during a deployment is remembered silently. You do nothing, and nothing is blocked — enabling it on a target that already has deployments is safe. Dry-runs and health checks never establish the pin.
  • Every later connection is checked against that key. A match proceeds as before.
  • A mismatch stops the deployment with SSH_HOST_KEY_CHANGED, records the new key, and raises a review prompt on the target. Nothing is written to the host. Dry-runs and health checks are checked against the same pin and will report the target as unreachable, but only a deployment records the key and raises the prompt.
  • If the original host comes back and matches the pin again, the prompt clears itself. A single transient mismatch does not leave you with a review you can only resolve by approving.
  • A blocked target does not drag its neighbours down: because nothing was written to it, a host-key failure alone never triggers auto-rollback, so the hosts that did receive the certificate keep it.
  • Approving queues the blocked distributions for redistribution, so the deployment resumes on its own — the confirmation tells you how many will re-run. Distributions the renewal sweep would skip anyway are not counted: one left partial (some targets already had the certificate), one whose module config has automatic distribution turned off, or one that was rolled back — re-run those from the certificate's Distributions tab.
The first key is trusted, not verified

CertAutoPilot cannot tell whether the first key it sees is the right one — that is what trust-on-first-use means. If something is impersonating the host at that moment, the impostor's key is what gets remembered, and connections to the genuine host are then rejected instead. Where host identity really matters, compare the pinned fingerprint against the host itself after the first deployment:

ssh-keyscan -p <port> <host> | ssh-keygen -lf -

Use ssh-keyscan rather than reading a single key file: a host normally offers several key types (Ed25519, ECDSA, RSA) and CertAutoPilot pins whichever one the connection negotiated, so ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub may well print a fingerprint that was never used. Once a key is pinned, later connections offer that key type first, so a host that keeps its key does not start failing because it also offers others. The other types stay available as fallback — a host that legitimately dropped the pinned type still connects far enough to be reported as a reviewable change, rather than failing negotiation with nothing to approve.

Approving is an operator decision, not a machine one: CertAutoPilot cannot tell a rebuilt host from an impostor. The review dialog gives you what makes the call possible — how long the old key had been trusted and how many deployments it served, the new fingerprint in the same form ssh-keygen -lf prints, and a command to compare it against the host itself. The approval is bound to the fingerprint you were shown: if the key changes again between opening the dialog and clicking approve, the request is refused rather than silently blessing a key you never reviewed. Approval is deliberately not routed through the configuration-approval workflow, so a rebuilt server does not hold up a renewal waiting for a second person; the decision is written to the audit log with both fingerprints instead.

Turning the setting back off leaves the remembered key in place, so re-enabling it later does not blindly trust whatever answers next.

The Host key column on the targets list shows what state a target is in — Awaiting first key before anything is pinned, Pinned once one is, Changed — review when a decision is waiting. Click Pinned or Changed — review to see the fingerprint, how long it has been trusted and how many deployments it has served; that is how you compare it against the host after a first deployment. (Awaiting first key has nothing recorded yet, so it is not clickable.)

Host certificates. If your servers present an OpenSSH host certificate, CertAutoPilot records the key inside it rather than the certificate. Certificates are short-lived by design, so pinning one would report a changed host key on every routine renewal. Note this is not CA-based verification — the certificate's signature is not checked against a trusted CA; only the underlying key is remembered.

Not for one name in front of several machines

A load balancer, HA pair or round-robin DNS name presents a different host key depending on which machine answers, so verification would fail at random. Leave the setting off for those targets and rely on network-layer control instead.

If you leave it off

Nothing changes from previous releases, and nothing warns you — which is the point of it being opt-in. Restrict SSH egress from the CertAutoPilot host to the intended targets (management VLAN, firewall allowlist, bastion) and treat DNS and ARP integrity on that path as part of your threat model, because a host-key change will not be detected.

Path set​

A path set defines the files you want CertAutoPilot to write, with ownership and mode. Example for nginx:

/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.crt    root:root   0644   <cert+chain>
/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.key    root:root   0600   <private key>
/etc/nginx/ssl/${{ CAP_DOMAIN_SLUG }}.ca     root:root   0644   <chain only>

owner (user or user:group — letters, digits, ., _, -; at most 32 characters per part) and mode (3–4 octal digits) are validated when the path set is saved. They are applied when a file is written and re-checked on every later run: if the mode or owner of an unchanged file drifted on the host, it is put back and a warning line is written to the job log.

Paths use ${{ NAME }} placeholder substitution — not Go templates. CAP_DOMAIN_SLUG is a reserved certificate-context variable (the primary domain lowercased to [a-z0-9-], wildcard *. stripped so it stays path-safe); the other reserved vars are CAP_PRIMARY_DOMAIN, CAP_CERT_TOKEN, and CAP_DEPLOY_NAME. You can reference project variables (Variables) the same way, but secret-flagged variables are rejected in paths. Typical practice is one path set per service template (nginx, haproxy, apache, postgres, dovecot, etc.) and reuse it across many targets.

Action set​

Actions run over SSH in sequence after every file has been written. Example:

sudo nginx -t
sudo systemctl reload nginx

Use nginx -t-style preflight to fail fast before reloading. A non-zero exit status marks the target as failed and triggers auto-rollback if it's enabled for the distribution.

By default the action runs only when a written file actually changes (hash-based skip), so reloads stay idempotent on no-op redistributes. Set run_always on the action set to run it on every distribution even when nothing changed — for verification/diagnostic actions that must run each time. There is a third trigger: CertAutoPilot records, per certificate and target, the version for which the action last completed; if that marker is missing or belongs to another version (a previous run wrote the files but crashed before the reload), the action runs once even though the files already match.

run_as (^[a-z_][a-z0-9_-]{0,31}$), shell (an absolute path to sh, bash, dash, zsh, ksh, mksh, ash or fish; default /bin/bash) and timeout_seconds (0–3600) are validated when the action set is saved.

Execute a distribution​

  1. On the cert detail page → Distribution tab → Add distribution.
  2. Pick the SSH module and a target or target group. The path set and action set come from the target (Settings → Distribution → Targets); a certificate can swap them per target from the distribution's Overrides drawer.
  3. Optionally enable validation endpoints so we confirm the cert is live post-deploy (see Validation).
  4. Click Dry-run first. It returns a plan, not an exact manifest: files whose hash already matches are omitted, and key-bearing rows (private_key, combined, pfx) report "would upload if changed" because the key is not loaded during a dry run. Review it as intent, then execute.
  5. Execute. Progress shows per target; a green tick means files written + action set returned 0. Red means failure — open the target's log to see whether it was transport (SSH_CONNECT, class network — including a handshake the peer dropped), rejected credentials (SSH_AUTH / CREDENTIAL_MISSING, auth), a transient file-transfer problem (SFTP_FAILED / FILE_UPLOAD, io_transient), a refusal by the host (SSH_PERMISSION, io_permanent — permission denied, missing directory, read-only filesystem; never retried), or a non-zero post-deploy action (SCRIPT_EXIT_NONZERO, io_transient — retried, not a validation failure). Deterministic pre-flight problems — a missing SSH spec, a path set or action set that no longer resolves, a bad path or variable — come back as SSH_VALIDATION and are never retried.

Rollback​

Supported via previous-version re-deploy: a rollback re-writes the previous retained certificate version's cert/key files at the PathSet paths and re-runs the ActionSet — the module's normal deploy path, fed older material. Pick the version in the rollback version picker; eligibility rules, auto-rollback, and the sticky rolled_back status are covered in Rollback.

.bak files are no longer created

The module no longer writes .bak backup copies before overwriting — rollback comes from CertAutoPilot's own version history, not from files on the host. Existing .bak files on hosts are inert leftovers from older versions and are safe to delete manually.

Troubleshooting​

SSH_AUTH although the key works in a terminal​

SSH_AUTH means the host rejected the credentials during the handshake — it is not a file-permission problem (those are SSH_PERMISSION, see below). The usual cause is a passphrase-protected key: it cannot be parsed, so it is offered as a password and refused. Remove the passphrase (ssh-keygen -p -f key -N "") and save the credential again. Otherwise check the target user, that ~/.ssh/authorized_keys holds the right public key, and — for password credentials — that sshd allows the password method (keyboard-interactive is never offered). A handshake that fails for a transport reason (the peer closes the connection, connection reset) is reported as SSH_CONNECT and retried.

SSH_PERMISSION — the host refused a file operation​

Creating or writing a path-set file was refused: permission denied on the file or its directory, a missing parent directory the SSH user could not create, a path that is not a directory, or a read-only filesystem. It is not retried — give the SSH user write access to the destination directories (file writes never go through sudo), create the directory, or fix the path. Passwordless sudo is only needed for the action-set commands.

A deploy was interrupted and the service will not start​

Files are written in place over the destination — the SFTP write truncates the existing file before the new content lands. If the transfer is interrupted (per-target timeout, job cancel, worker restart), the path is left holding a truncated or empty file rather than the previous certificate, and the post-deploy action may not have run to report it. Re-run the distribution to repair it. Putting a config test (nginx -t, haproxy -c) as the first ActionSet command catches it on the next run.

(The WinRM module behaves differently: it stages to a sibling file and renames it into place only after a clean decode, so an interrupted transfer there leaves the existing file intact.)

SSH_HOST_KEY_ALGO​

Host-key negotiation failed before any key was presented: the server offers only key types CertAutoPilot does not advertise once verification is on — in practice a DSA-only server. There is nothing to approve; add a modern host key to the server, or turn verification off for that target.

SSH_HOST_KEY_CHANGED​

The server presented a host key that does not match the one remembered for this target, so nothing was deployed. Open Settings → Distribution → Targets; the target carries a Changed — review tag and the page banner links straight to it. Compare the new fingerprint against the host — ssh-keyscan -p <port> <host> | ssh-keygen -lf -, run from a machine you trust — before approving. Do not read a single key file: CertAutoPilot pins whichever key type the connection negotiated, so a specific .pub file may print a fingerprint that was never used.

This error is never retried — retrying would just re-present the same key — so the deployment stays failed until you decide.

If the target does not have host-key verification enabled, a rebuilt VM (or an impostor on the same address) is accepted silently and there is nothing to review; see Host-key verification.

Action set times out​

Long-running reload (systemd's default is 90 s). Raise timeout_seconds on the action set itself (it is a field on the library resource, not on the distribution) — and note that leaving it unset means no per-command bound at all, only the 300 s per-target ceiling; the action will be killed and the target marked failed if it exceeds.

See also​

Multi-certificate hosts​

Two mechanisms let one SSH target serve many certificates:

  1. Per-distribution PathSet/ActionSet — each certificate's distribution can select its own PathSet (where files land) and ActionSet (what runs afterwards) from the certificate's Distributions → Overrides drawer (ssh { path_set_id, action_set_id }, with a * default row). A selected override must resolve — a deleted or wrong-kind resource fails the run instead of silently using the target's own libraries.
  2. Certificate-context variables in paths — PathSet paths may contain ${{ CAP_PRIMARY_DOMAIN }}, ${{ CAP_DOMAIN_SLUG }}, ${{ CAP_CERT_TOKEN }} or ${{ CAP_DEPLOY_NAME }}, rendered per certificate at distribution time: /etc/ssl/${{ CAP_DOMAIN_SLUG }}/fullchain.pem. Prefer CAP_DOMAIN_SLUG over CAP_PRIMARY_DOMAIN in paths: on a wildcard certificate the primary domain is *.example.com and the literal * lands in the directory name, while the slug renders as example-com. Project variables also render; secret variables are rejected in paths; a missing variable fails that target. Note that substitution always applies: a literal ${{ ... }} in a path or command must be escaped with a backslash (\${{ ... }}). A rollback re-deploys the previous version through the same PathSet rendering.

When several certificates share a host, put ${{ CAP_CERT_TOKEN }} or ${{ CAP_DEPLOY_NAME }} in the path (or use distinct PathSets) so files never collide.

Behaviours worth knowing​

  • Ownership/permission failures are warnings, not failures. If chmod or chown fails on a written file the run still reports success — the warning is a line in the job log, not an entry in the distribution table's Warnings column, so read the log after the first deploy to a new host: a private key can land world-readable. On every later run the mode and owner are re-checked even for unchanged files and corrected when they drifted (again logged as a warning).
  • Action output is truncated at 4 KB per command or per script in the job log (WinRM keeps 16 KB). Redirect verbose output to a file on the host if you need all of it.
  • pfx rows are rewritten on every run. A PKCS#12 build uses a fresh random salt, so its hash never matches the remote file; the file is uploaded and the action set runs on every deploy of that target.
  • The action set can run without a file change. Besides run_always, a missing or stale per-certificate action marker (a previous run wrote the files but crashed before the reload; the first run after binding an action set) makes it run once.
  • allowed_commands patterns are unanchored substring matches, applied in command mode only — anchor them (^…$); inline scripts are never checked.
  • No per-host lock. Two distributions (two certificates) can deploy to the same host at the same time; if they write the same paths or restart the same service, they race. Use distinct paths (${{ CAP_DOMAIN_SLUG }}) per certificate.