Multi-VM deployment

Two or more standalone hosts behind one load balancer, sharing a single external MongoDB. Same one-command bootstrap as standalone — every host beyond the first adopts the cluster's secrets via --secrets-from. Works with both KEK backends (env hex bytes in secrets.env, or PKCS#11 keys inside an HSM).

When to use this

• You want active/active API redundancy without running Kubernetes.
• You already have an HA MongoDB (Atlas, self-hosted replica set, DocumentDB) and want CertAutoPilot to point at it.
• You are comfortable with two to a handful of VMs. For larger fleets or autoscaling, jump to Helm + HA.

One-command alternative: --nodes orchestration

Since 1.5.48 the standalone installer can build the whole fleet from the FIRST host in one command — including a built-in 3-member MongoDB replica set (caprs, MongoDB 8.0) on the first three nodes, so an external MongoDB is no longer a prerequisite for multi-VM:

# On the first host (M1):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=<pinned> --nodes=10.0.0.11,10.0.0.12,10.0.0.13

M1 mints the shared secrets and the mongo keyfile once, ships them to every node over SSH (a 0600 transport file — never command-line arguments), and drives the install end-to-end; the manual --secrets-from copy below is not needed on this path. --mongo=external still composes with --nodes when you prefer a managed database. Details and the failure/firewall notes: High availability. The manual per-host procedure below remains fully supported.

Topology

          ┌─ LB / DNS round-robin ─┐
          │                         │
   ┌──────▼──────┐           ┌──────▼──────┐
   │   cap-a     │           │   cap-b     │       (2..N hosts)
   │ nginx :443  │           │ nginx :443  │
   │ backend     │           │ backend     │
   │ (loopback   │           │ (loopback   │
   │  18181)     │           │  18181)     │
   └──────┬──────┘           └──────┬──────┘
          │                         │
          └────────┬────────────────┘
                   ▼
        External MongoDB (replica set)
        - shared by every host
        - cluster-wide setup lock
        - leader-elected scheduler lease
        - kek_install singleton (provider lock)

Every host runs the same certautopilot.service in --mode=all. The scheduler is leader-elected through a MongoDB lock, so only one host enqueues time-driven work at any moment. Workers claim from the shared queue — adding a host adds parallelism with no extra config.

The shared-secret rule

CertAutoPilot envelope-encrypts every field-level secret in MongoDB (ACME private keys, DNS credentials, module secrets, license blob, …). The wrapping key is the KEK, and what each host needs to hold depends on the KEK provider:

	env provider	pkcs11 provider
Where the KEK lives	Raw 32-byte hex in secrets.env on each host (CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N}).	Inside the HSM token; never on the filesystem. Each host stores only the PIN needed to C_Login against the token.
What every host MUST share	The same secrets.env contents (KEK material + JWT secret + Mongo URI).	The same secrets.env (PIN + JWT + URI), plus network access to the same HSM token (or HA cluster mirroring it).
What's allowed to differ per host	Nothing in secrets.env. config.yaml may differ (port, instance_name).	The PKCS#11 module file path, if the OS or vendor packaging differs (e.g. Debian vs RHEL). The token label MUST match.

For both providers, the kek_install singleton in MongoDB records which provider was chosen on the first host's install, and refuses any later host that tries to use the other one. This is enforced by install.sh via pkcs11::check_provider_lock before any persistent state is written, and again by the backend at service start.

secrets.env is the cluster's master credential

secrets.env is the cluster's master credential

For env: it contains the actual KEK. For pkcs11: it contains the HSM PIN, which is enough to decrypt every envelope when combined with HSM access. Treat it as the most sensitive file on each host. Transfer only over SSH; never email, never paste, never put in a chat.

Env provider — first host

Run the standard one-command bootstrap with --mongo=external. The bundled install.sh mints fresh KEK / JWT and writes them to /etc/certautopilot/secrets.env.

# On cap-a (the first host):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0"

The full URI is persisted verbatim to secrets.env as CERTAUTOPILOT_DATABASE_URI so multi-host replica sets, authSource, tls=true, and any other query-string options survive the install. Credentials MUST be URL-encoded per RFC 3986 — any raw @, :, /, ?, #, or % in the username or password breaks the installer's URI parser.

Verify before proceeding to the second host:

sudo systemctl status certautopilot nginx
curl -k https://127.0.0.1/readyz

Share secrets.env with the other hosts

/etc/certautopilot/secrets.env is mode 0600 owned by the certautopilot service user. A regular SSH login cannot read it directly — pull through sudo cat over the SSH session, then push to each target host:

# On your workstation (or any host with SSH to cap-a + cap-b/c):
umask 077
ssh cap-a 'sudo cat /etc/certautopilot/secrets.env' > /tmp/cap-shared.env

# Sanity check — non-empty and contains the KEK V1 line:
grep -c '^CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V' /tmp/cap-shared.env   # expect >= 1

# Push to each additional host's /tmp:
scp -p /tmp/cap-shared.env cap-b:/tmp/
scp -p /tmp/cap-shared.env cap-c:/tmp/   # repeat per additional host

# After every additional host has finished installing, shred the local copy:
shred -u /tmp/cap-shared.env

scp -p preserves the 0600 mode. The installer's --secrets-from path validates that the file is non-empty and (for env provider) contains a CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} line, so a transfer error is caught up front.

Production alternative to scp

Production alternative to scp

For long-lived clusters, store secrets.env in your secret manager (Vault KV, 1Password shared vault, AWS Secrets Manager, SealedSecrets) and have your config-management tool fetch + place it. The bootstrap script still consumes it the same way via --secrets-from=<path>.

Env provider — additional hosts

Same one-liner as the first host plus --secrets-from. The installer detects the file, validates it contains a CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} line, and adopts every value verbatim instead of minting fresh material.

# On cap-b (and every additional host):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0" \
      --secrets-from=/tmp/cap-shared.env

sudo shred -u /tmp/cap-shared.env

What --secrets-from actually does

What --secrets-from actually does

The installer copies the source file to /etc/certautopilot/secrets.env (mode 0600, owned by certautopilot:certautopilot) and keeps a timestamped backup of any prior file at secrets.env.before-secrets-from.<ts>. Reruns are safe — only the most recent 3 backups are retained. --mongo-uri is technically optional on additional hosts (the installer reuses the URI already inside the source file via mongodb::_setup_external rerun-preservation), but passing it explicitly is more readable and lets the installer probe reachability.

PKCS#11 provider — multi-VM with HSM-backed KEK

For an HSM-backed KEK fleet, every host needs three things:

1. The vendor PKCS#11 SDK installed locally — Thales Luna client, AWS CloudHSM client, Fortanix DSM agent, Azure Key Vault HSM client, or SoftHSM2 for dev. Each vendor has its own install procedure (see PKCS#11 vendors). The installer only validates that the module path you point it at exists.
2. Network reachability to the same HSM token — every host must C_Login against the same token, with the same label, holding the same AES-256 key. That means a network-attached HSM (CloudHSM cluster, Luna with HA proxy, Fortanix DSM endpoint) or an HA partition that mirrors keys. Per-host SoftHSM tokens do NOT work for multi-VM — each one would hold a different key.
3. The same secrets.env as the env path. For pkcs11, this file holds CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN + JWT secret + Mongo URI. The PIN must match the HSM credential; JWT must match across the fleet for tokens to round-trip.

First host (HSM)

Identical to the standalone PKCS#11 quickstart but with --mongo=external. The installer mints fresh JWT, writes the PIN to secrets.env, runs certautopilot kek pkcs11-init --version=1 against the HSM (probes CKM_AES_GCM, generates the v1 AES-256 key inside the token, writes the kek_install singleton with provider=pkcs11 and the v1 metadata), then starts the service.

# On cap-a:
umask 077
printf '%s' "$HSM_PIN" > /tmp/cap-pin

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0" \
      --kek-provider=pkcs11 \
      --pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
      --pkcs11-token-label=certautopilot-prod \
      --pkcs11-pin-file=/tmp/cap-pin

shred -u /tmp/cap-pin

The PIN is copied once into secrets.env (mode 0600). Systemd loads it via EnvironmentFile= on every service start, so reboots and systemctl restart work without re-supplying the PIN.

Transfer secrets.env + PIN file to each additional host

The PIN inside secrets.env is the same one each host needs at install time — extract it on the receiving side rather than passing it around twice:

# On your workstation:
umask 077
ssh cap-a 'sudo cat /etc/certautopilot/secrets.env' > /tmp/cap-shared.env

# Per additional host: ship secrets.env, then derive a PIN file from it
# locally on that host (so the PIN never lands on the workstation as a
# separate file).
scp -p /tmp/cap-shared.env cap-b:/tmp/
ssh cap-b "umask 077 && sed -n 's/^CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN=//p' /tmp/cap-shared.env > /tmp/cap-pin"

# Repeat scp + ssh per additional host, then shred locally:
shred -u /tmp/cap-shared.env

Additional hosts (HSM)

Pass both --secrets-from (so JWT + PIN match the fleet) and --pkcs11-pin-file (so the installer can validate the HSM connection during pkcs11-init). The cmdline PIN and the PIN inside secrets.env must be identical — both target the same HSM token.

# On cap-b (and every additional host):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0" \
      --kek-provider=pkcs11 \
      --pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
      --pkcs11-token-label=certautopilot-prod \
      --pkcs11-pin-file=/tmp/cap-pin \
      --secrets-from=/tmp/cap-shared.env

sudo shred -u /tmp/cap-pin /tmp/cap-shared.env

pkcs11-init is idempotent on join

On the second host, certautopilot kek pkcs11-init --version=1 sees that kek_install is already locked for provider=pkcs11 and that v1 metadata exists with a matching HSM key handle. It returns success without creating a second key. If the lock instead reads provider=env, the installer aborts with a clear message and the host is left untouched.

Module path may legitimately differ per host

The vendor SDK package can place libCryptoki2_64.so under different paths depending on OS family (e.g. /usr/safenet/lunaclient/lib/libCryptoki2_64.so on Debian, /opt/safenet/lunaclient/lib/libCryptoki2_64.so on RHEL). Pass the actual local path to --pkcs11-module on each host. The token label must match across the fleet.

Run the setup wizard on exactly ONE host

Open https://<any-host>/setup on a single host and create the initial admin + default project. The backend guards setup with a cluster-wide $setOnInsert flag in MongoDB:

• Concurrent wizard submissions from two hosts still produce exactly one admin + one default project — the loser receives 400 setup already completed.
• Every host beyond the first sees the flag already set and redirects /setup to the login page.

From then on, log in via the load balancer; sessions and CSRF cookies are stateless JWTs signed with the shared JWT_SECRET, so any host can serve any request.

Load balancer & TLS

Each host installs nginx with its own TLS material. The cluster as a whole sits behind a single virtual hostname (cap.example.com) — pick whichever model fits your environment:

Model	TLS terminates	Setup
L7 proxy / cloud LB (recommended)	at the LB	Install each host with --tls=self-signed (loopback-only trust) and let the LB terminate the public cert. Health check: GET /readyz.
L4 / DNS round-robin	per host	Install each host with --tls=provided --cert=… --key=… using the same wildcard or SAN cert covering cap.example.com.

For non-default ports add --port=<n> (public HTTPS) and/or --backend-port=<n> (loopback). Use --extra-hostnames=cap.example.com,cap-a.internal on every host so the self-signed material covers both the LB hostname and the per-host name for in-fleet checks.

To have CertAutoPilot manage the per-host certificate itself (issue + distribute to every node + auto-renew, instead of hand-rolling --tls=provided), list every host as an SSH target and follow Manage CAP's own TLS certificate — the recipe is cluster-aware (one target per node, or a target group).

Rolling upgrade

The standalone update.sh swaps the binary + frontend atomically and restarts the service. For a multi-VM fleet, do it one host at a time so the LB always has a healthy member:

# Per host, in order, after draining the host from the LB:
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/update.sh \
  | sudo bash -s -- --version=1.4.5

# Wait for /readyz to return 200, then return the host to the LB before
# moving to the next.
curl -k https://127.0.0.1/readyz

There are no schema migrations: indexes are ensured at startup and backfills are additive and idempotent, so any host can be upgraded first and the rest pick up the same database with no extra work. secrets.env, config.yaml, TLS material, and the Mongo connection string are preserved byte-for-byte. Same procedure for both env and pkcs11 providers — update.sh never touches the KEK.

While hosts run mixed versions, don't configure features the new release introduced (e.g. a new distribution target-override type): a not-yet-upgraded worker silently ignores the unknown field and deploys with the pre-override behavior. (Current examples: a Kerberos KDC Address pin; IIS target-level auth — an old worker reads auth only from the credential; and an IIS per-certificate override with an empty Site name — an old worker ignores the whole override and deploys to the target's own binding.) Finish the fleet first — and don't downgrade below the introducing version while such configuration exists.

Don't run get.sh across hosts to upgrade

Don't run get.sh across hosts to upgrade

get.sh is for first-install topology + secrets bootstrap. For binary refresh use update.sh — it touches nothing else and never re-mints secrets.

KEK rotation across the fleet

On first contact with each node the installer prints the SSH host key fingerprint it just pinned. If the network between your nodes is not trusted, compare it against the node before continuing — that first handshake is the only unauthenticated one, and the pinned key is reused by every later operation.

One command on one host rotates the whole fleet, and the sequence is identical on a single-node install:

cap-cluster kek-rotate 2

It prints exactly what it is about to do — old and new version, provider, the nodes it will touch — and asks you to type yes before anything changes. Then it creates the new key material once, writes the same value into every node's secrets.env (so it survives restarts), restarts each backend, confirms every live instance loaded V2, flips the active key, and waits for the re-wrap to finish before reporting. Your data stays readable throughout: V1 is not touched here.

Add --yes to skip the prompt when driving this from a script. With no terminal and no --yes the command refuses rather than assuming consent, so an unattended run can never rotate a fleet by accident.

Prefer to flip the key yourself? cap-cluster kek-add 2 does only the distribution and the fleet check, then prints the cap kek rotate line for you to run.

Provider differences are handled for you:

• env — the key material is the secret, so it is distributed to each node's secrets.env.
• pkcs11 — nothing is copied: the key is created inside the HSM and its record lands in the shared MongoDB, so each node only restarts to pick it up. Every node must reach the same HSM or token; a per-host token (a local SoftHSM, say) leaves the other nodes unable to load the new version, and cap-cluster warns when it sees a multi-node install.

If a node is unreachable, cap-cluster stops there instead of leaving the fleet half-distributed; fix the host and re-run the same command — nodes that already have the identical value are skipped. It refuses outright when a node already holds a different value under that version number, because two keys under one version would split the fleet's data at rotation time.

Each host's heartbeat tick (~30s) detects the keystore flip in MongoDB and hot-reloads V2 as the active version with no restart. After your backup window, retire V1 — again from one host:

cap kek remove --version=1     # verifies nothing still references V1
cap-cluster kek-drop 1         # deletes the material from every node

If kek remove reports that the version is still referenced, a node wrote new data with the old key during the ~30-second window between the switch and that node picking it up. Sweep those leftovers forward and retry:

cap-cluster kek-resweep 1     # re-encrypts anything left on V1
cap kek remove --version=1

If instead it refuses because some documents hold encrypted data with no key version recorded, those documents are invisible to rotation. Repair them first — the version is read from each document's own envelope, never guessed:

cap kek repair-versions --dry-run
cap kek repair-versions
cap-cluster kek-resweep <the version it stamped>

If you already removed a version and only then discovered data still needs it, the removal can be undone as long as the key material is still on the nodes:

cap kek reinstate --version=1
cap-cluster restart            # every node loads V1 again
cap-cluster kek-resweep 1
cap kek remove --version=1

kek remove is the safety gate and cap-cluster kek-drop enforces it: it refuses to delete key material until the keystore reports that version as removed, so a half-finished rotation can never be "finished off" by throwing away the key that would have decrypted the leftovers. Deleting key material cannot be undone, so this command asks for confirmation too (--yes skips it). If kek remove reports that instances still run on V1, wait ~30 seconds for the keystore flip to reach every node and retry.

PKCS#11 provider: the new key is created inside the HSM by certautopilot kek pkcs11-init --version=N (run once, from any host); its record lands in the shared MongoDB. kek rotate does not create it and refuses to start if it is missing. Every other host must then be restarted to load the new version — a heartbeat only moves the active pointer among versions a process has already loaded. There is no secrets.env edit and no key distribution, but the restart is not optional. cap-cluster kek-add <N> / kek-rotate <N> does the init and the fleet restart for you. See KEK rotation for the full procedure (env, pkcs11, K8s).

Backup & disaster recovery

• MongoDB — owned by your DBA team or managed offering. Snapshot at the storage layer or mongodump daily.
• secrets.env — back up once from any single host (the file is identical fleet-wide) and store offsite with the same protection as the database backup. For env provider this contains the KEK; without it the Mongo snapshot is unrecoverable. For pkcs11 it contains the HSM PIN — losing it locks you out of the HSM (unless the PIN is also held by your secret manager).
• HSM (pkcs11 only) — back up via the vendor's mechanism (Luna key cloning, CloudHSM cluster snapshots, Fortanix DSM key export to wrapped blob). Without HSM key recovery, every encrypted record is permanently lost regardless of MongoDB or PIN.
• Surviving hosts as a recovery source — if a single VM dies, bootstrap a replacement with --secrets-from pointing at secrets.env from any surviving host. No data restore needed; the new host joins the cluster and starts processing immediately.

Removing a host

Use the gentle uninstall — it removes the binary and systemd units but leaves no orphaned state on the shared MongoDB:

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/uninstall.sh \
  | sudo bash

Don't --purge the last host without exporting secrets.env first

Don't --purge the last host without exporting secrets.env first

Every host holds the same shared secret in its local secrets.env. --purge on one host deletes the file only there — the surviving hosts still hold a working copy. But --purge'ing every host without first exporting a copy of secrets.env elsewhere loses the env-provider KEK or the pkcs11 PIN entirely; the encrypted data in MongoDB becomes unrecoverable.

Troubleshooting

Symptom	Likely cause
New host returns 500 decrypt failed when reading existing certificates.	Forgot --secrets-from on install — the host minted a fresh KEK / JWT. Re-run get.sh with --secrets-from=<shared.env>; the installer adopts the shared file and replaces the bad one.
--secrets-from: no CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} line in <path>	(env provider only) Source file is corrupted, empty, or pointed at the wrong file. Re-export secrets.env from a known-good host with ssh <host> 'sudo cat /etc/certautopilot/secrets.env'.
error: KEK provider lock mismatch. locked in MongoDB : env, --kek-provider : pkcs11	The first host installed with one provider; you're now trying to add a host with the other. Provider choice is install-time-immutable — re-run with the same --kek-provider as the first host, or follow the provider-migration guide.
pkcs11 host fails at pkcs11-init with CKR_TOKEN_NOT_PRESENT or CKR_PIN_INCORRECT.	Token label or PIN doesn't match the HSM. Verify --pkcs11-token-label matches the first host exactly; verify the PIN inside --pkcs11-pin-file matches what the first host used. The installer prints the PKCS#11 module path and token label in the error; cross-check with pkcs11-tool --module=<path> --list-token-slots.
pkcs11 host fails with CKR_GENERAL_ERROR opening the module.	Vendor SDK not installed locally, or the module path passed to --pkcs11-module is wrong for this host's OS. Each host needs its own vendor SDK install — the installer doesn't ship vendor binaries.
/setup redirects to login on every host.	Setup already completed by another host — use the admin credentials from that wizard run, or reset via the runbook (destructive).
Scheduler appears idle (no renewal jobs queued).	Leader-elected — only one host runs the scheduler tick at a time by design. journalctl -u certautopilot on each host shows which one currently holds the lease.

See also

• Requirements & sizing — host requirements, topology choice, and capacity planning
• Standalone deployment
• High availability (Kubernetes)
• PKCS#11 setup
• PKCS#11 vendors
• KEK rotation
• Provider migration (env ↔ pkcs11)