Multi-VM deployment

Two or more standalone hosts behind one load balancer, sharing a single external MongoDB. Same one-command bootstrap as standalone — every host beyond the first adopts the cluster's secrets via --secrets-from. Works with both KEK backends (env hex bytes in secrets.env, or PKCS#11 keys inside an HSM).

When to use this

You want active/active API redundancy without running Kubernetes.
You already have an HA MongoDB (Atlas, self-hosted replica set, DocumentDB) and want CertAutoPilot to point at it.
You are comfortable with two to a handful of VMs. For larger fleets or autoscaling, jump to Helm + HA.

Topology

          ┌─ LB / DNS round-robin ─┐
          │                         │
   ┌──────▼──────┐           ┌──────▼──────┐
   │   cap-a     │           │   cap-b     │       (2..N hosts)
   │ nginx :443  │           │ nginx :443  │
   │ backend     │           │ backend     │
   │ (loopback   │           │ (loopback   │
   │  18181)     │           │  18181)     │
   └──────┬──────┘           └──────┬──────┘
          │                         │
          └────────┬────────────────┘
                   ▼
        External MongoDB (replica set)
        - shared by every host
        - cluster-wide setup lock
        - leader-elected scheduler lease
        - kek_install singleton (provider lock)

Every host runs the same certautopilot.service in --mode=all. The scheduler is leader-elected through a MongoDB lock, so only one host enqueues time-driven work at any moment. Workers claim from the shared queue — adding a host adds parallelism with no extra config.

The shared-secret rule

CertAutoPilot envelope-encrypts every field-level secret in MongoDB (ACME private keys, DNS credentials, module secrets, license blob, …). The wrapping key is the KEK, and what each host needs to hold depends on the KEK provider:

	`env` provider	`pkcs11` provider
Where the KEK lives	Raw 32-byte hex in `secrets.env` on each host (`CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N}`).	Inside the HSM token; never on the filesystem. Each host stores only the PIN needed to `C_Login` against the token.
What every host MUST share	The same `secrets.env` contents (KEK material + JWT secret + Mongo URI).	The same `secrets.env` (PIN + JWT + URI), plus network access to the same HSM token (or HA cluster mirroring it).
What's allowed to differ per host	Nothing in `secrets.env`. `config.yaml` may differ (port, instance_name).	The PKCS#11 module file path, if the OS or vendor packaging differs (e.g. Debian vs RHEL). The token label MUST match.

For both providers, the kek_install singleton in MongoDB records which provider was chosen on the first host's install, and refuses any later host that tries to use the other one. This is enforced by install.sh via pkcs11::check_provider_lock before any persistent state is written, and again by the backend at service start.

secrets.env is the cluster's master credential

secrets.env is the cluster's master credential

For env: it contains the actual KEK. For pkcs11: it contains the HSM PIN, which is enough to decrypt every envelope when combined with HSM access. Treat it as the most sensitive file on each host. Transfer only over SSH; never email, never paste, never put in a chat.

Env provider — first host

Run the standard one-command bootstrap with --mongo=external. The bundled install.sh mints fresh KEK / JWT and writes them to /etc/certautopilot/secrets.env.

# On cap-a (the first host):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0"

The full URI is persisted verbatim to secrets.env as CERTAUTOPILOT_DATABASE_URI so multi-host replica sets, authSource, tls=true, and any other query-string options survive the install. Credentials MUST be URL-encoded per RFC 3986 — any raw @, :, /, ?, #, or % in the username or password breaks the installer's URI parser.

Verify before proceeding to the second host:

sudo systemctl status certautopilot nginx
curl -k https://127.0.0.1/readyz

Share `secrets.env` with the other hosts

/etc/certautopilot/secrets.env is mode 0600 owned by the certautopilot service user. A regular SSH login cannot read it directly — pull through sudo cat over the SSH session, then push to each target host:

# On your workstation (or any host with SSH to cap-a + cap-b/c):
umask 077
ssh cap-a 'sudo cat /etc/certautopilot/secrets.env' > /tmp/cap-shared.env

# Sanity check — non-empty and contains the KEK V1 line:
grep -c '^CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V' /tmp/cap-shared.env   # expect >= 1

# Push to each additional host's /tmp:
scp -p /tmp/cap-shared.env cap-b:/tmp/
scp -p /tmp/cap-shared.env cap-c:/tmp/   # repeat per additional host

# After every additional host has finished installing, shred the local copy:
shred -u /tmp/cap-shared.env

scp -p preserves the 0600 mode. The installer's --secrets-from path validates that the file is non-empty and (for env provider) contains a CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} line, so a transfer error is caught up front.

Production alternative to scp

Production alternative to scp

For long-lived clusters, store secrets.env in your secret manager (Vault KV, 1Password shared vault, AWS Secrets Manager, SealedSecrets) and have your config-management tool fetch + place it. The bootstrap script still consumes it the same way via --secrets-from=<path>.

Env provider — additional hosts

Same one-liner as the first host plus --secrets-from. The installer detects the file, validates it contains a CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} line, and adopts every value verbatim instead of minting fresh material.

# On cap-b (and every additional host):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0" \
      --secrets-from=/tmp/cap-shared.env

sudo shred -u /tmp/cap-shared.env

What --secrets-from actually does

What --secrets-from actually does

The installer copies the source file to /etc/certautopilot/secrets.env (mode 0600, owned by certautopilot:certautopilot) and keeps a timestamped backup of any prior file at secrets.env.before-secrets-from.<ts>. Reruns are safe — only the most recent 3 backups are retained. --mongo-uri is technically optional on additional hosts (the installer reuses the URI already inside the source file via mongodb::_setup_external rerun-preservation), but passing it explicitly is more readable and lets the installer probe reachability.

PKCS#11 provider — multi-VM with HSM-backed KEK

For an HSM-backed KEK fleet, every host needs three things:

The vendor PKCS#11 SDK installed locally — Thales Luna client, AWS CloudHSM client, Fortanix DSM agent, Azure Key Vault HSM client, or SoftHSM2 for dev. Each vendor has its own install procedure (see PKCS#11 vendors). The installer only validates that the module path you point it at exists.
Network reachability to the same HSM token — every host must C_Login against the same token, with the same label, holding the same AES-256 key. That means a network-attached HSM (CloudHSM cluster, Luna with HA proxy, Fortanix DSM endpoint) or an HA partition that mirrors keys. Per-host SoftHSM tokens do NOT work for multi-VM — each one would hold a different key.
The same secrets.env as the env path. For pkcs11, this file holds CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN + JWT secret + Mongo URI. The PIN must match the HSM credential; JWT must match across the fleet for tokens to round-trip.

First host (HSM)

Identical to the standalone PKCS#11 quickstart but with --mongo=external. The installer mints fresh JWT, writes the PIN to secrets.env, runs certautopilot kek pkcs11-init --version=1 against the HSM (probes CKM_AES_GCM, generates the v1 AES-256 key inside the token, writes the kek_install singleton with provider=pkcs11 and the v1 metadata), then starts the service.

# On cap-a:
umask 077
printf '%s' "$HSM_PIN" > /tmp/cap-pin

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0" \
      --kek-provider=pkcs11 \
      --pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
      --pkcs11-token-label=certautopilot-prod \
      --pkcs11-pin-file=/tmp/cap-pin

shred -u /tmp/cap-pin

The PIN is copied once into secrets.env (mode 0600). Systemd loads it via EnvironmentFile= on every service start, so reboots and systemctl restart work without re-supplying the PIN.

Transfer `secrets.env` + PIN file to each additional host

The PIN inside secrets.env is the same one each host needs at install time — extract it on the receiving side rather than passing it around twice:

# On your workstation:
umask 077
ssh cap-a 'sudo cat /etc/certautopilot/secrets.env' > /tmp/cap-shared.env

# Per additional host: ship secrets.env, then derive a PIN file from it
# locally on that host (so the PIN never lands on the workstation as a
# separate file).
scp -p /tmp/cap-shared.env cap-b:/tmp/
ssh cap-b "umask 077 && sed -n 's/^CERTAUTOPILOT_ENCRYPTION_PKCS11_PIN=//p' /tmp/cap-shared.env > /tmp/cap-pin"

# Repeat scp + ssh per additional host, then shred locally:
shred -u /tmp/cap-shared.env

Additional hosts (HSM)

Pass both --secrets-from (so JWT + PIN match the fleet) and --pkcs11-pin-file (so the installer can validate the HSM connection during pkcs11-init). The cmdline PIN and the PIN inside secrets.env must be identical — both target the same HSM token.

# On cap-b (and every additional host):
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- \
      --version=1.4.0 \
      --mongo=external \
      --mongo-uri="mongodb://capApp:<url-encoded-pwd>@db.internal:27017/?authSource=admin&replicaSet=rs0" \
      --kek-provider=pkcs11 \
      --pkcs11-module=/opt/thales/lib/libCryptoki2_64.so \
      --pkcs11-token-label=certautopilot-prod \
      --pkcs11-pin-file=/tmp/cap-pin \
      --secrets-from=/tmp/cap-shared.env

sudo shred -u /tmp/cap-pin /tmp/cap-shared.env

pkcs11-init is idempotent on join

On the second host, certautopilot kek pkcs11-init --version=1 sees that kek_install is already locked for provider=pkcs11 and that v1 metadata exists with a matching HSM key handle. It returns success without creating a second key. If the lock instead reads provider=env, the installer aborts with a clear message and the host is left untouched.

Module path may legitimately differ per host

The vendor SDK package can place libCryptoki2_64.so under different paths depending on OS family (e.g. /usr/safenet/lunaclient/lib/libCryptoki2_64.so on Debian, /opt/safenet/lunaclient/lib/libCryptoki2_64.so on RHEL). Pass the actual local path to --pkcs11-module on each host. The token label must match across the fleet.

Run the setup wizard on exactly ONE host

Open https://<any-host>/setup on a single host and create the initial admin + default project. The backend guards setup with a cluster-wide $setOnInsert flag in MongoDB:

Concurrent wizard submissions from two hosts still produce exactly one admin + one default project — the loser receives 400 setup already completed.
Every host beyond the first sees the flag already set and redirects /setup to the login page.

From then on, log in via the load balancer; sessions and CSRF cookies are stateless JWTs signed with the shared JWT_SECRET, so any host can serve any request.

Load balancer & TLS

Each host installs nginx with its own TLS material. The cluster as a whole sits behind a single virtual hostname (cap.example.com) — pick whichever model fits your environment:

Model	TLS terminates	Setup
L7 proxy / cloud LB (recommended)	at the LB	Install each host with `--tls=self-signed` (loopback-only trust) and let the LB terminate the public cert. Health check: `GET /readyz`.
L4 / DNS round-robin	per host	Install each host with `--tls=provided --cert=… --key=…` using the same wildcard or SAN cert covering `cap.example.com`.

For non-default ports add --port=<n> (public HTTPS) and/or --backend-port=<n> (loopback). Use --extra-hostnames=cap.example.com,cap-a.internal on every host so the self-signed material covers both the LB hostname and the per-host name for in-fleet checks.

Rolling upgrade

The standalone update.sh swaps the binary + frontend atomically and restarts the service. For a multi-VM fleet, do it one host at a time so the LB always has a healthy member:

# Per host, in order, after draining the host from the LB:
curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/update.sh \
  | sudo bash -s -- --version=1.4.5

# Wait for /readyz to return 200, then return the host to the LB before
# moving to the next.
curl -k https://127.0.0.1/readyz

Schema migrations are forward-only and idempotent — the first upgraded host runs them; subsequent hosts pick up the migrated schema with no extra work. secrets.env, config.yaml, TLS material, and the Mongo connection string are preserved byte-for-byte. Same procedure for both env and pkcs11 providers — update.sh never touches the KEK.

Don't run get.sh across hosts to upgrade

Don't run get.sh across hosts to upgrade

get.sh is for first-install topology + secrets bootstrap. For binary refresh use update.sh — it touches nothing else and never re-mints secrets.

KEK rotation across the fleet

Env provider: the V2 KEK material must be IDENTICAL on every host before triggering rotation. Generate it once, then add the SAME value to secrets.env on every host before running kek rotate:

# Step 1 — generate the new key ONCE (on any one host or your workstation):
NEW_KEK=$(openssl rand -hex 32)
echo "Distribute this V2 to every host: $NEW_KEK"

# Step 2 — on EACH host (cap-a, cap-b, cap-c), append the SAME value:
NEW_KEK='<paste-the-value-from-step-1>'
printf '\nCERTAUTOPILOT_ENCRYPTION_ENV_KEK_V2=%s\n' "$NEW_KEK" \
  | sudo tee -a /etc/certautopilot/secrets.env >/dev/null
sudo systemctl restart certautopilot

# Step 3 — on any ONE host, verify every node loaded V2, then rotate
# (cap = the installer's CLI wrapper; it loads secrets.env, sudo alone does not):
cap kek verify --target=2
cap kek rotate --from-version=1 --to-version=2
cap kek status   # poll until completed

If you have a config-management tool (Ansible, Salt, Puppet, …) drop the V2 line in via that instead of pasting per-host. The contract is the same — every host's secrets.env MUST end up with the identical V2 value before kek rotate runs.

Each host's heartbeat tick (~30s) detects the keystore flip in MongoDB and hot-reloads V2 as the active version with no restart. After your backup window, retire V1 from each host:

cap kek remove --version=1
# Then on every host:
sudo sed -i '/^CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V1=/d' /etc/certautopilot/secrets.env
sudo systemctl restart certautopilot

PKCS#11 provider: rotation is much simpler — the new key is generated inside the HSM by certautopilot kek rotate on a single host, and every other host pulls the new version handle from MongoDB on next heartbeat. No secrets.env edit, no fleet-wide distribution. See KEK rotation for the full procedure (env, pkcs11, K8s).

Backup & disaster recovery

MongoDB — owned by your DBA team or managed offering. Snapshot at the storage layer or mongodump daily.
secrets.env — back up once from any single host (the file is identical fleet-wide) and store offsite with the same protection as the database backup. For env provider this contains the KEK; without it the Mongo snapshot is unrecoverable. For pkcs11 it contains the HSM PIN — losing it locks you out of the HSM (unless the PIN is also held by your secret manager).
HSM (pkcs11 only) — back up via the vendor's mechanism (Luna key cloning, CloudHSM cluster snapshots, Fortanix DSM key export to wrapped blob). Without HSM key recovery, every encrypted record is permanently lost regardless of MongoDB or PIN.
Surviving hosts as a recovery source — if a single VM dies, bootstrap a replacement with --secrets-from pointing at secrets.env from any surviving host. No data restore needed; the new host joins the cluster and starts processing immediately.

Removing a host

Use the gentle uninstall — it removes the binary and systemd units but leaves no orphaned state on the shared MongoDB:

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/uninstall.sh \
  | sudo bash

Don't --purge the last host without exporting secrets.env first

Don't --purge the last host without exporting secrets.env first

Every host holds the same shared secret in its local secrets.env. --purge on one host deletes the file only there — the surviving hosts still hold a working copy. But --purge'ing every host without first exporting a copy of secrets.env elsewhere loses the env-provider KEK or the pkcs11 PIN entirely; the encrypted data in MongoDB becomes unrecoverable.

Troubleshooting

Symptom	Likely cause
New host returns `500 decrypt failed` when reading existing certificates.	Forgot `--secrets-from` on install — the host minted a fresh KEK / JWT. Re-run `get.sh` with `--secrets-from=<shared.env>`; the installer adopts the shared file and replaces the bad one.
`--secrets-from: no CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} line in <path>`	(env provider only) Source file is corrupted, empty, or pointed at the wrong file. Re-export `secrets.env` from a known-good host with `ssh <host> 'sudo cat /etc/certautopilot/secrets.env'`.
`error: KEK provider lock mismatch. locked in MongoDB : env, --kek-provider : pkcs11`	The first host installed with one provider; you're now trying to add a host with the other. Provider choice is install-time-immutable — re-run with the same `--kek-provider` as the first host, or follow the provider-migration guide.
pkcs11 host fails at `pkcs11-init` with `CKR_TOKEN_NOT_PRESENT` or `CKR_PIN_INCORRECT`.	Token label or PIN doesn't match the HSM. Verify `--pkcs11-token-label` matches the first host exactly; verify the PIN inside `--pkcs11-pin-file` matches what the first host used. The installer prints the PKCS#11 module path and token label in the error; cross-check with `pkcs11-tool --module=<path> --list-token-slots`.
pkcs11 host fails with `CKR_GENERAL_ERROR` opening the module.	Vendor SDK not installed locally, or the module path passed to `--pkcs11-module` is wrong for this host's OS. Each host needs its own vendor SDK install — the installer doesn't ship vendor binaries.
`/setup` redirects to login on every host.	Setup already completed by another host — use the admin credentials from that wizard run, or reset via the runbook (destructive).
Scheduler appears idle (no renewal jobs queued).	Leader-elected — only one host runs the scheduler tick at a time by design. `journalctl -u certautopilot` on each host shows which one currently holds the lease.

When to use this​

Topology​

The shared-secret rule​

Env provider — first host​

Share secrets.env with the other hosts​

Env provider — additional hosts​

PKCS#11 provider — multi-VM with HSM-backed KEK​

First host (HSM)​

Transfer secrets.env + PIN file to each additional host​

Additional hosts (HSM)​

Run the setup wizard on exactly ONE host​

Load balancer & TLS​

Rolling upgrade​

KEK rotation across the fleet​

Backup & disaster recovery​

Removing a host​

Troubleshooting​

See also​