Skip to main content

CertAutoPilot OS appliance

CertAutoPilot OS is a ready-made system image for CertAutoPilot: a Debian 13 appliance carrying the backend, the web UI, nginx, MongoDB 8.0 and a management console. It boots, asks for its network settings, forms — or joins — a cluster and then gets out of the way. The node-local layout is the one the standalone installer produces (the same files, owners, MongoDB users and connection string), so the backup, upgrade and runbook pages apply to an appliance as they stand.

When to use this​

  • You want CertAutoPilot on a VM or bare metal without preparing a Linux host first.
  • You want a self-contained three-node installation (a MongoDB replica set and an optional shared address) formed from the console in a few commands.
  • You prefer an appliance whose operating system is hardened, updated and supported as one unit with the application.

Downloads and sizing​

The images are published on the certautopilot-archive releases page (the releases named cap-os-v<version>), with a SHA256SUMS file signed by the CertAutoPilot OS release key, fingerprint 68FA 9CAD B2C0 5E10 4AE0 B1BB ECEF 4DC4 BC8D B41C:

FileUse it for
cap-os-<ver>.isobare metal and any hypervisor (BIOS and UEFI, interactive or unattended install)
cap-os-<ver>.ovaVMware vSphere / Workstation — the settings are asked in the deployment wizard
cap-os_<ver>.qcow2KVM, Proxmox, OpenStack
gpg --import cap-os-release-signing.asc
gpg --verify SHA256SUMS.asc SHA256SUMS && sha256sum -c --ignore-missing SHA256SUMS

Compare the key's fingerprint with the one above, not only with the key file that came with the download.

MinimumRecommended
vCPU2, with AVX4
RAM4 GB8 GB
Disk30 GiB50–100 GiB
MongoDB 8 needs AVX

MongoDB 8.0 does not start on a CPU without AVX. On Proxmox, set the VM's CPU type to host or x86-64-v3; the default x86-64-v2-AES has no AVX.

Installing from the ISO on Proxmox with UEFI (OVMF)

Put the CD first in the VM's boot order (Options → Boot Order). With the network device first, OVMF tries network boot (PXE and HTTP, IPv4 and IPv6) for about four minutes on every boot before it reaches the CD.

First boot​

Log in on the console (or over SSH) as admin; the default password is admin and must be changed at the first login. The admin login shell is the CertAutoPilot console, not bash: help lists every command, show health is the screen to read first.

Every node of a cluster needs, before cluster init or cluster join:

  1. A static management address. The replica set and every peer name the node by it, so it cannot move later.
  2. A synchronised clock (NTP); the members must be within seconds of each other.
  3. The same CertAutoPilot version on every node (see versions).

Set the address on the console, apply it and confirm it:

cap> set network static 10.0.0.11/24 10.0.0.1 10.0.0.1
cap> apply
cap> network confirm
danger
Confirm the network change before cluster init

A management network change applied from the console is commit-confirmed: unless network confirm runs within 120 seconds, the previous configuration comes back by itself. Run network confirm (from the new address, if it changed) before cluster init or cluster join — a cluster formed inside that window is left on the old address when the change reverts. network status shows whether a change is still waiting.

The same settings can be supplied without a console: the OVA's deployment wizard, a cloud-init write_files of /etc/cap/setup.yaml, or an answer file baked onto the ISO. A setting given there is applied at first boot and needs no confirmation.

A single node​

The web UI starts once the node forms a cluster — a single node is a one-member cluster:

cap> cluster init
cap> show cluster

cluster init creates the cluster's CA, the secrets (key-encryption key, JWT secret, database users, replica-set keyfile), starts MongoDB as the one-member replica set caprs, renders the backend's configuration and a self-signed certificate for the UI, starts the backend and nginx and waits for them to answer. When it prints cluster formed, open https://<address>/ and create the first administrator.

A single node can grow to three later with cluster join on the new nodes; the data is synchronised to them.

Three nodes​

Three nodes are the highly available shape: three voting MongoDB members, any one of which may fail. Every node serves the complete web UI and API.

node 1> cluster init [--vip 10.0.0.10/24] # node 1 is the leader
node 1> cluster token new # one join, 24 hours
node 2> cluster join 10.0.0.11 capjoin1.… # the leader's address and the token
node 1> cluster token new
node 3> cluster join 10.0.0.11 capjoin1.…
node 1> show cluster # 1 PRIMARY + 2 SECONDARY
  • Order. Run cluster init on the first node only, then cluster join on each of the others, one after the other. Each join asks for confirmation, copies the data (MongoDB's initial sync) and starts the backend; it takes a few minutes.
  • Tokens. cluster token new on the leader prints a single-use token valid for 24 hours. The token's secret never crosses the network: the joiner proves it holds it, bound to the TLS connection, and the token pins the leader's certificate.
  • Two nodes is a stop on the way to three, not a shape to stay in: only the leader votes, so a leader outage leaves no writable database. show health says so until the third node is in.
  • What the leader checks before it admits a node: the token, the joiner's static address, its clock (within 10 seconds), its CertAutoPilot and MongoDB versions, its database partition and an empty database. Every refusal names what to do next; nothing is changed on a refusal.
  • Checking the path first. diagnose join <leader-ip> on the new node tests DNS, TCP and TLS towards the leader's agent port (7443), its own clock and whether its address is static, without changing anything. With the host firewall on, the leader opens 7443 to non-members only while a join token is valid, so run it after cluster token new.

A node that already holds a database (it ran cluster init itself) cannot join another cluster as it is: cluster reset on that node wipes its database, secrets and cluster state first. Data from two clusters is never merged.

Unattended: three answer files​

The cluster can also be formed without a console session: each node gets an answer file (/etc/cap/setup.yaml, through cloud-init write_files, the ISO or the OVA) whose cluster: block says what the node is. All three share one pre-shared token that you generate yourself. It has the join token's shape, capjoin1. followed by 6 and then 32 characters from a-z and 2-7:

python3 -c 'import secrets;a="abcdefghijklmnopqrstuvwxyz234567";r=lambda n:"".join(secrets.choice(a) for _ in range(n));print(f"capjoin1.{r(6)}.{r(32)}")'

Node 1 forms the cluster and accepts the token:

version: 1
system:
hostname: cap-1
ntp: [ntp1.example.net]
network:
mode: static
address: 10.0.0.11/24
gateway: 10.0.0.1
dns: [10.0.0.2]
cluster:
role: init
token: capjoin1.… # the pre-shared token
vip:
address: 10.0.0.10/24

Nodes 2 and 3 join it with the same token (node 3: cap-3, 10.0.0.13/24):

version: 1
system:
hostname: cap-2
ntp: [ntp1.example.net]
network:
mode: static
address: 10.0.0.12/24
gateway: 10.0.0.1
dns: [10.0.0.2]
cluster:
role: join
token: capjoin1.… # the same token
join:
address: 10.0.0.11 # node 1
  • Any power-on order. A joining node retries for an hour until the leader answers, so the three can be started together or the leader last. show cluster on a waiting node says why it is still waiting.
  • The pre-shared token is valid for 72 hours or until the third member has joined, whichever comes first. Treat the answer files as secrets, and prefer interactive cluster token new tokens in production.
  • The full list of settings, including external MongoDB, extra_hostnames and a PKCS#11 key-encryption key, is in /etc/cap/setup.yaml.example on the appliance. A vendor HSM cannot be named in an answer file; see CertAutoPilot OS with an HSM.

The shared address (VIP)​

cluster init --vip <address/prefix> gives the cluster one more address in the management network that follows a healthy node (keepalived, unicast VRRP); browsers and API clients then need to know one name, and the UI certificate names it. Without a VIP, every node is reachable on its own address, and a load balancer in front of the three works as well: every node reads and writes through the replica set's primary, and sessions are valid on any node.

From CertAutoPilot OS 0.1.1 on, the leader can add, move or remove the VIP of a running cluster without a reset:

node 1> cluster vip set 10.0.0.10/24
node 1> cluster vip remove

The web UI's certificate​

Every node serves the UI with a self-signed certificate of its own. It names the node's hostname, its management address, the VIP and the extra_hostnames of the answer file. The node re-issues it when those names change and 30 days before it ends. Browsers warn about it until it is replaced, which can be done two ways:

  • By hand. Copy the certificate (its chain in the same file) and the key to /home/admin on a node, then run import tls-cert <crt> <key> on that node's console. Repeat it on every node. The appliance does not renew it: show health warns 30 days before it ends and turns critical 7 days before.
  • By CertAutoPilot itself. It issues the certificate, distributes it to every node over SSH and distributes it again at every renewal. The steps are in Manage CAP's own TLS certificate.

An HSM for the key-encryption key​

The cluster's key-encryption key can live in an HSM (Thales Luna, Fortanix DSM) instead of each node's secrets.env. It is chosen when the cluster is formed, and the HSM's client is installed on every node before it forms or joins. CertAutoPilot OS with an HSM walks through it node by node.

Versions​

Every member runs the same CertAutoPilot release. When a joining node runs a different one, what happens depends on how the cluster got its release:

  • The cluster was updated with system app update (or import app-release) on its leader. The leader keeps the signed files of that release, so the joining node fetches them from the leader, verifies the signature itself against the pinned release key, and installs the release before anything starts. Nothing needs to be done by hand.
  • The cluster still runs the release its image shipped, or the release was installed before the cluster was formed. The leader has no files to hand over, so the join is refused with the version it needs:
join: … version mismatch: the joining node runs CertAutoPilot 1.5.55, this cluster 1.5.56 —
bring that node to 1.5.56 first: 'system app update 1.5.56' or 'import app-release <file>' …

In that case, bring the joining node to the cluster's release first, then join:

node 2> system app update 1.5.56
node 2> cluster join 10.0.0.11 capjoin1.…

system app update installs only releases whose checksums are signed by the CertAutoPilot release key; on the leader of a formed cluster it updates every member. Without internet access, import app-release <file> takes a release brought in by hand, as shown below.

Updating without internet access​

An appliance that cannot reach ghcr.io installs a release from files you bring to it. On a machine with internet access, pull the release with oras; it is the bundle system app update downloads:

$ VERSION=1.7.3
$ oras pull ghcr.io/cloudnativeworks/standalone/certautopilot:${VERSION}

Three of the files it writes make up the update: the archive (certautopilot_<version>_linux_amd64.tar.gz), its checksums (certautopilot_<version>_checksums.txt) and their signature (certautopilot_<version>_checksums.txt.asc). Copy them, named as they are, to the home directory of admin on the leader:

$ scp certautopilot_${VERSION}_linux_amd64.tar.gz \
certautopilot_${VERSION}_checksums.txt \
certautopilot_${VERSION}_checksums.txt.asc admin@10.0.0.11:

Then name the archive on the leader's console (or on a node that is not in a cluster yet); the other two files are found beside it:

cap> import app-release /home/admin/certautopilot_1.7.3_linux_amd64.tar.gz
…
Install CertAutoPilot 1.7.3? (y/N): y
…
CertAutoPilot 1.7.3 is active on every member; the previous release was removed
cap> system app status
active: CertAutoPilot 1.7.3 (an installed release)
factory: CertAutoPilot 1.7.1 (shipped with this image)
on disk: 1.7.3 (active), signed files kept for joining nodes
cluster: every member runs 1.7.3

From there it is the update system app update performs. The signature is checked against the release key pinned in the image — a file that was changed, or a release that is not newer than the running one, is refused and nothing is installed — a backup is taken, and the members are switched one at a time. The other members take the files from the leader, so only the leader needs them.

A new release in the web UI​

Once a day the application reads the same release list system app update uses (the release tags on ghcr.io). When a newer release is there, org owners and org admins see x.y.z ready beside the version at the top of the sidebar; pointing at it shows the console command (system app update x.y.z) and a link to the release notes. Nothing is installed from the web UI. An appliance without internet access shows nothing — the check stays quiet and tries again the next day.

Checking the result​

cap> show cluster # every member, its replica-set state, the leader, the VIP
cap> show health # the one screen to read first: anything red is named

The web UI's Settings → Cluster page lists the same three instances, the database primary and which instance holds the scheduler, discovery and domain leader roles (one instance at a time, elected through the database).

Changing an address later​

A member's management address is its name in the replica set, in every peer's connection string and in its node certificate, so it cannot change in place: cluster remove <node> on the leader, cluster reset on that node, the new address (set network static …, apply, network confirm), then a fresh cluster join.