Skip to main content

Upgrade

CertAutoPilot upgrades in place. The binary is the only moving part; state lives in MongoDB and the secret store, both of which are preserved across upgrades. Index migrations run at startup. Rollback is "run the previous binary" — no schema migrations to reverse.

Version compatibility​

  • Patch bumps (e.g. 1.4.0 → 1.4.x) — always safe. Bug fixes + security patches only.
  • Minor bumps (1.3 → 1.4) — safe. Additive schema / API changes only.
  • Major bumps (1.x → 2.0) — read the CHANGELOG. May require ordered steps or a data migration script.

Deprecations are announced in the release notes; there is no deprecation response header.

Standalone upgrade​

Use update.sh when you only want to bump the version. It pulls the pinned tarball, verifies the checksum, and hands off to the bundled upgrade.sh. Binary, frontend, and systemd unit (with rollback snapshot) refresh only — config.yaml, secrets.env (including HSM PIN), TLS material, nginx config, and MongoDB all stay untouched.

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/update.sh \
  | sudo bash -s -- --version=1.4.5

Use get.sh instead when you also need to change install flags (--mongo, --tls, --kek-provider, --port, --extra-hostnames, …). install.sh is idempotent — it re-renders what the new flags affect while preserving every secret on disk.

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=1.4.5 --mongo=local --extra-hostnames=cap.new-domain.test

Or, if you already downloaded the tarball:

VERSION=1.4.5
oras pull ghcr.io/cloudnativeworks/standalone/certautopilot:${VERSION}
tar -xzf certautopilot_${VERSION}_linux_amd64.tar.gz
cd certautopilot_${VERSION}_linux_amd64
sudo ./upgrade.sh

upgrade.sh stops the service, atomically replaces the binary (mv onto the old path), refreshes the frontend assets under /usr/share/certautopilot/web/, takes a systemd unit snapshot + rolls back if the new binary fails to start, refreshes the journald retention drop-in, the cap / cap-kek / cap-cluster wrappers, and the KEK rotation runbook, then bounces the service. config.yaml, secrets.env, TLS, nginx config, and MongoDB stay untouched. Zero-downtime is NOT guaranteed on a single host: expect a few seconds when idle, and up to 30 s (the unit's TimeoutStopSec) when a job is in flight — after which systemd sends SIGKILL and the job is reclaimed on the 5-minute lock TTL.

Helm upgrade​

helm upgrade certautopilot oci://ghcr.io/cloudnativeworks/charts/certautopilot \
  --version 1.4.0 \
  -f values.yaml
kubectl rollout status deployment/certautopilot --timeout=5m

The chart performs a rolling restart. With ≥ 2 API replicas and maxUnavailable: 25% (the default), there's no downtime. Scheduler leader gracefully hands off. Workers get 55 s to drain, but the chart's terminationGracePeriodSeconds: 30 cuts that short — a job running longer than 30 s is killed and reclaimed after the 5-minute lock TTL.

Breaking changes​

When a breaking change is necessary, we ship it across two minor releases:

  1. 1.X: the new behaviour lands behind a feature flag. Old behaviour is default. Deprecation warnings flow into logs + API responses. Docs explain how to migrate.
  2. 1.Y (at least one quarter later): the old behaviour is removed. Operators who migrated during 1.X are unaffected.

This cadence lets you opt-in on your own schedule rather than being forced by the release train.

CGO / base image changes​

The Phase 2 PKCS#11 work required CGO, which in turn required switching the container base image from Alpine (musl) to Debian bookworm (glibc). No functional change for users — but be aware if you layer your own tooling on the image: apk add won't work; use apt install. Image size is larger by ~40 MB.

MongoDB version bumps​

  • Within a major (6.0 → 6.0.x), just upgrade Mongo.
  • Across majors (6 → 7): follow MongoDB's official upgrade path. CertAutoPilot supports 6.0+.
  • Bundled Bitnami chart in Helm: helm upgrade with a new mongodb.image.tag value handles the rolling step.

KEK during upgrade​

Syslog over mutual TLS: upgrade the fleet close together

The syslog TLS material moves into its own record the moment the first upgraded node starts. Nodes still on the previous version cannot see it there: they report no certificates and, if syslog is enabled over TLS, their own forwarding fails until they are upgraded. Audit events that exhaust their retries during that window are lost. On a multi-node fleet, either upgrade the nodes back to back or disable syslog forwarding for the duration.

An in-flight KEK rotation mid-upgrade is safe: the rotation handler is resumable. If the worker pod running the rotation restarts, another worker picks it up from the last batch boundary. Monitoring: kek status reports pending, in_progress, completed, completed_with_errors, failed or cancelled — there is no pause. Treat completed_with_errors as unfinished: some records were skipped and nothing retries them, so re-run the rotation before retiring the old version.

Rollback​

Standalone​

# Pull the older version's tarball, then
sudo ./upgrade.sh   # the script is version-agnostic; it runs whatever binary is in the dir

MongoDB data is forward-compatible within a major; running an older binary against a newer DB state usually works. The CHANGELOG lists incompatibilities (e.g. a new required field introduced in 1.4 would break a 1.3 rollback).

Helm​

helm rollback certautopilot 1

Helm keeps revision history. Previous pod image rolls out. Same compatibility caveat as standalone.

Index creation on startup​

The backend ensures indexes at startup via database.EnsureIndexes(). This is idempotent (safe to run repeatedly) and usually fast, but the first startup after a major upgrade can take tens of seconds on large collections while indexes build. Plan your restart window accordingly.

Pre-upgrade checklist​

  1. Read the CHANGELOG for the target version. Note any breaking changes.
  2. Back up MongoDB + the secret store. Always both, at the same point-in-time.
  3. Schedule the upgrade during a lull — renewals queuing during a restart is fine, but avoid stacking a KEK rotation and a version bump.
  4. Verify rollback path — can you pull the old tarball / image back?
  5. Alert your ops channel. Expect a short banner during the restart.

Troubleshooting​

Service hangs on startup after upgrade​

Usually an index build on a very large collection. Check mongodb logs; currentOp shows the build. Patience. If truly stuck for > 30 min, stop the service, drop the in-progress index manually (safe — EnsureIndexes will recreate it), restart.

Helm upgrade silently didn't re-roll pods​

No change in the rendered Secret checksum. Use kubectl rollout restart deployment/<release> to force a rollout.

See also​