Skip to main content

Backup & restore

Two things must be backed up together: MongoDB (all the data) and the secret store (the KEK that can decrypt it). Losing either makes the other useless. Every backup strategy in this doc backs up both at the same point in time and stores them off-host.

What to back up​

ComponentStandalone pathKubernetes source
MongoDBmongodump — as capApp from secrets.env on a single node; as capRoot from mongo-root.env, full-instance with --oplog, on a replica setmongodump against the chart's MongoDB service
Backend secret store/etc/certautopilot/secrets.envThe K8s Secret referenced in values.yaml
TLS material (the frontend cert)/etc/certautopilot/tls/The Ingress TLS Secret
Config/etc/certautopilot/config.yamlvalues.yaml in your GitOps repo
(PKCS#11 only) HSM key materialHSM vendor's backup procedureHSM vendor's backup procedure

Standalone — nightly automated backup​

The bootstrap ships a systemd timer + service, but both are opt-in — pass --enable-backup at install time (it is off by default so operators with their own backup orchestration — restic, borg, Ansible cron, etc. — aren't surprised by a second schedule). The flag is only honored together with --mongo=local; external deployments own their backup stack regardless of the flag.

curl -fsSL https://raw.githubusercontent.com/CloudNativeWorks/certautopilot-archive/main/get.sh \
  | sudo bash -s -- --version=<pinned> --mongo=local --enable-backup

# Verify the timer is armed
systemctl list-timers | grep certautopilot

Rerunning the bootstrap without --enable-backup disables the timer (the flag is the single source of truth for the installer's intent); upgrade.sh detects the timer's pre-existing enabled state and preserves it across version bumps.

The timer's service runs /usr/local/bin/certautopilot-backup, which:

  1. mongodump --gzip --archive against the local MongoDB using the capApp credentials from secrets.env.
  2. Copies /etc/certautopilot/secrets.env (and mongo-root.env when present) alongside the dump — the dump is useless ciphertext without the KEK, bundling them together makes restore a single-archive affair.
  3. Writes the bundle atomically as /var/backups/certautopilot/certautopilot-backup-<YYYYMMDD-HHMMSS>.tar.gz (mode 0600, root-owned).
  4. Rotates archives older than ${CAP_BACKUP_RETAIN_DAYS:-7} days — override via a systemd drop-in (systemctl edit certautopilot-backup.service) adding Environment=CAP_BACKUP_RETAIN_DAYS=30.

Off-host storage is your responsibility — sync the directory to S3 / Azure Blob / rsync to a separate host. Local retention alone does not protect against hardware loss on the backup host itself.

Standalone — on-demand​

sudo /usr/local/bin/certautopilot-backup

Produces the same single .tar.gz under /var/backups/certautopilot/ with a fresh timestamp. Use before every upgrade and before every major config change. Installed only when --enable-backup was passed — without it, run mongodump + snapshot secrets.env manually, or re-run the bootstrap with the flag.

Kubernetes backup​

No built-in timer. Three common patterns:

Velero​

Snapshots the whole namespace including the MongoDB PVC and the Secret. The backup's atomicity depends on storage-class snapshot semantics — on a CSI driver that supports it, you get a coherent point-in-time. Configure velero schedules as usual.

CronJob + mongodump + kubectl​

apiVersion: batch/v1
kind: CronJob
metadata:
  name: certautopilot-backup
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: mongo:7.0
              command:
                - /bin/sh
                - -c
                - |
                  mongodump --uri="$MONGO_URI" --archive --gzip > /backup/mongo-$(date -I).archive.gz &&
                  kubectl get secret cap-secrets -o yaml > /backup/secret-$(date -I).yaml &&
                  aws s3 cp /backup/ s3://my-bucket/certautopilot/ --recursive
              envFrom: [ { secretRef: { name: backup-env } } ]
              volumeMounts: [ { name: scratch, mountPath: /backup } ]
          volumes: [ { name: scratch, emptyDir: {} } ]

Needs RBAC for the CronJob's ServiceAccount to read the Secret. Rotate the S3 access key regularly.

External orchestration​

If you already have a backup platform (Kasten, TrilioVault, Portworx), point it at the namespace. Same principle: Mongo + Secret in the same snapshot.

Restore​

Restore MongoDB​

# Single-node archive (scoped to the app database, no oplog):
mongorestore --uri="$MONGO_URI" --drop --gzip --archive=mongo-dump.gz \
  --nsInclude='certautopilot.*'

# Replica-set archive (FULL instance, taken with --oplog):
mongorestore --uri="$MONGO_URI" --drop --gzip --archive=mongo-dump.gz --oplogReplay

Use the form that matches how the archive was taken — the nightly job produces two different shapes:

  • Single node: a dump of the certautopilot database only. Restore it scoped with --nsInclude.
  • Replica set (any --nodes install): a full-instance dump taken as capRoot, with --oplog. It must be restored with --oplogReplay — without it the oplog the backup deliberately captured is ignored and the restore lands at the moment the dump started, which is exactly the cross-collection inconsistency --oplog exists to prevent. Note this archive contains databases beyond certautopilot (admin included), so --drop reaches further than the app database.

--drop wipes the target collections before restoring.

Restore the secret store​

  • Standalone: copy the backed-up secrets.env to /etc/certautopilot/ (mode 0600, owner certautopilot:certautopilot).
  • Kubernetes: kubectl apply -f secret-backup.yaml. Pod rolling restart picks up the rendered env vars.

Restore config + TLS​

Copy config.yaml back; copy tls/. Restart the service (standalone) or kubectl rollout restart (K8s).

Start the service​

On first startup after a restore, the service loads the KEK material (env vars, or the HSM keys under pkcs11), verifies it against the kek_install lock, and checks that the version the restored keystore marks active actually has its material loaded. If it doesn't, startup aborts with an error naming the missing version — better than silently running on the wrong KEK.

Two things that check deliberately does not cover on the env provider:

  • It never decrypts stored data. A clean startup proves the active KEK is present, not that every record can be opened.
  • It says nothing about retired versions. If the restored database still holds records stamped with an older version whose env var is absent, the service starts perfectly healthy and those records fail only at the moment something reads them. After a restore, compare certautopilot kek status against secrets.env: every version the keystore lists as active or retired should have a matching CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V* line.

The pkcs11 provider is stricter — it loads every non-removed version from the keystore and refuses to start if any of their HSM keys is missing or renamed.

Partial restores are the risky ones

A full mongorestore --drop paired with the archive's own secrets.env is self-consistent, and a mismatch is caught loudly at startup. A scoped restore (--nsInclude, say to recover certificate_private_keys after a fat-fingered delete) is not: it reintroduces records stamped with an old KEK version without touching kek_versions or secrets.env, so nothing at startup notices. Before a scoped restore, confirm the key material for that era is still loaded.

Restore drill​

Run one quarterly:

  1. Spin up a scratch environment (Compose stack, empty MongoDB).
  2. Restore the latest backup.
  3. Log in with a known user.
  4. View a cert, trigger a renewal, watch the job complete.
  5. Diff the restored DB's cert count against production. Small drift (last hour's issuances) is expected; orders-of-magnitude drift means something's wrong with the backup.

Document the drill in your ops runbook. Auditors love it; production saves you next time it matters.

Retention strategy​

  • Daily: last 7 days on-host, last 30 off-host.
  • Weekly: 3 months.
  • Monthly: 12 months.
  • Align with your compliance requirement (SOC 2 often asks for 1-year retrievable; PCI-DSS is shorter).

Encrypted backups stored on an S3 bucket with server-side encryption + versioning + object lock give you both defence-in-depth (encryption at two layers) and ransomware resistance (object lock prevents deletion).

After a KEK rotation​

Backups taken before a KEK rotation are paired with the old secret-store. Keep the old secret-store alongside those backups until they age out of your retention policy — otherwise, an old-dated restore is unrecoverable. See KEK rotation.

Troubleshooting​

The restored database and the KEK material do not match​

There is no fingerprint check, and the two directions do not behave the same way — one stops the service, the other does not.

The service refuses to start when the version it must write with is unusable: keystore active version is vN but no key material is loaded for it, current KEK version N is removed or missing — cannot continue, or kek_versions has rows but none is marked active — the keystore is mid-transition or inconsistent.

It starts normally when only an older version is missing, logging KEK version N is "retired" in the keystore but no key material is loaded. Nothing else signals the problem: reads of records sealed under that version fail one at a time, whenever they happen. Run certautopilot kek status after every restore and confirm each active/retired row has loaded material.

A third case has nothing to do with the material: KEK provider mismatch: installed=… but config says …. The provider is fixed at install time, so a database installed under env cannot be brought up under pkcs11 or the reverse.

Restoring also replaces the keystore​

mongorestore --drop replaces kek_versions and kek_install too, so the restored keystore's active / retired / removed flags win over whatever the running fleet believed. A version the restored keystore marks removed is ignored even when its key material is still present on the host.

Restore a dump taken after a kek remove while you still hold that key, and the data it protects becomes unreadable until you undo the removal:

certautopilot kek reinstate --version=N     # removed → retired

Then restart every node so the key loads again, and move the affected records forward with certautopilot kek rotate --from-version=N --to-version=<current>. If a later kek remove refuses because some documents carry no recorded version, run certautopilot kek repair-versions --dry-run first and inspect what it would stamp.

"Database restored but UI shows empty lists"​

Wrong database name in the URI. mongodump's archive format preserves the original DB name; if your target environment uses a different name, run mongorestore --nsInclude='oldname.*' --nsFrom='oldname.*' --nsTo='newname.*'.

Mongo + secret store timestamps off by hours​

Your backup script doesn't snapshot them atomically. Fix the script: always take both in the same run, preferably against a quiesced DB (stop the backend for the duration, or use Mongo's read-replica for the dump).

See also​