Skip to main content

Troubleshooting

Symptom-driven catalog. Find the failure mode, jump to the fix. Index of the things that go wrong in production CertAutoPilot installs.

Install & bootstrap​

Installer fails: "port 443 in use"​

Another web server is bound. sudo systemctl stop apache2 / nginx, or re-run the installer with --port=8443.

First-run setup already completed​

First-run setup is a one-shot API call (POST /api/v1/setup) guarded by a cluster-wide flag — there is no setup token, no certautopilot setup command and no reset. Once the first admin exists, the endpoint returns 400 setup already completed.

If you have lost every admin account, reset a user's password from the host instead:

sudo certautopilot user list --org-id=<org>
sudo certautopilot user reset-password --username=<user> --password='<new>'

Failing that, restore from backup.

MongoDB won't start​

Check journalctl -u mongod first. The usual cause is ownership on the distribution's own data directory — /var/lib/mongodb on Debian/Ubuntu, /var/lib/mongo on RHEL — which must belong to mongodb:mongodb. MongoDB's data is not under /var/lib/certautopilot, and there is no certautopilot setup command to repair it.

ACME & CA​

ACME account registration fails​

The backend cannot reach the directory URL. Test from the host:

curl -sI https://acme-v02.api.letsencrypt.org/directory

Proxy support is asymmetric today: issuance, renewal, ARI refresh and revocation honour HTTPS_PROXY, but account registration does not — its HTTP client is built without a proxy, so setting HTTPS_PROXY will not help here. Register the account from a host with direct egress to the CA.

"Too many certificates already issued"​

You hit Let's Encrypt's per-registered-domain or duplicate-cert weekly cap. Switch to staging while iterating, and consolidate redundant cert requests.

EAB binding error​

The CA returned urn:ietf:params:acme:error:externalAccountRequired or unauthorized. Double-check the kid and HMAC: each EAB pair is single-use, and many providers regenerate them on the fly. See EAB-bound CAs.

DNS-01​

DNS challenge never resolves​

See DNS-01 troubleshooting. The most common cause is a stale TXT record with a long TTL.

"Insufficient DNS permissions"​

The credential's scope is narrower than the zone. Re-issue the credential with the minimum permission set per provider.

Distribution​

Fan-out partial failure​

Some targets succeeded, others failed. Open the certificate's Distribution tab — each target shows its own status and logs. Common causes: target offline, credential expired, post-deploy reload command non-zero.

Rollback failed​

A rollback is a normal deploy of a previous retained certificate version, so it fails for the same reasons a deploy does (target unreachable, credential expired, post-deploy action non-zero) — check the per-target logs on the Distribution tab and re-trigger. If the API returns 409, rollback is not available: the module is not rollback-capable or no previous version is eligible — the rollback-candidates endpoint reports the per-version reason.

Audit​

Audit verify reports broken chain​

This is serious. Either a bug or tampering. First capture the broken entry's id and the previous entry's hash, then escalate.

Audit-chain verification is an API call, not a CLI command:

curl -X POST -b cookies.txt -H "X-CSRF-Token: <token>" \
  https://<host>/api/v1/audit-logs/verify

Forward the JSON to support@cloudnativeworks.com along with the deployment version.

Upgrade​

Upgrade failed part-way​

There are no schema migrations to fail — indexes are ensured idempotently at startup and the few backfills are additive and safe to repeat. Re-running the same upgrade is therefore safe and is the recommended fix. If the service still will not start, check journalctl -u certautopilot: the common causes are a missing KEK version and a config value the new build validates more strictly.

KEK​

env KEK provider: version N not loaded​

The process is reading a record sealed under a KEK version it does not hold. Restarting does not help, and the version is almost never a new one: processes adopt a newly activated version on their next heartbeat without a restart, and a running process never loses a version it already loaded. It is an old version that was never provisioned here, or one the keystore has marked removed.

Check which: certautopilot kek status.

  • The version is missing from the fleet → add CERTAUTOPILOT_ENCRYPTION_ENV_KEK_V{N} (Kubernetes: the encryption-env-kek-v{N} Secret key) and roll the pods.
  • The version shows as removed → its material is deliberately ignored even when present. Run certautopilot kek reinstate --version=N, restart, then rotate those records forward.

The sibling message version N is marked removed — cannot unwrap is the same second case, reported at unwrap time.

See also​