Skip to main content

External secret stores

Some organisations rotate their infrastructure passwords continuously, often through a privileged-access product, and cannot copy a value into a second system where it would immediately be out of date. An external secret store solves that: the credential in CertAutoPilot holds only a reference, and the value is read from your secret manager at the moment it is needed.

Two consequences follow directly, and both are the point rather than side effects:

  • Rotating the value in your store takes effect immediately. There is no copy here to update, no "next renewal" delay, and no window in which CertAutoPilot deploys a password your systems no longer accept.
  • CertAutoPilot stores no encrypted value for that credential. A database backup no longer contains it, and neither does an export. The trade is that a deployment now depends on the store being reachable — see When the store is unavailable.
Scope

This applies to distribution target credentials only — the auth material modules use to reach hosts and devices (SSH, WinRM, F5, NetScaler, Vault, cloud APIs). ACME account keys, DNS provider credentials, AD CS connections, notification channels and project variables are unaffected and are still stored encrypted by CertAutoPilot.

Supported stores​

StoreStatus
HashiCorp Vault (KV v1 / v2)Supported
OpenBaoSupported — API-compatible with Vault; configure it as a Vault store
ManageEngine PAM360Supported — account passwords via the REST API

Everything product-specific lives inside the store implementation, so further products can be added later without changing how existing stores or credentials are stored.

How it fits together​

A secret store is its own object, shared by every credential that reads through it — so an address change, or rotating the login CertAutoPilot uses, is one edit rather than one per credential.

Secret store  "corp-vault"          →  address, TLS, KV mount, auth credential
     ▲
     │  referenced by
     │
Module credential "prod-f5-admin"   →  store + path + field   (no value stored)
     ▲
     │  referenced by
     │
Target  "f5-prod-01"

Manage stores under Settings → Distribution → Secret Stores.

Setting up a store​

Each store product has its own walkthrough — what to prepare on the store's side, which login CertAutoPilot needs, and how a credential addresses a value:

  • HashiCorp Vault / OpenBao — the minimum read-only policy, KV v1 vs v2 paths, namespaces, AppRole vs token login, path + field addressing.
  • ManageEngine PAM360 — the host-bound API user, the bootstrap-token exchange, resource sharing, resource + account addressing.

Both end the same way: add a credential with Secret Source set to External secret store, pick the store, and fill its reference fields. The Credential Type is chosen exactly as for a stored credential, and every module behaves identically — where a value comes from is independent of what it is.

Field mapping per credential type​

One field holds the whole value. For a credential type that expects a single string (SSH password, bearer token, SMTP password, an SSH private key in PEM form), put that string in the field. For a type that expects a JSON object, store the complete JSON object as the value of that one field:

Credential typeWhat to store in the field
SSH password, SMTP password, SMTP PKCS#12 passphrase, bearer token, HMAC secretthe string itself
SSH private key, kubeconfigthe full PEM / YAML text
F5 / NetScaler password{"username":"...","password":"..."}
WinRM, IIS, Exchange{"username":"...","password":"..."} — the authentication type (NTLM/Basic/Kerberos) is configured on the target, not inside the credential. IIS alone also accepts an optional "auth_type" of ntlm or basic in the credential (default ntlm); WinRM and Exchange ignore it in favour of the target's setting.
AWS AK/SK{"access_key_id":"...","secret_access_key":"..."}
Huawei AK/SK{"access_key":"...","secret_key":"..."} — add "domain_id":"..." for CDN targets, which require it
MerlinCDN token{"personal_access_token":"...","organization_id":"...","workspace_id":"..."} — all three required
Cloudflare API token{"api_token":"..."}
Azure service principal{"tenant_id":"...","client_id":"...","client_secret":"..."}
Vault token / AppRole{"token":"..."} / {"role_id":"...","secret_id":"..."}
The shape is not validated when you save

A stored credential's JSON is checked at save time. A store-backed one is not — its content lives in your secret manager and can change at any moment, so checking it now would prove nothing about what a deployment will actually read. A malformed or incomplete value therefore surfaces as a deployment failure, not as a rejected save. Use Test on the credential row after setting it up: it reports which keys the value contains.

A string field is passed through byte-for-byte, so a PEM key keeps its exact formatting. If the store holds a non-string value (a nested object or a number) it is serialised to JSON.

To check what CertAutoPilot actually receives, use Test on the credential row. It reports the shape — whether the value is JSON and which keys it contains, plus the store revision it read — and never the value itself.

Reading behaviour​

Every resolution reads the store. Ten targets sharing one credential produce ten reads; a second deployment reads again. Nothing is cached, because a cached value is exactly the stale value this feature exists to avoid.

What is reused within a single run is the authenticated connection: one login serves all the reads in that run, so a 500-target distribution does not create 500 Vault tokens.

A distribution large enough to be split into batches runs each batch as its own job, and each job logs in once — so the number of logins follows the number of batches, not the number of targets. Size any Vault token-count or login rate limits on that basis.

Two consequences worth knowing:

  • Within one job the value cannot change mid-run. Every credential is resolved before the first target is touched, so all targets in that job use the same value. A large distribution split into batches does resolve once per batch, so a rotation between batches can leave different batches on different values — re-run the distribution after a rotation if every target must be on the same one.
  • The store sees one read per credential slot per target. Most targets use one slot, but some use two (a webhook with both a signing secret and an auth credential, or SMTP with a relay password and a PKCS#12 passphrase). Size any request limits and audit retention accordingly — the store's own audit log will show these reads.

For Vault KV v2, the secret's revision is recorded in each deployment's job log. That is the only drift signal available without exposing anything about the value, and it lets you compare two runs and see that the underlying secret changed. KV v1 and PAM360 have no revision concept, so those reads are logged as (no revision).

Backup and disaster recovery​

A CertAutoPilot backup no longer contains these credential values — only the reference and the store's connection details. Restoring CertAutoPilot alone is therefore not enough to resume distributions: the secret manager must also be available, holding the same paths. Include the store in your recovery plan, and keep the login credential (token or AppRole) recoverable independently, since it is the one piece that still lives here.

When the store is unavailable​

If the store cannot be reached, or the reference no longer resolves, the distribution fails before any target is touched — CertAutoPilot does not deploy with a partial or guessed credential. Certificate issuance and renewal are unaffected; only distribution depends on the store.

The resolution phase is bounded by the same Distribution execution timeout (Settings → General) that bounds the deployment itself — each gets its own window of that size. A store that accepts connections but never answers therefore fails the job within that timeout instead of hanging it indefinitely.

The failure appears in the usual places (job status, last error, notifications) with enough detail to act on, and never includes secret material:

resolving target credentials: credential 68f0… (slot: default):
  credential "prod-f5-admin": secret store "corp-vault" (vault):
  reading secret/apps/f5/prod: read secret: HTTP 403

There is no new error code for this. The failure happens before any per-target result exists, so it is reported at the job level rather than against an individual target — a target-level code could never be produced.

A distribution is not retried — at any size

Every distribution job gets a single attempt, whether it is a small run or one batch of a large fan-out. A store outage that outlasts the one immediate in-process retry therefore fails the distribution outright, and it has to be re-run.

That immediate retry covers anything that might genuinely differ on a second attempt: a 5xx from the store, and every failure that never reached an HTTP status at all — a connection error, a timeout, an unparseable response, or a login that came back without a token. A 4xx is not retried, because repeating it would not change the answer.

Deletion and safety rules​

  • Deleting a secret store is refused while any credential still reads through it — those credentials would break at their next deployment, and the reference could not be repaired afterwards because the store's address and login would be gone with it.
  • Deleting the store's auth credential is refused while a store uses it.
  • A store and the credentials referencing it must belong to the same project. This is enforced when the reference is saved and again every time it is resolved.
  • A store's auth credential must be a stored credential, never another store-backed one.
  • Secret Source cannot be changed after creation. Switching a live credential between stored and store-backed changes its security posture silently; create a new credential and repoint the targets instead.

Creating, editing, deleting and testing a secret store requires admin, exactly as for module credentials. Reading the list is available to any project role — a store document holds no secret, only an address, its settings, and a reference to the credential it authenticates with.

Monitoring the store​

Because a deployment now depends on the store, alert on the store directly rather than waiting for deployments to fail. Three Prometheus series are exported on the usual metrics endpoint:

MetricLabelsUse it for
certautopilot_secret_store_read_totalstore_type, outcomeoutcome="error" is an outage; outcome="not_found" is a reference an operator must correct. They are separated on purpose — treating them alike makes the error rate meaningless.
certautopilot_secret_store_read_duration_secondsstore_typeCatches a store that is up but slow, which otherwise stays invisible until it crosses the request timeout.
certautopilot_secret_store_retry_totalstore_typeCounts the immediate second attempts (reads and AppRole logins alike; only transient failures qualify — see the retry note above). A rising count means the store is already failing a share of requests while deployments still succeed — the earliest warning available.

The store's name is deliberately not a label: it is operator-chosen text and would make the series unbounded. Use the logs to identify which store.

Every successful read is logged at info with the credential name, the store, the revision and the duration — never the value. Each deployment's job log also records the revision of every credential it read, so comparing two runs shows whether the underlying secret changed between them. If a run fails partway, the reads that already succeeded are still recorded.

Who can cause a secret to be read​

This is the one thing to decide before adopting store-backed credentials, because it works differently from a stored one.

CertAutoPilot fetches the value and hands it to whatever host the target names. With a stored credential that value is a fixed copy, so rotating the secret in your systems retires it. With a store-backed credential every run fetches the current value — which is the point, and also means that anyone who can run a distribution against a target receives the live secret each time. Rotating it does not take that access back.

Two controls decide who that is:

ControlDefaultWhat it gates
Require approval for configuration changesOnAn operator creating or editing a target — the moment a credential is bound to a host — becomes an approval request an admin reviews.
Require approval for distributionOffAn operator running a distribution against a target that already exists.

With the defaults, an operator cannot point a credential at a host of their choosing without an admin approving it — but once such a target exists, they can run it as often as they like. The approval screen flags any change that binds a store-backed credential, so review that warning on its merits: approving the target is the decision, not each run.

If operators can create targets, turn distribution approval on too

Enable Require approval for distribution (Settings → General → Approval Requirements) whenever store-backed credentials are in use and operators hold target-write access. Note that it gates operator-initiated distributions only — the automatic post-renewal sweep runs unattended by design, so a target that is already approved keeps deploying.

Both controls are bypassed by admins and owners, who are trusted with the credential itself.

What is not supported​

  • On Vault/OpenBao, anything other than the KV engine — no PKI, database or transit engine as a credential source.
  • Vault auth methods other than token and AppRole (no Kubernetes, JWT, userpass, or Vault Agent).
  • On PAM360, anything other than an account's password — no file/license-store attachments or SSH-key objects.
  • Caching, or any option to enable it.
  • Placeholders or templates inside the secret path — the path is literal.
  • Mapping several fields into one credential; exactly one field supplies the value.
  • Changing a credential's source after it is created.

See also​