Rollback

One uniform rollback mechanism: re-deploy a previous retained certificate version through the module's normal deploy path. Eligibility rules, the version picker, the rollback API, auto-rollback with flap guard, retention pinning, and the sticky rolled_back status.

How rollback works

A rollback re-deploys a previous retained certificate version — an artifact from the certificate's per-renewal version history (v1, v2, …) — through the module's normal deploy path. There is no separate rollback machinery per module: rolling back to v3 runs exactly the same Execute logic as deploying v3 in the first place, just with older material.

This replaces the old per-module rollback mechanisms entirely: the SSH .bak file restore, the Kubernetes store_content snapshot, the Vault KV version-restore, the Exchange previous-thumbprint re-enable, and the MerlinCDN conditional previous-cert re-bind are all removed. Any module that can deploy a certificate can re-deploy an older one — which is why rollback coverage is now all seventeen modules (the two push modules, Webhook and SMTP, re-send the previous version).

During a rollback run, delete_old_cert / remove_old_cert_after_bind behaviors are suppressed: the newer certificate stays on the remote, so rolling forward again is instant.

Capability by module

All seventeen modules support manual rollback (the two push modules re-send the previous version); auto-rollback is the fifteen stateful modules only (see below). The "what a rollback does" column is simply the module's normal deploy, fed the previous version:

Module	Rollback	What a rollback re-deploy does
SSH	Yes	Re-writes the previous cert/key files at the PathSet paths and re-runs the ActionSet.
Kubernetes	Yes	Re-applies the previous cert into the TLS Secret(s) and re-runs restart targets.
NetScaler	Yes	Updates the sslcertkey in place to the previous version.
Huawei Cloud	Yes	Re-uploads/re-binds the previous cert on ELB/WAF/CDN.
F5 BIG-IP	Yes	Updates the cert/key objects and every managed clientssl profile back to the previous version.
Vault	Yes	Writes the previous cert as a new KV version.
IIS	Yes	Re-imports the previous PFX into the certificate store and repoints the bindings.
WinRM (generic)	Yes	Re-writes files via the PathSet and re-runs the ActionSet.
AWS ACM	Yes	Re-imports the previous cert to the ACM ARN.
MerlinCDN	Yes	Re-uploads the previous cert and, on a binding target, re-binds it. An upload-only target (bind: false) has no CertAutoPilot-managed binding, so a rollback only re-uploads.
Exchange	Yes	Re-imports and re-enables the previous cert on the bound services.
Cloudflare	Yes	Re-uploads the previous cert to the zone. Additional eligibility rule: the version must have at least 14 days of validity left (below).
Azure Key Vault	Yes	Imports the previous version into the Key Vault certificate object.
cPanel / WHM	Yes	Re-installs the previous cert onto each domain (install replaces in place).
PAN-OS	Yes	Re-imports the previous version under the same object name and commits — bindings never move.
Webhook	Yes	Re-POSTs the previous retained version to the endpoint. Note: this re-sends the older cert; it cannot un-call the POST that already fired — the receiver processes the re-delivered payload.
SMTP	Yes	Re-emails the previous retained version to the recipients. Note: this re-sends the older cert as a fresh email; the already-sent one cannot be recalled.

Per-version eligibility

A previous version is eligible for rollback when:

• it is still retained — within the history retention setting (a version a distribution still references is pinned and never pruned);
• it is not revoked — a version revoked at the CA is excluded, whether it was revoked through CertAutoPilot's version-revoke API or out-of-band at the CA (the periodic revocation check also sweeps retained previous versions and flags CA-side revocations automatically);
• it is not expired;
• its private key is still retained — the check resolves the version's key fingerprint against the certificate's retained private-key rows, and that lookup uses the same 100-version ceiling as the artifact history, so a version is never reported key_missing merely because its key row fell outside a shorter page;
• Cloudflare only: it has ≥ 14 days of remaining validity — the same min_validity_floor Cloudflare imposes on any custom-certificate upload.

The rollback-candidates API returns a reason code for each ineligible version: current, revoked, expired, min_validity_floor, key_missing, module_not_capable.

A distribution's computed rollback_available field is: module is rollback-capable AND at least one previous version is eligible AND the distribution has deployed before AND the certificate is active AND no fan-out is in progress (see Why the Rollback button is greyed out). The UI shows the Rollback button only when it is true.

Manual rollback & the version picker

1. Open the certificate → Distribution tab.
2. On a distribution row with rollback available, click Rollback.
3. A version picker opens: a radio list of retained versions, newest first. Ineligible rows are disabled with their reason. Pick a version and confirm.

The artifact history table on the certificate detail page — the Certificates tab of the bottom card — shows a Version column (v1, v2, …) so you can correlate what you are rolling back to.

API

List candidates (newest first). Requires the project viewer role — the same floor as any other project read:

GET /api/v1/projects/{projectId}/certificates/{id}/distributions/{distId}/rollback-candidates

The listing covers the newest 100 retained versions. The retention setting itself caps at 100, so this ceiling only bites under unlimited retention (-1): on such a certificate, versions older than the newest 100 do not appear in the response or in the version picker.

{
    "candidates": [
        {
            "id": "9f2c4a1e-...",
            "certificate_id": "d41b0c88-...",
            "version": 3,
            "serial_number": "04:AB:...",
            "fingerprint_sha256": "5e884898da...",
            "key_fingerprint": "2c26b46b68...",
            "not_after": "2026-10-01T12:00:00Z",
            "created_at": "2026-07-03T09:15:00Z",
            "is_current": true,
            "eligible": false,
            "reason": "current"
        },
        {
            "id": "7d0e2b93-...",
            "certificate_id": "d41b0c88-...",
            "version": 2,
            "serial_number": "03:9F:...",
            "fingerprint_sha256": "6b86b273ff...",
            "key_fingerprint": "d4735e3a26...",
            "not_after": "2026-09-12T08:00:00Z",
            "created_at": "2026-06-01T09:15:00Z",
            "is_current": false,
            "eligible": true
        }
    ]
}

Trigger the rollback (project Operator role or higher):

POST /api/v1/projects/{projectId}/certificates/{id}/distributions/{distId}/rollback

{ "artifact_id": "7d0e2b93-..." }

The JSON body is optional — an empty or omitted body rolls back to the newest eligible previous version. The call is rate-limited to 5 requests per minute per project (shared with Execute), and returns 409 when rollback is not available.

The candidates listing does not check certificate status

rollback-candidates evaluates versions only — retention, expiry, private key, module capability. It never looks at the certificate's status. The certificate-status guard lives in the rollback call alone: it rejects any certificate that is not active (renewal-transient statuses excepted). So a revoked or expired certificate returns 409 from POST .../rollback even though its candidates list showed eligible versions. That is expected, not a bug in the listing.

Auto-rollback on failure

Every distribution has a per-distribution auto_rollback_on_failure toggle (off by default), available on the fifteen stateful modules. The two push modules (Webhook, SMTP) are excluded from auto-rollback — the system automatically re-sending an old certificate on a transient failure (SMTP would re-email it to recipients, unrecallable) is a surprise, and there is no partially-updated target state for auto-rollback to make consistent. Manual rollback stays available on all seventeen modules. Auto-rollback fires when:

• a normal run finishes failed or partial, and
• at least one target actually changed during that run.
• at least one failed target could actually have written something. A run whose only failures were SSH host-key verification — where the handshake is refused and nothing is sent — does not trigger auto-rollback, so one rebuilt server cannot drag its healthy neighbours back to an older certificate.

It picks the newest eligible previous version and rolls the distribution back to it. Auto-rollback is a blast-radius limiter for mixed fleets: if 3 of 40 hosts got the new cert before the run failed, rollback returns them to the previous, still-consistent state.

Flap guard: while the auto-rollback condition holds, the failed run is not auto-retried — the system never ping-pongs between re-pushing the new version and rolling it back.

Fan-out exclusion: auto-rollback triggers only for normal (non-fanned) runs. A distribution large enough to fan out into batch children never auto-rolls back — re-deploying the previous version while sibling batches are still pushing the new one would race. Roll such a distribution back manually after the fan-out completes.

If the rollback itself fails: the job retries transient failures up to 3 attempts, then parks as last_rollback_status=failed with a notification — no new auto-rollback is triggered and the sweep is not re-armed, so a failing rollback can never loop. The deployed-version fields are left untouched (the remote may hold a mix), and the operator can re-trigger from the version picker after fixing the cause.

After a rollback

A successful rollback sets the distribution status to rolled_back, and this state is sticky: the distribution sweep will not re-push the current version while the distribution is rolled back, so a rollback is never silently undone. The next renewal (a new fingerprint) rolls forward normally.

Because old-cert cleanup was suppressed during the rollback, the newer certificate is still present on the remote — rolling forward again (manually or at the next renewal) is instant.

Rollbacks are observable:

• the certificate timeline records the run's ordinary per-target entries — cert.distributed on success, cert.distribution_failed on failure. These are not marked as a rollback; because a rollback re-runs the normal deploy path, the timeline shows a distribution, not a distinct rollback event. Correlate by the job id, or by the distribution's rolled_back status;
• dedicated distribution.rollback_success / distribution.rollback_failed events are emitted to notification channels only — they do not appear on the certificate timeline;
• the Prometheus counter certautopilot_distribution_rollback_total increments;
• post-distribution validation endpoints run after a rollback too — they validate the rolled-back-to version's fingerprint, so a rollback gets the same "is it actually being served" confirmation as a deploy.

Retention pinning

The history retention setting prunes old artifact versions — but pruning pins any version a distribution still references. A referenced version is never pruned, even when it is past the retention count, so a rolled-back distribution's version (and its private key) cannot be deleted out from under it.

Migration notes

If you are upgrading from a release with the old per-module rollback mechanisms:

• SSH no longer creates .bak backup files. Existing .bak files on hosts are inert leftovers from older versions — safe to delete manually.
• Vault rollback writes the old cert content as a new KV version. It does not rewind Vault's version counter, and there is no KV v2 requirement anymore.
• The Kubernetes rollback.store_content option is removed. Rollback works without any opt-in.
• Exchange and MerlinCDN rollback no longer depend on delete_old_cert having been off. The previous version is re-deployed from CertAutoPilot's own artifact history, not recovered from the remote.
• Old last_rollback_meta data in MongoDB is ignored. No migration is needed.

Troubleshooting

Rollback button missing

rollback_available is false: no previous version is eligible — check the rollback-candidates API for the per-version reason (expired, key_missing, min_validity_floor, …). Raising the history retention count keeps more versions eligible going forward.

Rollback API returns 409

Rollback is not available for this distribution — same causes as above, plus: the certificate is no longer active (revoked/expired certificates cannot be rolled back). The candidates endpoint explains the per-version reasons, but it does not evaluate certificate status — a 409 with an all-eligible candidates list means the certificate itself is the blocker.

Rollback API returns 422

Either a rollback for this distribution is already pending or running (only one rollback runs at a time — wait for it to finish), or a fan-out is in progress (retry after it completes).

Old certificates accumulating on devices

Rollback deliberately never deletes the displaced newer version (that is what makes rolling forward instant). On stores with named objects this leaves inert leftovers over repeated rollback cycles: Windows certificate store entries (IIS/Exchange), F5 versioned cert/key objects, MerlinCDN unbound certificate uploads. They never serve traffic; clean them up manually when convenient.

A version I want is not in the picker

It was pruned by history retention (only versions still referenced by a distribution are pinned). Pruned versions cannot be recovered; raise History Versions in the Certificate Defaults card of Settings → General (API field cert_history_max_versions) to retain more versions in the future.

See also

• Distribution overview — rollback
• Certificate version history & download-by-version
• Post-distribution validation
• Fan-out execution

Why the Rollback button is greyed out

rollback_available is false — and rollback_unavailable_reason says which of these applies:

Reason	Meaning
module_not_capable	The module type does not support rollback.
no_eligible_version	Every retained previous version failed the per-version checks below.
never_deployed	This distribution has not deployed anything yet, so there is nothing to go back to.
cert_not_active	The certificate is not in an active state.
fan_out_in_progress	A fan-out run is still going; rollback is refused until it settles.
unknown	Eligibility could not be evaluated (a storage read failed) — retry.

If a rollback job dies

A rollback whose job is killed mid-flight used to leave the distribution reading Rolling back… forever and pinned its artifact against retention. The scheduler now reconciles it on the next cycle: it inspects the job and closes the state out as failed, cancelled or completed, releasing the retention pin.