Skip to main content
Video tutorial · 1:50

Rollback

One uniform rollback mechanism: re-deploy a previous retained certificate version through the module's normal deploy path. Eligibility rules, the version picker, the rollback API, auto-rollback with flap guard, retention pinning, and the sticky rolled_back status.

How rollback works​

A rollback re-deploys a previous retained certificate version — an artifact from the certificate's per-renewal version history (v1, v2, …) — through the module's normal deploy path. There is no separate rollback machinery per module: rolling back to v3 runs exactly the same Execute logic as deploying v3 in the first place, just with older material.

This replaces the old per-module rollback mechanisms entirely: the SSH .bak file restore, the Kubernetes store_content snapshot, the Vault KV version-restore, the Exchange previous-thumbprint re-enable, and the MerlinCDN conditional previous-cert re-bind are all removed. Any module that can deploy a certificate can re-deploy an older one — which is why rollback coverage is now all twenty-one modules (the two push modules, Webhook and SMTP, re-send the previous version).

During a rollback run, delete_old_cert / remove_old_cert_after_bind behaviors are suppressed: the newer certificate stays on the remote, so rolling forward again is instant.

Capability by module​

All twenty-one modules support manual rollback (the two push modules re-send the previous version); auto-rollback is the nineteen stateful modules only (see below). The "what a rollback does" column is simply the module's normal deploy, fed the previous version:

ModuleRollbackWhat a rollback re-deploy does
SSHYesRe-writes the previous cert/key files at the PathSet paths and re-runs the ActionSet.
KubernetesYesRe-applies the previous cert into the TLS Secret(s) and re-runs restart targets.
NetScalerYesUpdates the sslcertkey in place to the previous version.
Huawei CloudYesRe-uploads/re-binds the previous cert on ELB/WAF/CDN.
F5 BIG-IPYesUpdates the cert/key objects and every managed clientssl profile back to the previous version.
VaultYesWrites the previous cert as a new KV version.
IISYesRe-imports the previous PFX into the certificate store and repoints the bindings.
WinRM (generic)YesRe-writes files via the PathSet and re-runs the ActionSet.
AWS ACMYesRe-imports the previous cert to the ACM ARN.
MerlinCDNYesRe-uploads the previous cert and, on a binding target, re-binds it. An upload-only target (bind: false) has no CertAutoPilot-managed binding, so a rollback only re-uploads.
ExchangeYesRe-imports and re-enables the previous cert on the bound services.
CloudflareYesRe-uploads the previous cert to the zone. Additional eligibility rule: the version must have more than 14 days of validity left (below).
Azure Key VaultYesImports the previous version into the Key Vault certificate object.
cPanel / WHMYesRe-installs the previous cert onto each domain (install replaces in place).
PAN-OSYesRe-imports the previous version under the same object name and commits — bindings never move.
FortiWebYes8.0.7+: updates the same object in place back to the previous version. Earlier firmware: points the SNI members and multi-certificate entries back at the previous version's object (re-imported if it was deleted). Nothing is deleted during a rollback; no change drops a connection.
A10 ThunderYesUploads the previous version under the same file name — every template and virtual port keeps serving it. Nothing is deleted during a rollback.
FortiGateYesUpdates the same local certificate in place with the previous version — every VIP, SSL inspection profile and SSL VPN reference stays. Nothing is deleted during a rollback.
ElchiYesWrites the previous version into the same secrets (by resource id) and publishes it.
WebhookYesRe-POSTs the previous retained version to the endpoint. Note: this re-sends the older cert; it cannot un-call the POST that already fired — the receiver processes the re-delivered payload.
SMTPYesRe-emails the previous retained version to the recipients. Note: this re-sends the older cert as a fresh email; the already-sent one cannot be recalled.

Per-version eligibility​

A previous version is eligible for rollback when:

  • it is still retained — within the history retention setting (a version a distribution still references is pinned and never pruned);
  • it is not revoked — a version revoked at the CA is excluded, whether it was revoked through CertAutoPilot's version-revoke API or out-of-band at the CA (the periodic revocation check also sweeps retained previous versions and flags CA-side revocations automatically);
  • it is not expired;
  • its private key is still retained — the check resolves the version's key fingerprint against the certificate's retained private-key rows, and that lookup uses the same 100-version ceiling as the artifact history, so a version is never reported key_missing merely because its key row fell outside a shorter page;
  • Cloudflare only: it has more than 14 days of remaining validity (min_validity_floor) — a conservative floor: Cloudflare may refuse shorter-lived certificates when it builds the chain itself, though a bundle_method: force target would accept them.

The rollback-candidates API returns a reason code for each ineligible version: current, revoked, expired, min_validity_floor, key_missing, module_not_capable.

A distribution's computed rollback_available field is: module is rollback-capable AND at least one previous version is eligible AND the distribution has deployed before AND the certificate is active AND no fan-out is in progress (see Why the Rollback button is greyed out). The UI shows the Rollback button only when it is true.

Manual rollback & the version picker​

  1. Open the certificate → Distribution tab.
  2. On a distribution row with rollback available, click Rollback.
  3. A version picker opens: a radio list of retained versions, newest first. Ineligible rows are disabled with their reason. Pick a version and confirm.

The artifact history table on the certificate detail page — the Certificates tab of the bottom card — shows a Version column (v1, v2, …) so you can correlate what you are rolling back to.

API​

List candidates (newest first). Requires the project viewer role — the same floor as any other project read:

GET /api/v1/projects/{projectId}/certificates/{id}/distributions/{distId}/rollback-candidates

The listing covers the newest 100 retained versions. The retention setting itself caps at 100, so this ceiling only bites under unlimited retention (-1): on such a certificate, versions older than the newest 100 do not appear in the response or in the version picker.

{
"candidates": [
{
"id": "9f2c4a1e-...",
"certificate_id": "d41b0c88-...",
"version": 3,
"serial_number": "04:AB:...",
"fingerprint_sha256": "5e884898da...",
"key_fingerprint": "2c26b46b68...",
"not_after": "2026-10-01T12:00:00Z",
"created_at": "2026-07-03T09:15:00Z",
"is_current": true,
"eligible": false,
"reason": "current"
},
{
"id": "7d0e2b93-...",
"certificate_id": "d41b0c88-...",
"version": 2,
"serial_number": "03:9F:...",
"fingerprint_sha256": "6b86b273ff...",
"key_fingerprint": "d4735e3a26...",
"not_after": "2026-09-12T08:00:00Z",
"created_at": "2026-06-01T09:15:00Z",
"is_current": false,
"eligible": true
}
]
}

Trigger the rollback (project Operator role or higher):

POST /api/v1/projects/{projectId}/certificates/{id}/distributions/{distId}/rollback
{ "artifact_id": "7d0e2b93-..." }

The JSON body is optional — an empty or omitted body rolls back to the newest eligible previous version. The call is rate-limited to 5 requests per minute per project (shared with Execute), and returns 409 when rollback is not available.

A distribution whose SSH or WinRM targets (or its target overrides) run an ActionSet is the exception to the Operator rule: rolling it back by hand needs the Admin role, because the rollback runs those commands again. An Operator gets 403 ACCESS_DENIED — or, when the project's approval policy gates distribution actions for that Operator, an approval request that an admin decides on.

The candidates listing does not check certificate status

rollback-candidates evaluates versions only — retention, expiry, private key, module capability. It never looks at the certificate's status. The certificate-status guard lives in the rollback call alone: it rejects any certificate that is not active (renewal-transient statuses excepted). So a revoked or expired certificate returns 409 from POST .../rollback even though its candidates list showed eligible versions. That is expected, not a bug in the listing.

Auto-rollback on failure​

Every distribution has a per-distribution auto_rollback_on_failure toggle (off by default), available on the nineteen stateful modules. The two push modules (Webhook, SMTP) are excluded from auto-rollback — the system automatically re-sending an old certificate on a transient failure (SMTP would re-email it to recipients, unrecallable) is a surprise, and there is no partially-updated target state for auto-rollback to make consistent. Manual rollback stays available on all twenty-one modules. Auto-rollback fires when:

  • a normal run finishes failed or partial, and
  • at least one target actually changed during that run.
  • at least one failed target could actually have written something. A run whose only failures were SSH host-key verification — where the handshake is refused and nothing is sent — or IIS binding-name refusals (IIS_NAME_MISMATCH, nothing imported or bound), or FortiWeb refusals made before anything is written (FORTIWEB_CHAIN_GROUP_MISSING, FORTIWEB_CONFLICT, FORTIWEB_AUTH, and FORTIWEB_VALIDATION / FORTIWEB_PLACEMENT when the target reports nothing changed), or FortiGate refusals made before anything is written (FORTIGATE_CONFLICT, FORTIGATE_PLACEMENT, FORTIGATE_AUTH, FORTIGATE_PERMISSION, and FORTIGATE_VALIDATION when the target reports nothing changed) does not trigger auto-rollback, so one rebuilt server cannot drag its healthy neighbours back to an older certificate.

It picks the newest eligible previous version and rolls the distribution back to it. Auto-rollback is a blast-radius limiter for mixed fleets: if 3 of 40 hosts got the new cert before the run failed, rollback returns them to the previous, still-consistent state.

Flap guard: while the auto-rollback condition holds, the failed run is not auto-retried — the system never ping-pongs between re-pushing the new version and rolling it back.

Fan-out exclusion: auto-rollback triggers only for normal (non-fanned) runs. A distribution large enough to fan out into batch children never auto-rolls back — re-deploying the previous version while sibling batches are still pushing the new one would race. Roll such a distribution back manually after the fan-out completes.

If the rollback itself fails: the job retries transient failures up to 3 attempts, then parks as last_rollback_status=failed with a notification — no new auto-rollback is triggered and the sweep is not re-armed, so a failing rollback can never loop. The deployed-version fields are left untouched (the remote may hold a mix), and the operator can re-trigger from the version picker after fixing the cause.

After a rollback​

A successful rollback sets the distribution status to rolled_back, and this state is sticky: the distribution sweep will not re-push the current version while the distribution is rolled back, so a rollback is never silently undone. The next renewal (a new fingerprint) rolls forward normally.

Because old-cert cleanup was suppressed during the rollback, the newer certificate is still present on the remote — rolling forward again (manually or at the next renewal) is instant.

Rollbacks are observable:

  • the certificate timeline records the run's ordinary per-target entries — cert.distributed on success, cert.distribution_failed on failure. These are not marked as a rollback; because a rollback re-runs the normal deploy path, the timeline shows a distribution, not a distinct rollback event. Correlate by the job id, or by the distribution's rolled_back status;
  • dedicated distribution.rollback_success / distribution.rollback_failed events are emitted to notification channels only — they do not appear on the certificate timeline;
  • the Prometheus counter certautopilot_distribution_rollback_total increments;
  • post-distribution validation endpoints run after a rollback too — they validate the rolled-back-to version's fingerprint, so a rollback gets the same "is it actually being served" confirmation as a deploy.

Retention pinning​

The history retention setting prunes old artifact versions — but pruning pins any version a distribution still references. A referenced version is never pruned, even when it is past the retention count, so a rolled-back distribution's version (and its private key) cannot be deleted out from under it.

Migration notes​

If you are upgrading from a release with the old per-module rollback mechanisms:

  • SSH no longer creates .bak backup files. Existing .bak files on hosts are inert leftovers from older versions — safe to delete manually.
  • Vault rollback writes the old cert content as a new KV version. It does not rewind Vault's version counter, and there is no KV v2 requirement anymore.
  • The Kubernetes rollback.store_content option is removed. Rollback works without any opt-in.
  • Exchange and MerlinCDN rollback no longer depend on delete_old_cert having been off. The previous version is re-deployed from CertAutoPilot's own artifact history, not recovered from the remote.
  • Old last_rollback_meta data in MongoDB is ignored. No migration is needed.

Troubleshooting​

Rollback button missing​

rollback_available is false: no previous version is eligible — check the rollback-candidates API for the per-version reason (expired, key_missing, min_validity_floor, …). Raising the history retention count keeps more versions eligible going forward.

Rollback API returns 409​

Rollback is not available for this distribution — same causes as above, plus: the certificate is no longer active (revoked/expired certificates cannot be rolled back). The candidates endpoint explains the per-version reasons, but it does not evaluate certificate status — a 409 with an all-eligible candidates list means the certificate itself is the blocker.

Rollback API returns 422​

Either a rollback for this distribution is already pending or running (only one rollback runs at a time — wait for it to finish), or a fan-out is in progress (retry after it completes).

Old certificates accumulating on devices​

Rollback deliberately never deletes the displaced newer version (that is what makes rolling forward instant). On stores with named objects this leaves inert leftovers over repeated rollback cycles: Windows certificate store entries (IIS/Exchange), F5 versioned cert/key objects, MerlinCDN unbound certificate uploads. They never serve traffic; clean them up manually when convenient.

A version I want is not in the picker​

It was pruned by history retention (only versions still referenced by a distribution are pinned). Pruned versions cannot be recovered; raise History Versions in the Certificate Defaults card of Settings → General (API field cert_history_max_versions) to retain more versions in the future.

See also​

Why the Rollback button is greyed out​

rollback_available is false — and rollback_unavailable_reason says which of these applies:

ReasonMeaning
module_not_capableThe module type does not support rollback.
no_eligible_versionEvery retained previous version failed the per-version checks below.
never_deployedThis distribution has not deployed anything yet, so there is nothing to go back to.
cert_not_activeThe certificate is not in an active state.
fan_out_in_progressA fan-out run is still going; rollback is refused until it settles.
unknownEligibility could not be evaluated (a storage read failed) — retry.

If a rollback job dies​

A rollback whose job is killed mid-flight used to leave the distribution reading Rolling back… forever and pinned its artifact against retention. The scheduler now reconciles it on the next cycle: it inspects the job and closes the state out as failed, cancelled or completed, releasing the retention pin.

Video tutorial

Short, hands-on videos showing this feature working for real. English audio, English and Turkish subtitles. All video tutorials →

Roll Back a Certificate Deployment and Roll ForwardYouTube ↗