Fan-out execution
How large distributions are split into parallel batch child jobs: the fan-out threshold, per-target retry with error classification, aggregate status computation, and the dedicated distribution worker lane.
What fan-out is
Normally one distribution_execute job pushes the certificate to all targets of a distribution. With hundreds of targets that single job becomes a long-running, all-or-nothing unit: one worker holds it for the whole run, and a crash loses all progress.
Fan-out replaces the single job with N batch child jobs, each carrying a snapshot of target IDs. Children run in parallel on the distribution worker pool, report their result to the parent distribution document, and the final status is computed atomically once every child has finalized.
When it triggers
Fan-out activates when both are true:
- Fan-out Threshold (
distribution_fanout_thresholdin Settings → General) is greater than0. The default is0= disabled — every distribution runs as a single job regardless of target count. - The distribution's resolved target count is >= threshold.
When it triggers, targets are split into batches of Fan-out Batch Size (distribution_fanout_batch_size, default 50 when unset) — or fewer, see Batches that fit the execution timeout. All batch children are enqueued first; only then is fan_out_total set on the distribution. If any batch fails to enqueue, the whole fan-out is aborted — there is no partial enqueue, and the scheduler sweep retries later.
A fan-out run has no single parent job. The UI shows progress by polling the distribution's fan-out counters, and each child appears as its own job under Jobs.
Execution flow
Child job modes (the mode field of the distribution_execute payload):
| Mode | Payload | Purpose |
|---|---|---|
"" (normal) | all targets | No fan-out — single job, takes the per-cert/module distribution lock. When a system-started run (renewal, re-import, sweep) finds another run of the same distribution holding that lock — typically the previous version still deploying when a renewal enqueues the new one — the job does not fail: it completes without deploying, logs that it was deferred, and re-arms the distribution for the next distribution sweep, which pushes the new version once the earlier run has finished. A manual Execute that meets the held lock still fails visibly ("lock is held") and is simply re-run, because the sweep applies the enabled / auto-trigger / schedule / rolled-back rules a manual run is meant to bypass; a rollback keeps retrying instead. |
batch | target_id_batch | Fan-out child with a deterministic target-ID snapshot. Runs without the distribution lock so children can execute in parallel. |
retry | retry_target_ids, retry_attempt | Per-target retry of failed targets from a previous child. |
lock_retry flag | lock_retry (on a batch payload) | Not a mode of its own — a marker on the one delayed re-enqueue a batch child gets after waiting out the whole device-lock window. |
Device-lock contention is not a failure. Fan-out groups targets by device affinity so one device's targets land in one child — this applies to the appliance and Windows modules (F5, NetScaler, PAN-OS, FortiWeb, FortiGate, A10 Thunder, IIS, WinRM, Exchange); SSH targets are not grouped and take no device lock, so two SSH targets on the same host can land in different children and run concurrently. Children still contend for a device lock when another distribution holds the same appliance or Windows host. A child that waits the full lock window without acquiring it is re-enqueued once with a 5-minute delay (lock_retry) rather than failing terminally — the retry child reports the batch's result in the original child's stead, so the aggregate still completes. A second wait timeout on the lock_retry child is terminal, which keeps contention from looping.
Targets deleted between the snapshot and execution are reported as skipped ("target deleted after snapshot") instead of failing the batch. Because the run then mixes success and skipped, its overall status settles as partial — expected, not a fault.
Per-target retry
When a batch or retry child finishes partial or failed, its failed targets are classified by error class and only transient classes are retried:
| Error class | Examples | Retried? |
|---|---|---|
network | timeout, connection refused, DNS resolution | yes |
io_transient | broken pipe, connection reset, transient SFTP, script exit non-zero | yes |
io_permanent | permission denied on remote file, no space left | no |
auth | SSH/API auth failure, credential rejected | no |
validation | cert/key mismatch, invalid path, missing target | no |
| unknown / unclassified | — | no |
Retryable targets are collected (sorted, deduplicated) into a new mode=retry child. The retry wave counter (fan_out_retry_wave) is allocated atomically on the distribution; after 3 waves no further retries are enqueued. While a retry child is outstanding, fan_out_retrying > 0 blocks aggregate completion.
Per-target retry covers failures during a target's deployment. Resolving the batch's credentials happens first, so a failure there — most visibly an external secret store being unreachable — fails the whole child with no retry, because a batch child gets a single attempt. A single transient network error or 5xx from the store is retried once immediately; a longer outage needs the distribution to be re-run.
Aggregate counters & final status
Each child's terminal result increments exactly one counter on the CertificateDistribution document — fan_out_succeeded, fan_out_failed, fan_out_partial, or fan_out_cancelled — via an atomic $inc with an $addToSet duplicate guard on fan_out_finalized_jobs (a child job can never be counted twice).
When succeeded + failed + partial + cancelled >= fan_out_total and fan_out_retrying == 0, an atomic aggregation-pipeline update computes the final status:
| Condition | Final status |
|---|---|
| all children cancelled | cancelled |
| zero children succeeded | failed |
| zero failed, zero partial, zero cancelled | success |
| anything else | partial |
A child that exits without reporting (worker crash, cancel, timeout) is caught by a deferred safety net that records it as cancelled, so the parent never waits forever.
Fan-out children do not emit per-batch notifications. The child whose report finalizes the aggregate emits a single distribution success/failure notification for the whole fan-out, keyed off the aggregate's final status.
Dedicated worker lane
In --mode=worker (and all) two workers run side by side:
- the main worker handles ACME/MSCA/notification/discovery jobs and explicitly excludes distribution job types;
- a dedicated distribution worker claims only
distribution_executeanddistribution_rollbackjobs.
A burst of fan-out children therefore cannot starve certificate issuance or renewal jobs, and vice versa.
Monitoring progress
On the certificate's Distribution tab, a fan-out run shows a progress bar with Total, Succeeded, Failed, Partial, Cancelled counters, the current retry wave, and the list of finalized child jobs (each linking to its job log). Per-target results from all children are appended to the same distribution record, so the target results table covers the whole fleet.
Settings
These knobs live in Settings → General, distribution section:
| Setting | Field | Range | Default | Effect |
|---|---|---|---|---|
| Fan-out Threshold | distribution_fanout_threshold | 0–500 | 0 (disabled) | Target count at/above which distribution splits into child jobs. |
| Fan-out Batch Size | distribution_fanout_batch_size | 0–200 | 0 → 50 | Targets per child job. Smaller values = more parallel jobs. |
| SSH Max Concurrency | distribution_ssh_max_concurrency | 0–50 | 0 → module config default (10) | System-level override for concurrent SSH sessions within one job. |
| Parallel Windows Hosts | distribution_windows_max_concurrency | 0–32 | 0 → 8 | How many distinct hosts an IIS deploy works on at once within one job. Targets on the same host always run one after another. Parallel execution switched off forces 1. |
Batch size controls parallelism across jobs; SSH concurrency controls parallelism within one job. With threshold 100, batch size 50, and SSH concurrency 10, a 500-target distribution becomes 10 child jobs, each opening up to 10 SSH sessions at a time.
Batches that fit the execution timeout
Each child job has one Distribution Execution Timeout (default 30 minutes) for all its targets. IIS and Exchange report how long one target may take and how many hosts they work on at once, and fan-out then shrinks the batch so it fits:
batch = min(Fan-out Batch Size, ⌊0.8 × execution timeout ÷ slowest target's budget⌋ × hosts at once)
With the defaults an IIS target may take 330 s (five times the 60 s command timeout plus 30 s) and 8 hosts run at once, so a batch holds at most ⌊0.8 × 1800 ÷ 330⌋ × 8 = 32 targets. The reduction is logged on the server; the configured value is not changed.
When fan-out is off and one run's targets could need longer than the execution timeout, the job log warns before the deploy starts ("… may need up to … but the execution timeout is …"); the run itself is not changed. Turn on fan-out or raise the execution timeout — targets still running when it ends fail.
Troubleshooting
Distribution stuck "running" although all child jobs finished
If the final TryCompleteAggregate write fails transiently (e.g. a MongoDB hiccup at the exact moment the last child reports), the aggregate can be left uncompleted. The scheduler's distribution sweep runs a reconciliation pass (ReconcileStuckFanOuts) every scheduler cycle that finds distributions where all children have finalized but the status is not terminal, and completes them. No manual action needed — wait one scheduler cycle (default hourly, driven by the renewal check interval).
Children enqueued but total never set
If setting fan_out_total fails after all children were enqueued, the run is treated as failed and the certificate keeps distribution_pending; the scheduler sweep re-triggers the distribution. Idempotency keys on the batch children prevent duplicate execution.
Targets keep failing without retry
Only network and io_transient errors are retried. auth, validation, and io_permanent failures (wrong credential, cert/key mismatch, permission denied) will not resolve on their own — fix the cause and hit Retry on the distribution.
See also
Video tutorial
Short, hands-on videos showing this feature working for real. English audio, English and Turkish subtitles. All video tutorials →