Fan-out execution
How large distributions are split into parallel batch child jobs: the fan-out threshold, per-target retry with error classification, aggregate status computation, and the dedicated distribution worker lane.
What fan-out is
Normally one distribution_execute job pushes the certificate to all targets of a distribution. With hundreds of targets that single job becomes a long-running, all-or-nothing unit: one worker holds it for the whole run, and a crash loses all progress.
Fan-out replaces the single job with N batch child jobs, each carrying a snapshot of target IDs. Children run in parallel on the distribution worker pool, report their result to the parent distribution document, and the final status is computed atomically once every child has finalized.
When it triggers
Fan-out activates when both are true:
- Fan-out Threshold (
distribution_fanout_thresholdin Settings → General) is greater than0. The default is0= disabled — every distribution runs as a single job regardless of target count. - The distribution's resolved target count is >= threshold.
When it triggers, targets are split into batches of Fan-out Batch Size (distribution_fanout_batch_size, default 50 when unset). All batch children are enqueued first; only then is fan_out_total set on the distribution. If any batch fails to enqueue, the whole fan-out is aborted — there is no partial enqueue, and the scheduler sweep retries later.
A fan-out run has no single parent job. The UI shows progress by polling the distribution's fan-out counters, and each child appears as its own job under Jobs.
Execution flow
Child job modes (the mode field of the distribution_execute payload):
| Mode | Payload | Purpose |
|---|---|---|
"" (normal) | all targets | No fan-out — single job, takes the per-cert/module distribution lock. |
batch | target_id_batch | Fan-out child with a deterministic target-ID snapshot. Runs without the distribution lock so children can execute in parallel. |
retry | retry_target_ids, retry_attempt | Per-target retry of failed targets from a previous child. |
Targets deleted between the snapshot and execution are reported as skipped ("target deleted after snapshot") instead of failing the batch.
Per-target retry
When a batch or retry child finishes partial or failed, its failed targets are classified by error class and only transient classes are retried:
| Error class | Examples | Retried? |
|---|---|---|
network | timeout, connection refused, DNS resolution | yes |
io_transient | broken pipe, connection reset, transient SFTP, script exit non-zero | yes |
io_permanent | permission denied on remote file, no space left | no |
auth | SSH/API auth failure, credential rejected | no |
validation | cert/key mismatch, invalid path, missing target | no |
| unknown / unclassified | — | no |
Retryable targets are collected (sorted, deduplicated) into a new mode=retry child. The retry wave counter (fan_out_retry_wave) is allocated atomically on the distribution; after 3 waves no further retries are enqueued. While a retry child is outstanding, fan_out_retrying > 0 blocks aggregate completion.
Aggregate counters & final status
Each child's terminal result increments exactly one counter on the CertificateDistribution document — fan_out_succeeded, fan_out_failed, fan_out_partial, or fan_out_cancelled — via an atomic $inc with an $addToSet duplicate guard on fan_out_finalized_jobs (a child job can never be counted twice).
When succeeded + failed + partial + cancelled >= fan_out_total and fan_out_retrying == 0, an atomic aggregation-pipeline update computes the final status:
| Condition | Final status |
|---|---|
| all children cancelled | cancelled |
| zero children succeeded | failed |
| zero failed, zero partial, zero cancelled | success |
| anything else | partial |
A child that exits without reporting (worker crash, cancel, timeout) is caught by a deferred safety net that records it as cancelled, so the parent never waits forever.
Fan-out children do not emit per-batch notifications. The child whose report finalizes the aggregate emits a single distribution success/failure notification for the whole fan-out, keyed off the aggregate's final status.
Dedicated worker lane
In --mode=worker (and all) two workers run side by side:
- the main worker handles ACME/MSCA/notification/discovery jobs and explicitly excludes distribution job types;
- a dedicated distribution worker claims only
distribution_executeanddistribution_rollbackjobs.
A burst of fan-out children therefore cannot starve certificate issuance or renewal jobs, and vice versa.
Monitoring progress
On the certificate's Distribution tab, a fan-out run shows a progress bar with Total, Succeeded, Failed, Partial, Cancelled counters, the current retry wave, and the list of finalized child jobs (each linking to its job log). Per-target results from all children are appended to the same distribution record, so the target results table covers the whole fleet.
Settings
All three knobs live in Settings → General, distribution section:
| Setting | Field | Range | Default | Effect |
|---|---|---|---|---|
| Fan-out Threshold | distribution_fanout_threshold | 0–500 | 0 (disabled) | Target count at/above which distribution splits into child jobs. |
| Fan-out Batch Size | distribution_fanout_batch_size | 0–200 | 0 → 50 | Targets per child job. Smaller values = more parallel jobs. |
| SSH Max Concurrency | distribution_ssh_max_concurrency | 0–50 | 0 → module config default (10) | System-level override for concurrent SSH sessions within one job. |
Batch size controls parallelism across jobs; SSH concurrency controls parallelism within one job. With threshold 100, batch size 50, and SSH concurrency 10, a 500-target distribution becomes 10 child jobs, each opening up to 10 SSH sessions at a time.
Troubleshooting
Distribution stuck "running" although all child jobs finished
If the final TryCompleteAggregate write fails transiently (e.g. a MongoDB hiccup at the exact moment the last child reports), the aggregate can be left uncompleted. The scheduler's distribution sweep runs a reconciliation pass (ReconcileStuckFanOuts) every scheduler cycle that finds distributions where all children have finalized but the status is not terminal, and completes them. No manual action needed — wait one scheduler cycle (default hourly, driven by the renewal check interval).
Children enqueued but total never set
If setting fan_out_total fails after all children were enqueued, the run is treated as failed and the certificate keeps distribution_pending; the scheduler sweep re-triggers the distribution. Idempotency keys on the batch children prevent duplicate execution.
Targets keep failing without retry
Only network and io_transient errors are retried. auth, validation, and io_permanent failures (wrong credential, cert/key mismatch, permission denied) will not resolve on their own — fix the cause and hit Retry on the distribution.