Tasks die permanently after 3 consecutive failures — a transient network blip kills intake until a process restart #14
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Background tasks that hit three consecutive failures are marked
deadand never retried until the process restarts.src/tasks.rs(spawn_supervised, ~lines 186–200):The backoff is correct and capped at 60s — but at three strikes the supervisor
returns out of its loop and the task is gone for the life of the process./healthzthen reportsdegraded/503indefinitely.Consequence observed
Tenant
gal, appconstructionbot, 2026-10-28/29:login.microsoftonline.com(Azure IP rotation vs. a stale egress allowlist) hit three consecutive mailbox polls within ~35 seconds.Why this is the wrong trade
The intent (make a dead task visible to an operator) is sound. The mechanism is not: a transient fault and a permanent fault are indistinguishable here, and both produce the same permanent outcome. Network egress, a provider's API, and DNS are all failure domains that recover on their own — exactly the cases where a supervisor should keep trying.
Proposed fix
Keep retrying with capped backoff instead of returning:
failures >= MAX_FAILURES: log at error once per crossing, set aregistry.mark_degraded(name)state that/healthzstill surfaces as unhealthy (preserving operator visibility), and continue looping with backoff pinned at the 60s cap rather than exiting.dead, but transport-level errors (HTTP send, DNS, connect) should never escalate.Test impact (why this needs a plan, not a patch)
Existing tests pin the current contract deliberately:
src/tasks.rsmodule doc: "3 consecutive failures → task markeddead"supervised_failing_task_goes_dead(~line 372) polls until the task is dead and asserts itsupervised_task_stays_alive-family tests/healthzdegradedsemantics testsThose expectations must change with the behavior, so this wants a short plan + critic pass rather than a silent edit. The visibility guarantee ("an operator can tell a task is unhealthy") must survive the change — that is the acceptance criterion, not "the task never dies."
Verification
/healthzreports it unhealthy throughout.aliveafter a single transient failure.