Tasks die permanently after 3 consecutive failures — a transient network blip kills intake until a process restart #14

Open
opened 2026-11-02 17:54:16 +00:00 by gal-bot · 0 comments
Owner

Problem

Background tasks that hit three consecutive failures are marked dead and never retried until the process restarts.

src/tasks.rs (spawn_supervised, ~lines 186–200):

Ok(Err(e)) => {
    failures += 1;
    tracing::error!(task = name, error = %e, failures, "supervised task iteration failed");
    if failures >= MAX_FAILURES {          // MAX_FAILURES = 3 (line 28)
        registry.mark_dead(name).await;
        return;                            // <- supervisor exits permanently
    }
    tokio::time::sleep(backoff).await;
    backoff = (backoff * 2).min(Duration::from_secs(60));
    continue;
}

The backoff is correct and capped at 60s — but at three strikes the supervisor returns out of its loop and the task is gone for the life of the process. /healthz then reports degraded / 503 indefinitely.

Consequence observed

Tenant gal, app constructionbot, 2026-10-28/29:

  • A transient transport failure to login.microsoftonline.com (Azure IP rotation vs. a stale egress allowlist) hit three consecutive mailbox polls within ~35 seconds.
  • The mailbox task was marked dead at 22:08Z. Client mail intake — the app's primary input path — was dead for ~19 hours.
  • Nothing retried. Nothing recovered. The container ran healthy in every other respect; only a manual restart revived the task.
  • A brief network blip became an all-day intake outage.

Why this is the wrong trade

The intent (make a dead task visible to an operator) is sound. The mechanism is not: a transient fault and a permanent fault are indistinguishable here, and both produce the same permanent outcome. Network egress, a provider's API, and DNS are all failure domains that recover on their own — exactly the cases where a supervisor should keep trying.

Proposed fix

Keep retrying with capped backoff instead of returning:

  • On failures >= MAX_FAILURES: log at error once per crossing, set a registry.mark_degraded(name) state that /healthz still surfaces as unhealthy (preserving operator visibility), and continue looping with backoff pinned at the 60s cap rather than exiting.
  • Distinguish permanent from transient where cheaply possible: a task that fails the same deterministic error N times can escalate to dead, but transport-level errors (HTTP send, DNS, connect) should never escalate.
  • Alternative if a hard ceiling is wanted: after the ceiling, fall back to a long-interval retry (e.g. 15 min) rather than none — degraded but self-healing.

Test impact (why this needs a plan, not a patch)

Existing tests pin the current contract deliberately:

  • src/tasks.rs module doc: "3 consecutive failures → task marked dead"
  • supervised_failing_task_goes_dead (~line 372) polls until the task is dead and asserts it
  • supervised_task_stays_alive-family tests
  • /healthz degraded semantics tests

Those expectations must change with the behavior, so this wants a short plan + critic pass rather than a silent edit. The visibility guarantee ("an operator can tell a task is unhealthy") must survive the change — that is the acceptance criterion, not "the task never dies."

Verification

  • New/updated test: a task failing 5 consecutive times stays alive, keeps retrying, and /healthz reports it unhealthy throughout.
  • Regression: a healthy task still returns to alive after a single transient failure.
  • End-to-end on a tenant: break egress for one dependency, confirm the task rides through the failure and recovers without a restart.
## Problem Background tasks that hit three consecutive failures are marked `dead` and **never retried until the process restarts**. `src/tasks.rs` (`spawn_supervised`, ~lines 186–200): ```rust Ok(Err(e)) => { failures += 1; tracing::error!(task = name, error = %e, failures, "supervised task iteration failed"); if failures >= MAX_FAILURES { // MAX_FAILURES = 3 (line 28) registry.mark_dead(name).await; return; // <- supervisor exits permanently } tokio::time::sleep(backoff).await; backoff = (backoff * 2).min(Duration::from_secs(60)); continue; } ``` The backoff is correct and capped at 60s — but at three strikes the supervisor `return`s out of its loop and the task is gone for the life of the process. `/healthz` then reports `degraded` / `503` indefinitely. ## Consequence observed Tenant `gal`, app `constructionbot`, 2026-10-28/29: - A transient transport failure to `login.microsoftonline.com` (Azure IP rotation vs. a stale egress allowlist) hit three consecutive mailbox polls within ~35 seconds. - The mailbox task was marked dead at 22:08Z. Client mail intake — the app's primary input path — was dead for **~19 hours**. - Nothing retried. Nothing recovered. The container ran healthy in every other respect; only a manual restart revived the task. - A brief network blip became an all-day intake outage. ## Why this is the wrong trade The intent (make a dead task visible to an operator) is sound. The mechanism is not: a *transient* fault and a *permanent* fault are indistinguishable here, and both produce the same permanent outcome. Network egress, a provider's API, and DNS are all failure domains that recover on their own — exactly the cases where a supervisor should keep trying. ## Proposed fix Keep retrying with capped backoff instead of returning: - On `failures >= MAX_FAILURES`: log at error once per crossing, set a `registry.mark_degraded(name)` state that `/healthz` still surfaces as unhealthy (preserving operator visibility), and **continue** looping with backoff pinned at the 60s cap rather than exiting. - Distinguish permanent from transient where cheaply possible: a task that fails the same deterministic error N times can escalate to `dead`, but transport-level errors (HTTP send, DNS, connect) should never escalate. - Alternative if a hard ceiling is wanted: after the ceiling, fall back to a long-interval retry (e.g. 15 min) rather than none — degraded but self-healing. ## Test impact (why this needs a plan, not a patch) Existing tests pin the current contract deliberately: - `src/tasks.rs` module doc: "3 consecutive failures → task marked `dead`" - `supervised_failing_task_goes_dead` (~line 372) polls until the task is dead and asserts it - `supervised_task_stays_alive`-family tests - `/healthz` `degraded` semantics tests Those expectations must change with the behavior, so this wants a short plan + critic pass rather than a silent edit. The visibility guarantee ("an operator can tell a task is unhealthy") must survive the change — that is the acceptance criterion, not "the task never dies." ## Verification - New/updated test: a task failing 5 consecutive times stays alive, keeps retrying, and `/healthz` reports it unhealthy throughout. - Regression: a healthy task still returns to `alive` after a single transient failure. - End-to-end on a tenant: break egress for one dependency, confirm the task rides through the failure and recovers without a restart.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
gal/constructionbot#14
No description provided.