feat: user-specified retries for transient API failures

[timzsu]

Purpose

The API executor marks transient failures (5xx, 408, 429, connection errors) as retryable=True but never retried them — a request failed immediately on the first transient error. In production the LLM serving endpoint intermittently returns 504 Gateway Timeout on large requests; the same request succeeds when retried. A user-specified retry count lets the executor re-issue and succeed.

Changes

• spec.api.retries (new field, default 0 preserving current behavior) — how many times a transient failure is re-issued before the task fails.
• _request_with_retries — wraps the request in a retry loop with a fixed 1s backoff between attempts. A retryable failure (connection error, 5xx, 408, 429) retries up to retries times; non-retryable 4xx and cancelled tasks stop immediately.
• Cancellation during a retry. The backoff waits on the task's cancel event, so a cancel ends pending retries at once and the task reports cancelled; a request already inside its HTTP call is not interrupted.
• Cancellation is scoped to one task. A worker reuses one API executor across tasks, so the executor records which task it is running: a cancel addressed to another task is ignored, and a cancel that arrives before its task starts still takes effect when that task starts. Before this, a cancelled executor stayed cancelled for the next task it ran.
• Tests covering retries-exhausted (fails loudly), retry-succeeds, cancel during backoff, a late cancel for a finished task, and a cancel before its task starts; docs updated.

Design

The retry is per-request, preserving the executor's concurrency model. The final attempt's failure propagates to the caller — a request that exhausts retries still fails loudly, never silently swallowed.

Test Plan

The repo's own pre-commit gate over every changed file, and the full suite on the repo's interpreter.

Test Result

E2E against a live serving endpoint. A FlowMesh worker running this change serves Lumilake's paper-ingestion runs against qwen3.8-27b with spec.api.retries: 3. Its per-call log shows both outcomes:

• calls whose first attempt failed were re-issued and returned 200 on the second attempt, in single-request tasks and in multi-row batch tasks, where only the failed row was re-issued;
• a task whose endpoint kept returning 503 made four attempts per row, the first plus three retries, and then failed with the 503 rather than reporting success.

---

Pre-submission Checklist

• I have read the contribution guidelines.
• I have run pre-commit run --all-files and fixed any issues.
• I have added or updated tests covering my changes (if applicable).
• I have verified that uv run pytest tests/ passes locally.
• If I changed shared schemas or proto definitions, I have checked downstream compatibility across Server and Worker.
• If I changed the SDK or CLI, I have verified the affected packages work (uv sync --all-packages --group ci --frozen).
• If this is a breaking change, I have prefixed the PR title with [BREAKING] and described migration steps above.
• I have updated documentation or config examples if user-facing behavior changed.

[kaiitunnz]

A few comments. PTAL.

[kaiitunnz]

One more minor issue.

[kaiitunnz]

LGTM.