Skip to content

test: make timing-sensitive budget and task-timeout tests deterministic - #1861

Closed
bokelley wants to merge 1 commit into
nextfrom
claude/fix-parallel-flaky-tests
Closed

bokelley wants to merge 1 commit into
nextfrom
claude/fix-parallel-flaky-tests

Conversation

@bokelley

@bokelley bokelley commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Fixes the parallel-run test failures noted in #1860. Test-only change; no source changes.

What was failing

Under pytest -n auto these tests compared real wall-clock time against tight bounds:

Test Failure Root cause
test_branch_parity_absorbs_registry_variance intermittent Compared the slowest real sample from each PERMISSION_DENIED branch against a 15 ms tolerance. One scheduling stall exceeds it.
test_enforce_does_not_sleep_when_already_overrun at risk Required enforce() to return in under 5 ms of real time.
test_real_mcp_dispatch_reports_the_exact_pinned_wire_key intermittent Its 50 ms deadline also covered pre-dispatch work, so a slow preflight expired the deadline before the request reached the wire.
test_new_mutations_timeout_with_recovery_metadata[sync_principal] every time when its file runs on its own (serial or parallel); passes only when earlier tests have warmed the caches Same as above, with 100 ms and cold schema loading. The existing nullable-schema warm-up covered only part of the cold work.
test_mutation_timeout_carries_secret_safe_recovery at risk Same as above, with 10 ms.

Fix

Make the timing deterministic, without widening any tolerance.

  • Budget tests run adcp.decisioning._permission_denied_budget on a virtual clock: perf_counter reads it, and sleep and the fake registry advance it exactly. The parity test now asserts that both branches end at exactly the 50 ms budget, and adds a case showing a registry slower than the budget is not padded further. With the budget disabled, the test fails.
  • Post-dispatch timeout tests use a deadline long enough that pre-dispatch work can't expire it. They expire it through the adcp.client.run_with_timeout seam exactly when the test's dispatch signal fires, raising _TaskDeadlineExpiredError as the real timer does. The client's deadline state, dispatch marking and recovery assembly all stay real, and other tests in the file still exercise the real timer. If recovery is dropped from the timeout error, 5 tests fail. The partial warm-up workaround is removed.

Verification

  • tests/test_task_options.py: 23/23 passed serially and on 3 of 3 runs under -n auto. Before the fix, the sync_principal case failed on 4 of 4 runs.
  • Both files under -n auto with all 48 cores pegged by busy loops: 53/53 passed on 3 of 3 runs.
  • Full suite under -n auto: 21,708 passed, 0 failed. make lint and make typecheck-all pass.

🤖 Generated with Claude Code

Under pytest -n auto these tests compared real wall-clock time with tight
bounds and failed intermittently; one also failed every time its file ran
on its own.

test_permission_denied_budget: the branch-parity test compared the slowest
real sample from each branch against a 15 ms tolerance, and the overrun
test required enforce() to return in under 5 ms. Both now run the budget
module on a virtual clock: perf_counter reads it, and sleep and the fake
registry advance it exactly. Each branch ends at exactly the 50 ms budget,
and a registry slower than the budget is not padded.

test_task_options: three "timeout after dispatch" tests used a real 10-100 ms
deadline that also covered pre-dispatch work (cold schema loads,
validation, the MCP adapter path). When that work was slow the deadline
expired before dispatch, leaving no recovery metadata. They now use a
generous deadline and expire it through the run_with_timeout seam exactly
when dispatch is signalled, raising what the real timer raises. The
client's deadline state, dispatch marking and recovery stay real; other
tests in the file still exercise the real timer. This also removes the
partial nullable-schema warm-up workaround.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — test-only PR making timing-sensitive tests deterministic.

Checked:

  • Two test files changed only (tests/test_permission_denied_budget.py, tests/test_task_options.py); no src/, public surface, wire-shape, forward-compat, type-layering, generated code, or credential surface touched.
  • Reviewer confirmed the _VirtualClock faithfully models _permission_denied_budget (module-level time/asyncio seams; budget built before registry read at handler.py:774), so the approx(50.0) branch-parity assertions still catch a disabled budget.
  • _deadline_expires_on replaces only the run_with_timeout timer seam; deadline state, mark_task_dispatched, and recovery=state.mutation_recovery assembly stay real — a dropped recovery still fails.
  • Removed nullable_request_paths warm-up and import fully excised; no test disabled/weakened (CI gates stay armed).

No critical/high/medium findings. high_risk false, gated_paths false, no author team gate. Falls through to row 9.

@bokelley

bokelley commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing: these tests don't fail in CI.

I checked the last 40 CI runs on next (39 passed; the one failure was an unrelated test) and every failed CI run on any branch from 24 Oct to 7 Nov (60 runs). None of the five tests changed here ever failed; they appear in those logs only as PASSED. CI runs the unit suite with 2 workers, and in a full run earlier tests warm the caches these timeouts depend on.

The failures are real only locally: under pytest -n auto on a 48-core machine, and for test_new_mutations_timeout_with_recovery_metadata[sync_principal], every time tests/test_task_options.py runs on its own. The branch claude/fix-parallel-flaky-tests keeps the deterministic versions (virtual clock for the budget tests; deadline expired on dispatch via the run_with_timeout seam) if that becomes worth fixing.

@bokelley bokelley closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant