Skip to content

test(qed): cover worker import isolation and busy-worker recovery - #1492

Merged
burtenshaw merged 5 commits into
mainfrom
fix/qed-worker-import-startup
Oct 26, 2026
Merged

burtenshaw merged 5 commits into
mainfrom
fix/qed-worker-import-startup

Conversation

@burtenshaw

@burtenshaw burtenshaw commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add regression coverage for QED worker import isolation and busy-worker cleanup. Main now contains the lazy-export production fix from #1491; this PR adds tests only.

worker import → excludes client, models and full environment
busy worker → timeout/cancellation → admission released → subsequent verification succeeds
stop → executor reference cleared

Retains public export identity and fresh-process import coverage. Request deadlines, rewards and lifecycle implementation are unchanged.

Evidence

  • Before: main has cold-worker import tests, but lacks the explicit models exclusion and these busy-worker timeout/cancellation recovery cases.
  • After: at 68060a4391573a3d024af8a779d2c3a3933d2c28, both QED test files pass: 153 passed, 5 deselected on macOS/Python 3.11. Four fresh-process import probes and scoped Ruff/usort checks pass.
  • The 76 added test lines are unchanged by conflict resolution; both production initializers match main exactly.
  • New-head CI is pending. Earlier Linux CPU-pressure evidence validated the production optimization now on main; it was not rerun for this test-only head. These tests do not establish termination of arbitrary running jobs or resolve Add optional fastapi-guard security middleware for env servers (default off) #1361 policy gates.

Merge Danger

Door: two-way; revert the added tests.

Blast Radius: tests. No production behavior changes.

@burtenshaw burtenshaw added bug Something isn't working environment size: medium Medium pull request labels Oct 26, 2026 — with Cursor

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alignment Review Report

Automated Checks

  • Lint: PASS — uv run usort check / ruff format --check / ruff check all pass on the three files this PR touches (envs/qed_math_env/__init__.py, envs/qed_math_env/server/__init__.py, tests/envs/test_qed_math_environment.py). (Note: a full-repo ruff format --check reports 47 files needing reformatting, but none of them are touched by this PR — pre-existing drift, not introduced here.)
  • Debug code: CLEAN — no print/breakpoint/pdb/TODO/FIXME in the diff.

Verification performed

  • Installed the env's own deps (math-verify, datasets, fastmcp, trackio) and ran tests/envs/test_qed_math_environment.py: 151 passed, including the two new tests (TestVerifierImports::test_worker_import_does_not_load_environment_dependencies, TestVerifierImports::test_public_exports_preserve_identity) and the new test_busy_worker_deadline_and_cancellation cases. The 5 errors in TestQEDMathServerIntegration are pre-existing (that class predates this PR) and only fail here because no live server is running in this sandbox — unrelated to this change.
  • Confirmed the actual fix in a fresh interpreter: import qed_math_env.server.math_verify_service no longer pulls in qed_math_env.client, qed_math_env.models, datasets, fastmcp, or openai — exactly the problem a spawn-context ProcessPoolExecutor worker hits on re-import.
  • Confirmed the lazy __getattr__/__dir__ (PEP 562) pattern preserves the public API: from qed_math_env import QEDMathEnv, qed_math_env.server.QEDMathEnvironment, and dir(qed_math_env) all behave as before; identity checks in the new test confirm the lazily-loaded objects are the same objects as direct submodule imports.
  • Checked for the same risk elsewhere: no other environment under envs/ uses ProcessPoolExecutor/mp.get_context("spawn"), so this fix is correctly scoped to qed_math_env alone per the self-contained-environments guidance (no sibling copies need the same fix).

Open RFCs Context

Scanned all RFCs in rfcs/ (000, 001, 002, 003, 004, 005, 006, 008 — status In Review; 010, 011 — status Draft). None reference qed_math_env, process-pool verifier workers, or package __init__.py/lazy-import conventions, so there is no active design discussion this change intersects with.

Tier 1: Fixes Required

None — mechanical checks are clean on the changed files.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. This is an internal import-ordering fix inside a single self-contained environment:

  • No change to the Gym-like API signatures or MCP tool surface (Invariant: API Invariants).
  • No crossing of the client/server boundary — envs/qed_math_env/__init__.py still only re-exports client/model symbols, and server/__init__.py still only re-exports the server Environment class; both via lazy attribute access rather than eager imports (Invariant: client-server separation, git-blamed to @Darktex, who authored this section of INVARIANTS.md).
  • No change to reward computation or where it lives (Principle: rewards inside environment).
  • Pattern is isolated to qed_math_env and doesn't change the canonical openenv init scaffold in PATTERNS.md for other environments.

RFC Conflicts

None identified.

Summary

  • 0 mechanical issues to fix
  • 0 alignment points for human review
  • 0 RFC conflicts to discuss

This is a narrow, well-tested fix: it correctly diagnoses that ProcessPoolExecutor(mp_context=mp.get_context("spawn")) re-imports qed_math_env.server.math_verify_service in each spawned worker, which previously eagerly pulled in datasets, fastmcp, openai, and the full QEDMathEnvironment/QEDMathEnv stack on every worker start. The TYPE_CHECKING + __getattr__/__dir__ (PEP 562) lazy-export pattern keeps the public API identical while avoiding that cost, and the new tests lock in both properties (lightweight worker import, and preserved public export identity/AttributeError behavior).

Open in Web View Automation 

Sent by Cursor Automation: Pre-review

Comment thread tests/envs/test_qed_math_environment.py Resolved
Comment thread tests/envs/test_qed_math_environment.py Resolved
Comment thread tests/envs/test_qed_math_environment.py Resolved
@burtenshaw burtenshaw changed the title fix(qed): keep verifier worker imports lightweight test(qed): cover worker import isolation and busy-worker recovery Oct 26, 2026
@burtenshaw
burtenshaw merged commit d8f1f36 into main Oct 26, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working environment size: medium Medium pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant