Make execute_sample non-preemptible

[kevintli]

Summary

execute_sample is the CPU-only relay between the frontend and the SGLang sampler pool. It holds up to 128 in-flight sample inputs per container. Because it is a regular Modal Function, it is preemptible by default, and we see it preempted often: 8 times in one day on a single frontend, per the Container terminated due to preemption log. In one sec-search-rl run, a preemption that hit mid-batch was followed by 2 sample calls that never resolved and never sent traffic to the sampler, which stalled the training batch. We haven't proven a causal link.

 @app.function(
     image=image,
     min_containers=0,
     timeout=60 * 60,
     retries=2,
+    nonpreemptible=True,
     secrets=[...],
 )
 @modal.concurrent(max_inputs=128, target_inputs=16)
 async def execute_sample(task: dict) -> dict:

• Modal supports nonpreemptible only on CPU Functions, which this is. GPU sampler and trainer Functions are unaffected.
• Modal bills a 3x multiplier on CPU and memory list price for non-preemptible Functions. That should be negligible for a thin async relay next to the H200s.
• This removes one trigger, not the underlying failure mode. Detecting and recovering lost or stale calls is tracked in Detect and recover stale sampling calls (configurable liveness timeout, no hard cap on rollout length) #14.

Tests: pytest shows the same results as the base branch. tests/scoped/test_lifecycle.py::test_owned_children_stop_before_parent fails 4 times on the base branch as well, and the torch-dependent modules can't be collected without torch. ruff check and ruff format --check pass on the changed file.

Link to Devin session: https://modal.devinenterprise.com/sessions/f53cfabb210146de8f0338fe388d7973
Open in Devin Desktop: https://modal.devinenterprise.com/desktop/session/f53cfabb210146de8f0338fe388d7973?variant=devin
Requested by: @kevintli

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring