Target 16 concurrent inputs per execute_sample container

[kevintli]

Summary

-@modal.concurrent(max_inputs=128)
+@modal.concurrent(max_inputs=128, target_inputs=16)
 async def execute_sample(task: dict) -> dict:

When only max_inputs=128 is set, Modal's autoscaler aims for 128 in-flight inputs per container. So a client with ~500 concurrent asample calls (sec-search-rl: 64 groups x 8) ran on 1–2 execute_sample containers. Each container pulls inputs one FunctionGetInputs round-trip at a time, which made intake the bottleneck. Under load the function had a backlog of 100–200 while SGLang replicas sat almost idle, and most retrieve_future long-polls timed out (408).

Setting target_inputs=16 makes the autoscaler spread that load across ~30 cheap CPU containers. max_inputs=128 stays the burst ceiling. On the sec-search-rl traj-recall run (stacked on #8, deployed to kevinli-spindle-secrepro), throughput went from ~4 to ~26 asample/s at the frontend (21 runners, backlog 0), and 408s dropped from the majority of retrieve_future responses to ~3%.

tests/providers/test_modal_app.py passes.

Link to Devin session: https://modal.devinenterprise.com/sessions/f53cfabb210146de8f0338fe388d7973
Open in Devin Desktop: https://modal.devinenterprise.com/desktop/session/f53cfabb210146de8f0338fe388d7973?variant=devin
Requested by: @kevintli

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring