Skip to content

PT while_loop: torch while_loop op under static tracing - #2018

Draft
albertz wants to merge 3 commits into
masterfrom
albert-torch-while-loop
Draft

albertz wants to merge 3 commits into
masterfrom
albert-torch-while-loop

Conversation

@albertz

@albertz albertz commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

For #2014.

Under rf.is_static_traceable(), the PT backend now runs rf.while_loop as the torch while_loop op
(torch._higher_order_ops.while_loop.while_loop_op; the public torch.while_loop goes through Dynamo).
Captured into a CUDA graph under CUDAGraphCaptureControlFlowOpDispatchMode (torch >= 2.14),
the loop becomes a device-side CUDA graph while node:
one capture for any trip count, no host sync per iteration.
Outside static tracing, nothing changes: the eager Python loop as before.

Two commits:

  1. RF while_loop: graph-loop carry conversion backend-neutral:
    the conversion of the RF loop state to a fixed-shape raw carry and back
    (Tensors in their initial dim order, TensorArrays as buffer plus write index, dims in the state via their dyn size)
    moves from JaxBackend.while_loop to rf.loop._graph_while_loop, _BufferTensorArray to rf.tensor_array.
    JAX keeps only its own part (one device, lax.while_loop). No behavior change.
  2. PT while_loop: torch while_loop op under static tracing:
    TorchBackend.while_loop uses it; TensorArray buffer form for PT
    (push_back out-of-place via index_copy, beyond the capacity it fails;
    unstack under static tracing gives the buffer form, read by index on the device, no host read).

Tests in tests/test_rf_loop.py:

  • test_while_loop_torch_graph_loop_beam_search: frame-synchronous beam search
    (top_k over beam and vocab, label and backref TensorArrays with a capacity, backtracking in reverse),
    eager vs graph loop, trip counts 0, 1, 4, 6, exact.
  • test_while_loop_torch_graph_loop_tensor_array_overflow: a push beyond the capacity raises.
  • test_while_loop_torch_cuda_graph_while_node (CUDA, torch >= 2.14, skipped in CI):
    the same search captured once as a while node, replayed for trip counts 0, 1, 4, 6, exact vs eager.

Run at RZ:

  • CPU, torch 2.12: these plus the existing loop, JAX while_loop and bounded-cache tests, 15 passed, 1 skipped (CUDA).
  • H100, torch 2.14.1: the three new tests passed.
  • Negative control (new tests on master): "graph loop not taken", "capacity overflow not detected",
    and for the CUDA test "while_loop: cond should be on CPU".
  • Full test_rf_loop.py, test_rf_jax.py, test_torch_frontend.py, test_rf_attention.py, test_rf_decoder_transformer.py:
    the same 16 failures on master and here (RoPE / attention tests in that env, transformers not installed).

End to end, with the joint AED+CTC time-sync search model_recog_with_recomb_while_loop (i6_experiments),
Loquacious 560M AED+CTC, beam 64, recomb max, f32, 32 dev seqs in 4 batches of 8, H100 (shared, timings rough),
together with #2016 (RoPE with the bounded cache) and #2017 (mask_nested under static tracing)
and some recipe changes for the bound shapes:
the graph loop gives the same best hypotheses as the eager production search for all 32 seqs (score diff at most 1e-5),
warm time 6.4 s vs 17.0 s eager (batch 8).
The whole recognition call (feature extraction, encoder, search loop as while node, backtracking as while node)
captured once as one CUDA graph and replayed for 8 batches of 4 with different audio and lengths (batch 4 to fit next to other jobs):

batch 4, 32 seqs, 132 s audio warm time
eager model_recog_with_recomb (production) 8.50 s
graph loop, host-driven, no capture 6.50 s
captured, while node, one replay per batch 3.66 s

Best hypotheses identical for all 32 seqs in all pairs, max score diff 2.2e-5.
Warmup and capture 2.1 s, peak allocation 8.4 GiB.
Who captures the recognition step (bound shapes, one capture per bound, reset of the dims before capture)
is not part of this PR; the probe did that by hand.

The conversion of the RF loop state to a fixed-shape raw carry and back
(Tensors in their initial dim order, TensorArrays as buffer and write index,
dims in the state carried by their dyn size)
moves from JaxBackend.while_loop to rf.loop._graph_while_loop,
so that another backend's graph loop can use it.
JaxBackend.while_loop keeps only its own part: one device for the carry, lax.while_loop.
_BufferTensorArray moves to rf.tensor_array for the same reason.
The eager branch of rf.while_loop becomes _while_loop_eager.
No behavior change, except that an empty TensorArray with a capacity
which is still a plain list can now cross the graph loop as well.
Under rf.is_static_traceable(), TorchBackend.while_loop runs rf.while_loop
as the torch while_loop op over the fixed-shape carry of rf.loop._graph_while_loop
(the op directly, torch.while_loop goes through Dynamo).
Captured into a CUDA graph under CUDAGraphCaptureControlFlowOpDispatchMode (torch >= 2.14),
the loop becomes a device-side while node:
one graph for any trip count, no host sync per iteration.
Outside static tracing, the eager Python loop as before.

TensorArray in the buffer form for PT:
push_back writes out-of-place via index_copy (beyond the capacity it fails),
get_item and stack read the buffer,
unstack under static tracing gives the buffer form, read on the device by index,
and get_item of a list with a tensor index under static tracing stacks once, without host read.

#2014
The torch while_loop op exists since torch 2.3 (CI rf_loop runs torch 2.0).
noinspection PyProtectedMember for the _BufferTensorArray import in rf.loop
and for torch._higher_order_ops.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant