Repository navigation
Conversation
The conversion of the RF loop state to a fixed-shape raw carry and back (Tensors in their initial dim order, TensorArrays as buffer and write index, dims in the state carried by their dyn size) moves from JaxBackend.while_loop to rf.loop._graph_while_loop, so that another backend's graph loop can use it. JaxBackend.while_loop keeps only its own part: one device for the carry, lax.while_loop. _BufferTensorArray moves to rf.tensor_array for the same reason. The eager branch of rf.while_loop becomes _while_loop_eager. No behavior change, except that an empty TensorArray with a capacity which is still a plain list can now cross the graph loop as well.
Under rf.is_static_traceable(), TorchBackend.while_loop runs rf.while_loop as the torch while_loop op over the fixed-shape carry of rf.loop._graph_while_loop (the op directly, torch.while_loop goes through Dynamo). Captured into a CUDA graph under CUDAGraphCaptureControlFlowOpDispatchMode (torch >= 2.14), the loop becomes a device-side while node: one graph for any trip count, no host sync per iteration. Outside static tracing, the eager Python loop as before. TensorArray in the buffer form for PT: push_back writes out-of-place via index_copy (beyond the capacity it fails), get_item and stack read the buffer, unstack under static tracing gives the buffer form, read on the device by index, and get_item of a list with a tensor index under static tracing stacks once, without host read. #2014
The torch while_loop op exists since torch 2.3 (CI rf_loop runs torch 2.0). noinspection PyProtectedMember for the _BufferTensorArray import in rf.loop and for torch._higher_order_ops.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For #2014.
Under
rf.is_static_traceable(), the PT backend now runsrf.while_loopas the torchwhile_loopop(
torch._higher_order_ops.while_loop.while_loop_op; the publictorch.while_loopgoes through Dynamo).Captured into a CUDA graph under
CUDAGraphCaptureControlFlowOpDispatchMode(torch >= 2.14),the loop becomes a device-side CUDA graph while node:
one capture for any trip count, no host sync per iteration.
Outside static tracing, nothing changes: the eager Python loop as before.
Two commits:
RF while_loop: graph-loop carry conversion backend-neutral:the conversion of the RF loop state to a fixed-shape raw carry and back
(Tensors in their initial dim order, TensorArrays as buffer plus write index, dims in the state via their dyn size)
moves from
JaxBackend.while_looptorf.loop._graph_while_loop,_BufferTensorArraytorf.tensor_array.JAX keeps only its own part (one device,
lax.while_loop). No behavior change.PT while_loop: torch while_loop op under static tracing:TorchBackend.while_loopuses it; TensorArray buffer form for PT(
push_backout-of-place viaindex_copy, beyond the capacity it fails;unstackunder static tracing gives the buffer form, read by index on the device, no host read).Tests in
tests/test_rf_loop.py:test_while_loop_torch_graph_loop_beam_search: frame-synchronous beam search(top_k over beam and vocab, label and backref TensorArrays with a capacity, backtracking in reverse),
eager vs graph loop, trip counts 0, 1, 4, 6, exact.
test_while_loop_torch_graph_loop_tensor_array_overflow: a push beyond the capacity raises.test_while_loop_torch_cuda_graph_while_node(CUDA, torch >= 2.14, skipped in CI):the same search captured once as a while node, replayed for trip counts 0, 1, 4, 6, exact vs eager.
Run at RZ:
while_loopand bounded-cache tests, 15 passed, 1 skipped (CUDA).and for the CUDA test "while_loop: cond should be on CPU".
test_rf_loop.py,test_rf_jax.py,test_torch_frontend.py,test_rf_attention.py,test_rf_decoder_transformer.py:the same 16 failures on master and here (RoPE / attention tests in that env,
transformersnot installed).End to end, with the joint AED+CTC time-sync search
model_recog_with_recomb_while_loop(i6_experiments),Loquacious 560M AED+CTC, beam 64, recomb max, f32, 32 dev seqs in 4 batches of 8, H100 (shared, timings rough),
together with #2016 (RoPE with the bounded cache) and #2017 (
mask_nestedunder static tracing)and some recipe changes for the bound shapes:
the graph loop gives the same best hypotheses as the eager production search for all 32 seqs (score diff at most 1e-5),
warm time 6.4 s vs 17.0 s eager (batch 8).
The whole recognition call (feature extraction, encoder, search loop as while node, backtracking as while node)
captured once as one CUDA graph and replayed for 8 batches of 4 with different audio and lengths (batch 4 to fit next to other jobs):
model_recog_with_recomb(production)Best hypotheses identical for all 32 seqs in all pairs, max score diff 2.2e-5.
Warmup and capture 2.1 s, peak allocation 8.4 GiB.
Who captures the recognition step (bound shapes, one capture per bound, reset of the dims before capture)
is not part of this PR; the probe did that by hand.