Skip to content

perf: Move stolen items with bulk segmented copies - #12

Open
lxl66566 wants to merge 1 commit into
asynchronics:mainfrom
lxl66566:bulk-transfer
Open

lxl66566 wants to merge 1 commit into
asynchronics:mainfrom
lxl66566:bulk-transfer

Conversation

@lxl66566

@lxl66566 lxl66566 commented Sep 1, 2026

Copy link
Copy Markdown

Replace the scalar per-element transfer loop in steal/steal_and_pop with a transfer_items helper that splits the source and destination ring ranges at the union of their physical wrap points and moves the resulting ≤3 chunks with ptr::copy_nonoverlapping. Chunks of ≤8 items keep an inline scalar loop, and ≤1-item steals take a direct fast path, so small steals pay no extra cost.

The inlined transfer_items becomes:

  • an n <= 1 fast path: a single bare load+store, no chunk computation;
  • chunks of ≤ 8 items: fully unrolled scalar copies (8 load/store pairs dispatched by cmp/je), medium chunks via a 4-way unrolled loop;
  • chunks of ≥ 9 items: a direct memcpy call (3 call sites, one per chunk), delegating to libc's SIMD implementation;
  • no per-element bounds checks remain inside the transfer.

Soundness: the booking step already caps the transfer count by both the number of live items in the source queue and the spare capacity of the destination, so the two slot ranges can never overlap — even when stealing from one's own queue. The transfer is a bitwise move, exactly like the per-element read_at/write_at pair it replaces. Under loom, the helper falls back to the tracked per-element loop.

Benchmark: (benches/benchmark.rs on AMD Zen 4, executor scenario, interleaved A/B, medians of 7 rounds)

st3 fifo -7.4%, st3 lifo -5.5%; tokio/crossbeam controls and push/pop paths unchanged.

Split both ring ranges at the union of their physical wrap points and move the (at most 3) chunks with ptr::copy_nonoverlapping; small steals keep an inline scalar path, and loom falls back to the tracked per-element loop. Wraparound is exhaustively covered by new bulk steal/self-steal matrix tests.

Signed-off-by: lxl66566 <lxl66566@gmail.com>
@sbarral

sbarral commented Sep 9, 2026

Copy link
Copy Markdown
Member

Thanks @lxl66566 , I just wanted to let you know that we are looking into this.

I don't wan't to promise anything about the outcome of the review though. We are mindful about keeping an appropriate balance between performance and code complexity, and this might be one of those cases where we will elect to sacrifice a bit of perf, but let us get back to you on this in a few days so we can evaluate it fully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants