Skip to content

Ship backfill rows to the batcher in chunks - #271

Merged
serprex merged 1 commit into
mainfrom
split/row-chunks
Oct 23, 2026
Merged

serprex merged 1 commit into
mainfrom
split/row-chunks

Conversation

@serprex

@serprex serprex commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

WAL replay sent rows to batcher one at a time, adding channel overhead
for every row. Bootstrap already grouped rows into chunks.

Share bootstrap row buffer with WAL replay and allow chunks to span
transactions. Register transactions before sending rows, then report
row counts afterward so acknowledgments cannot arrive too early.

Flush buffered rows at size limits, input boundaries, idle, completion,
and errors. Send rows holding detoast memory permits immediately so
buffered memory cannot block later allocations. Keep failed transactions
incomplete so acknowledgments cannot advance past them.

Depends on #265.

Validation: cargo fmt --all -- --check, cargo clippy --workspace --all-targets --locked -- -D warnings, RUSTDOCFLAGS=-D warnings cargo doc --no-deps --lib --bins, full cargo nextest run --workspace --all-targets --locked against PG 17.

🤖 Generated with Claude Code

@serprex
serprex force-pushed the split/ack-allocator branch 2 times, most recently from 3f811d2 to d1b7c2f Compare October 22, 2026 20:38
Base automatically changed from split/ack-allocator to main October 23, 2026 01:25
@serprex

serprex commented Oct 23, 2026 •

Copy link
Copy Markdown
Member Author

WalReplaySink::on_record(COMMIT) over synthetic inline Int4 rows into tail::spawn_null. Release, Rust 1.99.1, Ryzen 7 7840U, taskset -c 2,4,6, median total of 7 runs.

Parent 0cfa6db, 568d6b6 (prior push), a610259 (seqs held across commits, rows ship on buffer trigger, segment boundary, finish, or error).

Rows/commit Int4 columns Commits Parent ms 568d6b6 ms a610259 ms a610259 vs parent
1 8 262,144 363 389 210 1.73×
32 8 32,768 387 292 240 1.61×
64 8 16,384 373 285 231 1.61×
1,024 8 1,024 362 235 223 1.62×
16,384 8 64 360 242 255 1.41×
1,024 64 1,024 588 431 414 1.42×
262,144 8 1 98 65 67 1.47×

16,384×64 reruns at 11 samples: 238–244 ms, noise.

cargo bench --bench wal_replay -- --rows 1 --transactions 262144 --columns 8 --repeats 7

WAL replay sent rows to batcher one at a time, adding channel overhead
for every row. Bootstrap already grouped rows into chunks.

Share bootstrap row buffer with WAL replay and allow chunks to span
transactions. Register transactions before sending rows, then report
row counts afterward so acknowledgments cannot arrive too early.

Flush buffered rows at size limits, input boundaries, idle, completion,
and errors. Send rows holding detoast memory permits immediately so
buffered memory cannot block later allocations. Keep failed transactions
incomplete so acknowledgments cannot advance past them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@serprex
serprex merged commit 4b0afb5 into main Oct 23, 2026
6 checks passed
@serprex
serprex deleted the split/row-chunks branch October 23, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants