Repository navigation
Conversation
The queueing sink ships records its inner sink does not want as a count and a high-water LSN, so they occupy channel batches while adding nothing to in_flight. walshadow_pump_queue_depth therefore undercounts by the skipped share, and on the write-heavy repro read a few hundred to about 1,500 records while the channel was full and the pump blocked on it, hiding that the queue worker bounds the drain. Count skipped records into in_flight as they ship and out as the worker settles them, as processed already does. The fork fence, which waits for in_flight to reach zero, now also waits for their positions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Queue depth omits skipped records whose positions still occupy channel batches. Count positions in buffered, queued and settling batches. Include skipped positions in fork fence and settle counts on errors.
Extracted from original #228 and based directly on
main. Contains no changes from #226 or other extracted PRs.Validation:
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --locked -- -D warnings, andcargo test --locked --lib source::queueing_record_sink(15 passed). Rebuild crate for each isolated diff.Measured impact (2026-10-22)
Compared
65437d21a132d8310d78c8c3d7febaec34c642a2against shared parent151fd47cbd75ec7b9bc63ffe7040f049e702ecd0, with each PR built separately.Result: position-only depth now reports expected 511 instead of 0. Queue timings have overlapping ranges; no clear CPU overhead or gain established.
Accounting probe buffers 511 unwanted records without yielding or shipping batch (batch size 512): expected depth 511. Throughput workload uses 262,144 records per case, no-op downstream batch sink, queue capacity 131,072, and two Tokio runtime workers. Timing includes allocation, dispatch, close, and runtime shutdown. Measured functional benefit is correct position-only queue depth; throughput results describe overhead of accounting change.
AMD Ryzen 7 7840U; rustc 1.99.1; locked dependencies; optimized release builds; mimalloc 0.1.52. Seven trials per revision, alternating base/PR order, pinned to CPUs 2,4,6 (three physical cores). Values show median [min, max]; negative time changes mean less work or waiting. Local component measurements, without PostgreSQL or ClickHouse services.