Repository navigation
Conversation
The pump awaited Filter::flush_shadow_rels after every fully parsed record. The flush future carries the TOML encoding and atomic write of the eligibility file, several hundred bytes that were built and moved twice per record, although the set changes only when a relation is admitted: about a fifth of the pump's cycles were these memmoves on the write-heavy repro. Ask whether the set is dirty first, which also makes decide synchronous. The check runs after either routing path and before the record stages, so a dirty set still persists before the record that dirtied it reaches the shadow wire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
looks like noise, prefer to avoid unnecessary branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pump constructs shadow-relation flush future for every parsed record even when relation set is unchanged. Check dirty state before awaiting persistence. Keep persistence ahead of record publication.
Extracted from original #228 and based directly on
main. Contains no changes from #226 or other extracted PRs.Validation:
cargo fmt --all -- --check,cargo clippy --workspace --all-targets --locked -- -D warnings, andcargo test --locked --lib source::(189 passed). Rebuild crate for each isolated diff.Measured impact (2026-10-22)
Compared
9d2d068a581d416bd0ced205870d9b329e3f670bagainst shared parent151fd47cbd75ec7b9bc63ffe7040f049e702ecd0, with each PR built separately.Result: no clear pump speedup in measured workload. Full-parse median changed -0.1% and header-routing median -1.3%, with overlapping ranges.
Workload: replay local
classify/segments/000000010000000000000001.gzfixture through production WAL pump 20 times (594,760 records per mode), after decompression outside timed scope. Includes segment buffering, parsing/routing, CRC work, and owned record collection. Uses no configured shadow-relation set, isolating clean no-op path. Persistence I/O and relation changes remain unmeasured; component timing does not establish end-to-end throughput.AMD Ryzen 7 7840U; rustc 1.99.1; locked dependencies; optimized release builds; mimalloc 0.1.52. Seven trials per revision, alternating base/PR order, pinned to CPUs 2,4,6 (three physical cores). Values show median [min, max]; negative time changes mean less work or waiting. Local component measurements, without PostgreSQL or ClickHouse services.