Conversation
Warm scans re-read and re-hashed every file on each boot. The cache in .bh_filesync/manifest-cache.json stores size + mtime + inode per path so unchanged files skip hashing entirely. Trust requires size, mtime (ns precision) and inode to all match, the file to be older than 5s (rapid rewrites always re-hash), and no pending watcher event for the path.
Replace serial WalkDir collect-then-hash with a jwalk parallel walk pipelined through a bounded channel into a rayon pool sized by core count instead of the fixed 4 threads, so walking overlaps hashing. Hashing is tiered by size: buffered streaming below 128KB, mmap for files 4MB and up with streaming fallback, one shared helper for the manifest, sync engine and bundler paths.
Covers cache roundtrip, hit-skip, mtime and inode invalidation, the 5s age gate, dirty-path bypass, prune-missing, corrupt-backup handling and the rapid-rewrite scenario.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a persistent manifest hash cache to the filesync crate, significantly improving scan performance by avoiding redundant file hashing when file metadata hasn't changed. It also refactors the manifest-building logic to support pipelined, parallel directory scanning and hashing, and updates the hashing logic to use blake3's mmap and rayon features for better performance on large files.
The most important changes are:
Manifest Hash Caching and API:
manifest_cachethat implements a persistent, metadata-sensitive hash cache (ManifestCache) stored in.bh_filesync/manifest-cache.json, with logic for atomic saves, corruption recovery, and stability gating. This cache is used to speed up repeated scans by skipping hashing for unchanged files.build_manifestinmanifest.rs) to use the hash cache, track cache hits/misses, and return updated cache and scan statistics alongside the manifest.Parallel and Tiered Hashing Improvements:
update_mmap_rayonfor large files (with streaming fallback), and pipelined directory traversal usingjwalkandrayonfor improved parallelism and throughput. [1] [2]hash_file_tieredfunction for consistent, efficient hashing. [1] [2] [3]Dependency and Infrastructure Updates:
jwalk,num_cpus) and enabled blake3'smmapandrayonfeatures to support the new parallel and memory-mapped hashing logic.These changes together provide a major performance upgrade for large and repeated syncs, while maintaining correctness and robustness.
References: