Repository navigation
NTFS MFT engine and a per-file parallel duplicate pipeline with hash cache (0.10.0) - #14
Merged
Merged
Conversation
…cache (0.10.0) Scanning: NTFS MFT engine (Windows, administrator) - New mft module: reads the volume's Master File Table straight from the volume (boot sector -> $MFT data runs -> 8 MB chunks read in parallel over four handles), applies update-sequence fixups and parses $STANDARD_INFORMATION, $FILE_NAME (8.3 aliases skipped, one entry per hard link), the unnamed $DATA stream (logical / allocated / compressed size) and reparse points. Metafiles, symbolic links and junctions are left out. All reads are bounds-checked. - Without exclusions the tree is built straight from the records (no paths); with exclusions, and for the duplicate search, it emits the usual batches. - Cloud placeholders and OneDrive-synced folders (from the OneDrive environment variables) are sized / listed through Windows, because only the cloud filter knows what is stored locally. - Options::mft, CLI --mft; ScanTree::engine() and mft_fallback() say which engine ran and why the MFT was not used (no rights, not NTFS, ...). - Measured (administrator, NVMe system drive): C:\Users\me 196 s -> 37 s with the walk vs the MFT, all of C: (4.8 M files, 4.9 M folders) in ~30 s; file counts and folder sizes match the walk. On a SATA SSD with millions of files the table is larger than the folders and the walk stays faster, which the settings text says. Duplicates - New dupes module: candidates of a repeated size are opened once, in parallel per file (not per size group), for their identity (hard links collapsed) and a BLAKE3 of the first and last 16 KB (files up to 32 KB are thereby hashed whole); only what still matches is hashed in full, big files on several threads via mmap. same-file dependency dropped. - HashCache: path + size + modification time -> identity and hashes, zstd-compressed; find_duplicates_cached reuses it. CLI --cache FILE; the GUI keeps one in its data folder. A repeated search of a 461-group Steam library: 1.2 s -> 0.1 s, identical groups. - Progress reports phases (list, compare ends, hash) with totals. - Exclude globs are case-insensitive on Windows. GUI - Engine "Ultra (NTFS MFT)" with a restart-as-administrator button (and the note that Windows blocks drag & drop into an elevated window); "Admin" next to the version; the scan card shows "NTFS MFT" and a status line explains a fallback. - Duplicate search skips program and game files unless included (the card offers a search that includes them); the progress card shows the three phases with counts, bytes and a real percentage. Tests: 34 core (cache round trip and reuse, same ends / different middle, in-order and out-of-order tree batches), 8 CLI, 23 UI helper tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scanning: NTFS MFT engine (Windows, administrator)
volume (boot sector -> $MFT data runs -> 8 MB chunks read in parallel over
four handles), applies update-sequence fixups and parses $STANDARD_INFORMATION,
$FILE_NAME (8.3 aliases skipped, one entry per hard link), the unnamed $DATA
stream (logical / allocated / compressed size) and reparse points. Metafiles,
symbolic links and junctions are left out. All reads are bounds-checked.
with exclusions, and for the duplicate search, it emits the usual batches.
environment variables) are sized / listed through Windows, because only the
cloud filter knows what is stored locally.
engine ran and why the MFT was not used (no rights, not NTFS, ...).
the walk vs the MFT, all of C: (4.8 M files, 4.9 M folders) in ~30 s; file
counts and folder sizes match the walk. On a SATA SSD with millions of files
the table is larger than the folders and the walk stays faster, which the
settings text says.
Duplicates
parallel per file (not per size group), for their identity (hard links
collapsed) and a BLAKE3 of the first and last 16 KB (files up to 32 KB are
thereby hashed whole); only what still matches is hashed in full, big files
on several threads via mmap. same-file dependency dropped.
zstd-compressed; find_duplicates_cached reuses it. CLI --cache FILE; the
GUI keeps one in its data folder. A repeated search of a 461-group Steam
library: 1.2 s -> 0.1 s, identical groups.
GUI
note that Windows blocks drag & drop into an elevated window); "Admin" next
to the version; the scan card shows "NTFS MFT" and a status line explains a
fallback.
offers a search that includes them); the progress card shows the three
phases with counts, bytes and a real percentage.
Tests: 34 core (cache round trip and reuse, same ends / different middle,
in-order and out-of-order tree batches), 8 CLI, 23 UI helper tests.