Skip to content

NTFS MFT engine and a per-file parallel duplicate pipeline with hash cache (0.10.0) - #14

Merged
kingchenc merged 1 commit into
mainfrom
feat/mft-fast-dupes
Oct 22, 2026
Merged

kingchenc merged 1 commit into
mainfrom
feat/mft-fast-dupes

Conversation

@kingchenc

Copy link
Copy Markdown
Owner

Scanning: NTFS MFT engine (Windows, administrator)

  • New mft module: reads the volume's Master File Table straight from the
    volume (boot sector -> $MFT data runs -> 8 MB chunks read in parallel over
    four handles), applies update-sequence fixups and parses $STANDARD_INFORMATION,
    $FILE_NAME (8.3 aliases skipped, one entry per hard link), the unnamed $DATA
    stream (logical / allocated / compressed size) and reparse points. Metafiles,
    symbolic links and junctions are left out. All reads are bounds-checked.
  • Without exclusions the tree is built straight from the records (no paths);
    with exclusions, and for the duplicate search, it emits the usual batches.
  • Cloud placeholders and OneDrive-synced folders (from the OneDrive
    environment variables) are sized / listed through Windows, because only the
    cloud filter knows what is stored locally.
  • Options::mft, CLI --mft; ScanTree::engine() and mft_fallback() say which
    engine ran and why the MFT was not used (no rights, not NTFS, ...).
  • Measured (administrator, NVMe system drive): C:\Users\me 196 s -> 37 s with
    the walk vs the MFT, all of C: (4.8 M files, 4.9 M folders) in ~30 s; file
    counts and folder sizes match the walk. On a SATA SSD with millions of files
    the table is larger than the folders and the walk stays faster, which the
    settings text says.

Duplicates

  • New dupes module: candidates of a repeated size are opened once, in
    parallel per file (not per size group), for their identity (hard links
    collapsed) and a BLAKE3 of the first and last 16 KB (files up to 32 KB are
    thereby hashed whole); only what still matches is hashed in full, big files
    on several threads via mmap. same-file dependency dropped.
  • HashCache: path + size + modification time -> identity and hashes,
    zstd-compressed; find_duplicates_cached reuses it. CLI --cache FILE; the
    GUI keeps one in its data folder. A repeated search of a 461-group Steam
    library: 1.2 s -> 0.1 s, identical groups.
  • Progress reports phases (list, compare ends, hash) with totals.
  • Exclude globs are case-insensitive on Windows.

GUI

  • Engine "Ultra (NTFS MFT)" with a restart-as-administrator button (and the
    note that Windows blocks drag & drop into an elevated window); "Admin" next
    to the version; the scan card shows "NTFS MFT" and a status line explains a
    fallback.
  • Duplicate search skips program and game files unless included (the card
    offers a search that includes them); the progress card shows the three
    phases with counts, bytes and a real percentage.

Tests: 34 core (cache round trip and reuse, same ends / different middle,
in-order and out-of-order tree batches), 8 CLI, 23 UI helper tests.

…cache (0.10.0)

Scanning: NTFS MFT engine (Windows, administrator)
- New mft module: reads the volume's Master File Table straight from the
  volume (boot sector -> $MFT data runs -> 8 MB chunks read in parallel over
  four handles), applies update-sequence fixups and parses $STANDARD_INFORMATION,
  $FILE_NAME (8.3 aliases skipped, one entry per hard link), the unnamed $DATA
  stream (logical / allocated / compressed size) and reparse points. Metafiles,
  symbolic links and junctions are left out. All reads are bounds-checked.
- Without exclusions the tree is built straight from the records (no paths);
  with exclusions, and for the duplicate search, it emits the usual batches.
- Cloud placeholders and OneDrive-synced folders (from the OneDrive
  environment variables) are sized / listed through Windows, because only the
  cloud filter knows what is stored locally.
- Options::mft, CLI --mft; ScanTree::engine() and mft_fallback() say which
  engine ran and why the MFT was not used (no rights, not NTFS, ...).
- Measured (administrator, NVMe system drive): C:\Users\me 196 s -> 37 s with
  the walk vs the MFT, all of C: (4.8 M files, 4.9 M folders) in ~30 s; file
  counts and folder sizes match the walk. On a SATA SSD with millions of files
  the table is larger than the folders and the walk stays faster, which the
  settings text says.

Duplicates
- New dupes module: candidates of a repeated size are opened once, in
  parallel per file (not per size group), for their identity (hard links
  collapsed) and a BLAKE3 of the first and last 16 KB (files up to 32 KB are
  thereby hashed whole); only what still matches is hashed in full, big files
  on several threads via mmap. same-file dependency dropped.
- HashCache: path + size + modification time -> identity and hashes,
  zstd-compressed; find_duplicates_cached reuses it. CLI --cache FILE; the
  GUI keeps one in its data folder. A repeated search of a 461-group Steam
  library: 1.2 s -> 0.1 s, identical groups.
- Progress reports phases (list, compare ends, hash) with totals.
- Exclude globs are case-insensitive on Windows.

GUI
- Engine "Ultra (NTFS MFT)" with a restart-as-administrator button (and the
  note that Windows blocks drag & drop into an elevated window); "Admin" next
  to the version; the scan card shows "NTFS MFT" and a status line explains a
  fallback.
- Duplicate search skips program and game files unless included (the card
  offers a search that includes them); the progress card shows the three
  phases with counts, bytes and a real percentage.

Tests: 34 core (cache round trip and reuse, same ends / different middle,
in-order and out-of-order tree batches), 8 CLI, 23 UI helper tests.
@kingchenc
kingchenc merged commit 43e7058 into main Oct 22, 2026
7 checks passed
@kingchenc
kingchenc deleted the feat/mft-fast-dupes branch October 22, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant