[laurigates]

Draft for #590, as discussed there. Adds an off-by-default --shared-blocks flag that counts blocks shared between files (APFS clones, btrfs/XFS reflinks) once instead of once per file.

The example from the issue, five cp -c clones of a 200 MiB file:

$ dust -d1 .
200Mi   ├── clone5.bin│███████                                  │  17%
...
1.2Gi ┌─┴ .           │████████████████████████████████████████ │ 100%

$ dust -d1 --shared-blocks .
   0B   ├── clone5.bin│█                                        │   0%
...
200Mi   ├── big.bin   │████████████████████████████████████████ │ 100%
200Mi ┌─┴ .           │████████████████████████████████████████ │ 100%

It's four commits; the second and the fourth can each be dropped on their own:

1. macOS, clone ids. One getattrlist per file reads the clone id and EF_SHARES_ALL_BLOCKS, without opening the file. Files with the same clone id are one family; clean_inodes charges the first one and gives the rest size 0, the same way it already handles hard links. A clone that has since been partly rewritten is still counted in full.
2. exact-shared-blocks cargo feature, off by default. Files flagged EF_MAY_SHARE_BLOCKS get their extents mapped with F_LOG2PHYS_EXT, and each file is charged only for byte ranges no earlier file has claimed, so partly rewritten clones are handled too.
3. Linux via FIEMAP. Linux has no clone id, so it always uses the extent approach from commit 2. One statfs per device skips filesystems that can't share extents, so files on ext4, tmpfs or NFS are never opened. A new CI job runs a reflink test on btrfs and XFS loop images.
4. Private fd tables on Linux. With the flag, every file is opened and closed, and the threads of one process share an fd table whose spinlock both open() and close() take, so they queue on it. Each rayon worker now calls unshare(CLONE_FILES) at start, Linux and --shared-blocks only. That is only safe while every fd stays on the thread that opened it, so walk_dir resolves file types and drops the ReadDir on the thread that read the directory. That rule is the extra complexity this commit adds to the walker.

On your complexity concern: the walk change is a few lines in process_entry (plus, with commit 4, keeping each directory's ReadDir on the thread that read it), and the rest is platform code in platform.rs plus the extra accounting in clean_inodes. Without the flag none of it runs and Node is unchanged; time and peak memory with the flag off match master on macOS and Linux.

With the flag on (hyperfine -N -w 2 -r 10, release builds; Linux rows on a GCP n2-standard-8 VM with kernel 7.0, before commit 4):

	master	--shared-blocks
macOS APFS, ~/.cargo/registry	258 ms	316 ms
Linux ext4, kernel source tree	60 ms	61 ms
Linux XFS, never reflinked	54 ms	102 ms
Linux XFS, tree + its reflink	105 ms	275–506 ms
Linux btrfs, never reflinked	56 ms	178 ms
Linux btrfs, tree + its reflink	114 ms	379 ms

Commit 4 on 100,000 never-reflinked files with --shared-blocks, builds run in both orders:

	before	after
GCP n2-standard-8 (KVM), 8 threads	99.0–99.1 ms	95.6–95.8 ms
bare metal, Ryzen 9 3900X, 24 threads	76.9–78.5 ms	69.1–69.8 ms
OrbStack VM on an M4 Pro, 13 threads	74.9–75.8 ms	29.8–35.2 ms

Output is identical to before at every thread count tested, and the default walk and reflinked trees are unchanged. Outside that VM the gain is 3–11%, so commit 4 is the first one I'd drop if the walker rule is unwelcome.

The reflinked XFS case doesn't scale with threads. The time goes into XFS itself: FIEMAP's shared-extent check takes a lock per allocation group, so the XFS range above is 32 vs 4 allocation groups.

cargo test passes in both builds on macOS, and on Linux with commit 4. The reflink test from the new CI job passed on btrfs and XFS in a privileged ubuntu:24.04 container, but the job itself hasn't run in Actions yet.

Happy to drop commit 2 or 4, or change anything else, if that makes it easier to take.

Closes #590

🤖 Generated with Claude Code

https://claude.ai/code/session_01JkCD9BwvWZwfV7FXbbuD2E