[trevorlasn]

Written by Claude Code, an AI agent, for trevorlasn.

Fixes #649

On macOS, dust now lists each folder with getattrlistbulk(2) and takes a regular file's stat fields from the batch. It no longer calls lstat once per file. Output does not change. On a 61,007-entry APFS tree the walk takes 57% less wall time and 58% less CPU time. On a freshly mounted ExFAT image it takes 37% to 75% less wall time.

What changed

• src/macos_listing.rs (new, macOS only): lists one folder with getattrlistbulk. For each regular file it returns the size, allocated size, I/O block size, inode, device, and modified, accessed and changed times. Blocks are ATTR_FILE_ALLOCSIZE / 512, rounded up. That is how xnu's vn_stat computes st_blocks, and how Apple's fts(3) does it.
• src/dir_walker.rs: list_dir returns a folder's entries as a Listing: the read_dir Vec, as before, or the bulk Vec. ignore_file and process_entry take an Entry that holds one item of either. A file whose stat fields came with the listing skips get_metadata. Linux and Windows build only the read_dir variants, so their types and sizes stay as they were.
• src/platform.rs: tuple_from_metadata fills a StatFields and calls the new tuple_from_stat. That function holds the existing arithmetic, NTFS cap included, so both paths share it.
• Cargo.toml: libc for macOS only. Cargo.lock already had libc 0.2.191 through 6 other crates, so nothing new is downloaded. 0.2.133 is the first version with getattrlistbulk.

Folders, symlinks and other entries are still stat'ed. A folder has to be: at a mount point or a firmlink, the batch describes the object itself, and lstat describes what it leads to. /usr/share/snmp is one; its batch inode differs from its lstat inode.

When it falls back

A folder is listed with read_dir and one stat per entry, as on master, when:

• open or any getattrlistbulk call fails. A folder that can be listed but not searched (mode 444) fails with EACCES. A file system without the call fails with ENOTSUP. dust then reports the same errors as before.
• a record has no name, type or inode, or does not parse.
• the folder's link count shows more than 50,000 entries. On APFS and HFS+ a folder's link count is its entry count plus 2, up to 65,535. The calls for one folder run on one thread, and on APFS their cost per entry grows with the folder: about 1.3 µs of system time at 30,000 entries and 9 µs at 1,000,000. With 1,000,000 files in one folder, the bulk listing took about twice the wall time of the parallel stat.
• -f is on without -L, -x, -M, -A or -y. There master stats no file (it uses d_ino), and read_dir costs less than the bulk call.

ExFAT reports a link count of 1, so its folders are listed in bulk at any size. On the 3 GB ExFAT image, whose largest folder holds 62,000 entries, the walk took 62% of master's wall time and 21% of its CPU time at the default thread count. A file whose record has an error or lacks a field is stat'ed on its own.

Why a small module and not dua-core

• dua-core 6.1.0 has a public read_dir(path, Options) for macOS and Windows. Using it would add 4 crates to Cargo.lock: dua-core, crossbeam, crossbeam-channel and crossbeam-queue. This PR adds none.
• dua-core had four major versions in under three months: 3.0.0 on 14 August 2026, then 4.0.0, 5.0.0 and 6.0.0. The latest is 6.1.0, from 3 November. dust would carry each upgrade.
• dua-core's macOS code is 1,210 lines, and its build script turns the bulk call on by deployment target. dust would still map dua-core's Entry, Metadata and Error to its own stat tuple and error lists, and keep its own fallbacks so that errors print as before.
• The module here needs only libc, which dust already locks. It has 3 unsafe blocks: open, getattrlistbulk and close. Records are parsed from byte slices with bounds checks; a short or malformed record sends the folder to read_dir.
• It is 312 lines with comments, plus 87 lines of tests.

Results

Machine: Apple M4 Max (14 cores), 36 GB, macOS 26.3.1 (25D771280a), internal SSD. Both binaries were built with cargo build --release --locked and rustc 1.99.1; master is 079c552.

Each tree ran 9 rounds. A round runs each binary once at the default thread count and once with -T 1, and alternates which binary and which thread setting go first. The command was /usr/bin/time -l dust -P -c -n 5 [-T 1] <tree>. The table gives medians. CPU is user plus system time. Peak is time -l's peak memory footprint. The machine was shared with other work, such as builds: the 1-minute load average, which also counts dust's own threads, was 9.6 to 25.0 at the start of the runs. Both binaries ran back to back in each round, so both saw the same load.

The trees:

• web/: a Next.js app with its node_modules: 61,007 entries (55,982 files, 4,007 folders, 1,018 symlinks).
• /usr/share: 20,030 entries (15,588 files, 886 folders, 3,556 symlinks), on the sealed system volume.
• A 1 GB ExFAT image made with hdiutil: 43,890 entries, in copies of the test tree (see below) and of web/src, and one folder of 29,354 entries. Copying to ExFAT adds an AppleDouble (._) file for each file with extended attributes. "Fresh mount" means the image was detached and attached again before each run.
• A 3 GB ExFAT image: 82,044 entries. One folder holds 62,000 (31,000 empty files and their ._ files), and 20 folders hold about 1,000 each.
• One APFS folder with 30,000, 100,000 or 1,000,000 empty files.

Tree	Threads	Wall s, master	Wall s, branch	CPU s, master	CPU s, branch	Peak MiB, master	Peak MiB, branch
web/, APFS	default	0.14	0.06	1.17	0.49	33.9	27.2
web/, APFS	-T 1	0.34	0.19	0.33	0.18	30.0	23.3
/usr/share, APFS	default	0.08	0.03	0.47	0.25	10.6	11.2
/usr/share, APFS	-T 1	0.11	0.06	0.10	0.05	8.1	7.7
1 GB ExFAT image, warm	default	0.08	0.06	0.35	0.11	31.5	27.3
1 GB ExFAT image, warm	-T 1	0.22	0.10	0.12	0.05	27.9	22.2
1 GB ExFAT image, fresh mount	default	1.61	1.02	0.46	0.12	32.3	28.2
1 GB ExFAT image, fresh mount	-T 1	2.73	0.69	0.42	0.06	28.5	21.5
3 GB ExFAT image, warm	default	0.13	0.08	0.70	0.15	54.0	44.1
3 GB ExFAT image, warm	-T 1	0.26	0.08	0.17	0.05	51.0	37.2
30,000 files in one folder, APFS	default	0.05	0.06	0.23	0.08	24.8	20.5
30,000 files in one folder, APFS	-T 1	0.08	0.05	0.07	0.04	21.3	17.1
100,000 files in one folder, APFS	default	0.23	0.23	0.98	1.02	71.9	71.9
100,000 files in one folder, APFS	-T 1	0.32	0.30	0.31	0.29	69.6	68.2
1,000,000 files in one folder, APFS	default	15.02	14.69	42.38	42.18	580.5	580.9
1,000,000 files in one folder, APFS	-T 1	53.59	49.96	15.28	14.02	592.1	592.7

• On web/, the walk takes 57% less wall time and 58% less CPU time at the default thread count, and the peak footprint is 20% lower. Part of that is paths: the branch allocates each listed file's path at its exact length, where DirEntry::path() can leave up to half of the buffer unused, and dust keeps every path in its tree.
• On a freshly mounted ExFAT image, the walk takes 37% less wall time at the default thread count and 75% less with -T 1.
• At the default thread count, the folder of 30,000 files took 0.01 s longer, with 65% less CPU time: one thread lists a folder, where master stats its files on all threads.
• The folders of 100,000 and 1,000,000 files are over the 50,000-entry limit, so both binaries run the same read_dir code there. Those differences are noise. With 1,000,000 files and -T 1, wall time ranged from 35 s to 82 s within each binary: the folder has more files than the vnode cache holds (kern.maxvnodes is 263,168).
• Each walker thread keeps one 128 KB buffer for the listing, up to 1.75 MB with 14 threads. On /usr/share at the default thread count, the peak was 0.6 MB higher.

The scripts and the raw numbers for every run are available if you want them.

Output equality

A script ran master (079c552) and this branch on each tree with -P -c -w 200 and each of 58 flag sets. It compared stdout, stderr and the exit code. The sets:

(none) -s -f -f -s -d 0 -d 2 -L -x -X <tree>/a -X <tree>/src -X <tree>/node_modules -e \.rs$ -v \.txt$ -e \.json$ -v test -t -t -s -z 10K -i -M -30 -M +30 -A -1 -A +1 -y -1 -y +3000 -m a -m c -m m -F -D -p -n 50 -r -b -T 1 --print-errors -f -L -f -x -f -M -30 -f -e \.rs$ -f -i -s -L -s -x -L -x -j -j -s -j -f -j -d 2 -j -o b -j -p -o b -s -o b -n 100000, and -o b -n 100000 with -s, -f, -L, -x, -i -p, -F -p, -D -p, -m m -p, and -T 1 --print-errors.

Tree	File system	Sets	Same
Test tree (below)	APFS	58	58
web/	APFS	58	58
An APFS clone (cp -c) of web/	APFS	58	58
/usr/share	APFS, sealed system volume	58	58
1 GB image	ExFAT	58	58
3 GB image	ExFAT	58	58
300 MB image: copies of the test tree, web/src and one node_modules package	HFS+	58	58
A folder holding the three mounted images	APFS, ExFAT, HFS+	58	58
One folder with 30,000 files	APFS	58	58
One folder with 100,000 files	APFS	58	58
One folder with 1,000,000 files	APFS	6	6
Test tree, x86_64 build under Rosetta 2	APFS	58	58
web/ clone, x86_64 build	APFS	58	58
1 GB image, x86_64 build	ExFAT	58	58

The 1,000,000-file folder ran (none), -s, -f, -x, -M -30 and -j -o b.

The test tree has: 3 hard links to one file across 2 folders, symlinks to a file, to a folder and to nothing, a FIFO, a file with APFS transparent compression, an APFS clone, a file with a resource fork, a 100 MB sparse file, Unicode names, hidden files, an empty folder, a mode 444 folder, a mode 000 folder, and a folder with 3,500 entries.

--print-errors and the error summary print the paths from a HashSet, so their order changes from run to run on master too. The script compares those lists as sorted items. 14 comparisons differed in that order only, all of them with -L.

With an earlier build of this branch, 3 of about 150 master runs on the mount-image tree lacked 1 or 7 AppleDouble (._) files on the 3 GB ExFAT image, with no error. The branch's totals were right in every run. master drops a file whose lstat fails and does not report it (build_node returns None). That is one possible cause; I did not confirm it. 30 more rounds of each binary showed no drop, and a probe that ran lstat on every listed entry saw no failure in 30 walks. The files had not changed since they were created.

A separate probe compared each field from the batch with read_dir and lstat, file by file. There were no differences on APFS (web/ 55,982 files, /usr/share 15,588, the test tree 3,520) or on HFS+ (15,474). On ExFAT, every difference was the inode of an empty file. An empty file there has no cluster, so FSKit gives it a new inode number on every lookup: lstat and the batch each get a fresh one. They are unique within a walk, so hard-link counting is unaffected; master's -f already takes d_ino from read_dir in the same way.

Tests

cargo test --locked on macOS 26.3.1 with rustc 1.99.1, on master and on this branch:

Test binary	master	branch
unit tests (src/main.rs)	36 passed, 1 ignored	38 passed, 1 ignored
tests/tests.rs	0	0
tests/test_exact_output.rs	8 passed	8 passed
tests/test_flags.rs	34 passed	34 passed
tests/tests_symlinks.rs	5 passed	5 passed

No existing test needed a change. The integration tests run the built binary, so on macOS they now go through the bulk listing. The 2 new unit tests are in src/macos_listing.rs:

• test_bulk_listing_matches_read_dir_and_lstat lists a temp folder of 3,007 entries, more than one buffer holds. Each name must be one that read_dir returned, and each type and inode must match lstat. Each regular file's size, blocks, block size, inode, device and three times must match lstat too. A link count over the limit must leave the folder to read_dir.
• test_bulk_listing_leaves_unsearchable_folder_to_read_dir checks that a mode 444 folder is left to read_dir. It checks nothing when run as root, who can search the folder.

Other checks, all passing:

• cargo fmt --check
• cargo clippy --locked -- -D warnings, and the same with --target x86_64-unknown-linux-gnu, --target x86_64-pc-windows-msvc and --target x86_64-apple-darwin
• cargo check --locked --target x86_64-unknown-linux-gnu
• typos 1.28.4, the version CI pins
• cargo build --release --locked --target x86_64-apple-darwin; that binary ran the equality sets above under Rosetta 2

For review

• The new module is 312 lines, plus 87 lines of tests, with 3 unsafe blocks. The whole diff is 646 lines added and 43 removed.
• tuple_from_metadata now calls tuple_from_stat on every Unix platform. The arithmetic is the same; only its place changed.
• Draft feat: add --shared-blocks to count cloned and reflinked blocks once #645 (--shared-blocks) also changes dir_walker.rs and platform.rs and adds libc, so whichever lands second needs a rebase. After this PR, feat: add --shared-blocks to count cloned and reflinked blocks once #645's getattrlist call per file for ATTR_CMNEXT_CLONEID and ATTR_CMNEXT_EXT_FLAGS could take both from the same batch.
• Symlinks and other entries that are not regular files are still stat'ed. web/ has 1,018 symlinks. Taking their fields from the batch too would save more calls, but would mean handling -L in the batch path.
• On ExFAT an empty file's inode number differs between the batch and lstat, as described above. Nothing dust prints shows it.

Not verified

• Intel hardware. The x86_64 build ran under Rosetta 2 only.
• SMB, NFS, AFP, FAT32, NTFS (macOS's read-only driver) and macFUSE volumes. A file system without the call should fail it with ENOTSUP and fall back; I did not run one.
• A physical external drive. The ExFAT and HFS+ volumes were disk images, so even a fresh mount read from the host's page cache.
• A cold APFS cache, after a reboot.
• Other macOS versions. getattrlistbulk has been in macOS since 10.10.
• Case-sensitive or encrypted APFS volumes, and Time Machine backups with folder hard links.
• The 50,000-entry limit comes from this one machine.
• A getattrlistbulk failure in the middle of a folder. The code hands the whole folder to read_dir; I never saw it happen.
• Linux and Windows builds were checked and linted, not run. Their code path is the same read_dir path as before.

🤖 Generated with Claude Code