[egibs]

Summary

Follow-up to #1777. Deeply nested paths no longer cost memory quadratic in their length, and paths past PATH_MAX are extracted and scanned instead of skipped or misread.

What was wrong

Measured on Linux with a zip whose single entry nests a file 32,765 directories deep (a 64 KB name, about 130 KB of zip):

• The directory walkers (file.WalkDir, the scan walk) recursed once per level and kept every level's path string and directory root alive, so memory grew with the square of the path's length.
• Paths past PATH_MAX (1,024 bytes on macOS, 4,096 on Linux) broke in three ways, because extractors and upx open and check paths whole:
  • a nested archive there was scanned as an opaque file, and its contents were never scanned;
  • an archive at such a path in a scanned directory was skipped entirely ("file name too long");
  • every file there holding the UPX! marker was classified as UPX, because upx could not open it and any failure other than "not packed" counts as packed.

	Before	After
mal scan, file 32,765 levels deep	scanned, 3.6 GiB RSS	scanned, 383 MiB RSS
ExtractArchiveToTempDir, same zip	failed, 3.4 GiB RSS	extracted, 220 MiB RSS
Nested archive past PATH_MAX	scanned as one opaque file	extracted and its files scanned
Archive past PATH_MAX in a scanned directory	skipped, nothing reported	extracted and its files scanned

Changes

• file.Walker: an iterative walker with fs.WalkDir's order, error entries, and SkipDir/SkipAll semantics (TestWalkDirMatchesFSWalkDir still compares it to fs.WalkDir). It keeps one path buffer and lets go of a directory once it yields the directory's last entry, so a chain holds one open directory. file.WalkDir runs on it. The scan walk keeps its 16-way parallelism by Detaching subdirectories into goroutines that each run their own Walker, with results spliced back in order (TestWalkScanPathMatchesASequentialWalk).
• file.OpenRoot: when the OS refuses a path as too long, it opens a root on a prefix of at most file.MaxDirectPath bytes and resolves the rest through that root. Shorter paths take the existing route unchanged. The pkg/file path helpers and programkind.File go through it.
• Staging (pkg/archive/stage.go): an archive at a path longer than file.MaxDirectPath is hard-linked (or copied) through its directory's root to a short path, extracted there, and the result moved into place with root.Rename. Extractors and their symlink validation are unchanged. Symlinks they create stay within what they wrote, and the move keeps their relative layout. Single-file extractors that name output after the archive's directory get a staged copy in a directory of that name. Files, and so report keys, land exactly where they would have.
• UPX: IsValidUPX checks a short temporary copy of a file at a long path. Passing the descriptor as /dev/fd/3 doesn't work: on Linux that path is a symlink, which upx refuses.
• Nested archive detection in nestedTree.extractFile reads content through the root.

macOS deadlock in yara-x

The first make verify run on this branch stopped making progress in pkg/report. Many goroutines were inside yrx_compiler_build and yrx_rules_destroy, all idle. A single-process reproducer (one thread compiling the bundled rules in a loop, twelve compiling, scanning, and destroying a small rule set) stalls on macOS after about 116,000 operations (40 seconds). Native stacks show every thread waiting for libunwind's DwarfFDECache write lock, which no thread holds:

Compiler::build → CodeMemory::publish → __register_frame   → DwarfFDECache::add → __psynch_rw_wrlock
drop(Rules)     → CodeMemory::drop    → __deregister_frame                      → __psynch_rw_wrlock

On Linux the same reproducer completed 361,680 operations over 10 minutes without stalling.

• internal/yrx wraps Build, Compile, ReadFrom, and Destroy, and runs them one at a time on macOS only. Build replaces the binding's finalizer, so rules the garbage collector frees are destroyed through the lock too.
• Through internal/yrx, the reproducer ran 219,318 operations over 3 minutes on macOS without stalling.
• A forbidigo rule with type analysis rejects direct calls outside internal/yrx. It caught three method-value uses (t.Cleanup(rules.Destroy)) that grep had missed.
• The upstream fix belongs in yara-x, which already disables wasmtime's native unwind info on musl. I'll file that separately.

Testing

• New tests fail on the old code and pass now:
  • memory held at the deepest file stays within 16× the path length: TestWalkDirDeepChainHoldsLinearMemory and TestWalkScanPathDeepChainHoldsLinearMemory (2.7 MB held before, against a 102 KB limit; 0 and 18 KB now). Their 6.4 KB path exceeds PATH_MAX but stays under the 8 KB that AppArmor checks; past that, GitHub's runners refuse to create the path at all;
  • TestScanNestedArchiveBeyondPathMax and TestScanArchiveBeyondPathMax;
  • TestExtractArchiveToTempDirStagesDeepNestedArchives: zip, gzip, zstd (parent-named output), and a corrupt archive produce identical trees at a short path and past PATH_MAX, with nothing left from staging;
  • TestIsValidUPXBeyondPathMax, with a stub upx that answers from the file it opens;
  • TestPathHelpersBeyondPathMax and TestSplitLong.
• internal/yrx tests: build, and round-trip through ReadFrom. TestCallsRunInsideTheLock records that every wrapper runs lock, then the yara-x call, then unlock, and TestLockHoldsTheMutex (macOS) checks that lock holds the mutex. Both are deterministic and fail when either piece is removed.
• Walker unit tests: Detach, NewDirWalker, SkipDir on files and directories, and Close partway through releasing every descriptor.
• make verify passes on darwin.
• make refresh-sample-testdata (run on Linux, as CI does) refreshed all 396 samples with no changes.