Skip to content

libfatx: Grow directories past their first cluster - #106

Merged
mborgerson merged 1 commit into
XboxDev:masterfrom
xmutantson:libfatx-grow-directories
Oct 29, 2026
Merged

mborgerson merged 1 commit into
XboxDev:masterfrom
xmutantson:libfatx-grow-directories

Conversation

@xmutantson

@xmutantson xmutantson commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #56.

fatx_alloc_dir_entry() never extended a directory correctly once its first cluster was full, and in both failing cases it writes outside the directory:

  • End marker in the last slot of a cluster. The marker is moved to entry + 1, which is past the end of the cluster. fatx_dev_seek_cluster() doesn't bound the offset, so this rewrites the first byte of whatever cluster follows on disk.
  • Every slot in use. A cluster is attached to the chain, but the entry is then written at slot entries_per_cluster of the old cluster, which is the first 64 bytes of the next cluster on disk.

In both cases the new entry is never found afterwards, so creating the file fails with EPERM. That is the error in #56, which shows up with rsync/cp into game folders with more than 256 entries (16 KiB clusters) or 512 entries (32 KiB clusters). What #56 doesn't show is the write outside the directory: because clusters are allocated sequentially, the cluster after a directory's first cluster is usually the first file or subdirectory created inside it. In a scratch image, the corrupted neighbor was a sibling directory created right after the full one, and the file inside it no longer read back correctly.

Reading a directory whose last cluster is completely full (no end marker) also failed, because fatx_next_dir_entry() treated reaching the end of the chain as an error.

Changes

  • fatx_alloc_dir_entry() walks the cluster chain instead of stepping past the end of a cluster. When the end marker is in the last slot, it moves into the next cluster of the chain; if there is none, a new one is appended. When every slot is used, the entry goes in the first slot of a new cluster.
  • New clusters for a directory are filled with 0xFF end markers (as fatx_init_root() does) before being attached, and are released again if they can't be initialized or attached.
  • fatx_next_dir_entry() returns FATX_STATUS_END_OF_DIR at the end of the chain after a full cluster; fatx_open_dir(), fatx_get_attr_dir(), fatx_unlink(), fatx_rmdir() and fatxfs read_dir treat that as the end of the directory.
    • In fatx_open_dir() this also stops a lookup that runs off the end of a directory from continuing with the next path component in the wrong directory.

Testing

Two tests in pyfatx/tests/test.py:

  • test_directory_grows_past_one_cluster: 600 files in one directory (three 16 KiB clusters) plus a canary file in a sibling directory created right after it; checks the listing, every file's content, the canary, slot reuse after unlink, and rmdir.
  • test_directory_with_full_last_cluster: builds a directory whose only cluster is full with no end marker (255 files, then the end marker in the last slot is overwritten with a 256th entry through fatx_write_dir()), then checks the raw statuses: walking the directory ends with FATX_STATUS_END_OF_DIR and a missing name gives FATX_STATUS_FILE_NOT_FOUND, where master returns FATX_STATUS_ERROR for both. It then checks that the directory grows from that state.

On master both new tests fail (the other 17 pass); with this change all 19 pass. I also ran a fatxfs mount test with 1,200 files in one directory on a 32 KiB-cluster F: image, checked before and after a remount, and used the patched fatxfs to copy ~430 GB of game folders (some directories with up to 4,000 entries) to a real 500 GB drive; a checksum comparison against the source afterwards matched every file.

Rebased onto master after #107.

Separately, the same copy hit the prefix match in fatx_open_dir() that #99 fixes (BloodRayne/ resolving to an earlier BloodRayne 2/). This PR doesn't touch that comparison, so the two shouldn't conflict.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An allocation failure during directory growth leaks an allocated cluster, and the tests neither exercise marker-free full-chain termination and growth nor verify the intended low-level statuses.

2 open issues
What changed in this PR

This PR addresses #56 by enabling FATX directories to grow beyond one cluster without writing into neighboring clusters. It also treats the end of a full directory chain as normal directory termination.

Changes:

  • Walk directory cluster chains and initialize new clusters with end markers before attaching them.
  • Handle marker-free directory termination in library operations and FUSE directory listing.
  • Add tests for multi-cluster growth, neighboring-file integrity, slot reuse, and cluster-boundary behavior.
File Description
pyfatx/​tests/​test.py Adds directory-growth and cluster-boundary regression tests.
libfatx/​fatx_dir.c Implements directory growth and updates chain-end handling.
libfatx/​fatx_attr.c Maps directory chain exhaustion to file-not-found.
fatxfs/​fatxfs_fuse.c Treats directory chain exhaustion as successful listing completion.

Review effort: Balanced

Comment thread libfatx/fatx_dir.c Resolved
Comment thread pyfatx/tests/test.py Resolved
@mborgerson

Copy link
Copy Markdown
Member

Thanks for the PR! Please rebase and check out the copilot reported issues

Adding an entry to a directory never extended it correctly once its
first cluster was full:

- With the end-of-directory marker in the last slot of a cluster, the
  marker was moved to the slot after it, which is the first entry of
  the next cluster on disk, not the next cluster of the directory.
- With every slot in use, a new cluster was attached but the entry was
  written one slot past the end of the old cluster, again landing in
  whatever cluster follows it on disk.

Both cases overwrite another file or directory, and the new entry is
never found, so creating the file then fails with EPERM (XboxDev#56). Reading
a directory whose last cluster is completely full also failed, because
reaching the end of its chain was treated as an error.

Allocation now walks the directory's cluster chain, moves the end marker
into the next cluster of the chain, and appends a cluster filled with
end markers when the chain is full. Reaching the end of the chain after
a full cluster is reported as the end of the directory.

Fixes XboxDev#56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the directory-growth and chain-termination fixes.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@mborgerson
mborgerson merged commit 488b53d into XboxDev:master Oct 29, 2026
18 checks passed
@mborgerson

Copy link
Copy Markdown
Member

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Operation not permitted

3 participants