[Japabu]

Problem

These open severity: [data-loss] records describe defects that still exist in the current code. Each commit fixes one, in the component that should own the invariant, and deletes its record.

commit	record	resulting behavior
fileserver write-back	a-write-back-the-device-refused-waits-for-the-next-write	A refused sync stays due, whether it was refused whole or for one file, and whether a client or the write-back asked for it. It is retried after 2 s, then at a doubling interval capped at 64 s. The first sync that lands resets the interval.
fileserver DATA reuse test	a-reallocated-extent-on-data-keeps-the-deleted-files-bytes	Adds the host test the record's exit asks for. A file written into blocks a delete freed reads back whole from the device after a remount. The BCFS DATA the record measured was replaced by bcachefs in #1152.
toyos-fat32 shrink	a-shrink-frees-clusters-before-the-entry-stops-naming-them	A shrink now writes the entry's shorter size, and the cleared first cluster for a shrink to zero, before it frees anything. A stop now leaves at most a leak.
toyos-fat32 recovery (r2: 958cc39cb)	a-stop-mid-append-…, nothing-reclaims-…, a-rename-that-stops-…	Fat32::recover runs on every writable FAT mount (LOG) and first reads the volume whole without writing: every directory walked, every file chain followed to its size, the FAT copies compared. Anything a ToyOS stop does not leave makes it refuse by name and write nothing; the file server then serves that volume read-only. Otherwise it repairs, in this order: split FAT entries take the active copy, a half-done rename's second name is erased and .. repointed, partial long names are erased, chains past their size are ended, and unreached clusters are freed.
shell (+ f8db19fc2, a type annotation the ToyOS build needed)	the-shell-runs-a-command-whose-redirect-it-could-not-open	If a redirection does not open, the shell says why, sets status 1 and runs nothing. 2>&1 onto a file shares stdout's open file instead of opening the path a second time.
nvme	the-nvme-driver-reads-a-namespaces-format-and-page-size-partly	The driver reads the full 6-bit LBA format index and checks it against NLBAF. It refuses, by name, any format carrying metadata and any controller whose CAP.MPSMIN is not 4 KiB.

Decisions

• Recovery runs on every writable mount, without the dirty bit. The record proposed gating it on fatgen103's ClnShutBitMask. That bit would add a dirty write and a clean write to FAT[1] around every sync, and logkeeper syncs every round. A full scan of the LOG volume costs a few hundred KiB of reads once per boot. A clean volume is left untouched.
• Ordered writes are a flush apart (r3): a remove's erase and its free, a shrink's entry and its free, and a rename's new name and the old one's erase. A write-back cache orders nothing within a flush. Without this, a stop could leave LOG refused for good, or a renamed file under neither name.
• Split FAT copies are accepted when each copy reads whole, not when there is at most one split. The file server's cache writes a flush's FAT blocks of both copies in any order. A stop inside that flush therefore splits every entry in the blocks it had not written yet, and the stopped-flush test met exactly that. A torn or foreign copy does not read whole, and then neither copy is written over.
• A FAT volume that recovery refuses is served read-only, not left unserved. logkeeper's writes are then refused.
• A half-done rename (coordinator's ruling): the write order stays as it is, and recovery keeps one of the two names.
• Recovery cuts a chain to its size rather than extending the size to the chain. After a stop, nothing shows whether the clusters past the entry's size hold the bytes that were written there.
• A refused redirection in a pipeline waits for the stages already started. They end on the pipe the refused stage would have read.

Checks

r3 (e41116092, 258e9197d):

• --ci host: exit 0, 109 steps.
• --build-only: exit 0.
• stopped_anywhere covers an unlink, a shrink and a rename: every stop is recovered and checker-clean, and no file is lost. The negative control removes each flush and turns its test red.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/4a65ca68ebd29b22833d126668cf3198d2fb2510/r4-review-r2

r2 (33291dbf0):

• cargo run -- --ci host: exit 0, 109 steps.
• cargo run -- --build-only and cargo run -- --build-only --arch aarch64: both exit 0.
• cargo clippy -p toyos-fat32 --all-targets -D warnings: exit 0.
• Recovery tests (lib/fat32/tests/refused_writes.rs):
  • Every scenario stopped at every write, each way, is recovered and then checker-clean, with rename no longer exempt. A second recovery finds nothing, and all six repairs are exercised.
  • Four volumes are refused byte-for-byte unchanged: a directory with a broken link, a file cross-linked into a directory, two different files on one chain, and FAT copies that differ where one does not read whole (FAT 0 torn, then the mirror).
  • Copies that differ in 128 entries, where each reads whole, are recovered.
• File server:
  • The stopped-flush test now recovers every stop state.
  • a_volume_recovery_refuses_is_served_read_only_and_untouched.
• Negative controls, each applied, run red and restored:
  • three refusals removed, each failing its refusal test;
  • six repairs skipped, each failing the stop or split test;
  • a refused volume served writable, failing the file server test.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/acb50cdf526761c716e1a9f3c00e46b64621d459/r3-review-r1

r1, unaffected by r2 (r2 changes only recovery, the file server's FAT mount, the shell's interrupt registration and a blank line):

• Write-back: a_refused_sync_stays_due_at_a_backed_off_interval.
• DATA reuse: a_file_in_a_deleted_files_blocks_holds_its_own_bytes_on_the_device. Negative control: dropping data writes fails it.
• Shrink: a_stop_at_any_write_leaves_only_the_named_windows, with both truncate own arms deleted. Negative control: the old order fails at truncate write 2.
• NVMe: five geometry tests.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/8ec7ddbb848ddb1936dc1408732d7402407d7244/r1

Guest: run 42030077569 is queued at 258e9197d (libc_children, nvme_disk_boots, the_log_lands_on_the_boot_stick).

• Run 42023343796 at 33291dbf0: the_log_lands_on_the_boot_stick passed and spawn_system passed.
• In that run, nvme_disk_boots was red on a race with READY, which 258e9197d fixes.
• tty_interrupt was red too. It reds the same way on kernel-no-tls, so it is a sighting independent of this branch.

Metal at 4937c2bdc: T14 usbload + testcases 2 passed (https://github.com/ToyOSOrg/evidence/tree/4bcf3c4d6243234fc2b10ede9744614ef6529562/r5-metal-t14); AMD testcases passed (https://github.com/ToyOSOrg/evidence/tree/7408fab3cd3fdc958780436231dfa5b0aea6ca9b/r6-metal-amd). Both drives up under the geometry checks; LOG served writable after recovery, both sticks clean.

• Metal, earlier:
• T14 at 668160b1e (r1's recovery): both rows passed. The NVMe geometry checks accepted the T14's drive.
• Recovery left a clean real stick unchanged. These boots show the clean-stick case only, not a repair.
• r2 needs fresh metal, to be staged after the focused review. AMD testcases at 668160b1e is still queued.

Risks

• Fat32::recover writes to a stick at mount, but only after reading the whole volume and finding nothing but stop states. Host tests cover every stop state the crate's calls leave, a stopped cache flush, and four kinds of volume it must refuse unchanged. The T14 boots at 668160b1e ran the r1 recovery on a clean stick only, and show no repair on metal. r2 needs fresh metal.
• A drive whose namespace format carries metadata, or whose MPSMIN is not 0, is now refused instead of mis-driven. No bench drive is expected to have either; the metal boots confirm it.

Not fixed here

The open severity: [data-loss] records left in issues/ are not in this PR. The logkeeper and held-file rulings are in #1190, which is stacked on this one.

🤖 Generated with Claude Code

[Japabu]

T14 metal at 668160b1e (boots run by the coordinator, judged here with the readbacks in place): nvme_leaves_the_benchs_disk_alone and the_disk_survives_a_reset_under_load both PASS, 2 passed, 0 failed.

• The T14's drive came up under the new geometry checks: 512-byte sectors, 4 I/O queues.
• Both log partitions read back clean under toyos-fat32-check. Fat32::recover on a host copy of each changed nothing, and a second recovery changed nothing either.
• This shows recovery leaves a real clean stick alone. It does not show a repair on metal: the reset left no stop state, and each boot reflashes LOG.

AMD testcases is still pending. Evidence: https://github.com/ToyOSOrg/evidence/tree/b06d62de9c5125008e84d2eb4dca7682186af061/r2-metal-t14

[Japabu]

Review r1 of #1185 at f8db19f

Assessment. Each fix sits with the component that owns it: write-back policy in the file server, write order in toyos-fat32, recovery in toyos-fat32 run by the server that owns the writable mount, redirect handling in the shell, and geometry in the NVMe driver. The PR adds about 650 lines and deletes about 340, of which roughly 250 are closed records. Most of the growth is recover.rs (263 lines) and tests. The write-back, shrink, shell and NVMe commits are correct as far as I can trace them. The NVMe field positions agree with NVM CS 1.0 §4.1.5.1 (FLBAS 3:0 plus 6:5, LBAF MS 15:0, LBADS 23:16) and Base 2.0 CAP.MPSMIN 51:48.

The problem is recovery. It decides what is unreachable from a tree walk that can be incomplete, and it writes before it knows whether it understood the volume.

Required

1. lib/fat32/src/recover.rs:89 (also :140–143, :150–153, :168–170). Recovery frees the clusters of every directory it could not walk.

free_unreached always runs, whatever reach_tree saw. A directory is not scanned in three cases:

• its chain fails reach_chain, either through a broken link (CorruptChain) or because one of its clusters was already marked by an earlier entry;
• DirScan returns CorruptDirectory partway through;
• the root itself fails.

In each case the code adds 1 to unrepaired and moves on. It never marks the clusters of the files and subdirectories below that directory, so free_unreached sets every one of them to 0. Their directory entries still name them, so later allocations overwrite those files.

Concrete cases:

• A subdirectory whose one-cluster chain reads FAT[x] = 0 loses every file inside it.
• A file whose chain, within its own size, is cross-linked into a directory scanned after it loses that directory's whole subtree. Which directory loses depends on walk order.
• A root chain with a bad link frees almost the whole volume.

So "counted and left as it is" (module header lines 14–16, the PR's "Cross-links and broken links are counted and left", and "its writes are leak-shaped") is false. LOG is on the boot stick, which macOS mounts read-write, so a FAT that ToyOS did not write is the expected case. This is what the brief means by repairing a FAT it does not understand.

2. recover.rs:85, 99–129. Mirrors are overwritten before anything has been checked.

reconcile_mirrors copies FAT 0 over every differing mirror entry before the walk runs, with no limit on how many entries it rewrites. A ToyOS stop leaves at most one split entry (repair.rs), and recovery's own writes go mirror first, active last, so they also leave at most one. Many differing entries mean something else happened: a torn or corrupt FAT 0 sector, or a foreign writer. In that case the mirror is the only redundant copy, and this code destroys it. Combined with finding 1, a damaged FAT 0 sector becomes freed files instead of a refusal.

Fix for 1 and 2. Classify the volume read-only first. Write nothing unless everything found is a state a stop can leave. Concretely:

• every directory was scanned completely, so the walk is complete;
• the split FAT entries are within what a stop leaves;
• the remaining inconsistencies are only the counted ones, such as rename's cross-link, which do not hide reachable clusters.

Otherwise refuse by name and leave the volume untouched. I would also make the mount read-only in that case, but how to handle it beyond "write nothing" is your call.

This is the step the closed nothing-reclaims-… record asked for first ("detects each of those states without writing anything"), and it would make the module header true.

The tests only start from states that ToyOS's own stops produce, so nothing exercises a volume recovery must refuse. Add host cases to the cheapest place, the refused_writes.rs fixture:

• a directory chain with a broken link;
• a file chain cross-linked into a directory;
• a FAT 0 sector that differs from FAT 1 in many entries.

Each should assert that recovery changes no byte of the medium. The current code is the negative control for the first two.

3. system/fileserver/src/fat.rs:120–122. The recovery message is false.

It prints {cut} clusters past a file's size … freed, but cut counts files whose chain was ended (recover.rs:177 and :189 add one per file, including an empty file's entry). The clusters past each file's size are counted in lost (Recovered::cut doc, line 34). This is the line the metal readback is told to look for, so it must say what was counted.

Suggestions

• recover.rs:14: "What a stop cannot leave … a cluster two entries reach" contradicts issues/a-rename-that-stops-…, which says a stopped rename leaves exactly that. Reword when fixing 1.
• fat.rs: when unrepaired > 0, consider mounting LOG read-only. On a cross-linked volume, a later remove of one name frees a chain the other name still uses. That predates this PR and the rename record owns it.
• Not gating on ClnShutBitMask is a sound decision. ToyOS's usual end is a reset, and the bit would cost ordered FAT[1] writes around every logkeeper sync. LOG is about 33 MiB (FAT32_MIN_BYTES), so the "few hundred KiB" scan holds. Running recovery on every mount does put every foreign volume through it, which is why findings 1 and 2 are required rather than optional.
• system/shell/src/main.rs:673: the wait after a refused last-stage redirect does not call waiting_for or waited. So sleep 1000 | cat > /bad waits for a child the interrupt path does not know about. Registering the children as the normal path does would keep Ctrl-C working.
• system/fileserver/src/data.rs:765: the new test is missing the blank line that separates tests in this module.
• PR body: "Not fixed here: the handoff lists…" points at a private scratch file. The remaining records are already in issues/, so point there. The task brief also mentions seven negative controls; the evidence README shows six (1 + 4 + 1). The PR body itself does not give a number. That correction is the orchestrator's.

Evidence

• ci-host-r2.log at 668160b1e: exit 0, 109 steps. f8db19fc2 changes only a type annotation in shell, which --ci host does not build, so reusing that result is valid. The image builds at f8db19fc2 are reported exit 0. I did not see their logs in the evidence tree.
• The six negative controls in r1/ each fail in the expected place. They show that the stop-state repairs are needed. They cannot show that recovery refuses a volume it does not understand (see finding 2's tests).
• Guest run 42016229966 is still queued at f8db19fc2. The shell claim depends on it (libc_children), as does any NVMe or LOG guest behaviour.
• T14 metal (PR comment, b06d62de9): recovery left a real clean stick unchanged, and the NVMe geometry checks accept the T14's drive. As the comment says, it shows no repair on metal. AMD is pending.

SEND BACK: findings 1–3.

[Japabu]

r2 at 33291dbf0: response to review r1

1 and 2: fixed, by checking first. Fat32::recover (958cc39cb) reads the whole volume before it writes anything:

• every directory is walked whole;
• every file chain is followed to its size;
• the FAT copies are compared.

It refuses by name and writes nothing (Recovery::Refused) on finding any of these:

• a link that is not one, or a directory that does not read whole, including the root;
• a cluster reached by two entries that differ in more than their name;
• a chain short of its size, or one ending in a bad link;
• an empty file that names a cluster;
• a directory with no ..;
• FAT copies that differ where some copy does not read whole.

free_unreached therefore runs only after a complete walk. The file server serves a refused volume read-only and says why.

One deviation from the suggested bound on split entries. Recovery accepts copies that differ where each copy reads whole; it does not cap them at one split. I tried the cap first: the file server's stopped-flush test (an_append_stopped_with_any_writes_of_a_flush_on_the_disk_reads_back_the_last_sync) failed, because a stop inside one flush of the server's cache splits every entry in the FAT blocks of either copy that the flush had not written yet. Those are ToyOS stop states on LOG's real path, so the cap would serve every such stick read-only.

What tells a stop from a torn or foreign copy is that each copy is then the volume before or after the flush, so it reads whole. walk_copy walks each mirror as if it were the active copy. If any copy fails that walk, recovery refuses and neither copy is written over.

copies_that_differ_where_one_does_not_read_whole_are_refused_and_left covers FAT 0 torn, then the mirror, each with 129 differing entries. copies_that_differ_where_each_reads_whole_are_recovered covers 128.

Refusal tests, each asserting no byte of the medium changes:

• a directory with a broken link;
• a file cross-linked into a directory;
• two different files on one chain;
• the two torn-copy cases above;
• the file server's a_volume_recovery_refuses_is_served_read_only_and_untouched (no write reaches the disk, and the volume is not writable).

Each has a negative control that removes its refusal and fails; nine mutations in total, three refusals and six repairs.

3: fixed. The fields are mirrors, renames, parents, long_names, ended and lost, and the line says {ended} files ended at their size, {lost} clusters no entry reached freed.

Suggestions:

• The recover.rs:14 wording is gone: half-done renames are now resolved here, by the coordinator's ruling, and the rename record is closed in this PR.
• Read-only mount on refusal: adopted.
• Shell Ctrl-C: adopted (33291dbf0).
• The blank line: added.
• The PR body now points to issues/ instead of the handoff.

Checks: --ci host exit 0, 109 steps; both image builds exit 0; evidence https://github.com/ToyOSOrg/evidence/tree/4a49c60d468b3d964b0aa7a83d3bff884622cdb7/r3-review-r1. Guest run 42023343796 is queued. Fresh metal is to be staged after the focused review.

[Japabu]

Review r2 of #1185 at 33291dbf0 (changes since f8db19fc2)

Architecture

The new shape is right. Fat32::recover runs a read-only examine pass, then either writes the repairs or returns Recovery::Refused without writing anything. The file server turns a refusal, or any recovery error, into a read-only mount. Recovery stays in toyos-fat32 and the policy for refused volumes stays in the file server, so each responsibility is with the component that owns it. r2 adds 409 lines and deletes 157. Most of the new lines are the examine pass (walk_tree, walk_copy, splits) and five refusal tests. The rename record is deleted because recovery now resolves that state.

Findings 1–3 from r1

1. Resolved. free_unreached runs only after walk_tree has finished on the active copy, and on every mirror when copies differ. Every path that cannot read something returns a refusal before the first write: a directory that does not read whole (root included, and MAX_DIR_ENTRIES overflow through CorruptDirectory), a cross-link, a chain short of its file's size, a chain ending in a bad link, an empty file that names a cluster, or a directory with no ... Device errors (Err(e)) propagate out of examine before anything is written, and the file server then serves the volume read-only.
   • Long names: only live LFN entries that DirScan would not attach to a short entry are freed. DirScan drops a run that is incomplete, has the wrong order or has the wrong checksum, which is how fatgen103 defines an orphan.
   • Reserved and bad entries: BAD is kept, and entries 0 and 1 and anything past max_cluster are skipped.
   • Volume size: only LOG is mounted writable (main.rs:348), so the bitmap and the full FAT scan stay small. A larger volume still gets a correct result, just a slower one.
2. Resolved. Splits are written only after both the main walk and every walk_copy pass. The torn-copy tests refuse both FAT 0 torn and the mirror torn.
3. Resolved. The counters are mirrors / renames / parents / long_names / ended / lost, and the log line reports exactly those.

The deviation (copies accepted when each reads whole)

I accept it as the safety test. A copy that is torn, zeroed or garbage almost always breaks a reached chain, which gives a free link, a cross-link or a short file, and then nothing is written. Differences confined to clusters nothing reaches are leaks under either copy.

The remaining risk is that a directory chain has no size field. If one copy ends a directory's chain early, that copy still reads whole. FAT 0 then wins, and the later directory clusters and their whole subtree are freed. For a ToyOS stop this is correct: FAT 0 is the "before" state, and entries lag their chains by a flush. For a foreign writer stopped in the same shape, FAT 0 wins too, which is what common fsck tools do by default on a mirrored volume. I do not count this as a blocker. It is worth knowing that "reads whole" does not prove which writer produced the volume.

The comment that justifies the deviation is false, though (blocker 1 below).

Rename ruling

Implemented as ruled. The second entry that the walk meets for an already-named chain is erased if its 21 bytes after the name are identical. .. is then repointed at the kept entry's directory. Every stop of the directory-move scenario is now checker-clean, a second recovery finds nothing, and the test asserts the moved file is under one of the two names. The renames and parents repairs each have a negative control.

Read-only serving

FatVolume::mount sets writable false on Refused and on Err. a_volume_recovery_refuses_is_served_read_only_and_untouched asserts that no write epoch reaches the disk, and r2-fs-control.log shows it fails when the refused volume is served writable.

Tests and evidence

• refused_untouched compares the whole medium after a refusal, which is the right oracle.
• The ten negative controls (r2-controls.log, r2-fs-control.log) each fail in the expected test. All nine fat32 mutations ran on the pre-commit working tree. The README says so, and the restored suite is the r2-fat32-b.log run at the head.
• Some refusals have no test of their own: a chain short of its size, an empty file naming a cluster, no .., and a bad link at the end. Each is a return before any write, so inspection is enough. Their failure mode would be a false refusal of a ToyOS stop, and the stop-at-every-write test already guards against that because it requires Recovered for every stop.
• The host CI run, both image builds and clippy ran at the head and exited 0.
• Guest run 42023343796 is still queued at 33291dbf0. The shell claim (libc_children) and the LOG behaviour on a guest still depend on it.

Blockers

1. lib/fat32/src/recover.rs:171-176: the deviation's stated reason is false.
The comment says: "Each copy then holds the volume before or after that flush, which the walk over it reads whole."

• The cache flush runs any number of runs concurrently with no ordering (system/fileserver/src/cache.rs:377-402, and the "disk keeps no order until a flush" contract in fat.rs).
• A flush that touches more than one FAT block per copy can therefore leave a copy that is neither the before state nor the after state. Examples are an append crossing a FAT block, or a sweep's frees plus a new part's growth in one flush.
• What actually makes such a copy read whole is that an entry reaches the disk only a flush after its chain (fat.rs header, level). Mixed blocks then differ only past each file's size or in unreached clusters.

This comment is the only written argument for accepting unbounded splits. State the real invariant in one or two lines, or delete the sentence. The r2 response comment repeats the same premise.

2. system/fileserver/src/fat.rs:406-417 (unlink), with lib/fat32/src/fs.rs:1089: a ToyOS stop state that recovery now refuses for ever.

• Fat32::remove erases the entry and then frees the chain. Both writes go into the cache in one flush, with no flush between them.
• Under the file server's own stop model (any subset of a flush epoch's writes, as an_append_stopped_with_any_writes_of_a_flush_on_the_disk_reads_back_the_last_sync uses), a stop can land the FAT frees but not the directory erase.
• walk_tree then finds a live entry whose first link is free. reach_chain and next_cluster return CorruptChain, and recovery refuses with "of a file does not read whole" or "ends in a link that is not one".
• logkeeper's sweep (system/logkeeper/src/store.rs:132) unlinks old parts on LOG whenever it holds more than MAX_LOG_FILES logs. One reset during that flush leaves LOG read-only on every boot after, and nothing in ToyOS takes it back.
• Truncation (set_len shrink) and release_replaced after a replacing rename have the same shape under the cache. The r1 shrink fix orders the writes inside the crate, but not across the cache.

This ordering defect existed before this PR. On main the same state mounted writable and later reallocated the freed clusters into a cross-link, so r2's refusal is strictly safer. It is still a stop that the closed nothing-reclaims-… record was about, and the PR describes its recovery as covering what ToyOS stops leave.

This is my inference from reading the code. The cheapest check is the existing stopped-flush test run over an unlink instead of an append. Fix it by keeping the erase and the free a cache flush apart, as growth already does. Otherwise, record it in issues/ with that test as its evidence and exit. Either is enough to land.

Suggestions

• recover.rs:229-241: the de-duplication in splits only checks splits.last(), but the loop runs mirrors outside entries. With more than two FATs, one cluster is pushed once per differing mirror, so it is rewritten and mirrors is over-counted. Comparing every mirror inside the entry loop fixes both.
• recover.rs:285: fields includes byte 12, the NT lowercase flags, which belong to the name. insert_entry zeroes byte 12 (dir.rs:544). A stopped rename of an entry a foreign writer stored with lowercase flags is therefore refused rather than resolved. Comparing from byte 11 and bytes 13 onward would match the ruling's "equal in every field but the name".
• recover.rs:12: this header line wraps at about 100 columns while the rest wrap at 78.

SEND BACK: blockers 1 and 2. Blocker 2 can be closed by an issues/ record if the ordering fix is out of this PR's scope.

[Japabu]

r3 at 258e9197d: response to review r2

1: fixed. My r2 comment said a stopped flush leaves each FAT copy before or after it. That is false: the cache writes a flush's blocks in no order. A copy still reads whole because an entry reaches the disk a flush after its chain, and, as of this round, an erase reaches it a flush before its chain's free. So the copies can differ only past a file's size or in clusters nothing reaches. recover.rs now states that.

2: fixed, not recorded. Fat32::remove (and so release_replaced), a shrink, and rename each flush between their two ordered writes: the erase before the free, the shorter entry before the free past it, and the new name before the old name's erase. The rule is stated once, in crate::repair's header.

The evidence is stopped_anywhere, the file server's stopped-flush test generalised:

• an_unlink_stopped_anywhere_is_recovered
• a_shrink_stopped_anywhere_is_recovered
• a_rename_stopped_anywhere_is_recovered

Each operation is stopped at every subset of every flush epoch. In every state:

• a writable mount recovers rather than refuses;
• the checker finds nothing afterwards;
• the file is whole under exactly one name, or at its old or new length, and its neighbour is untouched.

The negative control is the old ordering, removing each flush in turn:

• unlink: refused at epoch 4, writes 0b10.
• shrink: refused at epoch 3, writes 0b10.
• rename: the file under neither name at epoch 3, writes 0b1.

So on main, a rename through the cache could lose a file outright on a reset.

Suggestions, all taken:

• splits compares every mirror in one pass over each entry.
• The rename match leaves out byte 12.
• The header is rewrapped.

Guest run 42023343796 at 33291dbf0 has two reds.

• nvme_disk_boots: LOG mounted at 1.983 s and READY came at 2.004 s. The row read only the boot log before READY, and LOG's serving line comes after recovery has read the volume whole (19 ms on the T14). The row's claim is that every role is served, not that each is served before READY, so 258e9197d makes it wait on the guest for each role.
• libc_children: tty_interrupt stalls at "the nested line held". The same step stalls in run 42018194698 on kernel-no-tls, which contains main and does not touch system/shell, so I'm treating it as a sighting independent of this branch. spawn_system, the shell claim of this PR, passed: "a refused redirect ran nothing".
• the_log_lands_on_the_boot_stick passed.

Checks at e41116092:

• --ci host: exit 0, 109 steps.
• --build-only: exit 0.
• cargo clippy -p toyos-fat32 --all-targets: exit 0.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/b4f9fc04f14f87756465073cbf858b663dba937b/r4-review-r2
• Guest run 42030077569 is queued at 258e9197d.

[Japabu]

Review r3 of #1185 at 258e9197d (changes since 33291dbf0)

Architecture

The fix is in the right place and the right shape. The file server's cache keeps writes in order only across a flush, so each call with two dependent writes now puts a flush between them:

• remove (and through it remove_dir and release_replaced) flushes in erase_committed between the erase and commit();
• a shrink flushes between the shorter entry and repair.truncate(mark);
• move_entry flushes between insert_entry and erase_entries.

There is no new mechanism and no second path. The rule is written once, in repair.rs's header.

Code adds 3 lines in fs.rs, about 15 net in recover.rs/repair.rs, and 132 lines of file server tests. The test growth is a generalization of the existing stopped-flush harness. It is justified because these are exactly the states a host test can enumerate and no guest test can.

The rename flush keeps the coordinator's ruled write order (new name, then erase), so "the write order stays as it is" is still true.

Failure paths of the new flushes (traced)

Each flush sits before the point where the call commits, so a refused flush is rolled back like any refused write:

• erase_committed: a refused flush returns before commit(). The queued rollback restores the entry, and remove answers Err. This matches its doc ("a refusal while erasing restores the entry").
• Shrink: a refused flush returns before truncate(mark). The repair rewrites the old entry, and the handle's entry_size is untouched.
• move_entry: a refused flush rolls back the inserted run and any directory growth.

The invariant in repair.rs holds in all three.

A directory move writes set_dot_dot in the same epoch as the erase. Either partial landing is covered by r2's parents repair. The new rename test covers only a file, but r2's refused-writes suite already holds the directory-move scenario.

Previous blockers

1. Resolved. recover.rs:172-178 now states the real invariant: entries lag their chains by a flush, and erases lead their frees by one.
2. Resolved by fixing the ordering, as the orchestrator ruled.
   • stopped_anywhere stops each operation at every flush epoch from the op's start, with every subset of that epoch's writes. For every state it requires a writable mount, a clean checker run and a whole file under one name or length.
   • Each test asserts that some stop needed a repair, so a stop space that never exercises recovery fails rather than passing vacuously.
   • The reported controls (unlink refused at epoch 4 0b10, shrink at epoch 3 0b10, rename under neither name at epoch 3 0b1) are each the defect r2 described.

Suggestions taken

• splits reads every mirror and then compares each entry once. A cluster is pushed at most once, so the last() dedup is correctly gone.
• fields skips byte 12, as the ruling requires.

Blocker

1. tests/toyos.rs:5261-5269: the row's cleanliness check no longer covers LOG's mount.

• The row claims (doc at :5253-5254) that the machine "says nothing a clean boot does not". must_be_clean still runs only on Serial::boot's log, which ends at runner-ready.
• The new loop waits past READY for fileserver: Log …. By the r3 response's own timing, that is exactly where LOG's recovery output now lands.
• main.rs:243-247 prints fileserver: {role:?} serving … — {describe} for any volume fat_on returned. If a regression in this PR made recovery refuse a clean image, LOG's volume would come back read-only and the role line would still appear. The row would go green.
• Waiting for each role is a legitimate fix for the READY race. As written, though, it also hides the one regression this PR could plausibly bring to this row.
• Cheapest fix: after the loop, check the accumulated console the same way the virtio-gpu row does (serial::Serial::named("the nvme boot", console).must_be_clean()?).

Suggestion

• recover.rs:174-176 gives two of the three ordering rules and leaves out the shrink's entry before its free. The sentence stays true, because freed clusters past a size are covered by "past a file's size". Pointing at crate::repair's rule would avoid a second list that can drift.

Evidence

• Host CI at e41116092: green, as reported.
• 258e9197d touches only tests/toyos.rs, which --ci host does not run, so the earlier host result still holds.
• Guest run 42030077569 at 258e9197d: nvme_disk_boots must be green there after the blocker's fix, which needs only a compile and that row.
• Treating tty_interrupt as a sighting is reasonable: it stalls at the same step on kernel-no-tls, which does not contain this branch's shell change.

SEND BACK: blocker 1.

[Japabu]

r4 at 4937c2bdc: response to review r3

1: fixed. After its role loop, nvme_disk_boots checks the whole console it accumulated (serial::Serial::named("the nvme boot", console).must_be_clean()), which includes LOG's mount and recovery. A volume that recovery refused would also miss the role's marker: a read-only mount prints FAT32 read-only,, not FAT32, .

Suggestion: taken. recover.rs points at crate::repair's ordering rule instead of listing two of its three cases.

Checks:

• The harness compiles.
• cargo test -p toyos-fat32 --lib passes. The comment is the only change to the crate.
• nvme_disk_boots only, on the guest runner: run 42036235415, queued.
• Guest run 42030077569 at 258e9197d: nvme_disk_boots and the_log_lands_on_the_boot_stick passed. libc_children was red only in tty_interrupt, the sighting.

[Japabu]

Guest run 42036235415 at 4937c2bdc: nvme_disk_boots PASS (3s), with the accumulated console clean.

[Japabu]

Review r4 of #1185 at 4937c2bdc (changes since 258e9197d)

Architecture

No change to the design. r4 is one harness line and one comment.

r3 blocker 1: resolved

tests/toyos.rs:5270 now checks the whole accumulated console with must_be_clean. That console starts from boot_log() and extends through every role marker, so LOG's mount and recovery output are inside the checked text.

The read-only half of the concern was already covered, and I missed that in r3:

• NVME_ROLES[0] is "Log serving /log — FAT32, ".
• fat.rs:518-522 renders a refused recovery as FAT32 read-only, .
• A LOG volume that recovery refused therefore stalls the marker wait and the row goes red.

The new check adds what the marker could not catch: a death or NEVER_CLEAN line printed after runner-ready.

Evidence: guest run 42036235415 ran at head 4937c2bdc and concluded success (checked with gh run view).

r3 suggestion: taken

recover.rs:175-176 now points at crate::repair's rule instead of keeping its own partial list. repair.rs:33-36 names all three orderings.

Suggestion

• lib/fat32/src/recover.rs:177: the rewrap left a 111-column comment line ("differ only past … A copy torn or written by something else"), while its neighbours wrap at about 76. The clause "every write that must precede another a flush before it" is also hard to parse. A possible wording: "…an entry reaches the disk a flush after its chain, and each ordered write a flush before the one it precedes ([crate::repair]), so…". Rewrap it with the same edit.

LAND

[Japabu]

Fresh metal is staged for this head from a clean tree. The coordinator runs it.

• Revision: 4937c2bdcdbe486cd414e89a2f693fba392d35ce. git status --porcelain --ignore-submodules=none was empty.
• Staged with cargo test --test toyos-build -- --metal --metal-readback <scratch>/metal-r4 nvme_leaves_the_benchs_disk_alone the_disk_survives_a_reset_under_load. The command exits 2 because it touches no machine.
• Images (SHA-256):
  • testcases/image.img: de9d9b74259a6f6fb6a80a15326e139fc61d8fd6f8f273319c24fa5972c19739
  • usbload/image.img: 8c37a76edbcfe053534c4f3326b6ae60369978b54f6abd57c1da0da0dfc53f07
• Requested boots:
  • T14: usbload, then testcases.
  • AMD: testcases, into its own readback directory.

[Japabu]

T14 metal at 4937c2bdc, judged with the readbacks in place: nvme_leaves_the_benchs_disk_alone PASS and the_disk_survives_a_reset_under_load PASS, 2 passed, 0 failed.

• NVMe: the T14's drive came up under the new geometry checks on both boots.
• LOG: on both boots, LOG was served writable about 20 ms after its mount. The new recovery ran on the real stick and did not refuse it.
• Readbacks: both log partitions are clean under toyos-fat32-check. Recovery on a host copy changed nothing, twice.

This is the clean-stick case: the reset left no stop state, and each boot reflashes LOG. The host tests cover the repairs and refusals.

AMD testcases has not run; the bench did not answer. I'll judge it once it has.

Evidence: https://github.com/ToyOSOrg/evidence/tree/59439f381cc68973bccd6272ba4ce5d57825daf4/r5-metal-t14

[Japabu]

The evidence branch pr-1185 was rewritten to mask machine identifiers: the T14's vendor, product and BIOS as <vendor> <product>, BIOS <bios>; the guest runner's machine and runner names as <qemu-host> and <runner>; and the runner's home path as <runner>. The body's links are repinned. Links in earlier comments map as follows (old → new):

old	new
59439f381	4bcf3c4d6243234fc2b10ede9744614ef6529562
b4f9fc04f	4a65ca68ebd29b22833d126668cf3198d2fb2510
4a49c60d4	acb50cdf526761c716e1a9f3c00e46b64621d459
b06d62de9	8484fa8752cd9f9e8bb76082bcbfc5ee93cf5fa6

[Japabu]

AMD metal at 4937c2bdc (image sha256 de9d9b74…, matched before flashing): nvme_leaves_the_benchs_disk_alone PASS, judged with the readback in place.

• The AMD bench's drive came up under the new geometry checks: 512-byte sectors, 4 I/O queues.
• LOG was served writable 14 ms after its mount, so recovery ran and did not refuse.
• The log partition has 0 complaints under toyos-fat32-check. Recovery on a host copy changed nothing, twice.

With the T14 boots, both benches pass at this head. Evidence: https://github.com/ToyOSOrg/evidence/tree/7408fab3cd3fdc958780436231dfa5b0aea6ca9b/r6-metal-amd (the machine line is masked).