[Japabu]

Rebased onto main 3022fd718 (after #1185, #1195, #1196) as two commits, head 67601147b. Waiting for a focused review once the owner resumes.

• logkeeper: the console no longer waits for /log to open #1195 is reconciled per its reviewer's plan: the opener thread is kept, adopt only takes the volume, logkeeper: the console no longer waits for /log to open #1195's catch-up loop and Hub::owed(&mut u64) / serve::Owed are dropped, and the stop waits for a still-opening volume within STOP_CATCH_UP.
• Memory overwrite on every boot; the black box goes, and a clean stop proves an image #1196: the stop mark is written only over an open part.
• The full-log row is dropped, by the coordinator's ruling (option 3).
  • Its cleanup job cannot write /log after Memory overwrite on every boot; the black box goes, and a clean stop proves an image #1196.
  • Filling the volume with logkeeper's own old parts would not heal it either: retention sweeps only at open, before the first write, and at a rotation, which a refused part never reaches.
  • The logkeeper record closes on store.rs's host tests. The end-to-end evidence of its time is guest run 42040641298 (green at 6405def8d, before Memory overwrite on every boot; the black box goes, and a clean stop proves an image #1196).
• Checks:
  • Host CI at 3d7e96592: exit 0, 109 steps. The head's system/, lib/ and Cargo.lock are identical to it; the rest of its tree is main's, plus issues/.
  • Guest run 42133377735 at 3d7e96592: acpi_press_is_one_sci, the_log_lands_on_the_boot_stick and a_clean_stop_proves_its_image PASS. The row it stalled is the one dropped here.
• Evidence (range-diff, interdiff, logs): https://github.com/ToyOSOrg/evidence/tree/be2e31065e3ce55e82eb5181b7ef8c3c42dc9e25/r7-rebase

The sections below describe the pre-rebase rounds.

What

This PR implements the coordinator's rulings of 2026-11-05 on the data-loss records #1185 left open. The FAT rename ruling went into #1185's recovery rework, which needed it.

commit	record	resulting behavior
cd71613d2 + r2 564351057 logkeeper	logkeeper-ends-… (narrowed, open); a-file-held-across-… (narrowed, open)	A refused create, append, sync or continuation is asked again: after 1 s, doubling to 64 s, and at the stop's flush whatever the wait. Asking again reopens the part, cuts it to what its last sync made durable, and writes the rest of the boot from the replay (Hub::since). What the replay already let go is said in the file in its place. A LOG server restart (Gone) is the same refusal, so logkeeper reopens its own path.
r2 f3a8caa18 tests	logkeeper-ends-… (exit amended)	The coordinator's option (d): the new QEMU row a_refused_log_is_written_whole_once_the_volume_takes_it boots a stick whose log volume the harness filled. logkeeper's first write is refused; a job frees the room; the stick read back holds the whole boot (bootlog::verdict), with the refusal and the recovery in it.
4ddeb5a89 issues	std-file-lock-…; the storage track	File locks stay with M4. The storage track records #1131's finding: every native DATA journal entry costs two flushes (three when the entry names itself), and FUA is missing end to end.

r2: review r1's five blockers

1. The file's bytes and the replay's offsets are kept apart in Volume (size/durable and from/through/durable_through). The gap note moves only the file's bytes.
2. A part shorter than its last sync is rebuilt whole from its start in the replay, never padded with zeros. The record now names the swapped-stick hazard.
3. a_refused_file_is_given_back_the_boot_the_replay_holds drives a real Volume through an unsynced write with torn bytes, a replay gap, a refused continuation and a part cut behind its back. Blocker 1, blocker 2 and an oversized rewrite each turn it red.
4. The rewrite goes through Volume::catch_up: part-sized writes and continuations, at most MAX_LOG_BYTES a round.
5. The retry comment now states the actual tradeoff.

Suggestions taken: FUA is cited by its specifications, the duplicated paragraph is gone, and reopened(now_or_never) is now caught_up(at_stop).

r3: review r2's three blockers

• The stop's flush syncs what it wrote and says any shortfall, in the file and on the console, within half the supervisor's flush bound.
• The row's fill now holds until the job frees it.
• The row checks the stick's kernel records against the console's.

There is also a host case for a refused create at open.

Checks (r3, a99667bf0 to 17289151b)

• Host: cargo run -- --ci host: exit 0, 109 steps.
• Tests: logkeeper, 26 passed. Negative controls: blockers 1, 2 and 4 of r1, and A of r2, each fail when applied and pass when restored.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/858beb0c3d16f30ff5fd068c4602a46bd8cc76c2/r3; r4 and r5 (review r3 and r4 responses): https://github.com/ToyOSOrg/evidence/tree/29c1a3d1815ff76dab812969997cb3db20c50b61/r5 and https://github.com/ToyOSOrg/evidence/tree/29c1a3d1815ff76dab812969997cb3db20c50b61/r6 (r2: https://github.com/ToyOSOrg/evidence/tree/51ba7703a1b6ca4feeb180a9b197099c464f5a18/r2)
• Guest: run 42040641298 at 6405def8d. a_refused_log_is_written_whole_once_the_volume_takes_it PASS: 165 kernel records, the console's in order, from the boot's first. the_log_lands_on_the_boot_stick PASS.
  • Two harness fixes on the way: 0939e4e24 hands the job over as a test binary, and 6405def8d types the job at the guest so the console keeps its window.
  • Earlier reds: 42030108549 (the fill did not hold, blocker B), 42036984718 and 42038743967 (the two harness fixes).
  • Evidence: https://github.com/ToyOSOrg/evidence/tree/7b8b6a8a2469f6b27b43186a7d513b0d19e06e13/r4
• Record: 17289151b closes logkeeper-ends-…, its exit met. The swapped-stick hazard moves to a-file-held-across-….

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1190 (head cd47061b5, own commits on top of #1185's 33291dbf0)

Architecture

The split of responsibilities is right. Nothing in the kernel changes. The file, its retry policy and its recovery stay in logkeeper, which owns /log. Rebuilding a part from logkeeper's own replay (Hub::since) is a small, direct design: the replay already exists, and it is bounded (REPLAY_BYTES = 16 MiB). The ordering holds. write calls to_volume(text) before hub.append(text), and resume does the same. So a part reopened inside to_volume gets the replay up to the round before, and then writes the round once. The ordinary path does not duplicate bytes. The backoff is bounded (1 s doubling to 64 s, with one retry at the stop's flush), and the doubling test and its control show it.

The change is +262/−89. About 170 net production lines are in logkeeper, and two small host tests are added.

The weak point is the bookkeeping that joins file lengths to replay offsets. Volume::from/durable count bytes in the file, and Hub::since takes a replay offset. The PR assumes the two are always equal, and two of the findings below come from that assumption. The cut-and-rewrite step, which is the main claim of this change, has no test.

Findings

BLOCKER 1. system/logkeeper/src/main.rs:607-610 with store.rs:247,300: the "bytes were let go" note shifts the file's offsets away from the replay's. A later reopen then loses bytes without saying so and starts in the middle of a line.
When lost > 0, the note is spliced into owed. reopen counts it in size (size = durable + owed.len()). The next sync copies it into durable, and the next rotate adds it to from, because from grows by size. But the note is never appended to the hub. From then on, durable_end() is W bytes (the note's length) past the matching replay offset. On the next refusal, since(durable_end()) skips W bytes of real log. Replay::next does not snap to a line, so the rewrite starts mid-line and nothing reports the gap. When nothing new arrived, the offset can also be past the replay's end, and the result is an empty rewrite that holds the wrong position. This path is reachable on a flaky stick: a long outage evicts bytes, then a later refusal follows. That is exactly the case this PR exists for.
Fix: keep the file's offset and the boot's offset apart. For example, record the replay offset each sync made durable, rather than deriving it from file sizes. Alternatively, append the note to the hub as well, so both streams carry it. A host test of the offset arithmetic would catch this; see blocker 3.

BLOCKER 2. system/logkeeper/src/store.rs:242-244: reopen trusts durable when it truncates, and set_len makes a file longer as readily as it makes one shorter.
The brief asks that the rewrite be safe when the durable length is unknown and when the server restarts with a different volume state. reopen opens the file with create(true) and calls set_len(self.durable) without checking the file's actual length. Suppose the file is shorter than durable or missing after the restart: the stick was swapped, the file was deleted from the userland-writable /log, or a sync claimed more durability than it delivered. Then set_len pads it with up to 1 MiB of zero bytes, and the replay is appended after them. The real log bytes in that range are replaced by NULs, even though the replay may still hold them, and nothing reports it.
Fix: cut to min(file.metadata()?.len(), self.durable) and refill from that offset. since then reports whatever the replay no longer holds. The remaining hazard is a foreign volume that carries a file at the same path, such as unknown-NN on another stick. Reopening by path cannot detect it, and reopen would rewrite that other boot's file beyond durable. That waits on volume identity, as the ruling says, but neither record names it. Add one sentence about it to logkeeper-ends-… or a-file-held-across-….

BLOCKER 3. system/logkeeper/src/store.rs (reopen, rotate, durable_end): the claim "cut to what was durable and refilled, nothing lost or duplicated" has no test.
a_lost_file_is_given_the_boot_from_where_it_was_durable tests Hub::since and nothing else. The bookkeeping of from/size/durable across write, failed sync, rotate, a refused next-part create and reopen is never exercised, and blocker 1 sits exactly there. The cheapest sufficient check is a host test. Either make that bookkeeping a small pure type, or make Volume generic over its directory or file, then drive it through these steps: refusal, eviction with the note, a second refusal, a refused rotate. In each case assert that the file equals the replay plus the notes.

BLOCKER 4. system/logkeeper/src/store.rs:242-247: one reopen can write the whole replay into a single part, up to 16 MiB, ignoring MAX_LOG_BYTES.
owed can be as large as REPLAY_BYTES (16 × MAX_LOG_BYTES), and reopen writes all of it into the current part. That breaks contracts stated in this crate:

• MAX_LOG_BYTES is documented as "how large one file may get", and as the bound on what console reads off USB before it paints.
• Retention allows 16 files at 1 MiB within a volume of about 35 MB. A few 16 MiB parts exceed it, and the volume then refuses for good.
• The module header says a part-sized burst of writes "starves a tone playing beside it", and this is a burst sixteen times that size.
• At the stop's flush, the write sits inside the supervisor's bounded wait.

Fix: feed owed through the same part-sized write-and-rotate path, so the rewrite obeys every bound an ordinary write does.

BLOCKER 5. system/logkeeper/src/policy.rs:13-16: the comment's rationale is false.
It says "a stick that recovers within a second or two loses none of the boot's file". How much is lost depends on whether the replay (16 MiB) still holds the bytes, not on the retry wait. A stick that recovers after minutes also loses nothing. Delete the rationale, or state the actual tradeoff: how often a dead stick is asked again, against how soon a live one is used again. It is short and cheap to fix.

SUGGESTION. issues/storage-is-layers-and-a-role-is-a-filesystem.md:139: "Upstream writes the entry as one preflushed Force Unit Access write" puts a fact taken from another implementation's code into the repository. Root keeps that provenance in evidence. State the device fact with its specification instead: FUA makes a write durable without a trailing Flush (NVMe Write FUA, SBC WRITE(10) FUA). The flush-count and the FUA-plumbing claims check out against write_entries (lib/bcachefs/src/engine/mount/mod.rs:1448), which flushes, writes the entry and flushes, with one more round for an entry naming itself.

SUGGESTION. The same paragraph about reopening appears four times: the main.rs header, the Volume doc, Log::lost, and the issue. Keep the Volume doc. reopened(now_or_never: bool) reads as a riddle; ignore_wait or at_stop would say what it does.

Records

• std-file-lock-…: accurate.
• a-file-held-across-…: accurate. The volume id is the server's pid (system/fileserver/src/main.rs:262), and the Stale/Gone claims match toyos::fs. It should name the foreign-volume hazard from blocker 2.
• logkeeper-ends-…: accurate about what was built and about the missing staged break. It stays open honestly, and the choice among (a), (b) and (c) is correctly left to the coordinator. Its sentence "cut to what its last sync made durable" will need to match the fix for blocker 2.
• The FUA finding: accurate, apart from the sourcing in the suggestion above.

Evidence

• f-logkeeper-r3.log: 24 passed on ae16175b4's content. cd47061b5 touches only issues/, so that result carries over to the head.
• Both controls are real mutations, and each turns its test red. They cover the doubling and since's count only, not the rewrite.
• The pre-rebase build-lock red is outside the diff and was green at the head; I accept it as a sighting.
• Guest run 42023375706 was still in progress at review time. It is only a regression check of the ordinary path, which the PR states correctly.
• No run shows a boot that actually retries. The PR says so and keeps the record open.

SEND BACK

[Japabu]

r2 at 8523f4394: response to review r1

All five blockers are fixed in 9e04cbafc.

1. The file's bytes (size, durable) and the replay's offsets (from, through, durable_through) are separate fields. A gap note moves only the file's bytes, and a gap resumes at the replay's own line start.
2. reopen reads the part's length. A part shorter than its last sync is rebuilt whole from its start in the replay and says so. It is never padded. The record names the swapped-stick hazard.
3. a_refused_file_is_given_back_the_boot_the_replay_holds drives a real Volume in a scratch directory through four cases: an unsynced write with torn bytes after it, a refusal long enough for the replay to evict, a refused continuation, and a part cut behind its back. After each, it asserts that the parts are the boot with one note at the gap, and that no part exceeds its bound.
   • Reintroducing blocker 1 turns it red "across the gap".
   • Blocker 2 turns it red "after the part was cut".
   • Ignoring the part's room turns it red on the part's size.
4. Volume::catch_up feeds Hub::owed through part-sized writes and continuations, at most MAX_LOG_BYTES a round. Each part is synced as it fills, and the round's text waits in the replay until the file has caught up.
5. The comment is rewritten.

Both suggestions are taken.

Same round, 8523f4394: the coordinator's option (d), a_refused_log_is_written_whole_once_the_volume_takes_it. The harness fills the stick's log volume. A job removes the filler once logkeeper has been refused, and the stick is read back as a whole boot that carries the refusal and the recovery. The record's exit is amended to this row.

Checks:

• --ci host exit 0.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/5bbdf36084ea913dccbedf8ef68703f5fec57bf0/r2
• Guest run 42027419042 is queued.

[Japabu]

Review r2 of #1190 (head f3a8caa18, own commits on top of #1185's 258e9197d)

Architecture

The design is unchanged and still sound. logkeeper owns the file, the retry and the recovery, and the kernel is untouched. The r2 representation is the right fix: Volume now keeps the file's position (size/durable) apart from the replay's (from/through/durable_through), and Hub::owed → Owed puts the replay's answer behind one small type. catch_up uses the same part-sized write-and-rotate path as an ordinary round. Adding dir to Volume is a genuine boundary for the host test, not test-only machinery.

Size: about +240 net production lines in logkeeper, about +150 lines of host tests, and +100 lines of harness, comprising the row, fill_log_volume and log_unfill.

r1 blockers

1. Resolved. The note goes through put (file bytes only), and Gap sets through = at. rotate takes from from through. The control in f3-lk-controls.log (through = at + note.len()) turns the test red "across the gap".
2. Resolved. reopen checks metadata().len() < durable and rebuilds from from. It never pads. The control turns it red "after the part was cut". Both records now name the hazard of a same-named file on a different stick.
3. Resolved. a_refused_file_is_given_back_the_boot_the_replay_holds drives a real Volume through each step r1 asked for. I checked the three controls in the evidence; each one is a real mutation that the test catches.
4. Resolved for the ordinary path: room = min(rotate_at - size, budget - written). For the stop, see blocker A below.
5. Resolved. The comment in policy.rs now states the actual tradeoff.

The FUA paragraph in the storage track now cites the NVMe and SBC specifications. The duplicated paragraphs are gone.

Findings

BLOCKER A. system/logkeeper/src/main.rs:718-731 with store.rs:313-350: the stop's flush catches up at most 1 MiB, leaves that MiB unsynced, says nothing, and tells the supervisor the log is whole.
flushed calls caught_up(true), and from there catch_up(..., MAX_LOG_BYTES, ...).

• When the file owes more than one part (the replay holds up to 16 MiB), catch_up writes up to 1 MiB and returns Ok(false) once the budget runs out. It syncs only at a rotation or at Owed::Nothing.
• flushed skips both sync() and answered() when caught_up returns false. It then signals FLUSHED anyway, and stopping stops every later to_volume.
• Result: the bytes this catch-up just wrote are never made durable, the rest of the owed boot never reaches the file, and no record says so.

The same silent outcome follows when the stop's catch-up rotates and the next part's create is refused (Ok(false), part not open). Only the Err arm is loud.

The stop is also the backstop for an idle machine. caught_up runs only from to_volume, which runs only on a round that read lines. A due retry therefore waits for the next line or for the stop, and the stop is exactly where the budget truncates.

The case is reachable. The record's own scenario is a long refusal on a busy boot, and the_log_lands_on_the_boot_stick wrote 2.5 MB in this run.

Fix: at the stop, always sync what was written. Then either keep catching up, if the supervisor's bound allows the replay's 16 MiB, or say in the file and on the console how many bytes of the boot the file lacks. Add a host test: a Volume refused while more than MAX_LOG_BYTES arrives, then a stop. Assert either that the file is whole, or that it is synced and carries the shortfall note.

BLOCKER B. tests/toyos.rs:3377 and build/image.rs:403-435: the new row is red at the head (guest run 42030108549), because its fixture does not refuse logkeeper's first write.
The run checked out dataloss-followup at f3a8caa18. nvme_disk_boots and the_log_lands_on_the_boot_stick passed. The new row failed with STALLED: waiting for logkeeper to find the full volume refusing it. In the guest's output, logkeeper printed this boot's kernel log is /log/2026-11-05-130008.log at 2.233 s, and then only clock lines followed. No refusal came.

Probable cause, inferred from the code and not measured: FULL_LOG_KEPT sizes loader.log at 256 KiB, and the loader deletes and recreates that file at every boot (loader/src/loaderlog.rs:148-161). The deletion frees roughly 256 KiB minus the loader's own output, and logkeeper's early boot log fits in that space. The comment on fill_log_volume says the kept files only give a writer "its own room back", and that is false for a writer that deletes and recreates.

The fixture needs a fill that the loader's rewrite cannot undo. For example, size the kept files to what the loader actually writes, or fill after the loader has run. Then the row needs a green run on the guest runner. Until that happens, the record's exit (logkeeper-ends-…: "green on the guest runner") is unmet. The PR's "Guest: … queued" line is stale; correcting it is the orchestrator's job.

BLOCKER C. tests/toyos.rs:3406-3422: once green, the row still cannot show that no bytes were lost or duplicated, but the PR and the record present it as showing the whole boot.
bootlog::verdict (build/bootlog.rs:548) checks only three things: a Boot: complete record exists, no job ran past its bound, and the supervisor's stop line exists. The following would all pass it:

• a file that repeats the replay after an untruncated partial append;
• a file that is missing everything before the refusal;
• a file with a gap note in the middle.

The row's first line is printed, not asserted. In this fixture the first write is refused on an empty file, so blocker 2's cut never even runs. The host test is the real oracle for the bookkeeping, and that is fine. But the row is cited as the exit's proof of "whole".

Cheapest sufficient check, using data the row already has: the console is the independent source. Assert that the stick's kernel records, the non-program lines, equal in order the kernel records on the serial console from the first one to the stop. Also assert that the file has no "were let go" note, and that it begins with the boot's first kernel record. Otherwise, narrow the row's and the record's claim to "the file exists after the refusal and carries the boot record, the stop, the refusal and the recovery".

Records

• logkeeper-ends-…: accurate about what was built, and it stays open. Its exit names a row that is red at the head (B), and that exit should match whatever claim C settles on. The row exercises a refused append, not the refused create from run 69, which is the record's title. Volume::open's refused-create path (file: None at open) has no test, either on the host or in the guest. A one-line host case would cover it, for example a directory placed at the first part's path.
• a-file-held-across-…: accurate. The volume id is the server's pid. The ruling is attributed to the coordinator, not the owner, which is correct.
• std-file-lock-… and the storage-track FUA paragraph: accurate.

Evidence

• f3-lk-c.log: 24 passed on 9e04cbafc's content. f3-lk-controls.log holds four real mutations, each exiting 101. The rebase onto Data-loss burn-down: write-back retry, FAT shrink order and stop recovery, shell redirects, NVMe geometry #1185 r3 leaves logkeeper's diff unchanged, and the host CI at f3a8caa18 is reported green, so I accept that the host result carries over.
• Guest run 42030108549 has completed, and it failed: 2 passed, 1 failed (B). The failure was introduced by this change, not an unrelated red, so it blocks.

SEND BACK

[Japabu]

r3 at a99667bf0: response to review r2

A. Fixed. Volume::finish now does the stop:

• it catches up a part at a time while there is time;
• it syncs what it wrote;
• where the replay holds bytes the file doesn't, it writes a shortfall line into the file, and logkeeper says it on the console as an error.

A next-part create refused during that catch-up ends it the same way. The previous part was already synced by its rotation, and the shortfall goes to the console.

The time is STOP_CATCH_UP, half of the supervisor's bound on the whole flush (toyos_quiesce::FLUSH_MS, 5 s). The other half is for draining the rings, the final sync and the answer. It is a time bound, not a byte count, because how much a stick takes in that time depends on the device. FLUSHED now means "the flush is done", and FLUSH's doc says so.

Host test: a_stop_with_more_owed_than_its_time_syncs_and_says_the_shortfall. A stop with no time leaves a synced file ending in the shortfall; a stop with time leaves a whole file. Three controls each turn it red: no shortfall line, no sync, and no catching up while there is time.

B. Fixed. The cause was what you inferred: the loader deletes and recreates loader.log at every boot, which freed the 256 KiB I had kept for it. fill_log_volume now fills every cluster and keeps nothing. The loader's own writes are refused that boot, which it reports and then boots on (loaderlog's and attempt's refusal paths), and logkeeper's first write is refused. The red run was 42030108549; the row is redispatched as 42036984718, queued. The PR body is corrected.

C. Fixed. The row now checks, against the serial console:

• the stick's first line is the boot's first kernel record;
• each CPU's kernel records on the stick are the console's, in order, with none extra;
• the file has no gap note and no shortfall.

Host case: a_first_part_refused_at_open_is_made_and_given_the_boot. The first part's create is refused, using a directory nothing may be created in. The part is then made at the first catch-up and given the boot from its start.

Checks:

• --ci host at a99667bf0: exit 0, 109 steps.
• logkeeper: 26 passed.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/a9b38564a3aae49b83abc7f84a78c7ba2ba55eea/r3

[Japabu]

The row is green on the guest runner at 6405def8d, run 42040641298: a_refused_log_is_written_whole_once_the_volume_takes_it PASS ("3 file(s), 165 kernel records, the console's in order"), and the_log_lands_on_the_boot_stick PASS.

Two harness-only fixes got it there:

• 0939e4e24: the job is handed to the boot as a test binary. run_test had refused it.
• 6405def8d: the job is typed at the guest. run_test had taken the job's window off the console, so the stick's records looked out of step with a console that was missing lines; a local run showed both sides.

17289151b closes logkeeper-ends-…, since its amended exit is met. The swapped-stick hazard moves to a-file-held-across-….

Evidence: https://github.com/ToyOSOrg/evidence/tree/dfba7eb32340a5793c7da5054a7dc6514e2b571b/r4. Host CI at a99667bf0 still holds: the later commits touch only tests/toyos.rs and issues/, which --ci host does not run, and the harness compiles.

[Japabu]

Review r3 of #1190 at 17289151b (own commits a99667bf0..17289151b on #1185 r4)

Architecture

The design is still right: logkeeper owns the stop's catch-up, and the kernel and supervisor are untouched.

• Volume::finish reuses catch_up in a loop and adds only a sync and a shortfall line. It is a small extension of the existing representation, not a second path.
• A time bound derived from toyos_quiesce::FLUSH_MS is the right unit: the supervisor's bound is time, and the bytes a device takes in that time vary.
• The new toyos-quiesce dependency is an existing workspace crate, used for the constant it owns.

Net change this round:

• About +55 production lines in logkeeper.
• About +50 lines of host tests.
• About +45 net lines of harness. The row's check grows; the fixture loses FULL_LOG_KEPT.

r2 blockers

A. Resolved for the Ok path, but the reported count is wrong on the Err path (blocker 1 below).

• finish loops while more() holds, syncs, and puts the shortfall into the file.
• A refused continuation ends with Ok, with the previous part synced by its rotation and the shortfall stated on the console.
• FLUSH's doc and the panic text now say "the flush is done".
• The three controls in r3/f6-lk-controls.log are real mutations of the three claims (no note, no sync, no loop), and each one exits 101.

B. Resolved.

• fill_log_volume fills every free cluster and asserts free_bytes() == 0 afterwards.
• The loader's refused write is reported by the loader, which then boots on.
• Run 42040641298 (gh run view: head 6405def8d, success) shows both rows PASS.
• 6405def8d..17289151b touches only issues/, so that run stands for the head.

C. Resolved. The row now asserts three things:

• The file's first line is the console's first kernel record.
• Per CPU, the stick's records are an exact prefix of the console's: no mismatch at any position, and no extras.
• No "were let go" note and no shortfall line appear.

This catches the three cases r2 listed: a repeated replay misaligns, a missing head fails begins, and a gap note is matched by text. The begins check also stops the row passing on an empty record set; the run counted 165 records.

Host case and harness changes

• a_first_part_refused_at_open_is_made_and_given_the_boot covers the refused create at open, the record's own case. It checks !is_open() after open, then checks the whole boot after the volume answers.
• Test binary instead of extra_root_files: run_test/run require the binary on ROOT as a test (qemu.rs:1951-1956). This is plumbing and weakens nothing.
• Typed run instead of run_test:
  • The exit-code check is replaced by awaiting log_unfill: /log/FILLER.BIN removed. log_unfill prints that only after remove_file returns Ok, and otherwise exits 1 without it.
  • A failure therefore stalls inside the budget instead of returning at once. That makes the failure slower but no weaker.
  • Keeping the job's window in console is what makes the C comparison sound, so this change strengthens the row.
  • Typing run … is the pattern other rows already use (toyos.rs:3343, 5221, 6141).
• One small evidence gap: r4/guest-42038743967.log is 0 bytes, so the README's account of that intermediate red has no log behind it. It does not bear on the head; it is noted for the record.

Findings

BLOCKER 1. system/logkeeper/src/main.rs:740-746: on the Err arm the console understates the shortfall.

• The Err message prints hub.end() - v.through(). through advances on every successful write, including unsynced rounds (to_volume with sync == false) and rounds whose sync was then refused.
• Log::lost → Volume::lost clears only the file and leaves through ahead of durable_through. Only a successful reopen rolls it back.
• In the record's own scenario, the volume is still refusing at the stop: finish's first catch_up fails in reopen. The message then reports the file as short by end - through.
• finish's own doc says the file is "as its last sync left it". The file that last sync guarantees is short by end - durable_through.
• The same understatement follows a refused put or sync inside finish.
• Consequence: the one durable-loss statement A exists to make reports less loss than may have occurred.
• Fix (cheapest): report the count from the durable point on Err. For example, Volume::lost can roll (size, through) back to (durable, durable_through), which is what reopen does anyway, or finish can return that count with its Err. Then extend a_stop_with_more_owed_than_its_time_syncs_and_says_the_shortfall, or add a sibling, with a stop whose reopen is refused after an unsynced write. The 0o555 directory trick from the new test does this.

BLOCKER 2. system/logkeeper/src/main.rs:600-604: caught_up's at_stop parameter is dead, and its doc is false.

• The only caller left is to_volume's caught_up(false) (:584), now that flushed uses finish.
• The doc still says a refused part "is opened again … or at the stop whatever the wait".
• Fix: drop the parameter and its !at_stop branch, and delete that clause.

Suggestions

• system/logkeeper/src/main.rs:722-723: the flushed doc ("every line is written by now, so the volume is made durable") predates the shortfall path. Either cut it to the contract, or let FLUSH's doc in logstream be the single statement.
• store.rs:472-488: the permission trick refuses nothing when the tests run as root, and then the first assert fails. Host CI is a non-root ubuntu-24.04 runner, so this is not a blocker. A directory created at the first part's path refuses the create for any user.
• Volume::finish takes seven arguments behind allow(clippy::too_many_arguments), and three of them are passed straight through to catch_up. That is acceptable for one caller. If it grows, a small struct holding owed, note and say would serve both functions.

Records

• logkeeper-ends-… is closed by deletion. Its exit ("first log write refused … /log still holds that boot's records … green on the guest runner") is met by run 42040641298. The refused create itself is held by the new host case.
• The swapped-stick hazard moves to a-file-held-across-…, so it is not dropped.
• The commit message attributes the exit amendment to the coordinator, not the owner, which is accurate.

Evidence

• r3/f6-lk-b.log: 26 passed. r3/f6-ci-host.log: --ci host exit 0 at a99667bf0.
• The later commits touch only tests/toyos.rs and issues/, which the host checks do not run, so that result carries.
• Both fixes above are logkeeper-only and need only cargo test in system/logkeeper plus a control. They do not need another guest run: the row never reaches the Err arm.

SEND BACK: blockers 1 and 2.

[Japabu]

r4 at 701156896: response to review r3

1. Fixed. Volume::lost now rolls size and through back to the last sync, as a reopen already did. The stop also calls it on any refusal before counting, so the console's shortfall counts from the last durable position.
   • Host case: a_stop_refused_after_an_unsynced_write_counts_from_the_last_sync. It makes an unsynced write, refuses the volume, then puts a directory where the part was, so the reopen is refused whoever runs it.
   • Control: without the rollback, the test fails.
2. Fixed. The at_stop parameter and its doc are gone.

Root-safe refusal, taken: the first-part test puts a directory at the part's path. ours now lists files only, because a directory is no log part and should not take a part number. Control: listing directories again fails that test.

Checks:

• --ci host: exit 0, 109 steps.
• logkeeper: 27 passed.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/835b39e07505e80434fa51a462c1476bc7b686b2/r5
• The row is rerun at this head, because lost sits on its path: run 42071298691.

[Japabu]

Guest run 42071298691 at 701156896: a_refused_log_is_written_whole_once_the_volume_takes_it PASS ("3 file(s), 165 kernel records, the console's in order").

[Japabu]

Review r4 of #1190 at 701156896 (17289151b..701156896, one commit)

Architecture

Nothing structural changes.

• The rollback sits in Volume::lost. That is where the bookkeeping belongs: reopen already resets (size, through) to the durable point, so lost now records what the file is guaranteed to hold instead of waiting for a reopen.
• Net change: about +10 production lines (logkeeper only), +25 test lines, and −6 lines from removing at_stop.

r3 blockers

1. Resolved.

• Volume::lost rolls (size, through) back to (durable, durable_through) (store.rs:258-261).
• flushed calls v.lost() when finish returns Err (main.rs:740-743), so the console's short counts from the last sync. This covers both ways finish can fail: a refused reopen, and a refused put or sync on an open file that holds unsynced flush rounds.
• I checked the other callers of lost: caught_up's Err arm, Log::lost from answered, and rotate_if_full. After a rotation durable_through == through, so the rollback changes nothing there. On a refusal, publish now reports durable bytes, which is the truer number.
• Where reopen fails after it has reset durable_through to from (a short file, then a refused set_len), lost rolls back to from, which is correct.
• New test a_stop_refused_after_an_unsynced_write_counts_from_the_last_sync (store.rs:531-549). Its control in r5/f10-controls.log removes the rollback line, and the test fails on its shortfall assert (exit 101). The control is real.
• The test reaches only the refused-reopen path. The put/sync-refused path in flushed is held by inspection. See suggestion S1.

2. Resolved. caught_up has no at_stop parameter and no !at_stop branch, and the false clause is gone from its doc (main.rs:600-612).

Findings

BLOCKER 1. system/logkeeper/src/store.rs:74: filtering ours to files only gives a squatting directory the part number, and the boot's log is then refused for good.

• Before this commit, an entry with a part's name was counted whatever its type. Volume::open's part search (:220) skipped it, and the boot wrote to the next part number.
• Now a directory at <stem>-0001 is not in kept, so part 1 is chosen. create fails, and every reopen at each retry fails the same way, for as long as the directory stays. The boot's log reaches the console only.
• undated_stem (:149) has the same hole. A directory named unknown-00-0001 no longer reserves unknown-00, so an undated boot picks that stem and then fails in the same way.
• /log is userland-writable (this module's header names toybox), so any program can now deny logkeeper the boot's file, which it could not do before. That makes the answer to the brief's question yes: a squatting directory now leaves the part number stuck.
• On overwrites: on ToyOS the fileserver always gives the entry's kind, so file_type() does not fail and no real log part drops out of the list to be truncated by create. The filter does widen what create's truncate can reach, though. A symlink with a part's name is no longer skipped, and create follows it. /log is FAT, so that is theoretical today, but it is the wrong direction to move in.
• The change exists only so the r3 suggestion's test staging refuses the create. f10-controls.log says so: the test fails with ours listing directories, "because the part opens at the next number". Root's rule is "No production machinery ships solely to satisfy a test", and here the change also makes the shipped behaviour worse. My r3 suggestion did not account for ours, and that is my error.
• Fix (cheapest): revert the filter. Make the refusal root-safe in the test instead by opening the volume on a log directory that does not yet exist, for example tmp/log. read_dir fails, so ours is empty, part 1 is chosen, and create fails with NotFound for any user. Then fs::create_dir(dir) before the second round. The v.path() == first assert still holds. The new stop test puts its directory at the path reopen opens, which does not consult ours, so it needs no change.

Suggestions

• S1. main.rs:740-743 and store.rs:358-359: finish's doc promises "an Err is … the file as its last sync left it", but the bookkeeping only matches that once the caller calls lost. Calling self.lost() on finish's own Err paths would keep the contract and the code in one place. The new test, or a sibling with a refused sync, would then also hold the open-file path. This is optional, since the current code is correct by inspection.

Evidence

• r5/f10-ci-host.log: --ci host, 109 steps, all green, exit 0.
• r5/f10-lk.log: 27 passed.
• r5/f10-controls.log: two controls, each exit 101 on the intended assert.
• The guest row never reaches these paths (r3 said so). Run 42071298691 is extra; it is not required for this round.
• The fix above touches only store.rs and its test. It needs cargo test in system/logkeeper and no guest run.

SEND BACK: blocker 1.

[Japabu]

r5 at bc1285046: response to review r4

Blocker: fixed. The files-only filter in ours is reverted. An entry under a part's name takes that number whatever it is, and the boot writes the next part.

The first-part test is still root-safe without the filter. It opens the volume on a log directory that doesn't exist yet, so the create is refused with NotFound whoever runs it, and it makes the directory before the second round.

S1: taken. Volume::finish now marks the part lost on its own refusal, so its contract no longer depends on the caller, and the stop no longer calls lost itself. A refused write or sync to an open part can't be staged root-safely on the host, so that path is still held by inspection.

Check: cargo test in system/logkeeper, exit 0, 27 passed. Evidence: https://github.com/ToyOSOrg/evidence/tree/d174fde2b0f51efeec11ba66b767d25ac88935a6/r6

Host clippy on system/logkeeper fails on four lints that predate this branch: two collapsible ifs in inspect.rs, and wrong_self_convention on from_supervisor and to_volume. None is in the changed code, and --ci host's clippy, which uses the adopted lint set, was green at r4.

[Japabu]

Review r5 of #1190 at bc1285046 (701156896..bc1285046, one commit)

Architecture

• Nothing structural changes, and the change makes the code smaller: production code loses about 3 lines and the test loses 1.
• The refusal bookkeeping now lives in one place. Volume::finish's promise that "an Err is … the part lost, the file as its last sync left it" is kept inside finish itself, so the stop in main.rs no longer has to remember to call lost.

r4 blocker

1. Resolved.

• The files-only .filter is gone from ours (store.rs:70-81), and no file_type/is_file check remains anywhere in logkeeper.
• An entry with a part's name now counts as taken whatever kind of entry it is. A squatting directory therefore pushes Volume::open's part search to the next number again, and undated_stem keeps reserving the stem.
• a_first_part_refused_at_open_is_made_and_given_the_boot (store.rs:478-492) now opens the volume on tmp/log, which does not exist yet:
  • read_dir fails, so ours is empty and part 1 is chosen.
  • create fails with NotFound. That holds for any user, root included, so the test is root-safe.
  • create_dir(&log) comes before the second round, and on_disk reads path(dir, STEM, p).
• The test still covers its claim: a first part refused at open is created at a later catch-up and receives the whole boot. Dropping the v.path() == first assert loses nothing, because no squat is staged any more.

S1, as taken

• finish wraps its whole body in one closure and calls inspect_err(|_| self.lost()) on the result (store.rs:370-381). Every ? inside it goes through that one call: the reopen in catch_up, its writes and syncs, the shortfall put, and the final sync. No refusal path can skip the rollback.
• Bookkeeping at the stop. flushed (main.rs:724-747) computes short = hub.end() - v.through() after finish returns. On Err, through is now durable_through, which is exactly what r4's caller-side v.lost() produced. This holds for both ways the stop can be refused:
  • a refused reopen: the file is already None;
  • a refused put or sync on an open part holding unsynced rounds: size and through roll back.
• No harmful double lost. The call in main.rs is removed, and catch_up never calls lost itself.
  • Volume::lost is idempotent: it sets file = None and (size, through) = (durable, durable_through), and neither durable value is changed by lost.
  • The stop tests call v.lost() before finish to stage a refusal. A second call inside finish changes nothing.
  • One case to check: when reopen resets durable_through to from and then fails at set_len, lost rolls back to from, as it did in r4. That is correct.
• No retry is armed when the stop is refused. flushed called v.lost(), not Log::lost, in r4 as well, so this is unchanged. After a resume, caught_up's get_or_insert arms the retry anyway.

Refused write or sync on an open part: no test

• Inspection is enough here. The guarantee comes from the code's shape: one closure, one inspect_err, every fallible step uses ? inside it. A test could catch only a later edit that adds an early return Err outside that closure, and that would be visible in review.
• Volume::lost's own rollback is already guarded by r4's control on a_stop_refused_after_an_unsynced_write_counts_from_the_last_sync.
• Staging an EIO on an open fd on the host would need a fault-injection seam in production code solely for the test, which root forbids.

Findings

None.

Evidence

• I inspected the diff and the touched code in full. As instructed, I ran nothing.
• The implementer reports cargo test in system/logkeeper: exit 0, 27 passed, with the evidence at d174fde2/r6.
• The change touches only store.rs and the stop's call site, so no guest run is needed. r4's --ci host result still stands apart from the compile and test of these two files, which the logkeeper test run covers.
• The four host-clippy lints the implementer reports are in unchanged code and do not appear under --ci host's lint set. I did not verify that claim independently. It does not affect this diff.

LAND

[Japabu]

The evidence branch pr-1190 was rewritten to mask machine identifiers: the T14's vendor, product and BIOS as <vendor> <product>, BIOS <bios>; the guest runner's machine and runner names as <qemu-host> and <runner>; and the runner's home path as <runner>. The body's links are repinned. Links in earlier comments map as follows (old → new):

old	new
d174fde2b	29c1a3d1815ff76dab812969997cb3db20c50b61
835b39e07	ca0b1ad1904e6bc6d377c60a26856e78c134683e
dfba7eb32	7b8b6a8a2469f6b27b43186a7d513b0d19e06e13
a9b38564a	858beb0c3d16f30ff5fd068c4602a46bd8cc76c2
5bbdf3608	51ba7703a1b6ca4feeb180a9b197099c464f5a18