[Japabu]

Closes issues/a-test-build-waited-forty-minutes-on-its-own-processs-sysroot-lock.md and issues/a-suite-that-builds-its-sysroot-can-wedge-on-its-own-locks.md (both deleted). Records the polling compromise in issues/macos-lets-blocked-shared-flock-waiters-in-one-at-a-time.md.

Problem

In a 12-wide cargo test --test toyos-build, a guest build waited 2611 s on a sysroot key lock that its own process had finished with. I reproduced two causes:

1. macOS lets blocked shared flock waiters in one at a time. Measured on Darwin 27.0.0, across threads and across processes (probe, output). When an exclusive holder releases, macOS admits one of the shared waiters queued behind it. Each of the others is admitted only when the shared holder ahead of it releases, although a new non-blocking shared try succeeds at once. A build holds its uses (of a sysroot, of the worktree lock) for its whole length, so a waiter behind one waits that whole length, and forever behind a use held to the end of a run. That matches the incident log: one waiter was admitted after the sysroot build, five more after 17:45:06 when the failing shm_commit build dropped its use, and the seventh stayed behind a use that was never dropped.
2. keyed_made queued exclusively to make a key it had found unmade. When another waiter made the key in the meantime and its users kept their uses, the queued maker waited until every one of them finished.

The wedge issue is the first cause. Its recurrences on #836 r4 and #1052 r2 have the same signature: after the sysroot build, one worker is left alone on using sysroot … — the holder left no readable note, and lsof shows the lock open only in the harness. Its first case fits too: shared waits on the worktree lock's state file go through the same admission. The ordering inversion it names has no path: sysroot, compiler and freestanding keys are all taken with the worktree lock put down (Held::without_shared, at sysroot.rs:832 and compiler.rs:150; the freestanding make runs inside the sysroot build). The LLVM and ToyOS-hosted clang keys are taken inside it, which is the documented order.

Change (build/buildlock.rs)

• One poll loop. Every acquisition is one poll over an attempt:
  • take tries a lock mode;
  • keyed_using tries a use and dates it;
  • keyed_made uses the key if whole, or else tries to make it.
• What the loop does. It sleeps 1 ms at first, doubling up to 50 ms. It prints "waiting", then "still waiting" every 30 s, then "acquired". After a 6 h ceiling it panics, naming the holder, e.g. build lock: gave up waiting for the sysroot lock (using sysroot K) after … — held by pid N (building sysroot K). 6 h is above the 350 minutes .github/workflows/toolchain.yml allows the whole toolchain.
• The maker tries. keyed_building is a try that returns Option<Guard>.
• Removed: the flock FFI and the blocking take, replaced by std's File::try_lock{,_shared}, and the heartbeat thread with its channel.

Decisions and costs

• The poll is a workaround for the macOS admission defect, recorded with its exit in the new issue. Without that defect, a blocking wait with a timed join would work.
• Costs:
  • up to 50 ms between a release and a long waiter's next try;
  • no FIFO order among waiters of one mode, so a new waiter can overtake long ones, bounded only by the ceiling. The artifact lock is the exposed one.
• Kept: exclusive-before-shared ordering, through intent. A holder that dies still releases its lock.
• One ceiling for every lock, as a backstop. The heartbeat is what an agent sees.

Tests (host, buildlock::tests)

• waiters_on_a_build_all_get_in: eight threads wait on another thread's make of a sysroot. All eight must hold their use at once, both when that make succeeds (the incident) and when it fails and a waiter makes the key.
• a_would_be_maker_does_not_wait_out_another_use: the key is whole and in use by another process, and defect reports it unmade once.
• a_wait_past_its_deadline_names_the_holder: keyed_made with a 500 ms deadline on a key that another holds in the making must fail, naming held by pid <self> (building sysroot K).

Negative controls (patches and logs): each built (exit 0) and its test failed (exit 101).

• a: the base code with the scenario test admits 1 of 8.
• b: keyed_try_using blocking in the kernel admits 1 of 8.
• c: poll never checking its deadline never ends.
• d: keyed_building waiting instead of trying waits out another's use.

Checks

• End to end. cargo test --test toyos-build of seven of the incident's eight tests, 12 wide, on a head whose sysroot was not yet made (log): exit 0, 7 of 7. The eighth, shm_commit_alone, is not on main. To make the sysroot unmade I appended const _: () = (); to sdk/libc/src/alarmreq.rs, uncommitted, and reverted it afterwards. Six workers waited on building sysroot b68c8b152facb43c and all six were admitted in the same second, 178 s later.
• Clippy with the adopted lints over toyos-build --all-targets at the head, command line in the log: exit 0. The buildlock tests at the head (log): exit 0.
• Reused from r1: cargo run -- --ci host (log): exit 1. Its one red step was a clippy redundant_clone in a new test, since fixed. This round changes only build/buildlock.rs and issues/, so the other steps cannot change. --build-only (log): exit 0.
• Not run: the same unmade-sysroot suite on base code as a control. The three recorded incidents stand in for it.

Risks

• The admission defect is measured on one macOS version. The fix does not depend on it.
• Within-mode starvation is possible. See the new issue.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1145 at dce7a48

Architectural assessment

build/buildlock.rs still owns all build serialisation. Replacing the flock FFI with std's File::try_lock{,_shared} is a good change: less unsafe, and the same primitive on every host (flock(2) on Linux and macOS, LockFileEx on Windows). Production code goes from 511 to 484 lines and tests from 461 to 565. Dropping the heartbeat thread and its channel is a real simplification.

Is polling justified? Yes, as a workaround for a host defect. I looked for an event-based mechanism and found none that is simpler and correct:

• Exclusive-only locks with a reader count lose the crash release that flock gives readers. Getting it back needs one lock file per reader, and a writer then has to wait on N files. That is more mechanism, and it changes the ordering.
• A per-key build-done file watched through kqueue, inotify or ReadDirectoryChanges needs a different API on each host (or a notify dependency), has a race between checking and starting to watch, and does nothing for the worktree lock's shared waits.
• F_SETLKWTIMEOUT exists only on macOS. It also takes POSIX record locks, which belong to the process, so the threads of one harness (the incident) would not exclude each other.
• A blocking wait on a helper thread with recv_timeout gives a timed wait on Linux and Windows, but on macOS it still hits the defect: control b admits 2 of 8.

So the macOS defect is what forces the poll. The lack of a timed wait does not force it by itself, which means the commit message's "a wait with a failure timeout over it is a poll anyway" overstates the case. Under root's no-silent-debt rule the compromise needs a record (B2).

Starvation. There is no FIFO within a mode. A new waiter tries at once, then after 1, 2, 4 ms and so on, so it beats long waiters that sleep 50 ms for roughly its first 100 ms. Among long waiters the winner depends only on where each one's sleep falls. Being overtaken without limit is possible, and only the ceiling bounds it. This matters most on the artifact lock: the incident log shows five waiters there for over 100 s. Writer preference still holds, because intent is taken before state (a_queued_exclusive_phase_goes_first). It is acceptable once recorded.

Ceiling. toolchain.yml does have timeout-minutes: 350, and 6 h is a sound fence for hangs. The 30 s heartbeat is what an agent actually sees.

Crash. std's lock is flock(2) on Unix, scoped to the open file description, so a dead holder still releases it, and killed_holder_releases_the_lock still covers this. A waiter that panics at its ceiling inside acquire unwinds and drops intent.

Hosts. On Linux the semantics are unchanged. Linux admits all compatible waiters, so the poll costs only handoff latency there. The build system does not compile on Windows (an existing issue). This change makes Windows no worse, and the FFI it removes was Unix-only.

Probe. It can be reproduced from the evidence: a stand-alone program with its source, manifest, uname and output, independent of ToyOS code. The grants come at 730, 1140, 1942 and 3148 ms, and each lands within about 10 ms of when the previous holder's 400, 800 or 1200 ms hold ends. Controls a and b show the same behaviour through ToyOS's own code.

Tests and controls. Each of the three new tests has a control that builds and fails, so each can expose its defect. The control snapshot differs from the head only in two test lines (I checked with diff). waiters_on_a_failed_build_all_get_in has a 300 ms settle sleep. It can only weaken the test, never fail it falsely, which is acceptable because the "waiting" state cannot be observed without instrumentation. I confirmed there are no build/ changes in 4768c76..9b3711c or in 9b3711c..origin/main, so the reused host-CI evidence still applies.

Required

B1. build/buildlock.rs:262-277: keyed_made's retry is a second wait loop, and it is silent. It has its own deadline, its own backoff and an unannounced sleep. It also calls describe_holder (open, read, kill) on every round, even though it uses the result only at the ceiling. When the try fails because another process holds a use of a key that this caller's defect reports as not made, keyed_using succeeds at once each round. Nothing is printed for up to 6 h; the old code announced an exclusive wait with a heartbeat. Two worktrees at different revisions with different checks for whether a key is whole could trigger this (an inference; not observed). It breaks the contract take itself states (:372-375, "an agent staring at silence kills and retries"). The ceiling in this loop is also untested, and each keyed_using inside it starts a fresh 6 h deadline.

• Fix: one poll loop. For example, make take's loop generic over an attempt closure and give keyed_made the step "use it if whole, else try to make it". Deadline, heartbeat, backoff and the deadline test then each exist once.

B2. The polling compromise is not recorded in issues/. Root forbids a fixed delay where an event can be awaited, and says a discovered compromise is removed or recorded with evidence, owner and exit. The poll exists because of a host defect (see the assessment above), and it costs up to 50 ms per contended handoff and FIFO order within a mode. The PR body is not a record.

• Fix: file an issue linking the probe and controls, with the costs, an owner and an exit. A possible exit: every host admits all compatible waiters when a lock is released, so a blocking wait plus a timed join can replace the poll.
• Then shorten the comment at :377-382 to the constraint plus the issue name. The Darwin 27.0.0 measurement belongs in that record, not in the comment.

B3. Two comments are now false.

• :823-825: "waits on the builder's lock — not on a timer —". It now waits on a timer.
• :492-496: "a single-process test would prove nothing about the thing that actually races". This contradicts the new tests, whose docs say "The waiters are threads of one process, as a test run's builds are". It also contradicts the incident itself. flock belongs to the open file description, so threads that each open the file do contend. Narrow the comment to what the child processes are actually needed for: holder death, and holders that outlive a thread's drop.

B4. issues/a-suite-that-builds-its-sysroot-can-wedge-on-its-own-locks.md is directly affected and is not updated. Its recurrences (#836 r4, #1052 r2) have this defect's exact signature:

• after the sysroot build, one worker is left alone on using sysroot … — the holder left no readable note;
• lsof shows the lock open only in the harness.

The PR closes the forty-minutes issue and leaves this one open with a cause that may now be wrong. Do one of these:

• update it with this mechanism and what still remains of the ordering inversion it describes;
• close it with evidence;
• say why it is a different defect.

Suggestions

• Commit message and PR body: "six waiters went on at 17:45:06" is off by one. In qemu1.log the count of waiters drops from 7 to 6 between 17:44:02 and 17:44:32, when one waiter is admitted after the sysroot build. It drops from 6 to 1 between 17:45:02 and 17:45:32, so five went on after the failure. Serial admission of short holds fits the chain model better than "six at once" does. The PR-body correction is the orchestrator's to make.
• Commit message: reword "is a poll anyway". See the assessment: the macOS admission is what forces the poll.
• :344-346: "holds intent while it queues for state" / "line up behind it". Nothing queues any more, so "waits for" is accurate.
• MAX_PAUSE doc (:77): it says "([take])", but keyed_made also uses it. B1's fix removes this.
• Evidence: clippy-build.log does not record its command line. CI reruns clippy anyway.
• Optional: a cargo test --test toyos-build run on a head whose sysroot is not yet made, the condition all three incidents shared. It is the cheapest end-to-end check, and B4 needs it if that issue is to be closed.

SEND BACK