Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
status: open
kind: tooling
opened: 2026-10-09
---

# A shared member's allowance sets the kernel's deadline at several times the work, and no reading holds it

`toyos_tco::RUST_MEMBER_MS` and `toyos_tco::C_MEMBER_MS` are what one member
of a metal shared boot adds to its list's bound (`toyos_tco::list_bound_ms`).
The kernel's `boot-deadline=` is twice that bound and its hard-lockup bound is
half the deadline, so the two constants set how long a failed shared boot holds
the T14. They are 300 and 100 ms by the owner's ruling, not by a derivation:
nothing ties them to what the machine measures, and nothing reds when the
measurement moves.

## Readings

The T14, three boots at each of two heads; the judge's own lines. `eff8b20ee`
was armed with the 860 and 260 ms the allowances were then, `d6d008e88` with
300 and 100:

| boot | members | head | ms a member | the list's last record | allowance now | factor |
|---|---|---|---|---|---|---|
| `shared` | 88 | `eff8b20ee` | 80 | 8 264 ms | 300 | 3.7 |
| | | `d6d008e88` | 81 | 8 308 ms | 300 | 3.7 |
| `ccorpus` | 137 | `eff8b20ee` | 12 | 2 864 ms | 100 | 8.3 |
| | | `d6d008e88` | 15 | 3 223 ms | 100 | 6.7 |
| `shared-debug` | 7 | `eff8b20ee` | 32 | 1 380 ms | 300 | 9.4 |
| | | `d6d008e88` | 32 | 1 389 ms | 300 | 9.4 |

Each last record counts from the kernel's zero, so it holds about 1.2 s of
boot before the first job.

**The mean hides the member.** From each member's own start and end records in
the two kernel logs:

| boot | head | the members' sum | members past their allowance | their sum | the slowest |
|---|---|---|---|---|---|
| `shared` | `eff8b20ee` | 7 068 ms | 5 of 88 | 5 548 ms | `abuse_mmap_regions`, 2 875 ms |
| | `d6d008e88` | 7 115 ms | 5 of 88 | 5 616 ms | `abuse_mmap_regions`, 2 955 ms |
| `ccorpus` | `eff8b20ee` | 1 667 ms | 1 of 137 | 1 141 ms | `124_atomic_counter`, 1 141 ms |
| | `d6d008e88` | 2 031 ms | 1 of 137 | 1 517 ms | `124_atomic_counter`, 1 517 ms |

The five on `shared` at `d6d008e88` are `abuse_mmap_regions` 2 955 ms,
`fs_cache_eviction` 1 066, `abuse_thread_table` 787, `mutual_kill` 405 and
`allocator_stress` 403. The slowest Rust member takes nearly ten times its allowance
and the slowest C case fifteen times, and that case moved by a third between
two boots of one job list.

**The bound holds on the list's sum, not per member.** What the members add
without the 60 000 ms base, 26 400 ms on `shared` and 13 700 ms on `ccorpus`,
covers the 7 115 and 2 031 ms they took. It does not hold by construction: a
list made only of members as slow as `abuse_mmap_regions` passes its bound at
the 23rd.

## The price, on failure only

What `--metal --list` and `metal::return_secs` give at 300 and 100 ms, against
the 60 000 ms list bound and 120 000 ms deadline of a boot no member rides:

| boot | a hung member, and a CPU that takes no interrupt, hold the machine | a wedged kernel | a boot that never returns is waited |
|---|---|---|---|
| no member | 60.0 s | 120.0 s | 420 s |
| `shared-debug` | 62.1 s | 124.2 s | 425 s |
| `ccorpus` | 73.7 s | 147.4 s | 448 s |
| `shared` | 86.4 s | 172.8 s | 473 s |

**It grows with every member and has no ceiling**: 0.6 s of deadline a Rust
member and 0.2 s a C case. The 88 Rust members and 137 C cases on one boot,
which is what joining `shared` and `ccorpus` to `testcases` makes, arm a list
bound of 100 100 ms and a deadline of 200 200 ms, waited 501 s. A clamp is no
answer: a ceiling under the derived bound ends a healthy list.

**A late expiry adds to the longer bound.**
`issues/a-120000-ms-boot-deadline-fired-132859-ms-late-on-the-t14.md` is open:
that deadline was reached at 252 859 ms, and the same lateness on `shared` is
305 659 ms. The wait moves with the deadline, so what is left of it after such
an expiry is the same 167 s; the machine is held 52.8 s longer.

## Owner

The metal suite: `tests/common/metal.rs`, which derives the bounds, and
`toyos-tco`, which declares the allowances.

## Exit

Both, on the T14:

- Each allowance is derived at its declaration from what the T14 measures of a
list's total over its members, by one stated factor: a factor over a
per-member mean says nothing of a list of slow members. And the judge reds a
shared boot whose list's last record came past half its bound, which today
is a line a reader reads.
- The boot that carries `shared`, `ccorpus` and `testcases` together reads its
list's last record within half its list bound, with the pair read from the
kernel's own `boot deadline:` and `hard lockup:` lines.
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,10 @@ and takes no VFS lock. So it is not this.
`kernel/src/deadline.rs`: a bound armed off the parameter line and polled from
the timer interrupt entry in both rings on every CPU, which on expiry seals a
`WEDGED` record carrying **the tail of the log ring** into the black box and
writes the reset register itself. Every metal image carries it
(`toyos_tco::WEDGE_BOUND_MS`, 120 s). The next occurrence therefore ends itself
writes the reset register itself. Every metal image carries one: twice its
list's bound (`toyos_tco::wedge_bound_ms`), which is 120 s where no shared
member rides the boot, or `toyos_tco::STAGED_BOUND_MS` where the boot stages
its own wedge. The next occurrence therefore ends itself
without a hand and leaves the records `logd` never wrote, which is the one
channel that crosses a reset without `logd`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ opened: 2026-09-30
# `toyos_tco::HARD_LOCKUP_BOUND_MS` is read by nothing but its own assertion

`toyos-tco/src/lib.rs` declares `HARD_LOCKUP_BOUND_MS`,
`hard_lockup_bound_ms(WEDGE_BOUND_MS)`, as "the one a T14 boot runs under",
`hard_lockup_bound_ms(wedge_bound_ms(JOB_BOUND_MS))`, as "the one a T14 boot runs under",
and the `const _` beside it is its one reader. Its last other reader was the
metal profile's list of declared ceilings, which went with
`tests/metal-profile.toml`. The T14's `hardlockup` arm no longer runs under it
Expand Down
5 changes: 3 additions & 2 deletions issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,9 @@ Each was an issue of its own and is true of `809c33c0c`'s code, so it comes
back with it:

- A swap on the T14 lives inside a metal boot's bound: every metal image
carries `boot-deadline=` (`toyos_tco::WEDGE_BOUND_MS`) and a runner whose
list ends in `reboot` (`toyos_tco::JOB_BOUND_MS`), so the loop the owner
carries `boot-deadline=` (`toyos_tco::wedge_bound_ms` of its list's bound)
and a runner whose list ends in `reboot` under that bound
(`toyos_tco::list_bound_ms`), so the loop the owner
asked for — netstack rebuilt on the host and swapped in, over and over —
has the span from the first lease to the runner's bound, and then a flash.
Owed: a boot staged for the swap loop whose hold is not the runner's, and a
Expand Down
13 changes: 7 additions & 6 deletions issues/the-t14-reboots-through-ubuntu-for-every-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,18 @@ metal rows.
Every stage below has the host drive a T14 that runs ToyOS, so the track is
blocked on `issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md`.

**Next: two sessions in place of six boots.** `shared`, `shared-2`, `ccorpus`,
**Next: two sessions in place of five boots.** `shared`, `ccorpus`,
`testcases`, `testcases-mkdir` and `testcases-readdir` share a config, a
parameter line and the shipping kernel. The session image is `tests/testcases`
with the netd, sshd and streaming `logd` of `tests/lantalkcase`, which is
deleted (`issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md`),
under the same 120 s `boot-deadline=`. A session holds 74.8 s of members, priced at
under a `boot-deadline=` derived as a metal boot's is: twice its list's bound
(`toyos_tco::wedge_bound_ms`), which is `toyos_tco::JOB_BOUND_MS` and
`toyos_tco::RUST_MEMBER_MS` a Rust member, `toyos_tco::C_MEMBER_MS` a C case
and a list at twice its slowest on the T14: the bound less a tenth, less a
lease as late as 19.1 s
(`issues/most-t14-leases-land-one-dhcp-retry-late.md`), less one
per-job bound. Every judge reads the stick as today but `syscall_cost`, which
(`issues/a-shared-members-allowance-sets-the-kernels-deadline-at-several-times-the-work.md`).
A session's bound owes, beside its members, a lease as late as 19.1 s
(`issues/most-t14-leases-land-one-dhcp-retry-late.md`) and one per-job
bound. Every judge reads the stick as today but `syscall_cost`, which
reads the window. `loader_watchdog_arms`' control arm rides a session,
`mkdir_cap` and `readdir_bound` remove what they made, and `audio_idle_suspend`
waits for soundd's `inspect` to read `suspended`
Expand Down
Loading
Loading