diff --git a/issues/a-shared-members-allowance-sets-the-kernels-deadline-at-several-times-the-work.md b/issues/a-shared-members-allowance-sets-the-kernels-deadline-at-several-times-the-work.md new file mode 100644 index 00000000000..81e83d2a4cb --- /dev/null +++ b/issues/a-shared-members-allowance-sets-the-kernels-deadline-at-several-times-the-work.md @@ -0,0 +1,97 @@ +--- +status: open +kind: tooling +opened: 2026-10-09 +--- + +# A shared member's allowance sets the kernel's deadline at several times the work, and no reading holds it + +`toyos_tco::RUST_MEMBER_MS` and `toyos_tco::C_MEMBER_MS` are what one member +of a metal shared boot adds to its list's bound (`toyos_tco::list_bound_ms`). +The kernel's `boot-deadline=` is twice that bound and its hard-lockup bound is +half the deadline, so the two constants set how long a failed shared boot holds +the T14. They are 300 and 100 ms by the owner's ruling, not by a derivation: +nothing ties them to what the machine measures, and nothing reds when the +measurement moves. + +## Readings + +The T14, three boots at each of two heads; the judge's own lines. `eff8b20ee` +was armed with the 860 and 260 ms the allowances were then, `d6d008e88` with +300 and 100: + +| boot | members | head | ms a member | the list's last record | allowance now | factor | +|---|---|---|---|---|---|---| +| `shared` | 88 | `eff8b20ee` | 80 | 8 264 ms | 300 | 3.7 | +| | | `d6d008e88` | 81 | 8 308 ms | 300 | 3.7 | +| `ccorpus` | 137 | `eff8b20ee` | 12 | 2 864 ms | 100 | 8.3 | +| | | `d6d008e88` | 15 | 3 223 ms | 100 | 6.7 | +| `shared-debug` | 7 | `eff8b20ee` | 32 | 1 380 ms | 300 | 9.4 | +| | | `d6d008e88` | 32 | 1 389 ms | 300 | 9.4 | + +Each last record counts from the kernel's zero, so it holds about 1.2 s of +boot before the first job. + +**The mean hides the member.** From each member's own start and end records in +the two kernel logs: + +| boot | head | the members' sum | members past their allowance | their sum | the slowest | +|---|---|---|---|---|---| +| `shared` | `eff8b20ee` | 7 068 ms | 5 of 88 | 5 548 ms | `abuse_mmap_regions`, 2 875 ms | +| | `d6d008e88` | 7 115 ms | 5 of 88 | 5 616 ms | `abuse_mmap_regions`, 2 955 ms | +| `ccorpus` | `eff8b20ee` | 1 667 ms | 1 of 137 | 1 141 ms | `124_atomic_counter`, 1 141 ms | +| | `d6d008e88` | 2 031 ms | 1 of 137 | 1 517 ms | `124_atomic_counter`, 1 517 ms | + +The five on `shared` at `d6d008e88` are `abuse_mmap_regions` 2 955 ms, +`fs_cache_eviction` 1 066, `abuse_thread_table` 787, `mutual_kill` 405 and +`allocator_stress` 403. The slowest Rust member takes nearly ten times its allowance +and the slowest C case fifteen times, and that case moved by a third between +two boots of one job list. + +**The bound holds on the list's sum, not per member.** What the members add +without the 60 000 ms base, 26 400 ms on `shared` and 13 700 ms on `ccorpus`, +covers the 7 115 and 2 031 ms they took. It does not hold by construction: a +list made only of members as slow as `abuse_mmap_regions` passes its bound at +the 23rd. + +## The price, on failure only + +What `--metal --list` and `metal::return_secs` give at 300 and 100 ms, against +the 60 000 ms list bound and 120 000 ms deadline of a boot no member rides: + +| boot | a hung member, and a CPU that takes no interrupt, hold the machine | a wedged kernel | a boot that never returns is waited | +|---|---|---|---| +| no member | 60.0 s | 120.0 s | 420 s | +| `shared-debug` | 62.1 s | 124.2 s | 425 s | +| `ccorpus` | 73.7 s | 147.4 s | 448 s | +| `shared` | 86.4 s | 172.8 s | 473 s | + +**It grows with every member and has no ceiling**: 0.6 s of deadline a Rust +member and 0.2 s a C case. The 88 Rust members and 137 C cases on one boot, +which is what joining `shared` and `ccorpus` to `testcases` makes, arm a list +bound of 100 100 ms and a deadline of 200 200 ms, waited 501 s. A clamp is no +answer: a ceiling under the derived bound ends a healthy list. + +**A late expiry adds to the longer bound.** +`issues/a-120000-ms-boot-deadline-fired-132859-ms-late-on-the-t14.md` is open: +that deadline was reached at 252 859 ms, and the same lateness on `shared` is +305 659 ms. The wait moves with the deadline, so what is left of it after such +an expiry is the same 167 s; the machine is held 52.8 s longer. + +## Owner + +The metal suite: `tests/common/metal.rs`, which derives the bounds, and +`toyos-tco`, which declares the allowances. + +## Exit + +Both, on the T14: + +- Each allowance is derived at its declaration from what the T14 measures of a + list's total over its members, by one stated factor: a factor over a + per-member mean says nothing of a list of slow members. And the judge reds a + shared boot whose list's last record came past half its bound, which today + is a line a reader reads. +- The boot that carries `shared`, `ccorpus` and `testcases` together reads its + list's last record within half its list bound, with the pair read from the + kernel's own `boot deadline:` and `hard lockup:` lines. diff --git a/issues/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md b/issues/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md index 760bef657f4..81c6188d1bc 100644 --- a/issues/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md +++ b/issues/a-t14-boot-wedges-after-a-jobs-exit-and-nothing-said-why.md @@ -77,8 +77,10 @@ and takes no VFS lock. So it is not this. `kernel/src/deadline.rs`: a bound armed off the parameter line and polled from the timer interrupt entry in both rings on every CPU, which on expiry seals a `WEDGED` record carrying **the tail of the log ring** into the black box and -writes the reset register itself. Every metal image carries it -(`toyos_tco::WEDGE_BOUND_MS`, 120 s). The next occurrence therefore ends itself +writes the reset register itself. Every metal image carries one: twice its +list's bound (`toyos_tco::wedge_bound_ms`), which is 120 s where no shared +member rides the boot, or `toyos_tco::STAGED_BOUND_MS` where the boot stages +its own wedge. The next occurrence therefore ends itself without a hand and leaves the records `logd` never wrote, which is the one channel that crosses a reset without `logd`. diff --git a/issues/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md b/issues/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md index 10287d4472c..00702425cf4 100644 --- a/issues/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md +++ b/issues/hard-lockup-bound-ms-is-read-by-nothing-but-its-own-assertion.md @@ -7,7 +7,7 @@ opened: 2026-09-30 # `toyos_tco::HARD_LOCKUP_BOUND_MS` is read by nothing but its own assertion `toyos-tco/src/lib.rs` declares `HARD_LOCKUP_BOUND_MS`, -`hard_lockup_bound_ms(WEDGE_BOUND_MS)`, as "the one a T14 boot runs under", +`hard_lockup_bound_ms(wedge_bound_ms(JOB_BOUND_MS))`, as "the one a T14 boot runs under", and the `const _` beside it is its one reader. Its last other reader was the metal profile's list of declared ceilings, which went with `tests/metal-profile.toml`. The T14's `hardlockup` arm no longer runs under it diff --git a/issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md b/issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md index d2d15cf0b12..936917fbaa0 100644 --- a/issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md +++ b/issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md @@ -105,8 +105,9 @@ Each was an issue of its own and is true of `809c33c0c`'s code, so it comes back with it: - A swap on the T14 lives inside a metal boot's bound: every metal image - carries `boot-deadline=` (`toyos_tco::WEDGE_BOUND_MS`) and a runner whose - list ends in `reboot` (`toyos_tco::JOB_BOUND_MS`), so the loop the owner + carries `boot-deadline=` (`toyos_tco::wedge_bound_ms` of its list's bound) + and a runner whose list ends in `reboot` under that bound + (`toyos_tco::list_bound_ms`), so the loop the owner asked for — netstack rebuilt on the host and swapped in, over and over — has the span from the first lease to the runner's bound, and then a flash. Owed: a boot staged for the swap loop whose hold is not the runner's, and a diff --git a/issues/the-t14-reboots-through-ubuntu-for-every-test.md b/issues/the-t14-reboots-through-ubuntu-for-every-test.md index 5cd11a383dc..2278d88133e 100644 --- a/issues/the-t14-reboots-through-ubuntu-for-every-test.md +++ b/issues/the-t14-reboots-through-ubuntu-for-every-test.md @@ -44,17 +44,18 @@ metal rows. Every stage below has the host drive a T14 that runs ToyOS, so the track is blocked on `issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md`. -**Next: two sessions in place of six boots.** `shared`, `shared-2`, `ccorpus`, +**Next: two sessions in place of five boots.** `shared`, `ccorpus`, `testcases`, `testcases-mkdir` and `testcases-readdir` share a config, a parameter line and the shipping kernel. The session image is `tests/testcases` with the netd, sshd and streaming `logd` of `tests/lantalkcase`, which is deleted (`issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md`), -under the same 120 s `boot-deadline=`. A session holds 74.8 s of members, priced at +under a `boot-deadline=` derived as a metal boot's is: twice its list's bound +(`toyos_tco::wedge_bound_ms`), which is `toyos_tco::JOB_BOUND_MS` and `toyos_tco::RUST_MEMBER_MS` a Rust member, `toyos_tco::C_MEMBER_MS` a C case -and a list at twice its slowest on the T14: the bound less a tenth, less a -lease as late as 19.1 s -(`issues/most-t14-leases-land-one-dhcp-retry-late.md`), less one -per-job bound. Every judge reads the stick as today but `syscall_cost`, which +(`issues/a-shared-members-allowance-sets-the-kernels-deadline-at-several-times-the-work.md`). +A session's bound owes, beside its members, a lease as late as 19.1 s +(`issues/most-t14-leases-land-one-dhcp-retry-late.md`) and one per-job +bound. Every judge reads the stick as today but `syscall_cost`, which reads the window. `loader_watchdog_arms`' control arm rides a session, `mkdir_cap` and `readdir_bound` remove what they made, and `audio_idle_suspend` waits for soundd's `inspect` to read `suspended` diff --git a/src/metal.rs b/src/metal.rs index 40150e1ade2..c2abba366bf 100644 --- a/src/metal.rs +++ b/src/metal.rs @@ -37,29 +37,25 @@ const GOING_DOWN_SECS: u64 = 120; /// the firmware's pass and Ubuntu's own boot. const RETURN_ALLOWANCE_SECS: u64 = 300; -/// Every bound a metal boot runs under, by the constant that arms it: the -/// firmware's over the span before the handoff, the TCO the loader arms there -/// and the kernel keeps feeding, the runner's own over its job list — which no -/// watchdog covers, because a kernel with an unfinished job is alive — the -/// panicked kernel's own over its panel, and the kernel's own over the whole -/// boot, which is the only one that reaches a machine still executing and -/// making no progress. -const WATCHDOG_BOUNDS_MS: &[u64] = &[ - toyos_tco::FIRMWARE_BOUND_MS, - toyos_tco::BOUND_MS, - toyos_tco::JOB_BOUND_MS, - toyos_tco::PANIC_BOUND_MS, - toyos_tco::WEDGE_BOUND_MS, -]; - -/// How long the machine has to answer `ssh` again after `reboot`. +/// Every bound a metal boot runs under that its image does not carry, by the +/// constant that arms it: the firmware's over the span before the handoff, the +/// TCO the loader arms there and the kernel keeps feeding, and the panicked +/// kernel's own over its panel. +const WATCHDOG_BOUNDS_MS: &[u64] = + &[toyos_tco::FIRMWARE_BOUND_MS, toyos_tco::BOUND_MS, toyos_tco::PANIC_BOUND_MS]; + +/// How long a machine flashed with an image armed with `deadline_ms` has to +/// answer `ssh` again after `reboot`. /// /// **Derived, because a literal is wrong the day any of it moves.** A boot is -/// only certainly over once the longest of [`WATCHDOG_BOUNDS_MS`] could have -/// fired; [`RETURN_ALLOWANCE_SECS`] is what coming back costs after that. -pub fn return_secs() -> u64 { - let longest = - WATCHDOG_BOUNDS_MS.iter().copied().max().expect("a metal boot runs under a watchdog"); +/// only certainly over once the longest of [`WATCHDOG_BOUNDS_MS`] and the +/// kernel's own bound over the whole boot could have fired: that one is the +/// image's, the only one that reaches a machine still executing and making no +/// progress, and it outlasts the runner's bound over its list, which +/// `toyos_tco::wedge_bound_ms` derives it from. [`RETURN_ALLOWANCE_SECS`] is +/// what coming back costs after that. +pub fn return_secs(deadline_ms: u64) -> u64 { + let longest = WATCHDOG_BOUNDS_MS.iter().copied().fold(deadline_ms, u64::max); longest.div_ceil(1_000) + RETURN_ALLOWANCE_SECS } @@ -822,12 +818,13 @@ pub fn stages_a_wedge(armed: &[impl AsRef]) -> bool { armed.iter().any(|name| WEDGE_ARMS.contains(&name.as_ref())) } -/// The `boot-deadline=` bound an image armed with `armed` carries. -pub fn bound_for(armed: &[impl AsRef]) -> u64 { +/// The `boot-deadline=` bound an image armed with `armed` carries, whose +/// runner bounds its list at `list_ms`. +pub fn bound_for(armed: &[impl AsRef], list_ms: u64) -> u64 { if stages_a_wedge(armed) { toyos_tco::STAGED_BOUND_MS } else { - toyos_tco::WEDGE_BOUND_MS + toyos_tco::wedge_bound_ms(list_ms) } } @@ -850,16 +847,18 @@ pub fn flashable(name: &str) -> bool { /// The pre-flash gate: what the image is armed with, judged before it is /// written. Read off the image's own ESP, so it answers about the artifact /// rather than about whoever built it. -fn arms_are_admissible(path: &Path) -> Result, Refusal> { +fn arms_are_admissible(path: &Path) -> Result<(Vec, u64), Refusal> { let armed = crate::image::params_of(path) .map_err(|why| Refusal::File { path: path.display().to_string(), why })?; - judge_arms(&armed)?; - Ok(armed) + let deadline_ms = judge_arms(&armed)?; + Ok((armed, deadline_ms)) } /// The gate's decision, over the list alone — so it can be staged without an -/// image, which is the only way the refusals below get a test at all. -pub fn judge_arms(armed: &[String]) -> Result<(), Refusal> { +/// image, which is the only way the refusals below get a test at all. An +/// admitted image answers with the bound on its own boot, in milliseconds, +/// which is what the loop waits it out by. +pub fn judge_arms(armed: &[String]) -> Result { // **Every flashed image carries a bound on its own boot, and this is the // gate.** The bound was first asked for only of the image that stages a // wedge, which is the sharpest case and not the only one: a kernel that @@ -868,12 +867,13 @@ pub fn judge_arms(armed: &[String]) -> Result<(), Refusal> { // measured twice on the T14, once as a hang after the job list and once as // a `--install-sudoers` that fell through into flashing an ordinary // `target/bootable.img`, which has no job list and no deadline at all. - if !armed.iter().any(|name| name.starts_with(toyos_tco::DEADLINE_PARAM)) { + // A bound nothing can read is no bound: the kernel panics on it. + let Some(Ok(deadline_ms)) = toyos_tco::deadline_in(&armed.join(",")) else { return Err(Refusal::NoBound { staged_a_wedge: stages_a_wedge(armed) }); - } + }; match armed.iter().find(|name| !flashable(name)) { Some(name) => Err(Refusal::Armed { name: name.clone() }), - None => Ok(()), + None => Ok(deadline_ms), } } @@ -1345,13 +1345,12 @@ declare_flags!(METAL = { INSTALL_SUDOERS = "--install-sudoers", Next; READBACK = "--readback", Next; FAT32_CHECK = "--fat32-check", None; - WAIT_SECS = "--wait-secs", Next; DRY_RUN = "--dry-run", None; }); /// The flags that describe a boot, as against the ones that say which machine /// to reach: [`Args::parse`] refuses an `--install-sudoers` beside any of them. -const ABOUT_A_BOOT: &[&Flag] = &[&DRY_RUN, &IMAGE, &READBACK, &FAT32_CHECK, &WAIT_SECS]; +const ABOUT_A_BOOT: &[&Flag] = &[&DRY_RUN, &IMAGE, &READBACK, &FAT32_CHECK]; /// What the binary was asked to do. #[derive(Debug, Clone, PartialEq, Eq)] @@ -1377,7 +1376,6 @@ pub struct Args { /// The boot-describing flags this command line named, in the order given — /// so a refusal can say which ones rather than that there were some. about_a_boot: Vec<&'static str>, - wait_secs: u64, /// Where the stick's two files and this boot's own facts are written, for a /// judge that is not this process. Absent leaves the run's only account its /// standard output, which no per-test predicate can be held to. @@ -1412,7 +1410,6 @@ impl Args { .map(|seen| seen.flag.name) .filter(|name| ABOUT_A_BOOT.iter().any(|flag| flag.name == *name)) .collect(), - wait_secs: return_secs(), readback: value(&READBACK).map(PathBuf::from), fat32_check: METAL.present(args, &FAT32_CHECK), }; @@ -1430,10 +1427,6 @@ impl Args { if let Some(node) = value(&DEVICE) { out.target.node = Node::parse(node)?; } - if let Some(secs) = value(&WAIT_SECS) { - out.wait_secs = - secs.parse().map_err(|_| Refusal::Usage(format!("--wait-secs: {secs:?}")))?; - } // **`--install-sudoers` is an action, not a mode.** It installs the // rule and exits; a command line that also describes a boot is asking // for two things and would get both, which is how a reinstall came to @@ -1559,8 +1552,14 @@ pub fn run(args: &Args) -> Result, Refusal> { ); // The pre-flash gate, before the disk is even asked what it is: what this // image will arm, judged against the only table that has ruled on any of it. - let armed = arms_are_admissible(asked)?; - println!("image {}: armed with {armed:?}", image.path.display()); + let (armed, deadline_ms) = arms_are_admissible(asked)?; + // Off the image and not off this build's constants: the bound is the + // artifact's, and the wait for it is as long as that boot can last. + let wait_secs = return_secs(deadline_ms); + println!( + "image {}: armed with {armed:?}, so it is waited {wait_secs} s to come back", + image.path.display() + ); driver.require_sudo()?; let policy = @@ -1591,7 +1590,7 @@ pub fn run(args: &Args) -> Result, Refusal> { return Ok(None); } - let back = driver.ride_the_reboot(args.wait_secs)?; + let back = driver.ride_the_reboot(wait_secs)?; println!("the machine answered ssh again after {back} s"); // Before the mount, so the stick's own answer is a number rather than // the reason a mount failed. @@ -2048,8 +2047,21 @@ mod tests { assert!(refusal.to_string().contains("stops every CPU on purpose"), "{refusal}"); // And with the bound, both go through — the wedge because it is paired. - assert_eq!(judge_arms(&armed(&[&bound])), Ok(())); - assert_eq!(judge_arms(&armed(&[WEDGE_ARM, &bound])), Ok(())); + // The gate answers with the bound the image carries, which is what the + // loop waits it out by. + assert_eq!(judge_arms(&armed(&[&bound])), Ok(toyos_tco::wedge_bound_ms(toyos_tco::JOB_BOUND_MS))); + assert_eq!(judge_arms(&armed(&[WEDGE_ARM, &bound])), Ok(toyos_tco::wedge_bound_ms(toyos_tco::JOB_BOUND_MS))); + let longer = format!("{}271360", toyos_tco::DEADLINE_PARAM); + assert_eq!(judge_arms(&armed(&["watchdog", &longer])), Ok(271_360)); + // A bound nothing can read is no bound. + for unread in ["soon", "", "0"] { + let unread = format!("{}{unread}", toyos_tco::DEADLINE_PARAM); + assert_eq!( + judge_arms(&armed(&[&unread])), + Err(Refusal::NoBound { staged_a_wedge: false }), + "{unread}" + ); + } // The bound does not excuse an arm nobody ruled on. assert!(matches!( judge_arms(&armed(&["nvme-write-selftest", &bound])), @@ -2071,12 +2083,12 @@ mod tests { // And it is exactly what `tests/common/metal.rs` arms every image with: // the same two constants, so a change to either moves both. assert!(bound.starts_with(toyos_tco::DEADLINE_PARAM)); - assert_eq!(judge_arms(&[bound]), Ok(())); + assert_eq!(judge_arms(&[bound]), Ok(toyos_tco::wedge_bound_ms(toyos_tco::JOB_BOUND_MS))); } /// What every metal image is armed with, spelled the way `tests/common/metal.rs` /// spells it. fn alloc_deadline() -> String { - format!("{}{}", toyos_tco::DEADLINE_PARAM, toyos_tco::WEDGE_BOUND_MS) + format!("{}{}", toyos_tco::DEADLINE_PARAM, toyos_tco::wedge_bound_ms(toyos_tco::JOB_BOUND_MS)) } /// **`--install-sudoers` is an action and not a mode.** It installed the @@ -2096,7 +2108,6 @@ mod tests { vec!["--readback", "target/metal/x"], vec!["--fat32-check"], vec!["--dry-run"], - vec!["--wait-secs", "60"], ] { let mut words = vec!["--install-sudoers".to_string(), "/tmp/pw".to_string()]; words.extend(flag.iter().map(|w| (*w).to_string())); @@ -2155,9 +2166,9 @@ mod tests { let bound = alloc_deadline(); for arm in WEDGE_ARMS { assert!(flashable(arm), "{arm} reaches no stick"); - assert_eq!(judge_arms(&[arm.to_string(), bound.clone()]), Ok(()), "{arm}"); + assert_eq!(judge_arms(&[arm.to_string(), bound.clone()]), Ok(toyos_tco::wedge_bound_ms(toyos_tco::JOB_BOUND_MS)), "{arm}"); assert!(stages_a_wedge(&[arm.to_string()]), "{arm}"); - assert_eq!(bound_for(&[arm]), toyos_tco::STAGED_BOUND_MS, "{arm}"); + assert_eq!(bound_for(&[arm], 135_680), toyos_tco::STAGED_BOUND_MS, "{arm}"); // And with no bound behind it, the sharpest refusal names it as the // wedge it is rather than as a plain image. assert_eq!( @@ -2167,9 +2178,11 @@ mod tests { ); } // The negative half: an arm that stops nothing is not judged as one, - // and keeps the bound a wedge nobody staged is ended by. + // and keeps the bound a wedge nobody staged is ended by, which is + // twice its list's. assert!(!stages_a_wedge(&["watchdog".to_string(), bound])); - assert_eq!(bound_for(&["watchdog"]), toyos_tco::WEDGE_BOUND_MS); + assert_eq!(bound_for(&["watchdog"], 60_000), 120_000); + assert_eq!(bound_for(&["watchdog"], 135_680), 271_360); } /// A row for a name the kernel no longer declares is a ruling about @@ -2555,20 +2568,24 @@ mod tests { assert_eq!(entries_labelled(listing, "Setup"), [("0010".to_string(), String::new())]); } - /// The wait is the *longest* watchdog plus the allowance, computed here by + /// The wait is the *longest* bound plus the allowance, computed here by /// hand from the constants rather than from the expression under test — so - /// a `max` that became a `min` reds instead of agreeing with itself. + /// a `max` that became a `min` reds instead of agreeing with itself. The + /// image's own deadline is one of them: a boot armed past every constant + /// is waited for that long, and one armed under them for the longest + /// constant. #[test] fn the_wait_outlasts_every_watchdog_a_boot_runs_under() { assert_eq!(toyos_tco::FIRMWARE_BOUND_MS, 60_000); assert_eq!(toyos_tco::BOUND_MS, 9_600); - assert_eq!(toyos_tco::JOB_BOUND_MS, 60_000); assert_eq!(toyos_tco::PANIC_BOUND_MS, 60_000); assert_eq!(RETURN_ALLOWANCE_SECS, 300); - assert_eq!(return_secs(), 420); + assert_eq!(return_secs(120_000), 420); + assert_eq!(return_secs(271_360), 572); + assert_eq!(return_secs(271_001), 572); + assert_eq!(return_secs(toyos_tco::STAGED_BOUND_MS), 360); assert!(WATCHDOG_BOUNDS_MS.contains(&toyos_tco::FIRMWARE_BOUND_MS)); assert!(WATCHDOG_BOUNDS_MS.contains(&toyos_tco::BOUND_MS)); - assert!(WATCHDOG_BOUNDS_MS.contains(&toyos_tco::JOB_BOUND_MS)); assert!(WATCHDOG_BOUNDS_MS.contains(&toyos_tco::PANIC_BOUND_MS)); } @@ -2596,7 +2613,7 @@ mod tests { assert!(Refusal::Wedge { why: "x" }.about_the_boot()); assert!(Refusal::Log(bootlog::Unfit::NoBootRecord).about_the_boot()); assert!(Refusal::Log(bootlog::Unfit::Unfinished("x".to_string())).about_the_boot()); - assert!(Refusal::Silent { what: "come back", secs: return_secs() }.about_the_boot()); + assert!(Refusal::Silent { what: "come back", secs: 420 }.about_the_boot()); assert!(!Refusal::Node("/dev/nvme0n1p1".to_string()).about_the_boot()); assert!(!Refusal::Sudo("a password is required".to_string()).about_the_boot()); assert!(!Refusal::Landed { what: "dd".to_string(), want: 1, got: 2 }.about_the_boot()); @@ -2642,7 +2659,6 @@ mod tests { let args = Args::parse(&[]).unwrap(); assert_eq!(args.target.user, "t14"); assert_eq!(args.target.node.whole(), "/dev/sda"); - assert_eq!(args.wait_secs, return_secs()); assert!(!args.dry_run); let words: Vec = ["--dry-run", "--device", "/dev/sdb", "--host", "runner@box"] @@ -2661,10 +2677,6 @@ mod tests { assert_eq!(Args::parse(&part), Err(Refusal::Node("/dev/nvme0n1p2".to_string()))); assert!(matches!(Args::parse(&["--image".to_string()]), Err(Refusal::Usage(_)))); assert!(matches!(Args::parse(&["--flash".to_string()]), Err(Refusal::Usage(_)))); - assert!(matches!( - Args::parse(&["--wait-secs".to_string(), "soon".to_string()]), - Err(Refusal::Usage(_)) - )); } /// A disk carrying `parts` in order, so `admit`'s refusals are exercised on diff --git a/src/metalimage.rs b/src/metalimage.rs index 646a8bf2d79..ddc88933e50 100644 --- a/src/metalimage.rs +++ b/src/metalimage.rs @@ -5,7 +5,8 @@ //! no serial port has nothing at the other end to type a job — so the //! stdin path every `tests/*case` but `jobcase` leaves the runner on parks //! forever, and the loop refuses after `metal::return_secs`. What ends a boot -//! is `[programs.test-runner] args`: one binary name per job, then `reboot`. +//! is `[programs.test-runner] args`: the list's bound, one binary name per +//! job, then `reboot`. //! //! So a metal image is a committed boot config with that one field derived onto //! it. The derivation is here, pure, because what it changes about the config @@ -21,6 +22,9 @@ use toml::Value; /// The runner's manifest key, and the job that hands the machine back. const RUNNER: &str = "test-runner"; const REBOOT: &str = "reboot"; +/// The runner's own word for what its whole list gets, in milliseconds +/// (`userland/test-runner`), and its first argument. +const BOUND: &str = "--bound-ms="; const TOYBOX: &str = "toybox"; const REBOOT_LINK: &str = "bin/reboot"; const TOYBOX_PATH: &str = "/system/bin/toybox"; @@ -53,11 +57,13 @@ impl std::fmt::Display for Underived { /// The committed config with the batch's job list derived onto it. /// -/// `jobs` are the words the runner takes, in order; `reboot` is appended here -/// rather than by every caller, because a list that did not end in one is a -/// boot the loop waits 360 s for and then refuses. +/// `jobs` are the words the runner takes, in order, and `bound_ms` what it +/// gives the whole list; `reboot` is appended here rather than by every +/// caller, because a list that did not end in one is a boot the loop waits +/// out and then refuses. pub fn derive( config: &str, + bound_ms: u64, jobs: &[&str], extra_links: &[(String, String)], ) -> Result { @@ -91,7 +97,8 @@ pub fn derive( .get_mut(RUNNER) .and_then(Value::as_table_mut) .ok_or_else(|| Underived::Toml(format!("`programs.{RUNNER}` is not a table")))?; - let mut args: Vec = jobs.iter().map(|j| Value::String((*j).to_string())).collect(); + let mut args = vec![Value::String(format!("{BOUND}{bound_ms}"))]; + args.extend(jobs.iter().map(|j| Value::String((*j).to_string()))); args.push(Value::String(REBOOT.to_string())); runner.insert("args".to_string(), Value::Array(args)); @@ -138,11 +145,12 @@ mod tests { fn derived(dir: &str, jobs: &[&str]) -> String { let at = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join(dir).join("system.toml"); let text = std::fs::read_to_string(&at).expect("a committed boot config"); - derive(&text, jobs, &[]).unwrap_or_else(|why| panic!("{}: {why}", at.display())) + derive(&text, 133_100, jobs, &[]).unwrap_or_else(|why| panic!("{}: {why}", at.display())) } /// The whole point of the derivation, on every config it is asked of: the - /// runner ends the boot, and it holds the connector the last job needs. + /// runner is told its list's bound before any job, it ends the boot, and + /// it holds the connector the last job needs. #[test] fn every_derived_config_ends_its_own_boot() { for dir in CONFIGS { @@ -151,7 +159,7 @@ mod tests { let runner = &parsed["programs"][RUNNER]; let args: Vec<&str> = runner["args"].as_array().unwrap().iter().map(|v| v.as_str().unwrap()).collect(); - assert_eq!(args, ["test_rs_mkdir_cap", REBOOT], "{dir}"); + assert_eq!(args, ["--bound-ms=133100", "test_rs_mkdir_cap", REBOOT], "{dir}"); let receives: Vec<&str> = runner["receives"] .as_array() .unwrap() @@ -195,13 +203,13 @@ mod tests { /// producing an image that parks the machine for 360 s. #[test] fn a_config_with_no_runner_is_refused_by_name() { - let refusal = derive("[boot]\nstart = [\"logkeeper\"]\n", &[], &[]).unwrap_err(); + let refusal = derive("[boot]\nstart = [\"logkeeper\"]\n", 60_000, &[], &[]).unwrap_err(); assert_eq!(refusal, Underived::NoRunner("does not start `test-runner`".to_string())); let said = refusal.to_string(); assert!(said.contains("parks on a console nothing is on"), "{said}"); let refusal = - derive("[boot]\nstart = [\"test-runner\"]\n[programs.logkeeper]\n", &[], &[]).unwrap_err(); + derive("[boot]\nstart = [\"test-runner\"]\n[programs.logkeeper]\n", 60_000, &[], &[]).unwrap_err(); assert!(matches!(refusal, Underived::NoRunner(_)), "{refusal:?}"); } } diff --git a/src/testargs.rs b/src/testargs.rs index e1903aec0b8..4dab4808f77 100644 --- a/src/testargs.rs +++ b/src/testargs.rs @@ -203,9 +203,9 @@ mod tests { /// A boot word is `--metal`'s alone, and never one of the filters. #[test] fn a_boot_word_names_a_metal_boot_and_nothing_else() { - let line = owned(&["--metal", "boot:shared-2", "control_regs", "boot:ccorpus"]); + let line = owned(&["--metal", "boot:shared-debug", "control_regs", "boot:ccorpus"]); let parsed = parse(&line).unwrap(); - assert_eq!((parsed.filters, parsed.boots), (vec!["control_regs"], vec!["shared-2", "ccorpus"])); + assert_eq!((parsed.filters, parsed.boots), (vec!["control_regs"], vec!["shared-debug", "ccorpus"])); let refusal = parse_owned(&["boot:shared"]).unwrap_err(); assert!(refusal.contains("add --metal"), "{refusal}"); let refusal = metal_owned(&["--metal", "boot:"]).unwrap_err(); diff --git a/tests/checks.rs b/tests/checks.rs index 9156ac12a4e..a1b2ea80492 100644 --- a/tests/checks.rs +++ b/tests/checks.rs @@ -906,6 +906,11 @@ mod checks { metal_checks::a_boots_last_job_is_behind_every_other(); } + #[test] + fn metal_rows_run_before_members_under_a_bound_the_members_widen() { + metal_checks::rows_run_before_members_under_a_bound_the_members_widen(); + } + #[test] fn metal_words_take_rows_members_and_whole_boots() { metal_checks::words_take_rows_members_and_whole_boots(); diff --git a/tests/checks/metal.rs b/tests/checks/metal.rs index 5670ce65074..a3fa5d3b35f 100644 --- a/tests/checks/metal.rs +++ b/tests/checks/metal.rs @@ -391,7 +391,7 @@ pub fn a_failing_shared_member_fails_itself_alone() { config: "tests/testcases", params: &[], features: &[], - members: const { std::num::NonZeroUsize::new(38).expect("a chunk holds a member") }, + member_ms: toyos_tco::RUST_MEMBER_MS, jobs: Vec::from(jobs.map(String::from)), files: Vec::new(), links: Vec::new(), @@ -436,26 +436,84 @@ pub fn a_boots_last_job_is_behind_every_other() { assert!(refused.contains("other ends the boot \"own\" on later and another row ends it on hold"), "{refused}"); } +/// **A boot's rows' jobs run before its members, its last job behind both, +/// and its bounds follow from who rides it**: the runner's bound over the list +/// is the rows' and what each member adds, the kernel's over the boot twice +/// that, and a boot no member rides keeps the bounds a row's boot had. A +/// shared boot that disagrees with a row about the image is refused, and so +/// are two under one name. +pub fn rows_run_before_members_under_a_bound_the_members_widen() { + static EARLY: Metal = + Metal { arms: &[metal::once("shared", "tests/testcases", &[], &["tone", "cost"])], judge: |_| Ok(()) }; + static HELD: Metal = Metal { + arms: &[ + metal::Arm { last: Some("hold"), ..metal::once("shared", "tests/testcases", &[], &["late"]) }, + metal::once("own", "tests/testcases", &[], &["alone"]), + ], + judge: |_| Ok(()), + }; + static WEDGES: Metal = Metal { + arms: &[metal::once("wedge", "tests/jobcase", &[toyos_build::metal::WEDGE_ARM], &[])], + judge: |_| Ok(()), + }; + let members = |boot: &str, member_ms: u64, jobs: &[&str]| metal::SharedBoot { + boot: boot.to_string(), + config: "tests/testcases", + params: &[], + features: &[], + member_ms, + jobs: jobs.iter().map(ToString::to_string).collect(), + files: vec![(format!("expect/{}", jobs[0]), Vec::new())], + links: Vec::new(), + }; + let rows = [("early", &EARLY), ("held", &HELD), ("wedges", &WEDGES)]; + let shared = [members("shared", 860, &["m1", "m2", "m3"]), members("corpus", 260, &["c1", "c2"])]; + let boots = metal::batches(&rows, &shared).expect("one image a boot"); + let bounds = |boot: &str| (boots[boot].bound_ms(), boots[boot].deadline_ms()); + assert_eq!(boots["shared"].jobs, ["tone", "cost", "late", "m1", "m2", "m3", "hold"]); + assert_eq!(bounds("shared"), (62_580, 125_160)); + assert_eq!(boots["corpus"].jobs, ["c1", "c2"]); + assert_eq!(bounds("corpus"), (60_520, 121_040)); + // No member: the bounds every row's boot had before a list could widen. + assert_eq!(bounds("own"), (60_000, 120_000)); + // A staged wedge keeps its short deadline, whatever its list's bound. + assert_eq!(bounds("wedge"), (60_000, toyos_tco::STAGED_BOUND_MS)); + + let debug = metal::SharedBoot { features: toyos_build::build::TEST_KERNEL, ..shared[0].clone() }; + let Err(refused) = metal::batches(&rows, &[debug]) else { + panic!("a shared boot rode a row's boot on another kernel"); + }; + assert!(refused.contains("the shared boot \"shared\" rides the boot \"shared\" as"), "{refused}"); + let Err(refused) = metal::batches(&rows, &[shared[0].clone(), shared[0].clone()]) else { + panic!("two shared boots under one name were batched"); + }; + assert!(refused.contains("two shared boots are both named \"shared\""), "{refused}"); +} + /// **What a run's words take**: no word the whole profile; a name word every -/// row and member it is part of, chunked as they come; a boot word that boot -/// as the whole profile chunks it, with every row that rides it; and a word -/// that takes nothing is refused. +/// row and member it is part of, and of a shared boot only the files its +/// members name; a boot word that boot whole, with every row that rides it, +/// whatever name words stand beside it; and a word that takes nothing is +/// refused. pub fn words_take_rows_members_and_whole_boots() { static ROWS: [(&str, Metal); 2] = [ ("alpha_row", Metal { arms: &[metal::once("own", "tests/testcases", &[], &[])], judge: |_| Ok(()) }), - ("beta_row", Metal { arms: &[metal::once("shared-2", "tests/testcases", &[], &[])], judge: |_| Ok(()) }), + ("beta_row", Metal { arms: &[metal::once("shared", "tests/testcases", &[], &[])], judge: |_| Ok(()) }), ]; let boot = |boot: &str, jobs: &[&str]| metal::SharedBoot { boot: boot.to_string(), config: "tests/testcases", params: &[], features: &[], - members: const { std::num::NonZeroUsize::new(2).expect("a chunk holds a member") }, + member_ms: toyos_tco::RUST_MEMBER_MS, jobs: jobs.iter().map(ToString::to_string).collect(), - files: Vec::new(), - links: Vec::new(), + files: jobs + .iter() + .flat_map(|job| [(format!("expect/{job}"), Vec::new()), (format!("bin/test_c_{job}"), Vec::new())]) + .collect(), + links: jobs.iter().map(|job| (format!("bin/{job}"), "/system/bin/test_rs_ccheck".to_string())).collect(), }; - let profile = [boot("shared", &["test_rs_a1", "test_rs_a2", "test_rs_b1"]), boot("ccorpus", &["c1"])]; + let profile = [boot("shared", &["test_rs_a1", "test_rs_a2", "test_rs_b1"]), boot("ccorpus", &["c1", "c2"])]; let taken = |names: &[&str], boots: &[&str]| { metal::select(names, boots, &ROWS, &profile).map(|(rows, shared)| { let rows: Vec<&str> = rows.iter().map(|(name, _)| *name).collect(); @@ -467,18 +525,24 @@ pub fn words_take_rows_members_and_whole_boots() { let took = |rows: &str, shared: &str| Ok::<_, String>((rows.to_string(), shared.to_string())); assert_eq!( taken(&[], &[]), - took("alpha_row,beta_row", "shared=test_rs_a1+test_rs_a2,shared-2=test_rs_b1,ccorpus=c1") + took("alpha_row,beta_row", "shared=test_rs_a1+test_rs_a2+test_rs_b1,ccorpus=c1+c2") ); assert_eq!(taken(&["b1"], &[]), took("", "shared=test_rs_b1")); assert_eq!(taken(&["alpha", "c1"], &[]), took("alpha_row", "ccorpus=c1")); - assert_eq!(taken(&[], &["shared"]), took("", "shared=test_rs_a1+test_rs_a2")); - assert_eq!(taken(&[], &["shared-2"]), took("beta_row", "shared-2=test_rs_b1")); - assert_eq!(taken(&["alpha"], &["ccorpus", "shared-2"]), took("alpha_row,beta_row", "shared-2=test_rs_b1,ccorpus=c1")); + assert_eq!(taken(&[], &["shared"]), took("beta_row", "shared=test_rs_a1+test_rs_a2+test_rs_b1")); + // One boot, once and whole, where a name word takes a member of it too. + assert_eq!(taken(&["a2"], &["shared"]), took("beta_row", "shared=test_rs_a1+test_rs_a2+test_rs_b1")); + assert_eq!(taken(&["alpha"], &["ccorpus"]), took("alpha_row", "ccorpus=c1+c2")); assert_eq!(taken(&[], &["own"]), took("alpha_row", "")); + // A member taken by name brings its own files and links and no other's. + let (_, shared) = metal::select(&["c2"], &[], &ROWS, &profile).expect("a member's name"); + let paths: Vec<&str> = shared[0].files.iter().map(|(path, _)| path.as_str()).collect(); + assert_eq!(paths, ["expect/c2", "bin/test_c_c2"]); + assert_eq!(shared[0].links, [("bin/c2".to_string(), "/system/bin/test_rs_ccheck".to_string())]); let refused: [(&[&str], &[&str], &str); 3] = [ (&["a1", "nope"], &[], "\"nope\" is part of no"), (&["rs_a1"], &[], "\"rs_a1\" is part of no"), - (&[], &["shared", "shared-3"], "boot:shared-3 names no boot"), + (&[], &["shared", "shared-2"], "boot:shared-2 names no boot"), ]; for (names, boots, refusal) in refused { let said = taken(names, boots).expect_err("a word that takes nothing was accepted"); diff --git a/tests/common/metal.rs b/tests/common/metal.rs index 0c440f498d9..4fab02f6f74 100644 --- a/tests/common/metal.rs +++ b/tests/common/metal.rs @@ -17,7 +17,6 @@ use std::cell::RefCell; use std::collections::btree_map::Entry; use std::collections::{BTreeMap, BTreeSet}; use std::io::{BufRead, BufReader}; -use std::num::NonZeroUsize; use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; @@ -58,8 +57,8 @@ pub struct Arm { /// owner flashes. pub features: &'static [&'static str], /// The job this boot ends on, before `reboot`: [`batches`] puts it after - /// every job any arm on the boot names, so none can land behind it, and - /// refuses two arms that name different ones. + /// every job any arm on the boot names and every member riding it, so none + /// can land behind it, and refuses two arms that name different ones. pub last: Option<&'static str>, } @@ -86,17 +85,21 @@ pub const fn once( /// exit record for it. So they are a boot with a list rather than a row each, /// and each member is still reported under its own name. /// -/// A chunk rides one flash. A member that takes the machine down takes every -/// member after it *in its chunk* with it, and that is the honest price: on the -/// T14 there is no `MAX_SHARED_REBOOTS` to answer a dead guest with a new one. +/// The members ride one flash. A member that takes the machine down takes +/// every member after it with it, and that is the honest price: on the T14 +/// there is no `MAX_SHARED_REBOOTS` to answer a dead guest with a new one. #[derive(Clone)] pub struct SharedBoot { + /// The boot the members ride. A row whose arm names it rides it too, on + /// the same terms two arms share one, and its jobs run before any member. pub boot: String, pub config: &'static str, pub params: &'static [&'static str], /// The kernel build, empty for the one an image ships. pub features: &'static [&'static str], - pub members: NonZeroUsize, + /// What each member adds to the bound the runner gives the boot's whole + /// list (`toyos_tco::list_bound_ms`), in milliseconds. + pub member_ms: u64, /// What the runner spawns, in order — the whole binary name, `test_rs_` /// prefix and all, because that is what the kernel records it under. pub jobs: Vec, @@ -109,63 +112,36 @@ pub struct SharedBoot { pub links: Vec<(String, String)>, } -/// How many members allowed `allowance_ms` each one shared boot holds: the -/// runner's [`toyos_tco::JOB_BOUND_MS`] less a tenth of it, shared among them. -pub const fn members_fitting(allowance_ms: u64) -> NonZeroUsize { - let members = (toyos_tco::JOB_BOUND_MS - toyos_tco::JOB_BOUND_MS / 10) / allowance_ms; - NonZeroUsize::new(members as usize).expect("a chunk holds a member") -} - -/// The name of one chunk of a boot that had to be cut in two. -fn chunk_name(boot: &str, index: usize) -> String { - if index == 0 { - boot.to_string() - } else { - format!("{boot}-{}", index + 1) +impl SharedBoot { + /// What the members add to their boot's list bound. + fn members_ms(&self) -> u64 { + self.jobs.len() as u64 * self.member_ms } -} -/// **A chunk carries only the files and links its own members name.** The C -/// corpus stages a binary and an expectation per case; putting all of both on -/// every chunk would double a flash that is already written over `ssh`. -fn sized(shared: &[SharedBoot]) -> Vec { - let mut out = Vec::new(); - for boot in shared { - for (index, jobs) in boot.jobs.chunks(boot.members.get()).enumerate() { - let named: BTreeSet<&str> = jobs.iter().map(String::as_str).collect(); - let mine = |path: &str| { - let last = path.rsplit('/').next().unwrap_or(path); - named.contains(last) - || last.strip_prefix("test_c_").is_some_and(|c| named.contains(c)) - }; - out.push(SharedBoot { - boot: chunk_name(&boot.boot, index), - config: boot.config, - params: boot.params, - features: boot.features, - members: boot.members, - jobs: jobs.to_vec(), - files: boot - .files - .iter() - .filter(|(path, _)| mine(path)) - .cloned() - .collect(), - links: boot.links.iter().filter(|(from, _)| mine(from)).cloned().collect(), - }); - } + /// This boot with the members `named` and none else, or `None` where that + /// is none of them. **It carries only the files and links its own members + /// name**: the C corpus stages a binary and an expectation per case, and a + /// stick is written over `ssh`. + fn keeping(&self, named: impl Fn(&str) -> bool) -> Option { + let jobs: Vec = self.jobs.iter().filter(|job| named(job)).cloned().collect(); + let mine = |path: &str| { + let last = path.rsplit('/').next().unwrap_or(path); + let case = last.strip_prefix("test_c_").unwrap_or(last); + jobs.iter().any(|job| job == case) + }; + let files = self.files.iter().filter(|(path, _)| mine(path)).cloned().collect(); + let links = self.links.iter().filter(|(from, _)| mine(from)).cloned().collect(); + (!jobs.is_empty()).then(|| SharedBoot { jobs, files, links, ..self.clone() }) } - out } /// The rows and the shared boots a run takes. pub type Taken = (Vec<(&'static str, &'static Metal)>, Vec); /// What a run's words take of the profile: every row and shared member a -/// `names` word is part of, chunked as they come, and every boot a `boots` -/// word names, whole — its shared members as the whole profile chunks them, -/// and every row with an arm on it, which brings that row's other boots. No -/// word at all takes everything. +/// `names` word is part of, and every boot a `boots` word names, whole — all +/// its shared members, and every row with an arm on it, which brings that +/// row's other boots. No word at all takes everything. /// /// **A word that takes nothing is refused**: it would be dropped in silence. pub fn select( @@ -185,25 +161,17 @@ pub fn select( }) { return Err(format!("{dead:?} is part of no metal registration's name and no shared member's")); } - let whole = sized(shared); let arms = || rows.iter().flat_map(|(_, decl)| decl.arms).map(|arm| arm.boot); - let known: BTreeSet<&str> = whole.iter().map(|boot| boot.boot.as_str()).chain(arms()).collect(); + let known: BTreeSet<&str> = shared.iter().map(|boot| boot.boot.as_str()).chain(arms()).collect(); if let Some(dead) = boots.iter().find(|boot| !known.contains(**boot)) { return Err(format!("boot:{dead} names no boot of the metal profile, whose boots are {known:?}")); } let named = |name: &str| all || names.iter().any(|word| name.contains(word)); - let mut taken: Vec = - whole.iter().filter(|boot| boots.contains(&boot.boot.as_str())).cloned().collect(); - let by_name: Vec = shared + let taken: Vec = shared .iter() - .cloned() - .map(|mut boot| { - boot.jobs.retain(|job| named(bare(job))); - boot - }) + .filter_map(|boot| boot.keeping(|job| boots.contains(&boot.boot.as_str()) || named(bare(job)))) .collect(); - taken.extend(sized(&by_name)); let rows = rows .iter() .filter(|(name, decl)| named(name) || decl.arms.iter().any(|arm| boots.contains(&arm.boot))) @@ -542,6 +510,8 @@ pub struct Batch { features: &'static [&'static str], /// What the runner spawns, in order. pub jobs: Vec, + /// What the members riding it add to the bound of its list. + members_ms: u64, last: Option<&'static str>, files: Vec<(String, Vec)>, links: Vec<(String, String)>, @@ -555,6 +525,46 @@ impl Batch { } } } + + /// The bound the runner gives this boot's whole list, in milliseconds. + pub fn bound_ms(&self) -> u64 { + toyos_tco::list_bound_ms(self.members_ms) + } + + /// The bound the kernel gives this whole boot, in milliseconds. + pub fn deadline_ms(&self) -> u64 { + toyos_build::metal::bound_for(&self.params, self.bound_ms()) + } +} + +/// The batch of the boot `boot`, which `who` rides as `(config, params, +/// features)`: a new one, or the one an earlier rider made where the two agree. +fn ride<'a>( + out: &'a mut BTreeMap, + who: &str, + boot: &str, + config: &'static str, + params: &'static [&'static str], + features: &'static [&'static str], +) -> Result<&'a mut Batch, String> { + let batch = out.entry(boot.to_string()).or_insert_with(|| Batch { + config, + params: params.to_vec(), + features, + jobs: Vec::new(), + members_ms: 0, + last: None, + files: Vec::new(), + links: Vec::new(), + }); + if batch.config != config || batch.params != params || batch.features != features { + return Err(format!( + "{who} rides the boot {boot:?} as ({config}, {params:?}, {features:?}) and another rides \ + it as ({}, {:?}, {:?}); one boot is one image", + batch.config, batch.params, batch.features + )); + } + Ok(batch) } /// Where a batch's derived config, its image and its readback live. @@ -562,63 +572,24 @@ pub fn at(dir: &Path, label: &str) -> PathBuf { dir.join(label) } -/// The boots `tests` need, keyed by the name their arms give them. +/// The boots `tests` and `shared` need, keyed by the name their arms give them. /// -/// **Two arms naming one boot are refused where they disagree about it**: an +/// **Two riders naming one boot are refused where they disagree about it**: an /// image is one config armed one way, and a silent winner would give one of the -/// two tests a machine it did not ask for. +/// two a machine it did not ask for. +/// +/// **A boot's rows' jobs run before its members**, so a row that measures has +/// the machine its jobs alone would give it, and what the members write to the +/// log lands behind every row's. pub fn batches( tests: &[(&str, &'static Metal)], shared: &[SharedBoot], ) -> Result, String> { let mut out: BTreeMap = BTreeMap::new(); - // First, so a registration naming a shared boot rides it rather than - // minting a second one under the same name with a different list. - for boot in shared { - let was = out.insert( - boot.boot.clone(), - Batch { - config: boot.config, - params: boot.params.to_vec(), - features: boot.features, - jobs: boot.jobs.clone(), - last: None, - files: boot.files.clone(), - links: boot.links.clone(), - }, - ); - if was.is_some() { - return Err(format!("two shared boots are both named {:?}", boot.boot)); - } - } for (name, decl) in tests { let Metal { arms, .. } = decl; for arm in *arms { - let batch = out.entry(arm.boot.to_string()).or_insert_with(|| Batch { - config: arm.config, - params: arm.params.to_vec(), - features: arm.features, - jobs: Vec::new(), - last: None, - files: Vec::new(), - links: Vec::new(), - }); - if batch.config != arm.config - || batch.params != arm.params - || batch.features != arm.features - { - return Err(format!( - "{name} rides the boot {:?} as ({}, {:?}, {:?}) and another row rides it as \ - ({}, {:?}, {:?}); one boot is one image", - arm.boot, - arm.config, - arm.params, - arm.features, - batch.config, - batch.params, - batch.features - )); - } + let batch = ride(&mut out, name, arm.boot, arm.config, arm.params, arm.features)?; batch.add(arm.jobs.iter().map(|j| (*j).to_string())); match (batch.last, arm.last) { (Some(was), Some(now)) if was != now => { @@ -633,6 +604,18 @@ pub fn batches( } } } + let mut ridden: BTreeSet<&str> = BTreeSet::new(); + for boot in shared { + if !ridden.insert(&boot.boot) { + return Err(format!("two shared boots are both named {:?}", boot.boot)); + } + let who = format!("the shared boot {:?}", boot.boot); + let batch = ride(&mut out, &who, &boot.boot, boot.config, boot.params, boot.features)?; + batch.add(boot.jobs.iter().cloned()); + batch.members_ms += boot.members_ms(); + batch.files.extend(boot.files.iter().cloned()); + batch.links.extend(boot.links.iter().cloned()); + } for batch in out.values_mut() { if let Some(last) = batch.last { batch.jobs.retain(|job| job != last); @@ -687,7 +670,7 @@ fn build( let text = std::fs::read_to_string(&committed) .map_err(|e| format!("{}: {e}", committed.display()))?; let jobs: Vec<&str> = batch.jobs.iter().map(String::as_str).collect(); - let derived = metalimage::derive(&text, &jobs, &batch.links) + let derived = metalimage::derive(&text, batch.bound_ms(), &jobs, &batch.links) .map_err(|why| format!("{}: {why}", committed.display()))?; // **The job list is in the file name, not only in the file.** // `build_test_image` memoizes ROOT on the config's *path*, so two runs whose @@ -756,7 +739,9 @@ fn build( // chipset watchdog does not count on this PCH, the runner's own bound // reboots through the shutdown syscall the wedge may be inside, and the // loop then waits `metal::return_secs` and needs a hand on the power - // button. Armed after the kernel build is decided above, because a + // button. Derived from the bound the runner was just given, so the kernel + // never ends a list its runner would still have ended itself. + // Armed after the kernel build is decided above, because a // parameter carrying a value is not an actuator and must not pull the test // kernel in behind it. if let Some(own) = batch.params.iter().find(|p| p.starts_with(toyos_tco::DEADLINE_PARAM)) @@ -769,11 +754,7 @@ fn build( field on an arm" )); } - let deadline = format!( - "{}{}", - toyos_tco::DEADLINE_PARAM, - toyos_build::metal::bound_for(&batch.params) - ); + let deadline = format!("{}{}", toyos_tco::DEADLINE_PARAM, batch.deadline_ms()); let mut params: Vec<&str> = batch.params.clone(); params.push(&deadline); let plan = toyos_build::build::Plan::new(toyos_build::arch::Arch::X86_64, &config, features, ¶ms); @@ -919,7 +900,13 @@ pub fn run( let (dir, offline): (PathBuf, bool) = match &mode { MetalMode::List => { for (label, batch) in &batches { - println!("{label}: {} job(s) — {:?}", batch.jobs.len(), batch.jobs); + println!( + "{label}: {} job(s), a list bound of {} ms and a boot deadline of {} ms — {:?}", + batch.jobs.len(), + batch.bound_ms(), + batch.deadline_ms(), + batch.jobs + ); } return Verdict::Green; } @@ -941,9 +928,12 @@ pub fn run( match build(&root, dir, label, batch, rust_bins, quiet) { Ok(image) => { eprintln!( - "[metal] {label}: {} job(s), armed with {:?} — {}", + "[metal] {label}: {} job(s) under a list bound of {} ms, armed with {:?} and \ + a boot deadline of {} ms — {}", batch.jobs.len(), + batch.bound_ms(), batch.params, + batch.deadline_ms(), image.display() ); images.insert(label.as_str(), image); @@ -1176,6 +1166,15 @@ pub fn judge_readbacks( if let (Some(complete), Some(last)) = (back.complete_record_ms(), back.last_record_ms()) { let each = last.saturating_sub(complete) / ran as u64; eprintln!(" {} ms per member over the {ran} that ran", each); + // On the clock the runner counts its bound on, the kernel's + // own, whose reading `Boot: complete` states beside its stamp. + if let Some(boot_ms) = back.boot_ms { + eprintln!( + " its last record came {} ms into a list bound of {} ms", + (last + boot_ms).saturating_sub(complete), + toyos_tco::list_bound_ms(boot.members_ms()) + ); + } } } } diff --git a/tests/metal/lenovo-20w0003amz.toml b/tests/metal/lenovo-20w0003amz.toml index c33ec82e17b..fc3981df3d9 100644 --- a/tests/metal/lenovo-20w0003amz.toml +++ b/tests/metal/lenovo-20w0003amz.toml @@ -37,18 +37,12 @@ bios = "N34ET71W (1.71 )" "boot.proctreecase.complete_ms" = 1153 "boot.proctreecase.panel_max_us" = 3773 "boot.proctreecase.panel_us" = 21279 -"boot.shared-2.complete_ms" = 1165 -"boot.shared-2.panel_max_us" = 3721 -"boot.shared-2.panel_us" = 21855 "boot.shared-debug.complete_ms" = 1165 "boot.shared-debug.panel_max_us" = 3791 "boot.shared-debug.panel_us" = 21483 "boot.shared.complete_ms" = 1154 "boot.shared.panel_max_us" = 3608 "boot.shared.panel_us" = 21429 -"boot.testcases-bounds.complete_ms" = 1137 -"boot.testcases-bounds.panel_max_us" = 2493 -"boot.testcases-bounds.panel_us" = 12748 "boot.testcases-deaf.complete_ms" = 1165 "boot.testcases-deaf.panel_max_us" = 5247 "boot.testcases-deaf.panel_us" = 26659 diff --git a/tests/toyos.rs b/tests/toyos.rs index e67da326128..192f13fcba4 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -143,12 +143,6 @@ const RUST_SKIP: &[&str] = &[ "readdir_bound", // Fills the VFS `created_dirs` cap and leaves it there. `mkdir_cap` runs it. "mkdir_cap", - // Each fills a bound of its own process — tens of thousands of mappings, - // thousands of threads, a thousand 2 MiB images — which no shared member's - // allowance is sized for: the `process_bound_*` metal rows run them. - "abuse_mmap_regions", - "abuse_thread_table", - "abuse_dlopen_ledger", // Audio is judged on the T14 and nowhere else: the `hda_client_stall`, // `hda_tone`, `audio_idle_suspend`, `shipped_client_departures` and // `soundserver_log_stall` metal rows run these. @@ -177,6 +171,14 @@ const RUST_SKIP: &[&str] = &[ "bar_map_again", ]; +/// The shared boot's last members, in this order: each fills a bound of its +/// own process — tens of thousands of mappings, thousands of threads, a +/// thousand 2 MiB images — and its exit gives all of it back. Behind every +/// discovered member, so none of those runs after one of these. **This is +/// their one declaration**: [`discover_rust_tests`] leaves them to it, so a +/// name taken off this list is a discovered member again. +const LAST_MEMBERS: &[&str] = &["abuse_mmap_regions", "abuse_thread_table", "abuse_dlopen_ledger"]; + /// Binaries a metal row or a guest test drives that the shared boot also runs /// on purpose. /// @@ -367,18 +369,6 @@ const METAL: &[(&str, metal::Metal)] = &[ judge: |b| b[0].job_passed("test_rs_readdir_bound"), }, ), - ( - "process_bound_regions", - metal::Metal { arms: BOUNDS, judge: |b| b[0].job_passed("test_rs_abuse_mmap_regions") }, - ), - ( - "process_bound_threads", - metal::Metal { arms: BOUNDS, judge: |b| b[0].job_passed("test_rs_abuse_thread_table") }, - ), - ( - "process_bound_libraries", - metal::Metal { arms: BOUNDS, judge: |b| b[0].job_passed("test_rs_abuse_dlopen_ledger") }, - ), ( "wake_storm_cost", metal::Metal { @@ -942,15 +932,6 @@ const TESTCASES_MKDIR: &[metal::Arm] = const TESTCASES_READDIR: &[metal::Arm] = &[metal::once("testcases-readdir", "tests/testcases", &[], &["test_rs_readdir_bound"])]; -/// One boot for the three, which can share it: each fills a bound of its own -/// process, and its exit gives all of it back. -const BOUNDS: &[metal::Arm] = &[metal::once( - "testcases-bounds", - "tests/testcases", - &[], - &["test_rs_abuse_mmap_regions", "test_rs_abuse_thread_table", "test_rs_abuse_dlopen_ledger"], -)]; - /// The shipping kernel with the windows' instrument and nothing else, so what /// it reads is that kernel under the herd. Three exits, a report each: /// `idle_span`'s is the boot's first, where the kernel holds; `pwd`'s reads the @@ -1036,19 +1017,20 @@ const SELFTESTS: &[metal::Arm] = &[metal::Arm { }]; /// The boots every discovered Rust binary rides on the T14: the shipping -/// kernel's, and [`ACTUATOR_TESTS`] on the kernel that carries `SYS_DEBUG`, -/// armed with [`SELFTEST_PARAMS`]. +/// kernel's, with [`LAST_MEMBERS`] behind them, and [`ACTUATOR_TESTS`] on the +/// kernel that carries `SYS_DEBUG`, armed with [`SELFTEST_PARAMS`]. fn shared_metal() -> Vec { - let (debug, shipping): (Vec, Vec) = discover_rust_tests() + let (debug, mut shipping): (Vec, Vec) = discover_rust_tests() .into_iter() .partition(|name| ACTUATOR_TESTS.contains(&name.as_str())); + shipping.extend(LAST_MEMBERS.iter().map(ToString::to_string)); vec![ metal::SharedBoot { boot: "shared".to_string(), config: "tests/testcases", params: &[], features: &[], - members: const { metal::members_fitting(toyos_tco::RUST_MEMBER_MS) }, + member_ms: toyos_tco::RUST_MEMBER_MS, jobs: shipping.iter().map(|n| format!("test_rs_{n}")).collect(), files: Vec::new(), links: Vec::new(), @@ -1063,7 +1045,7 @@ fn shared_metal() -> Vec { config: "tests/testcases", params: SELFTEST_PARAMS, features: ACTUATOR_KERNEL, - members: const { std::num::NonZeroUsize::new(18).expect("a chunk holds a member") }, + member_ms: toyos_tco::RUST_MEMBER_MS, jobs: debug.iter().map(|n| format!("test_rs_{n}")).collect(), files: Vec::new(), links: Vec::new(), @@ -1128,7 +1110,7 @@ fn c_corpus_metal(c_bins: &[(String, Vec)]) -> metal::SharedBoot { config: "tests/testcases", params: &[], features: &[], - members: const { metal::members_fitting(toyos_tco::C_MEMBER_MS) }, + member_ms: toyos_tco::C_MEMBER_MS, jobs, files, links, @@ -1357,7 +1339,8 @@ fn discover_c_tests() -> Vec { names } -/// Discover the Rust test binaries: every one but the [`RUST_SKIP`] helpers. +/// Discover the Rust test binaries: every one but the [`RUST_SKIP`] helpers +/// and the [`LAST_MEMBERS`], which [`shared_metal`] places itself. /// /// **A name that arrives this way is registered by nothing but its file.** /// `tests/toyos-rust-tests/src/bin/.rs` is the whole declaration — no row @@ -1369,7 +1352,8 @@ fn discover_rust_tests() -> Vec { .filter_map(|e| { let path = e.ok()?.path(); let name = path.file_stem()?.to_str()?.to_string(); - (path.extension()? == "rs" && !RUST_SKIP.contains(&name.as_str())).then_some(name) + let placed = RUST_SKIP.contains(&name.as_str()) || LAST_MEMBERS.contains(&name.as_str()); + (path.extension()? == "rs" && !placed).then_some(name) }) .collect(); names.sort(); @@ -5137,10 +5121,11 @@ fn declared<'a>() -> impl Iterator { MACHINE_TESTS.iter().copied().chain(SCREEN_TESTS.iter().map(|(n, _)| *n)) } -/// What the shared boots answer for: every discovered Rust binary and every -/// corpus case. +/// What the shared boots answer for: every discovered Rust binary, the T14's +/// [`LAST_MEMBERS`] and every corpus case. fn shared_names() -> Vec { - discover_rust_tests().into_iter().chain(discover_c_tests()).collect() + let metal = LAST_MEMBERS.iter().map(ToString::to_string); + discover_rust_tests().into_iter().chain(metal).chain(discover_c_tests()).collect() } /// Every name this suite can produce a verdict for, on either machine: the diff --git a/toyos-tco/src/lib.rs b/toyos-tco/src/lib.rs index 587abe7d192..d0b6e9c0ab8 100644 --- a/toyos-tco/src/lib.rs +++ b/toyos-tco/src/lib.rs @@ -114,10 +114,17 @@ pub const FIRMWARE_BOUND_MS: u64 = 60_000; /// while a job never finishes is no wedge to it and nothing else ends the boot. pub const JOB_BOUND_MS: u64 = 60_000; -/// What one member of a metal shared boot is allowed of [`JOB_BOUND_MS`], in -/// milliseconds: a Rust test on the shipping kernel, and a C corpus case. -pub const RUST_MEMBER_MS: u64 = 860; -pub const C_MEMBER_MS: u64 = 260; +/// What one member of a metal shared boot adds to its list's bound, in +/// milliseconds: a Rust test, and a C corpus case. A share of the list's and +/// no bound of the member's own: nothing ends one member. +pub const RUST_MEMBER_MS: u64 = 300; +pub const C_MEMBER_MS: u64 = 100; + +/// The bound the runner gives a metal boot's whole list, in milliseconds: +/// [`JOB_BOUND_MS`] for the jobs its rows name, and what its members add. +pub const fn list_bound_ms(members_ms: u64) -> u64 { + JOB_BOUND_MS + members_ms +} /// The bound netstack gives this machine's first DHCP lease before it says it has /// none and serves anyway, in milliseconds. The harness waits it out on a wire @@ -129,33 +136,36 @@ pub const LEASE_BOUND_MS: u64 = 20_000; /// bound for every watchdog. pub const PANIC_BOUND_MS: u64 = 60_000; -/// The bound the *kernel* gives a whole boot, in milliseconds, before it seals -/// a record and writes the reset register itself. +/// The bound the *kernel* gives a whole boot whose runner bounds its list at +/// `list_ms`, in milliseconds, before it seals a record and writes the reset +/// register itself. /// -/// **Twice [`JOB_BOUND_MS`], and derived from it rather than chosen.** A job the +/// **Twice the list's, and derived from it rather than chosen.** A job the /// runner is still inside is not a wedge: the runner's own bound ends it and the /// boot goes on, so a kernel bound at or under that number resets a machine /// whose test was about to be reported. Twice it is wider than the runner's /// bound plus the boot around it — the slowest healthy T14 boot measured is /// 6.242 s of kernel time — and narrower than a wait for a hand. -pub const WEDGE_BOUND_MS: u64 = JOB_BOUND_MS * 2; +pub const fn wedge_bound_ms(list_ms: u64) -> u64 { + list_ms * 2 +} /// The kernel's bound has to outlast the one that ends a single job, or a slow /// test is a reset where it should have been a verdict. -const _: () = assert!(WEDGE_BOUND_MS > JOB_BOUND_MS); +const _: () = assert!(wedge_bound_ms(JOB_BOUND_MS) > JOB_BOUND_MS); /// The bound a boot that stages its own wedge carries instead of -/// [`WEDGE_BOUND_MS`], in milliseconds: it ends the machine seconds after the -/// wedge rather than at the bound every other boot keeps for a wedge nobody -/// staged. Its half is the lockup detector's bound, which still outlasts the -/// lockup probe's own reach to its lock. +/// [`wedge_bound_ms`] of its list, in milliseconds: it ends the machine seconds +/// after the wedge rather than at the bound every other boot keeps for a wedge +/// nobody staged. Its half is the lockup detector's bound, which still outlasts +/// the lockup probe's own reach to its lock. pub const STAGED_BOUND_MS: u64 = 10_000; /// The bound one *CPU* gets to take no interrupt at all while it is burning /// cycles, in milliseconds, before that CPU ends the machine from its own NMI. /// /// **Half the bound this boot gave itself, and derived rather than chosen.** -/// [`WEDGE_BOUND_MS`] is twice [`JOB_BOUND_MS`] because a job the runner is +/// [`wedge_bound_ms`] is twice the list's bound because a job the runner is /// still inside is not a wedge; that argument does not apply here, since a CPU /// that takes no interrupt runs no job — no scheduler pass reaches it. So the /// stronger evidence gets the earlier bound: a machine that has hard-locked one @@ -172,11 +182,11 @@ pub const fn hard_lockup_bound_ms(deadline_ms: u64) -> u64 { /// [`hard_lockup_bound_ms`] of the bound a metal image carries, which is the one /// a T14 boot runs under. -pub const HARD_LOCKUP_BOUND_MS: u64 = hard_lockup_bound_ms(WEDGE_BOUND_MS); +pub const HARD_LOCKUP_BOUND_MS: u64 = hard_lockup_bound_ms(wedge_bound_ms(JOB_BOUND_MS)); /// The detector has to fire before the deadline it is derived from, or a /// hard-locked machine is reported as an ordinary wedge and the CPU is unnamed. -const _: () = assert!(HARD_LOCKUP_BOUND_MS < WEDGE_BOUND_MS); +const _: () = assert!(HARD_LOCKUP_BOUND_MS < wedge_bound_ms(JOB_BOUND_MS)); /// How often an armed CPU samples itself against [`hard_lockup_bound_ms`], in /// nanoseconds of unhalted time, and so how late past that bound a stuck CPU @@ -187,11 +197,11 @@ const _: () = assert!(HARD_LOCKUP_BOUND_MS < WEDGE_BOUND_MS); /// period of the bound. It is also the period the report's ages are quoted at. pub const HARD_LOCKUP_SAMPLE_NS: u64 = 1_000_000_000; -/// The boot parameter that arms [`WEDGE_BOUND_MS`], with the bound in -/// milliseconds after it. A value and not a flag, because the bound is the one -/// thing about this mechanism a boot can legitimately differ on — a negative -/// control wants a short one, and a boot the loop waits longer for wants its -/// own. +/// The boot parameter that arms the kernel's bound ([`wedge_bound_ms`]), with +/// the bound in milliseconds after it. A value and not a flag, because the +/// bound is the one thing about this mechanism a boot can legitimately differ +/// on — a negative control wants a short one, and a boot the loop waits longer +/// for wants its own. pub const DEADLINE_PARAM: &str = "boot-deadline="; /// The bound `cmdline` arms, or `None` where it names none. @@ -323,7 +333,7 @@ mod tests { /// arm two numbers that could disagree. #[test] fn the_lockup_bound_follows_the_deadline_the_boot_named() { - assert_eq!(hard_lockup_bound_ms(WEDGE_BOUND_MS), 60_000); + assert_eq!(hard_lockup_bound_ms(wedge_bound_ms(JOB_BOUND_MS)), 60_000); assert_eq!(hard_lockup_bound_ms(20_000), 10_000); // A bound of one millisecond leaves none: the detector arms nothing // rather than firing on the first sample, which the kernel checks.