diff --git a/docs/architecture/04-execution.md b/docs/architecture/04-execution.md index 2078da0..90ba0c0 100644 --- a/docs/architecture/04-execution.md +++ b/docs/architecture/04-execution.md @@ -165,6 +165,8 @@ Timeouts and CPU-time limits run the same sequence with the reason `TIMED_OUT` / - `termination_reason`: `EXITED`, `CANCELLED`, `TIMED_OUT`, `CPU_TIME_EXCEEDED`, `OOM_KILLED` (Linux: `memory.events` oom_kill increased, or `SERVICE_RESULT=oom-kill`), `MEMORY_LIMIT` (Windows commit-limit notification observed and exit ≠ 0), `PREEMPTED`, `LOST`, `SPAWN_FAILED`, `NODE_SHUTDOWN`, `POLICY_DENIED`, `TERMINATION_INCOMPLETE`, `CACHE_RESTORED` (result-cache restore; never an execution), `OUTPUT_LIMIT` (§4), `DISK_LIMIT` (per-attempt temporary space, or critical free space on a written volume; [06](06-storage-workspaces.md) §5) or `ENDPOINT_TAKEN` (another process listened on the attempt's own sccache endpoint, or took the cache store's port; [05](05-toolchains-caches.md) §5); - `effect`: `DID_NOT_HAPPEN` only for `SPAWN_FAILED` and `POLICY_DENIED`; `HAPPENED` for completed processes; `UNKNOWN` for `LOST`. +**Windows node-gap memory evidence (MR-17, 2026-10-26).** A holder records `job_memory` when the attempt job's peak committed memory exceeds its enabled memory-notification threshold, queried from that job when writing `exit.json`. It uses the current notification threshold, not a startup-spec copy or the higher hard commit cap. This preserves the crossing after the processes release memory and a notification/accounting query leaves the final live violation query empty while the node is down; an unread notification may otherwise still be latched. The durable peak comparison does not depend on which consumer queried first. A restarted node applies the existing nonzero-exit condition to that durable evidence; an exit of zero remains `EXITED`. This is notification-threshold evidence, not proof that an allocation was refused by a hard cap. Limits changed by same-user code retain the guardrail boundary of AD-0005. + When an attempt's result records a stop during a running launch step (§3), its `effect` is `HAPPENED`, including when the command never started. Launch steps are part of the task's execution. This does not assert that the command ran or that effects of the step were rolled back. ## 8. Dependencies, services, retries diff --git a/docs/decisions/AD-0005-resource-enforcement.md b/docs/decisions/AD-0005-resource-enforcement.md index 7f885bc..70d4464 100644 --- a/docs/decisions/AD-0005-resource-enforcement.md +++ b/docs/decisions/AD-0005-resource-enforcement.md @@ -1,6 +1,6 @@ # AD-0005 — Resource enforcement, WSL model and privileges -- Status: Accepted (2026-10-12; revised after the independent review the same day: job handle holders, guest budgets, aligned parameters; revised 2026-10-13 after the findings review: verified console signalling, guest allocation split and lease, stated memory guarantees; revised 2026-10-13 after the mechanism review: the guest's reservation kept during link loss, holders as disk guards, compiler servers per attempt; amended 2026-10-16 from AF-EXEC-02 evidence: Linux forced phase by freeze, per-process kill and direct thaw, exact environment through exec-launch, the node's own rlimits, `CollectMode=inactive`, `TimeoutStopSec` formula; amended 2026-10-19 by the current-design corrections, which remove overstatements and change no mechanism: never elevating is not never causing a privileged change (Process model), the same-user qualifier on every resource guarantee (Windows and Linux), no ArcForges job sets UI limits (Windows §3, reinforced), and escape detection limited to the recorded ancestry (Windows §7); amended 2026-10-19 by AD-0017: direction, and a restatement of boundary 2: confined profiles will be standard-user features, effective at the first confined-backend promotion, and optional privileged steps follow AD-0017's boundary 2, which is in force now; amended 2026-10-19 by AD-0017 as amended: direction only: owner placement, a second placement (beside managed placement) of the same job and unit mechanisms, effective when RM-18 is promoted; amended 2026-10-21 by AD-0018: "WSL2 model" items 1–3 (the guest link, guest capacity and accounting, coordinated admission) superseded and kept as history; item 4 restated as AD-0018 §2; the Linux section no longer covers WSL2 guests; amended 2026-10-23 from the AF-RES-01 verification evidence (L-27): attempt units use `OOMPolicy=stop` instead of `OOMPolicy=kill`, because the group kill that `kill` enables can kill the user manager; amended 2026-10-23 from the AF-SCHED-01 measurements (L-28): Windows attempt jobs set the priority-class limit `BELOW_NORMAL_PRIORITY_CLASS`) +- Status: Accepted (2026-10-12; revised after the independent review the same day: job handle holders, guest budgets, aligned parameters; revised 2026-10-13 after the findings review: verified console signalling, guest allocation split and lease, stated memory guarantees; revised 2026-10-13 after the mechanism review: the guest's reservation kept during link loss, holders as disk guards, compiler servers per attempt; amended 2026-10-16 from AF-EXEC-02 evidence: Linux forced phase by freeze, per-process kill and direct thaw, exact environment through exec-launch, the node's own rlimits, `CollectMode=inactive`, `TimeoutStopSec` formula; amended 2026-10-19 by the current-design corrections, which remove overstatements and change no mechanism: never elevating is not never causing a privileged change (Process model), the same-user qualifier on every resource guarantee (Windows and Linux), no ArcForges job sets UI limits (Windows §3, reinforced), and escape detection limited to the recorded ancestry (Windows §7); amended 2026-10-19 by AD-0017: direction, and a restatement of boundary 2: confined profiles will be standard-user features, effective at the first confined-backend promotion, and optional privileged steps follow AD-0017's boundary 2, which is in force now; amended 2026-10-19 by AD-0017 as amended: direction only: owner placement, a second placement (beside managed placement) of the same job and unit mechanisms, effective when RM-18 is promoted; amended 2026-10-21 by AD-0018: "WSL2 model" items 1–3 (the guest link, guest capacity and accounting, coordinated admission) superseded and kept as history; item 4 restated as AD-0018 §2; the Linux section no longer covers WSL2 guests; amended 2026-10-23 from the AF-RES-01 verification evidence (L-27): attempt units use `OOMPolicy=stop` instead of `OOMPolicy=kill`, because the group kill that `kill` enables can kill the user manager; amended 2026-10-23 from the AF-SCHED-01 measurements (L-28): Windows attempt jobs set the priority-class limit `BELOW_NORMAL_PRIORITY_CLASS`; amended 2026-10-26 by MR-17: durable node-gap memory-notification evidence) - Evidence: [platform verification](../verification/platform-capabilities.md) V-W01–V-W11, V-L01–V-L08, L-01–L-16, L-18, L-24, L-25, L-27, L-28 The numeric limits and formulas live in [resource control](../architecture/03-resource-control.md) §2 and §5. The launch, cancellation and recovery sequences live in [execution](../architecture/04-execution.md) §3, §6 and §9. This decision fixes the mechanisms. @@ -38,7 +38,7 @@ Optional privileged features are designed but not part of the current deliverabl 3. **Handle holders keep jobs alive across node restarts.** A named job disappears from the namespace as soon as its last handle closes, even while processes still run inside it (L-14). So every attempt has a holder: the launcher role `arcforges node exec-wrap`. - **Where it runs.** The holder runs *outside* the job (it is not assigned to it). It inherits handles to the child job and the root job, with `JOB_OBJECT_ASSIGN_PROCESS`, `JOB_OBJECT_QUERY` and `JOB_OBJECT_TERMINATE`, and to the output files. - **What it does.** It creates the target process suspended, assigns it to the **root job first and then to the child job**, resumes it and waits for it. Assigning a process that is already in the root job to the child job is what nests the child under the root (V-W01); assigning it to the child alone would leave the child a top-level job outside the root caps. Where available, `PROC_THREAD_ATTRIBUTE_JOB_LIST` with `[root, child]` does the same at creation. No ArcForges job sets UI restrictions, because those prevent nesting. F-W01 verifies the resulting hierarchy. *Reinforced 2026-10-19:* no ArcForges job, root or attempt, sets UI limits. A workload that sets UI limits on jobs for its own children may fail to nest them inside its attempt job. Chromium's sandbox source sets such limits, so Chromium-family programs are expected to; this is unverified for ArcForges workloads, and whether they degrade or fail is unknown, a known compatibility risk ([03](../architecture/03-resource-control.md) §5.2). - - **Exit record.** It writes the durable exit record with job accounting ([04](../architecture/04-execution.md) §3). + - **Exit record.** It writes the durable exit record with job accounting ([04](../architecture/04-execution.md) §3). For node-gap recovery (MR-17), a peak above the attempt job's current enabled memory-notification threshold is retained as `job_memory` even after current usage/violation flags clear. The holder queries that threshold from the job when recording, rather than substituting the hard cap or cached startup value. The existing nonzero-exit condition still applies; this does not prove allocation refusal ([04](../architecture/04-execution.md) §7). - **Disk guard.** While the node's heartbeat is stale, it enforces the attempt's output, temporary-space and low-space limits, because it holds the job handles and the output files ([06](../architecture/06-storage-workspaces.md) §5). - **Lifetime.** It keeps both job handles until the node acknowledges the record, through the named event `Local\ArcForges...ack`.