Conversation
…he worker does not have `WorkerResources::n_resources` spans only the resources the worker declared (`from_description`), while a request may name a resource another worker provides. `get` already returns zero for such an id, but `remove`, `remove_multiple` and `remove_multiple_masked` indexed the vector directly, and the scheduler's gap computation (`GapCache::get_gap`) reaches them with such requests once task priorities are in use: the server died with `index out of bounds: the len is 3 but the index is 4` at workerload.rs:160 (issue It4innovations#1135). A worker holds none of a resource it did not declare, so there is nothing to subtract. Adds a gap test with a cpus-only worker and a request that names a resource the worker lacks (panics before this change).
a932f9b to
10fb418
Compare
|
For context, from our side: with this change (main + bounds-safe #1136 fixes the underlying gap computation, which is the real fix, and it also passes the same stream. This PR is a different, complementary kind of change: it only makes |
Fixes #1135.
WorkerResources::n_resourcesspans only the resources a worker declared (from_description), while a request may name a resource that another worker provides.getalready treats such an id as zero, butremove,remove_multipleandremove_multiple_maskedindexed the vector directly. With task priorities in use the scheduler's gap computation (GapCache::get_gap) reaches them with such requests (a multi-variant request whose variant needs agpus-like resource, evaluated on a CPU-only worker), and the server died withindex out of bounds: the len is 3 but the index is 4atworkerload.rs:160.A worker holds none of a resource it did not declare, so there is nothing to subtract: the three functions now go through one
subtracthelper that skips ids past the vector.Adds
test_compute_gap_resource_the_worker_lacksingap.rs(a cpus-only worker and a request naming a resource the worker lacks), which panics before this change.Verification: the production-shaped stream from the issue (
run.sh/driver.py) crashed the 2026-09-25 nightly within 16–18 s in every run; with this change the same stream ran 4/4 × 45 s without a crash.