Introduce Spectral Sentinel — an online subspace anomaly detector

[da2ce7]

A sentinel is not a judge. It stands watch, keeps its bearings, and reports what has changed.

That is the idea behind Spectral Sentinel. It does not decide whether a pattern is dangerous, important, or actionable. It measures structure in a stream, learns what has been ordinary so far, and returns statistical readouts when new observations depart from the learned geometry.

The "spectral" part is literal: each selected region is modelled with low-rank subspace trackers, and a second tier models the spectrum of those scores across related cells. The "sentinel" part is restraint: the crate observes, scores, and reports, but policy stays with the host.

---

Summary

This PR adds torrust-sentinel to the workspace: a library crate for hierarchical online subspace anomaly detection over positionally structured observation streams. It is rebuilt as a single commit on the current develop (after #884), so it sits on the MSRV-1.89 floor of ADR-T-011 and follows the per-crate versioning of ADR-T-012.

It is built on top of Mudlark. Mudlark provides the adaptive spatial substrate — regions that receive more observation volume earn finer resolution, quiet regions remain coarse. Spectral Sentinel uses that structure to decide where statistical trackers are worth maintaining. It selects significant V-Tree entries, closes them under G-tree ancestry so every selected cell has a complete ancestor chain back to the root, and scores incoming batches against learned subspace models at every selected scale.

The simplest way to think about it: Mudlark decides where the stream has shape; Sentinel measures whether the recent shape still looks like what that region has learned to expect.

The crate is deliberately policy-free. Reports carry raw measurements — four scoring axes (novelty, displacement, surprise, coherence), maturity, baselines, CUSUM drift accumulators, geometry, contour summaries, and health snapshots. They do not encode threat levels, recommended actions, or decisions. The host reads the measurements and decides what they mean.

The core invariant is feed-forward: every input value updates Mudlark with exactly one unit of observation volume. Anomaly scores never flow back into the spatial index. That keeps spatial adaptation driven by traffic structure, not by the detector's own conclusions. Temporal policy is host-controlled: Sentinel never applies decay automatically.

A second analysis tier — coordination trackers — runs at internal G-tree nodes whose subtrees both contribute competitive cells. It scores cross-cell patterns of the four axes, so a coordinated shift that no single cell would flag still surfaces in the report.

Contents

The addition is substantial, but almost entirely self-contained within packages/sentinel.

• A new torrust-sentinel crate (1.0.0) with a narrow public surface exposed through flat crate-root re-exports
• SpectralSentinel<C, V, N> as the generic engine, with Sentinel128 and Sentinel64 aliases for the common domain widths
• SentinelConfig, NoiseSchedule, and SvdStrategy for host-controlled measurement parameters, with structured ConfigError / ConfigErrors / ConfigWarning validation rather than panics
• Public readout types for batch, cell, coordination, contour, health, maturity, geometry, baseline, score, and analysis-set summaries — ordered deterministically by GNodeId
• Analysis-set selection over Mudlark's G-V Graph: top-K competitive V-entries plus ancestor closure into the investment set
• Per-cell subspace trackers scoring novelty, displacement, surprise, and coherence; EWMA baselines with upper-tail clipping; CUSUM drift accumulators with a separate slow EWMA
• Hierarchical coordination trackers scoring cross-cell score patterns at G-tree internal nodes
• Automatic synthetic noise warm-up for every newly created tracker, plus a deferred staging area and optional background warming thread so cell creation does not block the ingest hot path
• Brand incremental SVD for subspace evolution, with a naive thin-SVD path used both as a fallback for small/numerically sensitive cases and as a debug-mode oracle
• Optional serde support (off by default; pulls in torrust-mudlark/serde)
• A Criterion benchmark suite (15 bench functions across 8 groups: encoding, ingest, auxiliary, convergence, scaling, temporal, analysis)
• 21 architecture decision records covering measurement-not-opinions, the feed-forward invariant, Mudlark integration, deterministic ordering, analysis-set recomputation, automatic noise injection, scoring geometry, decay semantics, routing, degenerate-dimension guards, test budgets, warm-up convergence, visibility, cell-creation performance, Brand SVD, deferred warm-up, generic domain parameters, investment-set terminology, and the clip-pressure / mean-centred-variance EWMA refinements
• README, public API reference, algorithm document, and implementation notes (~4.7k lines of crate-level documentation total)
• 642 tests with all features enabled (631 in the default configuration) across crate-level tests (src/tests/), integration tests (tests/) and doc-tests, with the README compiled as a doc-test via #[cfg(doctest)] include_str!
• Two pedagogy integration tests (pedagogy.rs, pedagogy_advanced.rs) written to be read end-to-end as a walkthrough of the public surface

Changes outside sentinel

• Cargo.toml — packages/sentinel added as a workspace member
• packages/mudlark — unchanged: the series rides on the released Mudlark 1.1.0 that develop carries, whose structural mutation counters and semi_internal_count() accessor the report reads; the branch's own earlier cut of that feature is dropped, and no commit in the series touches the Mudlark package
• Cargo.lock — 76 entries added for the dependency closure (faer, rand_distr, plus dev-only criterion and tracing-subscriber) against the lockfile develop refreshed under the 1.90 floor; no existing entry moves, and fifteen bare dependency lines gain a version qualifier because the closure introduces a second compatible release of thiserror, thiserror-impl, rand_chacha and r-efi
• AGENTS.md — adds Sentinel's S- cross-reference prefix to the package table and ADR examples

Manifest under ADR-T-012

The dependency on the sibling torrust-mudlark pins version = "1.1.0" beside its path, because cargo publish writes that requirement into the published manifest and the report reads the structural mutation counters that arrive with that minor. faer, tracing and criterion name the 0.x line the sources are written against (0.24, 0.1, 0.8) instead of a bare 0, for the reason #884 gave for the root's requirements. cargo publish --dry-run -p torrust-sentinel stops at resolution because torrust-mudlark is not on crates.io yet; that is the publication order ADR-T-012 documents, and torrust-mudlark itself dry-runs cleanly.

Reviewing this

The best starting point is the public surface:

packages/sentinel/src/lib.rs → packages/sentinel/docs/api.md → packages/sentinel/README.md

From there, the main implementation path is src/sentinel/mod.rs for the orchestrator, src/analysis_set.rs for competitive selection and ancestor closure, src/sentinel/tracker.rs for per-cell scoring, src/sentinel/{cusum,staging,warming_thread}.rs for drift and warm-up, and src/maths/ for the SVD plumbing.

For a focused review, I would look at:

• the public API shape and the flat crate-root re-exports
• configuration validation, defaults, and the structured error/warning types
• the feed-forward Mudlark integration (Δ = 1 per observation, scores never feed back)
• report semantics, deterministic ordering, and what is and isn't part of the public surface
• tracker warm-up, the deferred staging area, and the optional background warming thread
• the Brand SVD fallback boundary (small d, narrow rank gaps) and the debug-mode oracle path
• integration tests that assert invariants through the public API only

The pedagogy tests are intended to be readable end-to-end; running cargo test -p torrust-sentinel --test pedagogy -- --nocapture produces a narrated walk through the public surface.

Verification

On the rebuilt commit: cargo fmt --check clean; cargo clippy --workspace --all-targets --all-features -- -D warnings clean under the workspace lint table; the crate's 565 tests and 15 doc-tests pass; the whole workspace passes (2,370 tests, none failed); cargo audit keeps the vulnerability count of develop (the one rsa advisory with no fixed release) and adds a single allowed unmaintained-crate warning, RUSTSEC-2024-0436 (paste, a proc-macro pulled by faer through gemm).

Notes

• Ships at 1.0.0: the public surface documented in docs/api.md is covered by semver guarantees from this release onwards. A sibling crate consumes it through a version-beside-path pin, the same discipline this manifest applies to torrust-mudlark, so the version a consumer pins is the one the manifest declares.
• MSRV 1.89, inherited from the workspace (ADR-T-011).
• No unsafe code; #![forbid(unsafe_code)] at the crate root.
• AGPL-3.0-only, inherited from the workspace. Unlike Mudlark, no linking exception is shipped with this crate.
• Default features: none. serde is opt-in.
• The crate measures only. Interpretation and response remain external host policy.
• Temporal policy is host-controlled: Sentinel never applies decay automatically.
• Configuration prefers structured errors over panics.
• Sentinel docs and ADRs use the S- cross-reference prefix added in this PR.

Review fixes

Since the previous head, nine commits on top of the three original ones fix every finding an automated review of the package produced and a code-level verification confirmed: the configuration validation refuses non-numbers, an unrepresentable depth-buffer headroom and a coordinate width below the tracker minimum (two additive error variants); the full-width cell owns the domain maximum and the root leaves the analysis candidates before the capacity cut; staged cells warm by their real volume and a pass with no competitive scores retires every coordination context; health and batch reports count the online sets, populate the semi-internal count from the graph (one additive mudlark accessor, and the headroom arithmetic in mudlark saturates instead of wrapping), count the whole contour and order coordination reports by depth then identifier; the geometric schedule reports its true maximum, a failed corrective factorisation reports failure so the dispatcher falls back, and the unread round scores are gone. Prose follows the code (the z-score denominator, the live-tracker figure, the open bit-source trait, the implemented dimension guard), and the one exact float equality in the invariants suite compares bit patterns. Nothing on the public surface is removed or reshaped.

Second round of review fixes

Five further commits fix every finding of a second automated review at the previous head. The warming thread's shutdown transition is made under the staging lock its wait is paired with, so a shutdown can no longer be lost between the worker reading its predicate and sleeping on it, which left the join — reached from Drop — waiting forever; the u128 centred-bit conversion caps the requested width at the type's own instead of indexing past its backing array; the centred bit vector gains a validated constructor and a length accessor so an implementation of the open bridge trait outside the crate can return the value its impl must produce; the online summary reports the investment count over the whole selection, warming cells included, as its contract states. The test support's four-bit generator refuses a nibble at sixteen or above (which shifted every set bit out of the coordinate and aliased the range sixteen below), a six-bit generator carries the sprays that claim sixty-four distinct ranges, and the ordering, budget and concentration witnesses assert the documented order, the structure's own budget and a report below the root. The upper bounds of analysis entries and coordination contexts document the top-of-domain exception, the analysis set's full field is named as the investment set it is, the thread-safety plan states the Send + Sync the crate asserts statically, and section-mark references with no referent leave the record and the test banners. Public surface: three additive constant functions on the centred bit vector; nothing is removed or reshaped. Verified on stable 1.98 (the toolchain this repository lints and tests with), 1.89.0 and nightly.

Third round of review fixes

Four further commits fix every finding of a third automated review at the previous head. The headroom a depth pair demands was computed with one checked step and three unchecked ones around it, so a creation depth of zero beside an eviction depth at the top of the range overflowed inside the very method that promises to hand back its faults as values; the computation now lives in a helper whose every step is checked, and any overflow reports the existing structured error for a buffer too large to honour, with the widest pair a budget can clear pinned as accepted. The centred-bit vector holds at most 128 values, but the coordinate trait it is fed from is open to wider types and the only width guard compared the tracker's dimension with the coordinate's declared bits, so a 200-wide tracker over a 256-bit coordinate was admitted and fed from a 128-slot vector; the sentinel now refuses a width above the vector's ceiling with a configuration error naming the width and the maximum, and the ceiling is documented on the bit source, on the bit vector and in the crate docs. The prefix generators in the test support guarded their ranges with assertions that release builds compile out; all three sites assert unconditionally. The dimension guard's doc said widths at or below the minimum are refused where the predicate refuses only widths below it, and now names the side of the boundary that is kept; a bare section ordinal in the exponential-average module is replaced by the sense it carried. Public surface: one additive configuration-error variant; nothing is removed or reshaped. Verified on stable 1.98 (the toolchain this repository lints and tests with), 1.89.0 and nightly.

Fourth round of review fixes

Three further commits correct every finding of a fourth automated review at the previous head; all eight are documentation, and no Rust moves. The two warming modes draw from two different generators: synchronous warming drains from the sentinel's own generator, while background warming draws from a second one seeded on the worker and promotes whatever the worker has finished at each ingest, against a map the main thread is concurrently writing. Four sites promised bit-for-bit reproducibility across runs without saying which mode delivers it; each now scopes the claim to synchronous warming on a fixed build, in one identical clause, and names the background-mode interleaving as the second source of randomness that reaches the scores. Three lifecycle records described mechanisms that no longer run where they said: the cell-width rejection lives in the suffix-width filter applied at runtime rather than in configuration validation, and its effective range is stated; creation schedules noise injection rather than performing it, now that the injection itself is deferred; and the bounded per-ingest work of the deferred warm-up record is stated for the background mode it holds in, with the default mode's in-line drain named beside it. The dimension guard's record said cells at or below the minimum are excluded where the filter keeps a cell at it, and now carries the same words as the constant's own documentation. Verified on stable 1.98 and nightly, with the crate's rustdoc and doc tests, since the README is the crate's front-page documentation.

Fifth round of review fixes

Three further commits correct every finding of a fifth automated review at the previous head. Construction asked the operating system for the background warming thread and aborted the host when the request was refused, over a resource limit that has nothing to do with the configuration's correctness; the request now returns the environment's own account as a configuration error naming the setting, and it arrives alone because the thread is asked for only once validation has passed. Reset, which has no error channel, keeps the sentinel running and warms cells synchronously instead, recording the refusal as a warning: the warm-up dispatch keys on whether a thread is present rather than on the flag, so the fallback is complete and every report is produced as before. Seeding one baseline from another copied the numbers but only ever raised warmth, so a receiver seeded from a cold source stayed warm over placeholder statistics and the cold path that replaces them never ran again; warmth is now part of what is handed over, in both directions, with a witness that fails at the previous head. Three tests and their prose claimed more, or other, than the engine guarantees: the determinism test compared three lengths and a few means where it now compares whole reports figure by figure with equal bit patterns; the coordination-report ordering test asserted ascending handle where the producer sorts by depth and then handle, and both prose statements of the handle-only order are corrected with it; and the reproducibility claim at the top of the determinism suite is scoped to synchronous warming on a fixed build, which is the configuration those tests share. Public surface: one additive configuration-error variant, and the configuration-error enumeration is marked non-exhaustive ahead of first publication so a later refusal is additive too; the warming-thread handle whose signature changed is crate-internal. Verified on stable 1.98 (the whole workspace lints clean; the crate's tests pass), 1.89.0 and nightly, with the crate's rustdoc and doc tests.

Rebased onto the released Mudlark

The series is rebased onto the develop that merged Mudlark 1.1.0. The rebase drops the branch's own cut of the structural mutation counters and the two Mudlark hunks two Sentinel commits carried, keeps every other commit byte-identical in patch, author and order, and re-resolves the lockfile against the refreshed one: the resolver accepts the result unchanged under --locked, and the full bar — nightly tests with and without features, nightly and stable clippy with warnings denied, stable tests, the 1.90 check, and rustdoc with warnings denied — is green at the tip.

Later rounds of review fixes

The remaining rounds of automated review, each verified at code level before a change was made, are answered by the commits after the fifth round. The CUSUM allowance now follows the algorithm text, κσ·√v_slow, with the denominator-protection constant kept out of it; the geometric noise schedule saturates an unrepresentable exponent instead of wrapping it, so a public caller with an arbitrarily deep argument still lands on the floor; coordination contexts are retained by online competitive membership rather than by which cells happened to score in the batch, so a quiet batch no longer destroys a context that the next joint batch would have to re-warm. Every section reference in source and tests uses the qualified §ALGO S-N form and points at the section that carries the cited content; the clip-pressure implementation plan moved to docs/plans/ so the ADR identifier it borrowed resolves to one record; the implementation guide describes the lazy coordination warm-up the code performs; a failed warming worker is recorded rather than allowed to take reset() down; the report's geometry record describes the model that produced the scores beside it; the compile-time width bound is named where a reader would look for it in the error list; and a test comment that cited a document the package never contained now derives its tolerance in place. Finally, the structural mutation counts in the contour snapshot are read from the spatial layer's own counters instead of being inferred from node and terminal deltas, an inference that a last-child eviction falsified; that is what the Mudlark minor bump carries.

The round that followed the rebase is answered by two further commits. The first repoints every stale algorithm citation in the Sentinel records, plans and source comments at the section that now specifies the behaviour each one describes, including the three that named a chapter the document no longer has and the one requirement the algorithm never specified at all. The second orders the warm-up queue by depth before identifier at both promotion sites, so an ancestor that ties with a descendant on volume is warmed first even when arena slot reuse has handed that descendant the smaller identifier, with a witness on each path that fails without the depth comparison.

The notes that round left on unchanged code are answered by four further commits. The first fixes a tracker report that mixed two models, publishing an energy share and a leading singular value read after the subspace had already been replaced, so that every reported model figure now describes the model that actually scored the batch. The second gives the API reference the two re-exported observation types it had never documented, states the real ordering of each report vector against the code that produces it, and extends the configuration-error table from a third of the enum to all of it. The third repoints four records at the specification sections that carry the material they cite, two of which named a section about something else and two a chapter the document no longer has. The fourth brings the batch warmer's equal-volume ordering into line with the two live drains, so an ancestor is warmed before the cells beneath it on whichever path serves the queue.

The round after that left four notes on unchanged records, answered by five further commits. The first states the warm-up ordering the staging area actually keeps, volume then depth then the identifier with the reason each layer exists, in both the deferred-warm-up record and the investment-set record's synchronous-drain decision, so the two describe one ordering rather than two. The second makes the contour snapshot's type-level contract count what its producer counts, terminal nodes together with semi-internal ones, and replaces the configuration record's overflow-prone headroom example with the checked helper and caller the validator actually runs. The third repoints all seventeen citations of the retired chapter 18 at the sections that replaced it. The fourth gives the specification the tie-breaks its ancestor-first guarantee depends on: the priority key becomes the lexicographic triple of volume, depth and node identifier, with the reason each layer is load-bearing. The fifth repoints thirty-four further citations left pointing at sections two renumberings removed, each target verified by reading the section rather than by arithmetic on the number.

The following round is answered by three further commits. The first recovers a sentinel whose warming worker died: a panicked background thread left its handle standing, so every later reconciliation notified a dead thread and every cell staged from then on stayed off the producing set while reports kept arriving; the worker is now reaped where it would otherwise be handed more work, the cells it was holding are rebuilt, and warming continues synchronously for the sentinel's remaining life, with a witness that fails without the detection. The second repoints fifty-six stale algorithm cross-references at the sections that actually specify what each citing line describes, after auditing all four hundred and seventeen citation sites in the package against the document's headings and bodies. The third marks the noise-schedule default's remaining-work row done, so the record stops contradicting the 450 root rounds the configuration ships.

One further commit repairs the recovery's witness, which could not reach its own claim on a small machine: its setup now warms synchronously so every scheduled cell is online before the sentinel crosses into background warming for the strand itself, and every assertion after the strand stands exactly as it did.

The round after that left three notes on unchanged code, answered by four further commits. A checked-out cell now returns with the volume the graph has now, so an ingest landing mid-warm-up can no longer leave the busiest cell queued at its pre-ingest importance. The analysis-set summary type says what each of its producers counts, instead of writing one producer's online scope into fields three producers fill. The three texts justifying the volume refresh now name the tie-breaks that would otherwise decide a newly queued cell, rather than a tie-break the queue no longer has. The specification states that the root is a member of the investment set by construction and never a competitive target, and the sites asserting it cite the section that now carries it.

[codecov]

Codecov Report

❌ Patch coverage is 91.76115% with 218 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.37%. Comparing base (843aaff) to head (bab66fa).

Files with missing lines	Patch %	Lines
packages/sentinel/src/sentinel/mod.rs	91.34%	53 Missing and 21 partials ⚠️
packages/sentinel/src/config.rs	78.02%	49 Missing ⚠️
packages/sentinel/src/maths/bench_tracing.rs	0.00%	45 Missing ⚠️
packages/sentinel/src/sentinel/tracker.rs	96.13%	13 Missing and 4 partials ⚠️
packages/sentinel/src/maths/mod.rs	91.25%	13 Missing and 1 partial ⚠️
packages/sentinel/src/maths/brand_svd.rs	91.46%	6 Missing and 1 partial ⚠️
packages/sentinel/src/sentinel/staging.rs	98.56%	3 Missing and 3 partials ⚠️
packages/sentinel/src/sentinel/warming_thread.rs	96.05%	3 Missing ⚠️
packages/sentinel/src/analysis_set.rs	98.33%	2 Missing ⚠️
packages/sentinel/src/maths/naive_svd.rs	97.61%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #827      +/-   ##
===========================================
+ Coverage    68.52%   72.37%   +3.85%     
===========================================
  Files          161      175      +14     
  Lines        13111    15757    +2646     
  Branches     13111    15757    +2646     
===========================================
+ Hits          8984    11404    +2420     
- Misses        3853     4048     +195     
- Partials       274      305      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[Copilot]

Copilot review overview

🟡 Changes recommended

Handle-based warming tie-breaks can violate ancestor-first promotion, and several specification references are unresolved or incorrect.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 17

Open (18)

• Tie-break can violate ancestor-first warming · New
• Wrong chapter for axis polarity convention · New
• Wrong references for volume accounting and ordering · New
• Seed parameter cites clip modulation · New
• Selector and delivery references are obsolete · New
• Wrong references for selection and closure · New
• Wrong reference for noise-injection triggers · New
• Scoring axes point to baseline chapter · New
• Wrong phase reference for observation delivery · New
• Wrong references for closure and tracker · New
• Tracker metadata points to wrong chapter · New
• Stale tracker and EWMA cross-references · New
• Wrong phase reference for subspace evolution · New
• Obsolete work-variance section references · New
• Obsolete deferred warm-up section reference · New
• Wrong CUSUM cross-reference · New
• Obsolete warm-up section reference · New
• Invalid section reference for Send + Sync rationale · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Report metadata is time-skewed, and the canonical API documentation contains unresolved contract inaccuracies and omissions.

Review effort: Balanced
Findings: None

Resolved since last review (18)

• Tie-break can violate ancestor-first warming
• Invalid section reference for Send + Sync rationale
• Obsolete warm-up section reference
• Wrong CUSUM cross-reference
• Obsolete deferred warm-up section reference
• Obsolete work-variance section references
• Wrong phase reference for subspace evolution
• Stale tracker and EWMA cross-references
• Tracker metadata points to wrong chapter
• Wrong references for closure and tracker
• Wrong phase reference for observation delivery
• Scoring axes point to baseline chapter
• Wrong reference for noise-injection triggers
• Wrong references for selection and closure
• Selector and delivery references are obsolete
• Seed parameter cites clip modulation
• Wrong references for volume accounting and ordering
• Wrong chapter for axis polarity convention

Previously missed (4)

In code that hasn't changed since last review

Coordination report mixes updated and prior model metrics

packages/​sentinel/​src/​sentinel/​tracker.rs:324

evolve_subspace has already replaced self.sigmas before this snapshot, so energy_ratio describes the updated model while rank and geometry describe the prior model that actually produced the scores. This contradicts the public coordination-report contract in docs/api.md:207 and makes one report internally time-skewed. Capture the energy ratio (and, if it is intended to describe the scoring model, the top singular value) before Phase 2, then emit that snapshot.

Document re-exported CentredBitSource and CentredBits APIs

packages/​sentinel/​docs/​api.md:106

The canonical API reference re-exports CentredBitSource and CentredBits here but never documents their public contract or the public CentredBits::{new,len,is_empty,from_u128,suffix} methods; §6 instead lists the observation module as internal machinery. This conflicts with the README's promise that every crate-root re-exported type, trait, and method is documented and semver-covered. Add a public-surface section for these APIs and remove their types from the internal-only table.

Document coordination report ordering by depth and node ID

packages/​sentinel/​docs/​api.md:156

This blanket ordering statement is incorrect for coordination_reports: the producer and determinism test order those reports by ascending depth and then GNodeId, not by handle alone. Document the distinct order so consumers do not incorrectly sort or compare coordination reports.

Document all ConfigError variants

packages/​sentinel/​docs/​api.md:771

This purportedly exhaustive public ConfigError table ends here, but the enum continues with coordination-decay, CUSUM allowance, clipping, split/depth/budget, noise, tracker-width, and warming-thread variants (config.rs:485-534). Since this document is the crate's definitive stable API reference, consumers cannot discover most construction failures from it. Extend the table to cover every current variant.

[Copilot]

Copilot review overview

🟡 Changes recommended

Several public and architectural documents still contradict the implemented contour, validation, and warm-up-ordering behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 4

Open (4)

• Use checked depth arithmetic in the example · New
• Document depth and ID tie-breaks for ancestor-first ordering · New
• Document depth and identifier tie-breaks in volume ordering · New
• Correct cell_count contract to include semi-internal nodes · New

[Copilot]

Copilot review overview

🟡 Changes recommended

A failed warming worker can permanently strand staged cells, and several documentation references remain stale.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 13

Open (14)

• Recover from panicked warming worker · New
• Correct ancestor closure cross-reference · New
• Correct competitive selection references · New
• Correct suffix extraction cross-reference · New
• Correct competitive selection cross-reference · New
• Correct tracker suffix cross-reference · New
• Correct CUSUM seeding cross-reference · New
• Correct coordination context cross-reference · New
• Correct running-mean cross-reference · New
• Correct competitive selection references · New
• Correct ingest procedure references · New
• Correct hierarchical coordination section marker · New
• Correct competitive selection test citation · New
• Correct latent initialization test citation · New

Resolved since last review (4)

• Correct cell_count contract to include semi-internal nodes
• Document depth and identifier tie-breaks in volume ordering
• Document depth and ID tie-breaks for ancestor-first ordering
• Use checked depth arithmetic in the example

Previously missed (1)

In code that hasn't changed since last review

Remove completed root-rounds TODO

packages/​sentinel/​adr/​013-warm-up-convergence-benchmark.md:273

This remaining-work item is already implemented: NoiseSchedule::default() uses 450 root rounds in src/config.rs:126-140. Leaving it marked high-priority TODO makes the implemented ADR contradict the shipped configuration; mark it completed or remove it.

[Copilot]

Copilot review overview

🔵 Needs a closer look

In-flight warm-up cells retain stale priority volumes across concurrent ingests, and several public API contracts remain inconsistent with staged-cell behavior.

Review effort: Balanced
Findings: None

Resolved since last review (14)

• Recover from panicked warming worker
• Correct latent initialization test citation
• Correct competitive selection test citation
• Correct hierarchical coordination section marker
• Correct ingest procedure references
• Correct competitive selection references
• Correct running-mean cross-reference
• Correct coordination context cross-reference
• Correct CUSUM seeding cross-reference
• Correct tracker suffix cross-reference
• Correct competitive selection cross-reference
• Correct suffix extraction cross-reference
• Correct competitive selection references
• Correct ancestor closure cross-reference

Previously missed (3)

In code that hasn't changed since last review

Summary type mixes selection counts with online tracker counts

packages/​sentinel/​src/​report.rs:779

AnalysisSet::summary() also returns this public type, but that method counts the whole selection whether or not trackers are online. Consequently its competitive_size and full_size values do not satisfy this type's “online” field contracts, and investment_set_size is a selection count rather than necessarily a count of allocated trackers. Please separate the selection and producing summary types, or define these fields context-neutrally and document how each producer populates them.

In-flight volume updates are lost when cells return to warming

packages/​sentinel/​src/​sentinel/​staging.rs:415

An ingest can run while the worker has removed a cell from warming, so this loop cannot refresh that cell's cached volume. return_warming then restores the old value, and the worker may immediately choose its next batch using pre-ingest importance, delaying a cell that has just become hottest and violating the documented volume-first policy. Please retain the latest volume for in-flight IDs and apply it when the cell is returned; a checkout → graph update → return regression test would pin the interleaving.

Test rationale uses outdated zero-volume tie-break behavior

packages/​sentinel/​src/​sentinel/​README.md:33

This test-index claim still says zero-volume ties put the newest/deepest cell first, but the queue now breaks ties by shallower depth and then smaller GNodeId. Please update the rationale to say that missing the refresh would make deterministic tie-breaks, rather than traffic volume, decide newly queued cells.

[Copilot]

Copilot review overview

🟡 Changes recommended

Out-of-domain narrow-width coordinates can update Mudlark totals while being omitted from every Sentinel tracker, violating the feed-forward reporting invariant.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Out-of-domain values bypass Sentinel trackers · New
• Use package-qualified output contract reference · New
• Point polarity invariant to §ALGO S-5.1 · New
• Point projection-energy proof to §ALGO S-15.2 · New