Read only the probe's slice of untranslatable overlay flakes

[bplatz]

Problem

Staged inserts into a SHACL-governed ledger were quadratic in batch size. A reported import of 42k records (two xsd:decimal values per record) as 11 plain inserts of ~4 MB spent 80–290 s per batch between staging and commit, extrapolating to ~95 min, while bulk import of the same data takes 3.6 s.

The cause is not SHACL itself (the sh:sparql constraint suspected in the report was <1% of samples) but two per-call overlay costs on the staged view that SHACL multiplies by (focus nodes × property shapes):

1. Raw-merge fallback in binary_range_eq_v3. Decimals new to a transaction are absent from the persisted NumBig arena, so they fail V3 translation and are merged as raw flakes. Every range call cloned the entire cached raw set, then sorted and hashed all of it in resolve_current_flakes before filtering to the one subject requested. Measured on one 3.4k-record batch: 98k calls × ~4.8k raw flakes = 472M flakes processed, ~0.75 µs each → 352 s.
2. Unbounded walk in overlay_only_flakes. Taken when a bound component has no persisted id (e.g. a predicate the index hasn't seen yet — every first batch of a new record type). It walked the whole overlay per call with None/None/leftmost=true, the same pattern fixed in range_with_overlay's genesis arm by feat: Add sh:sparql (SHACL-SPARQL) constraint support #1717.

Change

• The cached raw set is already sorted in the key's index order (every OverlayProvider yields in comparator order; asserted in debug). A probe now binary-searches its span with overlay_eq_bounds (raw_window) and keeps only flakes of a matching fact (raw_fact_may_match) before lifecycle resolution. The filter tests s/p/o and never t, so a cancelling retraction still reaches resolve_current_flakes. The uncached path applies the same filter.
• overlay_only_flakes seeks to the same prefix span via overlay_eq_bounds (made pub in core so both paths share one bound derivation).

Results

Replaying the reported workload (file storage, reindex between batches, shapes/policy/rdfs schema cross-ledger), dev-fast build:

batch	records	before	after
1	4,936	15.9 s	0.59 s
2	4,654	2.6 s	0.67 s
3	3,825	129.7 s	0.68 s
4	3,367	351.9 s	0.81 s
5–11	2,405–4,328	not run (trend: minutes each)	0.56–0.85 s

Full import: ~95 min extrapolated → ~8 s.

Tests

• raw_window_is_the_probe_span_with_its_retractions, raw_window_without_a_prefix_is_everything, raw_fact_filter_ignores_t — unit tests on the window and filter.
• overlay_only_probe_walks_only_its_span — a counting overlay honoring the (first, rhs] contract pins that the overlay-only lane visits only the probe's span and still resolves an in-novelty retraction.
• shacl_over_staged_view_sees_untranslatable_decimals (it_staged_view_dict) — end to end over both lanes (predicate unindexed → overlay-only; indexed → raw-merge, confirmed via the translation debug event). A valid batch of novel decimals passes sh:minCount; an out-of-range value is rejected by sh:maxInclusive.

Each was verified non-vacuous by mutation: an empty raw window fails the SHACL test (50 false minCount violations) and the window unit test; an unbounded overlay walk fails the span test; a t-sensitive filter fails the filter test.

Ran locally: cargo test for fluree-db-core, fluree-db-query, fluree-db-shacl, fluree-db-transact, and fluree-db-api --features shacl (0 failures); clippy on the touched crates with --all-targets; cargo fmt --check.

Relation to #1335

Independent; no file overlap. #1335's inline decimal encoding would stop small decimals from reaching the raw-merge path on ledgers whose root uses the new format (new ledgers, or after a full reindex). This change still matters for existing arena-format ledgers, decimals beyond the inline range, overflow integers, novel durations, and vectors — and the overlay-only walk has nothing to do with decimals.

Not addressed

overlay_only_flakes_bounded (the subject-interval range_bounded path) still walks the whole overlay per call. It has different boundary semantics and wasn't on this path.

[zonotope]

📲

[aaj3f]

✅ Approve with nits.

This is a really nice, tight fix, @bplatz — the diagnosis (not SHACL itself, but two per-call overlay costs that SHACL multiplies by focus nodes × shapes) holds up exactly against the code, and binary-searching the already-sorted raw set with the same overlay_eq_bounds sentinels the genesis arm uses is the right altitude: no new structure, all four index orders, and it composes with #1335 rather than competing with it. I tried hard to make the window change an answer — a 2,880-probe differential across every index order and probe shape, including cross-type numerics and raw retractions cancelling base facts — and it never did; CI is green including the five new tests by name. My notes are all about keeping the win: the sortedness precondition is only asserted in debug (and we've declined order-dependence without a fallback sort before), the raw-merge call site can be reverted without any test noticing, and the non-SPOT windows aren't tested.

Adherence to repo commitments:

• Patterns/abstractions: ✔ extends the shared overlay_eq_bounds bound derivation rather than re-deriving; honors the correctness-preserving fallback contract (the raw lane still merges retractions and resolves lifecycles; the window only narrows it).
• Performance (speed first, memory second): ✔ large improvement on the staged raw-merge and overlay-only lanes (O(raw) clone + sort + hash per probe → two binary searches plus the matching slice; whole-overlay walk → seek); only new constant is two sentinel flakes per call. No performance-degradation risk.
• Testing: ⚠️ unit + end-to-end tests that run in CI (own [[test]] target with required-features = ["native", "shacl"], seen by name in the nextest log); but the raw-merge call site has no test teeth and no bench covers the lane.
• Conventions: ✔ self-describing subject, thorough multi-line body; clippy/fmt green in CI; no behavior change to document.

Verified locally at branch HEAD: cargo test -p fluree-db-query --lib binary_range::tests (6 PR tests + a review-only differential, green), cargo test -p fluree-db-api --features native,shacl --test it_staged_view_dict (8 passed), and mutation checks on the overlay-only seek (red, as claimed) and on the raw-merge call site (green — see the inline note).

Approving so you can merge when ready, but maybe worth considering a few of these first — solo#1219 is pinned to this head, so if anything gets folded in, it'll need a repin there anyway.

[aaj3f]

Optional — this is more of a question than a suggestion: should the sortedness precondition be repaired in release too, rather than only asserted in debug?

Before this change the raw lane didn't care about order — resolve_current_flakes sorts whatever it's handed. raw_window binary-searches entry.raw with partition_point, so if a raw set ever arrived out of comparator order the window would silently drop matching flakes (and the retractions that cancel them) in a release build. I don't think that's live today: I went through every overlay that vouches a content_version and can therefore reach this cache (Novelty, StagedLedger, ReasoningOverlay, SchemaBundleOverlay, the api's composite overlay, DerivedFactsOverlay, and core's delegating SizedOverlayRef), and each merges or sorts. But a debug_assert! only fires when a debug test happens to push an untranslatable flake through a non-compliant overlay, which is a narrow net for a future OverlayProvider.

We've declined this exact trade before — the query_overlay_only_range.rs bench doc says the limit early exit "was not worth restoring at the price of depending on the walk order with no sort to fall back on", and collect_overlay_only keeps its sort as "a cheap … safety net for non-compliant overlays". Here the check runs once per cache fill (which already translates every overlay flake with dict probes), not per probe, so a release-mode repair should cost nothing measurable:

let cmp = index.comparator();
let mut raw = raw; // or bind `mut raw` in the destructure above
if !raw.is_sorted_by(|a, b| cmp(a, b).is_le()) {
    debug_assert!(false, "overlay yielded out of comparator order");
    raw.sort_by(cmp);
}

Minor and non-blocking — but if you agree it's right, I'd rather see it folded in now than lost in the backlog.

[aaj3f]

Optional — nothing in the repo would go red if this call site went back to cloning the whole raw set.

I reverted just this site — entry.raw in place of raw_window(&entry.raw, index, match_val), and both raw_fact_may_match filters to |_| true — and all 8 tests in it_staged_view_dict (including shacl_over_staged_view_sees_untranslatable_decimals) still pass. The new unit tests call raw_window and raw_fact_may_match directly, so they can't see this site either, and the SHACL test's 50 items are far too few to be slow, so the 95 min → 8 s win is guarded only by the replay outside the repo. (The overlay-only half does have teeth: unbounding its seek fails overlay_only_probe_walks_only_its_span.)

A quick scaling probe shows how cleanly this separates (debug build, one SHACL-governed staged insert of n items with novel decimals against your weighted ledger fixture): at n = 250 / 500 / 1,000 the head takes 46 / 63 / 101 ms, and with only this call site reverted it takes 468 / 1,768 / 6,888 ms — ~3.8× per doubling (quadratic) against roughly linear, 68× apart at n = 1,000. Every test still passes in both builds.

Cheapest teeth I can think of: have the raw-merge path report how many raw flakes a probe merged (a field on the existing debug event, or a trace!), and assert in shacl_over_staged_view_sees_untranslatable_decimals that no probe merges more than a handful. The fuller option is a bench scenario — a SHACL-governed staged insert with novel decimals, in transact_commit.rs or a new transact_shacl_staged.rs, plus its regression-budget.json entry — so nightly tracks the number. Minor and non-blocking, but if you agree, I'd rather see one of these land with the fix than rediscover the regression from the next customer import.

[aaj3f]

Optional — the non-SPOT windows, and cross-type numeric equality, aren't exercised by a test.

raw_window is correct as long as two things hold: overlay_eq_bounds' sentinels bracket every real flake (they do — t = i64::MIN/MAX), and FlakeValue::Ord says Equal wherever PartialEq (which both raw_fact_may_match and flake_matches_range_eq use) says equal — e.g. a POST or OPST probe for Long(3) has to find a raw BigInt(3) or Decimal("3") inside the window. That agreement holds for every variant I checked (scale-different decimals, cross-type numerics, -0.0, and Duration, whose Ord is explicitly storage order), but it's a per-type property, and the unit tests only cover SPOT s+p and a no-prefix probe.

Claude and I ran a differential locally to check it: 61 raw flakes (decimals at two scales, equal Double/Long/BigInt values, list-index and @en/@fr metadata, retractions including one written as "2.50" that cancels a base "2.5") × 4 index orders × 720 probe shapes = 2,880 probes, comparing the old answer (merge all raw, resolve, post-filter) against the new one (window + pre-filter). Identical on all 2,880; 2,211 of the windows actually narrowed and 1,224 answers were non-empty. Happy to share the test body — folded in, it'd be cheap regression-proofing for exactly the property the window leans on.

[aaj3f]

Praise — the fact-only filter.

Filtering on the fact identity and never t, with the doc comment saying why and raw_fact_filter_ignores_t pinning it, is exactly the subtle part — a filter that also matched t would drop the later-t retraction on any t-bound probe and quietly resurrect the fact it cancels.

[aaj3f]

Praise — one shared bound derivation.

Reusing overlay_eq_bounds (made pub) instead of re-deriving bounds means the genesis arm, the raw window and the overlay-only seek all share one bound derivation. Nice side effect worth knowing about, too: a probe whose window holds no raw flakes now has has_untranslated == false, so it goes back to the per-row filter and early-limit exit instead of resolving everything.

[bplatz]

Addressed in bbcd747:

• Sortedness in release. Took your snippet: the cache fill re-sorts an out-of-order raw set in release and still asserts in debug. It runs once per fill, not per probe. It has no test of its own, since a debug build hits the assert first.
• Teeth for the raw-merge call site. Went with the cheap option. Each probe now emits a trace! with the number of raw flakes it merged, and shacl_over_staged_view_sees_untranslatable_decimals asserts that no probe on the raw-merge lane merges more than 4. At the head the widest probe merges 1. With the call site reverted as you did it (entry.raw, both filters |_| true), the test fails with "a probe merged 50 raw flakes". The bench is filed as bench(transact): SHACL-governed staged insert with novel decimals #1913, with your scaling numbers.
• Non-SPOT windows and cross-type numerics. Added raw_window_answers_as_the_full_merge_in_every_order: all four index orders, one value as Long / BigInt / Decimal / Double, lang-tagged strings, a ref, and a "2.50" retraction cancelling a "2.5" assert, across every s/p/o probe combination. Each windowed answer is compared with resolving the whole raw set and then filtering. It also asserts that some window in each order narrowed and some answer is non-empty. A window shrunk by one flake only outside SPOT fails this test and nothing else; shrinking either end in every order also fails the existing SPOT test.

Local: fmt, clippy --all-features on the query and api crates, fluree-db-query unit tests (1596 passed), it_staged_view_dict (8 passed). solo#1219 needs repinning to bbcd747.