Skip to content

(perf): selective fallback checkpoint/rewind via touched-others stack - #50

Merged
mgyoo86 merged 10 commits into
masterfrom
perf/fallback
Jul 10, 2026
Merged

mgyoo86 merged 10 commits into
masterfrom
perf/fallback

Conversation

@mgyoo86

@mgyoo86 mgyoo86 commented Jul 9, 2026 •

Copy link
Copy Markdown
Member

Problem

Reported via FastInterpolations CI benchmarks: a @with_pool call site that acquires one fallback (non-fixed-slot) type slows down when unrelated code registers other fallback types in the same task-local pool — even with active=0, depth=1, and after reset!. Pool history, not the scope's own work, determined hot-path latency (order-dependent timing in benchmark suites; phase-dependent latency in applications).

Root cause

Fallback types had no per-type touch tracking — a single per-depth _touched_has_others::Bool covered all of them:

  • _lazy_checkpoint! eagerly checkpointed every registered fallback pool at every lazy scope entry (_typed_lazy_checkpoint! likewise).
  • When any fallback was touched, rewind scanned the whole registry (_others_values::Vector{Any}), paying a dynamic dispatch per pool.

Scope cost was O(all fallback types ever registered) — ~160 ns per registered type per scope on an M-series machine.

Fix

Give fallback pools the same first-touch lazy checkpoint discipline fixed slots already have, using an identity list instead of a bitmask (the fallback type set is open):

  • _touched_others::Vector{Any} + _touched_others_checkpoints::Vector{Int} — a flat stack of fallback pools first-touched at each open depth, with per-depth segment bases (same pattern as _others_ptr_bounds).
  • Producers (_touch_fallback_pool!, shared first-touch guard): typed checkpoint!(pool, T), @generated checkpoint!(pool, types...), _record_type_touch!, and get_typed_pool! new-type registration.
  • Rewind drains only the current depth's segment: O(types touched this scope). The eager checkpoint loops and full-registry rewind scans are deleted.
  • Invariant: a fallback pool has a checkpoint entry at depth d ⟺ it is in the depth-d stack segment — except under full checkpoint!(pool), whose eager sweep pairs with full rewind!(pool)'s sweep (segment stays empty, truncate-only).
  • _touched_has_others is kept (still consumed by _can_use_typed_path and the S=1 debug validation). reset! docs now note that registered-but-inactive fallbacks no longer add per-scope cost.

CPU only; CUDA/Metal extensions keep their current (correct, non-selective) behavior — GPU parity is the next PR.

Benchmarks

Master measured in a clean worktree with identical scripts and preferences (runtime_check=false), runs serialized, replicated with side order swapped. Julia 1.12, Apple Silicon.

Pollution — lazy @with_pool scope acquiring one fallback type; 1 vs 8 registered (inactive, reset!-ed) fallback types:

clean (others=1) polluted (others=8) ratio
master 55.7 ns 1175.0 ns 21.1×
this PR 55.1 ns 54.6 ns 0.99×

Scaling on master (same scope, others = 1..8; zero allocations throughout — this PR stays flat at ~55 ns across the whole range):

others 1 2 3 4 5 6 7 8
ns 55.8 95.9 146.7 195.9 382.8 731.9 1095.8 1275.0

The marginal cost is nonlinear: master's eager checkpoint sweep and full-registry rewind are two dynamic-dispatch sites cycling through every registered TypedPool{T}. Up to 4 distinct types Julia's per-call-site dispatch cache stays warm (~45 ns/type); from the 5th type on, every dispatch falls back to the global method-table lookup (~180–360 ns/type). This matches the original report (others=3, warm regime, ~35 ns/type) — and means real applications degrade sharply once a 5th fallback type registers.

Fixed-slot-only scopes (no fallback anywhere; measures the new bookkeeping itself):

path master this PR delta
typed (static Float64) 20.0–20.1 ns 21.6 ns +1.5 ns
lazy (dynamic type var) 20.6 ns 21.4 ns +0.8 ns

The ~1 ns/scope cost (per-scope segment-base push/pop + empty-segment drain; ≈ +0.3% on the originally reported workload) is accepted for this PR. A depth-tagged-stack variant that eliminates it is planned alongside the follow-up bookkeeping rework.

Testing

  • New test/test_touched_others.jl (285 lines): pollution regression (unrelated registered fallbacks are never visited), nested scopes (same type at multiple depths), helper-function untracked touches in typed scopes, statically-tracked fallback types, mid-scope type registration, full checkpoint!/rewind! pairing, similar! touch recording, @with_pool integration, exception-leak recovery, and @allocated == 0 regression for lazy and macro roundtrips.
  • Full suite green (src/state.jl coverage 98%), including the S=1 runtime-check paths.
  • Side effect: fixes a pre-existing manual-API leak — checkpoint!(pool, T) + helper fallback acquire! + rewind!(pool, T) previously never restored the fallback's n_active.

Follow-ups (tracked separately)

  1. GPU parity: mirror selective tracking in the CUDA/Metal extensions.
  2. Concrete PoolCheckpointState to remove the drain loop's dynamic dispatch, folding in the depth-tagged stack to erase the fixed-only ~1 ns cost.
  3. Macro: treat global-name parametric type literals (Foo{Float64}) as static so those scopes take the typed path.

mgyoo86 added 9 commits July 9, 2026 13:26
…tracked acquire handling, and vector resize memory behavior

- Deleted `selective_memory_reclamation.md` as the design has been implemented and is no longer needed.
- Removed `untracked_acquire_design.md` which outlined a previous approach to handling untracked acquisitions in AdaptiveArrayPools.
- Eliminated `vector_resize_memory_behavior.md` that discussed the memory behavior of vector resizing, as the design decisions have been finalized.
@with_pool scopes now checkpoint/rewind only the fallback typed pools they
actually touch (first-touch lazy checkpoint, per-depth stack segment) instead
of eagerly checkpointing and rewinding every registered fallback pool. Fixes
order-dependent latency where unrelated fallback registrations slowed hot
call sites (active=0 pollution).
…l state

The task-local pool is a process-wide singleton shared by every test file.
@with_pool's fast path does not guarantee cleanup for exceptions that escape
its own scope (only nested leaks within one invocation are caught by the
entry-depth guard), so earlier files that deliberately throw across a
@with_pool boundary (e.g. test_macro_expansion.jl's stack-trace tests) can
leave _current_depth elevated before this testset runs. Reset the task-local
pool at the start of the testset so its absolute-depth assertions hold
regardless of test file execution order. Root-caused by direct repro (with
and without the touched-others changes: identical leak in both), confirming
it is pre-existing and unrelated to the semantic switch.
@codecov

codecov Bot commented Jul 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.47826% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.88%. Comparing base (e1f67c0) to head (1ec0984).

Files with missing lines Patch % Lines
src/state.jl 93.02% 3 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #50      +/-   ##
==========================================
+ Coverage   95.82%   95.88%   +0.05%     
==========================================
  Files          16       16              
  Lines        3619     3645      +26     
==========================================
+ Hits         3468     3495      +27     
+ Misses        151      150       -1     
Files with missing lines Coverage Δ
src/acquire.jl 91.04% <100.00%> (+0.09%) ⬆️
src/types.jl 85.71% <100.00%> (+0.18%) ⬆️
src/state.jl 96.12% <93.02%> (-0.47%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

reference/api.md renders at 200.05 KiB in CI — just over Documenter's 200 KiB
hard limit — because GitHub source links (remotes, CI-only) add a few KiB that
local builds don't have. Exempt only this page; all others keep the guardrail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant