(feat): auto-trim — automatically reclaim idle pool memory

Project.toml

@@ -1,7 +1,7 @@
 name = "AdaptiveArrayPools"
 uuid = "4f381ef7-9af0-4cbe-99d4-cf36d7b0f233"
 authors = ["Min-Gu Yoo <mgyoo86@gmail.com>"]
-version = "0.3.7"
+version = "0.3.8"
 
 [deps]
 Preferences = "21216c6a-2e73-6563-6e65-726566657250"

docs/make.jl

@@ -135,6 +135,7 @@ makedocs(
             "Bit Arrays" => "features/bit-arrays.md",
             "CUDA Support" => "features/cuda-support.md",
             "Metal Support" => "features/metal-support.md",
+            "Automatic Memory Management" => "features/auto-manage.md",
             "Configuration" => "features/configuration.md",
         ],
         "Reference" => [

docs/src/features/auto-manage.md

@@ -0,0 +1,126 @@
+# Automatic Memory Management
+
+A pool keeps buffers around so the next `acquire!` is allocation-free — that reuse is the
+whole point. But over a long-running program two kinds of slack accumulate:
+
+- **Over-sized slots** — a slot that once held a large array keeps that large backing buffer
+  even after it is reused for small arrays (the buffer only ever grows).
+- **Unused slots** — a type or concurrency level that was needed at a peak but isn't part of
+  the recent working set still has its slots retained.
+
+`auto_manage` reclaims both in the background, GC-style, so you don't have to call
+`compact!` / `trim!` by hand. **It is on by default.**
+
+## What it does
+
+The reclaiming is done by two primitives — `compact!` and `trim!` — and `auto_manage` simply
+runs them for you on a timer. First, what each one does to a pool:
+
+```text
+compact — shrink ONE slot's over-grown backing buffer down to what it actually holds
+
+    before   ███░░░░░░░░░░░░░░░░░░░░░░░░   █ live data   ░ over-allocated slack
+    after    ███░                          buffer reallocated to ~1.5× the live data
+                                           (the slot stays; arrays you hold follow the move)
+
+trim — free the retained slots that sit BEYOND the ones currently in use
+
+    before   [█][█][░][░][░]               █ in use (2)   ░ idle, still retained (3)
+    after    [█][█]                        the idle tail is freed; the in-use slots stay
+```
+
+So `compact` works *within* a slot (a smaller buffer); `trim` drops *whole* slots. `auto_manage`
+runs both on a background `Timer` — a little more smartly than calling them by hand:
+
+| Runs | Action | …automatically, but only where it pays off |
+|------|--------|--------------------------------------------|
+| every `compact_interval` (**30 s**) | **auto-compact** | `compact!` restricted to genuinely bloated slots: capacity `≥ compact_bloat_factor ×` the live size **and** `≥ compact_min_bytes` (1 MiB) to reclaim — a steady-state pool is left untouched |
+| every `trim_interval` (**120 s**) | **auto-trim** | `trim!` down to the **recent working-set peak** (the widest concurrency seen since the last trim, not just the instantaneous in-use count) — so a pool that briefly widens isn't thrashed; a type unused all period trims to zero |
+
+## When (and where) the work happens
+
+The `Timer` thread only ever sets a flag; it never touches pool data. Your **owner task**
+performs the actual reclamation at the next `@with_pool` **entry**, at the top-level
+safepoint (`depth == 1`) where nothing is borrowed:
+
+```julia
+@with_pool p begin          # ← if a request is pending, it is serviced HERE, before your code
+    x = acquire!(p, Float64, 1000)
+    # ...
+end
+```
+
+This means reclamation never runs mid-computation and never on the background thread, so it
+**cannot change any result** — it only changes memory layout. An idle task that never enters a
+`@with_pool` scope is simply never serviced (and a finished task's pool is freed by the GC).
+
+!!! note "Targets the task-local pool"
+    Auto-manage acts on the **task-local pool** (`get_task_local_pool()`) — the one `@with_pool`
+    uses and the only one auto-registered. A hand-made `AdaptiveArrayPool()` is not auto-managed;
+    call `compact!` / `trim!` on it directly, or register it explicitly.
+
+## Controlling it
+
+The usual way to configure auto-manage is **once, in `LocalPreferences.toml`** — the timer reads
+these `auto_manage_*` keys at startup, so you never need to write any code:
+
+```toml
+# LocalPreferences.toml  (restart Julia to take effect)
+[AdaptiveArrayPools]
+auto_manage = true                    # master on/off (compile-time)
+auto_manage_compact_interval = 30.0   # seconds — how often to auto-compact
+auto_manage_trim_interval    = 120.0  # seconds — how often to auto-trim (Inf disables it)
+# advanced compaction tuning (rarely needed):
+auto_manage_compact_bloat_factor  = 10      # compact a slot at ≥ this × its live size
+auto_manage_compact_target_ratio  = 1.5     # shrink it down to this × live size
+auto_manage_compact_min_bytes     = 1048576 # skip if it would reclaim less
+```
+
+The same knobs are available at **runtime** (same names, minus the package-implied prefix) for
+ad-hoc tuning or in startup code — these override the Preference defaults:
+
+```julia
+enable_auto_manage!(; compact_interval = 30.0, trim_interval = 120.0,
+                      compact_bloat_factor = 10, compact_target_ratio = 1.5,
+                      compact_min_bytes = 2^20)
+
+enable_auto_manage!(; trim_interval = Inf)   # compact-only: disable auto-trim
+disable_auto_manage!()                       # stop the background timer (this session)
+auto_manage_enabled()                        # → Bool
+```
+
+`disable_auto_manage!()` only stops the timer for the current session; the feature re-arms on
+the next Julia start. To turn it off permanently, set the `auto_manage` preference (below).
+
+## Performance & turning it off
+
+The always-on cost is a single sub-nanosecond atomic flag read per `@with_pool` entry plus one
+`Int` comparison per `acquire!` — both dead-code-eliminated when the feature is compiled off.
+
+The one caveat: a `@with_pool` entry that *services* a pending request allocates the new
+right-sized buffer(s), so that particular entry is **not** zero-allocation. For a guaranteed
+zero-allocation hot path (e.g. a hard-real-time loop or an allocation benchmark), compile the
+feature out via the `auto_manage` preference (see [Configuration](@ref)):
+
+```toml
+# LocalPreferences.toml  (restart Julia to take effect)
+[AdaptiveArrayPools]
+auto_manage = false
+```
+
+With the preference off, the `@with_pool` hook is eliminated entirely — `enable_auto_manage!`
+becomes a no-op-with-warning, and you reclaim memory only when you call `compact!` / `trim!`
+yourself.
+
+## Manual control
+
+`auto_manage` is just a policy over the manual primitives, which you can always call directly:
+
+- `compact!(pool)` — shrink over-allocated backing buffers now.
+- `trim!(pool)` — drop inactive retained buffers now (see [Essential API](@ref)).
+
+## Backends & versions
+
+CPU, CUDA, and Metal, on Julia 1.12+. On Julia ≤ 1.11 the whole feature is a defined,
+exported **no-op** (`AUTO_MANAGE == false`, `auto_manage_enabled() == false`), so code using the
+API runs unchanged across the full supported Julia range.

docs/src/features/configuration.md

@@ -55,7 +55,7 @@ All pooling code is **completely eliminated at compile time** (zero overhead).
 
 ## Compile-time: RUNTIME_CHECK
 
-Enable runtime safety checks to catch pool-escape bugs. See [Safety](safety.md) for full details.
+Enable runtime safety checks to catch pool-escape bugs. See [Safety](../safety/overview.md) for full details.
 
 ```toml
 # LocalPreferences.toml
@@ -111,11 +111,39 @@ set_cache_ways!(8)
 
 **When to increase**: If your CUDA code or Julia ≤1.11 code alternates between more than 4 dimension patterns per `acquire!` call, increase `cache_ways` to avoid cache eviction (~100 bytes header per miss).
 
+## Compile-time: `auto_manage`
+
+Master switch for the background memory reclamation engine (see [Automatic Memory Management](@ref)). **Default `true`** — a background timer periodically auto-compacts and auto-trims the task-local pool.
+
+```toml
+# LocalPreferences.toml
+[AdaptiveArrayPools]
+auto_manage = false   # compile the feature out; restart Julia to take effect
+```
+
+Setting it to `false` (compile-time) dead-code-eliminates the `@with_pool` reclamation hook entirely, guaranteeing a zero-allocation hot path; `enable_auto_manage!` then becomes a no-op-with-warning and you reclaim memory only via manual `compact!` / `trim!`.
+
+### Tuning the timer (`auto_manage_*`)
+
+When `auto_manage` is on, the background timer's cadence and thresholds default from these flat keys (read once at startup; `enable_auto_manage!` overrides them at runtime). All optional:
+
+```toml
+[AdaptiveArrayPools]
+auto_manage_compact_interval = 30.0   # seconds — how often to auto-compact
+auto_manage_trim_interval    = 120.0  # seconds — how often to auto-trim (Inf disables)
+auto_manage_compact_bloat_factor = 10      # compact a slot at ≥ this × its live size
+auto_manage_compact_target_ratio = 1.5     # shrink it down to this × live size
+auto_manage_compact_min_bytes    = 1048576 # skip if it would reclaim less
+```
+
+See [Automatic Memory Management](@ref) for what each does.
+
 ## Summary
 
 | Setting | Scope | Restart? | Priority | Affects |
 |---------|-------|----------|----------|---------|
 | `use_pooling` | Compile-time | Yes | ⭐ Primary | All macros, `acquire!` behavior |
 | `runtime_check` | Compile-time | Yes | Safety | Poisoning, invalidation, escape detection |
 | `cache_ways` | Compile-time | Yes | Advanced | `acquire!` N-D caching (≤1.11 / CUDA only) |
+| `auto_manage` | Compile-time | Yes | Optional | Background auto-compact/auto-trim (default on) |
 | `MAYBE_POOLING` | Runtime | No | Optional | `@maybe_with_pool` only |

docs/src/reference/api.md

@@ -25,6 +25,7 @@
 | `get_task_local_pool()` | Returns the task-local pool instance. |
 | `reset!(pool)` | Resets active-slot counters to 0 but **keeps** all buffers for reuse. |
 | `trim!(pool; force_gc=false)` | Releases **inactive** retained buffers, keeping active arrays. Returns `(; slots_released, wrappers_released, estimated_bytes_released, gc_triggered)`. Works on CPU, CUDA, and Metal pools. Reclaims on Julia 1.12+; defined no-op (zero summary + one-time warning) on older Julia. |
+| `compact!(pool; factor=10, shrink_to=1.5, min_bytes=2^20, active=true, force_gc=false)` | Shrinks over-allocated backing buffers **in place** (held arrays follow). Returns `(; slots_compacted, bytes_reclaimed, gc_triggered)`. Usually run automatically — see [Automatic Memory Management](@ref). |
 | `empty!(pool)` | Clears all internal storage, releasing **all** memory. |
 
 ### Convenience Functions
@@ -56,6 +57,10 @@ Default element type is `Float64` (CPU) or `Float32` (CUDA).
 | `MAYBE_POOLING` | Runtime `Ref{Bool}` for `@maybe_with_pool`. (alias: `MAYBE_POOLING_ENABLED`) |
 | `RUNTIME_CHECK` | Compile-time `Int` constant (0=off, 1=on). Set via `runtime_check` preference. Restart required. |
 | `set_cache_ways!(n)` | Set N-way cache size (≤1.11 / CUDA only; no effect on Julia 1.12+ CPU). |
+| `enable_auto_manage!(; …)` | (Re)start the background auto-compact/auto-trim timer. See [Automatic Memory Management](@ref). |
+| `disable_auto_manage!()` | Stop the background timer for this session. |
+| `auto_manage_enabled()` | `Bool`: is the background timer running? |
+| `AUTO_MANAGE` | Compile-time `Bool` constant. Set via `auto_manage` preference (default `true`); `false` compiles the feature out. |
 
 ---
 

ext/AdaptiveArrayPoolsCUDAExt/AdaptiveArrayPoolsCUDAExt.jl

@@ -33,8 +33,8 @@ using CUDA
     # State management (checkpoint!, rewind!, reset!, empty!)
     include("state.jl")
 
-    # Auto-compact: service the timer-flagged request at `@with_pool :cuda` entry
-    include("auto_compact.jl")
+    # Auto-manage: service the timer-flagged request at `@with_pool :cuda` entry
+    include("auto_manage.jl")
 
     # Safety: poisoning, escape detection, borrow tracking
     include("debug.jl")

ext/AdaptiveArrayPoolsCUDAExt/acquire.jl

@@ -94,6 +94,10 @@ CUDA.jl's resize! unnecessarily (especially after safety invalidation sets dims=
 @inline function _cuda_claim_slot!(tp::CuTypedPool{T}, total_len::Int) where {T}
     tp.n_active += 1
     idx = tp.n_active
+    # Auto-trim working-set telemetry (parity with CPU `_claim_slot!`): track the peak
+    # concurrency since the last trim. Branchless `max` on host-side `Int` fields (NOT a GPU
+    # scalar op), gated by the compile-time const → DCE'd to nothing when AUTO_MANAGE is off.
+    AdaptiveArrayPools.AUTO_MANAGE && (tp._am_peak_n_active = max(tp._am_peak_n_active, idx))
     if idx > length(tp.vectors)
         push!(tp.vectors, allocate_vector(tp, total_len))
         push!(tp.slot_extents, total_len)        # parallel to `vectors` (for compact!/_slot_used)
@@ -119,6 +123,7 @@ the wrapper points to a different array's memory via `setfield!(:data)`.
 @inline function _cuda_claim_slot!(tp::CuTypedPool{T}) where {T}
     tp.n_active += 1
     idx = tp.n_active
+    AdaptiveArrayPools.AUTO_MANAGE && (tp._am_peak_n_active = max(tp._am_peak_n_active, idx))   # working-set peak (DCE'd off)
     if idx > length(tp.vectors)
         push!(tp.vectors, CuVector{T}(undef, 0))
         push!(tp.slot_extents, 0)        # placeholder slot: wrapper points elsewhere (reshape!)

ext/AdaptiveArrayPoolsCUDAExt/auto_compact.jl → ext/AdaptiveArrayPoolsCUDAExt/auto_manage.jl

@@ -1,25 +1,25 @@
 # ==============================================================================
-# CUDA Auto-Compact: service a flagged CUDA pool at its `@with_pool :cuda` entry.
+# CUDA Auto-Manage: service a flagged CUDA pool at its `@with_pool :cuda` entry.
 # ==============================================================================
 #
-# Parity with the CPU `_maybe_auto_compact!(::AdaptiveArrayPool)` in src/auto_compact.jl
+# Parity with the CPU `_maybe_auto_manage!(::AdaptiveArrayPool)` in src/auto_manage.jl
 # and the Metal mirror. The single global Timer (base module, CPU thread) sets this pool's
 # `@atomic _compact_requested` flag via the shared WeakRef registry; the owner task runs
 # the full `compact!` here, at the `_current_depth == 1` safepoint where nothing is borrowed.
 #
 # The scope-entry hook itself is already emitted by the SHARED `@with_pool` codegen (it
 # expands `@with_pool :cuda` through the same `_generate_block_inner`), so this concrete
 # method is all that's needed to make the hook act on CUDA pools — before it, the hook
-# resolved to the `::Any` no-op fallback. Reuses the base `_run_auto_compact!` (CUDA
+# resolved to the `::Any` no-op fallback. Reuses the base `_run_auto_manage!` (CUDA
 # `compact!` takes the same factor/shrink_to/min_bytes/active kwargs as CPU).
 #
 # `@inline` so the common case (flag clear) is a cheap inlined monotonic read + compare;
-# the cold `_run_auto_compact!` stays a non-inlined call. `:monotonic` matches CPU: the
+# the cold `_run_auto_manage!` stays a non-inlined call. `:monotonic` matches CPU: the
 # flag is a one-way eventual-visibility signal with no cross-field ordering invariant.
 
-@inline function AdaptiveArrayPools._maybe_auto_compact!(pool::CuAdaptiveArrayPool)
+@inline function AdaptiveArrayPools._maybe_auto_manage!(pool::CuAdaptiveArrayPool)
     if (@atomic :monotonic pool._compact_requested) && pool._current_depth == 1
-        AdaptiveArrayPools._run_auto_compact!(pool)
+        AdaptiveArrayPools._run_auto_manage!(pool)
     end
     return nothing
 end

ext/AdaptiveArrayPoolsCUDAExt/state.jl

@@ -473,16 +473,22 @@ end
 # makes inactive compaction allocate-only (no device copy / kernel launch).
 function AdaptiveArrayPools._compact_slot!(tp::CuTypedPool{T}, slot::Int, target::Int, used::Int, copy_live::Bool) where {T}
     v = @inbounds tp.vectors[slot]
+    old_rc = getfield(getfield(v, :data), :rc)   # OLD device-buffer identity, captured BEFORE the swap
     nv = CuVector{T}(undef, target)
     copy_live && copyto!(nv, 1, v, 1, used)
     _update_cuda_wrapper_data!(v, nv)           # backing takes nv's buffer (refcounted)
     setfield!(v, :dims, (target,))
+    # Re-point EVERY cached wrapper sharing this slot's OLD device buffer — not only the wrappers
+    # cached AT `slot`. A `reshape!` alias lives in a SEPARATE placeholder slot while sharing this
+    # buffer's DataRef; the per-slot scan missed it, stranding it on the freed buffer after an
+    # `active=true` compact!. Match by refcount identity (the same `.data.rc` test `reshape!` uses)
+    # and migrate each alias onto the new buffer via the refcount-balanced helper (dims kept).
     for wrappers in tp.arr_wrappers
         wrappers === nothing && continue
-        slot <= length(wrappers) || continue
-        w = @inbounds wrappers[slot]
-        w === nothing && continue
-        _update_cuda_wrapper_data!(w, v)        # held wrapper re-points to new buffer (dims kept)
+        for w in wrappers
+            w === nothing && continue
+            getfield(getfield(w, :data), :rc) === old_rc && _update_cuda_wrapper_data!(w, v)
+        end
     end
     unsafe_free!(getfield(nv, :data))           # drop the temporary's own reference
     return nothing

ext/AdaptiveArrayPoolsCUDAExt/task_local_pool.jl

@@ -36,10 +36,10 @@ Uses `Dict{Int, CuAdaptiveArrayPool{RUNTIME_CHECK}}` in task-local storage, keye
     if pool === nothing
         pool = CuAdaptiveArrayPool()  # Uses RUNTIME_CHECK for initial S
         pools[dev_id] = pool
-        # Register for auto-compaction (gated by the compile-time const; mirrors CPU
+        # Register for auto-management (gated by the compile-time const; mirrors CPU
         # get_task_local_pool). One entry per (task, device) pool. DCE'd when the
         # Preference is off, so no registry traffic on the zero-overhead build.
-        AdaptiveArrayPools.AUTO_COMPACT && AdaptiveArrayPools.register_auto_compact!(pool)
+        AdaptiveArrayPools.AUTO_MANAGE && AdaptiveArrayPools.register_auto_manage!(pool)
     end
 
     return pool::CuAdaptiveArrayPool{RUNTIME_CHECK}