Skip to content

fix(cache): retain execution backing and bound external cache transfers - #3

Merged
feichai0017 merged 4 commits into
mainfrom
feat/cache-identity-budgets
Nov 15, 2026
Merged

feichai0017 merged 4 commits into
mainfrom
feat/cache-identity-budgets

Conversation

@feichai0017

@feichai0017 feichai0017 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

External prefix keys previously omitted weights, layout and shard authority; queued transfers also had no borrowed-byte ceiling. More seriously, a poisoned Engine retained logical pages but dropped the actual runner and completion buffers at final-reference destruction. The deterministic controls fail on the old implementation at the backing weak-reference assertion.

This binds explicit CacheIdentity plus ordered cache geometry, validates all exports before submission, and retains byte charges through retirement. The DRAM reference provider now bounds queued jobs and bytes. Native EngineCore requires an actual backing owner; Python supplies its runner and retains uncertain completions and temporary failure frames. Successful close still releases normally. No second scheduler or page allocator is added.

Breaking development APIs: native EngineCore::new takes the backing Arc; external Python storage requires prefix_cache_identity and prefix_cache_max_bytes. Old external keys intentionally miss. These are recorded in releases.md.

Validation: uv run --no-sync python scripts/check.py passed on the combined #3/#4 tree (311 Rust controls; 2,461 Python passed, 551 explicit device/platform skips; packaged CPU example and all three model packages). Ruff, ty and pinned Clippy pass. Receipt/identity/owner controls passed 50 repetitions; all seven execution-drop controls are in the existing 50x CI step. New SM100 gate is written, not run. GPU/model correctness remains unqualified.

The first Linux run exposed an additional CPU-fixture shutdown gap (fatal-2, SIGABRT). A deterministic control proves that destroy_process_group leaves Gloo alive when the intentionally quarantined runner retains its handles. In pinned Torch 2.13.0, the Gloo shutdown hook is inherited and thread joins occur in the destructor. CPU-only fixtures now detach retired communication handles after synchronous work and serving/watch threads finish, assert destruction, and still retain the poisoned model backing. This also covers the multiphase distributed fault fixture. Removing the detach makes the new control fail at the live-native-owner assertion. Both the forced control and real two-rank fatal HTTP case passed 50 local repetitions and run 50x in CI. The standalone CI repetition step also uses the same explicit loopback interface as scripts/check.py (lo0 on macOS, lo on Linux); it must not depend on runner hostname routing. Actionlint passes. The complete 59-control combined tree passed 50 local repetitions (2,950 executions). This is consistent with the upstream Gloo shutdown race; no CUDA teardown guarantee is inferred from it.

Handoff:

  • Commit: 7fe76c8 (P0 identity, byte bounds and native execution lifetime audit).
  • Owners: existing EngineCore, PagedKvResources, Python storage binding and MemoryCacheStorage. No changes to OrbitKV or OrbiTE.
  • Contracts: unknown retirement retains actual backing, completions, logical pages and byte charges; no replay. Fixed metadata bounds and strict identity. No distributed wire change.
  • Commands: scripts/check.py; cargo clippy --locked --workspace --all-targets -- -D warnings; ruff check; ty check; focused owner/identity repeats; git diff --check. Results above; final head 7fe76c8 has 5/5 CI checks passed: Linux and macOS complete checks plus 41 controls x 50 repetitions each, Clippy and lint/types, and Apple/MLX controls.
  • Installed gate: uv run --no-sync python tests/hardware/cache_storage_cuda.py --device cuda:0 (SM100, unrun).
  • Limits: native callers must include every runner allocation in the backing owner; Python runners must classify post-accept errors correctly. CPU controls do not establish physical CUDA quiescence. OrbitKV integration follows separately.
  • Net delta: +881 lines (+973/-92), including tests/docs and the installed gate.

Copilot AI balanced review requested due to automatic review settings November 15, 2026 10:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@feichai0017
feichai0017 force-pushed the feat/cache-identity-budgets branch from 43afdd2 to 9655934 Compare November 15, 2026 11:19
@feichai0017
feichai0017 force-pushed the feat/cache-identity-budgets branch from 9d482fe to 7fe76c8 Compare November 15, 2026 11:43
@feichai0017
feichai0017 merged commit 0818c94 into main Nov 15, 2026
5 checks passed
@feichai0017
feichai0017 deleted the feat/cache-identity-budgets branch November 16, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants