Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
run: |
for iteration in $(seq 1 50); do
echo "CPU owner/fixture repetition $iteration/50"
uv run --no-sync python -m pytest -q tests/test_cache_retirement.py tests/test_execution_retirement.py tests/test_cache_identity.py \
uv run --no-sync python -m pytest -q tests/test_cache_retirement.py tests/test_execution_retirement.py tests/test_cache_identity.py tests/test_orbitkv_storage.py \
tests/test_distributed_http.py::test_rank_fixture_joins_watcher_before_destroying_process_groups \
tests/test_distributed_http.py::test_failed_runner_retention_does_not_leave_fixture_gloo_threads_alive \
tests/test_distributed_http.py::test_distributed_http_routes_real_replica_engines[fatal-2] \
Expand Down
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ Preserve third-party copyright, licenses, notices and the source manifest.
- A runner receives a frozen plan. Do not add a second scheduler, page allocator
or forwarding coordinator. Storage providers extend the existing interface.
- OrbitKV owns external cache identity and policy. OrbiTE owns byte movement.
Their integration is not yet implemented; do not create fake success adapters.
The first OrbitKV adapter uses fixed-block GPU registrations and native receipts.
Do not create fake success adapters or claim unsupported layouts.
- Cancel, timeout, logical completion and dropped handles are not physical
retirement. Preserve actual backing until its last device/transport borrower
retires. Uncertain cleanup retains ownership and stops admission.
Expand Down
7 changes: 4 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,10 @@ from installed validation; see the [completion plan](completion-plan.md).
## OrbitInfer integration boundaries

OrbitInfer owns inference scheduling and device-page leases. A selected external
provider owns cache residency and tier policy. The first planned integration is
OrbitKV, with OrbiTE owning its byte transfers; Mooncake Store and LMCache are
alternative providers. These integrations are not implemented yet. The
provider owns cache residency and tier policy. The first source-level adapter is [OrbitKV local DRAM](orbitkv-dram.md), using its
existing native registration and transfer owners for fixed GPU blocks. Window/
state mappings and installed validation remain open. Mooncake Store and LMCache
are separate planned providers. The
[hierarchical cache proposal](external-kv-design.md) extends the existing storage
owners without adding a second scheduler or allocator.

Expand Down
4 changes: 2 additions & 2 deletions docs/cache-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ custom providers can implement other storage tiers through the same interface.

The [hierarchical external KV proposal](external-kv-design.md) specifies identity,
byte budgets and the planned OrbitKV/Mooncake Store/LMCache adapters. Identity,
budgets and independent operation/retirement receipts are implemented below. The remaining proposal
is not an integration claim; none of those three adapters is available yet.
budgets and independent operation/retirement receipts are implemented below. The [OrbitKV local DRAM adapter](orbitkv-dram.md) accepts fixed full-history CUDA
blocks. Its installed profile is unqualified; other adapters remain planned.

## Using the DRAM cache

Expand Down
14 changes: 9 additions & 5 deletions docs/completion-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ Engine/page allocator, one selected provider and separate outcome/retirement
receipts. The proposal covers device residency, provider DRAM/SSD and peer
placement, with OrbitKV first, then independent Mooncake Store and LMCache
adapters. The first implementation makes load/store outcome and physical
retirement separate receipts through the existing Engine. No provider adapter
has been added.
retirement separate receipts through the existing Engine. A separate OrbitKV adapter maps the existing client to fixed, full-history GPU
blocks in same-host DRAM; its installed profile is unqualified.

Clean-environment validation includes the CPU test dependencies and model-source
paths inherited by subprocess controls. Current Clippy's nonzero-divisor checks
Expand All @@ -25,8 +25,7 @@ P0 is implemented at the Rust/Python owner boundary: final drops retain actual
backing, uncertain completions and page charges; storage keys bind representation
identity and borrowed bytes are bounded until retirement. CPU controls include
synchronous failure, asynchronous inspection/harvest failure and a successful
release control. Installed CUDA execution remains unqualified. P3 begins with the CPU provider contract and OrbitKV
DRAM integration; SSD and peer profiles follow separately. GPU integration still
release control. Installed CUDA execution remains unqualified. P3 now has CPU contract controls for the fixed-block OrbitKV DRAM adapter; SSD and peer profiles follow separately. GPU integration still
needs installed validation. The first deployment target remains DeepSeek on
SM100. Other imported runners are available for separate validation.

Expand All @@ -36,7 +35,7 @@ SM100. Other imported runners are available for separate validation.
| P0 | Final-reference/failed-completion lifetime audit, existing storage integration, real backing, identity and byte bounds | Implemented at source level; real backing, identity, byte and metadata bounds verified on CPU; installed CUDA gate unrun |
| P1 | Full-checkpoint offline DeepSeek numerical comparison on SM100 | Unrun; hardware required |
| P2 | CUDA graphs, overlap, persistent-state faults and cancellation | Source imported; installed profile unqualified |
| P3 | OrbitKV DRAM, then SSD/peer through OrbiTE; separate Mooncake Store and LMCache adapters | Design proposed; not implemented; sequence in external-kv-design.md |
| P3 | OrbitKV DRAM, then SSD/peer through OrbiTE; separate Mooncake Store and LMCache adapters | Local DRAM adapter implemented for one fixed full-history GPU group; CPU client/CUDA shim controls; installed SM100 gate unrun; SSD/peer/window/state mappings deferred |
| P4 | Packaged serving and streaming lifecycle acceptance | Sources imported; model-serving profile unqualified |
| P5 | Other CUDA, Torch and MLX runners; measured optimizations | Sources imported; complete-model validation deferred |

Expand Down Expand Up @@ -65,3 +64,8 @@ OrbiTE integrations remain separate work.
`uv run --no-sync python tests/hardware/cache_storage_cuda.py --device cuda:0`
checks GPU views, guards, prefix restore and final-owner quarantine on SM100.
Written but not run. It does not qualify full-model execution or OrbitKV.

OrbitKV local DRAM: `uv run --no-sync python tests/hardware/orbitkv_dram.py
--bootstrap-socket /run/orbitkv/orbitkv.sock --metrics-url http://127.0.0.1:9090/metrics
--device cuda:0 --output /tmp/orbitinfer-orbitkv-dram.json`.
Requires a DRAM-only Manager; written but not run. See [layout and route limits](orbitkv-dram.md).
4 changes: 2 additions & 2 deletions docs/external-kv-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Design and implementation sequence. The current interface and independent
retirement receipts are described in [prefix cache storage](cache-storage.md).
Identity binding and borrowed-byte budgets are implemented; provider adapters remain the next step.
Identity binding and borrowed-byte budgets are implemented; the fixed-block OrbitKV local DRAM adapter is implemented at source level; further profiles remain planned.
No OrbitKV, Mooncake Store or LMCache adapter is implemented yet.

## Scope
Expand Down Expand Up @@ -267,7 +267,7 @@ must force interleavings and inspect backing and budget receipts.
actual arena backing across final drops, and add identity and byte bounds.
The Python storage receipt/view-retention controls are implemented. Native
execution backing, identity binding and byte accounting now have CPU controls. Installed device validation is still required.
2. **P3a: OrbitKV local DRAM.** Add the thin adapter and a CPU contract test.
2. **P3a: OrbitKV local DRAM.** The [thin adapter](orbitkv-dram.md) and CPU contract tests cover fixed full-history GPU blocks. Variable window/state mappings remain open.
Write the SM100 runner export/producer/readiness gate; run it only on an
explicitly authorized installed host. Do not modify OrbitKV from this work.
3. **P3b: OrbitKV SSD and peers.** Separate byte-checked profiles through the
Expand Down
111 changes: 111 additions & 0 deletions docs/orbitkv-dram.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# OrbitKV local DRAM

`OrbitKVCacheStorage` maps fixed GPU KV blocks onto an existing same-host OrbitKV
Manager. OrbitInfer retains the existing prefix matcher, page allocator and
storage operation limits. OrbitKV owns DRAM residency, native registrations,
Publish/Restore and byte movement. The adapter issues no memcpy and does not add
an inference coordinator. No OrbitKV or OrbiTE source changes are required.

This is **implemented at source level; installed GPU profile unqualified**.
CPU tests drive the real adapter and Engine through an explicit CUDA/client shim,
including byte/guard checks. They are not a GPU or Manager data-path pass.

## Supported layout

The first profile is one full-history group, one block per physical slab, and a
single local GPU shard. Each registered layer is a contiguous CUDA `uint8` view
shaped `[physical_block_count, bytes_per_block]`; count includes slab zero. Layers
may have different byte widths but share the block count and device. The runner's
`cache_storage_buffers` must export exactly one whole block for every registered
layer, all at the same physical block index. No allocation or gather is needed
to form a contiguous byte view of an existing fixed block layout.

Window/state tails, multiple groups, packed slots, partial blocks, resharding and
CPU buffers are rejected. This does **not** yet integrate the full DeepSeek-V4.1
compressed/window/state layout. Its variable checkpoint slices need a separate
layout mapping; silently padding or concatenating them would change cache
semantics. SSD, peer and multi-shard profiles remain separate work.

## Binding

Install an OrbitKV client wheel exposing the native Save/Restore APIs. The source
contract was inspected at
[OrbitKV 93c1c55](https://github.com/OrbitInfra/OrbitKV/tree/93c1c55d795c19cb677f6661763d158a37a059af),
particularly `python/src/client.rs`, `python/src/local_restore.rs` and
`crates/orbitkv-channel/src/client.rs`. The adapter checks required methods before
registration. OrbitInfer does not import OrbitKV or Torch merely on import.

Configure the Manager for **local DRAM only**. Supply its bootstrap socket and
already allocated runner backing:

```python
from orbitinfer import CacheIdentity, Engine
from orbitinfer.integrations.orbitkv import OrbitKVCacheStorage

identity = CacheIdentity(
model="model-id", weights="immutable-weight-revision", format="runner-cache-v1",
dtype="bfloat16", shard="tp=0/1,pp=0/1", semantics="attention-rope-adapter-fingerprint",
)
# These byte views retain the existing GPU allocations. They include null slab 0.
layers = {"k": runner.key_blocks.view(torch.uint8).view(block_count, -1),
"v": runner.value_blocks.view(torch.uint8).view(block_count, -1)}
with OrbitKVCacheStorage(
"/run/orbitkv/orbitkv.sock", identity=identity, layers=layers,
block_tokens=block_tokens, max_pending_bytes=256 * 1024**2,
) as storage:
with Engine(runner, prefix_cache=True, prefix_cache_storage=storage,
prefix_cache_identity=identity, prefix_cache_max_bytes=256 * 1024**2):
... # Submit and drive requests through the usual Engine API.
```

Import `torch` at the caller. Names in `layers` must match the runner's exported
region names. One registration belongs to one Engine. Separate Engines with the
same representation use separate registrations and can share Manager objects.
The namespace includes representation identity, block width and layer names;
physical block capacity and process identity are excluded. Engine keys additionally
include its compiled cache geometry, namespace and token chain. A lookup is only
advisory: load claims a complete lease again before any GPU submission.

## Ownership

The adapter has one storage worker, at most 16 admitted operations by default
(configurable 1–64), a required byte limit, and bounded key/region metadata. This
worker bridges the existing blocking native client to `CacheTransfer`; it does
not schedule inference or move bytes. Producer events are recorded on the calling
stream and observed before Publish. Restore also passes that retained stream to
the native client's previous-user fence.

- `SaveNotSubmittedError` proves no Publish worker was launched and can retire a
failed operation immediately.
- A native `done` result is the client's retirement receipt, including a drained
failure. A pending result, exception or observation timeout is not a receipt.
- Unknown completion retains the client, registration, native handle, exact
views and charges. It fences new submissions and is never replayed.
- Query leases are released if no Restore starts. Once Restore is attempted,
the native client owns consumption; the adapter never guesses whether to
release or replay it after an exception.
- Close the Engine before the provider. Provider close joins its bounded worker,
then unregisters. Failed unregister/close is sticky and is never retried.
Dropping a provider without successful explicit close conservatively retains
its registration; use the context manager.

`timeout` defaults to 30 seconds (maximum 300) for observing a transfer, not for
releasing memory. No timer fabricates a retirement receipt. The Engine's separate
byte budget remains charged until it consumes the provider's successful receipt.

## Installed gate

On an explicitly authorized SM100 host with an installed OrbitKV Manager using
only DRAM, run:

```bash
uv run --no-sync python tests/hardware/orbitkv_dram.py \
--bootstrap-socket /run/orbitkv/orbitkv.sock \
--metrics-url http://127.0.0.1:9090/metrics \
--device cuda:0 --output /tmp/orbitinfer-orbitkv-dram.json
```

The gate checks GPU bytes and guards, prefix restore through public Engine calls,
registration cleanup and route counters. It fails if SSD or peer byte counters
advance. Keep the report outside Git. It is written, not run; no full-model or
hardware qualification is claimed. See the [verification plan](completion-plan.md#installed-validation).
7 changes: 6 additions & 1 deletion docs/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,10 @@
releases pages. Use `transfer.wait(timeout=...)` for standalone blocking calls.
- `MemoryCacheStorage.close()` reports a failed device fence through
`CacheRetirementError`; repeated close retains owners without retrying cleanup.
- No OrbitKV, Mooncake Store or LMCache adapter is shipped yet. See the
- `OrbitKVCacheStorage` adds source-level fixed-block GPU KV to local DRAM
through the existing OrbitKV client. One full-history group is supported;
variable window/state slices and SSD/peers are rejected or outside this profile.
CPU shim controls are not hardware qualification. Explicit provider close is
required; uncertain cleanup and unclosed registrations remain retained.
- Mooncake Store and LMCache adapters remain planned. See the
[external KV design](external-kv-design.md) and [implementation plan](completion-plan.md).
3 changes: 3 additions & 0 deletions orbitinfer/_cache_storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,9 @@ def __init__(self, store, runner, layout, identity, max_bytes):
if g.family != CacheGroupFamily.STATE
and g.window_resident_block_count is None
}
validate = getattr(store, "validate_binding", None)
if validate is not None:
validate(identity, layout)

def _keys(self, key):
namespace, tokens = key
Expand Down
1 change: 1 addition & 0 deletions orbitinfer/integrations/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Optional external providers. Importing OrbitInfer does not load their runtimes."""
Loading
Loading