Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,12 @@ jobs:
run: uv sync --locked --no-default-groups --group test
- name: Check source, Rust, Python and installed wheel
run: uv run --no-sync python scripts/check.py
- name: Repeat storage and CPU fixture controls
run: |
for iteration in $(seq 1 50); do
echo "CPU owner/fixture repetition $iteration/50"
uv run --no-sync python -m pytest -q tests/test_cache_retirement.py \
tests/test_distributed_http.py::test_rank_fixture_joins_watcher_before_destroying_process_groups \
tests/test_deepseek_auxiliary_graphs.py::test_auxiliary_callbacks_bind_model_projections_and_preserve_output_dtypes \
models/cuda/tests/test_deepseek_v41_projections.py::test_prepared_vocab_head_preserves_last_row_full_logits_and_fp32_output
done
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ The complete source import includes CUDA model runners, MLX/Metal runners,
Torch reference models, serving tools, benchmarks and tests. The first planned
installed target is DeepSeek-V4.1-Flash on SM100. No complete GPU model profile
has been qualified for OrbitInfer. See the [plan](docs/completion-plan.md) and
[source import record](docs/source-import.md).
[source import record](docs/source-import.md). Interface changes are recorded in
the [change notes](docs/releases.md).

## Features

Expand All @@ -33,7 +34,7 @@ Start with the two-page [architecture overview](docs/architecture.md).
- [Rust batch scheduler](docs/kernel-boundary.md)
- [Unified slab-based cache pool (KV, SWA, Recurrent)](docs/cache-pool.md)
- [Pluggable prefix cache storage](docs/cache-storage.md)
- [Hierarchical external KV design](docs/external-kv-design.md) (proposal)
- [Hierarchical external KV design](docs/external-kv-design.md) (adapters planned)
- [Speculative decoding (MTP, DSpark)](docs/model-forward-pass.md)
- [Zero-overhead overlap scheduling](docs/scheduler.md)
- [Engine metrics and Prometheus export](docs/metrics.md)
Expand Down
4 changes: 2 additions & 2 deletions crates/orbitinfer-core/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1239,8 +1239,8 @@ impl<P: Clone> EngineCore<P> {
group.append_block_offsets[row] = 0;
continue;
}
group.append_block_offsets[row] = position % layout.block_tokens().get();
let logical = position / layout.block_tokens().get();
group.append_block_offsets[row] = position % layout.block_tokens();
let logical = position / layout.block_tokens();
let relative = logical
.checked_sub(group.block_table_start_indices[row])
.ok_or_else(|| Error::internal("continuation precedes its KV lease"))?;
Expand Down
2 changes: 1 addition & 1 deletion crates/orbitinfer-core/src/planning.rs
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ pub fn schedule_ready_prefix(
} else {
proposed_end
};
let checkpoint_end = checkpoint_limit / alignment.get() * alignment.get();
let checkpoint_end = checkpoint_limit / alignment * alignment.get();
if checkpoint_end > request.logical_context && checkpoint_end < proposed_end {
count = checkpoint_end - request.logical_context;
}
Expand Down
2 changes: 1 addition & 1 deletion crates/orbitinfer-core/src/resources/kv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ impl KvLayout {
"shared prefix must align to full-history blocks",
));
}
let shared_blocks = u64::from(shared_prefix_tokens / group.block_tokens.get());
let shared_blocks = u64::from(shared_prefix_tokens / group.block_tokens);
let extra_copies = u64::try_from(context_lengths.len().saturating_sub(1))
.map_err(|_| overflow())?;
blocks -= shared_blocks
Expand Down
14 changes: 10 additions & 4 deletions crates/orbitinfer-core/src/resources/paged_kv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,13 @@ pub struct PrefixCacheMetrics {

pub type PrefixStorageGroup = (u32, Vec<(u32, u32)>);
pub type StorageOp = Box<dyn PrefixStorageOperation>;
/// A storage job retained with its page loans until physical retirement.
pub trait PrefixStorageOperation: fmt::Debug + Send + Sync {
/// Return a matched-prefix length for lookup, or success (one) / failure
/// (zero) for transfer. A terminal transfer result is a release receipt:
/// every access to borrowed backing must have ended, even on failure.
/// `None` retains the job and its loans, including after a failed fence.
/// `wait` permits blocking for the receipt; it never fabricates one.
fn poll(&mut self, wait: bool) -> Option<u32>;
}
pub trait PrefixStorage: fmt::Debug + Send + Sync {
Expand Down Expand Up @@ -307,7 +313,7 @@ impl PagedKvResources {
&& Some(read) != layout.retention().window_tokens()
{
let live_first =
(key.len() as u32).saturating_sub(read.get()) / layout.block_tokens().get();
(key.len() as u32).saturating_sub(read.get()) / layout.block_tokens();
let count = mask.len();
for logical in *first..live_first {
mask[logical as usize % count] = false;
Expand Down Expand Up @@ -469,7 +475,7 @@ impl PagedKvResources {
// Wide leases may leave idle spare pages; owner growth never shrinks.
let page_count = (group.active_blocks(context).1 as usize).max(table.pages().len());
demands.push(GroupDemand::new(GroupId::new(index as u32), page_count));
let first = start / group.block_tokens().get();
let first = start / group.block_tokens();
// A cyclic table is visited at most once, even for long prefills.
let count = context
.div_ceil(group.block_tokens().get())
Expand Down Expand Up @@ -1077,7 +1083,7 @@ impl PagedKvResources {
}

for position in step.token_start()..post_length {
let logical = position / group.block_tokens().get();
let logical = position / group.block_tokens();
let page = page_for(group, table.pages(), logical).ok_or_else(|| {
Self::invalid(format!(
"request {} cache group {group_index} is missing logical block {logical}",
Expand All @@ -1090,7 +1096,7 @@ impl PagedKvResources {
append_offsets.push(if group.is_recurrent() {
0
} else {
position % group.block_tokens().get()
position % group.block_tokens()
});
}
}
Expand Down
4 changes: 2 additions & 2 deletions crates/orbitinfer-py/src/cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ fn plan_error(error: KvPlanError) -> PyErr {
eq,
hash,
rename_all = "SCREAMING_SNAKE_CASE",
from_py_object,
skip_from_py_object,
module = "orbitinfer._native"
)]
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
Expand All @@ -46,7 +46,7 @@ pub(crate) enum CacheRetention {
eq,
hash,
rename_all = "SCREAMING_SNAKE_CASE",
from_py_object,
skip_from_py_object,
module = "orbitinfer._native"
)]
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
Expand Down
25 changes: 24 additions & 1 deletion crates/orbitinfer-py/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,29 @@ use runner::{
resolve_runner_config, resolve_runner_contract, runner_error_is_post_accept,
};

// PyO3's from_py_object expansion always clones. These enums are Copy; preserve
// its guard errors and inspection hints without suppressing clone_on_copy.
macro_rules! copy_from_py_object {
($($kind:ty),+ $(,)?) => {$(
impl<'a, 'py> FromPyObject<'a, 'py> for $kind {
type Error = pyo3::pyclass::PyClassGuardError<'a, 'py>;
const INPUT_TYPE: pyo3::inspect::PyStaticExpr =
<Self as pyo3::PyTypeInfo>::TYPE_HINT;

fn extract(obj: Borrowed<'a, 'py, PyAny>) -> Result<Self, Self::Error> {
Ok(*obj.extract::<pyo3::PyClassGuard<'_, Self>>()?)
}
}
)+};
}

copy_from_py_object!(
CacheRetention,
CacheGroupFamily,
recipe_types::Role,
BatchPhase
);

fn default_sampling_seed(engine_id: u64, request_id: u64) -> i64 {
// SplitMix64 gives stable, well-distributed row seeds without adding a
// control-plane RNG or making batch composition part of randomness.
Expand Down Expand Up @@ -137,7 +160,7 @@ impl SchedulerConfig {
eq,
hash,
rename_all = "SCREAMING_SNAKE_CASE",
from_py_object,
skip_from_py_object,
module = "orbitinfer._native"
)]
#[derive(
Expand Down
2 changes: 1 addition & 1 deletion crates/orbitinfer-py/src/recipe_types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ use serde_json::{Map, Value, json};
/// Some roles are model-specific; recipes validate supported roles.
#[non_exhaustive]
#[allow(non_camel_case_types, clippy::upper_case_acronyms)]
#[pyclass(frozen, eq, eq_int, from_py_object, module = "orbitinfer._native")]
#[pyclass(frozen, eq, eq_int, skip_from_py_object, module = "orbitinfer._native")]
#[derive(Clone, Copy, Debug, PartialEq)]
pub(crate) enum Role {
SYSTEM,
Expand Down
7 changes: 7 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@ enabling an installed profile, prove that actual arena backing, Python owners
and registrations survive failed device completion and final-reference drop.
The imported quarantine controls alone do not establish that proof.

External storage now reports logical outcome and borrowed-buffer retirement
separately through `CacheTransfer`. The native storage job stays charged until
the binding observes a successful retirement receipt. Failed receipts retain
exact exported views and fence that storage binding, including after final
Engine drop. This proves the Python storage boundary on CPU; it does not close
the remaining native-runner or installed GPU lifetime audit.

## Runtime owners

OrbitInfer separates request and resource ownership from model-specific execution.
Expand Down
73 changes: 47 additions & 26 deletions docs/cache-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ a storage provider keeps immutable checkpoints outside the
custom providers can implement other storage tiers through the same interface.

The [hierarchical external KV proposal](external-kv-design.md) specifies identity,
retirement, byte budgets and the planned OrbitKV/Mooncake Store/LMCache adapters.
It is a design for review. This guide describes the currently implemented API;
none of those three adapters is available yet.
byte budgets and the planned OrbitKV/Mooncake Store/LMCache adapters. Independent
operation and retirement receipts are implemented below. The remaining proposal
is not an integration claim; none of those three adapters is available yet.

## Using the DRAM cache

Expand Down Expand Up @@ -128,15 +128,16 @@ lengths contribute to request cached-token metrics.

## Implementing a provider

The [`CacheStorage` protocol](../orbitinfer/cache_storage.py) has three methods. Each
submits work and returns a `concurrent.futures.Future` promptly, without waiting
for I/O. These are ordinary methods returning futures, not `async def` methods.
The [`CacheStorage` protocol](../orbitinfer/cache_storage.py) has three methods.
Each returns promptly without waiting for I/O. Lookup returns a
`concurrent.futures.Future`; load and store return a `CacheTransfer` containing
two futures. These are ordinary methods, not `async def` methods.

| Method | Result |
|---|---|
| `lookup(keys)` | A tuple of presence booleans in key order. |
| `store(keys, buffers)` | Transfer success, or unknown completion as described below. |
| `load(keys, buffers)` | Transfer success, or unknown completion as described below. |
| `store(keys, buffers)` | `CacheTransfer`: publication outcome and borrowed-source retirement. |
| `load(keys, buffers)` | `CacheTransfer`: whole-load outcome and borrowed-destination retirement. |

Keys are opaque `bytes` values. For transfers, `keys` and `buffers` have equal
lengths, and `buffers[i]` maps string region names to borrowed, contiguous
Expand All @@ -158,24 +159,44 @@ providers or partitions for incompatible model or cache representations.

### Transfer completion

A transfer future also defines when OrbitInfer may recycle its buffers:

- `True` means the transfer succeeded and all accesses to the borrowed buffers
have finished. A successful load must fill every requested region.
- `False`, or a terminal exception, means the operation failed but all buffer
accesses have finished. The Engine can discard a failed restore safely.
- `None` means buffer access may still be outstanding. OrbitInfer retains the owners;
Engine shutdown raises instead of releasing memory that may still be in use.

Providers must order GPU copies after the work that produced their inputs. A
separate CUDA stream alone does not establish that ordering. Resolve a transfer
with `True`, `False` or an exception only after all borrowed-buffer accesses have
finished. `future.done()` must also return promptly. Cancellation is not proof of
completion.

Completion covers the borrowed buffers. A provider can manage several tiers by
staging data in its own allocation, then continuing remote I/O against that copy
after the GPU transfer has finished. The provider owns that copy and its lifetime.
`CacheTransfer(outcome, retirement)` separates the two facts:

- `outcome: Future[bool]` reports success (`True`), a miss/failure (`False`), or an
operation exception. A successful load fills every requested region. A
successful store publishes an immutable object that can be served completely.
- `retirement: Future[None]` succeeds with `None` only when all accesses to the
borrowed buffers have ended. Pending, cancellation, an exception or a non-None
result is not a release receipt. Failed retirement is sticky.

The Engine consumes an outcome only after successful retirement. A failed
outcome can arrive earlier without releasing pages. A retirement failure fences
new transfers to that storage binding and keeps the operation's exact exported
views, runner and page charges. Close reports failure; dropping the final Engine
reference still retains the unquiesced owners. A provider submission that raises
without returning a receipt is also conservatively unquiesced: it may have
accepted work before raising. Reject capacity before accepting access, then return
a failed outcome with successful retirement when nothing was posted.

Providers own and complete both futures. Consumers can inspect them but must not
complete them. Cancelling an outcome is not cancellation of DMA. For example,
the DRAM provider acknowledges a cancelled queued operation's retirement only
when its worker proves that no copy was started. `future.done()` alone never
authorizes buffer reuse. Providers must order GPU copies after their producer;
a separate CUDA stream alone does not establish that dependency.

For standalone use, `transfer.wait(timeout=...)` waits for both receipts within
one timeout budget and returns the bool outcome. It propagates operation errors
only after retirement and raises `CacheRetirementError` for an invalid or failed
retirement receipt. A timeout does not cancel, release or replay the transfer.
The Engine's normal polling uses the same receipts without blocking its scheduler.

A provider can release the device source after a proven copy into its own charged
staging allocation, then continue remote I/O against that copy. Retirement can
therefore precede publication. The Engine conservatively retains its operation
until both are observed; the provider cannot expose a partial object through
lookup. Its own staging allocation remains retained until the later I/O retires.
`MemoryCacheStorage.close()` raises `CacheRetirementError` after a failed fence;
repeated close does not retry that native cleanup.

## Runner support

Expand Down
26 changes: 15 additions & 11 deletions docs/completion-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,31 @@ deployment tools and authoring skills. Local validation is recorded in the
[import evidence](source-import.md). A complete source import does not establish
GPU model correctness, performance or production readiness.

Review the [hierarchical external KV design](external-kv-design.md) first: one
Follow the [hierarchical external KV design](external-kv-design.md): one
Engine/page allocator, one selected provider and separate outcome/retirement
receipts. The proposal covers device residency, provider DRAM/SSD and peer
placement, with OrbitKV first, then independent Mooncake Store and LMCache
adapters. It adds no adapter implementation.

Before provider implementation, make the clean-environment CI baseline pass.
The first import runs exposed newer-Clippy diagnostics and missing test/runtime
dependencies; local passes do not substitute for that gate.

The next implementation step remains P0: verify final-owner retention through
the existing native/Python storage path, then enforce representation identity
adapters. The first implementation makes load/store outcome and physical
retirement separate receipts through the existing Engine. No provider adapter
has been added.

Clean-environment validation includes the CPU test dependencies and model-source
paths inherited by subprocess controls. Current Clippy's nonzero-divisor checks
are addressed without changing scheduling arithmetic. Keep this CI baseline
passing before adding providers.

P0 remains open: the Python storage path now retains exact exported views on
failed retirement and final-reference drop, with forced-interleaving controls.
Next audit native/device execution backing and enforce representation identity
and borrowed-byte bounds. P3 begins with the CPU provider contract and OrbitKV
DRAM integration; SSD and peer profiles follow separately. GPU integration still
needs installed validation. The first deployment target remains DeepSeek on
SM100. Other imported runners are available for separate validation.

| Phase | Scope | State |
| --- | --- | --- |
| Foundation | Complete runtime, all model packages, tooling, tests and provenance | Source imported; local validation in source-import.md; CI environment/toolchain fixes pending |
| P0 | Final-reference/failed-completion lifetime audit, existing storage integration, real backing, identity and byte bounds | Open; imported quarantine tests alone are insufficient |
| Foundation | Complete runtime, all model packages, tooling, tests and provenance | Source imported; validation in source-import.md; clean CPU gate repairs included |
| P0 | Final-reference/failed-completion lifetime audit, existing storage integration, real backing, identity and byte bounds | Partial: explicit Python storage receipts and backing-retention controls; native/device audit, identity and byte bounds remain open |
| P1 | Full-checkpoint offline DeepSeek numerical comparison on SM100 | Unrun; hardware required |
| P2 | CUDA graphs, overlap, persistent-state faults and cancellation | Source imported; installed profile unqualified |
| P3 | OrbitKV DRAM, then SSD/peer through OrbiTE; separate Mooncake Store and LMCache adapters | Design proposed; not implemented; sequence in external-kv-design.md |
Expand Down
Loading
Loading