Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Start with the two-page [architecture overview](docs/architecture.md).
- [Rust batch scheduler](docs/kernel-boundary.md)
- [Unified slab-based cache pool (KV, SWA, Recurrent)](docs/cache-pool.md)
- [Pluggable prefix cache storage](docs/cache-storage.md)
- [Hierarchical external KV design](docs/external-kv-design.md) (proposal)
- [Speculative decoding (MTP, DSpark)](docs/model-forward-pass.md)
- [Zero-overhead overlap scheduling](docs/scheduler.md)
- [Engine metrics and Prometheus export](docs/metrics.md)
Expand Down
12 changes: 7 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ from installed validation; see the [completion plan](completion-plan.md).

## OrbitInfer integration boundaries

OrbitInfer owns inference scheduling and device-page leases. OrbitKV will own
external cache identity and DRAM/SSD/peer retention policy; OrbiTE will own the
byte transfers. Neither integration is implemented yet. They must extend the
existing storage interfaces without adding a second scheduler or allocator.
OrbitInfer owns inference scheduling and device-page leases. A selected external
provider owns cache residency and tier policy. The first planned integration is
OrbitKV, with OrbiTE owning its byte transfers; Mooncake Store and LMCache are
alternative providers. These integrations are not implemented yet. The
[hierarchical cache proposal](external-kv-design.md) extends the existing storage
owners without adding a second scheduler or allocator.

Cancellation and logical completion do not prove physical retirement. Before
enabling an installed profile, prove that actual arena backing, Python owners
Expand Down Expand Up @@ -216,7 +218,7 @@ Perfetto traces.

## Current limits

- no prefill/decode disaggregation;
- no encoder/prefill/decode disaggregation or pipeline parallelism;
- sparse graph-safe expert dispatch requires a shared NVLink domain and retains
static receive storage; RDMA graph transport and communication overlap are pending;
- model dimensions, quantization blocks, and kernel capabilities constrain layouts;
Expand Down
5 changes: 5 additions & 0 deletions docs/cache-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ a storage provider keeps immutable checkpoints outside the
[device cache pool](cache-pool.md). OrbitInfer includes a bounded CPU DRAM provider, and
custom providers can implement other storage tiers through the same interface.

The [hierarchical external KV proposal](external-kv-design.md) specifies identity,
retirement, byte budgets and the planned OrbitKV/Mooncake Store/LMCache adapters.
It is a design for review. This guide describes the currently implemented API;
none of those three adapters is available yet.

## Using the DRAM cache

Construct a provider and pass it to an Engine with prefix caching enabled. Given a
Expand Down
30 changes: 24 additions & 6 deletions docs/completion-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,38 @@ deployment tools and authoring skills. Local validation is recorded in the
[import evidence](source-import.md). A complete source import does not establish
GPU model correctness, performance or production readiness.

The next implementation step remains P0: verify final-owner retention and the
native storage integration point. The first deployment target remains DeepSeek
on SM100. Other imported runners are available for separate validation.
Review the [hierarchical external KV design](external-kv-design.md) first: one
Engine/page allocator, one selected provider and separate outcome/retirement
receipts. The proposal covers device residency, provider DRAM/SSD and peer
placement, with OrbitKV first, then independent Mooncake Store and LMCache
adapters. It adds no adapter implementation.

Before provider implementation, make the clean-environment CI baseline pass.
The first import runs exposed newer-Clippy diagnostics and missing test/runtime
dependencies; local passes do not substitute for that gate.

The next implementation step remains P0: verify final-owner retention through
the existing native/Python storage path, then enforce representation identity
and borrowed-byte bounds. P3 begins with the CPU provider contract and OrbitKV
DRAM integration; SSD and peer profiles follow separately. GPU integration still
needs installed validation. The first deployment target remains DeepSeek on
SM100. Other imported runners are available for separate validation.

| Phase | Scope | State |
| --- | --- | --- |
| Foundation | Complete runtime, all model packages, tooling, tests and provenance | Source imported; validation in source-import.md |
| P0 | Final-reference/failed-completion lifetime audit, native storage injection and real backing ownership | Open; imported quarantine tests alone are insufficient |
| Foundation | Complete runtime, all model packages, tooling, tests and provenance | Source imported; local validation in source-import.md; CI environment/toolchain fixes pending |
| P0 | Final-reference/failed-completion lifetime audit, existing storage integration, real backing, identity and byte bounds | Open; imported quarantine tests alone are insufficient |
| P1 | Full-checkpoint offline DeepSeek numerical comparison on SM100 | Unrun; hardware required |
| P2 | CUDA graphs, overlap, persistent-state faults and cancellation | Source imported; installed profile unqualified |
| P3 | OrbitKV adapter, DRAM first, then separate SSD and peer profiles through OrbiTE | Not implemented |
| P3 | OrbitKV DRAM, then SSD/peer through OrbiTE; separate Mooncake Store and LMCache adapters | Design proposed; not implemented; sequence in external-kv-design.md |
| P4 | Packaged serving and streaming lifecycle acceptance | Sources imported; model-serving profile unqualified |
| P5 | Other CUDA, Torch and MLX runners; measured optimizations | Sources imported; complete-model validation deferred |

Existing TP/attention-DP/CP/EP/expert-TP paths remain subject to runner capability
checks and installed validation. PP and independent E/P/D deployment are not
implemented and are outside the external-cache step. Do not infer them from
distributed rank layout or from restoring a stored prefix.

## Installed validation

Before a model profile is enabled, record its exact device, driver, runtime,
Expand Down
23 changes: 23 additions & 0 deletions docs/distributed.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,28 @@
# Distributed execution

## Deployment status

The imported runtime contains attention TP/DP, context parallelism (CP), expert
parallelism (EP) and expert TP. Their availability is runner-specific:

| CUDA runner | Declared axes |
| --- | --- |
| DeepSeek-V4.1 | Attention DP, CP, EP, expert TP; no attention TP |
| GPT-OSS | Attention TP/DP, CP, EP, expert TP |
| Qwen and Gemma | Attention TP and dense FFN TP (the `expert_tp` axis) |

Source and CPU topology/control tests do not establish an OrbitInfer multi-GPU
deployment qualification. Historical upstream runs below remain upstream
evidence; see OrbitInfer's [import validation](source-import.md) and
[installed validation plan](completion-plan.md#installed-validation).

Pipeline parallelism (PP), independent encoder/prefill/decode (E/P/D) services
and multi-node serving are not implemented. EP is MoE expert parallelism, not
E/P/D disaggregation. The next design work is the
[external KV boundary](external-kv-design.md); it does not create those deployment modes.

## Runtime boundaries

`orbitinfer.Parallelism` defines rank placement in the Rust core and exposes that
immutable configuration to Python. `orbitinfer.backend.topology.ParallelGroups` owns
the Python process-group handles. `orbitinfer.backend.distributed` coordinates
Expand Down
Loading
Loading