Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,27 @@ encoding, MoE routing, recurrent/sliding state, vocabulary, and multimodal detai

Reuse an existing recipe when its formatting, parsing and media semantics match.
When working in the OrbitInfer repository or a fork, implement a new native Rust recipe
in [`orbitinfer-recipe`](../../crates/orbitinfer-recipe/README.md) and its bindings. When
in [`orbitinfer-recipe`](../../../crates/orbitinfer-recipe/README.md) and its bindings. When
importing OrbitInfer as a library without its source, implement the
[`orbitinfer.recipe.Recipe`](../../orbitinfer/recipe.py) protocol in Python instead; a Rust
[`orbitinfer.recipe.Recipe`](../../../orbitinfer/recipe.py) protocol in Python instead; a Rust
change is not required. Keep formatting and parsing in the recipe rather than
duplicating them inside a deployment script.

Before implementation, read the relevant parts of:

- [Runner authoring](../../docs/model-forward-pass.md): model preparation,
- [Runner authoring](../../../docs/model-forward-pass.md): model preparation,
shapes, graphs, cache ownership, speculation, and validation.
- [CUDA package boundaries](../../models/cuda/ARCHITECTURE.md): where model and
- [CUDA package boundaries](../../../models/cuda/ARCHITECTURE.md): where model and
hardware code belongs.
- [Distributed execution](../../docs/distributed.md), when using several GPUs.
- [Distributed execution](../../../docs/distributed.md), when using several GPUs.

## 2. Implement and qualify the forward-pass Runner

Implement the reference-correct forward path and checkpoint loading in the owning
model package, importing or implementing the best suitable kernels. Keep cache
allocation, scheduling, prefix ownership and request lifetime in the Engine. Use
existing runtime staging, graph, sampling and distributed machinery. Extend the
shared [model registry](../../deploy/models.py) for hardware presets and runtime
shared [model registry](../../../deploy/models.py) for hardware presets and runtime
construction; do not create a per-model deployment file or a new Modal probe suite.

Compare logits and greedy tokens against an independent trusted reference before
Expand Down Expand Up @@ -87,14 +87,14 @@ requirements, including imported upstream kernels:
is not proof. If tooling lacks this evidence, add reusable instrumentation or
leave that qualification explicitly unresolved.

Read [CUDA graph contracts](../../docs/cuda-graphs.md) and
[kernel artifact caching](../../docs/kernel-cache.md) for the relevant invariants.
Read [CUDA graph contracts](../../../docs/cuda-graphs.md) and
[kernel artifact caching](../../../docs/kernel-cache.md) for the relevant invariants.

## 3. Implement fast, graphed and overlapped speculative decoding

Use the research to select the supported speculator and implement draft generation,
target verification, acceptance, commit and rollback inside the runner. Reuse
[`orbitinfer.backend.speculative`](../../orbitinfer/backend/speculative.py) and the shared
[`orbitinfer.backend.speculative`](../../../orbitinfer/backend/speculative.py) and the shared
CUDA completion/overlap machinery; do not introduce a separate scheduler.

Keep the GPU draft/verify path graphable and prepare its buffers before capture.
Expand Down Expand Up @@ -127,7 +127,7 @@ claiming the full integration is qualified.

Place the runner in the Engine and start its native HTTP server on the agreed GPU
host. This can be a temporary test server; a public Modal deployment is not needed
for validation. Use [`scripts/check_server.py`](../../scripts/check_server.py)
for validation. Use [`scripts/check_server.py`](../../../scripts/check_server.py)
and the normal runtime tests, rather than adding model-specific diagnostic CLIs.

Run ordinary JSON/streaming requests, token usage/logprobs, concurrency, context
Expand Down Expand Up @@ -163,9 +163,9 @@ runner-level evidence from step 2 to close gaps in HTTP observability.
Use the [performance optimization skill](../optimize-orbitinfer-performance/SKILL.md)
for the profiling and iteration loop, including lessons from earlier OrbitInfer work.

Use [`scripts/bench.py`](../../scripts/bench.py), backed by `orbitinfer.bench`, and
[`scripts/plot_bench.py`](../../scripts/plot_bench.py). Read the
[benchmark guide](../../docs/benchmarks.md) for timing and topology semantics.
Use [`scripts/bench.py`](../../../scripts/bench.py), backed by `orbitinfer.bench`, and
[`scripts/plot_bench.py`](../../../scripts/plot_bench.py). Read the
[benchmark guide](../../../docs/benchmarks.md) for timing and topology semantics.
Start with an 8K curve and traces on the target GPU configuration:

```bash
Expand Down Expand Up @@ -213,7 +213,7 @@ commands, natural acceptance, baseline comparisons, and unresolved limitations.

## 6. Deploy to Modal if requested

Use [`deploy/modal_serve.py`](../../deploy/modal_serve.py) with `ORBITINFER_MODEL`
Use [`deploy/modal_serve.py`](../../../deploy/modal_serve.py) with `ORBITINFER_MODEL`
selecting the shared registry entry. Keep checkpoint download/publication separate
from GPU startup; add configuration to the registry instead of a new deployment
script. When deployment is already authorized, proceed after qualification without
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ Include CPU quota/affinity, NUMA placement and NCCL settings when investigating
host or communication behavior. Prefer paired A/B on the same allocation; if the
machine or environment changes, label the comparison accordingly.

Use [`scripts/bench.py`](../../scripts/bench.py) and
[`orbitinfer.bench`](../../orbitinfer/bench.py); extend shared tooling when necessary instead
Use [`scripts/bench.py`](../../../scripts/bench.py) and
[`orbitinfer.bench`](../../../orbitinfer/bench.py); extend shared tooling when necessary instead
of adding a model-specific Modal diagnostic file. Consult `--help` and
[the timing protocol](../../docs/benchmarks.md). For example, on an existing GPU
[the timing protocol](../../../docs/benchmarks.md). For example, on an existing GPU
host with `MODEL_KEY` and `CHECKPOINT` set:

```bash
Expand Down Expand Up @@ -128,7 +128,7 @@ instead of lists, scalar boxing and repeated validation. Send immutable request
media/sampling once and typed mutable step data thereafter. A metadata cache needs
explicit invalidation on changed inputs, layout or lifecycle. Keep Engine-owned
allocation, leases and commit in the Engine; model-specific attention layout and
replay policy belong in the runner. See [the boundary](../../docs/kernel-boundary.md).
replay policy belong in the runner. See [the boundary](../../../docs/kernel-boundary.md).

**Keep context and sampling out of JIT keys.** Logical history lengths, page-table
widths, positions, temperature, top-k/top-p and seeds are runtime data. Audit
Expand All @@ -143,7 +143,7 @@ which rank compiled. If artifacts do not expose compilation reliably, instrument
the compiler/cache-miss entry point or leave that claim unresolved. Warm repeat TTFT,
mtime-only comparisons, and an exhaustive warmup grid do not establish this.
Also verify numerical parity on unseen lengths; see
[kernel caching](../../docs/kernel-cache.md).
[kernel caching](../../../docs/kernel-cache.md).

**Graph useful regions without changing the computation.** GPU kernel paths must
be graphable, but the selected full/breakable policy determines actual capture
Expand All @@ -157,7 +157,7 @@ parity and padding cost across the supported buckets.
Keep graph inputs/outputs at stable addresses, update mutable metadata, and consume
or copy shared output views before replay overwrites them. Shared prefill-region
buffers execute serially; overlapping streams cannot replay them concurrently.
See [graph policy and ownership](../../docs/cuda-graphs.md).
See [graph policy and ownership](../../../docs/cuda-graphs.md).

**Separate matched-batch parity from cross-batch variation.** Compare target-only
and speculative outputs with matched physical batch buckets, prompts, sampling,
Expand All @@ -178,7 +178,7 @@ across boundaries. Preserve collective participation, rank agreement on readines
execution/commit failures, and quarantine after uncertain mutation. Stable decode
may reduce transport work, not erase those agreements. Some persistent kernels
need the whole GPU: obey measured stream dependencies instead of overlapping every
NCCL operation. See [distributed execution](../../docs/distributed.md).
NCCL operation. See [distributed execution](../../../docs/distributed.md).

**Fuse with the real numerical contract.** Preserve accumulator precision, packed
scale/weight strides, sentinel ordering, masks, recurrent state and rollback.
Expand All @@ -200,7 +200,7 @@ counters during timing; retain detailed diagnostics outside it. GPU event storag
must survive until its corresponding completion, rather than being overwritten by
the next replay. Metrics collection/scraping must not synchronize or introduce
collectives. Enable capture-time instrumentation before capture and quantify its
overhead when interpreting small differences. See [metrics](../../docs/metrics.md).
overhead when interpreting small differences. See [metrics](../../../docs/metrics.md).

## Qualify the change and close the loop

Expand All @@ -221,7 +221,7 @@ Do not loosen tolerances to hide graph padding or parallel reduction drift.
Approximate algorithms need an explicit quality comparison and must remain labeled
as approximate; they cannot establish exact full-compute parity.

For serving changes, use [`scripts/check_server.py`](../../scripts/check_server.py)
For serving changes, use [`scripts/check_server.py`](../../../scripts/check_server.py)
against the test endpoint for prefix reuse, varying-context TTFT, concurrent
requests, media/reasoning where supported, stress/recovery and the relevant MMLU
subset. The checker's prefix test verifies telemetry; separately compare
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ Engine overlap; `4c38de5` changes control transport. Detailed graph statistics
cost about 2.4 ms/step in a local diagnostic versus about 0.004 ms for aggregate
counters; that is harness overhead, not faster model math.

Read [recorded V4.1 measurements](../../../docs/benchmarks.md#recorded-v41-measurements)
and [distributed contracts](../../../docs/distributed.md). First inspect repeated
Read [recorded V4.1 measurements](../../../../docs/benchmarks.md#recorded-v41-measurements)
and [distributed contracts](../../../../docs/distributed.md). First inspect repeated
work and transport between graph bodies when device work is already close to a
reference. Preserve exact catalog revalidation and readiness/commit agreement.

Expand All @@ -65,9 +65,9 @@ macOS host against `4775fd3`:

These are CPU microbenchmarks, not measured decode improvements. Preserve ownership,
invalidation, eviction and retirement checks while changing their complexity.
Current anchors: [paged resources](../../../crates/orbitinfer-core/src/resources/paged.rs),
[prefix resources](../../../crates/orbitinfer-core/src/resources/prefix.rs) and
[scheduler](../../../docs/scheduler.md).
Current anchors: [paged resources](../../../../crates/orbitinfer-core/src/resources/paged.rs),
[prefix resources](../../../../crates/orbitinfer-core/src/resources/prefix.rs) and
[scheduler](../../../../docs/scheduler.md).

## 3. Repeated preparation and object conversion accumulate

Expand All @@ -86,7 +86,7 @@ Avoid lists, per-token Python scalars, repeated hashing and copies at a boundary
that already validated the data. Retain ownership until asynchronous copies and
receives finish. `40a44fd` corrected reusable buffers that became inference tensors;
`9ffe6f3` retains pinned storage through asynchronous copies. See
[the runner boundary](../../../docs/kernel-boundary.md).
[the runner boundary](../../../../docs/kernel-boundary.md).

## 4. Runtime geometry prevents first-use compilation stalls

Expand All @@ -100,7 +100,7 @@ compiled-kernel counts remained unchanged.
The full-model EP8 128K prefill experiments measured 9.774 s with 1K chunks,
4.009 s with 4K and 3.691 s with 8K. This combines execution/configuration changes;
it is not an isolated speedup from disabling compilation. See
[benchmark evidence](../../../docs/benchmarks.md#recorded-v41-measurements).
[benchmark evidence](../../../../docs/benchmarks.md#recorded-v41-measurements).

Older precedents: `31ae8d8` keeps only model-fixed trailing cuTile dimensions
static; `76f8665` separates variable prefill from tiny decode convolution;
Expand All @@ -122,7 +122,7 @@ GEMM geometry. GPT-OSS's separate output-projection capture cost 24–30 µs ver
operations. DeepSeek exact buckets avoid a demonstrated 17 → 32 row parity failure.

These are region timings, not whole-model speedups. Read
[graph policy](../../../docs/cuda-graphs.md), including serial shared-workspace use.
[graph policy](../../../../docs/cuda-graphs.md), including serial shared-workspace use.
Maintain graphability, report actual capture scope, and distinguish intentional
prefill partial-chunk fallback from a missing capture. Do not loosen numerical
checks simply to improve a replay counter.
Expand All @@ -147,7 +147,7 @@ does not validate packing, scale axes, strides or accumulation precision.
Recent positive primitive cases (`41ad813`, `5bd97f0`, `d2a4fc3`) include GPT-OSS
router 14.39 → 4.12 µs at eight rows and Inkling 4K convolution 496.74 → 122.11 µs.
They remain primitive claims until a matched end-to-end run demonstrates impact.
Read [primitive checks](../../../docs/benchmarks.md#cuda-model-primitive-checks).
Read [primitive checks](../../../../docs/benchmarks.md#cuda-model-primitive-checks).

## 7. Overlap requires ownership and device feedback, not just an async API

Expand All @@ -174,8 +174,8 @@ read window plus one. `8335f67` adds speculative pre-anchor history. Recompute t
complete read/write envelope when changing chunk size, draft width or capacity.
Graph padding needs reserved scratch storage; masks alone did not prevent the
unscheduled-row corruption fixed by `c79fa23`. See
[the scheduler](../../../docs/scheduler.md) and
[model correctness contract](../../../docs/model-forward-pass.md).
[the scheduler](../../../../docs/scheduler.md) and
[model correctness contract](../../../../docs/model-forward-pass.md).

## 8. Boundary tests expose bugs that ordinary generation misses

Expand Down Expand Up @@ -204,7 +204,7 @@ This does not establish equality to full prefill. Approximate consumer state sta
out of prefix caching and prompt scoring takes the full path. Separate deterministic
qualification compared compact/masked replay and prefix restoration, including
unequal/idle cohorts; that does not turn the algorithm into exact full computation.
See [the model's validation notes](../../../models/cuda/orbitinfer_models_cuda/deepseek_v41/README.md#validation).
See [the model's validation notes](../../../../models/cuda/orbitinfer_models_cuda/deepseek_v41/README.md#validation).

Inspect a reference's actual work before tuning toward its published curve.
Larger chunks alone cannot reproduce this algorithm, and consumer-sized graph
Expand All @@ -226,7 +226,7 @@ cover actual head/channel/state geometry and collective order, not only flags.
Historical GPT-OSS 120B H100 sweeps at `b2f7c3c` measured best tokens/s/GPU of
574.750 for TP1, 488.915 TP2, 286.481 TP4, 988.479 EP2 and 1,013.632 EP4. These
workload-specific curves argue for measuring the mesh; they are not a general
ranking. See [recorded decode sweeps](../../../docs/benchmarks.md#recorded-decode-sweeps).
ranking. See [recorded decode sweeps](../../../../docs/benchmarks.md#recorded-decode-sweeps).

## 11. Communication overlap and NUMA need controlled evidence

Expand All @@ -246,7 +246,7 @@ topology/configuration, not a complete causal diagnosis.

Local evidence is under
`artifacts/benchmarks/2026-10-24-deepseek-v41-b300-ep8-numa/`.
The API commit `3f2893b` and [NUMA guide](../../../docs/numa.md) explain placement
The API commit `3f2893b` and [NUMA guide](../../../../docs/numa.md) explain placement
before child startup, container CPU masks/quotas and fallback reporting. Affinity
does not reserve CPUs or migrate existing pages. Do not claim that API presence
establishes a measured performance improvement.
Expand All @@ -269,7 +269,7 @@ was a candidate opportunity, not a completed speedup. Those sources used
top-k-before-top-p (`580dde5`, `b2f7c3c`). The later fused sampler deliberately
changed to top-p on the full scaled distribution followed by the top-k cap,
retaining cutoff ties, and CDF sampling changed seeded stochastic choices.
See the current [sampling contract](../../../docs/benchmarks.md#shared-sampler-checks)
See the current [sampling contract](../../../../docs/benchmarks.md#shared-sampler-checks)
before choosing an oracle; a faster upstream sampler with another contract is
not a drop-in replacement.

Expand Down
11 changes: 6 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# OrbitInfer agent guide

Read docs/architecture.md and docs/completion-plan.md before making changes.
This repository is an independent full-source adaptation of the snapshot in
third_party/upstream/import.json. Product names, modules and tools use OrbitInfer.
Preserve third-party copyright, licenses, notices and the source manifest.
Product names, modules and tools use OrbitInfer. Preserve third-party copyright,
licenses and notices; source attribution lives in NOTICE and package notices.
External projects are reviewed manually as needed, with no vendored source
manifest or automatic synchronization. Project skills live in .agents/skills/.

## Owners

Expand Down Expand Up @@ -37,8 +38,8 @@ Use the pinned Rust toolchain and Python 3.12 development environment. Run
`uv run --no-sync python scripts/check.py`. On Apple Silicon, `uv sync --locked`
also builds the MLX extension and installs all model workspace packages.
The script runs formatting, Clippy, Rust tests, Python tests, the installed CPU
example and source/package boundary controls. GPU skips are reported and never
counted as hardware evidence. Write installed gates before claiming GPU support.
runtime check and source/package boundary controls. GPU skips are reported and
never counted as hardware evidence. Write installed gates before claiming GPU support.

Keep changes with their owner. Record source-import adaptations separately
from behavior changes. Tests should force lifecycle interleavings with events
Expand Down
3 changes: 2 additions & 1 deletion NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ https://github.com/ekzhang/resin

Files are renamed and adapted as described in docs/source-import.md.
Original attribution follows unchanged; its paths refer to the upstream tree.
The local paths are mapped in third_party/upstream/import.json.
Local Python and crate prefixes use orbitinfer and orbitinfer_models_*;
the original attribution text below is preserved.

Resin

Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ OrbitInfer also has optional HTTP serving and Prometheus metrics.
> inference engine, please use SGLang or vLLM. Read on if you want to hack on
> LLM inference systems, new models or hardware.

The complete source import includes CUDA model runners, MLX/Metal runners,
The repository includes CUDA model runners, MLX/Metal runners,
Torch reference models, serving tools, benchmarks and tests. The first planned
installed target is DeepSeek-V4.1-Flash on SM100. No complete GPU model profile
has been qualified for OrbitInfer. See the [plan](docs/completion-plan.md) and
[source import record](docs/source-import.md). Interface changes are recorded in
[source and license notes](docs/source-import.md). Interface changes are recorded in
the [change notes](docs/releases.md).

## Features
Expand All @@ -50,9 +50,9 @@ Start with the two-page [architecture overview](docs/architecture.md).
Either fork the repository or import `orbitinfer` and `orbitinfer_models_cuda` as standalone
packages. Then use the following skills:

- [**add-orbitinfer-model**](skills/add-orbitinfer-model/SKILL.md) -
- [**add-orbitinfer-model**](.agents/skills/add-orbitinfer-model/SKILL.md) -
Create an inference engine for a new model from scratch, using OrbitInfer.
- [**optimize-orbitinfer-performance**](skills/optimize-orbitinfer-performance/SKILL.md) -
- [**optimize-orbitinfer-performance**](.agents/skills/optimize-orbitinfer-performance/SKILL.md) -
Iterate on your inference engine to improve performance with an "autoresearch"
workflow, diagnose bottlenecks and validate improvements, drawing on past
learnings.
Expand Down
Loading
Loading