Unified resource management for heterogeneous AI infrastructure

English · 한국어

Getting started · Architecture · Optimus Prime · Components · Status · Docs

---

AI-RM is the integration hub for the AI Semiconductor Integrated Resource Management research project (AI 반도체 통합 자원관리 기술 개발). It connects GPU, NPU (FuriosaAI RNGD, Rebellions ATOM), CPU and GB10 unified-memory research to Kubernetes + llm-d through shared contracts, pinned component versions and reproducible measurements. Inference-engine technology is being consolidated into Optimus Prime, with FASOP as its planning element.

Onggi is the artifact storage and delivery layer: it deduplicates model weights and precompiled RNGD/ATOM variants, restores missing chunks through node caches and peers, and exposes read-only CSI volumes. The serving runtime owns device-memory delta loading and prepared graph selection; compatible precompiled variants are still required when batch or sequence length changes. Automatic compile-farm and placement-driven prefetch integration remain planned. See the Onggi component card.

✨ What it brings together

Describe every device Discover GPUs, NPUs, CPUs and unified-memory systems and publish their capabilities through one AcceleratorProfile model.	Move KV and experts by name NPU↔NPU KV transfer, NPU↔host expert offload and CPU↔GPU unified memory behind shared KVTransport and memory-tier contracts.	One engine target Prefill/decode, KV and offload paths converge in Optimus Prime; FASOP plans parallelization and placement.
Kubernetes + llm-d native Gateway, InferencePool/EPP, model storage and monitoring through the extension points llm-d already has.	One metric language TTFT, TPOT, tok/s, power and MFU under the same airm_* names in every layer, from the console to the KPI table.	Connected by contracts Components never import each other. Each registers an rm-core adapter, checked by conformance tests.

🏗 Architecture

Five technical layers

Layer	Responsibility	Components
L5	Observability & operations	lmd-top · rm-bench
L4	Cluster & placement	hetero-llm-d · rm-controller · FASOP adapter (planned)
L3	Unified engine	Optimus Prime · rngd-pd (migration source) · npu-train
L2	Memory & transport	hetero-npu-p2p · npu-offload · onggi · duet
L1	Accelerator abstraction	rm-accel

More: architecture · interfaces · engine consolidation plan

Important

Integration in progress. Optimus Prime is the planned unified engine in aicomp; its integration branch and adapters are not active yet, and FASOP's inference planning is future work. Existing repositories remain the pinned, executable sources. See status for what is implemented versus planned.

🚀 Quick start

Explore without accelerator hardware or private-component access:

git clone https://github.com/ai-computing/ai-rm.git && cd ai-rm
python3 tools/airm.py list            # components · layers · versions · deployability
python3 tools/check_sensitive.py .    # scan for internal addresses, tokens and keys

Fetch components, install and test

Initialize the public components by path (a --recursive clone also attempts the private duet), then:

make doctor                     # prerequisites and missing pieces
make setup                      # install components (LAYER=l2-memory to limit)
make test                       # quick tests that need no accelerator
python -m rm_core.conformance   # contract checks: PASS / SKIP (no hardware) / FAIL

Full steps: getting started.

Deploy to a Kubernetes + llm-d cluster

mkdir -p deploy/clusters/local
cp deploy/clusters/example/cluster.example.env deploy/clusters/local/cluster.local.env
$EDITOR deploy/clusters/local/cluster.local.env    # site values, never tracked
make deploy CLUSTER=local DRY_RUN=1                 # review before applying

See deployment.

🧩 Ecosystem

	Area	Components
🚂	Engine & planning	Optimus Prime · opt_prime · FASOP · fasop
🟣	Resource discovery	rm-accel
🔵	Engine migration sources	rngd-pd · hetero-npu-p2p · npu-offload
🏺	Model & artifact store	onggi (P2: store, cache, peers, delta apply, CSI)
🟡	Cluster integration	hetero-llm-d · rm-controller · rm-core contracts
🟠	Operations & measurement	lmd-top · rm-bench
🟢	Training & search workloads	npu-train · duet (private, optional)

Responsibilities, source roles and integration tasks: component directory.

🔌 How it connects

Components implement an rm-core contract, register it as an entry point and are selected by name, so switching a KV path is a configuration change rather than a code change.

# hetero-npu-p2p/pyproject.toml
[project.entry-points."airm.kv_transports"]
p2p = "hetero_p2p.airm:P2PTransport"

from rm_core import registry
transport = registry.load("airm.kv_transports", "p2p")   # kvTransport: p2p | shm | loopback

Contract	Entry-point group	Layer
AcceleratorProvider	airm.accelerator_providers	L1
KVTransport	airm.kv_transports	L2
Workload runner	airm.workloads	L3
Placement	airm.placements	L4
Metrics (airm_*)	—	all

Details: interfaces · integration guide · ai-rm.yaml schema

🗺 Roadmap

• Five-layer layout, pinned submodules and setup tooling (list · doctor · setup · test · deploy · check)
• rm-core v0.1 contracts, reference implementations, conformance checks and AcceleratorProfile CRD
• Component adapters for protocol v0.1, verified together (status)
• Optimus Prime integration branch: PD, KV-transport and expert-offload modules from the migration sources
• FASOP inference planning and a reviewed llm-d placement adapter
• First end-to-end run on the live cluster: L1 placement → llm-d → PD serving → measurement (results)
• Real-trace replay and workload characterization (traces); rm-bench and rm-controller in-tree under packages/
• Placement controller, first version: topology-aware placement written to airm-placement (dry-run verified on the cluster)
• Serving pods that follow the placement decisions; load-aware placement; RNGD→ATOM decode and P2P with live buffers end to end
• Umbrella chart, simulated cluster and power / MFU evidence from every run
• Protocol v0.2: bind live buffers (MemoryRegion) to KV transfers

Closing the gaps

Compared with llm-d, Dynamo, vLLM + NIXL/LMCache, Mooncake and HexGen-2/Helix, ai-rm is the only stack serving prefill/decode across NPUs from different vendors. Seven gaps remain; engine-side parts are feature requests on aicomp, the rest is done here. Full plan: roadmap.

	Gap	Plan
W1	Serving surface	Streaming, chat API, sampling, cancel (#41–#43); GuideLLM and MLPerf on the PD path
W2	Scale and maturity	Batched decode (#44), S≥2048, node image + umbrella chart, sim cluster in CI, multi-node pools
W3	KV transport standards	NIXL backend for RNGD↔ATOM, vLLM KV connector, layer-wise and GB-scale transfer (#47, #48)
W4	Cache reuse	KV block events and prefix reuse (#46), EPP prefix-cache routing, host-DRAM KV tier
W5	Control loop	Placements enforced, P:D rebalancing, re-placement on failure, KEDA on /metrics (#45, #49)
W6	Formal scheduling	optimal placement over card types and KV link cost; FASOP inference cost model (#50)
W7	Evidence	Same-budget baselines, tok/J in every run, paper drafts

📚 Documentation

Goal	Guides
🧭 Understand the project	Overview · Architecture · Research scenarios · Glossary
🔧 Connect an implementation	Engine plan · Interfaces · Integration guide
📊 Run and evaluate	Getting started · Deployment · KPI and evidence
📍 Check progress	Status · Roadmap · Planning configuration

All documentation · 한국어 문서

🤝 Contributing

See CONTRIBUTING.md for implementation boundaries, documentation and validation, and use GitHub Issues for questions and proposals.

Note

This repository is public. Keep site configuration, internal addresses and credentials out of tracked files; site values live only in the untracked cluster.local.env. See the security guide.

📄 License

BSD-3-Clause for this integration repository. Each component retains its own license.

_{Built by the ai-computing team · ETRI}