refactor(runtime)!: adopt the modern Infini stack by voltjia · Pull Request #506 · InfiniTensor/InfiniLM

voltjia · 2026-07-21T13:30:37Z

Summary

Rebase and preserve the InfiniCore-to-InfiniLM runtime migration on InfiniLM main at 80bb09ecebc9aabf198b9b866a89456bca1df946.
Synchronize the effective InfiniCore changes that landed after the migration source diverged, while retaining InfiniLM's stronger graph ownership, cancellation, and allocation-lifetime behavior.
Replace legacy/deprecated operator paths with current InfiniRT, InfiniOps, and InfiniCCL APIs.
Enable the currently validated NVIDIA and Moore execution paths, including explicit FlashAttention, paged segmented graph replay, tensor parallelism, and real-weight model coverage.
Fix Moore communicator initialization so short-lived worker threads select the native device without constructing and tearing down thread-local InfiniCore runtimes.

Current head: 5de9a0283ec66ed30006372c535a75983407cdfb.

The branch contains 43 commits on top of current main and remains mergeable.

Related to InfiniTensor/InfiniCore#1373.

Migration and Runtime Changes

InfiniLM owns the migrated runtime context, tensors, graph integration, operator adapters, distributed wrappers, and Python bindings.
Segmented graph replay keeps capture-unsafe MhaKVCache work in host segments while the remaining operators run in device-graph segments; TP paged decode no longer falls back to a wholly eager engine.
Static graph cache metadata stays device-resident and observes in-place replay updates.
The deprecated causal-softmax backend is removed from execution; the adapter composes current triangular-mask and softmax operations.
Linear bias is implemented as Gemm with beta=0 plus broadcast Add, including correct row/column-parallel placement and pre-transposed weights.
The dense factory enables validated Baichuan, ChatGLM, FM9G, GLM4, InternLM3, Llama, MiniCPM/MiniCPM4, Qwen2, and Qwen3 families.
Moore accepts its native device name and selects a dedicated 23-operator InfiniOps manifest; NVIDIA retains the 24-operator manifest including sampling.
Communicator-init threads call InfiniRT's native backend/device selection directly before infinicclCommInitRank. This avoids heap corruption caused by destroying a short-lived InfiniCore Runtime in each worker thread.

Current Upstream Stack

InfiniCore #1406: 55cfe5e6761c4ebb5e8eb77b65301479a7b6032c
InfiniRT: 0cdbb16967e15f2e055dea1ec9641617bf3b6cf6
InfiniOps #819: 70e50081f181d8a3c4f9a6226c8524a648ed90c4
InfiniOps #962: c2d76051ca21774f0b515dfd8afe59537e200ac2
InfiniCCL 支持海光运行 #57 -> [BUG] test_ppl.py run fail #58 -> Feature/use logsoft max in ppl #59 API stack: e8ccc0cb23ca5b1b1d63be29ba61ea031de72807
InfiniCCL [DEV] 修改 lanch_server, 支持 vllm benchmark 测试补全文本 #69 Moore architecture fix: f74ce8033ce601dbb0bbf31277755d033693768c

#506 uses AllGather, Send, and Recv, so #57/#58/#59 remain required. #69 is an independent master-based prerequisite; validation combined its MARCH_TYPE=310 behavior with the API stack.

Newly Validated Moore Capability

InfiniOps #819/#962 provide the missing paged prefill/decode attention closure. InfiniCCL #69 passes the actual MUSA architecture into MCCL so S5000's existing BF16 collective support is visible.

The selected formal matrix passed 13/13 commands:

IDs	Workloads	Result
D01-D03	9g-8B explicit FlashAttention + graph, batches 1/4/16	PASS
D05	9g-8B paged attention + graph, batch 32	PASS
D14	Qwen3-32B paged FlashAttention + graph, TP4	PASS
D15	Llama-3.2-3B paged FlashAttention + graph	PASS
D17	Baichuan2-7B paged FlashAttention + graph, TP2	PASS
D18	ChatGLM3-6B paged FlashAttention + graph	PASS
D19	InternLM3-8B paged FlashAttention + graph	PASS
D20, D22	MiniCPM4-8B generation and benchmark graph paths	PASS
D21	GLM-4-9B paged FlashAttention + graph	PASS
D26	MiniCPM4-8B Eagle speculative decoding	PASS

Every row reported segmented graph execution with host_segments > 0; none used whole-engine eager fallback. D14 and D17 additionally prove TP4/TP2 communicator setup and BF16 AllReduce. D14 completed in 191.391s and D17 in 74.904s after the two single-purpose fixes.

The formal operator smoke also passed D64/D128 paged prefill/decode coverage: 4 passed, 8 deselected.

All selected Moore commands are greedy/default sampling. Non-greedy sampling remains gated because the Moore manifest does not yet include a supported top_k_top_p_sampling_from_logits implementation.

Preserved NVIDIA Validation

Real-weight two-token smokes passed for Llama-3.2-3B, FM9G 9g-8B, Baichuan2-7B, ChatGLM3-6B, InternLM3-8B, GLM-4-9B, MiniCPM4-8B, Qwen2-compatible FM9G-70B TP8, and Qwen3-0.6B.
Explicit FlashAttention passed eager and segmented graph paths for Qwen3 and Llama, including Qwen3-32B BF16 TP4.
Qwen3 TP2 paged segmented graph passed batch sizes 1 and 16 without CUDA_LAUNCH_BLOCKING.
Bias, pre-transposition, TP1/TP2, and paged FlashAttention combinations passed the existing focused matrix.

Gates Intentionally Retained

FlashInfer attention: the selected linked provider implements sampling, not an attention backend.
Compressed-tensors W8A8, AWQ/GPTQ, MXFP4/Quark, and INT8 KV cache: InfiniLM does not yet have complete execution chains for these formats.
GPT-2: the required NVIDIA LayerNorm provider is unavailable in the selected closure.
Mistral: the current implementation does not consume sliding_window, so a short prompt is not sufficient semantic validation.
Mamba, Qwen3-Next, and Qwen3.5: required causal-convolution/selective-scan/gated-delta provider chains are incomplete.
Complete MoE and multimodal model-specific paths remain out of scope.
Moore non-greedy sampling remains unsupported as described above.

Verification

Static contract suite: 83/83 passed.
Build-script unit suite: 20/20 passed.
Current GitHub Check Format and Ruff jobs: passed.
git diff --check: passed.
Independent Moore InfiniLM extension build: passed in 118.662s.
Two-device and four-device BF16 AllReduce probes: passed with correct numeric outputs.
InfiniCCL [DEV] 修改 lanch_server, 支持 vllm benchmark 测试补全文本 #69: explicit mixed-architecture and native-detection builds passed; CTest 2/2 in both configurations; default examples build passed; two-device BF16 AllReduce passed.
The InfiniOps pre-rebase hardware-tested commits map one-for-one to the current #819/#962 commits under git range-diff.

Type of Change

Landing Order

InfiniOps #819.
InfiniOps #962, then retarget it from #819 to master.
InfiniCCL [DEV] 修改 lanch_server, 支持 vllm benchmark 测试补全文本 #69 can land independently.
InfiniCCL 支持海光运行 #57 -> [BUG] test_ppl.py run fail #58 -> Feature/use logsoft max in ppl #59; rebase the stack onto the [DEV] 修改 lanch_server, 支持 vllm benchmark 测试补全文本 #69-updated master before updating the final component pin.
InfiniCore #1406 updates its InfiniCCL and InfiniOps component pins to the merged heads.
This PR.

This PR remains draft until the upstream component PRs and final pins land, but it is ready for code review against the dependency order above.

BREAKING CHANGE: legacy InfiniCore runtime and inference APIs are removed in favor of InfiniRT, InfiniOps, and the migrated InfiniLM runtime.

Carry segmented P2P graph replay, graph-aware tensor initializers, runtime compatibility helpers, and long-context chunking into the InfiniLM-owned runtime. Preserve the stronger migrated graph ownership, cancellation, and allocation lease semantics while reconciling current InfiniLM attention behavior.

…ention feat(cambricon): enable tensor-parallel flash attention

* feat(iluvatar): enable modern Infini stack * feat(iluvatar): enable canonical attention adapters * feat(iluvatar): enable flash attention backend * fix(iluvatar): address review feedback * fix(model-loading): mmap zip-format pytorch checkpoints * fix(bench): reuse paged cache during warmup

wooway777 · 2026-09-08T08:54:59Z

        throw std::invalid_argument("infinilm::config::ConfigFactory::createConfig: Unsupported model config type: " + model_type);
    }

+    static const std::unordered_set<std::string> kModernModelTypes{


wooway777 · 2026-09-08T08:57:06Z

    const std::string quant_method = quantization_config.value("quant_method", "");

-    // Determine the quantization scheme from the JSON config
    if (quant_method == "compressed-tensors") {


为啥在这儿就都不支持了，我感觉是不是到了算子调用再拦住比较好。不然回头把东西补回来的时候又要一串一串改。

以及之前的全量测试好像确实忘记加量化相关的东西了

Register Hygon with the canonical InfiniOps bridge, greedy sampling, RoPE cache, and FlashAttention adapters. Extend the integration builder with Hygon architecture and RCCL wiring. Keep the platform-specific InfiniOps operator selection outside the repository and require it through --operator-config, matching the Iluvatar workflow.

* perf(runtime): make stream access constant time * perf(mlp): consume packed gate-up output * perf(ops): cache default infiniops implementation * perf(paged): reuse decode metadata buffers * perf(cache): skip unused paged cache scale upload * perf(graph): avoid eager output snapshots * perf(inference): optimize reviewed execution paths * perf(speculative): reuse graphs for token verification * perf(paged): capture ChatGLM short decode graph * test(runtime): cover optimized inference paths * style: format optimized inference paths * refactor(graph): centralize reviewed profile checks

wooway777 · 2026-09-09T07:15:53Z

        throw std::invalid_argument("infinilm::config::ConfigFactory::createConfig: Unsupported model config type: " + model_type);
    }

+    static const std::unordered_set<std::string> kModernModelTypes{


所以现在是要求显示列举支持的模型类型了么？本来应该是不需要的

wooway777 · 2026-09-09T07:23:31Z

同上，这样以后恢复起来岂不是很费劲。由缺失算子支持造成的问题还是建议直接暴露在算子层，而不是直接从基建里把痕迹都移除了

voltjia force-pushed the refactor/adopt-modern-infini-stack branch 3 times, most recently from 7c36e2f to 077867b Compare August 13, 2026 15:26

voltjia mentioned this pull request Aug 25, 2026

refactor!: reduce InfiniCore to a component manifest InfiniTensor/InfiniCore#1406

Open

voltjia added 25 commits August 25, 2026 12:53

refactor(runtime)!: adopt modern Infini stack

04fcda0

BREAKING CHANGE: legacy InfiniCore runtime and inference APIs are removed in favor of InfiniRT, InfiniOps, and the migrated InfiniLM runtime.

docs: define Infini stack repository boundaries

e9496ca

refactor: own Infini stack integration build

05c92ba

docs: document the InfiniLM stack build

bca98df

ci: validate the modern NVIDIA stack

f46c074

fix: keep static graph cache metadata dynamic

dab847b

refactor(ops): use canonical InfiniOps APIs

d78dacb

fix(build): select required linked InfiniOps providers

d49427f

fix(build): limit compiled models to Qwen3

f128747

feat(distributed): add eager point-to-point wrappers

0cbc125

feat(ops): restore prepacked linear execution

9f40398

fix(ops): match canonical Gemm call schema

28cc18b

fix(sampling): support scalar sample outputs

7dd8ac6

fix(nn): pass RoPE output handles by value

80af271

fix(attention): include variable-length MHA declaration

3af4355

fix(engine): use local static graph assertions

0cb41af

fix(config): reject unsupported Qwen3 linear bias

6fcd5b7

test: cover modern runtime contracts

4e36168

docs: update modern Qwen3 support boundary

354372a

refactor(build): select InfiniOps implementations from JSON

1a89775

style: format InfiniOps attention adapters

2c2d94b

fix(graph): update paged replay metadata through tensors

baf3315

fix(graph): fall back to eager paged decode under tensor parallelism

1551767

fix: restore modern InfiniCore API compatibility

c76b105

voltjia force-pushed the refactor/adopt-modern-infini-stack branch from 9439ea6 to 9d64e14 Compare August 25, 2026 05:12

voltjia added 14 commits August 25, 2026 13:59

style: format migrated sources

5964127

fix(runtime): keep receive buffers mutable

e4a549b

fix(build): exclude unsupported MXFP4 path

1ed454f

fix(packaging): preload Torch shared libraries

33f0409

refactor(runtime): replace deprecated causal softmax backend

89a0fdb

style: format causal softmax adapter

df6f2a3

feat(runtime): unlock validated InfiniOps capabilities

8c2d3f5

fix(runtime): select available InfiniOps implementations

57e9d01

fix(runtime): enable available Moore InfiniOps paths

24f9d46

fix(runtime): accept native MetaX device name

7f958b7

fix(core): validate Cat inputs against tensor rank

1de38df

feat(models): enable validated MiniCPM Eagle path

63147e7

style(runtime): group ModelRunner imports

9eb004a

fix(graph): enable safe paged decode capture under TP

dcfbebc

voltjia force-pushed the refactor/adopt-modern-infini-stack branch from 03d7533 to dcfbebc Compare August 31, 2026 07:07

voltjia and others added 5 commits September 2, 2026 13:19

feat(moore): enable modern flash attention

5e9be02

style(moore): format attention integration

d2766a2

build(moore): configure the modern stack

4093b0b

feat(cambricon): enable tensor-parallel flash attention

bae5a89

fix(distributed): avoid transient runtime teardown

5de9a02

voltjia force-pushed the refactor/adopt-modern-infini-stack branch from 2a44095 to 5de9a02 Compare September 2, 2026 14:17

voltjia and others added 2 commits September 3, 2026 11:11

Merge pull request #556 from InfiniTensor/feat/cambricon-tp-flash-att…

88e4c6e

…ention feat(cambricon): enable tensor-parallel flash attention

wooway777 reviewed Sep 8, 2026

View reviewed changes

gongchensu and others added 3 commits September 9, 2026 09:41

feat(ascend): enable InfiniOps flash attention (#558)

6966281

wooway777 reviewed Sep 9, 2026

View reviewed changes

Navigation Menu

Uh oh!

refactor(runtime)!: adopt the modern Infini stack - #506

refactor(runtime)!: adopt the modern Infini stack#506
voltjia wants to merge 49 commits into
mainfrom
refactor/adopt-modern-infini-stack

voltjia commented Jul 21, 2026 •

edited

Loading

Uh oh!

wooway777 Sep 8, 2026

Uh oh!

wooway777 Sep 8, 2026

Uh oh!

wooway777 Sep 9, 2026

Uh oh!

wooway777 Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

voltjia commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Migration and Runtime Changes

Current Upstream Stack

Newly Validated Moore Capability

Preserved NVIDIA Validation

Gates Intentionally Retained

Verification

Type of Change

Landing Order

Uh oh!

wooway777 Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

wooway777 Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

wooway777 Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

wooway777 Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

voltjia commented Jul 21, 2026 •

edited

Loading