feat(iluvatar): add native canonical attention providers - #975
Open
gongchensu wants to merge 2 commits into
Open
feat(iluvatar): add native canonical attention providers#975gongchensu wants to merge 2 commits into
gongchensu wants to merge 2 commits into
Conversation
gongchensu
force-pushed
the
feat/iluvatar-canonical-attention
branch
from
September 4, 2026 01:43
6f93119 to
8f0770d
Compare
voltjia
previously approved these changes
Sep 4, 2026
gongchensu
force-pushed
the
feat/iluvatar-canonical-attention
branch
from
September 4, 2026 06:17
8f0770d to
572f6b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FlashAttnVarlenFuncandFlashAttnWithKvcacheAPIs, reusing the existing native paged prefill and decode kernels.ReshapeAndCacheFlash.CUDA::cuda_driverimported target; the backend only uses the CUDA runtime and cuBLAS APIs.Motivation
Modern InfiniLM dispatches paged prefill and decode through the canonical
InfiniOps
FlashAttnVarlenFuncandFlashAttnWithKvcacheAPIs. Before thischange, Iluvatar only had the legacy
_infinilmcompatibility providers, sothe canonical calls had no active Iluvatar implementation.
The canonical varlen API supplies cumulative KV offsets such as
[0, 130, 430], while the reused native paged-prefill kernel expects onelength per sequence, such as
[130, 300]. This change lets the kernel deriveadjacent differences directly, avoiding an additional device allocation and
conversion kernel during InfiniRT graph execution.
The Iluvatar CMake configuration also referenced
CUDA::cuda_driver, which isnot provided by the CoreX
FindCUDAToolkitpackage. InfiniOps does not use theCUDA Driver API on this backend, so only
CUDA::cudartandCUDA::cublasarerequired.
Validated with static, paged-attention plus InfiniRT graph, and explicit
FlashAttention plus InfiniRT graph inference using the 9G-8B model on
Iluvatar BI-V150.
N/A - no linked issue.
Type of Change
feat— new feature / new operator / new platformfix— bug fixperf— performance improvement (no behavioral change)refactor— code restructuring without behavior changetest— adding or fixing tests onlydocs— documentation onlybuild/ci— build system or CI configurationchore— tooling, formatting, or other non-code changes!in the Conventional Commits prefix or aBREAKING CHANGE:footer)Platforms Affected
WITH_CPU)WITH_NVIDIA)WITH_ILUVATAR)WITH_METAX)WITH_CAMBRICON)WITH_MOORE)WITH_ASCEND)WITH_TORCH)WITH_HYGON) — sharedReshapeAndCacheFlashtemplateWITH_MARS) — shared paged-attention and cache templatesWITH_THEAD) — sharedReshapeAndCacheFlashtemplateSmoke Test Result
Test Results on Supported Platforms
Full `pytest` output (optional)
Benchmark / Performance Impact
Notes for Reviewers