Skip to content

fix(kunlun): memset fallback, graph-op destructor UB, missing selectlast_token_hidden kernel - #1544

Open
rubik-hua wants to merge 1 commit into
InfiniTensor:mainfrom
rubik-hua:kunlun-graph-bugfix
Open

fix(kunlun): memset fallback, graph-op destructor UB, missing selectlast_token_hidden kernel#1544
rubik-hua wants to merge 1 commit into
InfiniTensor:mainfrom
rubik-hua:kunlun-graph-bugfix

Conversation

@rubik-hua

Copy link
Copy Markdown

Kunlun inference was broken by three independent issues; this commit
fixes all of them.

  1. Kunlun memset support (src/infinirt/kunlun/infinirt_kunlun.cc)

memsetDevice/memsetDeviceAsync returned INFINI_STATUS_NOT_IMPLEMENTED.
This surfaced after fafd723 (issue/1170), which made Tensor::zeros
actually zero the buffer via memset; previously the buffer from empty()
was left untouched, so the stub was never exercised. XRE exposes no
memset API, so emulate it with chunked host-to-device copies from a
16 MiB host staging buffer, keeping host memory bounded regardless of
fill size. The async variant falls back to the sync copy, consistent
with mallocAsync/freeAsync already falling back to sync XRE calls.

  1. Uninitialized DispatchableGraphOperator members (include/infinicore/graph/graph.hpp)

planned_meta_/runner_/deleter_ had no default initializers. When a
derived constructor throws (e.g. an operator unsupported on the current
device), stack unwinding still runs the base destructor, which read the
garbage deleter_ and jumped through it - a silent SIGSEGV that masked
the real "operator not supported" error and made the process exit with
no traceback. Default-initialize all three to nullptr so the existing
null check in the destructor skips cleanup safely. No functional change
for already-supported devices: every constructor overwrites the three
members via INFINICORE_GRAPH_OP_DISPATCH before they are read.

  1. Kunlun kernel for select_last_token_hidden for already-supported devices: every constructor overwrites the three members via INFINICORE_GRAPH_OP_DISPATCH before they are read.

  2. Kunlun kernel for select_last_token_hidden

Add an XPU kernel (f16/bf16/f32) and register it in the infiniop
dispatcher, unblocking the static-cache path on Kunlun. The build picks
the new files up through the existing ops//kunlun/.xpu glob; no build
change needed. Note that paged_caching/paged_attention still have no
Kunlun kernels, so --enable-paged-attn remains unsupported on this
platform for now.

Also fix an out-of-scope index in a cat.cc assert (tensors[i] -> tensor,
where no such i exists in that loop), which broke debug builds.

当前最新代码仓上昆仑芯推理报错:
image

还会有跟graph相关的coredump,堆栈如下:
image
image

修复后验证:
python examples/test_infer.py --device kunlun --model=/mnt/geogpt-doc-new/default/infinilm-models/Qwen3-0.6B --tp 1 --batch-size 1 --prompt "who are you"

image

@rubik-hua
rubik-hua requested a review from a team September 3, 2026 03:41
@wooway777

Copy link
Copy Markdown
Collaborator
image agent的结论,我没机器验证不了

@wooway777

Copy link
Copy Markdown
Collaborator

另外static cache的推理流程中有cpu交互,预期就是不能跑graph。需要补充paged attention或者flash attention相关的支持。

@rubik-hua

Copy link
Copy Markdown
Author

另外static cache的推理流程中有cpu交互,预期就是不能跑graph。需要补充paged attention或者flash attention相关的支持。

paged attention或者flash attention 我在实现,后面我单独提PR。 不过没编译graph,运行也没指定 --enable-graph情况下,会coredump在graph代码中,这个肯定是bug,需要修复的。 上面agent给的检视意见我会过一遍修复掉。

…last_token_hidden kernel

  Kunlun inference was broken by three independent issues; this commit
  fixes all of them.

  1. Kunlun memset support (src/infinirt/kunlun/infinirt_kunlun.cc)

  memsetDevice/memsetDeviceAsync returned INFINI_STATUS_NOT_IMPLEMENTED.
  This surfaced after fafd723 (issue/1170), which made Tensor::zeros
  actually zero the buffer via memset; previously the buffer from empty()
  was left untouched, so the stub was never exercised. XRE exposes no
  memset API, so emulate it with chunked host-to-device copies from a
  16 MiB host staging buffer, keeping host memory bounded regardless of
  fill size. The async variant falls back to the sync copy, consistent
  with mallocAsync/freeAsync already falling back to sync XRE calls.

  2. Uninitialized DispatchableGraphOperator members
     (include/infinicore/graph/graph.hpp)

  planned_meta_/runner_/deleter_ had no default initializers. When a
  derived constructor throws (e.g. an operator unsupported on the current
  device), stack unwinding still runs the base destructor, which read the
  garbage deleter_ and jumped through it - a silent SIGSEGV that masked
  the real "operator not supported" error and made the process exit with
  no traceback. Default-initialize all three to nullptr so the existing
  null check in the destructor skips cleanup safely. No functional change
  for already-supported devices: every constructor overwrites the three
  members via INFINICORE_GRAPH_OP_DISPATCH before they are read.

  3. Kunlun kernel for select_last_token_hidden
  for already-supported devices: every constructor overwrites the three
  members via INFINICORE_GRAPH_OP_DISPATCH before they are read.

  3. Kunlun kernel for select_last_token_hidden

  Add an XPU kernel (f16/bf16/f32) and register it in the infiniop
  dispatcher, unblocking the static-cache path on Kunlun. The build picks
  the new files up through the existing ops/*/kunlun/*.xpu glob; no build
  change needed. Note that paged_caching/paged_attention still have no
  Kunlun kernels, so --enable-paged-attn remains unsupported on this
  platform for now.

  Also fix an out-of-scope index in a cat.cc assert (tensors[i] -> tensor,
  where no such i exists in that loop), which broke debug builds.
@rubik-hua

Copy link
Copy Markdown
Author

image agent的结论,我没机器验证不了

这三个问题已经修改

@wooway777

Copy link
Copy Markdown
Collaborator
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants