Repository navigation
Copying HostBuffer is broken #458
Description
Device buffers:
julia> c = oneVector{Int}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.DeviceBuffer}:
1
julia> d = oneVector{Int}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.DeviceBuffer}:
1
julia> Array(c .+ d)
1-element Vector{Int64}:
2Host buffers:
julia> c = oneVector{Int,oneL0.HostBuffer}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.HostBuffer}:
1
julia> d = oneVector{Int,oneL0.HostBuffer}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.HostBuffer}:
1
julia> Array(c .+ d)
1-element Vector{Int64}:
0Mixed buffers:
julia> c = oneVector{Int,oneL0.HostBuffer}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.HostBuffer}:
1
julia> d = oneVector{Int,oneL0.DeviceBuffer}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.DeviceBuffer}:
1
julia> Array(c .+ d)
1-element Vector{Int64}:
1On PVC hardware, this triggers an UNKNOWN_ERROR in subsequent command list submissions:
julia> c = oneVector{Int,oneL0.HostBuffer}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.HostBuffer}:
1
julia> d = oneVector{Int,oneL0.DeviceBuffer}([1])
1-element oneArray{Int64, 1, oneAPI.oneL0.DeviceBuffer}:
1
julia> Array(c .+ d)
Array(c .+ d) = [1]
1-element Vector{Int64}:
1
# synchronize doesn't trap this error
julia> oneAPI.synchronize()
# new command list submissions does
julia> a = oneVector([1])
ERROR: ZeError: unknown or internal error (code 2147483646, ZE_RESULT_ERROR_UNKNOWN)
Stacktrace:
[1] throw_api_error(res::oneAPI.oneL0._ze_result_t)
@ oneAPI.oneL0 ~/oneAPI/lib/level-zero/libze.jl:8
[2] check
@ ~/oneAPI/lib/level-zero/libze.jl:19 [inlined]
[3] zeCommandQueueExecuteCommandLists
@ ~/oneAPI/lib/utils/call.jl:24 [inlined]It's idiotic that the driver doesn't report this earlier...
Activity
[-]HostBuffer broadcast is broken[/-][+]HostBuffer use in kernels is broken[/+][-]HostBuffer use in kernels is broken[/-][+]Copying HostBuffer is broken[/+]maleadt commentedon Aug 13, 2024
This apparently extends to all uses of host buffers:
The API trace looks correct though:
Note how there's no explicit API memcpy, as we should just be able to use a CPU-based
memcpybecause this memory is readable on the CPU. This appears to not be true, as removing ourmemcpyoptimization and always using an explicitzeCommandListAppendMemoryCopywith host memory does return the correct result.C reproducer:
This reads
kernel.spv, attached here: kernel.zip@kballeda @pengtu Can you explain what the issue is here? Shouldn't it be possible to read
zeMemAllocHost-allocated using a simple CPU-basedmemcpyafter having synchronized the command queue we issued a kernel on?Also note that on PVC we encounter a command list submission failure first, but let's debug the
memcpyproblem first.kballeda commentedon Aug 28, 2024
@maleadt File a ticket on intel compute runtime on this. We will internally follow up on this.
maleadt commentedon Oct 19, 2026
The broadcast reproducer from the original report works on current main (host USM is now made resident, and drivers have moved on).
The
memset(pointer(a))case was our bug, not the driver's:ZePtrkernel arguments were converted toPtr, which lowers to an addrspace(0) pointer. SPIR-V treats that as private memory, so the store never reached the buffer, for device and shared buffers too. That also explains the C reproducer, which used the same SPIR-V. Fixed in #711 by passing them asLLVMPtr{T,AS.CrossWorkgroup}.