Repository navigation
Report HIP's grid limits to KernelInterface [skip tests docs benchmarks] - #1192
Merged
Merged
Conversation
HIP rejects launches with more than 65536 workgroups along y or z (CLR's `hipModuleLaunchKernel`, since ROCm 7.0) and more than an Int32 along x, but `KI.max_num_groups` only accounted for the dispatch packet's work-item limit and reported about 4.2 million workgroups per dimension. KernelAbstractions 0.10.2 folds the dimensions of an N-d ndrange past the third onto the grid's z-dimension whenever the reported limits allow it, so a 4-d `getindex` with 240000 elements in the trailing dimension launched 240000 workgroups along z and failed with hipErrorInvalidValue (the volumerhs benchmark of #1191).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The benchmark step of #1191 fails in
perf/volumerhs.jlwithHIPError(hipErrorInvalidValue)fromhipModuleLaunchKernel, on a 4-dgetindexover a (5, 5, 5, 240000) index space. The build resolved KernelAbstractions 0.10.2 and KernelInterface 0.4.3, registered this morning; the previous benchmark run, two hours earlier with 0.10.1 and 0.4.2, passed. Main's next benchmark run will fail the same way.KernelAbstractions 0.10.2 (JuliaGPU/KernelAbstractions.jl#919) launches ndranges with more than three dimensions on the N-d grid, folding the dimensions past the third onto the grid's z-dimension, whenever
KI.max_num_groupssays the folded grid fits. Our implementation only accounted for the dispatch packet's UInt32 work-item limit and reported about 4.2 million workgroups per dimension. HIP's launch entry point in CLR, since ROCm 7.0, rejects more than 65536 workgroups along y or z (and more than an Int32 along x) with hipErrorInvalidValue, so the folded launch with 240000 workgroups along z fails. ROCm 6 has no such check.This caps the reported limits at
typemax(Int32)along x andtypemax(UInt16)along y and z, as CUDA reports them. KernelAbstractions then picks a linear launch for such ndranges. A KernelInterface test checks the reported limits and a KernelAbstractions test launches a kernel over a (4, 4, 4, 70000) ndrange, which fails with hipErrorInvalidValue before this change.Verified on beverin with ROCm 7.2 and KernelAbstractions 0.10.2: on main,
@roclaunches with 65537 workgroups along y or z fail while 65536, and 65537 along x, pass, and both the KernelAbstractions kernel and the benchmark'sgetindexfail; with this change they pass, the KernelInterface and KernelAbstractions test files pass on an MI250X, and the full benchmark script of the Buildkite step runs to completion on an MI250X and an MI300.