Repository navigation
Faster reductions of Broadcasted sources - #185
Merged
Merged
Conversation
…near index Reductions along `dims` and whole-array reductions of a `Broadcasted` source indexed it linearly, which decodes every index into Cartesian indices with a division per dimension and then computes each argument's index from those. That made `sum(lazy((a, m) -> abs2(a - m), x, μ); dims=1)` 2-3.5x slower than `sum(abs2, x; dims=1)` (#169), and slower than materializing the broadcast. `LinearBroadcast` instead indexes each argument with its own linear index: no decode for one of the broadcast's shape, and a division or remainder by a 32-bit multiplicative inverse per run of kept dimensions for an extruded one. The runs' types depend on the sizes, so the kernels specialize on them; a function barrier in `mapreduce_nd!` keeps the host overhead at that of an array source. Lazy reductions now run within a few percent of plain ones on an RTX 5080. `source_layout`, unused since the launch shapes were made structs, is removed.
Member
Author
|
CI failures unrelated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #169.
Reductions of a lazy
Broadcastedsource (what GPUArrays passes forsum(f, x .- μ; dims),var(x; dims, mean), multi-arraymapreduce, ...) indexed it linearly. ABroadcastedobject decodes every linear index into Cartesian indices, with a 64-bit division per dimension, and then computes each argument's index from those. That made them 2–3.5× slower than the same reduction of an array, and slower than materializing the broadcast first.This PR indexes each argument of the broadcast with its own linear index (
LinearBroadcast, insrc/reduce/layout.jl):Refs take none.The trivial divisions are removed at the type level, so the kernels specialize on the argument shapes. Uniform branches or divisions by 1 in the load loop were 2–4× slower. To keep the host overhead at that of an array source, the launch goes through a function barrier, and the setup no longer builds the linear view. The index is 32-bit because 64-bit inverses take 128-bit products, which SPIR-V can't compile. A source with 2³² or more elements keeps the old indexing.
RTX 5080, µs,
dims=1unless noted, inputs rotated over more than the L2:sum(abs2, x; dims)abs2(a - μ)exp(a - m)abs2(a - μ)abs2(a - μ)Host time of a small lazy reduction along
dims(64×64): 4.3 µs on main, 4.4 µs with this PR (an array source: 3.5 µs).Julia 1.12 and later only
On Julia 1.10 and 1.11, a
Broadcastedsource is still materialized into a dense array before the reduction runs (kernel_sourceinsrc/reduce/reduce.jl), so this PR changes nothing there. That workaround exists because scalar linear indexing of a multidimensionalBroadcastedobject only works from Julia 1.12 on. The kernels no longer rely on it, but other parts of the reduction still index the source linearly as given: reductions where every output reduces a single element (mapreduce_singles!) and the scalar first element of a whole-array reduction. Dropping the materialization would mean routing those throughLinearBroadcasttoo, and adding tests ofBroadcastedsources on those versions. It is left out to keep this change to the kernels' indexing. Materializing costs one extra write and read of the source, about the "materialized" column in #169 (~2× a plain reduction), which is what 1.10 and 1.11 users get today.Also
source_layout, unused since the launch shapes became structs (Choose the launch of reductions along dims once, as shape structs #181), is removed.linear_layoutno longer carries a buffer.LinearBroadcastadaptor.Tests
A new testset,
reduce: Broadcasted sources, covers arguments of the broadcast's shape, arguments extruded along the reduced or the kept dimensions, arguments with fewer dimensions, scalars,Refs, nested broadcasts, ordered reductions and an argument with offset axes, through every launch shape. Reduce tests pass on CUDA (RTX 5080, also on Julia 1.10), AMDGPU (iGPU), OpenCL (PoCL), the KernelAbstractions host backend, Metal (M1) and oneAPI (Arc A770). AMDGPU's one error, thefill!of aUnion{Missing, Bool}array, also happens on main.