Skip to content

Faster reductions of Broadcasted sources - #185

Merged
maleadt merged 1 commit into
mainfrom
tb/reduce-169-174
Oct 26, 2026
Merged

maleadt merged 1 commit into
mainfrom
tb/reduce-169-174

Conversation

@maleadt

@maleadt maleadt commented Oct 25, 2026

Copy link
Copy Markdown
Member

Fixes #169.

Reductions of a lazy Broadcasted source (what GPUArrays passes for sum(f, x .- μ; dims), var(x; dims, mean), multi-array mapreduce, ...) indexed it linearly. A Broadcasted object decodes every linear index into Cartesian indices, with a 64-bit division per dimension, and then computes each argument's index from those. That made them 2–3.5× slower than the same reduction of an array, and slower than materializing the broadcast first.

This PR indexes each argument of the broadcast with its own linear index (LinearBroadcast, in src/reduce/layout.jl):

  • an argument of the broadcast's shape takes the broadcast's index as is;
  • an extruded one (a row of means, say) takes a division and/or remainder by a 32-bit multiplicative inverse per run of the dimensions it is not extruded along;
  • scalars and Refs take none.

The trivial divisions are removed at the type level, so the kernels specialize on the argument shapes. Uniform branches or divisions by 1 in the load loop were 2–4× slower. To keep the host overhead at that of an array source, the launch goes through a function barrier, and the setup no longer builds the linear view. The index is 32-bit because 64-bit inverses take 128-bit products, which SPIR-V can't compile. A source with 2³² or more elements keeps the old indexing.

RTX 5080, µs, dims=1 unless noted, inputs rotated over more than the L2:

plain sum(abs2, x; dims) lazy, main lazy, PR
Float32 768×1024×8, abs2(a - μ) 32.7 67.4 32.9
Float32 768×8192 32.4 59.4 32.8
Float32 50304×8192, exp(a - m) 1817 2976 1833
BFloat16 768×1024×8, abs2(a - μ) 18.6 63.9 19.3
BFloat16 50304×8192, abs2(a - μ) 938 2765 1026
Float32 768×1024×8, whole array 30.3 65.7 30.5

Host time of a small lazy reduction along dims (64×64): 4.3 µs on main, 4.4 µs with this PR (an array source: 3.5 µs).

Julia 1.12 and later only

On Julia 1.10 and 1.11, a Broadcasted source is still materialized into a dense array before the reduction runs (kernel_source in src/reduce/reduce.jl), so this PR changes nothing there. That workaround exists because scalar linear indexing of a multidimensional Broadcasted object only works from Julia 1.12 on. The kernels no longer rely on it, but other parts of the reduction still index the source linearly as given: reductions where every output reduces a single element (mapreduce_singles!) and the scalar first element of a whole-array reduction. Dropping the materialization would mean routing those through LinearBroadcast too, and adding tests of Broadcasted sources on those versions. It is left out to keep this change to the kernels' indexing. Materializing costs one extra write and read of the source, about the "materialized" column in #169 (~2× a plain reduction), which is what 1.10 and 1.11 users get today.

Also

Tests

A new testset, reduce: Broadcasted sources, covers arguments of the broadcast's shape, arguments extruded along the reduced or the kept dimensions, arguments with fewer dimensions, scalars, Refs, nested broadcasts, ordered reductions and an argument with offset axes, through every launch shape. Reduce tests pass on CUDA (RTX 5080, also on Julia 1.10), AMDGPU (iGPU), OpenCL (PoCL), the KernelAbstractions host backend, Metal (M1) and oneAPI (Arc A770). AMDGPU's one error, the fill! of a Union{Missing, Bool} array, also happens on main.

…near index

Reductions along `dims` and whole-array reductions of a `Broadcasted` source indexed it linearly,
which decodes every index into Cartesian indices with a division per dimension and then computes
each argument's index from those. That made `sum(lazy((a, m) -> abs2(a - m), x, μ); dims=1)`
2-3.5x slower than `sum(abs2, x; dims=1)` (#169), and slower than materializing the broadcast.

`LinearBroadcast` instead indexes each argument with its own linear index: no decode for one of
the broadcast's shape, and a division or remainder by a 32-bit multiplicative inverse per run of
kept dimensions for an extruded one. The runs' types depend on the sizes, so the kernels
specialize on them; a function barrier in `mapreduce_nd!` keeps the host overhead at that of an
array source. Lazy reductions now run within a few percent of plain ones on an RTX 5080.

`source_layout`, unused since the launch shapes were made structs, is removed.
@maleadt maleadt added the enhancement New feature or request label Oct 25, 2026
@maleadt

maleadt commented Oct 26, 2026

Copy link
Copy Markdown
Member Author

CI failures unrelated.

@maleadt
maleadt merged commit 7a27309 into main Oct 26, 2026
34 of 35 checks passed
@maleadt
maleadt deleted the tb/reduce-169-174 branch October 26, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reductions over a Broadcasted source are 2–3.5× slower than over an array

1 participant