Repository navigation
Look up fold_threshold behind an inference barrier, so that loading a backend doesn't invalidate launches - #937
Merged
Merged
Conversation
Launches compiled before KernelAbstractions' extension for a backend is loaded, such as those precompiled by AMDGPU.jl and Metal.jl through GPUArrays and AcceleratedKernels, depended on the generic method of fold_threshold, so loading the extension that adds the backend's method invalidated them. Look tuned values up through a `tuning` helper that hides the call from inference, and only for the launches that can be folded: tuned ones of more than three dimensions.
Contributor
Benchmark resultsCompared with d2ada37c6, the latest measured ancestor of the merge base (1 commit before it). Times are the minimum time of every benchmark. A change is unusual when it exceeds both three times the benchmark's typical change between consecutive runs on the default branch and a minimum per category (performance 3%, latency 5%, applications 3%, other 3%). A benchmark that starts or stops allocating is listed however few the allocations. A single run can still be a fluke, so rerun before drawing conclusions.
No unusual changes. All benchmarks
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #937 +/- ##
==========================================
+ Coverage 82.87% 83.19% +0.31%
==========================================
Files 31 31
Lines 2564 2571 +7
==========================================
+ Hits 2125 2139 +14
+ Misses 439 432 -7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KernelAbstractions sets
fold_thresholdfor AMDGPU, Metal and oneAPI in its own package extensions, so that the value can be tuned without releasing the back ends. Those back ends precompile kernel launches (their own, and GPUArrays' and AcceleratedKernels' through them), andselect_launchcalledfold_threshold(backend)on every launch. The precompiled launches therefore depended on the generic method, and loading the extension that adds the backend's method invalidated all of them. With the back ends defining their KernelAbstractions methods insrc(JuliaGPU/AMDGPU.jl#1202, JuliaGPU/Metal.jl#1118),fold_thresholdis the largest remaining invalidation tree when loading them: 535 method instances on AMDGPU, 322 on Metal.This PR looks such values up through
tuning(hook, backend), which calls the hook behindBase.inferencebarrierand asserts its result type (declared per hook withtuning_type). Compiled code then no longer depends on the hook's methods, and the extensions keep defining them as before.fold_thresholdis also only looked up for tuned launches of more than three dimensions, the only ones that can be folded, so 1–3-d launches don't pay for the lookup at all. Further values tuned per backend can use the same helper.Unique method instances invalidated by
using <backend>, KernelAbstractions(SnoopCompile, Julia 1.13.2, back end at the PR in parentheses):oneAPI.jl doesn't precompile launches on its backend, so there is nothing to invalidate. On Metal, most of the rest of the launch path is invalidated by AcceleratedKernels' extension, which JuliaGPU/AcceleratedKernels.jl#189 addresses the same way.
The lookup is a dynamic dispatch, which adds about 3 ns to
select_launchfor a 4-d launch (107 vs 111 ns on a Radeon iGPU, for a launch of ~1.3 µs). The time to the first launches doesn't change measurably by itself: on an M1, broadcasting, indexing, reductions, scans, sorts and a 4-d broadcast take 4.69–4.79 s from a fresh process, against 4.75–4.85 s before; on an Arc A770 13.8–14.0 s either way.The test suite passes on PoCL.