[morousg]

Adds an AMD GPU backend (HIP/ROCm, ParArch::GPU_AMD) to LTS-C++17. It is written in strict C++17 and modelled on main's HIP support. It also fixes LTS bugs found while porting and in review, most of which affect CPU and CUDA too. Apart from those fixes, CPU and CUDA behaviour is unchanged: the HIP code is behind ENABLE_HIP, the FK_HIP_BACKEND macro and ParArch::GPU_AMD.

What it adds

• HIP backend. ENABLE_HIP exists when CMake finds a HIP compiler for AMD GPUs (amdclang++) and is then ON by default.
  • HIP sources are built with CMake's HIP language at HIP_STANDARD 17, with -Werror=c++20-extensions.
  • Each test, utest and benchmark gets a _hip target next to _cpp/_cu. A new __ONLY_GPU__ marker limits a test to the CUDA and HIP targets.
• FK_HIP_BACKEND selects the HIP code paths, not __HIPCC__. The HIP headers also define __HIPCC__ in nvcc compilations that include them, so that macro can't tell the two apart.
• GPU_AMD streams, Ptr memory, TransformDPP/DivergentBatchTransformDPP and executors. Main selects these with requires; here they are partial specializations. The HIP executors are the CUDA executors with the HIP API names, including the thread fusion alignment fallback of 28101e3/1a9bfe8.
• fk::float3, fk::uchar3... in every backend (as main's 943895c). On CPU and CUDA they are the existing global types. Under HIP they are FKL's own structs, because HIP's global vector types are not FKL vector types.
• Stream_ owns its native stream through a shared_ptr, for both GPU_NVIDIA and GPU_AMD (from main's fcfe515/fe3e17c). The LTS Stream_ never destroyed the cudaStream_t it created.
• Benchmarks: HIP event timers, and a HIP-graphs variant of benchmark_horizontal_fusion.
• Docs: the README and the build-and-test skill cover the HIP build, the _hip targets and the fk:: vector types. They also say not to link FKL code compiled for different backends into one program.

LTS bug fixes (CPU and CUDA too)

setTo

• Host memory with a GPU stream: the kernel faulted. It is now filled by the CPU after the stream's pending work.
• HostPinned memory and both copies of DeviceAndPinned memory are filled by kernels on the stream, so the fill is in stream order and graph captures record it.
• During a graph capture, setTo on Host memory throws and leaves the capture intact.
• With a CPU stream, Device and DeviceAndPinned memory throw.

Memory and resource bugs

• Ptr::crop/getPlane reference counts: backport of main d0588f8 (Ptr::crop and Ptr::getPlane leak the parent allocation #377).
• CircularTensor::update wrote one plane past the end of the tensor on every update: backport of main 1838493. On the RTX 4090, without it test_circular_tensor fails BATCH 2, 4 and 5 and compute-sanitizer reports 3041 out-of-bounds writes. Its Device-memory check now also covers HIP streams.
• Ref::operator= kept its old counter (6724ed1).
• A failed stream destroy no longer leaves its error for the next cudaGetLastError().

Grid sizes

• Grids and thread fusion thread spaces are now computed with integer ceilDiv instead of ceil(n / float(d)). With the float version, sizes above 2^24 lost their last block, and the last elements were never written.
• Zero-size launches still throw.

HIP Debug builds

• DivergentBatchTransformDPP calls each sequence through a lambda, not a function pointer.
• At -O0 clang kept the pointer as an indirect call, and the kernel's stack size left out the callee. test_circular_tensor_hip then crashed with a memory fault in a Debug build.

Headers

• raw_image.h is self-contained.
• C++17 conformance fixes in array.h, ptr_nd.h and fkBenchmarksCommon.h.

Benchmarks

• benchmark_horizontal_fusion returns non-zero when a comparison fails. The CSV output is unchanged.

Many of these also exist on main: the setTo ordering, grid sizes, __HIPCC__ selection, the CircularTensor check, the stream destroy error, __grid_constant__ and the HIP utest checks.

CI changes

• The Linux workflows compile the benchmarks (-DENABLE_BENCHMARK=ON). ctest skips them (-E "^benchmark_").
• The Linux jobs actually build CUDA now. Before, they pointed at CUDA toolkits the runners no longer have, and silently built and tested the CPU backend only.
  • They now use CUDA 13.4 (as main does) next to 12.9.
  • They pass -DCMAKE_CUDA_COMPILER, so a missing toolkit fails the job.
  • They configure with -DENABLE_HIP=OFF.
• The Linux x64, ARM64 and Windows jobs require the cuda runner label, so they can't run on the ROCm runner.
• The Windows workflow is unchanged and is to be disabled for LTS-C* in a follow-up. Its CUDA paths are stale: at the final tip only the clang-cl 14.44 + CUDA 12.9 job found a toolkit and built and tested CUDA. The other three jobs build CPU only.
• No workflow builds HIP. A ROCm job (main's cmake-linux-rocm-amd64.yml) follows in a separate PR.

Verification at the final tip

• AMD, RX 9060 (gfx1200), ROCm 10.1:
  • Release, g++ host compiler, benchmarks built: all 42 _hip and 39 _cpp ctest entries pass, benchmarks included (benchmark_thread_fusion_cpp is left out as too slow).
  • Release, amdclang++ as host compiler with -pedantic-errors -Werror=c++20-extensions: 38/38 _hip and 36/36 _cpp tests.
  • Debug and RelWithDebInfo, HIP and CPU: 74/74 each. ENABLE_CPU=OFF: 38/38.
  • Debug builds for gfx906, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1151, gfx1200 and gfx1201 compile without warnings, with no indirect calls in device code.
  • ASan + UBSan are clean on the new and changed CPU tests and on the HIP host code.
  • All 46 _cu targets compile with nvcc 13.4.
• NVIDIA, RTX 4090, CUDA 13.4 (separate agent):
  • 124/124 targets build, with the same warnings as before. ctest 87/87, compute-sanitizer clean, and -G Debug runs identical to the previous tip.
  • Release SASS: the 317 kernels shared with 1a9bfe8 are identical.
  • Against the previous tip, 331 of 332 kernels are identical. The one that differs is test_circular_tensor_cu's DivergentBatchTransformDPP kernel for BATCH 4, after the lambda change above. It has the same 80 instructions and the same loads and stores; only the circular-index wrap computation moved from uniform to per-thread registers. That is a codegen variation, not a behaviour change.
• CI: all 12 jobs pass.
  • Linux x64 (g++-15, clang++-21, CUDA 13.4), Ubuntu 24.04 (g++-13, clang++-21, CUDA 12.9) and ARM64 (4 jobs, CUDA 12.9 and 13.4): 79/79 tests in each job.
  • Windows: 4 jobs pass, one of them with CUDA (see above).
• The CUDA side can't be run on the authoring machine. It has no NVIDIA GPU; there, every _cu target was compiled with nvcc 13.4.

Review

• Eight review rounds, each by two independent reviewers (one for library/CMake/CI, one for tests/benchmarks/docs).
• An NVIDIA-side review with probes on the 4090.
• The eighth round found nothing to change.
• Each fix is its own commit; the commit messages give the evidence.

Known issue (ROCm, not FKL)

• HIP Debug builds can fault when several GPU processes run at once (ROCm 10.1, gfx1200).
  • With four -O0 HIP processes in parallel, some kernels abort with a memory fault. A small HIP program with no FKL code fails the same way.
  • The faults stop with HIP_FORCE_DEV_KERNARG=0 and in RelWithDebInfo. Run alone, every Debug test passes.
  • Worth remembering if a ROCm Debug job ever runs on a shared runner.

Not ported (intentionally)

• Main's image DPPs that LTS doesn't have.
• ParArch::CPU_MULTITHREAD.
• The attention/WMMA work (RDNA4 phase 1: make HIP_ARCH work, attention wave-size helpers #394–RDNA4 phase 4: FlashAttention on gfx12 WMMA (bf16, FP8 QK^T) #397, against main).

🤖 Generated with Claude Code

https://claude.ai/code/session_019iTqfS8rWiAyRMrM4uU64n