Work around amdflang whole-image device codegen instability

[sbryngelson]

What

Adds -attributor-max-pi-accesses=16384 to the offload link for LLVMFlang OpenMP-offload builds, and documents two amdflang-specific traps in gpuParallelization.md.

Why

amdflang generates device code for the whole image at link time. Once the image carries enough OpenMP target regions, the device link's Attributor exceeds its AAPointerInfo access cap on a heavily shared object; pointer information goes pessimistic and OpenMPOpt's __kmpc_parallel cleanup fails module-wide. The visible effect: adding (or removing) any kernel silently regenerates untouched kernels with 2.4–4.5× worse ISA — register spills and +512 B LDS in every kernel, image-wide. Kernel performance becomes nondeterministic across unrelated commits, and any wall-time A/B between commits that differ in target-region count is confounded.

Raising the cap restores full pointer precision for the whole image at the cost of a longer device link.

Evidence

• Deterministic from source; partition-independent (-flto-partitions=1 reproduces); reproduces on two AFAR drops sharing flang 23.0.0git.
• In-link A/B on identical objects: default cap → slow-class ISA (e.g. WENO scratch 28→140 B, uniform LDS bump); cap 16384 (or 0) → fast-class ISA restored.
• A full solver run with the flag: byte-exact MPI communication totals and wall parity vs an unflagged fast-class build.
• A minimal reproducer does not trigger it — the ~1000-kernel image is required — so the workaround is applied at the build level.

Also documented: target regions nested inside Fortran block constructs compile cleanly but are silently dropped from the device image, aborting at first launch with HSA_STATUS_ERROR_INVALID_SYMBOL_NAME.

Status

Draft until the flagged amdflang gfx90a build re-verification completes on our cluster (the flag itself has been verified via env-injected FFLAGS; this PR moves it into the build system). A compiler bug report to AMD is being prepared separately.

[Copilot]

Pull request overview

This PR introduces a build-system workaround for an amdflang (LLVMFlang) OpenMP-offload device-link optimization instability by passing an LLVM Attributor tuning flag at offload link time, and documents the failure modes so future GPU changes aren’t confounded by nondeterministic kernel codegen.

Changes:

• Add -attributor-max-pi-accesses=16384 to the LLVMFlang OpenMP offload link flags to stabilize whole-image device code quality.
• Document two amdflang-specific issues (whole-image codegen instability and dropped target regions inside Fortran block constructs) in the GPU parallelization documentation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
docs/documentation/gpuParallelization.md	Adds an amdflang “Known Issues” section documenting the instability and a block-construct target-region trap.
cmake/MFCTargets.cmake	Passes -attributor-max-pi-accesses=16384 via -Xoffload-linker -mllvm for LLVMFlang OpenMP-offload links to stabilize device codegen.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.67%. Comparing base (e2f0e26) to head (9ff6d42).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1759   +/-   ##
=======================================
  Coverage   61.67%   61.67%           
=======================================
  Files          84       84           
  Lines       21619    21619           
  Branches     3196     3196           
=======================================
  Hits        13334    13334           
  Misses       6093     6093           
  Partials     2192     2192           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.