Repository navigation
Conversation
PR ICLDisco#128 taught the triangular solve to run on a device, but only through hipBLAS, so the CUDA builds still sent every panel back to the host while the trailing updates ran on the GPU. This adds the matching cuBLAS call. The eight files differ in how they clip the trailing tile, which block holds each leading dimension, and whether alpha is folded into the first step, so each solve body is transcribed from the hipBLAS one next to it rather than written afresh. The new call uses cuBLAS v2, which means the file can no longer include the legacy header, and dplasmaaux_cuda.h is explicit that the two APIs cannot share a translation unit. So the zgemm body that was already here moves across too: it takes its handle from the stream info instead of cublasSetKernelStream, passes the scalars by address, names its operations with the CUBLAS_OP_* enums through dplasma_cublas_op, and reports failures with DPLASMA_CUBLAS_CHECK_STATUS, which decodes the status rather than printing a number. That is a wider diff than a new kernel wants to be, but leaving half the file on v1 was not an option.
Now that the triangular solve runs on the device, a GPU TRSM never moves a panel back to the host, and the round trip it used to make was the only thing in the suite exercising those transfers. PR ICLDisco#128 raised that as an open question; this is the answer. --trsm_gpu_solve 0 strips the GPU chores off the ztrsm task class between building the taskpool and handing it to the context, so the solve runs on the CPU while the trailing updates stay on the device. A taskpool holds private copies of the task classes it was built from, so the amputation reaches no further than the taskpool it is done to, and what is left is indistinguishable from a JDF that never declared a device body for that task. That is what TRSM was a commit ago, and what ztrmm still is today. Two other spellings are worse. time_estimate is never consulted here: parsec_select_best_device short circuits on data locality before it gets that far, because B is already on the device from the zgemm that produced it. An evaluate hook that turns the device down task by task looks equivalent to removing the chores and is not -- it leaves them in place and refuses them later -- and it is not a theoretical difference: spelled that way, all sixteen combinations came out wrong on a GPU runner while the device solve beside them was correct. The flag is applied in the common test setup rather than in the TRSM testing, so it also covers the solves the other testings run to check their results, which is where the transfers in question mostly happened. TRSM had no GPU test at all until now, so each of the four new entries is paired with an ordinary one: without a baseline beside it, a passing host-solve run would not tell anyone whether the device path works.
The LAPACK stage in read its source from the flow's data_in. That is not where the engine decided to read from: it records its choice in the flow's source, which the engine's own stage in uses, and it is free to pick a peer holding the same version, or the host mirror of a copy the device had to empty. In the latter case data_in is the destination itself, and the copy was issued from a buffer to itself and rejected.
The mechanism this branch added to take the accelerator incarnations away from one task class now exists in parsec, driven by a specification rather than by a task class name and a hardcoded device mask, so the local copy goes. The runtime also amends the pushout of a flow whose successor has lost its accelerator incarnations, so a host solve reads the tiles the GPU gemms produced instead of a stale host copy.
The trsm chore trimming needs parsec_taskpool_trim_chores(), and the GPU tests need the copy accounting and host mirror fixes that go with it. None of that has landed upstream yet, so this takes the submodule from the fork branch that carries all of it, and has to be pointed back at icldisco master before this can be merged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets the TRSM test that keeps the solve on the host say so by trimming the GPU incarnation out of the taskpool, instead of relying on a body that declines the device.
Draft, for two reasons:
parsec_taskpool_trim_chores(), which is still draft ICLDisco/parsec#808, so the submodule points at a fork branch carrying that PR and the accelerator memory work it sits on (#811, #812, #813, #814, #809)..gitmoduleshas to go back to icldisco master before this can merge.