[QingleiCao]

cudaEventQuery returns PARSEC_ERROR (-1) on illegal memory access, which collides with PARSEC_HOOK_RETURN_AGAIN. Remap negative rc so the device is disabled instead of retrying the same dead event forever.

Do not merge before #814.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[bosilca]

Logically correct, but from a performance point of view terrible, as it disables the tasks submission pipelining by returning PARSEC_HOOK_RETURN_AGAIN when no events are ready.

--- a/parsec/mca/device/device_gpu.c
+++ b/parsec/mca/device/device_gpu.c
@@ -2646,9 +2646,18 @@
             /* the task can be withdrawn by the system */
             return rc;
         }
-        if( 0 != rc ) {
-            return PARSEC_HOOK_RETURN_AGAIN;
+        if( 0 > rc ) {
+            /* A backend event error is fatal for this device: CUDA surfaces
+             * asynchronous illegal memory accesses here, and retrying the same
+             * event can never make it valid again. Note that PARSEC_ERROR and
+             * PARSEC_HOOK_RETURN_AGAIN share the same value, so the error must
+             * be remapped before it reaches the caller.
+             */
+            return PARSEC_HOOK_RETURN_ERROR;
         }
+        /* The oldest event is not ready yet. Fall through, so that the
+         * remaining event slots of this stream can still be filled.
+         */
     }
 
   grab_a_task:

[devreal]

Ahh yes, that one's been nagging me for a while. Hope to see this landing soon (preferably @bosilca's solution)

[bosilca]

It looks OK but I think it's diverging with what I have in the #814 chain. Let's revisit once #814 is merged.