Repository navigation
Keep a value reachable in main memory when a device cannot serve it - #812
Merged
Merged
Conversation
This was referenced Sep 27, 2026
devreal
reviewed
Sep 27, 2026
devreal
left a comment
devreal
left a comment
Contributor
There was a problem hiding this comment.
This is good, significantly less aggressive evictions. I would prefer not having to take locks all over the place though
Failing loudly when no source can be reached is right, but a host mirror in the middle of receiving a write-back is not unreachable: it is the one place the value is going, and it will hold it shortly. Reporting it as a source that cannot be reached turns a wait into a fatal error. Treat it the way a peer copy being filled is already treated, as a source worth coming back for.
jdf2c decides at compile time whether a flow needs a pushout, from whether the consuming task class has an accelerator body. A task class can lose its accelerator incarnations after that, either because the build does not support the device type or because a caller trimmed them, and the compiled decision then describes a program that is no longer the one being run: the producer leaves its result on the device while the consumer reads the host copy, and the consumer computes on stale data. The successor walk that already runs before the stage out is the place that knows what the successors can do, so ask it: a successor with no accelerator incarnation reads this flow in main memory wherever it is scheduled, which means the flow has to go there. That walk was entered only when a remote successor could not be served from device memory or when the accelerators could not all read each other, as those were the only two questions it answered. It now answers one that is independent of both, so the early return goes; the check for a task class with no successor iterator stays, as the walk cannot proceed without it. Widening when the walk runs also widens who is walked, and DTD cannot be. Its successor iterator is the dependency release itself: it hands a descendant its input and retains that copy whatever the action mask says, it reads the argument as a release context rather than as whatever the caller passed, it ignores a request to stop, and it expects every descendant it walks to have been inserted already, which says nothing about a task whose successors are still being submitted. Asking it a question before the task runs answers the question by changing the program. DTD does not need to be asked in any case: it is told at insertion time which flows to push out, and that is the answer this walk would be reconstructing.
bosilca
force-pushed
the
topic/host-mirror-reachable
branch
from
September 28, 2026 06:39
b3e3edb to
0b9fc2b
Compare
| parsec_warning("GPU[%d:%s] asked to release its memory while it is still being managed." | ||
| " The copies may be in use, so nothing is flushed.", | ||
| device->device_index, device->name); | ||
| return PARSEC_ERR_DEVICE; |
Contributor
Author
There was a problem hiding this comment.
Fixed, in two halves.
parsec_device_memory_release() only runs from device fini, where a live manager means we are tearing the device down under a thread still using its copies. Nothing up that call chain can recover from that, and the caller is already committed to destroying the streams next, so it is now a parsec_fatal rather than a silent teardown.
Separately, parsec_devices_release_memory() now reports the refusal to its caller. It still asks every device, since whether one can give its memory back says nothing about the others, and returns PARSEC_ERR_DEVICE if any was busy. A refusal is non-destructive: parsec_device_flush_lru() returns before touching a list, so the caller can simply ask again once the work it is running has drained.
Comment on lines
+968
to
+969
| if( !PARSEC_DEV_IS_GPU(parsec_task_class_device_types(fct)) ) { | ||
| output->data.preferred_device = -1; |
Contributor
Author
There was a problem hiding this comment.
Fixed, though not by reordering alone.
PARSEC_ITERATE_STOP ends the scan for that one output entry, so the successors behind the one that trips the packed fallback are never visited either. Hoisting the sentinel would have rescued only the successor that trips it. The packed branch now marks the dependency host-bound before returning, which speaks for all of them at once.
That is also right on its own account: packed bytes are unpacked into a main-memory copy whoever reads them, since reshape_copy_allocate() allocates the reshape destination on device 0. Receiving them on an accelerator only moves them straight back out to unpack.
The veto itself has moved to the top of the callback, right after the successor's task class is known, since where a task class can run is a property of the class alone and not of the dependency being examined.
| PARSEC_OBJ_RELEASE(gpu_task); /* no need to call release_device_task, just release the task */ | ||
| gpu_device->data_avail_epoch++; | ||
| return 0; | ||
| return nb_transferred; |
Contributor
Author
There was a problem hiding this comment.
Fixed. The counter is incremented only in the branch that puts the copy back on the clean LRU, and it is named nb_reclaimable to match the parameter of parsec_device_wake_starved_tasks().
| * | ||
| * A successor with no accelerator incarnation reads this data in main | ||
| * memory whatever its affinity says, and once one of them has asked for it | ||
| * there none of the others gets to move it onto a device: they can all |
…t there The receiver predicts where to place an incoming data from the affinity of the task that will consume it, which for an owner-computes algorithm is where that task will run. It never asked whether that task can run on a device at all, so a data consumed by a task class with no accelerator incarnation could still be received straight into device memory, where its consumer cannot read it. In a distributed trsm that shows up as a rank computing on an unwritten tile. A successor with no accelerator incarnation reads this data in main memory whatever its affinity says. Once one of them has asked for it, none of the others may move it onto a device either: they can all stage it in from main memory, while none of them can read it back out of an accelerator this one cannot address. preferred_device therefore grows a host pin, -1, which no later successor overrides, and the consumer honours a device placement only when the prediction is positive. Incoming temporaries are also initialized to main memory at both iterate_successors() call sites. Some paths return before the prediction runs, notably the packed fallback for mixed receive datatypes, and must not inherit a placement hint from a recycled remote_deps item.
A task that cannot reserve the device memory it needs gave back everything it had taken and asked to be rescheduled. Nothing had promised that memory would be there the next time, so it came straight back and failed again, and on the way it evicted the blocks the other starving tasks had just given back. On a device whose memory is all dirty that goes on forever: eviction never writes anything back, so a dirty block is never a candidate, and the only mechanism that turns dirty blocks into reclaimable ones is the write-back task the manager already knows how to build. It was unreachable, because it hangs off a negative return from the push stage and the reschedule was swallowed by parsec_device_progress_stream() to wait for an event that does not exist. The reservation now says which of the two happened. A task that ran out of memory is parked on the device's gpu_mem_starved list rather than rescheduled: it queued nothing and has no event to wait for, so it takes no event slot, and it stays counted against the manager so the manager cannot leave with waiters still on the list. Seeing that list non-empty, the manager starts a write-back. When the write-back completes it reports how many copies it moved, and that many tasks are woken; waking the whole list on the strength of one write-back would only put the device back to evicting what the others need. A task cannot be held for a release that is not coming, so when the manager runs out of work entirely the list is emptied and its tasks get to try again. The status the reservation returns is private to this file, travelling only as far as parsec_device_progress_stream(), which turns it back into an uneventful tour. It is not something a body can return and does not belong in parsec_hook_return_t. Reproduced with testing_dgemm -N 1940 -t 320 -g 1 and device_cuda_memory_number_of_blocks at 21, which hung and now completes.
A write-back compared the host version with the version the device copy carries to decide whether the value it just moved is still the current one. That comparison cannot tell success from failure. In both cases the host is behind: either because the transfer did not bring it up to date, or because the copy was handed to a writer while the transfer was in flight and now carries a newer value than the one that travelled. So the write-back always concluded that its work had been superseded, and never once reported success. Record the version being written back instead, and compare against that. A copy whose version has not moved has genuinely reached the host. The failure conclusion had a second cost. Creating the transfer takes the copy out of the owned list, and only the success path puts it back into a device list. Every write-back therefore dropped a dirty copy out of both lists, where no eviction can reach it and no later write-back can retry it. The device lost the memory of that copy for good, and after enough of them it could no longer make room for anybody. Put such a copy back in the owned list: it still holds the only current value, so it still owes the host a write-back. The recorded versions live in the task locals next to the flow count, so cap the number of flows a single write-back task can carry accordingly.
A device's lists may only be touched by the thread holding that device's manager token, either the manager itself or whoever took the token while there was none, and those two cannot run at the same time. The list lock was therefore being taken and released with no other contender for it. parsec_device_flush_lru() was the exception, since anybody can reach it through parsec_devices_release_memory(), so it now takes the token like every other outsider does. It hands the memory back to zone_malloc, which is not something to do to a device whose copies are still in play, so finding a manager there is reported and the memory left alone. parsec_device_dump_gpu_state() goes the same way as the rest: it has no caller and no token, and a lock nobody else takes would not have made it consistent anyway.
bosilca
force-pushed
the
topic/host-mirror-reachable
branch
from
September 28, 2026 22:16
0b9fc2b to
72e589e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A successor that can only run on the host, and the network, both need the value in main memory, and a device that is out of memory needs somewhere to put what it is holding. Five changes to that end: a task waits for a host mirror that is being written back; a flow is pushed out to the host when a successor cannot read it on a device; an incoming data stays in main memory when its successor can only read it there; dirty device memory is written back when tasks are starved for it; and a write-back reports whether it succeeded.
Stacked on #811, whose single commit shows up here until it merges.
Depends on [x] #811