Repository navigation
fix(graph): fix stuck claims, cancels and artifact ids in Graph V1 - #1
Merged
Merged
Conversation
The WS claim path rebuilt the HTTP claim request from the connection identity but dropped X-Daemon-Instance-ID. Graph claims bind an attempt to the daemon process identity and refuse a claim without it, so every Graph task a daemon claimed over WebSocket was rejected and requeued immediately, in a loop, and the node never started. Reproduced on a local build of main (ad25feb) with a real Codex runtime: 684 claim/requeue cycles in about a minute, the agent never launched and the node stayed queued. With this change the same stuck task is claimed at once and the node runs. Capture the header at connect, forward it when rebuilding the claim request, and cover the forwarded headers with a regression test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d timed-out attempts When the server cancels a task, executeAndDrain returns from its drainCtx.Done branch immediately with a bare cancelled result, before the backend has finished tearing down its process tree. The backend computes ExecutionStopped only after that, so the result reached the daemon without stop evidence. A Graph attempt is settled only from a stop receipt, so the daemon withheld it (execution_unknown) and the engine, by design, kept the run blocked: a run cancelled mid-flight stayed in cancel_requested indefinitely. The idle-watchdog and drain-timeout branches had the same gap. For Graph attempts (DisableWholeRunRetry), wait up to 30s for the backend's terminal result in those branches and carry its ExecutionStopped. Ordinary tasks never read the evidence and are not delayed. Reproduced on a local build of main (ad25feb) with a real Codex runtime: a run cancelled while its node was executing was still cancel_requested after 80s. With this change it reaches cancelled in about 9s. A run that was already stuck before the fix is not healed by it and needs a manual cancel/archive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s what to use A produce_commit agent declared its commit SHA as the git_commit artifact id. The prompt never said what the id must be and ValidateResult only checked that it was non-empty, so the submission was accepted. The id is stored in a uuid column, so reconciliation then failed with "invalid_id: invalid Graph resource UUID" every 3 seconds, permanently, and the run could not advance; only cancelling it recovered. Validate the artifact id as a UUID when the result is submitted, so the agent gets a correctable error, and state the requirement in the node prompt and in the built-in working-on-issues skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three fixes for Graph (V1, serial) found while running it end to end against a real Codex runtime on a local build of
main(ad25feb0c). Each has a regression test that fails on currentmain.fix(daemon): forward daemon instance id on WebSocket task claimsX-Daemon-Instance-ID, which Graph claims requirefix(graph): carry physical stop evidence on cancelled, idle-killed and timed-out attemptsexecuteAndDrainreturned a bare cancelled result before the backend finished tearing down its process tree, so no stop receipt was sentcancel_requestedindefinitelyfix(graph): reject non-UUID artifact ids at submission and tell agents what to useValidateResultonly checked that an artifact id was non-empty, but ids are stored in auuidcolumnproduce_commitagent that used the commit SHA as the id was accepted, then reconciliation failed every 3 seconds forever and the run could not advanceReproduction and verification
Local build of
mainatad25feb0c, real Codex CLI (gpt-6-sol), database migrated to 357:cancel_requestedafter about 85s. With the fix it reachescancelledin about 9s.main(artifact id "<sha>" accepted at submission). End to end on this build the agent happened to generate a UUID, so the wedge itself was observed on the Graph V2 branch rather than onmain; the validation gap is identical in both.produce_commitrun onmainwith the fixes completed, pushedrefs/heads/multica/graphs/<run>/implement/1, and left the repository'smainuntouched.Tests: the three new tests fail before and pass after. Across
internal/graph,internal/daemon,internal/daemonwsandinternal/handler, the failing set on a Windows host is identical with and without the change (48 tests, all pre-existing).Notes for release
cancel_requestedis not healed by this change; it needs a manual cancel or archive.executeAndDrainchange only waits (up to 30s) whenDisableWholeRunRetryis set, i.e. for Graph attempts. Ordinary tasks are not delayed; a test covers this.Not verified
-racewere not run (Windows host without CGO).mainwas not tested.Test plan
go test -race ./server/internal/...cancelledwithin secondsgit_commitartifact whoseidis a commit SHA; the submission is rejected with a message asking for a UUID🤖 Generated with Claude Code