Skip to content

fix: read the answer from the message_end event pi 0.87.x emits - #79

Merged
bborbe merged 1 commit into
masterfrom
fix/pi-event-vocabulary
Sep 26, 2026
Merged

bborbe merged 1 commit into
masterfrom
fix/pi-event-vocabulary

Conversation

@bborbe

@bborbe bborbe commented Sep 26, 2026

Copy link
Copy Markdown
Owner

The defect

A v0.4.0 agent-pi pod answered a prompt correctly and the runner reported failure:

W main.go:345] prompt intake failed: no result found in pi CLI output

pi was fine. Its stream ended with:

{"type":"message_end","message":{...,"role":"assistant","content":[
   {"type":"thinking",...},{"type":"text","text":"ACK"}]}}
{"type":"agent_settled"}

extractEventText switches on exactly two event types — agent_end and message_update — and matched neither, so resultText stayed empty. The symptom points at the model; the defect is in the parser.

Why it survived

pi is installed unpinned in the agent-pi image:

npm install -g --omit=dev --no-optional @earendil-works/pi-coding-agent

so its event vocabulary drifts with every rebuild — and the only path that exercises the parser is a real prompt through the runner. The fleet's working agents run older images (pi-agent is pinned at v0.1.7), and the recently-built ones had never been prompted. The drift is invisible precisely because the agents that work are the ones nobody rebuilt.

This is therefore fleet-wide latent, not service-mode-specific: any agent-pi image rebuilt today would break the same way on its next real prompt.

The fix

extractEventText gains a message_end case. piEvent.Message already carries Role and Content, so it is small — but the role guard is load-bearing: message_end fires for every role, so without it a run producing no assistant text would return the system prompt as its answer. That is the worst available failure shape — a broken run that looks like a successful one — so it has its own spec.

Both vocabularies are kept rather than replaced: agent_end/message_update still cover the older pi that pi-agent's pinned image runs, which is exactly why that agent kept working.

Tests

7 of 7 specs in the pi package, up from 4. Three new, using the file's existing pi-shim pattern so they run through the public Run:

  • the older agent_end vocabulary still reads
  • the message_end stream reads — thinking block then text, exactly as observed live
  • a system-only stream is rejected, not echoed back

make precommit PASS — go test -race all 8 packages, golangci-lint 0 issues, gosec clean, trivy 0 vulns / 0 secrets, osv clean.

Still to come

Pinning pi in agent-pi's Dockerfile to 0.87.1 — the version the v0.4.0 image actually runs, confirmed against npm — which is the half that stops the recurrence. This PR makes it work; that pin is what keeps it working.

pi is installed unpinned in the agent-pi image and its event vocabulary has
already changed once. A v0.4.0 agent-pi pod produced
{"type":"message_end","message":{...,"role":"assistant",...}} followed
by {"type":"agent_settled"}, and extractEventText's two-case switch matched
neither — so a run that had answered correctly came back as "no result found
in pi CLI output".

The symptom points at the model; the defect is in the parser. Only a real
prompt through the runner exercises the path at all, which is how it
survived: the fleet's working agents run older images, and the
recently-built ones had never been prompted.

The message_end case carries a role guard, because that event fires for
every role — without it a user or system message would be returned as the
answer, and a run that failed would look like a successful echo of its own
prompt.

Both vocabularies are kept: agent_end/message_update still cover the older
pi that pi-agent's pinned v0.1.7 image runs, which is exactly why that
agent kept working while the freshly-built one did not.
@bborbe
bborbe marked this pull request as ready for review September 26, 2026 21:09

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have completed the review. Here's my assessment:


Step 5: Consolidated Report

Must Fix (Critical)

None.

Should Fix (Important)

None.

Nice to Have (Optional)

None.

Notes

  • pi/pi-runner_test.go — The ast-grep finding (go-testing/suite-test-file-required) is a false positive. pi_suite_test.go exists in the same directory and is a proper Ginkgo suite (TestPi + RunSpecs). The new var _ = Describe("piRunner event vocabulary", ...) block is correctly placed in the same package (pi_test) and discovered by that suite.

  • Role guard correctness — Verified at pi-runner.go:246-248: the message_end case returns "" for non-assistant roles. This is load-bearing and well-documented in both code comments and test comments.

  • Design trade-off acknowledged — The CHANGELOG entry and function docstring (lines 225-235) explicitly acknowledge that a future vocabulary change would silently produce "no result found". This is an intentional, documented trade-off.

  • Test quality — Three specs cover: older agent_end vocabulary, new message_end vocabulary (with thinking block + text block), and the role-guard failure case. The shim pattern is clean and consistent with the existing tests in the file.

Selector Mode Traceability

Finding Rule Disposition
Suite file required go-testing/suite-test-file-required False positive — pi_suite_test.go exists and is proper

Verdict

{
  "verdict": "approve",
  "summary": "Bugfix adds `message_end` handling to `extractEventText` with a load-bearing role guard. Three new Ginkgo specs cover the old vocabulary, the new vocabulary, and the role-guard failure case. The ast-grep suite-file finding is a false positive — `pi_suite_test.go` already exists.",
  "comments": [],
  "concerns_addressed": [
    {
      "concern": "correctness: role guard on message_end is load-bearing",
      "disposition": "addressed",
      "detail": "Verified at pi-runner.go:246-248 — returns '' for non-assistant roles, preventing system/user messages from being returned as answers. Well-documented in code comments and test comments."
    },
    {
      "concern": "correctness: future pi releases with new event types will silently return 'no result found'",
      "disposition": "not-an-issue",
      "detail": "This is a documented, intentional trade-off. Function docstring (lines 225-235) and CHANGELOG entry explicitly acknowledge it. Not a defect."
    }
  ]
}

@bborbe
bborbe merged commit 1f7aee9 into master Sep 26, 2026
5 checks passed
@bborbe
bborbe deleted the fix/pi-event-vocabulary branch September 26, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant