Emit canonical transcript events from all workflow spawn modes

[jameswnl]

Closes #272. Companion verification work: lightspeed-stack fork #52.

Summary

• New executor/step/transcript_events.py: reconstructs canonical transcript events (tool_call / tool_result / thinking / result / error, {"ts", "type", "data"}) from the pydantic-ai message history. Data keys mirror the sandbox EventLogger contract (lightspeed-agentic-sandbox src/lightspeed_agentic/logging.py) — the canonical producer — so event types and data keys match across spawn modes.
• DirectExecutor (spawn: none) run/run_stream/model-request paths emit canonical events; all failure paths emit error events (previously empty transcripts).
• SubprocessExecutor child (spawn: local): agent path, model-request path, and main() failure path emit canonical events over the existing stdin/stdout protocol.
• chat/runner._save_turn now reads canonical data keys (the previous flat-key reads matched no producer); tool results pair with pending calls in FIFO order so replayed names and synthesized call IDs match. Malformed or potentially truncated tool arguments and their paired results are omitted from provider history while the bounded audit events remain intact.
• docs/workflow-transcript-contract.md: the cross-mode contract plus documented gaps for none/local — one aggregate result event per run, cost_usd: null, run-completion ts (pydantic-ai has no per-turn usage source). Streaming StreamEvents are explicitly out of scope.

Test plan

• tests/unit/workflow/executor/test_transcript_events.py (new, 26 tests): converter contract (event types/order/shape/data keys/truncation/args serialization), normalize_transcript_events passthrough, end-to-end TestModel-driven emission through DirectExecutor.run and subprocess_child.main() (real agent loop calling a registered tool), and error events on both failure paths.
• Updated shape-pinning tests in test_direct_executor.py / test_chat_workflow_runner.py to the canonical contract.
• uv run pytest tests/unit -q: 2188 passed, 5 skipped. black --check and ruff check clean.

Review regressions cover the complete save-to-replay path for single and batched calls, including batches containing large arguments. Both subprocess execution paths append an error event when output-schema parsing fails; tests cover non-JSON, null, and successful JSON responses.

Mixed explicit/synthesized-ID history now removes explicit returns from the pending queue by matching ID, including out-of-order returns. The regression spans old and new turns. Integration fixtures use canonical events and verify provider replay identities and payloads.

[beesarmy]

Reviewed head 91d48ab. The shared canonical event conversion is clear, but the new persistence path produces invalid conversation history after tool use, and one subprocess failure path does not satisfy the error-event contract. I reproduced the inline findings against this exact head using the existing virtual environment, exercising _save_turn -> _build_message_history with real pydantic-ai message parts and the subprocess model-request path with a stubbed model response. No external model calls were made; I did not rerun the full unit suite.

[beesarmy]

[P1] Preserve tool call/result pairing before replaying the next turn

Omitting tool_call_id does not synthesize a matching fallback in the current _build_message_history: that function increments tool_call_counter for both calls and returns. Saving a single read_file call/result here and replaying the saved messages produces call_read_file_0 for the call and call_read_file_1 for its return, leaving an unanswered call and an unmatched return in the next provider request. Batched calls are also misidentified: call(a), call(b), result(a), result(b) saves both results with tool_name=b because current_tool_name only remembers the last call. This now affects ordinary spawn:none chat turns because this PR starts emitting and saving these events. Preserve a matching identity/name for each call and return (or fix replay pairing together with this change), and cover the complete save-to-replay path for single and multiple tool calls.

[beesarmy]

[P2] Avoid replaying truncated transcript input as tool arguments

transcript_events_from_messages deliberately caps data.input at 2000 characters, so this JSONDecodeError fallback persists an incomplete JSON string as the actual tool-call arguments for subsequent turns. I reproduced this with write_file({content: 'x' * 2100}): after _save_turn and _build_message_history, the ToolCallPart.args is invalid JSON. Anthropic and OpenAI Responses adapters call args_as_dict() on historical tool calls, so mapping the next request can fail before any model call; other adapters receive corrupted arguments. Keep complete arguments in conversation persistence through a separate representation, or omit incomplete tool exchanges from provider history. The bounded audit transcript alone cannot serve as a lossless conversation history.

[beesarmy]

[P2] Emit an error event for subprocess output parsing failures

_parse_content returns status='failed' rather than raising when a response is non-JSON (or null) and output_schema was requested. This path unconditionally overwrites transcript with a result event, so main() never enters the exception handler that adds error_transcript_event. With a model response of 'not JSON' and output_schema={'type': 'object'}, I reproduced a failed StepResult whose only event is result. The agent path has the same issue when it assigns transcript_events_from_messages after _parse_content. Append an error event when parsed.status is failed in both paths so transcript consumers see the failure and spawn:local meets the documented failure contract, as spawn:none does.

[beesarmy]

Re-reviewed b5a1902. The three original findings are addressed for the new canonical-only history: FIFO call/result IDs and names pair correctly, malformed or potentially truncated argument exchanges are excluded from provider replay, and both child execution paths append error events for parsing failures. The new save-to-replay and parsing regressions are useful.

Validation: 194 focused tests passed (test_direct_executor.py, test_transcript_events.py, test_chat_workflow_runner.py) against this exact checkout using the existing cloud-agents virtual environment. Stack's updated parity/mock suites also passed 25 tests against this source. No live sandbox or full unit suite was run.

One remaining issue in the pairing fix: the explicit-ID return branch never removes its call from pending_calls. Mixing supported explicit-ID history with the ID-less messages now persisted by _save_turn causes a later return to reuse an already-completed call's ID. Reproduction and fix guidance are inline.

[beesarmy]

[P2] Remove completed explicit-ID calls from the pending queue

Every tool_call is added to pending_calls, including calls with metadata.tool_call_id, but this explicit-ID result branch leaves its corresponding call queued. If prior history has an explicit-ID exchange and a later turn has the ID-less exchange saved by the new _save_turn, the later result consumes the completed old call. I reproduced two turns: old_tool call/return both use old-id; new_tool's call uses call_new_tool_1, but new_tool's return incorrectly uses old-id. Thus the supposedly repaired history again has a call without its matching result and assigns a new output to the wrong call. Explicit IDs are already supported by this adapter and its existing tests, so mixed history should work too. When honoring an explicit result ID, remove the matching pending entry by ID (rather than blindly popping, which would break out-of-order explicit returns), and add a mixed explicit/synthesized-ID multi-turn regression.

[beesarmy]

Coverage follow-up for head b5a1902:

• Full unit suite: 2,186 passed, 5 skipped, run against this exact source checkout using the cloud-agents Python 3.14 environment with branch coverage enabled.
• Combined line/branch coverage for the four changed runtime modules: 90%. This is measured module coverage, not whole-repository coverage.

Module	Line coverage	Branch coverage	Added executable lines covered
transcript_events.py	100%	94%	44/44
subprocess_child.py	97%	85%	7/7
direct.py	91%	81%	12/13
chat/runner.py	92%	79%	16/21

Added-line counts are executable added lines compared with the PR base; comments/docstrings are excluded.

The explicit-ID tool-result branch (direct.py:328) has no test coverage, including in the full unit suite. That is exactly the branch implicated in my remaining mixed-history finding. Please add a multi-turn history test combining an explicit-ID exchange with a subsequent synthesized-ID exchange, plus explicit results arriving out of order, and assert both IDs and outputs.

Other uncovered new paths: malformed JSON tool input (chat/runner.py:635-637), parsed non-object input (639-640), and empty ThinkingPart text. Tests currently cover the large-input omission but do not exercise those distinct rejection paths.

One assertion limitation that line coverage does not reveal: the run_stream tests mock the stream and leave new_messages() without real tool-message parts, so they exercise the conversion call but assert only the final result event. A real streaming agent/tool run, or a stream fixture with concrete ToolCallPart/ToolReturnPart values, should assert the emitted canonical tool events. This is additional coverage guidance, not a separate runtime defect.

[beesarmy]

Approved after the requested 30-minute follow-up review of head 1194ff0. All previously reported runtime findings are addressed. Explicit-ID returns now remove their matching pending call by ID, preserving mixed explicit/synthesized histories and out-of-order explicit returns; my original two-turn reproduction now yields matching IDs and preserves the correct outputs. The added regression cases exercise both return orders, and the integration fixtures now use canonical events with assertions on replay identities, arguments, and outputs.

Validation and coverage:

• 206 focused unit/integration tests passed locally against this exact checkout with branch coverage enabled.
• The formerly uncovered explicit-ID return branch and all executable statements in the new matching/removal block are covered. Both matching and nonmatching pending-ID comparisons were exercised.
• The prior full unit coverage run on b5a1902 passed 2,186 tests with 5 skips; current-head CI is green for lint, unit tests, integration, e2e, and build. I did not repeat the full local suite after this small fix.
• Stack's broader cloud-agents suites and mock tests passed 149 tests with 1 skip against this exact dependency source.

Non-blocking coverage follow-ups remain from the earlier coverage comment: malformed/non-object bounded arguments, empty thinking blocks, and streaming transcript assertions using concrete tool-message parts. The orphan explicit-result/no-matching-pending-call loop exit is also unexercised. These do not invalidate the reproduced fixes or approval. No live ephemeral sandbox was exercised locally.