Repository navigation
feat: add Goose parser - #69
Conversation
EntelligenceAI PR Summaryflowchart TD
classDef newBehavior fill:#dcfce7,stroke:#16a34a,color:#14532d;
A["Goose data directory"] --> B["Snapshot sessions.db"]
B --> C["Synthesize session NDJSON"]
C --> D["Discover and index sessions"]
D --> E["Parse messages, tools, and usage"]
E --> F["Session, turn, and tool spans"]
class A,B,C,D,E,F newBehavior;
Review Scorecard
Issues found:
Safe to merge with a low blast radius; code quality rated Excellent (5/5): the review found nothing to flag. 1 comment from earlier reviews is still open. Evaluated against
|
WalkthroughAdds end-to-end local ingestion for Goose coding-agent sessions. Goose SQLite databases are snapshotted and materialized as NDJSON, discovered from configured platform-specific home directories, and parsed into session, turn, and tool spans with usage and edit evidence. The update also documents Goose support and adds fixtures and tests for schema variations, filtering, parsing, tool aliases, and discovery. ChangesFiles
Adds Goose database snapshotting and transcript materialization, local session indexing, and parser support for sessions, turns, tool calls, usage, costs, timestamps, and edit evidence. Registers Goose agent detection and normalizes supported current and legacy tool names. Files
Registers Goose in discovery, including platform-aware home defaults, environment overrides, candidate materialization, and cache selection. Documents local session reading, skipped hidden and subagent sessions, database locations, and Files
Adds redacted JSONL transcripts covering edit and shell workflows, tool failures, message metadata, usage records, and session attributes. Files
Adds coverage for Goose database schema compatibility, snapshots, filtering, caching, parsing, usage attribution, failures, privacy, and discovery. Extends shared parser robustness and verifies Goose tool aliases and alias-table size. |
![entelligence-ai-pr-reviews[bot]](https://avatars.githubusercontent.com/in/932812?s=60&v=4){kind=small}
| shutil.copy2(db, copy) | ||
| for suffix in ("-wal", "-shm"): | ||
| side = db.with_name(db.name + suffix) |
There was a problem hiding this comment.
If Goose commits or checkpoints during these separate copies, the copied database and WAL can describe different snapshots, causing SQLite to reject the copy or materialization to omit committed messages. Read from a consistent SQLite snapshot instead.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In python/src/agentplay/materialize.py:1742-1747, avoid copying the live Goose database and its WAL/SHM sidecars independently. Read through a consistent SQLite snapshot or backup API, then materialize from that snapshot so concurrent Goose writes cannot produce a torn database/WAL pair.
| {"type":"goose_message","id":"msg_1","role":"user","created":1793637899.0,"content":[{"type":"text","text":"PLACEHOLDER-TEXT-1"}],"metadata":{"userVisible":true,"agentVisible":true}} | ||
| {"type":"goose_message","id":"msg_2","role":"user","created":1793637901.0,"content":[{"type":"text","text":"PLACEHOLDER-TEXT-2"}],"metadata":{"userVisible":false,"agentVisible":true,"turnContext":true}} | ||
| {"type":"goose_message","id":"resp_1","role":"assistant","created":1793637902.0,"content":[{"type":"toolRequest","id":"call_1","toolCall":{"status":"success","value":{"name":"shell","arguments":{"command":"python ARG-1"}}},"_meta":{"goose_extension":"developer"}}],"metadata":{"userVisible":true,"agentVisible":true,"inference":{"provider":"openai","requestedModel":"gpt-6-luna"},"usage":{"inputTokens":5116,"outputTokens":29,"totalTokens":5145,"cacheReadTokens":0,"cacheWriteTokens":5113,"cost":0.000653925,"costSource":"estimated","elapsedMs":1166,"timeToFirstTokenMs":1166}}} | ||
| {"type":"goose_message","id":"msg_3","role":"user","created":1793637902.0,"content":[{"type":"toolResponse","id":"call_1","toolResult":{"status":"success","value":{"resultType":"complete","content":[{"type":"text","text":"PLACEHOLDER-OUT-1\n\nCommand exited with code 3","annotations":{"priority":0.0}}],"structuredContent":{"stdout":"","stderr":"","exit_code":3},"isError":true}}}],"metadata":{"userVisible":true,"agentVisible":true}} |
There was a problem hiding this comment.
This result has both isError: true and Command exited with code 3, so the test still passes if failure detection is based on output text instead of Goose's flag.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In python/tests/core/fixtures/goose/failed_shell.jsonl:5, preserve the real-session fixture, but add a synthetic regression case in python/tests/core/test_goose_parser.py where `isError` is true and the output contains no exit-code text; assert that the parsed tool status is ERROR.
Read Goose sessions from its SQLite store, <data-dir>/sessions/sessions.db (~/.local/share/goose on Linux and macOS, %APPDATA%\Block\goose\data on Windows; honours GOOSE_PATH_ROOT; override with AGENTPLAY_GOOSE_HOMES). The db is WAL-mode and live while Goose runs, so the decoder copies it with its -wal/-shm and reads the copy read-only, selecting only columns that exist so older schemas still decode. Each session becomes one NDJSON transcript in the cache: a goose_session header, then messages and per-request usage rows in time order. Subagent and hidden sessions are skipped (main lane only). Goose records tool failures itself (toolResult status "error" or isError), so no output text is parsed for them. Messages Goose hides from the user (turnContext, compaction summaries) open no turn. The older developer__text_editor is classified by its command. Fixtures are three real Goose 1.57 sessions, redacted. Closes Entelligence-AI#66
- Read the live sessions.db through SQLite's backup API, so a commit or checkpoint by a running Goose cannot tear the db and its WAL; fall back to copying the files when it cannot be opened read-only. - Bill each request to its turn: use the usage Goose stamps on the assistant message (metadata.usage) and keep only compaction rows from usage_ledger; for older sessions place each ledger row after the last assistant message at or before its time. - Stream the whole transcript when checking for messages, so a first message larger than the 5 MB head still indexes. - Normalize developer__text_editor old_str/new_str/file_text, drop user-only result text, and keep epoch-zero timestamps.
6d5af4c to
35ea6cf
Compare
Also noticed in code this push did not change at `35ea6cf` (1, not blocking)These lines were already in the PR at the last review, so they are listed here instead of opening new threads.
|
|
thanks @Monika-rolls for the contribution! LGTM |
Summary
Adds a Goose parser. Goose 1.10+ keeps sessions in SQLite (
<data-dir>/sessions/sessions.db), so this adds a decoder tomaterialize.pynext to the OpenCode one, plusparsers/goose.py.-wal/-shm. Only existing columns are selected, so older schemas decode. Each session becomes one cached NDJSON: agoose_sessionheader, then its messages, eachusage_ledgerrow placed after the assistant message it bills. Subagent and hidden sessions are skipped (main lane only).turnContext, compaction summaries) don't. Tool failures come from Goose's own flags (status: "error"/isError), not output text. Tokens come from the usage Goose stamps on each assistant message (metadata.usage), which ties each request to its turn;usage_ledgeradds compaction requests, and is the fallback for older sessions.input_tokensalready includes cache reads and writes.shell→Bash,write,edit,tree→ListDir,delegate→Task, plus the olderdeveloper__shellanddeveloper__text_editor(classified by itscommand).~/.local/share/goose(Linux/macOS, XDG),%APPDATA%\Block\goose\data(Windows),$GOOSE_PATH_ROOT/datawhen set; override withAGENTPLAY_GOOSE_HOMES. README tables updated.Closes #66
Test plan
make checkstep by hand (nomakeon this Windows machine):ruff check src testspasses,pnpm run typecheckpasses,scripts/check_clean.pyreports clean, viewer tests 354/354 pass.pytest -q: 1485 passed, 10 failed. 8 are Windows symlink-permission tests (test_fixes_safety.py,test_autotune_lifecycle.py,test_robustness.py::test_store_never_reads_outside_maps,test_search_health_fixes.py) that fail the same way onmainwithout this change; 2 aretest_robustness.pymalformed-host/origin server tests that fail intermittently on this machine with and without this change. CI passed on every platform.test_goose_parser.py(43 tests): builds a db with Goose 1.57's fresh-install DDL and decodes it (rows still in the WAL, an older schema, an unreadable db, skipped session types); parses three redacted real sessions (Goose 1.57, Windows,openai); asserts no placeholder text, path ordelegateinstruction reaches the built map. Review fixes each have a regression test (usage billed to the right turn when the next prompt shares its second, a first message past the 5 MB head,old_str/new_strnormalization, user-only result text, epoch-zero dates,isErroralone, the snapshot fallback); all but the last two fail without the fix.agentplay agentslists Goose;agentplay build --agents goosebuilt 4 sessions with 0 failures. After the review fixes, all four sessions' map tokens and cost match Goose's ownusage_ledgerexactly (e.g. 396,280 tokens / $0.0086 on a 5-turn session). Checked its map against every string in the session's messages: none leaked.etceteraXDG strategy, covered by a unit test only); legacy per-session JSONL files (Goose imports them into the db itself).Known builder gaps seen with Goose on Windows, not changed here:
git commitat the end of acmd&chain isn't counted as a commit, andfindstr/Select-Stringcount as exec rather than search.make checkpasses (its steps run individually; see above for the pre-existing Windows-only failures)