Repository navigation
Add opt-in OpenCode local history with incremental rowid-window ingestion - #4744
Danielweng0902 wants to merge 3 commits into
Conversation
Reimplements steipete#4574 in the shape requested in its close comment: one shared, fingerprint-gated OpenCodeLocalHistoryStore replaces the separate full-load reader. - Bounded memory: the store retains compact per-turn entries and windows them into day rows at read time; nothing pre-windowed is held. - Cache-only publication: decoded entries persist in the store's own SQLite database (never Codex cost-usage.sqlite, never the OpenCode Go reader's in-memory cache), and cached publications read that durable cache without touching the OpenCode database at all. - Fingerprint gating: scans re-read the database only when its size/mtime/WAL fingerprint changed, keeping the tightened scan I/O budget intact. The ambient merge keeps the steipete#3273 contract: opt-in and default-off, OpenAI-provider assistant records only, stable-ID v1/v2 deduplication, recorded cost surfaced as-is, provenance-labelled sourceRecorded/mixed, and permanent exclusion from scoped CODEX_HOME scans, managed accounts, and the Spend Dashboard. The lane/scope/account-switch behavior from steipete#4574 carries over unchanged.
The claudeConfigDirectory restoration prepared account-specific projects roots in `options` but the call still passed a fresh scannerOptions lookup, so a mapped Claude account validated its directory while the scan read ambient history (ClawSweeper P1 on steipete#4702). Pass the prepared options through as main did.
Replace the blob cache with the incremental-store shape from steipete#4702's close comment: per-table [low, high] rowid windows append new rows only, late step-finish parts re-evaluate their messages, a shrinking raw table count restarts fresh, and entries persist by part/message/session-message source ID so every pass is an idempotent upsert. Publications stay windowed SQL queries over the durable cache. Also wires the catch-up, CLI, chart, and inline hint call sites through the opt-in flag and provenance, and extends clear-cost-cache to the opencode-local-history directory.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex review: needs real behavior proof before merge. What this changesAdds a default-off setting that merges OpenCode’s recorded OpenAI usage into ambient Codex history through a persistent incremental SQLite cache. Example: An OpenCode OpenAI assistant turn records 150 tokens and $0 subscription cost.
Review scores
ProductKind: Feature · Worth it: Yes · Fix scope: Partial Merge readiness⛔ Blocked before merge - 14 items remain Keep open: this PR implements a useful, maintainer-welcomed feature, but ingestion and cache-consistency defects can omit or miscount history. Priority: P2 Before merge
Findings
Tests
Agent review detailsHow this fits togetherThe local-history store reads OpenCode usage metadata, caches individual turns, and supplies windowed reports to Codex menu and CLI aggregation while excluding scoped homes and account-owned spend views. flowchart LR
A[OpenCode SQLite] --> B[Incremental ingestion]
B --> C[CodexBar history cache]
C --> D[Windowed report]
E[Native Codex history] --> F[Ambient aggregation]
D --> F
F --> G[Menu and cost CLI]
Technical reviewBest possible solution: Use a bounded, transactionally validated ingestion adapter that observes completed updates, reconciles deletions, preserves cursor progress, and exposes coverage through the existing reporting owners. Do we have a high-confidence way to reproduce the issue? The reviewed source establishes the failing cursor, update, deletion, and aggregation paths; no target code was executed during this read-only review. Is this the best way to solve the issue? Indexed window queries improve the previous implementation, but rowid-only progress is insufficient for OpenCode’s mutable projections and needs transactional consistency and explicit coverage. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against d6c651ca26c4. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceWhat I checked:
Likely related people:
Root-cause clusterRelationship: Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 23, 2026, 11:02 AM ET / 15:02 UTC. |
|
Thanks @Danielweng0902 for sticking with this, and cached reads really did improve: in a synthetic check at 1k, 10k and 100k history rows, the indexed 100-row window stayed flat at about 1,333 SQLite instructions. I'm closing this version anyway, because it still adds a separate ~1,300-line SQLite store (its own schema, connections and persistence) instead of an adapter onto the existing incremental history store, so the existing history performance gates don't cover it. The checks also turned up three cursor problems: an invalid row can mark an unfinished range complete and skip an older valid row, a message with more parts than the per-pass budget never converges, and an append between the read and the later The shape we'd take is a source adapter that feeds OpenCode rows into the existing store and reuses its persistence, bounded publication and completion tracking. I know that's a different structure from what you've built, so if you'd like to try it, a short sketch in an issue first would let us confirm the approach before you invest in the full implementation. |
Stage 1+2 of the steipete#4752 checklist. The adapter decodes OpenCode assistant turns inside short read-only source transactions and commits usage rows, per-file day aggregates, global deltas and its typed checkpoint atomically through applyOpenCodeLocalHistoryPass (baseline revalidation, rollback on stale). Turns keep hash-stable identities over <table>:<sourceID>; windows walk by exclusive rowid boundaries pinned by their own SELECT with raw-lookahead exhaustion. Explicit regressions for the three steipete#4744 cursor failures: invalid rows consume budget without starving older valid rows, oversized messages checkpoint a strict (messageRowid, partRowid) continuation and resume mid-message without re-emitting, and appends after a snapshot stay eligible because boundaries never come from a later MAX(rowid). Tests: 5 adapter tests + 4 checkpoint seam tests, all passing.
Third attempt at the OpenCode local history source (#3273), replacing #4574 and #4702 with the incremental-store shape from their close comments. The opt-in contract is unchanged; the storage is new.
The store: incremental rowid windows, not a blob
OpenCodeLocalHistoryStorekeeps its own durable SQLite cache whose entries are individual turns keyed by stable source ID (part ID forstep-finishparts, message ID for fallback rows, session-message ID for v2 rows), with an index oncreatedMs:WHERE createdMs >= ? AND createdMs <= ?over the indexed cache. A publication materializes only the rows in its window; nothing scales with total history.[low, high]rowid window of ingested rows — the SQLite analogue of the OpenCodex usage store's byte cursors (a database has no byte offsets, but it does have monotonically assigned rowids). A scan costs one fingerprint stat; when the fingerprint moved, the pass reads only what the window is missing: new rows abovehigh(the common append case), and, for a first-ever pass on a database larger than the row budget, newest-first continuation belowlowso partial passes still prioritize recent history. Rows persist by upsert, so no pass rewrites rows it did not change.step-finishparts are handled: a part rowid cursor re-evaluates the messages new parts belong to, purging the message fallback row so a message that gained a part is never counted twice.Task.isCancelledchecks every 512 rows in both decode and write loops (a cancelled pass rolls back whole), fingerprint captured before the read and verified after — a moving source never marks itself complete, and cursors only advance to rows actually read.step-finishparts on one message count as separate turns, one entry per part, matching the OpenCode Go reader.The 129 MB question
Measured with this branch's CLI against a synthetic 100,000-row OpenCode database (22 MB, 9.7k rows inside the reporting window),
/usr/bin/time -l, ambient Codex history present:The native Codex scan dominates all rows. The store's marginal cost on a cached publication and on a one-row append is within run-to-run noise of the native-only baseline — the previously reported store-alone 129 MB peak on a cached refresh is gone, and a changed source no longer re-reads and rewrites history (one row changed, one row read: the appended $0.25 turn appeared in
Todayexactly).The #3273 contract (unchanged)
Opt-in default-off (
codexIncludeOpenCodeLocalHistory); reads usage metadata only withSQLITE_OPEN_READONLY; exactproviderID == "openai"assistant records; recorded cost surfaced as-is (OpenCode records$0for subscription usage);sourceRecorded/mixedprovenance with labelled hints; permanent exclusion from scopedCODEX_HOMEscans, managed accounts, and the Spend Dashboard;OPENCODE_DB/XDG_DATA_HOMEselection follows upstream's single-database contract.Wiring
loadCachedCodexTokenSnapshotResult).codexbar costreads the same default-off setting; session-grouped text excludes the merge (like Pi) with a stderr notice; JSON and the default view include it; all Codex cost views render provenance-aware hints.costProvenance.opencode-local-historycache directory.Tests
46 tests across the three OpenCode suites, including the incremental behaviors: multi-part messages, append-only ingestion with cursor state, deletion-triggered fresh re-ingest, late-part fallback replacement, multi-pass budget convergence, cross-cache source validation, damaged-cache recovery, and the usable-legacy v1/v2 projection. Related catch-up, CLI cost, and chart suites (217 tests) pass. The full suite has the two known parallel-load-flaky dashboard timing tests that fail identically on the base commit.
Closes nothing — #3273 stays the tracker.