Repository navigation
Add OpenCode local history adapter into the cost-usage store (#4752) - #4757
Danielweng0902 wants to merge 4 commits into
Conversation
Structural foundation for the steipete#4752-approved adapter direction: a versioned, backward-compatible source discriminator on the files scan_state JSON (absent = native Codex, so upgrade needs no migration), a typed versioned OpenCodeSourceCheckpoint payload for the adapter's table windows, part continuation, updated-watermarks and reconciliation progress, and source-aware restore dispatch so adapter records never decode as Codex JSONL state. Tests cover the legacy-payload upgrade path, checkpoint version gating, and a discriminated round trip through upsert/readSnapshot.
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
clawsweeper
Bot
added
P3
|
Codex review: needs real behavior proof before merge. What this changesThis PR adds an OpenCode checkpoint model, a backward-compatible source discriminator, and source-aware native cache restoration. Example: An OpenCode assistant message records $0 in /data/opencode.db.
Review scores
ProductKind: Feature · Worth it: Yes · Fix scope: Partial Merge readiness⛔ Blocked before merge - 2 items remain Keep this PR open as the owner-requested early draft: the source seam follows the approved design, and inspection found no concrete introduced defect. Priority: P3 Before merge
FindingsNone. Tests
Agent review detailsHow this fits togetherCostUsageStore persists derived local usage in SQLite; this seam identifies the source owning each file record and keeps OpenCode checkpoints out of native Codex cache restoration. flowchart LR
A[Native Codex logs] --> B[Native scanner]
B --> C[SQLite history store]
D[OpenCode checkpoint model] --> C
C --> E[Source-aware restore]
E --> F[Native Codex cache]
Technical reviewBest possible solution: Complete the approved bounded adapter using the existing scanner, transaction, read-view, and completion owners. Do we have a high-confidence way to reproduce the issue? This is an approved feature foundation rather than a bug reproduction; source confirms that ordinary OpenCode session ingestion remains absent. Is this the best way to solve the issue? The seam uses the approved persistence owner and preserves native decoding defaults, but checkpoint fields alone do not establish bounded ingestion or correct publication. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against c1ce9a4e6f1f. Provenance checked
TestingProof path: in-process harness. SecurityNone. EvidenceWhat I checked:
Root-cause clusterRelationship: Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. LabelsLabel changes:
Label justifications:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 24, 2026, 3:06 AM ET / 07:06 UTC. |
Stage 1+2 of the steipete#4752 checklist. The adapter decodes OpenCode assistant turns inside short read-only source transactions and commits usage rows, per-file day aggregates, global deltas and its typed checkpoint atomically through applyOpenCodeLocalHistoryPass (baseline revalidation, rollback on stale). Turns keep hash-stable identities over <table>:<sourceID>; windows walk by exclusive rowid boundaries pinned by their own SELECT with raw-lookahead exhaustion. Explicit regressions for the three steipete#4744 cursor failures: invalid rows consume budget without starving older valid rows, oversized messages checkpoint a strict (messageRowid, partRowid) continuation and resume mid-message without re-emitting, and appends after a snapshot stay eligible because boundaries never come from a later MAX(rowid). Tests: 5 adapter tests + 4 checkpoint seam tests, all passing.
Upstream UPDATEs rows in place while streaming, so the pass now walks time_updated watermarks per table and re-evaluates the containing message (or session row), replacing the prior contribution at the same identity: old payloads resolve via the store's readUsagePayloads point lookup, their negation folds into the global merge, and per-file aggregates recompute as absolute stored+new-old values. A pure delete (a session row that becomes excluded or invalid) removes its footprint. Late step-finish parts swap the fallback row for the part. Tests: covered-code mutation, late parts, and session updates all pass (8 adapter tests total).
Stage 4 of the steipete#4752 checklist. The adapter's contribution now lands in the Codex daily report with provenance resolved at read time: - sourceRecorded marker on ModelBreakdown (report-model case only, never persisted in store enums); merge keeps recorded and priced breakdowns of the same model apart via provenanceKey, so a recorded /bin/zsh subscription day can never fold into a same-named priced estimate. - Recorded zero stays a PRESENT zero in costUSD — presence is the signal, per the recorded-cost-as-is contract. - CostUsageStoreAccess.sharedScanStore gives the adapter the scanner's own shared store actor (one persistence owner); the pass runs in the fetcher's async seam beside the synchronous scan executor. - Options.includeOpenCodeLocalHistory (default off) flows from SettingsStore.openCodeLocalHistoryEnabled through UsageStore into the fetcher; the ambient gate excludes any scan with an explicit codexSessionsRoot or CODEX_HOME override. - Preferences toggle under the Codex provider detail. Tests: 4 new stage-4 tests (recorded-/bin/zsh conversion + provenance, merge keeps recorded/priced apart, ambient gate matrix, default-off). All 12 adapter tests pass; full-suite regressions are unrelated environment clusters (keychain access absent in this shell, one keyboard-layout- dependent shortcut test, load-timing flakes that pass in isolation).
|
Stages 1–4 of the #4752 checklist are now implemented on this branch (4 commits): 1+2: adapter pass + cursor regressions ( 3: mutation probe ( 4: provenance, scope, recorded-zero ( Verification
Known remaining (out of scope so far)
|
|
Thanks @Danielweng0902 — this is the right direction, and it follows the #4752 design closely: the typed source checkpoint, file-qualified aggregates, and the recorded-$0 handling are solid foundations. I audited the head against the design with extra synthetic diagnostics, and a few contract points still need work: duplicate contributions when two parts of one parent update, part continuations lost on reevaluation, timestamp ties across a budget boundary, deleted rows that stay counted, a native full save that drops the adapter's file row, cached publication that omits the source, and Spend Dashboard including it. The raw SQL work also still grows linearly with sparse history under a fixed row budget. No need to rework it yourself — I'm finishing it in place on your branch with maintainer commits (your commits and credit stay), and will turn the diagnostics into regression tests. I'll post here when it's ready. |
|
Thanks for the thorough audit — acknowledged on all eight points (the duplicate re-evaluation of one parent across part hits, the timestamp-tie watermark, and the cached-publication/dashboard paths especially). Stepping back from the branch now so your maintainer commits land cleanly. Happy to re-verify behavior locally once you post. |
|
Quick update @Danielweng0902: the remaining contract work is done, but GitHub won't let me push to this branch — your fork is a fork of |
OpenCode's local history (opencode.db) can feed the existing cost-usage store as an opt-in source for the ambient Codex ledger, implementing the design approved in #4752 and finishing #4757 by @Danielweng0902. A typed, versioned checkpoint lives in the existing files row; native saves preserve the source's rows; one resumable raw-row sweep reconciles mutated and deleted upstream rows within fixed row, payload and SQLite work budgets (flat across 1k/10k/100k histories) with cancellation; fresh, cached and catch-up views include it only for the opted-in ambient ledger, never scoped CODEX_HOME roots, managed accounts or Spend Dashboard; buckets follow the reporting calendar and recorded $0 stays authoritative. Closes #4752; supersedes #4757. Co-authored-by: Daniel930902 <daniel0902life@gmail.com>
|
Landed via #4786 (914204f) with your four commits preserved in its history — thank you @Danielweng0902 for working through the design and the adapter. Closing this one in favor of #4786; it'll ship in the next release. |
Draft PR per the #4752 flow — structural foundation first, opened early so the shape can be reviewed before the full test matrix. This branch implements the approved direction: a dedicated OpenCode adapter driven by the scanner, feeding the existing
CostUsageStorepersistence, bounded publication, and completion tracking. No new parallel store, no SQL schema migration.What this commit contains (structural seam, per adjustment 1)
CostUsageStoreScanState.sourcediscriminator (nativeCodexSessionLogs/opencodeLocalHistory), optional in thefiles.scan_stateJSON. Records persisted before this seam decode as native Codex by default — proven by a legacy-payload test; no SQL migration or table rebuild.OpenCodeSourceCheckpoint— versioned, typed JSON progress carried inscan_state.resumePayloadfor adapter records: source identity + WAL-aware DB fingerprint, per-table[low, high]rowid windows, per-tabletime_updatedwatermarks (mutation-aware future passes), a strict(messageRowid, partRowid)in-flight continuation (oversized-message regression from Add opt-in OpenCode local history with incremental rowid-window ingestion #4744), and resumable reconciliation progress. Rowids never enter the byte-oriented fields; a newer version never decodes as trusted progress.StoredFileDetails/CostUsageJsonl.ResumeState. Native behavior is unchanged (guards are no-ops for every existing record).upsertFile→readSnapshot), checkpoint encode/decode + version gating.Adjustment checklist for the follow-up commits on this branch
files.scan_state, source discriminator, no migration, source-aware dispatchfile_id+ bounded source summary in the read view (no aggregate columns, no model markers, presence overcost > 0)$0must surface like any other recorded cost (OpenCode-only / mixed / single-source windows)session_messagecreate-then-update, deletes; bounded rechecks, no per-refresh restart; short source transaction → atomic store commit under the writer lock)updatedMswatermarks)The #3273 contract (unchanged)
Opt-in default-off; read-only source access; exact OpenAI-provider assistant rows; valid
step-finishparts (one per part) with message fallback; recorded cost surfaced as-is ($0for subscription usage); ambient Codex scope only — never scopedCODEX_HOMEhomes, managed accounts, or Spend Dashboard; no account attribution.Marked draft: please point the review at the seam (discriminator placement, the checkpoint payload shape, dispatch gate) before the scan passes land.