[napetrov]

Part of #4.

Summary

A novelty routing --model run now records what it spent.

 routing.json
   candidates / queries / lexical / model        (unchanged)
+  cost
+    llm_calls, input_tokens, output_tokens
+    cache_read_tokens, cache_write_tokens
+    replayed_calls                              calls served from the response cache
+    usd                                         null when any call this run paid for had
+                                                no known price, or a call returned no usage
 stdout
   model    selection 0.5 (...)  [selector-m]
+  cost     2 calls, 200 input / 10 output tokens, cache read 120 / write 60, $0.0040

The totals are what this run spent: calls replayed from the response cache are counted in replayed_calls and left out of the token and usd sums (llm_calls still counts them), as the spend cap already does. A run with no counted calls prints cost 0 calls.

The per-role summary is shared with novelty analyze, so its cost block changes too: it gains the cache token counts and replayed_calls, leaves replayed calls out of input_tokens, output_tokens and usd (they were summed from the stored records before), and gives usd null when a call returned no usage. The Markdown report's cost line names the replayed calls when there are any. A run without --model makes no calls and writes no cost block.

Evidence

Routing cost tests in tests/test_novelty_routing_cost.py (split out so test_novelty_routing.py stays under the 1000-line cap), roll-up tests in tests/test_novelty_measure.py, and the report line in tests/test_novelty_report.py. The first ones listed below were the initial red/green check: The two model-run tests fail with the intel/main version of commands/novelty.py and novelty/client.py (no cost block, no cache fields) and pass on this branch:

before: commands/novelty.py and novelty/client.py reverted to intel/main
  FAILED test_a_model_run_records_its_calls_tokens_and_cost
  FAILED test_a_model_run_with_an_unpriced_call_reports_no_usd_figure
  1 passed (test_a_lexical_run_spends_nothing_and_has_no_cost_block)

after: this branch
  all three pass (full suite below)

Local checks on the rebased head:

uv run ruff check .                              All checks passed!
uv run mypy dokimi                               Success: no issues found in 222 source files
uv run python -m dokimi.schema_check             exit 0
uv run python -m dokimi.config_check             exit 0
uv run pytest -n auto -q                         6882 passed, 22 skipped

make ci-local after merging intel/main            0 failed, 28 passed, 7 skipped

Merge Danger

Door: two-way. The routing JSON gains keys. In the novelty analyze cost block existing keys change meaning: input_tokens, output_tokens and usd now exclude replayed calls, and usd is null when a call returned no usage (filed under Changed in the changelog). Reverting restores the old output.

Blast Radius: low, limited to --model routing runs and the analyze cost summary and report line. A cached analyze rerun now reports 0 tokens and usd 0 plus replayed_calls, where it reported the stored totals.

• Schema. schemas/novelty.v1.json gains cache_read_tokens, cache_write_tokens and replayed_calls in the cost summary. Readers that validate strictly will see three new optional integer fields.
• Unpriced calls. usd is null, not 0, when any call had no price, so a missing price is not read as a free run.
• Cache accounting. Whether cache tokens are included in input_tokens depends on the provider. The docs say so; the two counts are reported separately.
• No new flags. No user must pass anything new, and no persisted format changes beyond the additive keys.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated: docs/skill-novelty.md (routing cost block), docs/benchmarking-and-comparison.md (run totals leave replayed calls out)
• No ad-hoc run artifacts committed
• New code and test files carry the SPDX license header (the changelog fragment follows the existing fragments, which carry none)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.75%. Comparing base (a2b787a) to head (345c91b).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #164      +/-   ##
==========================================
+ Coverage   92.74%   92.75%   +0.01%     
==========================================
  Files         254      256       +2     
  Lines       32087    32425     +338     
  Branches     5344     5422      +78     
==========================================
+ Hits        29759    30077     +318     
- Misses       1560     1562       +2     
- Partials      768      786      +18     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/novelty.py	96.34% <100.00%> (+0.06%)	⬆️
dokimi/dokimi/novelty/client.py	100.00% <100.00%> (ø)
dokimi/dokimi/novelty/report.py	93.62% <ø> (ø)

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

The cost block is right when every call returns a priced usage record, and the JSON totals are correct. Three things are worth fixing before merge: the stdout line when calls are missing or failed, the replayed-call accounting, and the stdout assertions.

What I ran, on a Linux review host at 14a7e93:

• make ci-local: 20 passed, 8 failed, 7 skipped. 6883 tests passed, 21 skipped; changed-line coverage, schema-compat vs the base, wheel build and smoke all pass. The 8 failures are ruff, mypy, mkdocs, pip-audit and the other tool steps, missing from my venv because make dev stops at the tools install on this base, which #179 fixed on main. Static checks, mypy and the strict docs build pass on GitHub's CI at this head.
• Reproductions for the code findings, with the provider call stubbed.
• Mutation run on the new code and tests: mutants of client.py, the JSON totals and the cost block are all caught. Four mutants of the printed line pass the suite (inline comment).

Checked, no issues

• Schema: the two new fields are additive optional integers matching their neighbours, and the cost object has no additionalProperties: false, so no version bump is needed. routing.json itself is not validated by any schema.
• Other callers: roll_up_cost has one other caller (novelty/analyze.py); the markdown report prints only calls, tokens and usd, so it does not change.
• Lexical runs: no cost block is written and the print sits under the same if args.model guard.

[antonwolfy]

Review at d2fd404. The three earlier threads are fixed: the zero-call line, the replayed-call accounting and the full-line assertion all hold. Two things are worth fixing before merge: a change to novelty analyze's existing cost keys that the PR calls additive, and two test gaps found by mutation.

What I ran, on a Linux review host at d2fd404:

• make ci-local: 28 passed, 1 failed, 7 skipped. 7213 tests passed, 1 skipped; changed-line coverage, schema-compat vs the base, mypy, ruff, wheel build and smoke all pass. The one failure is pip-audit, which reports a gitpython 3.2.0 advisory (fix 3.2.1) from the lock files; the PR does not touch them.
• Reproductions with the LLM transport injected, comparing roll_up_cost at the base and at the head.
• Mutation run on the new code against test_novelty_routing.py and test_novelty_measure.py: 23 mutants, 4 of which pass the suite (two inline comments).

This PR changes existing novelty analyze cost values, not only adds keys. analyze.py:386 calls roll_up_cost, and cost_summary (client.py:171-178) now skips replayed rows and sets usd to null when a call returned no usage. Two calls, each 100 in / 10 out / $0.01, replayed=True:

• base: {'llm_calls': 2, 'input_tokens': 200, 'output_tokens': 20, 'usd': 0.02}
• head: {'llm_calls': 2, 'input_tokens': 0, 'output_tokens': 0, ..., 'replayed_calls': 2, 'usd': 0}
  One live call and one call with usage=None: base usd 0.01, head usd None. So a resumed analyze run with DOKIMI_RESPONSE_CACHE now reports llm_calls that no longer matches its token and usd totals, and the Markdown report (novelty/report.py:468-474, outside this diff) prints "2 LLM calls, 0 in / 0 out, usd 0" without mentioning replays. The new rules are the right ones, and they match the spend cap. But the PR description says "no existing key changes meaning", the changelog fragment files this under Added and names analyze only for the cache keys, and the docs paragraph sits in the routing section. Suggestion: say in the changelog fragment (as Changed) and in the cost schema descriptions of llm_calls, input_tokens, output_tokens and usd that replayed calls are excluded from the totals, print replayed_calls in the report.py cost line when it is non-zero, and correct the PR description. Related: with an unpriced replayed row and a priced live row, usd is 0.01, while the docs and schema say usd is null when any call had no known price. Say "any call this run paid for".

[xaleryb]

• Ran: full suite, ruff, mypy and bandit on the head merged with main, plus novelty analyze with every call served from the response cache (replayed=True).
• Open: novelty analyze with a cached rerun -> cost 0 in / 0 out, usd 0, and the md report prints 0 in / 0 out, usd 0. with no replay count -> the stored totals, as docs/benchmarking-and-comparison.md:691-693 says ("not $0/0s, so cost stays comparable across runs"). The change sits in the shared cost_summary, so it reaches analyze, not just routing. A call with no usage row now gives usd None. in the analyze md, and the changelog does not mention it.
• Pushed: a5b007e test: roll_up_cost sums cache_read/cache_write/replayed_calls over every role (a first-role-only total passed before).
• Left for the author: choose either stored totals plus a separate this-run spend field, or the exclusion applied to routing only; then update the doc and the report Cost line. Also state whether zero calls should give usd None or 0.0, and whether cache tokens on replayed rows are excluded, and pin both with tests.

[antonwolfy]

Review at a5b007e. The only change since my last review is a test that pins the cross-role roll_up_cost sums, and it holds. The analyze behaviour change and both test gaps from d2fd404 are still open, plus one more wording point on usd.

What I ran, on a Linux review host at a5b007e:

• make ci-local: 28 passed, 1 failed, 7 skipped. 7214 tests passed, 1 skipped; ruff, mypy, schema-compat vs the base, docs build, wheel build and smoke all pass. The one failure is pip-audit, which reports an advisory from the lock files; this PR does not touch them.
• Mutation run on cost_summary, roll_up_cost and the printed routing summary against test_novelty_routing.py and test_novelty_measure.py: 33 mutants, 8 of which pass both files. Three of them are the ones in my open threads; the rest are print-only lines of _print_routing_summary that predate the PR's logic.
• Reproductions with the LLM transport injected.

Still open from my earlier review:

• dokimi/tests/test_novelty_routing.py:545: nothing pins usd is None for a client with no calls; dropping self.calls > 0 at client.py:178 still passes.
• dokimi/tests/test_novelty_routing.py:569: the replay test gives the replayed rows no cache tokens; summing cache tokens over self.usage instead of billed (client.py:174-175) still passes, and so does printing the replayed suffix unconditionally.

1. This PR still changes existing novelty analyze cost values, not only adds keys. analyze.py calls roll_up_cost, which now takes cost_summary's rules: replayed rows are left out of tokens and usd, and usd is null when a call returned no usage. Two replayed calls, each 100 in / 10 out / $0.01, give input_tokens: 0, output_tokens: 0, usd: 0, replayed_calls: 2 at the head, against 200 / 20 / 0.02 at the base. llm_calls still counts them (2), so calls and tokens describe different sets. Nothing else moved with it:

   • novelty/report.py:468-473 (outside this diff) prints only calls, tokens and usd, so a cached analyze rerun reads "2 LLM calls, 0 in / 0 out, usd 0" with no mention of replays, and usd None for a call with no usage.
   • docs/benchmarking-and-comparison.md:691-693 (outside this diff) still says a replayed entry reports the stored usage "so cost stays comparable across runs".
   • The changelog fragment files it all under Added; the usd, llm_calls, input_tokens and output_tokens descriptions in schemas/novelty.v1.json do not mention the exclusion.
     Suggestion: move the analyze part of the fragment to Changed, add the exclusion to those schema descriptions, print replayed_calls in the report.py cost line when it is non-zero, fix the benchmarking doc, and correct the PR description ("no existing key changes meaning").

2. usd is not null "when any call had no known price" if the unpriced call was replayed. The check runs over non-replayed rows only (client.py:178). One live call (5 in, $0.1) plus one replayed call with no price gives llm_calls: 2, input_tokens: 5, replayed_calls: 1, usd: 0.1. A single unpriced replayed call gives usd: 0 and the routing line prints $0.0000. The changelog, docs/skill-novelty.md:611 and the cost_summary docstring all say "any call". Excluding replayed rows is the right rule; the wording is not. Suggestion: say "any call this run paid for" in the fragment, the doc paragraph, the docstring and the schema usd description.

@xaleryb's question on whether analyze keeps stored totals or this-run spend, and whether zero calls give usd None or 0.0, waits for the author.

Checked, no issues

• Schema: the new cost fields are optional integers added in place, so no version bump or SCHEMA_VERSIONS change is needed.
• roll_up_cost: zeroing any one of the calls, input, output, cache read, cache write or replayed sums fails the suite, and the new cross-role test covers the first-role-only total.
• Routing line: the zero-calls branch, the unknown-USD text, the .4f format and swaps of read/write or input/output are all caught.

[antonwolfy]

I pushed a commit per nit, up to 54f4ca8, which fix the nit from my review:

• 54f4ca8: dokimi/tests/test_novelty_measure.py:247: added the second blank line before test_a_failed_call_is_a_dropped_pair_not_a_graded_answer.

make ci-local at 54f4ca8: 28 passed, 1 failed (pip-audit, as at the reviewed head), 7 skipped.

Not approving yet. Still open: the two risks from my review; the two earlier threads of mine above nit level.

[napetrov]

Review-body items at 2594a80 (after merging main in 21a1e0d, which had conflicts in novelty.py, growth_baseline.json and test_novelty_routing.py):

• Risk 2 (usd wording for a replayed unpriced call): the changelog fragment, docs/skill-novelty.md, the cost_summary docstring and the usd description in novelty.v1.json now say "any call this run paid for" (2594a80).
• Risk 1 (novelty analyze totals now exclude replayed calls, report line, benchmarking doc, fragment under Changed): not changed. Whether analyze keeps stored totals or reports this-run spend is a design choice I have not made; it is waiting on the author.

[antonwolfy]

Review at f4df39a. Both threads from d2fd404 are fixed, and the printed line, the zero-call case, the usd wording and the replayed-row accounting in cost_summary all hold. Three things are worth fixing before merge: the novelty analyze cost values that this PR still changes, and two test gaps found by mutation.

What I ran, on a Linux review host at f4df39a:

• make ci-local: 28 passed, 1 failed, 7 skipped. 7769 tests passed, 1 skipped; ruff, mypy, bandit, schema-compat vs the base, changed-line coverage, docs build, wheel build and smoke all pass. The one failure is pip-audit, which reports advisories on filelock and setuptools from the lock files; this PR does not touch them.
• Mutation run on cost_summary, roll_up_cost and the printed routing summary against test_novelty_routing.py and test_novelty_measure.py: 46 mutants, 8 of which pass both files (126 passed). I re-ran five of them by hand; they are in the inline comments.
• Offline reproductions of cost_summary and roll_up_cost with synthetic usage rows.

Still open from my earlier review: none of my threads. The risk from my last review body is still open, as item 1.

1. This PR still changes existing novelty analyze cost values, and the all-replayed case now looks like a free run. analyze.py:386 calls roll_up_cost, which now leaves replayed rows out of tokens and usd. Two replayed calls, each 100 in / 10 out / $0.01, give llm_calls: 2, input_tokens: 0, output_tokens: 0, replayed_calls: 2, usd: 0 (an int 0, not 0.0). The usd description in schemas/novelty.v1.json:453 still says null is "distinct from a genuinely free run", yet a cached rerun and a free run now differ only in replayed_calls.

   • novelty/report.py:468-473 (outside this diff) prints calls, tokens and usd only, so a cached rerun reads "2 LLM calls, 0 in / 0 out, usd 0".
   • docs/benchmarking-and-comparison.md:693 (outside this diff) still says a replayed entry keeps the stored usage "so cost stays comparable across runs".
   • The changelog fragment files the analyze change under Added, and the PR description says no existing key changes meaning.
     Suggestion: move the analyze part of the fragment to Changed, print replayed_calls in the report.py cost line when it is non-zero, fix the benchmarking doc, and correct the PR description. @xaleryb's question on whether analyze keeps stored totals or this-run spend waits for the author.

2. Nit: llm_calls includes replayed calls, and the docs do not say so. The stdout line reads "2 calls, 0 input / 0 output tokens, ..., 2 replayed from the cache and not counted", where the 2 calls are the 2 replayed ones. docs/skill-novelty.md:612-618 and the changelog fragment say replayed calls are left out of the token and usd totals, but not that llm_calls still counts them. Suggestion: add "llm_calls still counts them" to the docs paragraph and the fragment.

Checked, no issues

• Schema: the new cost fields are optional integers added in place, and cost has no additionalProperties: false, so no version bump or SCHEMA_VERSIONS change is needed. routing.json itself is not schema-validated.
• Other callers: roll_up_cost is called only by analyze.py:386 and the new routing code; both copy the dict, so the extra keys break nothing.
• Lexical runs: no cost key is written, and the print sits under the same if args.model guard, so there is no KeyError.
• Mutants killed: dropping self.calls > 0, dropping the len(self.usage) == self.calls check, summing cache tokens over self.usage instead of the billed rows, zeroing any roll_up_cost sum except output_tokens, and swapping read/write or input/output in the printed line all fail the suite.
• Growth baseline: scripts/growth_baseline.json says 879 lines for commands/novelty.py and the file has 879.

[antonwolfy]

I pushed a commit per nit, up to 415c5cd, which fix the two nits from my review:

• 6ca352c: dokimi/tests/test_novelty_routing.py:547: the cost line is now compared as a whole line in out.splitlines(), so extra digits fail.
• 415c5cd: dokimi/docs/skill-novelty.md:618: the docs and the changelog fragment now say llm_calls still counts replayed calls.

make ci-local at 415c5cd: 28 passed, 1 failed (pip-audit, same as the reviewed head), 7 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving yet. Still open: the three risks from my review.

[napetrov]

Updated at 9993768 (merged intel/main in 438531d; conflicts in tests/test_novelty_routing.py and scripts/growth_baseline.json. The PR's routing cost tests moved to tests/test_novelty_routing_cost.py to stay under the 1000-line cap, and the commands/novelty.py baseline was raised to 999 with a rationale).

Review at f4df39a (@antonwolfy) and @xaleryb's question:

• Item 1, novelty analyze totals: decided to keep this-run spend for analyze too, the same rule the spend cap uses, rather than stored totals. fb43f33: the report.py cost line appends N replayed from the cache and not counted when replayed_calls is non-zero (test in tests/test_novelty_report.py); docs/benchmarking-and-comparison.md says the run's cost totals leave replayed records out and count them in replayed_calls, while each stored record keeps its original usage; the changelog fragment files the analyze change under Changed. The PR description now says existing analyze keys change meaning.
• Zero calls give usd None (an unknown, not a free run). That is pinned by test_a_model_run_whose_every_call_fails_prints_no_price_blame, and the schema usd description says so in 9993768. Cache tokens on replayed rows are excluded, pinned by test_replayed_calls_are_counted_apart_and_not_billed.
• Item 2 (llm_calls still counts replays): fixed in your 415c5cd.
• Inline threads: answered in-thread (fb43f33 for the replay-line/mixed-call and output_tokens gaps).

make ci-local on the merged tree: 0 failed, 28 passed, 7 skipped.

[antonwolfy]

Review at 9993768. Both earlier threads are fixed, and the novelty analyze decision (this-run spend, replayed calls named in the report line, benchmarking doc and changelog updated) closes my earlier risk. One nit inline, a test gap in the new report line.

What I ran, on Linux at 9993768:

• make ci-local: 28 passed, 0 failed, 7 skipped. Changed-line coverage, schema-compat vs the base, ruff, mypy, bandit, docs build, wheel build and smoke all pass.
• Mutation run on cost_summary, roll_up_cost, the printed routing line and the report line against test_novelty_routing_cost.py, test_novelty_measure.py and test_novelty_report.py: 37 mutants, 5 of which pass. One is the inline comment; I re-ran it by hand.

Checked, no issues

• Schema: the three new cost fields are optional integers added in place, and cost has no additionalProperties: false, so no version bump is needed. schema_compat reports only warnings for new properties.
• Accounting: replayed rows are left out of tokens, cache tokens and usd; llm_calls counts them; zero calls and a call with no usage give usd null. The docs, changelog and schema descriptions match this.
• Growth baseline: commands/novelty.py is 999 lines against a baseline of 999.

[xaleryb]

• Ran: the full suite, ruff, mypy and bandit on the head merged with main; the earlier cached novelty analyze inputs; mutants on the changed lines; and a live novelty routing --model run twice with DOKIMI_RESPONSE_CACHE.
• Earlier points hold at c880fc4. A cached analyze rerun prints 2 LLM calls, 0 in / 0 out, usd 0, 2 replayed from the cache and not counted.. Zero calls and a call with no usage both give usd null. The doc and the changelog match this.
• Found: a paid routing run of two short queries (usd 3.7e-05 in routing.json) prints $0.0000, which is the same figure an all-replayed run prints. Expected: a figure that does not read as free.
• Pushed 345c91b: prints <$0.0001 for a spend under $0.00005, the same form task_runs_report uses, with a test that fails without it.
• Nothing left for the author.

[xaleryb]

Approved at 345c91b. Every earlier point re-run at this head passes, no must-fix or mid item is open, and CI is green.