[napetrov]

#211 has merged; intel/main is merged in, so the diff is only the trigger-probe.

Summary

The skill-catalog: arm over-triggers (recall 0.77, false-trigger 0.46 on 30 skill-reach queries). This PR adds a way to measure the real harness instead, plus an audit that checks what dokimi sends carries none of its own artifacts.

 dokimi/harnesses/
+├── trigger_probe.py   # fresh `claude -p` per query: new git repo, catalog in .claude/skills,
+│                      # scratch HOME/CLAUDE_CONFIG_DIR, query verbatim, no notice, bundled skills on
+└── payload_audit.py   # audit_payload / new_leaks: arm names, label keys, qids, grading words, paths
 dokimi/commands/
+└── catalog_probe.py   # `catalog trigger-probe`, scored by summarize_skill_selection, one arm per trial

Why the emulated arm over-triggers. I captured both request bodies through a logging proxy, then ablated them one at a time (haiku-4.5, 30 queries x 2, single turn):

variant	recall	precision	false-trigger
skill-catalog as sent	0.84	0.70	0.56
no sub-file enum	0.77	0.67	0.44
neutral tool wording	0.77	0.63	0.62
CC listing truncation	0.73	0.57	0.75
+ CC system prompt and tools	0.50	0.69	0.19
+ thinking	0.86	0.79	0.25
full CC replay	0.23	0.56	0.12

The main driver is that the arm offers only skill tools and no system prompt. The enum and the wording are within noise.

Evidence

• catalog trigger-probe, stock, CC 2.1.304, haiku-4.5, 30 queries x 3 trials: recall 0.23 / 0.27 / 0.27, false-trigger 0.00, misroute 0.09, cost $2.30. This is far below the emulated arm and in line with the full-replay ablation.
• The same run with --listing-budget-fraction 0.03 (skill-reach's setting), 2 trials: recall 0.45 / 0.36, false-trigger 0.00, cost $2.59. skill-reach on the same set gives recall 0.41–0.50 and false-trigger 0.19–0.31. Recall matches. False-trigger is lower, but there are only 8 negatives, so one false trigger moves the rate by 0.125.
• Leak audit: on the stock CC payload, the only hits are CC's own "golden" and "judge" text. The skill-catalog request is clean. The first probe draft leaked dokimi-run- through the working directory that CC shows the model. Fixed with a neutral project- prefix and a neutral "Initial commit".
• Tests: tests/test_trigger_probe.py and tests/test_payload_audit.py, FakeRunner, no network; every CLI flag asserted into ProbeConfig.
• Output adds max_turns_hit, not_listed (exit non-zero when a catalog skill was not offered) and per-query <out>.partial.
• make ci-local: all pass except pip-audit. pip-audit fails on filelock/setuptools CVEs in the lock that this PR does not change.

Merge Danger

Door: two-way

The command and modules are new. The only edit to existing code is two lines in catalog.py.

Blast Radius: none

Existing arms and outputs are unchanged.

🤖 Generated with Claude Code

[github-actions]

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi catalog trigger-probe`
    + option `--catalog`
    + option `--claude`
    + option `--concurrency`
    + option `--help/-h`
    + option `--limit`
    + option `--listing-budget-fraction`
    + option `--max-turns`
    + option `--model`
    + option `--out`
    + option `--sealed`
    + option `--target`
    + option `--timeout`
    + option `--trials`
    + option `--trigger-set`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 89.56229% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.73%. Comparing base (a2b787a) to head (e4fbbf2).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/harnesses/trigger_probe.py	89.79%	9 Missing and 6 partials ⚠️
dokimi/dokimi/commands/catalog_probe.py	88.76%	7 Missing and 3 partials ⚠️
dokimi/dokimi/harnesses/payload_audit.py	89.83%	3 Missing and 3 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #217      +/-   ##
==========================================
- Coverage   92.74%   92.73%   -0.02%     
==========================================
  Files         254      259       +5     
  Lines       32087    32711     +624     
  Branches     5344     5469     +125     
==========================================
+ Hits        29759    30334     +575     
- Misses       1560     1579      +19     
- Partials      768      798      +30     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/catalog.py	96.75% <100.00%> (+0.03%)	⬆️
dokimi/dokimi/harnesses/payload_audit.py	89.83% <89.83%> (ø)
dokimi/dokimi/commands/catalog_probe.py	88.76% <88.76%> (ø)
dokimi/dokimi/harnesses/trigger_probe.py	89.79% <89.79%> (ø)

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

Ran the full suite and lint, plus live catalog trigger-probe runs against real Claude Code (stock and --sealed).

• Two Skill calls in one turn arrive as separate per-block assistant events that share one message.id. They were scored as two turns: first_turn_loaded=[pdf-merge], recall 1.0, multi_load_rate 0. Expected [pdf-merge, csv-summarize], recall 0, multi_load_rate 1.0, as the skill-catalog: arm scores it.
• A run that called a skill and then hung on a backgrounded Bash command was recorded as error='timeout' and counted in n_failed. Expected: the choice is scored.
• --sealed listed plugin-authoring, which the docker-claude arms switch off. Expected: the same enabledPlugins seal.
• A relative --claude, or one failed run, ends in a traceback and --out is never written. Expected: an absolute path, and the failure recorded as that trial's error.

Pushed:

• 63ea270: group turns by message.id, skip subagent events
• d996190: score a choice made before a timeout or hang
• 25c992b: merge the plugin seal into --settings in sealed mode
• 0d83412: absolute --claude, per-trial errors

Left for the author:

• The backgrounded shell survives the timeout kill and the workspace cleanup.
• The docs say the operator's config "never reaches" the run, but the model reads any absolute host path, trigger-set labels included.

[antonwolfy]

Review at 0d83412. The trigger-probe stream parsing, timeout handling and sealed mode hold up. The problems are a few gaps around the probe and a branch that needs main merged in. Before merge I'd fix the tools_before_choice count, the late --out write and the branch.

What I ran, on a Linux review host at 0d83412:

• make ci-local: 32 passed, 1 failed, 6 skipped. The failure is pip-audit, which reports filelock and setuptools advisories in the lock files; this PR does not touch them.
• 7,876 tests passed, 1 skipped; changed-line coverage 95% (1,184 lines).
• Reproductions for the code findings, with no model or network involved, and mutation runs on the new tests: 61 tests in test_trigger_probe.py and test_skill_selection.py stay green under 15+ mutations of the probe and the command, and 4 mutants of the payload audit pass its tests (both listed below).

Findings not anchored inline:

1. The branch conflicts with main and carries three commits that are already there. f4ce883 (tasks inspect), f7abf7c (trigger-set readers, catalog formats) and 19822ba (precision) reached main through #211. git merge-tree of the head with main conflicts in 6 files: catalog.py, trigger_set.py, formats.py, growth_baseline.json, test_skill_formats.py and test_trigger_set_readers.py. Merged into main, the PR adds only the trigger-probe: catalog_probe.py, trigger_probe.py, payload_audit.py, their tests and the docs. Suggestion: merge main in (or rebase on it), so the diff and CI show only that.
2. The probe's flag handling is nearly untested. The CLI test uses a stub claude that ignores its arguments. Mutation check, each applied alone to the probe and the command, all pass the full test_trigger_probe.py and test_skill_selection.py: dropping --verbose, --no-session-persistence or --max-turns from the argv; dropping the TMPDIR override or input=""; replacing timeout=cfg.timeout_sec with 1e9; flipping symlinks=False; inverting stock=not args.sealed; dropping --model, --limit, --trials, --max-turns, --timeout, --listing-budget-fraction, --target or --concurrency in the CLI. Suggestion: one test that runs the command with every flag set and asserts the argv, env and ProbeConfig the runner receives.
3. The payload audit has survivors too. Each of these passes its tests: no IGNORECASE on the marker pattern, no lookbehind on it, the longest-first sort of allowed reversed, and tuples not walked. No test uses a capitalised marker, a marker preceded by letters, or a tuple.

@xaleryb's two points for the author, the backgrounded shell that survives the timeout kill and the "never reaches" wording in the docs, are still open.

Checked, no issues

• Stream parsing: blocks of one turn group by message.id, subagent events are skipped, and a missing id falls back to a per-event key.
• Errors: a Skill call counts whatever follows it, error_max_turns is an observation, and --timeout -5 fails before any run is spent.
• CLI surface: the trigger-probe flags in evaluating-treatments.md match catalog_probe.py. The changelog fragment exists and CHANGELOG.md is untouched.

[vchamarthi]

No blockers. 3 non-blocking comments.

No blockers found beyond the open points from the earlier review (merge main, tools_before_choice, --out written after all runs). Three further points below, all in the trigger-probe code.

CI at send time: 0 failing, 0 pending.

[vchamarthi]

Non-blocking: trigger-probe accepts any --target; reject one that is not a skill in --catalog, as catalog trigger-set already does.

dokimi/dokimi/commands/catalog_probe.py:31 passes target=args.target straight to read_trigger_set and never loads the catalog skills. cmd_catalog_trigger_set in catalog.py (_trigger_set_sources) refuses with "skill-creator target ... is not a skill in --catalog". A typo'd target here makes every positive query expect a skill that cannot load, so recall reads 0% after the full run is paid for.

[napetrov]

Fixed in e27d2bd: --target must name a skill discovered in --catalog (same rule as catalog trigger-set), checked before any paid run; malformed catalog is an input error too (6901cde).

[vchamarthi]

Non-blocking: trigger-probe should compare the skills Claude Code reports in its init event (listed) with the catalog and fail when a catalog skill is missing.

dokimi/dokimi/harnesses/trigger_probe.py:147 stores listed from the init event and :162 returns it, but catalog_probe.py reads only first_turn_loaded and cost_usd. A catalog whose frontmatter name differs from its directory, or uses a nested layout, is copied to .claude/skills with no complaint, so every positive query expects a skill that was never offered. Same silent-zero shape as the --target finding, different cause. Read at head 0d83412; not run against a real claude.

[napetrov]

Fixed in e27d2bd: catalog skill names are compared with the init event's listed; missing ones go to not_listed in the output and the command exits non-zero after writing the file.

[vchamarthi]

Question: Is --max-turns default 3 right for stock mode, where the module docstring says the model may look around before loading a skill?

trigger_probe.py treats error_max_turns with no Skill call as an observation (run_query, error stays None), and parse_stream records tools_before_choice. A model that reads a file or two first reaches the turn cap before it calls Skill, and that run counts as a miss for recall and as a correct 'no load' for negatives, so a low cap biases both rates in opposite directions. Consider reporting how many runs ended on error_max_turns in the output so the cap's effect is visible.

[napetrov]

Kept the default at 3 (a cost cap for single-shot trigger checks), but made the effect visible in e27d2bd/6901cde9: each run records result_subtype, and the output carries max_turns_hit, so a run cut by the cap is countable; raise --max-turns when that number is non-zero.

[napetrov]

Review follow-up, head 7c7bb67:

• Branch vs main (antonwolfy Make dokimi public #1): merged intel/main (2ef5055, 7c7bb67); diff now shows only the trigger-probe.
• Flag handling untested (Make dokimi public #2): e27d2bd adds a test that sets every CLI flag and asserts the ProbeConfig, plus argv/env/timeout/stdin of a run and symlinked skills copied as files.
• Payload audit survivors (fix(novelty): recover truncated extraction batches and report unreadable spans #3): e27d2bd adds capitalised-marker, lookbehind, tuple and longest-first tests.
• xaleryb: backgrounded shell surviving the timeout kill: e27d2bd kills processes whose cwd is in the run's workspace (kill_strays); "never reaches" wording: docs now say the operator's config is not loaded but host paths stay readable.
• vchamarthi's three points: answered in-thread (e27d2bd, 6901cde).
  Local make ci-local: all pass except one test failing on this host only (AF_UNIX path too long under a long TMPDIR; passes with TMPDIR=/tmp).

[antonwolfy]

Both earlier threads are fixed and the new not_listed, max_turns_hit and .partial paths work. One thing is worth fixing before merge: the fix for the backgrounded-shell leak has no test that would notice if it were removed. Three nits inline.

What I ran, on Linux at 7c7bb67:

• make ci-local: 33 passed, 0 failed, 6 skipped.
• Reproductions for the CLI findings, with a stub claude and no model or network involved.
• Mutation runs on test_trigger_probe.py and test_payload_audit.py: 60 mutants of the probe, 27 pass the suite; 59 mutants of the payload audit, 8 pass.

Checked, no issues

• Stream parsing: tools_before_choice stops at the first Skill turn, and turns still group by message.id.
• Input checks: --out, --target and --catalog are rejected before any run; .partial is replaced atomically and removed on success.
• kill_strays: matches the root exactly. With real sleep processes in project-abc and project-abcd, cleaning project-abc kills only the first.
• Repo rules: changelog fragment present, docs page already in nav:, no generated artifacts in the diff, catalog.py stays inside its growth baseline.

[xaleryb]

Re-checked at ea9d304: full suite, lint, types, security; each earlier point re-run with its original input (stub and live Claude Code).

• All four earlier points and both points left to the author are fixed at this head. A backgrounded sleep left in the workspace is killed when the run ends.
• Trigger set whose expected_skill names no --catalog skill (typo pdf-mrege) -> paid run, recall 0.0, exit 0 -> refused before any run, as the skill-catalog: arm refuses it and as --target already is.

Pushed:

• 687f90f: test that run_query kills a job left in its workspace and spares a same-prefix sibling root. This covers the open untested-kill_strays thread.
• e4fbbf2: refuse a label that names no catalog skill (input error, exit 2). Reading the trigger set moves into a helper so the command stays under the complexity cap.

Left for the author: nothing blocking.

[xaleryb]

Approved at e4fbbf2. Every earlier point re-run at this head passes, no must-fix or mid item is open, and CI is green.