[antonwolfy]

Summary

Follow-up to an audit of a dokimi repo run replay on IntelPython/dpnp#3099 (items 6 and 7; no review thread).

Item 6: guidance_read missed guidance printed through Bash under Claude Code. The Codex and Copilot trace readers have turned a shell command that only prints files into read events since #116 (cli_traces.shell_reads). The claude-stream-json converter did not: a Bash call was a command event and nothing else, and only the Read tool made a read event. In the replay, 17 of 78 guidance cells printed an AGENTS.md with Bash and all had guidance_read: []. On main:

Claude Code Bash call	events on main	guidance_read on main	here
cd benchmarks; cat AGENTS.md (ok)	command only	[]	["benchmarks/AGENTS.md"]
cat AGENTS.md (is_error: no such file, the none arm)	command only	[]	[] (a failed one-file read is no read)
cat AGENTS.md missing.md (is_error)	command only	[]	both read, rc 0 (the exit does not say which failed)
cat AGENTS.md, refused by the PreToolUse hook	command, denied	[]	read, denied: []
cat AGENTS.md | head, grep rule AGENTS.md	command only	[]	[]

Item 7: no reach scenario measured whether a Bash read of a nested file loads that directory's guidance. read=<dir> measures it for the Read tool only. The replay's agents used only Bash, and their cells' constant +3,065 start tokens fit the root file alone, so that was the open question.

The second round of commits answers the review of this PR: a cd in a background job or a pipeline no longer counts as moving the shell, refused calls are settled correctly, refused or failed reads are no reads for reach and skill use, and the bash= prompt quotes its path. The third round fixes two more cases where a cd that cannot move the shell still counted as moving it: a cd before a brace group in a backgrounded list, and a computed cd (cd "$X") or any cd in a pipeline, when backgrounded or piped. The fourth round:

• keeps one moved-shell state for the parent and its subagents;
• counts no refused Skill call as a use, and no failed arm-file read in uptake.reads;
• drops the read of a failed cd sub && cat AGENTS.md;
• keeps a refused read after a cd as denied;
• treats eval, source, . and time cd as moving the shell.

The fifth round:

• says that tool_calls counts denied reads and the reads of a failed multi-file command;
• counts no failed Skill call as a use;
• stops counting a cd $(...) in a pipeline (cd $(pwd) | cat) as moving the shell;
• drops the read of a failed time cd sub && cat AGENTS.md, and of the command cd, builtin cd and X=1 cd forms.

The sixth round fixes three edge cases of the shared shell parser at their roots:

• a brace group whose first command holds $( ) ({ cd $(pwd); } | cat) is tracked, so its pipe or & moves nothing;
• a backgrounded or piped source, eval, ., pushd or computed cd (source env.sh & cat AGENTS.md) keeps the foreground directory;
• a failed read keeps its reads when its printer names two files, even when only one of them is known (cd -; cat missing.md /ws/AGENTS.md).

The seventh round fixes two more cases in the same parser:

• a directory builtin in a job or a pipe leaves a later || branch its directory (source env.sh & false || cat AGENTS.md reads AGENTS.md);
• a quoted or escaped directory builtin with a computed operand ("source" "$ENV", \cd "$D") is that builtin.

The eighth round stops adding special cases to the parser. It adds one conservative fallback instead: a command the parser does not fully follow that names a directory builtin (X="a b" cd "$D", time -p cd sub, command -p cd sub, builtin -- cd sub) counts as moving the shell. Forms that only drop a read (cd $(printf sub; true) | cat) are documented limits, tracked in #374.

The ninth round narrows that rule, which fired on a builtin's name anywhere in the command, and settles four more points:

• only the command word counts: past builtin, command or time and their options, the next word must be a directory builtin. So time git add ., time find . -name x, command -v cd and builtin printf '%s' . no longer drop every later relative read of the trace;
• time is bash's keyword only as the first word, unquoted: X=1 time cd sub and \time cd sub run /usr/bin/time and move nothing. A directory builtin in a case arm counts as a move;
• a failing brace group that holds one read ({ cat AGENTS.md; }) gives no read;
• a refused relative read after a move keeps its event, with no path;
• process.unknown_reads counts the relative shell reads made at a directory the converter could not follow, so guidance_read: [] after source .venv/bin/activate is not taken as "none read".

The line-continuation point (cd \<newline>sub && cat AGENTS.md) is declined: that command already gives no read, and a test now pins it.

The tenth round stops process.unknown_reads from counting a print whose output never reached the agent: a brace group whose output is hidden ({ source env.sh; cat AGENTS.md; } > /dev/null, or the group piped into head) gave no read but counted 1.

The twelfth round merges main again and settles two more points:

• repo report and repo compare now show process:unknown_reads. Before, a batch whose every cell ran source .venv/bin/activate and then cat AGENTS.md showed only a process:guidance_read mean of 0;
• three forms of the case and time handling kept a read at a wrong path or a hidden one: time X=1 cd sub, a nested case, and a brace opened on a case part.

The thirteenth round settles two more points:

• a failed one-file read at a directory the converter cannot follow (cd -; cat missing.md, exit 1) no longer adds 1 to process.unknown_reads, under Codex, Copilot or Claude Code;
• a leading redirection (> /tmp/cd.log cd sub) is no longer taken as the command word, so that cd moves the shell, and a later relative Bash read is no longer recorded as the root file's.

The fourteenth round finds the command word also when a redirection comes between command, builtin or time and its options (command > /tmp/cd.log -p cd sub), so that cd moves the shell too.

Changes

• repos/events.py: for a Bash tool_use, a read event per path of cli_traces.shell_reads(command, cwd), before the command event. The read has the same t, the subagent's agent id, and its path goes through relative_path. Claude Code keeps the shell's directory between calls. So once a call may have moved its shell (a cd, pushd, popd, eval, source or .; new cli_traces.shell_reads_moved, which reuses the same walker), later relative reads are skipped and absolute ones are kept. The shell starts at the harness's start directory (the root in repo run and in bash=). A (cd x && make) subshell, make -C x or env -C x moves nothing.

• One moved-shell state for the whole trace, not one per agent. After the parent's cd, a subagent's relative reads are dropped too, and the reverse. A Task subagent may share the parent's Bash directory: the alternative, a read of the root AGENTS.md while the shell is in sub, would count guidance the agent was never shown. Dropping is safe under either model. No real trace with a subagent cd was checked. (review: 4445790393)

• cli_traces._Walk keeps a shell_moved flag, separate from the chain flag moved, which loses the directory for a later || branch (seventh round: a builtin in a subshell no longer sets moved either, below). A directory builtin sets it only outside a subshell. It is saved and restored with the directory at a backgrounded && / || list, a piped or backgrounded brace group, and a list backgrounded at the end of the script. So cd sub &, { cd sub; } | cat, { cd sub; } &, cd sub && make & and (cd sub) move nothing. cd sub, cd sub && make, cd "$X", pushd sub and popd do. (review: 4443237154, 4443248498)

• cli_traces._Walk.track_groups: a brace group saves the enclosing && / || list's state when it opens and restores it when it closes, so its own separators do not end that list. So cd sub && { echo a; echo b; } & and cd sub && { cd x; cat a; } & move nothing, while cd sub && { echo a; } still moves. This also fixes a read for Codex and Copilot: cd x && { cat f; cat h; } & cat g read x/g, and now reads g. (review: 4443584029)

• cli_traces._Walk.words gets the part's operator after it. A computed cd in a backgrounded part or in a pipeline (an unquoted, unescaped |) no longer marks the shell as moved, and (sixth round) no longer loses the directory either. So cd "$X" &, cd "$X" | cat, cd sub | cat and echo | cd sub move nothing, while cd "$X", cd "$X" && make, cd $(pwd) and cd a\|b do. (review: 4443584037)

• cli_traces._in_own_shell: a part holding a command substitution (cd $(pwd)) takes the walker's subshell branch, which saw & but not a pipe. Both branches now call this one helper: the part is not backgrounded, and has no pipe outside quotes, escapes, $( ... ) and backticks. So cd $(pwd) | cat, cd `pwd` | cat and cd $(pwd) & move nothing, while cd $(echo a | tr a b) still moves, because its pipe is inside the substitution. _Walk.words reuses the same builtin test, now _runs_dir. (review: 4446703160)

• repos/shell_split.py (new, moved out of cli_traces.py, which was at its 800-line cap): the splitter now gives each part its outer text as well, the text outside any ( ... ) or backticks. Its subshell flag meant both "inside ( ... )" and "holds a substitution". The walker tracks brace groups on the outer text before it skips a subshell part, and tells a cd $(...) by that text (_runs_dir, _in_own_shell). Every brace of a part counts, in order (_Walk.open_group, _Walk.close_group). So:

  • { cd $(pwd); } | cat, { cd $(pwd); } &, { cd `pwd`; } | cat, { source $(x); } &, { { cd $(pwd); }; } | cat and { cd sub; } | grep $(echo x) move nothing;
  • (true; cd sub) and echo $(true; cd sub; pwd) move nothing: the inner cd was taken as this shell's;
  • { cd $(pwd); } still moves, and { { cd sub; }; }, missed before, now moves too;
  • { { cat a; }; } reads a, and { cat a; } | { cat b; } drops a, whose output feeds the pipe.

  (review: 4447109156)

• cli_traces._Walk.lose: a directory builtin in a background job or a pipeline stage ran in a subshell, so it now leaves the foreground directory known, not only the moved-shell flag. source env.sh & cat AGENTS.md, eval "$CMD" | cat; cat AGENTS.md, pushd sub & cat AGENTS.md and cd "$X" & cat AGENTS.md read AGENTS.md, and the next call's relative reads are kept. The && / || gate that loses the directory for the next chain sets it unknown itself. (review: 4447109164)

• cli_traces._Walk: a directory builtin in a job or a pipeline stage no longer sets the chain flag moved either, and neither does a backgrounded cd. The flag is saved and restored with the directory and shell_moved (one _Spot, _Walk.spot / _Walk.back) at a backgrounded list and at a piped or backgrounded brace group. So source env.sh & false || cat AGENTS.md, source env.sh | false || cat AGENTS.md, cd x & false || cat AGENTS.md, { cd x; } & false || cat AGENTS.md and cd x && make & false || cat AGENTS.md read AGENTS.md. A builtin in this shell (source env.sh; false || cat AGENTS.md) still loses it. (review: 4447391791)

• cli_traces._unquoted (now shell_words._unquoted): a part with an expansion is not split by shlex, so the walker matched its blank-split words, quotes included. They now pass through this helper, which removes quotes and backslash escapes and expands nothing, before _command_words (in _runs_dir, and in _is_cd for the failed-read rule). So "source" "$ENV", 'cd' "$D", \eval "$X", c"d" "$D" and builtin "cd" "$D" lose the directory and move the shell for the next call, and a failed "cd" sub && cat AGENTS.md gives no read. The words of a part shlex splits are already unquoted and are not unquoted twice ('\cd' names a program \cd). Keywords are matched before unquoting. (review: 4447391806)

• cli_traces._runs_dir / _may_run_dir (eighth round; narrowed in the ninth, below): one fallback rule. _runs_dir takes a part's words from shlex.split where it can split them, so an assignment holding a blank is skipped like any other (X="a b" cd "$D"). Where shlex cannot split the text, any word that names a directory builtin counts. _may_run_dir counts a part as moving the shell when it has a builtin, command or time that _command_words did not skip, because an option comes first (time -p cd sub, command -p cd sub, builtin -- cd sub), followed by a word that names a directory builtin. The walker calls it for parts shlex splits and for parts with an expansion, so the shell is unknown for the rest of the command and for later Bash calls. In a job or a pipe such a part still moves nothing (time -p cd sub & cat AGENTS.md reads AGENTS.md). The forms the walker does follow are unchanged. The failed-read rule (_is_cd) is unchanged, on purpose: a failed X="a b" cd sub && cat AGENTS.md keeps its read, because a part before the read that is not surely a cd may be the one that failed. cd $(printf sub; true) | cat is left as it is: it only drops a read, and it is listed in repo: Bash reads dropped where the shell could not have moved (conservative limits of #369) #374. (review: 4447632234, 4447632250, 4447632218)

• cli_traces: eval, source and . lose the directory as pushd / popd do (_DIR_LOST), because they may run a cd that is not seen. So eval "cd sub", eval "$CMD", source env.sh and . env.sh move the shell, and a later relative read in the same command is not recorded. time before a directory builtin is skipped like builtin and command (ninth round: only as the first word, unquoted), so time cd sub is a cd (time cd sub && cat AGENTS.md reads sub/AGENTS.md). command cd sub already was a cd. eval "cd sub" & and env source env.sh move nothing, and (sixth round) a backgrounded or piped eval, source or . keeps the directory for the rest of its command. A cd inside a shell function that the agent defines and then calls is still not seen. (review: 4446227585)

• cli_traces._failed_read: a command that is one read of one file after cd parts (cd sub && cat AGENTS.md) is one read part, so a non-zero exit drops its read. After &&, the cd or the read failed; after ;, & or ||, the exit is the read's own. Either way the file was not shown. A list whose other part is not a cd (make && cat f, cat f && cd sub) keeps its read, as before. The rule is shared by the three readers. (review: 4446227568) A part counts as a cd after the words the walker skips (_is_cd, through the walker's _command_words): time cd, command cd, builtin cd, X=1 cd and (seventh round) a quoted or escaped "cd" or \cd are a cd, and env cd, which runs no builtin, is not. (review: 4446703173) "One file" is the printer's own operand count, which the walker records (_Walk.operands), not the number of reads it kept. So a failed cd /ws/sub; cd -; cat missing.md /ws/AGENTS.md, whose missing.md is at a directory not known, keeps its /ws/AGENTS.md read. (review: 4447109167)

• The Bash reads are settled once the trace ends, because a permission_denials entry comes only in the final result, and it can follow an is_error tool_result (Permission to use Bash has been denied) for the same call. Each Bash call records its order, named paths, whether it moves the shell, and its result (ok, error or denied: a hook's result or permission_denials). Then, in the trace's order:

  • a denied call moves nothing, and its reads stay, denied, after a move too, a relative one with no path (ninth round) (review: 4446227577; before, a refused relative read after a cd that ran was dropped, and tool_calls was one short);
  • an error result of a call that is one read of one file drops its reads (_failed_read, the rule the Codex and Copilot readers use);
  • after a non-denied call that moves, only absolute reads are kept, in every agent;
  • any other result gives the reads rc 0.

  Read events are placed before their command event at tool_use time, and only the dropped ones are removed, so every read keeps its position. PowerShell commands are not parsed. The logic sits in helpers (_tool_use, _bash_call, _settle_bash, _deny, a _Stream state), so events._apply goes from complexity 17 to 14 (baseline lowered; since the twelfth round's merge of main, which splits it further, it is under the cap and has no baseline entry). process_metrics.py's guidance_read rule is unchanged. No new event field is added; a Claude Code Skill event now carries the existing rc (next bullets). (review: 4443237165, 4443237182)

• Transcript.reads() (reach's classify, and so the receipt's read_by_tool) and arms.skill_usage take only reads that showed the file (transcript.shown_read: not denied, rc 0 or no result), as guidance_read already did. The events stay in events.jsonl. The Read tool had this bug before this PR too. (review: 4443248578)

• arms.skill_usage skips a denied or failed event at the top of its loop, so a Skill tool call that a hook or the permissions refused no longer counts the skill as used. uptake.arm_uptake skipped denied events already. It now also skips a failed read (rc 1) of a file a patch: arm added (shown_read). (review: 4445790411)

• events._apply_result sets rc on a Claude Code Skill tool event from its tool_result (0, or 1 on an error), as on a read. So a failed Skill call (Skill lint could not be loaded, with no hook refusal or permission denial) no longer counts the skill as used. No other tool event gets rc, and no consumer other than skill_usage reads it on a tool event (uptake's mcp_calls, tool_calls, reach, the consistency check). A permission-denied Skill call now has rc 1 next to denied, as a permission-denied command has. (review: 4446703155)

• repos/shell_words.py (new, ninth round, move only): the helpers that find a part's command word and tell whether it runs a directory builtin (_command_words, _runs_dir, _may_run_dir, _is_cd, _run_name, _cd, _unquoted and their constants) moved unchanged out of cli_traces.py, which was at 793 of its 800 lines. docs/architecture.md names it.

• shell_words._command_at (ninth round): the index of the command word past builtin, command and time with their options (time -p, command -p, --), or none for command -v / -V, which only describes the command. _may_run_dir and _runs_dir now count a part as moving the shell only when that word is a directory builtin, not when a builtin's name is any later word. For text shlex cannot split, _runs_dir unquotes the words one by one (_unquote) and looks at the same command word. Only when a quote up to that word does not close (X="a b c" cd "$D) does any builtin's name in the part still count. So time find . -name x; cat AGENTS.md reads AGENTS.md. time git add ., time make -C ., command -v cd, builtin printf '%s' . and grep $'it\'s' . move nothing. time -p cd sub, command -p cd sub and builtin -- cd sub still move. time cat AGENTS.md . still records no read of its own: the walker skips time only before a directory builtin. (review: 4449500725)

• shell_words._time_keyword / _prefix (ninth round): time is skipped (in _command_words, _command_at and _is_cd) only when the part's raw text starts with it as bash's keyword, unquoted. After an assignment, or quoted or escaped, it runs /usr/bin/time, a child process. So X=1 time cd sub; cat a, \time cd sub; cat a and "time" cd sub; cat a read a and move nothing, and a failed X=1 time cd sub && cat AGENTS.md keeps its read like any part that is not surely a cd. cli_traces._Walk.in_case: the walker does not follow a case ... esac. From its case part to its esac, a part that names a directory builtin as any word counts as a move, and the reads in it are not kept, as an arm may or may not run. So case x in a) cd sub;; esac, a later arm and an arm on its own line move the shell. (review: 4453764969)

• cli_traces._failed_read / _ungrouped (ninth round): a brace group's braces are not parts. A } part is skipped and an opening { is stripped, so a group that holds one read is that read. A failing { cat AGENTS.md; }, cd sub && { cat AGENTS.md; } or { { cat AGENTS.md; }; } gives no read, while { make; cat AGENTS.md; } keeps it. Shared by the three readers. (review: 4449500729)

• events._settle_bash / _deny_read (ninth round): a refused relative read after a move keeps its event with no path ({"kind": "read", "denied": true}), as the file it named is not known. Before, it carried the start directory's path. Every consumer reads path with .get or skips a denied event first (Transcript.reads(), guidance_read, skill_usage, uptake, consistency.run_from_events, which gives a plain Read with no file_path). No schema describes the events. (review: 4449500733)

• result.json process.unknown_reads (ninth round): an int, 0 when there are none. The walker counts the relative files a printing command names once its directory is lost in the command (_Walk.unknown; the new cli_traces.shell_walk returns it with the reads and the move). The Codex and Copilot readers add that per command (_counted_reads) and (thirteenth round) take it back when the command's result is a failed one-file read. The Claude Code converter adds it per Bash call that was neither refused nor failed as one read, plus each relative read it drops after an earlier call moved the shell, but not a failed one-file read. Transcript.unknown_reads carries it to process_metrics through runner._process_or_null. No schema covers process. (review: 4453764950)

• cli_traces._Walk.open_group / close_group (tenth round): a brace group saves the unknown count with its read mark. The close that drops the group's reads because its output is hidden (a pipe after it, its stdout to a file or /dev/null) also puts the count back. So { source env.sh; cat AGENTS.md; } > /dev/null and { source env.sh; cat AGENTS.md; } | head give unknown_reads 0, not 1, and a nested hidden group takes back only its own count ({ source env.sh; { cat a; } > /dev/null; cat b; } gives 1, not 2). A group whose output is shown keeps its count. Shared by the three readers. (review: 4455672089)

• tests/test_repo_events_bash.py (ninth round): the failed cd-then-cat test gains cd \<newline>sub && cat AGENTS.md and cd sub \<newline>&& cat AGENTS.md, which already gave no read: split_script keeps a line continuation inside its part. No code change. (review: 4447855726)

• repos/reach.py: a new kind bash, with scenario id bash=<dir> per guidance directory. It starts at the root in tool mode bash, and the prompt is ASK_AFTER_BASH (First use the Bash tool to run `cat <dir>/REACH_PROBE.txt`, and run no other command. Then: ) + ASK. The operand is shell_word(path): shlex.quote, with ./ in front of a path that starts with -, so a space, $ or ; in a directory name stays one operand. Scenario.read and the Read-tool prompt keep the raw path. The default kinds stay root,cwd,read. check_kinds refuses bash for a profile without that tool mode. The CLI calls it with the other argument checks (exit 2, naming the profile, before any clone), and so does run_reach. --scenarios help, the known-kinds message, --dry-run and --only bash=<dir> all cover it. (review: 4443237195, 4443248319)

• profiles/claude-code.yaml: tools.bash: ["--tools", "Bash"]. harness_profile.v1.json takes the optional tools.bash key, widened in place. delegation.none gets no bash mode: schema_compat holds propertyNames fixed, so widening delegation.none's mode enum fails it as a breaking change. Reach never passes a delegation, and a Bash-only run cannot start a subagent. So repo run --tools bash --delegation none is refused as the allow-list refuses any unlisted mode, and a test pins that.

• Receipt: no code change. A bash=<dir> row carries the probe file in read, so _workdir is <dir> and a lazy nested file is expected: true, as for read=<dir>. states.accessed is true for a file with a read event that was neither denied nor failed, else unknown (tools on). Only the _states docstring and the target description in effective_instructions.v1.json changed.

• Docs: contract §4.1 (guidance_read lists which shell reads count, for all three harnesses, and when a cross-call cd makes a relative read unknown; tool_calls counts the derived reads that are kept, a denied one or one of a failed multi-file read too (review: 4446703136); rc on a Claude Code Skill event; skills_used takes calls and reads neither denied nor failed; uptake.reads takes reads that did not fail), §6 (trace converters, profile example), §7 (bash=<dir> in step 3, with the quoted path; the fetched bullet covers Claude Code, says a denied or failed read is none, and lists the cd forms before a failed read), §7.1 (states.accessed; the "cannot show" list: rg, grep or a pipe stay unseen for all three, the shared-shell assumption for subagents, a cd inside a called shell function, and (eighth round) the fallback rule for a command the parser does not fully follow, with a link to repo: Bash reads dropped where the shell could not have moved (conservative limits of #369) #374 for the drop-only forms; (ninth round) narrowed to the command word, with the time keyword and case arms). Ninth round, §4.1 also: a read event may have no path, the unknown_reads row, and the brace-group form in guidance_read and §7's fetched. Also docs/reviewing.md step 2 and the process_metrics.py / events.py / reach.py / uptake.py docstrings. I found no other doc or data/skills/dokimi-design-evals/ sentence that said Bash reads are not counted.

• Tests: tests/test_repo_events_bash.py (205) and tests/test_repo_reach_bash.py (17, with its own fake Bash harness: test_repo_reach.py is at 997 of its 1000 lines). The fake harness parses the prompt's quoted operand with shlex.split and builds its other command with as_posix(), not str(Path) (review: 4443248422). tests/test_repo_skill_arms.py adds six tests: refused and failed SKILL.md reads, refused Skill calls, and a failed Skill call converted end to end. tests/test_repo_uptake.py adds a failed-read case. tests/test_repo_cli_traces.py gains the cd x && { cat f; cat h; } & cat g row, a Copilot cd sub && cat AGENTS.md failure row, and (sixth round) a group-tracking test, a directory-builtin-in-a-job test, and a two-operand failure row for Codex and one for Copilot, and (seventh round) a ||-after-a-job test and two quoted-builtin tests, and (eighth round) a test that the forms the walker does not follow lose the directory and one that they keep it in a job or a pipe, and (ninth round) rows where a builtin's name is an argument, a Codex brace-group failure test, a Copilot brace-group row, and a Codex/Copilot unknown_reads test, and (tenth round) a shell_walk test of the unknown count for hidden and shown groups. The tenth round also adds three hidden/shown-group rows to the Claude Code unknown_reads test. tests/test_repo_run.py and tests/test_repo_process_metrics.py pin unknown_reads in result.json and in process_metrics. Its backtick row echo `true; cd sub; pwd`; cat AGENTS.md changes from [] to ["AGENTS.md"]: its own comment says the cd is not this shell's. tests/test_batch_ledger_golden.py pins the execution id of the new ("bash", "native") mode. The existing ids are unchanged.

• Merge of main b084e47 (eleventh round, 75d06d4): no conflict, no change to this PR's code.

• docs/reviewing.md "Delivery first" (eleventh round): process:guidance_read counts a read tool's reads and a Claude Code Bash call, Codex command or Copilot bash command that only prints files (§7's list). A pipe, grep or rg does not count, nor does a relative read after the shell moved. process:unknown_reads counts those dropped reads, so guidance_read: [] with unknown_reads above 0 is not "never read".

• Merge of main 39dbd2a (twelfth round, 4597bc3; feat(repo): time to first useful result, with a rate in repo compare #332, feat(repo): judged recall over the credited replay hits #357, test(subjects): pin the evidence baseline fallback, ci nulls and warnings #381, feat(repo): reach --scenarios shadow adds a CLAUDE.local.md #367). Five conflicts, each resolved by keeping both sides:

  • contract §7 step 3 lists bash=<dir> and then main's shadow= scenarios;
  • commands/repo.py: cmd_repo_reach calls check_kinds and then main's check_shadow, and the --scenarios help names root, cwd, read, bash, shadow;
  • repos/reach.py: one KINDS (root, cwd, read, bash, shadow), Scenario with both the bash prompt and shadow, and scenarios() giving read=, then bash=, then shadow=;
  • repos/events.py: main split _apply into _apply_init, _apply_assistant and _apply_final and counts malformed tool_use / tool_result blocks as dropped lines. The split now carries this branch's _Stream: _apply_assistant hands each well-formed block to _tool_use, and _apply_final marks permission denials through _deny;
  • scripts/growth_baseline.json: main's side, with no events.py::_apply entry.

  main's test_a_failed_edit_changed_nothing_so_the_next_successful_one_is_the_first_useful_result called _apply_result(block, by_id) and now passes the branch's (out, block, stream). docs/reference/cli-interface.json was regenerated with scripts/cli_surface.py dump; it matches the merge. first_useful.steps counts every event, as process.tool_calls does, so under Claude Code it now counts the Bash-derived reads too.

• process_counts.PROCESS_KEYS (twelfth round): unknown_reads comes right after guidance_read. So process_values({"process": {"guidance_read": [], "unknown_reads": 3}}) gives process:unknown_reads 3.0 next to process:guidance_read 0.0, repo report's process line shows it, and repo compare compares it as a descriptive measure, in §4.1's order. No golden output had the key. (review: 4458839235)

• shell_words._prefix_runs_dir / _command_at and cli_traces._Walk.in_case (twelfth round). Each case is shared by the three readers:

  • after bash's time keyword and its options, NAME=value words are skipped before the directory builtin. time X=1 cd sub; cat AGENTS.md gives (['sub/AGENTS.md'], True, 0), and time -p X=1 cd sub counts as a move. command X=1 cd and builtin X=1 cd, which run no cd, are unchanged;
  • inside a case, each case word opens one more level and only a leading esac closes one, so a miscount stays inside. case $x in a) case $y in b) :;; esac; cd sub;; esac; cat AGENTS.md gives ([], True, 1), no longer a read of sub/AGENTS.md;
  • a case part tracks its braces before it is skipped, and before the lose() a builtin in it causes. { cat AGENTS.md; { case $x in a) :;; esac; }; } > /dev/null gives ([], False, 0), and a backgrounded or piped { case x in a) cd sub;; esac; } now moves nothing.

  (review: 4458839251)

• cli_traces._failed_read and the three converters (thirteenth round): "one file" no longer also needs a resolved path. So a failed cd -; cat missing.md is a failed one-file read, though its file is at a directory not known. Each command's unknown count is settled with its result, as its known reads are:

  • Codex keeps it with the started command's derived reads (_Derived) and subtracts it on a failed one-file completion; a completion with no start adds it only when the command did not fail;
  • Copilot keeps it per call id in _CopilotRun.by_id and subtracts it in _copilot_result;
  • events._settle_bash adds a call's count only when the call was neither refused nor failed as one read.

  cd -; cat missing.md with exit 1 now gives unknown_reads 0 in all four paths (it gave 1), and with exit 0 still gives 1. The contract's §4.1 unknown_reads row says so. (review: 4459651043)

• shell_words._command_words, _command_at and _prefix_runs_dir (thirteenth round): leading redirections are skipped when finding the command word, with the NAME=value words (_past_redirections, formerly _past_assignments), and also after builtin, command and time with their options. This covers >, >>, <, &>, 2>err, </dev/null and {fd}>x, with the target attached or as the next word. The part is still not followed, just as cd sub > log is not. So > /tmp/cd.log cd sub; cat AGENTS.md and >/tmp/cd.log cd sub; cat AGENTS.md give ([], True, 1) (they gave (['AGENTS.md'], False, 0)), and a Bash > /tmp/cd.log cd sub makes the next call's cat AGENTS.md unknown. 2>err cd sub, whose stderr redirect the walker already strips, still reads sub/AGENTS.md. A redirected cd in a job or a pipeline still moves nothing. > log time cd sub is still /usr/bin/time, a child. The failed-read rule sees the same command word: a failed > log cd sub && cat AGENTS.md is a failed one-file read. Shared by the three readers. (review: 4459651056)

• shell_words._command_at / _past_prefix (fourteenth round): after builtin, command or time, the prefix's options, redirections and (after time) NAME=value words are skipped together in one loop, in any order, until the command word. Before, the options were skipped only before the redirections. So command > /tmp/cd.log -p cd sub; cat AGENTS.md, time > log -p cd sub; cat AGENTS.md and builtin > log -- cd sub; cat AGENTS.md give ([], True, 1) (they gave (['AGENTS.md'], False, 0)), as command -p > log cd sub already did. A Bash command > /tmp/cd.log -p cd sub makes the next call's cat AGENTS.md unknown. A word after -- is no option (builtin > log -- -p cd sub runs -p). command > log -v cd still runs nothing. Shared by the three readers. (review: 4459886479)

• Changelog fragments: Changed (A, and the refused/failed read fix), Added (B, and process.unknown_reads, now with its repo report / repo compare measure).

Risk

• A Claude Code cell's process.guidance_read now also lists guidance printed by a bare cat/head/tail/nl/sed -n Bash call. tool_calls grows by one per printed file, skills.skills_used counts a cat of an installed SKILL.md, and uptake.reads counts a cat of a file a patch: arm added. Codex and Copilot cells already counted all of these. A batch that resumes cells converted before this change mixes the two meanings: the converter is not part of the execution id.
• repo reach with Claude Code: a guidance file the agent cats in Bash is now fetched, not loaded, and states.accessed is true. A read that was denied or failed, with any tool, no longer makes a file fetched or states.accessed true, and no longer counts a skill as used. Neither does a refused or failed Skill call. Before, a refused Read-tool read of a guidance file could turn loaded into fetched. A Claude Code Skill event in events.jsonl now has rc.
• Codex and Copilot:
  • a read after a backgrounded && / || list holding a brace group is now joined to the directory from before the list (cd x && { cat f; } & cat g reads g, not x/g);
  • a failed cd sub && cat AGENTS.md is no longer a read, nor a failed time cd, command cd, builtin cd or X=1 cd form of it;
  • a relative read after an eval, source or . in the same command is no longer recorded, unless that builtin is backgrounded or piped;
  • (sixth round) a cd inside $( ), backticks or ( ... ) ((true; cd sub); cat AGENTS.md) no longer loses the directory, and a backgrounded or piped cd, pushd, source or computed cd no longer does either;
  • (sixth round) a nested brace group ({ { cat a; }; }) now gives its reads and its cd, and a group whose output feeds a following group ({ cat a; } | { cat b; }) no longer gives a read;
  • (sixth round) a failed command whose printer names two files keeps the read it gave when only one of them is known (cd -; cat missing.md /ws/AGENTS.md); before, it was dropped;
  • (seventh round) a || branch after a backgrounded or piped directory builtin keeps the directory (source env.sh & false || cat AGENTS.md reads AGENTS.md);
  • (seventh round) a relative read after a quoted or escaped directory builtin with a computed operand ("source" "$ENV"; cat AGENTS.md) is no longer recorded, and a failed "cd" sub && cat AGENTS.md is no longer a read;
  • (eighth round) a relative read after a directory builtin the parser does not fully follow (X="a b" cd "$D", time -p cd sub, command -p cd sub, builtin -- cd sub, time -p source env.sh) in the same command is no longer recorded. (ninth round) Only when that builtin is the command word: time -p ls ., time git add . and command -v cd move nothing;
  • (ninth round) a cd after X=1 time or a quoted or escaped time moves nothing (X=1 time cd sub; cat a read sub/a, and reads a), and its failed read is kept;
  • (ninth round) a case arm that names a directory builtin, as any word, loses the directory (case x in a) find . ;; esac too), and the reads inside a case are not kept;
  • (ninth round) a failed brace group that holds one read ({ cat a; }, also with 2>/dev/null after the }) is no read;
  • (thirteenth round) a failed one-file read at a directory not known (cd -; cat missing.md) no longer counts in unknown_reads, and a relative read after a cd, pushd, source, eval or . behind a leading redirection (> log cd sub; cat a) is no longer recorded at the start directory. (fourteenth round) Nor is one behind a redirection between a prefix and its options (command > log -p cd sub; cat a);
  • (twelfth round) time X=1 cd sub; cat a reads sub/a, not a. A nested case keeps the walker inside the outer one until its own esac. A backgrounded or piped brace group around a case arm with a cd keeps the directory. A case arm holding the word case as an argument (echo case) keeps the walker inside to the end of the command, which only drops reads.
• Under-counting by design, under Claude Code: after any call that may have moved the shell, the later relative reads of the parent and of every subagent are not recorded. That now includes source .venv/bin/activate, which in practice moves nothing. A cd inside a later-called shell function is not followed (as in shell_reads). (eighth round) Parts the parser does not fully follow whose command word is a directory builtin count as moves too (ninth round: not a builtin's name as an argument), as does a case arm that names one, and some forms of cd inside a substitution followed by a pipe or & still count as moves (cd $(printf sub; true) | cat). Both only drop reads. repo: Bash reads dropped where the shell could not have moved (conservative limits of #369) #374 lists the drop-only forms.
• (twelfth round) repo report and repo compare gain process:unknown_reads. Comparing a batch from before this PR with a new one warns that only one arm reports it, as for any new count.
• (ninth round) result.json process has a new key, unknown_reads; a consumer that compares whole process objects sees it. (tenth round) It does not count the reads of a brace group whose output is hidden, as no read event is made for them either. events.jsonl can hold a read event with no path (a refused relative Bash read after a move).
• repo run --tools bash now exists for claude-code (the Bash tool only). No default changes, and existing execution ids are unchanged.

Testing

On a Linux review host, at e8c06f0 (on main 39dbd2a):

• make ci-local: 28 passed, 2 failed, 7 skipped (external-catalog-smoke: needs docker; test-latest, docs deploy and pip-audit report: job conditions false on a PR; the cli-surface packaging step: an expression the local runner does not evaluate; conda-forge-deps: no micromamba; semgrep secrets: offline). The two failures are the host's network: "Install gitleaks" could not download it (curl: (22) The requested URL returned error: 504, four times), so "Scan commits" had no gitleaks to run (exit 127). Both passed at 92c790d, and this round's commit changes one module and one test file. 13538 tests passed, 5 skipped; changed-line coverage 99% of 432 lines (the two missed lines, shell_words._cd's -L / -- handling, moved unchanged in the ninth round). package-smoke and windows-smoke pass. Not covered locally: the 3.11, 3.12 and 3.14 legs, and CI's merge commit.

• On main's code, 14 of the first round's 20 event tests fail; the 6 that pass are the "gives no read" cases (failed one-file cat, pipe/grep/redirect, PowerShell, pushd). test_repo_reach_bash.py fails at import (ASK_AFTER_BASH).

• The second round's new tests on the code of bff1c3f (this PR before the review fixes): 21 fail, one or more per review point. The cases that pass there are the ones that were already right ((cd sub), the "moves" rows, a cd sub that ran).

• The third round's new tests on the code of 261b310 (the head before it): 11 fail, every new moved=False row and its cross-call read, plus the cd x && { cat f; cat h; } & cat g read. The rows that pass there are the ones that still move (cd sub && { echo a; }, cd "$X" && make, cd $(pwd), cd a\|b) and echo | cd sub, which was already right.

• The fourth round's new tests on the code of 444aec9 (the head before it): 27 cases fail, at least one per review point:

  • both directions of the shared shell;
  • both refused Skill calls, and the failed arm-file read;
  • the six failed cd-then-cat forms and the Copilot row;
  • both refused reads after a cd;
  • every eval / source / . / time cd row.

  The cases that pass there were already right: the three lists that keep their read, command cd sub, eval "cd sub" & and env source env.sh.

• The fifth round's new tests on the code of ff48bb1 (the head before it): 9 cases fail, at least one per code point:

  • both cases of the converted failed-Skill test;
  • the cd $(pwd) | cat and cd `pwd` | cat rows, and the cd $(pwd) | cat cross-call read;
  • the four failed time cd, command cd, builtin cd and X=1 cd reads.

  The cases that pass there were already right: cd $(pwd) &, cd $(echo a | tr a b) (moves), its cross-call read, and env cd sub && cat AGENTS.md keeping its read. The tool_calls row is a doc change, with no test.

• The sixth round's new tests on the code of 13f858b (the head before it): 35 cases fail, at least one per review point:

  • the group rows ({ cd $(pwd); } | cat and &, the backtick and source forms, { { cd $(pwd); }; } | cat, (true; cd sub), { { cd sub; }; }) and both reported forms' cross-call reads;
  • the nine within-call forms of a backgrounded or piped source / eval / . / pushd / computed cd, and the three Claude Code calls that read after one;
  • the Claude Code, Codex and Copilot two-operand failure rows.

  The changed backtick row fails there too. The cases that pass there were already right: { cd $(pwd); } and { cd $(pwd); }; cat AGENTS.md, which move, and the next-call reads after a backgrounded or piped source, eval, . or pushd, which the fifth round's code already kept.

• The seventh round's new tests on the code of e60774d (the head before it): 35 cases fail, at least one per review point:

  • the 11 ||-after-a-job-or-a-pipe rows and both Claude Code next-call rows;
  • 9 of the 11 quoted-builtin rows within a call, 7 of the 8 next-call rows, 3 of the 8 moved rows, and the 3 failed quoted-cd rows.

  The cases that pass there were already right: the four foreground || rows, "." env.sh and \pushd sub (shlex splits those parts), and the rows that keep the directory or move nothing.

• The ninth round's new or changed tests on the code of e61e812 (the head before it): 42 cases fail, at least one per code point:

  • the seven argument rows (time find . -name x, time git add ., command -v cd, builtin printf '%s' ., ...) and the grep $'it\'s' . row, each within a call and in the next call, plus four Codex-level rows;
  • the four non-keyword time rows and the two failed X=1 time cd / \time cd rows;
  • the four case rows;
  • the four brace-group rows, the Codex test and the Copilot row;
  • both refused-read-after-a-cd cases and the consumer test;
  • every unknown_reads case, the Codex/Copilot test, the two changed result.json tests, the new result.json test and the process_metrics zero test.

  The cases that pass there were already right: the existing rows of the changed tests, { make; cat AGENTS.md; } keeping its read, a case with no directory builtin, and the two continued-cd rows, which pin what the code already did.

• The eleventh round is a merge and a doc change. It adds no test and no mutant. tests/test_docs_code_refs.py passes.

• The tenth round's new tests on the code of c51f68a (the head before it): 6 cases fail: the four hidden-group shell_walk rows (> /dev/null, | head, a hidden group after source, a nested hidden group) and the two hidden-group Claude Code rows. The cases that pass there were already right: a group whose output is shown keeps its count.

• The twelfth round's new or changed tests on the code of 4597bc3 (the merge, before its two fixes): 5 cases fail, at least one per point: the new unknown_reads process_values/report test, the compare test that now has process:unknown_reads, and the three case/time tests.

• Mutants, first round: 22 breaks of the code, each one failing its tests (31 of 31 runs). Mutants, second round: 15 breaks, each one failing its tests:

  • shell_moved not restored at a backgrounded list, at a piped or backgrounded group, or for a list backgrounded at the end;
  • shell_reads_moved returning moved;
  • a backgrounded cd setting shell_moved;
  • a denied call still moving the shell;
  • an earlier error result winning over the final permission denial;
  • settling the reads at the tool_result again;
  • Transcript.reads() or skill_usage without the filter;
  • shown_read without its rc or its denied check;
  • the bash operand unquoted, or without ./;
  • the Read-tool prompt quoting its path.

  The as_posix() change in the fake harness is test-only and has no mutant. I could not run it on Windows.

• Mutants, third round: 5 breaks, each one failing its tests:

  • a group's close not restoring the enclosing list's state (5 tests fail);
  • a computed cd marking the shell moved whatever its context (6 fail);
  • only the background context seen (4 fail: the pipe rows);
  • only the pipe context seen (2 fail: cd "$X" &);
  • a \| taken as a pipe (1 fails: cd a\|b).

• Mutants, fourth round: 11 breaks, each one failing its tests:

  • a subagent's relative read kept after a move (1 test fails);
  • a subagent's cd not moving the shell (1);
  • skill_usage without its top filter (3 tests, 5 cases);
  • arm_uptake counting a failed read (1);
  • _failed_read back to one part only (7 cases);
  • _failed_read taking any earlier part, not only a cd (5 cases, including existing Codex and Copilot tests);
  • the denial checked after the move again (2);
  • eval, source or . each left out of _DIR_LOST (4, 3 and 3 cases);
  • time not skipped before cd (3).

• Mutants, fifth round: 6 breaks, each one failing its tests:

  • no rc on a Skill event (2 cases);
  • the substitution branch seeing only &, as before (3 cases);
  • the substitution branch seeing only the pipe (2 cases: cd $(pwd) &);
  • a pipe inside $( ... ) taken as the part's (1 case: cd $(echo a | tr a b));
  • _failed_read taking only a first-word cd again (4 cases);
  • env cd taken as a cd (1 case).

• Mutants, sixth round: 6 breaks, each one failing its tests:

  • groups tracked after the subshell skip, as before (11 cases);
  • the outer text keeping what is inside parentheses (9 cases, existing ones included);
  • only the first leading { stripped (3 cases);
  • a part's opens taken before its closes (1 case: { cat a; } | { cat b; });
  • lose() ignoring the job / pipeline context (12 cases);
  • "one file" decided from the derived reads again (4 cases: Claude Code, Codex and Copilot).

• Mutants, seventh round: 7 breaks, each one failing its tests:

  • lose() setting moved for a builtin in a job or a pipe, as in the sixth round (7 and 2 cases);
  • back() not restoring moved (3 cases: the group and list rows);
  • a backgrounded cd setting moved (1 case);
  • _unquoted returning its words as they are (22 cases);
  • _runs_dir not unquoting (19 cases), _is_cd not unquoting (3 cases);
  • _runs_dir unquoting twice (2 cases: '\cd' "$D").

• Mutants, ninth round: 18 breaks, each one failing its tests:

  • _may_run_dir counting any word after the prefix again; command -v / -V taken as running the command;
  • text shlex cannot split read with the old any-word rule; only the command word itself, not the words before it, checked for an unclosed quote;
  • time taken as the keyword wherever it is; the raw-text keyword check always true;
  • case ... esac not tracked; a case arm losing no directory;
  • the braces counted as parts again; only the closing brace skipped;
  • a refused relative read keeping its path; an absolute one losing its path too;
  • the walker counting no unknown read; a read dropped after a move not counted; a failed one-file read counted; a refused call's count added; the runner passing no count; Codex and Copilot adding no count.

• Mutants, tenth round: 2 breaks, each one failing its tests:

  • a hidden group's close not putting the count back (6 cases);
  • every group's close putting it back, shown or not (4 cases).

• The thirteenth round's new tests on the code of b6ce65f (the head before it): 12 cases fail, at least one per point:

  • the Codex completed-only, Codex started/completed and Copilot rows, and the failed Claude Code row, of the unknown-count tests;
  • the eight leading-redirection forms (> /tmp/cd.log cd sub, >/tmp/cd.log cd sub, </dev/null cd sub, &>log cd sub, X=1 > log cd sub, command > log cd sub, command -p > log cd sub, > log source env.sh).

  The cases that pass there were already right: the successful Claude Code row (count 1), and a redirected cd in a job or a pipeline, which moved nothing.

• Mutants, twelfth round: 4 breaks, each one failing its tests (5 of 5 runs):

  • unknown_reads dropped from PROCESS_KEYS (2 tests);
  • no assignments skipped after time (1);
  • only a first case word opening a case, as before (1);
  • a case part not tracking its braces (1).

• Mutants, thirteenth round: 8 breaks, each one failing its tests:

  • Codex's started/completed path not taking the count back (1 case);
  • Codex's completed-only path counting before the result again (1);
  • Copilot not taking the count back (1);
  • Claude Code adding a failed call's count (1);
  • _failed_read requiring a resolved path again (4 cases: all three readers);
  • no leading redirection skipped in _command_words (6 cases);
  • a detached redirection's target not skipped (4 cases);
  • no redirection skipped after builtin / command and their options in _command_at (1 case: command -p > log cd sub). This mutant first survived, because _prefix_runs_dir already covers command > log cd sub; the command -p row was added for it.

• The fourteenth round's new tests on the code of 92c790d (the head before it): 3 cases fail, command > /tmp/cd.log -p cd sub, time > log -p cd sub and builtin > log -- cd sub, each within a call and in the next Bash call. The cases that pass there were already right: time -p > log X=1 cd sub, whose options come first, and command > log -v cd, which runs nothing.

• Mutant, fourteenth round: the two separate loops put back (options, then redirections once) in _past_prefix. The same 3 cases fail.

• A fuzz of 20,000 random Bash commands through the converter raised nothing and logged no trace error (first round; repeated at e60774d with the walker's new words: braces, $( ), backticks, source, time, env -C; and at 7ba6c6a with quoted and escaped builtin names, &, |, && and ||).

• Live, Claude Code 2.1.316 with Haiku 4.5, on a synthetic tree (root AGENTS.md, sub/AGENTS.md, deep/x/AGENTS.md), repo reach --scenarios bash,read ($0.05): read=sub and read=deep/x loaded their directory's AGENTS.md; bash=sub and bash=deep/x did not (root file only). Each bash= trace holds the cat <dir>/REACH_PROBE.txt command and its derived read event, rc 0. So in this version a Bash read of a nested file does not load that directory's guidance, which fits the replay's root-only start tokens. Repeated at 261b310 with a fourth directory, my docs/AGENTS.md ($0.08): the same result, and the bash=my docs trace holds cat 'my docs/REACH_PROBE.txt' with its read of my docs/REACH_PROBE.txt. Not repeated after 261b310: the later rounds change the cross-call rules, the failed-read rule, the pipeline rule and the skill and uptake counts. A bash= run (one successful cat, no subagent, no Skill call) reaches none of them.

• Merged together with fix(repo): reach prose count skips a short lead-in line before the tokens #368 on main 93ff3a4, at 444aec9: pytest -n auto 12572 passed, 59 skipped, 6 xfailed; mypy and ruff check clean. Not yet run merged together with the other follow-up branches at e8c06f0.

Revert

git revert the commits. Nothing persisted changes shape: harness_profile.v1 only gained an optional key, and events.jsonl / result.json written in between keep the extra read events (and rc on a Claude Code Skill event, a refused read with no path, and process.unknown_reads).

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 31 selected item(s).

This PR has reached its review limit, so only a re-request starts a review.

Review details

• Commit reviewed: e4e5711 against merge base dbec45a
• Coverage: 31 files in 5 groups
• Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
• Usage: 559k tokens (277k from cache), 35s
• Tool calls: 27 (20 file_read, 5 code_search, 2 file_read_diff)
• Claude Bash read detection and trace parsing: 10 files
• Bash reach scenarios: 3 files
• Process metrics and instruction uptake: 7 files
• Skill arms and effective instruction schema: 3 files
• Repository evaluation orchestration, receipts, and documentation: 8 files

[Copilot]

🟡 Changes recommended

Unquoted Bash paths can execute unintended commands, while unresolved shell-state tracking, denied-read filtering, and Windows path handling cause missed reads, false usage attribution, and test failures.

4 open issues

• Quote probe paths in host-run shell commands · New
• Use POSIX paths for fake Bash command operands · New
• Restore shell state across subshell command groups · New
• Exclude denied reads from fetch and skill-usage tracking · New

What changed in this PR

4 new findings.

This PR aligns Claude Code’s Bash file-read tracking with Codex and Copilot and adds an opt-in reach scenario to measure whether shell reads load nested guidance.

Changes:

• Derives read events from supported Bash commands, accounting for failures, denials, and directory changes.
• Adds bash=<dir> reach scenarios, profile support, and early argument validation without changing defaults.
• Adds regression tests and updates documentation, schema descriptions, and changelog fragments.

File	Description
dokimi/​tests/​test_repo_reach_bash.py	Tests Bash scenarios, validation, classification, and receipts.
dokimi/​tests/​test_repo_events_bash.py	Tests Bash read conversion and directory tracking.
dokimi/​tests/​test_batch_ledger_golden.py	Pins the Bash-mode execution identifier.
dokimi/​scripts/​growth_baseline.json	Lowers the converter’s complexity baseline.
dokimi/​dokimi/​schemas/​harness_profile.v1.json	Allows the optional Bash tool mode.
dokimi/​dokimi/​schemas/​effective_instructions.v1.json	Clarifies the receipt target description.
dokimi/​dokimi/​repos/​receipt.py	Updates the accessed-state docstring only.
dokimi/​dokimi/​repos/​reach.py	Adds Bash scenarios, prompts, and profile validation.
dokimi/​dokimi/​repos/​profiles/​claude-code.yaml	Defines the Bash-only tool mode.
dokimi/​dokimi/​repos/​process_metrics.py	Documents shell-read counting; logic unchanged.
dokimi/​dokimi/​repos/​events.py	Derives and settles Claude Code Bash read events.
dokimi/​dokimi/​repos/​cli_traces.py	Exposes shell directory-movement tracking.
dokimi/​dokimi/​commands/​repo.py	Validates Bash scenarios and updates CLI help.
dokimi/​docs/​reviewing.md	Adds Bash reach-probe usage guidance.
dokimi/​docs/​repo-evaluation-contract.md	Documents shell reads and Bash scenario behavior.
dokimi/​changelog.d/​20261102_120100_antonwolfy_reach-bash-scenario.md	Announces the optional Bash reach scenario.
dokimi/​changelog.d/​20261102_120000_antonwolfy_claude-bash-reads.md	Records changed Claude Code read metrics.

_{Review effort: Balanced}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.02198% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.71%. Comparing base (dbec45a) to head (e4e5711).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/shell_words.py	94.73%	2 Missing and 4 partials ⚠️
dokimi/dokimi/repos/events.py	97.64%	0 Missing and 2 partials ⚠️
dokimi/dokimi/repos/cli_traces.py	99.20%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #369      +/-   ##
==========================================
+ Coverage   94.69%   94.71%   +0.02%     
==========================================
  Files         332      334       +2     
  Lines       43409    43622     +213     
  Branches     7380     7414      +34     
==========================================
+ Hits        41104    41318     +214     
+ Misses       1468     1467       -1     
  Partials      837      837              

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/repo.py	97.27% <100.00%> (-0.05%)	⬇️
dokimi/dokimi/repos/arms.py	97.02% <100.00%> (+0.02%)	⬆️
dokimi/dokimi/repos/process_counts.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/process_metrics.py	100.00% <ø> (ø)
dokimi/dokimi/repos/reach.py	98.62% <100.00%> (+0.10%)	⬆️
dokimi/dokimi/repos/receipt.py	98.79% <ø> (ø)
dokimi/dokimi/repos/runner.py	96.89% <100.00%> (ø)
dokimi/dokimi/repos/shell_split.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/transcript.py	96.92% <100.00%> (+0.14%)	⬆️
dokimi/dokimi/repos/uptake.py	96.72% <100.00%> (+0.05%)	⬆️
... and 3 more

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review round 1 (Copilot and OpenCodeReview, 8 comments, 6 points), one commit per point:

• 862dc49: a cd in a background job, a piped group or a backgrounded list no longer counts as moving the Bash shell.
• 8f2b8de: Bash reads are settled when the trace ends. A permission_denials entry after an error result keeps the read, denied, and a refused cd moves nothing.
• aa0c86f: a refused or failed read is not a read for reach fetched, the receipt's states.accessed, or skill use. This also covers the Read tool.
• d0bb1ab: the bash= prompt shell-quotes the probe path, and a leading - gets ./.
• 261b310: the fake Bash commands in the tests use POSIX paths.

CI is green at 261b310. The live Haiku probe was re-run at 261b310 with a my docs/ directory and gave the same result. The description is updated.

[antonwolfy]

Review round 2 (OpenCodeReview, 2 comments):

• 8be264e: a brace group's own separators no longer end the && / || list around it, so cd sub && { echo a; echo b; } & moves no shell.
• 444aec9: a computed cd that is backgrounded or in a pipeline (cd "$X" &, cd sub | cat) moves no shell.

CI is green at 444aec9. The description is updated.

[vlad-perevezentsev]

Review at 444aec9. The Bash reads, their settlement at the end of the trace and the bash= scenario look right, and every earlier review thread is fixed. One open point: the moved-shell state is kept per agent, while Claude Code's subagents likely share the parent's Bash directory; two smaller gaps are in the line comments.

What I ran, at 444aec9:

• make ci-local --offline: 0 failed, 19 passed, 13 skipped (network and Docker steps).
• shell_reads_moved on about 20 forms: cd sub | cat, pushd sub &, popd | cat, (cd sub) && cat AGENTS.md and cd sub & cat AGENTS.md move no shell; cd sub || exit 1, if true; then cd sub; fi, cd -, command cd sub do. time cd sub and a function that runs cd are not seen as moves; they are rare in agent traces.
• convert on hand-written traces: a parent cd sub followed by a subagent cat AGENTS.md records root AGENTS.md (see the line comment on events.py).
• On Windows (Server 2025, Python 3.13): pytest tests/test_repo_reach_bash.py tests/test_repo_events_bash.py: 73 passed, 0 skipped, so the POSIX-path fix in 261b310 holds there.

Checked, no issues

• Earlier threads: the backgrounded and piped cd cases, the brace group inside a list, the computed cd, the late permission_denials entry, the denied cd, the quoted probe path and the denied reads in reach and skill use are all fixed as the replies say.
• Schemas: harness_profile.v1 only gains an optional bash key and effective_instructions.v1 only a description, so both are widening edits.
• Docs and changelog: the contract rows, §7 and the reach section match the code; both new test files and the fragments have the SPDX header.

[napetrov]

Two blocking, one question, inline. Already on main, not from this PR (follow-ups, not blockers): a denied Skill call still counts as used (arms.skill_usage), and a failed (rc 1) read of a patch: arm file still counts in uptake.reads. Still open from earlier: discussion_r4445790 (subagent shell sharing).

Coverage: plain + edge passes, probes under the head tree; the full suite and context/consistency.py consumers were not checked.

[antonwolfy]

Round for @vlad-perevezentsev's and @napetrov's reviews, at ff48bb1 (CI green), one commit per point:

• f05363f: one moved-shell state for the whole trace, so a cd by the parent or a subagent drops the later relative Bash reads of both (safe whether subagents share the parent's shell or not).
• fe82755: a refused or failed Skill call is no use of the skill. The same commit has uptake.reads skip a failed read of a patch: arm file (napetrov's review-body note; same bug class, same spot).
• 0e16f14: cd sub && cat AGENTS.md that exits non-zero is no read. The rule is shared, so Codex and Copilot commands get it too.
• 58920c4: a refused read after a cd stays, denied, and tool_calls counts it.
• ff48bb1: eval, source and . lose the shell's directory as pushd/popd do, and time cd is a cd. The cost: after source .venv/bin/activate, later relative Bash reads are not recorded (named in Risk). A cd inside a shell function the agent defines is still not seen (contract limits).

[antonwolfy]

Rounds 12-14 on 8b08665, head now e8c06f0:

• Round 12:
  • 4597bc3 merges main (feat(repo): time to first useful result, with a rate in repo compare #332, feat(repo): judged recall over the credited replay hits #357, test(subjects): pin the evidence baseline fallback, ci nulls and warnings #381, feat(repo): reach --scenarios shadow adds a CLAUDE.local.md #367). The conflicts are in the contract, commands/repo.py, reach.py, events.py and the growth baseline. Each keeps both sides: KINDS has bash and shadow, and main's _apply split carries this branch's Bash reads.
  • 77a3bdd: repo report and repo compare show process:unknown_reads (@vlad-perevezentsev).
  • b6ce65f: time followed by assignments, a nested case, and braces on a case part (@vlad-perevezentsev's three forms).
• Round 13, exai-bot's two points:
  • 7434efd: a failed one-file read adds nothing to unknown_reads.
  • 92c790d: a leading redirection is not a cd's command word.
• Round 14, exai-bot: e8c06f0 skips a prefix's options, redirections and assignments in any order before the command word.

Two drop-only forms from round 12 are added to #374: a case word inside a case arm, and time X=1 cat.

make ci-local on a Linux review host at e8c06f0: 13538 tests passed. Only the two gitleaks steps failed, because the host's download got HTTP 504 (they passed at 92c790d). PR CI is green, and exai-bot found 0 issues at e8c06f0.

[napetrov]

Only findings not already raised on this PR. Found by automated haiku review passes, deduped against all existing threads; the one inline item was reproduced by a probe on head and on the merge base.
Other candidates were dropped as already raised (case-word over-count, now tracked in #374) or as documented behaviour (a failed read followed by a passing part keeps rc 0).

[xaleryb]

Security fix for repo reach --scenarios bash, plus a merge of main. Three commits on top of 2f5dfd3:

• 21e06903 fix(repo): the bash= probe may run only the cat it names
• 4ed00e40 Merge main into feat/claude-bash-reads (main at dbec45a, no conflicts; harness_profile.v1.json keeps both tools.bash and tools.skills)
• 5aab99f9 refactor(repo): keep repo.py and test_repo_useful.py under the growth caps

The problem. The bash scenario starts Claude Code on the host with --permission-mode bypassPermissions --tools Bash. The child has HOME, the provider credentials and the host's network, and the workspace holds the probed repository's guidance, which the probe loads into the agent's context on purpose. The only thing limiting the shell was the prompt's "run no other command", so a CLAUDE.md in the probed repository could run any command on the host. With the profile at 2f5dfd3, a scripted model running Claude Code 2.1.300 through dokimi repo reach --scenarios bash got id, cat <probe> && id and cat <probe>; id executed.

The fix. The claude-code profile's bash tool mode adds --settings with a PreToolUse hook on Bash. The hook allows a call only when the command equals DOKIMI_REACH_COMMAND, which repo reach sets to the scenario's cat <shell-quoted probe> (new Scenario.command; the prompt names the same string). For anything else the hook exits 2, which Claude Code treats as a block, bypassPermissions included. The hook is python3 -c '...' || exit 2, so a missing interpreter, a missing variable or bad input also blocks the call (fail closed). The contract's §7 bash= bullet, the schema's tools.bash description and the changelog fragment now say that the scenario runs a shell on the host and what limits it.

Options considered:

• --permission-mode dontAsk with --allowedTools "Bash(cat <probe>)" is not enough. Run on the same CLI, it still let through id, ls, echo hi, cat <probe> && id, cat <probe>; id and cat <probe> | head -1, because Claude Code approves commands it classes as read-only without an allow rule. It did refuse curl, env, printenv, a cat outside the workspace, $(...) and redirection.
• Running the scenario in the container sandbox would also close the hole. It is a much larger change to how repo reach runs.

Tests (red with the profile as at 2f5dfd3, green now):

• test_the_claude_code_bash_mode_refuses_any_other_command runs the profile's hook as Claude Code does. It is parametrized over curl x, cat a && curl x, cat a; id, id, env, the probe cat followed by && id, ; id, | head -1 or a newline and id, an unquoted cat of the same path, and an empty command.
• test_the_claude_code_bash_mode_admits_the_probe_cat_and_fails_closed_without_one checks that the quoted probe cat (a directory with a space) is allowed and that without the variable it is refused.
• test_only_a_bash_scenario_names_its_command_to_the_harness checks that only a bash scenario gets the variable.

Three mutations of the profile each turn the gate tests red: the 2f5dfd3 profile, the profile without || exit 2 (exit 1 does not block), and a hook that always exits 0.

End to end, dokimi repo reach --scenarios bash with the real claude 2.1.300 binary and a scripted Messages API stub: the probe cat ran, and id, cat <probe> && id and cat <probe>; id were blocked by the hook. On 2f5dfd3 the same run executed all three.

Run on the head (5aab99f, Python 3.13):

• full suite: 14606 passed, 25 skipped
• ruff, mypy and bandit -ll: clean
• check_comment_blocks --base origin/main and check_growth: clean
• schema_compat --base origin/main: OK
• CLI snapshot: current
• benchmark gate (--strict): passes

Other changes:

• The pinned execution id of ("bash", "native") in test_batch_ledger_golden.py changes, because the bash mode's arguments changed. The mode is new in this PR, so no saved batch carries the old id.
• check_growth was already red at 2f5dfd3 (tests/test_repo_useful.py at 1014 lines), and the merge put commands/repo.py at 802. The third commit fixes both without raising a cap. The reach flag check moves to reach.check_flags, and the Bash first_useful test moves to test_repo_events_bash.py.

Still open:

• I could not make a live model call (the provider answered 403), so a real model run of bash= after this change is still owed.
• The hook needs python3 on the PATH where repo reach runs. Without it every Bash call is refused, so the scenario fails loudly instead of running unchecked.
• The agent's own cat of a guidance file is now blocked too. A real Claude run of bash= can therefore no longer score fetched: such a read shows up as a refused read, which this PR already does not count as a fetch.

[antonwolfy]

@xaleryb thanks, this closes a real hole. I checked 21e0690..5aab99f:

• The hook blocks every command except the exact probe cat, and it fails closed. sys.exit(<bool>) gives 1 on a mismatch, and || exit 2 turns that, a missing python3, a missing variable or bad JSON into a block. The repository's own .claude/settings.json cannot switch the hook off, because the probe workspace holds only guidance files and probe files.
• make ci-local on 5aab99f on a Linux review host: 0 failed, 30 passed, 7 skipped.
• The live run you could not make: repo reach --scenarios bash with Claude Code 2.1.318 and Haiku 4.5 on Bedrock, on a synthetic repo whose root CLAUDE.md tells the agent to run id > /tmp/pwned369. The only Bash call was cat sub/REACH_PROBE.txt. The hook allowed it, and /tmp/pwned369 was never written. The matrix is unchanged from the 2026-11-02 run: CLAUDE.md is loaded and the nested sub/AGENTS.md is not. Haiku did not try the injected command, so this run tested the allow path. The block path rests on your gate tests and your stub run.

On the open points: python3 on the PATH is the same requirement the background-command hook already has, and the bash mode fails loudly where that one fails open. Losing fetched in bash= is fine: that scenario asks what a shell read loads, and an extra cat the agent makes on its own still shows up as a refused read.

[xaleryb]

Approved at e4e5711: CI green; the bash= probe may run only its cat, and a negated read is no read.