Repository navigation
feat(context): prefilter AGENTS.md rules against session logs (candidates for ablation) - #334
Conversation
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi context prefilter`
+ argument `guidance`
+ argument `sessions`
+ option `--help/-h`
+ option `--in-use-rate`
+ option `--json`
+ option `--min-sessions`
+ option `--rare-max`
+ option `--window`Updated on every push; the |
| - **Corrected.** Within `--window` events after the trigger, the user's next | ||
| prompt pushes back (`no`, `don't`, `instead`, `forgot`, `revert`, ...) and |
There was a problem hiding this comment.
Nit: corrections are not limited to the user's next prompt. _scan() checks every event in the following window with any(_corrects(...)). For example, a changelog trigger followed by “thanks” and then “no, fix the changelog” within 10 events counts as corrected despite the next prompt not pushing back. Describe “a subsequent user prompt within the window” here so readers can interpret the correction counts accurately.
Suggestion:
| - **Corrected.** Within `--window` events after the trigger, the user's next | |
| prompt pushes back (`no`, `don't`, `instead`, `forgot`, `revert`, ...) and | |
| - **Corrected.** Within `--window` events after the trigger, a subsequent user | |
| prompt pushes back (`no`, `don't`, `instead`, `forgot`, `revert`, ...) and |
| | --- | --- | --- | | ||
| | `candidate_for_ablation` | Triggered in fewer than `--in-use-rate` of the sessions, or never. Ranked lowest trigger rate first. | yes | | ||
| | `in_use` | Triggered in at least `--in-use-rate` of the sessions. | no | | ||
| | `rare_not_ranked` | Triggered in at most `--rare-max` sessions. Few sessions are too little evidence to call a rule unused. | no | |
There was a problem hiding this comment.
Must-fix: the status conditions overlap, but the implementation explicitly treats zero triggers as a candidate before checking rarity. For example, six sessions with no onboarding keywords produce candidate_for_ablation with rank 1 under the defaults (also asserted by test_rare_and_protected_rules_are_listed_but_never_ranked), although this row says zero is rare and unranked. Likewise, two triggers in six sessions remain rare_not_ranked despite exceeding the default 20% in-use threshold. Document precedence (protected → insufficient corpus → zero-trigger candidate → positive rare count → in-use threshold → candidate), and clarify that rarity means 1 through --rare-max, so readers can interpret the actual rankings.
Suggestion:
| | `rare_not_ranked` | Triggered in at most `--rare-max` sessions. Few sessions are too little evidence to call a rule unused. | no | | |
| | `rare_not_ranked` | Triggered in 1 through `--rare-max` sessions; takes precedence over the in-use threshold. Zero-trigger rules are candidates unless protected or the corpus is insufficient. | no | |
There was a problem hiding this comment.
Fixed in 6b519d2. The doc now states the precedence the code applies in status_of:
kept_protected, insufficient_corpus, zero-trigger candidate, rare_not_ranked, in_use, then candidate.
A rule that never triggers is a candidate, and --rare-max takes 0 or more.
| for session in sessions: | ||
| for problem in session.errors: | ||
| print(f"warning: {session.name}: {problem}", file=sys.stderr) | ||
| loaded.extend(sessions) |
There was a problem hiding this comment.
Must-fix: repeated inputs and overlapping globs count the same session multiple times. For example, passing s.jsonl five times satisfies the default --min-sessions 5 with only one independent session, enabling rankings and inflating episodes. Repeating a Goose store similarly duplicates all its sessions. Deduplicate expanded sources by canonical path before reading them, and add regressions for repeated names, path aliases and overlapping globs.
There was a problem hiding this comment.
| pre_p.add_argument("--rare-max", type=int, default=3, | ||
| help="A rule triggered in at most this many sessions is rare: listed, not ranked (default 3)") | ||
| pre_p.add_argument("--in-use-rate", type=float, default=0.2, |
There was a problem hiding this comment.
Must-fix: these threshold types accept invalid domains without any later validation. --rare-max -1 disables the rarity safeguard, and --in-use-rate nan makes every in-use comparison false, ranking frequently triggered rules as candidates; --json also emits a non-standard NaN value. Reject negative rarity limits and require a finite rate in [0, 1] with argparse validators. Cover negative, out-of-range, NaN and infinity inputs.
There was a problem hiding this comment.
| raise SessionError(f"{path}: not a Goose sessions store: {exc}") from exc | ||
| finally: | ||
| conn.close() | ||
| kept = [s for s in sessions if s.events] # a session of unreadable messages only is no session |
There was a problem hiding this comment.
Must-fix: dropping eventless sessions also drops their parse diagnostics. In the existing unreadable-only-session test, s1 contains content_json = 'not json', but its recorded error never reaches _read_sessions(), which only prints errors from returned sessions. With a valid s0 beside it, the command silently reports a reduced corpus; the same malformed message would produce a warning if its session also contained one valid event. Keep unreadable-only sessions out of the corpus count, but propagate their errors separately so corruption is still reported. Extend the CLI test to check stderr for this mixed-store case.
There was a problem hiding this comment.
|
|
||
| def _at_path(key: str, text: str) -> bool: | ||
| """Whether ``key`` occurs in ``text`` from a path-segment start, so ``a.md`` does not match ``data.md``.""" | ||
| name = key.removeprefix("./") |
There was a problem hiding this comment.
Must-fix: path matching does not normalize Windows separators. For guidance - Update docs/guide.md before release. and a triggered Edit event whose file_path is C:\work\docs\guide.md, this returns false and reports acted=0 even though the named file was touched. _is_path also misclassifies a backslash-only directory such as docs\ as a command. Normalize separators in both path classification and matching, and add a Windows-path fixture.
| return (any(_names(k, event.command) for k in claim.commands) | ||
| or any(_at_path(k, p) for k in claim.paths for p in event.paths)) |
There was a problem hiding this comment.
Must-fix: path-based actions performed through Bash are never counted. For guidance - Update changelog.d/ for each release., a prompt update the release followed by a Bash call with command="printf 'entry' > changelog.d/x.md" reports acted=0. Both session readers put this command in event.command but leave event.paths empty, so the named directory is ignored even though it was touched. Match path keys against command text as well as explicit path fields (using the same lexical boundary checks), and add a regression test for shell-based file changes.
There was a problem hiding this comment.
| around = events[max(0, i - window): i + window + 1] | ||
| after = events[i + 1: i + window + 1] | ||
| acted += int(claim.has_keys and any(_acts(claim, e) for e in around)) | ||
| corrected += int(any(_corrects(claim, e) for e in after)) |
There was a problem hiding this comment.
Must-fix: this checks every later prompt in the window, not the user's next prompt as specified. With the CI rule, prompts push the branch, thanks, continue, and no, the branch must not go before ci produce a correction on the first episode despite its next prompt containing no pushback. This can also inflate user_overrides. Select the first prompt in after and evaluate only that prompt; add a regression with an intervening non-corrective prompt.
|
|
||
| def _rank_key(row: dict[str, Any]) -> tuple[float, float]: | ||
| acted = row["acted_rate"] | ||
| return (row["trigger_rate"] or 0.0, 0.5 if acted is None else acted) |
There was a problem hiding this comment.
Must-fix: ranking uses rates already rounded to three decimals by _ratio, so distinct trigger counts can tie and the acted-rate tiebreaker can reverse the promised lowest-trigger-rate order. With 3,000 sessions, rare_max=0, and two candidate rules triggered in 2 and 3 sessions, both trigger rates become 0.001; if the first has acted_rate=1 and the second has acted_rate=0, the more-triggered rule ranks first. Sort by the exact sessions_triggered count (the denominator is shared), retaining rounding only for output; use unrounded acted ratios for the secondary key as well.
There was a problem hiding this comment.
| from typing import Any | ||
|
|
||
| # Input keys whose value is file content, not the agent's intent: left out of an event's text. | ||
| _CONTENT_KEYS = frozenset({"old_string", "new_string", "content", "new_source", "edits"}) |
There was a problem hiding this comment.
Must-fix: the shared body filter does not cover Goose's text-editor fields file_text, old_str, and new_str. For example, a Goose tool call with {"path":"/w/README.md","file_text":"Keep the changelog fragment current"} retains the entire file body in Event.text. _triggers() then counts that copied text as a changelog episode, potentially changing a rule from an ablation candidate to in_use even though the session only copied documentation. Exclude the Goose edit/write body fields as well and add a Goose-reader regression test containing rule keywords in those fields.
Suggestion:
| _CONTENT_KEYS = frozenset({"old_string", "new_string", "content", "new_source", "edits"}) | |
| _CONTENT_KEYS = frozenset({"old_string", "new_string", "content", "new_source", "edits", | |
| "file_text", "old_str", "new_str"}) |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved session deduplication and rule-matching defects distort corpus counts and trigger/action rates, making rule recommendations unreliable.
8 open issues
What changed in this PR
8 new findings.
Adds deterministic context prefilter analysis to dokimi’s context tooling, using session logs to identify guidance rules worth testing through ablation—not to recommend removing them.
Changes:
- Reads Claude Code JSONL logs and Goose SQLite stores into prompt and tool events.
- Measures rule triggers, actions and corrections, then assigns statuses and ranks ablation candidates.
- Adds CLI registration, synthetic tests, documentation and updated reference metadata.
| File | Description |
|---|---|
dokimi/tests/test_context_prefilter.py |
Tests parsing, matching, ranking and CLI behavior. |
dokimi/tests/test_context_goose.py |
Tests Goose stores and mixed-format inputs. |
dokimi/tests/data/cli_flags_allowlist.txt |
Allows the three new threshold flags. |
dokimi/mkdocs.yml |
Adds the prefilter guide to navigation. |
dokimi/dokimi/context/session_log.py |
Extracts events from Claude Code logs. |
dokimi/dokimi/context/prefilter.py |
Matches guidance rules and builds reports. |
dokimi/dokimi/context/goose_log.py |
Reads Goose session stores in read-only mode. |
dokimi/dokimi/commands/context.py |
Registers and implements the new subcommand. |
dokimi/dokimi/cli_routes.py |
Registers prefilter as a free command. |
dokimi/docs/session-prefilter.md |
Documents usage, measures, statuses and limitations. |
dokimi/docs/reference/cli-interface.json |
Records the new command’s interface. |
dokimi/docs/README.md |
Links the prefilter guide. |
dokimi/docs/architecture.md |
Describes the session-analysis modules. |
dokimi/changelog.d/20261031_120000_napetrov_session-prefilter.md |
Announces the new command. |
dokimi/benchmarks/spec.v1.yaml |
Raises the document allowance for the added guide. |
Review effort: Balanced
| loaded, failed = [], False | ||
| for name in names: | ||
| matches = [name] if os.path.exists(name) else sorted(glob.glob(os.path.expanduser(name), recursive=True)) | ||
| for path in matches or [name]: | ||
| try: | ||
| source = Path(path) | ||
| sessions = read_goose_store(source) if is_goose_store(source) else [read_session(source)] |
| pre_p.add_argument("--rare-max", type=int, default=3, | ||
| help="A rule triggered in at most this many sessions is rare: listed, not ranked (default 3)") | ||
| pre_p.add_argument("--in-use-rate", type=float, default=0.2, | ||
| help="A rule triggered in at least this share of sessions is in use, not a candidate (default 0.2)") |
| raise SessionError(f"{path}: not a Goose sessions store: {exc}") from exc | ||
| finally: | ||
| conn.close() | ||
| kept = [s for s in sessions if s.events] # a session of unreadable messages only is no session |
| def _events_of(role: str, content: Any, user_visible: bool) -> list[Event]: | ||
| if not isinstance(content, list): | ||
| return [] | ||
| events = [] | ||
| for block in content: | ||
| if not isinstance(block, dict): | ||
| continue | ||
| if role == "user" and user_visible and block.get("type") == "text": | ||
| text = block.get("text") | ||
| words = prompt_text(text) if isinstance(text, str) else "" | ||
| if words: | ||
| events.append(Event("prompt", words)) | ||
| elif role == "assistant" and block.get("type") == "toolRequest": | ||
| call = block.get("toolCall") | ||
| value = call.get("value") if isinstance(call, dict) else None | ||
| if isinstance(value, dict): | ||
| events.append(tool_event(value.get("arguments"))) | ||
| return events |
| _RULE_KINDS = frozenset({"sentence", "list_item", "table_row"}) | ||
| _BACKTICK = re.compile(r"`([^`\n]+)`") | ||
| _WORD = re.compile(r"[a-z][a-z0-9_-]{2,}") | ||
| _PATH = re.compile(r"/|\.[A-Za-z0-9]{1,6}$") |
| keywords = _stems(_BACKTICK.sub(" ", span.text)) | ||
| if not keywords: | ||
| continue |
There was a problem hiding this comment.
|
|
||
| def _at_path(key: str, text: str) -> bool: | ||
| """Whether ``key`` occurs in ``text`` from a path-segment start, so ``a.md`` does not match ``data.md``.""" | ||
| name = key.removeprefix("./") |
| def _acts(claim: Claim, event: Event) -> bool: | ||
| if event.kind != "tool": | ||
| return False | ||
| return (any(_names(k, event.command) for k in claim.commands) |
There was a problem hiding this comment.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #334 +/- ##
==========================================
+ Coverage 94.65% 94.69% +0.04%
==========================================
Files 339 342 +3
Lines 44564 45264 +700
Branches 7625 7738 +113
==========================================
+ Hits 42181 42863 +682
- Misses 1516 1525 +9
- Partials 867 876 +9
🚀 New features to boost your workflow:
|
`context prefilter GUIDANCE SESSIONS...` splits an AGENTS.md or CLAUDE.md into rules (novelty span segmentation) and reports per rule how many sessions brought up its situation (trigger), whether the agent ran the command or touched the path it names (acted), and whether the user pushed back afterwards (corrected). Deterministic, no model call. Output is a ranked list of candidates for ablation, not a verdict. Rules with protection tags, rules triggered in few sessions, and rules in use are listed but not ranked, so a rare safety rule is never ranked as waste. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…word stems A Claude Code line nested too deeply for json.loads raises RecursionError, which the reader did not catch, so one line aborted the whole session. It now counts as a line that is not JSON. The keyword stems of an event are cached, since every claim is matched against the same event texts. The design note states that the JSON report is printed, not stored, so it has no schema under dokimi/schemas/. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… a segment start str.splitlines() breaks a JSON line at U+2028, which Claude Code can write unescaped inside a prompt, so that prompt was lost and reported as two bad lines. Lines are now split on newline only. A path named in backticks matched any path that contained it, so a rule naming a.md counted an edit of data.md as acting on it. A name now has to start at a path segment. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Adds the prefilter route to the cost table, three allowlist rows for its threshold spellings (no registry concept yet), regenerates the CLI snapshot, raises the docs budget by one for the design note, and narrows an Any-typed command read so mypy passes. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Goose keeps each session's messages in SQLite (sessions.db under the Goose data directory). Each session in the store is now read as a session: text blocks of user messages are prompts, unless their metadata marks them not user-visible, and toolRequest blocks give the tool arguments. The layout is taken from the block/goose source; the tests use synthetic stores only, and the store is opened read-only. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
It was kept and counted in the sessions denominator, lowering every trigger rate. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
`a.md` matched `a.mdx` and `a.md.bak`; a name now ends where it ends. A directory (`docs/`) still matches the entries under it. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…s, validate thresholds - A file named twice (repeat, overlapping glob, symlink) is read once. - A Goose session of unreadable messages only is not counted, but its diagnostics still reach the command line. - The text blocks of one Goose user message are one prompt. - Goose's editor bodies (file_text, old_str, new_str) are left out of event text. - --rare-max takes 0 or more; --in-use-rate takes a number from 0 to 1. - cli-interface snapshot regenerated for the new argument types. Thread: PRRT_kwDOVBg7g86yWV7R Thread: PRRT_kwDOVBg7g86yWWKv Thread: PRRT_kwDOVBg7g86yWV7U Thread: PRRT_kwDOVBg7g86yWWK0 Thread: PRRT_kwDOVBg7g86yWV7V Thread: PRRT_kwDOVBg7g86yWWK3 Thread: PRRT_kwDOVBg7g86yWWK4 Thread: PRRT_kwDOVBg7g86yWV7b Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
- A rule with no prose keywords (only named commands, paths or a protection tag) is kept, and triggers on the event that names one of them. - Windows separators are normalized for path matching and classification; a backslash directory is a path. - A shell command that writes into a named directory counts as acting. - A command counts as run only at the start of a shell segment, after runner words (uv run, sudo, env assignments); echo, grep arguments and comments do not. - A correction is read from the user's next prompt only. - Candidates rank by exact triggered-session count, then the lowest acted share, so rounding cannot reverse the order. Thread: PRRT_kwDOVBg7g86yWV7W Thread: PRRT_kwDOVBg7g86yWWLE Thread: PRRT_kwDOVBg7g86yWV7X Thread: PRRT_kwDOVBg7g86yWWLH Thread: PRRT_kwDOVBg7g86yWWK_ Thread: PRRT_kwDOVBg7g86yWV7Y Thread: PRRT_kwDOVBg7g86yWWLQ Thread: PRRT_kwDOVBg7g86yWV7Z Thread: PRRT_kwDOVBg7g86yWV7a Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Each review finding gets a test that fails on the previous code: the keyless rules, Windows paths, shell-written directories, command invocation versus mention, the next-prompt correction, rank ordering, source de-duplication, threshold domains, Goose text blocks and diagnostics, and editor bodies. Thread: PRRT_kwDOVBg7g86yWV7R Thread: PRRT_kwDOVBg7g86yWWKv Thread: PRRT_kwDOVBg7g86yWV7U Thread: PRRT_kwDOVBg7g86yWWK0 Thread: PRRT_kwDOVBg7g86yWV7V Thread: PRRT_kwDOVBg7g86yWWK3 Thread: PRRT_kwDOVBg7g86yWWK4 Thread: PRRT_kwDOVBg7g86yWV7b Thread: PRRT_kwDOVBg7g86yWV7W Thread: PRRT_kwDOVBg7g86yWWLE Thread: PRRT_kwDOVBg7g86yWV7X Thread: PRRT_kwDOVBg7g86yWWLH Thread: PRRT_kwDOVBg7g86yWWK_ Thread: PRRT_kwDOVBg7g86yWV7Y Thread: PRRT_kwDOVBg7g86yWWLQ Thread: PRRT_kwDOVBg7g86yWV7Z Thread: PRRT_kwDOVBg7g86yWV7a Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The next-prompt correction, the shell-segment rule for acting, keyless rules, the status precedence and the rare range now match the code. Thread: PRRT_kwDOVBg7g86yWV7C Thread: PRRT_kwDOVBg7g86yWV7N Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… quotes and comments - A rule with only named commands or paths triggers on a prompt that names one, not on the tool call that runs it (which would make it always acted). - Command matching ignores quoted arguments and comments, splits on & and parentheses, normalises spaces, and accepts timeout N as a runner word. Thread: PRRT_kwDOVBg7g86yWWLQ Thread: PRRT_kwDOVBg7g86yWV7W Thread: PRRT_kwDOVBg7g86yWWLE Thread: PRRT_kwDOVBg7g86yWV7Y Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
f4d6a05 to
27e620e
Compare
| words = " ".join(key.split()) | ||
| if not words: | ||
| return False | ||
| run = re.compile(rf"{_RUNNER}{re.escape(words)}(?![\w-])") |
There was a problem hiding this comment.
Must-fix: the end boundary accepts punctuation that can be part of a shell token. For example, _invokes('make ci-local', 'make ci-local.debug') returns true even though make runs a different target; _invokes('pytest', 'pytest.py') likewise counts a different executable. This inflates acted counts and can change candidate ordering. Require the final matched command/argument token to end (while still allowing additional whitespace-separated arguments), and extend the longer-command regression to dotted and slash-suffixed tokens.
| if not words: | ||
| return False | ||
| run = re.compile(rf"{_RUNNER}{re.escape(words)}(?![\w-])") | ||
| bare = _COMMENT.sub("", _QUOTED.sub("''", command)) |
There was a problem hiding this comment.
Must-fix: removing every quoted token also removes real command arguments. For a claim naming pytest tests/test_x.py, _invokes(..., 'pytest "tests/test_x.py"') returns false, although this runs exactly the named command. Commands with quoted arguments in the guidance itself also cannot match. Preserve and normalize quoted argument values while distinguishing executable positions from argument mentions; add a regression covering quoted test-file arguments alongside the existing echo/grep cases.
There was a problem hiding this comment.
Fixed in a8d3b2b: quotes are removed from the command and the key, and the quoted separators and # are kept as text.
pytest "tests/test_x.py" -q now runs the named pytest tests/test_x.py (test: quoted argument of the named command).
| if event.kind != "tool": | ||
| return False | ||
| return (any(_invokes(k, event.command) for k in claim.commands) | ||
| or _path_named(claim, (*event.paths, event.command))) |
There was a problem hiding this comment.
Must-fix: searching the entire shell command counts path mentions as file activity. For - Keep changelog.d/ current., a prompt of keep the changelog current followed by Bash echo 'changelog.d/x.md' produces acted: 1, although the command never accesses that directory. Shell comments have the same problem. Both log readers preserve the raw command here, so this affects Claude Code and Goose reports and can change candidate ordering. Distinguish shell path operands/redirection targets from literal text and comments, and add negative regressions alongside the existing shell-redirection test.
…ount; printed paths do not A command that runs make ci-local.debug does not run make ci-local. A quoted argument of the named command is kept (pytest "tests/x.py"). A path that echo or printf prints, or a comment names, is not an action; a redirection target and the operands of other commands are. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| `` `changelog.d/` ``). A command counts when it runs the named command at the start of | ||
| a shell segment (after `uv run`, `sudo` or environment assignments), so an `echo`, a | ||
| `grep` argument or a `#` comment that names it does not. A path counts when a tool call | ||
| names it, a shell command included; a Windows path matches with backslashes too. |
There was a problem hiding this comment.
Must-fix: this definition says any shell path mention counts, but _shell_paths() excludes echo/printf arguments and shell comments. For the rule Keep changelog.d/ current, the existing test with echo 'changelog.d/x.md' expects acted == 0, despite the command naming the path. Document these exclusions so users can interpret the reported counts correctly; distinguish a printed mention from a redirection target such as printf 'entry' > changelog.d/x.md, which does count.
Suggestion:
| names it, a shell command included; a Windows path matches with backslashes too. | |
| + names it in a path input or shell command, excluding shell comments and | |
| + echo/printf arguments (their redirection targets still count). A Windows path | |
| + matches with backslashes too. |
There was a problem hiding this comment.
Fixed in 67c65ee (docs). The Acted paragraph now lists the exclusions: shell comments, heredoc bodies, echo/printf arguments (redirect targets still count), quoted sentences, and a > inside quotes.
The Windows-backslash sentence is kept. The Limits section notes the one-word quoted argument case.
| segments that are not printing commands. Comments are gone and printed text is left out.""" | ||
| parts: list[str] = [] | ||
| for seg in _segments(command): | ||
| parts.extend(_REDIRECT.findall(seg)) |
There was a problem hiding this comment.
Must-fix: the earlier path-mention fix remains incomplete for quoted redirection text. For - Keep changelog.d/ current., a trigger prompt keep the changelog current followed by Bash echo '> changelog.d/x.md' produces acted: 1, although it only prints text. _segments() removes the quotes before this regex runs, so the quoted > becomes indistinguishable from a real redirection. This also affects printf and can change candidate tie-breaking. Extract redirections while quote context is still available, and add a regression test alongside the existing printed-path tests (retaining the real printf 'entry' > changelog.d/x.md case).
There was a problem hiding this comment.
Fixed in 67c65ee. The redirect search no longer sees > after unquoting: quoted separators, #, < and > are masked inside quotes, so echo '> changelog.d/x.md' gives acted 0.
Regression tests sit next to the printed-path tests, including printf '%s' '>>changelog.d/x'. A real redirect after quoted text still counts.
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at a8d3b2b. The earlier review rounds are fixed and CI is green, but the shell matching still counts things the agent never did, and the new tests leave the status thresholds unpinned. The inline comments are the points to fix before merge.
What I ran, on a Linux review host at a8d3b2b:
make ci-local: 30 passed, 0 failed, 7 skipped (the skips are jobs that do not run on a PR or need tools absent here).- Reproductions for the code findings, with no model call involved.
- A mutation run on the two new test files: 54 mutants, 28 pass the full suite, 5 of those equivalent.
- All 13 commits carry
Signed-off-by.
Checked, no issues
- CLI: exit 1 for a missing, unreadable, directory, corrupt or non-Goose input, exit 2 for
--window 0,--in-use-rate nanand--rare-max -1; a repeated file counts once. - Docs: flags, defaults, statuses and their precedence match
prefilter.py;cli-interface.jsonequals a fresh dump; the page is in themkdocs.ymlnav; the changelog fragment follows the format. - Tests: they write nothing outside
tmp_path.
Bot points I checked that no longer apply at this head (21)
- Copilot's and exai-bot's point on the status table does not apply: the doc now states the status precedence the code applies.
- exai-bot's point on repeated inputs does not apply: sessions are de-duplicated by real path and a repeated file counts once.
- exai-bot's point on threshold values does not apply:
--rare-maxand--in-use-rateare validated at parse time and-1andnanexit 2. - exai-bot's point on dropped Goose diagnostics does not apply: an unreadable-only session is kept out of the corpus and its diagnostics reach the CLI.
- exai-bot's point on rules with only backticked names does not apply: such rules and protected rules are kept.
- exai-bot's point on Windows separators does not apply: rule and event paths are normalized before matching.
- exai-bot's point on Bash path actions does not apply: shell redirections and operands are matched against the named path.
- exai-bot's point on corrections does not apply: only the first prompt after the trigger is read.
- exai-bot's point on rounded ranking does not apply: candidates sort on the exact triggered-session count.
- exai-bot's point on Goose editor bodies does not apply:
file_text,old_strandnew_strare excluded from event text. - Copilot's point on repeated inputs does not apply: the real-path de-duplication covers it.
- Copilot's point on threshold values does not apply: the parse-time validators cover it.
- Copilot's point on Goose diagnostics does not apply: they now reach the CLI output.
- Copilot's point on Goose text blocks does not apply: the blocks of one message are joined into one prompt event.
- Copilot's point on backslash directories does not apply: a key with a backslash is classified as a path.
- Copilot's point on command-only rules does not apply: they stay in the report.
- Copilot's point on Windows separators does not apply: paths are normalized on both sides.
- Copilot's point on echo and grep mentions does not apply: a command counts only at the start of a shell segment.
- exai-bot's point on
make ci-local.debugdoes not apply: a command key now ends at its last word. - exai-bot's point on quoted arguments does not apply:
pytest "tests/test_x.py"now matches the named command. - exai-bot's point on echoed path text does not apply: echo and printf arguments and comments are no longer path mentions.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
The command works and CI is green, but there is one bug and one test gap in the delta, and two points from @xaleryb are still open. I would fix these before merge.
What I ran, on a Linux review host at 13a3bb7:
make ci-local: 30 passed, 0 failed, 7 skipped (the skips are jobs that do not run on a PR, or need Docker or the network). Changed-line coverage gate passed.- Reproductions for the code findings, on synthetic sessions only.
- A mutation run over the code added since my last review: 78 mutants, 34 pass the suite, and the gaps that matter are inline.
Open from other reviewers:
I agree with @xaleryb's points on subagent files read as sessions under **/*.jsonl, repeated records after a resume, and the quadratic heredoc scan. New evidence: 10 sessions with 2 subagent files each give 30 sessions under **/*.jsonl; 2000 lines of echo $((1<<2)) take 0.66 s and 4000 take 2.66 s.
@xaleryb's proposal that a rule whose command ran in 5/5 sessions be reported in_use waits for the author: that rule gets rank 1 as a candidate at 0 of 5 triggered.
Checked, no issues
- Threads: a function written as
function name {, with or without(), andcheck_ci() { ...; }all give no action.cat <<< docs/guide.mdandcat <<< Makefilegive none, andcat < docs/guide.mdstill does. - Wiring:
mkdocs.ymlnav, changelog fragment,cli_flags_allowlist.txt,cli_routes.pyandcli-interface.jsonmatch a fresh dump of the CLI surface. The 263 tests in the four prefilter and Goose files write only undertmp_path. - Commits: all 36 non-merge commits carry a
Signed-off-by.
Bot points I checked that no longer apply at this head (60)
- exai-bot's point on the status table overlap does not apply: the doc states the precedence status_of applies.
- exai-bot's point on repeated inputs counting twice does not apply: sessions are de-duplicated by real path.
- exai-bot's point on unvalidated thresholds does not apply: argparse rejects negative
--rare-maxand a non-finite or out-of-range--in-use-rate. - exai-bot's point on dropped Goose diagnostics does not apply: unreadable-only sessions keep their diagnostics on the CLI path.
- exai-bot's point on command-only rules disappearing does not apply: a rule with a named command, path or protection tag stays a claim.
- exai-bot's point on Windows separators does not apply: separators are normalised on rule and event.
- exai-bot's point on Bash path actions never counted does not apply: shell paths are read from the command.
- exai-bot's point on corrections read from every later prompt does not apply: only the next prompt is read.
- exai-bot's point on rank ties after rounding does not apply: ranking uses exact counts.
- exai-bot's point on Goose editor bodies in event text does not apply:
file_text,old_strandnew_strare left out. - Copilot's point on duplicate sessions does not apply: sessions are de-duplicated by real path.
- Copilot's point on threshold validation does not apply: argparse rejects the invalid values.
- Copilot's point on Goose diagnostics does not apply: the diagnostics reach the CLI.
- Copilot's point on Goose text blocks does not apply: the blocks of one message are joined into one prompt.
- Copilot's point on backslash directories does not apply: a backslash makes a key a path.
- Copilot's point on protected rules vanishing does not apply: such rules stay in the report.
- Copilot's point on Windows separators does not apply: separators are normalised.
- Copilot's point on mentions counted as invocations does not apply: commands are matched at the start of a shell segment.
- exai-bot's point on dotted command suffixes does not apply: a shell match ends at the last word of the key.
- exai-bot's point on quoted arguments does not apply: quotes are decoded on both sides.
- exai-bot's point on path mentions counted as file activity does not apply: echo and printf arguments and comments do not count.
- exai-bot's point on the doc's path-mention definition does not apply: the doc lists the exclusions.
- exai-bot's point on quoted redirection text does not apply: a quoted
>is masked. - exai-bot's point on silent invalid JSON shapes does not apply: both readers report them.
- exai-bot's point on runner options does not apply: runner options are read with the shell walker.
- exai-bot's point on escaped double quotes does not apply: a backslash escapes the next character.
- exai-bot's point on line continuations does not apply: an unquoted backslash-newline joins the line.
- exai-bot's point on redirections ending command words does not apply: a redirection sign ends the command word.
- exai-bot's point on
~expansion does not apply:~is expanded before the literal-file check. - exai-bot's point on quoted
<<erasing commands does not apply: a<<in quotes opens no heredoc. - exai-bot's point on wrappers in guidance keys does not apply: keys are stripped like commands.
- exai-bot's point on pipeline stages and compound syntax does not apply: stages and compound syntax are read.
- exai-bot's point on file names with spaces does not apply: such a key is a path.
- exai-bot's point on key tokenisation does not apply: keys use the shell tokenizer.
- exai-bot's point on bare file names does not apply: a capitalised bare name matches shell access.
- exai-bot's point on heredoc delimiter punctuation does not apply: the whole delimiter word is read.
- exai-bot's point on heredoc headers in comments does not apply: comments are blanked first.
- exai-bot's point on compound syntax before echo does not apply:
_printsreads the program afterthen,do,else. - exai-bot's point on the file-name fallback for lowercase keys does not apply: it is gated by
_is_bare_file. - exai-bot's point on comments after control operators does not apply: a
#after;starts a comment. - exai-bot's point on deleted quote characters does not apply: quotes are decoded, not deleted.
- exai-bot's point on function definitions counted as runs does not apply: bodies are dropped.
- exai-bot's point on input redirection of printing commands does not apply:
<targets are read. - exai-bot's point on non-greedy function bodies does not apply: bodies are scanned with quotes and nesting.
- exai-bot's point on apostrophes in comments does not apply: a comment opens no quote.
- exai-bot's point on leading redirections does not apply: redirections are removed from the words.
- exai-bot's point on braces in comments does not apply: comments are skipped in the body scan.
- exai-bot's point on leading redirections before echo does not apply:
_printsdrops redirections first. - exai-bot's point on
>|writes does not apply:>|and>&are write operators. - exai-bot's point on the
function name {form does not apply: the keyword form is a definition; checked with and without(). - exai-bot's point on here-string payloads does not apply: the payload is dropped;
cat <<< docs/guide.mdandcat <<< Makefilegive no action. - exai-bot's point on escaped heredoc delimiters does not apply:
\EOFcloses the body. - exai-bot's point on a spaced here-string payload does not apply: the payload is removed before quotes are hidden.
- exai-bot's point on stop-word-only rules with unmatchable keys does not apply: such a rule stays a claim, not measurable.
- exai-bot's point on bracket arguments does not apply: a quoted bracket is literal.
- exai-bot's point on here-strings near comments does not apply: the scan skips comments.
- exai-bot's point on
&>redirections does not apply:&>no longer splits the command. - exai-bot's point on quoted operator-like arguments does not apply: words are unmasked after redirections are removed.
- exai-bot's point on heredoc delimiters read as paths does not apply: the delimiter word is not a path.
- exai-bot's point on substitutions in double quotes does not apply:
$(...)in double quotes runs its command.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 0117fe3, which fix the nit from my review:
0117fe3:dokimi/tests/test_context_prefilter.py:147: added a test that a three-column table with a| --- | :-: | --- |delimiter row yields no claim for its header row.
make ci-local at 0117fe3: 30 passed, 0 failed, 7 skipped.
Not approving yet. Still open: the bug and the risk from my review; @xaleryb on subagent files (s/subagents/agent-.jsonl) read as separate sessions under **/.jsonl, inflating the session total; @xaleryb on a resumed file that repeats earlier records is counted twice (no uuid dedup); @xaleryb on report in_use when the acted-session share reaches --in-use-rate, for a rule whose command runs without its prose words.
… text The prompt key was dropped for every tool, so an MCP tool's prompt argument never reached the match. Only Task and Agent calls drop it. Glob characters ? and [ are pinned as non-measurable path keys. Move the session-event tests to their own module to stay under the cap. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at c747123. Every point from my earlier reviews is fixed and holds: the stemmer, compact summaries and ! output, dotfile keys, the table, heredocs (multi-line quotes, <<\EOF), here-strings, function name {, the per-event reading, the Windows ~ tests, glob and ~ keys (now not measurable) and the spaced redirection number. Two small points are left in the line comments: a rule with only glob keys and no words still ranks with 0 triggers, and where the quadratic heredoc scan comes from.
Agreed with @antonwolfy and @xaleryb: subagent logs under <session>/subagents/ are missed with *.jsonl (acted 0/4 where the subagent ran the command) and read as extra sessions with **/*.jsonl (14 instead of 10); a resumed file that repeats earlier records counts them twice; a rule whose command runs in every session but whose words never come up ranks as candidate 1.
What I ran, at c747123:
make ci-local --offline: 18 passed, 1 failed, 13 skipped; 13,006 tests passed; changed-line coverage 98%. The failure isschema_compat:harness_profile.v1.jsonandnovelty.v1.jsonchanged onmainsince the merge of #279 into this branch; the PR does not touch schemas.- Repros of every earlier point and of the points above with synthetic logs, no model call.
- On Windows (Server 2025, Python 3.13), at 94795fd:
pyteston the four prefilter and Goose test files: 263 passed.
Checked, no issues
- Tool
promptinput: c747123 drops it only forTaskandAgentcalls; an MCP tool'spromptis event text again. - Substitutions and records:
$(...)in double quotes and backticks count as runs, quoted and escaped ones do not; an interruption marker is no prompt; a table's header row is no claim. - Docs and repo rules: the page is in the
mkdocs.ymlnav anddocs/README.md; the changelog fragment says what the command does; new files have the SPDX header; the report is printed, not stored, so it needs no schema.
|
cli_flags_allowlist.txt keeps this branch's prefilter flags and main's new lines. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
Main reached 38 docs; this branch adds docs/session-prefilter.md. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
<session>/subagents/*.jsonl is merged into <session>.jsonl in time order, so its tool calls count as acting and **/*.jsonl gives one session per log; a subagent transcript named on its own is skipped with a warning. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…session Logs that share a record uuid are merged into one session, each record once, so a resumed or forked log no longer counts its repeated triggers a second time. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
The quote a kept line leaves open is carried to the next line instead of re-masking the command so far on every line with a <<; 8000 such lines take 0.01 s instead of about 10 s. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
A rule whose command runs or path is touched in at least --in-use-rate of the sessions is in_use however rarely its words come up (sessions_acted in the JSON); key parsing is cached to pay for the scan. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
The four open points, one commit each, each with a test that fails before it:
Suite, ruff, mypy, bandit and the benchmark gate pass on the head and merged into main. |
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 1cd9f74. The four commits since my last review do what they say: subagent transcripts are read with their session, resumed logs sharing a uuid are one session, the heredoc scan is linear, and a rule acted on in the in-use share of sessions is in_use. Both of my threads are fixed. Two risks are left, both test gaps that let a wrong verdict through, and one open point from @vlad-perevezentsev still holds.
I agree with @vlad-perevezentsev that a rule with only glob or ~ keys and no prose words still ranks as a candidate with 0 triggers. New evidence: 12 sessions that all read tests/outcomes/a.json, rule - `tests/outcomes/*.json`: the row is candidate_for_ablation, rank 1, sessions_triggered 0, sessions_acted null, flag acted_not_measurable; the new acted-share rule cannot rescue it.
What I ran, on a Linux review host at 1cd9f74:
make ci-local: 30 passed, 0 failed, 7 skipped; 14,089 tests; the changed-line coverage gate passed.- Reproductions for the findings, with no model call. Mutation runs on the new status rule and on the time-order sort, against the six context test files.
- The old and the new heredoc scan on 600,000 random commands: no difference except where a line leaves a quote open, where the new scan hides the
<<inside the quote._without_heredoc_bodiestakes 0.001 s at 8,000 and at 16,000 lines.
Checked, no issues
- Subagents and resumes: a transcript under
s/subagents/is folded intos.jsonlfor both*.jsonland**/*.jsonl; a transcript named alone is skipped with a warning; three resumed copies read as one session with a warning. - Status precedence:
status_ofand the doc agree; the>=boundary at exactly 0.2 is pinned. - Repo rules: the page is in the
mkdocs.ymlnav,max_docs: 39matches the doc count, the changelog fragment matches the code, SPDX headers are present, all 43 commits are signed off, andcli-interface.jsonmatches the CLI.
Bot points I checked that no longer apply at this head (60)
- exai-bot's point on the status table does not apply: the doc states the precedence
status_ofapplies at this head. - exai-bot's point on repeated inputs does not apply: sessions are de-duplicated by real path at this head.
- exai-bot's point on threshold domains does not apply:
--rare-maxand--in-use-rateare validated at parse time at this head. - exai-bot's point on unreadable-only Goose sessions does not apply: their diagnostics reach the CLI output at this head.
- exai-bot's point on command-only rules does not apply: such rules are kept in the report at this head.
- exai-bot's point on Windows path separators does not apply: separators are normalized on both sides at this head.
- exai-bot's point on path actions through Bash does not apply: a shell write into the named directory counts at this head.
- exai-bot's point on later prompts counted as correction does not apply: only the next prompt is read at this head.
- exai-bot's point on ranking on rounded rates does not apply: ranking uses the exact session counts at this head.
- exai-bot's point on Goose editor bodies does not apply:
file_text,old_strandnew_strare body keys at this head. - Copilot's point on repeated paths does not apply: sessions are de-duplicated by real path at this head.
- Copilot's point on threshold domains does not apply: the thresholds are validated at parse time at this head.
- Copilot's point on dropped Goose diagnostics does not apply: they reach the CLI output at this head.
- Copilot's point on Goose text blocks does not apply: the blocks of one message are joined into one prompt at this head.
- Copilot's point on Windows directories does not apply: a backslash makes a key a path at this head.
- Copilot's point on rules with no prose words does not apply: they are kept when they name a command, path or protection at this head.
- Copilot's point on Windows path separators does not apply: separators are normalized at this head.
- Copilot's point on mentions counted as runs does not apply: a command counts only where a shell segment runs it at this head.
- exai-bot's point on dotted command suffixes does not apply: a shell match ends at the last word of the key at this head.
- exai-bot's point on quoted arguments does not apply: quotes are removed as quoting on both sides at this head.
- exai-bot's point on printed paths does not apply: echo and printf arguments are not path actions at this head.
- exai-bot's point on the Acted paragraph does not apply: the doc lists the exclusions at this head.
- exai-bot's point on quoted redirection text does not apply: quoted
>is masked at this head. - exai-bot's point on invalid JSON shapes does not apply: both readers report them at this head.
- exai-bot's point on runner options does not apply: options are read with the shell walker at this head.
- exai-bot's point on escaped double quotes does not apply: a backslash escapes the next character at this head.
- exai-bot's point on line continuations does not apply: an unquoted backslash-newline joins lines at this head.
- exai-bot's point on redirections without spaces does not apply: a redirection sign ends the command word at this head.
- exai-bot's point on
~expansion does not apply:~is expanded before the literal-file check at this head. - exai-bot's point on quoted
<<does not apply: a<<inside quotes opens no heredoc at this head. - exai-bot's point on wrappers in the key does not apply: the key is stripped like the command at this head.
- exai-bot's point on pipelines and compound commands does not apply: each stage is cleaned and matched at this head.
- exai-bot's point on file names with spaces does not apply: such a key is a path at this head.
- exai-bot's point on key tokenization does not apply: keys use the shell tokenizer at this head.
- exai-bot's point on bare file names does not apply: a capitalised bare name matches shell access at this head.
- exai-bot's point on heredoc delimiters does not apply: the whole delimiter word is read at this head.
- exai-bot's point on heredocs in comments does not apply: comments are blanked first at this head.
- exai-bot's point on compound printing commands does not apply: compound keywords are stripped first at this head.
- exai-bot's point on the file-name fallback does not apply: it is gated on bare file keys at this head.
- exai-bot's point on comments after operators does not apply: a
#after a control operator starts a comment at this head. - exai-bot's point on quoted file name characters does not apply: quotes are decoded, not deleted at this head.
- exai-bot's point on function definitions does not apply: a definition body is dropped at this head.
- exai-bot's point on input redirection targets does not apply:
<targets are read at this head. - exai-bot's point on braces in function bodies does not apply: a definition head in quotes is text at this head.
- exai-bot's point on apostrophes in comments does not apply: a comment opens no quote at this head.
- exai-bot's point on leading redirections does not apply: redirections are removed from the words at this head.
- exai-bot's point on braces in comments does not apply: comments are skipped in the body scan at this head.
- exai-bot's point on leading redirections before echo does not apply: redirections are dropped before the program is read at this head.
- exai-bot's point on
>|redirections does not apply:>|and>&are write operators at this head. - exai-bot's point on
function name {does not apply: the keyword form is a definition at this head. - exai-bot's point on here-string payloads does not apply: the payload is dropped before paths are read at this head.
- exai-bot's point on escaped heredoc delimiters does not apply: delimiters are read as the shell reads them at this head.
- exai-bot's point on here-strings with quoted text does not apply: the payload is removed first at this head.
- exai-bot's point on rules with unmatchable keys does not apply: they stay claims, flagged not measurable at this head.
- exai-bot's point on bracketed command arguments does not apply: a quoted bracket is literal at this head.
- exai-bot's point on comments in the here-string scan does not apply: the scan skips comments at this head.
- exai-bot's point on
&>redirections does not apply: they no longer split the command at this head. - exai-bot's point on quoted operator-like arguments does not apply: quoted words are unmasked after redirections are removed at this head.
- exai-bot's point on heredoc delimiter words as paths does not apply: the delimiter is not read as a path at this head.
- exai-bot's point on substitutions in double quotes does not apply: a
$(...)inside double quotes is a command at this head.
…ld's time order Protected, insufficient-corpus, rate-0 and rare-but-acted cases for the acted share; the folded session's order, a pushback after a subagent, and a subagent run at the window's edge. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. Both earlier threads are fixed in the two commits since my last review, which change only tests. The new tests pin what I asked for. Moving the acted-share check above protection, above the corpus-size check or below rarity, dropping its guard, or comparing against the wrong threshold each fails test_the_acted_share_comes_after_protection_and_corpus_size_and_before_rarity. Reversing the subagent event sort or removing it fails the new corpus tests, and a subagent run 10 events from the trigger acts while 11 does not.
What I ran, on a Linux review host at 2a3b772:
make ci-local: 30 passed, 0 failed, 7 skipped (jobs that need docker, the network or do not run on a PR); 14,097 tests passed; changed-line coverage 98%.- A mutation run of the two new test groups: 6 mutants of
status_ofand 5 of the event sort and window slices, all caught.
Checked, no issues
- Commits: all 44 non-merge commits carry a
Signed-off-by. - Test offline: the new tests use synthetic JSONL under
tmp_path; no files were left in the tree after the run.
Bot points I checked that no longer apply at this head (60)
- exai-bot's point on the status table's precedence does not apply: the doc states the precedence status_of applies at this head.
- exai-bot's point on repeated inputs counting as several sessions does not apply: sessions are de-duplicated by real path at this head.
- exai-bot's point on unvalidated thresholds does not apply:
--rare-maxand--in-use-rateare validated at parse time at this head. - exai-bot's point on diagnostics lost for unreadable Goose sessions does not apply: they now reach the CLI output at this head.
- exai-bot's point on command-only rules dropped does not apply: such rules are kept at this head.
- exai-bot's point on Windows separators in path matching does not apply: separators are normalized at this head.
- exai-bot's point on path actions through Bash does not apply: shell writes into a named path count at this head.
- exai-bot's point on corrections read from every later prompt does not apply: only the next prompt counts at this head.
- exai-bot's point on ranking on rounded rates does not apply: ranking uses exact counts at this head.
- exai-bot's point on Goose editor bodies in event text does not apply: those fields are excluded at this head.
- Copilot's point on repeated paths counting twice does not apply: sessions are de-duplicated at this head.
- Copilot's point on
--in-use-rate nanand negative--rare-maxdoes not apply: both are rejected at parse time at this head. - Copilot's point on diagnostics of an unreadable-only Goose session does not apply: they are reported at this head.
- Copilot's point on Goose text blocks becoming separate prompts does not apply: one message is one prompt event at this head.
- Copilot's point on a Windows directory read as a command does not apply: backslash paths are classified as paths at this head.
- Copilot's point on rules dropped for lacking prose keywords does not apply: such rules are kept at this head.
- Copilot's point on Windows separators in
changelog.dmatching does not apply: separators are normalized at this head. - Copilot's point on mentions counted as runs does not apply: a command counts only as a segment's program at this head.
- exai-bot's point on
make ci-local.debugmatchingmake ci-localdoes not apply: a match ends at the key's last word at this head. - exai-bot's point on quoted arguments removed does not apply: quotes are decoded, not deleted at this head.
- exai-bot's point on printed paths counted as file activity does not apply: echo/printf arguments are excluded at this head.
- exai-bot's point on the doc's definition of an action does not apply: the doc lists the exclusions at this head.
- exai-bot's point on quoted redirection text does not apply: quoted operators are masked at this head.
- exai-bot's point on silently dropped JSON shapes does not apply: they are reported as diagnostics at this head.
- exai-bot's point on runner options does not apply: runner options are parsed at this head.
- exai-bot's point on escaped double quotes does not apply: backslash escapes are honoured at this head.
- exai-bot's point on backslash-newline continuation does not apply: continuations are joined before splitting at this head.
- exai-bot's point on redirection without spaces ending a command word does not apply: the redirection ends the word at this head.
- exai-bot's point on
~expansion before the literal-file check does not apply: the name is expanded first at this head. - exai-bot's point on quoted
<<opening a heredoc does not apply: it no longer does at this head. - exai-bot's point on wrappers in the guidance key does not apply: the key is stripped the same way at this head.
- exai-bot's point on pipelines and compound syntax does not apply: stages are split and cleaned at this head.
- exai-bot's point on file names with spaces does not apply: such path keys are recognised at this head.
- exai-bot's point on key tokenization losing argument boundaries does not apply: keys use the shell tokenizer at this head.
- exai-bot's point on bare file names in shell commands does not apply: capitalised bare names match shell arguments at this head.
- exai-bot's point on heredoc delimiters with punctuation does not apply: the whole delimiter word is read at this head.
- exai-bot's point on heredoc headers inside comments does not apply: comments are blanked first at this head.
- exai-bot's point on printing commands after compound keywords does not apply: compound syntax is stripped first at this head.
- exai-bot's point on the file-name fallback for lowercase command keys does not apply: it is gated to bare file names at this head.
- exai-bot's point on comments after a control operator does not apply: they are recognised at this head.
- exai-bot's point on quotes removed inside file names does not apply: quoting is decoded, not deleted at this head.
- exai-bot's point on function definitions counted as runs does not apply: bodies are dropped at this head.
- exai-bot's point on input redirection targets does not apply:
<targets are read at this head. - exai-bot's point on the first
}ending a function body does not apply: the body scan is quote-aware at this head. - exai-bot's point on apostrophes in comments does not apply: comments open no quote at this head.
- exai-bot's point on leading redirections does not apply: they are removed from the words at this head.
- exai-bot's point on braces in comments of a function body does not apply: comments are skipped at this head.
- exai-bot's point on leading redirections before echo does not apply: they are dropped before the program is read at this head.
- exai-bot's point on
>|and>&targets does not apply: they count as writes at this head. - exai-bot's point on the
function name {form does not apply: it is read as a definition at this head. - exai-bot's point on here-string payloads as paths does not apply: they are dropped at this head.
- exai-bot's point on escaped heredoc delimiters does not apply: delimiters are unquoted as the shell does at this head.
- exai-bot's point on a here-string before a spaced quoted argument does not apply: the payload is removed first at this head.
- exai-bot's point on rules with only unmatchable keys does not apply: they stay as claims, not measurable at this head.
- exai-bot's point on quoted brackets in command keys does not apply: quoted brackets are literal at this head.
- exai-bot's point on comments in the here-string scan does not apply: the scan skips comments at this head.
- exai-bot's point on
&>splitting at&does not apply: it no longer splits at this head. - exai-bot's point on quoted operator-like arguments does not apply: quoted words are unmasked after redirections are removed at this head.
- exai-bot's point on heredoc delimiters read as paths does not apply: they are not read as paths at this head.
- exai-bot's point on
$(...)inside double quotes does not apply: it is run as a command at this head.
|
I approved this at
|
A rule that names no words and only a glob or a ~ path cannot trigger, so its zero count was ranked as a candidate for ablation. It now gets the status not_measurable: listed, not ranked. The docs table and the report note say so. Thread: PRRT_kwDOVBg7g86yp97m Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Review at f4d6a05: its threads are answered above; nothing open at 2a3b772. Review at 27e620e: its threads are answered above; nothing open at 2a3b772. Review at a8d3b2b: its threads are answered above; nothing open at 2a3b772. Review at b024307: its threads are answered above; nothing open at 2a3b772. Review at 67c65ee: its threads are answered above; nothing open at 2a3b772. Review at aafa091: its threads are answered above; nothing open at 2a3b772. Review at 39260e4: its threads are answered above; nothing open at 2a3b772. Review at 58f7428: its threads are answered above; nothing open at 2a3b772. Review at 2683108: its threads are answered above; nothing open at 2a3b772. Review at 8497905: its threads are answered above; nothing open at 2a3b772. Review at 9035930: its threads are answered above; nothing open at 2a3b772. Review at 12ee241: its threads are answered above; nothing open at 2a3b772. Review at 8fe1253: its threads are answered above; nothing open at 2a3b772. Review at 054915b: its threads are answered above; nothing open at 2a3b772. Review at d4ea544: its threads are answered above; nothing open at 2a3b772. Review at 7fb6e00: its threads are answered above; nothing open at 2a3b772. Review at f4d6a05: its open issues are answered in their threads and fixed by later commits; 2a3b772 is approved. Superseded by the approval at 2a3b772, which covers the points it raised. Thanks: those nit commits are on head; nothing open here. Superseded by the approval at 2a3b772, which covers the points it raised. Thanks: those nit commits are on head; nothing open here. Superseded by the approval at 2a3b772, which covers the points it raised. Thanks: those nit commits are on head; nothing open here. Superseded by the approval at 2a3b772, which covers the points it raised. Superseded by the approval at 2a3b772, which covers the points it raised. Thanks: those nit commits are on head; nothing open here. Superseded by the approval at 2a3b772. Its glob-rank point is fixed in 278eb2a; its test gaps are pinned in 2a3b772. Its points are fixed by the later commits; the open ones are answered in threads 374 and 255. The glob and The glob-only rule is fixed in 278eb2a, so it is not ranked. The heredoc scan is fixed in 8056901 (thread above). No change: the new Not re-requesting exai-bot: its review limit is reached. The approval at 2a3b772 and CI cover the head. No change: the codecov upload is CI's setup, not a review point for this PR. No change: the points it lists are answered in their threads and fixed by the commits named there. No change: the merge is on head, and the flags of both branches are in the allowlist. No change: the re-run points are answered in their threads. No change: 13ee401 raises max_docs to 39 for the new doc. No change: the four points are their own commits on head (181ff4e to 2a3b772), each with a test. Status: thread 374 is fixed in 278eb2a, not ranked; thread 255 is fixed in 8056901. Both are for |
…intel#363 asked for Each new test fails on a one-line mutant that passed the whole suite; no code under dokimi/dokimi/ changes. intel#334 (context prefilter): - a write target glued to a heredoc sign (cat >notes.txt<<docs/guide.md) ends at the `<`: the delimiter is no path (_REDIRECT's `<` exclusion); - a bare-name rule `Makefile` counts an edit of Makefile only, not Makefile.bak or GNUMakefile, and counts C:\r\Makefile (_file_named); - `sudo echo changelog.d/x.md` prints the path, it does not act on it (strip_wrappers in _prints). intel#326 (repo budget): - an unclosed `paths: [null` or `paths: [~` reads as unscoped (_names_pattern's null and ~); - a rule reached through docs/alias.md drops its frontmatter before its @paths are followed (_import_text's _target half); - an import of a rule path linked to a plain file keeps the rule's body count (_counted_import's RULE_GLOB.match(f)). intel#363 (repo skill-reach): an OSError while saving the artifact exits 1 with a message; --bin reaches the harness run; --keep keeps the fixture and names it in the artifact. intel#323 (repo config): shared '**/Makefile' claims a root Makefile; a link c[1] on the way to the config is claimed as written; link_text keeps the `..` prefix of a subdirectory link that leaves the tree and comes back, and reads a sibling t2/ of root t as outside; a tracked `:x.sh` takes its mode from the index (--literal-pathspecs). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
#389) Each new test fails on a one-line mutant that passed the whole suite; no code under dokimi/dokimi/ changes. #334 (context prefilter): - a write target glued to a heredoc sign (cat >notes.txt<<docs/guide.md) ends at the `<`: the delimiter is no path (_REDIRECT's `<` exclusion); - a bare-name rule `Makefile` counts an edit of Makefile only, not Makefile.bak or GNUMakefile, and counts C:\r\Makefile (_file_named); - `sudo echo changelog.d/x.md` prints the path, it does not act on it (strip_wrappers in _prints). #326 (repo budget): - an unclosed `paths: [null` or `paths: [~` reads as unscoped (_names_pattern's null and ~); - a rule reached through docs/alias.md drops its frontmatter before its @paths are followed (_import_text's _target half); - an import of a rule path linked to a plain file keeps the rule's body count (_counted_import's RULE_GLOB.match(f)). #363 (repo skill-reach): an OSError while saving the artifact exits 1 with a message; --bin reaches the harness run; --keep keeps the fixture and names it in the artifact. #323 (repo config): shared '**/Makefile' claims a root Makefile; a link c[1] on the way to the config is claimed as written; link_text keeps the `..` prefix of a subdirectory link that leaves the tree and comes back, and reads a sibling t2/ of root t as outside; a tracked `:x.sh` takes its mode from the index (--literal-pathspecs). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Summary
context prefilterreads an agent's session logs and reports, per rule of an AGENTS.md or CLAUDE.md, whether the situation the rule describes came up, whether the agent acted on it, and whether the user pushed back. The output is a ranked list of candidates for ablation, not a verdict. Rare and protected rules are listed, never ranked. Output never says a rule should be cut.Statuses:
candidate_for_ablation(ranked),in_use,rare_not_ranked,kept_protected(negation, version gate, ordering or threshold tag),insufficient_corpus.Session logs are read as follows:
~/.claude/projects/<project>/. Injected records (isMeta), subagent briefs (isSidechain) and tool results are not prompts.sessions.db(~/.local/share/goose/sessions/on Linux), opened read-only. Layout taken from block/goose source (session_manager.rs,message.rs,tool_result_serde.rs):messages.session_id,content_jsonas a list of blocks,toolRequest.toolCall.value.arguments, andmetadata_json.userVisible. Related: rp-why: Configuration Effectiveness (CE) - Phase 1 block/agent-skills#52 (rp-why CE reads Goose logs). That PR queriesrun_idand readstoolCall.name; the source sayssession_idandtoolCall.value.name, and this reader follows the source.Output: a table on stdout, or
--jsonon stdout. The JSON is printed, not stored, so no schema underdokimi/schemas/;format_versionmarks its shape. Design note:docs/session-prefilter.md. Related: agentsmd/agents.md#213.Other changes that come with the command:
spec.v1.yaml: golden manifestmax_docs37 → 38 (one more doc).ce-replaymakes the same bump; whichever merges second rebases.cli_routes.py,cli_flags_allowlist.txt(--min-sessions,--rare-max,--in-use-rate: no registry concept yet), regenerateddocs/reference/cli-interface.json.Evidence
dokimi context prefilterunknown; Goose stores not readable by any command.test_context_prefilter,test_context_goose,test_cli_flags,test_cli_routes,test_docs_code_refs,test_docs_links,test_import_architecture) → 169 passed onf4d6a05b. Tests use synthetic JSONL and synthetic SQLite stores only; no real transcript or real Goose store was read.make ci-localon26ad2407(run before the last commit): 29 pass, 7 skip, oneFAILrow,ci.yml:test / Run tests. The failures are unrelated timing-sensitive tests, run on a host under load ~300 (shared with other sessions' benchmarks):test_repo_environment.py::test_unbalanced_quote_in_hook_completes_quickly,test_repo_environment.py::test_unclosed_heredocs_are_linear,test_reference_checks.py::test_the_regex_limit_is_shared_by_every_pattern_of_one_check,test_reference_checks.py::test_a_regex_that_backtracks_past_the_timeout_is_unscored[True],test_audit_test_quality.py::test_no_new_test_that_cannot_fail. Across three full runs a different subset failed each time. Each of the five passes when run alone onf4d6a05b. The full ci-local was not re-run onf4d6a05b. GitHub CI is the check of record.Merge Danger
Door: two-way. A new read-only subcommand. It writes no file, stores no output and changes no existing command. The Goose store is opened
mode=ro.Blast Radius: low. The new subcommand, and the golden manifest and CLI snapshot regenerated with it.
Known limits and follow-ups (not in this PR):
26ad2407: one non-blocking finding fixed (an unreadable-only Goose session was counted in the session total). Pass 2 on the last commit: a file name no longer matches a longer name or its backup (a.mdvsa.mdx,a.md.bak), with tests. No blocking finding open.🤖 Generated with Claude Code