Repository navigation
repo skill-reach: measure which Agent Skills a harness lists - #363
Conversation
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi repo skill-reach`
+ option `--bin`
+ option `--budget`
+ option `--catalogue`
+ option `--dry-run`
+ option `--harness`
+ option `--help/-h`
+ option `--keep`
+ option `--max-turns`
+ option `--model`
+ option `--only`
+ option `--out`
+ option `--salt`
+ option `--timeout`
+ option `--work-dir`
~ `dokimi runs list` parameters changed:
~ changed option `--kind`: choices: ["'answers'", "'arms'", "'cli_routes'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'guidance_reach'", "'headroom'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'plan'", "'plugin_delta'", "'questions'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_readiness'", "'result'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'cli_routes'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'guidance_reach'", "'headroom'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'plan'", "'plugin_delta'", "'questions'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_readiness'", "'result'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #363 +/- ##
==========================================
+ Coverage 94.36% 94.56% +0.19%
==========================================
Files 311 323 +12
Lines 39834 42016 +2182
Branches 6751 7113 +362
==========================================
+ Hits 37589 39731 +2142
- Misses 1431 1458 +27
- Partials 814 827 +13
🚀 New features to boost your workflow:
|
Places token-seeded SKILL.md fixtures from a catalogue (repos/skill_fixtures.yaml) in a fresh workspace, HOME and config dir and reports which skills a harness lists: listed / fetched / not_listed / error per case and scenario, and a verdict per same-name collision group. Profiles gain a skills block and a skills tool mode (claude-code, codex). Writes skill_discovery.v1. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…mmand module repo.py passed its 800-line cap after the merge, so cmd_repo_skill_reach and its parser move to dokimi/commands/repo_skill_reach.py. The CLI test runs in process. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The move of skill-reach into its own command module also moved the receipt helpers that `repo reach` uses, so repo.py raised NameError (ruff F821, mypy name-defined). Put them back in repo.py and drop the copies from the skill-reach module. Also: strict zip() in the collision verdict (B905); the fixture header is two lines, with the field reference in the skill_reach docstring (comment-block check); the CLI snapshot is regenerated under Python 3.13 to match the CI hashes. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The claude-code profile gained a `skills` tool mode for repo skill-reach, so the built-in profile now yields one more execution id. The six pinned ids are unchanged, so no existing batch's identity moves; the new (skills, native) id is added to the golden table. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
66db967 to
d778214
Compare
The changed-lines gate (diff-cover vs intel/main) measured 76% on this PR. Add behaviour tests for the CLI usage and failure exits, catalogue validation, collision verdicts, fixture placement (symlinks, undeclared dirs), the profile refusals, the trace clean-up, a timed-out and an unstartable harness, and the table rendering. No code changed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| except (ProfileError, SkillReachError, ReachError) as exc: | ||
| print(f"✗ {exc}", file=sys.stderr) | ||
| sys.exit(1) | ||
| path = save_skill_reach(payload, out_dir) |
There was a problem hiding this comment.
Must-fix: filesystem failures escape this handler and the outer CLI handler, producing a traceback instead of a diagnostic. For example, dokimi repo skill-reach --model m --work-dir /nonexistent/parent raises FileNotFoundError from mkdtemp; an existing file passed as --out raises an OSError while creating traces. Artifact writes can also raise OSError, and save_skill_reach is outside the try block. Catch OSError for the run and save operations, move saving into the guarded block, and add regression tests for missing work directories and invalid output destinations.
There was a problem hiding this comment.
Fixed in 09959b3: a run or save OSError is a CLI diagnostic (exit 1), not a traceback. Tests: missing work dir, blocked --out.
| except (OSError, yaml.YAMLError) as exc: | ||
| raise SkillReachError(f"{path}: cannot read the catalogue: {exc}") from exc |
There was a problem hiding this comment.
Must-fix: a --catalogue file containing invalid UTF-8 raises UnicodeDecodeError from read_text(), which is not covered here or by the CLI's ProfileError/SkillReachError handler. For example, a catalogue containing byte 0xff produces a traceback instead of the expected catalogue error and usage exit. Catch UnicodeDecodeError alongside OSError and yaml.YAMLError, and test this malformed-file input.
Suggestion:
| except (OSError, yaml.YAMLError) as exc: | |
| raise SkillReachError(f"{path}: cannot read the catalogue: {exc}") from exc | |
| except (OSError, UnicodeDecodeError, yaml.YAMLError) as exc: | |
| raise SkillReachError(f"{path}: cannot read the catalogue: {exc}") from exc |
There was a problem hiding this comment.
Fixed in 09959b3: a non-UTF-8 catalogue is a SkillReachError. Test: byte 0xff in a case.
| name = str(item.get("name") or cid) | ||
| case = Case(id=cid, at=str(item["at"]), name=name, dirname=str(item.get("dirname") or name), | ||
| description=str(item.get("description") or "single"), eol=str(item.get("eol") or "lf"), | ||
| link=item.get("link"), collision=item.get("collision"), cwd=item.get("cwd")) |
There was a problem hiding this comment.
Must-fix: optional catalogue fields retain arbitrary YAML types. {id: a, at: 'project:x', cwd: 123} raises TypeError in _relative, and collision: [g] raises TypeError when _check_groups uses it as a dictionary key. These escape the CLI's ProfileError/SkillReachError handlers and produce a traceback rather than a catalogue diagnostic. Validate cwd and collision as strings (or absent) before constructing the Case, and add these malformed-type inputs to BAD_CATALOGUES.
There was a problem hiding this comment.
Fixed in 09959b3: non-string cwd and collision are catalogue errors. Both added to the bad-catalogue tests.
| def _relative(rel: str) -> bool: | ||
| p = PurePosixPath(rel) | ||
| return not p.is_absolute() and ".." not in p.parts and "\\" not in rel |
There was a problem hiding this comment.
Must-fix: catalogue paths are not confined to the fixture on all platforms. _relative('C:/existing') returns true, so {id: a, at: 'project:C:/existing', dirname: skill} makes the Windows filesystem join discard the project root and write C:/existing/skill/SKILL.md. Separately, {id: '../../escape', at: 'project:x', dirname: skill, link: symlink} passes validation and escapes through outside / case.id / case.dirname even on POSIX. Reject Windows drive-qualified paths as well as POSIX absolute/traversal paths, and validate or safely encode IDs before using them in target paths. Add confinement regressions for both cases.
There was a problem hiding this comment.
Confirmed on d6c0920: a drive path (C:/x) and a symlink case id with .. or an absolute path wrote outside the fixture.
Fixed in a3267b4: : is refused in at, dirname and cwd; symlink targets use the case index, not the id.
Regression tests: drive and traversal cases in tests/test_skill_reach.py, red on d6c0920, green now.
| dest.mkdir(parents=True, exist_ok=True) | ||
| (dest / "SKILL.md").write_bytes(text.encode()) |
There was a problem hiding this comment.
Must-fix: two distinct case IDs can resolve to the same destination, and this write silently replaces the earlier SKILL.md while both cases remain in fixture.placed. For example, two catalogue entries with at: project:x and dirname: same report the overwritten case as not_listed, although it was never available to the harness. The bundled catalogue also hits this when a valid profile declares .agents/skills as its client project directory: dup-project-agents and dup-project-client2 both resolve to .agents/skills/dup-pa. Validate resolved destination uniqueness before writing and reject or explicitly skip overlapping cases instead of measuring nonexistent copies.
There was a problem hiding this comment.
Fixed in 09959b3: a case whose directory an earlier case holds is skipped with a reason, not overwritten. Test: two cases, one dir.
| if transcript is not None: | ||
| (trace_dir / f"{slug}.events.jsonl").write_text(transcript.events_jsonl(), encoding="utf-8") | ||
| reported = sorted(set(TOKEN_RE.findall(transcript.answer))) | ||
| looked = any(e.get("kind") in _LOOKED for e in transcript.events) |
There was a problem hiding this comment.
Must-fix: invoking the allowed Skill tool loads a skill's body, but events.convert() represents that invocation as kind='tool', name='Skill', so this check leaves looked=False. A response that invokes Skill and then reports its tokens is consequently classified as listed, with body=True interpreted as eager loading, even though the content was fetched explicitly. Include skill invocations in the contamination check and add a fake-harness mode that invokes Skill before answering.
| env = process.Env.inherit({"GIT_AUTHOR_NAME": "dokimi", "GIT_AUTHOR_EMAIL": "dokimi@localhost", | ||
| "GIT_COMMITTER_NAME": "dokimi", "GIT_COMMITTER_EMAIL": "dokimi@localhost"}) |
There was a problem hiding this comment.
Must-fix: inheriting Git's environment can redirect these commands into the caller's repository. For example, run the probe with GIT_DIR=/path/to/existing/.git and GIT_WORK_TREE=/path/to/existing: git init, git add -A, and git commit operate on that repository rather than the temporary workspace, potentially staging and committing the user's changes. Inherited global Git configuration can also introduce hooks into this supposedly fresh fixture. Use a scrubbed environment and isolated Git configuration, as the guidance-reach workspace builder does, and test with those variables set.
There was a problem hiding this comment.
Confirmed on d6c0920: git inherited GIT_DIR, GIT_WORK_TREE and global config, so it could touch another repo or run its hooks.
Fixed in a3267b4: _git_init runs git with an allow-listed env, GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_NOSYSTEM=1.
Regression tests set GIT_DIR and a global hooksPath, red on d6c0920, green now.
| "GIT_COMMITTER_NAME": "dokimi", "GIT_COMMITTER_EMAIL": "dokimi@localhost"}) | ||
| for argv in (["git", "init", "-q"], ["git", "add", "-A"], | ||
| ["git", "-c", "commit.gpgsign=false", "commit", "-q", "-m", "fixture", "--allow-empty"]): | ||
| proc = process.run(argv, cwd=workspace, env=env, timeout=60) |
There was a problem hiding this comment.
Must-fix: a missing Git executable raises FileNotFoundError here, which escapes the CLI's ProfileError/SkillReachError/ReachError handlers and produces a traceback. This differs from the handled failure when Git starts but exits unsuccessfully. Catch OSError around the Git invocation and re-raise SkillReachError with the command and cause; add a test for an unavailable Git executable, not only exit code 128.
There was a problem hiding this comment.
Fixed in 09959b3: a git that cannot start is a SkillReachError. Test: missing executable.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved path-validation and Git-environment isolation flaws can write outside fixtures or modify other repositories, while trace-classification errors and fixture collisions can produce misleading results.
8 open issues
What changed in this PR
8 new findings. This PR adds dokimi repo skill-reach, extending the existing guidance probe approach to measure which Agent Skills a harness lists. Live validation against Claude Code and Codex remains pending.
Changes:
- Adds catalogue-driven fixtures, scenario execution, discovery classification, and collision reporting.
- Extends harness profiles and registers the
skill_discovery.v1artifact. - Adds CLI integration, documentation, packaging, and fake-harness regression tests.
| File | Description |
|---|---|
dokimi/tests/test_skill_reach.py |
Tests fixture placement, classification, errors, and CLI behavior. |
dokimi/tests/test_runs_show.py |
Adds discovery artifact count expectations. |
dokimi/tests/test_runs_scan.py |
Adds discovery artifact metadata expectations. |
dokimi/tests/test_cli_routes.py |
Covers the new command route. |
dokimi/tests/test_batch_ledger_golden.py |
Records the new skills-mode identity. |
dokimi/tests/fixtures/artifacts/skill_discovery.v1.json |
Provides a sample discovery artifact. |
dokimi/tests/data/cli_flags_allowlist.txt |
Registers the catalogue flag. |
dokimi/pyproject.toml |
Packages the fixture catalogue. |
dokimi/dokimi/schemas/skill_discovery.v1.json |
Defines the discovery artifact schema. |
dokimi/dokimi/schemas/harness_profile.v1.json |
Adds skill locations and tool-mode fields. |
dokimi/dokimi/runs/kinds.py |
Enables discovery artifact indexing. |
dokimi/dokimi/repos/skill_reach.py |
Implements fixture execution and result reporting. |
dokimi/dokimi/repos/skill_fixtures.yaml |
Defines 25 discovery cases. |
dokimi/dokimi/repos/profiles/codex.yaml |
Declares Codex skill locations and mode. |
dokimi/dokimi/repos/profiles/claude-code.yaml |
Declares Claude Code skill locations and mode. |
dokimi/dokimi/repos/harness_profile.py |
Loads skill declarations into profiles. |
dokimi/dokimi/commands/repo.py |
Registers the new subcommand. |
dokimi/dokimi/commands/repo_skill_reach.py |
Implements command arguments and orchestration. |
dokimi/dokimi/cli_routes.py |
Classifies the command as paid. |
dokimi/dokimi/artifacts.py |
Registers the discovery artifact version. |
dokimi/docs/what-it-measures.md |
Describes the new measurement capability. |
dokimi/docs/reference/cli-interface.json |
Records the CLI interface changes. |
dokimi/changelog.d/20261101_120000_napetrov_skill-reach.md |
Announces the probe and artifact. |
Review effort: Balanced
|
|
||
| def _relative(rel: str) -> bool: | ||
| p = PurePosixPath(rel) | ||
| return not p.is_absolute() and ".." not in p.parts and "\\" not in rel |
There was a problem hiding this comment.
Already fixed in a3267b4: ':' is refused in at, dirname and cwd, so C:/x and ./C:/x are rejected. Tests: drive-path cases.
| rel = str(PurePosixPath(parent) / case.dirname) | ||
| tok, body_tok = token(salt, case.id, "description"), token(salt, case.id, "body") | ||
| text = skill_text(case, tok, body_tok) | ||
| dest = roots[scope] / rel |
There was a problem hiding this comment.
Fixed in 09959b3: the second case is skipped, so one copy is placed. Test: the .agents/skills pair.
| dest = roots[scope] / rel | ||
| link_kind = None | ||
| if case.link == "symlink": | ||
| target = outside / case.id / case.dirname |
There was a problem hiding this comment.
Already fixed in a3267b4: symlink targets use the case index, not the id. Tests: traversing and absolute ids.
| if placed.token not in reported and not (placed.case.description == "missing" and body): | ||
| return {"class": "not_listed", "body": body} |
There was a problem hiding this comment.
There was a problem hiding this comment.
Maintainer decision: a tokenless listing (name, then -) of a description-missing skill stays not_listed. Done in 75201aa: documented in what-it-measures.md, the module docstring and the schema.
| if transcript is not None: | ||
| (trace_dir / f"{slug}.events.jsonl").write_text(transcript.events_jsonl(), encoding="utf-8") | ||
| reported = sorted(set(TOKEN_RE.findall(transcript.answer))) | ||
| looked = any(e.get("kind") in _LOOKED for e in transcript.events) |
There was a problem hiding this comment.
Already fixed in a3267b4: a Skill event makes the scenario fetched. Test: skill mode.
| env = process.Env.inherit({"GIT_AUTHOR_NAME": "dokimi", "GIT_AUTHOR_EMAIL": "dokimi@localhost", | ||
| "GIT_COMMITTER_NAME": "dokimi", "GIT_COMMITTER_EMAIL": "dokimi@localhost"}) |
There was a problem hiding this comment.
Already fixed in a3267b4: git runs with an allow-listed env and no global config. Tests: GIT_DIR and hooksPath.
| print(f"{len(selected)} cases, {len(scenario_ids(selected))} scenarios, at most ${args.budget:g} each", | ||
| file=sys.stderr) |
There was a problem hiding this comment.
Fixed in 09959b3: --budget is refused for a harness with no spend-cap flag (codex, copilot); the banner says the timeout is the only bound.
| case = Case(id=cid, at=str(item["at"]), name=name, dirname=str(item.get("dirname") or name), | ||
| description=str(item.get("description") or "single"), eol=str(item.get("eol") or "lf"), | ||
| link=item.get("link"), collision=item.get("collision"), cwd=item.get("cwd")) |
There was a problem hiding this comment.
Fixed in 09959b3: non-string cwd and collision are catalogue errors, also under --dry-run.
… env - Reject drive-qualified paths (C:, C:escape) in at, dirname and cwd. - Place symlink-case targets under an index, not the raw case id, so an id cannot write outside the probe's temp dir. - Count a Skill invocation as the agent looking for itself (fetched, not listed). - Run git in the fixture with an allow-listed env and no global or system config, so a caller's GIT_DIR, GIT_WORK_TREE or hooks do not apply. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| "missing": "", | ||
| }[case.description] | ||
| body = f"Body token {body_tok}.\n" + (f"Description token {tok}.\n" if case.description == "missing" else "") | ||
| text = f"---\nname: {case.name}\n{desc}---\n\n# {case.name}\n\n{body}" |
There was a problem hiding this comment.
Must-fix: Serialize the frontmatter name as a quoted YAML string. A valid custom catalogue entry such as {id: a, at: 'project:{client_project}', name: 'null'} produces name: null, which YAML parses as null rather than the declared skill name; similarly, 'true' and '123' become a boolean and integer. A client requiring a string name can reject these fixtures, producing a false not_listed result unrelated to discovery. Use YAML-safe scalar serialization (a JSON-quoted string is also valid YAML) and add a round-trip test for these names.
There was a problem hiding this comment.
Fixed in 09959b3: a frontmatter name is quoted when the plain form would load as another type. Round-trip test added.
|
|
||
|
|
||
| def _looked_for_itself(events: list[dict[str, Any]]) -> bool: | ||
| return any(e.get("kind") in _LOOKED or (e.get("kind") == "tool" and e.get("name") == "Skill") for e in events) |
There was a problem hiding this comment.
Must-fix: File-search tools can fetch tokens without setting looked. For example, a custom Claude profile with tools.skills: ["--tools", "Skill", "Grep"] can grep SKILL.md descriptions and report their tokens; events.convert() represents Grep as kind='tool', name='Grep', so these results are incorrectly classified as listed. Account for file-search tools (at least Grep, and directory discovery via Glob) when detecting self-discovery, and add a regression test that reports a token after a Grep invocation.
There was a problem hiding this comment.
Fixed in 09959b3: any tool call (Grep, Glob, Skill, ...) makes the scenario fetched. Test: Grep, then report.
| for root in roots.values(): | ||
| root.mkdir() | ||
| fixture = build_fixture(cases, profile, roots, tmp_path / "outside", "s") | ||
| assert [(p.case.id, p.link_kind) for p in fixture.placed] == [("symlink-client", "symlink")] |
There was a problem hiding this comment.
Must-fix: On Windows without Developer Mode or symlink privileges, build_fixture() correctly skips symlink-client, so this unconditional success assertion fails for a supported fallback. Capability-check symlink creation and skip this success-path test when unavailable (as _symlink_or_skip does in tests/test_artifacts.py). Keep test_build_fixture_reports_a_symlink_the_os_refuses as the deterministic fallback regression test.
There was a problem hiding this comment.
Fixed in 09959b3: the success-path symlink test skips where this OS refuses symlinks; the refusal test stays.
Live run:
|
| Case | Claude Code | Codex |
|---|---|---|
project client dir (.claude/skills / .codex/skills) |
L L | L L |
project .agents/skills |
. . | L L |
user <config-dir>/skills |
L L | L L |
user ~/.agents/skills |
. . | L L |
user ~/.config/agents/skills |
. . | . . |
pkg/.agents/skills |
. . | . L |
pkg/<client dir> |
. L | . L |
nested skills/group/x (client dir) |
. . | L L |
nested two levels (.agents) |
. . | L L |
| symlinked skill dir, target outside the tree (client dir) | L L | L L |
dir name differs from name: (.agents) |
. . (.agents not read) |
L L |
| description: folded, literal, quoted, >1024 chars, CRLF | L for all | L for all |
| description missing | . . | . . |
Collisions (same name):
| Group | Claude Code | Codex |
|---|---|---|
| project client dir vs user config dir | one: the user copy (the project copy is hidden) | all (both listed) |
project .agents vs project client dir |
one: the client dir copy (.agents not read) |
all |
project .agents vs ~/.agents |
none | all |
The live run confirms what the prototype found:
- Claude Code reads neither
.agents/skillsscope nor nested dirs, and a user skill hides a project skill with the same name. - Codex lists every copy and follows symlinks.
New from the full catalogue:
- Both clients pick up a subdirectory's skill dir only when started inside that subdirectory.
- Neither client lists
~/.config/agents/skills. - Neither client lists a skill without a
description.
The profiles' skills.notes already match these results. No code change came out of the run.
Run notes:
- The ancestor guard refused a
--work-dirthat has the user's own CLAUDE.md above it, which is the intended behaviour. The fixture ran under/tmp. - Codex used a local profile copy that adds a
config_filesprovider for litellm (env_key = "CODEX_API_KEY"), as the codex profile comment describes.
🤖 Generated with Claude Code
…t run errors - A catalogue whose cwd or collision is not a string, or whose file is not UTF-8, is a catalogue error, not a traceback. - A case whose skill directory an earlier case already holds is skipped with a reason instead of overwriting that copy. - A missing git executable is a SkillReachError. - Frontmatter names that YAML would load as another type (null, true, 123) are quoted. - Any tool call (Grep, Glob, ...) makes a scenario fetched, not listed. - --budget is refused for a harness with no spend-cap flag (codex, copilot); the banner says the timeout is the only bound. Default is now unset. - A missing work dir or an unwritable --out exits 1 with a diagnostic. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| if args.dry_run: | ||
| argv = profile.argv(prompt="<ask>", model=args.model, tools="skills", max_turns=args.max_turns, | ||
| budget_usd=budget, bin_path=args.bin) if "skills" in profile.tools else None |
There was a problem hiding this comment.
Must-fix: dokimi repo skill-reach --harness copilot --model m --dry-run succeeds with argv: null and declared: {}, although the same invocation without --dry-run is refused by _check_profile(). Profiles with a skills tool mode but no skills block (or native_prompt: true) also bypass that validation. Run the shared profile-capability check before branching into dry-run, so an unsupported probe cannot pass preflight; add dry-run regression tests for these rejected profiles.
There was a problem hiding this comment.
Fixed in 75201aa: --dry-run runs the same profile check as a run, so it refuses the profiles a run refuses (native_prompt, no skills block, no skills tool mode). Tests added.
| raise SkillReachError(f"{path}: case {n} needs `id` and `at`") | ||
| extra = set(item) - set(Case.__dataclass_fields__) | ||
| if extra: | ||
| raise SkillReachError(f"{path}: case {item['id']}: unknown field(s) {', '.join(sorted(extra))}") |
There was a problem hiding this comment.
Must-fix: unknown catalogue keys are not necessarily strings. For example, cases: [{id: a, at: 'project:x', 123: value}] makes --dry-run --catalogue ... raise TypeError in join rather than SkillReachError; mixed numeric/string unknown keys also fail in sorted. Validate that field keys are strings, or stringify them before sorting and formatting, and add a malformed-key regression test.
Suggestion:
| raise SkillReachError(f"{path}: case {item['id']}: unknown field(s) {', '.join(sorted(extra))}") | |
| raise SkillReachError(f"{path}: case {item['id']}: unknown field(s) {', '.join(sorted(map(str, extra)))}") |
There was a problem hiding this comment.
Fixed in 75201aa: unknown catalogue keys are sorted as strings, so a numeric key is reported as an unknown field, not a TypeError. Test: a numeric and a named unknown key.
| def _relative(rel: str) -> bool: | ||
| """A path that stays under its root: no absolute, traversing, backslash or drive (``C:``) form.""" | ||
| p = PurePosixPath(rel) | ||
| return not p.is_absolute() and ".." not in p.parts and "\\" not in rel and ":" not in rel |
There was a problem hiding this comment.
Must-fix: _relative() accepts embedded NUL characters. A custom YAML case with at: "project:x\0y" passes resolve_at() but then raises ValueError: embedded null byte during fixture filesystem operations. Neither the command nor the outer CLI catches that exception, so malformed catalogue input produces a traceback instead of a catalogue diagnostic. The same gap affects dirname and cwd. Reject \x00 here and add regression cases for all three path fields.
There was a problem hiding this comment.
Fixed in 75201aa: _relative rejects a NUL byte, so at, dirname and cwd with one are a SkillReachError. Tests: dirname, cwd and at.
|
|
||
|
|
||
| def _looked_for_itself(events: list[dict[str, Any]]) -> bool: | ||
| return any(e.get("kind") in _LOOKED or e.get("kind") == "tool" for e in events) |
There was a problem hiding this comment.
Must-fix: edit tools bypass the contamination check. events._tool_event() normalizes Claude's Edit/Write/NotebookEdit calls to kind="edit", not kind="tool". Thus a valid custom skills-mode profile allowing Edit can invoke it and subsequently report fixture tokens, yet this function returns false and the records remain listed, contrary to the any-tool-call classification rule. Include normalized edit events and add a trace regression test, alongside the Read/Skill/Grep tests.
Suggestion:
| return any(e.get("kind") in _LOOKED or e.get("kind") == "tool" for e in events) | |
| return any(e.get("kind") in (*_LOOKED, "edit", "tool") for e in events) |
There was a problem hiding this comment.
Fixed in 75201aa: an edit (Edit, MultiEdit, Write, NotebookEdit) counts as looking, like any other tool call. Test: an Edit before the answer makes the scenario fetched.
| reported = sorted(set(TOKEN_RE.findall(transcript.answer))) | ||
| looked = _looked_for_itself(transcript.events) | ||
| row.update(reported=reported, unknown=[t for t in reported if t not in known], | ||
| prose=prose_words(transcript.answer), looked=looked) |
There was a problem hiding this comment.
Must-fix: prose_words() is imported from guidance reach and strips only RC[0-9A-F]{6} tokens. Skill tokens therefore contribute alphabetic fragments to the persisted prose count: project-client SKBCFC2F yields 4 words rather than 2. Strip this module's SK/SB tokens before calling the helper (or let the helper accept a token regex), and add a regression test comparing an answer with and without its verification token.
Suggestion:
| prose=prose_words(transcript.answer), looked=looked) | |
| prose=prose_words(TOKEN_RE.sub(" ", transcript.answer)), looked=looked) |
There was a problem hiding this comment.
Fixed in 75201aa: prose words are counted after the skill tokens are stripped, so SKBCFC2F adds no words. Test: an answer with a token counts only its words.
…rose clean - --dry-run runs the same profile check as a run (native_prompt, no skills block, no skills tool mode), so it cannot print an argv a run refuses - an Edit (or any other tool call) before the answer makes the scenario fetched, as a Read or a Skill call does - a non-string unknown catalogue key is reported as an unknown field, not a TypeError; a NUL byte in `at`, `dirname` or `cwd` is a catalogue error - the verification token's letters are not counted as prose words - document that a description-missing skill listed by name only (`name -`) is not_listed (maintainer decision) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| if re.match(r"^\d{2,}-.*\.(stdout|events\.jsonl)$", old.name) and old.is_file(): | ||
| old.unlink() |
There was a problem hiding this comment.
Must-fix: scope trace cleanup to this probe. Impact: running skill-reach in an output directory previously used by guidance reach deletes that probe's diagnostic evidence while leaving reach.json behind. Trigger: both commands use the same --out directory. Evidence: reach.py:317–330 writes traces with exactly this numbered filename convention, and the skill-reach CLI accepts an existing output directory without checking ownership. Workaround: use separate output directories. Severity justification: medium because the loss is bounded to persisted diagnostic traces, not primary artifacts. Use probe-specific trace subdirectories (and write new traces there), or reject directories owned by another probe before deleting files.
A bad scope, a traversing path or a placeholder typo in `at` passed --dry-run with exit 0 and was refused only while placing the fixture. Resolve `at` against stand-in skill dirs in _parse_case, so both modes refuse it as a usage error (exit 2) before anything runs. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
A profile check_profile refuses, a profile skill dir that escapes, an --out under a file and a missing --work-dir exited 1, the code for a run that ran and failed (docs/quickstart.md exit codes). Check them with the other usage errors, before the banner, in dry run and run alike. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
The digest came from a second read of the path after every scenario: a pipe recorded the hash of empty input, an edit during the run the hash of the edited file, and a deleted catalogue lost the finished run. Hash the bytes load reads and pass that digest into the run. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> # Conflicts: # dokimi/dokimi/commands/repo_skill_reach.py
The cleanup of an earlier run's traces matched the names repo reach gives its own, so a skill-reach run into a shared --out deleted the traces behind a stored reach.json. Each probe now owns its directory. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
No test failed when a shell command stopped counting as looking, when an empty answer with exit 0 stopped being an error, or when the body-token rule for a description: missing skill was dropped or widened. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
cli_flags_allowlist.txt: keep this branch's flags and the --ceiling, --floor and --min-reps lines from intel#279. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at e28968b. The probe is sound: the fixture is confined, git is isolated, and refused catalogues fail before any run. The fixes for the earlier bot threads hold on this head. What is left is how a few edge results are classed and how two refusals exit; it is in the line comments.
Agreed with @xaleryb: a collision with a fetched member still gets one/all, the same as a clean run.
What I ran, at e28968b:
make ci-local --offline: 12,834 tests passed, 0 failed, 58 skipped. The one FAIL isschema_compatonrepo_environment.v1.json(structureremoved). It comes from main: #349 added that property after this branch's base, and the check compares with currentmain. This PR does not touch that schema.- CLI with
--bin /bin/false(no model call):--dry-runvs. a run for a profile withoutuser_dirsand--only user-client, and for a--work-dirunder aCLAUDE.md. Both dry runs exit 0, and both runs exit 1 after the banner. classify/collisionscalled directly for a body-token-only answer and a collision with afetchedmember.- On Windows (Server 2025, Python 3.13):
pytest tests/test_skill_reach.py: 93 passed, 2 skipped (/dev/fdand the POSIX hook script). The symlink success path ran there, it was not skipped.
Checked, no issues
- Earlier threads: drive and traversal paths, symlink ids, occupied destinations, the git environment, non-string fields, NUL bytes, the Skill/Grep/Edit/Bash looking rule, the budget refusal for codex, and the traces moved to
skill-traces/(4b8869d) are all fixed in the code. - Isolation between scenarios: the config dir is shared, but both bundled profiles run without sessions (
--no-session-persistence,--ephemeral), so one scenario does not carry state into the next. - Schemas and packaging:
harness_profile.v1only gains optional fields.skill_discovery.v1is registered with a fixture, andskill_fixtures.yamlis in package data. - Docs and changelog: the fragment is under Added and says what a user gets.
what-it-measures.mdnames the command and thedescription: missingrule.
| listed = [m for m, c in zip(members, classes, strict=True) if c in ("listed", "fetched")] | ||
| verdict = ("none" if not listed else "all" if len(listed) == len(members) | ||
| else "one" if len(listed) == 1 else "some") |
There was a problem hiding this comment.
A collision group with a fetched member gets the same one/all verdict as a clean run.
collisions() counts fetched as listed. When the agent looked for the files itself (Codex can cat them in its read-only sandbox), which copy it reported says nothing about the client's precedence, but the record and the table still say collision g [cwd=.]: one: a. @xaleryb raised this in the conversation. With members a=fetched, b=not_listed the result is {'listed': ['a'], 'verdict': 'one'}.
Suggestion: return verdict fetched when any member is fetched, and add it to the verdict enum. skill_discovery.v1 is new in this PR, so the enum can still change in place.
There was a problem hiding this comment.
Done in 4f4fbc9: a collision group with a fetched member now has verdict fetched, and the schema enum has it.
Pinned in test_collision_verdicts.
| check_profile(profile) | ||
| for case in selected: | ||
| resolve_at(case, skill_dirs(profile)) | ||
| if args.dry_run: | ||
| return | ||
| existing = Path(args.out) / "skill-traces" | ||
| while not existing.exists() and existing != existing.parent: | ||
| existing = existing.parent | ||
| if not existing.is_dir(): | ||
| raise SkillReachError(f"--out {args.out}: {existing} is not a directory") | ||
| if args.work_dir and not Path(args.work_dir).is_dir(): | ||
| raise SkillReachError(f"--work-dir {args.work_dir} is not a directory") |
There was a problem hiding this comment.
Two refused inputs still pass --dry-run and exit 1 after the banner, not 2.
--harness <profile without user_dirs> --only user-client: the dry run lists the case and exits 0. The run prints1 cases, 1 scenarios, then✗ no case could be placedand exits 1.--work-dir /x/workwith/x/CLAUDE.md: the dry run exits 0. The run prints the banner, then✗ guidance files above the workspace…and exits 1.
Suggestion: in _refuse_what_a_run_would_refuse, refuse a selection where resolve_at gives None for every case, and run check_skill_ancestors(Path(work_dir or tempfile.gettempdir()).resolve() / "_"). Then a dry run and a run refuse the same inputs, with exit 2.
There was a problem hiding this comment.
Done in 4f4fbc9: a dry run runs the same refusals as a run (no placeable case, --work-dir, the ancestor guard
with the profile's guidance names), exit 2. Tests cover both inputs from the review.
| return {"class": "error", "body": False} | ||
| reported = set(TOKEN_RE.findall(transcript.answer)) | ||
| body = placed.body_token in reported | ||
| if placed.token not in reported and not (placed.case.description == "missing" and body): |
There was a problem hiding this comment.
When only a skill's body token is reported, the record says not_listed with body: true.
With no tool call (looked false), the agent can only know SB… because the client put the skill in its context, so the record is self-contradictory and shows as .b in the table. This can happen because the ask names both SK and SB tokens. classify(single case, answer="c SB000001", looked=False) gives {'class': 'not_listed', 'body': True}. test_only_a_description_missing_skill_is_listed_by_its_body_token pins this on purpose. Is that the intent? If it is, please say why in the docstring. If not, count it as listed.
There was a problem hiding this comment.
Intended: a body token alone counts as not_listed unless the case is description: missing, where the body
carries the description token. The docstring says so in 4f4fbc9; the existing test pins it.
| def check_profile(profile: HarnessProfile) -> None: | ||
| if profile.native_prompt: | ||
| raise SkillReachError(f"{profile.name} runs its own prompt (native_prompt); probe its exec profile") | ||
| if not profile.skills: | ||
| raise SkillReachError(f"{profile.path}: declares no `skills` block, so its own skill dirs are unknown") | ||
| if "skills" not in profile.tools: | ||
| raise SkillReachError(f"{profile.path}: declares no `skills` tool mode (the tools under which the agent " | ||
| "sees its skill list but can run nothing)") |
There was a problem hiding this comment.
With a profile that has trace: text, fetched can never come out, so every reported token reads as listed.
convert("text", …) returns a transcript with no events, so _looked_for_itself is always false. A custom profile with a skills block and trace: text passes check_profile, and a cat .agents/skills/*/SKILL.md before the answer shows as L. In that case the check was not measured, but the result reads as a pass.
Suggestion: check_profile refuses profile.trace == "text" ("its trace has no tool events, so a fetched skill cannot be told from a listed one").
There was a problem hiding this comment.
Done in 4f4fbc9: check_profile refuses trace: text (no tool events), so a fetched skill cannot pass as listed.
Covered by the dry-run refusal test.
| "records": records, | ||
| "collisions": collisions(fixture, records), | ||
| "summary": { | ||
| "cost_usd": round(sum(r.get("cost_usd") or 0 for r in rows), 6), |
There was a problem hiding this comment.
An unknown cost is summed as 0, so a Codex run reports cost=$0.0000 and summary.cost_usd: 0.
cli_traces gives cost_usd=None for Codex and Copilot, and a timed-out run has None too. sum(r.get("cost_usd") or 0 …) turns that into a number that reads as free.
Suggestion: make summary.cost_usd null when any scenario's cost is None (["number", "null"] in skill_discovery.v1, which is new here), and print cost=unknown in format_table.
There was a problem hiding this comment.
Done in 4f4fbc9: summary.cost_usd is null when any scenario's cost is unknown, and the table prints cost=unknown.
skill_discovery.v1 allows null there.
|
|
||
| def check_skill_ancestors(path: Path) -> None: | ||
| """Refuse to run under a directory that holds a skill dir or guidance some client would read.""" | ||
| check_ancestors(path, ANCESTOR_GUARD) |
There was a problem hiding this comment.
Nit: The ancestor guard skips the profile's own guidance names.
check_skill_ancestors checks only ANCESTOR_GUARD, but run_reach (reach.py:312) also adds profile.guidance_names, e.g. codex's AGENTS.override.md. The bundled profiles are not affected (Codex stops at the fixture's git root), but a custom profile's guidance file above --work-dir is not refused.
Suggestion: check_ancestors(path, tuple(dict.fromkeys(ANCESTOR_GUARD + profile.guidance_names))), as reach does.
There was a problem hiding this comment.
Done in 4f4fbc9: the ancestor guard also checks the profile's guidance names, as reach does.
Test: test_the_ancestor_guard_also_refuses_the_profiles_own_guidance_names.
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
Review at e28968b. The fixture confinement, git isolation and catalogue validation hold up, and the earlier bot threads are fixed in the code. Two points of mine are inline, and the merge also waits on four open points from @vlad-perevezentsev.
I agree with @vlad-perevezentsev's points on the fetched collision verdict (also raised by @xaleryb), the two dry-run gaps, trace: text and the unknown cost summed as 0.
@vlad-perevezentsev's question on a body-only token recorded as not_listed with body: true waits for the author.
What I ran, on a Linux review host at e28968b:
make ci-local: 29 passed, 1 failed, 7 skipped. 12,885 tests passed, 2 failed; changed-line coverage 96.1% forrepo_skill_reach.pyand 99.7% forskill_reach.py. The failing step istest_unbalanced_quote_in_hook_completes_quickly(1.09 s against a 1 s limit) andtest_the_regex_limit_is_shared_by_every_pattern_of_one_check, two wall-clock tests this PR does not touch.- 77 mutants of
skill_reach.pyandrepo_skill_reach.pyagainsttests/test_skill_reach.py: about 40 killed, about 30 survive. The ones that decide a result are in the inline comment. - A fake harness through
run_skill_reach, with the LLM transport mocked.
Checked, no issues
- Schemas:
schema_compatagainst the base passes;skill_discovery.v1is registered inSCHEMA_VERSIONS,harness_profile.v1only gains optional fields, and artifacts built for skipped, error and null-cost records validate. - Repo rules: changelog fragment, CLI snapshot, flags allowlist and package data are in place; all non-merge commits are signed off.
- Confinement: 25 bundled cases place with nothing skipped on both bundled profiles; the path, symlink and git-environment mutants are killed.
Bot points I checked that no longer apply at this head (25)
- exai-bot's point on filesystem errors escaping the handler does not apply: a run or save OSError is now a diagnostic with exit 1.
- exai-bot's point on a non-UTF-8 catalogue does not apply: it is now a SkillReachError.
- exai-bot's point on non-string
cwdandcollisiondoes not apply: both are catalogue errors now. - exai-bot's point on drive-qualified and traversing paths does not apply:
:is refused inat,dirnameandcwd, and symlink targets use the case index. - exai-bot's point on two cases resolving to one directory does not apply: the later case is skipped with a reason.
- exai-bot's point on a Skill call not counting as looking does not apply: any tool event now makes the scenario fetched.
- exai-bot's point on Git inheriting
GIT_DIRand global config does not apply: git runs with an allow-listed environment and no global or system config. - exai-bot's point on a missing git executable does not apply: it is now a SkillReachError.
- Copilot's point on Windows drive paths does not apply:
:is refused inat,dirnameandcwd. - Copilot's point on
dup-project-agentsanddup-project-client2sharing a directory does not apply: the second case is skipped. - Copilot's point on a case id escaping through a symlink target does not apply: targets use the case index.
- Copilot's point on tokenless
desc-missingreading as not_listed does not apply: the maintainer decided it stays not_listed, and the docs and schema say so. - Copilot's point on Skill invocations does not apply: a Skill event makes the scenario fetched.
- Copilot's point on inherited
GIT_DIRandGIT_WORK_TREEdoes not apply: git runs with an allow-listed environment. - Copilot's point on the Codex
--budgetpromise does not apply:--budgetis refused for a harness with no spend-cap flag. - Copilot's point on
cwdandcollisiontypes does not apply: both are validated as strings, also under--dry-run. - exai-bot's point on unquoted frontmatter names does not apply: names that YAML would load as another type are quoted.
- exai-bot's point on Grep and Glob does not apply: any tool call makes the scenario fetched.
- exai-bot's point on the symlink success-path test does not apply: it skips where the OS refuses symlinks.
- exai-bot's point on
--dry-runskipping the profile check does not apply: a dry run runs the same profile check as a run. - exai-bot's point on non-string unknown keys does not apply: they are sorted as strings and reported as unknown fields.
- exai-bot's point on NUL bytes in paths does not apply:
_relativerejects them. - exai-bot's point on edit tools bypassing the look check does not apply: an edit now counts as looking.
- exai-bot's point on skill tokens in the prose count does not apply: tokens are stripped before counting.
- exai-bot's point on trace cleanup deleting
repo reachtraces does not apply: skill-reach writes to its ownskill-traces/directory.
| transcript = convert(profile.trace, proc.stdout, str(roots["project"]), cwd) | ||
| row.update(returncode=proc.returncode, cost_usd=transcript.cost_usd, num_turns=transcript.num_turns, | ||
| _stdout=proc.stdout) | ||
| if proc.returncode != 0 or transcript.is_error or not transcript.answer: |
There was a problem hiding this comment.
Two of the three conditions that turn a run into error are not pinned.
if proc.returncode != 0 or transcript.is_error or not transcript.answer: decides whether a scenario is measured at all. Mutation check: replacing it with if not transcript.answer: leaves all 95 tests in tests/test_skill_reach.py passing. A harness that exits non-zero, or reports an error result, but still prints a line with tokens would be recorded as listed or fetched instead of error, and the exit code of the command would stay 0.
Only the empty-answer clause is covered. A related gap: no test drives a full cmd_repo_skill_reach run, so the run-path exit code (sys.exit(1) on a scenario error), the OSError clause of the handler and the pass-through of --budget, --timeout, --bin, --salt and --keep survive mutation too.
Suggestion: add two fake-harness modes, one that prints a token-bearing answer and exits 3, and one whose result event is an error with an answer, and assert class error for both. Add one CLI test that runs the fake harness and asserts the exit code and the --timeout and --salt values reach the argv or the artifact.
There was a problem hiding this comment.
Done in 4f4fbc9: fake-harness modes for a non-zero exit with a token and an error result with an answer, both
asserted as error; a CLI test checks --timeout, --salt and --budget. Mutating the error condition now fails two tests.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 569218e, which fix the nit from my review:
569218e:dokimi/dokimi/repos/profiles/codex.yaml:32: added the suggested comment above theskillstool mode saying codex has no allow-list, so the agent can still run commands and a look is caught afterwards as fetched.
make ci-local at 569218e: 30 passed, 0 failed, 7 skipped.
Not approving yet. Still open: the risk from my review; @vlad-perevezentsev on A collision group with a fetched member gets the same one/all verdict as a clean run (also @xaleryb); proposal: verdict fetched in the enum; @vlad-perevezentsev on --dry-run passes, and a run exits 1 after the banner, for a profile without user_dirs with --only user-client and for a --work-dir under a CLAUDE.md; should exit 2 in both; @vlad-perevezentsev on A profile with trace: text can never yield fetched, so every reported token reads as listed; check_profile should refuse it; @vlad-perevezentsev on An unknown cost (codex, copilot, timeout) is summed as 0, so the run reports cost=$0.0000 and summary.cost_usd 0; @vlad-perevezentsev on Is a body-only token (c SB000001, no tool call) meant to be not_listed with body true, or listed?.
… tests - A collision verdict is `fetched` when a member was fetched: which copy the agent named after looking says nothing about precedence. - summary.cost_usd is null when any scenario's cost is unknown (Codex, Copilot, a timeout); the table prints cost=unknown. - A dry run refuses what a run refuses: no selected case has a skill dir in the profile, a --work-dir that is not a directory, guidance above it. - The ancestor guard also checks the profile's own guidance names. - A profile with trace: text is refused (no tool events, so a fetched skill cannot be told from a listed one). - Tests: a non-zero exit or an error result with an answer is an error; the CLI passes --timeout, --salt and --budget to the run. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked 4f4fbc9. All six points from my last round are settled. The collision verdict is fetched when a member was fetched, dry runs refuse what a run refuses with exit 2, trace: text is refused, an unknown cost is null, and the ancestor guard checks the profile's guidance names. I accept the reason for keeping a body-only token as not_listed, now that the docstring explains it.
Merge: safe now. It merges cleanly into current main, 8 commits behind, and the full suite passes on the merge (13,141 passed). Once it is in, #364 (pyproject.toml), #369 (harness_profile.v1.json), #338 (profiles/codex.yaml) and #334 (cli_flags_allowlist.txt) conflict with it and need a rebase.
What I ran, at 4f4fbc9:
make ci-local --offline: 12,845 tests passed, 0 failed, 58 skipped. The one FAIL isschema_compatonharness_profile.v1.json(claude_displaces_agentsremoved). It comes from main: #366 added that property after this branch's base, and the check compares with currentmain.- The CLI with
--bin /bin/false(no model call): a profile withoutuser_dirswith--only user-client, a--work-dirunder aCLAUDE.md, and atrace: textprofile. Each one exits 2, in a dry run and in a run. - The new
tests/test_skill_reach.pyon the previous head e28968b: 8 tests fail there, one or more for each fix. - A mutation of the error condition to
if not transcript.answer:: the two new error-mode tests fail.
Checked, no issues
- @antonwolfy's threads: the non-zero-exit and error-result modes, the CLI pass-through test, and the codex
skillsmode comment are all in. - Schema: the
fetchedverdict and the nullablesummary.cost_usdchange onlyskill_discovery.v1, which is new in this PR.
…anner all() stopped at the first case it could place, so a profile skill dir that escapes the fixture passed a dry run of the whole catalogue (exit 0) and the run refused it after the banner (exit 1). Resolve every case first. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
pyproject.toml package data: keep repos/conformance_profiles/*.yaml from this branch and repos/skill_fixtures.yaml from intel#363. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
…te profile Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> intel#363 added tools.skills and a skills section to the built-in codex profile; the suite profile must equal it but for config_files.
…rix (#364) * feat(repo): effective-instructions receipt for repo reach `repo reach --format receipt` prints, and writes to effective_instructions.json in --out, the effective-instructions/v0 receipt (agentsmd/agents.md#213) of each probed scenario. It joins the probe's marker tokens with budget's start-up rules (Claude Code only): per source the path, sha256, scope, expected state, observed state, observation_method, the harness name and version, the displacements seen in the run, and the unknowns. followed is always unknown. The mapping and its limits are in docs/repo-evaluation-contract.md §7.1. The receipt is validated against schemas/effective_instructions.v1.json and registered in artifacts.SCHEMA_VERSIONS. The default output of repo reach is unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): keep local directories out of the receipt's tree label A local source's tree label is its absolute path, and the receipt copied it verbatim: a shared effective-instructions receipt then carried the home path. An absolute or drive-letter or ~ label now keeps only its last directory; URLs and relative labels are unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(repo): reach conformance suite and a nightly agent x version matrix The reach probe says what a harness loaded in one run. This adds the suite that runs it over a set of fixture repositories for several harnesses and versions, and compares each observed state with what the harness's vendor documentation says. - data/reach-conformance/: eight fixture repositories (root, nested, CLAUDE.md with AGENTS.md, a Claude Code @ import, an override file, a subdirectory shadow, a file over the size cap, a symbolic link), a guidance-only manifest that probes every guidance name for every harness, and expectations.yaml (reach_conformance.v1) with the documented state per file and scenario; anything undocumented is unknown and reported as no_claim. - repos/conformance.py: harness registry, availability (a harness with no model, credential or CLI is a skipped run), the agreement rules, one run per fixture from the effective-instructions receipt, and the merge and markdown rendering of reach_matrix.v1. - scripts/reach_conformance.py: check (expectations against the tree), run (paid calls, only for harnesses that can run here) and merge. - .github/workflows/reach-matrix.yml: nightly and workflow_dispatch, one job per harness with only its own secret, then a merge job that publishes the matrix as an artifact. Not a pull-request check. - docs/reach-matrix.md: design, limits and how to run and extend it. - reach_matrix.v1 is registered as an artifact kind and added to the runs views and the CLI snapshot. Offline tests cover the rules, the run outcomes with the harness faked, merge refusal, rendering, drift checks and the CLI. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): a malformed reach receipt fails its run, not the whole harness load_artifact raised on a receipt that did not validate, which aborted the remaining fixtures and harnesses of the nightly run. Such a run is now failed with the validation error as its reason. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): keep the reach matrix comment blocks to two lines The static comment-block check fails a block over two lines. The sources and the design move to docs/reach-matrix.md, which both comments point to. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): every unreadable reach receipt fails its run The catch named only the schema error, so a receipt that is not JSON or not UTF-8 still aborted the harness. Catch OSError and ValueError, which covers both, and test a truncated and a non-object receipt. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): a reach matrix names its expectations revision; merge refuses a mix reach_matrix.v1 gains expectations_digest, the sha256 of the expectations as canonical JSON. merge refuses matrices built from different revisions, and the merge command refuses matrices whose digest is not that of the expectations file it renders against. Failed and skipped runs, which name no version, no longer count as a repeated harness version on merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): skip only failed and skipped runs in the merge repeat check; document the digest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Create the markdown output directory before merge writes it Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Document that run_harness env only decides availability Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Reject a scenario key with no directory in the expectations check Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Reject a repeated harness name in the run command Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Let the test matrix helper take empty runs and cells Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Leave failed runs out of the markdown version columns Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): answer the reach matrix review - CI: exempt the fixtures' CLAUDE.md and AGENTS.override.md from the license header check, add the header to docs/reach-matrix.md, and regenerate the CLI snapshot (reach_matrix sorts before redteam). - Nightly workflow: default bash shell; pip cache; Node.js set up for the CLI installs; a CLI is installed only with its model variable; Claude Code uses Bedrock (CLAUDE_CODE_USE_BEDROCK, AWS_REGION) only when the Anthropic key is absent; the merge fails when a harness wrote no matrix. - Markdown: a multi-line stderr reason stays on one table row. - CLI: an unusable expectations file, matrix or merge input prints one line and exits 1, no traceback. - Docs: --budget is enforced by Claude Code only; "marker self-report" label; versioning rule matches the persisted-format policy. - Tests for the duplicate harness, the markdown directory, the failed-run column, the malformed scenario key, the receipt cleanup, the run exit code and the merge repeat rules. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): answer the reach matrix review - run_harness: each repo reach run gets exactly the environment it is given, not the inherited one - codex: the suite runs data/reach-conformance/codex.profile.yaml, whose provider reads CODEX_API_KEY (codex exec sends no bearer for the built-in profile) - run_case: a receipt that cannot be cleared fails the run instead of being read as this run's - merge: generated_at is compared as an instant; a timestamp that is not timezone-aware is refused - markdown: every dynamic table field goes through _table_text, not only the reason - docs: PYTHONPATH=. for the scripts; --budget is per Claude Code scenario Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): shorten the codex profile comment; list the reach matrix in runs.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com> * docs(repo): shorten the codex suite profile's comment block to two lines Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): ship the codex suite profile in the package; find its CLI on the run's PATH The reach suite's codex profile lived under data/, outside the package, so an installed wheel could not find it. It moves to repos/conformance_profiles/ and pyproject's package-data ships it; a test checks the wheel list and loads it through importlib.resources. Availability looked for the codex CLI on this process's PATH, but the run gets only the env it is given. It now searches the PATH of that env, so a CLI on the parent's PATH alone is not available and one only on the run's PATH is. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): say which preflight errors make run exit 1 run also exits 1 before any fixture runs: expectations drift, a harness given twice, an input that cannot be loaded. Errors-only and skipped runs still exit 0. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): check harness keys in the expectations; say what inconclusive and link cells mean The expectations check now names a harness key that no harness has, so a typo cannot silently leave its cells as no_claim. The matrix legend explains the inconclusive and link cells, and the run's --budget help says it caps Claude Code only. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): write reach outputs under results/ and reports/; reword the codex profile note The example run and merge write to the gitignored results/ and reports/ directories. A file cut at the size cap reads as unknown. The codex profile comment no longer names a PR number, which would go stale after merge. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): refuse an out directory that holds the fixtures; check only the scenarios reach runs A run clears <out>/<case> before it probes, so an out directory that holds the fixtures (or sits in them) deleted fixtures. run_harness and the run command now refuse it before anything is created or cleared. check also accepted scenario ids reach never runs (read=. and a cwd= directory with no guidance file); it now accepts only the ids reach produces. The receipt-failure doc now names invalid and unreadable receipts. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): fail a reach run in which no scenario answered A rejected key or a hung CLI left every scenario without an answer, yet the run was recorded as errors, so `run` exited 0 and the nightly published a matrix with no observation. Such a run is now failed, and an errors or failed run's reason is the first scenario error from reach.json, not reach's stderr. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): score no not_loaded from a reach scenario that answered in prose A model that refuses the probe answers in prose with no tokens; the receipt flags its not_loaded states as possible under-reports, but the matrix scored them against the docs and published false disagreements. Such a not_loaded is now observed unknown, so its cell is inconclusive. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): stamp the fixture tree in a reach matrix and refuse to merge two expectations_digest covers the documented states only. A matrix observed over an edited fixture tree merged with one over the original, and the edit read as a change between harness versions. reach_matrix.v1 now carries fixtures_digest, and merge refuses matrices with different ones. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(repo): hold the codex suite profile to the built-in one in every field The guard compared invoke, env_allow and egress only, so a change to the built-in profile's bin_companions, trace, tools or version_cmd passed while the nightly kept running the stale copy. The test now compares the whole document except the suite's config_files. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs(repo): say the fixtures digest covers the expectations and guidance files Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(reach): pin the fixtures digest's link, framing and walk-order rules The digest test only edited and renamed a regular file. Add three tests that each fail on one mutation of fixtures_digest (checked one by one): a link hashed by its target's bytes, a link hashed as a bare "L", links skipped, no length prefix on each part, and a walk-order-dependent sort. No code change. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): carry the skills sandbox and skill dirs into the codex suite profile Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> #363 added tools.skills and a skills section to the built-in codex profile; the suite profile must equal it but for config_files. --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>
* dokimi: add repo skill-reach, a skill-discovery probe Places token-seeded SKILL.md fixtures from a catalogue (repos/skill_fixtures.yaml) in a fresh workspace, HOME and config dir and reports which skills a harness lists: listed / fetched / not_listed / error per case and scenario, and a verdict per same-name collision group. Profiles gain a skills block and a skills tool mode (claude-code, codex). Writes skill_discovery.v1. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * skill-reach: record the catalogue sha256; test the fetched class Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(skill-reach): restore the reach helpers repo.py still calls The move of skill-reach into its own command module also moved the receipt helpers that `repo reach` uses, so repo.py raised NameError (ruff F821, mypy name-defined). Put them back in repo.py and drop the copies from the skill-reach module. Also: strict zip() in the collision verdict (B905); the fixture header is two lines, with the field reference in the skill_reach docstring (comment-block check); the CLI snapshot is regenerated under Python 3.13 to match the CI hashes. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(skill-reach): pin the execution id of the skills tool mode The claude-code profile gained a `skills` tool mode for repo skill-reach, so the built-in profile now yields one more execution id. The six pinned ids are unchanged, so no existing batch's identity moves; the new (skills, native) id is added to the golden table. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(skill-reach): cover the refusal and edge branches of the probe The changed-lines gate (diff-cover vs intel/main) measured 76% on this PR. Add behaviour tests for the CLI usage and failure exits, catalogue validation, collision verdicts, fixture placement (symlinks, undeclared dirs), the profile refusals, the trace clean-up, a timed-out and an unstartable harness, and the table rendering. No code changed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(skill-reach): confine fixture paths, count Skill calls, scrub Git env - Reject drive-qualified paths (C:, C:escape) in at, dirname and cwd. - Place symlink-case targets under an index, not the raw case id, so an id cannot write outside the probe's temp dir. - Count a Skill invocation as the agent looking for itself (fetched, not listed). - Run git in the fixture with an allow-listed env and no global or system config, so a caller's GIT_DIR, GIT_WORK_TREE or hooks do not apply. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(skill-reach): refuse bad catalogue types, skip shared dirs, report run errors - A catalogue whose cwd or collision is not a string, or whose file is not UTF-8, is a catalogue error, not a traceback. - A case whose skill directory an earlier case already holds is skipped with a reason instead of overwriting that copy. - A missing git executable is a SkillReachError. - Frontmatter names that YAML would load as another type (null, true, 123) are quoted. - Any tool call (Grep, Glob, ...) makes a scenario fetched, not listed. - --budget is refused for a harness with no spend-cap flag (codex, copilot); the banner says the timeout is the only bound. Default is now unset. - A missing work dir or an unwritable --out exits 1 with a diagnostic. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(skill-reach): refuse bad dry runs, count edits as looking, keep prose clean - --dry-run runs the same profile check as a run (native_prompt, no skills block, no skills tool mode), so it cannot print an argv a run refuses - an Edit (or any other tool call) before the answer makes the scenario fetched, as a Read or a Skill call does - a non-string unknown catalogue key is reported as an unknown field, not a TypeError; a NUL byte in `at`, `dirname` or `cwd` is a catalogue error - the verification token's letters are not counted as prose words - document that a description-missing skill listed by name only (`name -`) is not_listed (maintainer decision) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(skill-reach): check a case's `at` when the catalogue loads A bad scope, a traversing path or a placeholder typo in `at` passed --dry-run with exit 0 and was refused only while placing the fixture. Resolve `at` against stand-in skill dirs in _parse_case, so both modes refuse it as a usage error (exit 2) before anything runs. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(skill-reach): exit 2 for every input the command refuses A profile check_profile refuses, a profile skill dir that escapes, an --out under a file and a missing --work-dir exited 1, the code for a run that ran and failed (docs/quickstart.md exit codes). Check them with the other usage errors, before the banner, in dry run and run alike. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(skill-reach): take catalogue_sha256 from the bytes that were parsed The digest came from a second read of the path after every scenario: a pipe recorded the hash of empty input, an edit during the run the hash of the edited file, and a deleted catalogue lost the finished run. Hash the bytes load reads and pass that digest into the run. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> # Conflicts: # dokimi/dokimi/commands/repo_skill_reach.py * fix(skill-reach): write traces to skill-traces/, not reach's traces/ The cleanup of an earlier run's traces matched the names repo reach gives its own, so a skill-reach run into a shared --out deleted the traces behind a stored reach.json. Each probe now owns its directory. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(skill-reach): pin the guards that decide a fetched or error class No test failed when a shell command stopped counting as looking, when an empty answer with exit 0 stopped being an error, or when the body-token rule for a description: missing skill was dropped or widened. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * Note that the codex skills mode still allows commands Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(skill-reach): fix review findings on verdicts, cost, dry runs and tests - A collision verdict is `fetched` when a member was fetched: which copy the agent named after looking says nothing about precedence. - summary.cost_usd is null when any scenario's cost is unknown (Codex, Copilot, a timeout); the table prints cost=unknown. - A dry run refuses what a run refuses: no selected case has a skill dir in the profile, a --work-dir that is not a directory, guidance above it. - The ancestor guard also checks the profile's own guidance names. - A profile with trace: text is refused (no tool events, so a fetched skill cannot be told from a listed one). - Tests: a non-zero exit or an error result with an answer is an error; the CLI passes --timeout, --salt and --budget to the run. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(skill-reach): check every selected case before a dry run or the banner all() stopped at the first case it could place, so a profile skill dir that escapes the fixture passed a dry run of the whole catalogue (exit 0) and the run refused it after the banner (exit 1). Resolve every case first. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com>
…rix (intel#364) * feat(repo): effective-instructions receipt for repo reach `repo reach --format receipt` prints, and writes to effective_instructions.json in --out, the effective-instructions/v0 receipt (agentsmd/agents.md#213) of each probed scenario. It joins the probe's marker tokens with budget's start-up rules (Claude Code only): per source the path, sha256, scope, expected state, observed state, observation_method, the harness name and version, the displacements seen in the run, and the unknowns. followed is always unknown. The mapping and its limits are in docs/repo-evaluation-contract.md §7.1. The receipt is validated against schemas/effective_instructions.v1.json and registered in artifacts.SCHEMA_VERSIONS. The default output of repo reach is unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): keep local directories out of the receipt's tree label A local source's tree label is its absolute path, and the receipt copied it verbatim: a shared effective-instructions receipt then carried the home path. An absolute or drive-letter or ~ label now keeps only its last directory; URLs and relative labels are unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * feat(repo): reach conformance suite and a nightly agent x version matrix The reach probe says what a harness loaded in one run. This adds the suite that runs it over a set of fixture repositories for several harnesses and versions, and compares each observed state with what the harness's vendor documentation says. - data/reach-conformance/: eight fixture repositories (root, nested, CLAUDE.md with AGENTS.md, a Claude Code @ import, an override file, a subdirectory shadow, a file over the size cap, a symbolic link), a guidance-only manifest that probes every guidance name for every harness, and expectations.yaml (reach_conformance.v1) with the documented state per file and scenario; anything undocumented is unknown and reported as no_claim. - repos/conformance.py: harness registry, availability (a harness with no model, credential or CLI is a skipped run), the agreement rules, one run per fixture from the effective-instructions receipt, and the merge and markdown rendering of reach_matrix.v1. - scripts/reach_conformance.py: check (expectations against the tree), run (paid calls, only for harnesses that can run here) and merge. - .github/workflows/reach-matrix.yml: nightly and workflow_dispatch, one job per harness with only its own secret, then a merge job that publishes the matrix as an artifact. Not a pull-request check. - docs/reach-matrix.md: design, limits and how to run and extend it. - reach_matrix.v1 is registered as an artifact kind and added to the runs views and the CLI snapshot. Offline tests cover the rules, the run outcomes with the harness faked, merge refusal, rendering, drift checks and the CLI. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): a malformed reach receipt fails its run, not the whole harness load_artifact raised on a receipt that did not validate, which aborted the remaining fixtures and harnesses of the nightly run. Such a run is now failed with the validation error as its reason. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): keep the reach matrix comment blocks to two lines The static comment-block check fails a block over two lines. The sources and the design move to docs/reach-matrix.md, which both comments point to. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): every unreadable reach receipt fails its run The catch named only the schema error, so a receipt that is not JSON or not UTF-8 still aborted the harness. Catch OSError and ValueError, which covers both, and test a truncated and a non-object receipt. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): a reach matrix names its expectations revision; merge refuses a mix reach_matrix.v1 gains expectations_digest, the sha256 of the expectations as canonical JSON. merge refuses matrices built from different revisions, and the merge command refuses matrices whose digest is not that of the expectations file it renders against. Failed and skipped runs, which name no version, no longer count as a repeated harness version on merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): skip only failed and skipped runs in the merge repeat check; document the digest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Create the markdown output directory before merge writes it Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Document that run_harness env only decides availability Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Reject a scenario key with no directory in the expectations check Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Reject a repeated harness name in the run command Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Let the test matrix helper take empty runs and cells Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Leave failed runs out of the markdown version columns Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): answer the reach matrix review - CI: exempt the fixtures' CLAUDE.md and AGENTS.override.md from the license header check, add the header to docs/reach-matrix.md, and regenerate the CLI snapshot (reach_matrix sorts before redteam). - Nightly workflow: default bash shell; pip cache; Node.js set up for the CLI installs; a CLI is installed only with its model variable; Claude Code uses Bedrock (CLAUDE_CODE_USE_BEDROCK, AWS_REGION) only when the Anthropic key is absent; the merge fails when a harness wrote no matrix. - Markdown: a multi-line stderr reason stays on one table row. - CLI: an unusable expectations file, matrix or merge input prints one line and exits 1, no traceback. - Docs: --budget is enforced by Claude Code only; "marker self-report" label; versioning rule matches the persisted-format policy. - Tests for the duplicate harness, the markdown directory, the failed-run column, the malformed scenario key, the receipt cleanup, the run exit code and the merge repeat rules. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): answer the reach matrix review - run_harness: each repo reach run gets exactly the environment it is given, not the inherited one - codex: the suite runs data/reach-conformance/codex.profile.yaml, whose provider reads CODEX_API_KEY (codex exec sends no bearer for the built-in profile) - run_case: a receipt that cannot be cleared fails the run instead of being read as this run's - merge: generated_at is compared as an instant; a timestamp that is not timezone-aware is refused - markdown: every dynamic table field goes through _table_text, not only the reason - docs: PYTHONPATH=. for the scripts; --budget is per Claude Code scenario Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): shorten the codex profile comment; list the reach matrix in runs.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com> * docs(repo): shorten the codex suite profile's comment block to two lines Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): ship the codex suite profile in the package; find its CLI on the run's PATH The reach suite's codex profile lived under data/, outside the package, so an installed wheel could not find it. It moves to repos/conformance_profiles/ and pyproject's package-data ships it; a test checks the wheel list and loads it through importlib.resources. Availability looked for the codex CLI on this process's PATH, but the run gets only the env it is given. It now searches the PATH of that env, so a CLI on the parent's PATH alone is not available and one only on the run's PATH is. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): say which preflight errors make run exit 1 run also exits 1 before any fixture runs: expectations drift, a harness given twice, an input that cannot be loaded. Errors-only and skipped runs still exit 0. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): check harness keys in the expectations; say what inconclusive and link cells mean The expectations check now names a harness key that no harness has, so a typo cannot silently leave its cells as no_claim. The matrix legend explains the inconclusive and link cells, and the run's --budget help says it caps Claude Code only. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(repo): write reach outputs under results/ and reports/; reword the codex profile note The example run and merge write to the gitignored results/ and reports/ directories. A file cut at the size cap reads as unknown. The codex profile comment no longer names a PR number, which would go stale after merge. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): refuse an out directory that holds the fixtures; check only the scenarios reach runs A run clears <out>/<case> before it probes, so an out directory that holds the fixtures (or sits in them) deleted fixtures. run_harness and the run command now refuse it before anything is created or cleared. check also accepted scenario ids reach never runs (read=. and a cwd= directory with no guidance file); it now accepts only the ids reach produces. The receipt-failure doc now names invalid and unreadable receipts. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): fail a reach run in which no scenario answered A rejected key or a hung CLI left every scenario without an answer, yet the run was recorded as errors, so `run` exited 0 and the nightly published a matrix with no observation. Such a run is now failed, and an errors or failed run's reason is the first scenario error from reach.json, not reach's stderr. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): score no not_loaded from a reach scenario that answered in prose A model that refuses the probe answers in prose with no tokens; the receipt flags its not_loaded states as possible under-reports, but the matrix scored them against the docs and published false disagreements. Such a not_loaded is now observed unknown, so its cell is inconclusive. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * fix(repo): stamp the fixture tree in a reach matrix and refuse to merge two expectations_digest covers the documented states only. A matrix observed over an edited fixture tree merged with one over the original, and the edit read as a change between harness versions. reach_matrix.v1 now carries fixtures_digest, and merge refuses matrices with different ones. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * test(repo): hold the codex suite profile to the built-in one in every field The guard compared invoke, env_allow and egress only, so a change to the built-in profile's bin_companions, trace, tools or version_cmd passed while the nightly kept running the stale copy. The test now compares the whole document except the suite's config_files. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs(repo): say the fixtures digest covers the expectations and guidance files Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(reach): pin the fixtures digest's link, framing and walk-order rules The digest test only edited and renamed a regular file. Add three tests that each fail on one mutation of fixtures_digest (checked one by one): a link hashed by its target's bytes, a link hashed as a bare "L", links skipped, no length prefix on each part, and a walk-order-dependent sort. No code change. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * fix(repo): carry the skills sandbox and skill dirs into the codex suite profile Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> intel#363 added tools.skills and a skills section to the built-in codex profile; the suite profile must equal it but for config_files. --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com>
…intel#363 asked for Each new test fails on a one-line mutant that passed the whole suite; no code under dokimi/dokimi/ changes. intel#334 (context prefilter): - a write target glued to a heredoc sign (cat >notes.txt<<docs/guide.md) ends at the `<`: the delimiter is no path (_REDIRECT's `<` exclusion); - a bare-name rule `Makefile` counts an edit of Makefile only, not Makefile.bak or GNUMakefile, and counts C:\r\Makefile (_file_named); - `sudo echo changelog.d/x.md` prints the path, it does not act on it (strip_wrappers in _prints). intel#326 (repo budget): - an unclosed `paths: [null` or `paths: [~` reads as unscoped (_names_pattern's null and ~); - a rule reached through docs/alias.md drops its frontmatter before its @paths are followed (_import_text's _target half); - an import of a rule path linked to a plain file keeps the rule's body count (_counted_import's RULE_GLOB.match(f)). intel#363 (repo skill-reach): an OSError while saving the artifact exits 1 with a message; --bin reaches the harness run; --keep keeps the fixture and names it in the artifact. intel#323 (repo config): shared '**/Makefile' claims a root Makefile; a link c[1] on the way to the config is claimed as written; link_text keeps the `..` prefix of a subdirectory link that leaves the tree and comes back, and reads a sibling t2/ of root t as outside; a tracked `:x.sh` takes its mode from the index (--literal-pathspecs). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
#389) Each new test fails on a one-line mutant that passed the whole suite; no code under dokimi/dokimi/ changes. #334 (context prefilter): - a write target glued to a heredoc sign (cat >notes.txt<<docs/guide.md) ends at the `<`: the delimiter is no path (_REDIRECT's `<` exclusion); - a bare-name rule `Makefile` counts an edit of Makefile only, not Makefile.bak or GNUMakefile, and counts C:\r\Makefile (_file_named); - `sudo echo changelog.d/x.md` prints the path, it does not act on it (strip_wrappers in _prints). #326 (repo budget): - an unclosed `paths: [null` or `paths: [~` reads as unscoped (_names_pattern's null and ~); - a rule reached through docs/alias.md drops its frontmatter before its @paths are followed (_import_text's _target half); - an import of a rule path linked to a plain file keeps the rule's body count (_counted_import's RULE_GLOB.match(f)). #363 (repo skill-reach): an OSError while saving the artifact exits 1 with a message; --bin reaches the harness run; --keep keeps the fixture and names it in the artifact. #323 (repo config): shared '**/Makefile' claims a root Makefile; a link c[1] on the way to the config is claimed as written; link_text keeps the `..` prefix of a subdirectory link that leaves the tree and comes back, and reads a sibling t2/ of root t as outside; a tracked `:x.sh` takes its mode from the index (--literal-pathspecs). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
repo skill-reach: measure which Agent Skills a harness lists
Problem
Client behaviour for Agent Skills discovery (.agents/skills vs client dirs, user vs project scope, nesting, symlinks, same-name collisions, description formats) is documented only from reports (agentskills#15). Nothing measures it per client version.
Change
dokimi repo skill-reach: places token-seeded SKILL.md fixtures from a catalogue (repos/skill_fixtures.yaml, 25 cases, or--catalogue) in a fresh git workspace, HOME and config dir; runs the harness once per scenario (cwd root, plus each casecwd) with askillstool mode; asks it to list visible skills and their description tokens.bodymarks a body token reported too. Each same-name collision group gets a verdict one / all / some / none / error.skillsblock (project_dirs, user_dirs) andtools.skills; claude-code (--tools Skill) and codex (--sandbox read-only) declare both. Profiles without the block are refused.skill_discovery.v1(schema, SCHEMA_VERSIONS, runs kind, fixture artifact).dokimi/commands/repo_skill_reach.py(repo.py hit its line cap).Evidence
tests/test_skill_reach.py: fake harness behaving like Claude Code (ignores .agents/skills and nested dirs, user copy hides project copy) gives listed/not_listed and verdict
oneas expected; a crashing harness giveserror, notnot_listed; artifact validates.The artifact also records
catalogue_sha256, so a case id is tied to the catalogue that defined it.Live run: pending
The live run of this command against real Claude Code and Codex has not been done yet: SSH access to the lab machine is down. The prototype findings so far (Claude Code 2.1.304 does not list
.agents/skillsor nested dirs, and a user skill hides a project skill with the same name; Codex 0.166.0 lists both copies and follows symlinks) come from a hand-rolled script, not from this command. This PR stays a draft until the live results are added here.Open question
The artifact is
skill_discovery.v1while the command isskill-reach, the same pattern asreach->guidance_reach.v1. If a different name is preferred, it is cheap to change now and costly after merge.Follow-ups
skipped; consider a per-case placement field like guidance_reach'salias_kind.Local self-review (pr-review, lenses + design, L4): no blocking findings; the
fetchedclass test and the catalogue digest came out of it.🤖 Generated with Claude Code