feat(novelty): routing takes the catalog field, skips disabled and mirrored skills, adds all-vs-all

dokimi/changelog.d/20261021_130000_napetrov_routing-catalog-manifest.md

@@ -0,0 +1,9 @@
+### Changed
+
+- **`novelty routing --catalog <dir>` reads the catalog through its `skills.yaml`** — when the
+  catalog root has a `skills.yaml`, that manifest decides which skills are competitors, as it
+  does for `catalog scan`, so a `SKILL.md` it does not list is no longer a sibling. Without a
+  manifest, every `SKILL.md` under the directory counts as before. A declared skill that cannot
+  be loaded is skipped with a warning on stderr instead of making the whole field unreadable.
+  Two catalog skills with the same name and different content are refused, naming both paths,
+  rather than the run failing on a duplicate competitor name.

dokimi/changelog.d/20261021_131000_napetrov_routing-skip-non-skill.md

@@ -0,0 +1,7 @@
+### Changed
+
+- **`novelty routing` skips a non-`SKILL.md` competitor with a warning** — a
+  `skill:` competitor that names a file other than `SKILL.md`, or a directory without one,
+  is left out of the field and reported on stderr. It used to abort the whole run, so a
+  shell glob over a skills directory (which also matches helper scripts) could not be
+  passed. The artifact under test is still refused.

dokimi/changelog.d/20261021_132000_napetrov_routing-model-hidden-skills.md

@@ -0,0 +1,8 @@
+### Changed
+
+- **`novelty routing` leaves `disable-model-invocation` skills out of the field** — no
+  harness lists such a skill, so the model cannot select it, and a card for it scored a
+  choice that does not exist (`ultracave` and `megacave` in the caveman repo). They are
+  left out of a `--catalog` and of `--competitors`, each with a warning on stderr, unless
+  `--include-disabled` is passed. A disabled artifact is still routed, with a warning that
+  its rate is not a harness rate.

dokimi/changelog.d/20261021_133000_napetrov_routing-mirror-dedupe.md

@@ -0,0 +1,6 @@
+### Fixed
+
+- **`novelty routing --catalog` with a `skills.yaml` counts a mirrored skill once** — a catalog
+  that lists the same `SKILL.md` bytes under two paths (a plugin copy of the `skills/` tree)
+  used to put the name in the field twice, and the run was refused as naming a competitor more
+  than once. The copy nearest the catalog root stands for the card; the others are named on stderr.

dokimi/changelog.d/20261021_134000_napetrov_routing-all-vs-all.md

@@ -0,0 +1,14 @@
+### Added
+
+- **`novelty routing --all-vs-all`** routes every skill that has queries against the whole
+  field, in turn, and reports a confusion matrix plus an attractor table: the skills picked
+  for other skills' queries, with how many of those queries they took and from whom. Takes
+  `--queries-by-skill <json>` and `--out`, and no `--artifact` or `--repeats`. The lexical arm always runs;
+  `--model` adds the model arm.
+- `--all-vs-all` rates are over the scored queries (`n_scored`, `n_unscored`; `null` own rate
+  when none was scored; a query with no content word is unscored in both arms and not sent
+  to the model); a lexical tie among other cards is `(tie)`; a failed model call
+  exits non-zero; a repeated skill key in the queries file is refused; routing results record
+  the field rule under `field`.
+- `--all-vs-all` names skills with the same description as one tied group (`tied_groups`), not
+  as each other's attractor: a pick between them is in the confusion matrix only.

dokimi/docs/skill-novelty-walkthrough.md

@@ -41,12 +41,10 @@ python cli.py novelty routing --artifact skill:skills/my-skill \
 ```
 
 The competitors are the skills a harness lists next to this one: `--catalog <dir>` takes
-every other `SKILL.md` under the directory (a `template/` copy and a byte-identical mirror
-that loads under the same skill name are left out, and vendored and VCS directories such as
-`.git`, `node_modules`, `site-packages` and virtualenvs are not walked), and `--competitors`
-adds or names them one by
-one. With no real siblings at hand, skip this step and say so; a field of competitors made
-up for the run measures nothing.
+every other skill the catalog lists (a `template/` copy and vendored or VCS directories
+such as `.git` are left out, and a byte-identical copy under another path counts once), and
+`--competitors` adds or names them one by one. With no real siblings at hand, skip this step and say so; a field of
+competitors made up for the run measures nothing.
 
 Read `recall@1` against the competitors, not alone. Measured on the study corpus, every
 `dpnp-*` skill scored 0.0 against its siblings, and `linux-perf` scored 0.0 recall@1 on

dokimi/docs/skill-novelty.md

@@ -600,10 +600,23 @@ python cli.py novelty routing --artifact skill:path/to/skill \
     --queries queries.json [--model M [--repeats N]]
 ```
 
-`--catalog <dir>` takes the competitors from a skill directory instead: every `SKILL.md`
-under it except the artifact itself (a `template/` copy and a byte-identical mirror that
-loads under the same skill name are left out, and vendored and VCS directories such as `.git`,
-`node_modules`, `site-packages` and virtualenvs are not walked).
+`--catalog <dir>` takes the competitors from a skill directory instead: every skill the catalog
+lists except the artifact itself. A `skills.yaml` at the root decides what the catalog lists;
+without one, every `SKILL.md` under it counts, except a `template/` copy and vendored or VCS
+directories such as `.git` and `node_modules`, which are not walked. A byte-identical copy under
+another path counts once. A declared skill that cannot be loaded is skipped with a warning. Two
+catalog skills with the same name and different content are refused, with both paths named:
+rename one, or delete the copy you do not mean to route against.
+A `skill:` competitor that is not a `SKILL.md`, or a directory without one, is skipped with a
+warning. A shell glob must be expanded first, with the prefix on each path
+(`skill:dir/a/SKILL.md skill:dir/b/SKILL.md`): an unexpanded `skill:dir/*` is refused as a missing
+path.
+A skill that sets `disable-model-invocation` is left out of the field, both from a catalog
+and as `--competitors`: no harness lists it, so the model cannot select it. Pass
+`--include-disabled` to keep it. A disabled artifact is still routed, with a warning; the flag does not change that.
+A catalog that holds the same `SKILL.md` under two paths (a plugin copy of its `skills/`
+tree) counts it once: the copy nearest the catalog root stands for the card, and the
+others are named on stderr.
 
 Two numbers, reported side by side and never merged:
 
@@ -631,20 +644,50 @@ Two numbers, reported side by side and never merged:
   queries x repeats, `n_abstained` and `n_failed` count calls, and each `per_query` row
   carries its `repeat` index.
 
-A `--model` run also writes a `cost` block (calls, input and output tokens, cache read and
+A single-artifact `--model` run also writes a `cost` block (calls, input and output tokens, cache read and
 write tokens, and `usd`, which is `null` when any call this run paid for had no known price) and prints the
 same totals on stdout. Cache tokens are reported separately from input tokens, and depending on
 the provider they may or may not be counted inside `input_tokens`. A run without `--model` makes
 no calls and has no `cost` block. `usd` is also `null` when a call returned no usage, and a
 call that raised is not counted. A call replayed from the response cache is counted in
 `replayed_calls` and left out of the token and `usd` totals, since this run did not pay for it;
-`llm_calls` still counts it.
+`llm_calls` still counts it. `--all-vs-all` writes no `cost` block.
 
 `recall_at_3` comes back `null` with three or fewer candidates, where every rank is `<= 3`
 and the figure would be a structural 1.0. The lexical arm also reports `n_queries_given`
 next to `n_queries_scored`, since a query with no content words is dropped rather than
 counted as a miss.
 
+**All skills against each other.** `novelty routing --all-vs-all --catalog <dir>
+--queries-by-skill queries.json --out routing.json` (add `--model M` for the model arm) makes
+every skill that has queries the gold in turn, routed against the whole field. The file is a
+JSON object of skill name to the requests that should select it, each written without the
+skill's own text. A skill named twice in the file is refused. The output has a confusion
+matrix (gold to picked to count, with `(none)` for a query no card shares a word with or, in
+the model arm, an abstention, and `(tie)` for a query on which two or more other cards tie for
+the top lexical score, so no field order decides it), an own rate per skill, and an attractor
+table: a skill picked for other skills' queries, with `wrongly_picked` of `sibling_queries` and
+the skills whose queries it took (`from`). Skills whose descriptions have the same words (case and punctuation aside) are a tied group
+(`tied_groups`, named once on the console): a pick between them is in the confusion matrix but
+is no attractor, and a tied skill's own rate is lower by the picks inside its group. Every rate is over the queries that were scored:
+`n_scored` and `n_unscored` per skill say how many, a query with no content word or a failed
+model call is unscored, and a skill with none scored has an own rate of `null`, not 0.0. A run
+with any failed model call writes the result and exits non-zero. The saved file records the
+field rule under `field` (`include_disabled`, the `competitors` and what was `left_out`); a
+single-artifact run records it too. It takes no `--artifact`, `--queries`,
+`--queries-from-headings` or `--repeats` (one model call per query). The model arm also records the selector `model`, so two runs with
+different models can be told apart. It is refused with fewer than two skills, a name repeated
+in the field, a query file that names a skill outside the field, or no queries at all.
+
+**Cost of `--model` in all-vs-all.** The model arm makes one selector call per query, across
+every skill in the field: the call count is the number of queries in `queries.json` that have a
+content word (one without is unscored in both arms and not sent), and a failed call still counts. Each call sends the whole field's cards, so a large field costs
+more per call as well as in calls. Before the first call the run prints that count to stderr
+(`all-vs-all: --model makes N model calls, one per query, ...`). The run does not pause for it,
+so count the queries first, or pass `--max-usd` to stop the run at a spend limit. A lexical-only run (no `--model`) makes no model calls.
+A stored response in `DOKIMI_RESPONSE_CACHE` is replayed without a call, so a rerun can make
+fewer calls than the printed count.
+
 Three properties worth knowing before quoting either:
 
 - **Competitors are mandatory.** The existing `skill-agent:` arm offers one skill, which