0.11.17: a holdout outside the builder's read scope, the charter defect it found, and the two classes behind it (#112 #149 #148) by betmoar · Pull Request #146 · betmoar/cc-operator-plugin

betmoar · 2026-09-19T01:46:13Z

What this is

Everything gating this repo — validate_plugin.py, tests/, the workflows — is written by the agent that writes the code and readable by it. #112 named the consequence: given enough attempts a builder optimises against checks it can read. This adds the one check outside that loop.

The holdout lives in ci-admin/cc-operator-holdout on lokaal, not here. A session working in this repo never clones it, so the independence is structural — no prompt-level denial to forget, no PR guard to bypass. (The sibling project this argument came from keeps its holdout in-repo behind exactly those two mechanisms.)

The four properties, and how each was met

Outside the read scope — separate repo, plus a CLAUDE.md rule not to clone it.
Derived from the spec, not the code — holdout.sh was written by a claude -p process in an empty directory with Read/Write/Edit/Bash/Glob/Grep/Agent/WebFetch/WebSearch all denied. Not an agent told not to look at scripts/: one with no way to look at anything. Its entire context was templates/OPERATOR.md plus a black-box interface block. The prompts ship in derivation/ so the claim is auditable.
Exact values, not properties — every check names a count and a content. Proof: a mutation making ops-claims.sh examine only the first changed path (the LIMIT 1 shape) drives it red.
Absence fails closed — run-holdout.sh demands HOLDOUT_PASSED sha=<the sha it was asked to test> plus a check-count floor. On Forgejo 16 that is not optional: a queued job has no actions/tasks row and conclusion is always null, so "did not run" and "found nothing" are the same silence.

What it found on its first run

The charter prescribed ops-claims.sh --claimed "<paths>". The shipped CLI has required a mandatory --since <sha> since CR2 and exits 2 without it — an operator following the charter verbatim got a usage error.

All 1101 in-repo shell cases passed throughout, because every one of them was written against the CLI and so passed --since. Nothing in the repo compared the charter's prescription to the CLI's contract. That is the defect class an in-scope check cannot see: both sides individually correct, never read against each other.

Fixed here by correcting the charter line (144/150 lines, 8923/9000 bytes, 100-char max all within caps).

Measurements

Mutations against the gate CLIs, each restored byte-identical (shasum -c OK):

mutation	holdout
Stop gate `exit 2` → `exit 0`	4 failed
`+dirty` branch of the source stamp deleted	2 failed
`ops-claims.sh` examines only the first changed path	1 failed
ledger row's verdict word hardcoded to `PASS`	1 failed
`--defer` writes its row without clearing the sentinel	1 failed
an auto-bar that blocks without arming	1 failed
`ops-adopt.sh` replaced by `exit 0`	6 failed (0 before the review round)
an installer that creates no ledger	17 failed (14 before the vacuity fix)
restored	41 passed, 0 failed

The first attempt at mutation 3 was a syntax error — red for the wrong reason, proving nothing. Redone syntactically valid (bash -n clean).

Wrapper refusals, each with a crafted stub: absent suite 4, ran-but-reported-nothing 4, wrong sha 5, shrunken suite 6, bad sha 2, no sha named 2, control 0.

Live on the forge (ci-admin/cc-operator-holdout):

run 1458 / 1459 — HOLDOUT_VERIFIED sha=7057dcf7f2f6 checks=41 (runs 1455/1457 were the pre-review 33-check suite)
run 1456 — red on a nonexistent sha: could not check out ... the sha under test does not exist there, exitcode 2

Gates here: validator all contracts hold · shell 1112@1112 · python 411@411 · workflows 384@384 · compress 161@161 · release gate v0.11.17 OK · base-gate PASSED · shellcheck 0.11.0 clean at warning level (one pre-existing SC2329 in an untouched file).

Process note worth recording

Derivation rounds 1–3 went 27/4 → 31/2 → 21/10. Round 3 regressed checks round 2 passed, which tripped the same-target-rework cap at 2 (logged in DECISIONS.md). The escalation was to a different mechanism — a repair dispatch carrying the measured system output back to a denied-context process — not a third guess at the prompt. Round 4: 33/0.

Asymmetry to know about

holdout.sh is derived from the corrected charter, so it no longer re-finds the --since defect. The suite that found it ships as derivation/derived-round2.sh in the holdout repo. A holdout run once against a fixed system tells you nothing about what it caught — the derivation record is the evidence.

0.11.17 — the two classes the holdout's findings belonged to

v0.11.16 fixed the charter line. Both issues below are the same complaint about that:
the INSTANCE was fixed and the CLASS was left to a deferral.

#149 — check_prose_invocations. Correcting three prose copies by hand leaves
nothing that catches the fourth. The check extracts every ops-*.sh --flag
prescription from tracked prose and asserts the CLI would accept it. Both flag sets are
read off each CLI's own parser and usage: forms — a table in the validator would
be a second copy of the contract, drifting the moment the parser changes, which is the
defect the check exists to catch reintroduced one layer up. Mandatory is judged per
form (--owner is required by --mark-handoff and optional in the verdict form;
--expect-clean is a complete form needing no --since). [--since <sha>] in prose
counts as ABSENT — the PLAYBOOK shape, where a presence test reads the flag as
prescribed while the brackets tell the operator it is optional.

Red against the actual defect, three ways, each restored byte-identical:

reverted copy	result
`templates/OPERATOR.md`, `--since` removed	`OPERATOR.md:109` fires
`README.md`, `--since` removed	`README.md:109` fires
`docs/PLAYBOOK.md`, `[--since <sha>]`	`PLAYBOOK.md:565` fires

Ten mutations of the check itself, each named with the case it drove red (#111), each
restored byte-identical. Four defects in the check were found by RUNNING it on the
correct tree before any mutation — a line-anchored scan missing ops-render.sh's
;;-packed arms, mandatory flags treated globally rather than per form, a break that
read --mark-handoff's requirements onto the verdict form (five correct lines
condemned, the charter's among them), and a line-only marker scan that condemned
REPLAY-CHARTER.md's deliberate --ownr probe. A pin that only ever ran against its own
mutation would have shipped all four: the mutation proves the check catches the defect,
not that it leaves correct work alone.

#148 — eleven vacuous refusal assertions. Each proved a writer appended nothing by
comparing two reads of the same file. ABSENT, both reads are "" and [ "" = "" ] is
true, so each certified a refusal about a ledger that was never there. Measured in an
isolated git archive HEAD tree: ops-init.sh mutated to skip the VERDICTS.md copy
(still exit 0) → 991 passed / 110 failed, with eight of the eleven among the PASSES.
After the fix, 993 / 119 with all eight red and naming the missing file. A second
mutation (no DECISIONS.md) covers two more; the eleventh passes there honestly,
because an earlier case's >> "$DECISIONS" creates the file before it reads it.

Four helpers assert the precondition, compare with -eq (which errors on an empty
operand where = succeeds — exactly why the two pre-existing -eq sites failed closed
for free), and NAME the missing file on stderr. Each carries BOTH controls: refuse the
absent file AND accept the present unchanged one, since a helper that refused everything
would pass a refusal-only control set while failing all eleven real call sites.

Floors: FLOOR_shell 1101 → 1112, FLOOR_python 397 → 411, measurements in
tests/floors.env.

Not in scope

The holdout job is dispatched by hand today. Wiring it into this repo's release path needs the reading side (forge-run --expect, which shipped in cc-skills-plugin#5) and is filed separately rather than smuggled in here — #147.

Still open, deliberately: #150 (the derivation is a manual five-round ritual),
#151/#152 (the decision-engine probes this PR records but does not act on),
#153 (MECHANICAL's default binding is dominated on both axes). Each is its own
engagement rather than scope creep here.

…defect it found (#112) The holdout lives in ci-admin/cc-operator-holdout on lokaal and is never cloned by a session working here — independence that is structural rather than enforced, so there is no denial to forget and no guard to bypass. 33 checks derived by a claude -p process in an empty directory with every file tool denied, whose entire context was templates/OPERATOR.md plus a black-box interface block. It could not read scripts/ because it had no way to read anything. When two checks failed the repair was dispatched BACK to a denied-context process with measured evidence; hand-editing them here would have produced a mirror. What it found that 1101 in-repo shell cases did not: the charter prescribed ops-claims.sh --claimed without the --since the CLI has required since CR2, so an operator following it verbatim got exit 2. Every in-repo test passed throughout — they were written against the CLI, so all of them passed --since, and nothing compared the charter's prescription to the CLI's contract. Measured: four mutations red (29/4, 31/2, 32/1, 32/1), each restored byte-identical; six wrapper refusals each with a crafted stub; on the forge run 1457 logs HOLDOUT_VERIFIED sha=7057dcf7f2f6 checks=33 and run 1456 goes red on a nonexistent sha. Gates unchanged: validator green, shell 1101, python 397, workflows 384, compress 161, release gate v0.11.16 OK.

…alse green Two findings from four review panels, each reproduced before fixing. 1. F30, in the fix for a defect the holdout had just found. templates/OPERATOR.md was ONE of three places prescribing ops-claims.sh: README.md carried the same broken form, and docs/PLAYBOOK.md marked --since OPTIONAL in square brackets for a flag the CLI exits 2 without. Fixing the charter alone shipped a PR claiming the invocation was corrected while two copies still said otherwise. Measured both forms against the shipped CLI: --since <sha> --claimed rc=0, bare --claimed rc=2. The holdout cannot catch this — it tests the SYSTEM and all three copies are prose. 2. The holdout's own 33/0 was a false green. ops-adopt.sh appeared in it only as a string the SessionStart guidance must MENTION, never as a CLI it INVOKES. Measured: ops-adopt.sh replaced by 'exit 0' and the suite reported 33 passed, 0 failed — RECOVERY PROTOCOL step 6's whole mechanism deletable with the holdout certifying the release. Closed by a fifth denied-context repair dispatch (41 checks; the same mutation now drives 6 red), floor raised 30->41. Two further checks the review suspected of vacuity were proved real by mutation instead: defer-without-clear and arm-without-sentinel each drive one red alone. Counts corrected throughout, and the CHANGELOG now says plainly that the figure is point-in-time — the holdout is a separate repo on its own history and this line is not kept in sync with it. Gates after: validator all contracts hold, shell 1101@1101, python 397@397, workflows 384@384, compress 161@161, release gate v0.11.16 OK. Forge run 1458: HOLDOUT_VERIFIED sha=7057dcf7f2f6 checks=41.

…s not the mechanism Third review finding. A silent-failure audit caught three checks reporting ok having measured nothing, all the same arithmetic: $(( $(wc -c < absent) )) is 0 so a before/after byte comparison was 0 == 0; sed -n '1,p' on a missing file twice, compared, is equal, so the guard on the ledger's append-only-ness compared nothing to nothing; grep -c yields 0 for 'correctly absent' and 'file not there' alike. Measured with ops-init.sh mutated to create no ledger: 27 passed / 14 failed and all three were among the PASSES — fourteen checks caught the condition while these three reported success about it. After a denied-context repair: 24 / 17, each naming its precondition. Every earlier mutation still red, the gate one sharper (4 -> 10 red). Forge run 1459 verifies 41/0 on the fixed suite. Also answers the portability question: property 1 is 'outside the builder's read scope' and lokaal is one way to buy it, not the mechanism. A second GitHub repo gets it for free. run-holdout.sh contains no Forgejo — verified against a plain GitHub URL, HOLDOUT_VERIFIED sha=7057dcf7f2f6 checks=41 — and the holdout repo's new PORTING.md prices the weaker alternatives (orphan branch, local directory) and says what each gives up. Gates: validator all contracts hold, shell 1101@1101, python 397@397, workflows 384@384, compress 161@161, release gate v0.11.16 OK, shellcheck clean modulo the pre-existing SC2329.

betmoar · 2026-09-19T09:59:33Z

Review round: 4 panels, 3 real findings in my own change

Each reproduced by mutation before fixing. Pushed as 655e339 and 5605f96.

1. The charter fix was one copy of three (F30, in the fix for the defect the holdout found)

templates/OPERATOR.md was not the only place prescribing ops-claims.sh. README.md:109 carried the identical broken form, and docs/PLAYBOOK.md:565 had [--since <dispatch-sha>] — square brackets, marking a flag OPTIONAL that the CLI exits 2 without. Fixing the charter alone would have shipped a PR claiming the invocation was corrected while two of three prescriptions still said otherwise.

Measured against the shipped CLI: --since <sha> --claimed "s.txt" → rc=0; bare --claimed "s.txt" → rc=2. Both fixed, plus a LANDMINES paragraph naming the shape — the holdout structurally cannot catch this, because it tests the system and all three copies are prose.

2. The holdout's own 33/0 was a false green

ops-adopt.sh appeared in the suite only as a string the SessionStart guidance must mention — never as a CLI it invokes. Measured: ops-adopt.sh replaced by a body of exit 0 and the suite reported 33 passed, 0 failed. The entire re-claim mechanism RECOVERY PROTOCOL step 6 depends on could be deleted with the holdout certifying the release.

RECOVERY PROTOCOL promises it, so it was closable from the charter alone. A fifth denied-context repair dispatch (no file tools, no shell), carrying the mutation as evidence and the explicit bar that a no-op must not pass: 41 checks, same mutation now 6 red. HOLDOUT_MIN_CHECKS raised 30 → 41 — a floor below the true count is slack a deletion hides in; verified a 40-check suite now exits 6.

3. The holdout committed the defect it exists to refuse

Three checks reported ok having measured nothing, all the same arithmetic:

$(( $(wc -c < absent) )) is 0, so a before/after byte comparison was 0 == 0
sed -n "1,p" on a missing file, twice, compared → equal. That was the guard on the ledger's append-only-ness, comparing nothing to nothing.
grep -c yields 0 for "correctly absent" and "file not there" alike

Measured with ops-init.sh mutated to create no ledger: 27 passed / 14 failed, and all three were among the passes — fourteen checks caught the condition while these three reported success about it. After repair: 24 / 17, each naming its precondition.

The shape worth carrying: a check whose PASS condition is 0, or an empty string, or an equality between two reads of the same absent file, cannot tell success from never-having-measured.

Two reviewer claims adjudicated WRONG, by mutation rather than argument

The same panel flagged defer_clears_sentinel and the auto-bar block as possibly vacuous. Both are real: --defer writing its row without clear_sentinel drives the first red alone (32/1), and an auto-bar that blocks and prints "autobar" while arming no sentinel drives the second red alone (32/1). Two of three suspicions wrong, one right — only mutation separates them.

(One reviewer also reported README/PLAYBOOK as already-correct; it had read my uncommitted working tree. Settled with git show fix/112-holdout:README.md.)

Mutation ledger, all restored byte-identical

mutation	result
Stop gate `exit 2` → `exit 0`	10 failed (was 4 pre-fix — the guards sharpened it)
`+dirty` branch deleted	2 failed
`ops-claims.sh` first-path-only (`LIMIT 1`)	1 failed
verdict word hardcoded `PASS`	1 failed
`--defer` without `clear_sentinel`	1 failed
auto-bar blocks without arming	1 failed
`ops-adopt.sh` → `exit 0`	6 failed (0 before the review)
installer creates no ledger	17 failed (14 before the vacuity fix)
control	41 passed, 0 failed

Portability — asked mid-review, answered in the artifact

The forge is not the mechanism. Property 1 is "outside the builder's read scope"; lokaal is one way to buy it. run-holdout.sh contains no Forgejo — verified against a plain GitHub URL: HOLDOUT_VERIFIED sha=7057dcf7f2f6 checks=41. The holdout repo now ships PORTING.md pricing the alternatives: a second GitHub repo gets the same property for free; an orphan branch and a local directory get weaker versions, and it says exactly what each gives up. The one thing that does not port is putting the holdout inside the repo it tests, behind a denial.

Forge runs 1458 and 1459 both green at 41 checks. Gates unchanged: validator all contracts hold · shell 1101@1101 · python 397@397 · workflows 384@384 · compress 161@161 · release gate v0.11.16 OK · shellcheck clean modulo the one pre-existing SC2329.

…un them Four probe rounds against TypeSafe/jev-1.13.0 produced decisions that were expensive to reach and cheap to lose: the reasoning lived in issue comments, which no future session can address or search, and the raw data lived in a session scratchpad that does not survive. docs/DECISION-ENGINE-PROBES.md carries the measured record of five surfaces — cap detector, autonomous evidence gating, reviewer identity, model selection, task-nature routing, pre-dispatch packet triage — each with the number that decided it. The durable part is the filter, not the numbers: who authors the input, and do they benefit from the answer? A probabilistic judgment may gate a SPEND (recoverable) and never a GATE, because a sentinel, a ledger row and a Stop decision are facts about what happened while a judgment is an opinion about operator-authored text. Two rejections are worth not re-litigating: a fabricated 'GATE_OK: shell — 9999 cases' scores 0.91 and fabrication is undetectable at 0.08 separation (inside the +/-0.07 run-to-run drift); and 'Just a small tweak, should be quick' demotes a judgment task to MECHANICAL at conf 0.65 — the attack shaped like ordinary prose succeeds where the blatant one fails. docs/dev/decision-engine-probes/ preserves 29 request/response files including the hashed ground truth, so the claims are re-runnable rather than trusted. Scanned clean of credentials. Labels reproduce from .operator/VERDICTS.md at 5605f96. Gates: validator all contracts hold, shell 1101@1101, python 397@397, workflows 384@384, compress 161@161. CLAUDE.md 37,821/38,000 chars.

…sured nothing (#149 #148) Both found while reviewing the holdout's work in PR #146, and both the same shape: v0.11.16 fixed the instance and left the class unmechanized. #149 — check_prose_invocations. The charter, README and PLAYBOOK each prescribed an ops-claims.sh invocation the CLI refuses; correcting all three by hand left nothing that would catch the fourth. The check extracts every ops-*.sh --flag prescription from tracked prose and asserts the CLI would accept it, reading both flag sets off the CLI's own parser and usage: forms — a table in the validator would be the second copy the check exists to catch. Mandatory is per form. `[--since]` in prose counts as ABSENT: the brackets say optional for a flag the CLI exits 2 without. Verified against the real defect three ways (each copy reverted, each red at its own file:line, restored byte-identical) plus ten mutations of the check, each named with the case it drove red. Four defects in the check itself were found by RUNNING it on the correct tree before any mutation — each would have condemned a correct line, and a pin that only ever ran against its own mutation would have shipped all four. #148 — eleven assertions proved "nothing was written" by comparing two reads of one file. Absent, both reads are "" and [ "" = "" ] is true, so each certified a refusal about a ledger that was never there. ops-init.sh mutated to write no ledger: 991/110 with eight of the eleven among the PASSES; a second mutation (no DECISIONS.md) covers two more, and the eleventh passes there honestly. Fixed with four helpers that assert the precondition, compare with -eq, and NAME the missing file on stderr. Each carries both controls — refuse-absent and accept-present — since a helper that refused everything would pass a refusal-only control set and fail every real call site. Gates: validator all contracts hold · shell 1112@1112 · python 411@411 · workflows 384@384 · compress 161@161 · release gate v0.11.17 OK · shellcheck 0.11.0 clean at warning level (one pre-existing SC2329 in an untouched file).

…epeated #148 one level down Five reviewers, six findings in this release's own work. Each was REPRODUCED before it was believed, and each now carries a mutation that drives its case red. Four were in check_prose_invocations itself — none in the hard part (reading a shell parser), all in the suppression logic and scan boundaries, which is where a guard is least examined because they are what makes it quiet. Three were the check performing its own defect class: - It CONDEMNED CORRECT PROSE. re.finditer yields non-overlapping matches, so the citation tail swallowed the next invocation whole: `Run ops-verdict.sh and ops-claims.sh --since <sha> --claimed "<paths>"` reported ops-verdict.sh for flags it never took, while ops-claims.sh went unexamined. Bounding the tail at the next CLI name fixed both halves and exposed a second: a span of bare filenames has each entry reading as the next one's argument, so the flagless arm fired on the install-set list. - The negative-control exemption SUPPRESSED ITS NEIGHBOURS. Keyed on the paragraph, it exempted every invocation in it — a genuinely broken ops-claims.sh line appended to REPLAY-CHARTER.md's --ownr paragraph was reported by nothing. An exemption must attach to the thing it excuses. - delta_is committed #148's OWN defect one level down: ${2:-0} substituted 0 for an empty count, so 0-0 -eq 0 passed on a file that EXISTS. Worse than the absent-file case, because nothing looks wrong. The fourth is the one to carry forward. THE ROOT GLOBS WERE UNPINNED and the floor could not catch them: narrowing _roots to ["*.md", "docs/**/*.md"] stops reading templates/OPERATOR.md — the file #149's defect shipped in — with every case and the real tree green. _MIN counts invocations, not which files produced them. A count is not a selection. Also fixed: a FLAGLESS prescription was invisible (the check keyed on a flag being present to decide something had been prescribed, so `ops-adopt.sh <task-id>`, which exits 2, read as nothing to check). Also pinned, each confirmed unpinned by mutation first: the no-readable-CLIs guard, the docs/dev exemption, a CLI with no parseable usage form, and the per-form selection arm (whose only coverage was the real-tree case). THE FIRST FIXTURE FOR THAT LAST ONE WAS WRONG and passed under both mutations it named — its two forms shared no flag, so the flags_in & form[0] guard suppressed the report either way. Gates: validator all contracts hold · shell 1116@1116 · python 423@423 · workflows 384@384 · compress 161@161 · release gate v0.11.17 OK · shellcheck clean at warning level.

Sixth reviewer, comment-accuracy lens. Three findings, all confirmed by re-measurement; the two numeric ones are the same defect in different files. THE #148 SITE COUNT SAID ELEVEN IN FOUR FILES. Re-running both mutation pairs pre-fix and post-fix against HEAD gives 9 + 3 + 1 = 13 ledger sites: VERDICTS.md skipped 991/110 pre, 997/119 post -> 9 newly red DECISIONS.md skipped 1085/16 pre, 1097/19 post -> 3 newly red G1.3 passes both, HONESTLY (G1.2's >> creates the file) Plus a fourteenth site that takes the same guard without belonging to the count: the sessionstart steady-state check byte-compares an installed CLI, not a ledger, so neither mutation moves it. It was never vacuous. "Eleven" was the tally from the FIRST substitution batch. Three more sites were converted afterwards and nothing re-derived the number — then it was copied into floors.env, LANDMINES.md, CHANGELOG.md and the suite's own comments. A count written beside the code is a second copy of the code and drifts exactly like any other copy. The raw pass/fail totals in the prose were all correct; only the breakdown was wrong, which is what made it survive. check_prose_invocations' own comment had the same shape: "21 flagged invocations" against a measured 33 (REPLAY-CHARTER 10, README 6, tiers.md 5, PLAYBOOK 4, OPERATOR 4, handoff 1, HANDOUT 1, UNKNOWNS 1). Never gated by anything — _MIN is 15, below both. THIRD FINDING, and the reviewer's suggested fix was wrong: CLAUDE.md cited `ProseInvocationTest` in bare backticks, outside check_coupling_case_refs' _"..."_ scan. Wrapping the CLASS name as suggested FAILS the build — a carrier is a `def test_` line, not a class, which is why all five sibling classes use plain backticks too. Citing a method name alongside brings the row under the guard: measured, resolves. Gates: validator all contracts hold · shell 1116@1116 · python 423@423 · workflows 384@384 · compress 161@161 · release gate v0.11.17 OK · shellcheck clean at warning level.

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate validator findings, plus the inconsistent recorded shell-test floor, remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2

Open (3)

Exclude exempt prescriptions from the shrinking-scan floor count · New
Correct stale rationale for the 112-holdout pair · New
Align recorded test total with the executable floor · New

What changed in this PR

This pull request adds v0.11.17 safeguards against prose/CLI drift and vacuous shell assertions, while documenting holdout and decision-engine probe results.

Changes:

Adds validator coverage for CLI invocations in tracked prose.
Replaces vacuous assertions with fail-closed checks and raises suite floors.
Updates documentation, release metadata, and probe artifacts.

File	Summary and final review notes
`tests/test-scripts.sh`	Adds fail-closed assertion helpers and regression controls.
`tests/test_validate_plugin.py`	Adds prose invocation validation tests.
`tests/floors.env`	Raises suite floors. Nit (2 votes): recorded `1112 passed` conflicts with executable floor `1116`.
`templates/OPERATOR.md`	Documents mandatory `--since`.
`scripts/validate_plugin.py`	Implements prose/CLI contract checking. Critical (1 vote): exempt lines can satisfy the scan floor. Moderate (1 vote): the exemption can suppress unreadable-parser findings.
`README.md`	Updates CLI examples.
`docs/PLAYBOOK.md`	Corrects the claims procedure.
`docs/LANDMINES.md`	Records holdout and assertion lessons.
`docs/dev/decision-engine-probes/rows.json`	Stores probe rows.
`docs/dev/decision-engine-probes/routeadv.json`	Stores routing adversarial input.
`docs/dev/decision-engine-probes/routeadv-resp.json`	Stores routing adversarial output.
`docs/dev/decision-engine-probes/route.json`	Stores routing input.
`docs/dev/decision-engine-probes/route-resp.json`	Stores routing output.
`docs/dev/decision-engine-probes/rev.json`	Stores reviewer-attribution input.
`docs/dev/decision-engine-probes/rev-resp.json`	Stores reviewer-attribution output.
`docs/dev/decision-engine-probes/resp2.json`	Stores repeated cap-probe output.
`docs/dev/decision-engine-probes/resp.json`	Stores cap-probe output.
`docs/dev/decision-engine-probes/req.json`	Stores cap-probe input.
`docs/dev/decision-engine-probes/pktadv.json`	Stores packet-triage adversarial input.
`docs/dev/decision-engine-probes/pktadv-resp.json`	Stores packet-triage adversarial output.
`docs/dev/decision-engine-probes/pkt.json`	Stores packet-triage input.
`docs/dev/decision-engine-probes/pkt-resp.json`	Stores packet-triage output.
`docs/dev/decision-engine-probes/inj.json`	Stores injection-probe input.
`docs/dev/decision-engine-probes/inj-resp.json`	Stores injection-probe output.
`docs/dev/decision-engine-probes/groundtruth.json`	Stores hand-labelled probe pairs. Nits: rationales are mismatched at lines 376/418 (2 votes), 390 (1), 404 (1), and 432 (1).
`docs/dev/decision-engine-probes/fab.json`	Stores evidence-fabrication input.
`docs/dev/decision-engine-probes/fab-resp.json`	Stores evidence-fabrication output.
`docs/dev/decision-engine-probes/ev_resp3.json`	Stores evidence response batch.
`docs/dev/decision-engine-probes/ev_resp2.json`	Stores evidence response batch.
`docs/dev/decision-engine-probes/ev_resp1.json`	Stores evidence response batch.
`docs/dev/decision-engine-probes/ev_resp0.json`	Stores evidence response batch.
`docs/dev/decision-engine-probes/ev_ans.json`	Consolidates evidence scores.
`docs/dev/decision-engine-probes/ctl.json`	Stores control-probe input.
`docs/dev/decision-engine-probes/ctl-resp.json`	Stores control-probe output.
`docs/dev/decision-engine-probes/adv.json`	Stores adversarial evidence input.
`docs/dev/decision-engine-probes/adv-resp.json`	Stores adversarial evidence output.
`docs/DECISION-ENGINE-PROBES.md`	Documents probe methodology and conclusions. Nit (1 vote): committed token totals do not reproduce the stated headline measurement.
`CLAUDE.md`	Adds coupling rules and holdout guidance.
`CHANGELOG.md`	Adds v0.11.17 release notes.
`.claude-plugin/plugin.json`	Bumps the plugin version to 0.11.17.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+                        f"), so an operator following this prose verbatim gets "
+                        f"a usage error and exit 2 (#149)")
+                    continue
+                seen += 1


+   "criterion": "BAR criteria 1-6: an independent holdout exists, is derived by a denied-context agent, asserts exact values, fails closed, goes red on real mutations, and this repo's gates are unchanged"
+  },
+  "label": 0,
+  "why": "pr132 audit vs v0.11.15 release",


+#   bash tests/test-scripts.sh -> 1112 passed, 0 failed, 0 skipped = 1112
+FLOOR_shell=1116


This was referenced Sep 19, 2026

Every check in this repo is inside the builder's read scope — there is no holdout #112

Open

The holdout is dispatched by hand — nothing in the release path demands its marker #147

Open

betmoar added 2 commits September 19, 2026 11:39

betmoar added 2 commits September 19, 2026 13:33

betmoar changed the title ~~0.11.16: a holdout outside the builder's read scope, and the charter defect it found (#112)~~ 0.11.17: a holdout outside the builder's read scope, the charter defect it found, and the two classes behind it (#112 #149 #148) Sep 19, 2026

betmoar added 2 commits September 19, 2026 17:25

betmoar requested a lite review from Copilot September 21, 2026 09:36

Copilot started reviewing on behalf of betmoar September 21, 2026 09:36 View session

Copilot AI reviewed Sep 21, 2026

View reviewed changes

Navigation Menu

0.11.17: a holdout outside the builder's read scope, the charter defect it found, and the two classes behind it (#112 #149 #148) - #146

0.11.17: a holdout outside the builder's read scope, the charter defect it found, and the two classes behind it (#112 #149 #148)#146
betmoar wants to merge 7 commits into
mainfrom
fix/112-holdout

betmoar commented Sep 19, 2026 •

edited

Loading

Uh oh!

betmoar commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		# bash tests/test-scripts.sh -> 1112 passed, 0 failed, 0 skipped = 1112
		FLOOR_shell=1116

Conversation

betmoar commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is

The four properties, and how each was met

What it found on its first run

Measurements

Process note worth recording

Asymmetry to know about

0.11.17 — the two classes the holdout's findings belonged to

Not in scope

Uh oh!

betmoar commented Sep 19, 2026

Review round: 4 panels, 3 real findings in my own change

1. The charter fix was one copy of three (F30, in the fix for the defect the holdout found)

2. The holdout's own 33/0 was a false green

3. The holdout committed the defect it exists to refuse

Two reviewer claims adjudicated WRONG, by mutation rather than argument

Mutation ledger, all restored byte-identical

Portability — asked mid-review, answered in the artifact

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

betmoar commented Sep 19, 2026 •

edited

Loading