Gap found during a Phase 9 self-audit
What was observed
Phase 4's Tests-fix rubric item requires the revert-and-run experiment, and its fallback ladder is gated by an explicit entry condition at create-dev-loop.md:372:
Enter this ladder only when the tool/interpreter itself is unavailable or broken (the UNVERIFIED case above), so that no local technique can execute.
That condition covers one way the experiment can fail to produce evidence: the anchor cannot run. It does not cover a second, distinct way: the anchor runs perfectly and reaches a verdict, but its scope excludes the surface the fix changed, so it returns the same result on both sides of the revert.
The two are not the same condition and do not have the same remedy. When the anchor cannot run, nothing was measured and the ladder offers substitutes. When the anchor is scope-blind, something was measured — it just was not the thing the fix changed — and no rung helps, because every rung is a different way of obtaining the same scope-blind signal.
Phase 4 already recognizes scope-blindness elsewhere, in the paragraph beginning "Green CI is not verification when CI's scope excludes the changed files", which instructs the reader to state the gap and lean on adversarial hand-review. The Tests-fix item and its ladder are simply not connected to it, so a reader following the rubric top-to-bottom arrives at the base experiment, observes an identical PASS on both sides, and has no instruction other than rung 3's "score FAIL and hand to a human".
Why it matters
Scoring FAIL and handing off is the right outcome here, so the loop does not produce a wrong answer. What it produces is an unexplained one: the hand-off reads as "the regression test is a false negative — fix the test", when the accurate statement is "no test in this repo can reach this surface, and none can be written". Those have entirely different remedies, and only the second tells the maintainer that the condition is permanent for this class of change rather than a defect in this PR.
The distinction matters most on exactly the repos where the loop runs headlessly against a documentation product — this one included, where every fix to template prose lands in this state. It is the Tests-fix analogue of the structurally-red-anchor condition Phase 1 already records at triage.
Suggested change
Give the base Tests-fix experiment a scope check before the ladder is reached: if the revert is confirmed to have changed the tree and the anchor still returns an identical verdict, distinguish
- a false-negative test — the anchor covers this surface but the assertion is too weak, which is repairable with sentinel data, from
- a scope-blind anchor — no check in this repo reaches the changed surface, which is not repairable and should be named as such.
For the second, the ladder should be skipped rather than entered (no rung obtains a different signal), Tests-fix scored FAIL with the reason stated as scope-blindness, and the hand-off in Phase 8 required to name the condition — the same treatment Phase 8 already gives a structurally-red anchor, whose hand-off must "name the blocking condition" instead of reading like a queue.
Related
Distinct from #109, which concerns the /create-dev-loop behavioral validation run being unrunnable under a headless dispatch. This issue concerns the automated anchor, which runs fine and is simply blind to the changed surface.
Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable, confidence: high). An anchor that returns the same verdict regardless of the change carries no information about it, so a Tests-fix score taken from it is unanchored judgment — the same argument the PR #121 entry under that finding makes for a build tool reporting success having executed zero tests. RESEARCH.md §3 is also relevant: regression evidence is what distinguishes a real fix from a coincidental patch, so a condition under which such evidence is unobtainable deserves to be named rather than reported as a missing test.
Provenance
Observed during the Phase 4 self-review of PR #138. The changed files were reverted to the merge base, the tree was confirmed to have actually changed, and both scripts/check_docs.py and the unit tests passed identically against the reverted tree — because neither can reach the semantics of template prose.
This issue was filed during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson
Gap found during a Phase 9 self-audit
What was observed
Phase 4's Tests-fix rubric item requires the revert-and-run experiment, and its fallback ladder is gated by an explicit entry condition at
create-dev-loop.md:372:That condition covers one way the experiment can fail to produce evidence: the anchor cannot run. It does not cover a second, distinct way: the anchor runs perfectly and reaches a verdict, but its scope excludes the surface the fix changed, so it returns the same result on both sides of the revert.
The two are not the same condition and do not have the same remedy. When the anchor cannot run, nothing was measured and the ladder offers substitutes. When the anchor is scope-blind, something was measured — it just was not the thing the fix changed — and no rung helps, because every rung is a different way of obtaining the same scope-blind signal.
Phase 4 already recognizes scope-blindness elsewhere, in the paragraph beginning "Green CI is not verification when CI's scope excludes the changed files", which instructs the reader to state the gap and lean on adversarial hand-review. The Tests-fix item and its ladder are simply not connected to it, so a reader following the rubric top-to-bottom arrives at the base experiment, observes an identical PASS on both sides, and has no instruction other than rung 3's "score FAIL and hand to a human".
Why it matters
Scoring FAIL and handing off is the right outcome here, so the loop does not produce a wrong answer. What it produces is an unexplained one: the hand-off reads as "the regression test is a false negative — fix the test", when the accurate statement is "no test in this repo can reach this surface, and none can be written". Those have entirely different remedies, and only the second tells the maintainer that the condition is permanent for this class of change rather than a defect in this PR.
The distinction matters most on exactly the repos where the loop runs headlessly against a documentation product — this one included, where every fix to template prose lands in this state. It is the Tests-fix analogue of the structurally-red-anchor condition Phase 1 already records at triage.
Suggested change
Give the base Tests-fix experiment a scope check before the ladder is reached: if the revert is confirmed to have changed the tree and the anchor still returns an identical verdict, distinguish
For the second, the ladder should be skipped rather than entered (no rung obtains a different signal), Tests-fix scored FAIL with the reason stated as scope-blindness, and the hand-off in Phase 8 required to name the condition — the same treatment Phase 8 already gives a structurally-red anchor, whose hand-off must "name the blocking condition" instead of reading like a queue.
Related
Distinct from #109, which concerns the
/create-dev-loopbehavioral validation run being unrunnable under a headless dispatch. This issue concerns the automated anchor, which runs fine and is simply blind to the changed surface.Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable, confidence: high). An anchor that returns the same verdict regardless of the change carries no information about it, so a Tests-fix score taken from it is unanchored judgment — the same argument the PR #121 entry under that finding makes for a build tool reporting success having executed zero tests. RESEARCH.md §3 is also relevant: regression evidence is what distinguishes a real fix from a coincidental patch, so a condition under which such evidence is unobtainable deserves to be named rather than reported as a missing test.
Provenance
Observed during the Phase 4 self-review of PR #138. The changed files were reverted to the merge base, the tree was confirmed to have actually changed, and both
scripts/check_docs.pyand the unit tests passed identically against the reverted tree — because neither can reach the semantics of template prose.This issue was filed during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson