Gap found during a Phase 9 self-audit
What was observed
Phase 4's self-review rubric contains a Docs item, at
create-dev-loop.md:367:
- Docs: every row in the Phase 7 documentation sources-of-truth table reflects the new behavior.
That item is scored, and the self-review comment containing it is posted, at
Phase 4 step 5. The check it names is not performed until Phase 7, three
phases later — after the Phase 5 review wait and after Phase 6 has addressed
comments. Phase 7 is where the sources-of-truth table actually lives and where
the instruction to "read the implementation first, then check each doc against
it" is given.
Two consequences follow, and both were hit during the cycle that produced
PR #136:
- The Docs rubric item is scored before its work is done. Scoring it
honestly at Phase 4 requires performing Phase 7 early; scoring it from
judgment is exactly the unanchored critique RESEARCH.md §1 warns against, and
the rubric's own framing ("grounded in the diff or a command output — not in
judgment") forbids it. Nothing in the template tells the reader which of
those to do.
- Phase 7's result has nowhere to go. Phase 7 says only "if any inaccuracy
is found: fix it, commit, and restart this phase from the top." It never asks
for the outcome to be recorded, and the one artifact that would naturally
carry it — the self-review comment — was published before Phase 7 ran. A
complete Phase 7 pass that finds nothing therefore leaves no trace at all, so
a later reader cannot distinguish "the sweep ran and was clean" from "the
sweep was skipped".
Phase 4 step 6's cap ("do not re-run the rubric internally after Phase 6") makes
this harder to work around after the fact: re-scoring Docs once Phase 7 has
actually run would violate that cap.
Suggested change
Two options, either of which resolves both consequences:
- Move the check to where it is scored. Have the Phase 4 Docs item state
that Phase 7's table pass must be performed now, at Phase 4, so the score is
grounded — and reduce Phase 7 to a re-check against whatever changed during
Phases 5–6, restarted from the top only if a comment altered the
implementation.
- Move the scoring to where the check happens. Drop the Docs item from
Phase 4's rubric, and have Phase 7 end by posting its own short result comment
(clean, or "these rows were corrected in <sha>"). This keeps each phase's
output adjacent to its work, at the cost of a second PR comment.
The first is preferred, since it keeps the self-review a single artifact. Either
way, Phase 7 should be given an explicit instruction to record that a complete
pass ran, so a clean sweep is distinguishable from a skipped one.
Provenance
Observed during the cycle that produced PR #136, where Phase 7 was performed
early and its per-row findings were folded into the Phase 4 self-review comment
by hand. That workaround is not described anywhere in the template and was
arrived at only because scoring the Docs item otherwise would have been
ungrounded.
Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable,
confidence: high) is the relevant finding: a rubric item scored before the check
it names has been run is a judgment wearing a rubric's clothes, which is the
failure mode that section's implementations have repeatedly corrected elsewhere
in Phase 4 (PRs #54, #121, #136).
This issue was filed during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson
Gap found during a Phase 9 self-audit
What was observed
Phase 4's self-review rubric contains a Docs item, at
create-dev-loop.md:367:That item is scored, and the self-review comment containing it is posted, at
Phase 4 step 5. The check it names is not performed until Phase 7, three
phases later — after the Phase 5 review wait and after Phase 6 has addressed
comments. Phase 7 is where the sources-of-truth table actually lives and where
the instruction to "read the implementation first, then check each doc against
it" is given.
Two consequences follow, and both were hit during the cycle that produced
PR #136:
honestly at Phase 4 requires performing Phase 7 early; scoring it from
judgment is exactly the unanchored critique RESEARCH.md §1 warns against, and
the rubric's own framing ("grounded in the diff or a command output — not in
judgment") forbids it. Nothing in the template tells the reader which of
those to do.
is found: fix it, commit, and restart this phase from the top." It never asks
for the outcome to be recorded, and the one artifact that would naturally
carry it — the self-review comment — was published before Phase 7 ran. A
complete Phase 7 pass that finds nothing therefore leaves no trace at all, so
a later reader cannot distinguish "the sweep ran and was clean" from "the
sweep was skipped".
Phase 4 step 6's cap ("do not re-run the rubric internally after Phase 6") makes
this harder to work around after the fact: re-scoring Docs once Phase 7 has
actually run would violate that cap.
Suggested change
Two options, either of which resolves both consequences:
that Phase 7's table pass must be performed now, at Phase 4, so the score is
grounded — and reduce Phase 7 to a re-check against whatever changed during
Phases 5–6, restarted from the top only if a comment altered the
implementation.
Phase 4's rubric, and have Phase 7 end by posting its own short result comment
(clean, or "these rows were corrected in
<sha>"). This keeps each phase'soutput adjacent to its work, at the cost of a second PR comment.
The first is preferred, since it keeps the self-review a single artifact. Either
way, Phase 7 should be given an explicit instruction to record that a complete
pass ran, so a clean sweep is distinguishable from a skipped one.
Provenance
Observed during the cycle that produced PR #136, where Phase 7 was performed
early and its per-row findings were folded into the Phase 4 self-review comment
by hand. That workaround is not described anywhere in the template and was
arrived at only because scoring the Docs item otherwise would have been
ungrounded.
Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable,
confidence: high) is the relevant finding: a rubric item scored before the check
it names has been run is a judgment wearing a rubric's clothes, which is the
failure mode that section's implementations have repeatedly corrected elsewhere
in Phase 4 (PRs #54, #121, #136).
This issue was filed during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson