Ground Phase 4's Docs score in a real pass and fix two Phase 3 rules - #138
Merged
Conversation
Phase 4's Docs rubric item named a check that Phase 7 does not perform until
three phases later, so the item could only ever be scored from judgment — the
thing the rubric's own framing forbids — and a clean Phase 7 sweep left no
trace that would distinguish it from a skipped one. The table pass now happens
at Phase 4, where the score is taken; Phase 7 becomes a re-check scoped to
whatever Phases 5-6 changed, with an explicit instruction to record that it
ran.
Phase 3's zero-tests rule ended by describing {{TEST_CMD}} as a substitution,
which is the one context a generated child skill never has: after substitution
it read "fall back to whatever the ./gradlew test substitution names". It now
names the executed-test count as the gate, which reads correctly on both sides
of substitution. A scan of the rest of the template body found no other
sentence referring to a token as a token.
Phase 3's scratch-file rule gains a third cleanup form observed to work where
a bare rm is blocked: git clean -f on a single untracked path inside the
working tree, previewed with git clean -n.
Closes #126
Closes #134
Closes #137
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Docs rubric item was an anchor-shaped score with no anchor behind it, because the check it named ran three phases later. That is the same defect as finding 1's PR #121 entry, relocated from the signal to the schedule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
Author
Phase 4 self-reviewPosted as a plain comment, not a formal review — no independent reviewer is implied. This review was performed inline by the same session that authored the PR. Universal rubric
Repo-specific rubric
Out-of-diff observationsFolded into this body rather than posted inline, since none of these lines is in a diff hunk.
SummaryTwo FAILs found. The Retrofit note was mechanical and has been fixed in the PR description. Tests-fix is a genuine FAIL that cannot be repaired within this repo's tooling, and it — together with the absent behavioral validation run — is why this PR is handed to a human rather than merged: a template-prose fix has no reachable regression evidence, and the merge gate should not be satisfied by an anchor that measured nothing. This review comment was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener). drafted by Claude on behalf of Daniel Stephenson |
This was referenced Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three corrections to the generated-skill template, all in
create-dev-loop.md.Phase 4's Docs rubric item was scored three phases before its check was performed. The item names the Phase 7 documentation sources-of-truth table, but Phase 7 does not run until after the Phase 5 review wait and the Phase 6 comment pass. Scoring it at Phase 4 could therefore only be done from judgment — precisely what the rubric's own framing ("grounded in the diff or a command output — not in judgment") forbids — and a clean Phase 7 sweep left no artifact at all, so a later reader could not tell a completed pass from a skipped one. The table pass is now performed at Phase 4, where the score is taken, and Phase 7 is reduced to a re-check scoped to whatever Phases 5–6 changed, with an explicit instruction to record that the pass ran. The option chosen is the first of the two the issue proposed, which keeps the self-review a single artifact. A sentence was added to Phase 7 distinguishing this re-check from the intrinsic-critique re-run that Phase 4 step 6 caps, since a review comment that changed the implementation is an external signal. Closes #137
Phase 3's zero-tests rule described
{{TEST_CMD}}as a substitution. After substitution a generated skill read "fall back to whatever the./gradlew testsubstitution names as the real gate for this repo" — the substitution mechanism is the one context a generated child never has. The sentence now names an executed-test count from{{TEST_CMD}}as the gate, which reads correctly both in the template and in every generated instance. The rest of the template body was scanned for the same class of defect (a sentence referring to a{{TOKEN}}as a token rather than using it as a value); this was the only occurrence. Closes #126Phase 3's scratch-file rule gains a third cleanup form.
git clean -f <path>on a single untracked path inside the working tree succeeds where a barerm <path>is blocked by the harness classifier, and is a more native fit than shelling out to Python whenever the scratch file already sits in a repo — which is the common case, since the canonical use is a body drafted withWriteand passed togh … --body-file. The preview step (git clean -n) and the single-path constraint are stated alongside it. Closes #134One deviation from an issue's suggested text is worth flagging. #134 asks that the new rule defer to "the rule elsewhere in the template — never
git reset --hard/git cleana path this run did not create". No such rule exists increate-dev-loop.md; the onlygit reset --hardin the template is Phase 1's stuck-mid-revert recovery, which is unrelated. Per the Phase 3 localization rule that the source wins over an issue's description, the constraint is stated directly and self-containedly instead of as a cross-reference to something that is not there.Backlog skipped this cycle
Recorded here so deferral is auditable rather than silent. #106 (stale gardener owner link) is confined to
CLAUDE.md, which is agent-loaded config and harness-blocked for autonomous editing — it needs explicit maintainer authorization and is surfaced rather than attempted. #109, #114, #120, #122, #123, #124, #125, #127, #128, #129, #130, #131, #133, and #135 were deferred as out of scope for a batch already coherent around three small instruction-text corrections; none is blocked, and each remains available for a later cycle.Retrofit note
All three changes are rules in the generated-skill template, so the template only fixes them forward — every
<slug>-dev-loopskill generated before this commit carries all three defects and will keep carrying them until its nextMODE=updateregeneration or self-audit cycle. Only the person running those skills can enumerate which exist, so the affected surfaces are named here rather than the skills:{{TEST_CMD}}as this repo's real gate rather than the count-free banner") to the instance skills, so some instances may already read closer to this PR's wording than the template did; the wording adopted here is deliberately close to that variant to converge the two rather than introduce a third.git clean -fform.Research grounding
RESEARCH.md §1 (Self-critique without an external signal is unreliable, confidence: high) grounds the #137 change. A rubric item scored before the check it names has been performed is unanchored judgment wearing a rubric's clothes, which is the failure mode this finding's implementations have repeatedly corrected elsewhere in Phase 4 (PRs #54, #121, #136). An Implementations entry recording this PR has been added under that finding.
No finding applies to #126 or #134. #126 is a wording correction with no empirical claim behind it, and #134 records an observed harness-classifier behavior rather than a research result — consistent with how the two earlier rules from the same root cause (#62, #85) shipped without a RESEARCH.md entry. RESEARCH.md has no sandbox-behavior finding, and this change does not warrant creating one.
Doc sync check
RESEARCH.mdupdated (an Implementations entry under §1 for the Phase 4's Docs rubric item is scored three phases before Phase 7 performs the check it names #137 change)The other two boxes are deleted as not applicable: no Step was added, removed, or renamed, and no
{{placeholder}}was added or changed.Test plan
python3 scripts/check_docs.py— "Doc consistency check passed."python3 -m unittest discover -s tests— Ran 13 tests, OKdoc-consistencygreen on the PR head/create-dev-loopbehavioral validation run — not performed; this change is UNVERIFIED behaviorallyThe two gaps are stated plainly rather than papered over.
Behavioral validation. All three changes are to the generated-skill template body, which nothing in CI executes:
check_docs.pyverifies placeholder/substitution-table parity, README Step parity, and relative links, and none of those cover the semantics of a phase instruction. A greendoc-consistencyjob is therefore not evidence that these instructions behave correctly. The validation run described inCLAUDE.md's "Testing changes" section was not performed because this cycle ran headlessly under a Gardener dispatch, where the side effects that run requires are not authorized: Step 6 creates a GitHub repository, and Steps 3 and 5 write outside the checkout to~/local-skills/and~/.claude/commands/. None of the five checklist items were exercised. In place of the run, each changed instruction was hand-reviewed as a generated skill would receive it after substitution — which is how the #126 defect was confirmed and how the corrected wording was checked on both sides of substitution.Regression evidence. The Phase 4 revert-and-run experiment was performed and is recorded in the self-review comment: the changed files were reverted to the merge base, the tree was confirmed to have actually changed, and both
check_docs.pyand the unit tests passed identically against the reverted tree. That is a real false-negative, not a scoring artifact — no automated check in this repo can reach the semantics of template prose. Neither fallback rung applies: rung 1 (CI-based temporary revert) would produce the same green on both sides and is in any case barred under a headless dispatch, and rung 2 requires a pre-existing test asserting the old behavior, of which there is none. The Tests-fix item is therefore scored FAIL, and this PR does not clear the Phase 8 regression gate autonomously.The residual risk is bounded: no placeholder, Step, phase number, or fence was touched, so the mechanical failure modes CI does cover are not in play, and the changes are confined to instruction prose whose effect is visible on reading.
Closes #126
Closes #134
Closes #137
This PR description was drafted during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson