Routed up from Preponderous-Software/acsf-dev-loop#48, which was labelled template-rule and is therefore not implemented in that instance.
The pattern
An acceptance criterion sometimes depends on behaviour only a real model produces — divergence, variety, tone drift, refusal rates. A deterministic mock cannot exercise it. Two things can then happen, and only one of them is honest:
- A deterministic test is written that looks like it satisfies the criterion. It asserts a property of the mock, goes green, and hides the fact that the criterion was never checked.
- The causal lever that produces the behaviour is tested instead (that the prior outputs and the diverging instruction do reach the prompt), and the unautomatable part of the criterion is disclosed in the PR body and the self-review, with a stated way to validate it (a manual smoke run, or a run against a live provider).
Observed instance
Recorded on the ACSF loop while closing its issue #118. The criterion asked for an opening-overlap threshold (bigram Jaccard < 0.4 across five reflections); MockProvider output is deterministic and cannot demonstrate divergence at all. The lever was tested, the threshold was disclosed as unautomatable and deferred to a manual smoke run, and that was recorded as the approach worth retaining.
Suggested rule
For Phase 3's test-writing rules and the Phase 4 rubric: when an acceptance criterion depends on stochastic model behaviour a mock cannot exercise, a deterministic stand-in test must not be fabricated for it. Test the mechanism that produces the behaviour, and disclose the remaining criterion plus how it should be validated. An all-green suite that conceals an unchecked criterion is worse than an acknowledged gap — and the disclosure is what lets a reviewer decide whether the gap matters.
This issue was filed during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson
Routed up from
Preponderous-Software/acsf-dev-loop#48, which was labelledtemplate-ruleand is therefore not implemented in that instance.The pattern
An acceptance criterion sometimes depends on behaviour only a real model produces — divergence, variety, tone drift, refusal rates. A deterministic mock cannot exercise it. Two things can then happen, and only one of them is honest:
Observed instance
Recorded on the ACSF loop while closing its issue #118. The criterion asked for an opening-overlap threshold (bigram Jaccard < 0.4 across five reflections);
MockProvideroutput is deterministic and cannot demonstrate divergence at all. The lever was tested, the threshold was disclosed as unautomatable and deferred to a manual smoke run, and that was recorded as the approach worth retaining.Suggested rule
For Phase 3's test-writing rules and the Phase 4 rubric: when an acceptance criterion depends on stochastic model behaviour a mock cannot exercise, a deterministic stand-in test must not be fabricated for it. Test the mechanism that produces the behaviour, and disclose the remaining criterion plus how it should be validated. An all-green suite that conceals an unchecked criterion is worse than an acknowledged gap — and the disclosure is what lets a reviewer decide whether the gap matters.
This issue was filed during a Gardener session (https://github.com/Stephenson-Software/gardener).
drafted by Claude on behalf of Daniel Stephenson