fixtures: a corpus with ground truth, and the defect it found - #9
Merged
Merged
Conversation
Every claim this repository makes about the mastery model was checkable only by reading the code. test/fixtures/ makes them checkable by running it: eight synthetic learners, each built to exhibit one property, each carrying an expect block stating what is true of it by construction. They are also the substrate the pipeline needs. docs/OPENCLAW.md already says all pipeline work uses synthetic fixtures in test/fixtures/, and pointed at a directory that did not exist. Nothing that examines the record, the scheduler, or a generated activity could be measured against anything until it did. test/fixtures.test.ts holds the corpus to its own claims. That file is the point of the exercise. A persona that quietly stops exhibiting the property it is named for does not fail loudly, it makes everything measured against it report success against nothing. Two properties worth naming, because both were assumptions until they were arithmetic. Eight correct answers across widening gaps buy a half-life of about sixty-five days, so a well-spaced skill survives a summer away and only lapses around a hundred and fifty. The same eight answers crammed into six consecutive days lapse in sixty. lapsed-summer carries both, in one child, so the spacing effect is visible as a difference rather than asserted as a claim. vowel-confuser is deliberately unlabelled. It carries a real misconception in the response column, bed read as bid and pen as pin, with short i mastered so that not knowing short i is ruled out, and no misconception row at all. A detector scored against a record that already names the answer has found nothing. The corpus found something on the first run. nextTargets tested opportunities >= 6 && p < 0.5 before it tested lapsed, and six opportunities is an ordinary number of tries to learn something, so a child who mastered a skill in spring and took the summer off came back reported as stuck. The tutor was told to change modality or drop to a prerequisite for a child who needed a two minute review of something she used to know. peak_p_known is the only thing that can separate lost it from never had it, statusOf already reads it, and the stuck branch now consults it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The assertion read `validateInterface.length === 1`, meaning to fail the day the validator learned who an activity was for. It would not have. Giving a function a second parameter with a default leaves Function.length at 1, so the check would have kept passing after the capability landed and would have quietly asserted nothing. That is precisely the silent rot this corpus exists to prevent, in the corpus. A persona that stops exhibiting its property fails loudly; a pin that stops pinning does not. So the gap is now pinned by behavior: a page setting 9px text is accepted today, and nothing reports which accommodations went unchecked. Both statements become false the moment the constraint is enforced, and the test says what to assert instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
test/fixtures/, whichdocs/OPENCLAW.mdalready requires and which did not exist. Eight synthetic learners, each built to exhibit one property of the mastery model, each carrying anexpectblock stating what is true of it by construction.Independent of #7. Touches nothing under
world/.Why
Every claim about the model was checkable only by reading the code. Nothing that examines the record, the scheduler, or a generated activity could be measured against anything.
test/fixtures.test.tsholds the corpus to its own claims. A persona that quietly stops exhibiting the property it is named for does not fail loudly. It makes everything measured against it report success against nothing.The personas
cold-startcold-start-youngstuck-blendinglapsed-summercrammervowel-confuseraccommodatedabove-bandTwo properties are now arithmetic rather than assumption. Eight correct answers across widening gaps buy a half-life near 65 days, so a well-spaced skill survives a summer and only lapses around 150. The same eight crammed into six consecutive days lapse in 60.
lapsed-summercarries both in one child, so the spacing effect is visible as a difference rather than asserted.vowel-confusercarries a real misconception in the response column with nomisconceptionrow. A detector scored against a record that already names the answer has found nothing.The defect it found
nextTargetstestedopportunities >= 6 && p < 0.5before it tested lapsed. Six opportunities is an ordinary number of tries to learn something, so a child who mastered a skill in spring and took the summer off came back reported as stuck: the tutor was told to change modality or drop to a prerequisite for a child who needed a two-minute review of something she used to know.peak_p_knownis the only thing that separates "lost it" from "never had it".statusOfalready reads it. The stuck branch now consults it.stuck-blendingstill reports stuck, which is the case the branch is for.Checks
npm testpasses, 193 tests. CI cannot run: the upstream account is billing-locked and every workflow is refused before it starts.🤖 Generated with Claude Code