chore: merge test/node-conformance into tests/, and gate the partition - #60
Merged
Conversation
The repository had two top-level test trees whose names differed by one
character. `test/node-conformance/` held 14 `node --test` files run by
`bun run test:node` against the built dist/; `tests/conformance/xcut/` held
the Bun-run cross-package suites added by Phase 9. Both trees are necessary
and so is the split between them. Only the names were the problem, and no
file recorded that they differed at all.
Moves the Node tree to `tests/node-conformance/`, so one top-level tree holds
everything crossing a process, a network, or a runtime boundary, one
subdirectory per runner. Styleguide 11-testing scopes that rule to process and
network boundaries; reading a runtime boundary the same way is this repo's
extension, and it is now stated as one rather than cited as if the styleguide
said it.
What the move costs, and what pays for it:
Before, the file system held the separation — the Node tree sat outside
anything `bun test` could reach, so nothing could drift. Now one path written
into five files holds it, and every way that breaks is silent. Bun does not
error on a `node:test` import: it collects those files, runs them, and reports
them PASSING while proving nothing about Node. Bun accepts an unrecognized
`[test]` key with no warning, so `testPathIgnorePatterns` reads as configured
and does nothing. `node --test` over a glob matching nothing exits 0.
Measured on `bun run test`, pinned Bun 1.3.14:
with pathIgnorePatterns → 164 files, exit 0
without it → 178 files, exit 1
The 14 extra files are the Node suite collected by a runner that cannot prove
anything about Node. That run goes red only by accident — 13 of the 14 pass
silently and the 14th trips an unrelated timer assertion pointing nowhere near
the cause. The exit code is not a control.
So `scripts/verify-test-partition.mjs` reads the five files that must agree
(bunfig.toml, package.json, eslint.config.js, ci-preflight/run-ci.mjs, and the
tree's README) and blocks CI on disagreement. It runs neither suite. Seven
checks: the issue's five, plus two guarding rules CLAUDE.md states and nothing
enforced — the root script must name both trees whole, and `[test] root` must
stay "packages".
Adversarial review found six ways the first draft passed while the partition
was broken; each is now a check with a test:
- Bun applies pathIgnorePatterns while WALKING, so a pattern naming a
directory prunes the subtree without matching any file path. Adding
`tests/conformance/fixtures` silently dropped a real test file and every
full-path check stayed green. Directory prefixes now count.
- The comment added above eslint.config.js's `files:` entry quoted the glob
it documents, so deleting the entry left the check satisfied by the
sentence explaining the guarantee.
- A case named `retry.mjs` or `orphan.test.ts` was ignored by Bun, unmatched
by test:node, and exempted by the check's own extension filter.
- `**` compiled to a bare `.*`, matching paths Bun does not ignore — a gate
green-lighting a config Bun reads differently.
- A glob narrowed to one of 14 cases satisfied "matches at least one file".
- `statSync` per entry threw a bare ENOENT stack trace on a broken symlink.
Closes open-items H13 — `bun run test:scripts` now runs in CI. Its trigger had
already fired: `knowledge.test.mjs` had been failing on main since 36c3f96,
whose Phase 10 correction to docs/knowledge/deliberate-deviations.md cites
CFG-1 and so gave a previously ID-less topic its first requirement ID, moving
the `--list-topics` count from 16 to 15. A gate whose own logic degrades still
exits 0, so nothing else in the run would have noticed. The same count is
quoted in CLAUDE.md and knowledge-lookup/SKILL.md; all three now move together
and the assertion's failure message names the other two. Three neighbouring
counts in that paragraph had rotted the same way (592 KB not 518, 255 not 256,
386 not 385, and a fourth bucket of 4 cited nowhere that went unmentioned) and
are now pinned by a second canary against `--coverage`.
Records what the audit found and this change does not fix: the 80% coverage
floor measures only the Bun run, and `test:node` contributes nothing to it
(open-items H20, RECORDED with a trigger).
Deliberately unchanged: docs/superpowers/plans/ and specs/, the validation
prompts, and .changeset/*.md keep their `test/node-conformance/` paths — they
are dated records and were correct when written. The node-conformance CI job
calls `bun run test:node`, not a path, so its steps are untouched.
tests/tsconfig.json gains a comment only; it sets no allowJs, so tsc still
never opens the .mjs subtree, and that subtree remains the one testing the
shipped artifact with the fewest static checks over it. CI running it on two
Node versions is the compensating control. No changeset — nothing here is
visible to a consumer.
Verified: node .claude/skills/ci-preflight/run-ci.mjs --clean --node-floor,
all 18 steps. `bun test ./tests` collects 7 files, not 21, on 1.3.14 and 1.4.0.
`bun run test:node` runs 137 tests across all 14 files on Node 26 and on the
20.3.0 floor. test:scripts is 81 cases. Commenting out the gate's
`process.exit(1)` fails a test; before the review it passed all 64.
Wahbeh-Mohammad
deleted the
55-merge-testnode-conformance-into-tests-and-gate-the-partition-that-keeps-the-two-suites-apart
branch
August 31, 2026 19:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #55.
What changed
This repository has two top-level test trees. Their names differed by one
character.
test/node-conformance/held 14node --testfiles.bun run test:noderuns these files against the builtdist/.tests/conformance/xcut/holds Bun test files.
bun run testruns these files. Both trees are necessary.The separation between them is also necessary. Only the names were a problem. No
file recorded that the two names differed.
This change moves the Node test tree to
tests/node-conformance/. One top-leveltree now holds every test that crosses a process, a network, or a runtime
boundary. Each subdirectory holds one test runner.
Before this change, the file system kept the two trees apart. The Node test tree
was outside the reach of every Bun command. After this change, one path in five
files keeps the two trees apart. These five files must agree:
bunfig.toml[test] pathIgnorePatternskeeps the Node test tree out ofbun testpackage.jsontest:nodeglob is the only command that runs the Node test treeeslint.config.js.mjsoverride supplies the Node globals.claude/skills/ci-preflight/run-ci.mjs--node-floorpathtests/node-conformance/README.mdEach failure of this arrangement is silent:
node:testimport. Bun collects thesefiles. Bun runs these files. Bun reports these files as passed. The report
shows no evidence about Node.
[test]key. Bun shows no warning. The keytestPathIgnorePatternslooks correct. Bun does not read this key.node --testexits with code 0 when a glob matches no file.Measured with Bun 1.3.14, with
bun run test:pathIgnorePatternspresentpathIgnorePatternsabsentThe 14 additional files are the Node test tree. Bun runs these files and proves
nothing about Node. The exit code 1 is an accident. Thirteen of the 14 files
pass on Bun. The fourteenth file fails on an unrelated timer assertion. That
failure does not identify the cause. Do not use the exit code as a control.
This change adds
scripts/verify-test-partition.mjs. This script reads the fivefiles. This script runs no test suite. This script blocks CI when the five files
disagree. The script makes seven checks. Five checks come from issue #55. Two
more checks guard rules that
CLAUDE.mdstates and that no tool enforced. Theroot test script must name both trees. The
[test] rootkey must stay"packages".Other changes:
.github/workflows/ci.ymlgets two blocking steps. One step runs the newgate. One step runs
bun run test:scripts.docs/open-items.mdH13.bun run test:scriptsnowruns in CI.
scripts/knowledge.test.mjsfailed on
mainfrom commit36c3f96. That commit added a correction todocs/knowledge/deliberate-deviations.md. The correction citesCFG-1. Atopic file with no requirement ID received its first requirement ID. The
--list-topicscount moved from 16 to 15.CLAUDE.mdand.claude/skills/knowledge-lookup/SKILL.mdquote the samecount. Both documents were also incorrect. All three values now move together.
The assertion message names the two documents.
and not 518 KB. The roll-up count is 255 and not 256. The substantive count is
386 and not 385. A fourth group of 4 uncited IDs was absent from the text. A
second assertion now pins these values against
--coverage.docs/open-items.mdH20 records a gap that this change does not close. The80% coverage floor measures the Bun run only.
bun run test:nodeadds nothingto this floor.
These items stay unchanged, by decision:
docs/superpowers/plans/,docs/superpowers/specs/,docs/validation-prompts/and
.changeset/*.mdkeep the old paths. These files are dated records. Thepaths were correct on the date of each record.
node-conformanceCI job callsbun run test:node. This job does notname a path. The steps of this job are unchanged.
tests/tsconfig.jsongets a comment only. The compiler options are unchanged.This file sets no
allowJs. Thereforetscdoes not read the.mjssubtree.That subtree tests the shipped artifact and receives the fewest static checks.
CI runs that subtree on two Node versions. This is the compensating control.
Verification:
node .claude/skills/ci-preflight/run-ci.mjs --clean --node-floorpasses all18 steps.
bun test ./testscollects 7 files. Before the change it collected 21 files.This result is the same on Bun 1.3.14 and on Bun 1.4.0.
bun run test:noderuns 137 tests across all 14 files. This result is thesame on Node 26 and on the Node 20.3.0 floor.
bun run test:scriptsruns 81 tests.Reviews done
Three review passes ran in parallel against one frozen snapshot. Each pass ran
as a separate agent. Each pass received the same snapshot and its own
instructions. No pass received the results of the other passes. No pass could
edit a file. Each pass used the
knowledge-lookupskill to read the applicablerules and cited rule IDs in its findings.
Pass 1 approved the shape. Therefore the findings of pass 2 and pass 3 are
applicable. Pass 1 also confirmed three earlier decisions. The gate belongs in
scripts/and not in a skill directory. The newstatictier inrun-ci.mjsis correct. The H13 work belongs in this change. Without the H13 work, CI never
runs the 21 tests of the new gate.
Pass 3 reported one blocker. No test started the gate as a program. A person
could delete
process.exit(1), and all 64 tests still passed. The test filescripts/verify-seam-1.test.mjsstates this risk in its own header. This changeadds a test harness. The harness copies the gate into a temporary tree and runs
the gate as a program. The harness reads the exit code and the output.
Pass 2 reported six conditions where the first version of the gate passed and
the partition was broken. Each condition now has a check and a test:
retry.mjsororphan.test.tstests/**[test] rootis removedThe first condition is the most important. Bun applies
pathIgnorePatternsduring the directory walk. A pattern that names a directory removes the whole
subtree. That pattern matches no file path. The first version of the gate tested
file paths only.
Four more defects from pass 2 are also corrected. The
**operator now spanswhole path segments, as Bun does. The TOML reader now respects quoted strings.
The array reader now stops at the next key. The directory reader now uses
withFileTypesand does not fail on a broken symbolic link.Two defects appeared during the correction work. The new tests found both. A
general comment remover deleted the correct
files:entry, because the stringpackages/*/scripts/*.mjscontains the comment delimiters. A substring testaccepted
./tests/conformance, which is the exact narrowing that the ruleforbids.
Pass 1 and pass 3 disagreed about one fact. Pass 1 stated that the test names
follow the pattern of both sibling files. Pass 3 stated that the test names
deviate from
testing.md:14and fromscripts/verify-seam-1.test.mjs. Pass 3is correct.
verify-seam-1.test.mjsuses the required form. Onlyverify-sse-37.test.mjsdoes not. All test names now use the required form.Pass 3 also found incorrect statements in the documents. The
### HARD RULEheading contained the rest of the
## Commandssection, including the gatelist. The gate list claimed to be complete and omitted two blocking CI steps.
The README of the Node test tree listed 6 of 14 files. This change removes that
table. Nothing regenerated the table, and the phase checklists already record
the same information.
Pass 2 disproved one statement in this change. The comment in
bunfig.tomlstated that both configurations exit with code 0. That statement is incorrect.
The corrected measurement is in the table above.
Evidence for the blocker correction: a person removes
process.exit(1)from thegate, and one test fails. Before this review, the same removal caused no test to
fail.