Skip to content
This repository was archived by the owner on Oct 27, 2026. It is now read-only.

Add deterministic tests scope with once-per-contract Jev review - #16

Closed
brianorwhatever wants to merge 3 commits into
mainfrom
feat/deterministic-tests-gate
Closed

brianorwhatever wants to merge 3 commits into
mainfrom
feat/deterministic-tests-gate

Conversation

@brianorwhatever

Copy link
Copy Markdown
Contributor

Adds scope: tests, a REV gate that decides each push in code: no API key, no model call, no network.

  • Task v2: each criterion lists tests (file > suite > name) and paths (globs; catch-alls rejected).
  • Per push (code): each named test must pass on head and not pass when the head's test files run against the merge base; every changed file must match a glob; required CI must succeed.
  • Per contract (Jev): scripts/review-contract.js asks whether the criteria cover the request and whether each criterion's tests would establish it. It reads the exact test blobs and pins REV_CONTRACT_PR_<n> only on approval.
  • Digest: covers the task, request, required checks and the named test file blob hashes, so editing a test after review invalidates the contract.
  • scripts/junit.js: normalizes Bun and Node JUnit into identical test IDs. scripts/tests-change.js: collects the v2 change manifest from Git objects.

Existing pull-request and acceptance scopes are unchanged. npm run check and npm test pass (87 tests, 5 new).

🤖 Generated with Claude Code

Each criterion names tests and path globs. The gate checks in code that
every named test passes on head and not on the merge base, that every
changed file is mapped, and that required CI succeeded. Jev reviews the
contract once per digest, which binds the named test file blobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
brianorwhatever and others added 2 commits October 26, 2026 22:57
A combined question left a trivially correct contract below threshold.
Separate single-focus questions approve it at 0.93-1.0 and still reject a
weakened test (inadequate, p=1.0) and a broad glob (broad, p=0.94).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each tests question now carries the file's setup code and the named
test's block instead of every test file, falling back to the whole file
when the block cannot be located.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brianorwhatever

Copy link
Copy Markdown
Contributor Author

Closing: REV is being scrapped. The deterministic gate worked, but Jev's contract review could not approve real contracts (p 0.5–0.85) and varied by ±0.05 on identical input. Per owner decision, onionoriginals/sdk#1007 now removes REV from the SDK entirely.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant