Skip to content
This repository was archived by the owner on Oct 27, 2026. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ Version 0.3 defaults to `scope: pull-request`. It requires the original request,

Use the [PR coverage integration guide](docs/pr-coverage.md) for the complete collector, contract and final-job setup. Existing full-SHA pins remain unchanged. Updating an integration to 0.3 requires migration; do not switch to acceptance-only mode to bypass coverage failures.

## Deterministic tests gate

`scope: tests` needs no API key and makes no model call per push. Each criterion names tests and path globs. Code checks that every named test passes on the PR head and does not pass on the merge base, that every changed file is mapped, and that CI succeeded. Jev reviews the contract—request coverage and whether the named tests establish each criterion—once per digest, which binds the exact test file contents. See the [tests gate guide](docs/tests-gate.md).

## Acceptance-only example

**1. The task.** Write observable requirements before implementation. Store them in a reviewed task manifest:
Expand Down
14 changes: 10 additions & 4 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ branding:
color: purple
inputs:
scope:
description: pull-request requires reviewed request and full change coverage. acceptance explicitly checks only supplied criteria.
description: pull-request requires reviewed request and full change coverage. acceptance explicitly checks only supplied criteria. tests is deterministic, needs no API key, and checks named tests against head and merge-base reports.
default: pull-request
change-file:
description: Complete Git change manifest with original request, required CI jobs and per-file criterion mappings.
description: Complete Git change manifest with original request, required CI jobs and per-file criterion mappings. In tests scope, the version 2 manifest from scripts/tests-change.js.
required: false
base-sha:
description: Expected PR base SHA. Required in pull-request scope.
Expand All @@ -24,8 +24,14 @@ inputs:
description: JSON task and acceptance criteria manifest (version 1).
required: true
evidence-file:
description: JSON evidence manifest with headSha and named source items (version 1).
required: true
description: JSON evidence manifest with headSha and named source items (version 1). Required outside tests scope.
required: false
head-report:
description: Tests scope. REV test report (scripts/junit.js) of head tests run on head sources.
required: false
base-report:
description: Tests scope. REV test report of the head's test files run on merge-base sources.
required: false
head-sha:
description: Exact PR head commit expected in the evidence manifest. Not the synthetic merge SHA.
required: true
Expand Down
64 changes: 64 additions & 0 deletions docs/tests-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Deterministic tests gate

`scope: tests` decides each push in code: no API key, no model call, no network. Identical inputs produce an identical report. Jev reviews the contract once per digest, before the gate runs; it never runs per push.

| Question | Decided by | When |
| --- | --- | --- |
| Do the criteria cover the original request? | Jev (`scripts/review-contract.js`) | Once per contract digest |
| Would the named tests, if they pass, establish each criterion? Are path globs scoped? | Jev | Once per contract digest |
| Does each named test pass on head and not pass on the merge base? | Code | Every push |
| Does every changed file match a criterion's path glob? | Code | Every push |
| Did every required CI job succeed? Do head, base and contract match? | Code | Every push |

## Task (version 2)

```json
{
"version": 2,
"task": "Protect the export endpoint.",
"criteria": [
{
"id": "anonymous",
"description": "An anonymous GET /export returns HTTP 401 and no export data.",
"tests": ["test/export.test.ts > export > rejects anonymous requests"],
"paths": ["src/routes/export.ts", "test/export.test.ts"]
}
]
}
```

Test IDs are `repository-relative file > suite > … > test name`, exactly as written by `scripts/junit.js`. Globs support `**`, `*` and `?`; catch-all patterns such as `**` are rejected.

## Contract

`contract.json` holds `request` (`text`, `source`) and `requiredChecks` (CI job IDs). The digest covers the task, request, required checks and the Git blob hash of every file containing a named test. Editing a criterion, glob, required check or named test file changes the digest and requires another review, so tests cannot be weakened after review without detection.

```sh
node rev/scripts/tests-change.js BASE_SHA HEAD_SHA .rev/contract.json .rev/task.json change.json
TYPESAFE_API_KEY=… node rev/scripts/review-contract.js .rev/task.json change.json --pin OWNER/REPO PR_NUMBER
```

`review-contract.js` reads named test sources from Git blobs, asks Jev three separate questions (request coverage; whether each criterion's named tests assert all of its obligations; whether its path globs are scoped to it) and sets `REV_CONTRACT_PR_<n>` only if every answer is positive and meets the probability (0.9) and confidence (0.8) thresholds. It is operator-invoked; PR content never pins its own digest. Without `--pin` it prints the result only.

## Test reports

Run the named test files twice and convert each JUnit file:

1. **Head:** head sources, head tests. `junit.js CWD REPO HEAD_SHA HEAD_SHA head.json head.xml`
2. **Base:** merge-base sources with the head's named test files copied in. `junit.js CWD REPO MERGE_BASE_SHA HEAD_SHA base.json base.xml`

A test is accepted only if it passed on head and did not pass (failed, skipped or missing, for example because the file could not load) on base. Duplicate IDs take the worst status. Bun (`--reporter=junit`) and Node (`--test-reporter=junit`) produce identical IDs.

## Verdicts

| Criterion verdict | Reason |
| --- | --- |
| satisfied | Every named test passes on head and not on base. |
| violated | A named test fails on head. |
| unverified | A named test is missing or skipped on head (`test-not-run`), or passes on base (`passes-without-change`). |

The gate approves only when all required jobs succeed, every changed file is mapped, and every criterion is satisfied. Validation errors (`stale-change`, `stale-test-report`, `contract-mismatch`, `missing-reviewed-contract`, malformed input) deny.

## Trust boundary

Reports and the change manifest are produced by an unprivileged job that runs PR code, as in the other scopes. The gate job reads artifacts only. A malicious PR could forge a report; this gate checks agent work for mistakes, not for an adversarial author. A passing gate establishes that reviewed tests pin the change; it does not establish merge readiness.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
},
"scripts": {
"test": "node --test",
"check": "node --check src/index.js && node --check src/gate.js && node --check src/corpus.js && node --check src/io.js && node --check src/report.js && node --check src/guidance.js && node --check src/errors.js && node --check src/policy.js && node --check eval/run.js && node --check src/coverage.js && node --check scripts/change.js && node --check eval/coverage-fixtures.js && node --check src/focused.js && node --check eval/compare.js && node --check eval/real-pr-fixtures.js && node --check src/diagnose.js && node --check eval/diagnose-real.js && node --check eval/diagnostic-fixtures.js && node --check eval/diagnose-fixtures.js && node --check scripts/github-review.js && node --check scripts/local-review.js && node --check scripts/local-review-worker.js && node --check scripts/install-local-review.js && node --check scripts/local-review-gate.js && node --check src/local/protocol.js && node --check src/local/github.js && node --check src/local/lock.js",
"check": "node --check src/index.js && node --check src/gate.js && node --check src/corpus.js && node --check src/io.js && node --check src/report.js && node --check src/guidance.js && node --check src/errors.js && node --check src/policy.js && node --check eval/run.js && node --check src/coverage.js && node --check scripts/change.js && node --check eval/coverage-fixtures.js && node --check src/focused.js && node --check eval/compare.js && node --check eval/real-pr-fixtures.js && node --check src/diagnose.js && node --check eval/diagnose-real.js && node --check eval/diagnostic-fixtures.js && node --check eval/diagnose-fixtures.js && node --check scripts/github-review.js && node --check scripts/local-review.js && node --check scripts/local-review-worker.js && node --check scripts/install-local-review.js && node --check scripts/local-review-gate.js && node --check src/local/protocol.js && node --check src/local/github.js && node --check src/local/lock.js && node --check src/tests.js && node --check scripts/junit.js && node --check scripts/tests-change.js && node --check scripts/review-contract.js",
"eval": "node eval/run.js",
"eval:compare": "node eval/compare.js"
}
Expand Down
66 changes: 66 additions & 0 deletions scripts/junit.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import { readFile, realpath, writeFile } from 'node:fs/promises';
import { isAbsolute, relative, resolve, sep } from 'node:path';
import { pathToFileURL } from 'node:url';

// Normalize Bun and Node JUnit XML into REV test reports. IDs are "file > suite > ... > name".
const unescape = s => s.replace(/&(#x[0-9a-f]+|#\d+|amp|lt|gt|quot|apos);/gi, (_, e) => e[0] === '#'
? String.fromCodePoint(e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : Number(e.slice(1)))
: { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'" }[e.toLowerCase()]);
const attributes = s => Object.fromEntries([...s.matchAll(/([\w:-]+)="([^"]*)"/g)].map(([, k, v]) => [k, unescape(v)]));
const RANK = { passed: 0, skipped: 1, failed: 2 };

export function parseJUnit(xml, cwd, repo = cwd) {
// Attribute values may contain an unescaped ">", so match attributes explicitly.
const tags = /<(\/?)(testsuite|testcase|failure|error|skipped)((?:\s+[\w:-]+="[^"]*")*)\s*(\/?)>/g;
const body = xml.replace(/<!\[CDATA\[[\s\S]*?\]\]>/g, '').replace(/<!--[\s\S]*?-->/g, '');
const suites = [], tests = [];
let current;
const file = path => {
const absolute = isAbsolute(path) ? path : resolve(cwd, path);
const id = relative(repo, absolute).split(sep).join('/');
if (!id || id.startsWith('../')) throw new Error('test-file-outside-repository');
return id;
};
for (const [, close, tag, attrs, selfClosing] of body.matchAll(tags)) {
if (tag === 'testsuite') {
if (close) suites.pop();
else if (!selfClosing) suites.push(attributes(attrs));
} else if (tag === 'testcase') {
if (close) { tests.push(current); current = undefined; continue; }
const a = attributes(attrs);
const path = a.file || [...suites].reverse().find(s => s.file)?.file;
if (!path || a.name === undefined) throw new Error('testcase-without-file');
// Bun wraps each file in a suite named after the file; it is not part of the test name.
const names = suites.filter(s => !(s.file && s.name === s.file)).map(s => s.name);
current = { id: [file(path), ...names, a.name].join(' > '), status: 'passed' };
if (selfClosing) { tests.push(current); current = undefined; }
} else if (current && !close) {
current.status = tag === 'skipped' ? (current.status === 'failed' ? 'failed' : 'skipped') : 'failed';
}
}
return tests;
}

export function testReport(xmls, cwd, repo, sourceSha, testsSha) {
const merged = new Map();
// Duplicate IDs are ambiguous; the worst status wins so a duplicate cannot hide a failure.
for (const t of xmls.flatMap(xml => parseJUnit(xml, cwd, repo))) {
if (!merged.has(t.id) || RANK[t.status] > RANK[merged.get(t.id)]) merged.set(t.id, t.status);
}
return { version: 1, sourceSha, testsSha,
tests: [...merged].sort(([a], [b]) => a < b ? -1 : 1).map(([id, status]) => ({ id, status })) };
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
try {
// CWD is where the test runner ran; REPO is the repository root that test IDs are relative to.
const [cwd, repo, sourceSha, testsSha, output, ...inputs] = process.argv.slice(2);
if (!inputs.length || ![sourceSha, testsSha].every(x => /^[a-f0-9]{40}$/.test(x))) throw new Error('usage');
const report = testReport(await Promise.all(inputs.map(f => readFile(f, 'utf8'))), await realpath(cwd), await realpath(repo), sourceSha, testsSha);
await writeFile(output, JSON.stringify(report, null, 2) + '\n');
console.log(`${report.tests.length} tests: ${report.tests.filter(t => t.status === 'passed').length} passed`);
} catch (error) {
console.error(`Usage: junit.js CWD REPO SOURCE_SHA TESTS_SHA OUTPUT JUNIT_XML... (${error.message})`);
process.exitCode = 1;
}
}
Loading
Loading