Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions SCIENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -1060,6 +1060,22 @@ Scient's own tests live in `test/scient/` folders and run on macOS with the plat

The macOS native test process preloads `scripts/scient/macos-spelling-ready.ts`. AppKit can return no spans when its first XPC request times out during spelling-service startup; the preflight requires the service to identify a known misspelling within three probes before running the unchanged tests. Persistent unavailability fails qualification. This provisions the CI fixture; it does not retry failed tests, change runtime results, or qualify cold-service startup behavior.

### Flake quarantine

`KNOWN_FAILURES` is for tests that fail on untouched Oh My Pi. A flaky test that fails on Scient's tree, on content that also passed, can be quarantined instead. The rule is the one Scient Desktop's `Check` follows (owner decision, 2026-11-17).

- **Exact match.** A row names one test and a signature: text the failure's own output must contain, between the test before it and its `(fail)` line. The same test failing another way still fails the job, and so does everything `KNOWN_FAILURES` never excuses (a crash, an error outside any test, a report that does not add up).
- **Still visible.** The test still runs. Each quarantined failure is printed as a warning and listed in the job summary with its row.
- **Expiry.** Every row has an owner and expires 7 days after it was quarantined. From the next day (UTC) its failures fail the job again, unless the cause is fixed (the fix removes the row) or the owner extends it.
- **Releases.** A release run (`release: true`) takes no rows, so a release ships only from a run without a quarantined failure.
- **The rows** are in `scripts/scient/ci-quarantine.json`; each row's id and expiry must also be in the table below (a test checks this). The daily CI care task adds each occurrence (date, run link, running count) to its row.

| Row | Test | Signature | Owner | Quarantined → expires | Occurrences (running count) | Cause |
| --- | --- | --- | --- | --- | --- | --- |
| `browser-open-options-downloads` | `coding-agent` `browser-open-options.test.ts` › saves each tab's downloads into its own downloads directory, also after another tab closes | `.crdownload",` in the failed `readdir(second)` diff | Browser tool | 2026-11-18 → 2026-11-25 | 11-16 [main](https://github.com/ScientFactory/scient-agent/actions/runs/43954563060) (1); 11-17 [PR #70](https://github.com/ScientFactory/scient-agent/actions/runs/44078095729) (2) | Open. Upstream's test and code, unchanged by Scient. In 30 local runs Chrome's own download never left a `.crdownload` behind at completion, but Chrome also saved downloads that no page started (no `downloadWillBegin`) into the folder it was pointed at last, the second tab's here. Not yet reproduced in the test itself. |

The other two browser tests that failed `main` on 2026-11-16 need no row. "does not call a cancelled run's page listeners" (`browser-nav-frames-dialogs.test.ts`) is in `KNOWN_FAILURES` for 18.22.0. The relay default-tab test (`browser-relay-tab-isolation`) was fixed by #75 and has not failed since.

## Definition of done

Report work as done only when all of these hold on the final head. Without a pull request, the handback carries what the pull request would.
Expand Down
13 changes: 13 additions & 0 deletions scripts/scient/ci-quarantine.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"schemaVersion": 1,
"rows": [
{
"id": "browser-open-options-downloads",
"test": "packages/coding-agent/test/tools/browser-open-options.test.ts > browser open options > saves each tab's downloads into its own downloads directory, also after another tab closes",
"signature": ".crdownload\",",
"owner": "Browser tool",
"quarantined": "2026-11-18",
"expires": "2026-11-25"
}
]
}
3 changes: 2 additions & 1 deletion scripts/scient/divergences.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@
"scripts/scient/release-version.ts",
"scripts/scient/release-version.test.ts",
"scripts/scient/upstream-tests.ts",
"scripts/scient/upstream-tests.test.ts"
"scripts/scient/upstream-tests.test.ts",
"scripts/scient/ci-quarantine.json"
],
"upstream": ["scripts/host-detect.ts"],
"deleted": [
Expand Down
82 changes: 81 additions & 1 deletion scripts/scient/upstream-tests.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { describe, expect, it } from "bun:test";
import { judge } from "./upstream-tests";
import * as path from "node:path";
import { judge, type Quarantine, type QuarantineRow, rowIsValid } from "./upstream-tests";

// Shaped as `scripts/ci-test-ts.ts <mode> --full` prints a failed run, with
// Bun 1.4's own output inside each chunk.
Expand Down Expand Up @@ -44,6 +45,7 @@ describe("a failed bucket", () => {
it("passes when every failure is a known upstream failure", () => {
expect(judge(1, run([knownOnly]), KNOWN)).toEqual({
known: [`packages/coding-agent/${FILE} > ${KNOWN_NAME}`],
quarantined: [],
problems: [],
});
});
Expand Down Expand Up @@ -111,3 +113,81 @@ describe("a failed bucket", () => {
expect(judge(2, run([knownOnly]), KNOWN).problems).toEqual(["the runner exited 2, not with failed tests"]);
});
});

describe("the flake quarantine", () => {
const FLAKY = "saves each tab's downloads";
const row: QuarantineRow = {
id: "downloads",
test: `packages/coding-agent/${FILE} > ${FLAKY}`,
signature: '.crdownload",',
owner: "Browser tool",
quarantined: "2026-11-18",
expires: "2026-11-25",
};
const strayFile = ["error: expect(received).toEqual(expected)", "", "- []", "+ [", '+ "0a1d.crdownload",', "+ ]"];
const flaky = (error: string[]) =>
chunk(LABEL, 1, [
"",
`${FILE}:`,
"(pass) an earlier test [1.00ms]",
...error,
`(fail) ${FLAKY} [2199.58ms]`,
...summary(110, 1),
]);
const on = (today: string): Quarantine => ({ rows: [row], today });

it("lets a failure through when it shows the row's signature, and reports it", () => {
expect(judge(1, run([flaky(strayFile)]), new Set(), on("2026-11-25"))).toEqual({
known: [],
quarantined: [{ test: row.test, row: "downloads", owner: "Browser tool", expires: "2026-11-25" }],
problems: [],
});
});

it("blocks the same test failing another way", () => {
const timeout = ["error: Test timed out after 30000ms"];
expect(judge(1, run([flaky(timeout)]), new Set(), on("2026-11-20")).problems).toEqual([`failed: ${row.test}`]);
});

it("reads the signature only from the failing test's own output", () => {
const body = ["", `${FILE}:`, ...strayFile, "(pass) an earlier test [1.00ms]", `(fail) ${FLAKY} [2.00ms]`];
const judged = judge(1, run([chunk(LABEL, 1, [...body, ...summary(110, 1)])]), new Set(), on("2026-11-20"));
expect(judged.problems).toEqual([`failed: ${row.test}`]);
});

it("blocks again from the day after the row expires, and without rows", () => {
expect(judge(1, run([flaky(strayFile)]), new Set(), on("2026-11-26")).problems).toEqual([`failed: ${row.test}`]);
expect(judge(1, run([flaky(strayFile)]), new Set()).problems).toEqual([`failed: ${row.test}`]);
});

it("never matches a row whose dates are not canonical or span more than a week", () => {
for (const dates of [
{ quarantined: "2026-11-18", expires: "2026-11-2" },
{ quarantined: "2026-11-18", expires: "2026-11-31" },
{ quarantined: "2026-11-18", expires: "2026-11-26" },
{ quarantined: "2026-11-18", expires: "2026-11-17" },
]) {
const bad = { ...row, ...dates };
expect(rowIsValid(bad)).toBe(false);
const judged = judge(1, run([flaky(strayFile)]), new Set(), { rows: [bad], today: "2026-11-18" });
expect(judged.problems).toEqual([`failed: ${row.test}`]);
}
});

it("keeps every row exact, a week long and documented in SCIENT.md", async () => {
const root = path.resolve(import.meta.dir, "..", "..");
const register = await Bun.file(path.join(import.meta.dir, "ci-quarantine.json")).json();
const scientMd = await Bun.file(path.join(root, "SCIENT.md")).text();
const ids = new Set<string>();
for (const entry of register.rows as QuarantineRow[]) {
expect(ids.has(entry.id)).toBe(false);
ids.add(entry.id);
expect(entry.test).toMatch(/^packages\/[^ ]+\.test\.[cm]?[jt]sx? > .+ > .+/);
expect(entry.signature.trim().length).toBeGreaterThan(5);
expect(entry.owner.length).toBeGreaterThan(0);
expect(rowIsValid(entry)).toBe(true);
expect(scientMd).toContain(`| \`${entry.id}\``);
expect(scientMd).toContain(entry.expires);
}
});
});
119 changes: 110 additions & 9 deletions scripts/scient/upstream-tests.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@
* A failed bucket passes only when every failure is one `KNOWN_FAILURES` lists
* for the exact Oh My Pi commit `scient.json` names: a test that fails on that
* commit untouched, on the runner this workflow uses. The entries lapse at the
* next alignment, which checks them again. Anything else fails the run: an
* next alignment, which checks them again. A flaky test can also pass while an
* unexpired row of `ci-quarantine.json` names it and its failure shows the row's
* signature; a release run takes no quarantine. Anything else fails the run: an
* unlisted test, a chunk that crashed or was killed, an error outside any test,
* and a report this cannot reconcile in full.
*
Expand All @@ -18,6 +20,7 @@
*
* bun scripts/scient/upstream-tests.ts <mode>
*/
import * as fs from "node:fs/promises";
import * as path from "node:path";

/** A test that fails on untouched Oh My Pi at `commit`, with the evidence. */
Expand Down Expand Up @@ -53,13 +56,70 @@ const KNOWN_FAILURES: KnownFailure[] = [
},
];

/**
* A flaky test whose failure does not fail the run until `expires`, a week after `quarantined`. Only a failure whose
* own output contains `signature` matches, so the same test failing another way still blocks.
*/
export interface QuarantineRow {
id: string;
/** `<package>/<test file> > <describe> > <test>`. */
test: string;
signature: string;
owner: string;
/** `YYYY-MM-DD`, UTC. */
quarantined: string;
/** `YYYY-MM-DD`, UTC; the row blocks again from the next day. */
expires: string;
}

/** The rows in force and today's UTC date; a release run passes no rows. */
export interface Quarantine {
rows: readonly QuarantineRow[];
today: string;
}

/** A failure a quarantine row let through. */
export interface QuarantinedFailure {
test: string;
row: string;
owner: string;
expires: string;
}

/** Why a bucket's failure is or is not acceptable. */
export interface Judgement {
/** Known failures that occurred; the run passes when `problems` is empty. */
known: string[];
quarantined: QuarantinedFailure[];
problems: string[];
}

const NO_QUARANTINE: Quarantine = { rows: [], today: "" };

const DAY_MS = 86_400_000;

/** The UTC midnight a canonical `YYYY-MM-DD` names, or `undefined` for any other spelling. */
function utcDay(value: unknown): number | undefined {
if (typeof value !== "string" || !/^\d{4}-\d{2}-\d{2}$/.test(value)) return undefined;
const time = Date.parse(`${value}T00:00:00Z`);
return Number.isNaN(time) || new Date(time).toISOString().slice(0, 10) !== value ? undefined : time;
}

/** Whether a row can match at all: canonical dates, expiring no earlier than it started and within a week. */
export function rowIsValid(row: QuarantineRow): boolean {
const start = utcDay(row.quarantined);
const end = utcDay(row.expires);
return (
start !== undefined &&
end !== undefined &&
end >= start &&
end - start <= 7 * DAY_MS &&
typeof row.test === "string" &&
typeof row.signature === "string" &&
row.signature.trim().length > 0
);
}

const CHUNK_HEADER = /^==> \[\d+\/\d+\] (.+) \((?:ok|FAILED exit (\d+)), [\d.]+s\)$/;
const FAILED_FOOTER = /^(\d+) of \d+ test chunk\(s\) FAILED$/;

Expand All @@ -68,9 +128,15 @@ const FAILED_FOOTER = /^(\d+) of \d+ test chunk\(s\) FAILED$/;
* output: a `==> [i/n] <label> (FAILED exit N, …)` block per failed chunk with
* Bun's whole output, and a `N of M test chunk(s) FAILED` footer.
*/
export function judge(exitCode: number, output: string, known: ReadonlySet<string>): Judgement {
export function judge(
exitCode: number,
output: string,
known: ReadonlySet<string>,
quarantine: Quarantine = NO_QUARANTINE,
): Judgement {
const problems: string[] = [];
const seen: string[] = [];
const quarantined: QuarantinedFailure[] = [];
if (exitCode !== 1) problems.push(`the runner exited ${exitCode}, not with failed tests`);
const lines = output.split(/\r?\n/);
const failed = Number(lines.map(line => FAILED_FOOTER.exec(line)?.[1]).find(Boolean) ?? 0);
Expand All @@ -83,11 +149,12 @@ export function judge(exitCode: number, output: string, known: ReadonlySet<strin
chunks++;
let end = start + 1;
while (end < lines.length && !CHUNK_HEADER.test(lines[end]) && !FAILED_FOOTER.test(lines[end])) end++;
problems.push(...judgeChunk(header[1], Number(header[2]), lines.slice(start + 1, end), known, seen));
const chunkLines = lines.slice(start + 1, end);
problems.push(...judgeChunk(header[1], Number(header[2]), chunkLines, known, quarantine, seen, quarantined));
start = end - 1;
}
if (chunks !== failed) problems.push(`the runner says ${failed} chunks failed; ${chunks} were reported`);
return { known: seen, problems };
return { known: seen, quarantined, problems };
}

/** The problems with one failed chunk, given Bun's whole output for it. */
Expand All @@ -96,12 +163,16 @@ function judgeChunk(
exit: number,
lines: string[],
known: ReadonlySet<string>,
quarantine: Quarantine,
seen: string[],
quarantined: QuarantinedFailure[],
): string[] {
if (exit !== 1) return [`${label}: exited ${exit}, not with failed tests`];
const problems: string[] = [];
const pkg = label.split(" ")[0];
const failures: string[] = [];
const failures: Array<{ id: string; output: string }> = [];
// Bun prints a failure's error before its `(fail)` line; these are the lines since the last test or file.
let block: string[] = [];
let file: string | undefined;
let failCount: number | undefined;
let finished = false;
Expand All @@ -119,15 +190,32 @@ function judgeChunk(
const header = /^(\S.*\.test\.[cm]?[jt]sx?):$/.exec(line);
if (header) file = header[1].replaceAll("\\", "/");
const marker = /^\(fail\) (.+?)(?: \[[\d.]+m?s\])?$/.exec(line);
if (marker && !recap) failures.push(file ? `${pkg}/${file} > ${marker[1]}` : `${pkg}: ${marker[1]}`);
if (marker && !recap) {
const id = file ? `${pkg}/${file} > ${marker[1]}` : `${pkg}: ${marker[1]}`;
failures.push({ id, output: block.join("\n") });
}
if (header || marker || /^\((?:pass|skip|todo)\) /.test(line)) block = [];
else block.push(raw);
}
if (!finished || failCount === undefined) problems.push(`${label}: Bun did not finish its report`);
else if (failCount !== failures.length) {
problems.push(`${label}: Bun counted ${failCount} failures and named ${failures.length}`);
}
if (failures.length === 0) problems.push(`${label}: failed without naming a test`);
for (const id of failures) {
if (known.has(id)) seen.push(id);
for (const { id, output } of failures) {
if (known.has(id)) {
seen.push(id);
continue;
}
const row = quarantine.rows.find(
entry =>
rowIsValid(entry) &&
entry.test === id &&
output.includes(entry.signature) &&
utcDay(quarantine.today) !== undefined &&
quarantine.today <= entry.expires,
);
if (row) quarantined.push({ test: id, row: row.id, owner: row.owner, expires: row.expires });
else problems.push(`failed: ${id}`);
}
return problems;
Expand All @@ -154,8 +242,21 @@ if (import.meta.main) {
if (exitCode !== 0) {
const { upstream } = await Bun.file(path.join(repoRoot, "scient.json")).json();
const known = new Set(KNOWN_FAILURES.filter(entry => entry.commit === upstream.commit).map(entry => entry.test));
const { known: seen, problems } = judge(exitCode, output, known);
// A release ships only from a run without a quarantined failure.
const release = process.env.RELEASE === "true";
const register = await Bun.file(path.join(import.meta.dir, "ci-quarantine.json")).json();
const quarantine = { rows: release ? [] : register.rows, today: new Date().toISOString().slice(0, 10) };
const { known: seen, quarantined, problems } = judge(exitCode, output, known, quarantine);
for (const id of seen) console.log(`::warning::Fails on untouched Oh My Pi ${upstream.version} too: ${id}`);
const notes = quarantined.map(
entry => `Quarantined by \`${entry.row}\` (${entry.owner}, expires ${entry.expires}): ${entry.test}`,
);
for (const note of notes) console.log(`::warning::${note}`);
if (notes.length > 0 && process.env.GITHUB_STEP_SUMMARY) {
const lines = notes.map(note => `- ${note}`).join("\n");
await fs.appendFile(process.env.GITHUB_STEP_SUMMARY, `### Quarantined failures (${mode})\n\n${lines}\n`);
}
if (release && problems.length > 0) console.log("::notice::A release run takes no quarantine rows.");
for (const problem of problems) console.log(`::error::${problem}`);
process.exitCode = problems.length > 0 ? 1 : 0;
}
Expand Down
Loading