Repository navigation
ci(scient): quarantine exactly matched flaky Oh My Pi test failures - #76
Merged
Merged
Conversation
A row of scripts/scient/ci-quarantine.json names one test and a signature from the failure's own output; a matching failure passes the bucket with a warning and a job-summary line until the row expires, 7 days after it was quarantined. A release run takes no rows. KNOWN_FAILURES is unchanged. First row: browser-open-options downloads (a stray .crdownload in the second tab's folder), seen on main 11-16 and PR #70 on 11-17. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row's dates must be canonical UTC YYYY-MM-DD, expire no earlier than the quarantine and within 7 days of it; otherwise the row never matches. String comparison of "2026-11-2" kept an expired row in force (Codex review). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Scient Desktop's flake-quarantine rule (owner decision, 2026-11-17), applied to Oh My Pi's test buckets.
scripts/scient/ci-quarantine.jsonholds the rows. Each names one test, a signature (text the failure's own output must contain), an owner and an expiry 7 days after quarantine.upstream-tests.ts's judge lets a failure through only when an unexpired row names its test and the signature appears in that test's own output: the lines between the previous test and its(fail)line. The same test failing another way still fails the job. So does everythingKNOWN_FAILURESnever excuses: a crash, an error outside any test, or counts that don't add up.::warning::and listed in the job summary with its row.RELEASE=true) takes no rows, so a release ships only from a run without a quarantined failure.KNOWN_FAILURES(exact-upstream failures) is unchanged.SCIENT.md› Checks › "Flake quarantine" documents the rule and the register table. A test checks that every row is unique, at most a week long, and in that table.First row:
browser-open-options-downloads. "saves each tab's downloads into its own downloads directory, also after another tab closes" fails whenreaddir(second)finds a stray<guid>.crdownload.The other two 11-16 browser tests named for this register need no row:
KNOWN_FAILURESfor 18.22.0;Why
Since #69,
mainrequiresCI passed, so a flaky upstream test now blocks an unrelated PR until someone reruns it. This test failed on main 11-16 and on docs-only PR #70, both with the same stray.crdownload.The cause is open. It is upstream's test and code (
downloads.ts), unchanged in Scient. Evidence so far:.crdownloadat completion.Browser.downloadWillBegin) into the folder it was pointed at last, the second tab's in this test.Testing
bun test scripts/scient/upstream-tests.test.ts: 13 pass. New cases:bun check: passes.Platforms
upstream-tests, ubuntu-22.04). The change applies there.Review
Review: Light Codex (gpt-6.1-sol, high) — 1 finding (malformed expiry dates compared as strings could keep an expired row in force), fixed in 023bddb with a regression test; recheck clean.
bun checkpasses🤖 Generated with Claude Code