Skip to content

ci(repo): run the release gate on main only, with readable, bounded shards - #82

Merged
kiro-systemf[bot] merged 6 commits into
mainfrom
xs/release-gate-runner-heap
Oct 9, 2026
Merged

ci(repo): run the release gate on main only, with readable, bounded shards#82
kiro-systemf[bot] merged 6 commits into
mainfrom
xs/release-gate-runner-heap

Conversation

@systemfsoftware-maker

@systemfsoftware-maker systemfsoftware-maker commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Why

Mutation testing runs on main only, never on a pull request. #78 added a pull_request trigger scoped to each change, and this PR takes it out again. It also keeps the fixes that make main's gate readable and stable. Shards on the 4 vCPU, 16 GB ubuntu-latest runner were ending in exit 3 or a lost VM, and the shard wrapper hid the child's error: it ignores the child's stdout (main.mjs:113463), keeps 4,096 characters of its stderr (113472) and prints the first 1,024 of those (114204-114206).

Change

Removed: everything the pull-request gate needed.

  • The pull_request trigger and its cancel-in-progress.
  • The plan job's changed-files step and fetch-depth: 2.
  • The scope output, MUTATION_SCOPE and stryker.shared.ts's scopedMutate, so every enrolled file is mutated.
  • The pull-request-only incremental-cache skip.
  • stryker-plan-gate.ts's change scoping (GATE_FILES, scopeOf, resolveScope, readChanged, the out-of-scope report, the MalformedStrykerConfig refusal), its --changed and --scope-out flags, and their tests.
  • The text in README and CONTRIBUTING that described pull-request mutation.

The gate runs on push to main and on workflow_dispatch. Apart from the shard selection mode, stryker-plan-gate.ts and its tests are back to their pre-#78 form.

Kept: readable, bounded shards.

  • The gate runs the shard's children itself. stryker-plan-gate.ts shard decodes the plan with the gate's ShardPlan schema and finds the shard by its index/count label, as the wrapper does (main.mjs:112975-112980, 113345-113362). It refuses an unknown label and writes <index>\t<project> lines in plan order. .github/scripts/run-shard.sh then runs each project from its own directory with the wrapper's exact args (113415-113429) and incremental seeding (113431-113437). It writes stdout.log and stderr.log beside the progress stream under reports/shards/<index>/<project>/, which the shard artifact already uploads.
  • It settles each child as the wrapper does. Exit 0 carries on. Exit 1 logs the wrapper's below-threshold line and carries on (113385-113391, 113478). Any other exit prints the tails of both logs and stops the shard with that exit code. A child that leaves no progress stream prints both tails and stops with 3, the wrapper's RuntimeError code (113479, 83447). The wrapper runs projects in sequence and stops at the first failure (113474-113483, concurrency: 1). It exits 3 for any aborted child, while this step passes the child's own code through.
  • The shard step's failure output. The diagnostics re-run and its artifact are gone, because the main step now keeps both logs.
  • fileLogLevel is removed. This build accepts it, but nothing writes stryker.log (65208, 88970, 113889, 113985).
  • concurrency: 2 gives 1 checker and 1 test runner (104850-104853).
  • testRunnerNodeArgs: ['--max-old-space-size=2048'] caps each test-runner isolate (106954-106960).
  • Unchanged: the verdict job, stryker merge and stryker gate, apart from the no-shards message, which no longer mentions a change's scope.

Sandbox proof. sandbox-proofs/release-gate.test.ts used to find the mutation step by stryker run, which the direct-run change removed. It now finds that step by run-shard.sh. It runs the shard selection step and the shard step in the sandbox with a stub stryker, and checks that the guard's variables reach stryker, that ALLOW_LOCAL_MUTATION never does, and that the progress stream lands at <plan dir>/<SHARD_OUT_MARKER>/<index>/<project>/, where the shard artifact uploads it.

Smoke test with a stub stryker in a scratch tree (two projects, nothing committed):

  • exit 0: both projects ran; a's incremental file was seeded;
  • exit 1 on a: the line was logged, b still ran and the step exited 0;
  • exit 3 on a: the step exited 3 and printed a's stderr ("Initial test run failed. 1 of 2 test(s) failed: flaky") and stdout (the exact child argv); b did not run;
  • no stream on b: the step exited 3 with both tails.

shard --shard 1/1 on run 37970427513's real plan prints 1\tpackages/xstate.

Gate ownership

This edits a judgment surface. The conductor, who owns the release gate, asked for these changes in cycles 247 to 264, and the main-only direction is Ryan's. No threshold or mutant timeout changes. The mutated set grows back to every enrolled file.

Gates

  • dprint check, actionlint, tsc -p tsconfig.node.json --noEmit, oxlint, deno check and deno lint all exit 0.
  • stryker-plan-gate.test.ts passes 11 of 11.
  • sandbox-proofs/release-gate.test.ts passes 5 of 5.
  • git grep finds none of MUTATION_SCOPE, scopedMutate, scopeOf, resolveScope, readChanged, changed-files, GATE_FILES, renderOutOfScope or MalformedStrykerConfig outside repos/.
  • No changeset: no publishable package changes.
  • No local mutation testing of any kind. The proof is main's next gate run after merge, which must pass the verdict.

@systemfsoftware-maker
ci(repo): cap each mutation test runner isolate at a 2 GiB heap
c381345 
Two of three release-gate runs on #81 lost the whole ubuntu-latest VM
(16 GB) mid-shard. Node's default old-space ceiling is about 4 GiB per
isolate, and each test-runner child holds a main isolate plus a claimed
and a spare vitest worker thread, so two runners running a mutant that
allocates without bound can outgrow the VM before the 45 s mutant
timeout settles them.

testRunnerNodeArgs passes --max-old-space-size=2048 to every test-runner
child. The flag is process-wide in V8, so each worker thread gets the
same ceiling. A worker that reaches it is ended by Node; Stryker retries
the mutant twice and then records it as a RuntimeError, so the run
carries on. The full xstate suite passes on one thread with a 512 MiB
ceiling, so 2 GiB leaves room for the coverage instrumentation.
@systemfsoftware-maker systemfsoftware-maker changed the title xs/release gate runner heap ci(repo): cap each mutation test runner isolate at a 2 GiB heap Oct 9, 2026
@systemfsoftware-maker
systemfsoftware-maker added this pull request to stack #83 October 9, 2026 18:01
@systemfsoftware-maker
ci(repo): print a failed shard's logs before its diagnostics re-run
81c0115 
On #81 the shard ends with exit 3 about 26 s into mutation testing, and
the job log keeps only the first 1024 characters of the child's stderr,
which are startup lines. The diagnostics re-run then ran for 16 minutes
until the VM was lost, so its stderr.log never reached an artifact.

A new step prints the last 32 KiB of each package's stryker.log as soon
as the shard fails, before anything else runs. The re-run is bounded by
timeout 600 (kill after 30 s more) and --concurrency 1, which gives one
checker and one test runner, and prints the last 32 KiB of its stderr,
stdout and stryker.log into the job log before the upload step.
@systemfsoftware-maker
ci(repo): run mutation shards with one checker and one test runner
930de0a 
At Stryker's default concurrency on the 4 vCPU hosted runner, a shard
starts 2 checkers, each driving a native tsgo process, and 2 test
runners. Run 37975117762 on #82 (main's code) and every gate run on #81
ended with exit 3 about two minutes in, or lost the 16 GB VM. The
bounded diagnostics re-run of the same project at concurrency 1 passed
its dry run and tested 276 mutants in five minutes without a failure.

concurrency: 2 splits into one checker and one test runner. The 2 GiB
test-runner heap cap stays as a second guard.
@systemfsoftware-maker systemfsoftware-maker changed the title ci(repo): cap each mutation test runner isolate at a 2 GiB heap ci(repo): run mutation shards with one checker and one test runner Oct 9, 2026
@systemfsoftware-maker
ci(repo): print the failed shard's progress streams instead of stryke…
3e1f34f 
…r.log

Run 37978503748 printed "(no such file)" for packages/*/stryker.log.
This Stryker build accepts fileLogLevel but nothing writes the file:
the option appears only in the schema, the CLI table and the
fingerprint key list. The setting and every stryker.log path go.

The print step now tails the shard wrapper's own progress stream
(reports/mutation-stream.jsonl, the default for a run without
--progressStreamFile) and each project child's stream under
reports/shards. Every run writes its framed events to that file, one
synced line at a time. The diagnostics artifact keeps the wrapper's
stream, and the re-run prints its stderr and stdout.
@systemfsoftware-maker systemfsoftware-maker changed the title ci(repo): run mutation shards with one checker and one test runner ci(repo): make failed mutation shards readable and bound their workers Oct 9, 2026
@systemfsoftware-maker
ci(repo): run each shard project's stryker child with its output kept
7cd18c2 
`stryker run --shard` spawns one child per planned project with stdout
ignored, keeps 4096 characters of its stderr and prints 1024 of them
(stryker-js dist/main.mjs 113412-113483, 114204-114206). Every exit-3
run on #81 and #82 lost the error that way.

The shard step now runs the same children itself:
- stryker-plan-gate.ts gains a `shard` mode. It decodes the plan with the
  gate's ShardPlan schema and finds the shard by its index/count label,
  as the wrapper does (112975-112980, 113345-113362), refusing an unknown
  label. It writes "<index>\t<project>" lines in plan order.
- .github/scripts/run-shard.sh runs each project in its own directory
  with the wrapper's exact args (113415-113429) and seeding
  (113431-113437), writing stdout.log and stderr.log beside the stream
  under reports/shards/<index>/<project>/. Exit 1 logs the wrapper's
  below-threshold line and carries on (113478); any other exit, or a
  missing progress stream, prints both log tails and stops the shard,
  as the wrapper's sequential forEach does (113474-113483).

The diagnostics re-run and its artifact go: the shard artifact now
carries both logs. The verdict job, merge and gate are unchanged.
@systemfsoftware-maker
ci(repo): run the release gate on main only
f794dde 
Mutation testing runs on main, never on a pull request. The release
gate loses its pull_request trigger and everything that existed only
for it:
- the plan job's changed-files step and fetch-depth 2;
- the scope output, MUTATION_SCOPE and stryker.shared.ts's scopedMutate,
  so every enrolled file is mutated;
- stryker-plan-gate.ts's change scoping (GATE_FILES, scopeOf,
  resolveScope, readChanged, the out-of-scope report and the
  MalformedStrykerConfig refusal), its --changed and --scope-out flags,
  and their tests;
- the pull-request-only cache skip and cancel-in-progress.

The verdict job, the shard layout, run-shard.sh, concurrency 2, the
2 GiB heap cap and the log tails on failure stay.

The release-gate sandbox proof now finds the mutation step by
run-shard.sh instead of `stryker run`. It runs the shard selection and
shard steps with a stub stryker, and checks that the progress stream
lands where the shard artifact uploads it. README and CONTRIBUTING
describe the gate as main-only again.
@systemfsoftware-maker systemfsoftware-maker changed the title ci(repo): make failed mutation shards readable and bound their workers ci(repo): run the release gate on main only, with readable, bounded shards Oct 9, 2026
kiro-systemf[bot]
kiro-systemf Bot approved these changes Oct 9, 2026
View reviewed changes

@kiro-systemf kiro-systemf Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: 7/7 regular checks green on f794dde; run-shard.sh proven by run 37984176648 (shard success); no threshold/mutate-set change; pull_request trigger removed per Ryan (mutation on main only); 0 threads; hunt clean.

@kiro-systemf
kiro-systemf Bot merged commit c8c376c into main Oct 9, 2026
7 checks passed
systemfsoftware-maker added a commit that referenced this pull request Oct 9, 2026
@systemfsoftware-maker
ci(repo): take #82's release-gate commits back out of the guards branch
d7f31aa 
The guards capability no longer waits on a pull-request mutation gate.
Mutation runs on main only, so this branch goes back onto main without
#82's gate changes. release-gate.yml, the shard scripts,
stryker-plan-gate.ts with its tests, and stryker.shared.ts return to
main's content. They reach main through #82 itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kiro-systemf kiro-systemf[bot] kiro-systemf[bot] approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@systemfsoftware-maker