Skip to content

feat: distinguish smoke skips from passes and assert post-exit sidecars - #125

Merged
TMHSDigital merged 2 commits into
mainfrom
feat/smoke-skip-and-postexit
Sep 9, 2026
Merged

feat: distinguish smoke skips from passes and assert post-exit sidecars#125
TMHSDigital merged 2 commits into
mainfrom
feat/smoke-skip-and-postexit

Conversation

@TMHSDigital

Copy link
Copy Markdown
Owner

Summary

  • Host runner tests/smoke/run_example.py + catalog tests/smoke/catalog.json + summarize.py.
  • SKIP is exit 77 + SMOKE_SKIP: <reason> and only when --min-version is above this Blender. Exit 0 with that marker is FAIL (vacuous). Skip on a version that should run is FAIL. Zero PASSes is red (exit 2).
  • Post-exit sidecar is opt-in (--expect-sidecar / $BDT_SMOKE_SIDECAR), asserted after Blender dies. No gallery still.
  • No new shipped examples. Canaries live under tests/smoke/.

Today, before this PR (inspection)

blender-smoke.yml was 45 independent set -e steps. Exit 0 = green. There was no skip channel. run_smoke.py was pass-or-die then ALL SMOKE CHECKS PASSED. A 5.0-only example that sys.exit(0) on 4.5 would have been indistinguishable from a real pass.

Evidence

live-run-proven .scratch/blender-5.2.1-windows-x64/blender.exe Blender 5.2.1 LTS (9e2066aef7ef):

Probe Wrapper exit Notes
--min-version 99.0 + always-skip canary 0 [SKIP] proc_exit 77
--min-version 4.0 + always-skip canary (should run) 1 [FAIL] skipped on 5.2 but min-version 4.0 (should run) not a clean pass
sidecar written, --expect-sidecar after process death 0 [PASS]
--omit-sidecar + --expect-sidecar 1 missing post-exit sidecar
summarize 1 PASS + 1 SKIP 0
summarize all SKIP 2 every example skipped; job is not green

.scratch/blender-4.5.11-windows-x64/blender.exe Blender 4.5.11 LTS: expected skip wrapper exit 0 [SKIP].

python tests/smoke/test_harness.py -v — 16 tests OK (no Blender). Inverted canaries are also CI steps that assert wrapper exit 1.

run_smoke.py still green on 5.2.1: 12 passed, 0 skipped.

inspection-only: YAML splice, catalog listing matches plugin 45 example dirs.

Test plan

  • validate-harness + validate counts/manifest/frontmatter
  • blender-smoke 5.2 + 4.5: unit tests, red canaries (exit 1 asserted), expected SKIP in summary, sidecar PASS, catalog green
  • Socket not pending
  • squash feat:release.yml RELEASE (expect v0.48.0)

Exit 0 was the only green. A version skip would have been a vacuous pass. The host runner records SKIP only on exit 77 plus a SMOKE_SKIP reason below --min-version; a missing sidecar after Blender exits is FAIL; a leg with zero PASSes is red.

Signed-off-by: fOuttaMyPaint <TMhospitalitystrategies@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation ci labels Sep 9, 2026
GitHub rejects runner.temp at job-level env, so the workflow never
parsed and the PR never got a smoke job.

Signed-off-by: fOuttaMyPaint <TMhospitalitystrategies@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@TMHSDigital

Copy link
Copy Markdown
Owner Author

CI canaries on PR head d71160c (run 34408788566):

live-run-proven (CI Linux portable)

  • 5.2: /home/runner/work/_temp/bl/blender-5.2.1-linux-x64/blender Blender 5.2.1 LTS (9e2066aef7ef)
  • 4.5: /home/runner/work/_temp/bl/blender-4.5.13-linux-x64/blender Blender 4.5.13 LTS (daeeeca98fb0)

Both legs:

  • expected skip: [SKIP] canary-skip: harness canary (always skip) (wrapper 0; summary counts SKIP separately)
  • unexpected skip: [FAIL] skipped on X.Y but min-version 4.0 (should run) then unexpected skip exited 1 (correct)
  • sidecar: [PASS] canary-sidecar after process death
  • missing sidecar: [FAIL] missing post-exit sidecar then missing sidecar exited 1 (correct)
  • job summary: 47 passed, 1 skipped, 0 failed (canary-skip listed SKIP, not PASS)

First push (5cdd729) never got a smoke job: runner.temp is illegal in job-level env (actionlint). Fixed in d71160c.

@TMHSDigital
TMHSDigital merged commit 350e0a2 into main Sep 9, 2026
10 checks passed
@TMHSDigital
TMHSDigital deleted the feat/smoke-skip-and-postexit branch September 9, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant