Skip to content

test(visual): attribute synthetic capture failures without leaking browser details (#651) - #935

Merged
Fooftilly merged 3 commits into
masterfrom
test/visual-failure-stage-redaction-651
Oct 27, 2026
Merged

Fooftilly merged 3 commits into
masterfrom
test/visual-failure-stage-redaction-651

Conversation

@Fooftilly

@Fooftilly Fooftilly commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Why

The existing five-scene local visual regression pilot tests only disposable research fixtures, but its current Python/Playwright failures can be difficult to triage: a timeout or exception prints an unbounded Playwright message without immediately identifying which capture phase failed. Playwright traces can include selectors, URLs and page text, which should not become standard CI output. This targets #651 D3 and #857 without expanding the testing matrix.

Change

  • Name bounded stages for initial browser/server setup, Folder/Work/PDF/split/Settings readiness, each scene capture, and the final strict browser-error collector.
  • Raise only VisualCaptureFailure(stage, exception class), withholding the raw Playwright exception/message. Do not treat any failure as a pass or ignore the strict collector.
  • Finish context, browser, Playwright and disposable server cleanup independently; preserve the original capture failure if a second cleanup error occurs, but fail on a cleanup error when capture otherwise passed.
  • Add cheap mock regression tests proving the intended stage, redacted exception string and cleanup behavior. The real advisory visual workflow exercises all five capture phases and unchanged approved baseline.

No screenshot or baseline update, new framework/workflow/package, altered pixel thresholds, new required gate or external upload. The normal fixture-only visual pilot remains authoritative; do not merge without maintainer approval.

Related: #651 D3 and #857.


Devin Review

@coldtea-pr-lens

Copy link
Copy Markdown

Note

The title starts with test(visual):, so PR Lens left this pull request undrawn. Comment @pr-lens draw to draw it

github.comment.notice: false in .github/pr-lens.yml turns this note off

@Fooftilly
Fooftilly deployed to project-automation October 26, 2026 05:29 — with GitHub Actions Active
@sourcery-ai

This comment has been minimized.

@coderabbitai

This comment has been minimized.

@chatgpt-codex-connector

This comment has been minimized.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@Fooftilly
Fooftilly merged commit 326cbf4 into master Oct 27, 2026
28 checks passed
@Fooftilly
Fooftilly deleted the test/visual-failure-stage-redaction-651 branch October 27, 2026 16:28
@github-project-automation github-project-automation Bot moved this from Review to Done in PRKS Roadmap Oct 27, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Pull request is closed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant