Repository navigation
Demo: force an Android Release E2E failure to exercise CI retries (#59571) - #59571
Open
GijsWeterings wants to merge 2 commits into
Open
GijsWeterings wants to merge 2 commits into
GijsWeterings wants to merge 2 commits into
Conversation
meta-cla
Bot
added
the
CLA Signed
|
@GijsWeterings has exported this pull request. If you are a Meta employee, you can view the originating Diff in D128044163. |
GijsWeterings
force-pushed
the
export-D128044163
branch
from
October 28, 2026 11:51
fdcb530 to
5b2dbc9
Compare
Visual regressionNo visual differences at ios: no capture |
GijsWeterings
added a commit
to GijsWeterings/react-native
that referenced
this pull request
Oct 28, 2026
…act#59571) Summary: Demo only, stacked on the retry-chain fix so the exported PR shows the new behavior on real CI. Do not land. This adds an RNTester Maestro flow that always fails, scoped so that only the Android Release E2E job runs it: - `android-release-only` excludes it from the Android Debug job. - `local-screenshot-baseline` keeps it out of the Maestro Cloud runs. - A `when: platform: Android` condition makes it a no-op in the iOS E2E job. The file name sorts it last, so every other flow still runs first. What the PR run should show: - `test_e2e_android_rntester`: `test (debug)` passes and `test (release)` fails. Both checks are green (a soft failure in the step), and `retry_1` starts even if the passing Debug job finishes last. - `test_e2e_android_rntester_retry_1` behaves the same way, and `retry_2` starts. - `test_e2e_android_rntester_retry_2 / test (release)` is red on the "Run E2E Tests" step. It is the only red check from this chain, and there is no separate `report` job. - "Re-run failed jobs" re-runs `retry_2 / test (release)`, which runs Maestro again and fails again. Changelog: [Internal] Differential Revision: D128044163
meta-codesync
Bot
changed the title
GijsWeterings
force-pushed
the
export-D128044163
branch
from
October 28, 2026 13:24
5b2dbc9 to
b54bc13
Compare
Summary:
The E2E and Fantom retry chains reported their final failure through a separate `report` job. The `test` job always succeeded (its test step used `continue-on-error: true`), and `report` read its `status` output and exited 1 on the last attempt. This caused three problems:
- The red check was a five-second status job, not the job that ran the tests, so finding the real failure meant opening a green job and looking for the step with a warning icon.
- "Re-run failed jobs" re-ran only `report`. It read the previous attempt's cached `status` output and failed again without running any tests.
- The Debug and Release matrix legs wrote the same `status` output, and GitHub keeps only the last value written to a given output name. If one flavor failed and the other passed and finished later, `status` came out empty, so no retry ran and the failure was dropped silently.
This change:
- Removes the `report` jobs. The test step runs as two copies of the same step, gated on `inputs.fail-on-error`: earlier attempts use `continue-on-error: true`, so they stay green and record the failure in the `status` output, and the final attempt runs without it, so the test step itself fails the job. The two copies share their `with:` block through a YAML anchor. (A single step with `continue-on-error: ${{ !inputs.fail-on-error }}` doesn't work: GitHub evaluated it as true on the final attempt, so the failure stayed hidden. Step `if:` conditions read `inputs` correctly.)
- Has each matrix leg write a flavor-specific output (`debug_status` / `release_status`). GitHub combines outputs with different names across matrix legs. The reusable workflow's `status` output is `debug_status || release_status`, or `jobs.test.outputs.status` for the single-job Fantom workflow.
- Also triggers the retry jobs in `test-all.yml` on `needs.<attempt>.result == 'failure'`. The iOS `report` jobs counted a failed `build` or `test` job as a failure, and this keeps that behavior for every workflow.
The flavor is lowercased with `tr` rather than `${FLAVOR,,}` because the macOS runners' default `bash` may be 3.2. A flavor with no matching output fails the step with an error instead of writing an output nothing reads; a red `test` job still triggers the retry through `needs.<attempt>.result`.
Changelog: [Internal]
Differential Revision: D128043004
…act#59571) Summary: Demo only, stacked on the retry-chain fix so the exported PR shows the new behavior on real CI. Do not land. This adds an RNTester Maestro flow that always fails, scoped so that only the Android Release E2E job runs it: - `android-release-only` excludes it from the Android Debug job. - `local-screenshot-baseline` keeps it out of the Maestro Cloud runs. - A `when: platform: Android` condition makes it a no-op in the iOS E2E job. The file name sorts it last, so every other flow still runs first. What the PR run should show: - `test_e2e_android_rntester`: `test (debug)` passes and `test (release)` fails. Both checks are green (a soft failure in the step), and `retry_1` starts even if the passing Debug job finishes last. - `test_e2e_android_rntester_retry_1` behaves the same way, and `retry_2` starts. - `test_e2e_android_rntester_retry_2 / test (release)` is red on the "Run E2E Tests" step. It is the only red check from this chain, and there is no separate `report` job. - "Re-run failed jobs" re-runs `retry_2 / test (release)`, which runs Maestro again and fails again. Changelog: [Internal] Differential Revision: D128044163
GijsWeterings
force-pushed
the
export-D128044163
branch
from
October 28, 2026 15:58
b54bc13 to
6365b8e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Demo only, stacked on the retry-chain fix so the exported PR shows the new behavior on real CI. Do not land.
This adds an RNTester Maestro flow that always fails, scoped so that only the Android Release E2E job runs it:
android-release-onlyexcludes it from the Android Debug job.local-screenshot-baselinekeeps it out of the Maestro Cloud runs.when: platform: Androidcondition makes it a no-op in the iOS E2E job.The file name sorts it last, so every other flow still runs first.
What the PR run should show:
test_e2e_android_rntester:test (debug)passes andtest (release)fails. Both checks are green (a soft failure in the step), andretry_1starts even if the passing Debug job finishes last.test_e2e_android_rntester_retry_1behaves the same way, andretry_2starts.test_e2e_android_rntester_retry_2 / test (release)is red on the "Run E2E Tests" step. It is the only red check from this chain, and there is no separatereportjob.retry_2 / test (release), which runs Maestro again and fails again.Changelog: [Internal]
Differential Revision: D128044163