Report fixture and lifecycle failures without creating phantom tests - #1391
Open
baev wants to merge 7 commits into
Open
Report fixture and lifecycle failures without creating phantom tests#1391baev wants to merge 7 commits into
baev wants to merge 7 commits into
Conversation
baev
added
the
pr:improvement
A failed configuration method used to be reported as an invented test result. Such a result is counted in the run statistics, and its history id — md5(testCaseId + parameters) — is one no execution ever produces, so no retry could ever replace it: a green rerun still showed the error. The failure now becomes a run-level GlobalError instead, keeping the broken fixture and its evidence in the scope. The tests it prevented keep their own results: TestNG skips them with the real parameters of every data provider row, so their identity matches a green rerun. The one exception is a parameterised method whose values were never produced — its data provider failed, or a failed dependency or fixture kept the provider from running. TestNG ends such a method once with no parameters, and a result built from that end has an identity nothing can replace. It is reported as a run-level error too. Whether a method is parameterised is read from its own declaration, because ITestNGMethod#isDataDriven reads only the method-level annotation and misses a provider inherited from a class-level @test. A data provider that skips on purpose keeps its skipped result: TestNG wraps the SkipException, so the intent is found through the cause chain. After configuration failures are worded neutrally, since an alwaysRun after method also runs for a test a failed before fixture already skipped.
…s global errors A container that failed or aborted — typically a class whose @BeforeAll threw or failed an assumption — used to be reported as an invented test result. Such a result is counted in the run statistics, and its history id is one no execution ever produces, so no retry could ever replace it. The platform sends no events for the tests inside such a container, so nothing can be backfilled either; the failure now becomes a run-level GlobalError carrying the original exception. A skipped subtree still backfills a skipped result for every test the plan can name, with the identity it has in a real run. Test templates, test factories and parameterized classes are containers whose invocations only exist once they run: a skipped one has nothing to report, and a result for the container itself would carry a history id no invocation ever produces. Such containers are collected and reported once as a run-level error instead. With no non-test identifier reaching stopTest any more, the AS_ID=-1 marker that hid the invented results from the report goes too.
… errors A @BeforeClass, @afterclass or @ClassRule that threw, or a @BeforeClass assumption, used to vanish from the report entirely: JUnit 4 reports it on the description of the class itself, with no events for the tests it prevented, and the listener's update of a test it never started was a silent no-op. The run looked green with fewer tests than expected. Such a failure is now a run-level GlobalError naming the class, the count of tests inside it and the cause. Nothing is backfilled: JUnit 4 does not name the tests a class-level failure prevented, and the message claims nothing about whether they ran, since a before and an after failure arrive on the same description. An invalid test class, or a Parameterized class whose parameters method threw, is reported by JUnit 4 as a synthetic initializationError test. That used to become a broken test result whose identity matches none of the cases a green run produces, so no retry could replace it; it is now a run-level error too.
A failed setupSpec used to be reported as an invented test result, and every feature it prevented was backfilled as skipped — including data-driven features, backfilled once with no parameters. The fixture result is not a test case, and the parameterless feature result carries a history id that no iteration ever produces: both are counted in the run statistics under identities a green rerun cannot replace. The failure now becomes a run-level GlobalError, with the broken fixture kept in the spec scope. Plain features are still backfilled as skipped with the identity they have in a real run. Data-driven features are not: their iterations, and the values that tell them apart, only exist once the feature runs, so they are named in the run-level error instead. A failed cleanupSpec used to be visible only as a broken fixture inside the spec scope; it is now a run-level error too, its features keeping the results they already produced.
baev
force-pushed
the
fixture-failures-as-global-errors
branch
from
August 25, 2026 15:29
51f8e17 to
19b5524
Compare
pratushnyi
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Report failures outside individual tests as run-level errors across TestNG, JUnit Platform, JUnit 4, Spock 2, Cucumber 7, and JBehave 5. This covers configuration methods, data providers, failed containers, class and specification fixtures, run hooks, and story lifecycle steps.
These failures are no longer lost or represented as synthetic test cases that distort statistics and remain in history after successful retries. Real tests retain stable identities, blocked tests are reported as skipped when their identities are known, and unknown parameterized or template invocations are not invented.
fixes #643
fixes #912
fixes #1128
fixes #1155
Checklist