Report tests blocked by a broken @BeforeAll as separate results (fixes #1155) - #1390
Open
Yevhen-Bozhenko wants to merge 1 commit into
Conversation
When @BeforeAll fails, the class is reported as one fake test result carrying the class id. A run where it succeeds reports one result per method instead, so the two never share a history id and a passing retry cannot replace the failure. The class stays red even after every test passes. Report the blocked tests themselves, so both runs produce the same method-level history ids and the retry replaces them. Only a test with no result counts as blocked: a disabled test keeps its single skipped result, and tests a nested class already reported are not reported again by its parent. A broken @afterall blocks nothing, and an aborted container is never retried, so both keep the fake test result. fixes allure-framework#1155
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Fixes #1155.
When
@BeforeAllfails, the class is reported as one fake test result identified by the class.A run where
@BeforeAllsucceeds reports one result per method instead, so the two never share a history id and a passing retry cannot replace the failure - the class stays red even after every test passes.This reports the tests the failing container blocked, so both runs produce the same method-level history ids and the retry replaces them.
A blocked test is one with no result yet, which is not the same as one that never started - a
@Disabledmethod never starts but already has aSKIPPEDresult. Every result goes throughstartTest, so that is where the unique id is recorded. A broken@AfterAllblocks nothing, and an aborted container is never retried, so both keep the fake test result.Two existing tests change because they asserted the old class-level result:
shouldProcessBrokenInBeforeAllTests, andshouldLinkFailedContainerFakeTestToItsScope, which is renamed toshouldLinkBlockedTestsToFailedContainerScopesince there is no fake test in that scenario now.One thing I would value your view on: blocked tests are linked to the failed container's scope only, not to their ancestor scopes via
getParentScopeKeys, since they never started and have no scope of their own. Happy to change it if you would rather they were linked all the way up.Checklist