Skip to content

[Bug]: Playwright-Junit lifecycle incompatible with maven retries 'surefire.rerunFailingTestsCount' #1751

Open
#1988
Open
[Bug]: Playwright-Junit lifecycle incompatible with maven retries 'surefire.rerunFailingTestsCount'#1751
#1988
Labels
P3-collecting-feedback

Description

@rtretyak
Roman Tretiak (rtretyak)
opened on Feb 20, 2025

Version

1.50.0

Steps to reproduce

  1. Have a basic Playwright test that fails
  2. Try running it with maven retries: ./mvnw test -Dtest=MyTest -Dsurefire.rerunFailingTestsCount=2

Expected behavior

System is able to retry the failing test, system can work with Playwright context (browser, page etc.)

Actual behavior

First attempt is a legit failure, subsequent rerun attempts have broken Playwright context

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in fm.seller.onboardingweb.test.MyTest
[ERROR] fm.seller.onboardingweb.test.MyTest.test(Page) -- Time elapsed: 0.001 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[ERROR] Errors: 
[ERROR] fm.seller.onboardingweb.test.MyTest.test(Page)
[ERROR]   Run 1: MyTest.test:13
[ERROR]   Run 2: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
[ERROR]   Run 3: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed

Additional context

Minimal test:

@UsePlaywright(OptionsFactory.class)
public class MyTest {
    @Test
    public void test(Page page) {
        page.navigate("https://google.com/");
        assert false;
    }
}

Environment

  • OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"
  • Java version: 17.0.2, vendor: Oracle Corporation
  • Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
  • playwright 1.50.0
  • Junit 5.11.4
  • surefire 3.5.2

Activity

  1. Roman Tretiak (rtretyak)
    changed the title [-][Bug]: Playwight-Junit lifecycle incompatible with maven retries 'surefire.rerunFailingTestsCount'[/-] [+][Bug]: Playwright-Junit lifecycle incompatible with maven retries 'surefire.rerunFailingTestsCount'[/+] on Feb 20, 2025

  2. yury-s commented on Feb 21, 2025

    @yury-s
    Yury Semikhatsky (yury-s)
    on Feb 21, 2025
    Member
    • Have a basic Playwright test that fails

    Please share a complete example. I was not able to reproduce it with a simple test.

    uchagani (@uchagani) can you have a look?

  3. Yury Semikhatsky (yury-s)
    added
    P3-collecting-feedback
    on Feb 21, 2025

  4. uchagani commented on Feb 21, 2025

    @uchagani
    uchagani (uchagani)
    on Feb 21, 2025
    Contributor

    I cannot reproduce this. Please provide a complete example. Here is mine that works as expected:

    import com.microsoft.playwright.Page;
import com.microsoft.playwright.assertions.PageAssertions;
import com.microsoft.playwright.junit.Options;
import com.microsoft.playwright.junit.OptionsFactory;
import com.microsoft.playwright.junit.UsePlaywright;
import org.junit.jupiter.api.Test;

import static com.microsoft.playwright.assertions.PlaywrightAssertions.assertThat;

@UsePlaywright(SampleTests.PlaywrightOptions.class)
public class SampleTests {
    public static class PlaywrightOptions implements OptionsFactory {
        @Override
        public Options getOptions() {
            return new Options().setBaseUrl("https://www.playwright.dev");
        }
    }

    @Test
    void testPageTitle(Page page) {
        page.navigate("/");
        assertThat(page).hasTitle("foo", new PageAssertions.HasTitleOptions().setTimeout(1000));
    }
}

    pom.xml

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>playwright-java-sample</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.50.0</version>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.11.4</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>3.5.2</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

  5. rtretyak commented on Feb 21, 2025

    @rtretyak
    Roman Tretiak (rtretyak)
    on Feb 21, 2025
    Author

    pom:

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>playwrightRetries</artifactId>
    <version>1.0-SNAPSHOT</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.2</version>
        <relativePath/>
    </parent>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.5.2</version>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.50.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

    test:

    import com.microsoft.playwright.Page;
import com.microsoft.playwright.junit.UsePlaywright;
import org.junit.jupiter.api.Test;

@UsePlaywright
public class MyTest {
    @Test
    public void test(Page page) {
        page.navigate("https://google.com/");
        assert false;
    }
}

    command:
    mvn clean test -Dsurefire.rerunFailingTestsCount=1

    output:

    [INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.671 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.test(Page) -- Time elapsed: 3.640 s <<< FAILURE!
java.lang.AssertionError
        at MyTest.test(MyTest.java:10)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.test(Page) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] MyTest.test(Page)
[ERROR]   Run 1: MyTest.test:10
[ERROR]   Run 2: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

  6. rtretyak commented on Feb 22, 2025

    @rtretyak
    Roman Tretiak (rtretyak)
    on Feb 22, 2025
    Author

    uchagani (@uchagani) I tried your project and I'm getting the same problem. The retry is not able to start properly:

    [INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.565 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.testPageTitle(Page) -- Time elapsed: 5.528 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 
Page title expected to be: foo
Received: Fast and reliable end-to-end testing for modern web apps | Playwright

Call log:
  - Locator.expect with timeout 1000ms
  - waiting for locator(":root")
    5 × locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-hydrated="true" class="plugin-pages plugin-id-default" data-rh="lang,dir,class,data-has-hydrated">…</html>
      - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:74)
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:50)
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:42)
        at com.microsoft.playwright.impl.PageAssertionsImpl.hasTitle(PageAssertionsImpl.java:45)
        at MyTest.testPageTitle(MyTest.java:22)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.testPageTitle(Page) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page arg0] in method [void MyTest.testPageTitle(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] MyTest.testPageTitle(Page)
[ERROR]   Run 1: MyTest.testPageTitle:22 Page title expected to be: foo
Received: Fast and reliable end-to-end testing for modern web apps | Playwright

Call log:
  - Locator.expect with timeout 1000ms
  - waiting for locator(":root")
    5 × locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-hydrated="true" class="plugin-pages plugin-id-default" data-rh="lang,dir,class,data-has-hydrated">…</html>
      - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
[ERROR]   Run 2: MyTest.testPageTitle(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page arg0] in method [void MyTest.testPageTitle(com.microsoft.playwright.Page)]: Playwright connection closed
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

    Here is also my mvn version output:

     % mvn -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /opt/homebrew/Cellar/maven/3.8.5/libexec
Java version: 23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/23/libexec/openjdk.jdk/Contents/Home
Default locale: en_UA, platform encoding: UTF-8
OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"

    And I also tried with this with the same result:

     % ./mvnw -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /Users/user/.m2/wrapper/dists/apache-maven-3.8.5-bin/67203e94/apache-maven-3.8.5
Java version: 17.0.2, vendor: Oracle Corporation, runtime: /Users/user/Library/Java/JavaVirtualMachines/openjdk-17.0.2/Contents/Home
Default locale: en_UA, platform encoding: UTF-8
OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"

  7. uchagani commented on Feb 23, 2025

    @uchagani
    uchagani (uchagani)
    on Feb 23, 2025
    Contributor

    I'm not sure what could be causing this and I can't seem to reproduce it.

  8. rtretyak commented on Feb 23, 2025

    @rtretyak
    Roman Tretiak (rtretyak)
    on Feb 23, 2025
    Author

    I have also tried it on my home Windows PC which is completely independent from the previous environment and was able to reproduce it with your project.

    mvn -version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: E:\workspace\apache-maven-3.8.4
Java version: 23.0.2, vendor: Oracle Corporation, runtime: C:\Users\38097\.jdks\openjdk-23.0.2
Default locale: en_US, platform encoding: UTF-8
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

    Let me know if there is any other debug information I can provide you with...

  9. uchagani commented on Feb 23, 2025

    @uchagani
    uchagani (uchagani)
    on Feb 23, 2025
    Contributor

    Are you configuring your Java or maven in any specific way?

    Can you check and post your settings.xml file (without any private info).

    Are you running these tests from inside of an IDE or just the command line/terminal?

  10. rtretyak commented on Feb 23, 2025

    @rtretyak
    Roman Tretiak (rtretyak)
    on Feb 23, 2025
    Author

    Are you configuring your Java or maven in any specific way?

    No, nothing special. I tried with multiple different Javas as you can see above. Not doing anyhting special to them. 1 was installed with homebrew, other 2 just a default installation by IntelliJ IDEA

    settings.xml file

    default one, just empty tags, no additional configurations

    Are you running these tests from inside of an IDE or just the command line/terminal?

    I'm running from terminal. Tried both in terminal inside IntelliJ and native terminal apps on windows/mac, same result.

  11. rtretyak commented on Mar 3, 2025

    @rtretyak
    Roman Tretiak (rtretyak)
    on Mar 3, 2025
    Author

    You can also reproduce it in the official playwright-java image..
    docker run --rm -v "$(pwd):/app" -w /app mcr.microsoft.com/playwright/java:v1.50.0-noble mvn clean test -Dsurefire.rerunFailingTestsCount=1

  12. Develop-KIM commented on Sep 16, 2026

    @Develop-KIM
    monkey (Develop-KIM)
    on Sep 16, 2026
    Contributor

    I hit this too and dug into it. PlaywrightExtension keeps the Playwright in a static ThreadLocal, and the PlaywrightRegistry in the root store closes it when the launcher finishes the first run, but the ThreadLocal is never cleared. Surefire's rerun happens in the same JVM on the same thread, so getOrCreatePlaywright hands back the closed instance and BrowserExtension fails on launch():

    ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page arg0] ...: Playwright connection closed
  at com.microsoft.playwright.impl.BrowserTypeImpl.launch(BrowserTypeImpl.java:48)
  at com.microsoft.playwright.impl.junit.BrowserExtension.getOrCreateBrowser(BrowserExtension.java:82)

    Repro on 1.62.0 is one @UsePlaywright test that always fails plus mvn test -Dsurefire.rerunFailingTestsCount=1. One thing to be aware of: since #1860 (1.57) the registry is AutoCloseable rather than CloseableResource, and JUnit only closes AutoCloseable store values from 5.13 on. With junit-jupiter 5.11 the first Playwright is simply never closed, so the retry reuses it and the problem is invisible; with 5.14 it fails every time.

    The fix I have in mind is to only reuse the ThreadLocal instance when it belongs to the current run's registry and create a fresh one otherwise (clearing the ThreadLocal from close() can't reach other threads). Happy to send a PR if that direction works for you.

  13. Develop-KIM commented on Sep 16, 2026

    @Develop-KIM
    monkey (Develop-KIM)
    on Sep 16, 2026
    Contributor

    Opened #1988 with the fix and a regression test. If you'd rather handle this differently I'm fine closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3-collecting-feedback

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions