Repository navigation
The PostCommit XVR Spark3 job is flaky #30602
Description
The PostCommit XVR Spark3 is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml?query=is%3Afailure+branch%3Amaster to see the logs.
Activity
github-actions commentedon May 15, 2024
Reopening since the workflow is still flaky
github-actions commentedon May 25, 2024
Reopening since the workflow is still flaky
Abacn commentedon May 26, 2024
looks like thus time the breakage is due to python3.12 support:
File "/tmp/bootstrap_beam_venv7899546459285591331.py", line 33, in
from pkg_resources import parse_version
ModuleNotFoundError: No module named 'pkg_resources'
70 remaining items
github-actions commentedon Jul 3, 2026
Reopening since the workflow is still flaky
aIbrahiim commentedon Jul 8, 2026
I inevstigated the issue and here's my findings as until July 1 both Python 3.10 and 3.14 were passing after that, the job started failing, but at first only the 3.14 job was red but the 3.10 job was still green until today
so the 3.14 failure seems to be in the Go cross language test TestKafkaIO_BasicReadWrite and the Java io expansion service does not start that's why I saw connection refused after 30 retries and JDBC tests pass, so it looks specific to the Kafka expansion service. This started around July 4, before PR #39205 also one thing to note: the 3.10 job uses skipNonPythonTask=true, so it skips the Go test that fails on 3.14. That is why 3.10 still looked fine even when the overall workflow was failing
#39205 changed workflow default Java from 11 to 21 and after that, the 3.10 job also started failing with IllegalAccessError on sun.nio.ch.DirectBuffer. My understanding is that XVR Spark3 starts Spark in a separate process through run_job_server.sh but before #39205, that process was effectively running on Java 11 (from the setup in #39058) and after #39205, it runs on Java 21, but it does not get the same --add-opens flags that Gradle Spark tests already use in sparkTestJvmArgs(). So #39205 added a new regression on 3.10.
so for 3.10 I would suggest passing the required --add-opens flags to the Spark job server on Java 17+, maybe through run_job_server.sh and for 3.14, I think the Kafka expansion service issue and I'm investigating it further.
@Abacn @Amar3tto @shunping
github-actions commentedon Jul 22, 2026
Reopening since the workflow is still flaky
Amar3tto commentedon Jul 23, 2026
Stabilized
github-actions commentedon Aug 7, 2026
Reopening since the workflow is still flaky
aIbrahiim commentedon Aug 10, 2026
fixed by #39683