The PostCommit Python job is flaky

The PostCommit Python is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python.yml?query=is%3Afailure+branch%3Amaster to see the logs.

It first failed on https://github.com/apache/beam/actions/runs/8210266873.

The failed task is :sdks:python:test-suites:portable:py38:portableWordCountSparkRunnerBatch.

Traceback:

INFO:apache_beam.utils.subprocess_server:Starting service with ('java' '-jar' '/runner/_work/beam/beam/runners/spark/3/job-server/build/libs/beam-runners-spark-3-job-server-2.56.0-SNAPSHOT.jar' '--spark-master-url' 'local[4]' '--artifacts-dir' '/tmp/beam-temp8q8022zi/artifactsg6e8usou' '--job-port' '56313' '--artifact-port' '0' '--expansion-port' '0')
INFO:apache_beam.utils.subprocess_server:Error: A JNI error has occurred, please check your installation and try again
INFO:apache_beam.utils.subprocess_server:Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/vendor/grpc/v1p60p1/io/grpc/BindableService
INFO:apache_beam.utils.subprocess_server:	at java.lang.ClassLoader.defineClass1(Native Method)
INFO:apache_beam.utils.subprocess_server:	at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
INFO:apache_beam.utils.subprocess_server:	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
INFO:apache_beam.utils.subprocess_server:	at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
INFO:apache_beam.utils.subprocess_server:	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
INFO:apache_beam.utils.subprocess_server:	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
INFO:apache_beam.utils.subprocess_server:	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
INFO:apache_beam.utils.subprocess_server:	at java.security.AccessController.doPrivileged(Native Method)
INFO:apache_beam.utils.subprocess_server:	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
INFO:apache_beam.utils.subprocess_server:	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
INFO:apache_beam.utils.subprocess_server:	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
INFO:apache_beam.utils.subprocess_server:	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
INFO:apache_beam.utils.subprocess_server:	at java.lang.Class.getDeclaredMethods0(Native Method)
INFO:apache_beam.utils.subprocess_server:	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
INFO:apache_beam.utils.subprocess_server:	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
INFO:apache_beam.utils.subprocess_server:	at java.lang.Class.getMethod0(Class.java:3018)
INFO:apache_beam.utils.subprocess_server:	at java.lang.Class.getMethod(Class.java:1784)
INFO:apache_beam.utils.subprocess_server:	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:670)
INFO:apache_beam.utils.subprocess_server:	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:652)
INFO:apache_beam.utils.subprocess_server:Caused by: java.lang.ClassNotFoundException: org.apache.beam.vendor.grpc.v1p60p1.io.grpc.BindableService
INFO:apache_beam.utils.subprocess_server:	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
INFO:apache_beam.utils.subprocess_server:	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
INFO:apache_beam.utils.subprocess_server:	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
INFO:apache_beam.utils.subprocess_server:	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
INFO:apache_beam.utils.subprocess_server:	... 19 more
ERROR:apache_beam.utils.subprocess_server:Started job service with ('java', '-jar', '/runner/_work/beam/beam/runners/spark/3/job-server/build/libs/beam-runners-spark-3-job-server-2.56.0-SNAPSHOT.jar', '--spark-master-url', 'local[4]', '--artifacts-dir', '/tmp/beam-temp8q8022zi/artifactsg6e8usou', '--job-port', '56313', '--artifact-port', '0', '--expansion-port', '0')
ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
  File "/runner/_work/beam/beam/sdks/python/apache_beam/utils/subprocess_server.py", line 175, in start
    raise RuntimeError(
RuntimeError: Service failed to start up with error 1
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/runner/_work/beam/beam/sdks/python/apache_beam/examples/wordcount.py", line 111, in <module>
    run()
  File "/runner/_work/beam/beam/sdks/python/apache_beam/examples/wordcount.py", line 106, in run
    output | 'Write' >> WriteToText(known_args.output)
  File "/runner/_work/beam/beam/sdks/python/apache_beam/pipeline.py", line 612, in __exit__
    self.result = self.run()
  File "/runner/_work/beam/beam/sdks/python/apache_beam/pipeline.py", line 586, in run
    return self.runner.run_pipeline(self, self._options)
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/runner.py", line 192, in run_pipeline
    return self.run_portable_pipeline(
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/portability/portable_runner.py", line 381, in run_portable_pipeline
    job_service_handle = self.create_job_service(options)
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/portability/portable_runner.py", line 296, in create_job_service
    return self.create_job_service_handle(server.start(), options)
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/portability/job_server.py", line 81, in start
    self._endpoint = self._job_server.start()
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/portability/job_server.py", line 110, in start
    return self._server.start()
  File "/runner/_work/beam/beam/sdks/python/apache_beam/utils/subprocess_server.py", line 175, in start
    raise RuntimeError(
RuntimeError: Service failed to start up with error 1
> Task :sdks:python:test-suites:portable:py38:portableWordCountSparkRunnerBatch FAILED

@damccorm

Added the owner of the commit whose post-commit job failed at the first time.
@damccorm

I think we can pretty comfortably rule out that change, it was to the yaml sdk which is unrelated to portableWordCountSparkRunnerBatch. Note that this runs on a schedule, not on commits, though none of the commits in that scheduled time look particularly harmful

I see. It was red for the last two weeks and flaky before that too.

Permared right now

Only sorta - each component job is actually not permared - e.g. there are 2 successes here, https://github.com/apache/beam/actions/runs/8873798546

The whole workflow is permared just because our flake percentage is so high

Yea, let's work out how to get top-level signal.

The lowest and highest Python version (3.8, 3.11) are running more tests than (3.9, 3.10), could be those tests or task permared

Could make sense to find a way to get separate top-level signal for Python versions, assuming we can use software engineering to share everything necessary so they don't get out of sync.

Reopening since the workflow is still flaky

@Abacn

@Abacn @shunping this is also the 2.44 and 2.42 DirectRow mismatch, will be fixed by #40042 and the new beam-master image

Stabilized

Reopening since the workflow is still flaky

The test LeaderBoardIT.test_leader_board_it has been flaky recently.

  E         AssertionError: 
  E         Expected: (Test pipeline expected terminated in state: RUNNING and Expected checksum is de00231fe6730b972c0ff60a99988438911cda53 and Expected checksum is de00231fe6730b972c0ff60a99988438911cda53)
  E              but: Expected checksum is de00231fe6730b972c0ff60a99988438911cda53 Actual checksum is da39a3ee5e6b4b0d3255bfef95601890afd80709
  
  apache_beam/runners/dataflow/test_dataflow_runner.py

Something wrong with reading from BQ

WARNING  apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:235 Timing out on waiting for job 2026-09-22_06_19_14-11343197862403999659 after 604 seconds
  INFO     apache_beam.io.gcp.tests.bigquery_matcher:bigquery_matcher.py:121 Attempting to perform query SELECT total_score FROM `apache-beam-testing.leader_board_it_dataset1790083151d5bbb9.leader_board_users` WHERE total_score=5000 LIMIT 1 to BQ
  INFO     apache_beam.io.gcp.tests.bigquery_matcher:bigquery_matcher.py:96 Read from given query (SELECT total_score FROM `apache-beam-testing.leader_board_it_dataset1790083151d5bbb9.leader_board_users` WHERE total_score=5000 LIMIT 1), total rows 0
  INFO     apache_beam.io.gcp.tests.bigquery_matcher:bigquery_matcher.py:101 Generate checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709

@shunping

@shunping I looked at a few recent PostCommit Python runs. Same failure: empty checksum da39a3ee..., and the BQ query for total_score=5000 returns 0 rows aslo the Timing out on waiting for job after around 600s part seems normal for this streaming IT and what stands out is that it fails on different SHAs, and the same SHA can fail once then pass later so I’m thinking toward a race on the BQ side rather than a recent commit breaking the pipeline

Also the LeaderBoardIT writes with streaming inserts but the matchers are created without timeout_secs, so they only query once after the wait. If the rows arent visible yet, we get exactly this empty checksum

I was thinking we could pass timeout_secs (maybe 5 min) on both users/teams matchers in leader_board_it_test.py keep the same query and expected checksum, and let it retry until the data shows up

The query of SELECT total_score FROM apache-beam-testing.leader_board_it_dataset*.leader_board_users WHERE total_score=5000 LIMIT 1 is running against the user score table in the code.

beam/sdks/python/apache_beam/examples/complete/game/leader_board.py

Lines 354 to 363 in d1ebefb

    
           events 
        
           | 'CalculateUserScores' >> CalculateUserScores(args.allowed_lateness) 
        
           | 'FormatUserScoreSums' >> beam.Map(format_user_score_sums) 
        
           | 'WriteUserScoreSums' >> WriteToBigQuery( 
        
               args.table_name + '_users', 
        
               args.dataset, { 
        
                   'user': 'STRING', 
        
                   'total_score': 'INTEGER', 
        
               }, 
        
               options.view_as(GoogleCloudOptions).project))

I see 4 rows should be written in that step.

I was thinking we could pass timeout_secs (maybe 5 min) on both users/teams matchers in leader_board_it_test.py keep the same query and expected checksum, and let it retry until the data shows up

Sounds good. Please try that and see if we can reduce the test flakiness.

@aIbrahiim

@aIbrahiim: have you experimented on your idea in #30513 (comment)? LeaderBoardIT.test_leader_board_it is still flaky.

@shunping

@aIbrahiim: have you experimented on your idea in #30513 (comment)? LeaderBoardIT.test_leader_board_it is still flaky.

@shunping yes I tried that approach in #40241

What I did there:

timeout_secs on both BigQuery matchers (poll instead of one-shot)
wait on Pub/Sub publish futures before asserting

On the PR PostCommit runs, LeaderBoardIT passed on 3.10 / 3.11 / 3.12, but we still saw at least one empty-checksum fail (e.g. 3.14), so it looks reduced, not fully gone

I wasn’t able to dig further at the time because PostCommit was also hitting IAM permission issues (403 PermissionDenied on GCP APIs like Vertex AI / Recommendations AI, and earlier some Pub/Sub / BigQuery 403s), which made the suite red for unrelated reasons.

I’ll retrigger PostCommit on that PR and check the latest results

On the PR PostCommit runs, LeaderBoardIT passed on 3.10 / 3.11 / 3.12, but we still saw at least one empty-checksum fail (e.g. 3.14), so it looks reduced, not fully gone

there is a PR that might address this: #40297

Navigation Menu

Repository navigation

The PostCommit Python job is flaky #30513

240 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	events
	\| 'CalculateUserScores' >> CalculateUserScores(args.allowed_lateness)
	\| 'FormatUserScoreSums' >> beam.Map(format_user_score_sums)
	\| 'WriteUserScoreSums' >> WriteToBigQuery(
	args.table_name + '_users',
	args.dataset, {
	'user': 'STRING',
	'total_score': 'INTEGER',
	},
	options.view_as(GoogleCloudOptions).project))

The PostCommit Python job is flaky #30513

Description

Activity

shunping commented on Mar 11, 2024

shunping commented on Mar 11, 2024

damccorm commented on Mar 11, 2024

shunping commented on Mar 11, 2024

kennknowles commented on Apr 29, 2024

damccorm commented on Apr 29, 2024

kennknowles commented on Apr 29, 2024

Abacn commented on Apr 29, 2024

kennknowles commented on Apr 29, 2024

240 remaining items

github-actions commented on Sep 4, 2026

aIbrahiim commented on Sep 9, 2026

Amar3tto commented on Sep 10, 2026

github-actions commented on Sep 22, 2026

shunping commented on Sep 22, 2026

aIbrahiim commented on Sep 23, 2026

shunping commented on Sep 23, 2026

shunping commented on Sep 28, 2026

aIbrahiim commented on Sep 28, 2026

tvalentyn commented on Sep 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions