[tkaymak]

Part of #36841.

This adds the streaming ValidatesRunner suite for the Spark 4 Structured Streaming runner, :runners:spark:4:validatesStructuredStreamingRunnerStreaming. It runs as part of :runners:spark:4:validatesRunner, so beam_PostCommit_Java_ValidatesRunner_Spark4 covers it. The suite runs with --streaming and stops each pipeline after 5 idle batches. The batch and streaming tasks now share one setup in spark_runner.gradle, the effective configuration of the batch task on Spark 3 and 4 is unchanged.

In streaming mode, a Flatten of no inputs is replaced by EmptyFlattenAsCreateFactory, as Dataflow does. It has to come before the Impulse override from #40766, because the replacement expands to an Impulse.

Exclusions are by category where one exists, grouped by the runner gap. Tests that fail on a gap without a category are excluded by name and cite their issue: ParDo with additional outputs #40949, Redistribute #40950, merging windows, side inputs and splittable DoFn under #36841, bounded source metrics under #32021.

Locally the suite passes with 191 tests, 0 failures and 8 skipped in about 22 minutes. Four Flatten and multiple apply tests that were excluded before now pass with the empty Flatten override and #40932. Full :runners:spark:4:test and :runners:spark:3:test pass. The Spark 4 ValidatesRunner trigger file runs it on this PR.

The CHANGES.md entry for Spark 4 streaming follows once this is approved.

[tkaymak]

R: @Abacn

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[Abacn]

Let's use the existing https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Java_ValidatesRunner_Spark4.yml since the tests are fast, most of the time spend on build

that is add the structured-streaming streaming task to here if spark major version is 4:

beam/runners/spark/spark_runner.gradle

Line 562 in 4b5b9c8

dependsOn validatesRunnerStreaming

[tkaymak]

Thanks, done in 65c3494. The streaming task now runs from validatesRunner when the Spark major version is 4, the separate workflow is removed and the Spark 4 ValidatesRunner trigger file is bumped so it runs here.

[Abacn]

Thanks!