Repository navigation
[Feature Request]: Support Apache Spark 4.x in Beam Spark Runner #36841
Description
What would you like to happen?
I would like Beam's Spark Runner to officially support Apache Spark 4.0. Spark 4.0 brings multiple improvements and new features, including performance enhancements, security updates, and updated APIs (e.g., improvements in shuffle, catalog features, Kubernetes operator, and structured streaming). As major platforms (such as Dataproc, EMR, and internal platforms) start adopting Spark 4.0, supporting this version will enable users to migrate without friction and leverage new Spark functionalities. Current Beam documentation lists support up to Spark 3.x, and 4.0 support is not mentioned. Attempting to run the Spark Runner on Spark 4.0 results in compatibility issues (such as dependency conflicts and API changes). Official CI matrix coverage, documentation, and dependency updates for 4.0 are needed for safe production use. This would help unblock organizations intending to move to Spark 4.0 and keep Beam competitive as Spark evolves.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
liferoad commentedon Nov 24, 2025
Spark 4.0 requires Java 17+.
Fix Spark 4.x compilation errors for Beam Spark Runner
Abacn commentedon Dec 1, 2025
We can set applyJavaNature(..., requireJavaVersion: JavaVersion.VERSION_17) for BeamModulePlugin.
nia-potato commentedon Mar 4, 2026
hi, any updates on this? we are currently blocked by this issue. thanks
tkaymak commentedon Apr 14, 2026
@Abacn is this issue still valid to be targeted for the 3.0 release? Can I help here?
docs: add Spark 4 runner entry to CHANGES.md
docs: add Spark 4 runner entry to CHANGES.md
[Spark Runner] Hoist Spark 3 structured-streaming to shared base, ado…
docs: add Spark 4 runner entry to CHANGES.md
15 remaining items
tkaymak commentedon Sep 15, 2026
Thanks for the suggestion, I looked into wiring
ValidatesRunnerfor streaming:ValidatesRunnertests in sdks/java/core useCreate.oforPAssert, which requireGroupByKey(Slice 5c) andImpulse(Slice 5d), so zero SDKValidatesRunnertests can pass on master yet.My plan is to introduce
validatesStructuredStreamingRunnerStreamingalongside Slice 5c and 5d, so standard SDK tests start validating the runner end to end as soon asGroupByKeyandPAssertland.[Spark][apache#36841] Add an opt-in Dataset-based backend for portabl…
[Spark][apache#36841] Add an opt-in Dataset-based backend for portabl…
[Spark][apache#36841] Prune dead DStream helpers and add streaming li…
[Spark][#36841] Prune dead DStream helpers and add streaming lifecycl…
[Spark][apache#36841] Select the Dataset backend by flag and add comp…
[Spark][#36841] Add an opt-in Dataset-based backend for portable pipe…
tkaymak commentedon Sep 25, 2026
Status update on slice 5b. Stateful ParDo, meaning state plus event time timers, now runs on the Spark 4 Structured Streaming runner through
transformWithState. It is coded and tested locally. The PR is not opened yet.There is one known blocker for 5c. Spark has no watermark hold. So output from an event time timer is emitted after the watermark has already passed the timer timestamp. That is harmless when only stateless ParDos follow. A downstream GroupByKey would drop that output as late. Before 5c supports a GroupByKey after a stateful ParDo, it needs a way to hold back output timestamps, or it has to reject that pipeline shape.
[Spark 4] Support stateful ParDo in Structured Streaming via transfor…
[Spark 4] Support stateful ParDo in Structured Streaming via transfor…
[Spark 4] Filter late data against the key's previous watermark in th…
tkaymak commentedon Oct 12, 2026
Status update. Slice 5b, stateful ParDo via
transformWithState, merged in #40281. The blocker from my last comment is resolved by #40447: late data is now filtered against the key's previous watermark, so timer output behind the shared watermark survives into a downstream stateful operator. The bound is per key rather than per batch, tracked in #40446. Next is windowed GroupByKey and Combine on the same processor, then Impulse and PAssert so a streaming ValidatesRunner suite can run.[Spark 4] Group the streaming stateful ParDo by encoded key bytes to …
[#36841] Spark 4 streaming: windowed GroupByKey and Combine via trans…
tkaymak commentedon Oct 23, 2026
Status update. Windowed GroupByKey and Combine are in with #40703, on top of the stateful ParDo fixes #40447, #40516 and #40550. Next is Impulse and PAssert, so a streaming ValidatesRunner suite can run, the CHANGES.md entry comes with it.