[Task]: Add perf tests #38559
Description
What needs to happen?
For context, please see https://s.apache.org/beam-delta-lake-source.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
durgaprasadml commentedon May 30, 2026
Hi @chamikaramj, I would like to work on this issue.
I’ve been analyzing the Delta Lake source implementation and the existing Beam IO integration/performance test patterns. My plan is to:
I’ll follow the existing Beam IO connector architecture and keep the implementation modular and contribution-ready.
chamikaramj commentedon May 30, 2026
Sounds good. As discussed in the PR, let's wait for the source implementation with SDF support (#38706) to go in first and then you can add perf testing on top of that.
chamikaramj commentedon May 30, 2026
For perf testing, let's make sure that that we deploy a real Delta Lake instance with a large input size (for example, 100 GB).
We should also use a distributed Beam runner instead of a local runner so that the work get parallelized.
Results can be published to here: https://metrics.beam.apache.org/d/1/getting-started?orgId=1&viewPanel=123125
Documentation on testing: https://beam.apache.org/documentation/io/testing/
durgaprasadml commentedon May 30, 2026
I’ll wait for the SDF-based source implementation in #38706 to land first and then continue the perf-testing work on top of that implementation.