Skip to content

[GH-3124] Share Spark parser and Arrow strategy sources - #3526

Draft
jiayuasu wants to merge 1 commit into
apache:masterfrom
jiayuasu:codex/oss-shared-spark41-sources
Draft

jiayuasu wants to merge 1 commit into
apache:masterfrom
jiayuasu:codex/oss-shared-spark41-sources

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Nov 6, 2026

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Yes, I have read the contributor rules and development guide.

Is this PR related to a ticket?

Related to #3124. Independent source preparation for Spark 4.2; this PR does not enable native types or add a Spark version.

What changes were proposed in this PR?

Move four existing Spark 4.1 parser/Arrow UDF sources into a shared source directory, with their contents unchanged. The Spark 4.1 module compiles that directory and still owns the resulting classes. A later Spark 4.2 module can reuse these 326 lines instead of copying them.

The review consists of four exact renames and two POM edits (40 XML lines). The POM changes preserve Spotless and Scalastyle coverage of the moved files. Runtime behavior, dependencies, schemas, serializers, and older-runtime module selection are unchanged.

How was this patch tested?

  • Clean Spark 4.1.1 / Scala 2.13 and Spark 3.5.8 / Scala 2.12 shaded builds passed.
  • All 18 tests passed across SQLSyntaxTestScala, Box2DCastSuite, Box3DCastSuite, and StrategySuite; the latter executes the Python Arrow UDF path.
  • All seven compiled classes generated by the four moved sources are byte-identical to the frozen unchanged OSS shaded artifact.
  • Static verification confirms identical source contents and unchanged dependencies, profiles, and module selection. The shared compile root is added only to the Spark 4.1 version module.
  • Spotless and Scalastyle passed, including the moved source directory.

Did this PR include necessary documentation updates?

No user-facing behavior changes; this only reorganizes version-shared sources.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant