Skip to content

[GH-3124] Isolate DataFrame and spatial RDD geometry boundaries - #3525

Draft
jiayuasu wants to merge 1 commit into
apache:masterfrom
jiayuasu:codex/oss-spatial-boundary-preparation
Draft

jiayuasu wants to merge 1 commit into
apache:masterfrom
jiayuasu:codex/oss-spatial-boundary-preparation

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Nov 6, 2026

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Yes, I have read the contributor rules and development guide.

Is this PR related to a ticket?

Related to #3124. This is a standalone preparation step; it does not enable Spark 4.2 native types.

What changes were proposed in this PR?

Centralize the geometry representation boundary used by Adapter, StructuredAdapter, and DfUtils in a small version-selected SpatialTypeSupport implementation. The existing implementation keeps GeometryUDT schemas and JTS values. A later Spark 4.2 change can supply native conversion at this boundary without duplicating adapter logic.

Preserve explicit GeometryUDT schemas, SRIDs, user data, full source rows, geography columns, and the existing null internal-geometry carrier. SQL, RDD/index, and S2 serialization remain unchanged. The build selects the legacy source by default and includes it in formatting/style checks.

Seven files: the helper, three callers, two build files, and one focused test suite.

How was this patch tested?

  • Five new boundary checks passed against unchanged production code before applying the refactor.
  • Spark 3.5.8 / Scala 2.12: 30 Scala and 10 Java adapter tests passed.
  • Clean Spark 4.1.1 / Scala 2.13: 30 Scala and 10 Java adapter tests passed.
  • Both runs include the existing adapterTestScala, structuredAdapterTestScala, and adapterTestJava suites. One existing ignored Scala test remains ignored.
  • Spotless, Scalastyle, and git diff --check passed. Review found no changes to the RDD/index formats or geography codec.

Validation command: mvn -pl spark/common -am verify with the appropriate Spark/Scala profile, the three selected Scala suites, and -Dtest=adapterTestJava (the 4.1 run also used clean).

Did this PR include necessary documentation updates?

No user-visible behavior changes. The new helper documents the version-selected boundary.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant