[jiayuasu]

Did you read the Contributor Guide?

Yes.

Is this PR related to a ticket?

Closes #3124.

What changes were proposed in this PR?

Use Spark 4.2's native GEOMETRY and GEOGRAPHY types with Spark's shipped spatial serializer. Sedona retains JTS geometry computation and its newer S2 geography implementation. Native boundary validation remains active, including nested calls and computed join keys. The five overlapping Spark ST functions remain registered to Spark.

The SQL/DataFrame boundary is separate from RDD and spatial-index serialization. Older Spark versions and explicit Sedona UDT schemas use the all-version public EWKB migration in #3528; RDD/Kryo, spatial-index, Python RDD and Flink formats remain unchanged. Direct and computed spatial join keys avoid the redundant Sedona SQL serialization/decode cycle. Spark's own encoding and validation are retained where required.

Native adapters cover file I/O, aggregate/container values, Python Classic/Connect, Arrow/UDF values and Sedona-owned GeoPandas boundaries. GeoParquet requires fixed per-column CRS on writes and rejects incompatible inferred CRS metadata; explicit ANY reads can preserve supported per-file SRIDs. Geography defaults distinguish omitted CRS from explicit invalid SRIDs. The migration guide documents the changed JVM/Python value contracts and file-I/O rules.

Review prerequisites

Independent fixes and preparation are separated for review:

• [GH-3124] Fix Python geography SQL serialization #3524: public Python geography WKB framing.
• [GH-3124] Isolate DataFrame and spatial RDD geometry boundaries #3525: common spatial boundary API preparation.
• [GH-3124] Share Spark parser and Arrow strategy sources #3526: share the unchanged Spark 4.1 source implementations.
• [GH-3124] Preserve S2 semantics in geography distance joins #3527: preserve S2 distance semantics when considering an index join.
• [GH-3124] Use public EWKB for SQL geometry transport #3528: migrate SQL geometry serialization across supported versions.
• Fix KNN input binding for computed geometry operands #3529: bind computed KNN operands by attribute identity.
• Preserve geometry column order in GeoArrow schema inference #3530: preserve GeoArrow geometry-column order.
• Prepare shared geometry result tests for native spatial values #3531: representation-neutral shared geometry-result tests.

The cumulative branch includes these prerequisites. They should land before the native integration; no PR has been merged by this work.

Performance scope

The isolated SQL serializer microbenchmarks are documented in #3528, including the approximately 8x point and 28x polygon encode slowdown of public JTS EWKB versus the previous compact representation. The all-version migration is intentional. Those measurements concern that SQL serializer change and are not end-to-end native Spark 4.2 query benchmarks. This PR does not install a substitute for Spark's serializer. Join tests verify the removed legacy round trip and required native validation; no unmeasured native speedup is claimed.

How was this patch tested?

Local validation of the integrated source tree (23ff9cf5e6dca41585531e9731a11c3bb33797fd; the history-preserving merge leaves the tested tree unchanged):

• Spark 4.2: full native module 85 Scala + 13 Java checks after the final geography ownership fix. Earlier shared-contract validation passed 61 Scala + 238 applicable Java checks, with one intended legacy-only Scala cancellation; that shared run preceded the ownership-only adjustment, which is covered by the final full native module and an independent mutable-value regression.
• Python 4.2 with installed public artifacts: 60 Classic/GeoPandas checks and 8 Connect checks, including strict spatial Arrow/UDF paths and bounded display. One empty ordinary NullType control permits standard Arrow fallback.
• Pure Python boundaries: 96 checks on 4.2; 69 checks on 4.1 with 27 intended native-only skips.
• Clean builds on Spark 3.5.8, 4.0.2 and 4.1.1. Shared compatibility suites passed 325 checks on each of 3.5 and 4.1; the embedded Python UDF strategy test passed on all three versions.
• Integrated Python 4.1: 29 GeoPandas/SQL transport checks, including all-NaN, mixed-dimensional, dimensional-empty and CRS controls. The isolated subset explicitly initializes Sedona before pandas-on-Spark, matching the full CI session setup.
• Connect-only distributions: four import-only checks each on fresh pyspark-client 4.1.1 and 4.2.0 environments, without a JVM.
• Prerequisites: KNN 192 checks and shared test preparation 686 checks across clean Spark 3.5 and 4.1 builds. GeoArrow ordering: reproduced the old integration assertion failure, then six checks passed on Spark 3.5.8/Shapely 1.8.5, including distinct values in each geometry column.

mvn -B -pl spark/spark-4.2 -Dspark=4.2 -Dscala=2.13 verify discovers the full native module. Focused shared suites and clean shaded-artifact builds were run separately for each supported profile. The final Python pass uses pytest tests/sql/test_native_spatial.py tests/geopandas/test_native_spatial.py for Classic and SPARK_REMOTE=local pytest tests/sql/test_native_spatial.py for Connect. Spotless, Scalastyle, repository hooks and git diff --check passed. Full remote CI for this replacement head is pending.

Runtime checks verify that the public Spark Geometry, Geography, WkbReader and WkbWriter class bytes are loaded unchanged. Exact commands and raw evidence are kept outside the repository.

Did this PR include necessary documentation updates?

The spatial type/serialization guide and Python/Scala installation pages document migration and verified upstream limitations. Native cache/persist and direct spatial grouping remain unsupported; raw pandas-on-Spark and native toPandas also have upstream conversion limitations. Sedona-owned adapters do not globally modify Spark's APIs. Spark 4.2 managed-runtime compatibility is not established by local OSS validation.

The native CI lane compiles the full reactor, runs applicable shared Java tests and selected shared spatial contract suites, discovers every native version-module suite, and exercises native Python Classic and Connect boundaries. Older runtime matrices retain their existing full coverage. Hadoop's MiniDFS WebHDFS fixture is excluded only on 4.2 because Hadoop's Jersey 2/javax server conflicts with Spark's Jersey 3/jakarta classes; older-runtime MiniDFS tests and native shapefile I/O remain covered. This is not a claim that every historical shared test or arbitrary pandas-on-Spark operation supports native spatial types.