[jiayuasu]

Did you read the Contributor Guide?

Yes, I have read the contributor rules and development guide.

Is this PR related to a ticket?

Related to #3124. Required prerequisite for the Spark 4.2 integration in #3391. Depends on #3524 to reuse its public Shapely WKB compatibility helper; merge #3524 first. Both target master because their branches live in the contributor fork. Until #3524 merges, this PR includes its five files; the incremental SQL migration is 17 files, including the CI worker and GeoPandas expectation updates.

What changes were proposed in this PR?

Replace compact geometry bytes at the JVM/Python SQL boundary with public JTS/Shapely EWKB across supported Spark versions. This makes SQL geometry transport use standard WKB while leaving the RDD/Kryo, spatial-index, Python RDD and Flink formats unchanged. S2 geography and its JVM UDT format remain unchanged.

Keep the existing SQL serializer APIs, JVM coordinate-layout validation, supported dimensions, dimensional empties, SRIDs, null handling and direct SerdeAware object evaluation. The JVM writer rejects explicitly conflicting layouts within polygons or multipart geometries, while general collection members retain independent layouts. Python follows public Shapely WKB, which preserves mixed member layouts instead of the old compact SQL serializer's single-layout promotion: an XY/XYZ MultiPoint remains distinct from an all-XYZ MultiPoint with a NaN first Z in geom_equals_identical. Reuse the shared Python WKB helper from #3524.

This changes persisted ordinary-Parquet GeometryUDT payloads. Existing compact UDT data must be exported as EWKB with the old runtime before upgrading; the new migration guide includes both commands. Standard GeoParquet and explicit WKB/EWKB columns are unaffected. The guide also records public-writer NaN canonicalization, NaN/NaN point normalization to empty, and the full signed 32-bit EWKB SRID field.

Spark 4.2 native types and Spark's default native serializer are introduced separately in #3391. This PR adds no native codec or serializer replacement.

Measured performance impact

The public JTS implementation is substantially slower than the existing compact SQL serializer in isolated boundary microbenchmarks. The all-version SQL migration incurs these costs; this is not a performance optimization.

Operation	Compact baseline, ns/op range	Public EWKB, ns/op range	Median paired latency ratio	Allocation ratio
XY point encode	9.9–10.4	80.8–81.7	8.04×	3.67×
XY point decode	12.0–15.1	75.8–79.6	5.59×	3.36×
256-vertex XY polygon encode	217.5–227.2	6090.6–6157.5	27.71×	4.98×
256-vertex XY polygon decode	619.0–656.3	5881.9–6125.5	9.55×	2.02×

Measured against unchanged Apache base 20efe7d4c4ff13dac154181ace13818f29087991 and the JVM implementation in 55ef4d066cb (unchanged by the subsequent Python-helper consolidation). Six paired fresh JVM runs, alternating before/after order, Corretto 17, fixed 1 GiB heap and two active CPUs, prebuilt geometries with SRID 4326, two-second prewarm, three warmup and five measured rounds, one million point or twenty thousand polygon operations per round, and a retained-result ring. All 12 forks are included; correctness comparisons found zero differences for the measured fixtures. CPU time and allocation corroborate the slowdown.

These are local serializer-only measurements, not end-to-end query, Python, or Spark 4.2 native-serializer timings. Tools and raw traces are kept outside the repository.

How was this patch tested?

• Spark 3.5.8/Scala 2.12, Spark 4.0.2/Scala 2.13 and Spark 4.1.1/Scala 2.13: 154 Scala tests and 18 Java tests passed on each. Includes SQL serializer, predicate, SerdeAware, Python framing, SRID/factory-SRID behavior and unchanged compact geometry/index controls. New independent WKB fixtures failed on the old transport before implementation.
• Spark 4.1.1 actual Python geometry/Arrow interoperability: 88 passed; RDD/DataFrame boundary controls: 4 passed.
• After integrating [GH-3124] Fix Python geography SQL serialization #3524 and consolidating the Python helper: 102 passed with actual Spark 4.1.1/Shapely 2.1.2, covering geometry, geography, vectorized Arrow callbacks and JVM/Python transport together. The frozen OSS shaded artifact hashes and unchanged JVM source/test files were rechecked.
• Consolidated pure helper/UDT fixtures: 61 passed on Shapely 2.1.2; 58 passed, 3 unsupported-M skips on Shapely 1.8.5. GEOS <3.9 branches use explicit version/writer-failure simulations; no actual GEOS <3.9 runtime is claimed.
• Follow-up CI correction (e7c7af6ab782): the embedded Pandas UDF test workers now use SQL EWKB and the Java workflow imports Python from this checkout. The exact previously failing StrategySuite passed on Spark 3.5.8, 4.0.2 and 4.1.1 after clean module builds. The stale worker used the compact RDD decoder; production RDD serialization is unchanged.
• Python CI follow-up (6974170ca13256b8651f90400b03e02579ef7424): a fresh shaded Spark 4.1.1 build and Shapely 2.2.0 reproduced both original failures, then both corrected tests passed against the same artifact. Tests now assert both-NaN points become empty while partial-NaN points remain nonempty, preserve mixed MultiPoint child dimensions [2, 3], and check both negative mixed/homogeneous and positive homogeneous identity. Shapely 2.1.2/2.2.0 pure probes produced identical fixture WKB; this was an outdated compact-format expectation, not a Shapely-version regression. Production is unchanged. Local Python was 3.12.11; full current-head CI remains pending.
• Spotless, Scalastyle, repository-pinned Black and commit hooks.

Did this PR include necessary documentation updates?

Yes. Added the SQL geometry serialization migration guide and linked it from Scala installation instructions.