Skip to content

Preserve geometry column order in GeoArrow schema inference - #3530

Draft
jiayuasu wants to merge 2 commits into
apache:masterfrom
jiayuasu:codex/oss-geoarrow-column-order
Draft

jiayuasu wants to merge 2 commits into
apache:masterfrom
jiayuasu:codex/oss-geoarrow-column-order

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this PR?

GeoArrow schema inference puts multiple geometry columns in the wrong positions. For example, [first, second, id] becomes [second, first, id], and [first, id, second] becomes [first, second, id].

Record each geometry field's actual DataFrame position using enumerate. The old counter advanced only for non-geometry fields. Add two pure schema regression cases covering adjacent and separated geometry columns, including the retained scalar and geometry field types. Update the existing Arrow integration test to expect the input order and use distinct geometry-column values, verifying their association with each row name.

How was this patch tested?

  • Both new cases failed on unchanged Apache base 20efe7d4c4f with the incorrect column order.
  • Both passed after the fix with PySpark 4.1.1 and 4.2.0 Python dependencies.
  • The two pure schema tests call schema inference without starting Spark; the test preserves gateway identity, and the standalone runs verified no gateway existed before or after execution.
  • CI follow-up (b4967e7db037aa01bcee06a940c7c433c7deb3d9): reproduced the original stale expected-order assertion on Spark 3.5.8/Shapely 1.8.5, then all 6 focused tests passed (four Arrow input integration cases and two schema-position cases). The integration test now checks both column order and distinct geometry values for each row name. Verified runtime class origins and unchanged compact SQL serialization. Local Python was 3.11.13; the failing CI lane used Spark 3.5.0/Python 3.8. Fresh current-head CI remains pending.
  • Repository-pinned Black, pre-commit and automatic commit hooks passed.

Did this PR include necessary documentation updates?

No public API changes; no documentation update is needed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant