Spark: Restore 4.2 commit history

[manuzhang]

Summary

• Revert the squashed Spark 4.2 merge (031135677395a3fbc90a2e33f58a9fa3983335f5) and its follow-up cleanup (e76d63584d7f83b102026749e1ae0f91813cb78e).
• Replay the changes from Spark: Add support for 4.2.0 #14984 as three commits: move spark/v4.1 to spark/v4.2, copy it back, and add Spark 4.2 support. The support commit includes all follow-up fixes.
• Carry compatible Spark 4.1 changes that landed after the original branch point into Spark 4.2: executor-side delete-file caching for rewrite actions (Spark 4.1: Add rewrite option to enable executor cache for delete files #17868) and try-with-resources for StreamingOffset JSON generation (Core, Hive, Spark: Auto close JsonGenerator using try-with-resources #17931).
• Keep the Spark 4.1-only TableInfo API changes from Spark 4.1: Mark overrides of deprecated Spark API as deprecated #16028 out of Spark 4.2, where the required CatalogV2Util signatures are unavailable.

The cleanup from #17952 remains present because the replay starts from the current Spark 4.1 sources.

Merge Requirement

This PR must not be squash-merged because squashing would discard the commit and rename history it restores. Please use a merge commit or rebase merge.

Validation

• ./gradlew --no-daemon :iceberg-spark:iceberg-spark-4.2_2.13:assemble :iceberg-spark:iceberg-spark-extensions-4.2_2.13:assemble :iceberg-spark:iceberg-spark-runtime-4.2_2.13:assemble
• ./gradlew --no-daemon :iceberg-spark:iceberg-spark-4.2_2.13:test --tests org.apache.iceberg.spark.TestSparkExecutorCache --tests org.apache.iceberg.spark.actions.TestRewriteDataFilesAction
• git diff --check upstream/main...HEAD
• Verified spark/v4.1 is unchanged from upstream/main

---

AI Disclosure

• Model: GPT-5
• Platform/Tool: Codex
• Human Oversight: partially reviewed
• Prompt Summary: Revert the squashed Spark 4.2 commits and replay the original commits from PR Spark: Add support for 4.2.0 #14984 while preserving current Spark 4.2 compatibility.

[manuzhang]

@szehon-ho @huaxingao @nssalian This is to fix the Spark 4.2 commit history and should be Rebase and merge. Here are the detailed changes. We'd better move fast to avoid further conflicts and complexity. Please take a look. Thanks!

Summary

• Revert the squashed Spark 4.2 merge (031135677395a3fbc90a2e33f58a9fa3983335f5) and its follow-up cleanup (e76d63584d7f83b102026749e1ae0f91813cb78e).
• Replay the changes from Spark: Add support for 4.2.0 #14984 as three commits: move spark/v4.1 to spark/v4.2, copy it back, and add Spark 4.2 support. The support commit includes all follow-up fixes.
• Carry compatible Spark 4.1 changes that landed after the original branch point into Spark 4.2: executor-side delete-file caching for rewrite actions (Spark 4.1: Add rewrite option to enable executor cache for delete files #17868) and try-with-resources for StreamingOffset JSON generation (Core, Hive, Spark: Auto close JsonGenerator using try-with-resources #17931).
• Keep the Spark 4.1-only TableInfo API changes from Spark 4.1: Mark overrides of deprecated Spark API as deprecated #16028 out of Spark 4.2, where the required CatalogV2Util signatures are unavailable.

The cleanup from #17952 remains present because the replay starts from the current Spark 4.1 sources.

[nssalian]

Let's get this merged in so we can clean up the history and put the 4.2 baseline back. Thanks @manuzhang

[szehon-ho]

Thanks, sorry that was my fault!

[szehon-ho]

Looks like this is going to be a bit hard.

Looks like the current conflict is caused by #17972 updating sqlite-jdbc.

What do you think @manuzhang , can you rebase again and then me or @huaxingao can try again? I think only file that would conflict outside 4.2 line is gradle/libs.version

[manuzhang]

@szehon-ho @huaxingao It should be good for Rebase and merge now.

[szehon-ho]

ok ill try to do it tonight, thanks again! @huaxingao can also do it if the CI finish

[szehon-ho]

"Rebase and merged", thanks @manuzhang! Let me know if anything else is wrong

[manuzhang]

@szehon-ho Thanks for the prompt action. I excluded Spark 4.2 from source release in this PR, but that would crash our source release process. To fix it, I've opened #18024 to revert the change. Please help review. cc @huaxingao @nssalian