[ErikBPF]

Which issue does this PR close?

Closes #7455.

Rationale for this change

The Spark 4.2 profile currently inherits Delta 4.3.1, which targets Spark 4.1. Delta 4.4.1 publishes the matching delta-spark_4.2_2.13 artifact.

Hadoop 3.5 also introduces fs.s3a.encryption.context. Treating it as an unconditional discovery-test exemption is unsafe: a local Delta shallow clone can reference S3 data files without initializing that bucket during metadata reads. A real MinIO reproducer showed stock Spark rejecting malformed bucket context while CometDeltaNativeScan returned all four rows. The previous 281-case contrib run passed and missed that shape.

What changes are included in this PR?

• Pair Spark 4.2 with io.delta:delta-spark_4.2_2.13:4.4.1, leaving older profile pins unchanged.
• Add the Spark 4.2 Delta contrib lane to nightly and run-delta-tests routing, including required-check aggregation. Run required MinIO coverage on Spark 4.2 as well as 3.5.
• Add a dedicated context admission gate for selected S3 data and deletion-vector buckets. It uses existing propagated alias and credential-provider resolution and declines nonempty or unverifiable effective context without printing values. SSE-C/CSE decline precedence is unchanged.
• Conservatively fall back to Spark for valid configured contexts too, until Comet reproduces Hadoop's setup validation. Unconfigured reads retain the native Delta scan. Document this behavior.
• Add the cold metadata failure regression, provider/alias/redaction checks, and a MinIO local-clone regression covering valid and malformed global, short-bucket and long-bucket context settings. The integration test asserts external S3 AddFile paths and an unconfigured native-scan control.

The separate kernel-backed -Pcontrib-delta surface is unchanged; this pairing is for contrib/delta-spark under -Pdelta.

How are these changes tested?

Validated on Apollo with a freshly built matching JNI. Production code is unchanged between the full-run head fea770f91 and review-follow-up head 60deafe942261abba4577645d0478a0ca3f10007:

• Full Spark 4.2 / Delta 4.4.1 contrib run at fea770f91: 287 passed, zero failures/errors/cancellations/skips. Native scan: 119; DML: 3; MinIO: 4; admission/reference checks: 161.
• At 60deafe94, both complete affected suites pass 166 cases, zero failures/errors/skips. This includes a planner regression with local data and an S3 DV, plus all four MinIO cases. Mutations dropping DV URIs or moving the context gate before the algorithm gate each fail that regression as expected.
• Tests now detect Hadoop’s S3_ENCRYPTION_CONTEXT capability rather than a Spark version string. The cold driver test is documented as metadata-setup parity, and the MinIO cache setting explains why fresh filesystem initialization is required. Fallback documentation explicitly includes older Hadoop profiles.
• Spark 4.2 root-reactor test compilation and Spark 3.5 / Delta 3.3.2 root-reactor compilation with -Pstrict-warnings pass.
• The unchanged original cross-store reproducer fails before the guard and passes afterward. A mutation that changes only the captured Comet error message causes exactly the parity regression to fail.
• CI routing assertion RED/GREEN, suite registry, Actionlint and Spotless checks pass.
• Independent review and Hackerman source/static review completed. The bot did not independently execute JVM tests. Its review of the test-only maintainer follow-up is pending.

(cd native && cargo build --locked --profile ci -j 8)
mkdir -p native/target/release
cp native/target/ci/libcomet.so native/target/release/libcomet.so
./mvnw -B -Prelease -Pspark-4.2,delta test \
  -Dtest=none -DwildcardSuites=org.apache.comet.contrib.delta

The observed run used a 2 GiB test JVM and ActiveProcessorCount=2, with the repository's module-opening arguments, and required MinIO through COMET_DELTA_S3_REQUIRED=1. Archive checkouts skipped Git metadata extraction only.

These are local Apollo results. The upstream draft currently reports CI (fork) neutral; fork workflow dispatch was denied with HTTP 422. No GitHub runtime success is claimed.

[ErikBPF]

@dwsmith1983 The upstream draft is now open, and I addressed your fork review in 60deafe94:

• A real planner regression covers local data with an S3-only DV bucket and checks SSE-C/CSE reason precedence. Removing DV URIs or reversing the gate order each makes it fail as expected.
• Both test guards now detect the Hadoop key instead of a Spark version string.
• Comments clarify the cold driver's scope and why the MinIO test disables filesystem caching. Docs explicitly cover conservative fallback on older Hadoop profiles.

All 166 cases in the two affected suites pass, including four MinIO cases, as do Spark 4.2 and Spark 3.5 strict compilation. Production code is unchanged from the earlier 287-case full run. Could you review the follow-up coverage when you have time? I am keeping this as a draft; parser-based admission for valid contexts remains separate work.

[dwsmith1983]

Could you review the follow-up coverage when you have time?

Looks good to me. The new DV test fails the way it should: with dvUris dropped from the gate call, the context-only DV bucket gets a native scan, and with the gates swapped, the SSE-C/CSE cases report the context reason instead of the algorithm. The baseline really reaches a native claim, since planning only decodes the DV descriptor and never opens the file. supportsS3EncryptionContext picks the right profiles today: the constant is in the hadoop-aws 3.5.0 that Spark 4.2 pulls in, and not in the 3.3.4 and 3.4.x jars the 3.5, 4.0 and 4.1 profiles use.