AWS: Add s3.checksum-algorithm property to configure the SDK checksum algorithm

[moomindani]

This PR adds a new S3FileIO property to select the checksum algorithm the AWS SDK uses for data integrity protection on S3 upload requests:

• s3.checksum-algorithm: valid values are the algorithms supported by S3 — CRC32, CRC32C, CRC64NVME, SHA1, SHA256, and (since AWS SDK 2.42.x) MD5, SHA512, XXHASH3, XXHASH64, XXHASH128 — case-insensitive. When unset, the AWS SDK default algorithm (CRC32) is used, so there is no behavior change.

Background

S3FileIO has no way to choose the checksum algorithm today; the SDK always uses its default (CRC32). Users may want CRC64NVME for efficient full-object integrity checks of multipart uploads, or SHA256 where a cryptographic hash is required. See #17178 for details.

Usage

Set the catalog property s3.checksum-algorithm to the desired algorithm name, e.g. with Spark:

spark-sql --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \
    --conf spark.sql.catalog.my_catalog.s3.checksum-algorithm=CRC64NVME \
    ...

Naming

The property is named s3.checksum-algorithm to sit alongside the existing s3.checksum-enabled, which also concerns upload checksums. An alternative considered was s3.write.checksum-algorithm, following the s3.write.* convention used for other upload-only properties (e.g. s3.write.storage-class), since this property only affects upload requests. Open to renaming if reviewers prefer that convention. For reference, Hadoop S3A calls the equivalent option fs.s3a.create.checksum.algorithm.

Implementation notes

• The algorithm is applied per request in S3OutputStream (PutObjectRequest, CreateMultipartUploadRequest, UploadPartRequest).
• Part checksums returned by UploadPart are passed on to CompleteMultipartUpload via CompletedPart; S3 rejects the completion with InvalidPart otherwise when an algorithm is specified.
• One interaction worth noting: setting a request-level checksum algorithm causes the SDK to calculate checksums even when the SDK request checksum calculation is configured to when_required (see AWS: Add properties to configure SDK request checksum calculation and response checksum validation #17177).

Testing

• Unit tests in TestS3FileIOProperties for parsing (case-insensitivity, invalid values, unset leaves the SDK default untouched)
• Integration test TestS3OutputStream#testWriteWithChecksumAlgorithm verifying the algorithm is applied to PutObject, CreateMultipartUpload, and UploadPart requests
• Verified against MinIO (testcontainers) and real AWS S3 with CRC64NVME and SHA256, for both single PUT and multipart uploads: request headers carry the configured algorithm (x-amz-sdk-checksum-algorithm / x-amz-checksum-* trailers), uploads and reads succeed, and content roundtrips correctly.

Fixes #17178

[moomindani]

Not stale — no review yet.

CI is green (39/39). Follow-up to #17177: adds s3.checksum-algorithm so users can select the algorithm the SDK uses for upload integrity protection (CRC32, CRC32C, CRC64NVME, SHA1, SHA256, and MD5 since AWS SDK 2.42.x), rather than only choosing when checksums are calculated.

This one stacks on #17177, so it makes sense to review that one first.

@nastra @singhpk234 @amogh-jahagirdar a review would be appreciated whenever you have a moment.

[moomindani]

@singhpk234 first, a correction to what I wrote here on 7 August: I said this PR stacks on #17177 and that it made sense to review that one first. That is not true — this branch comes off main, shares no commits with #17177, and stands on its own. I am sorry for parking it behind a dependency that does not exist.

On its own terms: s3.checksum-algorithm lets users choose the algorithm the SDK uses for upload integrity protection (CRC32, CRC32C, CRC64NVME, SHA1, SHA256, plus the MD5 / SHA512 / XXHASH3 values S3 added in April 2026), applied per request in S3OutputStream for PutObject, CreateMultipartUpload and UploadPart. Part checksums from UploadPart are carried into CompletedPart; without that, CompleteMultipartUpload fails with InvalidPart once an algorithm is set. Verified against MinIO and real S3 with CRC64NVME and SHA256, single-part and multipart. Fixes #17178, CI green.

One open naming question, also in the PR body: s3.checksum-algorithm, sitting next to the existing s3.checksum-enabled — versus s3.write.checksum-algorithm, matching s3.write.storage-class, since this only affects uploads. Hadoop S3A calls the equivalent fs.s3a.create.checksum.algorithm. No strong preference on my side; I will rename if you have one.

[moomindani]

@CTTY this is the companion to #17177 and independent of it — separate branch, no shared commits, reviewable on its own and in either order. s3.checksum-algorithm selects the algorithm the SDK uses for upload integrity protection, applied per request in S3OutputStream for PutObject, CreateMultipartUpload and UploadPart. Open since 13 July, no review, CI green (44/44).

There is one naming question in the PR body — s3.checksum-algorithm, next to the existing s3.checksum-enabled, versus s3.write.checksum-algorithm, matching s3.write.storage-class, since this only affects uploads. Either works for me and it is a one-line rename, so please do not let it block the review.