AWS: Add properties to configure SDK request checksum calculation and response checksum validation

[moomindani]

This PR adds two new S3FileIO properties to control the AWS SDK checksum behavior introduced in AWS SDK 2.30.0:

• s3.request-checksum-calculation: controls when the SDK calculates a checksum for S3 requests (when_supported / when_required)
• s3.response-checksum-validation: controls when the SDK validates checksums on S3 responses (when_supported / when_required)

When a property is not set, the corresponding AWS SDK default (WHEN_SUPPORTED since SDK 2.30.0) is left unchanged, so this PR introduces no behavior change for existing users.

Background

Since AWS SDK 2.30.0, the SDK calculates a CRC32 checksum for all requests that support one (RequestChecksumCalculation.WHEN_SUPPORTED by default) and sends it as x-amz-sdk-checksum-algorithm / x-amz-trailer: x-amz-checksum-crc32 headers. Some S3-compatible object stores (e.g. Dell ECS, GCS S3-compatible API) reject these headers, which breaks writes through S3FileIO. S3FileIO currently has no way to configure this SDK behavior; s3.checksum-enabled only controls the legacy Content-MD5 checksum. Setting s3.request-checksum-calculation=when_required stops the SDK from sending these headers.

This revives and reworks #15391 by @rcjverhoef (thanks!), preserving the original commit. The rework decouples the new knobs from s3.checksum-enabled into standalone properties that follow the SDK defaults when unset.

Design notes

• Defaults intentionally follow the AWS SDK defaults (no behavior change). Flipping the default to when_required for better out-of-the-box compatibility with S3-compatible stores would be a behavior change and can be discussed separately on the dev list.
• A follow-up PR (Add a property to configure the checksum algorithm in S3FileIO #17178) will add an s3.checksum-algorithm property to select the checksum algorithm per request.
• The properties are applied in all S3 client factories (default, assume-role, LakeFormation, S3FileIO default factory), for sync, async, and CRT async clients.

Testing

• Unit tests in TestS3FileIOProperties for parsing, defaults (unset leaves SDK defaults untouched), and builder application
• Verified against MinIO (testcontainers) and real AWS S3 with a header-capturing interceptor: with the default settings PutObject carries x-amz-sdk-checksum-algorithm: CRC32 and x-amz-trailer: x-amz-checksum-crc32; with when_required no checksum headers are sent. S3FileIO write/read/delete succeed under both settings.

Fixes #14439

[moomindani]

Not stale — this has had no review yet.

CI is green (40/40). Adds two S3FileIO properties for the checksum behavior introduced in AWS SDK 2.30.0 (s3.request-checksum-calculation, s3.response-checksum-validation), wired through all four client factories. Validated against MinIO and real S3, as described in the PR body.

Context: this is the fix for #14439, where the SDK's default when_supported breaks third-party S3-compatible stores that reject checksum headers. The commit from the original reporter is preserved.

@nastra @singhpk234 @amogh-jahagirdar you've been handling recent AWS changes — would any of you have time for a review?

[moomindani]

@danielcweeks would you be willing to take a look? This is the S3FileIO side of #14439 (S3-compatible stores rejecting the SDK's CRC32 trailer headers), reviving #15391 by @rcjverhoef with his commit preserved. It has been open six weeks without a review; I rebased it onto current main and CI is green.

The part I would most like a maintainer's read on is the default. This PR deliberately leaves the AWS SDK defaults in place when the properties are unset, so there is no behavior change and s3.request-checksum-calculation=when_required is strictly opt-in. The alternative — defaulting to when_required so S3FileIO works out of the box against Dell ECS, GCS's S3-compatible API and similar stores — is a behavior change I did not want to make unilaterally. If you think that is the better default, I am happy to change it here or take it to dev@.

#17179 stacks on this one and adds s3.checksum-algorithm; no need to look at that one until this shape is settled.

[moomindani]

@singhpk234 I mentioned you here on 7 August along with two others, so apologies for coming back — this time I have a more specific reason to ask you in particular. #15242, which you merged in April, is the same shape as this PR: an S3FileIOProperties knob that lets users opt out of an AWS SDK default that some S3-compatible stores reject. You have already ruled on that pattern once, which is why I am asking you rather than casting wider.

What this does: since SDK 2.30.0 the default RequestChecksumCalculation.WHEN_SUPPORTED attaches a CRC32 trailer to every PutObject even when s3.checksum-enabled=false, and Dell ECS and the GCS S3-compatible API reject it (#14439). The PR adds s3.request-checksum-calculation and s3.response-checksum-validation (when_supported / when_required), both leaving the SDK default untouched when unset, wired through all four client factories. It revives #15391 by @rcjverhoef, whose commit is preserved as the first one here, at his suggestion. Rebased onto current main, CI green.

The one question I would rather not settle myself is the default: keep the SDK default, as this PR does, so there is no behavior change and when_required is strictly opt-in — or flip it so S3-compatible stores work out of the box. I put the same question to @danielcweeks a week ago. Either answer works for me; I can change it here, or take the flip to dev@ as its own discussion.

A correction to my earlier comments here: I wrote that #17179 stacks on this PR and should wait for it. That is wrong. #17179 branches from main, shares no commits with this branch, and can be reviewed on its own, in either order. Sorry for the misdirection.

@steveloughran you pointed at the S3A settings on #14439 back in April; this is the Iceberg-side equivalent, and your view on which default is right would be valuable.

[moomindani]

@CTTY would you be able to review this one? You merged #16794 and #16853 in the AWS module, and this PR has had no review since it was opened on 12 July.

To make it easier to say yes, I am withdrawing the design question I raised earlier in this thread. This PR leaves every AWS SDK default untouched when the new properties are unset, so there is no behavior change and s3.request-checksum-calculation=when_required is strictly opt-in. Whether Iceberg should flip the default for S3-compatible stores is a separate question, and I will raise it separately rather than hold this up.

Why it matters: on #14439, users on Dell ECS and the GCS S3-compatible API have no in-Iceberg way out today. @patricklucas pinned the AWS SDK back to 2.29.52 in April to get PUTs working, and the only other workaround is the AWS_REQUEST_CHECKSUM_CALCULATION environment variable. The first commit here is @rcjverhoef's from #15391, preserved at his suggestion.

Rebased onto current main, CI green (44/44). #17179 is independent of this PR and can be reviewed in either order.

@danielcweeks to close the loop on my 22 August comment: the default question above is withdrawn from this PR, so there is nothing for you to rule on here.