[cavanaug]

Summary

Part 3 of the Firehose S3 delivery series tracked in #2708. Stacked on GZIP #2745 and its metadata-aware S3 writer: until #2745 merges, this diff also contains its commits, and the Snappy changes are the commits from feat(firehose): support Xerial Snappy delivery to S3 onward.

• Implement the modeled Snappy compression format using pure-Rust raw Snappy blocks (snap 1.1.2, BSD-3-Clause, no dependencies, compression only) in the Xerial/snappy-java container.
• Use .snappy, Content-Type: application/octet-stream, and Content-Encoding: snappy-java, preserving extension overrides, metadata, exact decoded bytes, and persistent restart behavior.
• Fill 32 KiB input blocks across record boundaries, including delimiter bytes, as snappy-java's SnappyOutputStream does; each block is compressed independently. Test decoding is independent and bounded.
• Include independently generated snappy-java fixtures, reproduction details, unit regressions, and official-SDK delivery/persistence/UpdateDestination coverage. Java is not a runtime or test dependency.
• Keep immediate delivery unchanged; size/time buffering follows in its own increment.
• Retain upstream fix(s3): complete flexible checksums across every upload path #2730's default CRC64NVME full-object checksums, with assertions covering stored Snappy bytes (not decoded input), persisted checksum values, and SDK GET/HEAD checksum metadata across restart.

Why the Xerial container

Three incompatible Snappy containers exist. A published probe of real Amazon Data Firehose (floci-io/floci#2328, eu-west-1, 2026-08-15) found that CompressionFormat: Snappy writes .snappy objects with Content-Type: application/octet-stream, Content-Encoding: snappy-java, and the Xerial header 82 53 4e 41 50 50 59 00; snappy-java 1.1.10.8's SnappyOutputStream reproduced the AWS object byte for byte, while HADOOP_SNAPPY used Hadoop block framing with Content-Encoding: hadoop-snappy (still rejected at admission here). That probe delivered one 690-byte payload, so it confirms the header, metadata, and records sharing a block; the 32,768-byte block size for larger payloads follows snappy-java's default. The container bytes were also checked against snappy-java 1.1.10.8's SnappyCodec/SnappyOutputStream source. Compressed bytes may differ from AWS's (encoder heuristics), but decoded bytes and framing match.

Test plan

• Verified on c5907ef3a (stacked on GZIP dfd71471c, upstream 774c772d6): 1,727 focused CloudFormation/Firehose/Logs/S3 unit tests, 86 rebuilt-server codec/admission/SES/checksum/persistence/encryption/write-side-effect SDK tests, one Athena regression, and 107 Firehose/SES/S3 conformance tests.
• Workspace all-target Clippy, formatting, documentation counts, and diff checks passed on that revision.
• A pre-publication code/design review checked the container bytes against snappy-java's source, re-verified the fixture and JAR SHA-256 values, and decoded the fixtures with a separate decoder; a STRIDE security review found no Snappy-specific issues (no untrusted Snappy is ever decoded; output is at most about 1.17× the input).
• Unrestricted workspace testing of the GZIP predecessor stopped on the unchanged API Gateway fixed-port timeout (expected 502, actual 504 at closed 127.0.0.1:54321), independently reproduced on upstream. Snappy's final focused run does not claim that its workspace gate passes; no test exclusions were added. Revalidate after rebasing onto the merged predecessor before submission.

Series context

Refs #2708; does not close the tracking issue. Depends on #2745; branch feat/firehose-snappy is stacked on feat/firehose-gzip. Once #2745 merges, I will rebase onto main so only the Snappy commits remain and revalidate. Size/time buffering follows, stacked on this PR, in #2747.

[vieiralucas]

Thanks a lot for this, @cavanaug. Snappy output that matches what real Firehose writes (the Xerial/snappy-java container, Content-Encoding: snappy-java, 32 KiB input blocks across record boundaries) is exactly the kind of detail people hit when reading delivered objects with Spark or Hadoop tooling, and the independently generated snappy-java fixtures plus the bounded test decoder make it easy to trust. I pushed a couple of follow-ups before merging: rebased onto main after #2745 landed (folding in its destination SSE-KMS support, so the docs no longer list KMS as rejected), and added assertions that Snappy objects carry the MD5 of the stored bytes as their ETag on GET, list, and HEAD. Merged. Really appreciate the contribution.