feat(artifact-cas): verify uploaded content against the declared digest by javirln · Pull Request #3365 · chainloop-dev/chainloop

javirln · 2026-08-25T06:24:20Z

The Artifact CAS keyed stored artifacts by their client-declared SHA256 without ever
checking that the uploaded bytes hash to that digest. Only the OCI backend verified, via
its own layer-digest check. On S3, S3 access point and Azure Blob any identity holding
artifact:upload could store arbitrary content under an arbitrary digest key.

The CAS now streams each upload to a local staging file, hashes it while receiving, and
hands the backend a file only once its digest matches the one the client declared. A
mismatch is rejected with InvalidArgument and nothing reaches the backend.

What changed

A single spill-then-upload path shared by all backends, replacing the per-backend
streaming branch and the StreamingUploader interface. Verification is no longer
something each backend implements, and any future backend inherits it.
Staging files are unlinked as soon as they are created and used through the open
descriptor, so the kernel reclaims the space whenever the process exits, including when
it is killed mid-upload. Nothing needs to sweep or reap the staging volume.
staging_dir is required: the CAS refuses to start without one and proves it is
writable before serving traffic, so a misconfiguration fails at boot rather than
producing a healthy service that rejects every upload.
Helm: a dedicated staging emptyDir mounted unconditionally at a fixed path. It sets no
sizeLimit, since exceeding one evicts the Pod rather than failing the upload that
overran it.
Staging on disk keeps the service's own memory use independent of artifact size. The
OCI backend still requires the whole layer in memory, unchanged here.
The S3 backend relies on this service-layer verification rather than a per-object
checksum precondition, which cannot be expressed for multipart uploads.

Addresses PFM-6988.

This contribution was produced with the assistance of Claude Code.

chainloop-platform · 2026-08-25T06:24:29Z

AI Session Checks — ⚠️ 1 session(s) missing

Missing AI Coding Sessions

We detected commits in this PR that were AI-assisted, but the matching Chainloop Trace session(s) could not be found in Chainloop.

Please make sure the AI coding session evidence has been sent by the Chainloop CLI, or add the skip-ai-session label to this PR to bypass this check.

Learn more about Chainloop Trace.

Security Checks — ⚠️ 1 failing

✅ `secret-scan`

Status	Policy	Messages
✅ Passed	`secrets-detection`	-

✅ `sast-scan`

Status	Policy	Messages
✅ Passed	`owasp-top10-2025`	-
✅ Passed	`sast`	-
✅ Passed	`cwe-top25`	-
✅ Passed	`cwe-top26-40-cusp`	-

⚠️ `iac-scan` — 1 failing

Status	Policy	Messages
⚠️ Failed	`iac-misconfiguration`	Base64 High Entropy String in "app/artifact-cas/configs/config.devel.yaml" (error) Base64 High Entropy String in "deployment/chainloop/values.yaml" (error)

PR info

Status	Policy	Messages
✅ Passed	`pr-min-approvals`	-
✅ Passed	`pr-description-required`	-
✅ Passed	`pr-user-story-linked`	-

⏭️ 2 scans not applied

Scan	Reason
`vulnerability-scan`	no manifest/lockfile changed
`github-actions-scan`	no workflow files changed

View attestation ↗
View attestation ↗

Powered by Chainloop and Chainloop Trace

cubic-dev-ai

All reported issues were addressed across 25 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

migmartri

In general it looks good to me, see my comments, some thoughts though

I'd make this mode to be the only mode, not configurable, if it's going to be tmpDir that's ok
What happened to the streaming capabilities you added in the other PR, why is it no longer needed? can we confirm that during the end to end input->stagingFS->blobFS we never load the whole file in memory?

javirln · 2026-09-14T13:36:15Z

Posted by Claude Code on behalf of @javirln.

Verification

Unit / integration (app/artifact-cas, pkg/blobmanager)

Digest mismatch rejected with InvalidArgument, nothing sent to the backend.
Staging cleanup on every exit path, table-driven: success, digest mismatch, backend
failure, size cap exceeded, client disconnect.
staging_dir startup contract: unconfigured rejected, missing directory created,
existing-but-unwritable rejected, probe file removed.
Upload refused when the staging dir is unset, so os.CreateTemp cannot silently fall
back to the OS temp dir.

End to end against a real S3 (MinIO, testcontainers, runs in CI)

12 MB multipart upload: the object in the bucket re-hashes to the declared digest.
Tampered digest: InvalidArgument, and NoSuchKey at the canonical key.

End to end with the real CLI against a locally built control plane + CAS + MinIO

25 MB and 1 GB uploads stored with an exact digest match; download round-trip identical.
Memory stays flat: peak RSS +11.6 MB for a 250 MB artifact, +29 MB for a 1 GB artifact.
1 GB upload observed spilling to disk, then reclaimed.

End to end on a kind cluster, chart-deployed, scratch image built from this branch

Pod runs with readOnlyRootFilesystem: true, runAsUser: 1001, the jwt secret at
/tmp and the staging emptyDir alongside it; 40 MB upload verified in the bucket.
Removing staging_dir from the deployed config crash-loops the pod with
staging_dir is required instead of producing a healthy pod that fails every upload.

Interrupted uploads

Client killed mid-upload: partial staging file released in 12 ms, nothing written to the
bucket.
CAS killed mid-upload: before unlinking, a 403 MB partial survived a restart; after,
the directory holds nothing and the space is reclaimed by the kernel.

javirln · 2026-09-14T13:37:50Z

Claude Code, on behalf of @javirln.

@migmartri - the two points from your review, answered in the threads but summarised here:

Make staging the only mode. Done. cas.staging.enabled and the configurable mount path
are both gone; the emptyDir is mounted unconditionally at a fixed path, and staging_dir is
required rather than falling back. It was worse than a preference: the fallback resolved to
/tmp, which is the read-only secret mount, so the CAS booted healthy and failed every
upload.

What happened to streaming. SupportsStreaming had a single caller - the branch
choosing between the piped path and the buffered one. Verification must happen before any
bytes reach a backend, so there is now one path and the flag had nothing left to select.
Streaming is not lost: the backend still gets a seekable *os.File and reads it in bounded
parts.

End to end memory. Bounded for the object stores, measured: +11.6 MB peak RSS for a
250 MB artifact, +29 MB for 1 GB. The exception is OCI, which still reads the whole layer
into memory - unchanged here, that is PFM-6924.

cubic-dev-ai

All reported issues were addressed across 27 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

jiparis

Looks good to me. Thanks @javirln

The Artifact CAS keys stored artifacts by their client-declared SHA256. This change adds end-to-end content-integrity verification to the upload path so the canonical key is only ever written from bytes CAS has confirmed hash to that digest, across all object-store backends (S3, S3 access point, Azure Blob), in addition to the OCI backend's existing layer-digest check. The CAS service streams each upload to a temporary file on a local staging volume, computes its SHA256 while receiving, and verifies it against the declared digest before handing the verified file to the backend. A mismatch is rejected with InvalidArgument and nothing is written to the backend. The upload path is consolidated into a single spill-then-upload flow shared by all backends, replacing the per-backend streaming branch and the StreamingUploader interface. The staging directory is configured through staging_dir and is required: the CAS refuses to start without one, and proves it is writable before serving traffic, so a misconfiguration surfaces at boot rather than as a healthy service that fails every upload. The Helm chart mounts a dedicated emptyDir for it. Each staging file is unlinked as soon as it is created and the upload proceeds through the open descriptor, so the kernel reclaims the space whenever the process exits, including when it is killed mid-upload, and the volume cannot accumulate partial artifacts. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 5d0fc605-25b0-4bb3-900c-2c2a8fb4922f

migmartri · 2026-09-20T21:40:25Z

      type: "RuntimeDefault"
-
+
+  ## CAS upload staging.


this is outdated no?

cubic-dev-ai Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread deployment/chainloop/values.yaml Outdated

Comment thread deployment/chainloop/templates/cas/deployment.yaml Outdated

Comment thread app/artifact-cas/cmd/main.go

Comment thread deployment/chainloop/README.md Outdated

migmartri reviewed Aug 26, 2026

View reviewed changes

javirln force-pushed the javirln/PFM-6988 branch 4 times, most recently from d7ae4fa to 93c9dbf Compare September 14, 2026 13:34

javirln marked this pull request as ready for review September 14, 2026 13:39

cubic-dev-ai Bot reviewed Sep 14, 2026

View reviewed changes

Comment thread app/artifact-cas/internal/service/service.go Outdated

javirln force-pushed the javirln/PFM-6988 branch from 93c9dbf to 901b2d8 Compare September 14, 2026 13:46

javirln requested review from a team, jiparis and migmartri September 15, 2026 06:13

jiparis previously approved these changes Sep 15, 2026

View reviewed changes

Comment thread app/artifact-cas/internal/service/bytestream.go Outdated

Comment thread app/artifact-cas/internal/service/bytestream.go

Comment thread pkg/blobmanager/s3/backend.go Outdated

javirln dismissed jiparis’s stale review via afff50c September 16, 2026 03:40

javirln force-pushed the javirln/PFM-6988 branch 2 times, most recently from afff50c to 557ef34 Compare September 16, 2026 03:43

javirln requested a review from jiparis September 16, 2026 11:55

javirln force-pushed the javirln/PFM-6988 branch from 557ef34 to 5384cdb Compare September 18, 2026 07:58

jiparis approved these changes Sep 18, 2026

View reviewed changes

javirln merged commit 26a6cec into chainloop-dev:main Sep 18, 2026
15 of 17 checks passed

migmartri reviewed Sep 20, 2026

View reviewed changes

Comment thread deployment/chainloop/values.yaml

type: "RuntimeDefault"

## CAS upload staging.

migmartri Sep 20, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is outdated no?

This was referenced Sep 22, 2026

artifact-cas: verify downloads against the requested digest via disk staging before sending to the client #3460

Closed

feat(artifact-cas): verify downloads against the requested digest via disk staging #3461

Merged

Conversation

javirln commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Uh oh!

chainloop-platform Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AI Session Checks — ⚠️ 1 session(s) missing

Missing AI Coding Sessions

Security Checks — ⚠️ 1 failing

✅ secret-scan

✅ sast-scan

⚠️ iac-scan — 1 failing

PR info

⏭️ 2 scans not applied

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

migmartri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javirln commented Sep 14, 2026

Verification

Uh oh!

javirln commented Sep 14, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jiparis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

migmartri Sep 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

javirln commented Aug 25, 2026 •

edited

Loading

chainloop-platform Bot commented Aug 25, 2026 •

edited

Loading

✅ `secret-scan`

✅ `sast-scan`

⚠️ `iac-scan` — 1 failing

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading