Skip to content

fix: remote attestation state writes fail on second save during initialization #2908

Closed
#2909
Closed
fix: remote attestation state writes fail on second save during initialization#2908
#2909
Assignees
migmartri
Labels
bugSomething isn't workingcomponent/backend

Description

@migmartri
migmartri
opened on Mar 21, 2026

Problem

When the attestation crafter performs multiple state saves during initialization (e.g., running collectors that each update state), the second save fails with an OCC (Optimistic Concurrency Control) conflict.

The root cause is that AttestationStateServiceSaveResponse did not return the digest of the newly saved state. After a successful save, the client's UpdateCheckSum remained stale, so the next write sent an outdated base_digest that no longer matched the server-side state.

attestation_id=$(chainloop attestation init --workflow ${CHAINLOOP_WORKFLOW_NAME} --project ${CHAINLOOP_PROJECT_NAME} --release --remote-state -o json | jq -r .attestationID)
INF discovered AI agent config files agent=claude files=11
INF uploading ai-agent-config-claude-3801167160.json - sha256:f8e8a2449ae98ccc676de3dc6b41d0c3ee24e28ea7da8b3ad990df5f508ae827
INF successfully collected AI agent configuration agent=claude
ERR retrying in 652.072141ms error="evaluating attestation policies: failed to persist crafting state: failed to save state: rpc error: code = Aborted desc = saving attestation: failed to save attestation state: the provided base digest doesn't match the stored state, it might have changed, got=\"16b1ddff286db369d3256bb23b1ad1e7acf7954f2f88f07b2108861674692421\", want=\"e44e6cc538cf4e6010758567460080745c1393d60f28b3374f6b7666ed875b09\""
INF discovered AI agent config files agent=claude files=11
INF uploading ai-agent-config-claude-4821673.json - sha256:7490d637cef40c2f608f620aa5c7f58b5a9255fe48086e9433b33a3ed30b2400
INF successfully collected AI agent configuration agent=claude
INF discovered AI agent config files agent=claude files=11
INF uploading ai-agent-config-claude-365014333.json - sha256:c828347c1382cf90b75fcf140486275b5c147c5dcaea83e55a9c0ceacfc583d9
WRN collector failed error="adding AI agent config material for claude: failed to persist crafting state: failed to save state: rpc error: code = Aborted desc = saving attestation: failed to save attestation state: the provided base digest doesn't match the stored state, it might have changed, got=\"bfc6193105731b17a746772ec06df20454c64e701b7bd4cd2fc07cf76e355b55\", want=\"1fe2fccd9d55519460d56997f01e7ab618f56d05e87181c17f65314895a2a054\"" collector=ai-agent-config
ERR retrying in 677.93603ms error="evaluating attestation policies: failed to persist crafting state: failed to save state: rpc error: code = Aborted desc = saving attestation: failed to save attestation state: the provided base digest doesn't match the stored state, it might have changed, got=\"bfc6193105731b17a746772ec06df20454c64e701b7bd4cd2fc07cf76e355b55\", want=\"1fe2fccd9d55519460d56997f01e7ab618f56d05e87181c17f65314895a2a054\""
INF discovered AI agent config files agent=claude files=11

Activity

  1. migmartri
    added a commit that references this issue on Mar 21, 2026

    fix: return digest from attestation state save for OCC support

    79bf736
  2. linear
    added
    bugSomething isn't working
    component/backend
    on Mar 21, 2026
  3. migmartri
    self-assigned this
    on Mar 21, 2026

  4. migmartri commented on Mar 21, 2026

    @migmartri
    migmartri
    on Mar 21, 2026
    MemberAuthor

    Additional research

    Remote State Digest Mismatch Regression

    Summary

    A race condition in remote state attestation causes digest mismatch errors when collectors write materials and EvaluateAttestationPolicies subsequently tries to save state. The root cause: Remote.Write() never updated the in-memory UpdateCheckSum after a successful save, so any second write in the same flow sent a stale digest.

    Bug Mechanism

    1. AutoCollectPRMetadata (or RunCollectors) calls LoadCraftingState → gets digest D1
    2. addMaterial → stateManager.Write() sends D1 as BaseDigest, server saves and computes new digest D2
    3. EvaluateAttestationPolicies → stateManager.Write() still sends D1 (stale)
    4. Server compares D1 against stored D2 → conflict

    The fundamental flaw: Remote.Write() discarded the Save RPC response and never updated state.UpdateCheckSum. The AttestationStateServiceSaveResponse proto was empty — no digest field existed to return.

    Regression Timeline

    PR Date Effect
    #2617 2025-12-15 Added AutoCollectPRMetadata — single write at end of init, no issue
    #2765 2026-02-20 Regression introduced — added EvaluateAttestationPolicies after AutoCollectPRMetadata, creating two sequential writes without a Read in between
    #2843 2026-03-14 Worsened — collector framework added more writes before EvaluateAttestationPolicies
    Key Insight

    Before PR #2765, PR info worked with remote state because there was only one Write at the end of init. The second Write from EvaluateAttestationPolicies is what broke it.

    Fix

    Return the new digest from the Save RPC so Remote.Write() can update UpdateCheckSum immediately — same pattern as HTTP ETags after a successful PUT with If-Match.

    Changes

    • Proto: Added string digest = 1 to AttestationStateServiceSaveResponse
    • Data layer: Save() now returns (string, error), computing digest of newly stored state
    • Biz layer: Updated AttestationStateRepo interface and AttestationStateUseCase.Save
    • Service layer: Returns digest in the response
    • Client: Remote.Write() updates state.UpdateCheckSum from resp.GetDigest()

    Key Files

    • app/controlplane/api/controlplane/v1/attestation_state.proto
    • app/controlplane/pkg/data/attestationstate.go
    • app/controlplane/pkg/biz/attestationstate.go
    • app/controlplane/internal/service/attestationstate.go
    • pkg/attestation/crafter/statemanager/remote/remote.go
    • pkg/attestation/crafter/crafter.go

    Reproduction

    go run app/cli/main.go --insecure att init \
  --workflow foo --project bar \
  --collectors aiconfig --replace -y --debug --remote-state

    Error: the provided base digest doesn't match the stored state, it might have changed, got="...", want="..."

  5. migmartri
    closed this as completedin #2909on Mar 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingcomponent/backend

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions