Repository navigation
/sales/mcp: exact replay with one idempotency key returns two media-buy IDs (September follow-up) #7899
Description
Summary
This is a route-specific follow-up to #2346 and its fix in #2367. The earlier detail is in the first and second follow-up comments on #2346. #2346 itself remains closed for the original /mcp route.
On 2 and 21 September 2026, repeated byte-identical create_media_buy requests to https://test-agent.adcontextprotocol.org/sales/mcp returned two distinct media_buy_id values for one idempotency key. Each ID was later returned with replayed: true.
This does not claim the /mcp fix in #2367 was ineffective. It asks whether /sales/mcp is intended to share the same idempotency scope and backing state.
The primary retained evidence is the 21 September run. The 2 September result is included as previously reported corroboration; its raw responses are not retained in this evidence set.
These are historical observations. I have not re-tested the route under stable AdCP 3.2 and am not describing its current behavior.
Version and capability context
21 September run:
- Request declared
adcp_version: "3.1"andadcp_major_version: 3. Served version:3.1. get_adcp_capabilitiesdeclaredadcp.idempotency.supported: trueandadcp.idempotency.replay_ttl_seconds: 86400. Nobuild_versionwas present.
2 September run: no adcp_version was declared. Served version: 3.2-beta.10.
Method
For the 21 September shared-key run:
- The
create_media_buyrequest body was serialized once and sent ten times, sequentially. - The JSON-RPC request ID, payload, bearer credential, account and idempotency key were unchanged.
- Positive control, same run: ten requests with ten distinct idempotency keys returned ten distinct media-buy IDs.
The recorded request-body SHA-256 was identical across all ten shared-key calls:
a68636d3a72d1d96bc0475539198e0f421708c88fb248e52b55496958d865b3a
The 2 September run's method, the same in substance, is described in the first follow-up comment.
Observed results
| Run | Returned IDs | Responses per ID |
|---|---|---|
| 2 September | mb_4e89b550, mb_5d77a8c0 |
7 / 3 |
| 21 September | mb_a3729437, mb_f0045bfd |
5 / 5 |
The pattern was the same in both runs:
- Request 1 returned the first ID and was not marked as a replay.
- Requests 2 and 3 returned that ID with
replayed: true. - Request 4 returned a second ID and was not marked as a replay.
- Requests 5–10 returned one or other of the two IDs, each with
replayed: true.
On 21 September the two unmarked responses carried no replayed field at all.
Both IDs were subsequently returned with replayed: true, but the identical request did not converge on a single media-buy ID.
Expected behavior
#2367 describes the replay cache as scoped by (auth principal, account scope, idempotency_key). The client held the bearer credential, requested account, idempotency key and tool constant across all ten requests. The server-resolved principal and account scope were not independently observed. Within the advertised replay window, requests resolving to the same scope would be expected to return one committed operation and its cached response.
Scope and limits
- The deployment topology was not established. Separate replay stores would be one possible explanation, not an observed cause.
- These runs did not test expiry, purge, failover, concurrency, changed-payload conflicts or cross-principal access.
- These runs reused the same JSON-RPC ID; a separate same-key test with fresh JSON-RPC IDs was not performed.
- I have made no new writes to the public agent for this follow-up.
- Related compliance storyboard:
static/compliance/source/universal/idempotency.yaml. It was not run against this route.
Clarification requested
- Is
/sales/mcpintended to use the idempotency middleware and backing state addressed by fix(training-agent): enforce idempotency replay/conflict/expired semantics (#2346) #2367? - An automated triage reply on Training agent declares idempotency capability but does not enforce replay/conflict semantics #2346 suggested a separate issue. Could a maintainer confirm this is the appropriate tracker for the September observations?
- Is an isolated sandbox available for authorized verification against the current implementation?
Activity
bokelley commentedon Oct 3, 2026
Thanks for the detailed reproduction, the positive control, and the cross-release follow-up, @jaanijuk — picking this up now.
Generated by Claude Code
bokelley commentedon Oct 3, 2026
Triage
Classification: Bug — conformance failure (data integrity)
Bucket(s): compliance suite, training / certification
Status: Flag for human review
Milestone: P0 Bugs
What experts said:
Protocol expert: Clear spec violation. A server advertising
adcp.idempotency.supported: truewithreplay_ttl_seconds: 86400MUST return the same canonical response — including the samemedia_buy_id— for byte-identical replays within the window. The idempotency storyboard (static/compliance/source/universal/idempotency.yaml, narrative items 1–2) states this as an explicit MUST. Two distinct IDs for ten byte-identical requests is a direct violation. The pattern reproduces on stable 3.1 (September 21) and 3.2-beta.10 (September 2), meets P0 data-integrity criteria, and belongs inP0 Bugs. DR-0021 is adjacent but addresses a different failure mode (COMMITTED_RESOURCE_PURGED). The storyboard coverage gap (idempotency.yamlnever run against/sales/mcp) warrants separate tracking.Code reviewer (updated 2026-10-05): The lazy-init-to-in-memory-fallback hypothesis from the initial analysis does not hold against the actual boot sequence. In
server/src/http.ts,initializeDatabase(dbConfig)(line 10197) runs before the boot probegetIdempotencyStore().probe?.()(line 10218), and both run beforelisten()(line 10297). BecauseisDatabaseInitialized()returnspool !== null, the pool exists at the momentgetIdempotencyStore()is first called, so every replica selectsfencedPgBackend()on the standard boot path — there is no silent fallback tomemoryBackend(). The tenant-levelgetSdkIdempotencyStore()call insidebuildDefaultServerOptions()(registry.ts line 204) is lazy (first request triggers it), but by that point the process-widestoreInstanceis already PG-backed. @jaanijuk's read of the code is correct. Root cause is unconfirmed. The non-alternating pattern (A A A B B A B B B A on 21 Sep; A A A B A A A A B B on 2 Sep) is inconsistent with a simple two-instance round-robin and a different split mechanism must explain it.My take:
The spec violation and P0 classification stand. The specific mechanism previously identified (lazy-init race locking replicas into in-memory) is not supported by the boot sequence. Why byte-identical
create_media_buycalls to what should be a shared PG-backed store returned two distinct IDs across replicas remains unexplained. An on-host investigation is needed before a fix can be written and validated.Updated blocking for @bokelley:
/sales/mcpare PG-backed at time of first request? Specifically: doesisDatabaseInitialized()return true whengetIdempotencyStore()is called at boot (http.ts line 10218) on every machine? A Fly log showing backend selection per replica would clarify./sales/mcp→handleCreateMediaBuythat bypasses the singleton inidempotency.ts— i.e., creates a second store or uses a distinct pool? The non-alternating sequence cannot be explained by 2-instance round-robin and may indicate 3+ replicas or a path-level isolation.idempotency.yamlscope excludes/sales/mcp) be tracked here or in a separate issue?Generated by Claude Code
jaanijuk commentedon Oct 5, 2026
Thanks for picking this up. One note on the hypothesized cause, from reading source only. I have not run anything. The commits are
mainat8f7c110andmainas of the 21 September run ate3693a4; the code below is the same at both.isDatabaseInitialized()returnspool !== null(server/src/db/client.ts). It checks that the pool object exists, not that it is ready.HTTPServer.start()(server/src/http.ts),initializeDatabase(dbConfig)runs synchronously before the boot probegetIdempotencyStore().probe?.(), and both run beforelisten(). On that path, the firstgetIdempotencyStore()call would selectfencedPgBackend().getSdkIdempotencyStore()wiring intenants/registry.tsis built in a registry holder created on first request, so it comes afterstart().On the production entry path (
dist/index.js), I can't see how a replica would lock intomemoryBackend()through the ordering described. I don't know which build was deployed in September, or whether another path applies.Separately, the observed orders don't alternate. On 21 September they were A A A B B A B B B A; on 2 September, A A A B A A A A B B.
I'm not proposing a cause. I'm noting this so a fix can be checked against the mechanism that actually produced the split.
bokelley commentedon Oct 16, 2026
Verified against the deployed route on 2026-10-16 with the public test token from the quickstart (sandbox account
test_account):create_media_buypayload, 10 sequential requests then 5 concurrent: 15/15 returned the samemedia_buy_id(mb_d3203011), and all 14 repeats carriedreplayed: true.The fix is #7969, which makes production always use the durable Postgres replay store (resolved lazily so a cold replica can't fall back to in-memory), so
/sales/mcpreplays are scoped the same way #2367 describes for/mcp: by auth principal, account scope and idempotency key.On the questions here: yes,
/sales/mcpuses the same idempotency middleware and backing state; and the public test token plustest_accountis the sandbox for this kind of verification (anything created there is sandbox data). Runninguniversal/idempotency.yamlagainst this route is still worth adding to the storyboard matrix; reopen or file separately if you see a split again.jaanijuk commentedon Oct 16, 2026
Thanks — independently re-checked on the deployed
/sales/mcproute this morning (2026-10-16T23:53Z) with the quickstart test token andtest_account, clientadcp_version3.2:create_media_buy, 10 sequential + 5 concurrent: 15/15 samemedia_buy_id, 14replayed: true.media_buy_idacross all 7 accepted responses, 1 fresh. One of the five concurrent requests receivedIDEMPOTENCY_IN_FLIGHT(recovery: transient,retry_after: 30); the others blocked (~11 s) and then replayed.No split observed. Bounded run, so this doesn't prove the fix complete. I'll reopen or file separately if I see one again.