Skip to content

test(mcp): simulate real library shapes in legacy paid_call tests - #608

Merged
Miracle656 merged 1 commit into
Divinity-Labs:mainfrom
AGWAM001:test/mcp-realistic-paid-call-fixtures
Oct 23, 2026
Merged

Miracle656 merged 1 commit into
Divinity-Labs:mainfrom
AGWAM001:test/mcp-realistic-paid-call-fixtures

Conversation

@AGWAM001

Copy link
Copy Markdown
Contributor

Summary

tests/mcp.test.ts still carries the four original paid_call tests. Three of them simulated the upstream with material no dependency produces, so the suite proved the classifier's branch selection works against shapes that never arrive. This rewrites those three fixtures to what the installed payment libraries actually throw and return. Fixture construction only — no titles, no assertions, no source files.

  • ERR_EXPIRED_AUTHORISATION — the old message, Transaction signature expired. Signature expiration ledger 123 is less than current ledger 125, is not a string any dependency emits (grep -ri "signature expired" node_modules → nothing). @x402/fetch 2.24.0 rethrows a scheme failure as Failed to create payment payload: <cause> (dist/esm/index.mjs:48), and the SDK's expiry text is The transaction has expired (@stellar/stellar-sdk 13.3.0, lib/webauth/utils.js:124, thrown as InvalidChallengeError).
  • ERR_PAYMENT_REJECTED — a rejected payment does not throw. The wrapper pays, retries and return secondResponse (@x402/fetch/dist/esm/index.mjs:80), so the seller's second 402 reaches the server as a Response with an x402 error body, not a { ok, status, text } stand-in.
  • success — a settled call carries X-PAYMENT-RESPONSE, which PaymentLedger base64-decodes in src/mcp/ledger.ts:163. The previous stand-in had no headers property at all, so the call never entered the one code path that reads the settlement echo. It is now a real Response, and readBodyCapped consumes its stream the way it does in production.

The fourth is left as-is, deliberately. Horizon answers op_no_trust with BadResponseError, whose result_codes live at err.response.extras.result_codes, while extractStructuredSignal reads err.result_codes (src/mcp/codes.ts:118). Measured through the classifier, the faithful fixture returns ERR_BAD_REQUEST, not ERR_MISSING_TRUSTLINE. Flipping that assertion inside a tests-only PR would ship a behaviour change disguised as a fixture, so it stays, and the gap belongs in its own issue. Note that main's own describe('BazaarMcpServer paid_call error classification') block sets err.result_codes directly, so it exercises the same field no dependency actually populates.

Related issue

Closes #421 — its other four acceptance criteria landed with #521 (c55a980); this is the remaining "simulated errors resemble real ones" criterion. See the issue thread for the per-criterion audit.

Type of change

  • Tests

Checklist

  • I have read CONTRIBUTING.md
  • npx tsc --noEmit passes — exit 0
  • npm run build passes — prisma generate + tsc, exit 0
  • I added / updated tests where relevant — npx vitest run tests/mcp.test.ts: 75 passed; full suite vitest run: 1445 passed | 1 skipped
  • node scripts/check-dropped-tests.mjs origin/main HEAD — 1340 cases on base, 1340 on the merge result, exit 0
  • I updated docs where relevant — no docs affected; no changeset either, since nothing here is user-facing

…ibraries throw

Three of the four simulated errors were invented: the ledger-number sentence
appears in no installed package, and the two Response stand-ins carried only
{ok, status, text}, so the ledger's settlement-header decode never ran.

Fixtures now match what the payment path actually produces: @x402/fetch rethrows
a scheme failure as "Failed to create payment payload: <cause>", a rejected
payment comes back as the seller's second 402 Response rather than a thrown
Error, and a settled call echoes X-PAYMENT-RESPONSE.

The trustline case is left as a plain string on purpose. Horizon answers it as
BadResponseError with result_codes at err.response.extras.result_codes, and
extractStructuredSignal reads err.result_codes, so the faithful fixture
classifies as ERR_BAD_REQUEST instead. That is a source gap, not a test one, and
out of scope here.
@AGWAM001

Copy link
Copy Markdown
Contributor Author

review and magre pls

@Miracle656
Miracle656 merged commit 1fb9908 into Divinity-Labs:main Oct 23, 2026
2 checks passed
Miracle656 added a commit that referenced this pull request Oct 23, 2026
…loses #421)

The four `paid_call` tests built their stand-ins by hand — `{ ok, status, text }`
object literals and error strings invented for the test. That is what made the
text-matching in `handlePaidCallError` look safer than it is: the suite was
asserting against strings the test itself wrote, not against anything either
library produces.

Now the error is what `@x402/fetch` actually rethrows for a lapsed signature
(`Failed to create payment payload: The transaction has expired`, the cause
coming from `@stellar/stellar-sdk`'s own `webauth/utils.js`), the 402 is a real
`Response` carrying the x402 error body rather than an object with a `text()`,
and the success case is a `Response` whose `X-PAYMENT-RESPONSE` header holds a
base64url settlement echo.

Each change buys a code path the old stand-in skipped:

- The realistic expiry string matches `paymentCouldNotHaveSettled`'s
  `'failed to create payment payload'` branch, so the test now exercises the
  spend classification the ledger records — the invented
  `'Signature expiration ledger 123...'` only reached the
  `'signatureexpirationledger'` branch.
- `fetchWithPayment` does not throw on a rejected payment: it pays, retries,
  and returns the seller's second response. A real 402 `Response` is what the
  code actually receives.
- A hand-rolled object has no `Headers`, so the success test never ran the
  header decode that reads the settlement echo.

This completes #421. `bazaar_search` (success, empty → `ERR_NO_RESULTS`,
non-2xx, non-JSON and unparseable bodies, SSRF refusals), `ListTools` (costs
and Zod-derived schemas) and the unknown-tool path were all covered by earlier
work; "simulated errors resemble real ones" was the one criterion left.

129 test files / 1,553 tests pass; tsc clean.

Claude-Session: https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB
@Miracle656

Copy link
Copy Markdown
Collaborator

Merged as 07189a7, and this closes #421.

This is a small diff doing something genuinely valuable, which is the hardest kind to get credit for. The four paid_call tests were asserting against strings the test itself had written — so the text-matching in handlePaidCallError looked far safer than it is. A suite that invents its own inputs cannot tell you whether the matcher still lines up with what the libraries throw.

What I liked is that each replacement buys a code path the stand-in was skipping, not just realism for its own sake:

The expiry string reaches a different branch. paymentCouldNotHaveSettled tests a list of substrings. The invented 'Signature expiration ledger 123 is less than current ledger 125' matched 'signatureexpirationledger'. Your 'Failed to create payment payload: The transaction has expired' matches 'failed to create payment payload' — which is the branch the real client's rethrow actually hits, and it is the one that decides whether the ledger records the call as possibly-settled. That is a money-relevant classification, and it was untested.

fetchWithPayment does not throw on a rejected payment. Your comment says it exactly: it pays, retries, and hands back the seller's second response. So a 402 arrives as a resolved Response, not a rejection, and the old { ok: false, status: 402, text } object was testing a shape the code never sees.

A hand-rolled object has no Headers. The success test never ran the X-PAYMENT-RESPONSE decode, which is the only path that reads the settlement echo. Using a real Response with a base64url header is what makes that line covered.

And citing provenance in the comments — @stellar/stellar-sdk lib/webauth/utils.js for The transaction has expired — is the detail that makes this maintainable. The next person who sees a mapping break can check whether the library changed its wording rather than guessing.

Why this closes the issue

The other four acceptance criteria were already met by work merged earlier, which is easy to miss when the issue text predates it. For the record:

  • bazaar_search — success (returns results from a permitted public host), empty (maps an empty resources array to ERR_NO_RESULTS, and the top-level-array variant), non-2xx (rejects a non-2xx response naming the upstream and status), malformed body (rejects a JSON content type with an unparseable body, plus the non-JSON content-type guard), and four SSRF refusals on top.
  • ListTools — tells the caller what each tool costs and when not to reach for it and advertises schemas derived from Zod definitions.
  • Unknown tool — returns structured ERR_UNKNOWN_TOOL error with valid tool names and the bounded-name variant.

"Simulated errors resemble real ones" was the one box left, and this is it. 76 tests in tests/mcp.test.ts now.

One observation, not a request

maps an empty resources array to ERR_NO_RESULTS treats "the node has nothing for this query" as an error. There is an open PR (#562) arguing it should be a successful empty collection instead, which I think is the better reading — an empty result is a valid answer, and clients should not have to catch an error to learn there were no matches. Not your problem here, and your test correctly pins current behaviour; flagging it so you are not surprised if that test is inverted later.

129 test files / 1,553 tests pass. tsc --noEmit clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Only the error paths of one tool are tested

2 participants