feat(x402): harden facilitator settle — spend ceiling, allow-list, rate limit

[therealbibson]

Overview

This PR hardens the x402 facilitator POST /settle path against sponsored-fee drain: a per-settlement fee ceiling, a rolling daily spend ceiling per network (Redis, fail-closed), an optional caller allow-list, and a tighter per-route rate limit — on top of the existing settle idempotency from #126/#149.

Related Issue

Closes #147

Changes

Spend ceiling

• [ADD] src/x402/settleGuards.ts — fee extraction from the payment envelope, per-settlement cap check, atomic Redis daily reservation (INCRBY + rollback on overrun), fail-closed on store outage.
• [MODIFY] src/config.ts / .env.example — FACILITATOR_DAILY_SPEND_STROOPS_{TESTNET,MAINNET} with tighter mainnet defaults; existing FACILITATOR_FEE_STROOPS_* is the per-settlement cap.

Caller allow-list

• [ADD] Optional FACILITATOR_ALLOWED_ORIGINS (Origin or x-facilitator-caller). Empty keeps today's open demo behaviour.

Settle path

• [MODIFY] src/routes/facilitator.ts — apply allow-list + fee cap before any submission; reserve daily spend only for new attempts (replays do not double-count); refusals return SettleResponse with errorReason: facilitator_declined (403); /settle rate limit default 20/min (FACILITATOR_SETTLE_RATE_MAX).

Tests

• [ADD] src/__tests__/facilitatorHarden.test.ts — fee cap, daily cap, per-network keys, allow-list open/closed, idempotent replay without second spend, rate-limit config.
• [MODIFY] src/__tests__/facilitatorSettle.test.ts — mock Redis spend ledger so existing settle/idempotency coverage still runs.

Verification Results

Acceptance criteria covered by facilitatorHarden + facilitatorSettle tests:
✅ Over-cap fee rejected before create/submit
✅ Daily cap refusal + Redis counter survives restart (keyed + TTL)
✅ Per-network ceilings (testnet key ≠ mainnet key)
✅ Allow-list refuse / empty = unchanged
✅ Identical settle replays once; one submit + one spend incr
✅ /settle rateLimit.max = 20 (< global 100)
✅ Refusals use SettleResponse + facilitator_declined (not bare 500)
✅ Rejection paths log reason / network / fee / caller

Acceptance Criteria	Status
Settle whose fee exceeds per-settlement cap rejected before submit	✅
Daily cap reached → further settles refused; counter survives restart	✅ Redis key + 48h TTL
Caps per network; exhausting testnet does not block mainnet	✅
Allow-list configured → outsider refused; empty → unchanged	✅
Replay returns original result; exactly one submission	✅ (existing + harden test)
/settle has its own tighter rate limit	✅ 20/min default
Refusals are spec-valid SettleResponse shape	✅ facilitator_declined
Rejection paths logged with abuse vs misconfig detail	✅

Notes

• Fail closed: Redis errors refuse settle rather than allowing unlimited sponsored fees.
• Rate limits bound frequency; daily/per-settlement ceilings bound balance.

[drips-wave]

@therealbibson Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

Several controls here are done right, and on a money path that is worth saying first: the allow-list uses allowed.includes(identity) — exact match, not substring; the daily reserve sits after the idempotency check so a replay cannot double-charge; and reserveDailySpend catches store errors and refuses, i.e. fail-closed. It is also genuinely wired — @fastify/rate-limit registers at src/index.ts:115, before registerSettleRoute at :161, so the route-level override takes effect.

Three things block it.

1. It ships a test that fails deterministically

src/__tests__/facilitatorHarden.test.ts — "registers a /settle rate limit tighter than the global 100/min default":

   → expected undefined to be defined
      Tests  1 failed | 15 passed (16)

Failed on both full runs. It reads (app as any).routes, which is not a Fastify API — it is undefined, ?? [] makes it empty, and max is undefined. The test app also never registers the rate-limit plugin.

Either drop the assertion or register the plugin in buildApp() and assert behaviourally — fire 21 requests, expect a 429. The latter actually tests the thing.

2. The spend ceiling fails OPEN on a typo'd env var

src/config.ts, the new facilitatorDailySpendCeilingStroops block: parseInt(process.env[...] || default, 10) yields NaN for a non-numeric value, and newTotal > NaN is false — so the cap is silently never enforced.

""           -> 100000000  | enforced? true
"abc"        -> NaN        | enforced? false   <-- cap disabled, no error
"1e8"        -> 1          | enforced? true    <-- bricks settlement
"100000000"  -> 100000000  | enforced? true

On a money path a mistyped FACILITATOR_DAILY_SPEND_STROOPS_MAINNET turns off the control you are adding, with nothing in the logs. And 1e8 — a natural way to write 100000000 — parses to 1.

Validate with Number.isSafeInteger(n) && n > 0, else fall back to the default and log loudly. A safety control that silently disables itself is worse than no control, because it is believed.

3. Internal infrastructure leaks to an unauthenticated caller

settleFailure (src/routes/facilitator.ts:83-92) puts errorMessage straight into the JSON body, and /settle is config: { public: true }. reserveDailySpend's catch builds:

`Spend ledger unavailable; refusing settle (fail-closed): ${(err as Error).message}`

An ioredis failure is typically connect ECONNREFUSED <host>:<port> — internal topology handed to any anonymous caller. Your own test confirms errorMessage reaches the body. Send a fixed string to the client, keep err.message in req.log.

Worth raising, not blocking

• The allow-list is not an auth control. resolveCallerIdentity trusts the Origin header and falls back to x-facilitator-caller — both trivially spoofable by any non-browser client (curl -H "Origin: https://allowed.example"). Fine as a misconfiguration guard; the .env.example wording should say that rather than implying access control.
• Fail-closed becomes fail-permanent. The attempt row is created before reserveDailySpend, so on store_unavailable it is finalised as failed — and the idempotency check then replays that stored 403 forever. A transient Redis blip permanently bricks that payment payload even after recovery.
• Reservation leak. If store.expire throws after incrby succeeded, the catch returns without rolling back the incrby, and the key now has no TTL — so the leaked reservation never expires.
• Test fixture payTo: 'G' + 'A'.repeat(55) is not a valid address. Inert here (never passed to Address.fromString), but it is another invented one. The asset contract id does validate.

tsc --noEmit clean, merges clean with main.

[therealbibson]

All three blockers from the review:

1. the rate-limit test asserts behaviour now — buildApp() registers @fastify/rate-limit, and the test fires 21 requests expecting exactly one 429 on the 21st, instead of reading the non-existent app.routes
2. facilitatorDailySpendCeilingStroops is validated with Number.isSafeInteger(n) && n > 0, so a typo falls back to the default with a loud warning rather than silently disabling the cap (and 1e8 no longer parses to 1)
3. the store-error errorMessage no longer embeds the raw driver error; the detail is logged server-side and the public response carries a static message

[Miracle656]

Re-reviewed at c38ab03. All three blockers are properly fixed — thank you, and the rate-limit test in particular is better than what I asked for. I am not merging yet: the merge result is red against a guard main has gained since, and while reading the upstream scheme to check your fee accounting I found that the ceiling is counting the wrong number. Both are specific and neither needs guessing.

Fixed

#	Point	Status
1	facilitatorHarden.test.ts rate-limit test fails deterministically	Fixed
2	Spend ceiling fails OPEN on a typo'd env var	Fixed
3	Store error leaked to an unauthenticated caller	Fixed

1. buildApp() now registers @fastify/rate-limit and the assertion fires 21 requests and expects exactly one 429 at index 20. That is the behavioural test, and expect(app.hasRoute(...)) first is a nice touch — it fails for the right reason if the route ever moves. Verified on the merge result:

$ npx vitest run src/__tests__/facilitatorHarden.test.ts src/__tests__/facilitatorSettle.test.ts
 Test Files  2 passed (2)
      Tests  38 passed (38)

2. Number() + Number.isSafeInteger(n) && n > 0 + console.warn + fall back to the default. "1e8" now parses to 100000000 instead of 1, and "abc" no longer silently disables the cap. The comment explaining why is the part I most wanted and it is there.

3. reserveDailySpend's catch now sends a fixed 'Spend ledger unavailable; refusing settle (fail-closed).' and keeps the driver error in console.error. Your own test pins the message shape (toMatch(/Daily facilitator spend ceiling/)), so a regression would be caught.

Blocking — the daily ledger charges a number the facilitator never pays

This is the one that matters, and it is not something you could have seen without reading the scheme's source, so: extractFeeStroops reads tx.fee out of the caller-supplied envelope, and reserveDailySpend(network, feeStroops, ceiling) books that against the ceiling. But @x402/stellar's ExactStellarScheme.settle() throws that envelope away and rebuilds the transaction on the facilitator's own account:

const facilitatorAccount = await server.getAccount(signer.address);
const rebuiltTx = new TransactionBuilder(facilitatorAccount, {
  fee: BASE_FEE,
  networkPassphrase,
  …
  sorobanData
}).setTimeout(…).addOperation(Operation.invokeHostFunction(invokeOp)).build();

and verify() derives the real number from its own simulation:

const minResourceFee = parseInt(simResponse.minResourceFee, 10);
const settlementFeeStroops = minResourceFee + parseInt(BASE_FEE, 10);
if (settlementFeeStroops > this.maxTransactionFeeStroops) { … }

(node_modules/@x402/stellar/dist/cjs/exact/facilitator/index.js.)

So the fee the facilitator actually sponsors is minResourceFee + BASE_FEE, computed from simulation. The envelope's fee field never reaches a ledger — it is a free-text number the caller chooses.

The consequence is that the control can be evaded by exactly the party it is meant to bound. Declare fee: 100 and the daily ledger books 100 stroops, while the facilitator sponsors up to FACILITATOR_FEE_STROOPS_MAINNET (50000). Against the 10,000,000-stroop mainnet ceiling that is a 500x under-count: the ceiling would report ~1% of actual exposure and never trip. A spend ceiling that reads a caller-controlled field is the kind of control that is worse for being believed — which is the argument you yourself made in the comment on point 2.

assertFeeWithinCap has the same problem but it is harmless: the scheme already enforces maxTransactionFeeStroops against the real figure in verify(), so your check is just comparing a discarded number against the same ceiling. Worth a comment saying so, or dropping it.

Two ways out, either is fine by me:

• Reserve the worst case. Book getNetworkConfig(network).facilitator.feeStroops — the per-settlement ceiling the scheme itself enforces — before settling, then reconcile downward from the settled transaction's feeCharged (via rpc.getTransaction(hash)) on success. Conservative, never under-counts, and the reconciliation is best-effort.
• Simulate in the guard and use minResourceFee + BASE_FEE, mirroring what verify() does. Accurate, but it costs an extra simulation per settle and can drift from what the scheme computes a moment later.

Blocking — the merge result is red

Main has gained an .env.example drift guard since you branched (you are 21 commits behind). Merged locally onto current main:

 FAIL  src/__tests__/envDrift.test.ts > .env.example drift guard
AssertionError: Found 1 environment variable(s) read in source but missing from .env.example and not in allow-list:
  - FACILITATOR_DAILY_SPEND_STROOPS (read at src\config.ts:229)

 Test Files  1 failed | 55 passed | 1 skipped (57)
      Tests  1 failed | 552 passed | 1 skipped (554)

.env.example documents FACILITATOR_DAILY_SPEND_STROOPS_TESTNET and _MAINNET but not the un-suffixed fallback your config reads second. One line. Everything else in the merge is clean — no conflicts, and the other 552 tests pass.

The three I raised last time, still open

Not re-litigating — I called them non-blocking then and two of them still are. But one has got worse because of what this PR adds.

Fail-closed became fail-permanent, and now on a routine path. The attempt row is created before reserveDailySpend, so the daily-cap refusal runs finalise(attemptId, 'failed', response) — your own test asserts it:

expect(mockUpdate).toHaveBeenCalledWith(
  expect.objectContaining({ data: expect.objectContaining({ state: 'failed' }) }),
)

and the idempotency check then replays that stored decline forever:

if (existing?.state === 'settled' || existing?.state === 'failed') {
  return (existing.response as unknown as SettleResponseShape) ?? …
}

When this was only reachable on store_unavailable it was an edge case. Hitting the daily ceiling is a normal, self-healing condition that clears at the next UTC midnight — but the payer's payload is bricked permanently, and the replay comes back 200 with the success: false body rather than the 403 the first attempt returned. Either do not create the attempt row until after the reservation succeeds, or do not finalise a guard decline as a terminal failed state.

Reservation leak on expire. Unchanged: if store.expire throws after incrby succeeded, the catch returns store_unavailable without rolling back the increment, and the key now has no TTL — so that leaked reservation never expires and the ceiling drifts down permanently.

.env.example allow-list wording. Still describes it as an allow-list of origins without saying that Origin and x-facilitator-caller are both caller-supplied and trivially spoofed by any non-browser client. One sentence saying it is a misconfiguration guard, not access control, would stop someone deploying it as the latter.

Small

The payTo fixture is still not a valid address — StrKey.isValidEd25519PublicKey returns false on it. Inert, since it never reaches Address.fromString, but we have had five fabricated addresses land in a week and I would rather not have one more sitting in a test file. GBBD47IF6LWK7P7MDEVSCWR7DPUWV3NY3DTQEVFL4NAT4AQH3ZLLFLA5 is already used elsewhere in the suite and does validate. (The asset contract id does validate — that one is fine.)

Where this leaves it

The structure is right and I would keep all of it: the guards module, the fail-closed posture, the per-network keys, the reserve-then-rollback ordering, and the test file. What is left is pointing the ledger at the fee that is actually spent, one line in .env.example, and deciding whether a guard decline should be a terminal attempt state. Nothing here is a rewrite.

[therealbibson]

@Miracle656 Both blockers are fixed, plus the three remaining items from the first review. Head is 78ccc758; it merges clean with main and CI (Typecheck & build: tsc --noEmit, npm run build, npm test) is green on it.

Blocking 1 — the daily ledger now books the fee that is actually sponsored

• src/routes/facilitator.ts reserves getNetworkConfig(network).facilitator.feeStroops — the per-settlement ceiling the scheme enforces itself via maxTransactionFeeStroops — instead of extractFeeStroops(...) from the caller's envelope. A payload declaring fee: 100 now books 50000, not 100, so the 500x under-count is gone.
• On success, reconcileDailySpend(network, reservedStroops, normalised.transaction) reads feeCharged out of rpc.getTransaction(hash) and releases the unused headroom. It is best-effort and one-directional: a non-SUCCESS tx or an unreadable ledger keeps the full reservation, so the ceiling can over-count but never under-count.
• The abort path releases the same worst-case number, so a submission that never landed doesn't strand the larger reservation.
• extractFeeStroops / assertFeeWithinCap are kept with the comment you asked for: they are a cheap malformed-payload guard on a number the scheme discards, not the spend control.

Blocking 2 — the .env.example drift guard

• Added the unsuffixed FACILITATOR_DAILY_SPEND_STROOPS= fallback that src/config.ts reads second, so envDrift.test.ts passes on the merge result.

The three from last time

• Fail-closed no longer becomes fail-permanent. A ceiling refusal now withdraws the pre-submission attempt row (settlementAttempt.delete) instead of finalising it as terminal failed, so the payer's payload is retryable rather than replayed as a stored decline. New test: first attempt 403 with mockSettle untouched, second attempt after the ceiling clears settles 200. I took your first option — the row is still created before the reservation; withdrawing it also drops the implicit "payload consumed" lock.
• Reservation leak on expire. If expire throws after incrby landed, the increment is rolled back (decrby) before failing closed, instead of leaving a TTL-less reservation behind.
• .env.example allow-list wording. It now says Origin and x-facilitator-caller are both caller-supplied and trivially spoofed by any non-browser client, and that this is a misconfiguration guard, not access control.
• payTo fixture is now GBBD47IF6LWK7P7MDEVSCWR7DPUWV3NY3DTQEVFL4NAT4AQH3ZLLFLA5 in both facilitatorHarden.test.ts and facilitatorSettle.test.ts.
• While there, the concurrent-withdrawal case on the idempotency read (existing is null after the unique-constraint rejection) returns a retryable unexpected refusal instead of throwing on the non-null assertion.

Files: src/x402/settleGuards.ts, src/routes/facilitator.ts, .env.example, src/__tests__/facilitatorHarden.test.ts, src/__tests__/facilitatorSettle.test.ts.

[therealbibson]

@Miracle656 Re-verified at head 78ccc758 — everything from your last review holds on the current commit.

• Fee accounting. src/routes/facilitator.ts now books the worst case the scheme can sponsor — getNetworkConfig(network).facilitator.feeStroops — instead of the caller-supplied envelope fee, and reconcileDailySpend(network, reservedStroops, normalised.transaction) releases the unused headroom from the settled transaction's feeCharged. A non-SUCCESS or unreadable tx keeps the full reservation, and the abort path releases the same number, so the ledger can over-count but never under-count. extractFeeStroops/assertFeeWithinCap are kept with the note that they are a malformed-payload guard, not the spend control.
• .env.example drift. The unsuffixed FACILITATOR_DAILY_SPEND_STROOPS= fallback is documented, so envDrift.test.ts passes on the merge result with current main.
• Fail-closed is no longer fail-permanent. A ceiling refusal withdraws the pre-submission attempt row (settlementAttempt.delete) rather than finalising a terminal failed, so the payload is retryable instead of replayed as a stored decline.
• Reservation leak on expire. If expire throws after incrby landed, the increment is rolled back with decrby before failing closed.
• Allow-list wording. .env.example now says Origin and x-facilitator-caller are both caller-supplied and spoofable by any non-browser client, and that this is a misconfiguration guard, not access control.
• payTo fixture. Now GBBD47IF6LWK7P7MDEVSCWR7DPUWV3NY3DTQEVFL4NAT4AQH3ZLLFLA5 in both facilitatorHarden.test.ts and facilitatorSettle.test.ts.

Verification on 78ccc758:

npx tsc --noEmit
  -> exit 0

npx vitest run
  Test Files  47 passed | 1 skipped (48)
       Tests  417 passed | 1 skipped (418)

Ready for re-review.