feat(mcp): add authenticated Streamable HTTP transport alongside stdio (#419)

.changeset/mcp-http-transport.md

@@ -0,0 +1,42 @@
+---
+'lens': minor
+---
+
+The MCP server can now serve Streamable HTTP, selected with `MCP_TRANSPORT=http`.
+Until then stdio was the only transport, so the server could only be used by a
+client able to launch this process on the same machine — which rules out every
+hosted agent, and means the agent's secret key has to be installed on each
+machine that wants to pay.
+
+HTTP reaches the network, so it is only allowed to start authenticated.
+`resolveMcpTransportConfig` (`src/mcp/config.ts`) requires a non-empty
+`MCP_HTTP_AUTH_TOKEN` before anything else happens, and `runHttp()` throws again
+if handed a blank one, so an unauthenticated remote server with `paid_call` in
+its tool list cannot be brought up by configuration or by an embedding caller.
+The example entrypoint resolves the transport before it builds a signer: a
+server that cannot authenticate callers never reaches the point where it can
+spend. Every request must then present `Authorization: Bearer <token>`, compared
+over SHA-256 digests with `timingSafeEqual`; a missing, malformed or wrong token
+is refused with `401`, `www-authenticate: Bearer realm="lens-mcp"` and the new
+`ERR_UNAUTHORIZED` code, before the body is read and before any tool runs. The
+token is never echoed back into an error message or a log line.
+
+It binds `127.0.0.1` unless `MCP_HTTP_HOST` says otherwise, answers MCP only on
+`MCP_HTTP_PATH` (default `/mcp`), rejects anything else with `404` without
+revealing whether the path exists, and requires `POST` — `GET` gets `405`, which
+spec-compliant clients read as "no server-initiated stream" rather than an
+error. Bodies are capped at 1 MiB and must be `application/json`, so a caller
+cannot make the payment path stream an unbounded request.
+
+Both transports register the same tools from the same code path, and share
+one spend budget and payment ledger per process, so a remote client cannot spend
+past a ceiling a local one already used. State is deliberately not shared
+across requests: the SDK's stateless Streamable HTTP transport is single-use — a
+second request on one instance throws, because message ids from different
+clients would collide — so each `POST` gets its own protocol server and
+transport.
+
+stdio remains the default: with `MCP_TRANSPORT` unset, or set to `stdio`, the
+server talks to stdin and stdout exactly as before. An unrecognised value throws
+instead of quietly falling back, because a server that silently ignored the
+transport it was told to serve would be hard to trust with a key.

.changeset/mcp-payment-audit.md

@@ -0,0 +1,55 @@
+---
+'lens': patch
+---
+
+Every `paid_call` now leaves a record of what it cost. A successful call used to
+return the response body and nothing else: no log line, metric or persisted trace
+of which URL was paid, how much, on which network, with which transaction. For a
+tool that signs with a real key, the audit trail is not an enhancement — without
+it there is no answer to "where did today's money go", and the cumulative ceiling
+had no measurement of spend to read.
+
+`PaymentLedger` (`src/mcp/ledger.ts`) writes one JSON line per attempt to
+`MCP_PAYMENT_LEDGER_FILE` (default `~/.lens-mcp/payments.jsonl`): timestamp, full
+URL, host, method, price, asset, network, transaction hash, the call's outcome,
+what the ledger can vouch for about the money (`settled`, `possible` or `none`)
+and the error code if the call failed. A failed call is recorded too, because a
+payment can settle on a call that then errors — `outcome` and `spend` are separate
+fields so "the call failed" never implies "nothing left the account".
+
+The price is not on the `Response` the x402 client hands back; it is only visible
+on the payment header the transport puts on the request. So the ledger wraps that
+transport's fetch (`instrumentFetch`) and correlates the offer with the response
+it produced. The settlement header is read for the transaction hash and for
+whether the seller confirmed it.
+
+`mcp_payment_attempts_total` (labels `host`, `network`, `outcome`) and
+`mcp_payment_amount_usd_total` (labels `host`, `network`) export the same spend to
+Prometheus on the shared registry, which `MCP_TRANSPORT=http` now serves from the
+MCP process itself at `GET /metrics` — a counter nobody can scrape is not an
+export. That route answers only to the same bearer token as the tools: what an
+agent bought, on which host and for how much describes a key that moves money.
+`MCP_HTTP_PATH=/metrics` is refused at boot rather than resolved by precedence, so
+one path never has two meanings. Under stdio there is no listener, so the running
+total there is what `payment_budget` reports and what the audit file holds.
+
+Labels come from `MCP_METRIC_HOSTS` and `MCP_METRIC_NETWORKS` (default
+`stellar:testnet,stellar:pubnet`, the networks this agent signs for): series are
+permanent, so labelling by whatever URL an agent asks for would let one caller mint
+unbounded time series, and anything outside an allow-list is reported as `unknown`.
+The audit file always keeps the real host. Amounts that are not a USD money string
+are recorded verbatim but cannot join the USD sum, so `mcp_payment_unquantified_total`
+and `totalSpend().unquantifiedPayments` make that understatement visible instead of
+silent.
+
+No secret material is recorded. Only the seller's echoed `accepted` requirements are
+read out of the payment header; the signed envelope alongside it never reaches a
+record, a metric or the file, and every header field is length-capped so a seller
+cannot forge extra lines into the trail.
+
+Accounting fails closed in both directions: if the audit file cannot be created or a
+record cannot be written, `paid_call` is refused with `ERR_INTERNAL_FAILURE` rather
+than spending unrecorded. And `payment_budget` now reports `recordedSpend` beside the
+budget's own worst-case reservations, so the ceiling has an actual measurement to
+read: the budget holds the per-call maximum because it cannot see the price, while
+the ledger reports the price it did see.

.env.example

@@ -350,6 +350,22 @@ MCP_AGENT_SECRET_KEY_TESTNET=
 # and can spend real funds. Keep this out of any other process and fund only
 # what you are willing to lose.
 MCP_AGENT_SECRET_KEY_MAINNET=
+# Transport to serve: "stdio" (default) or "http". stdio speaks over the process's
+# stdin/stdout and only works for a client that can launch this process on the
+# same machine; "http" serves Streamable HTTP so remote/hosted clients can connect.
+MCP_TRANSPORT=stdio
+# Required when MCP_TRANSPORT=http, and refused without it. Every request must
+# carry `Authorization: Bearer <token>`. Generate a strong random value
+# (e.g. `openssl rand -hex 32`); it authorises paid_call spending, so treat it
+# as a credential, and never put it in a URL.
+MCP_HTTP_AUTH_TOKEN=
+# Bind address for the HTTP transport (default: 127.0.0.1). Reachability beyond
+# loopback is only safe behind a TLS-terminating reverse proxy.
+MCP_HTTP_HOST=127.0.0.1
+# Port for the HTTP transport (default: 3003; the REST API listens on PORT=3002).
+MCP_HTTP_PORT=3003
+# Path the HTTP transport answers MCP POSTs on (default: /mcp).
+MCP_HTTP_PATH=/mcp
 # Ceiling per paid_call, enforced by the x402 fetch wrapper.
 MCP_MAX_PAYMENT_PRICE=$1.00
 # Comma-separated origins (or URL prefixes) paid_call may pay for, e.g.
@@ -391,6 +407,30 @@ MCP_DAILY_SPEND_CEILING=$10.00
 # are refused until it is fixed (fail closed). Keep it off ephemeral filesystems
 # (default: ~/.lens-mcp/spend-state.json).
 MCP_SPEND_STATE_FILE=
+# Where the payment audit trail is appended, one JSON record per paid call:
+# timestamp, full URL, host, method, price, asset, network, transaction hash and
+# the outcome. Never the secret key and never the signed payment envelope. A call
+# that failed is recorded too, since a payment can settle on a call that errored.
+# If this file cannot be created or written, paid calls are refused: a tool that
+# moves money does not get to spend without a record (fail closed). Keep it off
+# ephemeral filesystems (default: ~/.lens-mcp/payments.jsonl).
+MCP_PAYMENT_LEDGER_FILE=
+# Comma-separated hosts allowed to become the `host` label of the spend metrics.
+# Prometheus series are permanent, so labelling by whatever URL an agent asks for
+# lets one caller mint unbounded time series; hosts outside this list are reported
+# as `unknown`, and an empty value reports every host as `unknown`. The audit file
+# above always keeps the real host regardless (e.g. bazaar.example,price.feed).
+MCP_METRIC_HOSTS=
+# Comma-separated network ids allowed to become the `network` label of the spend
+# metrics. Same bounded-series rule as above. The value here is the default — the
+# networks this agent signs for — spelled out, because the .env.example drift
+# guard reads a commented entry as undocumented. Setting it to an empty value
+# instead reports every network as `unknown`.
+MCP_METRIC_NETWORKS=stellar:testnet,stellar:pubnet
+# With MCP_TRANSPORT=http the registry above is scrapeable from the MCP server
+# itself: GET /metrics on MCP_HTTP_PORT, requiring the same MCP_HTTP_AUTH_TOKEN
+# bearer token. stdio starts no listener, so there the running total is what the
+# payment_budget tool reports and what the audit file holds.
 
 # Where `npm run validate:live-spec` points when checking the published
 # OpenAPI document against a running service. LIVE_API_URL wins if both are

README.md

@@ -49,8 +49,10 @@ curl -X POST http://localhost:3002/graphql \
 ### MCP
 
 ```bash
-# The MCP server speaks MCP over stdin and stdout — it does not listen
-# on a port. Port 3002 is the HTTP API, which knows nothing about MCP.
+# By default the MCP server speaks over stdin and stdout and listens on no
+# port; MCP_TRANSPORT=http serves Streamable HTTP on MCP_HTTP_PORT instead
+# (see "MCP server" below). Port 3002 is the HTTP API, which knows nothing
+# about MCP either way.
 npm run mcp:server
 
 # To drive it interactively, point the inspector at that command:
@@ -205,6 +207,9 @@ The GraphQL schema is exported as a canonical Schema Definition Language (SDL) a
 ## Observability
 
 Prometheus metrics are exposed on `GET /metrics` (public, no API key required).
+The MCP server is a separate process with its own `GET /metrics` on its own
+port, and that one does require its bearer token — see
+[MCP server](#mcp-server).
 
 Alongside the ingestion and database metrics, Lens exports HTTP request,
 latency, authentication and rate-limit signals:
@@ -321,19 +326,157 @@ const unsubscribe = client.subscribe(
 > the exact keys being indexed. Only the pair you subscribe to is delivered;
 > updates for other pairs are filtered out server-side. For full protocol handshake details and a raw WebSocket implementation, see [`docs/graphql.md`](docs/graphql.md#live-price-subscriptions-graphql-transport-ws).
 
-## MCP (Model Context Protocol)
+## MCP server
 
-Lens exposes its Bazaar catalog of x402-discoverable resources as an MCP
-server, so any MCP-compatible client or agent can discover and pay for price
-data through two tools: `bazaar_search` and `paid_call`. The server signs
-x402 payment challenges with Stellar secret keys on both `stellar:testnet`
-and `stellar:pubnet`.
+`npm run mcp:server` starts the Bazaar MCP server ([src/mcp/server.ts](src/mcp/server.ts), wired up in [examples/mcp-server/run.ts](examples/mcp-server/run.ts)). It advertises:
+
+| Tool | What it does | Spends? |
+| --- | --- | --- |
+| `bazaar_search` | Reads a Bazaar base URL's `/discovery/resources` or `/discovery/search?q=` and lists what is available. | No |
+| `paid_call` | Fetches an x402-gated URL and handles the `402 → sign → retry` loop. | **Yes — it signs with a real Stellar key** |
+| `payment_budget` | Reports the configured limits and what is left of today's budget. | No |
+| `lens_get_price` | Reads Lens's own aggregated price for a watched pair. | No |
+
+`payment_budget` appears whenever a spend budget is configured, which the example always sets, and `lens_get_price` only where the server can reach Lens's own price data. Every one of them is served over **either** transport, and the tool list, schemas and error codes are identical whichever you pick. Which transport runs is selected by configuration; **stdio is the default**, so an existing setup needs no changes.
 
 **⚠️ Warning:** the mainnet key signs with real XLM. Read the configuration,
 spending warning, and worked example before running.
 
+The server signs with a real Stellar key, so it needs `MCP_AGENT_SECRET_KEY_TESTNET`
+and `MCP_AGENT_SECRET_KEY_MAINNET` in the environment — keep them in `.env`
+(`run.ts` loads it with `dotenv`) rather than in a shell history line or a
+client config file that gets shared. Never commit them. Optional spend
+controls: `MCP_MAX_PAYMENT_PRICE` (per call, default `$1.00`),
+`MCP_DAILY_SPEND_CEILING` (default `$10.00`) and `MCP_SPEND_STATE_FILE` — the
+running total survives a restart, because a restart does not refund what the
+previous process spent.
+
 Full documentation: [examples/mcp-server/README.md](examples/mcp-server/README.md)
 
+### stdio (default)
+
+For a client that spawns the server itself as a child process — Claude
+Desktop, Cursor, or the runnable [agent example](examples/mcp-server/agent.ts)
+(`npm run mcp:agent`, which prints discovery and a paid call end to end).
+
+```bash
+npm run mcp:server   # reads MCP_TRANSPORT=stdio, talks JSON-RPC on stdin/stdout
+```
+
+Point a client at the command — the keys come from `.env`, so nothing secret
+has to be written into client config. This is the same invocation
+[agent.ts](examples/mcp-server/agent.ts) uses:
+
+```json
+{
+  "mcpServers": {
+    "lens-bazaar": {
+      "command": "npx",
+      "args": ["tsx", "examples/mcp-server/run.ts"]
+    }
+  }
+}
+```
+
+Because the process is spawned by the client, there is no port, no token and
+no remote attack surface: whoever can spawn it can spend, so keep it to
+your own machine.
+
+### Streamable HTTP
+
+For a client that connects over the network instead of spawning a process —
+a hosted agent, a shared dev box, a container. Set `MCP_TRANSPORT=http`:
+
+```bash
+# MCP_AGENT_SECRET_KEY_TESTNET / MCP_AGENT_SECRET_KEY_MAINNET live in .env
+export MCP_TRANSPORT=http
+export MCP_HTTP_AUTH_TOKEN="$(openssl rand -hex 32)"   # required, see below
+export MCP_HTTP_HOST=127.0.0.1                   # default
+export MCP_HTTP_PORT=3003                        # default
+export MCP_HTTP_PATH=/mcp                        # default
+npm run mcp:server
+# [mcp-server] Listening at http://127.0.0.1:3003/mcp
+```
+
+**Authentication is mandatory and fails closed.** Without a non-empty
+`MCP_HTTP_AUTH_TOKEN` the process exits before it builds a signer, so an
+unauthenticated remote server with `paid_call` on its tool list can never come
+up. Every request must then carry it as a bearer token; anything else gets a
+JSON-RPC error with `ERR_UNAUTHORIZED` and is rejected before the payment path
+is reached:
+
+```bash
+curl -sS -X POST http://127.0.0.1:3003/mcp \
+  -H "Authorization: Bearer $MCP_HTTP_AUTH_TOKEN" \
+  -H "Content-Type: application/json" \
+  -H "Accept: application/json, text/event-stream" \
+  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
+```
+
+A client connects the same way:
+
+```ts
+import { Client } from '@modelcontextprotocol/sdk/client/index.js';
+import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';
+
+const transport = new StreamableHTTPClientTransport(new URL('http://127.0.0.1:3003/mcp'), {
+  requestInit: { headers: { authorization: `Bearer ${process.env.MCP_HTTP_AUTH_TOKEN}` } },
+});
+
+const client = new Client({ name: 'lens-agent', version: '1.0.0' }, { capabilities: {} });
+await client.connect(transport);
+```
+
+Notes for this transport:
+
+- It binds `127.0.0.1` unless you set `MCP_HTTP_HOST` deliberately. Do not
+  expose it on a routable address over plain HTTP: the bearer token and every
+  paid result cross the wire, so terminate TLS in a reverse proxy and forward
+  to loopback.
+- The token is a credential — the bearer who holds it can spend. Keep it out
+  of URLs, logs, and shell history; rotate it if it leaks.
+- Requests are stateless: no `Mcp-Session-Id` is issued and each `POST` is
+  served by a fresh transport, so a client cannot hold a session open across
+  requests. Every MCP message must be a `POST` (`GET` returns `405`, which
+  spec-compliant clients treat as "no server-initiated stream"). Bodies are
+  capped at 1 MiB.
+- The daily spend ceiling is shared across everything this process serves, so
+  one HTTP client cannot spend past a budget another one already used up.
+
+#### Payment spend metrics
+
+What the agent spent is scrapeable from the same listener, at `GET /metrics`,
+guarded by the same bearer token — it is operator data about a key that moves
+money, so there is no anonymous scrape:
+
+```bash
+curl -sS http://127.0.0.1:3003/metrics \
+  -H "Authorization: Bearer $MCP_HTTP_AUTH_TOKEN"
+```
+
+| Metric | Type | Labels |
+| --- | --- | --- |
+| `mcp_payment_attempts_total` | Counter | `host`, `network`, `outcome` |
+| `mcp_payment_amount_usd_total` | Counter | `host`, `network` |
+| `mcp_payment_unquantified_total` | Counter | `host`, `network` |
+
+A refused payment is an attempt (`outcome="failure"`) and adds nothing to
+`mcp_payment_amount_usd_total`. `host` may only be a value from
+`MCP_METRIC_HOSTS` and `network` only one from `MCP_METRIC_NETWORKS` (default
+`stellar:testnet,stellar:pubnet` — the networks this server signs for):
+Prometheus series are permanent, so anything outside an allow-list is reported
+as `unknown` instead of letting one caller mint a time series per URL an agent
+tries. Amounts that are not a USD money string cannot join the USD sum, so they
+are counted in `mcp_payment_unquantified_total` rather than dropped silently.
+Per-payment detail — full URL, transaction hash, asset, error code — is in the
+`MCP_PAYMENT_LEDGER_FILE` audit trail, which keeps the real host whatever the
+allow-list says.
+
+`MCP_HTTP_PATH` cannot be `/metrics`; the process exits at boot rather than
+answering MCP and Prometheus on one path. Under **stdio** there is no listener
+to scrape, so read the running total from the `payment_budget` tool or the
+audit file.
+
 ### WebSocket backpressure
 
 `/ws` counts a message as sent only once the socket reports the write completed,