feat: x402 e2e test, structured LLM outputs, prompt-injection guard and shared LLM budgets (#64 #65 #70 #71)

.env.example

@@ -108,6 +108,52 @@ IPFS_LOCAL_DIR=.ipfs-local
 NEWSAPI_KEY=
 FRED_API_KEY=
 
+# ── LLM budgets and shared guards (#71) ────────────────────────────────────
+# Per UTC day, across every provider and instance (state lives in Redis).
+LLM_DAILY_TOKEN_BUDGET=2000000
+# USD micro-units (1_000_000 = $1); an estimate from a static price table.
+LLM_DAILY_COST_BUDGET_MICROS=20000000
+# Percent of both budgets reserved for admin-triggered generation; cron work
+# stops at budget - reserve.
+LLM_ADMIN_RESERVE_PCT=10
+LLM_ANTHROPIC_RPM=12
+LLM_GEMINI_RPM=15
+LLM_BREAKER_FAILURE_THRESHOLD=3
+LLM_BREAKER_OPEN_MS=3600000
+# closed: refuse LLM calls when Redis is down (recommended in production).
+# open: fall back to a per-process limiter and log a warning.
+LLM_REDIS_FAIL_MODE=closed
+
+# ── LLM router (#157) ──────────────────────────────────────────────────────
+# OpenAI or any OpenAI-compatible endpoint. The router can use them in a task
+# policy even if Anthropic/Gemini are configured.
+OPENAI_API_KEY=
+OPENAI_BASE_URL=
+# Optional JSON policy overriding some/all tasks, e.g.
+# {"market_generation":[{"provider":"anthropic","model":"claude-sonnet-4-6",
+#   "temperature":0.2,"maxTokens":4096,"timeoutMs":60000}]}
+# Or a path to a JSON file. Missing tasks keep the built-in default policy.
+LLM_POLICY=
+LLM_POLICY_FILE=
+
+# ── IPFS content store (#100) ──────────────────────────────────────────────
+# Per-object cap and total Postgres budget for verified pinned bytes.
+IPFS_CACHE_MAX_OBJECT_BYTES=262144
+IPFS_CACHE_BUDGET_BYTES=268435456
+
+# ── Transparency reports (#159) ────────────────────────────────────────────
+REPORTS_ENABLED=false
+REPORT_CRON=10 0 * * *
+# dry-run: pin + persist, never publish on-chain. live: publish with the agent.
+REPORT_MODE=dry-run
+# URL placed in the report's method.doc field.
+REPORT_METHOD_DOC=https://github.com/OracleDesk/oracledesk-backend/blob/main/docs/transparency-reports.md
+
+# ── Treasury admin console (#156) ──────────────────────────────────────────
+# Lifetime of a prepared unsigned transaction.
+TREASURY_OP_TTL_SECONDS=3600
+# Optional stricter backend policy for withdraw operations.
+TREASURY_WITHDRAW_MIN_SIGNERS=0
 # ── Resilience & Timeouts ──────────────────────────────────────────────────
 TIMEOUT_RPC_READ_MS=8000
 TIMEOUT_RPC_WRITE_MS=45000

.github/workflows/ci.yml

@@ -59,7 +59,20 @@ jobs:
       - name: Audit database constraints
         run: npx tsx scripts/audit-constraints.ts --json audit.json
       - run: npm run build
+
+      # #64: the contracts repo's x402 service runs against backend-produced
+      # traces. Its dependencies are installed from the submodule's lockfile.
+      - name: Cache x402 dependencies
+        uses: actions/cache@v4
+        with:
+          path: contracts/oracledesk-stellar/x402/node_modules
+          key: x402-node-modules-${{ runner.os }}-${{ hashFiles('contracts/oracledesk-stellar/x402/package-lock.json') }}
+      - name: Install x402 test dependencies
+        run: npm run test:x402:prepare
+
       - run: npm test
+      - name: End-to-end x402 test
+        run: npm run test:e2e:x402
       # Integration tier (#135): real Postgres 16 + Redis 7 from the service
       # containers above. TEST_DATABASE_URL is the admin DB, used to clone a
       # migrated template database per test file.

docs/architecture.md

@@ -53,6 +53,37 @@ Publication is durable (#131). Each trace that should be on-chain gets a `trace_
 
 Transient errors retry with backoff `min(2^attempts × 30 s, 1 h)`. Rows are claimed with `FOR UPDATE SKIP LOCKED` and leased, and chain calls run after the claim commits, so several processes can run the worker. `verified` is set only by indexer confirmation (or the duplicate checks, which read the same on-chain record) and by `POST /traces/verify`; dry-run never confirms a row or sets `verified`. The creation trace is pinned with `onChainMarketId: null` when its market is still pending and is **not** re-pinned later: that would change `trace_hash`; the `market_id` argument of `publish_trace` provides the link. Action strings live in `src/lib/trace-actions.ts`.
 
+### End-to-end x402 test (#64)
+
+The paid product relies on one invariant: the bytes `pinJson` pins are the
+bytes whose sha256 is recorded on-chain, and they are exactly what the
+contracts repo's x402 service re-hashes before serving. `tests/x402-e2e.test.ts`
+proves it by running the submodule's real `createTraceApi` (loaded in place from
+`contracts/oracledesk-stellar/x402/` through `tsx`, nothing copied into `src/`)
+against traces produced by `generateReasoningTrace` and `pinAndPublishDocument`.
+IPFS is an in-memory map, the facilitator is an in-process stub (it accepts any
+well-formed payload: the test covers the server path and content parity, not
+payment cryptography) and global `fetch` is guarded so any non-loopback request
+fails the test. It checks the 402 challenge, a paid 200 whose content equals the
+canonical payload, a tampered-bytes 502, an unknown-trace 404 and that a failed
+settlement serves nothing. `tests/canonical-parity.test.ts` compares
+`canonicalJson` with the contracts `canonicalize` over an edge-case corpus; the
+one known divergence is `Date` (ISO string vs `{}`), so `pinJson` rejects
+non-plain objects (`IPFS_NON_PLAIN_DOCUMENT`) and a test walks `tracePayload`
+for class instances.
+
+```
+git submodule update --init
+npm run test:x402:prepare   # npm ci in contracts/oracledesk-stellar/x402
+npm run test:e2e:x402       # builds the tests and runs the e2e file
+```
+
+Locally the tests skip with guidance when the submodule or its dependencies are
+missing; with `CI=true` they fail instead. The bindings package the x402
+`package.json` links to is only dynamically imported by
+`registry-trace-repository.ts`, so it does not need building. CI caches the
+package's `node_modules` keyed on its lockfile.
+
 ## Payments
 
 A daily pass is granted only after `payment-verification.service.ts` reads the transaction with Stellar RPC `getTransaction` and confirms all of the following:
@@ -85,6 +116,182 @@ Outcomes are decided on-chain by the resolver contract. The backend only reads `
 
 `src/generated/` is copied from the pinned `contracts` submodule by `scripts/sync-contracts.mjs`, with a do-not-edit header and `// @ts-nocheck`. CI fails if it's stale.
 
+## LLM router (#157)
+
+All generation goes through `src/lib/llm/router.ts`. Each `LlmTask`
+(`market_generation`, `hedge_conditions`, `forecasting`, `attestation`,
+`generic_json`, `generic_text`) maps to an ordered list of `ModelSpec`s
+(provider, model, temperature, max tokens, timeout, optional base URL). The
+default policy reproduces the old Anthropic → Gemini behaviour. Override any
+task with `LLM_POLICY` (JSON) or `LLM_POLICY_FILE`; missing tasks keep the
+default, and an invalid task/spec fails at boot with a per-task message.
+
+Providers live in `src/lib/llm/providers/` (Anthropic, Gemini,
+OpenAI-compatible). They normalise SDK errors into `LlmError.kind`
+(`rate_limited`, `quota_exhausted`, `auth`, `invalid_request`, `timeout`,
+`server`, `content_filtered`, `empty`, `parse`). The router:
+
+- skips providers with no key (without counting a failure);
+- retries inline only when `retryAfterMs <= 5s`; longer waits cool the
+  provider down and fail over;
+- opens a 1-hour circuit on quota exhaustion;
+- cools a provider for 60s after 3 server/timeout failures;
+- fails over on auth/invalid-request/empty/parse;
+- parses JSON in the router, so an invalid response tries the next model;
+- returns provenance (`provider`, `model`, `attempts`, `latencyMs`, `usage`)
+  for agent logs and trace payloads (`llm: { provider, model } | null`).
+
+Text mode never sets a JSON response format; only `mode: "json"` does.
+
+## Structured LLM output (#65)
+
+Output that must fit a shape (market proposals, hedge conditions) is never
+parsed out of prose. `src/lib/llm-schema.ts` is the single registry: each shape
+is a zod schema from which `defineStructured()` derives the Anthropic tool
+`input_schema` (`z.toJSONSchema`, draft-7) and the Gemini `responseSchema`
+(`toGeminiSchema`, an OpenAPI-subset dialect: types mapped to `SchemaType`,
+`const` becomes a one-value `enum`, `propertyOrdering` added, and
+`additionalProperties`, `$schema`, `minLength`, `pattern`, `format` and
+exclusive bounds dropped, since Gemini answers 400 on them). Wire schemas avoid
+`.catch`, `.default` and transforms; the lenient domain parse
+(`marketProposalSchema`) applies defaults separately, and a bad currency is an
+error on the wire. Rules JSON Schema cannot express (the confidence interval
+must contain the probability, expiry must be in the future) are `postValidate`
+functions whose message goes into the repair turn.
+
+`callLLMStructured(spec, { system, user }, { maxTokens, ctx, task })` returns
+`{ value, provider, model, attempts }`:
+
+1. **Anthropic**: `tools: [{ name, description, input_schema }]` with a forced
+   `tool_choice`; the `tool_use` block is searched for (a text block may come
+   first). `stop_reason: "max_tokens"` is never repaired at the same budget: the
+   budget is doubled once, then the call fails with a specific reason.
+2. **Gemini**: `responseMimeType: "application/json"` plus `responseSchema`;
+   the text is `JSON.parse`d with no fence stripping.
+3. Each provider gets **one repair round**: the zod issues (or the `postValidate`
+   message) go back as an `is_error` `tool_result` (Anthropic) or a correction
+   turn (Gemini). Anthropic that still fails falls back to Gemini.
+4. Failure is `AppError(422, "LLM_STRUCTURED_OUTPUT_FAILED", { provider, issues })`
+   when output never validated, otherwise the last transport/guard error.
+   `INVALID_LLM_JSON`, `SCHEMA_VALIDATION_FAILED` and
+   `INVALID_CONFIDENCE_INTERVAL` remain on the `validateMarketQuestion()` wrapper.
+
+Every attempt, repairs included, takes a lease on the shared guards below. Prompts
+and outputs are not logged at info; counts and provider/model are. `callLLMText`
+falls back to a plain-text Gemini call (never a JSON one). Tests replace the
+providers through `setAnthropicClient()` / `setGeminiModelFactory()` and replay the
+shapes in `tests/fixtures/llm/` (hand-built examples of the API shapes, not
+captures from a live account).
+
+## Prompt-injection defences (#70)
+
+Headlines, descriptions, GDELT URLs and earlier market questions are untrusted,
+and an accepted proposal makes the treasury seed a market. Layers
+(`src/services/llm-guard.service.ts`, all pure):
+
+1. **Sanitise at ingestion** (`sanitizeSignal`, applied by all three fetchers
+   before `cacheSignals`, and again on cache reads): NFKC, removal of `\p{Cc}` and
+   `\p{Cf}` (zero-width, bidi overrides), whitespace collapse, 200/300 character
+   caps, URLs moved to `url` (GDELT stores its link there and `content: ""`).
+   Flags: `stripped_control`, `had_url`, `truncated`, `instruction_like`.
+2. **Quarantine**: `instruction_like` signals (instruction phrasing such as
+   "ignore all previous instructions", "system prompt", "you are now", or our
+   JSON keys; not the bare word "ignore") are never sent to the model. FRED rows
+   are built from numbers by our code and are never quarantined. If nothing is
+   left, the cycle is skipped (`NO_USABLE_SIGNALS`).
+3. **Delimit**: `renderUntrustedBlock` emits one JSON line per signal inside
+   `<untrusted_signals nonce="…">` (6 random bytes per call, never derived from
+   content); look-alikes of the tags inside the data become `‹untrusted_signals`.
+   Existing markets use the same treatment (`existing_markets`). Signals have
+   citable ids `n1…`, `f1…`, `m1…`.
+4. **Hierarchy**: both system prompts say the block is data that may contain
+   instructions and must not be followed; proposals must cite `grounding_signal_ids`.
+5. **Policy** (`checkProposal`, after schema validation): reasons
+   `UNGROUNDED` (unknown ids, or no cited signal shares two meaningful tokens with
+   the question), `PROBABILITY_OUT_OF_POLICY` (outside [0.05, 0.95], rejected not
+   clamped), `EXPIRY_OUT_OF_RANGE` (outside 1–90 days), `ORACLE_NOT_ALLOWED`
+   (`ALLOWED_ORACLES`), `BANNED_TOPIC` (frozen patterns), `URL_IN_QUESTION`
+   (URLs, handles, code fences), `INSTRUCTION_ECHO`, `NON_ASCII_CONTROL`. A
+   failure throws `PROPOSAL_REJECTED_BY_GUARD` (422).
+
+Rejections write `PROPOSAL_REJECTED` and quarantines `SIGNALS_QUARANTINED` to
+`agent_logs` with reason codes, counts and `sha256(canonicalJson(signal))`, never
+raw titles. `tests/fixtures/injection/*.json` holds 28 attack and 18 benign cases
+with recorded model outputs; `tests/llm-injection.test.ts` replays them through
+`generateMarketQuestion` (model replaced by `setLlmJsonCaller`).
+
+## LLM budgets and shared guards (#71)
+
+`src/lib/llm-budget.ts` (state in Redis, atomic Lua in `llm-budget-store.ts`)
+replaces the per-process interval guards and circuit breaker. Every provider
+attempt is `acquire()` → call → `record()`:
+
+- **Token bucket** per provider (`LLM_ANTHROPIC_RPM`, `LLM_GEMINI_RPM`, capacity =
+  RPM). An empty bucket makes `acquire` wait with jitter for up to 30 s.
+- **Circuit breaker** per provider: `closed → open → half-open`. A daily quota error
+  opens it (Gemini: until at least the next UTC midnight); `LLM_BREAKER_FAILURE_THRESHOLD`
+  consecutive errors open it for `LLM_BREAKER_OPEN_MS`. After that exactly one probe
+  is admitted (its holder has a TTL, so a hung probe cannot block recovery);
+  success closes the breaker, failure reopens it. `LlmCircuitOpen` is `503
+  LLM_CIRCUIT_OPEN`; the router then tries the next provider.
+- **Daily budgets** per UTC day across providers and instances:
+  `LLM_DAILY_TOKEN_BUDGET` and `LLM_DAILY_COST_BUDGET_MICROS`. A call reserves
+  `ceil(chars/4) + maxTokens`; `record()` releases the reservation (also on errors
+  and responses without usage) and adds the real input/output tokens and an
+  estimated cost from `LLM_PRICES` (an estimate; update it when prices change).
+  Crashed calls are reclaimed when their lease TTL passes. Cron work stops at
+  `budget × (1 − LLM_ADMIN_RESERVE_PCT%)`; admin work runs to the full budget.
+  `LlmBudgetExceeded` is `429 LLM_BUDGET_EXCEEDED` and never falls back to another
+  provider (the budget is global). The market cycle logs `LLM_BUDGET_EXCEEDED`
+  once per day (Redis `SET NX`) and returns `null`.
+- **Redis down**: `LLM_REDIS_FAIL_MODE=closed` (recommended) refuses calls with
+  `LLM_CIRCUIT_OPEN` / `GUARD_STORE_UNAVAILABLE`; `open` falls back to a per-process
+  limiter and warns at most once a minute.
+
+Time comes from Redis `TIME` inside the scripts, so clock skew between instances
+does not matter; this is also why budgets reset at UTC midnight, unrelated to
+`checkDailyDrawdown`'s local midnight. The Anthropic client uses `maxRetries: 0` so
+the limiter sees every HTTP attempt. Callers tag calls with a `CallContext`
+(`purpose`: `market_generation`, `trace_hedge`, `admin_generation`; `priority`:
+`cron` or `admin`); `POST /markets/generate` runs as `admin`. `getLlmUsage()` (tokens,
+cost, reserved, breaker states) is part of `GET /admin/status` and logged after each
+cycle.
+
+Tuning: raise `LLM_DAILY_*` for more volume; keep `LLM_ADMIN_RESERVE_PCT` large
+enough that an admin can still trigger a market when cron has stopped; set the RPM
+values at or below the provider's account limits; look for repeated
+`LLM call recorded` lines with `outcome: "error"` before lowering the breaker
+threshold.
+
+## IPFS content store (#100)
+
+`pinJson()` writes the exact pinned bytes to `ipfs_content` (Postgres
+`bytea`) after the pin succeeds; a store failure is logged and does not fail
+the pin. `fetchVerifiedBytes(cid, expectedSha256, verifiedBy)`:
+
+1. reads the local row and re-hashes its bytes;
+2. returns it only when both the stored hash and the freshly computed hash
+   equal the expected hash (on-chain or pin hash);
+3. otherwise consults the gateway, stores bytes only after they verify, and
+   returns unverified bytes without caching so callers can report a mismatch.
+
+sha256 is the admission criterion; the CID is only a lookup key. A CID can
+never map to two hashes (`IPFS_CID_HASH_CONFLICT`). Objects above
+`IPFS_CACHE_MAX_OBJECT_BYTES` are not stored. An hourly job evicts
+`GATEWAY_VERIFIED` rows by least-recent access, then 30-day-old `PINNED` rows,
+never touching content referenced by an unpublished trace or a `PENDING`
+market (one SQL `NOT EXISTS` query, batches of 500).
+
+## Transparency reports (#159)
+
+`src/services/transparency-report.service.ts` builds a deterministic
+`oracledesk.transparency-report` v1 document from chain reads and
+`chain_events` only (no DB floats). Amounts are base-unit decimal strings.
+The daily job pins the canonical bytes, publishes the hash through the
+reasoning registry under the sentinel `u64::MAX` market id with action
+`report`, and persists the index row. A crash after pin re-publishes the
+stored bytes rather than recomputing. See `docs/transparency-reports.md`.
+
 ## Database Invariants & Schema Hardening
 
 The Postgres database enforces key invariants at the storage layer via Postgres enums and `CHECK` constraints:

docs/contract-requests.md

@@ -63,6 +63,22 @@ If dry-run simulated (and returned the simulated result), the bindings were
 injectable, and `agentSell` existed, `src/services/chain.service.ts` here
 could become a thin wrapper around it. Its method names already match.
 
+## 6. Treasury `admin()` and `is_paused()` views
+
+The admin console has to read `DataKey::Admin` and `DataKey::Paused` from the
+treasury's contract-instance storage via `getLedgerEntries`. That is a
+storage-layout workaround that breaks if the contract's keys change. Expose
+`admin() -> Address` and `is_paused() -> bool` (or include them in
+`get_config`) so clients do not depend on instance storage internals.
+
+## 7. A dedicated `publish_report` entrypoint
+
+Transparency reports use `market_id = u64::MAX` as a documented sentinel in
+`publish_trace`, because reports are not about a single market. A
+`publish_report(agent, action, trace_hash, ipfs_cid)` entrypoint (with its own
+report counter/index) would remove the sentinel hack and let
+`market_trace_count(u64::MAX)` stay a valid market-only index.
+
 ## Note: `publish_trace` action strings differ between repos
 
 The backend publishes traces with `create_market`, `buy_yes` and `buy_no`