feat(traces): durable trace publication queue — wait for on-chain market, retry, confirm via indexer (#131)

[abayomiwav]

Problem

Publishing a reasoning trace's hash to reasoning-registry was best-effort and happened once. If the market wasn't on-chain yet (creation failed or ran in dry-run), pinAndPublishTrace pinned, logged "not published" and returned, and nothing ever came back for that trace. Trade traces were published fire-and-forget, and skipped entirely when the market had no on-chain id, so an RPC timeout lost them silently. Such traces stayed onChainTraceId: null, verified: false, and POST /traces/verify returned 409 TRACE_NOT_PUBLISHED forever.

Changes

Schema (commit 1)

• TracePublicationStatus enum and TracePublication model / trace_publications table, with @@index([status, nextAttemptAt]). Fields per the issue, plus submittedTraceId, which stores the trace id returned by the write so the chain can be re-checked. payload is nullable only for backfilled rows.
• Migration 20261017000008_trace_publications also backfills pinned-but-unpublished MARKET_CREATION traces as AWAITING_MARKET, marked as a data migration at the top of the SQL. Trade traces can't be backfilled: the trace doesn't record its direction, so buy_yes / buy_no can't be reconstructed.
• ReasoningTrace.traceHash was already indexed on develop (@@index([traceHash]), from the trace-audit migration), so no change was needed.

Queue and worker (commit 2)

• src/lib/trace-actions.ts: TRACE_ACTIONS (create_market, buy_yes, buy_no) plus tradeTraceAction(). Both agents use it.
• enqueueTracePublication() replaces pinAndPublishTrace in runMarketMakerCycle() step 7 and executeTrade() step 5. The insert is awaited (durable, idempotent per trace), and a best-effort pass then runs in the background. Trade traces are queued even when the market isn't on-chain yet.
• src/jobs/trace-publication.worker.ts (processTracePublications) claims due rows with SELECT … FOR UPDATE SKIP LOCKED in a short interactive transaction and leases them. Chain calls run after the claim commits. Per row:
  • AWAITING_PIN: pin the stored payload and save ipfsCid/traceHash. IPFS_NOT_CONFIGURED parks the row for 1 h without counting an attempt; other pin errors retry. The payload is kept.
  • AWAITING_MARKET: wait for onChainMarketId (re-checked every 5 min, not counted as an attempt). A CANCELLED market fails the row with MARKET_CANCELLED.
  • READY: duplicate check first. A trace already linked by the indexer, or the same agent + trace_hash among the market's 10 newest registry traces (market_trace_count/market_trace_at/get_trace), is confirmed without a write. Otherwise publish_trace is called once and the row moves to SUBMITTED. In dry-run, the simulated publish is logged and the row stays READY with a long backoff.
  • SUBMITTED: after TRACE_CONFIRM_TIMEOUT_MS without indexer confirmation, the submitted trace id is re-read on-chain before anything is retried. This covers the ambiguous signAndSend failure.
  • Errors: AgentNotRegistered/EmptyAction/EmptyCid move the row to FAILED with lastError. Others retry with min(2^attempts × 30 s, 1 h), and TRACE_PUBLISH_MAX_ATTEMPTS moves it to FAILED.
  • The chain, IPFS and Prisma dependencies are injectable (TracePublisher, the existing RegistryReader, pin, prisma, claim).
• src/cron/trace-publication.cron.ts runs the worker every TRACE_PUBLISH_CRON (default every 30 s) and is registered in server.ts.
• Indexer: the TracePublished branch now sets verified: true when linking (hash equality with our stored traceHash is what verification means for our own traces) and moves the trace's queue row to CONFIRMED. Nothing sets verified from a write result alone, and dry-run never confirms or verifies.
• Config: new env vars TRACE_PUBLISH_ENABLED, TRACE_PUBLISH_CRON, TRACE_PUBLISH_MAX_ATTEMPTS (integer 1–100) and TRACE_CONFIRM_TIMEOUT_MS (integer ≥ 10000), validated in src/config/index.ts and listed in .env.example.

Tests and docs (commit 3)

• New tests/trace-publication.test.ts with 13 tests:
  • a pending market waits, then publishes once the market gets its on-chain id
  • a transient error backs off, while AgentNotRegistered fails permanently
  • a pin failure keeps the payload and pins on the next pass
  • a missing Pinata config parks the row without counting an attempt
  • an already-linked trace is confirmed without publishing
  • an on-chain duplicate found via market_trace_at is confirmed without publishing
  • a SUBMITTED row is re-checked on-chain, then confirmed or retried
  • dry-run never confirms or sets verified
  • a voided market moves the row to FAILED
  • retries stop at the max-attempts limit
  • the backoff cap, the claim loop, and idempotent enqueue
• tests/indexer.test.ts: TracePublished now asserts verified: true and a CONFIRMED queue update.
• docs/architecture.md: the lifecycle table and the rules (no re-pinning, SKIP LOCKED, dry-run). docs/api.md: the REASONING_PUBLISHED example now uses buy_yes, with notes on action values and what verified means. docs/contract-requests.md: a note that the contracts agents use buy.

Verification

Check	Result
tests/trace-publication.test.ts + tests/indexer.test.ts	19/19 pass
Full compiled suite on this branch	52 tests: 42 pass, 10 fail
Full compiled suite on untouched develop (2e1b725)	39 tests: 29 pass, the same 10 fail (audit-constraints, constraints, copy-trade-verify, resilience, trace-audit). None are introduced here.
tsc --noEmit	14 errors, all in files this PR doesn't touch (monitor.cron.ts, resilience.ts, copy-trade.service.ts, stellar/clients.ts, subscription.service.ts, trace-audit.service.ts), and present on develop too. None in the changed files.
npx prisma generate	ok

Not passing on develop either: npm run build and npm test fail on develop because of those 14 type errors. tsconfig.test.json has noEmitOnError: true, so the test build never emits. I ran the suites with tsc -p tsconfig.test.json --noEmitOnError false. npm run check:contracts needs the contracts/oracledesk-stellar submodule, which isn't initialised in a plain clone. The migration hasn't been applied to a live Postgres (no database here). Please run npx prisma migrate dev against a dev database.

Notes and deviations

• The trace row is created in generateReasoningTrace() before the queue row, not in one transaction. The insert is awaited right after, and upsert makes it idempotent.
• The duplicate scan checks the newest 10 market traces and matches on agent and hash. It doesn't stop at the first foreign-agent trace as the issue suggested: interleaved agents could otherwise hide our own earlier publish.

Closes #131
Closes #126
Closes #129
Closes #130

[drips-wave]

@abayomiwav Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits