Skip to content

[web] Research Hub /research: trials ledger, evidence matrix, gauntlet, slippage universe #708

Description

@eaitbrahim

Why this is category-of-one

No broker (Alpaca), SaaS platform, or open-source competitor (Jesse, Freqtrade) shows historical trials and negative results — their business models depend on trading looking easy. Keel's research rigor is already public and hash-chained; this issue puts it in the navigation. Parallel work stream: read-only, no dependency on #698#707, shares no code with the cancel action — start after Sprint 1 establishes payload/nav patterns.

Context (all verified)

  • Trials ledger: keel/research/ledger.py — append-only, SHA-256 hash-chained (canonical_json + prev_hash, genesis zero-hash), at docs/experiments/trials-ledger.jsonl (93 rows, git-tracked). verify_chain reports per-row breaks. Per trial: id, timestamp, session, rule, params, provenance (a_priori|fitted), kind (7 values), decision (selected|rejected|diagnostic_only), per-trade/per-bar P&L series, summary.
  • Trial budget is derived, not consumed: tuning.declared_cells / explored_vs_declared enforce a sweep-vs-declaration refusal; trial_counts returns (M, N_decisions). There is no decrementing counter — the UI reports explored vs declared, never "budget remaining".
  • Gauntlet machinery, all present: DSR / E[max SR] / MinBTL (research/deflate.py), CSCV PBO (research/cscv.py + matrix.py, refuses series-missing columns, warns N < 10), Monte Carlo (research/montecarlo.py — reshuffle + moving-block bootstrap with seeded reports), DCA benchmark at real fees/slippage (sim/benchmark.py).
  • Slippage universe: slippage_for_quote_volume — 5bp floor, $500M anchor, 183.8bp TON cap (strategy/backtest.py); per-product table renderer with fallback/capped flags (sim/report.py _render_slippage_rows); PR feat(rules): the promotion gate prices fills per product, and the restatement that fires it (#335) #686 landed per-product pricing with the 240-trial restatement (5 rules x 24 assets x 2 regimes; median PF 0.309 -> 0.219; 0 of 120 cells clear 1.0; corpus null 0 of 138 at honest cost).
  • (rule, asset, regime) buckets exist (portfolio_sim.per_bucket_pnl) but report losing buckets only — no standing matrix artifact.

Scope — four views

  1. Trials Ledger — the hash-chained table: timestamp, trial hash, kind, decision, params; M/N counts; explored-vs-declared per rule; chain-verification badge; Export Provenance Ledger (JSONL). The anti-p-hacking proof in plain sight.
  2. Evidence Matrix — rule x asset x regime grid assembled from ledger rows (each trial already records rule/params/series/summary — a thin assembly layer, not a new store). Cells report profit factor, fee drag, trade count with pass/fail verdicts in the real refusal grammar (e.g. n_trades 58 < min_trades 100). Negative cells are highlighted with why they failed, never hidden. PR feat(rules): the promotion gate prices fills per product, and the restatement that fires it (#335) #686's null is the headline exhibit.
  3. Promotion Gauntlet scorecard — DSR (trial-count and skewness corrected), PBO, Monte Carlo histogram as pre-computed SVG polylines (the server-quantised pattern the equity curve already uses — zero client-side math), DCA benchmark comparison.
  4. Slippage Universe — per-product measured bp vs the 5bp floor (PR feat(rules): the promotion gate prices fills per product, and the restatement that fires it (#335) #686's measured figures; BTC ~5.5bp at 1.1x floor, TON at the 183.8bp cap, ~10x median), fallback and capped flags carried honestly.

API: GET /api/research/trials, /api/research/matrix, /api/research/gauntlet (+ slippage data on the same surface). All read-only.

Constitutional guardrails (hard requirements)

  • The Strathern rail in the UI: no sorting or ranking by profit factor anywhere in /research. Views group by rule family; columns report pass/fail gates. (Performance view's realized track-record sorting is untouched — that is reporting, not selection.)
  • Zero client-side math: every quantile, DSR, PBO percentage arrives as a presentation-ready string or SVG polyline from Python.
  • Feed fields never rank and never place orders — this view is display-only research state.

Acceptance

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureNew capability (groups under Features)parallel-okNo intra-phase dependency; parallelizable in a worktreeresearchMeasurement, validation or KB work (Research & validation)

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions