[rNLKJA]

Career-aligned upgrade of the Cachex revival: statistical rigour around the original 2022 results, and optional bring-your-own-key AI features with transparency and an audit trail. The original coursework and its results are unchanged; the upgrade adds analysis around them.

Live: https://cachex-ai.vercel.app

What's new

Statistics and evaluation

• web/src/lib/stats/: Wilson and Newcombe intervals, seeded percentile and paired bootstraps, the Wilcoxon signed-rank test, Bradley-Terry fits on the Elo scale, and an exact McNemar test. Unit tests check them against scipy, statsmodels and R.
• /tournament: a seeded, colour-swapped round robin. You can rerun it in the browser in Web Workers, with CSV export. It also has the 1,200-game reference run, a cross-check against the original Python, the original 144/160 benchmark (90.0%, 95% CI 84.4% to 93.8%) and an alpha-beta efficiency study.
• /astar: a paired study of Manhattan versus Euclidean (980 boards). It reports a bootstrap CI and Wilcoxon test for node counts, and a BFS optimality check that adds a paired difference and McNemar test.
• /methods: data provenance, methods, evaluation design, assumptions, limitations and "what I'd change". It renders the agent card and decision records DR-001 to DR-006.

Bring-your-own-key AI (optional; the site works fully without a key)

• AI settings: Anthropic is the default (Claude Haiku 4.5, or Sonnet 5.5) and OpenAI is optional. The key is kept in sessionStorage unless you choose to remember it, and "Forget key" removes every saved key. Calls go directly from the browser to the provider.
• The move commentator on /play and /spectate is grounded in the agent's own numbers, including what the score means at the depth searched. An automatic grounding check runs on every reply and is stored with the call; you can accept, edit or reject it.
• /llm-arena: an evaluation harness in which a language model plays the minimax agent. It is compared with random, greedy and a scripted first-legal-cell baseline on the same seeds.
• /ai-log: every call is recorded in IndexedDB with its input, output (including the raw reply when validation failed), latency, tokens, grounding check and human decision, with JSON and CSV export. The key is never recorded.

Second review round

Governance and grounding

• Key storage: the AI settings dialog keeps every change as a draft until Save. A key saved for this tab only was being moved into local storage after a provider switch; the "remember" box now follows the selected provider's key and storage only changes when you change the box. Closing the dialog changes nothing.
• Commentary at depth 2+: the facts now say what the score means (score_scope, forced_win). The feature breakdown describes the position one ply after the move, so it no longer gets presented as the reason for a deeper search's score or a forced win. A fifth check fails a reply that misses, misattributes or invents a forced win. Regression test: 4×4, seed 76, a depth-3 forced win whose features sum to −12. The shared empty-hex feature may be called neutral.
• Audit trail: the grounding result is written into the same audit entry as the output and exported (grounding_passed, grounding_failures). /ai-log shows it above the review buttons, flags and counts failures, and re-checks older entries from the logged prompt.

Statistics

• Tournament reproducibility: records are analysed and exported in schedule order, so the same seed now gives the same Bradley-Terry intervals and CSV however the Web Workers finish. The reference run is unchanged.
• Alpha-beta: "92% to 100%" was a range of averages. 444 of 480 searches visit the whole tree, and the 36 that prune visit 0.6% to 92% of it, because the beta bug only allows a cut-off after a forced win for Red. DR-005 supersedes DR-002's sentence; DR-002 itself is unedited.
• LLM arena: the rejected-answer interval accounts for turns being clustered in games (Wilson on the effective sample size after a design effect, floored at 1, checked against statsmodels). Each baseline is compared with the model game by game, with an exact McNemar test.
• Display: p-values read "p = 0.012" or "p < 0.001".

Site

• The AI use statement's subsections are h3 under their h2; header nav focus rings are no longer clipped; wide tables fade at the right edge while columns are hidden.
• "Source" links point at the deployed commit, so files that exist only on this branch no longer 404 from production. A decision-record page notes when a later record supersedes part of it.
• DR-005 (report uncertainty at the unit that was sampled) and DR-006 (ground commentary in what the search found, and keep the check with the decision).

First review round

• Anthropic adapter: checks stop_reason before parsing, so truncated replies and refusals are no longer reported as format errors. It sends a schema that keeps enum (the SDK helper had been turning enums into descriptions).
• OpenAI adapter: now uses the same token budget.
• Audit log: failed calls keep their raw reply and the tokens they were billed for.
• LLM arena scoring: both providers are scored by the same rule. Illegal moves and replies with no usable move are counted separately, and the headline is a per-turn rate with forfeits reported over games. Baselines use only the games the model finished, and the arena warns when colours are unbalanced.
• Grounding check: now catches invented numbers at the end of a sentence.
• Spectate: autoplay pauses while commentary loads, and late commentary is kept. Entries left pending can be reviewed on /ai-log.
• Paired comparisons: Elo differences now come from the same bootstrap refits (original minus greedy: +15, 95% CI −24 to +56; depth 3 minus original: +151, 95% CI 106 to 195). For the A*, the paired optimality difference is −11.3 pp (95% CI −13.3 to −9.1), with an exact McNemar test (1 vs 106 discordant boards). DR-004 records this.
• New agent-card failure mode: the agent does not block one-move wins. A scripted Blue that fills row 0 wins 50/50 games on 4×4, 43/50 on 5×5 and 34/50 on 6×6, and tests pin these counts.
• Accessibility:
  • Checked checkboxes are visible in dark mode.
  • The AI settings dialog returns focus to the button that opened it.
  • Scrollable tables and code blocks can be focused and have names.
  • Two-column markdown tables now wrap.
  • The home-page benchmark table shows a Wilson interval on every row.
• Wording: claims now match what the evidence shows ("no detectable difference" plus the size of difference each check can detect).

Checks

• pnpm lint && pnpm typecheck && pnpm test && pnpm build all pass on the final commit, with 979 tests (up from 957). pnpm format:check is clean.
• Playwright on system Chrome against a local production build covered 15 routes in light at 1440 px and dark at 375 px: all 200 (the 404 page 404), no horizontal overflow and no console errors.
• In the browser:
  • Two in-browser tournaments at seed 4399 gave identical Elo intervals and the same games-CSV order.
  • The review's key-storage reproduction no longer moves the tab-only key, and Escape leaves preferences unchanged.
  • With a mocked Anthropic endpoint, commentary that names the wrong mover is stored with its failed check, shown on /ai-log, and exported in JSON and CSV. An entry logged before this change is re-checked and the check is stored with the decision.
  • A 4-game mocked arena run shows the paired column and the clustered interval.
  • The test key never reaches IndexedDB or the exports, and no request goes anywhere except the intercepted provider endpoint.
• The AI features have still not been run against a real provider. A manual check with your own Anthropic key before relying on them would confirm that browser CORS and structured outputs work on Haiku 4.5.

The AI use statement is informed by the Australian DTA responsible-AI policy, EU AI Act transparency principles and the NIST AI RMF. It is not a claim of compliance.

🤖 Generated with Claude Code