Track disparity history and fail CI on unexplained regression

[godofecht]

Parent: #202

Problem

A disparity can remain within an eligibility tolerance while getting materially worse from one benchmark freeze to the next. A single current-state disparity_report.json does not reveal that regression.

Work

Persist a compact history keyed by environment/toolchain/commit for every canonical row, including score/objective delta, tolerance-normalized disparity, model-state/semantic/configuration differences, and runtime ratio. Define regression budgets independent from final parity eligibility and render trend/history views on Pages.

Implementation ready

PR #329 adds a persistent disparity regression policy, compares each canonical run against the previous frozen report before accepting it, records a bounded history snapshot, freezes the history alongside canonical evidence, and publishes a dedicated disparity-history view on Pages. Pages also regenerates an in-workspace history snapshot so deployment cannot race the benchmark-freeze workflow.

The branch was rebuilt on current main after the direct MLIR WebAssembly work landed and preserves that deployment path. It intentionally excludes stale generated benchmark/architecture snapshots from the review diff.

Fresh CI is fully green: Benchmark disparity audit, Remaining issue implementation, and the full Flow workflow all pass. PR #329 is ready for review. Keep this issue open until it lands and the first main-branch freeze establishes the persisted history baseline.

[godofecht]

Triage: done on main

PR #329 landed as 3deac85 ("Track benchmark disparity history and restore data-driven plots"), and the first main-branch freeze c1e01df ("Freeze validated benchmark, disparity history, and architecture evidence") established the persisted baseline. That was the stated condition for closing this issue.

Verified on origin/main:

• benchmarks/disparity_history.json holds a snapshot keyed by commit (8936641c0cc977e53b5ca6cba80230412c3d8736) and environment_id (c6f5f30d77c794d7) with all 19 canonical rows. Each row carries score_abs_diff, score_tolerance_fraction, runtime_log2_ratio, configuration_difference_count, semantic_difference_count, strict_diagnostic_status and final_parity_status.
• benchmarks/disparity_regression_policy.json defines regression budgets independent of parity eligibility: tolerance-fraction max absolute increase 0.1, score delta max relative increase 0.5 above a 1e-7 noise floor, runtime log2 ratio max absolute change 1.0 (same environment only), and zero tolerance for new configuration or semantic differences.
• benchmarks/check_disparity_regression.py compares the current report against a baseline, fails on any budget breach, and appends the bounded history snapshot.
• CI enforces it. .github/workflows/remaining-issues.yml preserves the frozen disparity_report.json as a baseline before the run, then runs check_disparity_regression.py --baseline /tmp/disparity_report.baseline.json and asserts history['snapshots'] is non-empty. .github/workflows/disparity.yml regenerates and validates the disparity accounting on every benchmarks/** change.
• Pages renders the trend view. docs/disparity-history.html plots worst tolerance-normalized disparity per snapshot against the acceptance boundary and tables the latest snapshot; .github/workflows/pages.yml materializes the history for each build before publish_headline_v2.py runs.

One residual nuance worth a separate issue if it matters: the history rows record configuration and semantic difference counts but not model-state diagnostics. Model-state coverage is tracked in #207 and is currently 3/19, so there is little to trend yet.

Closing as done.