Issues

• Live evaluation progress is never visible: the worker holds one uncommitted transaction for the whole run

  mini-router/minirouter#251 · Filter by author kai392 opened on Jul 24, 2026

• eval_runner: a re-evaluation that produces no results.json is scored from the previous run's file

  mini-router/minirouter#249 · Filter by author kai392 opened on Jul 24, 2026

• fugu: workflow parser rejects a bare-int access index (accepts "0"/[0] but not 0)

  mini-router/minirouter#225 · Filter by author philluiz2323 opened on Jul 17, 2026

• Per-role decoding config is loaded but never applied (dead config)

  mini-router/minirouter#189 · Filter by author real-venus opened on Jul 13, 2026

• reward: BFCL multi-call grading misaligns candidate/gold calls, scoring correct answers 0

  mini-router/minirouter#180 · Filter by author philluiz2323 opened on Jul 13, 2026

• reward: RLPR WebInstruct math grader ignores \boxed{...} answers, scoring correct answers 0

  mini-router/minirouter#170 · Filter by author philluiz2323 opened on Jul 13, 2026

• worker: one-shot (non --loop) run exits 1 on success, so supervisors see every processed job as a failure

  mini-router/minirouter#168 · Filter by author RealDiligent opened on Jul 13, 2026

• validate_bundle trusts summary.json n_total over the canonical ParamSpec, so a wrong-size theta passes

  mini-router/minirouter#167 · Filter by author philluiz2323 opened on Jul 13, 2026

• cost ledger: model ids with a quote or backslash write malformed JSON and are dropped from the cost total

  mini-router/minirouter#165 · Filter by author RealDiligent opened on Jul 13, 2026

• reward: normalize_math_answer leaves a trailing backslash on LaTeX-escaped set braces \{...\}

  mini-router/minirouter#162 · Filter by author philluiz2323 opened on Jul 13, 2026

• CI red on main: test_wrong_theta_length_fails not updated for the warning-only size check (#150)

  mini-router/minirouter#160 · Filter by author philluiz2323 opened on Jul 13, 2026
  1

• benchmarks: rlpr / ifeval / bfcl_simple load() all raise TypeError (same bug as #86, masked by the same test pattern)

  mini-router/minirouter#152 · Filter by author philluiz2323 opened on Jul 13, 2026

• Multiple-choice grading picks the first listed option, not the model's answer

  mini-router/minirouter#124 · Filter by author rust-toml opened on Jul 12, 2026

• RLPR MMLU-Pro is scored as 4-option (A–D), so any correct answer in E–J is always marked wrong

  mini-router/minirouter#122 · Filter by author rust-toml opened on Jul 12, 2026

• Validator crashes on first startup: ensure_schema alters columns that no longer exist on a fresh DB

  mini-router/minirouter#120 · Filter by author rust-toml opened on Jul 12, 2026

• reward: MMLU-Pro (RLPR) answers E–J always score 0 — choice grading is capped at A–D

  mini-router/minirouter#116 · Filter by author philluiz2323 opened on Jul 12, 2026

• reward: math grader misses common MATH-500 answer forms (x=5, scientific notation, empty oxed{})

  mini-router/minirouter#107 · Filter by author joaovictor712 opened on Jul 11, 2026

• Add end-to-end GSM8K, HumanEval, and BIG-Bench Hard (BBH) benchmark support

  mini-router/minirouter#99 · Filter by author claytonlin1110 opened on Jul 10, 2026

• test(dataset): offline unit tests for sample_minibatch; normalize stdout in LCB test parser

  mini-router/minirouter#88 · Filter by author luciferlive112116 opened on Jul 9, 2026
  

• sep-CMA-ES ships the noisy per-eval xbest, not the distribution mean

  mini-router/minirouter#69 · Filter by author dale053 opened on Jul 9, 2026

• load_tasks must fail loudly instead of silently using the offline toy set

  mini-router/minirouter#65 · Filter by author pekar9781 opened on Jul 8, 2026

• Eval runs TRINITY/random once each; no noise estimate or significance for R1/R2/R4

  featurenot planned
  mini-router/minirouter#62 · Filter by author philluiz2323 opened on Jul 8, 2026

• No per-question agreement diagnostic / contested-subset export for routing analysis

  mini-router/minirouter#60 · Filter by author philluiz2323 opened on Jul 8, 2026

• Validator worker re-queues submissions forever after uncaught evaluation errors

  mini-router/minirouter#50 · Filter by author RealDiligent opened on Jul 8, 2026

• Default remote helper scripts to GPU 5 instead of GPU 0

  mini-router/minirouter#33 · Filter by author rust-toml opened on Jul 8, 2026