Evaluate held-out structured answers across four-model heterogeneous serving routes #263

Problem

The four-model serving matrix measures transport/lifecycle and exact output parity, but does not establish answer quality. Existing four-question fixtures include both formatting-only failures and true errors already present in native backends. BF16 tie cases also show why one-token equality is not a quality metric.

Evidence

Historical192model/route/case rows (#227) audited without new inference. After explicitly offline EOS truncation, exact-format scores are identical across four routes per model (Qwen0.6:6/12,Qwen4:11/12,KONI:9/12,Gemma:10/12). These are NOT semantic-accuracy scores: some responses include correct equations/prose; others answer Incheon or40/30/25 incorrectly. KONI nativeGPU original records contain the prior EOS bug, so offline truncation must not be called a live fix or current-runtime certification. Numerical controls #256/#261 show head-onlyFP32 does not generally improve free-generation equality.

Scope / preregistered hypothesis

Add a new deterministic synthetic validation set before inspecting model outputs: arithmetic, passkey retrieval, Korean table lookup; four unseen variants per task at short/~2K/~8K contexts (36cases per model). Use each model's chat template with thinking disabled and serialized identical input IDs across routes. Compare nativeGPU, RNGD→GPU, nativeATOM, RNGD→ATOM under normalEOS, max128output. Score JSON answer correctness and format separately; retain raw text/tokens and exact-token parity as an independent gate. No model-specific prompt/answer tuning after seeing results.

Done when

Fixture/scorer edge tests pass; all4models/routes run or explicit unsupported/failed conditions remain visible; paired correct→wrong and wrong→correct counts are reported separately from identical-token and format counts. Publish reproducible artifacts, configuration and limits. This small synthetic suite does not certify general model quality or statistical equivalence, and existing parity failures are not relabeled.

Owner

Codex; maintainer review. Related #201,#227,#233,#256,#261. Compatibility preference question remains pending; this work is useful under either exact-token or quality-oriented acceptance.