[gspark-etri]

Background

Existing four-question fixtures cannot determine whether cross-backend token differences change structured-answer correctness. Track this independently of unresolved strict-token failures in #256/#261.

Changes

Freeze 36 unseen synthetic cases per model for Qwen3-0.6B, Qwen3-4B, KONI-8B and Gemma3-1B: arithmetic, passkey retrieval and Korean table lookup at short/~2K/~8K contexts. Include compressed exact input IDs and hashes, greedy normal-EOS HTTP runner, separate parsed-answer/JSON-format scoring, and paired correct→wrong/wrong→correct/token comparison. Refuse overwriting prior outputs. No production inference changes.

Verification

• Test-first scorer/fixtures: 9 expected failures → 9 passes.
• Real loopback HTTP boundary test: missing request ID reproduced, corrected; 10 focused tests passed.
• Full suite collected before last HTTP test: 533 passed, 15 skipped, 253 subtests in160.17s.13 existing pinned-memory deprecation/flex-attention warnings retained.
• git diff --check and scoped sensitive scan passed. Published log copies only trim trailing whitespace.
• All21 source weight/config/tokenizer files (~27.67GB/host) SHA256-identical across hosts.
• Published auditor reproduces saved summary after unpacking frozen inputs; initial manifest-filename packaging mismatch corrected and rerun.
• All 576 responses completed with normal EOS and no inference failures; final published auditor reproduces the saved summary exactly. Owned evaluation Pod deleted and absence checked.
• Structured answer counts (native GPU / RNGD→GPU / native ATOM / RNGD→ATOM): Qwen3-0.6B 32/34/33/33; Qwen3-4B 36/36/36/36; KONI 31/31/31/31; Gemma 0/0/0/0 out of 36. Gemma emits code fences, so zero strict-format scores do not imply zero content accuracy.
• All six route pairs audited. Qwen3-4B is token-exact across all routes on these fixtures. Qwen0.6 passkey-3-2048 is correct on RNGD→GPU but wrong on both ATOM routes. Gemma korean_table-0-2048 returns the correct city on RNGD→GPU but null on both ATOM routes. Preserve these content counterexamples and all raw changed texts; aggregate scores do not establish equivalence.

Compatibility and limits

Stacked on #260 for previously validated runtime context. Expected four routes: nativeGPU, RNGD→GPU, nativeATOM, RNGD→ATOM. This small synthetic structured-answer test is not a general quality benchmark or statistical equivalence proof. Exact-token failures remain failures, regardless of answer scores. Local nativeGPU/RNGD-GPU uses BF16 runtime32ce5d5, four slots, page64, context9216, caches off; startup is excluded from request timing. Frozen fixtures, exact GPU launch argv/environment, HTTP responses and source hashes are recorded in evidence263; all four routes and final cleanup evidence are included.

Related Issue

Refs #263, #201, #256, #261. Does not close them yet.

Exploratory content diagnostic (frozen primary unchanged)

Re-analysis of the same576 responses removes only an outer JSON/unlabelled code fence and accepts integer arithmetic answers as exact decimal strings. Gemma content scores are29/30/29/29 outof36 (nativeGPU/RNGD-GPU/nativeATOM/RNGD-ATOM), while primary remains0/0/0/0. All36 outputs per route are fenced and8 additionally use integer arithmetic answers. Six or seven genuine content errors and the retained single Korean city-lookup cross-route difference remain. Other models’ scores are unchanged. These post-hoc rules are diagnostic, not a revised acceptance gate.

Analyzer has rejection self-checks, verifies paired input hashes/expected answers, and reproduces the published JSON byte-for-byte. All16 raw inputs match the previous published evidence; no new inference or engine change. See evidence263/analyze_quality263_content.py and quality263-content-diagnostic.json.

[gspark-etri]

Landed on optimus-rm via #286. Closing; the branch is archived locally (experiment records and evidence included).