hydrangeas20 · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026 · Oct 25, 2026
diff --git a/.gitignore b/.gitignore
@@ -228,3 +228,5 @@ results/
 # are listed in SHA256SUMS and must be committed. Without this exception the
 # *.log rule above drops them silently. Other logs stay ignored.
 !docs/benchmarks/*/raw/**/server.log
+# Real-model verification evidence: the pytest log is listed in SHA256SUMS.
+!docs/benchmarks/whisper-t4/raw/pytest.log
diff --git a/README.md b/README.md
@@ -222,9 +222,10 @@ The repository contains the following benchmark evidence, with checksums recorde
 | Phase 1    |              5 |
 | Phase 3    |             60 |
 | Phase 4    |            105 |
-| **Total**  |        **170** |
+| Real-model verification (T4) |  3 |
+| **Total**  |        **173** |
 
-All three expected manifests were present, and the referenced files matched their recorded checksums.
+All four expected manifests were present, and the referenced files matched their recorded checksums.
 
 Verify the evidence with:
 
@@ -236,15 +237,16 @@ The verifier checks the manifests that exist. A complete repository audit should
 
 ### Test validation
 
-The recorded test validation covered:
+The offline suite (fake and gated backends) runs in CI on every pull request:
 
-- Python 3.11 and Python 3.13.
-- All four combinations of the two experimental scheduling flags.
-- Eight configurations in total.
+- Linux: Python 3.11 and 3.13 × all four combinations of the two experimental scheduling flags (eight configurations).
+- macOS: Python 3.13, default flags.
 
-**Result: 209 tests passed per configuration, with one real-model test deselected.**
+**Result: 269 tests passed per configuration, with the 8 real-model tests deselected.**
 
-The recorded validation included baseline compatibility checks with scheduling flags disabled, deterministic inference-order traces, and WebSocket protocol tests.
+It includes baseline compatibility checks with scheduling flags disabled, deterministic inference-order traces, WebSocket protocol tests, server-shutdown reliability, and the real-model recorder's validation rules.
+
+**Real-model tests: 8 of 8 passed on an NVIDIA T4** (faster-whisper `base.en`, CUDA FP16, commit `096a034`). The run covered the model adapter, server and WebSocket levels. Evidence: [docs/benchmarks/whisper-t4](docs/benchmarks/whisper-t4/README.md).
 
 ### Reproducing the analyses
 
@@ -259,6 +261,7 @@ Detailed procedures and evidence are available in:
 - [Phase 1 T4 Pilot](docs/benchmarks/phase1-t4/README.md)
 - [Phase 3 Scheduling Results](docs/benchmarks/scheduling-t4/README.md)
 - [Phase 4 Results and Evidence](docs/benchmarks/phase4-t4/README.md)
+- [Real-Model Verification Evidence](docs/benchmarks/whisper-t4/README.md)
 - [Colab GPU Workflow](benchmarks/COLAB.md)
 - [Measurement Definitions](docs/measurements.md)
 
@@ -319,13 +322,15 @@ python clients/stream_wav.py --synthetic 3
 
 The fake adapter emits synthetic transcript tokens and is useful for exercising the WebSocket lifecycle without GPU access.
 
-### Real-model smoke test
+### Real-model tests
 
 ```bash
-pytest -m whisper
+python -m pytest -q -m whisper
 ```
 
-This test requires access to the actual Whisper model weights and may download them on first use. It was excluded from the recorded validation because model downloads were unavailable in that environment. A successful real-model smoke test should be confirmed separately before claiming end-to-end model validation.
+Eight tests run the real `base.en` model at three levels: the adapter (device and compute type in effect, transcripts of reference speech, error wrapping), the server (`/health` after loading from settings, clean shutdown), and a real WebSocket stream of reference speech. They need the model weights (downloaded on first use) and, for the speech tests, the 5-utterance reference set; `ECHORELAY_TEST_DEVICE=cuda` selects the GPU.
+
+**Verified on an NVIDIA T4:** all 8 passed with CUDA FP16 (corpus WER 0.0531, streamed-utterance WER 0.1154). That is one configuration; CPU, other GPUs and other model sizes are not verified. See [Real-model verification](docs/whisper-verification.md) for what each test claims, the Colab procedure (including a required CUDA library-path fix), and the verification log.
 
 ### Docker
 
@@ -546,6 +551,7 @@ These limitations are part of the engineering record and inform the next researc
 | [Scheduling Design](docs/scheduling-design.md)                          | Scheduler mechanics, invariants, and race conditions |
 | [Measurement Definitions](docs/measurements.md)                         | Metrics, timing boundaries, and analysis methodology |
 | [GPU Workflow](benchmarks/COLAB.md)                                     | Colab experiment execution                           |
+| [Real-Model Verification](docs/whisper-verification.md)                 | Real Whisper test suite, T4 result, troubleshooting  |
 | [Protocol](docs/protocol.md)                                            | WebSocket messages and error behavior                |
 
 ---

diff --git a/benchmarks/COLAB.md b/benchmarks/COLAB.md
@@ -1,7 +1,8 @@
 # Running the GPU checks and benchmarks on Google Colab
 
-Nothing in this directory has been run on a GPU yet. Run the stages **in
-order** and stop at the first failure: the smoke stages exist so that a
+These stages produced the committed T4 evidence in `docs/benchmarks/`: the
+Phase 1–4 benchmarks, and the real-model verification in stage 2b. Run the
+stages **in order** and stop at the first failure: the smoke stages exist so that a
 broken CUDA setup or model path is found in two minutes, not halfway
 through a benchmark.
 
@@ -27,7 +28,7 @@ Runtime → Change runtime type → **T4 GPU** (or better).
 ### Public repository
 
 ```bash
-!git clone -b feat/streaming-mvp https://github.com/hydrangeas20/echorelay.git
+!git clone https://github.com/hydrangeas20/echorelay.git
 ```
 
 ### Private repository (no credentials in the notebook, URL, or `.git/config`)
@@ -58,8 +59,7 @@ env = {
     "GIT_CONFIG_VALUE_0": f"AUTHORIZATION: basic {auth}",
 }
 subprocess.run(
-    ["git", "clone", "-b", "feat/streaming-mvp",
-     "https://github.com/hydrangeas20/echorelay.git"],
+    ["git", "clone", "https://github.com/hydrangeas20/echorelay.git"],
     env=env, check=True,
 )
 del token, auth, env
@@ -103,11 +103,37 @@ If loading fails with a `libcudnn` / `libcublas` error: faster-whisper ≥1.1
 `nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*` and set `LD_LIBRARY_PATH` as
 described in the faster-whisper README, then rerun the smoke test.
 
+Observed on Colab (October 2026): `Library libcublas.so.12 is not found or
+cannot be loaded`, because the image exposes CUDA 13's `libcublas.so.13`
+while the pip-installed `libcublas.so.12` sat outside `LD_LIBRARY_PATH`
+(`/usr/lib64-nvidia`). Prepending its directory, and cuDNN 9's if present,
+fixed it. The exact steps are in
+[docs/whisper-verification.md](../docs/whisper-verification.md#troubleshooting-incident-colab-cuda-libraries).
+
 `prepare_audio` downloads Mini LibriSpeech dev-clean-2 (~126 MB) from OpenSLR
 and writes the first 50 utterances of 3–20 s (sorted by id) with
 references. Keep `eval_audio/` unchanged across all runs; every result file
 records each file's SHA-256.
 
+### 2b. Real-model test suite (`pytest -m whisper`)
+
+The adapter, server and WebSocket tests against the real model, with a
+record of the environment and results. Needs the 5-utterance reference set
+(not the 50-file `eval_audio`):
+
+Use the two cells in [docs/whisper-verification.md](../docs/whisper-verification.md#how-to-run).
+They prepare `eval_audio5`, fix the cuBLAS/cuDNN library path, and run
+`python -m benchmarks.whisper_verification --device cuda` **with Colab's
+default `PYTHONPATH=/env/python` removed**. The recorder rejects any
+`PYTHONPATH` or `PYTEST_ADDOPTS` override as an evidence-integrity violation,
+so a plain `!python -m …` in Colab reports `NOT COMPLETE` even when all tests
+pass.
+
+Pass criterion: the last line starts with `COMPLETE: 8 passed, 0 failed, 0
+errors, 0 skipped` and the exit code is 0. Otherwise the recorder prints
+`NOT COMPLETE` and each problem. The verified run is in
+[docs/benchmarks/whisper-t4](../docs/benchmarks/whisper-t4/README.md).
+
 ## 3. Server smoke test (one real stream)
 
 ```bash

diff --git a/benchmarks/whisper_verification.py b/benchmarks/whisper_verification.py
@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""Run the real-model test suite (``pytest -m whisper``) and record what ran.
+
+    python -m benchmarks.prepare_audio --out eval_audio5 --num 5
+    python -m benchmarks.whisper_verification --audio-dir eval_audio5 --device cuda
+
+Writes to ``--out`` (default ``results/whisper_verification``):
+
+* ``record.json``  UTC time, git commit, GPU/driver (``nvidia-smi``), Python and
+                   package versions, model, device, compute type, the exact
+                   command, per-test outcomes, the runtime identity the tests
+                   observed, and every reason the run is not complete.
+* ``pytest.log``   the complete pytest output.
+* ``junit.xml``    per-test results, including recorded properties.
+
+``complete`` is true only if all of these hold (``problems`` lists each one
+that does not):
+
+* exactly the expected real-model tests ran (``EXPECTED_TESTS``), all passed,
+  and none failed, errored or was skipped;
+* the identity the loaded runtime reported during the run matches the
+  request: faster-whisper's ``WhisperModel`` over CTranslate2's ``Whisper``,
+  the requested device and compute type, the ``base.en`` weights snapshot,
+  and (for CUDA) a visible CUDA device and the expected GPU in ``nvidia-smi``.
+  The requested settings in the record are not evidence of any of this;
+* no ``PYTEST_ADDOPTS`` or ``PYTHONPATH`` override (they could inject plugins
+  or substitute modules), and the checkout had no local edits.
+
+Exit status: pytest's if non-zero, otherwise 0 if complete, else 3.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import shutil
+import subprocess
+import sys
+import xml.etree.ElementTree as ET
+from datetime import datetime, timezone
+from pathlib import Path
+
+from .common import environment, write_json
+
+REPO = Path(__file__).resolve().parents[1]
+MODEL = "base.en"  # what tests/test_whisper.py loads
+EXPECTED_TESTS = frozenset({
+    "test_adapter_loads_with_requested_device_and_compute_type",
+    "test_adapter_decodes_silence_and_empty_audio",
+    "test_adapter_transcribes_real_speech",
+    "test_adapter_final_decode_is_deterministic",
+    "test_adapter_wraps_decode_errors",
+    "test_server_loads_real_model_from_settings",
+    "test_websocket_streams_real_speech",
+    "test_server_shutdown_releases_inference_workers",
+})
+IDENTITY_TEST = "test_adapter_loads_with_requested_device_and_compute_type"
+WHISPER_CLASS = "faster_whisper.transcribe.WhisperModel"
+CT2_CLASS = "ctranslate2._ext.Whisper"
+OVERRIDE_VARS = ("PYTEST_ADDOPTS", "PYTHONPATH")
+
+
+def _nvidia_smi() -> str | None:
+    if not shutil.which("nvidia-smi"):
+        return None
+    q = subprocess.run(["nvidia-smi"], capture_output=True, text=True, timeout=30, check=False)
+    return q.stdout if q.returncode == 0 else f"nvidia-smi exited {q.returncode}: {q.stderr}"
+
+
+def _junit_results(path: Path) -> dict:
+    root = ET.parse(path).getroot()
+    suites = [root] if root.tag == "testsuite" else root.findall("testsuite")
+    tests = []
+    for case in (c for s in suites for c in s.iter("testcase")):
+        outcome, message = "passed", None
+        for tag, name in (("failure", "failed"), ("error", "error"), ("skipped", "skipped")):
+            el = case.find(tag)
+            if el is not None:
+                outcome, message = name, (el.get("message") or "")[:500]
+                break
+        entry = {"name": case.get("name"), "classname": case.get("classname"),
+                 "outcome": outcome, "time_s": float(case.get("time", 0))}
+        if message is not None:
+            entry["message"] = message
+        props = {p.get("name"): p.get("value") for p in case.iter("property")}
+        if props:
+            entry["properties"] = props
+        tests.append(entry)
+    counts = {k: sum(t["outcome"] == k for t in tests) for k in ("passed", "failed", "error", "skipped")}
+    return {"collected": len(tests), **counts, "tests": tests}
+
+
+def validate(*, results: dict, device: str, compute_type: str, expect_gpu: str | None,
+             gpus: list[str], overrides: dict, git: dict | None, exit_code: int) -> list[str]:
+    """Every reason the run does not verify the requested configuration."""
+    problems: list[str] = []
+    if exit_code != 0:
+        problems.append(f"pytest exited {exit_code}")
+    if results.get("junit_missing"):
+        return [*problems, "no JUnit results (pytest did not run the suite)"]
+
+    tests = results["tests"]
+    names = [t["name"] for t in tests]
+    if len(names) != len(set(names)):
+        problems.append(f"duplicate test names: {sorted(names)}")
+    if set(names) != EXPECTED_TESTS:
+        missing, extra = EXPECTED_TESTS - set(names), set(names) - EXPECTED_TESTS
+        problems.append(f"ran the wrong test set (missing {sorted(missing)}, unexpected {sorted(extra)})")
+    for t in tests:
+        if t["outcome"] != "passed":
+            problems.append(f"{t['name']}: {t['outcome']}" + (f" ({t['message']})" if t.get("message") else ""))
+    if results["passed"] != len(EXPECTED_TESTS):
+        problems.append(f"{results['passed']} passed, expected exactly {len(EXPECTED_TESTS)}")
+
+    seen = next((t.get("properties", {}) for t in tests if t["name"] == IDENTITY_TEST), {})
+
+    def check(key: str, ok, want: str) -> None:
+        value = seen.get(key)
+        if value is None:
+            problems.append(f"runtime did not report {key}")
+        elif not ok(value):
+            problems.append(f"{key} = {value!r}, expected {want}")
+
+    check("observed_whisper_class", lambda v: v == WHISPER_CLASS, WHISPER_CLASS)
+    check("observed_ct2_class", lambda v: v == CT2_CLASS, CT2_CLASS)
+    check("observed_ct2_device", lambda v: v == device, repr(device))
+    if device == "cuda":
+        check("observed_ct2_compute_type", lambda v: v == compute_type, repr(compute_type))
+        check("observed_cuda_devices", lambda v: v.isdigit() and int(v) >= 1, ">= 1")
+    else:
+        check("observed_ct2_compute_type", lambda v: v.startswith(compute_type), f"{compute_type}*")
+    snapshot = f"models--Systran--faster-whisper-{MODEL}"
+    check("observed_model_path", lambda v: snapshot in Path(v).parts, f"a path inside {snapshot}/")
+    check("observed_ct2_is_multilingual", lambda v: v == str(not MODEL.endswith(".en")),
+          str(not MODEL.endswith(".en")))
+    check("observed_model_bin_sha256", lambda v: len(v) == 64, "a SHA-256 of model.bin")
+
+    if device == "cuda" and expect_gpu and not any(expect_gpu in g for g in gpus):
+        problems.append(f"nvidia-smi lists {gpus or 'no GPU'}, expected one containing {expect_gpu!r}")
+    for k, v in overrides.items():
+        if v:
+            problems.append(f"{k} is set ({v!r}); it can inject plugins or substitute modules")
+    if not git or git.get("dirty") is not False:
+        problems.append(f"checkout is not a clean commit: {git}")
+    return problems
+
+
+def main() -> None:
+    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    p.add_argument("--audio-dir", required=True, help="output of prepare_audio --num 5")
+    p.add_argument("--device", default="cuda", choices=["cuda", "cpu"])
+    p.add_argument("--compute-type", default=None, help="default: float16 on cuda, int8 on cpu")
+    p.add_argument("--expect-gpu", default="T4",
+                   help="substring nvidia-smi must report for --device cuda ('' to skip the check)")
+    p.add_argument("--out", default="results/whisper_verification")
+    a = p.parse_args()
+
+    compute_type = a.compute_type or ("float16" if a.device == "cuda" else "int8")
+    out = Path(a.out)
+    out.mkdir(parents=True, exist_ok=True)
+    junit = out / "junit.xml"
+    junit.unlink(missing_ok=True)  # never read a previous run's results
+    # junit_family=xunit1: the format that carries record_property values.
+    cmd = [sys.executable, "-m", "pytest", "-m", "whisper", "-rA", "-s", "-p", "no:cacheprovider",
+           "-o", "junit_family=xunit1", f"--junitxml={junit}", "tests/test_whisper.py"]
+    env = {**os.environ, "ECHORELAY_TEST_DEVICE": a.device,
+           "ECHORELAY_TEST_COMPUTE_TYPE": compute_type,
+           "ECHORELAY_TEST_AUDIO_DIR": str(Path(a.audio_dir).resolve())}
+    overrides = {k: os.environ.get(k) for k in OVERRIDE_VARS}
+
+    started = datetime.now(timezone.utc)
+    print("$", " ".join(cmd), flush=True)
+    proc = subprocess.run(cmd, cwd=REPO, env=env, capture_output=True, text=True, check=False)
+    finished = datetime.now(timezone.utc)
+    (out / "pytest.log").write_text(proc.stdout + proc.stderr)
+    print(proc.stdout + proc.stderr)
+
+    results = _junit_results(junit) if junit.exists() else {"collected": 0, "junit_missing": True}
+    env_info = environment()
+    problems = validate(results=results, device=a.device, compute_type=compute_type,
+                        expect_gpu=a.expect_gpu or None, gpus=env_info.get("gpus", []),
+                        overrides=overrides, git=env_info.get("git"), exit_code=proc.returncode)
+    complete = not problems
+    write_json(out / "record.json", {
+        "kind": "real_model_verification",
+        "started_utc": started.isoformat(), "finished_utc": finished.isoformat(),
+        "command": " ".join(["python", *cmd[1:]]),
+        "requested": {"model": MODEL, "device": a.device, "compute_type": compute_type,
+                      "expect_gpu": a.expect_gpu or None,
+                      "note": "requested settings; the observed identity below is the evidence"},
+        "env": {k: env[k] for k in ("ECHORELAY_TEST_DEVICE", "ECHORELAY_TEST_COMPUTE_TYPE",
+                                    "ECHORELAY_TEST_AUDIO_DIR")},
+        "pytest_overrides": {**overrides, "ECHORELAY_MODEL_DIR": os.environ.get("ECHORELAY_MODEL_DIR")},
+        "observed": next((t.get("properties", {}) for t in results.get("tests", [])
+                          if t["name"] == IDENTITY_TEST), {}),
+        "environment": env_info,
+        "nvidia_smi": _nvidia_smi(),
+        "pytest_exit_code": proc.returncode,
+        "results": results,
+        "expected_tests": sorted(EXPECTED_TESTS),
+        "complete": complete,
+        "problems": problems,
+    })
+    if complete:
+        print(f"COMPLETE: {results['passed']} passed, 0 failed, 0 errors, 0 skipped; "
+              f"observed {a.device}/{compute_type}; wrote {out}/")
+    else:
+        print(f"NOT COMPLETE ({len(problems)} problem(s)); wrote {out}/")
+        for prob in problems:
+            print(f"  - {prob}")
+    sys.exit(proc.returncode if proc.returncode else (0 if complete else 3))
+
+
+if __name__ == "__main__":
+    main()