[hydrangeas20]

Recorder and real-model test changes are not yet re-verified on a GPU. benchmarks/whisper_verification.py and tests/test_whisper.py changed here, and have been tested offline only. PR #9's verified T4 result (8/8, complete: true) remains historical evidence tied to commit 096a034. Its raw files are unchanged.

Review fixes F1–F6 (cac3739)

These came from the independent review of 1380606. Each was reproduced first, then fixed with regression tests, then re-run against the original fault.

#	Finding	Fix	Re-run of the fault
F1	An unreadable LD_LIBRARY_PATH entry made Path.exists/is_dir raise PermissionError, aborting record.json after the tests ran	Probes record the error (ld_library_path_entries_exist, search_errors). Each provenance section is collected independently; one that raises is recorded as collection_error and fails the run closed.	As a non-root user on 3.11 and 3.13: no crash, and the error is recorded
F2	/proc/self/maps paths with spaces were truncated (Drive/libcublas.so.12)	Pathname = everything after field 5; (deleted) dropped; parser shared and tested offline	A real library loaded from …/My Drive/… is reported with its full path
F3	"Loaded plugins" overstated: pytest's header omits -p, PYTEST_PLUGINS, conftest.py and pytest_plugins plugins	Renamed to installed_entry_points / header_reported, with a note; docs say these routes are covered by the override and checkout rules, not the inventory	A stub injected via conftest.py is absent from header_reported, as documented
F4	A password containing @ leaked a fragment; one containing / leaked whole; bare tokens weren't redacted	Userinfo removed up to the last @; secret-like KEY=value redacted anywhere, including query strings; GitHub / HF / AWS / sk- / Slack / GitLab / JWT formats redacted. Documented as best effort.	All four probe cases now redacted
F5	Integrity gap (since PR #9): the dirty check ignores untracked files, so an untracked root conftest.py ran inside the tests with dirty: False	Checkout audit, fail closed, taken before pytest starts: any untracked non-ignored file anywhere, or any ignored *.py/*.pyc/*.pyo/*.pth/*.so or pytest/Python config file in the repo root, tests/ or src/, makes the run incomplete; so does an audit failure	The stub was injected only via an untracked root conftest.py, with no overrides. All 8 tests "passed"; the recorder reported untracked files present (1): ['conftest.py']
F6	LD_PRELOAD mixed separators; the lifecycle test leaked its task on failure	Split on spaces and colons; finally cancels and awaits the task	unit test

Unchanged: the existing requirements (exactly 8 tests, model identity, WER bounds 0.12 / 0.25, clean commit, overrides), src/, dependencies, CI and raw evidence.

Problem

• Provenance gaps in the recorder. The PR test: verify real Whisper inference on NVIDIA T4 #9 run needed a LD_LIBRARY_PATH fix (cuBLAS 12) that the record could not capture. pytest auto-loaded plugins from the Colab image that the record listed only implicitly, in pytest.log. Nothing in the record showed which CUDA libraries were actually loaded.
• Stale baseline comment. The real-model test file still cited Phase 1's 0.0708 as the WER baseline, but that figure was scored with the v1 normalizer.
• Remaining timing fragility. After the two macOS flakes fixed in PR test: verify real Whisper inference on NVIDIA T4 #9, other tests still asserted after fixed sleeps or polled without a bound.

Changes

1. Test reliability (d8b24e2)

I audited the timing-sensitive tests in test_lifecycle.py, test_reliability.py, test_live_server.py, test_streaming.py, test_errors.py, test_load_test_timeout.py, test_observability.py and test_priority.py. Tests changed only where fragility was demonstrated, and every behavioural assertion is kept.

Test	Problem	Fix
test_lifecycle::…mid_partial_cancels_receive_and_partial	Fixed sleep(0.4) after a 0.3 s decode, then asserts the slot was released: ~100 ms of slack (the PR #9 flake class). Also an unbounded while in_flight == 0 poll.	Bounded until() for both; runner shut down in finally
test_reliability::test_slow_sender_within_idle_timeout_completes	Frames every 0.35 s against a 0.5 s idle limit: 150 ms margin	0.1 s of audio every 0.15 s. Still slower than real time (1.5×) and still longer than 0.5 s in total, with a 0.35 s margin.
test_reliability final-decode disconnect, queued disconnect, malformed message, shutdown	Unbounded while …: sleep polls: a regression hangs until the CI job timeout	_await_metrics(): bounded wait that fails with a message

Shown to detect what they're meant to catch. Each old test (from main) and new test ran under the same injected condition:

Test	Condition	Old (main)	New
lifecycle	none	pass	pass
lifecycle	worker thread returns 0.25 s late (loaded runner)	fail (assert 1 == 0 on in_flight)	pass
lifecycle	slot never released (mutation)	fail	fail ("timed out waiting for decode slot to be released")
lifecycle	partials never scheduled (mutation)	hang (killed at 25 s)	fail in 5 s ("timed out waiting for partial decode to start")
slow sender	none	pass	pass
slow sender	each client pacing sleep wakes 0.2 s late	fail (idle_timeout)	pass
slow sender	idle timer stops resetting per frame (mutation)	fail	fail (idle_timeout)
malformed message	partials never scheduled (mutation)	hang (killed at 25 s)	fail in 5 s

Left as is:

• live-server polls (already bounded)
• wall-clock bounds of 2–3 s on 0.3–0.5 s operations (idle, send-timeout, load-test)
• the event-loop-lag test (a deterministic lower bound)
• test_inference_does_not_block_event_loop (its < 0.5 s bound during a 1 s decode is the property under test, with 2× headroom)
• gate pumps (bounded overall)

2. Verification provenance (1380606, revised in cac3739)

The record gains a provenance section. It never changes complete. It separates what the environment said from what the test process loaded:

• environment_metadata: what the environment said. Explicitly labelled as not proof of loading.
  • Variables: only allowlisted ones are read, as passed to pytest: LD_LIBRARY_PATH (as entries), LD_PRELOAD, CUDA_HOME, CUDA_PATH, CUDA_VISIBLE_DEVICES, CUDA_MODULE_LOADING, NVIDIA_VISIBLE_DEVICES, NVIDIA_DRIVER_CAPABILITIES, PYTEST_DISABLE_PLUGIN_AUTOLOAD, and any CT2_*. Nothing outside the allowlist is read.
  • Libraries on the search path: which LD_LIBRARY_PATH directories hold libcublas.so.12, libcublasLt.so.12 and libcudnn.so.9.
  • Wheels: installed nvidia-* pip distributions and their versions.
  • Redaction: URL credentials and secret-looking KEY=value entries are removed before path lists are split.
• pytest_plugins: installed_entry_points (pytest11 entry points with distribution and version: what pytest can auto-load) and header_reported (the plugins: line pytest printed). Neither lists every plugin that ran. Plugins loaded via -p, PYTEST_PLUGINS, conftest.py or pytest_plugins don't appear in either; those routes are covered by the override and checkout rules instead.
• loaded_libraries: what was actually loaded. CUDA / CTranslate2 shared objects mapped into the pytest process after the real-speech decodes, read from /proc/self/maps (Linux). The real-speech test records this, before its assertions.

Stricter, not looser. PYTEST_PLUGINS now counts as an override, like PYTEST_ADDOPTS and PYTHONPATH, because it loads arbitrary plugin modules. No existing requirement is relaxed: the model-identity, exact-8-tests, WER and clean-checkout rules are unchanged.

Defect found and fixed during this work. My first redaction split LD_LIBRARY_PATH-style lists on : before redacting. That cut a URL apart, so the remaining user:pw@ piece was no longer recognised as a URL and would have been recorded. Redaction now runs on the whole value first. A regression test asserts that no fragment of the credentials survives.

Offline tests (tests/test_whisper_verification.py, +9) cover:

• the allowlist (secrets in other variables are never read)
• redaction, including a URL inside a path list
• search-path detection
• nvidia wheel listing
• plugin inventory and header parsing
• loaded-library parsing (present, absent, non-list, malformed)
• that provenance never affects completeness
• PYTEST_PLUGINS as an override

End-to-end dry run. With the stub model from PR #9, the recorder:

• wrote provenance, with the URL credential redacted
• captured the plugins and the loaded CTranslate2 library
• still reported NOT COMPLETE, naming the stub classes and the injected overrides

3. Documentation

• tests/test_whisper.py: the 0.0708 comment now explains the v1/v2 distinction. The v2 corpus value the tests compute is 0.0531; Phase 1's 0.0708 is the same transcripts scored with v1. The bounds (0.12 and 0.25) are unchanged.
• docs/whisper-verification.md:
  • a callout that the current code is not yet re-verified on a GPU
  • a "Status of later changes" table
  • an Environment provenance section: each field, what it is, what it is not, and the limits
  • "Reproducing the environment": pinned faster-whisper 1.2.1 / CTranslate2 4.8.3, the library-path fix, the recorder environment, and the weights snapshot
• docs/benchmarks/whisper-t4/README.md (narrative, not raw evidence): no longer claims the test file and recorder are unchanged since 096a034; states that the result applies to 096a034.
• README.md: the T4 statement names 096a034 and notes the offline-only status of later changes.

Guardrails

• Untouched: src/ (inference, scheduling, WebSocket protocol, model behaviour), WER thresholds, all raw benchmark artifacts including docs/benchmarks/whisper-t4/raw/, dependencies, and CI. Confirmed with git diff --quiet origin/main.
• No LICENSE, and no new GPU results.

Test results (exact)

Local, Linux, no GPU:

Check	Python 3.11.17	Python 3.13.16
python -m pytest -q -m "not whisper" × all 4 flag combinations (F1–F6 code, before two cosmetic lint edits)	307 passed, 8 deselected (each)	307 passed, 8 deselected (each)
same, default flags, at the final tree cac3739	307 passed, 8 deselected	307 passed, 8 deselected
python -m pytest -m whisper --collect-only -q	8/315 collected	8/315 collected
lifecycle + reliability + verification tests, 10 repeats	10/10	10/10
python -m benchmarks.verify_evidence	OK: 4 manifests, 173 files, 0 problems	—
git diff --check	clean	—
ruff --select F,E9 on changed files	1 finding (numpy unused in test_reliability.py, already on main)	—

307 = 269 on main + 38 new offline tests (9 provenance + 29 for F1–F6). The repo has no configured lint or type checker. With extended ruff rules, this PR adds no findings beyond those already on main; one deliberate broad except (provenance collection must never abort the record) is annotated.

CI (GitHub Actions, separate from the local runs): run 40214206943 at cac3739, 10/10 passed on the first attempt, including macOS. The previous head 1380606 also passed 10/10 (run 40207706030).

Real-model tests: not run. There's no GPU here and Hugging Face is unreachable. GPU verification at cac3739 is pending: the Colab cells are prepared, and the run must not use 1380606.

Remaining limitations

• Not GPU-verified.
  • The changed recorder and tests/test_whisper.py have not been run against the real model.
  • On a real GPU, the loaded-library filter's patterns are unconfirmed, as is what /proc/self/maps shows for the pip CUDA wheels.
  • Re-running the Colab cells from docs/whisper-verification.md at this commit would verify it and produce the first record with provenance.
• Provenance values are recorded, not enforced. A surprising LD_PRELOAD or an unexpected entry-point plugin does not make a run incomplete; only failing to collect provenance does.
• Redaction is best effort. The allowlist is the main control; an unknown token format in an allowlisted variable would be recorded.
• Checkout audit scope. It checks git's view of the checkout. Files outside the repository (site-packages, ~/.config) are not audited; they are covered only by the identity checks and by running in a fresh runtime.
• loaded_libraries is only as trustworthy as the test code. It comes from the tests' own process. The existing integrity rules (no overrides, clean checkout, exact test set) are what protect it.
• Search-path metadata is partial. It covers LD_LIBRARY_PATH only, not RUNPATH, ld.so.cache or system directories, which is why the actually-loaded list exists.
• No log access from here. Other timing-sensitive tests could still exist. A future macOS failure should be traced from the Actions log, which this environment cannot read.

Pytest plugin enforcement: decided, inventory only

Per review, this PR keeps the inventory-only approach. It records the installed and reported plugins without allowlisting them or disabling auto-loading. Plugin enforcement may be evaluated separately in a future change; options include an allowlist, or PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 -p asyncio.

https://claude.ai/code/session_017DfLG9q5h7eiawGNwaBswq

---

Generated by Claude Code

[hydrangeas20]

Final Engineering Review — PR #10

Review status: Conditionally approved, pending NVIDIA T4 re-verification.

Reviewed the verification-hardening changes at 1380606 and confirmed that GitHub Actions run 40207706030 passed all 10 checks, including macOS.

What looks good

• Test reliability: Timing-sensitive assertions now use bounded, condition-based waits where genuine fragility was demonstrated. Fault-injection tests provide evidence that the revised tests still detect lifecycle, timeout, and scheduling regressions.
• Verification provenance: The recorder now distinguishes environment configuration, installed pytest plugins, and libraries actually loaded into the test process.
• Evidence integrity: Existing completeness requirements remain intact, with PYTEST_PLUGINS added to the prohibited overrides.
• Documentation accuracy: The v1/v2 WER distinction is corrected without changing thresholds or claiming an inference improvement.
• Reproducibility: Historical T4 evidence remains unchanged and tied to commit 096a034. The updated recorder is correctly identified as not yet GPU-reverified.

Final verification required

Before merging, rerun the eight real-model tests on an NVIDIA T4 using the exact PR head (1380606).

Confirm:

1. All eight tests pass and the recorder reports complete: true.
2. provenance.loaded_libraries captures the expected CUDA/CTranslate2 libraries after real inference.
3. CUDA library paths, installed NVIDIA packages, and pytest plugin metadata are recorded correctly.
4. No credentials or sensitive environment values appear in the evidence.
5. Existing WER thresholds and evidence-integrity requirements remain satisfied.

Preserve the new verification output separately from PR #9's historical evidence.

Pytest plugin policy

Decision: Defer plugin enforcement to a separate follow-up.

Keep the current inventory-only approach for PR #10. For future verification runs, prefer evaluating PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 with explicitly loaded required plugins, rather than maintaining an environment-specific allowlist.

This requires separate validation because it changes the test environment and potentially the definition of a complete run.

Merge decision

No blocking issues identified in the reviewed changes beyond the outstanding real-GPU verification.

Once the T4 run passes and its provenance is inspected, PR #10 is cleared for squash merge.

Do not merge until that verification is complete.

[hydrangeas20]

Final Verification — NVIDIA T4 / CUDA FP16

PR #10 verification is complete. The final run was executed in a Google Colab NVIDIA Tesla T4 environment against the exact PR head:

Commit: cac3739b09395bb04a37f92c14fecd23e2f837a3

Verification Results

• 8/8 real-model tests passed — 0 failures, 0 errors, 0 skipped.
• Recorder: exit code 0; complete: true; 0 recorded problems.
• Runtime: faster-whisper WhisperModel backed by CTranslate2 Whisper.
• Model: base.en, with model snapshot and binary SHA-256 recorded.
• Execution: CUDA / FP16 on NVIDIA Tesla T4.
• Loaded-library provenance: CUDA driver and cuBLAS libraries observed in the pytest process memory maps.
• Checkout integrity: clean Git working tree; no unexpected executable files identified by the checkout audit.
• Environment: no PYTHONPATH, PYTEST_ADDOPTS, or PYTEST_PLUGINS overrides.
• Provenance: required sections collected without errors.

Speech Evaluation

• Final corpus WER: 5.31% across five reference utterances.
• WebSocket final WER: 11.54% for the streaming test.

These are workload-specific measurements, not claims of improved decoding accuracy or broader production performance.

Environment Issue and Resolution

The initial Colab run failed because CTranslate2 could not resolve libcublas.so.12. After configuring the CUDA 12 cuBLAS and cuDNN 9 library search paths, the unchanged verification tests passed.

No test thresholds, application logic, or historical benchmark evidence were modified to obtain the passing result.

Final Assessment

The PR #10 verification-hardening changes have passed the available CI checks and the final real-GPU verification. The evidence archive, pr10_t4_verification.zip, preserves the verification record, JUnit results, console output, checkout audit, and runtime provenance.

Recommendation: Approved for squash merge, provided the PR head remains at the verified commit and required CI checks remain green.