diff --git a/HANDOFF.md b/HANDOFF.md index 8c11410..ad07ad6 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -756,6 +756,6 @@ Resolved since rev 1: ~~license~~ (MIT), ~~core language~~ (header-only C++20), Resolved since rev 3: ~~Max external naming~~ — settled (Rev 4): **`mutap.afc~`** (rename from the `mutap.defeed~` placeholder) and **`mutap.aec~`** for the new echo canceller, an acronym pair matching the literature. The rename executes in Stage 2 of "The next effort" above. Still open: -- **Wake-word detection — in progress (M0–M3 done).** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. **M2 is done** on MuTap's `claude/mutap-wake-word-plan-2i63pe`: the learned suppressor has its oracles — typed float/double tests with a −120 dB cross-precision pin, a float32 chain gate, a Python↔C++ parity CI job in both profiles on random and exported weights, a Cortex-M33 QEMU leg (mps2-an505, Ooura float32 FFT) with the float suppressor suite on-target on every leg, and `nn_suppressor` icount scenarios with baselines on m55, m33 and hexagon. The oracles found and fixed a Nyquist-bin contract defect at 48 kHz (C++ notched bin N/2, numpy did not; fixed on both sides). **M3 is done**: the dense/GRU kernels are DspTap's `tap::dsp::nn` (tap/DspTap#15) and `nn_suppressor` consumes them, bit-identically in both profiles on the shipping v2 model, with the M2 battery, the six parity cases and the ratchet unchanged; MuTap's DspTap pin points at the M3 tree (repoint at `main` after the DspTap PR merges) and MuTap-Max's pin follows once this merges. Alongside it, the M33 finding's root cause was fixed in the library: the speech predictor's pitch search now accumulates in `Sample` (the double profile is bit-for-bit unchanged, the float rows unchanged to the MSG bisection's quantum), the tonal PEM headline dropped from 1085 s to 31 s on the M33 emulator, and the M33 leg runs the full float selection (174 s) — the M2 exclusion is gone. Next: M4, the corpus and dataset builder, on the Apple Silicon Mac — **after** the amendments in [`docs/wake-word-m4-review.md`](docs/wake-word-m4-review.md) (8 September 2026): an adversarial review of the M4 section in the rev-1 audit's format found the skeleton right but fifteen root issues to settle before building, one critical (the recorded hold-out is human voice data going into a public MIT repo with consent scope, audio licence, storage form and size undecided, and the RIR fixture generator stamps MIT on everything it emits), plus two facts that moved under the plan — the maintained Piper is GPL-3 (espeak-ng embedded) and Common Voice is now account-gated on Mozilla Data Collective with a no-re-hosting clause. It proposes staging M4 as M4a (builder, contracts, Speech Commands, a toy CI fixture) → M4b (full corpus, phrase, splits, lock, card) → M4c (hold-out, host path only; the Pico path at M7), with the ≥ 10-talker / ≥ 200-utterance target enforced at M6's pass. Rev 3 of the M4 section landed on 9 September 2026 with every amendment and both decisions below written in, staged M4a → M4b → M4c; next is M4a on the Mac — `kws_features.py` over the bridge with its self-check, the `manifest.json` / `lock.json` schema, `kws_build.py`, `verify_splits.py`, `kws_dataset_card.py`, Speech Commands v2, the toy fixture and the `kws-dataset` CI job — which unblocks M5. **M4a is done (9 September 2026, tap/MuTap#48):** `tools/ml/kws/` carries `kws_features.py` (the front end through the bridge, the band-support assertion, the self-check at the reference and manifest geometries, the bridge keyed to the DspTap commit), the manifest/lock schemas with refuse-by-name validation, `kws_build.py` with its seven stages and `--jobs`-invariant lock, `verify_splits.py` R1–R5 with planted fixtures, `kws_dataset_card.py`, a 662 KB toy fixture whose committed lock, shards and card CI rebuilds (bit-for-bit on the M0 Mac; inside the 1e-6 pin on ubuntu-latest, 1 m 20 s), the `synth` hand-run over `en_US-kristin-medium` (piper-tts 1.8.0 as an isolated subprocess) recorded as `fixtures/synth_hand_run/`, and the M5 bring-up corpus `manifests/speech_commands_v2_bringup.json` built on the Mac in 396 s with `--jobs 8` (Speech Commands v0.02 in full, MUSAN noise, 60,000 SLR28 RIRs; 200,653 featurized rows, 27.5 M frames, 784 shards; `marvin` 1,710 / 195 / 195; eval-speech 3.0 h, so its zero-event bound is 1.0 FA/h; R1–R5 clean, licence checks pass; card committed beside the manifest); 74 tests. The plan's M4a Done record carries every number. Next: M5's harness bring-up on that corpus, in parallel with M4b, whose first act is the development phrase. **Two decisions it needed were taken on 8 September 2026:** (1) **The hold-out audio stays out of git.** It lives as FLAC in the off-git store (or a private release asset) and the repository commits only the record, `holdout.json` — per utterance: sha256, talker pseudonym, microphone path, distance, SNR, phrase, endpoint sample; per talker: pseudonym, consent-form version, permitted uses. The M5 harness refuses a hold-out whose file hashes do not match the committed rows (the RIR precedent's provenance-or-refuse rule, without its C-header encoding). Consent scope is evaluation use by the project, indefinite retention, loudspeaker replay at M7, publication of aggregate numbers, pseudonymous handling and a working withdrawal path — never public-repo permanence — plus one optional opt-in per talker, "may be published under CC BY 4.0, attributed to the collection, not to me", so an opted-in subset can be released later without re-consenting anyone. The M4c pilot is n = 1 (Tim). (2) **Training features come from the shipping front end**, generated through the DspTap C ABI bridge (`dsptap_py.LogMel`), which takes every `log_mel_geometry` value and agrees with the numpy reference to 1.5e-14 at the reference geometry; `make_frontend_reference.py` stays what M1 built it as, the oracle, and parameterizing it is a small DspTap follow-up so the parity self-check can also run at a tuned geometry — not an M4a prerequisite. So nothing blocks M4a on a submodule round-trip, and geometry stays tunable from the first build. +- **Wake-word detection — in progress (M0–M3 done).** A background briefing, a staged implementation proposal (rev 2) and the adversarial audit that produced it landed in [`docs/wake-word-briefing.md`](docs/wake-word-briefing.md), [`docs/wake-word-plan.md`](docs/wake-word-plan.md) and [`docs/wake-word-audit.md`](docs/wake-word-audit.md). The proposal reuses the `nn_suppressor` / `tools/ml` patterns rather than importing a runtime; the audit found those patterns sound but the learned path's *oracles* missing (the suppressor is double-only everywhere, never on-target, never instruction-counted, parity not in CI), so rev 2 adds an M2 that builds them before the kernels are promoted — a milestone worth doing whether or not the spotter ships. The named M33 target is the Raspberry Pi Pico 2 W (RP2350: single-precision FPU, so float32 is its profile; the M33 QEMU rig is ported from RatioTap in M2, and M7 adds a `pico2w` CI job that builds the board example against a pinned Pico SDK, uploads the UF2 and asserts its footprint — detection itself is a bench step with a loudspeaker-playback protocol and committed numbers, since QEMU has no RP2350 model), and the docs plan includes a user guide to training a phrase whose commands CI runs on a toy corpus. **M0 is decided (4 September 2026):** repository MuTap, charter widened; host rate fixed at 16 kHz in `kws.h` with conversion as an `@resample` option on the Max external, backed by a new DspTap `decimate.h` (2/3/6) and composed with RatioTap for 44.1 kHz, since `poly~` is powers-of-two only and neither RatioTap nor SampleRateTap covers 48 → 16; release shape runtime-first, no bundled phrase, the training guide as the primary document; development phrase `marvin` from Speech Commands for M5 bring-up, a synthesized four-syllable phrase from M4, never shipped; TTS voices lineage-verified from the Piper model cards — `en_US-libritts-high` (from scratch, CC BY 4.0, 904 speakers), `en_US-kristin-medium`, `en_GB-cori-high` (from scratch, public domain), `en_US-john-medium` (from Kristin), with every Lessac-derived voice (most of the English set, `libritts_r` and `vctk` included) and the sample generator's bundled `.pt` generator excluded; training on an Apple Silicon Mac via `--device mps`, under an hour per run on the development set. **M1 is done** on DspTap's `claude/mutap-wake-word-plan-2i63pe`: `log_mel.h` and `decimate.h` with their typed batteries, the numpy reference generator (`tools/reference/make_frontend_reference.py` — the family's single numpy copy of the formulas), C ABI and bridge; every tolerance in the plan's M1 record is a measured number. **M2 is done** on MuTap's `claude/mutap-wake-word-plan-2i63pe`: the learned suppressor has its oracles — typed float/double tests with a −120 dB cross-precision pin, a float32 chain gate, a Python↔C++ parity CI job in both profiles on random and exported weights, a Cortex-M33 QEMU leg (mps2-an505, Ooura float32 FFT) with the float suppressor suite on-target on every leg, and `nn_suppressor` icount scenarios with baselines on m55, m33 and hexagon. The oracles found and fixed a Nyquist-bin contract defect at 48 kHz (C++ notched bin N/2, numpy did not; fixed on both sides). **M3 is done**: the dense/GRU kernels are DspTap's `tap::dsp::nn` (tap/DspTap#15) and `nn_suppressor` consumes them, bit-identically in both profiles on the shipping v2 model, with the M2 battery, the six parity cases and the ratchet unchanged; MuTap's DspTap pin points at the M3 tree (repoint at `main` after the DspTap PR merges) and MuTap-Max's pin follows once this merges. Alongside it, the M33 finding's root cause was fixed in the library: the speech predictor's pitch search now accumulates in `Sample` (the double profile is bit-for-bit unchanged, the float rows unchanged to the MSG bisection's quantum), the tonal PEM headline dropped from 1085 s to 31 s on the M33 emulator, and the M33 leg runs the full float selection (174 s) — the M2 exclusion is gone. Next: M4, the corpus and dataset builder, on the Apple Silicon Mac — **after** the amendments in [`docs/wake-word-m4-review.md`](docs/wake-word-m4-review.md) (8 September 2026): an adversarial review of the M4 section in the rev-1 audit's format found the skeleton right but fifteen root issues to settle before building, one critical (the recorded hold-out is human voice data going into a public MIT repo with consent scope, audio licence, storage form and size undecided, and the RIR fixture generator stamps MIT on everything it emits), plus two facts that moved under the plan — the maintained Piper is GPL-3 (espeak-ng embedded) and Common Voice is now account-gated on Mozilla Data Collective with a no-re-hosting clause. It proposes staging M4 as M4a (builder, contracts, Speech Commands, a toy CI fixture) → M4b (full corpus, phrase, splits, lock, card) → M4c (hold-out, host path only; the Pico path at M7), with the ≥ 10-talker / ≥ 200-utterance target enforced at M6's pass. Rev 3 of the M4 section landed on 9 September 2026 with every amendment and both decisions below written in, staged M4a → M4b → M4c; next is M4a on the Mac — `kws_features.py` over the bridge with its self-check, the `manifest.json` / `lock.json` schema, `kws_build.py`, `verify_splits.py`, `kws_dataset_card.py`, Speech Commands v2, the toy fixture and the `kws-dataset` CI job — which unblocks M5. **M4a is done (9 September 2026, tap/MuTap#48):** `tools/ml/kws/` carries `kws_features.py` (the front end through the bridge, the band-support assertion, the self-check at the reference and manifest geometries, the bridge keyed to the DspTap commit), the manifest/lock schemas with refuse-by-name validation, `kws_build.py` with its seven stages and `--jobs`-invariant lock, `verify_splits.py` R1–R5 with planted fixtures, `kws_dataset_card.py`, a 662 KB toy fixture whose committed lock, shards and card CI rebuilds (bit-for-bit on the M0 Mac; inside the 1e-6 pin on ubuntu-latest, 1 m 20 s), the `synth` hand-run over `en_US-kristin-medium` (piper-tts 1.8.0 as an isolated subprocess) recorded as `fixtures/synth_hand_run/`, and the M5 bring-up corpus `manifests/speech_commands_v2_bringup.json` built on the Mac in 396 s with `--jobs 8` (Speech Commands v0.02 in full, MUSAN noise, 60,000 SLR28 RIRs; 200,653 featurized rows, 27.5 M frames, 784 shards; `marvin` 1,710 / 195 / 195; the eval-speech negatives 2.95 h, so the zero-event bound is 1.02 FA/h; R1–R5 clean, licence checks pass; card committed beside the manifest); 74 tests. The plan's M4a Done record carries every number. **M5 is done (9 September 2026, `feat/wake-word-m5`):** the evaluation harness before any model — `kws_scoring.py` (the scoring semantics as numbers: hit window [h − T, h + 20 + T], the reference decision stage W = 10 / R = 100 that `kws.h` must match, exact Poisson and Wilson intervals, ln 20 / H at zero events), `kws_streams.py`, `kws_detectors.py` (band-energy baseline through the bridge; the oracle's planted detector), `kws_holdout.py` (M4c's `holdout.json`, hash- and consent-verified before scoring), `kws_eval.py` (sweep + the committed report with speech / music / TTS FA/h columns and the eval-tts recall beside the hold-out's); the planted-event oracle and the mis-accounted variants are the pass (44 tests in CI); on the bring-up corpus the eval-speech denominator is 2.9513 h (bound 1.015 FA/h) and the sanity curve in the executed `notebooks/kws_det.ipynb` (script-built) reads best recall 34/195 at 1,977 FA/h — useless, as expected, not a pass. The plan's M5 Done record carries every number. Next: M4b (the development phrase first; MSWC and Common Voice need Tim's account steps) and M6 (`kws.h`, which pins the decision-stage parity against this harness). **Two decisions it needed were taken on 8 September 2026:** (1) **The hold-out audio stays out of git.** It lives as FLAC in the off-git store (or a private release asset) and the repository commits only the record, `holdout.json` — per utterance: sha256, talker pseudonym, microphone path, distance, SNR, phrase, endpoint sample; per talker: pseudonym, consent-form version, permitted uses. The M5 harness refuses a hold-out whose file hashes do not match the committed rows (the RIR precedent's provenance-or-refuse rule, without its C-header encoding). Consent scope is evaluation use by the project, indefinite retention, loudspeaker replay at M7, publication of aggregate numbers, pseudonymous handling and a working withdrawal path — never public-repo permanence — plus one optional opt-in per talker, "may be published under CC BY 4.0, attributed to the collection, not to me", so an opted-in subset can be released later without re-consenting anyone. The M4c pilot is n = 1 (Tim). (2) **Training features come from the shipping front end**, generated through the DspTap C ABI bridge (`dsptap_py.LogMel`), which takes every `log_mel_geometry` value and agrees with the numpy reference to 1.5e-14 at the reference geometry; `make_frontend_reference.py` stays what M1 built it as, the oracle, and parameterizing it is a small DspTap follow-up so the parity self-check can also run at a tuned geometry — not an M4a prerequisite. So nothing blocks M4a on a submodule round-trip, and geometry stays tunable from the first build. - **Default engine in the external** — `@kalman` off (classic NLMS) is the shipping default purely on seniority; the measured case for flipping it is in `tests/test_fd_kalman.cpp` and book chapter 1. Decide after real-room listening. - **RIR fixtures, the measured half** — the fixture pipeline is built and three physically-modeled rooms (image-source, documented geometry) are committed baselines with regression tests. What remains yours: which MEASURED rooms join them — an academic dataset room (MYRiAD is the PEM-AFROW group's own database; openAIR is the other usual source; check each room's license allows redistribution in an MIT repo) and/or your own swept-sine measurements. Either way it is one command per room: `python3 tools/fixtures/make_rir_fixtures.py --from-wav room.wav myroom --source ""`, then a test with a freshly measured threshold. (The dataset hosts are unreachable from the remote dev container's network policy, so the WAVs have to enter via a commit.) diff --git a/docs/wake-word-plan.md b/docs/wake-word-plan.md index 1c10b22..1c40108 100644 --- a/docs/wake-word-plan.md +++ b/docs/wake-word-plan.md @@ -849,10 +849,11 @@ simulated RIRs, the keyword list; *K* = 2, *M* = 1, dry share 0.25, context 200,653 rows, 27.5 M frames, in 49.6 s; `shard` 784 shards in 227.6 s; 396 s in all at 2.3 GB peak resident; the lock 164 MB, the features 3.6 GB, the augmented tier 4.9 GB. `marvin` positives 1,710 / 195 / 195 (train / dev / -eval) and negatives 83,133 / 9,786 / 10,810; the eval-speech share is 3.0 h -and the eval-noise share 0.5 h, so its zero-event bound is ln 20 / 3.0 = -1.0 FA/h — the bring-up figure this section predicted, and the reason M4b's -eval floors are 20 h; 27,455 of the 60,000 RIRs lie inside `rt60_s` +eval) and negatives 83,133 / 9,786 / 10,810; the eval-speech negatives are +2.95 h (3.0 h with the positives, which the FA/h denominator excludes) and the +eval-noise share 0.5 h, so the zero-event bound is ln 20 / 2.95 = 1.02 FA/h — +the bring-up figure this section predicted, and the reason M4b's eval floors +are 20 h; 27,455 of the 60,000 RIRs lie inside `rt60_s` [0.2, 1.0] (median RT60 0.67 s); `verify_splits` clean over 263,487 rows in 2.4 s; the card's licence checks pass, and the card is committed beside the manifest. Not in M4a, by design: any TTS beyond the hand-run, music in @@ -935,6 +936,46 @@ harness against hand-computed recall and FA/hour, exact to the utterance; a deliberately mis-accounted variant must be rejected. The trivial band-energy baseline is run as a sanity curve, not as the pass. +**Done, 9 September 2026** (MuTap branch `feat/wake-word-m5`; DspTap pin +`5ca3b1c`). Measured, not estimated. `tools/ml/kws/`: `kws_scoring.py` — the +scoring semantics as numbers, checked against hand-computed values: the hit +window [*h* − *T*, *h* + 20 + *T*] inclusive with *h* = ⌊*e*/160⌋; the +reference decision stage — a trailing moving average over *W* = 10 hops, an +upward crossing of the threshold fires, a crossing fewer than *R* = 100 hops +after an event merges into it (the numbers `kws.h` must match at M6); the +exact two-sided Poisson 95 % interval, the zero-event bound ln 20 / *H* and the +Wilson interval. `kws_streams.py` assembles every eval positive as the mixture +`extract` featurized and packs each share's negatives into streams of at most +60 s, hours from decoded lengths, every mis-accounting refused by name; +`kws_detectors.py` carries the detector contract (one score per completed hop, +through the bridge), the band-energy baseline and the oracle's planted +detector; `kws_holdout.py` carries M4c's `holdout.json`, verified by file hash +and consent row before anything is scored, and the hold-out set id; +`kws_eval.py` the sweep and the committed report — speech, music and TTS +speech as separate FA/h columns, an absent share absent, every figure with its +hours and interval, the eval-tts recall beside the hold-out's. The pass, in CI +(44 tests in 2 s): the planted-event oracle — hits exact to the utterance at +both inclusive window edges and one hop outside each, four planted events of +which two merge under the refractory period (3 in 0.5 h → 6.0 FA/h, +[1.24, 17.53]), events exactly *R* apart both firing and *R* − 1 apart +merging, a zero-event share reporting ln 20 / *H*, spurious events never in +FA/h — and the mis-accounted variants each refused by name (a wrong or +non-integer hours count, an endpoint or window beyond the stream, duplicate +ids, a positive stream with hours, NaN, short or out-of-range scores). On the +bring-up corpus (M0 Mac): 412 streams — 195 positives; the eval-speech +negatives 2.9513 h in 179 streams (zero-event bound 1.015 FA/h) and eval-noise +0.5021 h in 38; the band-energy baseline through the bridge scores about +2,500 s of audio per second including decoding; the sweep 8.6 s wall; the +executed notebook `notebooks/kws_det.ipynb` (built by +`tools/ml/build_kws_det_notebook.py` in 14 s, 0.12 MB) carries the sanity +curve — best recall 34/195 at 1,977 FA/h on speech, useless as the plan expects +of a band-energy mean, and not a pass. Every stream is decided after a fresh +reset, so each FA/h row depends on the packing bound by at most streams / *H* +(60.65 FA/h at θ = 0 on this corpus; 12–40 FA/h, 0.6–3.1 %, at thresholds +0.1–0.9), which the report records as `max_stream_s`. Not in M5, by design: +the engine through the `mutap_kws_*` C ABI (M6, which pins the harness ↔ +`kws.h` decision-stage parity) and any recorded hold-out (M4c). + ### M6 — The spotter *(MuTap)* `kws.h` per the §5 contract, over the M1 features and M3 kernels, with the diff --git a/notebooks/kws_det.ipynb b/notebooks/kws_det.ipynb new file mode 100644 index 0000000..28bf00d --- /dev/null +++ b/notebooks/kws_det.ipynb @@ -0,0 +1,660 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "3ab8da15", + "metadata": {}, + "source": [ + "# Wake word — the M5 evaluation harness on the bring-up corpus\n", + "\n", + "The wake-word plan's M5 ([`docs/wake-word-plan.md`](../docs/wake-word-plan.md) §6) builds the\n", + "evaluation harness *before any model*: the scoring semantics defined as numbers, a threshold sweep, a\n", + "committed report format with false accepts per hour on speech, music and TTS speech separately, each\n", + "figure with its hours and its interval. This notebook is that harness run end to end through the\n", + "shipping front end (DspTap's C ABI, `kws_features.FrontEnd`) on the corpus a manifest names, with the\n", + "plan's **trivial band-energy baseline** as the detector.\n", + "\n", + "It is the **sanity curve the plan asks for, not a pass**. The pass is\n", + "[`test_kws_eval.py`](../tools/ml/kws/test_kws_eval.py)'s planted-event oracle, which runs in CI; the\n", + "DET evaluation is deliberately not in CI (§7) and lives here, executed and committed, rebuilt by\n", + "[`tools/ml/build_kws_det_notebook.py`](../tools/ml/build_kws_det_notebook.py) whenever behaviour\n", + "changes. The operating point is M6's, measured on the hold-out.\n", + "\n", + "Manifest: `tools/ml/kws/manifests/speech_commands_v2_bringup.json`. The store comes from `MUTAP_KWS_STORE` (no default); nothing in the\n", + "corpus enters git." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "4bbc4760", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:12.004498Z", + "iopub.status.busy": "2026-09-09T16:44:12.004204Z", + "iopub.status.idle": "2026-09-09T16:44:14.252702Z", + "shell.execute_reply": "2026-09-09T16:44:14.252257Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "manifest speech_commands_v2_bringup (54b94a324b4b), store mutap-kws-store (from $MUTAP_KWS_STORE)\n", + "lock: 263487 rows read in 1.67 s; eval_set_id per share:\n", + " eval-noise: a7e8313152310948e177e5177b4c53f2e00674f16efc6fba5f7482d334123e63\n", + " eval-speech: ad456333ea8761839fe78fbc450c8bd63d29d905ec8cfaf7305501bb9cd5705f\n", + "front end: log_mel_contract_version 1, DspTap 5ca3b1cdffc4d724fdcef4bafb06e2908defda1d\n" + ] + } + ], + "source": [ + "import json, os, pathlib, subprocess, sys, time\n", + "import numpy as np\n", + "\n", + "REPO = pathlib.Path.cwd().parent if pathlib.Path.cwd().name == \"notebooks\" else pathlib.Path.cwd()\n", + "sys.path.insert(0, str(REPO / \"tools\" / \"ml\" / \"kws\"))\n", + "import kws_detectors, kws_eval, kws_features, kws_scoring, kws_streams\n", + "from kws_manifest import load_manifest, manifest_hash, read_lock\n", + "from kws_store import ENV_VAR, Store, resolve_store\n", + "\n", + "STORE = resolve_store(None) # refuses without MUTAP_KWS_STORE: there is no default store\n", + "MANIFEST = REPO / 'tools/ml/kws/manifests/speech_commands_v2_bringup.json'\n", + "manifest = load_manifest(MANIFEST)\n", + "LOCK = Store(STORE).features(manifest_hash(manifest)) / 'lock.json'\n", + "t0 = time.perf_counter()\n", + "lock = read_lock(LOCK)\n", + "t_lock = time.perf_counter() - t0\n", + "if lock.manifest_hash != manifest_hash(manifest):\n", + " raise RuntimeError(f\"lock {LOCK} was built from manifest {lock.manifest_hash}, this manifest hashes \"\n", + " f\"to {manifest_hash(manifest)}\")\n", + "print(f\"manifest {manifest.name} ({manifest_hash(manifest)[:12]}), store {STORE.name} \"\n", + " f\"(from ${ENV_VAR})\")\n", + "print(f\"lock: {len(lock.clips)} rows read in {t_lock:.2f} s; eval_set_id per share:\")\n", + "for share, sid in sorted(lock.eval_set_id.items()):\n", + " print(f\" {share}: {sid}\")\n", + "print(f\"front end: log_mel_contract_version {kws_features.contract_version()}, \"\n", + " f\"DspTap {kws_features.dsptap_commit()}\")" + ] + }, + { + "cell_type": "markdown", + "id": "1f57cc11", + "metadata": {}, + "source": [ + "## The scoring semantics, as numbers\n", + "\n", + "Everything below is `tools/ml/kws/kws_scoring.py` (checked against hand-computed values by its own\n", + "self-check, which the next cell runs) with the manifest's tolerance filled in; the notebook restates\n", + "nothing.\n", + "\n", + "- **Hop alignment.** A detector yields one score per completed front-end hop of 160 samples at\n", + " 16000 Hz: score[*t*] belongs to frame *t*, complete when sample (*t* + 1) · 160 − 1\n", + " arrives (`log_mel.h`'s alignment). An endpoint at sample *e* has hop index *h* = ⌊*e* / 160⌋.\n", + "- **The hit window** of a positive is frames [*h* − *T*, *h* + *L* + *T*], both ends inclusive, with\n", + " *T* = 3 hops (the manifest's `label.tolerance_hops`) and *L* = 20 hops\n", + " (`kws_scoring.LATENCY_CEILING_HOPS`, the §7 detection-latency ceiling of\n", + " 200 ms). One hit per utterance.\n", + "- **The reference decision stage.** The score is smoothed by a trailing moving average over\n", + " *W* = 10 hops; an event fires at hop *t* when the smoothed score crosses the threshold\n", + " upward (s[*t*] ≥ θ and s[*t* − 1] < θ, or *t* = 0 and s[0] ≥ θ) and at least\n", + " *R* = 100 hops have passed since the previous event (*t* − last ≥ *R*: a crossing\n", + " exactly *R* hops after an event fires, one *R* − 1 hops after it merges into it) — the refractory\n", + " period, under which false accepts merge. (`kws.h` carries these numbers at M6 and must match them.)\n", + "- **False accepts.** An event on a negative stream; FA/h = events / *H*, *H* the negative streams'\n", + " decoded duration in hours, per share (speech, music, TTS speech and noise separately). Every FA/h\n", + " carries *H* and the exact two-sided 95 % Poisson interval on the count (chi-square form); at zero\n", + " events the one-sided 95 % upper bound is ln 20 / *H*. An event on a positive stream outside every hit\n", + " window is *spurious*: reported, never in FA/h.\n", + "- **Recall** = hits / positives with its Wilson 95 % interval; FRR = 1 − recall. A share the lock lacks\n", + " is reported as *absent*, never as 0 FA/h.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "1cdfa88b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:14.254073Z", + "iopub.status.busy": "2026-09-09T16:44:14.253892Z", + "iopub.status.idle": "2026-09-09T16:44:14.716620Z", + "shell.execute_reply": "2026-09-09T16:44:14.716141Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'hop': 160, 'sample_rate': 16000, 'tolerance_hops': 3, 'latency_hops': 20, 'smoothing_hops': 10, 'refractory_hops': 100}\n", + "an endpoint at sample 24000: hop 150, hit window (147, 173)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "kws_scoring: self-check ok\n" + ] + } + ], + "source": [ + "# the scoring numbers are built from the manifest (hop and rate from the geometry, T from the\n", + "# label rule); kws_streams refuses any other value, since the lock's rows were cut for these\n", + "g = manifest.recipe.geometry\n", + "scoring = kws_scoring.Scoring(hop=g.hop, sample_rate=int(g.sample_rate),\n", + " tolerance_hops=manifest.recipe.label.tolerance_hops,\n", + " smoothing_hops=10, refractory_hops=100)\n", + "print(scoring.to_dict())\n", + "e = 24000\n", + "h = scoring.endpoint_hop(e)\n", + "print(f\"an endpoint at sample {e}: hop {h}, hit window {scoring.hit_window(h)}\")\n", + "r = subprocess.run([sys.executable, str(REPO / \"tools/ml/kws/kws_scoring.py\")], capture_output=True,\n", + " text=True)\n", + "print(r.stdout.strip() or r.stderr.strip())\n", + "if r.returncode != 0:\n", + " raise RuntimeError(\"kws_scoring self-check failed\")" + ] + }, + { + "cell_type": "markdown", + "id": "0362ed58", + "metadata": {}, + "source": [ + "## The streams\n", + "\n", + "`kws_streams.streams_from_lock` turns the lock's eval shares into the streams the detector scores after\n", + "a single reset, so the smoothing window and the refractory period run across clip boundaries as they do\n", + "in a room. Every variant-0 eval positive becomes one stream whose audio is the row's mixture — exactly\n", + "what `extract` featurized (context + keyword + the (L + T) hops the hit window needs). The negatives of\n", + "each share are sorted by id and packed into streams of at most 60 s, a clip never split (a\n", + "clip longer than that stands alone). The hours denominator is the sum of the negative streams' decoded\n", + "lengths — the plan's rule — so the positives' audio never enters it. Every stream is decided after a\n", + "fresh reset, so the packing bound is provenance the report records (`max_stream_s`): the *t* = 0 rule and\n", + "the refractory restart at each stream boundary make every FA/h figure depend on it by at most\n", + "streams / *H* per share, the ceiling printed below and reached at θ = 0. The accounting of every stream\n", + "is checked against its decoded audio (`kws_streams.validate_stream`, one rule set) by the scoring pass\n", + "in the next section, which decodes each stream exactly once." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "9dca11d3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:14.718015Z", + "iopub.status.busy": "2026-09-09T16:44:14.717913Z", + "iopub.status.idle": "2026-09-09T16:44:14.818173Z", + "shell.execute_reply": "2026-09-09T16:44:14.817852Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "412 streams assembled in 0.10 s (audio is decoded lazily)\n", + "positives per subshare: {'eval-speech': 195}\n", + "eval-speech: 179 streams, H = 2.9513 h (zero-event bound ln 20 / H = 1.015 FA/h; longest stream 60.0 s; packed at <= 60 s, so the per-stream reset can add at most streams / H = 60.65 FA/h to any row)\n", + "eval-music: absent from this lock\n", + "eval-tts: absent from this lock\n", + "eval-noise: 38 streams, H = 0.5021 h (zero-event bound ln 20 / H = 5.967 FA/h; longest stream 113.0 s; packed at <= 60 s, so the per-stream reset can add at most streams / H = 75.68 FA/h to any row)\n" + ] + } + ], + "source": [ + "t0 = time.perf_counter()\n", + "streams = kws_streams.streams_from_lock(manifest, lock, Store(STORE), scoring, max_stream_s=60.0)\n", + "t_assemble = time.perf_counter() - t0\n", + "hours = kws_streams.hours_per_share(streams, scoring)\n", + "positives = kws_streams.positives_per_subshare(streams)\n", + "counts = {}\n", + "for s in streams:\n", + " counts[s.share] = counts.get(s.share, 0) + 1\n", + "print(f\"{len(streams)} streams assembled in {t_assemble:.2f} s (audio is decoded lazily)\")\n", + "print(f\"positives per subshare: {positives}\")\n", + "for share in kws_eval.NEGATIVE_SHARES:\n", + " if share in hours:\n", + " longest = max(s.negative_samples for s in streams if s.share == share) / scoring.sample_rate\n", + " print(f\"{share}: {counts[share]} streams, H = {hours[share]:.4f} h \"\n", + " f\"(zero-event bound ln 20 / H = {kws_scoring.zero_event_bound(hours[share]):.3f} FA/h; \"\n", + " f\"longest stream {longest:.1f} s; packed at <= 60 s, so the per-stream reset \"\n", + " f\"can add at most streams / H = {counts[share] / hours[share]:.2f} FA/h to any row)\")\n", + " else:\n", + " print(f\"{share}: absent from this lock\")" + ] + }, + { + "cell_type": "markdown", + "id": "cad4afc7", + "metadata": {}, + "source": [ + "## The band-energy baseline\n", + "\n", + "`kws_detectors.BandEnergyBaseline`: the mean, over the mel bands whose centre lies in\n", + "[300, 3000] Hz, of the shipping front end's plain-log feature, clipped to [0, 1]. It has no notion\n", + "of the keyword — it is the plan's *trivial* detector, run so the harness's plumbing (alignment, hours,\n", + "intervals, the report) is exercised on real audio before a model exists. The log affine maps a band\n", + "energy of 1 to a feature of exactly 1 and is unbounded above; clips that exceed that level saturate the\n", + "baseline's clip to [0, 1], so its per-stream maximum sits at 1.0 on such streams. The next cell measures\n", + "how many, and the sweep below adds an even grid of thresholds because the quantiles of saturated maxima\n", + "collapse." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "eb6a685b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:14.819405Z", + "iopub.status.busy": "2026-09-09T16:44:14.819331Z", + "iopub.status.idle": "2026-09-09T16:44:20.033695Z", + "shell.execute_reply": "2026-09-09T16:44:20.033235Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "detector band-energy: bands 5..26 (329.698..2901.893 Hz), stored path log\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "scored 13158 s of audio (3.655 h, positives included) in 5.2 s: 2537 s of audio per second, decode included\n", + "per-stream maximum smoothed score, quantiles 0/10/50/90/100 %: [0.4764, 1.0, 1.0, 1.0, 1.0]\n", + "streams whose smoothed maximum reaches 1.0: 394 of 412 (96 %)\n", + "default_thresholds (50 quantiles) yields only 5 distinct thresholds; the sweep adds a grid of 41\n" + ] + } + ], + "source": [ + "detector = kws_detectors.BandEnergyBaseline(g)\n", + "print(f\"detector {detector.name}: bands {detector.bands[0]}..{detector.bands[-1]} \"\n", + " f\"({detector.params['band_centres_hz'][0]}..{detector.params['band_centres_hz'][-1]} Hz), \"\n", + " f\"stored path {detector.params['stored_path']}\")\n", + "t0 = time.perf_counter()\n", + "scores = kws_eval.score_streams(streams, detector, scoring) # decode, check the accounting, score: once\n", + "t_score = time.perf_counter() - t0\n", + "audio_s = sum(y.size for y in scores.values()) * scoring.hop / scoring.sample_rate\n", + "print(f\"scored {audio_s:.0f} s of audio ({audio_s / 3600:.3f} h, positives included) in \"\n", + " f\"{t_score:.1f} s: {audio_s / t_score:.0f} s of audio per second, decode included\")\n", + "maxes = np.array([kws_scoring.smooth(y, scoring.smoothing_hops).max() for y in scores.values()])\n", + "q = np.quantile(maxes, [0.0, 0.1, 0.5, 0.9, 1.0])\n", + "print(\"per-stream maximum smoothed score, quantiles 0/10/50/90/100 %:\", np.round(q, 4).tolist())\n", + "saturated = int((maxes >= 1.0).sum())\n", + "print(f\"streams whose smoothed maximum reaches 1.0: {saturated} of {maxes.size} \"\n", + " f\"({100.0 * saturated / maxes.size:.0f} %)\")\n", + "quantile_thresholds = kws_eval.default_thresholds(scores, scoring, n=50)\n", + "print(f\"default_thresholds (50 quantiles) yields only {len(quantile_thresholds)} distinct \"\n", + " f\"thresholds; the sweep adds a grid of 41\")" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "e6e7f4fa", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:20.034856Z", + "iopub.status.busy": "2026-09-09T16:44:20.034781Z", + "iopub.status.idle": "2026-09-09T16:44:24.156397Z", + "shell.execute_reply": "2026-09-09T16:44:24.155995Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "44 thresholds; scoring + sweep 9.3 s; report at build-kws-det/report.json and build-kws-det/report.md\n" + ] + } + ], + "source": [ + "grid = {kws_eval.threshold_value(v) for v in np.linspace(0.0, 1.0, 41)}\n", + "thresholds = sorted(set(quantile_thresholds) | grid)\n", + "provenance = {\"manifest_name\": manifest.name, \"manifest_hash\": lock.manifest_hash,\n", + " \"eval_set_id\": dict(lock.eval_set_id), \"holdout_set_id\": lock.holdout_set_id,\n", + " \"max_stream_s\": 60.0,\n", + " \"front_end\": {\"log_mel_contract_version\": kws_features.contract_version(),\n", + " \"dsptap_commit\": kws_features.dsptap_commit()}}\n", + "t0 = time.perf_counter()\n", + "report = kws_eval.evaluate(streams, detector, scoring, thresholds, scores=scores, provenance=provenance)\n", + "report.wall_s = t_score + time.perf_counter() - t0\n", + "OUT = REPO / \"build-kws-det\"\n", + "json_path, md_path = kws_eval.write_report(report, OUT)\n", + "print(f\"{len(report.rows)} thresholds; scoring + sweep {report.wall_s:.1f} s; report at \"\n", + " f\"{json_path.relative_to(REPO)} and {md_path.relative_to(REPO)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "8afb222f", + "metadata": {}, + "source": [ + "## The report\n", + "\n", + "The committed format (`kws_eval.Report.markdown`): the header names the manifest and its hash, the\n", + "detector, the scoring numbers, the front end's contract version and DspTap commit, the `eval_set_id`\n", + "per share, the hours per share with their stream counts and packing bound, and the positives per\n", + "subshare; then one row\n", + "per threshold — recall with its Wilson interval, FRR, spurious events, and FA/h **per share** with the\n", + "Poisson interval and the hours behind it (`<= ln 20 / H` where a share saw no event). A share this\n", + "lock lacks reads `absent`, and the eval-tts and hold-out recall columns stand beside each other as the\n", + "plan's descriptive gap figure." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "758af840", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:24.157770Z", + "iopub.status.busy": "2026-09-09T16:44:24.157690Z", + "iopub.status.idle": "2026-09-09T16:44:24.160934Z", + "shell.execute_reply": "2026-09-09T16:44:24.160582Z" + } + }, + "outputs": [ + { + "data": { + "text/markdown": [ + "# KWS evaluation report\n", + "\n", + "- manifest: speech_commands_v2_bringup (`54b94a324b4bbeef7dd7ec36bfaffb1337a60db3bfad92b41cfbae9801dda0a0`)\n", + "- detector: `band-energy` {\"band_centres_hz\": [329.698, 392.965, 460.119, 531.398, 607.058, 687.366, 772.609, 863.088, 959.128, 1061.068, 1169.271, 1284.122, 1406.031, 1535.429, 1672.778, 1818.566, 1973.312, 2137.565, 2311.911, 2496.968, 2693.396, 2901.893], \"band_hi_hz\": 3000.0, \"band_lo_hz\": 300.0, \"bands\": [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26], \"geometry\": {\"bands\": 40, \"fft_size\": 512, \"fmax_hz\": 7600.0, \"fmin_hz\": 20.0, \"frame\": 400, \"hop\": 160, \"log_floor\": 1e-10, \"log_scale\": 5.0, \"log_shift\": 5.0, \"pcen\": {\"alpha\": 0.98, \"delta\": 2.0, \"enabled\": false, \"epsilon\": 1e-06, \"power\": 0.5, \"smoother\": 0.025}, \"preemphasis\": 0.0, \"sample_rate\": 16000.0, \"window\": \"hann\"}, \"stored_path\": \"log\"}\n", + "- scoring: hop 160 at 16000 Hz; T = 3 hops, L = 20 hops (hit window [h - T, h + L + T]); W = 10 hops, R = 100 hops\n", + "- front end: log_mel_contract_version 1, DspTap `5ca3b1cdffc4d724fdcef4bafb06e2908defda1d`\n", + "- eval_set_id: eval-noise `a7e8313152310948e177e5177b4c53f2e00674f16efc6fba5f7482d334123e63`, eval-speech `ad456333ea8761839fe78fbc450c8bd63d29d905ec8cfaf7305501bb9cd5705f`\n", + "- holdout_set_id: absent\n", + "- hours: eval-speech H = 2.951 h (179 streams, packed at <= 60 s), eval-music absent, eval-tts absent, eval-noise H = 0.5021 h (38 streams, packed at <= 60 s)\n", + "- positives: eval-speech 195, eval-tts absent, holdout absent\n", + "- wall time: 9.3 s (scoring and sweep)\n", + "\n", + "| threshold | recall [95 %] (hits/positives) | FRR | spurious | FA/h speech | FA/h music | FA/h TTS | FA/h noise | recall eval-tts | recall hold-out |\n", + "|---|---|---|---|---|---|---|---|---|---|\n", + "| 0.0 | 0.000 [0.000, 0.019] (0/195) | 1.000 | 195 | 60.65 [52.09, 70.22] (179 in 2.951 h) | absent | absent | 75.68 [53.56, 103.88] (38 in 0.5021 h) | absent | absent |\n", + "| 0.025 | 0.051 [0.028, 0.092] (10/195) | 0.949 | 345 | 1023.28 [987.11, 1060.44] (3020 in 2.951 h) | absent | absent | 203.15 [165.65, 246.61] (102 in 0.5021 h) | absent | absent |\n", + "| 0.05 | 0.056 [0.032, 0.098] (11/195) | 0.944 | 357 | 1113.75 [1076.00, 1152.49] (3287 in 2.951 h) | absent | absent | 201.16 [163.85, 244.43] (101 in 0.5021 h) | absent | absent |\n", + "| 0.075 | 0.067 [0.039, 0.111] (13/195) | 0.933 | 369 | 1207.95 [1168.62, 1248.27] (3565 in 2.951 h) | absent | absent | 221.08 [181.87, 266.23] (111 in 0.5021 h) | absent | absent |\n", + "| 0.1 | 0.062 [0.036, 0.104] (12/195) | 0.938 | 375 | 1283.85 [1243.29, 1325.39] (3789 in 2.951 h) | absent | absent | 254.94 [212.69, 303.12] (128 in 0.5021 h) | absent | absent |\n", + "| 0.125 | 0.072 [0.043, 0.117] (14/195) | 0.928 | 391 | 1378.38 [1336.35, 1421.41] (4068 in 2.951 h) | absent | absent | 302.74 [256.52, 354.87] (152 in 0.5021 h) | absent | absent |\n", + "| 0.15 | 0.082 [0.051, 0.129] (16/195) | 0.918 | 404 | 1475.97 [1432.46, 1520.46] (4356 in 2.951 h) | absent | absent | 336.59 [287.76, 391.34] (169 in 0.5021 h) | absent | absent |\n", + "| 0.175 | 0.103 [0.067, 0.153] (20/195) | 0.897 | 414 | 1582.02 [1536.97, 1628.07] (4669 in 2.951 h) | absent | absent | 366.47 [315.43, 423.42] (184 in 0.5021 h) | absent | absent |\n", + "| 0.2 | 0.133 [0.093, 0.188] (26/195) | 0.867 | 432 | 1685.03 [1638.52, 1732.53] (4973 in 2.951 h) | absent | absent | 398.34 [345.04, 457.53] (200 in 0.5021 h) | absent | absent |\n", + "| 0.225 | 0.149 [0.106, 0.205] (29/195) | 0.851 | 444 | 1793.46 [1745.46, 1842.44] (5293 in 2.951 h) | absent | absent | 416.26 [361.74, 476.69] (209 in 0.5021 h) | absent | absent |\n", + "| 0.25 | 0.144 [0.101, 0.200] (28/195) | 0.856 | 454 | 1869.02 [1820.02, 1919.00] (5516 in 2.951 h) | absent | absent | 470.04 [411.98, 533.99] (236 in 0.5021 h) | absent | absent |\n", + "| 0.275 | 0.169 [0.123, 0.228] (33/195) | 0.831 | 461 | 1928.31 [1878.54, 1979.08] (5691 in 2.951 h) | absent | absent | 483.98 [425.04, 548.81] (243 in 0.5021 h) | absent | absent |\n", + "| 0.3 | 0.174 [0.128, 0.234] (34/195) | 0.826 | 464 | 1977.11 [1926.70, 2028.50] (5835 in 2.951 h) | absent | absent | 487.96 [428.77, 553.04] (245 in 0.5021 h) | absent | absent |\n", + "| 0.325 | 0.174 [0.128, 0.234] (34/195) | 0.826 | 465 | 2025.56 [1974.53, 2077.57] (5978 in 2.951 h) | absent | absent | 487.96 [428.77, 553.04] (245 in 0.5021 h) | absent | absent |\n", + "| 0.35 | 0.174 [0.128, 0.234] (34/195) | 0.826 | 470 | 2074.01 [2022.38, 2126.63] (6121 in 2.951 h) | absent | absent | 507.88 [447.45, 574.19] (255 in 0.5021 h) | absent | absent |\n", + "| 0.375 | 0.159 [0.114, 0.217] (31/195) | 0.841 | 476 | 2135.68 [2083.28, 2189.07] (6303 in 2.951 h) | absent | absent | 511.86 [451.19, 578.42] (257 in 0.5021 h) | absent | absent |\n", + "| 0.4 | 0.149 [0.106, 0.205] (29/195) | 0.851 | 484 | 2177.70 [2124.78, 2231.60] (6427 in 2.951 h) | absent | absent | 523.81 [462.41, 591.10] (263 in 0.5021 h) | absent | absent |\n", + "| 0.425 | 0.138 [0.097, 0.194] (27/195) | 0.862 | 489 | 2211.92 [2158.58, 2266.24] (6528 in 2.951 h) | absent | absent | 549.71 [486.76, 618.53] (276 in 0.5021 h) | absent | absent |\n", + "| 0.45 | 0.128 [0.088, 0.182] (25/195) | 0.872 | 492 | 2246.14 [2192.39, 2300.88] (6629 in 2.951 h) | absent | absent | 561.66 [498.01, 631.18] (282 in 0.5021 h) | absent | absent |\n", + "| 0.475 | 0.133 [0.093, 0.188] (26/195) | 0.867 | 494 | 2269.18 [2215.16, 2324.19] (6697 in 2.951 h) | absent | absent | 573.61 [509.26, 643.82] (288 in 0.5021 h) | absent | absent |\n", + "| 0.47642819919 | 0.144 [0.101, 0.200] (28/195) | 0.856 | 493 | 2269.86 [2215.83, 2324.88] (6699 in 2.951 h) | absent | absent | 571.61 [507.39, 641.72] (287 in 0.5021 h) | absent | absent |\n", + "| 0.5 | 0.144 [0.101, 0.200] (28/195) | 0.856 | 500 | 2296.97 [2242.61, 2352.31] (6779 in 2.951 h) | absent | absent | 587.55 [522.41, 658.57] (295 in 0.5021 h) | absent | absent |\n", + "| 0.525 | 0.128 [0.088, 0.182] (25/195) | 0.872 | 503 | 2322.38 [2267.72, 2378.02] (6854 in 2.951 h) | absent | absent | 607.46 [541.20, 679.61] (305 in 0.5021 h) | absent | absent |\n", + "| 0.55 | 0.149 [0.106, 0.205] (29/195) | 0.851 | 499 | 2342.03 [2287.14, 2397.91] (6912 in 2.951 h) | absent | absent | 627.38 [560.00, 700.63] (315 in 0.5021 h) | absent | absent |\n", + "| 0.575 | 0.133 [0.093, 0.188] (26/195) | 0.867 | 498 | 2346.78 [2291.83, 2402.71] (6926 in 2.951 h) | absent | absent | 657.26 [588.25, 732.14] (330 in 0.5021 h) | absent | absent |\n", + "| 0.6 | 0.123 [0.084, 0.177] (24/195) | 0.877 | 500 | 2356.60 [2301.54, 2412.65] (6955 in 2.951 h) | absent | absent | 707.05 [635.40, 784.56] (355 in 0.5021 h) | absent | absent |\n", + "| 0.625 | 0.103 [0.067, 0.153] (20/195) | 0.897 | 502 | 2349.83 [2294.84, 2405.79] (6935 in 2.951 h) | absent | absent | 685.14 [614.64, 761.51] (344 in 0.5021 h) | absent | absent |\n", + "| 0.65 | 0.097 [0.063, 0.147] (19/195) | 0.903 | 500 | 2340.34 [2285.47, 2396.19] (6907 in 2.951 h) | absent | absent | 709.04 [637.29, 786.66] (356 in 0.5021 h) | absent | absent |\n", + "| 0.675 | 0.092 [0.059, 0.141] (18/195) | 0.908 | 502 | 2337.29 [2282.45, 2393.11] (6898 in 2.951 h) | absent | absent | 742.90 [669.41, 822.25] (373 in 0.5021 h) | absent | absent |\n", + "| 0.7 | 0.092 [0.059, 0.141] (18/195) | 0.908 | 497 | 2328.14 [2273.41, 2383.85] (6871 in 2.951 h) | absent | absent | 758.83 [684.54, 838.99] (381 in 0.5021 h) | absent | absent |\n", + "| 0.725 | 0.082 [0.051, 0.129] (16/195) | 0.918 | 491 | 2316.28 [2261.69, 2371.85] (6836 in 2.951 h) | absent | absent | 774.77 [699.68, 855.72] (389 in 0.5021 h) | absent | absent |\n", + "| 0.75 | 0.082 [0.051, 0.129] (16/195) | 0.918 | 490 | 2290.19 [2235.91, 2345.45] (6759 in 2.951 h) | absent | absent | 748.87 [675.08, 828.53] (376 in 0.5021 h) | absent | absent |\n", + "| 0.775 | 0.077 [0.047, 0.123] (15/195) | 0.923 | 483 | 2266.13 [2212.14, 2321.11] (6688 in 2.951 h) | absent | absent | 720.99 [648.62, 799.22] (362 in 0.5021 h) | absent | absent |\n", + "| 0.8 | 0.092 [0.059, 0.141] (18/195) | 0.908 | 471 | 2245.80 [2192.06, 2300.53] (6628 in 2.951 h) | absent | absent | 701.07 [629.74, 778.28] (352 in 0.5021 h) | absent | absent |\n", + "| 0.825 | 0.082 [0.051, 0.129] (16/195) | 0.918 | 465 | 2205.82 [2152.56, 2260.07] (6510 in 2.951 h) | absent | absent | 673.19 [603.33, 748.92] (338 in 0.5021 h) | absent | absent |\n", + "| 0.85 | 0.082 [0.051, 0.129] (16/195) | 0.918 | 458 | 2167.87 [2115.07, 2221.65] (6398 in 2.951 h) | absent | absent | 653.27 [584.48, 727.94] (328 in 0.5021 h) | absent | absent |\n", + "| 0.875 | 0.082 [0.051, 0.129] (16/195) | 0.918 | 452 | 2120.43 [2068.22, 2173.63] (6258 in 2.951 h) | absent | absent | 786.72 [711.04, 868.26] (395 in 0.5021 h) | absent | absent |\n", + "| 0.9 | 0.072 [0.043, 0.117] (14/195) | 0.928 | 439 | 2064.87 [2013.34, 2117.37] (6094 in 2.951 h) | absent | absent | 687.13 [616.53, 763.60] (345 in 0.5021 h) | absent | absent |\n", + "| 0.925 | 0.077 [0.047, 0.123] (15/195) | 0.923 | 425 | 1997.44 [1946.77, 2049.09] (5895 in 2.951 h) | absent | absent | 571.61 [507.39, 641.72] (287 in 0.5021 h) | absent | absent |\n", + "| 0.95 | 0.067 [0.039, 0.111] (13/195) | 0.933 | 398 | 1894.77 [1845.43, 1945.09] (5592 in 2.951 h) | absent | absent | 470.04 [411.98, 533.99] (236 in 0.5021 h) | absent | absent |\n", + "| 0.975 | 0.067 [0.039, 0.111] (13/195) | 0.933 | 368 | 1760.25 [1712.71, 1808.78] (5195 in 2.951 h) | absent | absent | 426.22 [371.02, 487.31] (214 in 0.5021 h) | absent | absent |\n", + "| 0.988058451045 | 0.067 [0.039, 0.111] (13/195) | 0.933 | 347 | 1641.32 [1595.42, 1688.21] (4844 in 2.951 h) | absent | absent | 414.27 [359.88, 474.56] (208 in 0.5021 h) | absent | absent |\n", + "| 0.999877052675 | 0.036 [0.017, 0.072] (7/195) | 0.964 | 310 | 1403.12 [1360.71, 1446.52] (4141 in 2.951 h) | absent | absent | 420.25 [365.45, 480.94] (211 in 0.5021 h) | absent | absent |\n", + "| 1.0 | 0.036 [0.017, 0.072] (7/195) | 0.964 | 306 | 1390.92 [1348.69, 1434.13] (4105 in 2.951 h) | absent | absent | 444.15 [387.76, 506.42] (223 in 0.5021 h) | absent | absent |\n", + "\n", + "FA/h cells read `rate [Poisson 95 % lo, hi] (events in H)`; at zero events `<= ln 20 / H` is the one-sided 95 % bound. Spurious events lie on positive streams outside every hit window and never enter FA/h. Every negative stream is decided after a fresh reset, so each FA/h row depends on the stream packing by at most streams / H (the threshold-0 row). The eval-tts and hold-out recalls stand beside each other as a descriptive gap figure, never a pass.\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from IPython.display import Markdown, display\n", + "display(Markdown(report.markdown()))" + ] + }, + { + "cell_type": "markdown", + "id": "cb3fa34b", + "metadata": {}, + "source": [ + "## The DET\n", + "\n", + "False-rejection rate against false accepts per hour, one curve per share present, FA/h on a log axis.\n", + "Each point is one threshold; the horizontal bar is the exact Poisson 95 % interval on that share's\n", + "count. The dashed vertical line per share is the zero-event bound ln 20 / *H*: nothing to the left of\n", + "it is measurable on this many hours, and a threshold at which the share saw no event is drawn *at* the\n", + "bound with an open, left-pointing marker (its rate is ≤ that bound, not 0). Missing shares are named in\n", + "the title rather than drawn as empty curves." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "015fb960", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:24.161837Z", + "iopub.status.busy": "2026-09-09T16:44:24.161781Z", + "iopub.status.idle": "2026-09-09T16:44:24.810912Z", + "shell.execute_reply": "2026-09-09T16:44:24.810523Z" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3oAAAHWCAYAAAAhEvvEAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAA3fpJREFUeJzs3Qd8U+X6B/BfdveCspGNTAdbcaGioqI4ce+LAzdur/zVe8UJbq9X5br3FsQFKE62yJC9Z+ne2ef/ed40IU2TNk3TJm1/337ySXpycvLmZD7nfd/n0WmapoGIiIiIiIhaDH2sG0BERERERETRxUCPiIiIiIiohWGgR0RERERE1MIw0CMiIiIiImphGOgRERERERG1MAz0iIiIiIiIWhgGekRERERERC0MAz0iIiIiIqIWhoEeERERERFRC2OMdQNak1WrVmHRokXqsl6vR0pKCg466CAceuihSExMrHX9QCNGjMDy5cvhdDprvc9TTjkFXbp0QWNbuHAhcnJycOaZZyKWKioq8N5772H8+PFo3759TNvS2nz11VfIzs7GEUcc0Sjb/+yzz9C1a1cMHz4c8fh4G/vxU/N6fbREGzduxG+//YYrrriiye87Ly8PS5YsQXl5OQYNGoR+/fo1+WukJb3HP//8cwwePBi9e/cO+zZ8jzWOaL+u5LfYrFmzal1n+/bt6venTqcLuU7Hjh1x2mmnqcv79u3Dn3/+iaKiIvWaGTJkCAwGA1qSyspKvPvuuzj11FPRqVOnkOsVFxfj008/xWWXXQajMf7DqPhvYQvy3Xff4c4771RfkvIGKSsrw5o1a7Bt2zb84x//wL///W8kJSWFXN9f586dVRDocDjU/zabDe+88w5Gjx5d7QtQPjiaItD74IMP8Ouvv8Y80CsoKFD78pdffmGg18SmTZuGYcOG+b6svvzyS/UcjBo1Kirbnzp1Kk4//fS4+SEf+HgD/6emFW+vj5ZIgrzrrruuSQM9TdNw9dVX4/3338eRRx6JjIwMfP/99zj55JPx1ltvVfvObOzXSEt5j//xxx9qn27ZsqXZv8ei/T0TC9F+XZWWlqqD717yPP/4448477zzkJaW5gv09uzZ4wv05Pfb7t27MXHiRN/tDj74YBXo3XHHHXjxxRcxcuRI9XtStpefn4+nnnpKHVRvKYqLi9Xvxx9++KHWQC89PR3//e9/1W/4m2++GfGOgV4M/Oc//0FCQoLv/wULFuDcc89VQd+3335b4whL4Ppe48aNq3a0UwK9Sy65RH0REzU1CfK7devm+/9f//oXjjrqqGb9BdyQx09EDedyufD111+rH1/yeeLtWTz88MPVD2Q5QNpUWsp7/O6778bkyZNV0NzctYTvmWi/rqTH7bXXXqt2IF4CPXm/hOrBveaaazB37txqtxPy3ps+fbrqAZ4wYYJvuQSK69evR2t133334aqrrlKBYbARefGEgV4cOPbYY/HCCy/gggsuwJw5c3xd5c2163vp0qXYsWMHBg4ciMMOO6za9X///Td+//13dVmC1x49eqgPaP8eS+8wUAlkZaiObKtPnz7qiFcg6dGcP38+9u/fr44+1XYUJhhpz4oVK2CxWDBmzBhkZWVF1I5wtyVDaX/66Sd1JE0uy9AIOTImH8LyWGRfyHBc/+FRcp0MV5Ijqf5kG/J6kYMEtX1hL1u2DBs2bEDbtm3V9lNTU33XyQe79Ch7j1L1799fDYsKNlxHviBkP8hRLzmi271792rryfMtw0+8Q1Hk4IMcvPB+cZx44onYunVrgx6LyM3NVY+psLAQxxxzjOrd9hfOY6prv3h/YMprVbbVrl079ZyazeaQ7fJ//PXZb5Hcl/9QZTlQJPtEfijIDx7/91J92hBMbfvIu23ZVm3PR7iPL5x13G63ei/JEWW5X+nhkWHw/qQdkTzW+txPuPtdPt9k38h7XNaRoVJ2ux0///yz+jyQoXOHHHJItftuyG2j+X727st58+b5PptCqeu9FIp8dsu+DPV5cM4556jHIevJ+8tLPoflfuQ5iERdnyH+r20ZHSL76Morr2zQe7yuz3kh0w5kKof/Y5Uf0/JavPzyyyN67wX666+/1GN6/fXXI/rcDGf/RePzNZx9G+p7Rtbx/ta46KKLau31bcj7Ldznq679Efi6Cvezril4e32PPvroasvlcy+c4FRe7/J6k6Gfso9l2pH3s9J/qKR8DshrU4ZCynOYnJwctKdStiXPkYxcC9azHM46ZWVlqgdTRoDJb9QBAwbUWEdelzJU1WQyqddD4O8S+Z0uj0NeA9I7Htc0ajJPPvmkJru8srKyxnV2u11LSEjQbrjhhrDWD5Sbm6vW/c9//qPFwi233KJ16dJF69evn3bCCSdo48eP1ywWizZ58uRq682fP1+7+uqr1emiiy7SevbsqQ0aNEjbvXt3jW0deuih2sknn6yde+65WmJiYrV9I/Ly8tQ6nTt31s4//3x13yeeeKLaD7/88kut7a2oqNDOOeccLSkpSbV1zJgxWlpamvb999/Xux312dbAgQO1sWPHqse/bt06bcGCBWpduQ/ZvqxzzDHHqH3n9dZbb6n/5Tn2d9ttt2ndu3fXXC5X0MfocDi0U089VWvfvr12wQUXaOPGjdP69u2r/fzzz751XnjhBd/zIY8hPT1dmzhxouZ2u33rSJtHjBihnitZR55fs9msffDBB9Xub+TIkb7ne9q0aVrbtm3Vbb3bX7p0acSPxduO4cOHa506ddLOOussbfTo0Wpbge2o6zGFs182btyo7q9jx47aeeedp/Xv31+9vnbt2hX08Qb7P9z9Fs59BbNkyRKtQ4cOan25neyXQw45RNuzZ0+92xAonH0U7vMRzuMLZ53169erdeT+5LHI++TII4/USkpKGvRYA9V1P+Hu98MOO0y9n2UdeW2YTCbt+eefV7eTzxPZZ7Lsqaeeqnb/DbltNN/Pv//+u5aZmen7bJLHGfjZFM7rpDavvPKK+kwtKiqqtvyee+5R9+d0OoPeTh6PtF/usz7Cfc161+vWrZtab9KkSQ16j4fzOS9kPz799NPVlr366qtacnJyRI8jmAceeEDr1atXjeXhvnbqut9ofb6Gs29Dfc8Ieb/Ib4GdO3fWuj8a8n4L5/kKZ38Evq7q+gyqr/fff1/tC9nvoci+k9d7oL/++kszGo3aKaecon7TBL5Xa7Njxw71vS6/8S6++GL1epHHun//fnX93r17VbuOP/54dd/yO65Pnz7qN538PvL35ZdfahkZGWpb8jy1a9dOnft/RoSzzueff64+12QdeX3Lfr7pppuqtUd+o8lzJM+XrCfP3YYNG2o8Pu/zGe8Y6DWhugI3eWHJB0Lg+hK8yYeH/2nfvn1xF+jJ/cuXhdePP/6oln333XchbydvQAmMrrnmmhrb+uabb3zLvvrqK7Vs8+bNvmXXX3+9+lAoLCz0BcsnnXRSWIGe3IcE1vIh5vXwww9rXbt21axWa73aUZ9tffTRR9W+AKT9l1xyie/LVD5o5Eek/w8A2UZ2drb2xBNP+JbJaygrK0t79NFHQz7GX3/9Vd2n/5enfMAuWrQo5G3kx6ps1//LVD4I5QPT/4fszTffrPXo0aPabQO/rIYOHaoet79IH4u3HfJlu3LlSt+y+++/X32w5+fnh/2Y6tovEmwOHjxY/Xj3f23JD40LL7ww5OMN9iOwrv0W7n0FktfOgAED1IEFuSwKCgrUZ4h8WdanDZG+dsJ5PsJ5fOGsI58T8qPr2GOP1UpLS333t3DhQt99RfpY/dV1P/XZ7/LjwP9z+uijj1b79IsvvvAtmz59ujpA5N1WQ28brfezdz/IwTjvZ5OsL7fz/2yK5DPGn/xwlcfg/70l9y0/cO+7776Qt3vuuefU/coPu/oI9zNE1pPAbNu2bdVuH8l7PNzP+foGepF8Fgp5X/m/Vuv72qnrfqP1+Rru+znY94z/QWV5f9amIe+3cJ6vcN4j/q+rcD7rmjLQE/PmzVO/0yTQlu1IYCyPO9SBGK8HH3xQ7V//A7jLly/3Hdj3BlbymVpcXOz7jSCPXe7Pa+vWreo3lrTR+x6SbUiQP3PmzLDX2bhxo3oMciDJ/wDGnDlzqrVHfkfKa1LIY5TX6z/+8Y8aj09eE6mpqVq8Y6AXR4Ge9+hR4PpXXHGF72iV9xT4hq1PoCcfFoGBYzgn/+AmkHzQyhHAwDe+fIBdeeWV1ZbJh5ccGZLeHdmuHLGSN5L/tuQonj85ihQYdMmXS+ARNtmuf6AX+FhnzZqlPnSkrd6jtP49c/Il5v1wD6cd9dmWHL30Jz0Dsi3/AFHcfffdNX4A3Hvvvaot3g+nN954Q31g5eTkaKGsWrVKbf/NN9+s9cegfAHJfvnf//6n9pG8DqWHzUs+qAMf3yeffKLpdDrNZrPVK9CL9LF42zFhwoQaPxSlHe+++27Yj6mu/SJfwIFBuZDnXO7Le1Q1nECvrv0W7n0FWrZsmbqd9Lr4e/bZZ9XRV++XVLjPXSSvnXCej3AeXzjryOOUdaRnJJRIH6u/uu6nIfv9zjvvVD9c/f32229qe1u2bKn1cYR722i9n6VXRLa9YsWKauvddddd1T6bwv2Mqc1ll12meom8Zs+eXeuP0rlz56rP1sDHEI5wP0NkvUsvvbTG7SN5j9fnc74+gV64n4WBpOfM24MRKJzXTl33G63P13Dfz6G+Z8LVkPdbOM9XOO8R/9dVOJ91TR3o+f+u+eOPP9Trx2AwBH2P+JN9I/tR3gPBeAMr/w4Cb6+bLPf+JnjkkUfU8y6juPxNmTJFGzVqVNjrPPzww+q5CfV94G1P4GvzuuuuU8FnoLffflut7w1S4xXn6MUJCbol41GweRChkrFESuZI+WdkCpeMRe/Zs2fI62Wce2B2UBlb7h33LyTl76WXXqq207dvX1ViQh63jPn3l5mZWe1/mfcmrFarOi8pKVFpfgPnQsgY8Noeq8ztkLHuMt5fxmkHTjyW9mzatCnsdkgbwt2WzMfzJ3P+ROA492DzO66//no88cQTvmxz8pqQ+SsytyEUmV8hmaFksvpNN92kXlsy6VsmXXvnQsiE4meffVZl05J5FvL45LGE83zIa1bmMIQzl6yhjyXUvpF5DjIX0v81Vtdjqmu/eLcl8yn8n1OZpyGPWeYsyHshHHXtt0jvy3u7wNe7vK9k7s+uXbt810Xy3IXz2gnn+Qjn8YWzjve9Iu/f2jT0dVrX/TRkv8v9B1vmzZpc2+MI97bRej9LogVR1+druK+T2khCg+OOO07Ns5K5SjJ3TOatB0sasXjxYpUQQjL9vfTSS4hEOJ8hwT6vQ6lrX9bnc74xHkcgeZ0GS4sf7munrvuN5udrNL93atOQ92pd6vseCfezLhYk6Yi0X07ym/TJJ59U3+UdOnQIur4kBpR5+ZKPQNaXeX6SMDAwD0Xga8r7v3f+ppzLHDlJCONP5v15f2OFs86OHTvU+7Cu106w1533N58/mb8n6ipzFmsM9OKETAyV4EUmJDc2CcgCg5JoCPxCEDJR2v9DQFLRSpYi+YDwuueee9SHQX3Il4t86ATeZ+D/wR6rfEnIG1QCzMCA9+yzz0avXr3CbocEc+FuKzCbqjewkX0kk9+95P9A8jjkx40ERXI7Ka3hvw9DmTRpkjrJD1CZbP/Pf/5TTWyXH1ObN2/Go48+qjK9SsDlJclm5Mu0sUT6WII9v/IBK8G29zUW7mOqbb94P+Tlh2fg61ImXUfzoEuk9+V9vLI//N9f8r+8zsIJmutS2z7yv7/ano9wHl8467Rp08b3pR3uD/BI1HU/TbHfIxXN97O3Bqk8Lv/PpmCf8eG8TmrjDer+97//4d5771UHA1999dWgdWUlMZasL1kEI63hVddr1qu2+mL1UZ/PeXlMkqTEnySpaMjjCPbcBt53fV474dxvvHy+NrZwn6/6vEea6rMuXJLUxNsmf95gXB5TqNecPJdy8ODpp59WyXGkFIYcqHnjjTdw8cUXh3xNeV+f/t8j8joM/I0l25ffWeGu06ZNG3VAIVqknfL7LzAwjDcM9OKAZIyUVMeSNdK/hklzI28g+RCTjElCjgpL9iP/I6/yxvA/Kiwfih9//HG970u+hI8//niV8ejaa6/1fSm/+eabdd5WjuZIlic5ShMYBMoHjvcoTTgasi35oJQjofKj5f7771fL5Evjo48+Crr+jTfeqL6EJYuVHPkOzIIVSD6A5YNNAmKpfSPZ3eTL9Ztvvqn2Yer/fEjWK8kcGkkx4kBSryfUj5T6PhYvycQnmd68H6yy7+S5l8xv4T6muvaL1KKUDGhy0EUOSviTtO7RPNIa6X1J9je5nbzepZaRkC+5t99+W9ViCpaxrD7q2kfhPh/hPD4J/MNZR7LVyWeJfxAgWdPkvdfQx+slvf213U9j7/eGiOb7WT6bZD9IyZ4HH3xQLZP3qtSyi+R1Uhe53XPPPad+2MoPM8m+G/g6GDt2LIYOHaoKFdfnMzpQXa/ZaKvP57xkepQfxIHtjebjkKyHkkk10tdOXffb1J+vob5nws262RDhPF/1fY/U9RkknzFN8di85PmVA9nSI+l/cEVG5Eh7anu+vM+nZAuVnk05ffLJJyrI9Q/0JPOmZCn1/o6Tzx0Z8SWf+0KCwxkzZuCuu+5SywPvI9x1zjzzTDz++OP48MMPq/3Wlp4+eS7rS94jktUzWgeFGgsDvRiQoxmSQlaGFcobXgIdSe8qL/ZgXcre9f1J6mP5wIwn0m1+2WWXqaE48uEjwxWGDBlSLX20vLkfeOAB9cUiHxLyA0mGh0TyRpEhA/Lj6qSTTlLBlqQvDjfdthT/lIBUhgzJUWJpy+rVq1X6a0nlXZ/6QpFuS3ol5UNHhjfIl4F8IMqwA/kSDbY/TjjhBPUBJl8kUo6jLpLmWY4iSupy2bakLH/55Zfx8MMPq+u9aYXlR5U8ZxKYynMWrdpKErzJ/ckPBfky9qa9juSxeMmXpbzu5TUlB0jktlLM1TvMK5zHVNd+kdeuHECQcifyepKj2vJDQnoe5ctWDl5ES6T3JT3Jzz//vHo/yWOUxy0/HOQ9IOU7GqqufRTu8xHO4wt3HSmOLc/rzp071Y9+uT/p/ZH059EKsOq6H/kB1pj7vSGi+X6W15f0sksPizz38h6WH2iSujyS10ld5PUjxbj/7//+Tw3t8v/xKq8FeR68ZVnkO8NLesikCHR91PWajbb6fM7fcMMN6rmT28hnpby21q5dG9XHIcPmpJdFXh/elP71ee3Udb9N/fka6ntGvntlqKQMG2ysYCic56u+75FwPuua4rF5yeOSYb1y/xJYyu8buX8J9CSw8u+lDiQHhqQXT9oppa8kwJP9IY/fn+wz+Q0nv+WkR05KXsyePdv3/pDfeXLAScocyGevjJTau3evaoP85pJ9Ge46DzzwgPqMkU4JOcgsIwXkOanvwSkhn/n+v2/jFQO9JiRHg+WLU+rMyREOeYNKwCaFYIPVZvNfP5Ac9fQP9OR/WVe2FwvyBpIfQfLhLW9SOUIiRVnlQ9D/KJAcpZIPZplrIUdlpYCnfHHIkJHAbfmTQFcen/9YbvliWrlypRr+IEds5MiKBH/yg6Gu4SsyB0GOVkpbpS3yfMgHhHwBen80htuOSLclZJy+9OTKF7/0gN5yyy1qOIsUKA1GjkjJvpV5jnWRYEqO+smRY/lwlaOK8uEmczCEfGDL9TNnzlT7T446y2tRhhH7f3nI/LnAeWJy9Ev2g/+R9cCir3L0WvaTHPWSYF4KHPvvt/o8Fv92SDsliJYhuHKQRIaBeoXzmOraL0J+LMhRUzmaKXNJ5Oi1fFnLl1Goxxv4f7j7LZz7CkaOSMp7QH6Ay2OVxyXvBf+aVuG2IVA4+0jIcydf4qGej3AfXzjryJf1unXr1I8HmccjX+TyQ9H7vor0sQaq634i3e/yGR/4w15+aEv7/H9QR3rbaL+f5QeM/DCVzzX5bLr99tvVjzX/QCvc10ldZN/JDzH57JPe/sChgd5RIvJ5Hzgssj6BXjifIf7rBYr0PR7u57y8n6TXR74P5cf9rbfeqoIIeV8ECue9F4yMhJHnVQ4sy/Yjee3Utv+i9fka7r4N9T0jP+Jl3boOAjXkvRrO8xXO/gh8XdX1GSTfm9L7Gu7QTrm9tL22oEx+lwX7jSIHBqRHTwI2OYAtc9Xk+ZPfcnX1gslvMelpk6BNHo88J3IeeDt53cnBf/kdJY9LfvMGPifyPJ9xxhmqHRIYSm/fK6+8op7v+qzz8MMPq6BbajDKe1Aet7d3UV7rsp8C60LK7zfvcHYvCUjlvRz3NfRkBJxkZIl1I4haKxnuKj9WvF8q8naUIFGWBU4qlgKq8gNTvjikF7E5a0mPpbWSYTjyhfnYY4/FuilELeZzvineexJAyDwx+dHdkGGwFBsyvFN61wIPijQ33nmIclDBezCnuTj33HNVkP/MM88g3rFHjyiG/vzzTzXcQL60ZdjuF198oSbGS6ISfzL/T47oyQejJK9pzlrSYyEiitbnfFO58MILVQ+YTB2RYZvUvMhc1SuvvDLWzWi1iouLVc+sd/5yvGOgRxRDMvRGjup+/fXXajy5DCGQI3WBWZxkmICsJwGSd4Jyc9WSHktrFmp4W7yS9NsyZCoUGUo1ePDgJm1TS9aU+zven9twP+eb8r3HnvjmS+YJtwShhkrGu/T0dDWHtbng0E0iImrxJGFKbT0o559/vkoGQM1vf/O5JSIKjoEeERERERFRC6OPdQOIiIiIiIgouhjoERERERERtTCtMtCT1MYlJSXqnIiIiIiIqKVplYFeaWmpypqzbeumet82L3df1NeV9YoqXJi9olidt2b12b8tuW2NcV8N3abbVgDblveQt3ttk7WhsdaP59dZU4rn/dDc32/R2m5DthHJbRvrO47iez80ddvi8TuuoduJl/dbpG1pifLieD80VdtaZaDnpWnuRr1NuOvKenanG1tz7eq8NYvkOWmJbWuM+2rwNl1WuErWQXNVNlkbGmv9eH6dNaV43g/N/f0Wre02ZBvx9B1H8b0fmrptcfkd18DtxMv7LdK2tERaHO+HpmpbXNTR+/XXX7Fu3Tqcdtpp6NixY53rOxwO/PTTT8jJyVG1cZpTLadQ2qWZcNPY7Fg3gygkfVInJB32MMr37+FeIiIiIopzMQ30pHjo3XffDaPRiL/++gs//vhjnYFefn4+TjjhBJSVlakgb/Lkybj00kvxwgsvoDlzuzXYnBosRh30el2sm0MU/OiTywrE8REyopZkV4EdHy+2otS5Fx3STTj10DR0yTLHullERNRMxHTopl6vxwcffIDZs2eHfZt7771X9ehJYPj5559j7ty5eOmll/DNN9+gOdtV6MCU93erc6J45C7birKfJwKVO2LdFKJmJ6/UiTd+yUde7m5UrpkOd2VOret/s7IEF7y4De8stuGr5cV45ac89b8sD/f+PlxmU+eBHDm/oPibo9R5U5PHHc7jJyKiZh7ojRs3DoMGDQp7fbfbjQ8//BBXXXUVkpOT1bLhw4dj1KhReP/99xuxpURERJErKJdArwDFRXth/Xs63Nb9tfbkPfLlPlgdbiSZgJQEA5LNelidbrV8d6E9rPv7aJlNnfuTbNPWVdPgyluszps6+7Q87roePxERtaA5euHauXOnKoswYMCAassHDhyIZcuWhbydzWZTJy/ZBhERUVOSkMrmcKvgqqKiBHpTUdD1vlhS5gnyzECFHXDbndDpZEi/hjKbC9O+3IVTBiUiNUGPFIseaYk6dZ5s0cFQNfTfWmFDgq4cOmcpNMeBkSLSi+fI+UkNwXbs+xGVqx6Fpdel0JszPCs4y6A5SsN7QOGu67+eszy8bRMRUesK9LwBWmZmZrXlWVlZtQZvjz76KB566KEay51OB3LrmViiPrcJd11Zr6AoB3abFQX5OUh0GdBaRfKctMS2NcZ9NXibFTnQ26xwOp0Rb6e+bWis9eP5ddaU4nk/NPf3W+B2CwtcqLA58a8v9+LJjqVwfDcerhBfwVv33w638yS43eVwutPg1A58J2jQ4/cNRfhr0+6gt03SlyPFUIZkfRl66svwv/d/QJqxTC1LMZRidNIcZOlt0EnY6XKhcsX9qFzxT0BnBPQm6GFEgd4E6AyACi5D02saCupYp8Z6bodErigs2A844vO111j4fmvcfRGtbUa6nXj5TRlpW1oiZxzvh4a2Lbtdp5YX6CUmJvrq4PmT/73XhZrXd/vtt/v+l6Cwa9euMBpNYe8oL3lSwr1NuOvKellt2sJsyUFWm/bIbtN6J9vXZ/+25LY1xn01dJuu0kpUWBLgNhoj3k5929BY68fz66wpxfN+aO7vt8DtFrqtSLJY8cCZHZG0OhXG0Z9Anz4w6O16/FaGBb+VQW9OQoJTUwGXW/Mk7XK4gF7t09ApIxOlVjfKrBpKbW5U2j1DMG1Vpxy3hpJyF9LdRhir4sSBlkUYm/whKtwpgA4wwwYDnCh3p8IFs+o1lABQEoI5dckoNA1EacKhsCcfCnf6YUhMSle9iOmJenVuL8tB544dq3obQ8vL3Yu22Z5Ea66iNSj96WykZ7WDMTM+X3uNhe+3xt0X0dpmpNuJ5HaN8Zsy0ra0RLlxvB+aqm3NKtA76KCDYDKZsG3btmrLt27dit69e4e8ncViUad41inDhMcndkKiqVWXNqQ4pk/uhuSj3oG1oDjWTSFqliQcspj0KjBKSkqDMaVquGSACcOT8MHiClQ43Eg0AgaDQQ33LLe7VYD17KUHoXNm9QOCDpeGMqsLJZVulFpd+Hu3FdO/2YcLj8hCklmPkgonjil4AyanEzYtGTI1zwoTLFo5tjoG4PPSf6CX+W/0NK5GD/NaJOjLkeFcjIzKxUChTBwE9joPwmr7IGyWk2MgdtnlOzkHaYkGpCUYkKoCQEPV/3qkJhpUe2G3oLPV83+SLRl6VxKSm3ZqIBFRqxT3gZ7U2Nu/fz/OPvtsmM1mnHzyySpT5zXXXKO+LPfu3avKMkjmzebMaNAh1dB6h2xS/NPpjdCZ0wE959gQ1VdWshFXHJ2F9IwkJAyYAn1Cu5DrSgmF+8/soBKvVEoXnpx0QIJRr5YHBnnCZNAhM9mITE+eMlWOYX9BESaOzETbVCM0lx3Fs3PgrrDABP8ELRYcmlGMvuechxKbETv37keOJQ3O4nUwlixHUsUKpFtXIMW1A10NO9DFvANHa3NUD2OlO0kFiZvtg7G5dCA25w9EhZZao21utwt6fdUQJc0EzfUx9FtMSE3crIK/dL8gUc7TVYDouVwtgJQ5iQkG9X1JRERxHuht3LgRCxYsQGFhoa+u3qZNmzBkyBB1Em+88QYWLlyoAj3x+OOP48gjj8Q555yjsm2++eabGDp0qKql15zlljjwyZIinDs8A9lpplg3h6gGd8Ve2Da9BmScLn3Q3ENE9SDB1hVHt/H8kz2lzvXHHZKGQ7om4KPfdqHUmagCt9MOSwsa5IW6v4lDLepc6AxmpI39AW5bfo119Za2yEpKQWe5nRqaLT2No6pOHm5bAVz5y+CUU8EyuAr+hNNWhk66P3Ek/oRb01QvYYW5JwpNh2Kf/lDscg/CLns35BZXwqaZUap6G3WwOlJUYhrpfZTTbtSvrFCyxTN0VIJECf6kB9Hbe6h6E6suq2DRt0wPs5EjZoiodYlpoJebm6uCOHH11VergE/+b9eunS/QO/roo9G5s3z9eEjGzZUrV6oAcPv27bjppptw5ZVXqiGdzVmlQ8OqXVacdhjHs1B80lwVcOYtBtJOiHVTiFoFCeouGZGA7Hae+W0NpU/qpE4R3daSBX2nsTB1Gqv+19xO5G39BanYoYI/V/5SuMq2wezeigzbVvTAF2o9nSkNzq79kNz5aBjbDIUxawgc+hTfEFMJ/krk3OpGaaXLc1kFgJ5lnnPP5TKrW22z3OZWp33FNWsE1sZi1Hl6Cat6DNP8hpp6A0L/ANITPBqQZNbVOQ+RiCgexTTQk545OdXm8ssvDzpXb+rUqY3YMiIiIqptKDdSD4al3RhYenm+p93WfLgKlsOZv9QT/BX8Cc1RAl3hQliLF3tvCUNaHyS1GYrUNsNU8KdP7Q2dru7eNqfMQ7R5AkQVAFYFiRIAyv/e4NATMLqr/S9DTW1ODbZSF/JKXfV6Yg16BASBB4JE7/BST6+iN3D0nEvPo7fcBRFRLMT9HD0iIiKKf/qENjV6/VzF61C09QeYHVvgkiGfZdvgKtmgTtj6vq/XT3r6DNLjJ8Ff1uHQmdNqbF/m5mUkGdSpPiRbaYVdgkJP0OftKZSTChL9kth4g0Q5L650wekCXG6gsNylTqjHMFPpBPTUOfQEiWadHW0z9nrmIAb0Inp7Gb3BpMy5JCJqKAZ6zYyUhti6ZQusNqtvWc+evZCdnR3TdhH527JlC/LzctW8HSGJlA4/3DMcm4haT6+fMXMQNEcWkqvSiLuteQG9fitUr58UcVeF3D23hCGtr+rtM/h6/XqF1esXjJSLkCQuckJG+NM8JNOp9AJW6yGsCgh9Q0ut/ssODDWVchfy8ad6FKuGnKqkNHvKwrrvRLPuQAbTgN7DGnMR/YaayvBUDjMlIi8GenFCjlCeMyyj1iOVG9avx7/+9RD69O2DxIQDdQOTk5MZ6FGj01nawNL7algNmbWuN/2pJ7B582Z0797d94MjNTWNgR4RQZ/QFvpOJ8HU6SS/Xr+1ao6fSvSSvwzu8u1wlaxXJ2x9z/P5Y06v2etnSm3czzydDgkmOemRXbODsVZS7sK/91CCv1378qCzpPuGk0qPofeydz0ZmioBogSKlXYn9pfUbx6i9ASqwDBgnmFaQAZTT6/igYQ1EtQSUcvDQC9OyJG5EwbW/qX1y68/Y/wZZ+D88y9osnYReenNGTAfNAHYX5UmPYjKygosW7YUM//3JhITDxyMICIK3es3WJ0sva9Uy9zWXDjzl3uCP5XhcwU0ezEc+35Up6pbwpB+MIxZgb1+8THkUQKurBQjslIOLMtNL0F2u9oPlLncmko0E9hD6Eta4997WDXU1BskyhBTCTDzy1zqVC+aC+nJWw70FvrVQQzWeyj/q15Si57lLkjZVWDH+38UYtm2CvX/0B5JuHBUpioXQ80o0Fu8eLGqY/fzzz9j165dalnXrl1xzDHH4MILL8SwYcMao50tnnywr9trRb+OCWoCdzBt2rSBJjPKiWJAc5TBWbACcLUPuY7FkqB6mI1GHkMiosjoE7Jh7nwyICfV6+eo6vWTHj9Pz5+7fIea/ycnbH3Xr9dPevyqgr+swxq91y/aJHmLN7DqXHtMWI30yFXYtWrzDH1zEAN6Fr3zEb1DUWV4qvy0KK5wqVN95iGKFFXKoqp30OxAkn0TMtv1RVpqalUvogGp+hJY8r9Fm16nIj2jnQpI/TlyfkHF8nuRNORRmNofHXKZl7syB7Yt78DS8xLoE6u+k2y5qFzzfvVl1OjySp3491f78NPaUlXy00syyb/7eyGO65+Cf57RwVfqhZpW2Ht9yZIluO2227Bs2TJV8kAKl7dv73kj5eTkqABQipZLoPf0008z4Kun/DInZi7Ixz2nt0ey5cDRj+3bt6Egv0Bd7tChI9579x2kpKaibZu2vnUO6tZNBYFEjcltzYF1zeNAz3sB9PEtdzgcWL1qle//YcOG47lnn8YxxxwLo9EzH8ZkNmHQoMF8goio3nR6E4yZh6iTr9evcj+cBVW9fjLXr9Db6zdfnapuCUN6P0/gJwFg22HQp/SMm16/aJLHlGyRk17VXKwPm8ONbbv2wJScXaOkRbVMpn6JamSZHKAWEkzKaS+cgNsBtz0J+j2VgN5v2KlafjD0vxQB+nI1XzHJskkFh20SK3GT+Xq0w3psm/9/+KPdh2qo6ejdtyKlciVyFv8L1uFfIS3J04uYaNLBbd0P69/T1RBgX1Bnz6u5jBrd2j2V+GF1abVl8haT0cAS0M9dXYpzh6fjqNTmddCl1QV6UqD87rvvVkXN09PTg65TXFyMd955RxU337FjRzTb2Wr9tWIF/vprhe9/Sbqy8I/fq61zxhkTGOhRzFRWVmL27K9qLP/22298l9PS0xnoEVHU6BPbwdz5FEBO3l6/or8P9PoVSK/fTk9PYPFaYMs7aj2dOaOqnp9/r5/f+MpWyGLSIytZj+x2lnrdTn7El0vPoEo44+kpLMrbirw/34C9+2SUIa0qYY2GktJKFO3fh3JLOsocennCYLU71eli451ob/obGnRoY1uMVX9+g07GrTgp80+4oQPyf8fT73yENbYR6n6NBiDVpCHJOgMZxRrSUrer4aQGawLS885G21U6ZLYtrVEOI8msV4l5KLp+WV9eY5kEebKntarTz+vKcVRfBnpxHeht2LABCQkJta4jAeDkyZNV8XOKjjPOnKBORPEqLS0ND0x9MNbNIKLW3uuXdag6Wfpc5Rve5+n18wR/rsK/oNmL4Ng7T52qbghD2sEqwYs30Ys+pUeL7PVrjHmIGclGZCQfWOZso6Fk6ydAyVfqOVGMgJbuABLy1b9uTY8KdxLK3Gkoc6Wih2UDdCrM02DS2XBtxj/VUE9dVZhggBPnpTyDtcVPwAkT7DIKSp06Yuf2YkBX7LkfNRr0fOgL9YB+X432SoyXLPMLq9VBPDDn0Jucxm114iBHpSeJTYJe9SJKaQ8KLq8seMIgLYx1KI4CvbqCvEjXpbq9+ebrKC4qwoiRo3DYYYdz/1LcKSwowGOPTcPwESMwYvhINZyYiCiWZPieufM4QE6+Xr81VYHfEs9cv4rdfr1+b3tuZ86sCvqqhnyqDJ9+0QzVSmdMRupxn8GQMdC3TPZ76Y8TkHL0uzCk94eWtw9d2naAY//vKP/lAkCTQEoHkykVXQy5gNum/pc/i9mCQ8wb8c0phXBmHolSq4ai3M1qSKdr4GMoN3RV2Ur37tmMil3zYOtwHsrcidXmJlodnnmIKplNpRu7C0O3X5XB0HtyUHhJb6CvvIWc+2Uz9Z67bQ50tVZWq48ovaWxsLfYjc/X5GFfsUMN5T310LRGS4rSNsUYlXWocUS85/ft24dPP/1U1cuaPn26WrZgwQKMHj2aiRgiPDLWNcsUtEjqxRdfitWrV2HJ4kV4++030blzF4wYPgLDh49AekZGpE8hUf3ozWp+C3Q1539kZmXhzrvuVnN1//f6a2pe6ZAhQzFi5Ej069cfen1svuyIiKr3+h2mTpY+Vx/o9ZPePl+v30q47YVw750Lx965aoih5rJ65ghmH+lL9KJPOVA+pj5qSzBSX0ETksQLY3L1RDjGZNV7qktoB31SJyAB0CV2hH3TTM84P0XG++mrgjxZ5h38JxO+nLBv+h9STjgXyek6tDUXon3q30jta4Ex0/PYczf/DSNeQeqJ58CY2aVac+zO6olqfNlKq2Uy9Qw/zS+pgM1lUtfJ3ENRYXerU06xs14Botmoq5m51FvuItEA2O3olF8akOnUgCRz/eshSlKU2SuKVVD6zHdlsLvKfLvvjV/ycczBKbjj1PZRT4py9MHJePeP0JGzPIpj+vFASaxE9GwvXboUY8eORY8ePfDnn3/6Ar2PP/4Y69atw7XXXhvtdrZ4HTNMuHd8h6DXSQZD6cmTk2TW2rx5ExYvXoSHHvo/lcJeelGOO+54ZDDoo0ZkSO6K5BHPoiJEeYW2bbNx6qmnqVN5eRmWLV2KOV/PxvPPP4tBAwfhiCNHo2uXjnyOiCi+ev26nAbIydvrV7iqqpj7MhWYaVXZPV2lm2Db8pbndpYsT2+ff4ZPY1Ltd6ZpsK6aBlfeYnVubDenQUNEgyYkiTF9QjskDJiizutcLvu6fAegNwDuqgDK5Q3y/Ejgpzd71nU7AIM5+PbMbYPet7rKqEObFKM61SV3/x5kt+vkaY5bU72FgcloJAAMrIOYW1QOm2byBY5S7kICzNrKXXiCw5rDTA16T9ktT0ZTT91Do1aJdlm5Klj0r4PoCSL12F/iwGs/5Xnu16UhxWJQry/53Sjtnf1XCU4/PE0lRZFSCHP+KolKj1//TokYOygV8/8uVfftT+ZTjumfin4dWW6pWQV6d9xxB6ZOnaqycPp/SE2aNEmVWGCg13hkf/fu3UedLrroEuzbtxeLFy3C7t27GOhR3EhOTsExxx6nTpKVc+XKv7Bu3VoGekQU/71+bYaok7vyDGDNk7BZc5Aw6C5otgK4Cv5UwxDdtny4934Px97vq25ogCGtr6eoe9ZhMGQeBn3yQdUDuS0vwbHnO7WuY/+vcOz+JuJePck6atvwqqp/F08k4EwcOCWs5TqDGWljf0D5snvg2ONJ3pU88iWUL75VolgYMwbCWbwW5oPORUK/G6C3tFW3CXk/luyg993QchcSVMmpPgGip9yFDBv1BH0S/ElgKAGXf+9hblEpHEio+t+zTJLcSMBUWO5SJ2+5C09QWBTy/p1uDUUVbt98xGK3WwWMMlLMZASsTuC7VaWY/WcJfl4vPX5aVHr8ZP1nLu6Cr1cUY+pne9U+k/t3ujTcNDYblx/NrPDNLtCTEguzZs1Sl/0/xHr16oWNGzdGr3WtyM4CO578Ogd3ntYeXcM8qiLZDtesWYNevftg4MADY+GJGoOrdAsqlk4BetwFwPNlFo7du3Zh65Yt6sCEfBESETUH0mNm2/wm4LKicvXjvuQi8iNehhKq3iV1sqshns6KvXDuW3BgAzIMUW4jJ80AvbNqeJvc1lmGsh8nAJY2EfXqSc8jbPlAXb2IcU6GcerN6b5968z9AzqdBn1qT1VKw/Xn/Z6gL3NwMyx3YVCnjghd7iJ3v8sXHPqXu/DWQfQNMa10YW9uATRTao06iN7hp3Lu7QuVGM7l0lRdO5mf6LVgbRnyy10wGzw9hqF6/IKpqxcwUeYxJhhwWLdEDO2epEqGbciRHlpqdoGe2WxWpRRSA14MEnS0bXugvhuFT743nG6/oeq1WLv2b8yfNxcLF/6hevauuNKTYYyocVX9uAkcVhNEWVkZfvllAebPm4eCggKcOHYsnxwiapZ0hgSkHPOBSiJS23w5yerpLFwFV9FKT3IX73BE+dR01ExBL5+npuxR0GcMgKXb+apQfLhk+6ULzodOgslozP2LZbFx7UCBdseOT9W5ueuZah6kcJdvR2siCVyy5RTQs5a7vxzZ7UL/xv57dyUu++92FeRZDBp0ej2cLk9Pn8OpqW/uokpPD7DdBRRIwGfUIdEkSWN0sDq1oGUQZO7fU3Ny6uwFlMBTyNDSET09gd7ybRWqhzJY/gmK40Bv/PjxeOCBB/Dqq6/6jkRJT951112HCRNYCqCxshr++NN8/Dh/HsxmC5wuJy697AqcdNLJjXJ/RPXldruxauVKzJs/Fyv/WoF+/QeoIO+VV2fCYKh72AsRUdxxVUJzVqL05wsOlAsIg6fXz+Xp7XPaAC1Yz4YGx84vgJ1fwLbuP9AZE+rXoydz1wxJ6r4aPPcvhsXG1WMJuGw66CzoDJ7eSnfZdvUYWfKidlICQnrVZMikBHbJJh0sRk+PnfQKynzBThkm7CyoGgqqeXr7rNLtF1AGwb/3zmp3Y/aKEphNOqRa9CF7Ab2Ja2RuYJ/2FmQmG9TQ05U7K1UPX11q6zGM9DqKMNB76qmncPLJJ6vi3fLjrk+fPir75uGHH45p06Zxv0bZJx9/hG++mYOjjjoKU6bche49eqgEF5KkhSgelJaW4o4pt6F9+/Y47rgxuP76yagoL8c999zFII+Imi9jMiy9r0JCv1tUkfZQ8+Xs2z6EufvEoOvY985H+U9nBWSSVBld1JBP6dnTGSxIGHgn9MndYF39GBIH3Qtj9shq29HcTrgr93oKwecvhnXdC9BZ2qD854lwl6xX23bsm4fK5ffC1Gmcmgdo6XFByHZX46qIcAcFzwIqwskIat/xJexbP1QJVgA94CiBIb0fDOkDPCNIdHpoEmxb90OX2N7Tc7n0DpUAJ+nQqdXup642xUvCmsaSlWzE1ce2qcq6maPmCKpRYjogwaTDSYNSVQ/ezoJipCd6AjarwxMAStAnFm+uwNPf7senS4pgVcPMoHrk5GoJHkvcbrg0TQWQZpNn3p+3F1CGk3p79KQwvfTqyZzAJVsqag30ausxHNHLk61z8eZy33Vy9Ss/5uGGE9piwz5btdt5r7vrtPY4f2Rm0+z4OBdRpCDDMyWN+pw5c1QGTgn2hgwZonr6GHxEX0JiopqEK8Nli0uK1f4miifyoZ6amoLi4iKUlJTAWlkZ6yYRETWYTmeApc8/YEjrFXIdrWwbrOtfgKnr+OolBap69uwbXlZZI1VgVzXUUgIrKc5u6n8rKn+5UAUylTIHWgWCbpT9PBEwSKZCl6dn0Hfy37q7KsDzK0/gdsO65nFY1zyh/reu+hdgsHjuX2cM2Sumd9qgoWFFrf2zgIq6eghl31SueRJwlXuybUoZHs0BTXpBqxLj6JM6q8BWhm9KaQbVc1mwHK7CFbD0vdYTrPjus02zyEzaWGT45BVViU/6ZZVj4a4EXy/XaYeloXOmGb+sL8Vny4phc2hISZDSDwY4XZ45gSoBTIVLDblUxeUtOrghwaDn9SXBoAqoVLmJAy/E/aVO1av2+8ZyFeyt2mVV/4/slawCvUWby3Hd8aGHnK7dU6l6BlUpioAewx9Wl6p3hH9vosPlVklnXpqfB6tdC3qdBKuj+yZDDh+09h6/iAK9119/Heecc44K7OREDScvvn+e2SFoUcnTTx+PE044Ab/99hs+/OB9vPTiCzAaDeh2EItSU9PRJ3VF0ogXYS1zB82y+dT0Z1TPvswfnTLlVnTq3BkulxOVlRVITGzeCQOIqPUJVSogJGc5NEdptUWayw5X2VZPoCW9d369eZK50yjBhzkLCb0uU4lfNFte1Q2lN8sAnQr2DtBJPdPkLtBZsqE5iqBL6QXHtg9UEKduU9VDCGOq6h1Tgafe6LutIWMwDJJVNEvKQhyuisOLvL9ehGHtVFVWor6JT7y9ZlJk3rsfPI/BrQJYd0WQJFzWfbBv/x2uvEXenSdxq2d75Tth3/IOjG1HQm/Ogrt0qyqs7iper3osPY9TB+feH1RZC/WY5T6dlhr739eWeqpPT6B3fiR6TAHanRWVbTZUx3Q9JvVpG7QUwumHpqleMP8evxSLHv06WrBsW6XKFyFBXZlNrvT05nmZDFBz+iocbjX/T/y8rkwFedI7KMHiwk3lOP+FrRjV29MbtzXXrko/tEsLPvT5l/Xlqh0JRs8cQr1Op2oKeucNyv1L3XmZUyhDU2W+nwR/9tquc2n4aFERtudUYPH2ra26xy+iQG/KlCm44YYbcMYZZ+CSSy7BKaecApMp/LHrVJMcyZCx06HID+UTTxyrTnv27Ma8eXPx1VdfYP78ebhh8o3o2/dg7lZqVJLW2pByEBDsS7tKz5490bPnJFx+xZWq1qPMKb3m6itx9NHH4rrrb+AzRETNRqhSAcFoznKUzB8fdB6fJj1x0qsmo3Gk16qK216A0p/OBmwFsG56XQVm1XrnHMXQ3EmeQE1nUPXmZA6b9CCibJtnHmDuQs88QP/i4nIuAY4+AZa+k+Cu2A1X3hJVCN5ZsFydbHhN3YshtbenAPz2L1Vvmm3ja7D0urxe8+G8vWZJI5737QfFlo/SHyQRV81t6aGhPFRiL1cFyn+7rNqiyqW3Be5VVC67o+qyTt2nHgYU+e1ftZY3K2o9hdsTqHolV/yfmh+p156DdvCEkPsuHnoXpdfvsYmdsbvQjq9XlFTr8Su3uXHe81sPtDfI06OrShYjv1klKJNgz+bU1Emuk55Ai0kyh2r4fnUpBnW2YF+xSw0JPf3w9KBt8s4LLKn0BpWaCtz8794TdEK10bt7vckLy22e2/lf53ZrmLumBNvyHDDqoeoRyhxGmb/o3+MnvZxo4b1+EQV6+/btw9dff413330X559/PpKTkzFx4kQV9B1xxBHRb2UrkF/mxDd/lWDcoWk1Cnq++OLzGDhgII4bczy+//47dOnSBZdeerlKV7/izz+h9w4FIWpEnnkoHwApx9Uor5Cfn4/777sHL//3VZVx88MP3sPV10zC6NFHqeuWLl3M54aIWiydMRmpx30GQ0boUkd5uXvRNrtj0Pl9MjevfNH1gD5FzdfTZCij26myfYbKoim9hSXfjq7qMZNerQrPcE+dHnpLG7idlTB3Ga/aJAGJu2yrGvboKQa/XPU0uko3wlm0GrB76rO58haictldSBhwK3SmtPAevF+vmXc/iNIfJyDl6HeDZivN3zgbhjW31+hx05kyPH1ImkvV1HOVbIJtw39gyDgEzvxFqtSFp3agphLRJB72MKxrnkTqmC9Q6MhCRoqh2nxJqXmogukgwspUWtVLq+oWbpTgWAdLn6t98x4r/34WzpyfpOFA4VJVG1GC8sq/HkTioQ/C1O7IoPvJ28On2/oy3KmTmzzwkwBn0pjqPX4b9llhNuphcEuxdZ0vYJWAT8o8OLw9fTIXT4Z2mvVITzJgR75DBXneIEtuJ8Ge9MaZpJAfXFi8pTxkoOcdyWY2ynxAzzLZrgRrxZWebl7ptZNOOaNehySLHsUVLs/9SU+gBJZOT83DJLNO3Uayikq7hPRQSnAnTZGhqt4ePwl0zx6WUWN+YEvr9Yu4vMJZZ52lTjJv7NNPP1VBnyQL6dGjBzZt2hT9lrZwciTi903lOKZfCtqkVL9O5uS5qw5drF+/Ts2DHDBgoEpyMXTYsNg0mFodzVkKx94fgJ5BXnOaBpfL8wlts1qxcOFCFeiJNm3a4OSTxzV1c4mImpYxucYcverXl9a43mBKRcLge1E271RPmQGdRSVdUT14WiVs656HqfO4oL1EEluknTRfDQF1lWxA+W+XexbqjXDLEEZ7UcheRqF6BF326j2JmgvWv5+C9e8ZgCHBc9Kba+3hq9FrZvQM2ZOAU+bVSa28wPvV737fU4MwcFsSDJnTVaZS+9b3YD74BmgbDXAV/VVV3sevHZoL9h2fQ/UlyX1qKdBc+dXnS3rbEuSxV66YqoaOhspU6t9L66lbWOCZ97jpVc8y2X+23KrSQ3a1L1VtRHVjB8p+PEMNy/VuN3A/SQ+fbtsrcPc9L27mD0rSFumlq3BoSDZ7gj3pfZUgyuHWVM9fgknv6wWc/s1+X0Dl7QGUDJ/+23O5Ncz7uwx6/V41ci2wt+zog5Px3sJC2KumiMrukiBPegXV/1XBmvD2+Mm59CrKHD0JDr29ezLf0Et6F91+3YJyVVFFVXkJp4avlhfhy+VFqtdPgsBEE+BUyWc86z46ax96tjNjWI/gr6HmosFpG9PT03H22WerYGT//v1YvXp1dFpGRERE1NLm8gVyO+Aq3+GZx+cfMOnNnuUSIKislCEKjlcFUtID5+1VrC0TqP91rpL1KPv5AjWYUmeUOW5lnrINpjTVs+h5fO1h6nomzAedpXqhAnurvL1m0ovovx8C94l3jpq5+/lA5e4DiWn8qZIUNt9j1yd18SyWYFSCTndVdlA1lNUEzZaLhP63eu7Hr0yAtydOSjQkHHyjOvefv2ffMxfOHClur8Gx/xeU//4PJA6+17Ppza/C3eUYNT/S2yMpdQslQY70okpQlnTov+DIXQjb35L0Ru0l1R7PA616DjUNSYf9q6qH7/+Q0O96WNc+jfJfLoG552Wwb3nL81gD9o83g6h1/X88+/Hg65skEPTP2vnc97nV5vDJfLmxg1Jx+ymemnmBvXFpiZ75cV6SqVN61vYUOlRwJQHXV8uL1cjlwPp7Mm/wxIGpmLu6VAVwcp+VDg2JJh2O7peiAsUf/y5T18nQS6tDgjIdbjjek3VTErZIrUC5zhvYWYyeXr5QZL2teQcONEhgVxZQAUV6Ba98dQceOLNDs+7ZizjQs1qtvuGbkn1TAj4Zvjlz5szotpCU5cuWoqioEDu2b1c9JnLZa+jQYejWzVNYlChWrNZKfPbZJ6isqPRd9kpKTMIp407lk0NErXouX6j5z2ljf1A9czW2bWmrrq9vr6L0FCYecl+tWUKNXU5XPYaqJ1Ev9fjcgDFJdZ9IcGNsfywcOz9XPU+2ja/CuuEV6DQnNHshrH8/reaaqd6qql4zKfXgvx8C94n/HDX38A+QuOtF2Hd+VZVNVEPioHtgbDscuqrC8eqxm1I8WRjNGUgb8yUqlt0JZ/E6mDqOVYGZrKNPqhoOW7on5HxJeby+xy+RhF32tbcbqBz2zTNhl5qGOh10tgKUbfuv6un01k9UvXHOSjV01F20BmULzvckkDnw6KoFbZ47cqBi4fVV9+OC9a+H1LnLXojK5d75hQZPe4JlLV3/IiS1iLnbOU0S6Pln7ZT5a4Fz+Lzz2fx5e+Ok900lQqnKmOntjdue71A9azpvnT+Trkb9Pbnf0w5Nw7KtFSrDp4xgO7JPMu45vb26z3V7rVixvVJdJz1vVx3Tplp7ki178dWfJTAagPZpRmzMsft6BxtKAkLptfSfz9cqAr2rr75aDdd0Op2qQPrnn3+Ok046ifWyGkn/fv2xdt1a7N69Gwd182TalMu+6/sPaKy7JgqLJSEBI0aO8r0u/S+LlJSA8chERBS0Z67JehUdxZ6MoDLcU3qhNO8kK+kpK0DigCnq5Nw3Xw2RtO/9AZoKRjXPsl2zYO4wJmSvWQ3+c9QSOgDS46aCMU9AZup8sieLZuBjs2TBLcMmjSmq11Juo9MbQmYHrWu+pMzNU+UrZIisd9ins0wFjnL/JQsuQNLw6XAX/Q1TlzOgT2jr6flcID2fMjRQX60cYnDeFTxBnmeRlI/w63WU7cANx67Z0BsS4JYkO5pLZSpVJPCOUQ6GYHP4ggmVxVN6ADtkGNWwSJU9UxVmd8NsNPiyaXrr7wnJ9inDQ1MTdCq4GtApwRdYSdZOuU7KPchxhWuObaNKOnlJAJls0avafiN6JqtAL9GsU72CqJpz1xCVdrcKesPZHy0m0Nu7dy9efPFFFeRJIhZqOEklK8Us5TzQiWNPUieiWJJJ8uZu58JqrDlBXwK5m266JSbtIiJq7erbqyi9XipJiQQSITKCFs86EEipXidHsd9cPgfKfzwL5ZZs6Kpu599rFvQ+A+aouUoD8jmEmEuoT+6uAj13+bYD27IdGNVUn/mSqrdJejFVkKeWVAVTmio0L0NWYS9AxW9XqIjFU4+waj0fKWFRVz1j7/p+gZ30nFbj2YZ1xf3q5FX6w4kHbhJijmG8qC2Lp/SESaBXaa/KpqkBheUu3yxLb7ZNeU6kN08M65GExVsqkFd2YL9tyz3QUyovQwkoUxIMvmUqoJP8AE7pCUzyzPerSqxikNi6amhmpGRTP64tbV2BngzVpOjKSDZiwtAM7laKW/qENirtNvaHLq9ARETxL7DXK1hG0Jq9YBd4etWgg6aydGowpHZH6nGfhpWhs1oGTGcF3JX7qrcpVKCX0g0oWA532XbfMs1eEFnPpm8+pPRiVgUQKimMTtUnTB79Jkp+PAepx75/YG7eLxfD0vd6WFc9XBXgeXriaqWS6fj33oVaT5LnmJAw+H5Y1zylbiP7IXHIo6hYfg90zSSrerAeQP/5e9IjV1LpmasnnXGSXMV7/c4CB3JLnWro55gBqZ5Ar/TA2Evp0QtMXlgt0LMfeC56tktQPYxz15SqOXsu6RT1y7ETmKAlXGv3WFUw2xyHb4Yd6M2dO1edn3jiib7Locg6VD/Spb0j346D2phVRiOieKM5Kz1HYF0sfk5E1Oz593oFyQhaoxfMmxVUFkpgJ0M/C/5E6S+XIfWY92skfQl2fz6VB4I2H0lGE6JHT6ihjVWkJmAkPZu++ZBl21EiWTFlLuHh01D5531IOuI16MwZ1bKFyrxJmbto2/w/v069uruHdBmHQiv8s67xnZ4uKs0Fh8wPVDeUYaE66CxZVT2N4dczjDf+8/dSEnRISzSoxCzeDJpHHez5LeHtzRvcNcFXT9rb2ye25VUP9MpsbvjPWPQP9CSRjPQw/nd+Ll5bUAAd3BjUxdNL6O1VjIRkIm2uwzfDDvTGjh3re8N7L4finVhK4dtf4sQz3+WqyacS7BHFG3flHvVliJ6SmaxXrJtDRERNMZ8vVFZQSdzidsJV/DfKfpqA5GM+hCG5a3j3W7GkxvU6nTF0j540o9y/R68Emlt6vw707IRLAjid9ExW9SCasg4HJBFMVe+m1n1S9X0jGTxVGYXwaYXLw1zT6dmH+VX7Q58ATTOqxDYJB0+uWhRhNtcYCzZ/T3rtZFil2aDDvDVlWLfHrrJxSi9dr3YWtE31PJ/5pU4VS8hy+X3sra0n9fNkGUIEeraqKDLZYlDz9nTQMLpvCtbsrlRF1yONTuR2m/db0RyFHej5B28M5IiIiIha5nw+/1T/sn6orKAy0qNiyU1wlW1D2fwzkDDwTljXvxS0CLn//eoqtgVsyA3rxteClhIwpHh69Fzl21WWzaobQHMUe3q+Gkhnyay2P7Qe16nzyjXTYWwz1BPQSlAoP4MlsJSeNilo7yz1C36DZGeRYZmqKJx/j5Spal6glIcwwt3zNmR0P07t24rFNyNp2FOoWHaXKlWRdNiDaM5Czd+ToZxPf7sfnywphsng6S2TPffWbwXYmmtTWTelhIIEh9vzPftOAkAp/xAs0JNEL14yH9Bb3sFXNsHqQqLZALNRU1NRrVXBoKxTn7l7+4qilMqziUU0RrBfv34RXUcNU1CQr+oV+svNzUVpaS1ZroiagM1mw4oVf/qKphMRUfPlTfUv595eMMlyGXgyZY9A6pgvYUjrC1flPpQvnOQrQl5rp0CwQG/Dy77786dPrurRq9gNzS9okjIP0XmwztCP3xvcuqsCO5mbKBlAvRlEfe0J8lhlqGu1IE/IfECnZJPx3KbdcTB1PEHNB/QkxklES+Odvzd1Qkd1flTfFN/wTSl9KLGwmrfn1PDtqlIYqzJq5pW6sHW/Z/91b2tBSlWywnJb9d8ZlQ53jd49lYylSmFVkXSZJ2gySmZPgzqlJxrqNTDWL1dRsxJRs9evXx90uZRb2Lx5c0PbRAEqKipw151TcN21/8A/rrkSy5cv8133wQfvYcmSxdxnFFNlpaX418MP4rprr8G7776Nffv28hkhImruqgqP13aSxC7JR72lkplIkhU15yznZzh2fxP6NuVb/MoPeE/ByXw5nRRL19xwV+wKP/NmuGpJmiJF0y3dL1S9czpzpqrxp06WNoCvd7E2gaGEp5yDzpxVbc6id1irIe3g+pfJaGbm/FXiu+xNlCJDOqXsghwb8IZtMk9va9X8vJ7ZZjUUU0gdvlBDNyXfhXA4/QK98uDPr7dkQ7i8tQFbdNbNn376KehlIT1Nf/zxB7p3Z+HuSEgK2IwkgzoPNG/eD0hJTcG7732InTt24NnnnobD4cDIkaMiui+iiOgMni83BJ8TkZXVBvfe90/Mm/s97r7rDnTr3h0nnDAWo0YdAYvFwp1ORNSMBCs8HnJdVYRcMmFW/Rh2VaDsp7MAc5anqHrgutZcGXxZPTtliFICcnvp1ZNadpq9OOo9eprmDPn4y3673POPswxataCtqmBcnfQqU6mnJIMTicOfgyl7JNzWXJT/eknQYa3GzEFoyWQIp7wE1GDXqjp3dqfm64XzZsbMl0CvqrRC92wzSqoCPP+hm263DPM8EIB5L3uHborC8tBDLr1DOr11/mrLyClzBFt8oDdmzJigl4Ver8dBBx2Ep59+Onqta0Wka3vaecGLpe7buxfDh4+AyWRCz1698OCD/8LDD/1fjQ9PosYk8yRSRr+BylrKK/Ts2RM9J12Hyy6/En/88Tvm/vA9Zr72iqoDedllUpeIiIiag7oKjwctv2CQ4uOlVcGbhpQjZ9aYqydlFYrmjIbOaIYxayic+YvV8EktRDIWb0IWCfSqbacRh276P36dJRtlC86Du3K3r4C5pubolQGGdE+wqoLcqgBE5u3pPQc39UmdkXbKr3CXblSlJSTIkyGvMgdSeu7KzM0vi2NDyTw9789X6eCQywadJzgrrnSrZQVlLlViwZtxs0e2BZurhnH6B3pSO89/hLDM60NV4Bg4dDMU/5497+2D6ZplbvmBnvQiibZt2yIvL69GoCcnir5evXpj7bq1vv8zMzPxz39OxYMPTVWHQwYObNlHf6j5kR68444bo0579uzB4sULY90kIiKqrxCFx0OWX5Di69KDJcXV3Q5UrpgK02lLqh2Ydhf8pQImQ3I36GR91bejq/XgtbfEQrX7jVagV1u9O2MyDGm9oJOELBLQeXskZY6enHuzfqpyE0USHcLcZxIS+k7ytNvSVtWglazV1R5PVQ9eWSusS3vqoWl4/ef8qt43DUa93vMacmgq6OvXwYLfN1Vgy367CvhEt7ZmpHiHbvoFev7DNqsN3fTr0SutDC/jSoJJh0p79UHEEoAaJY53Acf0i+/i9aHUKzIzGo3qVFRU5LvsPTHIaxjJSnTfx3vUeaBjjxuDwsJC5PsF123atsUDD/wfTCYzEhISGnjvRHVTWdV+uwKo2FnjOvkMCDVsu1OnTpgwoapILhERtawSDDXKL7irauLp4CpaVW1enVpd6rHKfaT29gzfFDp9rfdnqCqx4C9ac/SCDd30f/ya21GttIPnRq6qouv2qscsAYknUNWZ0n3JavRJHSMradGCdcky47ZT2iHRpFPBnWTFLLe71Rw9KcdwcCfPb9pl2zz19TqkG1V9vANz9FxBE7GEGroZrqBz9nQSAOpVu/p1bJ6JclgwPU5IilcpJBks1avBYFA9eIHatWuPGU8/2zQNJNJc0FQGsppHP9MzMnB/kNcoERG1zBIMNYqQ+5dfcDtQvmgyXGVbULn8HiQf9Y6vx85V6knaZ0jtBWfhX1Ub0dd6f/qqEgv+GjPrpv/jd5VsgKa5VO09KYkgmTc1KX8wbDqMmYf6blP87XGAwQid29qg/dkanD8yE6P7Jlcru3DaYWlqGtOCdaXVkqjIsE3hDfQq/Hr0/OfnVcu66Td0sz68c/ZkCKfTpan7PHtYBi4YlanKRTRHLJjeTEja+j//XI4dO7ajvLwcGRkZ6N9/AHr37hPrphH5lJaWYPny5cjPz1PDN7t378GhxURELZyUX5CTv5Sj3kTp3JPh2Pcj7JvfgKX3lWq5uyrQ06f2AgpXhLf9Rh26WXt9NGfeEpWYRUogyH2qNCw6A0ztjw0oEK8BtkK4Sjw9luHUKGzNvGUXAgUGVJKIRXjLK9Q+dDPyHj0hNfxkDqAEirKFUqsb7y8sxAcLC3HXae1VgNrcsGB6MyDDNv9v6j9VMpaDD+6HxMRE7Ni+HV99+SUO7tcPU6bcycQsFHPz58/DO2+/qebw7ty5E4cccig++/RTNXTzrrvvQWpqWqybSERETURq6yUeMhUVf96PypUPw9juKBjS+sBVdqBHT0omhEOVbpBEKH7rN3YyFi9n4SrAUQZ9Yge4XZWehTq9+t/rwLw9B5z5S9T/oeYcemv0mTqdJBNxovMYWpA2KQGBXtuqQM9iqJGMJZw5enVxVWXulCBPLkttPy/JhC9z94oq3KrIu/RCSoDanEQle4okafn111+xY8eOaGyOAnz7zRz06t0b02c8g0nXXodLL7sck2+8Gc+/8BK2b9uK1atXcZ9RTEkNzTden4l//XsannhyBq67fjJSUlPx8n9fRZs2bfDO22/xGSIiamXMva6AqcMYlaVShnJKDT13+a4DPXq1FVX3IyUeJIOlP00Kl/v1kum2vqzOg5HllWumq3PJEFoy92RoLk9OBBmWWU3BIhR/c5RaT9Z37v3e096EDirZivTcqbl2LitcJZtRuXIa7Fve8ySjUe3KrbWGoK/YOgWVlVw90JMaeiKpauhm9UAv+NDNcAO9SrsLBeUuNVRTgj3/IM+zHU9mTwnZpfyDDDVtbiIacDp//ny8/fbbeP3119X/p556KubOnat6nL744gv1P9VPuzQjbj05W50HKi0tRbeDak5EliQs7Tt0RElJ83vhUfOjT+yExMOnwWpNqnFdZaXnKGfnzl3Uee/evfHxRx+qz4QJZ52DJ598rMnbS0REsSW9WknDZqD0++PhKlqN8sU3eYY4GlOgUwXHw+vRE5Kl012+M+jQTekl0217Be6+56n5cKF60Ywdx6Lyr4fgyl+ismhq+qxqQzelJ06//hG4Sv6CddU0JAx5HK7C1arN1o2vAW6bJ/mMrRBFXxysErVA5iZWBYCeO3OErCGo7kNuoxK4UDD7SxxwaZqai2fU63z1pb1DN6sFegHJWCq9QzfDmKPncsswzbrX8waTckxC5hO2ikDvvvvuw4svvqguL168GH/99Rd27dqFOXPm4MEHH2SgFwHJ6tO3Q/DsmVJD75lnpiMxKRH9+vVHQkIiSktKsHDRH9i4YQNuuumWSO6SqF50xkSVRQxB0kGnpqYiNS0Nc+Z8jWHDhuOrr75UNfXUdSkpsNv4pUZE1JL4zzULFlxVS0Qy9EmU/3EN7DtnA44SIG2wJwgKs0dP7kuKjKuhm95adjWGbmoqEKzcOQvm7hOhT/TLcKmGVbrh2PM9nDk/ebJmysmWC+f+36BP6gr7jk+hOcqB4uUq3aJj/68w7PnBl4DM1PlUuEvWwVW0BsYOxyPlyNfgKl6L0p/Ok8ZInxN0pkRPjT23E8lDn1LZOgPbIreXmnpU00eLCvHE1zmqF02Kl+t1Gi55eTuOOTgF1xzrGeYqvW9SKF2vl3II7oh79Crt7mqlFOoiL1dJGtMqAr1Vq1Zh4EBPAc158+bh7LPPRufOnXHJJZfgtttui3YbW4Wicid+WleG4/qlICOg2/rwIUNw88234euvZ6lekoqKCqSnp6Nf/wH49yPT1GWixua25sOxezZgGSpFE2pcf+utt+OlF1/AW2++jr4HH4ybb75VLU9MSsL4M87kE0RE1IL4zzWrLdAT5i6nwt5tIuwbXvIMcbTleOa1hdmjJ/flLFgOnRQi9ytaLmV/7Ns/hbHNUMBZgbJfLlaBZOXap9VwTy9Pz1serMvvrL5hzQXr6sfUKeAKVRDduvwO3xLH9g+qSihocO75HkWzh6reOzhLPD16Bgs0me8n9fa0Slg3/EdlGDV1HV+9FqGxedZja2y7CuxqHpz0ypmqevFMBqgSDLP/KsHJgz37UF42EuylJBh8gZ3Mo5Ohl+HO0au0u3y9f+GS3kXJDNoqAr2srCwV7A0fPhyff/457rjD80bIyclRiRio/kqsbny/uhRDuichI8hnwNBhw9SJKFY0RxHs2z8BegbP9NqnT188/cxzNZYnJSXhDAZ6REQtk7PcM/esDqZ2R8G+vuo7wl41j03Nk6saGlfbNqRHrqpguT8JpCTYTBrxPGBMQvLw6ahYegdSjn5XZcn03bzob5TNPcWvPJAnYPNcNCLh0IdgXflv6Tqsul5flfylalinuWqIp7PCc24wIXnE89CZUlA2/3RP2QUZjqlVDdXUmaAFFEn3qlZTL3BSWCs2568SNQ9O9qD01rlcnmLqUrDc6tTwx6YKddnp8gzflJJk0kFSanXBoDd4kqpUDbOsrbyCK8whm4GG90xqdolYIg70rrrqKpxyyino1q2bCu68c/K+/PJLnHXWWdFuIxERERHFGSk7UDJ/fLXes6DrSTeMveBA1ky33TOPTZarYEqn5ryFvH3VvDZN51Z17GBKU/X7fMGhZMPUNLjtJdDsRSj96RzPfRnMgAST6n79gyq/H/qaE9YV/6y+THoa/TOCStv9uawo//UiwJhWNZzU5AkM9QdyHLrtxUFLN1SrqVcaPBhsjWT+m3ckrzdQq5AeuqppcXllTpV5UwK8T5YU4fWf81XPndMt8/qc6rZr91qRV+qstUfP6vCUTqgP6VkMlkOjOYio1Q899BAOOeQQbN++Heeccw5SUlLUcqvVivvvvz/abaRazJ83F506d1Zz94jiUVlZGT777BNcdtkVsW4KERFFkc6YjNTjPoMhwzOdJxTJYFn28wVq2KJOhjg6pOfMrWrvuUs2QmfJQsaE9SFvL/PaSuafCci8PBn66SxViVQ0e7EKNssX36ICusrl93oSpshQSlUKQafW9fTg1aa2n/466LOGwl2wtKr9idAkAYvejOThz/rm4RWUudA2u2O1NnMuXvhk/ps3d016kgHGqphZgrbiSjcSjDqUWV0ornTi1Z88QZ5FOlK1qiGeTk+R9XV7K2sN9Nxhzgv1Z9DrmuX8vAaVV5AA7/bbb1e9el533303h242sYTERJXZkChe6fV6pCR7DgYREVELI8GPKRWaswLWDa+oc/nfe5IMm7Z1z3sCNMl06ZvH5oC7cp8K+KQXzrH/99rvRw2NrOrekR4+ezHcZVtU4JV67EdwH70AScOeUj9tLf1uAuT+zVLg2i8jps5c+09fnRHu3ncjYeA9nt46z0IVzHnoPe1XbXDCvvUdJAy+Vw3h1O18p9pj51y8+jn10DSYDTrvQF5fxlKZoye+WVWKnBIn7M4Dc/Bk5Ktc0vtlN/15XXmtQzfdEhlGoDnOzxMR90NKN/yCBQuwdu1adXnAgAE49thjWbg7QskWPY7snazO6+PII0dHepdE9aIzpsLUcSyshvoFbTJH7+xzzuXeJiJqQarNNastOYuUIyjfoXrAfGUF3G7P/y6r53/NibIfz6ilJIH8uvcvJaWp+XTW1U+qHrzSny+AHgZUqEDQDdsaT3IVzVEcsKE6MkBrTug3PY6qVnkfgMrOqfh6ByUlpNHzuNyOoOUdAvcP1a5Llhm3ndLOk3XTocEmUZzO01sncZ0Eb9LL5x+neS/LcEwvGeIZqkfP5dZgj2BaZGShYTMO9Pbs2aPm4i1ZsgTt2rVTb0qZqzdixAh89tln6NSpZkY+ql2bFCMuGZ1V7920ffs2pKSkqqLURI1J0kMn9L8ZpUHKKwSzb99e7Nu3Dz169GRmWCKiFqbaXLM6krOkHvc5NL95boWFucjMzEbpzxdBK99atUGzSnBiandkjU1Kb1/5b5d7hmMaUzwBouaAscMxMLYdAfNB56CwzIUMUyHKfrkIxk7j4Njy5oENyBw6g8VzWcofeHsG9Yl+CVh0nm1rLiSPehm2rR/AuWdO1XoJgN6EpGHT1b8Vy+9B8ohnYWw70jNUVBVB18LbPxTS+SMzMbpvsipMLnP2ZLhkSaUL7y8shNPlKamgc1efYyeBYFqiAfllngguI6lm0h4v/4CwPiS0lzZNGtO2dQR6N998M5KTk7F582b06NFDLdu6datK0nLLLbfg448/jnY7Wzw5UiFHIdqmGGE21jWW/ACpVzZw4CAcf/wJjdo+IsmOpobZyJHYAMXFxXj1lZdxx513q//n/vA9Xn99Jjp27Ijc3Fw88MCD6N0neLZOIiJqXclZ9G43SqTnTmrjeTnLUP7rxTV69XyJXLy9gapXrerinh/gLPgLto2vqW2WSsIVWx4cW94KaJj0wHkDPb/+Ol+Qp1bybbv896ur395tVacKmQMobbMXoXzJbb7HqZLFOKv3A1JkJLOlf0D18Bd7ayRp8SfDNv1/NR/eLRHz/y6L6u7Xmmmx9IgDve+//16VV/CfnycB3xtvvIHBgwdHs32thryAHpudg3tOb4+D2lRP37pw4R/YsmVz0Ntt3bJFBXpEjc1dsRMVS24Fet4LoHu165wOB9avPzCR/u2331LBXb/+/fHVl1/gk08+wj33MlETEVFLFm5ylrzcvUh3b0LZzxPVXDc1x08yVLqdSDlyJkztj1bruSv3w7blXdi3vg135V5PeQNDoqdHToIrUwoyzlitMnDKNtOsy1G+6HrAUeRpjzkTmgRgegNSjnpL9cCVfD3MM+dOeukkgPPvHzIkqXl4OkummjcIXZIneYxk7nRVquQrxuxRsG/7sFohdEm8UizJYqhRk7R4e+scLrevRIJTyiXY3WodmeMXKmlKpd2FiqryCwjoEQxV5SLRrFMlG5prsfSIk7G43e6gCUCMRqO6jqJryZLF2LRxI6yV1honl6tm6l6iWCopKVEfihLkiWHDR2DXrl18UoiIWlFyltpOMCRXJWhxqmBNDcarStCilhtTPMGfqwK2jS8jadTLSB79JmBpg+Sj3oKl7w1qQJ0M29QntPFt07rpf74gT4I2TbZpSlb3I9uV7J5JR77m2c7RbyNx1H8BUwYsA+5QPYmybffIz2FI7e0JJvUmaFI+Qerk6XQq+Yo+tScSD7kPhrReAYlXwh+NRZEnaTEZdEg06WGuGqE5uk8yJh3XFl0yTWoIpwzvrE/tPEdVkCdF2r03NRuBrGQDLEbP/cpIu1aVjGXMmDGYPHkyXn31VV+Wzby8PNxwww04/vjjo91GAnDU0ccEHZ75/PPPcv9QXLBaK/HJxx+pgz0WS9UQGQAWsxk2my2mbSMiosZVr+QjWpAELWojZl+CE1UDz7tYgsfk7kjsfxuMmYepsgpqPp3bdWA+YO58uHIW+N+JZz6e0JngKtsGzVYAg/92bLmwGhNh6nIa9IkdVFF3FFfCbc0Ju23ex651n8TEK02YpCXRrMfJB6fgjlPbo22qEXPXlMLqcKo5fbXVztMFSa4ihdjvG99BXZb7EZV2t1ov0aTDbSe3a5bF0iMO9J577jmcdtpp6NKlC3r27KmWbdmyBb169cLs2bOj3cZWb8SIkSrpTTAnn3wK0tMzWv0+othKTErCuFNPg93h+VKUy156gwHt2/tlYCMiohanXslH9Gakjf0BbqlHF3iVpa0aihlq3p91/QvQJABzlsOZ+6sqtC7z+PQ2KcjuF5ipoZ1Jnjl1qjpDAYpnHZhe5NmOpxC73tIGZr8i5uG2zf+xaz2uq55tlBo1SYv0sHmDr10FdhSWO1FqdeGr5cWqB09q3wWrnRcY5Bn0wCmDU9V9iNrup9UEejIfb+XKlSqoW7NmjZo0K+UVxo8fD4MhdLYbCk0+hyRtbJCswhg5clTI2/XtezB3KzURnacIbZDhKVJC4aKLLgl6q8zMTPz7kUeboH1ERNRcSLF0OXm5K3Ng2/IOLD0vqXPen6tkI0p+GOuZN3fk6ypTZukCme/nlswovhp2Mi/Pf75fsLl1wYqaB7aN4i9Ji9dHiwo9vX1OTZVb+HFtKZwuz3DLJLNnjpqnzp7mm+cnv7edbqC4wqWGgvZql1Dn/TRXEdfRk/l4EyZMUCdquK5ZZjx3aVfuSopbhtSeSB3zOaxhllcgIiIKV8g6fH7z/rx0iZ1Vlk7pjav8e4YnS6fMqZMgT9W4MwEyv2/d8zB1Hqeu18q2qV48U9fxB7ZVFRBS8yQ9eU9/ux+VDk+NPSEdeRLSSeDndEmPnbN67T23GzrVKeVu9vPvGjXQW7duHZ5//nlVMF1Ij56UXejbt28020dERERELZyzZKOaQ+csXAlDiiezs+aogEHm0jkqqtXmc+Yv8gzNlF65vd8D5kxPMhdUzcmTent+8/LUnDqp7+d2oHzxTUge8QL0Cdmwb/8Ulj6cW9dczfmrBHaXpsYZSTDvdmuw+eUodMk0zYApeyVWDQabEyajrtnPv2u0QE+Kop9//vkYNWoUhg8frpZJ8fSBAwfik08+wZlnMsVsfe0tcuCNX/JxxdFt0DGjeaZwpZbNVb4T1jVPAe1lWA2HtBARUfRIj5sUH69YeD0ql93pWSbBnK0ApflLDtSs89bV859tZS+U1F815uf5z8vz1LorhyvnF7hKpByQBtvGV5B64necW9dMyTw67/Q7h0R1YTDoJQsncPrgVEw+MbtFB3kRB3p33XUXnnjiCdx+++3Vls+YMQN33HEHA70IyAt0Z4Ej7BcqUZNz2+Eu2wK0a55FQ4mIKP5ZDp6MhN5XqssSkEnx8qThz8OQ1kctc+YvRcWy2z3z8eBfYkq6blTfDmCwIOnwaTC2GXbg2pKNKF84SdKweHr8nOVN/tio8WrsyRDMYAXVAxmrAr0Ui6HFB3kRB3r79u3D1VdfXWO5LJs6dWo02kUhbNiwXmXZLC4uUufMZkjxRkoprPxrBQ459DB1PnzEyFg3iYiImgnbmsdgW/NE1X/yw11D+a8XhHFLb9CnqSCuYtG1IdcsX3STpzaef/kEapY19l7/Od9TPsEvs2ZtbFUvk7yy1lGHOqKC6TJEc9myZTWWL126FIMGDYpGuyiE7777FmvWrPadE8WbstJSvPLKf33nREREddGZpVSUDolDn0LGuTvVKXXsD9AltEPKmK+QfuZ6pI1fBX1KLzX/Lih9AnSWbMCUrursJY9+S91OTqkn/wZjF0kgqEfyyOeRdvwsldGTmn+NPZlr559wpTaWqi6utikRpylpViJ6lDI/T04ydFPm6EkULUGeDN287777sHDhQt+6Mo+PiIiIiCgUnTldBXr6lO6+0gaqlp3OoJYZ0j3J/lJO+gklMu/OUVRzI24bNKR4sm46bbBvfQ/mnpeoRB1ye7f1Ejh3fQUYEplxs4XV2Hvxh1x8+WcJDDqpm1ezXp4/Ge55TL/WEeRHFOjJPDxx//3317huypTqxTLD7Upt7dqkGHH1sW3UOVE80ie0R8LAu2F1Zce6KURE1MIY0g6Gscup6txLn9AOCQOmqHPfMun5c1WE2IqkWbR5fsnrzXCV7/AkZqkqcu5/H5J1M3Db1DzJXLvbx7VXAd7P68tQYXfDEZBtU0jNvASTDicNSkG/joloDSKKKkpLD6S4baj8/Hx89NFHyMnJweDBg3HWWWdBr699ROn69evx3XffobCwEAcddBDOOeccpKU17xoYyRY9hnZPinUziELSmVJgan8UwDp6REQUZcbMQWo4pT+pp5c4sHoHgt6cgrRxf8BVttW3zG0rQFnhLqS1GwRDRr8D61raQlcV5AW7j8BtU/PVNtWIxyZ2xu5CO75eUYLN+63YV1CBtOREVDrcqiOlVzuLqpnXGpKwNCjQS0lJicqdb926FaNHj1a192QIqPQG/u9//8OsWbNCBnuvv/46rr32Wlx44YXo3r07Zs6cibvvvhu///47evfujeaqpNKFJVsqMLxnEtISpRYMUXxx24vg3PcTYJCjrSyvQEREsWFsM0Sd/JXt3wNLO343tXYSxE0a01Zdzt2/B9mt/DURUTKWaJEATYK1efPm4cknn8T8+fPx/fff48MPPwx5GynrMGnSJLz55pt46KGH8NNPPyEpKUkFiM1ZUYULny4tUudE8Uiz5cO2aSbgkHpFRERERBTPYhboOZ1O1XN3ySWXwGDw9GD17NkTxx57rCrIHkpWVhYcjgN1vNxut9qWLG8Nhg4dhm7duvnOieJNYlISxo071XdORERERE0vZpk/duzYAavVWmO4pfz/xx9/hLydDNWUoZsyl08CHVl3/PjxuPHGG2ut6yUnr5KSEjRXRx45Wp336tV8h6lSyyY97Gefc6667D0nIiIiomYQ6L311ls499xz1Q+6SJWXl6vzwCQq6enpvuuC2bVrF7Zv344uXbqo20pv4Lp161BcXIyEhISgt3n00UfVMM9ATqdDjd+tj/rcJtx1Zb2CohzYbVYU5Ocg0dV65+hF8py0xLY1xn01eJsVOdDbrKoHPdLt1LcNjbV+PL/OmlI874fm/n6L1nYbso14+o6L19dZU4rn/dDUbYvL77gGbide3m+RtqUlcsbxfmho28Kde6jTIqh/kJGRocomTJw4EVdddVVEtfIkEYsM1fz2229x8skn+5ZLb92iRYuwYsWKGrex2+3o1KkTbr75ZkydOtU3dFMSuUjGzjfeeCPsHr2uXbti86Z16NnrQBrfcNRnYme466onOiEbnywpwrnDM5CdFqIQaCsQzxNnm7JtjXFfDd2mu2IvbJteQ1nG6cg+6PAmaUNjrR/Pr7OmFM/7obm/36K13YZsI5LbNtZ3XLy+zppSPO+Hpm5bPH7HNXQ78fJ+i7QtLVFuHO+HpmpbRHP09u7di5deegmbN2/GkUceiYEDB2L69OnYv39/2NuQsgjJycmqVII/+b9///5BbyMlGKQcg39gKdk5R4wYgdWrV4e8L4vFonr//E/xRoK760/IbtVBHsU3fVJHJB7yAJDQPtZNISIiIqLGCPQSExNx8cUXq2yZW7ZsUcM4n3/+eTWcUubOzZ49W/W01UaGXMq6MgxUeurE33//jV9//RXnnXeebz1JzPLMM8+oy507d1ZB2o8//ui7XoaRyW1CBYfNhdOlodTqUudE8UhzO+G2FwNuZ6ybQkRERESNnYxFyiNITTuZVyfB3s8//6yGY0qPnZRAqG1Y52OPPYajjz4aRxxxBIYOHYqvvvpKBY0SAHrNmTMHCxcuxK233qp671588UX84x//wKpVq9TQTwn6ZH7eww8/jOZsT5EDj83OwT2nt8dBbaoXclyyZDG2b9sW8rbDho9QzwNRY3KXb0fFkluBnvdKn3y16yorK/D17Nkhb5uYlIjTThvPJ4iIiIgo3gO90tJSVe9O6tdJIHb88cerwE6CNAn6HnnkEVU6YdOmTSG3IT10K1euVAGeDMs8//zzceKJJ1Zb55xzzlGBoJds87jjjlP182QYp9zvKaecEjIRS0tQVlaG/Pw8dXn16lXIzMpC506dfdfbbNYYto7IM1fW+xq12mxYvGghjjnmWN+uSbalcDcRERERxXugd8UVV+CTTz5RGTLl8jvvvKN61/znxD3++OOYMWNGndtKSUnBRRddFPL6cePG1VgmQ0Ql4Gstxow5Xp3E888/i4EDB+H440+IdbOIfJKTU3DtdTeoy/l5eVi9apXvfyIiIiJqJoFeQUEB3nvvPZx22mm+Yuc1Nmw0YsmSJQ1tHxERERERETVFoCfDJc8444w61xs2bFgkmyciIiIiIqKmDvTuuusu3HjjjarXjqKjS6YJ0y/sDItRV+O6ZUuXYvsOTzKWHdu3w2a1oqio0Hf90KHD0K0bk7FQ49Kn9EDKMR/Cml9Q4zpJxvLNN3M8lysqYbVW4rPPPvFdn5SYhFPGncqniIiIiKiJRBSpSXFyGZbpnySFGkav1yHRXDPIE4WFBdi9e7e6fFC3burc+7/o338Adz81Op1ODxiTAF1RjetcLne11+SIkaOq/S9zcYmIiIgozgM9SYQyceJE3HfffRgwYADM5urlAGorqUDB7S9x4MNFRZg4MgPtAoqmnzj2JHUiiiV3xR5YN7wMZE0A0KnadRLI3XTTLTFrGxERERFFIdC7/fbb1fn1118f9HpNY9Hv+rI6NKzdY1XnRPFIc1XCVfAnkHFKrJtCRERERI0R6EkNPSIiIiIiImpBgR7n2xAREREREcUvfaQ33LdvH1588UVMmTLFt2zBggVwOp3RahsRERERERE1VaC3dOlS9O/fHzNnzsSMGTN8yz/++GO1jOovM9mgErHIeaA333gdTz89Hb/99qtKY08UCzpLW1j6XgeYsmpcV1hQgDvvuB0fffgBtm7dEpP2EREREVEDh27ecccdmDp1Km677TbodAdKAkyaNAkXXnghrr322kg226qlJhhwbL/UoNddcull+PvvNVi8eBE+eP9dtG/fAcNHjMCI4SORmVXzRzdRY9Cb02Huchqwf0+N6+R1eO9992PJksV45+23sH9/Dg47fAhGjBiJAQMGwmCoeQCDiIiIiOIs0Fu2bBlmzZqlLvsHer169cLGjRuj17pWpNzmwupdVgzqkoBkS/UfxfIjefDgQ9RJSI/J4kWL8Mgj/1JF6+XH9JjjT0BmZmaMWk+tgeYohTN/KeDuHPT6rKw2OPnkceokPc/Lly3DDz98j5deegH9+/XHEUeORvduXZq83UREREStUUSBntTNKy4uRmpq9R6oNWvWoG3bttFqW6uSX+bCm78W4J7T29cI9AL16NFTnSZecCH279+vevr27NnDQI8aldu6H9a/ZwA97wXQt9Z1ExOTMPqoo9VJ5u2uXr0K69atZaBHREREFM9z9MaPH48HHnhA/YDz9uhJT951112HCROkmDI1lXbt2uH008dj4MCB3OkUl6TX+bDDDscFF1wU66YQERERtRoRBXpPPfUUVq5ciezsbLjdbvTp0wf9+vWDXq/HtGnTot9KIiIiIiIiatyhmzI8c/HixZgzZ47KwCnB3pAhQ1RPnxy9JyIiIiIiotiJKCp7/fXXcc4556jATk7UcBajDj2yzeqcKB7pDAkwpB0M6C2xbgoRERERNUagJ0XSb7jhBpxxxhm45JJLcMopp8BkMkWyKarSPt2EO09tH3J/uFwu/PnncuzYsR3l5eXIyMhA//4D0Lt3H+5DahL6pM5IGvYUyoOUV/DKz8vDkqVLkJ+fB5PRpOaQjhw1SiVnISIiIqI4n6O3b98+vPfeeyr4OP/889GxY0dMnjwZf/zxR/RbSCgsLMRtt96M9997F3m5eYAG7Ni+HY8/9iieeuoJaJrGvUQxN3v2LNx6681YvWolnA4nSstK8fPPC3DdtZOwZs3qWDePiIiIqFWJuLzCWWedpU5SZuHTTz/Fu+++i6OOOgo9evTApk2bot/SFm5Hvh2Pzc5R5RUOamOudt2338xBr969ccstt1VbbrVacecdt6nU9d4ae0SNxVW6GRVLbq0qr9Cp2nV2ux3vvfs2nnhyOrp06Vrtuvnz5+Hdd97CtEef4JNDRERE1EQanDklPT0dZ599tkrIIjXdVq/mkftoKy0tRbeDutVYnpCQgPYdOqKkpCTq90lUHzabTZ137Fg9ABTdu3VHSUkpdygRERFRcwj0pDfp66+/Vj15kn1TAr6JEydi5syZ0W0hYfjwEXjmmelITEpEv379kZCQiNKSEixc9Ac2btiAm266hXuJYio1NRU9e/bCk08+jlNPPQ3Z2e3U0O7du3bio48+xMiRo/gMEREREcV7oHf11Ver4ZpSMF0KpH/++ec46aSTYDAYot9CwuFDhuDmm2/D11/PwscffYiKigoVWPfrPwD/fmSaukwUa3ffcx+++OIzzHztFRQUFKjPg3bt2uOEE07EyaeMi3XziIiIiFqViAK9vXv34sUXX1RBXnJycvRbRTUMHTZMnYjiuVfv0ksvVyciIiIiaoaBngzVpOjqkG7CQ2d1QHpS6KekoCAfGRmZ0OsPJEvNzc1Vc/XkRzZRY9IndUXyqFdgLfHMxws1pNvhcFR7PUrP/969e9C160F8goiIiIjiubyCt8SC9OpJTT2vBQsWqB91VH9mow7ZaSZ1HkiGat515xRcd+0/8I9rrsTy5ct8133wwXtYsmQxdzk1Op3BDH1SR0BfPSus188LfsLll12Mq6+6HNOfegKVlZVqeXFRER5+6EE+Q0RERETxHugtXboU/fv3V4lXZsyY4Vv+8ccfMxlLhPJKnXjjl3x1HmjevB+QkpqCd9/7EPffPxVvvvk6Fi1aGOldEUXEXZmDyjXTAVtujeukluPrr8/Effc9gLffeV+VWHjkkYd9wR4RERERNYNA74477sDUqVOxfPnyassnTZqE5557Llpta1Uq7G4s3lKhzgPt27tXZd40mUzo2asXHnzwX/jg/fewePGimLSVWifNWQZnzk+Aq7zGdeXl5SrL5qGHHQaLxYKJF1yII48YrYI9q80ak/YSERERtWYRBXrLli3DNddcoy7rdAeGGvbq1QsbN26MXuuoar/2xpYtW3x7IzMzE//851S8/fab2LB+HfcSxVxKSoqalydDur1OPe10DB0yDE89yULpRERERM0i0DObzSguLq6xfM2aNWjbtm002kV+jj1uDAoLC5Gfl+db1qZtWzzwwP/BZDKrZCxEsXb5FVdh7tzvqy076+xzMHr0UergBBERERHFedbN8ePH44EHHsCrr77q69GTnrzrrrtOlVyg6JJ6ZNKDF0hqlM14+lnubooLI0aMVKdA5553vjoRERERUZz36D311FNYuXIlsrOz4Xa70adPH/Tr10+l/Z82bVr0W9kKpCUacOqhaeqcKB7pzJkwd78QMKbHuilERERE1Bg9ejI8c/HixaqenmTglGBvyJAhqqfPaIxok61eRpIBpx/GH9AUv/SWLFh6XgTs3xPrphARERFRHYwNGU4ogZ2cqOEq7W5szbWhR7YFieaIyxsSNRrNWQFX8TrAxQMSRERERPGOEUWcyC114oW5eeqcKB65K/ei8q//A2w5sW4KEREREdWB4yybmeKiImzeshk264HaZL1690G7du1i2i4if5s2bkRefh40t6cupNliwdChw7iTiIiIiJoIA71mZN26tXjk3/9SiW8SExN9y9PS0xnoUdx44onHsGP7dnTv0QP6qqy8UmOPgR4RERFRnAd6zzzzDG699dbot4Zq9duvv+CMM87EeedP5J6iuFRRUYEVfy7H/15/i/UdiYiIiJrbHL277roLTifnkkWT0aBDdqpRnYfStm02DEaWX6AY0ZmgT+yozkORnubk5BSVrImIiIiImlmP3uDBg7FkyRIcccQR0W9RK9Upw4SHzu5Y6zr9+vfHCy88h6SkZFXiwqtH9x5o4/c/UWMwpByE5CNeQUUt5RUcDocaovnM09Nx9DHH+sqtmM1mHHLIoXxiiIiIiOI50LvkkkswceJE3HfffRgwYID6Eedv1KhR0Wof+dm4YQM6duiI5cuWVtsvp48/g4EexQWbzYqCgnx1ef68ub7lqWlpDPSIiIiI4j3Qu/3229X59ddfH/R6TdMa1qpWaHehHc9+l4tbTs5G58zqgbN/QCcnolhwlW1D5Z/3A11vlj7ooOukpqbhvvsfaPK2EREREVEUAr3S0tJIbka1cLmBMptbnfvbtGkjcvfvr3XfsbwCNQnNBc1RIq/WaottNluNXuZALK9ARERE1AwCvZSUlOi3hILauHEDVq9aVevekWFxrKNHsWK32/Hrr7/Uuo709LG8AhEREVEzqKO3b98+fPrpp9iyZQumT5+uli1YsACjR4/2JWCghhs37jR1IopXUiPvzrvuiXUziIiIiKih5RWWLl2K/v37Y+bMmZgxY4Zv+ccff6yWERERERERUTML9O644w5MnToVy5cvr7Z80qRJeO6556LVtlalfZoRd4xrp86J4pE+qTOShj4JWDrEuilEREREVIeIooply5Zh1qxZ6rJOd6DAd69evbBx48ZINtnqWUx69GxnafX7geKXzpAAQ3o/wBa6jh4RERERNeMePambV1xcXGP5mjVrqhXypvAVljvxyZJCdU4Uj9zWPFg3vgbYC2LdFCIiIiJqjEBv/PjxeOCBB+B0On09etKTd91112HChAmRbLLVK7W6Mf/vMnVOFI80RzEcO78EnFJigYiIiIhaXKD31FNPYeXKlcjOzobb7UafPn3Qr18/6PV6TJs2LfqtJOWrL7/AXytW1LmMKFZKSkrw7LNP17mMiIiIiOJwjp4Mz1y8eDHmzJmjMnBKsDdkyBDV08fSCo2nR8+eyMrKqnMZUazIsO7DDju8zmVEREREFIeB3uuvv45zzjlHBXZyoqYxePAhYS0jipWEhAQce+xxdS4jIiIiojgcujllyhS0b98eEydOVNk3HQ5H9FvWyqRY9Djm4BR1XpvioiIsX74Mf/z+m++0f//+JmsntV46UxpMnU8FjKm1ric9/H/88TvmzPla/S+fD+vXr2uiVhIRERFRxIHevn378N5778HlcuH8889Hx44dMXnyZPzxxx/cqxHKSjHiglGZ6jyUdevW4sYbb8A3c75WP6S9p9xcBnrU+PQJ2Ug4+HrA3KbW9V7+z4v4+utZ+PyzT9X/JcXFeOrJJ/gUEREREcX70E2Zc3PWWWepk5RZ+PTTT/Huu+/iqKOOQo8ePbBp06bot7SFszvd2FfsRId0I8zG4PH3b7/+gjPOOBPnnT+xydtHpLlscFfsAtyhjw9J793vv/+Gp6Y/gwf+eZ9vuV+5TSIiIiKK1x49f+np6Tj77LNx4YUXYsCAAdi8eXN0WtbKSJD32OwcdR5K27bZMBgNTdouIi8J8iqW3ApY94bcKVJyRRIymYwHjiEVFhUiLS2dO5KIiIgo3nv0hNVqxddff6168iT7pgR8Mmdv5syZ0W1hK7dt61bk5eepy9nt2uH9999FUlJytcL0Pbr3QBsWqqc4kJiYiA4dOmL1mtXqf+nxf/+9d3HY4cy6SURERBT3gd7VV1+thmvK0XspkP7555/jpJNOgsHA3qZoW716FVau/Mv3f8cOHbF82dJq65w+/gwGehQ3brn1drz4wnMoKirEtZOuxqhRR+Dcc8+PdbOIiIiIWpWIAr29e/fixRdfVEFecnJy9FtF1YI4ORE1F5Kc6d+PPAq73Q69Xs/amkRERETNZY6eDNW8+OKLGeRFkSSrSDDpak1a8e2332BN1ZC42pYRNQ4ddIZEdR6KZOLdvXuXL2nTqpUr8cMP36vefyIiIiJqBslYNE3DTz/9hP/85z946aWX1GVZRpHpmmXGjIu6qPNQNm7cgJycnDqXETUGQ2pPpBz7EZDULeQ6RYWFePD/pqrLmzdvwksvvaCyxb7z9lt8UoiIiIjifejmnj17VGmFJUuWoF27dtDpdCrYGDFiBD777DN06tQp7G1t3LgRr732mrr94MGDcd1119XZUygFmT/55BPMnz8fSUlJuOKKK3DIIYegpUtJSUFZaWm1ZaWlpexZpfihuqQ9B3wWLvwDY8eehCNHj8aTTz6OK3BVrFtHRERE1GpE1KN38803q+BCSilI8XSZsyeXJePeLbfcEvZ2VqxYgcMPP1wFjsOHD1cZPI855hg1tycUuW7cuHG49957cfDBB6vTpEmT8NdfBxKWNEd7ixx4+Iu96jyUwYMPwXfff6uGxkmwu3z5MqxftxZ9+x7cpG2l1slVvhPlC28AKneHXCctLQ0VFRXqtGrlXxg0eDASExJRUV7RpG0lIiIiau0i6tH7/vvvsWrVKnTrdmAIlxRKf+ONN1SvXLjuueceHH300Xj77bfV/1Ke4aCDDsLrr7+Oa6+9NuhtnnzySSxevBhr165Fhw4d1LKrrroK5eXlaM4cLk3V0JPzUIYNG46tW7bg7rvugM1mQ5s2bXHzzbchMzOzSdtKrZTbDnfFTkALfTDCZDJh3KmnYdI/rkLv3n3Qr19/FBYUNGkziYiIiCjCQE96k+QHXSAplCzXhUMClXnz5uG///2vb5nUhjv++ONVfb5Qgd6rr76KSy+91BfkCWlLRkZGq3g+zzt/ojpVVlYgMTEp1s0hquGSSy7DBRdc5Mu2mZGZiSeenM49RURERBTvgd6YMWMwefJkFXR5C3fn5eXhhhtuUIFaOHbs2KEy8UkPnj/pJVywYEHQ2xQVFWH79u1qmOcLL7yAZcuWqfmAkgF0wIABtQaVcvIqKSlBcyXDZJcvX4qBAweje/fu2LVrlxpGy149iiV5f0nNx0GDBteo8yjMFguGDh0Wk7YRERERtUYRBXrPPfccTjvtNHTp0gU9e/ZUy7Zs2YJevXph9uzZYW3DarX6Eoz4k/+91wUqKytT5//85z9xwgknqGGfixYtwmGHHabuV4q2B/Poo4/ioYceqrHc6XQgd/+esNobyW3CXVfWKyjKgd1mRUF+DhJdwQvP79mzF88//zwsFgucDjuSk8x4/7230bdvXxx55BFoCSJ5Tlpi2xrjvhq8zYoc6G1WdYAmcDsydPqXn39Em6x0zJv3Q42bpqQk46CunerdhsZaP55fZ00pnvdDc3+/RWu7DdlGPH3HxevrrCnF835o6rbF5XdcA7cTL++3SNvSEjnjeD80tG3Z7cJLfKnTIqyJID/2JLhas2aNyropPWrjx4+HwRA8SAnWoye9dzJM89RTT/Utv+aaa1SSlqVLa/YKSIZJSfZw+umnY9asWb7l5513nurp+vXXX8Pu0evatSs2b1qHnr3ql8hEnpRwd26468p6yRkdsGmfDb07WJBkDp4j55133lJZRnfv3o2BAwfh+ONPwPPPP6t6UcaMCa8nNd7VZ/+25LY1xn01dJuaoxyuotUodLRBdqfeTdKGxlo/nl9nTSme90Nzf79Fa7sN2UYkt22s77h4fZ01pXjeD03dtnj8jmvoduLl/RZpW1qi3DjeD03VNmPENzQaMWHCBHWKhARaMq9u9erV1QI9SfISKqFLamqqSvrSr1+/asvlfxnGGYr0gMkpnklwd8hBUow6NLvNjuzs7Bp1yyT4JWpsOlMyjNkjgTCOQEkmXanxaDaZMGDgIKSnp/MJIiIiImoOBdMbSnoBL7roIsycOdM3Z0565CSj5iWXXOJb75VXXsGdd97p+//yyy/HN998g8rKSvW/DPOUXsFRo0ahOSuudOG7VSXqPJRDDj0USxYvhtvlUsXppU6ZFKWWoZtEjc1tK4R928eAo6jW9b7/7luVGXbRwj/www/f4+abbsC6dWv5BBERERE1oYh79KLhkUceUT1xMuxTTr///jvuuusuNf/OSwK/hQsXqrIK4u6771aF2iW4kRp8Uj9PerlmzJiB5qy4woUvlxejf6cEpCcaQpZX2LF9Oz7//DP8+usvqrzCrbdNQWoqe/So8Wn2Ati2vAX0vDf0OpqGt99+E49Me9yXaGnu3B/wwQfv4cEH/8WniYiIiKg1BHoydFOCOwnkcnJy8NJLL6F37+pzf6TMwrnnnuv7PyEhQc0N/PPPP1UGTvkxKQGf9BC2Bmefc646sbwCxSMplC78s+nK0Oqvvvwihq0iIiIian1iGugJvV6PI488MuT1UkohGAnu5NTasLwCxTNPqY8szJ71FU4ZdyocDju++PwzDi8mIiIiitdAb+7cuWFv9MQTT4y0PVSLnTt34N//ekj1akqxdKmj9/nnn/oycBLFg9tvvwMvvvg83nrrDdXTfuhhh+PGG2+OdbOIiIiIWpWwA72xY8eGvdEIKza0aolmPQ7vlqjOQ1mw4CecfMo4VV7BX2sZtkqxpTNK1s3RgCGp1vW69+iBJ5+aoUqaSLkVydBLRERERE0r7F9gDN4aV3aqEf84rm2t67C8AsWSPrEDEgffg7Iwyits2rgRefl50Nxu9b/ZYsHQocOaoJVEREREJHioPU44XRpKrS6kJhhgNOhCllf49ps5qp4gyytQU9PcTmj2IsDtrHW9J554TGWHlZ49fVVvs7xmGegRERERNR3O0YsTe4oceGx2Du45vT0OamMOug7LK1Asucu3o2LJrVXlFQ5k1QzMurniz+X43+tvqbmkRERERBQbnKPXzLC8AsWzxMREJCenqLl5RERERBQ7nKPXzOzatQvz589FUWEhOnfugpNOPkUNiyOKBw6HQw3RfObp6Tj6mGN9iVjMZjMOOeTQWDePiIiIqNUIneKR4s7mzZtw/313q96SQYMGY8+e3bj7rjt8RaqJYs1ms6KgIF8FfPPnzcX3332rTpIxloiIiIiaDpOxNCM//7wAp556OiZecKH6//gTTsQD/7wPa1avwvARI2PdPCKkpqbhvvsf4J4gIiIiao49etKDdPfdd6Nnz55qaJbUcfM/Uf11zTLh2Uu6qPNQMtIzYDBWn/sk/2dkZnKXU6PTp/REynGfAYnduLeJiIiIWmKP3v33348FCxbgiSeewHnnnYdZs2Zh8eLFePrpp3HHHXdEv5WtgATIpjryV/TvPwDPPjsDBr0BHTp0wJo1a1BRXoHi4mIsXboEPbr3QJu2tdfiI2rIaxQ6k1zgTiQiIiJqiT16n376Kd566y2ce+656v/TTjsNDz/8MN58801899130W5jq5BT7MDT3+5X56Fs2rQRXbsehLVr/8aPP87H/v05yMjI8M2D2r1nd5O2mVoXd8VuVCy/F7DujXVTiIiIiKgxevQk82P//v3V5ZSUFBQVFSEzMxMnn3wyLr744kg22erZnBo25tjUeSinjz9DnYhiQXNZ4SpaDWSN5xNARERE1BJ79DRN89XJ6tOnD7799lt1+Y8//kB6enp0W0g++fn5KC8vU5dluObXX8/Ctq1buYeIiIiIiKjhgV779u19l2VO3hVXXIFBgwapIZyTJ0+OZJMUhvfeeweLFi1Sl59//lk1R+/hh/8PJSUl3H9ERERERNSwoZv79u3zXb7oootUr54EIP369cOJJ54YySYpDCoFhqahsrIC69etxetvvI0ZM57C2r/XYOSoI7gPiYiIiIgoenX0hg8frk4UuaxkAy4+IlOdh5KZmYWcnBysXrUKffr2VaUtEhMTUc6C6dQE9JZsJPS7CVa04f4mIiIiaolDNyXYeOGFF2osl2VyHdVfSoIBo/umqPNQpED6Tz/9iBdeeB5nnDGBu5malM6cBlOnkwBjKvc8ERERUUvs0bv55ptxzjnn1Fjerl073HrrrXj//fej0bZWpczqwl87KnHoQYkhg72OHTviv6+8BpfL5UuGc+WVV8NkCl1knShaNHsJnHkLAbBgOhEREVGL7NGTLJtSSiGQLGMdvcgUlLvw7h+F6rwu3iDPW97CYrFEeK9E4XPbcmFd9zxgz+duIyIiImqJgZ4EFjt37qyxfMeOHdDrI9okERERERERRUlEUdnpp5+OG264Abt3765WRP26665T1xEREREREVEzC/SeeOIJVbutR48eqqTCwQcfjJ49e6K8vBxPPfVU9FtJREREREREjZuMpW3btli6dClmzZqF5cuXQ6fT4fDDD8f48eNVyn+qP4tRhz7tLeqcKB7pDAkwZAySOguxbgoRERER1SHiqEwCurPOOkudqOHap5tw2yntuCspbumTOiNpyKMo378n1k0hIiIiojpEnDll3759ePHFFzFlyhTfsgULFsDpdEa6yVZN0zQ4XJo6J4pH8trU3A65EOumEBEREVFjBHoybLN///6YOXMmZsyY4Vv+8ccfq2VUfzsLHLjlnV3qnCgeucu2oOyns4HK7bFuChERERE1RqB3xx13YOrUqWp+nr9Jkybhueeei2STREREREREFMs5esuWLVOJWIQkYvHq1asXNm7cGK22ERERERERUVP16JnNZhQXF9dYvmbNGpWRk4iIiIiIiJpZoCdlFB544AGVeMXboyc9eVIwfcKECdFuIxERERERETV2oCdF0VeuXIns7Gy43W706dNHFU7X6/WYNm1aJJts9TplmPDIuR3VOVE80id3Q/KRrwMJXWLdFCIiIiJqrILpixcvxpw5c1QGTgn2hgwZwoLpDWA06JCZzGLzFL90eiN0CW0BPevoEREREcW7iCKLMWPG4Mcff1SBnZyo4XJLnfhiWREmDM1AdioDPoo/7sp9sG16A8g4BUCnWDeHiIiIiKI9dFOybpaVlUVyUwqh0u7Gn9sr1TlRPNKc5XDm/ga4KmLdFCIiIiJqjEBv3LhxeP/99yO5KRERERERETWyiMYIpqam4tprr8Wnn36KAQMGqHIL/h577LFotY+IiIiIiIiaItDbtm0bjj/+eFVeQbJvEhERERERUfyIKNCbO3du9FvSyqUnGXDmkHR1ThSPdOYsWHpeBqspI9ZNISIiIqLGmKNH0ZeeaMDJg9PUOVE80lsyYe5+HsBAj4iIiCjuMdCLExV2N1buqFTnRPFIc0jWzUWAk1k3iYiIiOIdA704kVfqxMs/5qlzonjktu5D5ap/A/b9sW4KEREREdWBgR4REREREVELE5VA74033ojGZoiIiIiIiCheAr0rr7wyGpshIiIiIiKiKODQzThhMujQId2ozonikt4MfVJXQGeKdUuIiIiIqDHq6FH0dcwwYeqEjty1FLcMyV2RPOolVOzfE+umEBEREVFT9Oh9/vnn0dgMERERERERxUugN2HChGhsplXbWWDH7e/tUudE8chVugVlC84HKrbHuilEREREVAfO0YsTmgZYHZo6J4pPGjRXpTonIiIiovjGQI+IiIiIiKiFYaBHRERERETUwjDQIyIiIiIiamGiHuhdc8010d5kqyA19O45vb06J4pH+qQuSBr+DJDAMiBERERErS7QmzlzZrQ32SqYjXoc1Maszonikc5ggSG1F6C3xLopRERERFSHenUfvfPOO/VZneqhoMyJ71eX4qRBqchKYa8exR+3NRf27Z8AyUcB6BTr5hARERFRLeoVUVx66aX1WZ3qoczmxs/ry3Bkn2RkpXDXUfzRHCVw7J4D9Dw01k0hIiIiojrUa5zgYYcdhqVLl0LTtJAnIiIiIiIiakaB3vXXX4+XX3658VpDRERERERETTt08+KLL4bdbq91nR9++KGhbSIiIiIiIqKm6tFLTk7GjTfeWOs6J554YkPa02qlJuhx/IAUdU4Uj3SmdJi6ngkY02LdFCIiIiKqQ9hRxa5du9AY65JHZrIR5w7PVOdE8Uif0BYJfa4BzFmxbgoRERERRSvQGzp0KG677TasW7cu5DqrV6/GLbfcgiFDhoS7Wfz4448455xzcNRRR6k5gLt37w77tm+99RaGDRuGGTNmoLmzOdzYst+mzonikeaywlW8DnBZY90UIiIiIopWoLdixQqUlJTg0EMPRd++fXHhhRfi1ltvVYHdxIkT0bNnTxUMlpWVYeXKlWFtc968eTjppJMwePBgPPDAA9ixYwdGjx6t7qcua9euxT//+U/k5OSo2zV3OSVOPPXNfnVOFI/cFbtRsexOwLYv1k0hIiIiomgFeh07dsTMmTOxfft2FeDp9XosX75cBYBGoxF33nmnCrhknQ4dOoS1TQnuzjvvPDz44IM4+eST8cknn6CoqAj//e9/a72d1WpVweXTTz+NNm3ahPsQiIiIiIiIWoV6TwiTIO6GG25Qp4YoLy/HwoULMXnyZN+yxMREnHDCCaqnTwLHUGQI6fDhw9WQz3/9618NagcREREREVFLE7PMH5KwRQqsd+rUqdpy+V8CvVA+++wzVcJBehLDZbPZ1MkrnKGhREREREREzVXMAj2Hw6HOLRZLteXSq+e9LpAMG73uuuswa9YspKSkhH1fjz76KB566KEay51OB3L376lXu+tzm3DXlfWKivfDBBuKCvYj0dV6SyxE8py0xLY1xn01eJsVedC5THC63BFvp75taKz14/l11pTieT809/dbtLbbkG3E03dcvL7OmlI874embltcfsc1cDvx8n6LtC0tkTOO90ND25bdrnpHWdwFet65dfn5+dWW5+XlhZx3JwFeRUVFteGeGzZswN69e/Hrr79i0aJFMBgMNW5377334vbbb6/Wo9e1a1cYjaawd5SXPCnh3ibcdWW9Q/p0wrN96tWUFqk++7clt60x7qvh2+wEdP+sQdup720ba/14fp01pXjeD839/Rat7Tbl+62+t+H7rfGfj5batvj8jmvYduLl/RZpW1qi3DjeD03VtpgFepLcRU5LlizB+PHjfcslWDv22GOD3kYSsIwaNarasosvvlhl+5RALliQ5+01DOw5JCIiIiIiaqliOkbw6quvxmuvvYadO3eq/z/88ENVNkGWe/373//GBRdcoC5nZ2erunn+Jxnq2a5dO3W5OdtT5MD/fbZXnRPFI1fZDpT/MQmoDL/WJRERERE1s0Dvgw8+wNixY9GrV69qQdn+/fvD3oaUVzjuuOPQp08fVYdPAryXX35Z9dB5bdu2TRVib+mcLg25pU51ThSXNAfclXvVORERERHFt4iGbr766qu4//77ceONN2Lu3Lm+5VlZWZg2bRqeeeaZsLZjNpvx3nvvqaLnEiBK0JiUlFQjGJQi7KHI7VNTUyN5GERERERERC1SRD16M2bMUMXNp06dWm35qaeeqoZf1lf79u0xePDgGkGe6NatGwYOHBjytgMGDFCJVYiIiIiIiKgBgd7WrVtVwXKh0+l8yzMzM1FQUBDJJomIiIiIiCiWgV7nzp2xZs2aGoHe119/jd69e0erba1KdqoRN57YVp0TxSN9YkckHvoQYGkf66YQERERUWMEetdffz2uvPJKNT9PAj0J+qZPn66W+9e4o/AlmvUY0DlRnRPFI50xCcY2QwBDYqybQkRERER1iKj7aMqUKSgtLcWZZ54Jl8uFQYMGqTIHd911F2644YZINtnqFVW48OuGMhzVNwUZScHrARLFkttWAMfubwHzoZ7i6UREREQUtyLqPlq/fj0eeugh5OfnY/ny5Vi6dCny8vLw4IMPYt26ddFvZStQUunCnL9K1DlRPNLshbBvex9wFse6KURERETUGD16/fv3h6ZpSEhIwOGHHx70OiIiIiIiIoqNqE4Ik3p3ycnJ0dwkERERERERNWaP3q233hr0snC73VixYgWGDBlS3zYQERERERFRrAK9TZs2Bb0sTCYThg4diltuuSV6rWtFksx6jOiZpM6J4pHOmAJj++MAA3vtiYiIiFpUoDd79mx1fsUVV+CNN95orDa1Sm1Tjbji6DaxbgZRSPrE9kgcOAVl+/dwLxERERHFuYi6jxjkRZ/dqSG3xKHOieKR5rLDXbEXcNtj3RQiIiIiaoysm8LpdOLvv//Gjh071GV/EyZMiHSzrda+Ygcem52De05vj4PamGPdHKIa3BU7UbHkVqDnvQC6cw8RERERtbRAb+PGjSqYk3p6UjBd5uc5HA51nWTdlOybRERERERE1IyGbkrGzWOPPRbl5eXqf6vVimXLluGwww7DI488Eu02EhERERERUWMHegsXLsSDDz4Ii8Wi/pdePSmr8Oabb+K5556LZJNEREREREQUy0CvoKAA7dq1U5fbtm2Lffv2qcu9evXC7t27o9U2IiIiIiIiikCDi7aNGDECjz32mKqrJ8M2e/fu3dBNtkqSgOWly7syEQvFLUNqL6QePwtIYiIWIiIiohaZjOXaa6/1XZYg7/TTT8dLL72EzMxMfPTRR9FsHxERERERETVFj97LL7/suzx48GBs27YNO3fuRE5ODk488cRINtnq5RQ78OScHHVOFI/cFbtRsfQOwLo31k0hIiIiosYI9MaMGVPtf51Ohy5duqgyCxQZm1PD1ly7OieKR5rLClfJesBti3VTiIiIiKgxAj0ppcBaeURERERERC0o0Bs3bhzef//96LeGiIiIiIiIYpOMJTU1VSVk+fTTTzFgwACYzeZq10uCFiIiIiIiImpGgZ4kXzn++OPhdDqxcuXK6LeqFWqTYsDlR2Wpc6J4pE9oh4QBt8PqbhvrphARERFRYwR6c+fOjeRmVItkiwEjeyVzH1Hc0plSYeowBti/J9ZNISIiIqLGLphO0VFqdWHBulJ1ThSP3PZi2Hd9DThKYt0UIiIiIqoDA704UVjuwoeLitQ5UTzSbHmwbXgZcBTEuilEREREVAcGekRERERERC0MAz0iIiIiIqIWhoEeERERERFRCxNR1k0pq/Dmm2/it99+Q0FBzfk6X3zxRTTa1qokmHTo3ylBnRPFI50hEYasw6XOQqybQkRERESNEejdcsstePfdd3HaaaehS5cukWyCArRLM+GmsdncLxS39EmdkHTYwyhneQUiIiKilhnoffTRR6qW3rBhw6LfolbK7dZgc2qwGHXQ69mrR/FH09yAywrIORERERG1vDl6er0e/fr1i35rWrFdhQ5MeX+3OieKR+6yrSj7eSJQuSPWTSEiIiKixgj0xo4di88++yySmxIREREREVE8Dt1MSEjAVVddhS+//BK9e/eGTld9qOFjjz0WrfYRERERERFRUwR627Ztw3HHHYfi4mIsW7Yskk0QERERERFRPAV6koiFiIiIiIiIWlCgR9HXKcOExyd2QqKJNewpPumTuyH5qHdgLSiOdVOIiIiIqLECva1bt+LZZ5/F2rVroWkaBgwYoOrr9ejRI9JNtmpGgw6pBkOsm0EUkk5vhM6cDujLuZeIiIiI4lxE3Ufz589H//79sWDBAhXY9erVS12WZXId1V9uiQP/mZerzonikbtiLypX/guw5sS6KURERETUGD16d999N+655x48+OCD1ZbL/3LdkiVLItlsq1bp0LBqlxWnHabFuilEQWmuCjjzFgNpJ3APEREREbXEHr2VK1fi1ltvrbFchm7KdURERERERNTMAr3MzExs2LChxvL169cjKysrGu0iIiIiIiKipgz0LrnkEpx//vl45513VMAnp7ffflstk+uIiIiIiIiomc3RmzZtGvR6Pa655hrYbDa1zGKx4Oabb8a///3vaLexVchIMuCcYRnqnCge6SxtYOl9NayGzFg3hYiIiIgaI9Azm8144okn8PDDD2Pz5s3Q6XTo2bMnEhISItkcAUhLNOCEgancFxS39OYMmA+aAOzfE+umEBEREVFjFkyXwG7gwIEN2QRVKbe5sW6vFf06JiDZwqLpFH80RxmcBSsAV/tYN4WIiIiIohXo3Xjjjer8hRde8F0ORdah+skvc2Lmgnzcc3p7JFvM3H0Ud9zWHFjXPA70vBdAn1g3h4iIiIiiEejt2rUr6GUiIiIiIiJqpoHeF1984bv82GOPoV+/fkHXW7duXXRaRkRERERERBGJaDJY//79I7qOiIiIiIiIGl9Us36UlZUhOTk5mptsNUwGHbpmmdQ5UVzSm6FP6QnoTLFuCRERERFFM+vmrbfeGvSycLvdWLFiBYYMGVKfTVKVjhkm3Du+A/cHxS1Dclckj3gWFSyvQERERNSyAr1NmzYFvSxMJhOGDh2KW265JXqtIyIiIiIiosYN9GbPnq3Or7jiCrzxxhv1vzcKaWeBHU9+nYM7T2uPrlksr0Dxx1W6BRVLpwA97gLQKdbNISIiIqJaRDRHj0Fe9Gka4HR7zonikwZoTs85EREREbW8QO/XX3/F9ddfX2O5LJPriIiIiIiIqJkFerfffjuuvvrqGsuvuuoq3HHHHdFoFxERERERETVloLdy5UocfPDBNZbLMrmOiIiIiIiImlmg17VrV/z00081lv/444/o1IlJGiLRId2Ef57ZQZ0TxSN9UlckjXgRSOB7nIiIiKhFBnrXXXedGrr5yiuvYN26dVi7di3++9//4pprrlHXUf2ZjTp0yjCpc6J4pDOYYUg5SBVOJyIiIqIWOkfvH//4hyqa3r9/fwwYMAC33XYbJk2ahClTptRrW2+99Zaqv9elSxeMGzeuzqGfq1atwpVXXqmGiQ4cOBDXXnst9uzZg+Yuv8yJd34rUOdE8chduR/Wtc8BtrxYN4WIiIiIGiPQ0+l0eOSRR5CXl4elS5di2bJl6rIsk+vC9dFHH6lewJtuugnz5s1Twd6YMWOQk5MTdH2Xy4VLL70Uxx13HGbNmoV3330XGzduxAknnIDKyko0Z+U2N37fVK7OieKR5iyFY+8PgKss1k0hIiIiomgWTA+UlJSkeuMiJYGh9M5JAXbx8ssv46uvvsJLL72Ehx56qMb6BoMBf/75Z7Vg8tVXX0Xv3r2xcOFCFSQSERERERG1dhH16IkPPvgAY8eORa9evXzL/v3vf2P//v1h3b64uFgN0zzxxBOrBXLHH388fvnll5C3C+wx9PbkJSQkRPAoiIiIiIiIWp6IAj3pRbv55ptx9NFHY8uWLb7lWVlZmDZtWljb8M6ra9++fbXl8v/evXvD2oamabj77rvRr18/DBs2LOR6NpsNJSUl1U5EREREREQtlU6TaKmeJAGLZNk85phjVA+bdxPbtm3DEUccEVag9vfff6tkKr/++itGjx7tWy4F12X+3fr16+vchiSAef311/Hzzz/jkEMOCbnegw8+GHQo6Pp1q5CZmYX6cDodMBpNUV1X1it3GPDHFieO6GlEemLEHa3NXn32b0tuW2PcV4O3aS+ALncuHJnHwZjUrkna0Fjrx/PrrCnF835o7u+3aG23IduI5LaN9R0Xr6+zphTP+6Gp2xaX33EN3E68vN8ibUtL5Izj/dDQtmW369R4c/S2bt2K4cOH1xhKmZmZiYKCgvAamJ2tznNzc6stl/+919Xmnnvuwf/+9z/88MMPtQZ54t5771WZQr2kR09qAcoODndH+dq3f0/Ytwl3XVmvd6dO6N2tXk1pkeqzf1ty2xrjvhq+zU5Al0EN2k59b9tY68fz66wpxfN+aO7vt2httynfb/W9Dd9vjf98tNS2xed3XMO2Ey/vt0jb0hLlxvF+aKq2RdR11LlzZ6xZs6ZGoPf111+rxCjhkGCuR48eqkfPn/TOjRw5stbb3nffffjPf/6D77//HiNGjKjzviwWC9LS0qqd4o3V4caGfVZ1ThSPNGclnIWrAFfzznBLRERE1BpEFOhdf/31Klvm3LlzVaAnQd/06dPV8smTJ4e9HZnn99prr2HRokVwOp144okn1Nw9qY3nPzxT5gJ6PfDAA3jxxRdVkFdXQNic7C9x4pnvctU5UTxyV+5B5Z/3Abbg5U+IiIiIKH5ENHRTiqKXlpbizDPPVLXtBg0ahMTERNx111244YYbwt7OLbfcorJ0SuZNSZgiPYWfffYZ+vbt61tH7qewsFBdzs/PV5k9pYfutNNOq7atZ555BpdcckkkD4eIiIiIiKhFiSjQk148SW4ic9/Wrl0Lt9utErRIXb36bkeydErwVl5ejtTU1BrrSAAnvX3erJ6Bc/q8gt2WiIiIiIioNWpQwXSpXXf44Yc3uBF6vT5koJaSklItMGzbtm2D74+IiIiIiKglCzvQu/HGG9X5Cy+84LscivTsSQ/fhRdeyELmYTLogYwkgzoniks6A3SWNvJqjXVLiIiIiChagd6uXbuCXg7G4XDgvffew7fffosPP/ww3Lto1TpnmjHtvPhMAUskDCndkTL6DVTu38MdQkRERNRSAr0vvvgi6OVQ9u3bhz59+kTeMiIiIiIiIopIow0U7NChA2bNmtVYm29xdhfacd/He9Q5UTxylW1D2W9XABU7Y90UIiIiImqsQO+DDz7A2LFj0atXL98yyZ4p5RK8jjvuuEg33+q43EBRhUudE8UlzQXNli+v1li3hIiIiIgaI9B79dVXVbFzKWS+ZcsW33IpfyDlEoiIiIiIiKiZBXozZszAJ598gqlTp1ZbfuqppzL5ChERERERUXMM9LZu3Yrhw4f7att5ZWZmoqCgIHqtIyIiIiIioqYJ9Dp37ow1a9bUCPS+/vpr9O7dO5JNtnrt0oy49eRsdU4Uj/SJnZB4+DTA0j7WTSEiIiKixgj0rr/+elx55ZWYO3euCvQk6Js+fbpaPnny5Eg22eolmPTo2yFBnRPFI50xEcbMwYAhMdZNISIiIqI6RNR9NGXKFJSWluLMM8+Ey+XCoEGDkJiYiLvuugs33HBDJJts9YrKnfhpXRmO65eCjGT26lH8cVvz4dg9G7AMBdAp1s0hIiIiolpEFFEUFxfjoYcewr333ou1a9fC7Xajf//+SEpKimRzBKDE6sb3q0sxpHsSMpK5Syj+aI4i2Ld/AvTsE+umEBEREVFjBHpSRkGCu4SEBBx++OGRbIKIiIiIiIgaSUQTwjp06IC9e/dGvzVEREREREQUm0DvpptuUvP0SkpKGt4CIiIiIiIiiv3Qzddffx0bN25URdO7dOkCs9lc7fp169ZFq32tRrJFjyN7J6tzonikM6bC1HEsrIaUWDeFiIiIiBoj0Lv55psjuRnVok2KEZeMzuI+orilT2yHhP43o3T/nlg3hYiIiIgaI9C78cYbI7kZ1cLu1JBX5kTbFCPMxgNF6Iniheayw125D3C7Y90UIiIiIqpDVMYJTpgwIRqbadX2FTvw7y/3qXOieOSu2ImKxZMBK3v0iIiIiFpFoPfll19GYzNEREREREQUBcz8QURERERE1MIw0CMiIiIiImphohLoaZoWjc20ajodYNR7zonik05qLHjOiYiIiKjlBnplZWVYtWpV9FrTinXNMuO5S7uqc6J4ZEjtidQxnwNJ3WLdFCIiIiJqjECvpKQEF1xwAdLS0nDIIYf4lp977rlYsmRJJJskIiIiIiKiWAZ699xzD3Jzc7FixYpqy6+++mo8/PDD0Wpbq7K3yIFHZ+1T50TxyFW+E+WLbwEqd8e6KURERETUGAXTpZzCb7/9hu7du1dbPmrUKJx33nmRbLLVc7g07CxwqHOiuOS2w122BWjHgxFERERELbJHLz8/H23btlWXdX7ZQyoqKqr9T0RERERERM0k0BsyZAhmzZqlLvsHdtOnT8fIkSOj1zoiIiIiIiJqmqGbjzzyCM444ww1fFNKK8j/3333HRYtWoQff/wxkk0SERERERFRLAO9MWPGYN68eXj88cfRtWtXvPzyy6qX75dffsGIESPQnLlcLjgcoecgORxOWK3WsLYV7rqyXkqCE1cflYIUk9zGjdaqPvu3JbetMe6rodvUkAH0vQcOd1bE26lvG7zrm0wmGAyGiO6TiIiIqDWKKNATEtB9+umnaEmkLuCuXbtqLQAvgWBxSXlY2wt3XVnPUFKOJAA5rTyhYX32b0tuW2PcV3S2mQWXy4ni8q1N0gbv+jJEvEuXLkhJSYnofomIiIham/9v7zzAoyjeMP6l00LoHUSl9yJNQAFFuqCiiCDwBxuiUm2IYkNFQQURKwhSVBBEka7YkaYgCChdem8BEtL2/7yDc24ut3d7l7vkcnl/z5Mnudvd2dkp2Xnn++Ybn4TesWPHZM6cOfLQQw+pz++//768+uqrcvXVV8u0adOkdOnSktPAgBIiL1++fFK8eHHLoDLJyUkSFWVvU3O75+K88IgoSUhKk7zR4RIRnnsD2nhTvqGct0DcK7NpGmkpYqRckFSJkajoPFmSB5wfGRmltnNB/6xcuTIte4QQQgghgRJ6I0aMkA4dOqi/Dx8+LIMHD5ZRo0bJ6tWrZdiwYfLJJ59ITgPumrDkQeTlzZvX8ryIiHDbA1W75+K8tLBIuXAxRWILREpMpE8xckICb8o3lPMWiHtlNk0jNVHSUs5JWkxpiYrJkyV50OejX+7du1f1U7pwEkIIIYQESOgtXbpUJk6c6Pj7hhtukKeeekoOHTok9erVk5wMt4cgJPhgvySEEEII8Q6fTEdJSUly6dIl9fc333wjN954o/o7f/786hgJHX7++We59957bZ+fkJAgXbt2lZSUFAlVvC0TYg+4fX/88ccsLkIIIYSQ7BJ6LVu2lIEDByqr3hdffCFdunRR38N1s1mzZpLbOHAqSd7/7oQ8v+Cw+o3PoRSgZv/+/bbPnzx5sgrUExkZKVu2bJFWrVqlO75jxw7VfvxFWlqavPfee3L99dere33wwQcexQTOQx61VRqcOHFCqlWrlu4HefVHmZg5c+aMI33nCLXoS/pYrVq11e++fft6zLtd0D/79+8vwcqtt94qL774otuot4QQQgghJICum2+//bYMGjRIPvzwQ3njjTdUEBY9yH/66acllDl5PkWW/hkvnevFSbHYSFmy6ZyM+fKIJKakIf68SJjIxz+fkqe6lpL65SPTnesOxF/JGx2mfudksNXGihUrHNa9nTt3pjsOS7CVgPKFqVOnysaNG+WFF16Qs2fPyn333SeFChWS22+/PcO5S5YskZEjR6p2W7BgQRVMqEyZMtK9e3dlgTx58qTaIkRzxRVXiL+JjY2VBQsWyL59++SOO+5Id6xNmzbqmA5CMnjwELVtiae82+XChQvqvr4TLmGR+X2dH/IIngvPu3DhQiX6CCGEEEKI7/g0YqtQoYIajG3atEnuv/9+x/dffvmlXHvttRIKIDALomA6/xw+kywf/XhSDp9Jkp1HE+VFiLzkNMkXHSb584Sr3/iM7zftT3Cc6yot8zYOURHhUqJglPpttjL169dP6tatq6w7zz77rMN1cMCAAepz06ZN5X//+5+cOnXKcR3caTt16iTXXHONEgSnT5+2dQwCrXPnzlKzZk11P1ifdFmMGTNG3atXr15KELliz549as+zihUrSlYBi9c777wj1113nbIsQ/hA+LkCez8i/x07dpQWLVrIkCFDVMRYDYJ8mC160dHWQUPslokz+h5XXXVVhmNxcXGOexcrVkzWr18vvXv3tpV3M1btBlb4X3/9VX2HtNCO7rnnHiWSmzRpIr///rt6LkzkwHKIvvzaa6852umMWZ9KjQZtpG79a1QbwvWa119/XVnj0BYbNmyo9tiEyIcbL9KeMWOG41zcx9zOFi9e7DgGyyzW/RJCCCGEkGzaRy/USUw2pMO4XRm+T0lNk7MJafLAtAOSlJImF5MMGPHkUsp/og3j4oTkVHny8+OSlGLIwOkHJNKFqW7JiKuVFe/yNYakGZctezrwxCuvvCIxMTFqkAzRUbhwYYfr4PTp09UAfdKkScqNDwP2+fPnK3GAwDgvvfSSFC1aVK15evjhh2XmzJluj61Zs0buvPNOefPNN5VVBYIE1iewcuVKFWUVFlsM4HH9+PHjMzzP9u3bM4i8o0ePqsG8xt0aTpwLweYq8AbEw6xZszJ8j420NRcvXpTly5dbCiBY6LAtCKyKKM/vv/9edu36r44hbBs1aqTWmt52221KCFsFAbFbJr6COkf6qCc7eTdj1W4QGXfKlKnKCoptRLZu3arcQbE1ypQpU5RlHhb6DRs2qO/gFgsLPa5H+4I4a3RNA0lKTpGdO3cpEbpt2zYVpRZbrqA9wY0W4hvWSkwGQWSizu+66y71PHFxBZXVFWsc4cIbHh4uZcuWdeT9yiuvlE8//dRv5UgIIYQQkluxLfSwWbFdsN9VKAMhdzYhVf1Wn//9zpVYtOuJmZRqyJEzKVKqELZXuHwVBti//fabbN68Wbn1lSxZ0nF+jRo15Mknn1R/w6IFQQD3Q4i2f/75R7nWAnyn1zy5OzZ79mwl+u6+++4MeYP1DxYkAAsuhIQrkJZZeAHkS7sjArhtWgUywbmffz7XZfh9CBN3QADBXRPiA1YhV+C+sB6hHCFO2rZt63h+hO+HJRCCG3nENiEQeXqvSF/LxFc++mhaunV47vLujFW7gaBCOWrhDaEHqx+eVQOxdu7cOVm3bp36jL/hzgqhdyH+lLz68rPyx5ZdcvFighLmEJu1atVS58KKqNfrtm7dWtWDDtSE9oqJgEaNrlH5w6QD8oPzIKw1EKY60BMhhBBCCMkCoQe3rNxEnqgwZXFz5q+D8TJ49lEZd2cZWbXjgsxYdVq5a5otPxALsPS1r11Aftp+Ud7uU04qlczj8h7ueOKJJ6R8+fLKJRaDcQi0Z555xrGeSYPBOywjGCDHx8erATd+NAiMAtwdg5VQr7V05VJoHohbWeUwcMfg3wzSN1v03EXjhPtj9+63e2XR02sBu3XrplwN3a0RzZMnj1rrpt0s586d65iU0C6VoHr16mqDbpS7ldCzWya+8Msvv6hnwrYldvLuTbtxBgLXDNrIuHHjlDDT6LY28MFHpNKVJWXypAlSqEgJ5ZYJV13n8wAEv/mzuYxgmcRaw7feeku5esLy2L59e8e+nMg7IYQQQgjJIqFnFge5AYgN7VZpJiYqXMLDwqRQ/kjp3riwfL7urCQkp0n+f8WeEnnJhuSNCpe2NfPLLzsS1DV5o71fDgnx0adPH/WzatUqNSjWA3ZYXGCxqV27trLUQaTBMoK1VbAuDR06NJ0FELg7BusL3Pawdku7C3pLnTp15MiRI0o0FihQwOvrfbHo4V5wKYTlykrMuLoPgpLAPdGVJQ4i6+uvv3a5ji4rgAhCnUO8e5t3d+0GdYKANe5AG5k3b55agwf3TzP79u+XR+7vIQ0bNpA16zbK7t27fXo+WPQeffRR9QOXV0Qb1UIP7Rruu4QQQgghJHNwjZ6XFMkfIf1aFpEi+SNVJE1E10TUzYsquMrlqJt5IsPV99VKRzrO9QVYkxA8BWulEGADa8E0cJdDqHxYQGAp0e6RcF1EkBwIv1KlSimLWrt27WTChAluj8Flc+3atSrQDixzEBkIx+8NEBg9evRQedFBRLxBW/9cCT13UT4RFAQCE+6nAC6cWL8I6yIELJ4ZFiWswUPwFFgVDx48qFxVdXROuCxCdECo41jz5s1l9OjR4gsIMrJs2TJLd2dsQYI1bbCe4Xnhvgj3W4DvILR+//23dNe4y7vddgMhDldMrPdDQBdXbroQkBCICAaDqJ6YvICVFBMAQwY/LLf0HCjFij4tpUqXlkqVKvlUPtgeAvmABRrPg7WlAGW/aNEiJU4JIYQQQkjmCDPMoR+9ANYkzMTDuuDsjocAD8EMBplwvdu18y+56uqq6ju4oCFqJIJBwE3OCoS9dxYiB08nyaKN5+TI2WQpFRclneoVlLKFo12ea5VmWlikaY3eZUsOBvQY+MMNDuJL5wtRCRE0Bb8h9OB+p10wNRB/um7gQodBu51jsGZhjziIhMqVK6tyQWROLVpwHG6NEISu2Lt3rxJ5EF+4Fs9gdgnV97YSCXbLTINooxBNZrC9AoQsng+RH6tWraoES2pqqlp/BxGLZzZbHfGMEIY4Vrp0aUcgGld5wzYFVmUCsfz8888ri6AVyJO5zyAf5rQOHTokFSqUT1cO7vLujFW7Qf7DwyNU/SIdWAfNz+FsKUU6+PeAskSZGqmJcubwNkmQQlKm3JVqvScsw0gfzw+hX6RIEXU92iUszNp9E3UO8RgVFSl79/6jngdWWuQP14HPP/9crQfExINzmdvtn8ePHZLiJf5rz5k9L9QJ5nLIyrwF6l7+SDczafhyrTfXsL8Fvj5CNW+BuJ+/0vQ1nWDpb77mJRQ5HsTlkFV588nUBIGHKHpYQ4QZeEQoRPAHDPJ1MIbcBETdfa2LZSqN6IgwKV8kSszL08zRCK2AKHGZXnS0pZhydwyBPqpUqeL4jMG6OVgGjluJPICom1hLh4E8BuTO6/7c3dsXICy0uHDGeX2geR2eK3dCHZ3SE+7KBGXnKWqku+dHWigzCBwz7vLujLt2g3TMkVHNz2EGQhICOR3hMVKobB0pkJKaYZ9B57V+zu1Slw+eCxMIroCVES6jhBBCCCEkm4Tec889Jx999JEKxw9LCWbiYaGAyxgj5vkGytEikn8GEJZeRzoMRgKx0XhOAZavUOVykBxY3y4LPX8DyyEhhBBCCMlGoff333/LzTfffDmByEjlbgZLBFzWzNH6iH2SU9Pk1IVUtQbQvGm6HYsSIVmBkZokaZeOi0RAzNp3ryWEEEIIIVmP96Eg/13PpqMgYr0Q9scC2NfLHG6d2AebpScmXd40nZDgJE0k5eK/O0cSQgghhJCQjroJyx7Ct2PNHtbuWW1WTQghhBBCCCEkiC162LhZg728EIoeASgQEQ97gBFCCCGEEEIIyQEWvVGjRsmLL76o/jaHY8dasbfeeiswuSOEEEIIIYQQEjiLnt5MGtSuXdv7OxG3RIaHqUAs+B2MoO5HjhyZYc/E3AL2jXzjjTckVxMWKWF5iouEXd73zi4LFixQ+yoSQgghhJAgFHqw4s2ZM0dttgwOHDhg+UO8JyI8TGLzRqjfmeWvv/5SLrX+5LPPPlObcCPKKvZLHDZsWLrj2Fx78ODBfr0nNjB/5pln5MEHH5QVK1ZYnnf27Fnp169fuh/kx046v//+uzz++OPyxBNPyIYNGyzvgfS+/fZbn58FZTZ69GgZMmSIfPXVV+mObdy4UeVh0KBBMnXqVBXUyBu2bt0qr732mgSasPBICY8u5LXQq1+/vnpuQgghhBAShEIPe+dhAK03Pi5fvrzlTyiTlnBUEraMV7/NJB/9Sc4uaaF+ezrXFalphpxPTFW//bGXW7169cSfTJw4Ufr376/+PnHihBL9Zk6fPq3EoL/AhusI7LNz50619rNnz56WYg/beyAQUKtWrRw/evsJd+ngmoEDB0rRokXVRuLYn3DVqlXib3bt2iXdu3dXG8hjwgSC7v3331fHduzYIdddd53aSL5mzZqqnJ988kmv0j906JAsW7ZMAo2Rlippyefwh9f7KmJT+x9//DFgeSOEEEIIIT6u0UNkTQySYbGrXLmybNu2TUIZ5aaampDh+7SL+yRhyziJLNFSwqJiHecmbHpBUk+sVb8jWn2hNpd2dW46IvL+uwm1SEqaISfPp0qpQmEOqx7cBZcvXy516tSRRYsWydVXX62ECaxqAHUwc+ZMtUl9t27dpEWLFur7M2fOKCtR+/btlXVoypQpylqF81q3bi19+/Z1WKlgQYLFq2nTpipyanh4Ru1//PhxtXciLDNZxeLFiyUqKkpmz57tEK+vv/66tG3b1uX52McRExHepNOkSRP59ddfHc986tQp+frrr+Xaa6+1zBdEoqu6cEexYsWU6yKEnrl+wJ49e1RaL7zwgvqMbUucRbTGqi7Hjx+vrHp4/ipVqkiXLl1Uu6lVq5Z6/oceekj12XXr1slXXy2UpKQk1V4gMAHORflA7FasWFHuueceKV26tDo2d+5cVX5oI3/8sUG6tW8mjVu0l3fee0OOHDmi7qnbBdrJe++9p54JAhvlgzIGN954o3z55ZfSrFlTj+VFCCGEEEKyeHsFDFQrVaokM2bMkGrVqklIk5ogZ76olPH7tGSRS6fk/MpOIuGXi89IvSSSdEbtL5Zy+Bs5M6+8hEXEiJGaLJJ0Ws6v7Ow410yhW3aKRF7ej9AVEGJjxoyRDh06qMimGORfuHBBuRli70IIEgymixcvrgbu06dPl06dOqkB+NKlS9V548aNUwN5iHRYjSAEAAT7TTfdJL169VKC4KOPPpItW7bIyy+/nCEfEJz6OrMFzyysIF6swLlDhw51eQzpYu2fq3uaBReeH1ZlK1AuDz/8sLLk3XrrrdK4cWOP6WAPSOd7QhRZsW7depk2bVqGuvBEXFycXLx4UZUXyuLcuXOqrrQAguCC1bFkyZKqXrS1zxmrusREwL59+5QlE2nodtOuXTu54YYb1P0XLlwozz//vNx9992SlpamxByCKOEcWBlxLb7H5AHy9McffygR+9tvv6m2gfPLlysrt/UaJE2bNJHrW7VW6aINwS0V5d67d29l8Uf5QDyXKFHCkfcaNWpkidWREEIIIYRkYh89DOhyN4YYl045/k6/gbShxJ0hsMrhJ3OumLCszJo1S/0N18NJkyYpcYFtLOBKiSA5AO6HkydPVkLPDKwssOZgwA/Lkeadd95RVjC4DuIHg/Z58+a5FHrnz593uEJqYmJilDjQQFysXr3a5TM4n2sGwsRq3V2BAgUcn2NjYy3FJPIO0QLLKp4F1jqUWefOnW2nA7EJYdKjRw+xolSpUi7rwg4QTSiDw4cPK6sXBBRE0bFjx5R4gxUObs8QebD+Va2aXli7q0s8L9LTwhsiH+0BVjptMcaazeLFi6k1idrKiPqG0LvqqqvUPSHuIF6Rx927dzuE5O23366Eo5GaKKtX/SCNm7aUp0aNUsdgMYQ1sVGjRip/t9xyi3oWTD6YQfvBGk9CCCGEEJJDNkwPWSLyXra4OZF4YoMk/NhdYq/7RCLiqkvysV/kwi99RcKiJCwiWozUJBEjWfI3ny6pEbGSuKqPFGj9pUQWruXyHp4wr3mESMFAHMBq16xZM8exqlWrqu9cCRgEIoGbH1zz4LaIwTgsR7CymAWYlciBKIAlygxcDM0WvT///NNymw24GX7//fcuj0FMwGLpDETXpk2bHJ8hIqxEobPbJs6D5Q1Cz046EGuwZMFNUQsjV5QvX85lXdgBFjidRwirsWPHqnqAZa9Bgwby6quvqmNwdYR1dsCAy+sh7dSlK3AP87Ogvtu1G+BYY4t6h1gFsITCKte1a1clir/77jtlddSYt1MpkD+/lC1b9r/PBQo4ygGWP1gNkU+0R7gFV69e3eEWa7bwEUIIIYSQwEKhZ4EaJLtwqwz7d11dWExRCctTQpJ2fCBipIhE5BHDSBUJjxBJSVDfR9Z58fK5kXklzI2LJsAqsZioMNvRceA6C/EE102AiJCu3Gkh0mDpww8sPY888ogSB9giA5akPn36uFyXZwaBXRDMBFsr2FmT5g+LHtYbwlUR1kSIifnz5zvWIHoCkWG1Fc9TOigPnA+RBzEWCGBtgyWuYMGC6jPcLAsXLuwQqbDqabAWDt+5wqouUb5Yd2fGWbCivmHxdLWOEesU4Y6NtXaw5lm7r4b/G3HTtRiuW7eushLiPk899ZTajkK7ocJaiHWghBBCCCEka6DQ85KwmOKSp8ZwCc9TQq3XS72wTyQ8WiTNNNAOj1bfR0UX+u9cD0RFhkupONtBUNVWAbDoYT0UBAQCdMAS4wyiOMJdD+uvEIxDW88Q+RFh/hHpsWHDhkrAwf0O3zsD6x3WbX3zzTcqwIu3OFv/7ADLFn6QN6wLhVuo3otNr/n74IMPVKAQHUxEu27COrVy5UqP6eB6iCZYMu+77z71HdbKIfCQtwwYMEC5vbqyWsGiibqClQt5x1pIBH0BWCP55ptvKhEEyxmCvVjt12dVl7CaYX0hngNiCxZCZyB2sZ4OdaitfVivBzdQpHPbbbep+of1EwFrXAGLtZqwsNheAW0SaxHxvGiLb7/9tuMYnvfTTz+1WZqEEEIIISRkhF5iYqIjKmEgr8ks4XlLSkzN4Y7PBduukLRLJzOeF1NMUqOKSpTpXG+BFWb48OHp3Bz1mjAMxiHuELIeA2tYqbSVCJY9fR4CrUAIQhDBRQ+DeQCr0Q8//KCED8L/w1qHiItWPPbYY0rIQOjB5c9ZjECkTJgwQfwJrG8QDCdPnlRugVpEaQuhtkTqYCL4jPWjKAuzVcwqHYgrrHU04xx0xlwX5r3gzHWB9Wxr1661FEgIBgNxibpCe4U41xZHrKXDGjfUBdawwYUTQiw5Ob2Fzl1dwtKHtoC6xHpF53aj28TmzZtUQBmsp4Qo1i6Y2IOvY8eO6v4QZ2vWrHG4dWJ9ntnSef/990uJEv+tv0MZaEsyyh2WRQh7rOnDcwDkC1ssoMxcPRchhBBCCAkARjbzzDPPGHFxcUZERIRRqVIlY8mSJQG5xszZs2dVBJVdO/9yfJeQkGBs3bpV/XZHUtIl2/exey7OS0xONfYev/w7WFmwYIGRnJwc0Ht4U75ZjVXedu7caWzfvj1L7pWdaaalJBgpZ7cbSYnxXl23Zs0aY9++fT7lQZ9vt38eO3rQVrp2zwt1grkcsjJvgbqXP9LNTBq+XOvNNexvga+PUM1bIO7nrzR9TSdY+puveQlFjgVxOWRV3uz7CgYABO+A2xrcuuDyBZc5BKLAejB/XkP8A4J1+LJGL9RBBEyswSOuwVYX5qBChBBCCCEk8GSr0INgw/5ccPmCexii9SGE/bvvvuvXawghhBBCCCEkN5FtQu/EiRNqbVPLli3TfY/No7FGyF/XEEIIIYQQQkhuI9uEng4pj0ASZhAowxxuPrPXAAQrwb5g5h8rEKSCEBJcsF8SQgghhHhHti+4Qqh4M4j+6G7Tal+uQbTI5557LsP3KSnJcvzYIUeaqakpah+zIkWKiFVy0IGI9mkHu+dqbRkXLXLpokiS+8cPabwp31DOWyDu5Z80i4lxKVUSk85lSR5wfkJCotpwHf3z7JkTEu9m30dzn3aH3fNCnWAuh6zMW6Du5Y90M5OGL9d6cw37W+DrI1TzFoj7+StNX9MJlv7ma15CkZQgLofM5q14iTLBLfTKlLmcQQgrM7DM6WP+uAY8+eSTMmzYMMdnWPQQHCIyMipdQeUvUEgOHDigws9bkZqaKhERrvcR8/Vcb9IMdYK5LLIyb4G4l7/SzEw63l6rz8dETsWKVzq2pbAC/zTt/POze16oE8zlkJV5C9S9/JFuZtLw5VpvrmF/C3x9hGreAnE/f6XpazrB0t98zUsocjyIyyGr8pZtQg97jmFfsG+//Vbt1aWtatjoWm9eDRISEpTFLjY21vY1zmDfNfx4AoNIRE9MTk62POfUyWNSpKjnDdC9ORfnheUpIt9vi5dW1WOlcP5sN7RmG96UbyjnLRD3ymyaaYknJGn/QrlYoJkUKV0pS/Kgz8fegcE6AUAIIYQQEoxkq6JAxMx+/fqp4CrNmjVTGzdjPd3AgQMd52BjaGy4/Oeff9q+JjNgMOluQBkVFWl7k3a75+K8BImWn3alSPPq0ZInz38bVOc2vCnfUM5bIO6V2TRTky9J6vHFEhVb1+d0vM1DMLcHQgghhJBgJluFXs+ePZXFbuzYscods3bt2so6V7p0acc5+fLlk4IFC3p1DSGEEEIIIYTkZrLdR7B///7qx4qJEyd6fQ0hhBBCCCGE5GayXehlZ6j2+PjzbrdacEV8fLzE5Dnn13NxXkJEtCQlxEt8fF45F5V7XTe9Kd9Qzlsg7pXZNFPj4+XihWS5FH9BYrzsN77mIVDnB3M7y0qCuRxyen/zV7qZScOXawP1jgvWdpaVBHM5ZHXegvEdl9l0gqW/+ZqXUCQ+iMvBH3lD/BJPOxWEGblwgypE1kTUTUIIIYQQQgjJaZw9ezbd8jZX5Eqhh0idhw4dkjZt2sj69eu9urZRo0aybt06v52rt3rYv3+/x8rKDXhTvqGct0Dcy19pZiYdb68NxPnsc76Xb1aS0/ubv9LNyv7m7TXsb4Gvj1DNWyi+44KhvwG+4zJXJzkpb3YsernSdTM8PFzKlSsnkZGRXosrROS0e4035+I8Cj3vyiyU8xaIe/krzcyk4+21gTyffY79zdd2Foj2GIg0fLmW77jAwfdbaL/jgqm/Ab7j2OdAuORiBg0aFNBrfEk/txPMZZaVeQvEvfyVZmbS8fbaQJ+f2wnm8srp/c1f6WZlf/P2mmBuP8FIMJdXVuctFN9x7G/BxyD2udzpuhlMwMQeFxdny8+WEMI+R0hOgu84QtjnSPaRqy16wUBMTIyMHj1a/SaEsM8REkrwHUcI+xzJPmjRI4QQQgghhJAQgxY9QgghhBBCCAkxKPQIIYQQQgghJMTIldsrEEKIK/bt2ydJSUlq65WKFSuykAghhBCSY6FFL4hBQNQnnnhCihQpIldccYUsWLAgu7NESEhz7733yk033SQtWrTI7qwQEvKsWbNGmjRpInny5JHmzZvL33//nd1ZIiRkSUlJkaefflqKFi0qJUuWlPHjx2d3lkgWQKEXxCxbtkwWL14s27Ztk7lz58r999+vtmEghASuz61du5bFS0gW8Omnn8qkSZPkzJkz0rRpUxk5ciTLnZAAsXXrVilUqJDs3r1bfvrpJ3nllVeUFwsJbei6GWDQobZv3y6NGzdWljln0tLSZOPGjXL+/HmpX7++xMbGOo6tWLFC+vXrp2Ze8NOoUSP59ddfpX379oHONiE5kv3798vSpUulUqVK0rp1a5fn/Pzzz7Jjxw5lJW/VqpWEh3O+ixBfWb9+vZw4ccLyvZSQkCC//fabREdHS4MGDZRbtOaNN95w/I33X3x8PCuCEDccPnxY/vjjD6ldu7aULVvW5TlbtmyR48ePS61ataRYsWKO7+vUqaN+YNlLTU2V/PnzS4ECBVjeIQ5HOAFi1apV6sXXsmVL6dChg2zatMnloBSdrkuXLvLwww+rTgvLnQYvz+LFizs+lyhRQn1HCEnP0aNHpWvXrqq/wTVl+vTpGYoIL7du3bpJ9+7dlRjEJAqE3sWLF1mchHgJ+ph+f+Ed54pvv/1Wypcvr7xR7rjjDqlSpYqyKjjz119/yeTJk+W5555jPRBiId5uv/12ZTTo2LGjMgQ4A4+v66+/Xq677joZMWKEVKhQQSZOnJghHbhK16xZU/VLVwYIElpQ6AWIf/75RwYPHqwscFbcc889qpPt2bNHzdBg4/S+ffuqGRsAkYcBrAZ/m4UfIeQyycnJ0r9/f9m1a5eaxXTFhx9+KCtXrlTrgj777DNlZYBl77XXXmMxEuIlx44dk5kzZ1r2H1jn7rzzTvWew+AS3i2wQvTq1SvdeatXr1ZrY+fMmSOlS5dmPRDiAhgGIPTQj6y8UB577DE1TsQ5sLTPmDFDhgwZIhs2bHCcA4GHSU+4bOI9+Msvv7C8QxwKvQDRs2dPNctp1SEh5jAjM2zYMOXSAgYNGqTcWvDCAwgKMXXqVCUa4U+NjtusWbNAZZmQHEu5cuWURS8iIsLteqDOnTsrl02ASZMePXqo7839EhMvcGvZuXOnnDp1KkvyT0hO49FHH1UWPSsWLVokp0+fVucBvAsff/xxtVQBE5sAAcaGDx+uPFlKlSql+h0hJCPwEINVPCoqymXxQLzNnj1bHnjgAYmLi1Pf3XbbbWoZw8cff6w+4/j8+fPl3LlzypMFE6SIMk1CGwq9bAIvOkTVxLoEDczp1atXd7wEIfTgZoZF6gMGDJApU6ZIwYIFsyvLhORoYFWoUaNGuu/wGWto8ZIEsKpjkgZrF/BihcWCEOI9EHRw20SEPw3W6AH9jnv22WeVhR0TNTryJiHEezAxqWM9mMFn3d86deqkJlcw2Yn3G6zrVmvZSejAYCzZBKKMAWf/aLwUMQuqef7559UPISRzYBYTEcfMFC5cWAVEwgsSx95//30WMyF+esc5v98g5vLmzet4x0EMEkL809+sxpSYzASw9GnrHsk90KKXTWh3TUQkMwNzuj5GCPEfGGA6R/WD+NPHCCH+A+8x5/cbJlUuXbrEdxwhfoZjSmIFhV42UbFiRccCWzMHDhxwHCOE+I/KlSur9Xdm8BnuZTExMSxqQvwI3mOHDh1S4k6jP/MdR4h/4ZiSWEGhl03UrVtXLT6Hv7QGftSIGsh98gjxPzfffLMsXLhQuWkCWBYQBALfE0L8S7t27VS49++++87x3bx58yRfvnxqGxRCiP+Ayyb2WjaPKbGXHgL5cUyZu+EavQBx8OBB2bx5s2Pfu7Vr10piYqKKgIQfRAd89dVXVZAVrFvAHnovvfSS2h+Fi2MJ8Z53333X0fcuXLigPsfGxjrCuWO7E0TYbNOmjdxyyy2ybNky5Vo2atQoFjchXoL3G/qa3iMWe1MC7POFQSfCuGPLkz59+qigK5hgQV/D39ykmRDvQARojCMBAvmh/6HPYeyIbUvA2LFjVRC/kiVLqiAsEyZMUAHHnLc0IbmLMAMthvgdbJ0wfvz4DN/37t1b/Wgw2ERkP7wEMcuJLRboRkaI9yCstDNYiD5mzBjHZ/SzadOmqf3zsJksNk03RwUkhNgD7zdXmzZjAlNvu4DtErB/5fLly9UaIoR7RyRpQoh3YEIF++Q507ZtW7VFiQZRbBFU7OTJk0rsDR06lNHaczkUeoQQQgghhBASYnCNHiGEEEIIIYSEGBR6hBBCCCGEEBJiUOgRQgghhBBCSIhBoUcIIYQQQgghIQaFHiGEEEIIIYSEGBR6hBBCCCGEEBJiUOgRQgghhBBCSIhBoUdIDiUlJUV++OEH+eyzz2T37t22rvnzzz/l22+/DXjeiP/ZuHGjqu9gZNWqVbJ9+/Yc18b8nddgrqPcxrFjx+TTTz8VwzCyOys5jlApO+fn2Ldvn6xcuTK7s0VIlsIN0wnJgaSmpsq1114rCQkJUqNGDbn//vuldevWHq979tln5ZtvvpGff/5ZcjubN2+W48ePS5s2bSQnMGLECCVMli5dKsHEwYMHpV69evL7779L+fLlc1Qb83deg7WOgg2I65IlS0qtWrUsz0GdHDhwQP0dHR0tFSpUkPr160tERISte3z//ffqf2JycrJERkZKsLJz507566+/JC4uTpo1a5Yur5jM+/zzzzNc06RJE7nyyivdCpz169dL7dq1VZ80s2fPHiV4rr/+er+W3dGjR+W7776TO++8U4IF5+c4c+aMVKlSRX2P9yYhuYHg/e9HCHFrOVi7dq2cPXtWChYsyJLyAVhCMZjMKUIvWHnhhRfk1ltvzTCgzAm4ExokcIwePVpuvPFGt+U/btw4JVZatGghSUlJsnr1ailUqJAsXrxYKlas6PEeJUqUkB49ekh4eHjQTtb17dtXvv76a/WMe/fuVcJu2bJlcsUVV6hzEhMTpWfPnqqsihYt6ri2dOnSlkIPEw34n9agQQNVfhCKrVq1ctwTZfLcc8/5vewwcYa8BpPQcwbt55577pGnn35a5s2bl93ZISRLoNAjJIexdetW9fLGSxiDHtC9e3f5559/ZN26depzbGys1KxZ09aA6NChQ0o4QjA2bNhQ8ubNm+44Zpvxg8EFBg9RUVGWae3atct2Hnbs2KGeBYMaWIS8Pe4uX3geiODGjRvLhg0b5OTJk9KyZUv1ogdwM0Ta2rUHwEIKq4Gn8rB7D1/yChfIc+fOyS233GJZxpcuXVLXnDp1Sho1aiTFihVLdxxuSqgDWENQ9rifmTVr1qhzmjZt6vhu27Ztsn//frnpppu8yhO+nzFjhrKKucNTngBm3X/66Sc1GEV94/nQpnWevG1rGDTPnz9f6tatK1WrVnV8j9n8tLQ0NRiuVq2aFC5cOF26duvfHbC0W7UJO33EXfn7I39W7fHw4cPK9bRdu3aOckHdYVCMsoIw+/XXX1W9hIWFqbaHujKLEA3K+LffflP5xTlavGBy5cSJE0qQ6L7XrVs3yZMnT4Y0rrnmGsc558+fVxa9Rx99VObOneuwhkFgFClSRFnDYPnTIG9IF/nUxMfHq7JHW0Pa5ny7O+bpXnjGX375RW6//Xb1XBBt1atXl6uvvtqyDt5991358ssvZdOmTUq0oZz79Okj9957ryxfvjzDZIq5v7rj9ddfl/vuu09efPFFeeedd9S1WuhNmDBBKlWqJB06dHCbhnPZeXo+WMrQr4CuL9wH5aitfZiYLFCggGpvsF46lx3eYTgH/4fQZ2AdvOGGG6R48eKOcyF8FyxYoPou2v2iRYvU96gL5AV93RN33323snTCE6Fs2bK2ypSQHI1BCMlRzJ8/37j++uuNsLAwo0ePHuonMTHRWLFiheNzhw4djAIFChiPP/54umtHjx5tNG/e3PH57bffVufddNNNRuvWrY3q1asbGzduVMeQ5h133GGULFnS6Ny5s1GrVi2jRo0axq5duyzzZicPCQkJKt3Y2Fijbdu2Rv369Y0uXboYaWlpto7bydfw4cONqlWrGlWqVDHatGlj1KlTxyhSpIixZs0adXzx4sXqmuLFizvyu2rVKrfl4Yyne3ibV9zrhhtuMPr37295v2rVqqk0UC6NGjUy4uLi0t3vzJkzqn5LlSqlyr9YsWLqWS5evOg4p2/fvkavXr3Spf3yyy8bDRs29DpPX3zxhZE/f34jJSXFso3ZydPx48fVc5UrV06dU7ZsWdXGzXnypa0NHTrUqFixonH27Fn1GXUcGRlpLF261GVe3dU/nuOTTz4xjhw5YpknO23CTr6tyj+z+fPUHpOTk43GjRsbt956q+OasWPHGkWLFjUOHjyoPk+YMMGR/xYtWhgFCxY0Pv/883T32bNnj1G3bl2jTJkyRseOHY2rrrrKeOWVV9SxcePGqTZQs2ZNRzrIuzNdu3ZVP2YeeOABo1KlSurvhx56SLU9lAW+wz127NjhOPe7777Dwiz1TGD16tWqLpo2baryhHYxa9Ysj8fs3GvhwoVGVFSU0b59e6NJkyZGu3btjOjoaGPy5MmWddG7d2+jVatW6b5bsGCByrMu6/j4ePUZ9Y5jmzdvdvwftAJ1Mm/ePPX3+vXrjfLlyzvqpEKFCsaxY8fcXu+q7Dw93969e9Wz4Bpdpx9++KGj/eD/FK5Bn0ZbWrRoUYayQ9lec8016lqkh/91L7zwQrp8zZkzx8iXL58qF5SRvhfaSenSpVWfwPvD6jk0OPeDDz7wWA6EhAIUeoTkQJYsWWJERES4PQcDEbwU8bLXOA9sMeCaPXu24/O+ffuMdevWqb9HjhxpNGvWzLhw4YLj+L333qte2HZxlYdHH31UDeTxMtd89dVXRmpqqq3jdvKFgTJe8HPnznV8169fP6NevXqOgdJTTz2lBh5m3JWHM3bu4U1eV65c6bYscR7qHINSTffu3Y2bb77Z8XnEiBFKZJw6dUp9Pnz4sBrUjBkzxmuhZydPTz/9tNGgQYN03zm3MTt5euSRR5ToOHfunPqMuoeAcCf07LQ1CBsIDjwvxN6VV15pDB482DKv7uofg2yUCYSaFXbahJ18W5V/ZvNnpz0iPxCTGAgjTxjQQ9BbAUGEfKGsNRCLZjGPvosBvQZljrJ3hyuhB9HbsmVL4+uvv1biYMOGDer7pKQkdT9MgFgN8tFX7rvvPsdx5A3/Rz0ds3MvPBvuBRGrmTRpkhI4VvX+2GOPKeGF9DQQw0hH31sLPbRhiKzChQsrIaeFoFW5QRjqfKHtAdTx9OnTjRMnTqiJrt27d3sl9Dw9H9qds+1g+fLlSkCbJ7Zmzpyp2guezZy2c3uAyIPYc362nj17uswz6gwTK+Y8Wgk91J25vgkJZei6SUgIcfHiRRUU48iRI8p1DQEP4A4DFy9XwP1ly5Yt6lwsVsc6K73W6qOPPpKbb75ZuYf+Oymk1m7AVQ9uWVbrNzzlYfr06TJs2DCHKxfo0qWL429Px+3mC+5wcAfSPPbYY2oBPtw2za58dsvDFZ7uYTevcCWyE0wHbk8IxKCBS9Zbb73l+Ay3KZSddrsrVaqUDBgwQH0/cuRI8QY7eYILnrProzN28gRXZPwNV0aAur/jjjuU+6M7PLW1mJgYmT17tnIhg4scXMfGjh1rmZ67+of7JdYtIf/usNPu7PRTV+Wf2fzZaY9wuZs4caI8/PDDKl9YRwY3PjNwxYMLH+of6+fwG5F/4c4HN2A8C9bUabdSpNu5c2fxFrjXoZ3gHoiWiB+4O+I7uB9ql264nqKcsZYNLr9wr3QGeYHLIVxi4TqIz+3bt/d4zO694Ob4wAMPpOubSA9l5apOHnroIZkyZYp06tRJ1RuCpOjAK7hO3wuBa/Q6YtwP6cK9U7stOoN1csgfynzy5MnKHVTXb/PmzVW7wg/cavG9JzdOjbfPp9sb2j7aOu6no1/CpRlusHCB1TzyyCPpru3Vq5daS4dr8X/v9OnTsmTJEvniiy8c5yA9HIe7J9w60RfQ9jyB/0Vos4TkBij0CAkRsE4KC+HLlCkjV111lVrzgnUnWIdmxbRp09SA4+2335brrrtODVDxgsVAEut1MGjD+gszXbt2VS/VfPnyeZ0HrC/D34h85gpPxzHgs5sv53VPOngB1hdZCT2r8rASte7ugb/t5hVrpezgPICFkEE6AHWGgTHK3QzWriA/3mInTxBOFy5csDxuJ0+6rTmXJT67E3p22zsGmhj8Tp06VQkclJkV7uq/XLlyjvVH7vDU7uzm21X5ZyZ/3vSd//3vfzJmzBi1pvLVV19Ndy6E8vPPP6/Wy0EI6qiMyD+EHiI6Aqs+7A3IL9ZkYQ0WBvFYm1inTh157bXXMkxe6fViKGdXQg9r1TDBgHLF2taOHTvKgw8+qCYX3B1DenbuhTzmz5/fcY5uZ7p/OoPnwTrhDz/8UK1bRHuYNWuWmpRAv9JpmINF4V5Dhw5Va/Dwv9JVW4ZoBBCIaCtY24k0seYNa/QwgfLmm2+qeyEoi12h5+3zAYhnCDTnyKHIg3mdMtJ2rjP0G6ybRj4h9LAuE5MZes0u1vZBbGPtJoQr6gpiGRMXnsD/LOe1zYSEKhR6hIQIsJoMGjQoXUQ1BFBwtxcSBhEYbOCFjGhvCHSAgSAGeRiAYtCAAY+/8oDBAWbLMaPrCk/HMTiwmy8MMFx9dveCtyqPl156yet7eJNXc8AIX8GAGwMhzPqbwWfzM0MUYHbfjKvBmp08YTD/8ccfZypPOAdWFGfx4Vy2vrZ3iMWZM2cqwYd2jYGiVYh+b+vfl3ZnN9+uyj8z+fOmPUJIwuKBgCSvvPKK+gHol08++aQSEdraCKvknDlzHPnXgWdwridrryfMwVjMoCxdtSl9zBWwEkNk4zwEDnn55ZeVdRDBbtwd8+VedoEoMVvaUY7AXVARBOHB5Aj6C4S2K1DHWvD17t1bWWcxqQCrMiY9dNnqvS8DBfJqZ4LE6n8N8o6gMhD2EHx4Jj2xgDaJ54dlWU/EweKIIEOegCCEdZOQ3EBwxh0mhHgNBlxmSxVc1RC50gpEN4TbjbZCYC8+WAfgcoUXLyLvYbYZ55mBhSYzeWjbtq1yGTIPbDGo1J/dHfcmX3ANQoRDDSIwYtYYg2qAWXOzwHFXHla4u4evZZgZEKYdedDoiIn4XoNIc4ggaEZHzPMWRMVDmZnLwJc8YeYeA2sNhCjCzrvDTltD9EvUYb9+/ZTbH45biSJP9Q83NQxY9TlWeGp33vZTf+XPbnuEkISAhNsf+uH48eMdm8AjbdSfOf/O1hpY+hAp0XkCAHtWapz7nreg7WCvQrjAamDxQblAWLhCPyPqAtuBPPPMMyrKJtqau2O+3MsOSNssICHeYHGDNVGni7biDNoTogNbiTwzmAyA8IEVEEC46zRhLXUVLdVXtBXSXK9wf4WLKVwrzeDedjZjh+UP7Qbu/IjIC+GnwXNUrlzZIfLQ13UUanegzDE5AmsgIbkBWvQICRGwjgazw5jphWvKG2+84Xj5ugJhxDGIweAPgzOIKaypgNVDh+nGproYhOMFixfzjz/+qNxssO7J1zxgfyyEz8aLFq5neJFj4PTHH3+owain43bzBSsR3JIwm41ng8sZ0tbuaZjRxobZGFxh0AT3IKyXsSoPV3i6hy9lmBlgicC6F8zaw/oDsYSBHmbDNTiG2XBYdeCShoEYBvcYPHoL3Ksg9j755BMZNWqUz3mCdQttES6DKCuIB1iE9Jo9X9va8OHDlahBPcDtDO6bcGVDHWPrAm/6A6xoeAbUn7tBtqc24W0/tZs/DKaRvxUrVljmz1N7hHsnxCPEzl133aWuGTJkiApJD0EKCy7cM3EfrN3DgNlZ0CEtbB2AcyCgsC0AxC/6MdqJ2VIHwQgLvtX2Clag7WJ9G6yKaDN///23skLCImZlHcIaMFhy8fx6/RqeE3+7O+bLveyAdon+ACsVLIOwOkNIY39PDdakoY6xvlHvIYj6teNCjLYF103879QWbNQJ1vehX6EvoK79BSYy0I6feOIJtY4YImzgwIHy1VdfqTYAKzaeE+63sJ7if4Cn8oMQhVhE/SA9c59Fm4G7LVxgkS7qCC7QnkB5wNUTLrqE5AZo0SMkB4K1JNo1RzNp0iQZMWKEY78yvNAwk2velBh/65lMDKzw0oVLG/bGwgwprsHLGcDVBwM0DPiw4B3uLhgcmgfoztjJA17YSBf50NYIvPj1zKyn43bzhUEBBklYM4S8YHCEgY8G6WPNE0QOrEkQFu7KwxWe7mEnrxi0632u3OHqPKwVMgeqwZ5seAYIMMyAwwUMAtm8bkw/HwbkcGvEYA9WHggIb/MEYP14//331Zoh5zZmN08Y+CNPGCgiTwgggY2N3Qk9T20NwgdukxAwem0RhDxcwRYuXOh1f0BbxGSAO5cvlBsCYbhrE3b6iKvy95Q/O8FYPLVHrOPCIBhCRgMhCeEK0QG3OVj3kDdYgeFqjbzgvua1URBJCL6BATiOQwTAKqOB0IWAgsjEGjzddsxgsgc/rkBZIF3s64bfEKxY52YOGuO86TcmD2AhgjhFGcBqCYulp2N27gUrOa43g7aL+5vXtTm70sLiBgsY0sWkFurXbCWEsMR6OkwKoL1AQMHd0tznrUD9DB48ON0epGj/ELHoY7Byo++6wrns7Dwf2h/+T6N80L/Qx9E+sCcgJjrQH9AmdP93l7YZPAPy7TyRhDaMvoU+hOfBpA72DTSvaXR+DlhR0batJqUICUXCEHozuzNBCCH+BANprN2Ay1VOvkdOARY5CCZf171gsItXkY7SiL9hbYS1EOtzggEMECH0zJECCSE5B/y/hiUTlm1Ccgt03SSEEJIpRo8enanr4WYGiyKsJnB9hKUK63jg/hgswPWMEJJzgdWcIo/kNui6SQgJObxxPQzme+QWsBYHLpYIlb5+/Xol+rBfnC/rBgkhhBByGbpuEkIIIYQQQkiIQYseIYQQQgghhIQYFHqEEEIIIYQQEmJQ6BFCCCGEEEJIiEGhRwghhBBCCCEhBoUeIYQQQgghhIQYFHqEEEIIIYQQEmJQ6BFCCCGEEEJIiEGhRwghhBBCCCEhBoUeIYQQQgghhEho8X+Pro8FWQ1YqAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "SHARE_STYLE = { # colour follows the share, in the report's column order, never the rank\n", + " \"eval-speech\": (\"speech\", \"#2a78d6\", \"o\"), \"eval-music\": (\"music\", \"#eb6834\", \"s\"),\n", + " \"eval-tts\": (\"TTS speech\", \"#1baf7a\", \"D\"), \"eval-noise\": (\"noise\", \"#eda100\", \"^\"),\n", + "}\n", + "present = [sh for sh in kws_eval.NEGATIVE_SHARES if sh in report.hours]\n", + "absent = [SHARE_STYLE[sh][0] for sh in kws_eval.NEGATIVE_SHARES if sh not in report.hours]\n", + "fig, ax = plt.subplots(figsize=(9, 4.8), dpi=100)\n", + "for sh in present:\n", + " label, colour, marker = SHARE_STYLE[sh]\n", + " H = report.hours[sh]\n", + " rows = [(r.threshold, r.recall.frr, r.shares[sh]) for r in report.rows if r.recall.frr is not None]\n", + " seen = [(f.fa_per_hour, frr, f.interval) for _, frr, f in rows if f.events > 0]\n", + " unseen = [(f.zero_event_bound, frr) for _, frr, f in rows if f.events == 0]\n", + " bound = kws_scoring.zero_event_bound(H)\n", + " if seen:\n", + " x = np.array([v[0] for v in seen]); y = np.array([v[1] for v in seen])\n", + " lo = np.array([v[2][0] for v in seen]); hi = np.array([v[2][1] for v in seen])\n", + " order = np.argsort(x)\n", + " ax.errorbar(x[order], y[order], xerr=[x[order] - lo[order], hi[order] - x[order]], fmt=marker + \"-\",\n", + " color=colour, ecolor=colour, elinewidth=0.8, capsize=2, lw=1.5, ms=5, alpha=0.9,\n", + " label=f\"{label} (H = {H:.3g} h, {report.negative_streams[sh]} streams)\")\n", + " if unseen:\n", + " ax.plot([v[0] for v in unseen], [v[1] for v in unseen], marker=\"<\", ls=\"none\", mfc=\"none\", mec=colour,\n", + " ms=8, label=f\"{label}: no event (rate <= ln 20 / H)\")\n", + " ax.axvline(bound, color=colour, ls=\"--\", lw=1, alpha=0.7)\n", + " ax.annotate(f\"{label}: ln 20 / H = {bound:.2f} FA/h\", xy=(bound, 0.98), xytext=(4, 0),\n", + " textcoords=\"offset points\", rotation=90, va=\"top\", ha=\"left\", fontsize=8, color=\"#52514e\")\n", + "ax.set_xscale(\"log\")\n", + "ax.set_ylim(-0.02, 1.02)\n", + "ax.set_xlabel(\"false accepts per hour (log axis; bars: exact Poisson 95 % interval)\")\n", + "ax.set_ylabel(\"false-rejection rate (1 - recall over every positive)\")\n", + "title = f\"DET — band-energy sanity baseline on {manifest.name}\"\n", + "if absent:\n", + " title += f\" (absent: {', '.join(absent)})\"\n", + "ax.set_title(title, fontsize=11)\n", + "ax.grid(True, which=\"both\", color=\"#e6e5e0\", lw=0.6)\n", + "ax.set_axisbelow(True)\n", + "for side in (\"top\", \"right\"):\n", + " ax.spines[side].set_visible(False)\n", + "ax.legend(fontsize=8, loc=\"lower left\")\n", + "fig.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "009f8089", + "metadata": { + "execution": { + "iopub.execute_input": "2026-09-09T16:44:24.812014Z", + "iopub.status.busy": "2026-09-09T16:44:24.811899Z", + "iopub.status.idle": "2026-09-09T16:44:24.816945Z", + "shell.execute_reply": "2026-09-09T16:44:24.816588Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "positives: {'eval-speech': 195}; thresholds swept: 44\n", + "speech: H = 2.9513 h, zero-event bound 1.015 FA/h\n", + "noise: H = 0.5021 h, zero-event bound 5.967 FA/h\n", + "the baseline's best recall over the sweep: 34/195 = 0.174 [0.128, 0.234] at threshold 0.3, where FA/h speech = 1977.11 [1926.70, 2028.50] (5835 in 2.951 h)\n", + "below 1000 FA/h on speech the best recall is 0/195 at threshold 0.0 (60.65 [52.09, 70.22] (179 in 2.951 h))\n", + "threshold 0.0: every stream fires once at hop 0 (the t = 0 rule), so FA/h speech = 60.65 [52.09, 70.22] (179 in 2.951 h) is the stream count over H, and recall is 0/195 with 195 spurious events\n", + "the earliest hit window on this corpus starts at hop 264: hop 0 lies inside no window, which is why recall there is 0\n", + "the plan's floors are >= 20 h of speech and >= 20 h of music (M4b's): this corpus has speech 2.951 h, noise 0.502 h\n", + "streams whose smoothed maximum reaches 1.0: 394 of 412\n", + "the saturated end of the sweep, thresholds >= 0.95 (5 rows): speech events [5592, 5195, 4844, 4141, 4105], hits [13, 13, 13, 7, 7] of 195\n" + ] + } + ], + "source": [ + "# what the curve says, in the numbers it was measured with (nothing below is typed by hand)\n", + "best = min((r for r in report.rows if r.recall.recall is not None), key=lambda r: r.recall.frr)\n", + "speech = \"eval-speech\"\n", + "print(f\"positives: {report.positives}; thresholds swept: {len(report.rows)}\")\n", + "for sh in present:\n", + " H = report.hours[sh]\n", + " print(f\"{SHARE_STYLE[sh][0]}: H = {H:.4f} h, zero-event bound {kws_scoring.zero_event_bound(H):.3f} FA/h\")\n", + "print(f\"the baseline's best recall over the sweep: {best.recall.hits}/{best.recall.positives} = \"\n", + " f\"{best.recall.recall:.3f} [{best.recall.interval[0]:.3f}, {best.recall.interval[1]:.3f}] at threshold \"\n", + " f\"{best.threshold!r}, where FA/h speech = {kws_eval.format_fa(best.shares.get(speech))}\")\n", + "under_1000 = [r for r in report.rows if speech in r.shares and r.shares[speech].fa_per_hour < 1000.0]\n", + "if under_1000:\n", + " top = max(under_1000, key=lambda r: r.recall.recall)\n", + " print(f\"below 1000 FA/h on speech the best recall is {top.recall.hits}/{top.recall.positives} at \"\n", + " f\"threshold {top.threshold!r} ({kws_eval.format_fa(top.shares[speech])})\")\n", + "zero = report.rows[0]\n", + "print(f\"threshold {zero.threshold!r}: every stream fires once at hop 0 (the t = 0 rule), so FA/h speech = \"\n", + " f\"{kws_eval.format_fa(zero.shares.get(speech))} is the stream count over H, and recall is \"\n", + " f\"{zero.recall.hits}/{zero.recall.positives} with {zero.spurious} spurious events\")\n", + "earliest = min(scoring.hit_window(p.endpoint_hop)[0] for s in streams if s.share == \"positives\"\n", + " for p in s.positives)\n", + "print(f\"the earliest hit window on this corpus starts at hop {earliest}: hop 0 lies inside \"\n", + " f\"{'no' if earliest > 0 else 'a'} window, which is why recall there is {zero.recall.hits}\")\n", + "print(f\"the plan's floors are >= 20 h of speech and >= 20 h of music (M4b's): this corpus has \"\n", + " + \", \".join(f\"{SHARE_STYLE[sh][0]} {report.hours[sh]:.3f} h\" for sh in present))\n", + "print(f\"streams whose smoothed maximum reaches 1.0: {saturated} of {maxes.size}\")\n", + "top = [r for r in report.rows if r.threshold >= 0.95]\n", + "print(f\"the saturated end of the sweep, thresholds >= 0.95 ({len(top)} rows): speech events \"\n", + " f\"{[r.shares[speech].events for r in top if speech in r.shares]}, hits \"\n", + " f\"{[r.recall.hits for r in top]} of {report.rows[0].recall.positives}\")" + ] + }, + { + "cell_type": "markdown", + "id": "83dc7b62", + "metadata": {}, + "source": [ + "## What this is, and what it is not\n", + "\n", + "This is the **sanity curve** the plan's M5 asks for: the harness's plumbing — hop alignment, the hit\n", + "window, the refractory merge, the hours from decoded durations, the Poisson and Wilson intervals, the\n", + "per-share report — exercised end to end on real audio through the shipping front end. The pass is the\n", + "planted-event oracle in `test_kws_eval.py` (exact to the utterance, in CI); the curve above is *not* a\n", + "pass and *not* a performance claim. A band-energy mean has no notion of the keyword: its recall is what\n", + "loud speech at the right moment buys, and its false-accept rate is whatever the printed numbers above\n", + "say — read them, not the shape. Every figure in this cell's argument is printed by the code cell above\n", + "it, from `report`; nothing here is typed by hand.\n", + "\n", + "Three things the curve makes visible that M6 inherits:\n", + "\n", + "- **The hours.** The FA/h denominator is the eval negatives' decoded duration only; the positives'\n", + " audio never counts. The hours per share and the zero-event bound ln 20 / *H* are printed above beside\n", + " the plan's floors (≥ 20 h speech and ≥ 20 h music), which are M4b's: until they exist, no figure can\n", + " distinguish a good detector from a lucky one below that bound.\n", + "- **The saturation.** The count of streams whose smoothed band-energy maximum reaches 1.0 is printed\n", + " above; where most do, the quantile thresholds collapse and an even grid fills the sweep. A real\n", + " spotter's scores are calibrated by its training and the quantiles spread on their own. Because the\n", + " reference stage smooths with a trailing mean over *W* hops, the smoothed score reaches 1.0 only where\n", + " all *W* frames sit at the baseline's clip, so the rows at the saturated end of the sweep (thresholds\n", + " ≥ 0.95, their speech counts and hits printed above) select among fully saturated stretches and say\n", + " nothing about a detector.\n", + "- **The threshold-0 row, and the packing.** Under the reference decision stage an event fires at hop 0\n", + " when s[0] ≥ θ, and every stream is decided after a fresh reset, so θ = 0 fires exactly once per stream:\n", + " FA/h = streams / *H*. Recall in that row is 0 wherever no hit window reaches hop 0 — the earliest window\n", + " start on this corpus is printed above; a window includes hop 0 only when the endpoint lies within\n", + " *T* + 1 hops of the stream start. streams / *H* is also the ceiling of what the negative-stream packing\n", + " bound (`max_stream_s`, recorded in the report) can add to any FA/h row through the *t* = 0 rule and the\n", + " refractory restart at each stream boundary. It is the rule, stated as a number, not a defect.\n", + "\n", + "Absent negative shares read *absent* in the report and are named in the plot title; an absent hold-out\n", + "reads *absent* in the report's positives line. The report never prints 0 FA/h for hours that were not\n", + "scored. **The operating point is M6's**, chosen on dev, and its recall is measured on the recorded\n", + "hold-out alone (`kws_eval.py holdout`, which refuses a hold-out whose file hashes do not match\n", + "`holdout.json`); the eval-tts recall beside it is a descriptive gap figure, never a pass." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tools/ml/build_kws_det_notebook.py b/tools/ml/build_kws_det_notebook.py new file mode 100644 index 0000000..3f77d2c --- /dev/null +++ b/tools/ml/build_kws_det_notebook.py @@ -0,0 +1,439 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""Assemble and execute notebooks/kws_det.ipynb — the wake-word plan's M5 DET notebook (§6 M5, §7). + +Same convention as tools/ml/build_ml_notebook.py: the committed notebook is a build product of this +script — edit THIS file, rerun, and the .ipynb is overwritten with fresh outputs. The notebook loads a +corpus from the feature store (never from git), assembles the evaluation streams from its lock, runs the +band-energy sanity baseline through the DspTap C ABI, sweeps the thresholds with `kws_eval`, renders the +committed report format and draws the DET (FRR against FA/h per share) with the zero-event bound marked. +It is the sanity curve the plan asks for at M5, not a pass: the operating point is M6's, measured on the +hold-out. + + .venv/bin/python tools/ml/build_kws_det_notebook.py \\ + --manifest tools/ml/kws/manifests/speech_commands_v2_bringup.json [--store DIR] [--grid 41] + +Refuses to run without a store (`--store` or MUTAP_KWS_STORE) and a manifest whose lock exists under +`/features//lock.json` (or `--lock`). The manifest is recorded in the notebook by +its repository-relative path; the store reaches the kernel through MUTAP_KWS_STORE, so a committed +notebook never carries a personal path. Requires tools/ml/kws/requirements-notebook.txt on top of +requirements.txt (nbformat, nbclient, ipykernel, matplotlib). +""" +from __future__ import annotations + +import argparse +import os +import pathlib +import sys +import time + +import nbformat as nbf +from nbclient import NotebookClient + +REPO_ROOT = pathlib.Path(__file__).resolve().parents[2] +KWS = REPO_ROOT / "tools" / "ml" / "kws" +sys.path.insert(0, str(KWS)) + +import kws_scoring # noqa: E402 +from kws_manifest import load_manifest, manifest_hash # noqa: E402 +from kws_streams import DEFAULT_MAX_STREAM_S # noqa: E402 +from kws_store import ENV_VAR, Store, StoreError, resolve_store # noqa: E402 + +DEFAULT_OUT = REPO_ROOT / "notebooks" / "kws_det.ipynb" +DEFAULT_GRID = 41 # evenly spaced thresholds added to the quantiles (the baseline saturates at 1) +DEFAULT_THRESHOLDS = 50 # the quantile count of kws_eval.default_thresholds + + +def scoring_text(scoring: kws_scoring.Scoring) -> str: + """The scoring semantics as a text cell, every number quoted from kws_scoring and the manifest.""" + s = scoring + return f"""## The scoring semantics, as numbers + +Everything below is `tools/ml/kws/kws_scoring.py` (checked against hand-computed values by its own +self-check, which the next cell runs) with the manifest's tolerance filled in; the notebook restates +nothing. + +- **Hop alignment.** A detector yields one score per completed front-end hop of {s.hop} samples at + {s.sample_rate} Hz: score[*t*] belongs to frame *t*, complete when sample (*t* + 1) · {s.hop} − 1 + arrives (`log_mel.h`'s alignment). An endpoint at sample *e* has hop index *h* = ⌊*e* / {s.hop}⌋. +- **The hit window** of a positive is frames [*h* − *T*, *h* + *L* + *T*], both ends inclusive, with + *T* = {s.tolerance_hops} hops (the manifest's `label.tolerance_hops`) and *L* = {s.latency_hops} hops + (`kws_scoring.LATENCY_CEILING_HOPS`, the §7 detection-latency ceiling of + {s.latency_hops * s.hop * 1000 // s.sample_rate} ms). One hit per utterance. +- **The reference decision stage.** The score is smoothed by a trailing moving average over + *W* = {s.smoothing_hops} hops; an event fires at hop *t* when the smoothed score crosses the threshold + upward (s[*t*] ≥ θ and s[*t* − 1] < θ, or *t* = 0 and s[0] ≥ θ) and at least + *R* = {s.refractory_hops} hops have passed since the previous event (*t* − last ≥ *R*: a crossing + exactly *R* hops after an event fires, one *R* − 1 hops after it merges into it) — the refractory + period, under which false accepts merge. (`kws.h` carries these numbers at M6 and must match them.) +- **False accepts.** An event on a negative stream; FA/h = events / *H*, *H* the negative streams' + decoded duration in hours, per share (speech, music, TTS speech and noise separately). Every FA/h + carries *H* and the exact two-sided 95 % Poisson interval on the count (chi-square form); at zero + events the one-sided 95 % upper bound is ln 20 / *H*. An event on a positive stream outside every hit + window is *spurious*: reported, never in FA/h. +- **Recall** = hits / positives with its Wilson 95 % interval; FRR = 1 − recall. A share the lock lacks + is reported as *absent*, never as 0 FA/h. +""" + + +def build_notebook(manifest_rel: str, lock_rel: str | None, grid: int, thresholds: int, smoothing: int, + refractory: int, max_stream_s: float, scoring: kws_scoring.Scoring) -> nbf.NotebookNode: + nb = nbf.v4.new_notebook() + nb.metadata.kernelspec = {"display_name": "Python 3", "language": "python", "name": "python3"} + cells: list = [] + + def md(s: str) -> None: + cells.append(nbf.v4.new_markdown_cell(s)) + + def code(s: str) -> None: + cells.append(nbf.v4.new_code_cell(s)) + + md(f"""# Wake word — the M5 evaluation harness on the bring-up corpus + +The wake-word plan's M5 ([`docs/wake-word-plan.md`](../docs/wake-word-plan.md) §6) builds the +evaluation harness *before any model*: the scoring semantics defined as numbers, a threshold sweep, a +committed report format with false accepts per hour on speech, music and TTS speech separately, each +figure with its hours and its interval. This notebook is that harness run end to end through the +shipping front end (DspTap's C ABI, `kws_features.FrontEnd`) on the corpus a manifest names, with the +plan's **trivial band-energy baseline** as the detector. + +It is the **sanity curve the plan asks for, not a pass**. The pass is +[`test_kws_eval.py`](../tools/ml/kws/test_kws_eval.py)'s planted-event oracle, which runs in CI; the +DET evaluation is deliberately not in CI (§7) and lives here, executed and committed, rebuilt by +[`tools/ml/build_kws_det_notebook.py`](../tools/ml/build_kws_det_notebook.py) whenever behaviour +changes. The operating point is M6's, measured on the hold-out. + +Manifest: `{manifest_rel}`. The store comes from `MUTAP_KWS_STORE` (no default); nothing in the +corpus enters git.""") + + lock_expr = (f"STORE / {lock_rel!r}" if lock_rel + else "Store(STORE).features(manifest_hash(manifest)) / 'lock.json'") + code(f'''import json, os, pathlib, subprocess, sys, time +import numpy as np + +REPO = pathlib.Path.cwd().parent if pathlib.Path.cwd().name == "notebooks" else pathlib.Path.cwd() +sys.path.insert(0, str(REPO / "tools" / "ml" / "kws")) +import kws_detectors, kws_eval, kws_features, kws_scoring, kws_streams +from kws_manifest import load_manifest, manifest_hash, read_lock +from kws_store import ENV_VAR, Store, resolve_store + +STORE = resolve_store(None) # refuses without MUTAP_KWS_STORE: there is no default store +MANIFEST = REPO / {manifest_rel!r} +manifest = load_manifest(MANIFEST) +LOCK = {lock_expr} +t0 = time.perf_counter() +lock = read_lock(LOCK) +t_lock = time.perf_counter() - t0 +if lock.manifest_hash != manifest_hash(manifest): + raise RuntimeError(f"lock {{LOCK}} was built from manifest {{lock.manifest_hash}}, this manifest hashes " + f"to {{manifest_hash(manifest)}}") +print(f"manifest {{manifest.name}} ({{manifest_hash(manifest)[:12]}}), store {{STORE.name}} " + f"(from ${{ENV_VAR}})") +print(f"lock: {{len(lock.clips)}} rows read in {{t_lock:.2f}} s; eval_set_id per share:") +for share, sid in sorted(lock.eval_set_id.items()): + print(f" {{share}}: {{sid}}") +print(f"front end: log_mel_contract_version {{kws_features.contract_version()}}, " + f"DspTap {{kws_features.dsptap_commit()}}")''') + + md(scoring_text(scoring)) + + code(f'''# the scoring numbers are built from the manifest (hop and rate from the geometry, T from the +# label rule); kws_streams refuses any other value, since the lock's rows were cut for these +g = manifest.recipe.geometry +scoring = kws_scoring.Scoring(hop=g.hop, sample_rate=int(g.sample_rate), + tolerance_hops=manifest.recipe.label.tolerance_hops, + smoothing_hops={smoothing}, refractory_hops={refractory}) +print(scoring.to_dict()) +e = 24000 +h = scoring.endpoint_hop(e) +print(f"an endpoint at sample {{e}}: hop {{h}}, hit window {{scoring.hit_window(h)}}") +r = subprocess.run([sys.executable, str(REPO / "tools/ml/kws/kws_scoring.py")], capture_output=True, + text=True) +print(r.stdout.strip() or r.stderr.strip()) +if r.returncode != 0: + raise RuntimeError("kws_scoring self-check failed")''') + + md(f"""## The streams + +`kws_streams.streams_from_lock` turns the lock's eval shares into the streams the detector scores after +a single reset, so the smoothing window and the refractory period run across clip boundaries as they do +in a room. Every variant-0 eval positive becomes one stream whose audio is the row's mixture — exactly +what `extract` featurized (context + keyword + the (L + T) hops the hit window needs). The negatives of +each share are sorted by id and packed into streams of at most {max_stream_s:g} s, a clip never split (a +clip longer than that stands alone). The hours denominator is the sum of the negative streams' decoded +lengths — the plan's rule — so the positives' audio never enters it. Every stream is decided after a +fresh reset, so the packing bound is provenance the report records (`max_stream_s`): the *t* = 0 rule and +the refractory restart at each stream boundary make every FA/h figure depend on it by at most +streams / *H* per share, the ceiling printed below and reached at θ = 0. The accounting of every stream +is checked against its decoded audio (`kws_streams.validate_stream`, one rule set) by the scoring pass +in the next section, which decodes each stream exactly once.""") + + code(f'''t0 = time.perf_counter() +streams = kws_streams.streams_from_lock(manifest, lock, Store(STORE), scoring, max_stream_s={max_stream_s}) +t_assemble = time.perf_counter() - t0 +hours = kws_streams.hours_per_share(streams, scoring) +positives = kws_streams.positives_per_subshare(streams) +counts = {{}} +for s in streams: + counts[s.share] = counts.get(s.share, 0) + 1 +print(f"{{len(streams)}} streams assembled in {{t_assemble:.2f}} s (audio is decoded lazily)") +print(f"positives per subshare: {{positives}}") +for share in kws_eval.NEGATIVE_SHARES: + if share in hours: + longest = max(s.negative_samples for s in streams if s.share == share) / scoring.sample_rate + print(f"{{share}}: {{counts[share]}} streams, H = {{hours[share]:.4f}} h " + f"(zero-event bound ln 20 / H = {{kws_scoring.zero_event_bound(hours[share]):.3f}} FA/h; " + f"longest stream {{longest:.1f}} s; packed at <= {max_stream_s:g} s, so the per-stream reset " + f"can add at most streams / H = {{counts[share] / hours[share]:.2f}} FA/h to any row)") + else: + print(f"{{share}}: absent from this lock")''') + + md("""## The band-energy baseline + +`kws_detectors.BandEnergyBaseline`: the mean, over the mel bands whose centre lies in +[300, 3000] Hz, of the shipping front end's plain-log feature, clipped to [0, 1]. It has no notion +of the keyword — it is the plan's *trivial* detector, run so the harness's plumbing (alignment, hours, +intervals, the report) is exercised on real audio before a model exists. The log affine maps a band +energy of 1 to a feature of exactly 1 and is unbounded above; clips that exceed that level saturate the +baseline's clip to [0, 1], so its per-stream maximum sits at 1.0 on such streams. The next cell measures +how many, and the sweep below adds an even grid of thresholds because the quantiles of saturated maxima +collapse.""") + + code(f'''detector = kws_detectors.BandEnergyBaseline(g) +print(f"detector {{detector.name}}: bands {{detector.bands[0]}}..{{detector.bands[-1]}} " + f"({{detector.params['band_centres_hz'][0]}}..{{detector.params['band_centres_hz'][-1]}} Hz), " + f"stored path {{detector.params['stored_path']}}") +t0 = time.perf_counter() +scores = kws_eval.score_streams(streams, detector, scoring) # decode, check the accounting, score: once +t_score = time.perf_counter() - t0 +audio_s = sum(y.size for y in scores.values()) * scoring.hop / scoring.sample_rate +print(f"scored {{audio_s:.0f}} s of audio ({{audio_s / 3600:.3f}} h, positives included) in " + f"{{t_score:.1f}} s: {{audio_s / t_score:.0f}} s of audio per second, decode included") +maxes = np.array([kws_scoring.smooth(y, scoring.smoothing_hops).max() for y in scores.values()]) +q = np.quantile(maxes, [0.0, 0.1, 0.5, 0.9, 1.0]) +print("per-stream maximum smoothed score, quantiles 0/10/50/90/100 %:", np.round(q, 4).tolist()) +saturated = int((maxes >= 1.0).sum()) +print(f"streams whose smoothed maximum reaches 1.0: {{saturated}} of {{maxes.size}} " + f"({{100.0 * saturated / maxes.size:.0f}} %)") +quantile_thresholds = kws_eval.default_thresholds(scores, scoring, n={thresholds}) +print(f"default_thresholds ({thresholds} quantiles) yields only {{len(quantile_thresholds)}} distinct " + f"thresholds; the sweep adds a grid of {grid}")''') + + code(f'''grid = {{kws_eval.threshold_value(v) for v in np.linspace(0.0, 1.0, {grid})}} +thresholds = sorted(set(quantile_thresholds) | grid) +provenance = {{"manifest_name": manifest.name, "manifest_hash": lock.manifest_hash, + "eval_set_id": dict(lock.eval_set_id), "holdout_set_id": lock.holdout_set_id, + "max_stream_s": {max_stream_s}, + "front_end": {{"log_mel_contract_version": kws_features.contract_version(), + "dsptap_commit": kws_features.dsptap_commit()}}}} +t0 = time.perf_counter() +report = kws_eval.evaluate(streams, detector, scoring, thresholds, scores=scores, provenance=provenance) +report.wall_s = t_score + time.perf_counter() - t0 +OUT = REPO / "build-kws-det" +json_path, md_path = kws_eval.write_report(report, OUT) +print(f"{{len(report.rows)}} thresholds; scoring + sweep {{report.wall_s:.1f}} s; report at " + f"{{json_path.relative_to(REPO)}} and {{md_path.relative_to(REPO)}}")''') + + md("""## The report + +The committed format (`kws_eval.Report.markdown`): the header names the manifest and its hash, the +detector, the scoring numbers, the front end's contract version and DspTap commit, the `eval_set_id` +per share, the hours per share with their stream counts and packing bound, and the positives per +subshare; then one row +per threshold — recall with its Wilson interval, FRR, spurious events, and FA/h **per share** with the +Poisson interval and the hours behind it (`<= ln 20 / H` where a share saw no event). A share this +lock lacks reads `absent`, and the eval-tts and hold-out recall columns stand beside each other as the +plan's descriptive gap figure.""") + + code('''from IPython.display import Markdown, display +display(Markdown(report.markdown()))''') + + md("""## The DET + +False-rejection rate against false accepts per hour, one curve per share present, FA/h on a log axis. +Each point is one threshold; the horizontal bar is the exact Poisson 95 % interval on that share's +count. The dashed vertical line per share is the zero-event bound ln 20 / *H*: nothing to the left of +it is measurable on this many hours, and a threshold at which the share saw no event is drawn *at* the +bound with an open, left-pointing marker (its rate is ≤ that bound, not 0). Missing shares are named in +the title rather than drawn as empty curves.""") + + code('''import matplotlib.pyplot as plt + +SHARE_STYLE = { # colour follows the share, in the report's column order, never the rank + "eval-speech": ("speech", "#2a78d6", "o"), "eval-music": ("music", "#eb6834", "s"), + "eval-tts": ("TTS speech", "#1baf7a", "D"), "eval-noise": ("noise", "#eda100", "^"), +} +present = [sh for sh in kws_eval.NEGATIVE_SHARES if sh in report.hours] +absent = [SHARE_STYLE[sh][0] for sh in kws_eval.NEGATIVE_SHARES if sh not in report.hours] +fig, ax = plt.subplots(figsize=(9, 4.8), dpi=100) +for sh in present: + label, colour, marker = SHARE_STYLE[sh] + H = report.hours[sh] + rows = [(r.threshold, r.recall.frr, r.shares[sh]) for r in report.rows if r.recall.frr is not None] + seen = [(f.fa_per_hour, frr, f.interval) for _, frr, f in rows if f.events > 0] + unseen = [(f.zero_event_bound, frr) for _, frr, f in rows if f.events == 0] + bound = kws_scoring.zero_event_bound(H) + if seen: + x = np.array([v[0] for v in seen]); y = np.array([v[1] for v in seen]) + lo = np.array([v[2][0] for v in seen]); hi = np.array([v[2][1] for v in seen]) + order = np.argsort(x) + ax.errorbar(x[order], y[order], xerr=[x[order] - lo[order], hi[order] - x[order]], fmt=marker + "-", + color=colour, ecolor=colour, elinewidth=0.8, capsize=2, lw=1.5, ms=5, alpha=0.9, + label=f"{label} (H = {H:.3g} h, {report.negative_streams[sh]} streams)") + if unseen: + ax.plot([v[0] for v in unseen], [v[1] for v in unseen], marker="<", ls="none", mfc="none", mec=colour, + ms=8, label=f"{label}: no event (rate <= ln 20 / H)") + ax.axvline(bound, color=colour, ls="--", lw=1, alpha=0.7) + ax.annotate(f"{label}: ln 20 / H = {bound:.2f} FA/h", xy=(bound, 0.98), xytext=(4, 0), + textcoords="offset points", rotation=90, va="top", ha="left", fontsize=8, color="#52514e") +ax.set_xscale("log") +ax.set_ylim(-0.02, 1.02) +ax.set_xlabel("false accepts per hour (log axis; bars: exact Poisson 95 % interval)") +ax.set_ylabel("false-rejection rate (1 - recall over every positive)") +title = f"DET — band-energy sanity baseline on {manifest.name}" +if absent: + title += f" (absent: {', '.join(absent)})" +ax.set_title(title, fontsize=11) +ax.grid(True, which="both", color="#e6e5e0", lw=0.6) +ax.set_axisbelow(True) +for side in ("top", "right"): + ax.spines[side].set_visible(False) +ax.legend(fontsize=8, loc="lower left") +fig.tight_layout() +plt.show()''') + + code('''# what the curve says, in the numbers it was measured with (nothing below is typed by hand) +best = min((r for r in report.rows if r.recall.recall is not None), key=lambda r: r.recall.frr) +speech = "eval-speech" +print(f"positives: {report.positives}; thresholds swept: {len(report.rows)}") +for sh in present: + H = report.hours[sh] + print(f"{SHARE_STYLE[sh][0]}: H = {H:.4f} h, zero-event bound {kws_scoring.zero_event_bound(H):.3f} FA/h") +print(f"the baseline's best recall over the sweep: {best.recall.hits}/{best.recall.positives} = " + f"{best.recall.recall:.3f} [{best.recall.interval[0]:.3f}, {best.recall.interval[1]:.3f}] at threshold " + f"{best.threshold!r}, where FA/h speech = {kws_eval.format_fa(best.shares.get(speech))}") +under_1000 = [r for r in report.rows if speech in r.shares and r.shares[speech].fa_per_hour < 1000.0] +if under_1000: + top = max(under_1000, key=lambda r: r.recall.recall) + print(f"below 1000 FA/h on speech the best recall is {top.recall.hits}/{top.recall.positives} at " + f"threshold {top.threshold!r} ({kws_eval.format_fa(top.shares[speech])})") +zero = report.rows[0] +print(f"threshold {zero.threshold!r}: every stream fires once at hop 0 (the t = 0 rule), so FA/h speech = " + f"{kws_eval.format_fa(zero.shares.get(speech))} is the stream count over H, and recall is " + f"{zero.recall.hits}/{zero.recall.positives} with {zero.spurious} spurious events") +earliest = min(scoring.hit_window(p.endpoint_hop)[0] for s in streams if s.share == "positives" + for p in s.positives) +print(f"the earliest hit window on this corpus starts at hop {earliest}: hop 0 lies inside " + f"{'no' if earliest > 0 else 'a'} window, which is why recall there is {zero.recall.hits}") +print(f"the plan's floors are >= 20 h of speech and >= 20 h of music (M4b's): this corpus has " + + ", ".join(f"{SHARE_STYLE[sh][0]} {report.hours[sh]:.3f} h" for sh in present)) +print(f"streams whose smoothed maximum reaches 1.0: {saturated} of {maxes.size}") +top = [r for r in report.rows if r.threshold >= 0.95] +print(f"the saturated end of the sweep, thresholds >= 0.95 ({len(top)} rows): speech events " + f"{[r.shares[speech].events for r in top if speech in r.shares]}, hits " + f"{[r.recall.hits for r in top]} of {report.rows[0].recall.positives}")''') + + md("""## What this is, and what it is not + +This is the **sanity curve** the plan's M5 asks for: the harness's plumbing — hop alignment, the hit +window, the refractory merge, the hours from decoded durations, the Poisson and Wilson intervals, the +per-share report — exercised end to end on real audio through the shipping front end. The pass is the +planted-event oracle in `test_kws_eval.py` (exact to the utterance, in CI); the curve above is *not* a +pass and *not* a performance claim. A band-energy mean has no notion of the keyword: its recall is what +loud speech at the right moment buys, and its false-accept rate is whatever the printed numbers above +say — read them, not the shape. Every figure in this cell's argument is printed by the code cell above +it, from `report`; nothing here is typed by hand. + +Three things the curve makes visible that M6 inherits: + +- **The hours.** The FA/h denominator is the eval negatives' decoded duration only; the positives' + audio never counts. The hours per share and the zero-event bound ln 20 / *H* are printed above beside + the plan's floors (≥ 20 h speech and ≥ 20 h music), which are M4b's: until they exist, no figure can + distinguish a good detector from a lucky one below that bound. +- **The saturation.** The count of streams whose smoothed band-energy maximum reaches 1.0 is printed + above; where most do, the quantile thresholds collapse and an even grid fills the sweep. A real + spotter's scores are calibrated by its training and the quantiles spread on their own. Because the + reference stage smooths with a trailing mean over *W* hops, the smoothed score reaches 1.0 only where + all *W* frames sit at the baseline's clip, so the rows at the saturated end of the sweep (thresholds + ≥ 0.95, their speech counts and hits printed above) select among fully saturated stretches and say + nothing about a detector. +- **The threshold-0 row, and the packing.** Under the reference decision stage an event fires at hop 0 + when s[0] ≥ θ, and every stream is decided after a fresh reset, so θ = 0 fires exactly once per stream: + FA/h = streams / *H*. Recall in that row is 0 wherever no hit window reaches hop 0 — the earliest window + start on this corpus is printed above; a window includes hop 0 only when the endpoint lies within + *T* + 1 hops of the stream start. streams / *H* is also the ceiling of what the negative-stream packing + bound (`max_stream_s`, recorded in the report) can add to any FA/h row through the *t* = 0 rule and the + refractory restart at each stream boundary. It is the rule, stated as a number, not a defect. + +Absent negative shares read *absent* in the report and are named in the plot title; an absent hold-out +reads *absent* in the report's positives line. The report never prints 0 FA/h for hours that were not +scored. **The operating point is M6's**, chosen on dev, and its recall is measured on the recorded +hold-out alone (`kws_eval.py holdout`, which refuses a hold-out whose file hashes do not match +`holdout.json`); the eval-tts recall beside it is a descriptive gap figure, never a pass.""") + + nb.cells = cells + return nb + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("--manifest", required=True, + help="a manifest under the repository (recorded by its repository-relative path)") + ap.add_argument("--store", default=None, help=f"the feature store (or ${ENV_VAR}); no default") + ap.add_argument("--lock", default=None, + help="the lock to score (default /features//lock.json)") + ap.add_argument("--grid", type=int, default=DEFAULT_GRID, + help=f"evenly spaced thresholds added to the quantiles (default {DEFAULT_GRID})") + ap.add_argument("--thresholds", type=int, default=DEFAULT_THRESHOLDS, + help=f"quantile count of kws_eval.default_thresholds (default {DEFAULT_THRESHOLDS})") + ap.add_argument("--smoothing", type=int, default=kws_scoring.Scoring.smoothing_hops, help="W in hops") + ap.add_argument("--refractory", type=int, default=kws_scoring.Scoring.refractory_hops, help="R in hops") + ap.add_argument("--max-stream-s", type=float, default=DEFAULT_MAX_STREAM_S, + help=f"the negative-stream packing bound (default {DEFAULT_MAX_STREAM_S:g}, " + "kws_streams.DEFAULT_MAX_STREAM_S; recorded in the report)") + ap.add_argument("--out", default=str(DEFAULT_OUT), help=f"the notebook to write (default {DEFAULT_OUT})") + ap.add_argument("--timeout", type=int, default=3600, help="per-cell execution timeout in seconds") + args = ap.parse_args(argv) + try: + store = resolve_store(args.store) + manifest_path = pathlib.Path(args.manifest).resolve() + if not manifest_path.is_relative_to(REPO_ROOT): + raise StoreError(f"manifest {manifest_path} is not under the repository {REPO_ROOT}; the " + "notebook records manifests by repository-relative path") + manifest = load_manifest(manifest_path) + scoring = kws_scoring.Scoring(hop=manifest.recipe.geometry.hop, + sample_rate=int(manifest.recipe.geometry.sample_rate), + tolerance_hops=manifest.recipe.label.tolerance_hops, + smoothing_hops=args.smoothing, refractory_hops=args.refractory) + lock_rel = None + if args.lock: + lock_path = pathlib.Path(args.lock).resolve() + if not lock_path.is_relative_to(store): + raise StoreError(f"lock {lock_path} is not under the store {store}") + lock_rel = lock_path.relative_to(store).as_posix() + else: + lock_path = Store(store).features(manifest_hash(manifest)) / "lock.json" + if not lock_path.is_file(): + raise StoreError(f"no lock at {lock_path}: build the manifest into the store first " + "(kws_build.py all), or pass --lock") + except (StoreError, ValueError, OSError) as e: + print(f"build_kws_det_notebook: {type(e).__name__}: {e}", file=sys.stderr) + return 1 + nb = build_notebook(manifest_path.relative_to(REPO_ROOT).as_posix(), lock_rel, args.grid, args.thresholds, + args.smoothing, args.refractory, args.max_stream_s, scoring) + os.environ[ENV_VAR] = str(store) # the kernel inherits it; the notebook itself carries no store path + t0 = time.perf_counter() + client = NotebookClient(nb, timeout=args.timeout, kernel_name="python3", + resources={"metadata": {"path": str(REPO_ROOT / "notebooks")}}) + client.execute() + wall = time.perf_counter() - t0 + out = pathlib.Path(args.out) + nbf.write(nb, str(out)) + print(f"wrote {out} ({out.stat().st_size / 1e6:.2f} MB) after executing {len(nb.cells)} cells in " + f"{wall:.1f} s") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/ml/kws/README.md b/tools/ml/kws/README.md index dd44dae..3cddc69 100644 --- a/tools/ml/kws/README.md +++ b/tools/ml/kws/README.md @@ -286,3 +286,183 @@ without a `noise_start` field; a TTS clip's synthesis scales come from `draw_rng(seed, clip_id + "#tts", 0)`, so the variant-0 key stays reserved for the eval positive's context draw. A hold-out mixed take (M4c) draws `{"noise", "level_db"}`. + +## M5 — the evaluation harness + +The harness of the wake-word plan's M5 (`docs/wake-word-plan.md` §6 M5 and +§7), built before any model so that M6 is measured by tooling that can fail +on its own. Host-side only, like the builder; the same pinned environment. + +**Modules.** + +- `kws_scoring.py` — the scoring semantics **as numbers**, checked against + hand-computed values by its own self-check (`python kws_scoring.py`): + `Scoring` (hop, rate, *T* = `tolerance_hops`, *L* = `LATENCY_CEILING_HOPS` + = 20, *W* = `smoothing_hops`, *R* = `refractory_hops`), `Positive`, + `smooth`, `decide` (the reference decision stage: an upward crossing of + the smoothed score at least *R* hops after the previous event — *t* − last + ≥ *R*, so a crossing exactly *R* hops after an event fires and one *R* − 1 + hops after it merges), `hits` (one + per utterance, window [*h* − *T*, *h* + *L* + *T*] inclusive with + *h* = ⌊*e* / hop⌋), `spurious`, `poisson_interval` (exact two-sided 95 %, + chi-square form), `zero_event_bound` (ln 20 / *H*), `wilson_interval`. + `kws.h` carries the same numbers at M6 and must match them. +- `kws_streams.py` — `Stream` (id, share, positives, `negative_samples`, + subshare, eager `audio` or lazy `load`, `members`), `streams_from_lock`: + every variant-0 eval positive becomes one stream whose audio is the row's + mixture (`kws_build.mixture` over its draw — what `extract` featurized, so + the stream has exactly the lock's `extra.frames` hops and the hit window + fits); the negatives of each eval share, sorted by id, are packed into + `/stream-NNNN` streams of at most 60 s, a clip never split (an + over-long clip stands alone). Hours per share = Σ `negative_samples` / + 16000 / 3600 — the decoded durations, the plan's denominator; positives + never enter it. `validate_streams` decodes every stream once and refuses by + name: no streams, duplicate ids, an unknown share, an empty stream, a + `negative_samples` that is not an integer, a positive stream with hours or + without positives or without a subshare, an endpoint at or beyond its + audio, an `endpoint_hop` that disagrees with its sample, a hit window + ending beyond the stream's hops, a positive id used twice (within a stream + or across streams), a negative stream with positives or whose + `negative_samples` ≠ its decoded length. Its per-stream rules are + `validate_stream(stream, n_samples, scoring, seen_positives)`, which takes + the decoded length, so the harness applies the same rules — one rule set, + not a copy — to the audio it decodes for scoring. A `Scoring` whose hop, + rate, *T* or *L* differs from the manifest is refused + (`require_scoring_matches`): the lock's rows were cut for those. +- `kws_detectors.py` — the `Detector` contract (one float64 score in [0, 1] + per completed hop; `frames_for(n, hop)` = n // hop, measured through the + bridge); `BandEnergyBaseline` (the plan's trivial sanity detector: the + mean over the mel bands whose centre lies in [300, 3000] Hz of the shipping + front end's plain-log feature, clipped to [0, 1], PCEN forced off and + recorded in `params`); `PlantedDetector` (the oracle's: zeros except + planted (hop, score) pairs per stream id, refusing a hop beyond the stream). +- `kws_holdout.py` — M4c's `holdout.json` (version 1: talkers with + `consent_form_version` and `permitted_uses`; utterances with a FLAC path + under `/holdout/`, sha256, talker, microphone path, distance, SNR, + phrase, 16 kHz `endpoint_sample`), refused at load on any schema + deviation (a path only in its canonical spelling — no `.`, `..` or empty + segment — and one sha256 per row, so one file is one utterance under any + spelling, case-folded included); `verify_holdout` refuses by file a FLAC + that is missing or whose sha256 differs, a stray FLAC no row names, two + rows resolving to one file, a talker row missing, without a consent form + version, or whose uses lack `evaluation` or `m7-replay`; `holdout_set_id` + (sha256 of the sorted FLAC hashes, each once); `streams_from_holdout` + (one positive stream per utterance, share `positives`, subshare `holdout`; + a FLAC that is not 16 kHz is refused, never resampled here). +- `kws_eval.py` — `evaluate(streams, detector, scoring, thresholds)`: every + stream decoded exactly once by `score_streams`, which checks its accounting + on that decode (`kws_streams.validate_stream`, so a hand-built stream + cannot mis-account) and scores it, returning a `Scored` (the scores by id + plus the sample counts) that `evaluate` takes as verified — re-running only + the accounting rules against the recorded counts, no second decode; a bare + dict of scores is re-checked in full. Per threshold `decide` → hits / + spurious on the positive streams, events on the negative streams per + share; hours = the integer sample sum per share divided once (bit-identical + to `hours_per_share` and the lock's summary); `Report` with `to_dict` / + `from_dict` (schema-key diffing through `kws_manifest.check_keys`, a wrong + `report_version` refused) and `markdown()`. `default_thresholds` = 0, 1 + and the quantiles of the per-stream maximum smoothed score. Refused by + name: a detector whose score count is not n // hop, a score that is not a + real number (a complex or bool array is never cast) or lies outside + [0, 1], a threshold outside [0, 1] or repeated, scores for a stream the + list does not carry, and on the CLI a `--grid` of −1 or 1. + +**The report** (`report.json` + `report.md`, version 1): the manifest name and +hash; `eval_set_id` per share from the lock (and `holdout_set_id`); the +scoring numbers; the detector name and `params`; the front end's +`log_mel_contract_version` and DspTap commit; hours per share with stream +counts and the negative-stream packing bound they were assembled with +(`max_stream_s`, printed as `packed at <= 60 s`); positives per subshare +(eval-speech / eval-tts / holdout); then one +row per threshold — recall with its Wilson 95 % interval, FRR, spurious +events, and **FA/h on speech, music, TTS speech and noise as separate +columns**, each cell `rate [Poisson lo, hi] (events in H)` and `<= ln 20 / H` +where a share saw no event — with the eval-tts recall printed beside the +hold-out recall as the plan's descriptive gap figure. A share the lock lacks +reads `absent`, never 0 FA/h. + +**Running it.** + +``` +# the sweep: the lock's eval shares through a detector +python tools/ml/kws/kws_eval.py sweep --manifest M --lock L --store DIR \ + [--detector band-energy] [--thresholds 50] [--grid N] [--smoothing 10] \ + [--refractory 100] [--max-stream-s 60] --out DIR +# the hold-out: verified (hashes, consent rows), then scored — beside the +# lock's shares when --lock is given +python tools/ml/kws/kws_eval.py holdout --holdout holdout.json --manifest M \ + --store DIR [--lock L] --out DIR +# the accounting of a lock alone +python tools/ml/kws/kws_streams.py --manifest M --lock L --store DIR --validate +# the DET notebook (executed and committed; needs requirements-notebook.txt) +python tools/ml/build_kws_det_notebook.py --manifest tools/ml/kws/manifests/speech_commands_v2_bringup.json +``` + +`--grid N` adds N evenly spaced thresholds to the quantiles: the band-energy +baseline's maximum saturates at 1.0 on most streams, so its quantiles +collapse (5 distinct thresholds from 50 quantiles on the bring-up corpus). +`--max-stream-s` (default `kws_streams.DEFAULT_MAX_STREAM_S` = 60) is +recorded in the report because every negative stream is decided after a +fresh reset: the *t* = 0 rule and the refractory restart at each stream +boundary make every FA/h row depend on the packing by at most streams / *H* +per share — the θ = 0 row, 60.65 FA/h on the bring-up speech share; the +decision-stage part of it measures 12–40 FA/h on speech (0.6–3.1 % of the +baseline's figure) at thresholds 0.1–0.9 against a run that decides the +concatenated per-stream scores, 9 September 2026. + +Measured 9 September 2026 on the M0 Mac, the bring-up corpus +(`speech_commands_v2_bringup`, 263,487 lock rows read in 1.7 s, the streams +assembled in 0.08 s): 412 streams — 195 eval-speech positives, 179 +eval-speech negative streams = **2.9513 h** (bit-identical to the lock's own +`summary.splits.eval.negative.hours`; the plan's "3.0 h" counted the +positives' 0.053 h, which the denominator excludes; zero-event bound 1.015 +FA/h) and 38 eval-noise streams = 0.5021 h (bound 5.97 FA/h; three MUSAN +files longer than 60 s stand alone, the longest 113 s); music and TTS +absent. `kws_eval.py sweep` 8.6 s wall (6.2 s scoring + sweep, one decode +per stream, 5 thresholds); with `--grid 21` 9.9 s (24 thresholds); the +baseline scores about 2,500 s of audio per second including decoding and +the accounting check, 8,400 s/s scoring alone; peak RSS 1.04 GB, the parsed +lock. `notebooks/kws_det.ipynb` executes in 14 s (44 thresholds; 30 s on a +first run, while matplotlib builds its font cache) and is 0.12 MB. The +baseline's best recall over the sweep is 34/195 at 1,977 FA/h on speech — +the sanity curve, as expected useless, not a pass. + +**The pass** (`test_kws_eval.py`, `test_kws_streams.py`; part of the +`kws-dataset` CI job, 44 tests in 2 s): the planted-event oracle — synthetic +silent streams scored by `PlantedDetector`, the per-utterance hit map +asserted through the harness's own `decide` → `hits` path with events inside +a window, at its two inclusive edges and one hop outside each (one at the +lower edge and one below, two at the upper edge and two above, so a rigid +shift of the window of up to 13 hops changes the aggregates as well as the +map), negative streams totalling exactly 0.5 h with four planted events of +which two fall inside one refractory period (3 counted → 6.0 FA/h, Poisson +[1.237, 17.535]), the refractory boundary as a number (events exactly *R* +hops apart both fire, *R* − 1 merge: 2 → 8.0 FA/h and 1 → 4.0 FA/h in 0.25 h), +one hit per utterance when two events fall in one window, a share with zero +events reporting ln 20 / *H*, spurious events never in FA/h, every figure +asserted against a hand computation, and the `_self_check`s of +`kws_scoring` and `kws_detectors` run under `unittest`; the mis-accounted +variants (a wrong, non-integer or negative `negative_samples`, an endpoint +or window beyond the stream, duplicate stream ids, a positive id used twice +within or across streams, a positive stream with hours) each refused by name +by both the harness and `validate_streams`, plus NaN / short / out-of-range +/ complex scores handed to `evaluate`, thresholds outside [0, 1] or +repeated, and a stream edited after scoring; the report round trip (the +packing bound included) and its markdown carrying every figure with its +hours; the toy rebuilt into a temporary store and swept end to end through +the bridge as a library call and through `kws_eval.py sweep` (hours = the +pcm tier's decoded lengths, bit-identical to `hours_per_share` and the +lock's summary, positives = the eval positives, `eval_set_id` = the lock's, +`max_stream_s` recorded, `--grid -1` and `--grid 1` refused by name, the +positive's frame count = `extra.frames` and its features = the committed +shard's rows); the band-energy alignment on a planted 1 kHz burst (peak at +or before hop *e* // 160 + 1, the score at *h* ≥ 0.32 and at *h* + 1 ≥ 0.96, +at most 0.979 at *h* + 2, 0 from *h* + 3, measured); the hold-out record +over soundfile-written FLACs (verify passes, one flipped byte refused naming +the file, the talker / consent / uses refusals, the set id changing when a +row is dropped and unchanged by a duplicate row, a second spelling or a +second row with one sha256 refused at load, two rows resolving to one file +refused by `verify_holdout`, a 22.05 kHz FLAC refused by name, and +`kws_eval.py holdout` writing a report with the record's set id then refusing +the altered tier with nothing written); and the three CLIs refusing a +missing store with `refused:` and rc 2. diff --git a/tools/ml/kws/kws_detectors.py b/tools/ml/kws/kws_detectors.py new file mode 100644 index 0000000..ea5781a --- /dev/null +++ b/tools/ml/kws/kws_detectors.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""kws_detectors — the detector contract the evaluation harness scores, and its two M5 detectors. + +A detector maps a 16 kHz stream to one score per completed front-end hop: score[t] belongs to frame t, +complete when sample (t + 1) * hop - 1 arrives (log_mel.h's alignment), so a stream of n samples yields +n // hop scores — the frame count the shipping front end returns for n samples (measured through the +bridge on 9 September 2026 for n in {0, 159, 160, 161, 400, 16000, 48000, 62555, 1000000}: exactly +n // 160 every time). Scores are float64 in [0, 1]. + +- `BandEnergyBaseline` is the plan's trivial sanity detector: the mean, over the mel bands whose centre + lies in [band_lo_hz, band_hi_hz], of the shipping front end's plain-log feature, clipped to [0, 1]. It + runs through the C ABI bridge (`kws_features.FrontEnd`), so it scores what ships. It is the sanity + curve, never the pass. +- `PlantedDetector` is the oracle's detector: zeros except the planted (hop, score) pairs per stream id, + so the harness's recall and FA/h can be checked against hand-computed values. +""" +from __future__ import annotations + +import dataclasses +import pathlib +import sys +from typing import Any, Protocol, Sequence + +import numpy as np + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +import kws_features # noqa: E402 + +DEFAULT_BAND_LO_HZ = 300.0 +DEFAULT_BAND_HI_HZ = 3000.0 + + +class DetectorError(ValueError): + """A detector configuration or call the harness refuses.""" + + +class Detector(Protocol): + """One score per completed hop, float64 in [0, 1], len = n_samples // hop.""" + + name: str + params: dict[str, Any] + + def score(self, x: np.ndarray) -> np.ndarray: ... + + +def frames_for(n_samples: int, hop: int) -> int: + """The front end's frame count for n_samples samples after a reset: n // hop (the module docstring).""" + if hop < 1: + raise DetectorError(f"hop must be positive, got {hop}") + return int(n_samples) // int(hop) + + +def band_centres_hz(g: kws_features.Geometry) -> np.ndarray: + """The centre frequency of every mel band: the reference's mel edges (HTK scale, bands + 2 points + from fmin to fmax), the inner ones being the triangle peaks log_mel.h uses.""" + ref = kws_features.reference_module() + edges = ref.mel_to_hz(np.linspace(ref.hz_to_mel(g.fmin_hz), ref.hz_to_mel(g.fmax_hz), g.bands + 2)) + return np.asarray(edges[1:-1], dtype=np.float64) + + +class BandEnergyBaseline: + """The trivial band-energy detector through the shipping front end's log path.""" + + name = "band-energy" + + def __init__(self, geometry: kws_features.Geometry, band_lo_hz: float = DEFAULT_BAND_LO_HZ, + band_hi_hz: float = DEFAULT_BAND_HI_HZ): + if not (0.0 <= band_lo_hz < band_hi_hz): + raise DetectorError(f"band [{band_lo_hz}, {band_hi_hz}] Hz must satisfy 0 <= lo < hi") + # the plain-log feature: the same geometry with PCEN off (the log affine keeps ~[0, 1] for band + # energies in [1e-5, 1] under the reference constants log_floor 1e-10, shift 5, scale 5) + log_geometry = dataclasses.replace(geometry, pcen=dataclasses.replace(geometry.pcen, enabled=False)) + self._fe = kws_features.FrontEnd(log_geometry) + centres = band_centres_hz(log_geometry) + bands = [b for b in range(log_geometry.bands) if band_lo_hz <= centres[b] <= band_hi_hz] + if not bands: + raise DetectorError(f"no mel band centre lies in [{band_lo_hz}, {band_hi_hz}] Hz at geometry " + f"{log_geometry.bands} bands {log_geometry.fmin_hz}-{log_geometry.fmax_hz} " + f"Hz (centres {np.round(centres, 1).tolist()})") + self.geometry = log_geometry + self.bands = bands + self.params: dict[str, Any] = { + "geometry": log_geometry.to_dict(), "stored_path": "log", + "band_lo_hz": float(band_lo_hz), "band_hi_hz": float(band_hi_hz), + "bands": list(bands), "band_centres_hz": [round(float(centres[b]), 3) for b in bands], + } + + @property + def hop(self) -> int: + return self.geometry.hop + + def features(self, x: np.ndarray) -> np.ndarray: + """The front end's (frames, bands) log features for a whole stream after one reset.""" + x = np.asarray(x, dtype=np.float64) + if x.ndim != 1: + raise DetectorError(f"a stream is one-dimensional, got shape {x.shape}") + return self._fe.extract(x) + + def score(self, x: np.ndarray) -> np.ndarray: + feats = self.features(x) + if feats.shape[0] == 0: + return np.zeros(0, dtype=np.float64) + return np.clip(feats[:, self.bands].mean(axis=1), 0.0, 1.0).astype(np.float64) + + +class PlantedDetector: + """Zeros except the planted (hop, score) pairs of each stream id; a stream absent from the plan scores + all zeros. `score_stream` refuses a planted hop at or beyond the stream's frame count.""" + + name = "planted" + + def __init__(self, planted: dict[str, Sequence[tuple[int, float]]], hop: int): + if hop < 1: + raise DetectorError(f"hop must be positive, got {hop}") + self.hop = int(hop) + self.planted: dict[str, list[tuple[int, float]]] = {} + for stream_id, pairs in planted.items(): + seen: set[int] = set() + out: list[tuple[int, float]] = [] + for pair in pairs: + if len(pair) != 2: + raise DetectorError(f"stream {stream_id!r}: expected (hop, score) pairs, got {pair!r}") + h, s = pair + if isinstance(h, bool) or not isinstance(h, (int, np.integer)) or h < 0: + raise DetectorError(f"stream {stream_id!r}: planted hop {h!r} must be a non-negative " + "integer") + if not (0.0 <= float(s) <= 1.0): + raise DetectorError(f"stream {stream_id!r}: planted score {s!r} at hop {h} is outside " + "[0, 1]") + if int(h) in seen: + raise DetectorError(f"stream {stream_id!r}: hop {h} is planted twice") + seen.add(int(h)) + out.append((int(h), float(s))) + self.planted[stream_id] = sorted(out) + planted_doc = {k: [[h, s] for h, s in v] for k, v in self.planted.items()} + self.params: dict[str, Any] = {"hop": self.hop, "planted": planted_doc} + + def score_stream(self, stream_id: str, n_samples: int) -> np.ndarray: + n = frames_for(n_samples, self.hop) + out = np.zeros(n, dtype=np.float64) + for h, s in self.planted.get(stream_id, []): + if h >= n: + raise DetectorError(f"stream {stream_id!r}: planted hop {h} is beyond the stream's {n} hops " + f"({n_samples} samples at hop {self.hop})") + out[h] = s + return out + + def score(self, x: np.ndarray) -> np.ndarray: + raise DetectorError("PlantedDetector scores by stream id: call score_stream(stream_id, n_samples)") + + +# ---------------------------------------------------------------- self-check: alignment on a planted burst + + +def _self_check() -> None: + """A 1 kHz burst ending at sample e in digital silence, through the bridge at the reference geometry. + + Measured 9 September 2026 on the M0 Mac (bursts ending at 24000, 24080, 24159, 24001 and 8400), with + h = e // 160: argmax(score) <= h + 1; the score at h is at least 0.3218 (a burst ending early in a hop + fills only part of frame h) and at h + 1 at least 0.9613 (the offset edge splatters across every + band); at h + 2 it is 0.0 except for the burst ending late in its hop (24159), which reads 0.979018; + from h + 3 on it is 0 again, and it is 0 before hop start // 160, the first frame that reaches the + burst's onset. The plain 1 kHz interior reads ~0.33 on the 22-band mean: a tone excites two bands. + """ + g = kws_features.Geometry() + det = BandEnergyBaseline(g) + assert det.bands == list(range(5, 27)), det.bands # centres 329.7 .. 2901.9 Hz at the reference geometry + hop = g.hop + for start, end in ((16000, 24000), (16000, 24159), (8000, 8400)): + x = np.zeros(48000) + t = np.arange(start, end) + x[start:end] = 0.5 * np.sin(2.0 * np.pi * 1000.0 * t / g.sample_rate) + s = det.score(x) + assert s.shape == (frames_for(x.size, hop),), s.shape + h = end // hop + assert int(np.argmax(s)) <= h + 1 + assert s[h] >= 0.3 and s[h + 1] >= 0.3, (end, h, s[h], s[h + 1]) # measured minima 0.3218, 0.9613 + assert s[h + 2] <= 0.99, (end, h, s[h + 2]) # measured 0.979018 for 24159, 0.0 for the others + assert np.all(s[h + 3:] == 0.0), (end, h) + assert np.all(s[:start // hop] == 0.0) and s[start // hop] > 0.0, (start, h) + pd = PlantedDetector({"a": [(3, 0.9), (10, 0.5)]}, hop) + got = pd.score_stream("a", 20 * hop) + assert got.shape == (20,) and got[3] == 0.9 and got[10] == 0.5 and got.sum() == 1.4 + assert pd.score_stream("b", 5 * hop).sum() == 0.0 + try: + pd.score_stream("a", 10 * hop) + except DetectorError as e: + assert "beyond" in str(e) + else: + raise AssertionError("a planted hop beyond the stream was not refused") + print("kws_detectors: self-check ok") + + +if __name__ == "__main__": + _self_check() diff --git a/tools/ml/kws/kws_eval.py b/tools/ml/kws/kws_eval.py new file mode 100644 index 0000000..ae956b0 --- /dev/null +++ b/tools/ml/kws/kws_eval.py @@ -0,0 +1,700 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""kws_eval — the evaluation harness: threshold sweep, per-share false accepts per hour, the committed +report (wake-word plan §6 M5). + +Scores every stream once (`detector.score`, or `PlantedDetector.score_stream` for the oracle), then per +threshold runs the reference decision stage of `kws_scoring` (`decide`): hits and spurious events on the +positive streams, events on the negative streams per share. Recall = hits / positives with its Wilson 95 % +interval, FRR = 1 - recall; per negative share events, hours H, FA/h, the exact two-sided 95 % Poisson +interval, and at zero events the one-sided bound ln 20 / H. A share absent from the streams is absent in +the report — never 0 FA/h. Every FA/h figure is printed with its H. + + kws_eval.py sweep --manifest M --lock L --store DIR [--detector band-energy] [--thresholds N] + [--grid N] [--smoothing W] [--refractory R] [--max-stream-s S] --out DIR + kws_eval.py holdout --holdout holdout.json --manifest M --store DIR [--lock L] ... --out DIR + +`sweep` assembles the eval streams from the lock (`kws_streams.streams_from_lock`), `holdout` verifies +the hold-out's file hashes and consent rows (`kws_holdout.verify_holdout`) and scores its utterances, +beside the lock's eval streams when `--lock` is given, so the hold-out recall lands beside the eval-tts +recall in one report. Both write DIR/report.json and DIR/report.md. + +The harness refuses a mis-accounted stream before any figure is computed (`check_stream`, which is +`kws_streams.validate_stream` applied to the audio the scoring pass decodes — one rule set, not a copy, so +a stream built by hand cannot mis-account and nothing is decoded twice), a detector whose score count is +not the stream's hop count, a score that is not a real number or lies outside [0, 1], a threshold outside +[0, 1] or repeated, and scores supplied for a stream it was not given. +""" +from __future__ import annotations + +import argparse +import dataclasses +import json +import math +import pathlib +import sys +import time +from collections.abc import Mapping +from typing import Any, Iterable, Iterator, Sequence + +import numpy as np + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +import kws_streams # noqa: E402 +from kws_manifest import check_keys # noqa: E402 +from kws_scoring import (Scoring, decide, hits, poisson_interval, smooth, spurious, # noqa: E402 + wilson_interval, zero_event_bound) +from kws_streams import (DEFAULT_MAX_STREAM_S, EVAL_SHARES, POSITIVE_SHARE, STREAM_SHARES, # noqa: E402 + SUBSHARES, Stream) + +REPORT_VERSION = 1 +# the report's FA/h columns, in this order (the plan: speech, music and TTS speech separately; then noise) +NEGATIVE_SHARES = ("eval-speech", "eval-music", "eval-tts", "eval-noise") +assert set(NEGATIVE_SHARES) == set(EVAL_SHARES) +DEFAULT_THRESHOLDS = 50 +ABSENT = "absent" + + +class EvalError(ValueError): + """A stream, detector output or report the harness refuses, with the reason.""" + + +def _check_keys(cls: type, d: Any, where: str) -> None: + check_keys(cls, d, where, EvalError) # kws_manifest's schema-key check, raising the harness's error + + +# ---------------------------------------------------------------- the report + + +@dataclasses.dataclass +class RecallFigure: + """Hits over positives with the Wilson 95 % interval; `recall` is None when there is no positive.""" + + hits: int + positives: int + recall: float | None + interval: list[float] | None + frr: float | None + + @classmethod + def of(cls, n_hits: int, n_positives: int) -> "RecallFigure": + if n_positives <= 0: + return cls(hits=0, positives=0, recall=None, interval=None, frr=None) + lo, hi = wilson_interval(n_hits, n_positives) + return cls(hits=int(n_hits), positives=int(n_positives), recall=n_hits / n_positives, + interval=[float(lo), float(hi)], frr=1.0 - n_hits / n_positives) + + @classmethod + def from_dict(cls, d: dict[str, Any], where: str) -> "RecallFigure": + _check_keys(cls, d, where) + return cls(hits=int(d["hits"]), positives=int(d["positives"]), + recall=None if d["recall"] is None else float(d["recall"]), + interval=None if d["interval"] is None else [float(v) for v in d["interval"]], + frr=None if d["frr"] is None else float(d["frr"])) + + +@dataclasses.dataclass +class ShareFigure: + """Events on one negative share: FA/h = events / hours with the exact Poisson 95 % interval, and at zero + events the one-sided 95 % bound ln 20 / H (None otherwise).""" + + events: int + hours: float + streams: int + fa_per_hour: float + interval: list[float] + zero_event_bound: float | None + + @classmethod + def of(cls, events: int, hours: float, streams: int) -> "ShareFigure": + lo, hi = poisson_interval(events, hours) + return cls(events=int(events), hours=float(hours), streams=int(streams), fa_per_hour=events / hours, + interval=[float(lo), float(hi)], + zero_event_bound=zero_event_bound(hours) if events == 0 else None) + + @classmethod + def from_dict(cls, d: dict[str, Any], where: str) -> "ShareFigure": + _check_keys(cls, d, where) + return cls(events=int(d["events"]), hours=float(d["hours"]), streams=int(d["streams"]), + fa_per_hour=float(d["fa_per_hour"]), interval=[float(v) for v in d["interval"]], + zero_event_bound=None if d["zero_event_bound"] is None else float(d["zero_event_bound"])) + + +@dataclasses.dataclass +class Row: + """One threshold of the sweep.""" + + threshold: float + recall: RecallFigure # over every positive + spurious: int # events on positive streams inside no window (never in FA/h) + by_subshare: dict[str, RecallFigure] # recall per subshare (eval-speech, eval-tts, holdout) present + shares: dict[str, ShareFigure] # per negative share present + + def to_dict(self) -> dict[str, Any]: + return {"threshold": self.threshold, "recall": dataclasses.asdict(self.recall), + "spurious": self.spurious, + "by_subshare": {k: dataclasses.asdict(v) for k, v in self.by_subshare.items()}, + "shares": {k: dataclasses.asdict(v) for k, v in self.shares.items()}} + + @classmethod + def from_dict(cls, d: dict[str, Any], where: str) -> "Row": + _check_keys(cls, d, where) + for k in ("by_subshare", "shares"): + if not isinstance(d[k], dict): + raise EvalError(f"{where}.{k}: expected an object") + return cls(threshold=float(d["threshold"]), + recall=RecallFigure.from_dict(d["recall"], f"{where}.recall"), spurious=int(d["spurious"]), + by_subshare={k: RecallFigure.from_dict(v, f"{where}.by_subshare[{k}]") + for k, v in d["by_subshare"].items()}, + shares={k: ShareFigure.from_dict(v, f"{where}.shares[{k}]") + for k, v in d["shares"].items()}) + + +@dataclasses.dataclass +class Report: + """The committed report: provenance, the scoring numbers, hours and counts, one Row per threshold.""" + + scoring: dict[str, Any] # Scoring.to_dict() + detector: dict[str, Any] # {"name": str, "params": dict} + hours: dict[str, float] # per negative share present: sum of negative_samples / rate / 3600 + negative_streams: dict[str, int] # per negative share present + positives: dict[str, int] # per subshare present + rows: list[Row] + manifest_name: str | None = None + manifest_hash: str | None = None + eval_set_id: dict[str, str] = dataclasses.field(default_factory=dict) # per share, from the lock + holdout_set_id: str | None = None + # {"log_mel_contract_version", "dsptap_commit"} of the front end the detector ran through + front_end: dict[str, Any] = dataclasses.field(default_factory=dict) + wall_s: float | None = None # measured wall time of the scoring + sweep, when the CLI ran it + # the negative-stream packing bound (kws_streams.streams_from_lock's max_stream_s) the streams were + # assembled with, when they came from a lock: every stream is decided after a fresh reset, so the + # t = 0 rule and the refractory restart make every FA/h row depend on it by at most streams / H + max_stream_s: float | None = None + report_version: int = REPORT_VERSION + + def to_dict(self) -> dict[str, Any]: + d = dataclasses.asdict(self) + d["rows"] = [r.to_dict() for r in self.rows] + return d + + @classmethod + def from_dict(cls, d: dict[str, Any]) -> "Report": + _check_keys(cls, d, "report") + if int(d["report_version"]) != REPORT_VERSION: + raise EvalError(f"report: report_version {d['report_version']}, expected {REPORT_VERSION}") + if not isinstance(d["rows"], list): + raise EvalError("report.rows: expected a list") + return cls(scoring=dict(d["scoring"]), detector=dict(d["detector"]), + hours={k: float(v) for k, v in d["hours"].items()}, + negative_streams={k: int(v) for k, v in d["negative_streams"].items()}, + positives={k: int(v) for k, v in d["positives"].items()}, + rows=[Row.from_dict(r, f"report.rows[{i}]") for i, r in enumerate(d["rows"])], + manifest_name=d.get("manifest_name"), manifest_hash=d.get("manifest_hash"), + eval_set_id=dict(d.get("eval_set_id") or {}), holdout_set_id=d.get("holdout_set_id"), + front_end=dict(d.get("front_end") or {}), + wall_s=None if d.get("wall_s") is None else float(d["wall_s"]), + max_stream_s=None if d.get("max_stream_s") is None else float(d["max_stream_s"]), + report_version=int(d["report_version"])) + + def markdown(self) -> str: + return markdown(self) + + +def format_hours(hours: float) -> str: + return f"{hours:.4g} h" + + +def format_recall(f: RecallFigure | None) -> str: + if f is None or f.positives == 0 or f.recall is None or f.interval is None: + return ABSENT + return f"{f.recall:.3f} [{f.interval[0]:.3f}, {f.interval[1]:.3f}] ({f.hits}/{f.positives})" + + +def format_fa(f: ShareFigure | None) -> str: + """FA/h with its interval, the zero-event bound when there was no event, and always its hours.""" + if f is None: + return ABSENT + bound = f" <= {f.zero_event_bound:.2f}" if f.zero_event_bound is not None else "" + return (f"{f.fa_per_hour:.2f} [{f.interval[0]:.2f}, {f.interval[1]:.2f}]{bound} " + f"({f.events} in {format_hours(f.hours)})") + + +def markdown(r: Report) -> str: + """The committed table: one row per threshold; speech, music, TTS speech and noise FA/h as separate + columns (`absent` when the streams had no such share), eval-tts recall beside the hold-out recall.""" + s = r.scoring + lines = ["# KWS evaluation report", ""] + lines.append(f"- manifest: {r.manifest_name or ABSENT}" + + (f" (`{r.manifest_hash}`)" if r.manifest_hash else "")) + lines.append(f"- detector: `{r.detector.get('name', '?')}` " + f"{json.dumps(r.detector.get('params', {}), sort_keys=True)}") + lines.append(f"- scoring: hop {s['hop']} at {s['sample_rate']} Hz; T = {s['tolerance_hops']} hops, " + f"L = {s['latency_hops']} hops (hit window [h - T, h + L + T]); " + f"W = {s['smoothing_hops']} hops, R = {s['refractory_hops']} hops") + fe = r.front_end + lines.append(f"- front end: log_mel_contract_version {fe.get('log_mel_contract_version', ABSENT)}, " + f"DspTap `{fe.get('dsptap_commit', ABSENT)}`") + ids = ", ".join(f"{k} `{v}`" for k, v in sorted(r.eval_set_id.items())) or ABSENT + lines.append(f"- eval_set_id: {ids}") + lines.append(f"- holdout_set_id: `{r.holdout_set_id}`" if r.holdout_set_id + else f"- holdout_set_id: {ABSENT}") + packed = f", packed at <= {r.max_stream_s:g} s" if r.max_stream_s is not None else "" + hours = ", ".join(f"{sh} H = {format_hours(r.hours[sh])} ({r.negative_streams.get(sh, 0)} streams" + f"{packed})" if sh in r.hours else f"{sh} {ABSENT}" for sh in NEGATIVE_SHARES) + lines.append(f"- hours: {hours}") + pos = ", ".join(f"{sub} {r.positives[sub]}" if sub in r.positives else f"{sub} {ABSENT}" + for sub in SUBSHARES) + lines.append(f"- positives: {pos}") + if r.wall_s is not None: + lines.append(f"- wall time: {r.wall_s:.1f} s (scoring and sweep)") + lines.append("") + lines.append("| threshold | recall [95 %] (hits/positives) | FRR | spurious | FA/h speech | FA/h music " + "| FA/h TTS | FA/h noise | recall eval-tts | recall hold-out |") + lines.append("|---|---|---|---|---|---|---|---|---|---|") + for row in r.rows: + frr = ABSENT if row.recall.frr is None else f"{row.recall.frr:.3f}" + # the threshold as its shortest exact repr: quantiles 1 - 1e-8 apart stay distinct + cells = [repr(row.threshold), format_recall(row.recall), frr, str(row.spurious)] + cells += [format_fa(row.shares.get(sh)) for sh in NEGATIVE_SHARES] + cells += [format_recall(row.by_subshare.get("eval-tts")), + format_recall(row.by_subshare.get("holdout"))] + lines.append("| " + " | ".join(cells) + " |") + lines.append("") + lines.append("FA/h cells read `rate [Poisson 95 % lo, hi] (events in H)`; at zero events `<= ln 20 / H` " + "is the one-sided 95 % bound. Spurious events lie on positive streams outside every hit " + "window and never enter FA/h. Every negative stream is decided after a fresh reset, so each " + "FA/h row depends on the stream packing by at most streams / H (the threshold-0 row). The " + "eval-tts and hold-out recalls stand beside each other as a descriptive gap figure, never a " + "pass.") + return "\n".join(lines) + "\n" + + +# ---------------------------------------------------------------- streams and scores + + +def _samples(stream: Stream) -> np.ndarray: + """The stream's audio (`Stream.samples`, decoded on every call when lazy), its refusals as EvalErrors.""" + try: + return stream.samples() + except kws_streams.StreamError as e: + raise EvalError(str(e)) from None + + +def _detector_name(detector: Any) -> str: + return str(getattr(detector, "name", type(detector).__name__)) + + +def _require_stream(s: Any) -> None: + if not isinstance(s, Stream): + raise EvalError(f"expected a Stream, got {type(s).__name__}") + + +def check_stream(s: Stream, n_samples: int, scoring: Scoring, seen_positives: set[str] | None = None) -> None: + """One stream's accounting against its decoded length: `kws_streams.validate_stream` (one rule set, not + a copy), its StreamError re-raised as EvalError. Refuses, by stream id: an unknown share; a non-integer + negative_samples; an empty stream; a positive stream without positives, with negative_samples != 0 or + with a subshare outside SUBSHARES; a positive whose endpoint lies beyond its audio, whose endpoint_hop + is not the endpoint's hop, whose window ends beyond the stream's hops, or whose id is already in + `seen_positives` (one hit per utterance needs unique ids across the list); a negative stream with + positives or whose negative_samples != n_samples.""" + _require_stream(s) + try: + kws_streams.validate_stream(s, n_samples, scoring, seen_positives) + except kws_streams.StreamError as e: + raise EvalError(str(e)) from None + + +def check_streams(streams: Sequence[Stream], scoring: Scoring) -> dict[str, int]: + """The harness's accounting preconditions over a stream list — no streams at all, a duplicate stream + id, and `check_stream` per stream against its decoded audio with one seen-positives set; returns the + sample count per stream id. Decodes every stream once: `score_streams` makes the same checks on the + audio it decodes to score, so this is the path for scores supplied as a bare dict.""" + if not streams: + raise EvalError("no streams to score") + n_samples: dict[str, int] = {} + seen_positives: set[str] = set() + for s in streams: + _require_stream(s) + if s.id in n_samples: + raise EvalError(f"duplicate stream id {s.id!r}") + n = int(_samples(s).size) + check_stream(s, n, scoring, seen_positives) + n_samples[s.id] = n + return n_samples + + +def check_scores(stream_id: str, y: Any, n_samples: int, scoring: Scoring, detector_name: str = "?") \ + -> np.ndarray: + """The score contract, refused by stream and detector name: a one-dimensional array of real numbers + (a float or integer dtype; complex, bool, object and string arrays are refused rather than cast, since + a dropped imaginary part would otherwise surface only as numpy's warning) with exactly + n_samples // hop finite values in [0, 1]. Returns the scores as float64.""" + y = np.asarray(y) + if not (np.issubdtype(y.dtype, np.floating) or np.issubdtype(y.dtype, np.integer)): + raise EvalError(f"stream {stream_id!r}: detector {detector_name!r} returned dtype {y.dtype}; scores " + "must be real numbers (float64 in [0, 1])") + y = y.astype(np.float64) + n_hops = int(n_samples) // scoring.hop + if y.ndim != 1 or y.size != n_hops: + raise EvalError(f"stream {stream_id!r}: detector {detector_name!r} returned {y.shape} scores for " + f"{n_hops} hops ({n_samples} samples at hop {scoring.hop})") + if y.size and (not np.all(np.isfinite(y)) or y.min() < 0.0 or y.max() > 1.0): + raise EvalError(f"stream {stream_id!r}: scores must be finite and within [0, 1]") + return y + + +def score_stream(stream: Stream, detector: Any, scoring: Scoring, audio: np.ndarray | None = None) \ + -> np.ndarray: + """One score per completed hop: `detector.score_stream(id, n)` when the detector plants scores, else + `detector.score(audio)`; the result checked by `check_scores`. `audio` is the stream's samples when + the caller already loaded them (lazy streams decode on every `samples()` call).""" + if audio is None: + audio = _samples(stream) + n_samples = int(audio.size) + if hasattr(detector, "score_stream"): + y = detector.score_stream(stream.id, n_samples) + else: + y = detector.score(audio) + return check_scores(stream.id, y, n_samples, scoring, _detector_name(detector)) + + +class Scored(Mapping): + """What `score_streams` returns: the scores keyed by stream id — a read-only mapping, so + `scored[id]`, `scored.values()` and `default_thresholds(scored, ...)` read as with a dict — plus the + sample count every stream was checked and scored at. `evaluate` takes it as already verified (each + stream was decoded once, its accounting checked against that decode and the score contract enforced) + and re-runs only the accounting rules against the recorded counts, without decoding, so a stream + edited after scoring still cannot mis-account.""" + + def __init__(self, scores: dict[str, np.ndarray], n_samples: dict[str, int]): + if set(scores) != set(n_samples): + raise EvalError("Scored: scores and sample counts must cover the same stream ids") + self.scores = dict(scores) + self.n_samples = {k: int(v) for k, v in n_samples.items()} + + def __getitem__(self, stream_id: str) -> np.ndarray: + return self.scores[stream_id] + + def __iter__(self) -> Iterator[str]: + return iter(self.scores) + + def __len__(self) -> int: + return len(self.scores) + + +def score_streams(streams: Sequence[Stream], detector: Any, scoring: Scoring) -> Scored: + """Every stream decoded once, checked (`check_stream` with one seen-positives set; no streams at all + and a duplicate stream id refused) and scored (`score_stream`).""" + if not streams: + raise EvalError("no streams to score") + scores: dict[str, np.ndarray] = {} + n_samples: dict[str, int] = {} + seen_positives: set[str] = set() + for s in streams: + _require_stream(s) + if s.id in scores: + raise EvalError(f"duplicate stream id {s.id!r}") + x = _samples(s) + check_stream(s, int(x.size), scoring, seen_positives) + scores[s.id] = score_stream(s, detector, scoring, audio=x) + n_samples[s.id] = int(x.size) + return Scored(scores, n_samples) + + +def default_thresholds(stream_scores: Iterable[np.ndarray] | Mapping[str, np.ndarray], scoring: Scoring, + n: int = DEFAULT_THRESHOLDS) -> list[float]: + """0, 1 and the n quantiles (linspace(0, 1, n)) of the per-stream maximum smoothed score, deduplicated + and sorted — so the sweep's thresholds land where the streams' scores are. Each is clipped to [0, 1] + and rounded to 12 decimals: a moving average of scores at 1.0 carries ~1e-14 of summation noise + (measured max 1.0000000000000455 over the 412 bring-up streams, 92 of them above 1.0 and 302 at + exactly 1.0, BandEnergyBaseline, W = 10, 9 September 2026), which is not a second threshold.""" + if n < 2: + raise EvalError(f"default_thresholds: n must be at least 2, got {n}") + if isinstance(stream_scores, Mapping): + stream_scores = stream_scores.values() + maxes = [float(smooth(y, scoring.smoothing_hops).max()) for y in stream_scores if np.asarray(y).size] + values = {0.0, 1.0} + if maxes: + values.update(threshold_value(q) for q in np.quantile(np.asarray(maxes), np.linspace(0.0, 1.0, n))) + return sorted(values) + + +def threshold_value(v: float) -> float: + """A sweep threshold: clipped to [0, 1], rounded to 12 decimals (float noise is not a threshold).""" + return min(1.0, max(0.0, round(float(v), 12))) + + +# ---------------------------------------------------------------- the sweep + + +PROVENANCE_FIELDS = ("manifest_name", "manifest_hash", "eval_set_id", "holdout_set_id", "front_end", + "wall_s", "max_stream_s") + + +def _verified_scores(streams: Sequence[Stream], detector: Any, scoring: Scoring, + scores: Scored | Mapping[str, Any] | None) -> Scored: + """The scores `evaluate` sweeps, every stream's accounting checked: scored here when None; a `Scored` + re-checked against its recorded sample counts without decoding; a bare mapping re-checked against a + fresh decode (`check_streams`) and put through the score contract (`check_scores`). Scores for a + stream id the list does not carry, or a stream without scores, are refused.""" + if scores is None: + return score_streams(streams, detector, scoring) + if isinstance(scores, Scored): + n_samples = scores.n_samples + if not streams: + raise EvalError("no streams to score") + seen_ids: set[str] = set() + seen_positives: set[str] = set() + for s in streams: + _require_stream(s) + if s.id in seen_ids: + raise EvalError(f"duplicate stream id {s.id!r}") + seen_ids.add(s.id) + if s.id in n_samples: + check_stream(s, n_samples[s.id], scoring, seen_positives) + else: + n_samples = check_streams(streams, scoring) + missing = [s.id for s in streams if s.id not in scores] + if missing: + raise EvalError(f"no scores for stream(s) {missing[:5]}") + unknown = sorted(set(scores) - {s.id for s in streams}) + if unknown: + raise EvalError(f"scores for unknown stream id(s) {unknown[:5]}") + if isinstance(scores, Scored): + return scores + name = _detector_name(detector) + return Scored({s.id: check_scores(s.id, scores[s.id], n_samples[s.id], scoring, name) for s in streams}, + n_samples) + + +def evaluate(streams: Sequence[Stream], detector: Any, scoring: Scoring, thresholds: Sequence[float], + scores: Scored | Mapping[str, Any] | None = None, + provenance: dict[str, Any] | None = None) -> Report: + """Score once, sweep the thresholds, and fill the report. + + `scores`: None scores the streams here (`score_streams`); the `Scored` that `score_streams` returned is + taken as verified and only its accounting is re-checked against the recorded sample counts (no second + decode); a bare mapping of id -> scores is re-checked in full (the streams decoded once, the score + contract enforced). `thresholds` must be finite, within [0, 1] (scores are) and distinct. + `provenance` carries the report's header fields (PROVENANCE_FIELDS: manifest_name, manifest_hash, + eval_set_id, holdout_set_id, front_end, wall_s, max_stream_s). + """ + thresholds = [float(t) for t in thresholds] + if not thresholds or not all(math.isfinite(t) for t in thresholds): + raise EvalError(f"thresholds must be a non-empty list of finite numbers, got {thresholds}") + outside = [t for t in thresholds if not 0.0 <= t <= 1.0] + if outside: + raise EvalError(f"thresholds must lie in [0, 1] (scores do), got {outside}") + if len(set(thresholds)) != len(thresholds): + repeated = sorted({t for t in thresholds if thresholds.count(t) > 1}) + raise EvalError(f"duplicate threshold(s) {repeated}") + prov = dict(provenance or {}) + unknown = sorted(set(prov) - set(PROVENANCE_FIELDS)) + if unknown: + raise EvalError(f"provenance: unknown field(s) {unknown}") + scored = _verified_scores(streams, detector, scoring, scores) + positive_streams = [s for s in streams if s.share == POSITIVE_SHARE] + negative_streams = [s for s in streams if s.share != POSITIVE_SHARE] + # hours: the integer sample sum per share divided once, as kws_streams.hours_per_share and the lock's + # summary do (a running float sum differs from them by an ulp) + samples: dict[str, int] = {} + n_negative: dict[str, int] = {} + for s in negative_streams: + samples[s.share] = samples.get(s.share, 0) + int(s.negative_samples) + n_negative[s.share] = n_negative.get(s.share, 0) + 1 + hours = {sh: scoring.hours(n) for sh, n in samples.items()} + positives: dict[str, int] = {} + for s in positive_streams: + positives[s.subshare] = positives.get(s.subshare, 0) + len(s.positives) + # the smoothed maximum bounds the events: below it no crossing exists, so the sweep skips those streams + smoothed_max = {sid: (float(smooth(y, scoring.smoothing_hops).max()) if y.size else -math.inf) + for sid, y in scored.items()} + rows: list[Row] = [] + for theta in sorted(thresholds): + n_hits = 0 + n_spurious = 0 + sub_hits: dict[str, int] = {} + for s in positive_streams: + ev = decide(scored[s.id], scoring, theta) if smoothed_max[s.id] >= theta else [] + h = sum(hits(ev, s.positives, scoring).values()) + n_hits += h + sub_hits[s.subshare] = sub_hits.get(s.subshare, 0) + h + n_spurious += spurious(ev, s.positives, scoring) + events: dict[str, int] = {} + for s in negative_streams: + ev = decide(scored[s.id], scoring, theta) if smoothed_max[s.id] >= theta else [] + events[s.share] = events.get(s.share, 0) + len(ev) + rows.append(Row(threshold=theta, recall=RecallFigure.of(n_hits, sum(positives.values())), + spurious=n_spurious, + by_subshare={sub: RecallFigure.of(sub_hits.get(sub, 0), positives[sub]) + for sub in SUBSHARES if sub in positives}, + shares={sh: ShareFigure.of(events.get(sh, 0), hours[sh], n_negative[sh]) + for sh in NEGATIVE_SHARES if sh in hours})) + return Report(scoring=scoring.to_dict(), + detector={"name": _detector_name(detector), + "params": dict(getattr(detector, "params", {}) or {})}, + hours={sh: hours[sh] for sh in NEGATIVE_SHARES if sh in hours}, + negative_streams={sh: n_negative[sh] for sh in NEGATIVE_SHARES if sh in hours}, + positives={sub: positives[sub] for sub in SUBSHARES if sub in positives}, rows=rows, **prov) + + +def write_report(report: Report, out: pathlib.Path) -> tuple[pathlib.Path, pathlib.Path]: + out.mkdir(parents=True, exist_ok=True) + json_path, md_path = out / "report.json", out / "report.md" + json_path.write_text(json.dumps(report.to_dict(), indent=1, sort_keys=True) + "\n", encoding="utf-8") + md_path.write_text(report.markdown(), encoding="utf-8") + return json_path, md_path + + +def read_report(path: pathlib.Path) -> Report: + with pathlib.Path(path).open(encoding="utf-8") as f: + return Report.from_dict(json.load(f)) + + +# ---------------------------------------------------------------- CLI + + +def _check_args(args: argparse.Namespace) -> None: + """The sweep options refused by name before anything is read or decoded.""" + if args.grid < 0 or args.grid == 1: + raise EvalError(f"--grid must be 0 (off) or at least 2, got {args.grid}") + if args.thresholds < 2: + raise EvalError(f"--thresholds must be at least 2, got {args.thresholds}") + if not args.max_stream_s > 0.0: + raise EvalError(f"--max-stream-s must be positive, got {args.max_stream_s}") + + +def _scoring_from(manifest: Any, args: argparse.Namespace) -> Scoring: + g = manifest.recipe.geometry + return Scoring(hop=int(g.hop), sample_rate=int(round(g.sample_rate)), + tolerance_hops=int(manifest.recipe.label.tolerance_hops), + smoothing_hops=int(args.smoothing), refractory_hops=int(args.refractory)) + + +def _detector_from(manifest: Any, name: str) -> Any: + import kws_detectors # B's module; imported here so the report and the oracle need no bridge + if name == "band-energy": + return kws_detectors.BandEnergyBaseline(manifest.recipe.geometry) + raise EvalError(f"unknown detector {name!r} (known: band-energy)") + + +def _front_end_record() -> dict[str, Any]: + import kws_features + return {"log_mel_contract_version": kws_features.contract_version(), + "dsptap_commit": kws_features.dsptap_commit()} + + +def _run(streams: list[Stream], detector: Any, scoring: Scoring, args: argparse.Namespace, + provenance: dict[str, Any]) -> Report: + """One decode per stream: `score_streams` checks every stream's accounting (the rules of + kws_streams.validate_streams) on the audio it scores, and `evaluate` takes its `Scored` as verified.""" + t0 = time.time() + scored = score_streams(streams, detector, scoring) + thresholds = default_thresholds(scored, scoring, n=int(args.thresholds)) + if args.grid: # a detector whose maxima saturate collapses the quantiles; an even grid fills the curve + grid = {threshold_value(v) for v in np.linspace(0.0, 1.0, int(args.grid))} + thresholds = sorted(set(thresholds) | grid) + report = evaluate(streams, detector, scoring, thresholds, scores=scored, provenance=provenance) + report.wall_s = time.time() - t0 + return report + + +def cmd_sweep(args: argparse.Namespace) -> int: + from kws_manifest import load_manifest, manifest_hash, read_lock + from kws_store import Store, resolve_store + _check_args(args) + manifest = load_manifest(args.manifest) + lock = read_lock(args.lock) + if lock.manifest_hash != manifest_hash(manifest): + raise EvalError(f"lock {args.lock} was built from manifest hash {lock.manifest_hash}, the manifest's " + f"is {manifest_hash(manifest)}") + store = Store(resolve_store(args.store)) + scoring = _scoring_from(manifest, args) + streams = kws_streams.streams_from_lock(manifest, lock, store, scoring, + max_stream_s=float(args.max_stream_s)) + detector = _detector_from(manifest, args.detector) + provenance = {"manifest_name": manifest.name, "manifest_hash": lock.manifest_hash, + "eval_set_id": dict(lock.eval_set_id), "holdout_set_id": lock.holdout_set_id, + "front_end": _front_end_record(), "max_stream_s": float(args.max_stream_s)} + report = _run(streams, detector, scoring, args, provenance) + json_path, md_path = write_report(report, pathlib.Path(args.out)) + print(f"kws_eval: {len(streams)} streams, {len(report.rows)} thresholds, {report.wall_s:.1f} s -> " + f"{json_path}, {md_path}") + return 0 + + +def cmd_holdout(args: argparse.Namespace) -> int: + import kws_holdout + from kws_manifest import load_manifest, manifest_hash, read_lock + from kws_store import Store, resolve_store + _check_args(args) + manifest = load_manifest(args.manifest) + store = Store(resolve_store(args.store)) + scoring = _scoring_from(manifest, args) + holdout = kws_holdout.load_holdout(args.holdout) + kws_holdout.verify_holdout(holdout, store) + streams = kws_holdout.streams_from_holdout(holdout, store, scoring) + provenance: dict[str, Any] = {"manifest_name": manifest.name, "manifest_hash": manifest_hash(manifest), + "holdout_set_id": kws_holdout.holdout_set_id(holdout), + "front_end": _front_end_record()} + if args.lock: + lock = read_lock(args.lock) + if lock.manifest_hash != manifest_hash(manifest): + raise EvalError(f"lock {args.lock} was built from manifest hash {lock.manifest_hash}, the " + f"manifest's is {manifest_hash(manifest)}") + streams += kws_streams.streams_from_lock(manifest, lock, store, scoring, + max_stream_s=float(args.max_stream_s)) + provenance["eval_set_id"] = dict(lock.eval_set_id) + provenance["max_stream_s"] = float(args.max_stream_s) + detector = _detector_from(manifest, args.detector) + report = _run(streams, detector, scoring, args, provenance) + json_path, md_path = write_report(report, pathlib.Path(args.out)) + print(f"kws_eval: hold-out {provenance['holdout_set_id'][:12]}, {len(streams)} streams, " + f"{len(report.rows)} thresholds, {report.wall_s:.1f} s -> {json_path}, {md_path}") + return 0 + + +def _common(ap: argparse.ArgumentParser) -> None: + ap.add_argument("--store", default=None, help="the feature store (or MUTAP_KWS_STORE); no default") + ap.add_argument("--detector", default="band-energy", help="band-energy (the sanity baseline)") + ap.add_argument("--thresholds", type=int, default=DEFAULT_THRESHOLDS, + help=f"quantile count of default_thresholds (default {DEFAULT_THRESHOLDS})") + ap.add_argument("--grid", type=int, default=0, + help="also sweep N evenly spaced thresholds on [0, 1] (default 0: the quantiles only)") + ap.add_argument("--smoothing", type=int, default=Scoring.smoothing_hops, + help=f"W, the trailing moving average in hops (default {Scoring.smoothing_hops})") + ap.add_argument("--refractory", type=int, default=Scoring.refractory_hops, + help=f"R, the refractory period in hops (default {Scoring.refractory_hops})") + ap.add_argument("--max-stream-s", type=float, default=DEFAULT_MAX_STREAM_S, + help="negative clips are concatenated into streams of at most this many seconds " + f"(default {DEFAULT_MAX_STREAM_S:g}, kws_streams.DEFAULT_MAX_STREAM_S; recorded in " + "the report)") + ap.add_argument("--out", required=True, help="directory for report.json and report.md") + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + sub = ap.add_subparsers(dest="command", required=True) + sweep = sub.add_parser("sweep", help="the eval shares of a lock through a detector") + sweep.add_argument("--manifest", required=True) + sweep.add_argument("--lock", required=True) + _common(sweep) + hold = sub.add_parser("holdout", + help="verify a hold-out and score it (with the lock's eval shares if given)") + hold.add_argument("--holdout", required=True, help="holdout.json") + hold.add_argument("--manifest", required=True, help="for the geometry and the tolerance T") + hold.add_argument("--lock", default=None, help="score the lock's eval shares in the same report") + _common(hold) + args = ap.parse_args(argv) + try: + return cmd_sweep(args) if args.command == "sweep" else cmd_holdout(args) + except (EvalError, ValueError, RuntimeError, OSError) as e: + print(f"kws_eval: {type(e).__name__}: {e}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/ml/kws/kws_holdout.py b/tools/ml/kws/kws_holdout.py new file mode 100644 index 0000000..4e5a7a0 --- /dev/null +++ b/tools/ml/kws/kws_holdout.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""kws_holdout — the recorded hold-out's record (M4c's `holdout.json`), verified before it is scored. + +`holdout.json` (version 1) is the committed record of the hold-out: its talkers (pseudonym, consent form +version, permitted uses) and its utterances (a FLAC path relative to `/holdout/`, the file's sha256, +the talker, the microphone path, distance, SNR, phrase and the 16 kHz endpoint sample annotated on the +close-microphone take). The audio never enters git; the store's `holdout/` tier holds the FLACs. + +The plan's rule (§6 M5): the harness "refuses a hold-out whose file hashes do not match the committed +holdout.json rows". `load_holdout` refuses any schema deviation by name, like `kws_manifest`; +`verify_holdout` refuses, by file, a FLAC that is missing or whose sha256 differs from its row, a FLAC +under the tier that no row names, two rows that resolve to one file, and any utterance whose talker row +is missing, lacks a consent_form_version, or whose permitted_uses do not include both "evaluation" and +"m7-replay". One file is one utterance: a path is accepted only in its canonical spelling (no '.' or '..' +segment, no '//'), and a sha256 listed under two rows is refused at load. +`holdout_set_id` is the sha256 over the sorted per-utterance FLAC hashes — the lock's hold-out set id. +`streams_from_holdout` makes one positive stream per utterance (share "positives", subshare "holdout"), +decoded through soundfile; a FLAC that is not 16 kHz is refused rather than resampled here (the hold-out +is decoded to the pcm tier by the builder's one resampler, never by the harness). + + python3 tools/ml/kws/kws_holdout.py --holdout holdout.json --store DIR # verify, print the set id +""" +from __future__ import annotations + +import argparse +import dataclasses +import hashlib +import json +import math +import pathlib +import sys +from typing import Any + +import numpy as np + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +import kws_audio # noqa: E402 +from kws_manifest import check_keys # noqa: E402 +from kws_scoring import Positive, Scoring # noqa: E402 +from kws_store import Store, StoreError, resolve_store, sha256_file # noqa: E402 +from kws_streams import HOLDOUT_SHARE, POSITIVE_SHARE, Stream # noqa: E402 + +HOLDOUT_JSON_VERSION = 1 +REQUIRED_USES = ("evaluation", "m7-replay") +FLAC_SUFFIX = ".flac" + + +class HoldoutError(ValueError): + """A hold-out record or tier the harness refuses; the message names the file, talker or field.""" + + +def _is_real(v: Any) -> bool: + return isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v) + + +def _is_int(v: Any) -> bool: + return isinstance(v, int) and not isinstance(v, bool) + + +@dataclasses.dataclass(frozen=True) +class Talker: + pseudonym: str + consent_form_version: str | None = None # verify_holdout refuses a talker without one + permitted_uses: tuple[str, ...] = () # verify_holdout requires REQUIRED_USES + + +@dataclasses.dataclass(frozen=True) +class Utterance: + file: str # relative FLAC path under /holdout/ + sha256: str + talker: str # a talkers[] pseudonym + microphone_path: str + distance_m: float + snr_db: float | None # None for a clean (unmixed) take + phrase: str + endpoint_sample: int # 16 kHz sample index of the keyword end (the trim rule on the close take) + + @property + def id(self) -> str: + return f"{HOLDOUT_SHARE}/{self.file[:-len(FLAC_SUFFIX)]}" + + +@dataclasses.dataclass(frozen=True) +class Holdout: + talkers: tuple[Talker, ...] + utterances: tuple[Utterance, ...] + holdout_version: int = HOLDOUT_JSON_VERSION + path: pathlib.Path | None = None + + def talker(self, pseudonym: str) -> Talker | None: + for t in self.talkers: + if t.pseudonym == pseudonym: + return t + return None + + def to_dict(self) -> dict[str, Any]: + return {"holdout_version": self.holdout_version, + "talkers": [{**dataclasses.asdict(t), "permitted_uses": list(t.permitted_uses)} + for t in self.talkers], + "utterances": [dataclasses.asdict(u) for u in self.utterances]} + + @classmethod + def from_dict(cls, d: Any, path: pathlib.Path | None = None) -> "Holdout": + if not isinstance(d, dict): + raise HoldoutError(f"holdout: expected an object, got {type(d).__name__}") + unknown = sorted(set(d) - {"holdout_version", "talkers", "utterances"}) + if unknown: + raise HoldoutError(f"holdout: unknown top-level field(s) {unknown}") + missing = sorted(k for k in ("holdout_version", "talkers", "utterances") if k not in d) + if missing: + raise HoldoutError(f"holdout: missing top-level field(s) {missing}") + if d["holdout_version"] != HOLDOUT_JSON_VERSION: + raise HoldoutError(f"holdout_version {d['holdout_version']!r}, expected {HOLDOUT_JSON_VERSION}") + for key in ("talkers", "utterances"): + if isinstance(d[key], (str, dict)) or not isinstance(d[key], list): + raise HoldoutError(f"holdout.{key}: expected a list") + talkers = [] + for i, t in enumerate(d["talkers"]): + where = f"talker {t.get('pseudonym', i)!r}" if isinstance(t, dict) else f"talkers[{i}]" + check_keys(Talker, t, where, HoldoutError) + uses = t.get("permitted_uses", []) + if isinstance(uses, str) or not isinstance(uses, list) \ + or not all(isinstance(u, str) for u in uses): + raise HoldoutError(f"{where}.permitted_uses: expected a list of strings, got {uses!r}") + version = t.get("consent_form_version") + if version is not None and not isinstance(version, str): + raise HoldoutError(f"{where}.consent_form_version: expected a string, got {version!r}") + if not isinstance(t["pseudonym"], str) or not t["pseudonym"]: + raise HoldoutError(f"{where}: pseudonym must be a non-empty string") + talkers.append(Talker(pseudonym=t["pseudonym"], consent_form_version=version, + permitted_uses=tuple(uses))) + utterances = [] + for i, u in enumerate(d["utterances"]): + where = f"utterance {u.get('file', i)!r}" if isinstance(u, dict) else f"utterances[{i}]" + check_keys(Utterance, u, where, HoldoutError) + utterances.append(Utterance(**u)) + return cls(talkers=tuple(talkers), utterances=tuple(utterances), + holdout_version=int(d["holdout_version"]), path=path) + + +def _check_file(where: str, file: Any) -> None: + """A relative POSIX path in its one canonical spelling: no '..' or '.' segment, no empty segment + (leading '/', '//', trailing '/'), no backslash — checked on the raw string, since PurePosixPath drops + '.' segments and collapses '//', so two spellings of one file could otherwise pass as two rows.""" + if not isinstance(file, str) or not file or "\\" in file: + raise HoldoutError(f"{where}: file must be a relative POSIX path under the hold-out tier (no '..', " + f"no '.', no leading '/', no '\\\\'), got {file!r}") + segments = file.split("/") + if any(seg in ("", ".", "..") for seg in segments) \ + or file != pathlib.PurePosixPath(file).as_posix(): + raise HoldoutError(f"{where}: file must be a relative POSIX path under the hold-out tier (no '..', " + f"no '.', no leading '/', no '\\\\'), got {file!r}") + if not file.endswith(FLAC_SUFFIX): + raise HoldoutError(f"{where}: file {file!r} is not a {FLAC_SUFFIX} master") + + +def validate(h: Holdout) -> None: + """Every schema-level rule: types, canonical paths, hex digests, unique files (by spelling and by + digest) and pseudonyms.""" + seen_talkers: set[str] = set() + for t in h.talkers: + if t.pseudonym in seen_talkers: + raise HoldoutError(f"talker {t.pseudonym!r}: duplicate pseudonym") + seen_talkers.add(t.pseudonym) + seen_files: set[str] = set() + seen_digests: dict[str, str] = {} + for u in h.utterances: + where = f"utterance {u.file!r}" + _check_file(where, u.file) + if u.file in seen_files: + raise HoldoutError(f"{where}: listed twice") + seen_files.add(u.file) + if not isinstance(u.sha256, str) or len(u.sha256) != 64 \ + or any(c not in "0123456789abcdef" for c in u.sha256): + raise HoldoutError(f"{where}: sha256 must be 64 lowercase hex characters") + # one file, one utterance: the same bytes under a second spelling (a case-folded path on a + # case-insensitive filesystem, which resolve() does not unify) would double the recall + # denominator and hash twice into the set id + if u.sha256 in seen_digests: + raise HoldoutError(f"{where}: sha256 already recorded for utterance {seen_digests[u.sha256]!r} " + "(one file is one utterance)") + seen_digests[u.sha256] = u.file + for name in ("talker", "microphone_path", "phrase"): + if not isinstance(getattr(u, name), str) or not getattr(u, name): + raise HoldoutError(f"{where}: {name} must be a non-empty string") + if not _is_real(u.distance_m) or u.distance_m < 0.0: + raise HoldoutError(f"{where}: distance_m must be a non-negative number, got {u.distance_m!r}") + if u.snr_db is not None and not _is_real(u.snr_db): + raise HoldoutError(f"{where}: snr_db must be a finite number or null, got {u.snr_db!r}") + if not _is_int(u.endpoint_sample) or u.endpoint_sample < 0: + raise HoldoutError(f"{where}: endpoint_sample must be a non-negative integer (16 kHz), got " + f"{u.endpoint_sample!r}") + + +def load_holdout(path: str | pathlib.Path) -> Holdout: + path = pathlib.Path(path) + with path.open(encoding="utf-8") as f: + doc = json.load(f) + try: + h = Holdout.from_dict(doc, path=path) + except HoldoutError: + raise + except (TypeError, KeyError, AttributeError) as e: + raise HoldoutError(f"{path}: does not match holdout.json version {HOLDOUT_JSON_VERSION}: " + f"{e}") from None + validate(h) + return h + + +def save_holdout(h: Holdout, path: str | pathlib.Path) -> None: + text = json.dumps(h.to_dict(), indent=1, ensure_ascii=False) + "\n" + pathlib.Path(path).write_text(text, encoding="utf-8") + + +# ---------------------------------------------------------------- the tier against the record + + +def holdout_path(store: Store, u: Utterance) -> pathlib.Path: + tier = store.holdout().resolve() + p = (tier / u.file) + if not p.resolve().is_relative_to(tier) or p.resolve() == tier: + raise HoldoutError(f"utterance {u.file!r}: escapes the hold-out tier {tier}") + return p + + +def verify_holdout(holdout: Holdout, store: Store) -> None: + """Refuse, by file or talker, a tier that does not match the record or a talker without consent.""" + tier = store.holdout() + if not tier.is_dir(): + raise HoldoutError(f"hold-out tier {tier} does not exist — the FLAC masters are placed there by " + "hand, never downloaded") + if not holdout.utterances: + raise HoldoutError("holdout.json lists no utterances") + listed: dict[pathlib.Path, str] = {} + for u in holdout.utterances: + p = holdout_path(store, u) + resolved = p.resolve() + if resolved in listed: + raise HoldoutError(f"utterance {u.file!r}: names the same file as utterance " + f"{listed[resolved]!r} ({resolved})") + listed[resolved] = u.file + if not p.is_file(): + raise HoldoutError(f"utterance {u.file!r}: {p} is missing from the hold-out tier") + digest = sha256_file(p) + if digest != u.sha256: + raise HoldoutError(f"utterance {u.file!r}: sha256 {digest} != the recorded {u.sha256} — the " + "file under the hold-out tier is not the one holdout.json describes") + t = holdout.talker(u.talker) + if t is None: + raise HoldoutError(f"utterance {u.file!r}: talker {u.talker!r} has no talkers[] row") + if not t.consent_form_version: + raise HoldoutError(f"utterance {u.file!r}: talker {u.talker!r} has no consent_form_version") + lacking = [use for use in REQUIRED_USES if use not in t.permitted_uses] + if lacking: + raise HoldoutError(f"utterance {u.file!r}: talker {u.talker!r} permitted_uses " + f"{list(t.permitted_uses)} lack {lacking}") + stray = sorted(str(p.relative_to(tier.resolve())) for p in tier.resolve().rglob("*" + FLAC_SUFFIX) + if p.resolve() not in listed) + if stray: + raise HoldoutError(f"hold-out tier {tier} holds {len(stray)} FLAC(s) no holdout.json row names: " + f"{stray[:5]}{' ...' if len(stray) > 5 else ''}") + + +def holdout_set_id(holdout: Holdout) -> str: + """sha256 over the sorted per-utterance FLAC sha256s, one per line, each file once (validate() refuses + a digest listed twice; the set here keeps the id defined if it was bypassed): the plan's hold-out set + id.""" + return hashlib.sha256("\n".join(sorted({u.sha256 for u in holdout.utterances})).encode()).hexdigest() + + +def streams_from_holdout(holdout: Holdout, store: Store, scoring: Scoring) -> list[Stream]: + """One positive stream per utterance (the room take as recorded), share "positives", subshare + "holdout"; the audio is decoded lazily through soundfile and must already be 16 kHz.""" + streams: list[Stream] = [] + for u in holdout.utterances: + p = holdout_path(store, u) + positive = Positive(id=u.id, endpoint_sample=int(u.endpoint_sample), + endpoint_hop=scoring.endpoint_hop(int(u.endpoint_sample))) + + def load(p: pathlib.Path = p, u: Utterance = u) -> np.ndarray: + x, fs = kws_audio.read_audio(p) + if fs != kws_audio.RATE: + raise HoldoutError(f"utterance {u.file!r}: {fs} Hz, the harness scores {kws_audio.RATE} Hz " + "audio (its endpoint_sample is a 16 kHz index); decode the hold-out to " + "the pcm tier with the builder's resampler first") + return np.asarray(x, dtype=np.float64) + + streams.append(Stream(id=u.id, share=POSITIVE_SHARE, positives=[positive], negative_samples=0, + subshare=HOLDOUT_SHARE, load=load, members=[u.file])) + return streams + + +# ---------------------------------------------------------------- CLI + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Verify a hold-out record against the store's hold-out tier " + "and print its set id.") + ap.add_argument("--holdout", required=True, help="holdout.json") + ap.add_argument("--store", help="the feature store (or $MUTAP_KWS_STORE); no default") + args = ap.parse_args(argv) + try: + h = load_holdout(args.holdout) + store = Store(resolve_store(args.store)) + verify_holdout(h, store) + except (HoldoutError, StoreError, ValueError, OSError) as e: + print(f"refused: {e}", file=sys.stderr) + return 2 + print(f"hold-out verified: {len(h.utterances)} utterances from {len(h.talkers)} talkers under " + f"{store.holdout()}; set id {holdout_set_id(h)}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/ml/kws/kws_manifest.py b/tools/ml/kws/kws_manifest.py index 0133e8b..6fafb47 100644 --- a/tools/ml/kws/kws_manifest.py +++ b/tools/ml/kws/kws_manifest.py @@ -52,19 +52,24 @@ class ManifestError(ValueError): """A manifest the builder refuses, with the reason.""" -def _check_keys(cls: type, d: dict[str, Any], where: str) -> None: - """Refuse a dict whose keys are not exactly the dataclass's fields (unknown keys, missing required).""" +def check_keys(cls: type, d: Any, where: str, error: type[ValueError] = ManifestError) -> None: + """Refuse a dict whose keys are not exactly the dataclass's fields (unknown keys, missing required), + raising `error` (ManifestError here; kws_holdout and kws_eval pass their own class — M5's additive + change so the three modules share one schema-key check).""" if not isinstance(d, dict): - raise ManifestError(f"{where}: expected an object, got {type(d).__name__}") + raise error(f"{where}: expected an object, got {type(d).__name__}") fields = {f.name: f for f in dataclasses.fields(cls)} unknown = sorted(set(d) - set(fields)) if unknown: - raise ManifestError(f"{where}: unknown field(s) {unknown} (known: {sorted(fields)})") + raise error(f"{where}: unknown field(s) {unknown} (known: {sorted(fields)})") required = sorted(n for n, f in fields.items() if f.default is dataclasses.MISSING and f.default_factory is dataclasses.MISSING and n not in d) if required: - raise ManifestError(f"{where}: missing required field(s) {required}") + raise error(f"{where}: missing required field(s) {required}") + + +_check_keys = check_keys # the manifest's own call sites # ---------------------------------------------------------------- sources diff --git a/tools/ml/kws/kws_scoring.py b/tools/ml/kws/kws_scoring.py new file mode 100644 index 0000000..2069f7e --- /dev/null +++ b/tools/ml/kws/kws_scoring.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""kws_scoring — the evaluation harness's scoring semantics, defined as numbers (wake-word plan §6 M5). + +- A detector yields one score per completed front-end hop: score[t] belongs to frame t, which is complete + when sample (t + 1) * hop - 1 arrives (log_mel.h's alignment), so an endpoint at sample e has hop index + h = e // hop and the first frame that can see the whole keyword is h. +- The hit window of a positive is frames [h - T, h + L + T], both ends inclusive: T the manifest's + endpoint tolerance in hops, L the §7 detection-latency ceiling (20 hops = 200 ms). One hit per utterance. +- The reference decision stage (the numbers `kws.h` carries at M6 and must match, pinned there): the score + is smoothed by a trailing moving average over W hops, an event fires at hop t when the smoothed score + crosses the threshold upward (s[t] >= theta and s[t-1] < theta, or t = 0 and s[0] >= theta) and at + least R hops have passed since the previous event (t - last >= R): a crossing exactly R hops after an + event fires, one R - 1 hops after it merges into it — the refractory period, under which false accepts + merge. +- A false accept is an event on a negative stream; FA/h = events / H with H the negative streams' decoded + duration in hours. Every FA/h carries H, the exact two-sided 95 % Poisson interval on the count + (chi-square form), and at zero events the one-sided 95 % upper bound ln 20 / H. Recall carries its + Wilson 95 % interval. + +Checked against hand-computed values in `_self_check` (run this file). +""" +from __future__ import annotations + +import dataclasses +import math +from typing import Sequence + +import numpy as np +from scipy import stats + +LATENCY_CEILING_HOPS = 20 # the §7 ceiling: detection within 20 hops (200 ms at hop 160 / 16 kHz) + + +@dataclasses.dataclass(frozen=True) +class Scoring: + """The scoring numbers; everything a report prints beside its figures.""" + + hop: int = 160 # samples per hop at the manifest geometry + sample_rate: int = 16000 + tolerance_hops: int = 3 # T (the manifest's label rule) + latency_hops: int = LATENCY_CEILING_HOPS # L + smoothing_hops: int = 10 # W: trailing moving average (100 ms at the reference hop) + refractory_hops: int = 100 # R: 1 s at the reference hop + + def __post_init__(self) -> None: + if self.hop < 1 or self.sample_rate < 1 or self.tolerance_hops < 0 or self.latency_hops < 0 \ + or self.smoothing_hops < 1 or self.refractory_hops < 1: + raise ValueError(f"invalid scoring numbers: {self}") + + def endpoint_hop(self, endpoint_sample: int) -> int: + return int(endpoint_sample) // self.hop + + def hit_window(self, endpoint_hop: int) -> tuple[int, int]: + """[h - T, h + L + T], inclusive.""" + return endpoint_hop - self.tolerance_hops, endpoint_hop + self.latency_hops + self.tolerance_hops + + def hours(self, samples: int) -> float: + return samples / self.sample_rate / 3600.0 + + def to_dict(self) -> dict: + return dataclasses.asdict(self) + + +@dataclasses.dataclass(frozen=True) +class Positive: + """One keyword utterance inside a stream: its id and its endpoint (stream-relative sample and hop).""" + + id: str + endpoint_sample: int + endpoint_hop: int + + +def smooth(scores: np.ndarray, window: int) -> np.ndarray: + """Trailing moving average over `window` hops (shorter at the start: the mean of what exists).""" + s = np.asarray(scores, dtype=np.float64) + if s.ndim != 1: + raise ValueError("scores must be one value per hop") + if window <= 1 or s.size == 0: + return s.copy() + c = np.cumsum(np.concatenate([[0.0], s])) + n = np.arange(1, s.size + 1) + lo = np.maximum(0, n - window) + return (c[n] - c[lo]) / (n - lo) + + +def decide(scores: np.ndarray, scoring: Scoring, threshold: float) -> list[int]: + """Event hops under the reference decision stage: upward threshold crossings of the smoothed score + that come at least `refractory_hops` (R) after the previous event — t - last >= R, so a crossing + exactly R hops after an event fires and one R - 1 hops after it merges.""" + s = smooth(scores, scoring.smoothing_hops) + events: list[int] = [] + last = -scoring.refractory_hops - 1 + prev_above = False + for t in range(s.size): + above = bool(s[t] >= threshold) + if above and not prev_above and t - last >= scoring.refractory_hops: + events.append(t) + last = t + prev_above = above + return events + + +def hits(events: Sequence[int], positives: Sequence[Positive], scoring: Scoring) -> dict[str, bool]: + """One hit per utterance: whether any event falls inside its window.""" + out: dict[str, bool] = {} + ev = np.asarray(sorted(events), dtype=np.int64) + for p in positives: + lo, hi = scoring.hit_window(p.endpoint_hop) + i = int(np.searchsorted(ev, lo, side="left")) + out[p.id] = bool(i < ev.size and ev[i] <= hi) + return out + + +def spurious(events: Sequence[int], positives: Sequence[Positive], scoring: Scoring) -> int: + """Events on a positive stream that fall inside no utterance's window (reported, never in FA/h).""" + n = 0 + for t in events: + if not any(lo <= t <= hi for lo, hi in (scoring.hit_window(p.endpoint_hop) for p in positives)): + n += 1 + return n + + +# ---------------------------------------------------------------- intervals + + +def poisson_interval(events: int, hours: float, confidence: float = 0.95) -> tuple[float, float]: + """Exact two-sided Poisson interval on the rate (events per hour), chi-square form: + lower = chi2.ppf(a/2, 2k) / 2 / H (0 at k = 0), upper = chi2.ppf(1 - a/2, 2k + 2) / 2 / H.""" + if hours <= 0.0: + raise ValueError("the hours denominator must be positive") + if events < 0: + raise ValueError("events must be non-negative") + a = 1.0 - confidence + lo = 0.0 if events == 0 else float(stats.chi2.ppf(a / 2.0, 2 * events)) / 2.0 / hours + hi = float(stats.chi2.ppf(1.0 - a / 2.0, 2 * events + 2)) / 2.0 / hours + return lo, hi + + +def zero_event_bound(hours: float, confidence: float = 0.95) -> float: + """The one-sided upper bound on the rate when no event was seen: -ln(1 - confidence) / H = ln 20 / H.""" + if hours <= 0.0: + raise ValueError("the hours denominator must be positive") + return -math.log(1.0 - confidence) / hours + + +def wilson_interval(successes: int, trials: int, confidence: float = 0.95) -> tuple[float, float]: + """Wilson score interval for a proportion.""" + if trials <= 0: + raise ValueError("trials must be positive") + if not 0 <= successes <= trials: + raise ValueError("successes must be within [0, trials]") + z = float(stats.norm.ppf(1.0 - (1.0 - confidence) / 2.0)) + p = successes / trials + denom = 1.0 + z * z / trials + centre = (p + z * z / (2.0 * trials)) / denom + half = z * math.sqrt(p * (1.0 - p) / trials + z * z / (4.0 * trials * trials)) / denom + return max(0.0, centre - half), min(1.0, centre + half) + + +# ---------------------------------------------------------------- self-check against hand-computed values + + +def _self_check() -> None: + sc = Scoring() + assert sc.endpoint_hop(9000) == 56 and sc.hit_window(56) == (53, 79) + # smoothing: the mean of the last W values, shorter at the start + s = smooth(np.array([1.0, 0.0, 0.0, 0.0]), 2) + assert np.allclose(s, [1.0, 0.5, 0.0, 0.0]) + # decision: an upward crossing fires once; a second crossing inside R is merged; after R it fires again + x = np.zeros(400) + x[10:20] = 1.0 + x[30:40] = 1.0 # inside the refractory period of the first event + x[200:210] = 1.0 # after it + ev = decide(x, Scoring(smoothing_hops=1), 0.5) + assert ev == [10, 200], ev + # the refractory boundary as a number: a crossing exactly R = 100 hops after an event fires, one at + # R - 1 hops merges + x = np.zeros(400) + x[10], x[110] = 1.0, 1.0 + assert decide(x, Scoring(smoothing_hops=1), 0.5) == [10, 110] + x = np.zeros(400) + x[10], x[109] = 1.0, 1.0 + assert decide(x, Scoring(smoothing_hops=1), 0.5) == [10] + # hits: the window edges are inclusive, one hit per utterance + p = [Positive("a", 9000, 56), Positive("b", 20000, 125)] + assert hits([53], p, sc) == {"a": True, "b": False} + assert hits([52], p, sc) == {"a": False, "b": False} + assert hits([79, 80], p, sc)["a"] is True and hits([80], p, sc)["a"] is False + assert spurious([52, 53, 300], p, sc) == 2 + # intervals: k = 3 events in 0.5 h -> 6.0 FA/h, exact 95 % [1.24, 17.53] (chi2 0.025,6 = 1.237; 0.975,8 = 17.535) + lo, hi = poisson_interval(3, 0.5) + assert abs(lo - 1.2373) < 0.01, lo # chi2(0.025, 6) / 2 = 0.6187 per 0.5 h + assert abs(hi - 17.535) < 0.01, hi + # k = 0 in 3.0 h: one-sided bound ln 20 / 3 = 0.9986 FA/h; two-sided upper chi2(0.975, 2)/2/3 = 1.229 + assert abs(zero_event_bound(3.0) - 0.99858) < 1e-4 + assert abs(poisson_interval(0, 3.0)[1] - 1.2296) < 1e-3 + # Wilson: 190/200 -> [0.9104, 0.9726]; 1/1 -> [0.2065, 1.0] + lo, hi = wilson_interval(190, 200) + assert abs(lo - 0.9104) < 5e-4 and abs(hi - 0.9726) < 5e-4, (lo, hi) + lo, hi = wilson_interval(1, 1) + assert abs(lo - 0.2065) < 5e-4 and hi == 1.0, (lo, hi) + print("kws_scoring: self-check ok") + + +if __name__ == "__main__": + _self_check() diff --git a/tools/ml/kws/kws_streams.py b/tools/ml/kws/kws_streams.py new file mode 100644 index 0000000..52da046 --- /dev/null +++ b/tools/ml/kws/kws_streams.py @@ -0,0 +1,358 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""kws_streams — the audio streams the evaluation harness scores, assembled from a lock (plan §6 M5). + +A `Stream` is one continuously running deployment: the detector scores it after a single reset, so the +smoothing window and the refractory period run across clip boundaries the way they do in a room. + +- **Positive streams**: every variant-0 eval row with label 1 becomes one stream whose audio is the row's + mixture — `kws_build.mixture` over the row's resolved draw, i.e. the raw concatenation of the drawn + context and the keyword that `extract` featurized, so the stream has exactly the lock's `extra.frames` + hops and the hit window [h - T, h + 20 + T] around the row's endpoint lies inside them. One `Positive` + per stream; `negative_samples` is 0: a positive stream never enters the hours denominator. +- **Negative streams**: per eval share, that share's variant-0 rows with label 0 (eval-speech, eval-music, + eval-tts) or label None (eval-noise), sorted by id and concatenated in that order into streams of at + most `max_stream_s` seconds, a clip never split (a clip longer than `max_stream_s` stands alone, so the + bound is on the packing, not on a single clip). `negative_samples` is the stream's length — the sum of + its members' decoded lengths, the plan's FA/h denominator: hours = negative_samples / 16000 / 3600. + +Audio is loaded lazily (`Stream.load`) so a corpus of hours is never resident at once; `samples()` decodes +on every call and caches nothing. `validate_streams` decodes every stream once and refuses, by name, any +accounting that does not match the audio; its per-stream rules are `validate_stream`, which takes the +decoded length, so a caller that already holds the audio (kws_eval's scoring pass) applies the same rules +without a second decode. Refusals are `StreamError`s; nothing here warns. + +Measured 9 September 2026 on the M0 Mac (Apple silicon, CPython 3.12, numpy 2.5.3, soundfile 0.14.0), +the bring-up corpus (speech_commands_v2_bringup, 263,487 lock rows): the figures are in the M5 section +of README.md and in test_kws_streams.py's docstring; the toy fixture assembles in well under 0.1 s. +""" +from __future__ import annotations + +import argparse +import dataclasses +import pathlib +import sys +import time +from typing import Any, Callable, Sequence + +import numpy as np + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +import kws_audio # noqa: E402 +import kws_build # noqa: E402 +from kws_manifest import Clip, Lock, Manifest, load_manifest, manifest_hash, read_lock # noqa: E402 +from kws_scoring import Positive, Scoring # noqa: E402 +from kws_store import Store, StoreError, resolve_store # noqa: E402 + +EVAL_SHARES = ("eval-speech", "eval-music", "eval-noise", "eval-tts") +POSITIVE_SHARE = "positives" +HOLDOUT_SHARE = "holdout" # a subshare and an id prefix: a hold-out utterance is a positive stream +STREAM_SHARES = (POSITIVE_SHARE, *EVAL_SHARES) # there is no hold-out negative share +SUBSHARES = ("eval-speech", "eval-tts", HOLDOUT_SHARE) # where a positive stream's utterance came from +NEGATIVE_LABELS = {"eval-speech": 0, "eval-music": 0, "eval-tts": 0, "eval-noise": None} +DEFAULT_MAX_STREAM_S = 60.0 + + +class StreamError(ValueError): + """A stream, or a stream list, the harness refuses; the message names the stream and the rule.""" + + +@dataclasses.dataclass +class Stream: + """One continuously scored stream: its audio (eager or lazy), its positives and its hours accounting.""" + + id: str # unique across the list + share: str # one of STREAM_SHARES + positives: list[Positive] # stream-relative endpoints; empty for a negative stream + negative_samples: int # samples in the hours denominator (0 for a positive stream) + subshare: str | None = None # a positive stream: one of SUBSHARES (required there) + audio: np.ndarray | None = None # float64, 16 kHz + load: Callable[[], np.ndarray] | None = None # lazy alternative to `audio` + members: list[str] = dataclasses.field(default_factory=list) # the lock rows the audio is made of + + def samples(self) -> np.ndarray: + """The stream's audio as a one-dimensional float64 array (decoded on every call when lazy).""" + if self.audio is not None: + x = np.asarray(self.audio, dtype=np.float64) + elif self.load is not None: + x = np.asarray(self.load(), dtype=np.float64) + else: + raise StreamError(f"stream {self.id!r}: has neither audio nor a loader") + if x.ndim != 1: + raise StreamError(f"stream {self.id!r}: audio must be one-dimensional, got shape {x.shape}") + return x + + +# ---------------------------------------------------------------- assembly from a lock + + +def require_scoring_matches(manifest: Manifest, scoring: Scoring) -> None: + """The scoring numbers the lock's rows were cut for: hop and rate from the geometry, T from the label + rule, L the ceiling `extract` reserved after every positive. Any other value scores windows the lock + never guaranteed to fit inside a positive's rows, so it is refused rather than reinterpreted.""" + g = manifest.recipe.geometry + if scoring.hop != g.hop: + raise StreamError(f"scoring.hop {scoring.hop} != the manifest geometry's hop {g.hop}") + if scoring.sample_rate != kws_audio.RATE or int(g.sample_rate) != kws_audio.RATE: + raise StreamError(f"scoring.sample_rate {scoring.sample_rate} and geometry.sample_rate " + f"{g.sample_rate:g} must both be the pcm tier's {kws_audio.RATE} Hz") + want_t = manifest.recipe.label.tolerance_hops + if scoring.tolerance_hops != want_t: + raise StreamError(f"scoring.tolerance_hops {scoring.tolerance_hops} != the manifest's label " + f"tolerance {want_t} (the hit window is a manifest rule, plan §6 M4 \"Label " + "data\")") + if scoring.latency_hops != kws_build.LATENCY_CEILING_HOPS: + raise StreamError(f"scoring.latency_hops {scoring.latency_hops} != the " + f"{kws_build.LATENCY_CEILING_HOPS} hops `extract` reserved after every positive " + "(the §7 ceiling)") + + +def trailing_samples(manifest: Manifest) -> int: + """`extract`'s n_trail: (L + T) hops plus one frame of context after the keyword.""" + g = manifest.recipe.geometry + return (kws_build.LATENCY_CEILING_HOPS + manifest.recipe.label.tolerance_hops) * g.hop + g.frame + + +def _pack(rows: Sequence[Clip], max_samples: int) -> list[list[Clip]]: + """Greedy in the given order: a clip joins the open group unless that would exceed max_samples; a + clip that alone exceeds it forms its own group (never split).""" + groups: list[list[Clip]] = [] + have = 0 + for c in rows: + if groups and have + c.length > max_samples: + groups.append([]) + have = 0 + if not groups: + groups.append([]) + groups[-1].append(c) + have += c.length + return groups + + +def streams_from_lock(manifest: Manifest, lock: Lock, store: Store, scoring: Scoring, + shares: Sequence[str] = EVAL_SHARES, + max_stream_s: float = DEFAULT_MAX_STREAM_S) -> list[Stream]: + """The positive streams (every eval positive of the selected shares) followed by the negative streams + of each selected share, in `shares` order. Audio is lazy; nothing is decoded here.""" + require_scoring_matches(manifest, scoring) + unknown = [s for s in shares if s not in EVAL_SHARES] + if unknown or not shares: + raise StreamError(f"shares {list(shares)}: must be a non-empty subset of {list(EVAL_SHARES)}") + if len(set(shares)) != len(shares): + raise StreamError(f"shares {list(shares)}: a share is named twice") + if not (max_stream_s > 0.0): + raise StreamError(f"max_stream_s must be positive, got {max_stream_s!r}") + want = manifest_hash(manifest) + if lock.manifest_hash != want: + raise StreamError(f"the lock was built from manifest {lock.manifest_hash}, this manifest hashes to " + f"{want}: assemble streams from the lock of the manifest whose store tiers exist") + build = kws_build.Build(manifest, store) + n_trail = trailing_samples(manifest) + clips = {c.id: c for c in lock.clips if c.variant == 0} + eval_rows = [c for c in clips.values() if c.split == "eval" and c.share in shares] + # kws_build's loaders take dict rows; only the eval rows and the rows their draws reference are ever + # read, so only those are converted (asdict over every variant-0 row of the bring-up lock, 166,758 + # rows, took 0.83 s of a 0.9 s assembly, measured 9 September 2026; the needed 11,105 take 0.05 s) + by_id: dict[str, dict[str, Any]] = {} + + def row_of(clip: Clip) -> dict[str, Any]: + if clip.id not in by_id: + by_id[clip.id] = dataclasses.asdict(clip) + return by_id[clip.id] + + streams: list[Stream] = [] + for c in sorted((c for c in eval_rows if c.label == 1), key=lambda c: c.id): + if c.endpoint_sample is None: + raise StreamError(f"row {c.id}: an eval positive without an endpoint_sample") + if c.share not in SUBSHARES: + raise StreamError(f"row {c.id}: a positive in share {c.share!r}; positives come from " + f"{list(SUBSHARES)}") + row = row_of(c) + refs = kws_build.draw_refs(row.get("draw")) + missing = [i for i in refs if i not in clips] + if missing: + raise StreamError(f"row {c.id}: its draw names {missing}, not variant-0 rows of the lock") + for i in refs: + row_of(clips[i]) + positive = Positive(id=c.id, endpoint_sample=int(c.endpoint_sample), + endpoint_hop=scoring.endpoint_hop(int(c.endpoint_sample))) + + def load_positive(row: dict[str, Any] = row) -> np.ndarray: + y = kws_build._read_clip(build, row) + return kws_build.mixture(build, y, row, by_id, n_trail) + + streams.append(Stream(id=c.id, share=POSITIVE_SHARE, positives=[positive], negative_samples=0, + subshare=c.share, load=load_positive, members=[c.id])) + max_samples = int(round(max_stream_s * kws_audio.RATE)) + for share in shares: + want_label = NEGATIVE_LABELS[share] + rows = sorted((c for c in eval_rows if c.share == share and c.label != 1), key=lambda c: c.id) + for c in rows: + if c.label != want_label: + raise StreamError(f"row {c.id}: label {c.label!r} in share {share!r}; a {share} negative " + f"carries label {want_label!r}") + if c.length <= 0: + raise StreamError(f"row {c.id}: decoded length {c.length} — nothing to score") + for n, group in enumerate(_pack(rows, max_samples)): + member_rows = [row_of(c) for c in group] + + def load_negative(member_rows: list[dict[str, Any]] = member_rows) -> np.ndarray: + return np.concatenate([kws_build._read_clip(build, r) for r in member_rows]) + + streams.append(Stream(id=f"{share}/stream-{n:04d}", share=share, positives=[], + negative_samples=int(sum(c.length for c in group)), load=load_negative, + members=[c.id for c in group])) + return streams + + +# ---------------------------------------------------------------- accounting + + +def hours_per_share(streams: Sequence[Stream], scoring: Scoring) -> dict[str, float]: + """The hours denominator of every negative share present: sum of negative_samples / rate / 3600.""" + samples: dict[str, int] = {} + for s in streams: + if s.share != POSITIVE_SHARE: + samples[s.share] = samples.get(s.share, 0) + int(s.negative_samples) + return {share: scoring.hours(n) for share, n in samples.items()} + + +def positives_per_subshare(streams: Sequence[Stream]) -> dict[str, int]: + counts: dict[str, int] = {} + for s in streams: + if s.share == POSITIVE_SHARE: + key = s.subshare or "unspecified" + counts[key] = counts.get(key, 0) + len(s.positives) + return counts + + +def validate_stream(s: Stream, n_samples: int, scoring: Scoring, + seen_positives: set[str] | None = None) -> None: + """One stream's accounting against its decoded length `n_samples`; every per-stream rule of + `validate_streams` (which decodes and calls this), so a caller already holding the audio applies the + same rules without a second decode. Refuses, by stream id: an unknown share; a negative_samples that + is not an integer; an empty stream; a positive stream with negative_samples != 0, with no positives, + or with a subshare outside SUBSHARES; a positive whose endpoint lies at or beyond the audio, whose + endpoint_hop disagrees with its endpoint_sample, whose window ends beyond the stream's hops, or whose + id is already in `seen_positives` (added here, so one set threaded through a list refuses an id used + twice within a stream or across streams); a negative stream with positives or whose negative_samples + != n_samples.""" + if s.share not in STREAM_SHARES: + raise StreamError(f"stream {s.id!r}: unknown share {s.share!r} (known: {list(STREAM_SHARES)})") + if isinstance(s.negative_samples, bool) or not isinstance(s.negative_samples, (int, np.integer)): + raise StreamError(f"stream {s.id!r}: negative_samples must be an integer, got " + f"{s.negative_samples!r}") + n = int(n_samples) + if n == 0: + raise StreamError(f"stream {s.id!r}: empty stream (no audio)") + hops = n // scoring.hop + if s.share == POSITIVE_SHARE: + if s.negative_samples != 0: + raise StreamError(f"stream {s.id!r}: a positive stream with negative_samples " + f"{s.negative_samples} (it must be 0: positives never enter the hours " + "denominator)") + if not s.positives: + raise StreamError(f"stream {s.id!r}: a positive stream with no positives") + if s.subshare not in SUBSHARES: + raise StreamError(f"stream {s.id!r}: subshare {s.subshare!r} is not one of {list(SUBSHARES)} " + "(a positive stream names the share its utterance came from)") + for p in s.positives: + if seen_positives is not None: + if p.id in seen_positives: + raise StreamError(f"stream {s.id!r}: positive {p.id!r} appears twice (one hit per " + "utterance needs unique ids)") + seen_positives.add(p.id) + if not 0 <= p.endpoint_sample < n: + raise StreamError(f"stream {s.id!r}: positive {p.id!r} endpoint sample " + f"{p.endpoint_sample} lies beyond its {n} samples of audio") + if p.endpoint_hop != scoring.endpoint_hop(p.endpoint_sample): + raise StreamError(f"stream {s.id!r}: positive {p.id!r} endpoint_hop {p.endpoint_hop} != " + f"{scoring.endpoint_hop(p.endpoint_sample)} = endpoint_sample " + f"{p.endpoint_sample} // {scoring.hop}") + lo, hi = scoring.hit_window(p.endpoint_hop) + if hi >= hops: + raise StreamError(f"stream {s.id!r}: positive {p.id!r} window [{lo}, {hi}] ends beyond " + f"the stream's {hops} hops (the (L + T) hops after the keyword are " + "missing)") + else: + if s.positives: + raise StreamError(f"stream {s.id!r}: a {s.share} stream carrying {len(s.positives)} " + "positive(s)") + if s.negative_samples != n: + raise StreamError(f"stream {s.id!r}: negative_samples {s.negative_samples} != its " + f"{n} decoded samples (the hours denominator is the decoded duration)") + + +def validate_streams(streams: Sequence[Stream], scoring: Scoring) -> None: + """Refuse, by name, a stream list whose accounting does not match its audio. + + Decodes every stream once. Refusals: no streams at all; a non-Stream; a duplicate stream id; and, + per stream, every rule of `validate_stream` (an unknown share; a non-integer negative_samples; an + empty stream; a positive stream with negative_samples != 0, with no positives, or with a subshare + outside SUBSHARES (None included); a positive whose endpoint lies at or beyond the end of the audio, + whose endpoint_hop disagrees with its endpoint_sample, or whose window [h - T, h + L + T] ends beyond + the stream's hops; a positive id used twice; a negative stream with positives, or whose + negative_samples != len(audio)). + """ + if not streams: + raise StreamError("no streams to score") + seen_ids: set[str] = set() + seen_positives: set[str] = set() + for s in streams: + if not isinstance(s, Stream): + raise StreamError(f"expected a Stream, got {type(s).__name__}") + if s.id in seen_ids: + raise StreamError(f"stream {s.id!r}: duplicate stream id") + seen_ids.add(s.id) + validate_stream(s, s.samples().size, scoring, seen_positives) + + +# ---------------------------------------------------------------- CLI: the accounting of a lock + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description="Assemble the evaluation streams of a lock and print the " + "per-share accounting (stream counts, hours from decoded " + "lengths).") + ap.add_argument("--manifest", required=True) + ap.add_argument("--lock", required=True, help="the lock.json the streams are assembled from") + ap.add_argument("--store", help="the feature store (or $MUTAP_KWS_STORE); no default") + ap.add_argument("--shares", nargs="+", default=list(EVAL_SHARES), help="the eval shares to assemble") + ap.add_argument("--max-stream-s", type=float, default=DEFAULT_MAX_STREAM_S) + ap.add_argument("--validate", action="store_true", + help="decode every stream and check the accounting against the audio") + args = ap.parse_args(argv) + try: + manifest = load_manifest(args.manifest) + store = Store(resolve_store(args.store)) + t0 = time.perf_counter() + lock = read_lock(args.lock) + t_lock = time.perf_counter() - t0 + scoring = Scoring(hop=manifest.recipe.geometry.hop, + tolerance_hops=manifest.recipe.label.tolerance_hops) + t0 = time.perf_counter() + streams = streams_from_lock(manifest, lock, store, scoring, args.shares, args.max_stream_s) + t_assemble = time.perf_counter() - t0 + print(f"lock: {len(lock.clips)} rows read in {t_lock:.2f} s; {len(streams)} streams assembled in " + f"{t_assemble:.3f} s") + for share, n in sorted(positives_per_subshare(streams).items()): + print(f"positives from {share}: {n}") + counts = {s.share: 0 for s in streams} + for s in streams: + counts[s.share] += 1 + for share, h in hours_per_share(streams, scoring).items(): + print(f"{share}: {counts[share]} streams, {h:.4f} h") + if args.validate: + t0 = time.perf_counter() + validate_streams(streams, scoring) + print(f"validated (every stream decoded) in {time.perf_counter() - t0:.1f} s") + except (StreamError, StoreError, ValueError, OSError) as e: + print(f"refused: {e}", file=sys.stderr) + return 2 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/ml/kws/requirements-notebook.txt b/tools/ml/kws/requirements-notebook.txt new file mode 100644 index 0000000..fae2154 --- /dev/null +++ b/tools/ml/kws/requirements-notebook.txt @@ -0,0 +1,7 @@ +# The DET notebook's toolchain (tools/ml/build_kws_det_notebook.py -> notebooks/kws_det.ipynb), on top of +# requirements.txt. Not installed in CI: the DET evaluation is the notebook verification layer, executed +# and committed by hand on the M0 Mac (wake-word plan §7). Versions as installed 9 September 2026. +nbformat==5.11.1 +nbclient==0.11.0 +ipykernel==7.3.0 +matplotlib==3.11.1 diff --git a/tools/ml/kws/test_kws_eval.py b/tools/ml/kws/test_kws_eval.py new file mode 100644 index 0000000..0a9f2e1 --- /dev/null +++ b/tools/ml/kws/test_kws_eval.py @@ -0,0 +1,765 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""The M5 pass for the evaluation harness (wake-word plan §6 M5), items 1-4 of the M5 brief. + + .venv/bin/python -m unittest tools/ml/kws/test_kws_eval.py # from the repo root + .venv/bin/python -m unittest discover -s tools/ml/kws -p 'test_*.py' -v # the kws-dataset CI job + +1. The planted-event oracle: synthetic silent streams scored by a PlantedDetector, with known positive + endpoints and known event placements, against hand-computed recall and FA/h, exact to the utterance — + the per-utterance hit map asserted through the harness's own decide/hits path, with events inside a + window, at its two inclusive edges and one hop outside each, the plants deliberately not mirror-symmetric + about the window (one at the lower edge and one below it, two at the upper edge and two above it), so a + rigid shift of the window in either direction changes the aggregates as well as the map; negative streams + totalling exactly 0.5 h with four planted events of which two fall inside one refractory period (3 + counted -> 6.0 FA/h, Poisson [1.237, 17.535]); the refractory boundary as a number (a crossing exactly R + hops after an event fires, R - 1 merges); one hit per utterance when two events fall in one window; a + share with zero events reporting ln 20 / H; a positive-stream event outside every window counted as + spurious and never as a false accept; absent shares reported absent; the self-checks of kws_scoring and + kws_detectors run here so their hand-computed edge values are part of CI. +2. A deliberately mis-accounted variant is refused by name: a negative stream whose declared + negative_samples differs from its audio length (or is not an integer); a positive whose endpoint lies + beyond its stream (and one whose window ends beyond its hops); duplicate stream ids; a positive id used + twice, within one stream or across two; a positive stream with non-zero (or negative) negative_samples — + each by `kws_eval.check_streams` / `evaluate` and by `kws_streams.validate_streams` (one rule set: + the harness delegates to `kws_streams.validate_stream`). Scores handed to `evaluate` are re-checked + (NaN, a wrong length, a value above 1, an unknown stream id); a detector returning a complex or bool + array is refused, not cast; thresholds outside [0, 1] or repeated are refused. +3. The report round-trips (to_dict -> JSON -> from_dict, and through report.json on disk) and the markdown + carries every figure with its hours; `from_dict` refuses an unknown field and a wrong version by name. +4. Toy end to end: the toy rebuilt into a temporary store (as test_kws does), streams assembled from the + committed expected lock, the BandEnergyBaseline through the bridge, a sweep; the hours equal the sum of + the eval negatives' decoded lengths (bit-identical to `hours_per_share` and to the lock's summary), the + positives count the eval positives, and the report's eval_set_ids equal the lock's; the `sweep` CLI on + the same store writes a report carrying the lock's eval_set_id and the packing bound, and refuses a + `--grid` of -1 or 1 by name. + +Every expected figure below is computed by hand in the comment beside its assertion (z = 1.959964 for the +Wilson interval; chi-square quantiles chi2(0.025, 6) = 1.2373, chi2(0.975, 8) = 17.5345, chi2(0.975, 2) = +7.3778) and asserted to 5e-4, the hand precision — the harness's own arithmetic is never the reference. +""" +from __future__ import annotations + +import contextlib +import dataclasses +import io +import json +import pathlib +import sys +import tempfile +import unittest +import warnings +from typing import Callable + +import numpy as np +import soundfile + +HERE = pathlib.Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) + +import kws_build # noqa: E402 +import kws_detectors # noqa: E402 +import kws_eval # noqa: E402 +import kws_features # noqa: E402 +import kws_scoring # noqa: E402 +import kws_streams # noqa: E402 +import test_kws # noqa: E402 +from kws_detectors import PlantedDetector # noqa: E402 +from kws_eval import EvalError, Report, check_streams, default_thresholds, evaluate # noqa: E402 +from kws_manifest import load_manifest, read_lock # noqa: E402 +from kws_scoring import Positive, Scoring # noqa: E402 +from kws_store import Store # noqa: E402 +from kws_streams import Stream, StreamError, validate_streams # noqa: E402 + +RATE = 16000 +HOP = 160 +TOL = 5e-4 # the hand precision of the figures in the comments + + +def silent(n: int) -> Callable[[], np.ndarray]: + """A lazily materialised silent stream (zeros are never touched, so a 0.5 h stream costs no RSS).""" + return lambda: np.zeros(n, dtype=np.float64) + + +def positive_stream(stream_id: str, endpoints: list[int], seconds: float, scoring: Scoring, + subshare: str = "eval-speech") -> Stream: + n = int(seconds * RATE) + positives = [Positive(id=f"{stream_id}#{i}", endpoint_sample=e, endpoint_hop=scoring.endpoint_hop(e)) + for i, e in enumerate(endpoints)] + return Stream(id=stream_id, share="positives", positives=positives, negative_samples=0, subshare=subshare, + load=silent(n)) + + +def negative_stream(stream_id: str, share: str, seconds: float) -> Stream: + n = int(seconds * RATE) + return Stream(id=stream_id, share=share, positives=[], negative_samples=n, load=silent(n)) + + +# The oracle. Scoring: hop 160 at 16 kHz, T = 3, L = 20, W = 1 (the smoothed score is the raw score, so a +# planted 1.0 at hop t is an upward crossing at exactly t for every threshold in (0, 1]), R = 100. +# Positive streams are 4 s = 64000 samples = 400 hops; endpoint 24000 -> h = 24000 // 160 = 150, window +# [150 - 3, 150 + 20 + 3] = [147, 173]. The second utterance of "pos/two": endpoint 48000 -> h = 300, +# window [297, 323]. The plants around the window are not a mirror image of it: one at the lower edge and +# one below, two at the upper edge and two above, so shifting the window by k hops in either direction +# (any |k| <= 13, where "inside" leaves it) changes the hit count — a mirror-symmetric set would trade one +# edge hit for one outside hit and leave every aggregate unchanged. +ORACLE_SCORING = Scoring(hop=HOP, sample_rate=RATE, tolerance_hops=3, latency_hops=20, smoothing_hops=1, + refractory_hops=100) +ORACLE_PLANTED: dict[str, list[tuple[int, float]]] = { + "pos/inside": [(160, 1.0)], # inside [147, 173] -> hit + "pos/edge-lo": [(147, 1.0)], # the lower inclusive edge -> hit + "pos/edge-hi": [(173, 1.0)], # the upper inclusive edge -> hit + "pos/edge-hi-2": [(173, 1.0)], # the upper inclusive edge again -> hit + "pos/below": [(146, 1.0)], # one hop below the window -> miss, spurious + "pos/above": [(174, 1.0)], # one hop above the window -> miss, spurious + "pos/above-2": [(174, 1.0)], # one hop above the window again -> miss, spurious + "pos/inside+late": [(160, 1.0), (300, 1.0)], # a hit, then an event 140 hops later (past R) outside + # every window -> hit, spurious + "pos/silent": [], # no event -> miss + "pos/two": [(160, 1.0), (310, 1.0)], # two utterances, one event in each window -> 2 hits + # eval-speech: 1000 s + 800 s = 1800 s = 0.5 h exactly; events at 1000, 1050 (50 hops after the + # first: inside R = 100, merged), 5000 and 20000 -> 4 planted, 3 counted + "eval-speech/stream-0000": [(1000, 1.0), (1050, 1.0), (5000, 1.0)], + "eval-speech/stream-0001": [(20000, 1.0)], + # eval-music: 900 s = 0.25 h, no event + "eval-music/stream-0000": [], +} +# the hit map the oracle must reproduce, utterance by utterance, at every threshold in (0, 1] +ORACLE_HITS = {"pos/inside#0": True, "pos/edge-lo#0": True, "pos/edge-hi#0": True, "pos/edge-hi-2#0": True, + "pos/below#0": False, "pos/above#0": False, "pos/above-2#0": False, "pos/inside+late#0": True, + "pos/silent#0": False, "pos/two#0": True, "pos/two#1": True} +ORACLE_POSITIVE_STREAMS = 10 # 11 utterances (pos/two carries two) on 10 streams + + +def oracle_streams(scoring: Scoring = ORACLE_SCORING) -> list[Stream]: + return [positive_stream("pos/inside", [24000], 4.0, scoring), + positive_stream("pos/edge-lo", [24000], 4.0, scoring), + positive_stream("pos/edge-hi", [24000], 4.0, scoring), + positive_stream("pos/edge-hi-2", [24000], 4.0, scoring), + positive_stream("pos/below", [24000], 4.0, scoring), + positive_stream("pos/above", [24000], 4.0, scoring), + positive_stream("pos/above-2", [24000], 4.0, scoring), + positive_stream("pos/inside+late", [24000], 4.0, scoring), + positive_stream("pos/silent", [24000], 4.0, scoring), + positive_stream("pos/two", [24000, 48000], 8.0, scoring, subshare="holdout"), + negative_stream("eval-speech/stream-0000", "eval-speech", 1000.0), + negative_stream("eval-speech/stream-0001", "eval-speech", 800.0), + negative_stream("eval-music/stream-0000", "eval-music", 900.0)] + + +SPEECH_0, SPEECH_1, MUSIC_0 = 10, 11, 12 # indices of the negative streams in oracle_streams() + + +def oracle_report(thresholds: list[float] = (0.0, 0.5, 1.0)) -> Report: + streams = oracle_streams() + return evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, list(thresholds)) + + +class PlantedEventOracle(unittest.TestCase): + """Item 1.""" + + @classmethod + def setUpClass(cls) -> None: + cls.report = oracle_report() + + def test_hours_and_counts(self) -> None: + r = self.report + # 1000 s + 800 s = 28 800 000 samples / 16000 / 3600 = 0.5 h; 900 s = 0.25 h; TTS and noise absent + self.assertEqual(r.hours, {"eval-speech": 0.5, "eval-music": 0.25}) + self.assertEqual(r.negative_streams, {"eval-speech": 2, "eval-music": 1}) + self.assertNotIn("eval-tts", r.hours) + self.assertNotIn("eval-noise", r.hours) + # 9 utterances from eval-speech streams, 2 from the hold-out stream + self.assertEqual(r.positives, {"eval-speech": 9, "holdout": 2}) + self.assertEqual([row.threshold for row in r.rows], [0.0, 0.5, 1.0]) + self.assertEqual(r.scoring, ORACLE_SCORING.to_dict()) + self.assertEqual(r.detector["name"], "planted") + self.assertIsNone(r.max_stream_s) # hand-built streams: no packing bound to record + + def test_the_hit_map_utterance_by_utterance(self) -> None: + # the harness's own path — score_stream -> decide -> hits — reproduces ORACLE_HITS for every + # positive stream at thresholds 0.5 and 1.0, and is exactly the window's inclusive edges: 147 and + # 173 hit, 146 and 174 do not + det = PlantedDetector(ORACLE_PLANTED, HOP) + for theta in (0.5, 1.0): + got: dict[str, bool] = {} + for s in oracle_streams(): + if s.share != "positives": + continue + ev = kws_scoring.decide(det.score_stream(s.id, s.samples().size), ORACLE_SCORING, theta) + got.update(kws_scoring.hits(ev, s.positives, ORACLE_SCORING)) + self.assertEqual(got, ORACLE_HITS, theta) + for value in got.values(): + self.assertIs(type(value), bool) # one hit per utterance, never a count + # and the window itself, as numbers + self.assertEqual(ORACLE_SCORING.hit_window(150), (147, 173)) + + def test_recall_exact_to_the_utterance(self) -> None: + for row in self.report.rows[1:]: # thresholds 0.5 and 1.0 see the same crossings (planted score 1.0) + # hits: inside, edge-lo, edge-hi, edge-hi-2, inside+late, two (x2) = 7 of 11; + # misses: below, above, above-2, silent + self.assertEqual((row.recall.hits, row.recall.positives), (7, 11), row.threshold) + # 7/11 = 0.636364; Wilson: z^2/n = 3.841459/11 = 0.349224, denom = 1.349224, + # centre = (0.636364 + 0.174612) / 1.349224 = 0.601070, + # half = 1.959964 * sqrt(0.231405/11 + 3.841459/484) / 1.349224 = 1.959964 * 0.170218 / 1.349224 + # = 0.247269 -> [0.353801, 0.848339] + self.assertAlmostEqual(row.recall.recall, 7 / 11, places=12) + self.assertAlmostEqual(row.recall.frr, 4 / 11, places=12) + self.assertAlmostEqual(row.recall.interval[0], 0.35380, delta=TOL) + self.assertAlmostEqual(row.recall.interval[1], 0.84834, delta=TOL) + # spurious: below, above, above-2, and the late event of inside+late — never in any FA/h + self.assertEqual(row.spurious, 4) + # per subshare: eval-speech 5/9 (Wilson: z^2/n = 0.426829, denom 1.426829, centre + # (0.555556 + 0.213414) / 1.426829 = 0.538930, half = 1.959964 * sqrt(0.246914/9 + 3.841459/324) + # / 1.426829 = 1.959964 * 0.198219 / 1.426829 = 0.272285 -> [0.266645, 0.811215]); + # hold-out 2/2 = 1.0 (z^2/n = 1.920729, denom 2.920729, centre 0.671191, half = 1.959964 * + # sqrt(3.841459/16) / 2.920729 = 0.328810 -> [0.342381, 1.0]) + self.assertEqual(set(row.by_subshare), {"eval-speech", "holdout"}) + speech, hold = row.by_subshare["eval-speech"], row.by_subshare["holdout"] + self.assertEqual((speech.hits, speech.positives, hold.hits, hold.positives), (5, 9, 2, 2)) + self.assertAlmostEqual(speech.interval[0], 0.26665, delta=TOL) + self.assertAlmostEqual(speech.interval[1], 0.81122, delta=TOL) + self.assertAlmostEqual(hold.interval[0], 0.34238, delta=TOL) + self.assertEqual(hold.interval[1], 1.0) + self.assertEqual(hold.recall, 1.0) + + def test_false_accepts_per_hour(self) -> None: + for row in self.report.rows[1:]: + self.assertEqual(set(row.shares), {"eval-speech", "eval-music"}) + speech = row.shares["eval-speech"] + # 4 planted, 1050 merged into 1000's refractory period -> 3 events in 0.5 h = 6.0 FA/h; + # exact Poisson 95 %: chi2(0.025, 6)/2/0.5 = 1.2373/1 = 1.2373, chi2(0.975, 8)/2/0.5 = 17.5345 + self.assertEqual((speech.events, speech.hours, speech.streams), (3, 0.5, 2)) + self.assertEqual(speech.fa_per_hour, 6.0) + self.assertAlmostEqual(speech.interval[0], 1.2373, delta=TOL) + self.assertAlmostEqual(speech.interval[1], 17.5345, delta=TOL) + self.assertIsNone(speech.zero_event_bound) + music = row.shares["eval-music"] + # 0 events in 0.25 h: one-sided bound ln 20 / 0.25 = 2.995732 / 0.25 = 11.98293; the two-sided + # interval [0, chi2(0.975, 2)/2/0.25] = [0, 7.3778/0.5] = [0, 14.7555] + self.assertEqual((music.events, music.hours, music.streams, music.fa_per_hour), (0, 0.25, 1, 0.0)) + self.assertAlmostEqual(music.zero_event_bound, 11.9829, delta=TOL) + self.assertEqual(music.interval[0], 0.0) + self.assertAlmostEqual(music.interval[1], 14.7555, delta=TOL) + + def test_the_refractory_boundary_is_a_number(self) -> None: + # decide: a crossing exactly R = 100 hops after an event fires, one R - 1 hops after it merges + x = np.zeros(400) + x[10], x[110] = 1.0, 1.0 + self.assertEqual(kws_scoring.decide(x, ORACLE_SCORING, 0.5), [10, 110]) + x = np.zeros(400) + x[10], x[109] = 1.0, 1.0 + self.assertEqual(kws_scoring.decide(x, ORACLE_SCORING, 0.5), [10]) + # and through the harness, on its own negative stream: 900 s = 0.25 h of eval-noise + for hops, n_events, fa, lo, hi in ( + # 2 events in 0.25 h = 8.0 FA/h: chi2(0.025, 4)/2/0.25 = 0.48442/0.5 = 0.96884, + # chi2(0.975, 6)/2/0.25 = 14.44938/0.5 = 28.8988 + ((1000, 1100), 2, 8.0, 0.96884, 28.8988), + # 1 event (1099 merges into 1000's period): 4.0 FA/h, chi2(0.025, 2)/2/0.25 = 0.10127, + # chi2(0.975, 4)/2/0.25 = 22.2866 + ((1000, 1099), 1, 4.0, 0.10127, 22.2866)): + streams = [negative_stream("eval-noise/stream-0000", "eval-noise", 900.0)] + det = PlantedDetector({"eval-noise/stream-0000": [(h, 1.0) for h in hops]}, HOP) + noise = evaluate(streams, det, ORACLE_SCORING, [0.5]).rows[0].shares["eval-noise"] + self.assertEqual((noise.events, noise.hours, noise.fa_per_hour), (n_events, 0.25, fa), hops) + self.assertAlmostEqual(noise.interval[0], lo, delta=TOL) + self.assertAlmostEqual(noise.interval[1], hi, delta=TOL) + + def test_one_hit_per_utterance_when_two_events_fall_in_one_window(self) -> None: + # R = 5: pos/inside carries events at 160 and 170, both inside [147, 173] and both decided (10 hops + # apart > R); the utterance still counts one hit: 7 of 11, subshares unchanged, spurious unchanged. + # On eval-speech the events at 1000 and 1050 no longer merge: 4 events in 0.5 h. + sc = dataclasses.replace(ORACLE_SCORING, refractory_hops=5) + planted = {**ORACLE_PLANTED, "pos/inside": [(160, 1.0), (170, 1.0)]} + det = PlantedDetector(planted, sc.hop) + ev = kws_scoring.decide(det.score_stream("pos/inside", 64000), sc, 0.5) + self.assertEqual(ev, [160, 170]) + inside = next(s for s in oracle_streams(sc) if s.id == "pos/inside") + got = kws_scoring.hits(ev, inside.positives, sc) + self.assertEqual(got, {"pos/inside#0": True}) + self.assertIs(got["pos/inside#0"], True) + row = evaluate(oracle_streams(sc), det, sc, [0.5]).rows[0] + self.assertEqual((row.recall.hits, row.recall.positives, row.spurious), (7, 11, 4)) + self.assertEqual((row.by_subshare["eval-speech"].hits, row.by_subshare["holdout"].hits), (5, 2)) + self.assertEqual(row.shares["eval-speech"].events, 4) + + def test_threshold_zero_fires_once_per_stream_at_hop_0(self) -> None: + # At theta = 0 the smoothed score is >= 0 from hop 0 on: one event at hop 0 per stream, then no + # further upward crossing. Hop 0 lies inside no window here (every window starts at 147 or 297) -> + # 0 hits, one spurious event per positive stream (10); one event per negative stream: 2 in 0.5 h + # = 4.0 FA/h, 1 in 0.25 h = 4.0 FA/h — the stream count over H, the ceiling of what the packing + # bound can add to any row. + row = self.report.rows[0] + self.assertEqual(row.threshold, 0.0) + self.assertEqual((row.recall.hits, row.recall.positives, row.spurious), + (0, 11, ORACLE_POSITIVE_STREAMS)) + self.assertEqual(row.recall.recall, 0.0) + self.assertEqual((row.shares["eval-speech"].events, row.shares["eval-speech"].fa_per_hour), (2, 4.0)) + self.assertEqual((row.shares["eval-music"].events, row.shares["eval-music"].fa_per_hour), (1, 4.0)) + # 1 event in 0.25 h: chi2(0.025, 2)/2/0.25 = 0.050636/0.5 = 0.10127; chi2(0.975, 4)/2/0.25 = + # 11.14329/0.5 = 22.2866 + self.assertAlmostEqual(row.shares["eval-music"].interval[0], 0.10127, delta=TOL) + self.assertAlmostEqual(row.shares["eval-music"].interval[1], 22.2866, delta=TOL) + self.assertIsNone(row.shares["eval-music"].zero_event_bound) + + def test_smoothing_and_refractory_come_from_the_scoring(self) -> None: + # W = 2: a single planted 1.0 at hop t gives smoothed 0.5 at t and t + 1; at theta 0.5 the event + # still fires at t (0.5 >= 0.5), at theta 0.6 nothing fires: recall 0/11, spurious 0, 0 events. + sc = dataclasses.replace(ORACLE_SCORING, smoothing_hops=2) + r = evaluate(oracle_streams(sc), PlantedDetector(ORACLE_PLANTED, HOP), sc, [0.5, 0.6]) + self.assertEqual((r.rows[0].recall.hits, r.rows[0].spurious, r.rows[0].shares["eval-speech"].events), + (7, 4, 3)) + self.assertEqual((r.rows[1].recall.hits, r.rows[1].spurious, r.rows[1].shares["eval-speech"].events), + (0, 0, 0)) + # ln 20 / 0.5 = 5.99146 + self.assertAlmostEqual(r.rows[1].shares["eval-speech"].zero_event_bound, 5.99146, delta=TOL) + # R = 40: the events at 1000 and 1050 no longer merge -> 4 events in 0.5 h = 8.0 FA/h + sc = dataclasses.replace(ORACLE_SCORING, refractory_hops=40) + r = evaluate(oracle_streams(sc), PlantedDetector(ORACLE_PLANTED, HOP), sc, [0.5]) + speech = r.rows[0].shares["eval-speech"] + self.assertEqual((speech.events, speech.fa_per_hour), (4, 8.0)) + + def test_default_thresholds_are_the_quantiles_of_the_stream_maxima(self) -> None: + sc = ORACLE_SCORING # W = 1: the per-stream maximum smoothed score is the planted maximum + scores = [np.array([0.0, 0.2, 0.0]), np.array([0.4, 0.0]), np.array([0.0, 0.6]), np.array([0.8]), + np.array([0.9, 0.9, 0.1])] + # n = 5 over 5 maxima: linspace(0, 1, 5) lands on every sorted maximum, plus 0 and 1 + self.assertEqual(default_thresholds(scores, sc, n=5), [0.0, 0.2, 0.4, 0.6, 0.8, 0.9, 1.0]) + # n = 3: the 0, 0.5 and 1 quantiles -> 0.2, 0.6, 0.9 + self.assertEqual(default_thresholds(scores, sc, n=3), [0.0, 0.2, 0.6, 0.9, 1.0]) + # a maximum of exactly 1.0 deduplicates against the fixed 1; a dict of scores is accepted too + self.assertEqual(default_thresholds({"a": np.array([1.0, 0.0]), "b": np.array([0.0])}, sc, n=2), + [0.0, 1.0]) + # W = 2: a lone 1.0 smooths to 0.5, which is where the threshold lands + self.assertEqual(default_thresholds([np.array([0.0, 1.0, 0.0])], + dataclasses.replace(sc, smoothing_hops=2), n=2), [0.0, 0.5, 1.0]) + with self.assertRaisesRegex(EvalError, "n must be at least 2"): + default_thresholds(scores, sc, n=1) + # summation noise above 1.0 (a moving average of ones) is clipped, and it is not a second threshold + noisy = [np.full(30, 1.0), np.array([1.0])] + self.assertEqual(default_thresholds(noisy, dataclasses.replace(sc, smoothing_hops=10), n=5), + [0.0, 1.0]) + # the bring-up corpus's extreme (max over 412 streams, W = 10, measured 9 September 2026) and one + # stream's noise + self.assertEqual(kws_eval.threshold_value(1.0000000000000455), 1.0) + self.assertEqual(kws_eval.threshold_value(1.0000000000000113), 1.0) + self.assertEqual(kws_eval.threshold_value(0.15000000000000002), 0.15) + + def test_a_detector_that_breaks_the_score_contract_is_refused(self) -> None: + class Short: + name, params = "short", {} + + def score(self, x: np.ndarray) -> np.ndarray: + return np.zeros(x.size // HOP - 1) + + class OutOfRange: + name, params = "range", {} + + def score(self, x: np.ndarray) -> np.ndarray: + return np.full(x.size // HOP, 1.5) + + class Complex: + name, params = "complex", {} + + def score(self, x: np.ndarray) -> np.ndarray: + return np.full(x.size // HOP, 0.2 + 0.9j) + + class Boolean: + name, params = "bool", {} + + def score(self, x: np.ndarray) -> np.ndarray: + return np.zeros(x.size // HOP, dtype=bool) + + class Strings: + name, params = "strings", {} + + def score(self, x: np.ndarray) -> list[str]: + return ["a"] * (x.size // HOP) + + streams = oracle_streams()[:1] + with self.assertRaisesRegex(EvalError, r"'pos/inside'.*returned \(399,\) scores for 400 hops"): + evaluate(streams, Short(), ORACLE_SCORING, [0.5]) + with self.assertRaisesRegex(EvalError, r"'pos/inside'.*within \[0, 1\]"): + evaluate(streams, OutOfRange(), ORACLE_SCORING, [0.5]) + # a complex, bool or string array is refused by name, never cast (numpy's ComplexWarning would be + # the only trace of a dropped imaginary part; here any warning is an error) + with warnings.catch_warnings(): + warnings.simplefilter("error") + for det in (Complex(), Boolean(), Strings()): + with self.assertRaisesRegex(EvalError, + rf"'pos/inside': detector '{det.name}' returned dtype"): + evaluate(streams, det, ORACLE_SCORING, [0.5]) + with self.assertRaisesRegex(EvalError, "thresholds must be a non-empty list"): + evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, []) + with self.assertRaisesRegex(EvalError, "thresholds must be a non-empty list"): + evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, [float("nan")]) + # scores are within [0, 1], so a threshold outside it measures nothing; a repeated one is refused too + with self.assertRaisesRegex(EvalError, r"thresholds must lie in \[0, 1\].*\[-1\.0, 1\.5\]"): + evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, [-1.0, 0.5, 1.5]) + with self.assertRaisesRegex(EvalError, r"duplicate threshold\(s\) \[0\.5\]"): + evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, [0.5, 0.5, 1.0]) + + def test_supplied_scores_are_rechecked(self) -> None: + # a bare dict of scores goes through the same contract as a detector's output, by stream id + streams = oracle_streams() + det = PlantedDetector(ORACLE_PLANTED, HOP) + good = {s.id: det.score_stream(s.id, s.samples().size) for s in streams} + r = evaluate(streams, det, ORACLE_SCORING, [0.5], scores=good) + self.assertEqual(r.rows[0].recall.hits, 7) + self.assertEqual(r.rows[0].shares["eval-speech"].events, 3) + for name, bad in (("NaN", np.full(400, np.nan)), ("short", np.ones(5)), + ("above 1", np.full(400, 7.0)), ("complex", np.full(400, 1j))): + with self.subTest(name): + with self.assertRaisesRegex(EvalError, r"stream 'pos/inside'"): + evaluate(streams, det, ORACLE_SCORING, [0.5], scores={**good, "pos/inside": bad}) + with self.assertRaisesRegex(EvalError, r"unknown stream id\(s\) \['zzz'\]"): + evaluate(streams, det, ORACLE_SCORING, [0.5], scores={**good, "zzz": np.zeros(400)}) + with self.assertRaisesRegex(EvalError, r"no scores for stream\(s\) \['pos/inside'\]"): + evaluate(streams, det, ORACLE_SCORING, [0.5], scores={k: v for k, v in good.items() + if k != "pos/inside"}) + # the Scored that score_streams returns is taken as verified, but a stream edited afterwards is + # still refused against the recorded sample counts (no second decode) + scored = kws_eval.score_streams(streams, det, ORACLE_SCORING) + self.assertEqual(evaluate(streams, det, ORACLE_SCORING, [0.5], scores=scored), r) + streams[SPEECH_0].negative_samples -= 1 + with self.assertRaisesRegex(EvalError, r"'eval-speech/stream-0000': negative_samples 15999999"): + evaluate(streams, det, ORACLE_SCORING, [0.5], scores=scored) + with self.assertRaisesRegex(EvalError, r"unknown stream id\(s\) \['eval-music/stream-0000'\]"): + evaluate(oracle_streams()[:-1], det, ORACLE_SCORING, [0.5], scores=scored) + + +class SelfChecksRunInCI(unittest.TestCase): + """The hand-computed edge values of kws_scoring (hit_window(56) == (53, 79); hits at 53 and 79, none at + 52 or 80; the refractory pair) and of kws_detectors (the planted 1 kHz burst through the bridge) live in + their `_self_check`s; running them here makes them part of `unittest discover`.""" + + def test_kws_scoring_self_check(self) -> None: + with contextlib.redirect_stdout(io.StringIO()): + kws_scoring._self_check() + + def test_kws_detectors_self_check(self) -> None: + with contextlib.redirect_stdout(io.StringIO()): + kws_detectors._self_check() + + +class MisaccountedStreamsAreRefused(unittest.TestCase): + """Item 2: each variant refused by name — by the harness's own check, and by validate_streams.""" + + @staticmethod + def variants() -> list[tuple[str, str, list[Stream]]]: + """(name, the id the refusal must name, the streams).""" + sc = ORACLE_SCORING + out = [] + s = oracle_streams() + bad = negative_stream("eval-speech/stream-0000", "eval-speech", 1000.0) + bad.negative_samples = 1000 * RATE - 1 # declared one sample short of its audio + out.append(("negative_samples != audio length", "eval-speech/stream-0000", + s[:SPEECH_0] + [bad] + s[SPEECH_0 + 1:])) + s = oracle_streams() + s[SPEECH_0].negative_samples = float(1000 * RATE) # the right length, but not an integer + out.append(("negative_samples not an integer", "eval-speech/stream-0000", s)) + s = oracle_streams() + s[SPEECH_0].negative_samples = True # a bool is not an integer count either + out.append(("negative_samples a bool", "eval-speech/stream-0000", s)) + s = oracle_streams() + s[0] = positive_stream("pos/inside", [70000], 4.0, sc) # endpoint 70000 beyond 64000 samples + out.append(("endpoint beyond the stream", "pos/inside", s)) + s = oracle_streams() + s[0] = positive_stream("pos/inside", [63000], 4.0, sc) # h = 393: window [390, 416] beyond 400 hops + out.append(("window beyond the stream's hops", "pos/inside", s)) + s = oracle_streams() + s[1] = positive_stream("pos/inside", [24000], 4.0, sc) # a second "pos/inside" + out.append(("duplicate stream id", "pos/inside", s)) + s = oracle_streams() + s[0].positives = [Positive("u", 16000, 100), Positive("u", 48000, 300)] # one id, two utterances + out.append(("positive id used twice in one stream", "pos/inside", s)) + s = oracle_streams() + s[1].positives = list(s[0].positives) # pos/edge-lo carries pos/inside's utterance id + out.append(("positive id used twice across streams", "pos/edge-lo", s)) + s = oracle_streams() + s[0].negative_samples = 64000 # a positive stream counting its audio as negative hours + out.append(("positive stream with hours", "pos/inside", s)) + s = oracle_streams() + s[0].negative_samples = -1 # negative hours + out.append(("positive stream with negative hours", "pos/inside", s)) + s = oracle_streams() + s[SPEECH_0].negative_samples = -1 # negative hours on a negative stream + out.append(("negative stream with negative hours", "eval-speech/stream-0000", s)) + return out + + def test_the_oracle_itself_passes(self) -> None: + n = check_streams(oracle_streams(), ORACLE_SCORING) + self.assertEqual(n["pos/two"], 128000) + self.assertEqual(n["eval-speech/stream-0000"], 16000000) + validate_streams(oracle_streams(), ORACLE_SCORING) + + def test_refused_by_the_harness(self) -> None: + for name, who, streams in self.variants(): + with self.subTest(name): + with self.assertRaisesRegex(EvalError, repr(who)): + check_streams(streams, ORACLE_SCORING) + with self.assertRaisesRegex(EvalError, repr(who)): + evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, [0.5]) + + def test_refused_by_validate_streams(self) -> None: + for name, who, streams in self.variants(): + with self.subTest(name): + with self.assertRaisesRegex(StreamError, repr(who).replace("+", r"\+")): + validate_streams(streams, ORACLE_SCORING) + + def test_a_duplicate_positive_id_would_miscount_so_it_is_refused_not_scored(self) -> None: + # kws_scoring.hits keys its map by utterance id: two utterances under one id would collapse to one + # entry against a denominator of two (recall 1/2 with both windows hit) — hence the refusal above, + # here shown to name the id + s = oracle_streams() + s[0].positives = [Positive("u", 16000, 100), Positive("u", 48000, 300)] + with self.assertRaisesRegex(EvalError, r"'pos/inside': positive 'u' appears twice"): + evaluate(s, PlantedDetector({"pos/inside": [(100, 1.0), (300, 1.0)]}, HOP), ORACLE_SCORING, + [0.5]) + + def test_other_refusals_name_the_stream(self) -> None: + s = oracle_streams() + s[0].share = "eval-holdout" + with self.assertRaisesRegex(EvalError, r"'pos/inside': unknown share 'eval-holdout'"): + check_streams(s, ORACLE_SCORING) + s = oracle_streams() + s[SPEECH_0] = negative_stream("eval-speech/stream-0000", "eval-speech", 0.0) + with self.assertRaisesRegex(EvalError, r"'eval-speech/stream-0000': empty stream"): + check_streams(s, ORACLE_SCORING) + s = oracle_streams() + s[SPEECH_0].positives = [Positive("x", 24000, 150)] # a negative stream carrying a positive + with self.assertRaisesRegex(EvalError, + r"'eval-speech/stream-0000': a eval-speech stream carrying 1 positive"): + check_streams(s, ORACLE_SCORING) + s = oracle_streams() + s[0].positives = [Positive("x", 24000, 151)] # endpoint_hop is not the endpoint's hop + with self.assertRaisesRegex(EvalError, r"'pos/inside': positive 'x' endpoint_hop 151 != 150"): + check_streams(s, ORACLE_SCORING) + s = oracle_streams() + s[0].positives = [] + with self.assertRaisesRegex(EvalError, r"'pos/inside': a positive stream with no positives"): + check_streams(s, ORACLE_SCORING) + # a negative stream in a share the report has no column for is refused, never silently dropped + # from the hours (a hold-out utterance is a positive stream with subshare "holdout", not a share) + s = oracle_streams() + s[SPEECH_0].share = "holdout" + with self.assertRaisesRegex(EvalError, r"'eval-speech/stream-0000': unknown share 'holdout'"): + check_streams(s, ORACLE_SCORING) + with self.assertRaisesRegex(StreamError, r"'eval-speech/stream-0000': unknown share 'holdout'"): + validate_streams(s, ORACLE_SCORING) + s = oracle_streams() + s[0].subshare = None # a positive stream must say where its utterance came from + with self.assertRaisesRegex(EvalError, r"'pos/inside': subshare None"): + check_streams(s, ORACLE_SCORING) + with self.assertRaisesRegex(StreamError, r"'pos/inside': subshare None"): + validate_streams(s, ORACLE_SCORING) + # no streams at all is a refusal, not a vacuous report; so is something that is not a Stream + planted = PlantedDetector({}, HOP) + for fn in (check_streams, lambda st, sc: kws_eval.score_streams(st, planted, sc), validate_streams): + with self.assertRaisesRegex((EvalError, StreamError), "no streams to score"): + fn([], ORACLE_SCORING) + with self.assertRaisesRegex(EvalError, "no streams to score"): + evaluate([], PlantedDetector({}, HOP), ORACLE_SCORING, [0.5]) + with self.assertRaisesRegex(EvalError, "expected a Stream, got dict"): + check_streams([{"id": "x"}], ORACLE_SCORING) + + +class ReportRoundTrip(unittest.TestCase): + """Item 3.""" + + @classmethod + def setUpClass(cls) -> None: + cls.report = oracle_report() + cls.report.manifest_name = "oracle" + cls.report.manifest_hash = "0" * 64 + cls.report.eval_set_id = {"eval-speech": "a" * 64, "eval-music": "b" * 64} + cls.report.front_end = {"log_mel_contract_version": 1, "dsptap_commit": "c" * 40} + + def test_to_dict_from_dict_and_the_json_file(self) -> None: + r = self.report + d = json.loads(json.dumps(r.to_dict())) + self.assertEqual(Report.from_dict(d), r) + with tempfile.TemporaryDirectory(prefix="mutap-kws-eval-") as tmp: + json_path, md_path = kws_eval.write_report(r, pathlib.Path(tmp)) + self.assertEqual(kws_eval.read_report(json_path), r) + self.assertEqual(md_path.read_text(encoding="utf-8"), r.markdown()) + self.assertEqual(d["report_version"], kws_eval.REPORT_VERSION) + self.assertEqual(d["hours"], {"eval-speech": 0.5, "eval-music": 0.25}) + self.assertEqual(d["eval_set_id"], r.eval_set_id) + self.assertEqual(d["rows"][1]["shares"]["eval-speech"]["events"], 3) + self.assertIsNone(d["rows"][1]["shares"]["eval-speech"]["zero_event_bound"]) + self.assertNotIn("eval-tts", d["rows"][1]["shares"]) + self.assertIsNone(d["max_stream_s"]) + # the packing bound is provenance: it round-trips and lands beside the stream counts + with_bound = dataclasses.replace(r, max_stream_s=60.0) + self.assertEqual(Report.from_dict(json.loads(json.dumps(with_bound.to_dict()))), with_bound) + self.assertIn("- hours: eval-speech H = 0.5 h (2 streams, packed at <= 60 s), eval-music H = 0.25 h " + "(1 streams, packed at <= 60 s), eval-tts absent, eval-noise absent", + with_bound.markdown()) + + def test_from_dict_refuses_by_name(self) -> None: + d = self.report.to_dict() + d["extra"] = 1 + with self.assertRaisesRegex(EvalError, r"report: unknown field\(s\) \['extra'\]"): + Report.from_dict(d) + d = self.report.to_dict() + d["report_version"] = 99 + with self.assertRaisesRegex(EvalError, "report_version 99"): + Report.from_dict(d) + d = self.report.to_dict() + del d["rows"][0]["shares"]["eval-speech"]["hours"] + with self.assertRaisesRegex(EvalError, r"rows\[0\]\.shares\[eval-speech\]: missing required " + r"field\(s\) \['hours'\]"): + Report.from_dict(d) + + def test_markdown_carries_every_figure_with_its_hours(self) -> None: + md = self.report.markdown() + lines = md.splitlines() + rows = [ln for ln in lines if ln.startswith("| ") and not ln.startswith("| threshold")] + self.assertEqual(len(rows), 3) + # the header lines + self.assertIn("- manifest: oracle (`" + "0" * 64 + "`)", md) + self.assertIn("- hours: eval-speech H = 0.5 h (2 streams), eval-music H = 0.25 h (1 streams), " + "eval-tts absent, eval-noise absent", md) + self.assertIn("- positives: eval-speech 9, eval-tts absent, holdout 2", md) + self.assertIn("- eval_set_id: eval-music `" + "b" * 64 + "`, eval-speech `" + "a" * 64 + "`", md) + self.assertIn("- front end: log_mel_contract_version 1, DspTap `" + "c" * 40 + "`", md) + self.assertIn("T = 3 hops, L = 20 hops", md) + self.assertIn("W = 1 hops, R = 100 hops", md) + # the 0.5 row: every FA/h figure with its interval and its hours; absent columns read "absent" + cells = [c.strip() for c in rows[1].strip("|").split("|")] + self.assertEqual(len(cells), 10) + self.assertEqual(cells[0], "0.5") # the shortest exact repr (thresholds 1 - 1e-8 apart stay distinct) + self.assertEqual(cells[1], "0.636 [0.354, 0.848] (7/11)") + self.assertEqual(cells[2], "0.364") + self.assertEqual(cells[3], "4") + self.assertEqual(cells[4], "6.00 [1.24, 17.53] (3 in 0.5 h)") + self.assertEqual(cells[5], "0.00 [0.00, 14.76] <= 11.98 (0 in 0.25 h)") + self.assertEqual(cells[6], "absent") # eval-tts + self.assertEqual(cells[7], "absent") # eval-noise + self.assertEqual(cells[8], "absent") # eval-tts recall + self.assertEqual(cells[9], "1.000 [0.342, 1.000] (2/2)") # hold-out recall + # the 0.0 row + cells = [c.strip() for c in rows[0].strip("|").split("|")] + # Wilson 0/11: centre = 0.174612 / 1.349224 = 0.129417, half = 1.959964 * sqrt(3.841459/484) / + # 1.349224 = 0.129416 -> [0, 0.258833] + self.assertEqual(cells[1], "0.000 [0.000, 0.259] (0/11)") + # 2 in 0.5 h: chi2(0.025, 4)/2/0.5 = 0.4844, chi2(0.975, 6)/2/0.5 = 14.449 + self.assertEqual(cells[4], "4.00 [0.48, 14.45] (2 in 0.5 h)") + self.assertEqual(cells[5], "4.00 [0.10, 22.29] (1 in 0.25 h)") + # every FA/h cell that is not absent names its hours + for row in rows: + for cell in [c.strip() for c in row.strip("|").split("|")][4:8]: + self.assertTrue(cell == "absent" or cell.endswith(" h)"), cell) + + def test_an_empty_positive_set_reports_absent_recall(self) -> None: + streams = [s for s in oracle_streams() if s.share != "positives"] + r = evaluate(streams, PlantedDetector(ORACLE_PLANTED, HOP), ORACLE_SCORING, [0.5]) + self.assertEqual(r.positives, {}) + self.assertIsNone(r.rows[0].recall.recall) + self.assertEqual(Report.from_dict(json.loads(json.dumps(r.to_dict()))), r) + rows = [ln for ln in r.markdown().splitlines() + if ln.startswith("| ") and not ln.startswith("| threshold")] + self.assertEqual(len(rows), 1) + cells = [c.strip() for c in rows[0].strip("|").split("|")] + self.assertEqual(cells[1:4], ["absent", "absent", "0"]) + self.assertEqual(cells[8:10], ["absent", "absent"]) + + +class ToyEndToEnd(unittest.TestCase): + """Item 4: the toy rebuilt into a temporary store, streams from the committed expected lock, the + band-energy baseline through the bridge, a sweep — as a library call and through the `sweep` CLI.""" + + def test_toy_sweep(self) -> None: + manifest = load_manifest(test_kws.MANIFEST) + lock = read_lock(test_kws.EXPECTED / "lock.json") + with tempfile.TemporaryDirectory(prefix="mutap-kws-eval-toy-") as tmp: + store = pathlib.Path(tmp) / "store" + store.mkdir() + r = test_kws.run_build("all", test_kws.MANIFEST, store, jobs=1) + self.assertEqual(r.returncode, 0, r.stdout + r.stderr) + g = manifest.recipe.geometry + scoring = Scoring(hop=g.hop, sample_rate=int(g.sample_rate), + tolerance_hops=manifest.recipe.label.tolerance_hops) + streams = kws_streams.streams_from_lock(manifest, lock, Store(store), scoring) + kws_streams.validate_streams(streams, scoring) + detector = kws_detectors.BandEnergyBaseline(g) + scores = kws_eval.score_streams(streams, detector, scoring) + thresholds = default_thresholds(scores, scoring, n=10) + report = evaluate(streams, detector, scoring, thresholds, scores=scores, provenance={ + "manifest_name": manifest.name, "manifest_hash": lock.manifest_hash, + "eval_set_id": dict(lock.eval_set_id), "max_stream_s": kws_streams.DEFAULT_MAX_STREAM_S, + "front_end": {"log_mel_contract_version": kws_features.contract_version(), + "dsptap_commit": kws_features.dsptap_commit()}}) + # hours: the sum of the eval negatives' decoded lengths, read back from the pcm tier + build = kws_build.Build(manifest, Store(store)) + decoded: dict[str, int] = {} + for c in lock.clips: + if c.split == "eval" and c.variant == 0 and c.label != 1: + info = soundfile.info(str(build.pcm_path(c.source, c.id[len(c.source) + 1:]))) + self.assertEqual(info.samplerate, RATE) + self.assertEqual(info.frames, c.length, c.id) + decoded[c.share] = decoded.get(c.share, 0) + info.frames + self.assertEqual(set(decoded), {"eval-speech", "eval-music", "eval-noise"}) # the toy's shares + self.assertEqual(sum(decoded.values()), 112000) # 7 s: measured from the committed toy lock + self.assertEqual(set(report.hours), set(decoded)) + # bit-identical: the integer sample sum divided once, as hours_per_share and the lock's summary + # (which counts the featurized negatives: speech + music, never noise) compute it + for share, n in decoded.items(): + self.assertEqual(report.hours[share], n / RATE / 3600.0, share) + self.assertEqual(report.hours, kws_streams.hours_per_share(streams, scoring)) + self.assertEqual(scoring.hours(decoded["eval-speech"] + decoded["eval-music"]), + lock.summary["splits"]["eval"]["negative"]["hours"]) + # positives: the toy's one eval positive (label 1, share eval-speech, variant 0) + n_pos = sum(1 for c in lock.clips if c.split == "eval" and c.variant == 0 and c.label == 1) + self.assertEqual(n_pos, 1) + self.assertEqual(report.positives, {"eval-speech": n_pos}) + self.assertEqual(report.eval_set_id, lock.eval_set_id) + self.assertEqual(report.max_stream_s, 60.0) + self.assertNotIn("eval-tts", report.hours) + # the scores are one per completed hop; the positive's count is what extract featurized + frames = {c.id: c.extra["frames"] for c in lock.clips + if c.split == "eval" and c.variant == 0 and c.label == 1} + for s in streams: + self.assertEqual(scores[s.id].size, s.samples().size // g.hop, s.id) + self.assertEqual(scores.n_samples[s.id], s.samples().size, s.id) + if s.share == "positives": + self.assertEqual(scores[s.id].size, frames[s.positives[0].id], s.id) + self.assertEqual(len(report.rows), len(thresholds)) + for row in report.rows: + self.assertEqual(set(row.shares), set(decoded)) + for share, f in row.shares.items(): + self.assertEqual(f.hours, report.hours[share]) + self.assertEqual(Report.from_dict(json.loads(json.dumps(report.to_dict()))), report) + md = report.markdown() + self.assertIn("eval-tts absent", md) + self.assertIn("packed at <= 60 s", md) + self.assertIn(f"DspTap `{kws_features.dsptap_commit()}`", md) + print(f"\ntoy sweep: {len(streams)} streams, {len(thresholds)} thresholds, hours {report.hours}") + # the sweep CLI on the same store: the report carries the lock's eval_set_id, the same hours and + # the packing bound it was run with + out = pathlib.Path(tmp) / "sweep" + argv = ["sweep", "--manifest", str(test_kws.MANIFEST), "--lock", + str(test_kws.EXPECTED / "lock.json"), "--store", str(store), "--thresholds", "10", + "--out", str(out)] + with contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(kws_eval.main(argv + ["--max-stream-s", "2"]), 0) + cli = kws_eval.read_report(out / "report.json") + self.assertEqual(cli.eval_set_id, lock.eval_set_id) + self.assertEqual(cli.hours, report.hours) + self.assertEqual(cli.max_stream_s, 2.0) + self.assertEqual(cli.negative_streams["eval-speech"], 2) # three 1 s clips at a 2 s bound + self.assertIn("packed at <= 2 s", (out / "report.md").read_text(encoding="utf-8")) + self.assertIsNotNone(cli.wall_s) + # a --grid of -1 or 1 is refused by name before anything is decoded (1 would add only 0.0) + for grid in ("-1", "1"): + err = io.StringIO() + with contextlib.redirect_stderr(err): + self.assertEqual(kws_eval.main(argv + ["--grid", grid, "--out", str(out / "never")]), 1) + self.assertIn("--grid must be 0 (off) or at least 2", err.getvalue()) + self.assertFalse((out / "never").exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/ml/kws/test_kws_streams.py b/tools/ml/kws/test_kws_streams.py new file mode 100644 index 0000000..fb77267 --- /dev/null +++ b/tools/ml/kws/test_kws_streams.py @@ -0,0 +1,610 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Copyright 2026 MuTap contributors +"""The M5 stream, detector and hold-out tests (wake-word plan §6 M5; the brief's owner-B items). + + .venv/bin/python -m unittest tools/ml/kws/test_kws_streams.py # from the repo root + .venv/bin/python -m unittest discover -s tools/ml/kws -p 'test_*.py' -v # the kws-dataset CI job + +- Streams from the toy lock (`ToyStreams`): the toy is rebuilt into a temporary store as test_kws does; + the streams assembled from the committed expected lock have the expected counts, members and hours + (hours = the decoded lengths, and equal to the lock's own eval-negative hours where it featurizes + them); the positive stream's mixture is exactly what `extract` featurized — its frame count equals the + lock's `extra.frames`, its features through the bridge equal the committed eval shard's rows — and + its hit window lies inside the stream; the packing bound; the assembly's refusals. +- `validate_streams` refuses every mis-accounting of the brief's item 2 by name (`Validation`). +- The band-energy baseline's alignment on a planted 1 kHz burst (`Detectors`; the measured numbers are + beside the assertions), its band selection, and the planted detector's contract. +- The hold-out (`HoldoutRecord`, the brief's item 5): a synthetic holdout.json and a FLAC written by + soundfile verify; one altered byte is refused naming the file; a missing talker row, an absent + consent_form_version and permitted_uses lacking either required use are each refused by name; the set + id changes when a row is dropped; the schema, path and tier refusals — including one file under two + spellings ('./' or a case-folded path) or one sha256 under two rows, so a file is one utterance; and + `kws_eval.py holdout` end to end (a report whose holdout_set_id is the record's, refused with nothing + written once a byte is flipped). +- The three CLIs refuse a missing store (no --store, no MUTAP_KWS_STORE) with `refused:` and rc 2, never a + traceback. + +Measured 9 September 2026 on the M0 Mac (Apple silicon, CPython 3.12.14, numpy 2.5.3, scipy 1.18.1, +soundfile 0.14.0, the C ABI built Release): the toy rebuild in setUpClass (`all --jobs 1`, a subprocess) +0.87 s wall; the whole file 1.5 s. On the bring-up corpus (263,487 lock rows) the same day: read_lock 1.7 s, +streams_from_lock 0.08 s for 412 streams (195 positives, 179 eval-speech streams = 2.9513 h, 38 eval-noise +streams = 0.5021 h, hours from the decoded lengths; 0.83 s before the asdict conversion was narrowed to the +rows the loaders read), validate_streams 1.5 s (every stream decoded), every +positive mixture's frame count equal to its lock row's `extra.frames`, and the band-energy baseline at +8,461 s of audio scored per second (decode + score 4,478 s/s), 1.04 GB peak RSS (the lock dominates). +""" +from __future__ import annotations + +import contextlib +import copy +import dataclasses +import io +import json +import os +import pathlib +import subprocess +import sys +import tempfile +import time +import unittest +from unittest import mock + +import numpy as np +import soundfile + +HERE = pathlib.Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) + +import kws_audio # noqa: E402 +import kws_detectors # noqa: E402 +import kws_features # noqa: E402 +import kws_eval # noqa: E402 (owner A's harness: the holdout CLI is exercised on this fixture) +import kws_holdout # noqa: E402 +import kws_streams # noqa: E402 +from kws_manifest import load_manifest, manifest_hash, read_lock # noqa: E402 +from kws_scoring import Positive, Scoring # noqa: E402 +from kws_store import ENV_VAR, Store, sha256_file # noqa: E402 + +REPO_ROOT = HERE.parents[2] +PYTHON = sys.executable +BUILD = HERE / "kws_build.py" +TOY = HERE / "fixtures" / "toy" +MANIFEST = TOY / "manifest.json" +EXPECTED = TOY / "expected" +RATE = kws_audio.RATE +# The same relative bound test_kws.py uses for a rebuild against the committed shards: bit-identical on +# the M0 Mac (measured 0.0 on 9 September 2026), 1e-6 to cover another toolchain's FMA / libm rounding. +FEATURE_TOLERANCE = 1e-6 + + +def run_build(stage: str, store: pathlib.Path, jobs: int = 1) -> subprocess.CompletedProcess: + env = dict(os.environ) + env[ENV_VAR] = str(store) # the temporary store, whatever the developer's shell says + return subprocess.run([PYTHON, str(BUILD), stage, "--manifest", str(MANIFEST), "--store", str(store), + "--jobs", str(jobs)], capture_output=True, text=True, check=False, env=env, + cwd=str(REPO_ROOT)) + + +def toy_scoring(manifest) -> Scoring: + return Scoring(hop=manifest.recipe.geometry.hop, tolerance_hops=manifest.recipe.label.tolerance_hops) + + +def positive_stream(stream_id: str, n: int, endpoint: int, scoring: Scoring, **kw) -> kws_streams.Stream: + p = Positive(id=stream_id + "/kw", endpoint_sample=endpoint, endpoint_hop=scoring.endpoint_hop(endpoint)) + fields = {"id": stream_id, "share": "positives", "positives": [p], "negative_samples": 0, + "subshare": "eval-speech", "audio": np.zeros(n)} + fields.update(kw) + return kws_streams.Stream(**fields) + + +def negative_stream(stream_id: str, n: int, share: str = "eval-speech", **kw) -> kws_streams.Stream: + fields = {"id": stream_id, "share": share, "positives": [], "negative_samples": n, "audio": np.zeros(n)} + fields.update(kw) + return kws_streams.Stream(**fields) + + +class ToyStreams(unittest.TestCase): + """Streams from the committed toy lock over a temporary rebuild of the toy store.""" + + tmp: tempfile.TemporaryDirectory + store: Store + seconds: float + + @classmethod + def setUpClass(cls) -> None: + cls.tmp = tempfile.TemporaryDirectory(prefix="mutap-kws-streams-") + root = pathlib.Path(cls.tmp.name) / "store" + root.mkdir() + t0 = time.time() + r = run_build("all", root) + cls.seconds = time.time() - t0 + if r.returncode != 0: + cls.tmp.cleanup() + raise AssertionError(f"toy build failed (rc {r.returncode}):\n{r.stdout}\n{r.stderr}") + cls.store = Store(root) + cls.manifest = load_manifest(MANIFEST) + cls.lock = read_lock(EXPECTED / "lock.json") + cls.scoring = toy_scoring(cls.manifest) + cls.streams = kws_streams.streams_from_lock(cls.manifest, cls.lock, cls.store, cls.scoring) + + @classmethod + def tearDownClass(cls) -> None: + cls.tmp.cleanup() + + def test_1_counts_members_and_hours_from_decoded_lengths(self) -> None: + streams = self.streams + self.assertEqual(self.lock.manifest_hash, manifest_hash(self.manifest)) + by_id = {c.id: c for c in self.lock.clips if c.variant == 0} + eval_rows = [c for c in by_id.values() if c.split == "eval"] + positives = [s for s in streams if s.share == "positives"] + self.assertEqual([s.id for s in positives], sorted(c.id for c in eval_rows if c.label == 1)) + self.assertEqual(len(positives), 1) # the toy's one eval positive (marvin) + p = positives[0] + row = by_id[p.id] + self.assertEqual(p.subshare, "eval-speech") + self.assertEqual(p.negative_samples, 0) + self.assertEqual(p.members, [p.id]) + self.assertIsNone(p.audio) # lazy + self.assertEqual(p.positives, [Positive(p.id, row.endpoint_sample, row.endpoint_sample // 160)]) + self.assertEqual(p.positives[0].endpoint_hop, row.extra["endpoint_hop"]) + # negatives: per share, sorted by id, one stream each at the default 60 s bound + shares = [s.share for s in streams if s.share != "positives"] + self.assertEqual(shares, ["eval-speech", "eval-music", "eval-noise"]) # no eval-tts in the toy + for s in streams: + if s.share == "positives": + continue + want = sorted(c.id for c in eval_rows if c.share == s.share and c.label != 1) + self.assertEqual(s.members, want, s.id) + self.assertEqual(s.id, f"{s.share}/stream-0000") + self.assertEqual(s.negative_samples, sum(by_id[i].length for i in s.members)) + self.assertEqual(s.positives, []) + hours = kws_streams.hours_per_share(streams, self.scoring) + self.assertEqual(hours, {"eval-speech": 48000 / RATE / 3600, "eval-music": 32000 / RATE / 3600, + "eval-noise": 32000 / RATE / 3600}) + # the featurized eval negatives (speech + music; noise is never featurized) are the lock's own hours + self.assertAlmostEqual(hours["eval-speech"] + hours["eval-music"], + self.lock.summary["splits"]["eval"]["negative"]["hours"], places=15) + self.assertEqual(kws_streams.positives_per_subshare(streams), {"eval-speech": 1}) + kws_streams.validate_streams(streams, self.scoring) # decodes every stream: the accounting holds + # a negative stream is the concatenation of its members' pcm-tier clips, in member order + speech = next(s for s in streams if s.share == "eval-speech") + x = speech.samples() + self.assertEqual(x.dtype, np.float64) + parts = [kws_audio.from_int16(kws_audio.read_pcm( + self.store.pcm(by_id[i].source, kws_audio.DECODER_ID, kws_audio.resampler_id( + self.manifest.recipe.resampler.window)) / (i[len(by_id[i].source) + 1:] + ".wav"))) + for i in speech.members] + np.testing.assert_array_equal(x, np.concatenate(parts)) + print(f"\ntoy streams: {len(streams)} streams from {len(eval_rows)} eval rows; toy rebuild " + f"{self.seconds:.2f} s wall") + + def test_2_positive_mixture_is_what_extract_featurized(self) -> None: + p = next(s for s in self.streams if s.share == "positives") + row = next(c for c in self.lock.clips if c.id == p.id and c.variant == 0) + x = p.samples() + self.assertEqual(x.size, row.extra["mixture_samples"]) + g = self.manifest.recipe.geometry + det = kws_detectors.BandEnergyBaseline(g) + score = det.score(x) + self.assertEqual(score.shape, (row.extra["frames"],)) # the lock's frame count for this row + self.assertEqual(score.shape[0], kws_detectors.frames_for(x.size, g.hop)) + self.assertTrue(np.all((0.0 <= score) & (score <= 1.0))) + lo, hi = self.scoring.hit_window(p.positives[0].endpoint_hop) + self.assertGreaterEqual(lo, 0) + self.assertLess(hi, score.shape[0]) # the window `extract` reserved lies inside the rows + # and the features through the bridge are the committed eval shard's rows for this clip + with np.load(EXPECTED / "features" / "eval" / "shard-0000.npz") as shard: + ids = [str(i) for i in shard["clip_ids"]] + k = ids.index(p.id) + a, b = int(shard["clip_offsets"][k]), int(shard["clip_offsets"][k + 1]) + expected = shard["features"][a:b].astype(np.float64) + self.assertEqual(int(shard["clip_endpoints"][k]), p.positives[0].endpoint_hop) + got = det.features(x).astype(np.float32).astype(np.float64) + self.assertEqual(got.shape, expected.shape) + diff = np.abs(got - expected) + self.assertTrue(np.all(diff <= FEATURE_TOLERANCE * np.maximum(1.0, np.abs(expected))), + f"max |bridge - shard| = {diff.max():.3e} (measured 0.0 on the M0 Mac)") + + def test_3_the_packing_bound_never_splits_a_clip(self) -> None: + # the toy's eval-speech negatives are three 1 s clips: at a 1 s bound each stands alone, at 2 s + # the first two share a stream, and a bound below one clip still yields whole clips + for max_s, want in ((1.0, [1, 1, 1]), (2.0, [2, 1]), (0.5, [1, 1, 1]), (60.0, [3])): + streams = kws_streams.streams_from_lock(self.manifest, self.lock, self.store, self.scoring, + shares=("eval-speech",), max_stream_s=max_s) + speech = [s for s in streams if s.share == "eval-speech"] + self.assertEqual([len(s.members) for s in speech], want, max_s) + self.assertEqual([s.id for s in speech], + [f"eval-speech/stream-{n:04d}" for n in range(len(want))]) + self.assertEqual(sum(s.negative_samples for s in speech), 48000) # the hours never change + self.assertEqual(sorted(i for s in speech for i in s.members), + [i for s in speech for i in s.members]) # id order across the streams + kws_streams.validate_streams(streams, self.scoring) + + def test_4_the_assembly_refuses_a_mismatched_scoring_share_or_lock(self) -> None: + m, lock, store, sc = self.manifest, self.lock, self.store, self.scoring + with self.assertRaises(kws_streams.StreamError) as cm: + kws_streams.streams_from_lock(m, lock, store, dataclasses.replace(sc, hop=80)) + self.assertIn("scoring.hop 80", str(cm.exception)) + with self.assertRaises(kws_streams.StreamError) as cm: + kws_streams.streams_from_lock(m, lock, store, dataclasses.replace(sc, tolerance_hops=5)) + self.assertIn("tolerance_hops 5", str(cm.exception)) + with self.assertRaises(kws_streams.StreamError) as cm: + kws_streams.streams_from_lock(m, lock, store, dataclasses.replace(sc, latency_hops=10)) + self.assertIn("latency_hops 10", str(cm.exception)) + with self.assertRaises(kws_streams.StreamError) as cm: + kws_streams.streams_from_lock(m, lock, store, sc, shares=("eval-speech", "dev")) + self.assertIn("'dev'", str(cm.exception)) + with self.assertRaises(kws_streams.StreamError): + kws_streams.streams_from_lock(m, lock, store, sc, max_stream_s=0.0) + other = copy.deepcopy(lock) + other.manifest_hash = "0" * 64 + with self.assertRaises(kws_streams.StreamError) as cm: + kws_streams.streams_from_lock(m, other, store, sc) + self.assertIn("built from manifest " + "0" * 64, str(cm.exception)) + # a share the lock does not have simply yields no streams of it (reported as absent downstream) + only_tts = kws_streams.streams_from_lock(m, lock, store, sc, shares=("eval-tts",)) + self.assertEqual(only_tts, []) + + def test_5_the_cli_refuses_a_missing_store_by_name(self) -> None: + argv = ["--manifest", str(MANIFEST), "--lock", str(EXPECTED / "lock.json")] + err = io.StringIO() + with mock.patch.dict(os.environ), contextlib.redirect_stderr(err): + os.environ.pop(ENV_VAR, None) + self.assertEqual(kws_streams.main(argv), 2) + self.assertTrue(err.getvalue().startswith("refused: no feature store"), err.getvalue()) + with contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(kws_streams.main(argv + ["--store", str(self.store.root)]), 0) + + +class Validation(unittest.TestCase): + """The brief's item 2: every mis-accounting refused by name by validate_streams.""" + + def setUp(self) -> None: + self.sc = Scoring() # hop 160, T 3, L 20: the window of an endpoint at 9000 is [53, 79] + + def _refused(self, streams, *needles: str) -> None: + with self.assertRaises(kws_streams.StreamError) as cm: + kws_streams.validate_streams(streams, self.sc) + for needle in needles: + self.assertIn(needle, str(cm.exception)) + + def test_a_valid_list_passes_eager_or_lazy(self) -> None: + lazy = negative_stream("n1", 16000, audio=None, load=lambda: np.zeros(16000)) + kws_streams.validate_streams([positive_stream("p1", 16000, 9000, self.sc), lazy], self.sc) + + def test_negative_samples_must_equal_the_decoded_length(self) -> None: + self._refused([negative_stream("n1", 16000, negative_samples=16001)], "stream 'n1'", + "negative_samples 16001 != its 16000 decoded samples") + self._refused([negative_stream("n1", 16000, negative_samples=-16000)], "stream 'n1'", + "negative_samples") + self._refused([negative_stream("n1", 16000, negative_samples=16000.0)], "must be an integer") + + def test_a_positive_beyond_its_stream_is_refused(self) -> None: + self._refused([positive_stream("p1", 16000, 16001, self.sc)], "positive 'p1/kw'", "beyond its 16000") + # the endpoint is inside the audio but the window [h - T, h + L + T] is not: 100 hops, window to 118 + self._refused([positive_stream("p1", 16000, 15200, self.sc)], "window [92, 118]", + "beyond the stream's 100 hops") + ok = positive_stream("p1", 16000, 12000, self.sc) # window [72, 98] fits + kws_streams.validate_streams([ok], self.sc) + bad_hop = positive_stream("p1", 16000, 9000, self.sc) + bad_hop.positives = [Positive("p1/kw", 9000, 57)] + self._refused([bad_hop], "endpoint_hop 57 != 56") + + def test_duplicate_ids_are_refused(self) -> None: + self._refused([negative_stream("n1", 16000), negative_stream("n1", 16000)], "stream 'n1'", + "duplicate") + a = positive_stream("p1", 16000, 9000, self.sc) + b = positive_stream("p2", 16000, 9000, self.sc) + b.positives = list(a.positives) + self._refused([a, b], "positive 'p1/kw' appears twice") + + def test_a_positive_stream_with_negative_hours_is_refused(self) -> None: + self._refused([positive_stream("p1", 16000, 9000, self.sc, negative_samples=-16000)], "stream 'p1'", + "positive stream with negative_samples -16000") + self._refused([positive_stream("p1", 16000, 9000, self.sc, negative_samples=16000)], + "positive stream with negative_samples 16000") + + def test_empty_unknown_and_malformed_streams_are_refused(self) -> None: + self._refused([], "no streams") + self._refused([negative_stream("n1", 0)], "stream 'n1'", "empty stream") + self._refused([negative_stream("n1", 16000, share="train")], "unknown share 'train'") + self._refused([negative_stream("n1", 16000, audio=None)], "neither audio nor a loader") + self._refused([negative_stream("n1", 16000, audio=np.zeros((2, 8000)))], "one-dimensional") + self._refused([positive_stream("p1", 16000, 9000, self.sc, positives=[])], "no positives") + self._refused([positive_stream("p1", 16000, 9000, self.sc, subshare="eval-music")], + "subshare 'eval-music'") + neg = negative_stream("n1", 16000) + neg.positives = [Positive("x", 9000, 56)] + self._refused([neg], "eval-speech stream carrying 1 positive") + + +class Detectors(unittest.TestCase): + """The band-energy baseline through the bridge, and the planted detector.""" + + def test_band_energy_alignment_on_a_planted_1khz_burst(self) -> None: + # Measured 9 September 2026 on the M0 Mac at the reference geometry (bursts ending at 24000, 24080, + # 24159, 24001, 8400 in digital silence), h = e // 160: argmax <= h + 1; s[h] >= 0.321757 (the + # minimum, for a burst ending early in its hop) and s[h + 1] >= 0.961318 (the offset edge splatters + # across every band); s[h + 2] is 0.0 except for the burst ending late in its hop (24159), where it + # reads 0.979018; 0 from h + 3 on; 0 before hop start // 160 and > 0 at it. + g = kws_features.Geometry() + det = kws_detectors.BandEnergyBaseline(g) + self.assertEqual(det.name, "band-energy") + self.assertEqual(det.bands, list(range(5, 27))) # centres 329.7 .. 2901.9 Hz + self.assertEqual(det.params["bands"], det.bands) + self.assertEqual(det.params["stored_path"], "log") + self.assertEqual(det.params["band_centres_hz"][0], 329.698) # band 5's centre, rounded to 3 places + self.assertEqual(len(det.params["band_centres_hz"]), 22) + for start, end in ((16000, 24000), (16000, 24080), (16000, 24159), (16003, 24001), (8000, 8400)): + x = np.zeros(48000) + t = np.arange(start, end) + x[start:end] = 0.5 * np.sin(2.0 * np.pi * 1000.0 * t / g.sample_rate) + s = det.score(x) + h = end // g.hop + self.assertEqual(s.shape, (300,)) + self.assertEqual(s.dtype, np.float64) + self.assertLessEqual(int(np.argmax(s)), h + 1, (start, end)) + self.assertGreaterEqual(s[h], 0.3, (start, end, s[h])) # measured minimum 0.321757 + self.assertGreaterEqual(s[h + 1], 0.3, (start, end, s[h + 1])) # measured minimum 0.961318 + self.assertLessEqual(s[h + 2], 0.99, (start, end, s[h + 2])) # measured 0.979018 at most + self.assertTrue(np.all(s[h + 3:] == 0.0), (start, end)) + self.assertTrue(np.all(s[:start // g.hop] == 0.0), (start, end)) + self.assertGreater(s[start // g.hop], 0.0) + # the frame count is n // hop for every length, through the bridge + for n in (0, 159, 160, 161, 400, 62555): + self.assertEqual(det.score(np.zeros(n)).shape, (n // g.hop,), n) + self.assertEqual(kws_detectors.frames_for(n, g.hop), n // g.hop) + # a PCEN geometry scores through the plain-log path: identical scores, and the params say so + pcen = dataclasses.replace(g, pcen=dataclasses.replace(g.pcen, enabled=True)) + det_pcen = kws_detectors.BandEnergyBaseline(pcen) + x = np.random.default_rng(1).standard_normal(16000) * 0.1 + np.testing.assert_array_equal(det_pcen.score(x), det.score(x)) + self.assertFalse(det_pcen.params["geometry"]["pcen"]["enabled"]) + + def test_band_energy_refuses_a_band_with_no_centre(self) -> None: + g = kws_features.Geometry() + with self.assertRaises(kws_detectors.DetectorError) as cm: + kws_detectors.BandEnergyBaseline(g, 7200.0, 7300.0) # the top centre is 7119.6 Hz + self.assertIn("no mel band centre lies in [7200.0, 7300.0] Hz", str(cm.exception)) + with self.assertRaises(kws_detectors.DetectorError): + kws_detectors.BandEnergyBaseline(g, 3000.0, 300.0) + with self.assertRaises(kws_detectors.DetectorError): + kws_detectors.BandEnergyBaseline(g).score(np.zeros((2, 800))) + + def test_planted_detector(self) -> None: + pd = kws_detectors.PlantedDetector({"a": [(10, 0.5), (3, 0.9)], "b": []}, hop=160) + self.assertEqual(pd.name, "planted") + self.assertEqual(pd.params, {"hop": 160, "planted": {"a": [[3, 0.9], [10, 0.5]], "b": []}}) + s = pd.score_stream("a", 20 * 160 + 159) + self.assertEqual(s.shape, (20,)) + self.assertEqual((s[3], s[10], s.sum()), (0.9, 0.5, 1.4)) + self.assertEqual(pd.score_stream("b", 800).tolist(), [0.0] * 5) + self.assertEqual(pd.score_stream("unplanted", 800).tolist(), [0.0] * 5) + with self.assertRaises(kws_detectors.DetectorError) as cm: + pd.score_stream("a", 10 * 160) # hop 10 is beyond a 10-hop stream + self.assertIn("planted hop 10 is beyond the stream's 10 hops", str(cm.exception)) + with self.assertRaises(kws_detectors.DetectorError): + pd.score(np.zeros(1600)) + for bad in ({"a": [(3, 1.5)]}, {"a": [(-1, 0.5)]}, {"a": [(3, 0.5), (3, 0.6)]}, {"a": [(3,)]}): + with self.assertRaises(kws_detectors.DetectorError): + kws_detectors.PlantedDetector(bad, 160) + with self.assertRaises(kws_detectors.DetectorError): + kws_detectors.PlantedDetector({}, 0) + + +class HoldoutRecord(unittest.TestCase): + """The brief's item 5, over a synthetic holdout.json and a temporary store.""" + + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory(prefix="mutap-kws-holdout-") + self.dir = pathlib.Path(self.tmp.name) + self.store = Store(self.dir / "store") + self.store.holdout().mkdir(parents=True) + self.sc = Scoring() + self.files = {} + for talker, rel, endpoint in (("T01", "T01/room-a/take-001.flac", 24000), + ("T02", "T02/room-b/take-007.flac", 30000)): + self.files[rel] = self._write_flac(rel, endpoint, RATE) + self.doc = { + "holdout_version": kws_holdout.HOLDOUT_JSON_VERSION, + "talkers": [{"pseudonym": "T01", "consent_form_version": "consent-v1", + "permitted_uses": ["evaluation", "m7-replay"]}, + {"pseudonym": "T02", "consent_form_version": "consent-v1", + "permitted_uses": ["m7-replay", "evaluation", "publication"]}], + "utterances": [ + {"file": "T01/room-a/take-001.flac", "sha256": self.files["T01/room-a/take-001.flac"], + "talker": "T01", "microphone_path": "close", "distance_m": 0.3, "snr_db": None, + "phrase": "marvin", "endpoint_sample": 24000}, + {"file": "T02/room-b/take-007.flac", "sha256": self.files["T02/room-b/take-007.flac"], + "talker": "T02", "microphone_path": "room", "distance_m": 2.0, "snr_db": 10.0, + "phrase": "marvin", "endpoint_sample": 30000}]} + + def tearDown(self) -> None: + self.tmp.cleanup() + + def _write_flac(self, rel: str, endpoint: int, fs: int) -> str: + p = self.store.holdout() / rel + p.parent.mkdir(parents=True, exist_ok=True) + x = np.zeros(3 * fs) + t = np.arange(endpoint - 8000, endpoint) + x[endpoint - 8000:endpoint] = 0.4 * np.sin(2.0 * np.pi * 440.0 * t / fs) + soundfile.write(str(p), x, fs, format="FLAC", subtype="PCM_16") + return sha256_file(p) + + def _holdout(self, doc: dict) -> kws_holdout.Holdout: + path = self.dir / "holdout.json" + path.write_text(json.dumps(doc, indent=1) + "\n", encoding="utf-8") + return kws_holdout.load_holdout(path) + + def _refused(self, doc: dict, *needles: str, at_load: bool = False) -> None: + with self.assertRaises(kws_holdout.HoldoutError) as cm: + h = self._holdout(doc) + if at_load: + self.fail("refusal expected at load") + kws_holdout.verify_holdout(h, self.store) + for needle in needles: + self.assertIn(needle, str(cm.exception)) + + def test_5a_a_matching_tier_verifies_and_yields_positive_streams(self) -> None: + h = self._holdout(self.doc) + kws_holdout.verify_holdout(h, self.store) + set_id = kws_holdout.holdout_set_id(h) + self.assertRegex(set_id, r"^[0-9a-f]{64}$") + streams = kws_holdout.streams_from_holdout(h, self.store, self.sc) + self.assertEqual([s.id for s in streams], + ["holdout/T01/room-a/take-001", "holdout/T02/room-b/take-007"]) + for s, u in zip(streams, h.utterances): + self.assertEqual((s.share, s.subshare, s.negative_samples, s.members), + ("positives", "holdout", 0, [u.file])) + self.assertEqual(s.positives, [Positive(s.id, u.endpoint_sample, u.endpoint_sample // 160)]) + self.assertEqual(s.samples().size, 3 * RATE) + kws_streams.validate_streams(streams, self.sc) + self.assertEqual(kws_streams.positives_per_subshare(streams), {"holdout": 2}) + # the round trip and the CLI + kws_holdout.save_holdout(h, self.dir / "again.json") + self.assertEqual(kws_holdout.load_holdout(self.dir / "again.json").to_dict(), h.to_dict()) + self.assertEqual(kws_holdout.main(["--holdout", str(self.dir / "holdout.json"), + "--store", str(self.store.root)]), 0) + + def test_5b_one_altered_byte_in_a_flac_is_refused_naming_the_file(self) -> None: + p = self.store.holdout() / "T02/room-b/take-007.flac" + data = bytearray(p.read_bytes()) + data[len(data) // 2] ^= 0x01 + p.write_bytes(bytes(data)) + self._refused(self.doc, "utterance 'T02/room-b/take-007.flac'", "sha256", "!= the recorded") + self.assertEqual(kws_holdout.main(["--holdout", str(self.dir / "holdout.json"), + "--store", str(self.store.root)]), 2) + + def test_5c_talker_and_consent_refusals_by_name(self) -> None: + missing = copy.deepcopy(self.doc) + missing["utterances"][1]["talker"] = "T99" + self._refused(missing, "utterance 'T02/room-b/take-007.flac'", "talker 'T99' has no talkers[] row") + no_version = copy.deepcopy(self.doc) + del no_version["talkers"][0]["consent_form_version"] + self._refused(no_version, "utterance 'T01/room-a/take-001.flac'", + "talker 'T01' has no consent_form_version") + null_version = copy.deepcopy(self.doc) + null_version["talkers"][0]["consent_form_version"] = None + self._refused(null_version, "talker 'T01' has no consent_form_version") + for dropped in ("evaluation", "m7-replay"): + doc = copy.deepcopy(self.doc) + doc["talkers"][1]["permitted_uses"].remove(dropped) + self._refused(doc, "utterance 'T02/room-b/take-007.flac'", "talker 'T02' permitted_uses", + f"lack ['{dropped}']") + no_uses = copy.deepcopy(self.doc) + del no_uses["talkers"][1]["permitted_uses"] + self._refused(no_uses, "lack ['evaluation', 'm7-replay']") + + def test_5d_the_set_id_changes_when_a_row_is_dropped(self) -> None: + full = kws_holdout.holdout_set_id(self._holdout(self.doc)) + dropped = copy.deepcopy(self.doc) + del dropped["utterances"][1] + self.assertNotEqual(kws_holdout.holdout_set_id(self._holdout(dropped)), full) + reordered = copy.deepcopy(self.doc) + reordered["utterances"].reverse() + self.assertEqual(kws_holdout.holdout_set_id(self._holdout(reordered)), full) # order-free + + def test_5d2_one_file_is_one_utterance(self) -> None: + # the same FLAC under a second spelling is refused by name at load: a './' or '//' spelling by the + # path rule, a case-folded spelling (which APFS resolves to the file, Linux does not) by its digest + # being already recorded — so neither the recall denominator nor the set id can count a file twice + first = self.doc["utterances"][0] + for spelling in ("./" + first["file"], first["file"].replace("room-a/", "room-a//"), + first["file"].replace("T01/", "T01/./")): + doc = copy.deepcopy(self.doc) + doc["utterances"].append({**first, "file": spelling}) + self._refused(doc, repr(spelling), "relative POSIX path", at_load=True) + doc = copy.deepcopy(self.doc) + doc["utterances"].append({**first, "file": first["file"].replace("take-001", "TAKE-001")}) + self._refused(doc, "utterance 'T01/room-a/TAKE-001.flac'", "sha256 already recorded for utterance " + "'T01/room-a/take-001.flac'", at_load=True) + # the set id counts each digest once even on a record that bypassed validate() + h = self._holdout(self.doc) + doubled = dataclasses.replace(h, utterances=h.utterances + (h.utterances[0],)) + self.assertEqual(kws_holdout.holdout_set_id(doubled), kws_holdout.holdout_set_id(h)) + # and two rows that resolve to one file are refused by verify_holdout even if both spellings pass + # the path rule (a symlink beside the master); the digest rule fires first at load, so the row + # carries a wrong digest to reach verify + link = self.store.holdout() / "T01/room-a/alias.flac" + link.symlink_to(self.store.holdout() / "T01/room-a/take-001.flac") + doc = copy.deepcopy(self.doc) + doc["utterances"].append({**first, "file": "T01/room-a/alias.flac", "sha256": "0" * 64}) + self._refused(doc, "utterance 'T01/room-a/alias.flac'", "names the same file as utterance " + "'T01/room-a/take-001.flac'") + + def test_5f_the_holdout_cli_scores_a_verified_record_and_refuses_an_altered_one(self) -> None: + # kws_eval.py holdout: verify, then score through the band-energy baseline; the report carries the + # record's set id and its two positives; after one flipped byte nothing is written + self._holdout(self.doc) + out = self.dir / "report" + argv = ["holdout", "--holdout", str(self.dir / "holdout.json"), "--manifest", str(MANIFEST), + "--store", str(self.store.root), "--thresholds", "5", "--out", str(out)] + with contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(kws_eval.main(argv), 0) + report = kws_eval.read_report(out / "report.json") + self.assertEqual(report.holdout_set_id, kws_holdout.holdout_set_id(kws_holdout.load_holdout( + self.dir / "holdout.json"))) + self.assertEqual(report.positives, {"holdout": 2}) + self.assertEqual(report.hours, {}) # no lock: no negative share, no FA/h column that is not absent + self.assertIsNone(report.max_stream_s) + self.assertTrue((out / "report.md").is_file()) + p = self.store.holdout() / "T02/room-b/take-007.flac" + data = bytearray(p.read_bytes()) + data[len(data) // 2] ^= 0x01 + p.write_bytes(bytes(data)) + err = io.StringIO() + with contextlib.redirect_stderr(err): + self.assertEqual(kws_eval.main(argv[:-1] + [str(out / "altered")]), 1) + self.assertIn("utterance 'T02/room-b/take-007.flac': sha256", err.getvalue()) + self.assertFalse((out / "altered").exists()) + # and the verifier CLI refuses a missing store by name, rc 2, never a traceback + err = io.StringIO() + with mock.patch.dict(os.environ), contextlib.redirect_stderr(err): + os.environ.pop(ENV_VAR, None) + self.assertEqual(kws_holdout.main(["--holdout", str(self.dir / "holdout.json")]), 2) + self.assertTrue(err.getvalue().startswith("refused: no feature store"), err.getvalue()) + + def test_5e_schema_path_and_tier_refusals(self) -> None: + def mutate(fn): + doc = copy.deepcopy(self.doc) + fn(doc) + return doc + first = lambda d: d["utterances"][0] # noqa: E731 + self._refused(mutate(lambda d: first(d).update(bogus=1)), "utterance 'T01/room-a/take-001.flac'", + "unknown field(s) ['bogus']", at_load=True) + self._refused(mutate(lambda d: first(d).pop("phrase")), "missing required field(s) ['phrase']", + at_load=True) + self._refused(mutate(lambda d: d["talkers"][0].update(email="x")), "talker 'T01'", "['email']", + at_load=True) + self._refused(mutate(lambda d: d.update(holdout_version=2)), "holdout_version 2", at_load=True) + self._refused(mutate(lambda d: d.update(extra=1)), "unknown top-level field(s) ['extra']", + at_load=True) + for bad in ("../outside.flac", "/abs/take.flac", "T01\\take.flac", "T01/take.wav", ""): + self._refused(mutate(lambda d, bad=bad: first(d).update(file=bad)), repr(bad), at_load=True) + self._refused(mutate(lambda d: first(d).update(sha256="abc")), "64 lowercase hex", at_load=True) + self._refused(mutate(lambda d: first(d).update(endpoint_sample=-1)), "endpoint_sample", at_load=True) + self._refused(mutate(lambda d: first(d).update(file="T02/room-b/take-007.flac")), "listed twice", + at_load=True) + self._refused(mutate(lambda d: d["talkers"].append(dict(d["talkers"][0]))), "duplicate pseudonym", + at_load=True) + # the tier: a missing file, a stray FLAC no row names, no tier at all + self._refused(mutate(lambda d: d["utterances"].pop(1)), "holds 1 FLAC(s) no holdout.json row names", + "T02/room-b/take-007.flac") + (self.store.holdout() / "T02/room-b/take-007.flac").unlink() + self._refused(self.doc, "utterance 'T02/room-b/take-007.flac'", "is missing from the hold-out tier") + # a listed FLAC that is not 16 kHz is refused when its stream is decoded, by name + sha = self._write_flac("T02/room-b/take-007.flac", 30000, 22050) + doc = mutate(lambda d: d["utterances"][1].update(sha256=sha)) + h = self._holdout(doc) + kws_holdout.verify_holdout(h, self.store) + streams = kws_holdout.streams_from_holdout(h, self.store, self.sc) + with self.assertRaises(kws_holdout.HoldoutError) as cm: + streams[1].samples() + self.assertIn("utterance 'T02/room-b/take-007.flac': 22050 Hz", str(cm.exception)) + with self.assertRaises(kws_holdout.HoldoutError): + kws_holdout.verify_holdout(h, Store(self.dir / "no-such-store")) + + +if __name__ == "__main__": + unittest.main()