From 0d1fc83767a7b9197c9ee92bc096587b8b1d31bc Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Fri, 6 Nov 2026 04:57:40 +0000 Subject: [PATCH 1/3] docs(compat): register AAC compatibility plan Signed-off-by: gunwooterry --- docs/design/github-shapes-200.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/design/github-shapes-200.md b/docs/design/github-shapes-200.md index 3b68df1a8..58ba90a45 100644 --- a/docs/design/github-shapes-200.md +++ b/docs/design/github-shapes-200.md @@ -65,7 +65,7 @@ Unlike V2's lossy quality predicates, this measurement requires exact decoded eq The headline is `counted shapes / 200`. The weighted headline is the sum of repository counts for counted shapes divided by 1,470. It is the covered share of the top-200 repository-shape pairs, not 200 unique repositories and not 44% of real executions. Also report the numerator over all 3,338 sample pairs, explicitly as a lower-bound sample presence share. There is no bar, required count, release verdict or performance claim. -Each shape receives exactly one result in this order: `uninstantiable`, `reference_failed`, `timeout`, `refused`, `failed`, `unexpected_program`, `no_oracle`, `different`, `compatible_avc1`, or `counted`. The separately registered `compatible_avc1` outcome below is a miss for the exact headline. Break out both shape counts and repository weights for every miss class and both tools. A malformed fallback log, wrong oracle, changed registration, changed fixture or binary, or missing/duplicate result makes the record incomplete rather than reducing a denominator. +Each shape receives exactly one result in this order: `uninstantiable`, `reference_failed`, `timeout`, `refused`, `failed`, `unexpected_program`, `no_oracle`, `different`, `compatible_avc1`, `compatible_aac`, or `counted`. The separately registered compatibility outcomes below are misses for the exact headline. Break out both shape counts and repository weights for every miss class and both tools. A malformed fallback log, wrong oracle, changed registration, changed fixture or binary, or missing/duplicate result makes the record incomplete rather than reducing a denominator. An acquisition bundle retains generated fixtures, commands, binary identities, complete stdout, output files, fallback and program logs, and per-command process outcomes. The manifest binds evidence and method files by SHA-256. Replay verifies these identities, regenerates the commands from the frozen set, checks the complete roster and reapplies the output comparisons with the pinned oracle. It does not trust stored pass booleans or summary totals. Replacing a result label cannot create coverage. Artifacts use only generated media and relative names. Summaries contain closed reason labels, not arbitrary process diagnostics or machine paths. @@ -79,6 +79,26 @@ An acquisition bundle retains generated fixtures, commands, binary identities, c No commands-v1, commands-0.1.0, judging policy, known-divergence record, held-out input, existing bar, tolerance, baseline or V1 to V3 record changes. +## Separate AAC compatibility registration + +Registered on 2026-11-06 before implementation or decoder comparison. The maintainer requested that qualified AAC decode-tolerance commands count as compatible, separately from exact output, following coreav-dev/coreav#433. The new `compatible_aac` result never increases `counted`, `counted_pairs`, `top_200_pair_share` or `whole_sample_presence_lower_bound`. PNS is never exact. Frozen results, gate records, published numbers, population, fixtures and command assignments do not change. A new measurement result requires a separate later submission. + +### Predicate and scope + +Reuse the complete [AAC PCM qualification](aac-pcm-eligibility.md) and its [PNS extension](aac-pns-eligibility.md), including the existing production preflight. This section adds AAC scope separately; the avc1 registration's statement that it does not admit AAC tolerance describes only that registration. There is no new tolerance. No-PNS PCM requires complete finite F32 samples within 2^-16 and actual S16 samples within one LSB. PNS requires independently checked per-band energies within 0.1 dB for both decoders, the registered deterministic-bin and non-PNS projection bars, each decoder's actual S16 packing against its own F32, and exact metadata-predicted presentation intervals. Cross-decoder pointwise comparison of complete PNS waveforms is neither required nor sufficient. + +Only a single seekable MP4 input with one explicit mono or stereo AAC-LC audio track and a single audio-only output can qualify. Source video may be present but cannot be output. Retain the qualification's bounded nonfragmented timelines, complete packet scan, clean decoding, window-history rules and excluded PNS tool combinations. Output is raw F32, raw S16 or S16 WAV, with native rate and channel count. Explicit `-ar` and `-ac` must be identities. Permit only the existing default audio selection, optional input `-threads 1`, output `-vn`, `-c:a`, `-f`, `-ar` and `-ac`. The complete reference mapping must show AAC decoding to the corresponding PCM encoder, including when encoder selection is implicit. No seek, trim, filter, mixing, other encode, copied stream, multiple input or multiple output is qualified. + +The candidate ranks within this scope are 5, 13, 33, 43, 53, 57, 67, 72, 77, 89, 98, 139, 168 and 174. These are candidates, not observed passes. Ranks 15 and 188 use input pipes, which the existing seekable-input qualification does not cover. Rank 44 writes PCM in MP4, whose complete framing is not qualified. F32 WAV, other containers and unsupported or specification-undecidable presentations remain out. Do not infer qualification from a rank or codec label. + +### Implementation and verification order + +1. Commit this plan first. Implement only measurement tooling, qualification-output export, generated controls and workflow wiring. Do not change native decoding, MP3 or any frozen artifact. +2. For an eligible command, rerun the existing complete qualification with the same bound CoreAV, pinned FFmpeg 9.0.2 and opt-in diagnostic binaries. Export the qualified raw outputs for both arms without changing acceptance. Require each measured payload to equal its own arm's qualified payload byte for byte. This binds actual outputs to the numerical proof and exact count, priming and channel order. A content-admission flag or stored pass label cannot substitute for this proof. +3. Require everything outside the AAC PCM payload to be exact. Raw outputs have no framing. For S16 WAV require valid complete extents, sample fields and byte-identical framing and metadata, with payloads equal to their respective qualified raw outputs. File-output stdout must be empty, with no undeclared files or external encoder starts. A framing or payload mutation fails even if separate qualification passes. Compare WAV framing against the same-command reference output, not the qualification's metadata-discard control. Keep the exact decoded comparator first. Only a decoded-sample mismatch may use tolerance; other mismatches remain misses. Before exact credit for AAC-decoded PCM, a successful complete diagnostic scan must establish that no packet contains PNS. Unknown diagnostics fail closed. Even accidentally equal PNS samples require full qualification and receive only compatible credit. Qualification failures remain `different`, not `no_oracle`. +4. Keep existing non-AAC exact and avc1 behavior. Report the AAC category and its repository weights separately. Replay requalifies the bound input and checks actual output bytes, rather than trusting labels. Bind diagnostic build command, binary digest, qualification dependencies and method files to the acquisition manifest. Historical manifests cannot be rejudged under the new method. +5. Before merge, run only independent generated controls and fake-executable or pure-accounting tests. Cover no-PNS and separable PNS, both PCM widths, raw file and stdout, S16 WAV file and stdout, identity options, unsupported profiles or timelines or PNS tools, count drift, nonfinite samples, changed payloads, metadata and framing, and separate shape and weighted accounting. Use the verified pinned oracle and unchanged transform controls. Do not execute judged commands, the judged fixture generator, held-out inputs or retained judged outputs with this predicate before merge. + ## Separate avc1 compatibility registration Registered on 2026-11-06 before implementation or execution of this predicate. The maintainer permits a separate `compatible_avc1` outcome for copied H.264 parameter-set carriage. Like the separate [AAC decode-tolerance qualification](aac-pcm-eligibility.md), it never increases an exact-output count. The existing `counted` predicate, frozen exact results, published numbers, population, fixtures and gate records stay unchanged. This registration does not admit AAC tolerance into this measurement. From e1790f16c8585893c31cf0dece1f35daf3c8ee4f Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Fri, 6 Nov 2026 04:59:35 +0000 Subject: [PATCH 2/3] docs(compat): guard PNS exact credit by decoder Signed-off-by: gunwooterry --- docs/design/github-shapes-200.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/design/github-shapes-200.md b/docs/design/github-shapes-200.md index 58ba90a45..3e3586795 100644 --- a/docs/design/github-shapes-200.md +++ b/docs/design/github-shapes-200.md @@ -91,6 +91,8 @@ Only a single seekable MP4 input with one explicit mono or stereo AAC-LC audio t The candidate ranks within this scope are 5, 13, 33, 43, 53, 57, 67, 72, 77, 89, 98, 139, 168 and 174. These are candidates, not observed passes. Ranks 15 and 188 use input pipes, which the existing seekable-input qualification does not cover. Rank 44 writes PCM in MP4, whose complete framing is not qualified. F32 WAV, other containers and unsupported or specification-undecidable presentations remain out. Do not infer qualification from a rank or codec label. +The exact-credit guard follows AAC decoding in the reference mapping, not the output codec. Every non-copy output decoded from AAC needs a successful complete stored-packet diagnostic scan showing no PNS before exact credit. Unknown scans and PNS in unqualified output forms are `different`. This can only lower exact credit; it does not qualify an encoded output or alter its native path. Copied AAC keeps its existing packet and decoded predicates. The scan never reads a judged input before merge. + ### Implementation and verification order 1. Commit this plan first. Implement only measurement tooling, qualification-output export, generated controls and workflow wiring. Do not change native decoding, MP3 or any frozen artifact. From a090444d25525f302ccfe1dc6a71498f91514aac Mon Sep 17 00:00:00 2001 From: gunwooterry Date: Fri, 6 Nov 2026 05:06:55 +0000 Subject: [PATCH 3/3] feat(compat): count AAC tolerance separately Signed-off-by: gunwooterry --- .github/workflows/github-shapes.yml | 42 +++ docs/agents/compat-corpus.md | 1 + scripts/aac_pcm_eligibility.py | 8 +- tools/compat-corpus/github_shapes.py | 43 ++- tools/compat-corpus/github_shapes_aac.py | 138 ++++++++ .../compat-corpus/tests/test_github_shapes.py | 10 +- .../tests/test_github_shapes_aac.py | 324 ++++++++++++++++++ 7 files changed, 554 insertions(+), 12 deletions(-) create mode 100644 tools/compat-corpus/github_shapes_aac.py create mode 100644 tools/compat-corpus/tests/test_github_shapes_aac.py diff --git a/.github/workflows/github-shapes.yml b/.github/workflows/github-shapes.yml index a5d6db069..eefe47918 100644 --- a/.github/workflows/github-shapes.yml +++ b/.github/workflows/github-shapes.yml @@ -14,6 +14,13 @@ on: - scripts/install-oracle-ffmpeg.sh - scripts/install-x264.sh - scripts/oracle* + - scripts/aac_pcm_eligibility.py + - scripts/aac_pns.py + - scripts/aac-pns-requirements.txt + - docs/design/aac-pcm-eligibility.md + - docs/design/aac-pns-eligibility.md + - docs/compat/audio-decoder-eligibility.json + - tools/compat-corpus/tests/test_github_shapes_aac.py workflow_dispatch: inputs: commit: @@ -71,10 +78,14 @@ jobs: sudo apt-get install -y --no-install-recommends libdav1d-dev libvpx-dev libopus-dev nasm pkg-config scripts/install-oracle-ffmpeg.sh "$RUNNER_TEMP/oracle" scripts/install-x264.sh "$RUNNER_TEMP/oracle" + python3 -m venv "$RUNNER_TEMP/aac-python" + "$RUNNER_TEMP/aac-python/bin/pip" install --quiet --only-binary=:all: --require-hashes -r scripts/aac-pns-requirements.txt + echo "$RUNNER_TEMP/aac-python/bin" >> "$GITHUB_PATH" - name: Build the full feature set run: | set -euo pipefail cargo build --release --locked -p coreav-cli --features release-full + cargo build --locked --release -p coreav-aac --features diagnostics --example pns_diagnostic python3 - <<'PY' import hashlib, json, os from pathlib import Path @@ -91,6 +102,7 @@ jobs: python3 tools/compat-corpus/github_shapes.py run --commit "$COMMIT" \ --build-record "$RUNNER_TEMP/shape-build.json" \ --coreav "$PWD/target/release/coreav" --ffmpeg "$RUNNER_TEMP/oracle/ffmpeg" \ + --pns-diagnostic "$PWD/target/release/examples/pns_diagnostic" \ --ffprobe "$RUNNER_TEMP/oracle/ffprobe" --x264 "$RUNNER_TEMP/oracle/x264" \ --bundle "$RUNNER_TEMP/shape-evidence" --scratch "$RUNNER_TEMP/shape-replay" \ --summary "$RUNNER_TEMP/shape-summary.json" @@ -104,3 +116,33 @@ jobs: ${{ runner.temp }}/shape-build.json if-no-files-found: error retention-days: 30 + aac-controls: + if: github.event_name == 'pull_request' + runs-on: ubuntu-24.04 + timeout-minutes: 15 + env: + OPENBLAS_NUM_THREADS: '1' + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + persist-credentials: false + - name: Install the pinned oracle and transform dependency + run: | + set -euo pipefail + scripts/install-oracle-ffmpeg.sh "$RUNNER_TEMP/aac-oracle" + python3 -m venv "$RUNNER_TEMP/aac-python" + "$RUNNER_TEMP/aac-python/bin/pip" install --quiet --only-binary=:all: --require-hashes -r scripts/aac-pns-requirements.txt + echo "$RUNNER_TEMP/aac-python/bin" >> "$GITHUB_PATH" + - name: Build the AAC command and opt-in diagnostic + run: | + cargo build --locked --release -p coreav-cli --features aac + cargo build --locked --release -p coreav-aac --features diagnostics --example pns_diagnostic + - name: Check independent AAC command controls + env: + COREAV_SHAPES_ORACLE: ${{ runner.temp }}/aac-oracle + COREAV_SHAPES_COREAV: ${{ github.workspace }}/target/release/coreav + COREAV_SHAPES_AAC_DIAGNOSTIC: ${{ github.workspace }}/target/release/examples/pns_diagnostic + run: | + python3 -I -m unittest discover -s tools/compat-corpus/tests -p test_github_shapes_aac.py + python3 -I -m unittest discover -s scripts/tests -p test_aac_pns.py + python3 -I -m unittest discover -s scripts/tests -p test_aac_pcm_eligibility.py diff --git a/docs/agents/compat-corpus.md b/docs/agents/compat-corpus.md index d851a3b04..bb96066cd 100644 --- a/docs/agents/compat-corpus.md +++ b/docs/agents/compat-corpus.md @@ -4,6 +4,7 @@ Corpus exclusions, compatibility workflows, fixture comparisons, the HEVC confor | Rule | Enforced by | |---|---| +| AAC compatibility binds each measured PCM payload to its own fully qualified decoder output, and no AAC-decoded PNS output earns exact credit | `github_shapes_aac` reuses the complete registered qualification, checks same-command WAV framing and scans every stored packet before AAC-decoded exact credit. Generated raw/WAV, PNS, payload/framing mutation, sibling-copy, encoded-PNS and separate-accounting controls in `test_github_shapes_aac.py` enforce the boundary. One qualification owner prevents tolerance drift; external process outputs and PNS content still require runtime proof rather than declared codec types | | New concat output qualification stays outside the registered frozen harness module graph | `ffmpeg_concat` and `ffmpeg_aac_filter` own the non-frozen comparator, while `test_the_harness_driver_and_its_path_modules_are_frozen` rejects an unregistered path module in `tests/ffmpeg.rs`. Separate test targets preserve the existing source registry without expanding frozen measurement material; a Rust module type cannot enforce that provenance boundary | | avc1 compatibility requires known presentation timestamps and positive durations for every presented video and audio frame in every output, without relaxing the exact category | `exact_frame_timeline` in `decoded_output.compare` owns compatibility-only validation; `avc1_carriage` and the runner's final output checks require it. Generated missing-video-time, nonpositive-duration, copied and encoded audio, and multi-output regressions in `test_github_shapes.py` enforce the boundary. External probe fields require runtime checks, not declared stream types | | HEVC oracle comparisons describe the base reconstruction, not optional enhancement-decoder output | `HEVC_DECODE_INPUT` owns `-export_side_data +enhancements` for frame probing, whole-output hashes, picture hashes and numeric diagnostics. Command regressions require the shared option before input on every decoded-pixel path; original geometry, complete-output and published-MD5 checks remain unchanged. A native type cannot govern external decoder options | diff --git a/scripts/aac_pcm_eligibility.py b/scripts/aac_pcm_eligibility.py index ccf37824d..09940c5b4 100644 --- a/scripts/aac_pcm_eligibility.py +++ b/scripts/aac_pcm_eligibility.py @@ -148,7 +148,7 @@ def presentation_interval(metadata, listing): return skip, count -def check(coreav, oracle, source, work, diagnostic): +def check(coreav, oracle, source, work, diagnostic, capture=None): with tempfile.TemporaryDirectory(prefix='aac-diagnostic-', dir=work) as path: directory = Path(path) captured = run([diagnostic, source, directory]) @@ -158,13 +158,13 @@ def check(coreav, oracle, source, work, diagnostic): noise = sum(len(channel['noise']) for frame in metadata['frames'] for channel in frame['channels']) if noise and not metadata['unsafe']: pns = pns_check(oracle, source, directory, metadata) - row = check_output(coreav, oracle, source, work, metadata, pns) + row = check_output(coreav, oracle, source, work, metadata, pns, capture) if captured.returncode: row['diagnostic_refusal'] = captured.stderr.decode()[-500:].strip() return row -def check_output(coreav, oracle, source, work, metadata, pns): +def check_output(coreav, oracle, source, work, metadata, pns, capture=None): row = {'sha256': hashlib.sha256(source.read_bytes()).hexdigest(), 'formats': {}} probe = run([oracle / 'ffprobe', '-v', 'error', '-select_streams', 'a:0', '-show_entries', 'stream=sample_rate,channels,profile', '-of', 'json', source]) @@ -274,6 +274,8 @@ def check_output(coreav, oracle, source, work, metadata, pns): compare_wav(wave_file.read_bytes(), pin_file.read_bytes(), actual.stdout, reference.stdout, row['channels'], row['rate']) row['formats'][fmt] = result + if capture is not None: + capture[fmt] = (actual.stdout, reference.stdout) row['status'] = 'ineligible' if refusal else 'pass' if refusal: row['reason'] = refusal diff --git a/tools/compat-corpus/github_shapes.py b/tools/compat-corpus/github_shapes.py index 60e801528..299dbbb8a 100644 --- a/tools/compat-corpus/github_shapes.py +++ b/tools/compat-corpus/github_shapes.py @@ -21,6 +21,7 @@ import github_shapes_fixtures as fixtures import github_shapes_inputs as inputs import github_shapes_set as roster +import github_shapes_aac as aac import oracle_pin from oracle_audio import INPUT_ARGS as AUDIO_DECODE_INPUT @@ -32,7 +33,10 @@ COMPATIBLE = 'compatible_avc1' MAPPING = re.compile(r'^\s*Stream #\d+:\d+(?:\[[^]]*\])?(?: \([^)]*\))? -> #(\d+):(\d+) \((.*)\)\s*$', re.M) METHOD = ('docs/design/github-shapes-200.md', 'scripts/oracle-ffmpeg.env', - '.github/workflows/github-shapes.yml', 'Cargo.lock') + '.github/workflows/github-shapes.yml', 'Cargo.lock', + 'docs/design/aac-pcm-eligibility.md', 'docs/design/aac-pns-eligibility.md', + 'docs/compat/audio-decoder-eligibility.json', 'scripts/aac-pns-requirements.txt', + 'scripts/oracle-audio-args.txt') class EvidenceError(ValueError): @@ -291,6 +295,7 @@ def compare_outputs(command, base, programs, env, scratch): mapping = jobs(ref) if not mapping: return 'no_oracle' + aac_decoded = aac.decodes_aac(ref) outcome, unavailable = 'counted', 'no_oracle' timeline_checks = [] controls = {'command.stdout', 'command.stderr', 'stdout', 'fallback.jsonl', 'starts.json'} @@ -320,8 +325,21 @@ def compare_outputs(command, base, programs, env, scratch): if source is None or any(s.get('codec_type') == 'video' and not s.get('pix_fmt', '').startswith(('rgb', 'bgr', 'gbr')) for s in source['streams']): return unavailable - verdict, _ = decoded_output.compare(actual, expected, programs['$FFMPEG'], programs['$FFPROBE'], + verdict, reason = decoded_output.compare(actual, expected, programs['$FFMPEG'], programs['$FFPROBE'], env, scratch, inputs=raw, registered_codecs=EXACT_CODECS) + aac_pcm = aac_decoded and len(command.sources) == len(command.outputs) == len(left) == 1 and ( + mapping.get((0, 0)) in aac.PCM and len(mapping) == 1) + if aac_pcm and (verdict == 'verified' or (verdict, reason) == ('failed', 'decoded samples')): + result = aac.compare(command, base, source, mapping, (verdict, reason), programs, scratch) + if result == aac.COMPATIBLE: + checked = decoded_output.compare(actual, expected, programs['$FFMPEG'], programs['$FFPROBE'], + env, scratch, inputs=raw, registered_codecs=EXACT_CODECS, exact_frame_timeline=True) + if checked not in (('failed', 'decoded samples'), ('verified', 'exact decoded output')): + return 'different' + return result + if aac_decoded and verdict == 'verified' and not aac.exact_without_pns( + command, base, source, programs, scratch): + return 'different' if verdict != 'verified': return unavailable if verdict == 'unverified' else 'different' if copied: @@ -421,7 +439,7 @@ def summarize(rows, outcomes): counts, weights, tools = collections.Counter(), collections.Counter(), {} for row, outcome in zip(rows, outcomes): code = outcome['outcome'] - if code not in (*MISSES, COMPATIBLE, 'counted'): + if code not in (*MISSES, COMPATIBLE, aac.COMPATIBLE, 'counted'): raise EvidenceError('result_class') counts[code] += 1 weights[code] += row['repository_count'] @@ -431,10 +449,11 @@ def summarize(rows, outcomes): return {'set': 'github-shapes-200', 'set_sha256': roster.DIGEST, 'shapes': 200, 'counted': counts['counted'], 'repository_pairs': 1470, 'counted_pairs': weights['counted'], COMPATIBLE: counts[COMPATIBLE], COMPATIBLE + '_pairs': weights[COMPATIBLE], + aac.COMPATIBLE: counts[aac.COMPATIBLE], aac.COMPATIBLE + '_pairs': weights[aac.COMPATIBLE], 'top_200_pair_share': weights['counted'] / 1470, 'whole_sample_presence_lower_bound': weights['counted'] / 3338, - 'outcomes': {key: counts[key] for key in (*MISSES, COMPATIBLE, 'counted')}, - 'outcome_pairs': {key: weights[key] for key in (*MISSES, COMPATIBLE, 'counted')}, 'tools': tools, 'bar': None} + 'outcomes': {key: counts[key] for key in (*MISSES, COMPATIBLE, aac.COMPATIBLE, 'counted')}, + 'outcome_pairs': {key: weights[key] for key in (*MISSES, COMPATIBLE, aac.COMPATIBLE, 'counted')}, 'tools': tools, 'bar': None} def inventory(bundle): @@ -448,7 +467,13 @@ def inventory(bundle): def verify_programs(opts): + import numpy as np + required = (ROOT / 'scripts/aac-pns-requirements.txt').read_text().splitlines()[0].split()[0].removeprefix('numpy==') + if np.__version__ != required: + raise EvidenceError('pns_dependency_version') identities = breadth.oracle_programs(opts) + identities['pns_diagnostic'] = {'sha256': digest(Path(opts.pns_diagnostic)), 'command': aac.DIAGNOSTIC_BUILD} + identities['numpy'] = {'version': np.__version__} pin = oracle_pin.load() version = subprocess.run([opts.x264, '--version'], capture_output=True, text=True, timeout=10, check=True).stdout if pin['X264_COMMIT'][:7] not in version: @@ -489,7 +514,8 @@ def replay(opts): if [record['rank'] for record in records] != [row['rank'] for row in rows]: raise EvidenceError('receipt_roster') programs = {'$COREAV': str(Path(opts.coreav).resolve()), '$FFMPEG': str(Path(opts.ffmpeg).resolve()), - '$FFPROBE': str(Path(opts.ffprobe).resolve()), '$X264': str(Path(opts.x264).resolve())} + '$FFPROBE': str(Path(opts.ffprobe).resolve()), '$X264': str(Path(opts.x264).resolve()), + '$AAC_DIAGNOSTIC': str(Path(opts.pns_diagnostic).resolve())} opts.scratch.mkdir(parents=True, exist_ok=False) outcomes = [{'rank': row['rank'], 'outcome': judge(row, record, bundle, programs, environment(), opts.scratch)} for row, record in zip(rows, records)] @@ -513,7 +539,8 @@ def run(opts): 'lock_sha256': digest(ROOT / 'Cargo.lock')}: raise EvidenceError('build_record') programs = {'$COREAV': str(Path(opts.coreav).resolve()), '$FFMPEG': str(Path(opts.ffmpeg).resolve()), - '$FFPROBE': str(Path(opts.ffprobe).resolve()), '$X264': str(Path(opts.x264).resolve())} + '$FFPROBE': str(Path(opts.ffprobe).resolve()), '$X264': str(Path(opts.x264).resolve()), + '$AAC_DIAGNOSTIC': str(Path(opts.pns_diagnostic).resolve())} bundle = opts.bundle.resolve() bundle.mkdir(parents=True, exist_ok=False) env = environment() @@ -544,7 +571,7 @@ def run(opts): def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('mode', choices=('run', 'replay')) - for name in ('coreav', 'ffmpeg', 'ffprobe', 'x264'): + for name in ('coreav', 'ffmpeg', 'ffprobe', 'x264', 'pns-diagnostic'): parser.add_argument('--' + name, required=True) for name in ('bundle', 'scratch', 'summary'): parser.add_argument('--' + name, type=Path, required=True) diff --git a/tools/compat-corpus/github_shapes_aac.py b/tools/compat-corpus/github_shapes_aac.py new file mode 100644 index 000000000..11e263b25 --- /dev/null +++ b/tools/compat-corpus/github_shapes_aac.py @@ -0,0 +1,138 @@ +"""Bind measured AAC PCM outputs to the registered decoder qualification.""" +import json +from pathlib import Path +import re +import subprocess +import tempfile + +import aac_pcm_eligibility as qualification + + +COMPATIBLE = 'compatible_aac' +DIAGNOSTIC_BUILD = ['cargo', 'build', '--locked', '--release', '-p', 'coreav-aac', + '--features', 'diagnostics', '--example', 'pns_diagnostic'] +PCM = {'pcm_f32le', 'pcm_s16le'} +MAPPING = re.compile(r'^\s*Stream #0:\d+(?:\[[^]]*\])?(?: \([^)]*\))? -> #0:0 ' + r'\(aac \(native\) -> (pcm_f32le|pcm_s16le) \(native\)\)\s*$', re.M) + + +def decodes_aac(reference): + return bool(re.search(r'^\s*Stream #.* -> #\d+:\d+ \(aac \(', + (reference / 'command.stderr').read_text(), re.M)) + + +def noise_count(metadata): + frames = metadata['frames'] + if metadata['schema'] != 'coreav.aac-pns-diagnostic/1' or type(metadata['packets']) is not int or not ( + 0 < metadata['packets'] == len(frames) <= 16384) or metadata['channels'] not in (1, 2): + raise ValueError('incomplete diagnostic scan') + count = 0 + for frame in frames: + if len(frame['channels']) != metadata['channels']: + raise ValueError('incomplete diagnostic channels') + for channel in frame['channels']: + if not isinstance(channel['noise'], list): + raise ValueError('incomplete diagnostic noise') + count += len(channel['noise']) + return count + + +def exact_without_pns(command, base, source, programs, scratch): + audio = [s for s in (source or {}).get('streams', []) if s.get('codec_type') == 'audio'] + if len(command.sources) != 1 or len(audio) != 1 or audio[0].get('codec_name') != 'aac': + return False + try: + with tempfile.TemporaryDirectory(prefix='shape-aac-scan-', dir=scratch) as temporary: + directory = Path(temporary) + scan = qualification.run([programs['$AAC_DIAGNOSTIC'], + base / 'reference' / command.sources[0][0], directory]) + if scan.returncode: + return False + metadata = json.loads((directory / 'metadata.json').read_text()) + return noise_count(metadata) == 0 + except (ValueError, OSError, KeyError, TypeError, subprocess.SubprocessError): + return False + + +def output_format(command, source): + if command.tool != 'ffmpeg' or len(command.sources) != 1 or len(command.outputs) != 1 or command.stdin: + return None + audio = [s for s in (source or {}).get('streams', []) if s.get('codec_type') == 'audio'] + if len(audio) != 1 or audio[0].get('codec_name') != 'aac': + return None + prefix = list(command.args[:command.args.index('-i')]) + if prefix not in ([], ['-threads', '1']): + return None + options = command.output_options[0] + values = {} + at = 0 + while at < len(options): + key = options[at] + if key in values or key not in ('-vn', '-c:a', '-f', '-ar', '-ac'): + return None + at += 1 + if key == '-vn': + values[key] = True + else: + if at == len(options): + return None + values[key] = options[at] + at += 1 + for key, field in (('-ar', 'sample_rate'), ('-ac', 'channels')): + if key in values and int(values[key]) != int(audio[0][field]): + return None + form = values.get('-f', 'wav' if command.outputs[0].endswith('.wav') else None) + codec = {'f32le': 'pcm_f32le', 's16le': 'pcm_s16le', 'wav': 'pcm_s16le'}.get(form) + if codec is None or values.get('-c:a', codec) != codec: + return None + return form + + +def compare(command, base, source, mapping, verdict, programs, scratch): + reference = base / 'reference' + candidate = base / 'candidate' + audio = [s for s in (source or {}).get('streams', []) if s.get('codec_type') == 'audio'] + if len(audio) != 1 or audio[0].get('codec_name') != 'aac': + return 'different' + form = output_format(command, source) + exact = verdict == ('verified', 'exact decoded output') + if not exact and verdict != ('failed', 'decoded samples'): + return 'different' + try: + with tempfile.TemporaryDirectory(prefix='shape-aac-', dir=scratch) as temporary: + work = Path(temporary) + diagnostic = work / 'diagnostic' + diagnostic.mkdir() + input_path = reference / command.sources[0][0] + scan = qualification.run([programs['$AAC_DIAGNOSTIC'], input_path, diagnostic]) + if scan.returncode: + return 'different' + metadata = json.loads((diagnostic / 'metadata.json').read_text()) + noise = noise_count(metadata) + if exact and not noise: + return 'counted' + if form is None or metadata['unsafe']: + return 'different' + codec = 'pcm_f32le' if form == 'f32le' else 'pcm_s16le' + log = (reference / 'command.stderr').read_text() + if mapping != {(0, 0): codec} or MAPPING.findall(log) != [codec]: + return 'different' + if command.outputs[0] != '-' and any((folder / 'stdout').read_bytes() for folder in (reference, candidate)): + return 'different' + oracle = Path(programs['$FFMPEG']).parent + pns = qualification.pns_check(oracle, input_path, diagnostic, metadata) if noise else None + capture = {} + row = qualification.check_output(programs['$COREAV'], oracle, input_path, work, metadata, pns, capture) + if row['status'] != 'pass' or set(capture) != {'f32', 's16'}: + return 'different' + name = 'stdout' if command.outputs[0] == '-' else command.outputs[0] + actual, expected = [(folder / name).read_bytes() for folder in (candidate, reference)] + fmt = 'f32' if form == 'f32le' else 's16' + ours, theirs = capture[fmt] + if form == 'wav': + qualification.compare_wav(actual, expected, ours, theirs, row['channels'], row['rate']) + elif (actual, expected) != (ours, theirs): + return 'different' + return COMPATIBLE + except (ValueError, OSError, KeyError, TypeError, IndexError, subprocess.SubprocessError): + return 'different' diff --git a/tools/compat-corpus/tests/test_github_shapes.py b/tools/compat-corpus/tests/test_github_shapes.py index ba4f5cc67..181b5f945 100644 --- a/tools/compat-corpus/tests/test_github_shapes.py +++ b/tools/compat-corpus/tests/test_github_shapes.py @@ -190,7 +190,7 @@ def test_reference_failure_keeps_the_row(self): def test_replay_checks_inventory_and_recomputes_instead_of_trusting_labels(self): opts = types.SimpleNamespace(bundle=self.bundle, scratch=self.root / 'scratch', summary=self.root / 'summary.json', **{ - k: str(self.fake) for k in ('coreav', 'ffmpeg', 'ffprobe', 'x264')}) + k: str(self.fake) for k in ('coreav', 'ffmpeg', 'ffprobe', 'x264', 'pns_diagnostic')}) identity = {'coreav': {'sha256': run.digest(self.fake)}} build = {'commit': 'a' * 40, 'command': run.BUILD, 'coreav_sha256': run.digest(self.fake), 'lock_sha256': run.digest(run.ROOT / 'Cargo.lock')} @@ -219,6 +219,14 @@ def test_replay_checks_inventory_and_recomputes_instead_of_trusting_labels(self) self.assertEqual(run.replay(opts), 0) summary = json.loads(opts.summary.read_text()) self.assertEqual((summary['counted'], summary[run.COMPATIBLE]), (1, 0)) + self.record['outcome'] = run.aac.COMPATIBLE + run.write(self.bundle / 'records.json', records) + manifest['inventory'] = run.inventory(self.bundle) + run.write(self.bundle / 'manifest.json', manifest) + opts.scratch = self.root / 'replay-aac-label' + self.assertEqual(run.replay(opts), 0) + summary = json.loads(opts.summary.read_text()) + self.assertEqual((summary['counted'], summary[run.aac.COMPATIBLE]), (1, 0)) self.fake.write_text('changed executable') with mock.patch.object(run, 'verify_programs', return_value={'changed': {}}), \ self.assertRaisesRegex(ValueError, 'program_identity'): diff --git a/tools/compat-corpus/tests/test_github_shapes_aac.py b/tools/compat-corpus/tests/test_github_shapes_aac.py new file mode 100644 index 000000000..1950ad91c --- /dev/null +++ b/tools/compat-corpus/tests/test_github_shapes_aac.py @@ -0,0 +1,324 @@ +"""Independent generated controls, never the judged fixture generator.""" +import json +import os +from pathlib import Path +import shutil +import struct +import subprocess +import sys +import tempfile +import types +import unittest +from unittest import mock + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +import github_shapes as run +import github_shapes_aac as aac +import github_shapes_inputs as inputs +import github_shapes_set as roster +import aac_pcm_eligibility as qualification +import test_oracle + + +SOURCE = {'streams': [{'codec_type': 'audio', 'codec_name': 'aac', 'sample_rate': '48000', 'channels': 1}]} + + +def command(form='f32le', options=(), pipe=False): + name = '-' if pipe else 'control.' + form + opts = ('-f', form, *options) + return inputs.Command('ffmpeg', ('-i', 'in0/control.m4a', *opts, name), + (('in0/control.m4a', 'control.m4a'),), (name,), (opts,)) + + +def wave(pcm, info=b''): + body = b'WAVEfmt ' + struct.pack(' #0:0 (aac (native) -> pcm_f32le (native))\n') + self.capture = {'f32': (struct.pack(' #0:0 (aac (native) -> {codec} (native))\n') + return name + + def test_each_output_is_bound_to_its_own_qualified_decoder_bytes(self): + for form in ('f32le', 's16le', 'wav'): + for pipe in (False, True): + value = command(form, pipe=pipe) + name = self.outputs(value) + self.assertEqual(self.check(value), aac.COMPATIBLE) + for arm in ('candidate', 'reference'): + path = self.root / arm / name + original = path.read_bytes() + for mutated in (original[:-1], original + b'\0', original[:-1] + b'\x7f'): + path.write_bytes(mutated) + self.assertEqual(self.check(value), 'different') + path.write_bytes(original) + if pipe: + for arm in ('candidate', 'reference'): + (self.root / arm / 'stdout').write_bytes(b'') + + def test_wav_metadata_and_file_stdout_are_not_ignored(self): + value = command('wav') + name = self.outputs(value) + path = self.root / 'candidate' / name + path.write_bytes(wave(self.capture['s16'][0], b'LIST\x04\0\0\0INFO')) + self.assertEqual(self.check(value), 'different') + self.outputs(value) + (self.root / 'candidate/stdout').write_bytes(b'unexpected') + self.assertEqual(self.check(value), 'different') + + def test_exact_no_pns_stays_exact_but_equal_pns_never_does(self): + value = command() + self.outputs(value) + exact = ('verified', 'exact decoded output') + self.assertEqual(self.check(value, exact), 'counted') + self.metadata['frames'][0]['channels'][0]['noise'] = [{'coded': 'noise'}] + self.assertEqual(self.check(value, exact), aac.COMPATIBLE) + self.metadata['unsafe'] = ['tns_pns'] + self.assertEqual(self.check(value, exact), 'different') + + def test_failed_or_unavailable_qualification_cannot_create_exact_or_no_oracle(self): + self.outputs(command()) + for error in (ValueError('energy'), OSError('missing'), subprocess.TimeoutExpired('diagnostic', 1)): + with mock.patch.object(qualification, 'check_output', side_effect=error), \ + mock.patch.object(qualification, 'run', side_effect=self.scan): + self.assertEqual(aac.compare(command(), self.root, SOURCE, {(0, 0): 'pcm_f32le'}, + ('failed', 'decoded samples'), {'$AAC_DIAGNOSTIC': 'd', '$FFMPEG': '/o/ffmpeg', '$COREAV': 'c'}, + self.root), 'different') + for reason in ('stream fields or timing', 'decoded pixels', 'invalid audio timing'): + self.assertEqual(self.check(verdict=('failed', reason)), 'different') + + def test_wrong_decoder_and_extra_mapping_do_not_qualify(self): + self.outputs(command()) + (self.root / 'reference/command.stderr').write_text('Stream #0:0 -> #0:0 (mp3 (native) -> pcm_f32le (native))\n') + self.assertEqual(self.check(), 'different') + self.assertFalse(aac.decodes_aac(self.root / 'reference')) + + def test_compatibility_cannot_waive_output_fields_or_frame_timing(self): + value = command() + self.outputs(value) + programs = {'$FFMPEG': 'ffmpeg', '$FFPROBE': 'ffprobe'} + for failure in ('invalid audio timing', 'stream fields or timing', 'decode or incomplete samples'): + with mock.patch.object(run.decoded_output, 'probe', return_value=SOURCE), \ + mock.patch.object(run.decoded_output, 'compare', side_effect=[('failed', 'decoded samples'), + ('failed', failure)]), \ + mock.patch.object(aac, 'compare', return_value=aac.COMPATIBLE): + self.assertEqual(run.compare_outputs(value, self.root, programs, {}, self.root), 'different') + + def test_no_pns_scan_guard_also_covers_encoded_outputs_and_fails_closed(self): + programs = {'$AAC_DIAGNOSTIC': 'diagnostic'} + with mock.patch.object(qualification, 'run', side_effect=self.scan): + self.assertTrue(aac.exact_without_pns(command(), self.root, SOURCE, programs, self.root)) + self.metadata['frames'][0]['channels'][0]['noise'] = [{}] + self.assertFalse(aac.exact_without_pns(command(), self.root, SOURCE, programs, self.root)) + self.metadata['frames'] = [] + self.assertFalse(aac.exact_without_pns(command(), self.root, SOURCE, programs, self.root)) + for mapping in ('aac (native) -> aac (native)', 'aac (native) -> mp3 (libmp3lame)', + 'aac (native) -> opus (libopus)'): + (self.root / 'reference/command.stderr').write_text(f'Stream #0:0 -> #0:0 ({mapping})\n') + self.assertTrue(aac.decodes_aac(self.root / 'reference')) + (self.root / 'reference/command.stderr').write_text('Stream #0:0 -> #0:0 (copy)\n') + self.assertFalse(aac.decodes_aac(self.root / 'reference')) + + def test_aac_scan_does_not_bypass_a_sibling_copied_stream_packet_gate(self): + value = command() + self.outputs(value) + (self.root / 'reference/command.stderr').write_text( + 'Stream #0:0 -> #0:0 (copy)\nStream #0:1 -> #0:1 (aac (native) -> pcm_f32le (native))\n') + programs = {'$FFMPEG': 'ffmpeg', '$FFPROBE': 'ffprobe'} + with mock.patch.object(run.decoded_output, 'probe', return_value=SOURCE), \ + mock.patch.object(run.decoded_output, 'compare', return_value=('verified', 'exact decoded output')), \ + mock.patch.object(aac, 'exact_without_pns', return_value=True), \ + mock.patch.object(aac, 'compare') as compatible, \ + mock.patch.object(run, 'packet_rows', side_effect=[[(0, 'changed')], [(0, 'original')]]), \ + mock.patch.object(run.decoded_output, 'avc1_carriage', return_value=False): + self.assertEqual(run.compare_outputs(value, self.root, programs, {}, self.root), 'different') + compatible.assert_not_called() + + def test_encoded_pns_cannot_receive_exact_credit(self): + value = command() + self.outputs(value) + (self.root / 'reference/command.stderr').write_text('Stream #0:0 -> #0:0 (aac (native) -> opus (libopus))\n') + with mock.patch.object(run.decoded_output, 'probe', return_value=SOURCE), \ + mock.patch.object(run.decoded_output, 'compare', return_value=('verified', 'exact decoded output')), \ + mock.patch.object(aac, 'exact_without_pns', return_value=False), \ + mock.patch.object(aac, 'compare') as compatible: + self.assertEqual(run.compare_outputs(value, self.root, {'$FFMPEG': 'f', '$FFPROBE': 'p'}, {}, self.root), 'different') + compatible.assert_not_called() + + +@unittest.skipUnless(os.environ.get('COREAV_SHAPES_COREAV'), 'AAC command control binaries not configured') +class PinnedAacControls(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.oracle = test_oracle.require(ffmpeg=str(Path(os.environ['COREAV_SHAPES_ORACLE']) / 'ffmpeg')) + cls.coreav = Path(os.environ['COREAV_SHAPES_COREAV']).resolve() + cls.diagnostic = Path(os.environ['COREAV_SHAPES_AAC_DIAGNOSTIC']).resolve() + cls.tmp = tempfile.TemporaryDirectory() + cls.addClassCleanup(cls.tmp.cleanup) + cls.root = Path(cls.tmp.name) + cls.tone = cls.root / 'tone.m4a' + cls.call([cls.oracle.ffmpeg, '-v', 'error', '-f', 'lavfi', '-i', + 'sine=frequency=673:sample_rate=44100:duration=0.0928798185941043', '-c:a', 'aac', '-aac_pns', '0', cls.tone]) + packets = cls.root / 'packets' + packets.mkdir() + cls.call([cls.diagnostic, '--fixtures', packets]) + for source in packets.glob('*.aac'): + cls.call([cls.oracle.ffmpeg, '-v', 'error', '-f', 'aac', '-i', source, + '-c:a', 'copy', cls.root / (source.stem + '.m4a')]) + + @staticmethod + def call(args): + result = subprocess.run(list(map(str, args)), capture_output=True, timeout=120) + if result.returncode: + raise AssertionError(result.stderr.decode(errors='replace')[-1000:]) + return result + + def capture(self, source, form, pipe=False): + temporary = tempfile.TemporaryDirectory(dir=self.root) + self.addCleanup(temporary.cleanup) + base = Path(temporary.name) + value = command(form, pipe=pipe) + for arm in ('reference', 'candidate'): + folder = base / arm + (folder / 'in0').mkdir(parents=True) + shutil.copyfile(source, folder / 'in0/control.m4a') + argv = run.expected_invocation(value, arm == 'candidate') + programs = {'$COREAV': str(self.coreav), '$FFMPEG': str(self.oracle.ffmpeg), + '$FFPROBE': str(self.oracle.ffprobe), '$AAC_DIAGNOSTIC': str(self.diagnostic)} + env = run.environment() + env.update(COREAV_FALLBACK='/nonexistent/no-fallback', COREAV_STRICT_SAFE='1') + receipt = run.execute([programs.get(arg, arg) for arg in argv], folder, env, 120, 'command') + if receipt['code']: + raise AssertionError((folder / 'command.stderr').read_text()[-1000:]) + (folder / 'stdout').write_bytes((folder / 'command.stdout').read_bytes()) + return value, base, programs + + def test_real_no_pns_and_pns_raw_and_wav_file_and_stdout(self): + for source in (self.tone, self.root / 'pns-independent.m4a', self.root / 'pns-independent-stereo.m4a'): + for form in ('f32le', 's16le', 'wav'): + for pipe in (False, True): + with self.subTest(source=source.name, form=form, pipe=pipe): + value, base, programs = self.capture(source, form, pipe) + name = 'stdout' if pipe else value.outputs[0] + path = base / 'candidate' / name + expected = (base / 'reference' / name).read_bytes() + actual = path.read_bytes() + if form == 'wav': + channels = 2 if 'stereo' in source.name else 1 + actual = qualification.wav(actual, 's16', channels, 44100) + expected = qualification.wav(expected, 's16', channels, 44100) + outcome = 'counted' if source == self.tone and actual == expected else aac.COMPATIBLE + self.assertEqual(run.compare_outputs(value, base, programs, run.environment(), base), outcome) + path.write_bytes(path.read_bytes()[:-1]) + self.assertEqual(run.compare_outputs(value, base, programs, run.environment(), base), 'different') + + def test_generated_excluded_pns_tools_refuse_and_cannot_qualify(self): + for name in ('pns-tns', 'pns-ms', 'pns-illegal-window'): + source = self.root / (name + '.m4a') + result = subprocess.run(list(map(str, qualification.native(self.coreav, source, + ['-f', 's16le', '-']))), capture_output=True, timeout=120) + self.assertEqual((result.returncode, result.stdout), (4, b'')) + with tempfile.TemporaryDirectory(dir=self.root) as temporary: + row = qualification.check(self.coreav, self.oracle.ffmpeg.parent, source, + Path(temporary), self.diagnostic, {}) + self.assertEqual(row['status'], 'ineligible') + + def test_capture_exports_both_complete_qualified_arm_payloads(self): + source = self.root / 'pns-independent.m4a' + with tempfile.TemporaryDirectory(dir=self.root) as temporary: + capture = {} + row = qualification.check(self.coreav, self.oracle.ffmpeg.parent, source, + Path(temporary), self.diagnostic, capture) + self.assertEqual(row['status'], 'pass') + self.assertIn('pns', row) + for fmt, codec in (('f32', 'pcm_f32le'), ('s16', 'pcm_s16le')): + actual = self.call(qualification.native(self.coreav, source, ['-c:a', codec, '-f', fmt + 'le', '-'])) + reference = self.call([self.oracle.ffmpeg, '-v', 'error', *qualification.AUDIO_DECODE_INPUT, + '-i', source, '-c:a', codec, '-f', fmt + 'le', '-']) + self.assertEqual(capture[fmt], (actual.stdout, reference.stdout)) + + +if __name__ == '__main__': + unittest.main()