Skip to content

feat!: add PE-A evidence retrieval and agent evaluation - #153

Open
tulayha wants to merge 57 commits into
mainfrom
codex/fix-macos-benchmark-setup
Open

feat!: add PE-A evidence retrieval and agent evaluation#153
tulayha wants to merge 57 commits into
mainfrom
codex/fix-macos-benchmark-setup

Conversation

@tulayha

@tulayha tulayha commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Related issue

None.

Summary

  • Replace the product sound-retrieval path with PE-A-Frame Small frame embeddings, bounded overlapping inference sections, and distinct 10-second evidence windows. Existing sound indexes must be rebuilt because the embedding schema and distance metric changed.
  • Add shared local-answer routing through the CLI, application layer, managed Ollama runtime, and Desktop, including separate one-request router and planner policies.
  • Make the Promptfoo agent ablation reproducible across VidXP, direct-local, and clean-user conditions; preserve auditable reports, call/token/latency accounting, machine provenance, and lightweight result artifacts.
  • Add modality-specific gates and research-adoption documentation covering the measured model choices, product defaults, limitations, and paper-facing claims.

Diff-size note: 183,770 of the 201,021 added lines (91.4%) are sanitized, retained Promptfoo JSON runs. The remaining additions are 12,064 implementation/harness lines, 2,656 documentation lines, 2,120 test lines, and 411 configuration/other lines; the JSON occupies 8.2 MiB in a checkout and approximately 1.11 MiB in Git-compressed storage.

Validation

  • ./benchmarks/codex-mcp/run pilot — 81/81 isolated agent cases completed and were scorable. VidXP Success@3 was 15/27 versus 18/27 for direct-local; VidXP used 20.6% fewer tokens and was 18.3% faster. The quality gate remains explicitly failed.
  • ./benchmarks/codex-mcp/run slm — 54/54 local-SLM cases completed and were scorable. Router/planner Success@3 was 15/27 and 19/27, with one local model request per case and no external-provider cost.
  • ./benchmarks/codex-mcp/run slm-smoke — router and planner each returned three bounded candidates and passed Success@1/Success@3 on the development task.
  • PE-A product-path smoke on mac-m2-01 — indexed 1,896 unique frames from a 75.81-second video; the two checked sound queries ranked their target 10-second windows first.
  • AEGBench frozen 50-recording/149-query component gate — PE-A exceeded FineLAP on frame AUROC, frame AP, top-point accuracy, mean IoU, and R-IoU@0.5.
  • Fresh Python 3.14 base-wheel environment with NumPy absent: python utils/verify_runtime.py minimal --vidxp-executable <venv>/bin/vidxp completed successfully; both vidxp --help and vidxp benchmark --help loaded without optional scientific dependencies.
  • uv run --no-sync pytest -q tests/test_packaging.py — 20 passed.

Detailed configurations, machine profile, limits, metrics, and retained artifacts are linked from docs/benchmarking/metric_database.md and docs/benchmarking/results.md.

BREAKING CHANGE: Existing environmental-sound indexes use the prior representation and must be rebuilt for PE-A-Frame search.

This reverts commit daabb930ad6aa9d63d0b40a3a9222f87e9ae5064.
Internal-only benchmark tooling and documentation; no product runtime behavior changes.
Replace the product FineLAP selector with pinned PE-A frame indexing, bounded overlapping inference sections, inner-product ranking, and fixed evidence windows. Keep FineLAP dependencies and loading isolated to the reproducible benchmark control.

BREAKING CHANGE: index schema 8 requires existing repositories to be rebuilt.
Read VidXP's canonical index schema during setup and place derived Promptfoo index data in a versioned directory so upgrades rebuild without deleting the prior index.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant