feat!: add PE-A evidence retrieval and agent evaluation - #153
Open
tulayha wants to merge 57 commits into
Open
Conversation
This reverts commit daabb930ad6aa9d63d0b40a3a9222f87e9ae5064.
Internal-only benchmark tooling and documentation; no product runtime behavior changes.
Replace the product FineLAP selector with pinned PE-A frame indexing, bounded overlapping inference sections, inner-product ranking, and fixed evidence windows. Keep FineLAP dependencies and loading isolated to the reproducible benchmark control. BREAKING CHANGE: index schema 8 requires existing repositories to be rebuilt.
Read VidXP's canonical index schema during setup and place derived Promptfoo index data in a versioned directory so upgrades rebuild without deleting the prior index.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None.
Summary
Diff-size note: 183,770 of the 201,021 added lines (91.4%) are sanitized, retained Promptfoo JSON runs. The remaining additions are 12,064 implementation/harness lines, 2,656 documentation lines, 2,120 test lines, and 411 configuration/other lines; the JSON occupies 8.2 MiB in a checkout and approximately 1.11 MiB in Git-compressed storage.
Validation
./benchmarks/codex-mcp/run pilot— 81/81 isolated agent cases completed and were scorable. VidXP Success@3 was 15/27 versus 18/27 for direct-local; VidXP used 20.6% fewer tokens and was 18.3% faster. The quality gate remains explicitly failed../benchmarks/codex-mcp/run slm— 54/54 local-SLM cases completed and were scorable. Router/planner Success@3 was 15/27 and 19/27, with one local model request per case and no external-provider cost../benchmarks/codex-mcp/run slm-smoke— router and planner each returned three bounded candidates and passed Success@1/Success@3 on the development task.mac-m2-01— indexed 1,896 unique frames from a 75.81-second video; the two checked sound queries ranked their target 10-second windows first.python utils/verify_runtime.py minimal --vidxp-executable <venv>/bin/vidxpcompleted successfully; bothvidxp --helpandvidxp benchmark --helploaded without optional scientific dependencies.uv run --no-sync pytest -q tests/test_packaging.py— 20 passed.Detailed configurations, machine profile, limits, metrics, and retained artifacts are linked from
docs/benchmarking/metric_database.mdanddocs/benchmarking/results.md.BREAKING CHANGE: Existing environmental-sound indexes use the prior representation and must be rebuilt for PE-A-Frame search.