Repository navigation
Conversation
…ression gate Recipes (versioned, printed into every result), manifest hashes of the 200 PDFs and ground-truth files, a scoreless tune/holdout split file, per-metric scores per half, a baseline gate (tolerance 0.005 and per-document canaries), a loud skip when the models are absent, and the tests for all of it, run in CI. The full class labels (#969) stay private and are passed with --labels. Refs pdfluent-internal#960 and #969. Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…P-Bench fingerprints (#968) Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…nly gate Runs the pinned olmOCR-bench dataset and scorer for one PDFluent recipe and gates the overall score and each category at 0.005 against a baseline file. Per-document scores are never computed, printed or written. Data is downloaded at run time to a private root outside the home directory and is never committed. Licence findings and the internal-only rule are in the README; no scores are published. Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
jasperdew
force-pushed
the
feat/odl-bench-job-960-v2
branch
from
November 18, 2026 14:16
228e70d to
1532dc5
Compare
Member
Author
|
Superseded by #9, which carries the same runner and gate with an aggregate-only manifest (no document names, no per-document hashes). Closing; the old branch should be removed by the owner since it still holds the per-document manifest in its history. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3. Adds
python -m bench.olmocr: pinned olmOCR-bench dataset revision and scorer commit, recipes as in odl-bench, gate at 0.005 on the overall score and each category against a baseline file.make testin CI): threshold logic, manifest and pin, blindness, loud skip (exit 3), and a mutation check that the gate bites. Two existing tests in test_workflows.py fail on this base independent of this change.Data: olmOCR-bench, Allen Institute for AI, ODC-BY-1.0.