Skip to content

olmocr-bench: internal blind holdout with an aggregate-only gate (#959) - #6

Closed
jasperdew wants to merge 3 commits into
feat/odl-bench-job-960-v2from
feat/olmocr-blind-holdout
Closed

jasperdew wants to merge 3 commits into
feat/odl-bench-job-960-v2from
feat/olmocr-blind-holdout

Conversation

@jasperdew

Copy link
Copy Markdown
Member

Stacked on #3. Adds python -m bench.olmocr: pinned olmOCR-bench dataset revision and scorer commit, recipes as in odl-bench, gate at 0.005 on the overall score and each category against a baseline file.

  • Blind: aggregates only; no per-document score is computed, printed or written. Tests fail if an output file names a document.
  • Internal evaluation only: data is downloaded at run time to a private root outside $HOME, never committed. No scores here. Licence findings and the rule are in olmocr-bench/README.md.
  • Tests (tests/test_olmocr_bench.py, run by make test in CI): threshold logic, manifest and pin, blindness, loud skip (exit 3), and a mutation check that the gate bites. Two existing tests in test_workflows.py fail on this base independent of this change.

Data: olmOCR-bench, Allen Institute for AI, ODC-BY-1.0.

…ression gate

Recipes (versioned, printed into every result), manifest hashes of the 200
PDFs and ground-truth files, a scoreless tune/holdout split file, per-metric
scores per half, a baseline gate (tolerance 0.005 and per-document canaries),
a loud skip when the models are absent, and the tests for all of it, run in CI.
The full class labels (#969) stay private and are passed with --labels.
Refs pdfluent-internal#960 and #969.

Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…P-Bench fingerprints (#968)

Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
…nly gate

Runs the pinned olmOCR-bench dataset and scorer for one PDFluent recipe and
gates the overall score and each category at 0.005 against a baseline file.
Per-document scores are never computed, printed or written. Data is
downloaded at run time to a private root outside the home directory and is
never committed. Licence findings and the internal-only rule are in the
README; no scores are published.

Signed-off-by: jasperdew <10383561+jasperdew@users.noreply.github.com>
@jasperdew

Copy link
Copy Markdown
Member Author

Superseded by #9, which carries the same runner and gate with an aggregate-only manifest (no document names, no per-document hashes). Closing; the old branch should be removed by the owner since it still holds the per-document manifest in its history.

@jasperdew jasperdew closed this Nov 18, 2026
@jasperdew
jasperdew deleted the feat/olmocr-blind-holdout branch November 18, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant