Repository navigation
Judged metrics: Mod-POPE, Mod-ROPE, Mod-CHAIR and class-wise THRONE F0.5 - #6
SalimAlkharsa wants to merge 48 commits into
Conversation
…NE F0.5 Scores cached model outputs against static judge verdicts in judgments/ (never calls a judge; missing verdicts raise). Reproduces the reference table in tests/data/final_tables.json exactly from the results tree. The judge ships as tools/judge (driver and the three skills). Homepage and README tables become a headline-only placeholder; docs/methodology.mdx documents each modification with a raw vs Mod appendix.
There was a problem hiding this comment.
Dont change docs in this PR we will cross that bridge I care just about docs changing as far as numeric reporitng goes
There was a problem hiding this comment.
does judgements need to be in repo? Does it better fit in a HF data repo later? How does it fit I dont want to bloat
There was a problem hiding this comment.
Agreed, it belongs on HF. These files are what the Mod metrics are scored from. Without them, reproducing the published numbers means re-running the Claude judge, which costs API calls and only agrees with itself about 97% of the time. Right now they're still in git under judgments/, about 1 MB including the new 680 KB committee.json and final_tables.json. Plan: upload judgments/ to an HF dataset, pin its revision and sha256s in judge/manifest.json, and delete judgments/ from the repo. Do that before #6 merges, or squash-merge #6; otherwise the files stay in main's history.
There was a problem hiding this comment.
i want this to be native not post mortem I dont like this code set up, obviously it can run post mortem. Im a bit concerned by design of code here relative to mirage and my design goals
There was a problem hiding this comment.
I think a judge should be its own class and addition to the modular design, Ill discuss w agent
There was a problem hiding this comment.
Rewritten, and metrics/judged.py is gone. The judge is its own package, judge/:
Judge(judge/judge.py) answers from a frozen, sha256-checkedVerdictStore(judge/store.py).- It can fall back to a live
ClaudeCLI(judge/claude.py), which is off by default.
Metrics take verdicts natively, e.g. ChairMetric.score(images, verdicts, refs, committee=...). run.py writes a judged block per cell, or missing_verdicts when an image hasn't been judged. The committee is frozen at the 9 judged models, so adding a model doesn't shift existing rows.
Current limit: Mod-POPE and Mod-CHAIR only cover the frozen 500 images per dataset, because there's no live POPE drop-list writer yet.
There was a problem hiding this comment.
Again not a fan of the fact teh judge is little bits of jerry rigged code
There was a problem hiding this comment.
Partly fixed. The judging itself now lives in judge/. What's left in rope.py is judged_answer(), about 40 lines of regex plus NLTK that turn a response into the answer key the stored verdicts were graded on. It has to reproduce that key exactly or the stored verdicts are orphaned, so it can't be simplified without re-judging ROPE.
There was a problem hiding this comment.
is this needed
There was a problem hiding this comment.
Removed from tests/. The published table is now judgments/final_tables.json, which repro.py --check compares against. It moves to HF with the rest of judgments/.
There was a problem hiding this comment.
do we realy need thsi
There was a problem hiding this comment.
Deleted. Its job is now python repro.py --root TREE --check: 612/612 values, max diff 0. It needs the cached predictions, so it's a manual check, not a unit test.
There was a problem hiding this comment.
dont think we need this
There was a problem hiding this comment.
dont need this if we will document
There was a problem hiding this comment.
Deleted, along with tools/; run_judge.py is now judge/claude.py. The one short file table left is judgments/README.md, which becomes the HF dataset card.
There was a problem hiding this comment.
Yeah keep these minimal
…sts and the judge README
rope_records uses rope.strict(pred, gt), the same expression as the deleted _rope_match_strict, and constructs ROPEMetric(num_objects=5) (class_list was unused, so the names lookup is dropped too).
throne_classwise and fbeta now live in metrics.throne and metrics.base_metric (same bytes); judged.py imports them, with MIN_WORDS and MIN_SUPPORT, and drops its copies.
POPEMetric.score, ROPEMetric.score, ChairMetric.score and ChairMetric.verified_present are mod_pope, mod_rope, mod_chair and _verified_present moved verbatim; QUESTION_RE is _POPE_RE and MissingVerdicts lives in base_metric. repro.py imports them.
…e metrics ChairMetric.for_dataset and reference_objects are mention_records' blocks moved; ChairMetric.score takes an optional committee pool (none in repro, so the pool is today's union); THRONE goes through ThroneMetric.score and completed_labels. _rows refuses a cache with two rows for one (prompt, image_hash) instead of keeping the last.
…ault) VerdictStore opens judgments/ (or $MIRAGE_JUDGMENTS_DIR), checks each file against judge/manifest.json, and keeps new verdicts in a separate live shard that can never overwrite a frozen key. Judge answers from the store; with a live ClaudeCLI it grades the misses first. POPE drop decisions and the committee exist only for covered items (committee.json, written by repro.py --export-committee). judgments/final_tables.json is the 612-value reference from daebc84.
build_table(root, store=None) reads the frozen store (default judgments/). --check rebuilds every value, compares exactly with final_tables.json (and committee.json when the store has one) and exits 1 on any difference; it first asserts Python 3.12+ and the NLTK data. --export-committee writes the frozen pool's per-item present and verified words. docs_tables is gone.
POPE: (image_id, object, annotated, said_yes), with the object read back from the question as repro.py does. ROPE: (item, box, judged answer key, strict). CHAIR: tagged mentions per image under the model's registry key (ChairMetric.images). The questions asked, their order and format_results are unchanged.
…THRONE The frozen verdicts open before any generation (--judgments, default $MIRAGE_JUDGMENTS_DIR, else judgments/). Each metric's score runs on its run() records; CHAIR pools coverage with the committee, and VG THRONE completes labels with the committee's and the model's verified words. Items without verdicts are listed under missing_verdicts instead. The rest of the result JSON, cache_stats included, is unchanged.
… checked when read
The manifest lists final_tables.json under "reference". VerdictStore.open
checks the files it loads ("files"), so a judgments dir with only the
verdict files serves run.py; reference_table() checks the table's sha256,
and repro --check reads it before the rebuild. open() takes live=True for
the default <directory>/live shard.
The live ROPE requests had no raw reply and no box-annotated images, so ROPE misses now raise MissingVerdicts like POPE until that pipeline exists. Judge.load(live=True) opens the store once with its live shard, and judgments/live/ is ignored.
main() and ClaudeCLI both use claude_binary(): the argument, else $MIRAGE_CLAUDE_BIN, else claude on PATH, and raise when none is found.
…t-key rule POPE/ROPE/CHAIR/THRONE cells check COCO CHAIR's reference captions and the NLTK data the tagging needs (punkt_tab, the POS tagger, WordNet, VG stopwords) before the model runs, so a missing input fails the cell up front instead of losing it after generation. ChairMetric.run and run.py name a model's verdicts with metrics.base_metric.verdict_key.
caption_images and run.py's Visual Genome THRONE labels both take the first n annotated samples; run.py no longer rebuilds them from image ids.
main(argv, model=..., judge=..., evaluator=...) threads them through _run_dataset_metric (and the model through NOPE) in place of the ones the flags build, so a caller can drive the CLI without weights or verdict files.
…THRONE scores run's records Until committee.json ships, POPE and the CHAIR committee raise MissingVerdicts for every image; demo.py reported that as a crash after generation. Each judged score now prints the missing items and the demo continues, as run.py does. THRONE scores what ThroneMetric.run returns.
…ines between functions
With a live judge and no image_paths, mention_requests failed with a TypeError on None once a mention needed grading; it now raises a ValueError that names the missing argument.
Exported with python repro.py --root <mirage-prod> --export-committee from the frozen verdicts and caches (CPU only); its sha256 is in the manifest's files, so POPE coverage and Judge.committee work without a rebuild.
# Conflicts: # slurm/eval_cell.slurm
…ks the judged block
# Conflicts: # run.py
… results tree load_records reads the model replies off the tree (the step that needs images); rebuild scores them and builds committee.json. --check gives the same 612 values either way.
…the cache test_grading: the frozen replies rescored must give all 612 values of final_tables.json and committee.json. test_wiring: the first COCO and Visual Genome images through run.py --cache-only, with gemma4-e4b's published rows; every request must hit and the results match expected.json. tests/data/freeze.py rebuilds the fixtures from a results tree. The smoke test is folded in.
ab26d49 to
822a3a3
Compare
4bafea5 to
a63b8ba
Compare
…g; loaders name their dataset Loaders iterate in file order (as every caller already did), carry KEY (coco / vg) for dataset_key, and record how many images were skipped before the pool filled. One helper resolves Visual Genome object names. No evaluated image changes.
…pty denominators; THRONE settings come from flags only THRONE no longer reads the distractor ratio, window or dtype from the environment (the evaluator path still can). No published value changes.
… identity_for and half-wired quantization A miss under --cache-only names the model revision and the cache file. Unpinned model paths log a warning; an empty model-path variable counts as unset. PaliGemma and Qwen2-VL raise like the other wrappers. Cache identities are unchanged for every registry key.
… and checks the rubric against the manifest Absolute image paths, a clear error on a torn live shard, workdirs created and resumable, the judge model read from judge/manifest.json, and Judge.load can read a live shard without the claude binary.
… cover it, and a config block in every result --live-judge grades a new model's missing caption mentions (CHAIR, Visual Genome THRONE). Visual Genome THRONE now reports missing_verdicts instead of scoring unjudged mentions as absent, and POPE is judged only at the published 6 questions per image. Results record the model identity and settings; tests/data/wiring/expected.json gains only that block. The docs-coverage and old results-table checks are removed ahead of the docs rewrite.
Summary
Brings the final judged scoring rules into Mirage: Mod-POPE, Mod-ROPE, Mod-CHAIR and THRONE class-wise F0.5. Scores are computed from cached model outputs plus static judge files; scoring never calls a judge. Stacked on #5.
metrics/judged.py: four pure scorers, record builders over a results tree (prediction caches, caption and THRONE result JSONs), andpython -m metrics.judged(--rootto rescore,--tableto regenerate the docs CSV and appendix). A missing verdict raisesMissingVerdictswith the list of items to grade.judgments/pope_drop_list.tsv; headline balanced accuracy, plus F1 and yes-ratio.metrics/chair.py:ChairMetric.tag_mentions(mentions with their in-context POS tag) andvariant_of_gt(relation to an annotation). Both reuse the existing synonym tables, tokenization and singularization.metrics/rope.py:judged_answer, the answer key the ROPE verdicts are stored under (first noun phrase). It sits next to the strict extractor but can't reuse it: changing the key would orphan the 28,897 stored verdicts.predict/cache.py:SQLiteResponseCache.rows()for offline rescoring.judgments/:mentions.tsv.gz(43,499 verdicts, 197 KB),rope.tsv.gz(28,897, 139 KB) andpope_drop_list.tsv(569 rows, 15 KB). Large judge inputs (thumbnails, mention pools) stay out of the repo.tools/judge/: the judge as one tool. It containsrun_judge.py(headlessclaude -pdriver, plus--exportto TSV rows) and the three skills verbatim:fcbf47e812280835e4524094fa3c04727715docs/methodology.mdx(new): per metric, what breaks, the Mod, and the evidence; the judge audit; footnotes; and a raw → Mod appendix per dataset, generated from the reference table.docs/model-results.csvholds the four headline columns and is generated bypython -m metrics.judged --table tests/data/final_tables.json --csv docs/model-results.csv.Validation
tests/test_final_tables_repro.pyrebuilds the table from the results tree and compares every value intests/data/final_tables.json, a byte copy of the eval-analysisfinal_tables.json. That is 612 values (324 floats, plus ranks and sample sizes), including the 198 reported metric values. Result: all equal, max float difference 0 (job 20172265; rerun on this commit: job 20172348, all equal, max difference 0). The test allows 1e-12 because the class-wise THRONE average sums over a set in the reference script, but no difference occurred.tests/test_judged_metrics.py: hand-built fixtures, one per rule (drop list, balanced accuracy, unsure exclusion, missing verdict, mention resolution and dedup, reference-caption objects, 5-word minimum, completed labels, support threshold). These need no datasets, caches or NLTK data. Full suite on the committed tree: 70 passed, 1 skipped (the reproduction test, which runs separately when MIRAGE_REPRO_ROOT is set), job 20172348.Notes