Skip to content

Judged metrics: Mod-POPE, Mod-ROPE, Mod-CHAIR and class-wise THRONE F0.5 - #6

Closed
SalimAlkharsa wants to merge 48 commits into
eval/throne-windowfrom
eval/judged-metrics
Closed

SalimAlkharsa wants to merge 48 commits into
eval/throne-windowfrom
eval/judged-metrics

Conversation

@SalimAlkharsa

@SalimAlkharsa SalimAlkharsa commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Brings the final judged scoring rules into Mirage: Mod-POPE, Mod-ROPE, Mod-CHAIR and THRONE class-wise F0.5. Scores are computed from cached model outputs plus static judge files; scoring never calls a judge. Stacked on #5.

  • metrics/judged.py: four pure scorers, record builders over a results tree (prediction caches, caption and THRONE result JSONs), and python -m metrics.judged (--root to rescore, --table to regenerate the docs CSV and appendix). A missing verdict raises MissingVerdicts with the list of items to grade.
    • Mod-POPE: removes the no-questions in judgments/pope_drop_list.tsv; headline balanced accuracy, plus F1 and yes-ratio.
    • Mod-ROPE: judge grades; correct ÷ (correct + related + wrong), with unsure excluded.
    • Mod-CHAIR: completed labels, COCO reference-caption objects, noun mentions counted once per caption, a 5-word minimum, coverage, and F0.5.
    • THRONE: class-wise F0.5 with FLAN-T5-XL, a 5-word minimum, and on VG completed labels with category support ≥5.
  • metrics/chair.py: ChairMetric.tag_mentions (mentions with their in-context POS tag) and variant_of_gt (relation to an annotation). Both reuse the existing synonym tables, tokenization and singularization.
  • metrics/rope.py: judged_answer, the answer key the ROPE verdicts are stored under (first noun phrase). It sits next to the strict extractor but can't reuse it: changing the key would orphan the 28,897 stored verdicts.
  • predict/cache.py: SQLiteResponseCache.rows() for offline rescoring.
  • judgments/: mentions.tsv.gz (43,499 verdicts, 197 KB), rope.tsv.gz (28,897, 139 KB) and pope_drop_list.tsv (569 rows, 15 KB). Large judge inputs (thumbnails, mention pools) stay out of the repo.
  • tools/judge/: the judge as one tool. It contains run_judge.py (headless claude -p driver, plus --export to TSV rows) and the three skills verbatim:
    • judge-mentions: rubric v3.1, sha fcbf47e81228
    • judge-rope: v1, 0835e4524094
    • judge-pope: v1, fa3c04727715
  • Docs:
    • docs/methodology.mdx (new): per metric, what breaks, the Mod, and the evidence; the judge audit; footnotes; and a raw → Mod appendix per dataset, generated from the reference table.
    • The homepage and README tables become a headline-only placeholder (pending the redesign).
    • docs/model-results.csv holds the four headline columns and is generated by python -m metrics.judged --table tests/data/final_tables.json --csv docs/model-results.csv.

Validation

  • Record-level parity against the eval-analysis outputs, on the cluster, read-only (job 20172265), all identical:
    • POPE records: 18/18 model × dataset cells.
    • ROPE verdict keys: 22,500 per dataset.
    • ROPE strict flags.
    • Tagged mentions: 500 images × 9 models per dataset, 0 differing.
    • COCO reference-caption objects: 0 differing.
  • Reproduction: tests/test_final_tables_repro.py rebuilds the table from the results tree and compares every value in tests/data/final_tables.json, a byte copy of the eval-analysis final_tables.json. That is 612 values (324 floats, plus ranks and sample sizes), including the 198 reported metric values. Result: all equal, max float difference 0 (job 20172265; rerun on this commit: job 20172348, all equal, max difference 0). The test allows 1e-12 because the class-wise THRONE average sums over a set in the reference script, but no difference occurred.
  • tests/test_judged_metrics.py: hand-built fixtures, one per rule (drop list, balanced accuracy, unsure exclusion, missing verdict, mention resolution and dedup, reference-caption objects, 5-word minimum, completed labels, support threshold). These need no datasets, caches or NLTK data. Full suite on the committed tree: 70 passed, 1 skipped (the reproduction test, which runs separately when MIRAGE_REPRO_ROOT is set), job 20172348.

Notes

  • The headline values now use the shared first 500 images for all nine models. The old PaliGemma-3B and Qwen2-VL rows used 1,000 images for POPE/ROPE/CHAIR.
  • THRONE uses XL alone (3 phrasings). The paper's 9-vote ensemble was declined on cost; this is footnoted.
  • The ROPE and POPE-negative judges are unaudited; the mention judge audit is in-sample. Both points are stated in the methodology.

…NE F0.5

Scores cached model outputs against static judge verdicts in judgments/
(never calls a judge; missing verdicts raise). Reproduces the reference
table in tests/data/final_tables.json exactly from the results tree.
The judge ships as tools/judge (driver and the three skills). Homepage
and README tables become a headline-only placeholder; docs/methodology.mdx
documents each modification with a raw vs Mod appendix.
Comment thread docs/docs.json

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont change docs in this PR we will cross that bridge I care just about docs changing as far as numeric reporitng goes

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted. #6 changes no docs at all: no .mdx, docs.json, README or CSV. The homepage stays at #5's numbers until the homepage rework.

Comment thread judgments/mentions.tsv.gz

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does judgements need to be in repo? Does it better fit in a HF data repo later? How does it fit I dont want to bloat

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it belongs on HF. These files are what the Mod metrics are scored from. Without them, reproducing the published numbers means re-running the Claude judge, which costs API calls and only agrees with itself about 97% of the time. Right now they're still in git under judgments/, about 1 MB including the new 680 KB committee.json and final_tables.json. Plan: upload judgments/ to an HF dataset, pin its revision and sha256s in judge/manifest.json, and delete judgments/ from the repo. Do that before #6 merges, or squash-merge #6; otherwise the files stay in main's history.

Comment thread metrics/judged.py Outdated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i want this to be native not post mortem I dont like this code set up, obviously it can run post mortem. Im a bit concerned by design of code here relative to mirage and my design goals

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a judge should be its own class and addition to the modular design, Ill discuss w agent

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten, and metrics/judged.py is gone. The judge is its own package, judge/:

  • Judge (judge/judge.py) answers from a frozen, sha256-checked VerdictStore (judge/store.py).
  • It can fall back to a live ClaudeCLI (judge/claude.py), which is off by default.

Metrics take verdicts natively, e.g. ChairMetric.score(images, verdicts, refs, committee=...). run.py writes a judged block per cell, or missing_verdicts when an image hasn't been judged. The committee is frozen at the 9 judged models, so adding a model doesn't shift existing rows.

Current limit: Mod-POPE and Mod-CHAIR only cover the frozen 500 images per dataset, because there's no live POPE drop-list writer yet.

Comment thread metrics/rope.py

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again not a fan of the fact teh judge is little bits of jerry rigged code

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly fixed. The judging itself now lives in judge/. What's left in rope.py is judged_answer(), about 40 lines of regex plus NLTK that turn a response into the answer key the stored verdicts were graded on. It has to reproduce that key exactly or the stored verdicts are orphaned, so it can't be simplified without re-judging ROPE.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed from tests/. The published table is now judgments/final_tables.json, which repro.py --check compares against. It moves to HF with the rest of judgments/.

Comment thread tests/test_final_tables_repro.py Outdated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we realy need thsi

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted. Its job is now python repro.py --root TREE --check: 612/612 values, max diff 0. It needs the cached predictions, so it's a manual check, not a unit test.

Comment thread tests/test_judged_metrics.py Outdated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont think we need this

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted.

Comment thread tools/judge/README.md Outdated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont need this if we will document

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted, along with tools/; run_judge.py is now judge/claude.py. The one short file table left is judgments/README.md, which becomes the HF dataset card.

Comment thread README.md

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah keep these minimal

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#6 makes no README change. Across the stack, the README only gains the model-key list (#2, #4) and the results tables (#3, #5).

rope_records uses rope.strict(pred, gt), the same expression as the
deleted _rope_match_strict, and constructs ROPEMetric(num_objects=5)
(class_list was unused, so the names lookup is dropped too).
throne_classwise and fbeta now live in metrics.throne and metrics.base_metric
(same bytes); judged.py imports them, with MIN_WORDS and MIN_SUPPORT, and drops
its copies.
POPEMetric.score, ROPEMetric.score, ChairMetric.score and
ChairMetric.verified_present are mod_pope, mod_rope, mod_chair and
_verified_present moved verbatim; QUESTION_RE is _POPE_RE and
MissingVerdicts lives in base_metric. repro.py imports them.
…e metrics

ChairMetric.for_dataset and reference_objects are mention_records' blocks
moved; ChairMetric.score takes an optional committee pool (none in repro,
so the pool is today's union); THRONE goes through ThroneMetric.score and
completed_labels. _rows refuses a cache with two rows for one
(prompt, image_hash) instead of keeping the last.
…ault)

VerdictStore opens judgments/ (or $MIRAGE_JUDGMENTS_DIR), checks each file
against judge/manifest.json, and keeps new verdicts in a separate live
shard that can never overwrite a frozen key. Judge answers from the store;
with a live ClaudeCLI it grades the misses first. POPE drop decisions and
the committee exist only for covered items (committee.json, written by
repro.py --export-committee). judgments/final_tables.json is the 612-value
reference from daebc84.
build_table(root, store=None) reads the frozen store (default judgments/).
--check rebuilds every value, compares exactly with final_tables.json (and
committee.json when the store has one) and exits 1 on any difference; it
first asserts Python 3.12+ and the NLTK data. --export-committee writes the
frozen pool's per-item present and verified words. docs_tables is gone.
POPE: (image_id, object, annotated, said_yes), with the object read back
from the question as repro.py does. ROPE: (item, box, judged answer key,
strict). CHAIR: tagged mentions per image under the model's registry key
(ChairMetric.images). The questions asked, their order and format_results
are unchanged.
…THRONE

The frozen verdicts open before any generation (--judgments, default
$MIRAGE_JUDGMENTS_DIR, else judgments/). Each metric's score runs on its
run() records; CHAIR pools coverage with the committee, and VG THRONE
completes labels with the committee's and the model's verified words.
Items without verdicts are listed under missing_verdicts instead. The rest
of the result JSON, cache_stats included, is unchanged.
… checked when read

The manifest lists final_tables.json under "reference". VerdictStore.open
checks the files it loads ("files"), so a judgments dir with only the
verdict files serves run.py; reference_table() checks the table's sha256,
and repro --check reads it before the rebuild. open() takes live=True for
the default <directory>/live shard.
The live ROPE requests had no raw reply and no box-annotated images, so
ROPE misses now raise MissingVerdicts like POPE until that pipeline exists.
Judge.load(live=True) opens the store once with its live shard, and
judgments/live/ is ignored.
main() and ClaudeCLI both use claude_binary(): the argument, else
$MIRAGE_CLAUDE_BIN, else claude on PATH, and raise when none is found.
…t-key rule

POPE/ROPE/CHAIR/THRONE cells check COCO CHAIR's reference captions and the
NLTK data the tagging needs (punkt_tab, the POS tagger, WordNet, VG
stopwords) before the model runs, so a missing input fails the cell up
front instead of losing it after generation. ChairMetric.run and run.py
name a model's verdicts with metrics.base_metric.verdict_key.
caption_images and run.py's Visual Genome THRONE labels both take the first n
annotated samples; run.py no longer rebuilds them from image ids.
main(argv, model=..., judge=..., evaluator=...) threads them through
_run_dataset_metric (and the model through NOPE) in place of the ones the flags
build, so a caller can drive the CLI without weights or verdict files.
…THRONE scores run's records

Until committee.json ships, POPE and the CHAIR committee raise MissingVerdicts
for every image; demo.py reported that as a crash after generation. Each judged
score now prints the missing items and the demo continues, as run.py does.
THRONE scores what ThroneMetric.run returns.
With a live judge and no image_paths, mention_requests failed with a TypeError
on None once a mention needed grading; it now raises a ValueError that names
the missing argument.
Exported with python repro.py --root <mirage-prod> --export-committee from the
frozen verdicts and caches (CPU only); its sha256 is in the manifest's files, so
POPE coverage and Judge.committee work without a rebuild.
# Conflicts:
#	slurm/eval_cell.slurm
# Conflicts:
#	run.py
… results tree

load_records reads the model replies off the tree (the step that needs images); rebuild scores
them and builds committee.json. --check gives the same 612 values either way.
…the cache

test_grading: the frozen replies rescored must give all 612 values of final_tables.json and
committee.json. test_wiring: the first COCO and Visual Genome images through run.py --cache-only,
with gemma4-e4b's published rows; every request must hit and the results match expected.json.
tests/data/freeze.py rebuilds the fixtures from a results tree. The smoke test is folded in.
…g; loaders name their dataset

Loaders iterate in file order (as every caller already did), carry KEY (coco / vg) for dataset_key, and record how many images were skipped before the pool filled. One helper resolves Visual Genome object names. No evaluated image changes.
…pty denominators; THRONE settings come from flags only

THRONE no longer reads the distractor ratio, window or dtype from the environment (the evaluator path still can). No published value changes.
… identity_for and half-wired quantization

A miss under --cache-only names the model revision and the cache file. Unpinned model paths log a warning; an empty model-path variable counts as unset. PaliGemma and Qwen2-VL raise like the other wrappers. Cache identities are unchanged for every registry key.
… and checks the rubric against the manifest

Absolute image paths, a clear error on a torn live shard, workdirs created and resumable, the judge model read from judge/manifest.json, and Judge.load can read a live shard without the claude binary.
… cover it, and a config block in every result

--live-judge grades a new model's missing caption mentions (CHAIR, Visual Genome THRONE). Visual Genome THRONE now reports missing_verdicts instead of scoring unjudged mentions as absent, and POPE is judged only at the published 6 questions per image. Results record the model identity and settings; tests/data/wiring/expected.json gains only that block. The docs-coverage and old results-table checks are removed ahead of the docs rewrite.
@SalimAlkharsa
SalimAlkharsa deleted the eval/judged-metrics branch October 19, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant