Repository navigation
Re-judge THRONE with a 1024-token evaluator window - #5
Closed
SalimAlkharsa wants to merge 22 commits into
Closed
SalimAlkharsa wants to merge 22 commits into
SalimAlkharsa wants to merge 22 commits into
Conversation
Captions longer than the 512-token window minus the question prompt were trimmed from the end, so the evaluator never saw objects named late in long captions. --throne-evaluator-max-tokens (or MIRAGE_THRONE_EVALUATOR_MAX_TOKENS) sets the window; the default stays 512 so earlier results reproduce from their caches. The window is part of the evaluator cache key and is reported in the results JSON.
… rows Re-judged COCO for all rows and Visual Genome for PaliGemma-3B and Gemma-4-12B with --throne-evaluator-max-tokens 1024, so no caption is trimmed. Other Visual Genome rows had no trimmed captions at 512 tokens; one Qwen2-VL-7B Visual Genome caption is footnoted.
Closed
SalimAlkharsa
commented
Oct 16, 2026
| </Tabs> | ||
|
|
||
| ROPE strict accuracy requires the extracted answer to match the label exactly. Before matching, the extractor removes answer formatting: markdown, surrounding quotes, a parenthetical aside, and lead-ins such as "appears to be" or "is pointing to". Every row, including PaliGemma-3B and Qwen2-VL, is scored this way from its cached answers. This replaces the earlier extractor, which scored Gemma-4-E4B's markdown answers ("The object inside box 4 is a **zebra**.") as 0.0%; Qwen2-VL-7B gains 0.8 points on each dataset from "appears to be" answers. Answers that name a part or a broader category ("a piece of broccoli", "a vehicle") still count as wrong. Ministral 3 answers POPE with "No" about twice as often as "Yes" on balanced questions, which lowers its POPE accuracy. THRONE F1 uses a FLAN-T5-XL evaluator and 38 distractor categories per ground-truth object for every row (1,000 images for the PaliGemma-3B and Qwen2-VL rows, rescored from their cached captions); it supersedes the earlier FLAN-T5-base THRONE values. To reproduce it, pass `--throne-evaluator google/flan-t5-xl --throne-distractor-ratio 38`. Metrics measure different behaviors and should be compared within each metric, not combined into a single model ranking. The source values are in the [model results CSV](https://github.com/Advanced-Vision-and-Learning-Lab/mirage/blob/main/docs/model-results.csv). A new model belongs in these tables only after its results have been measured and reviewed. | ||
| ROPE strict accuracy requires the extracted answer to match the label exactly. Before matching, the extractor removes answer formatting: markdown, surrounding quotes, a parenthetical aside, and lead-ins such as "appears to be" or "is pointing to". Every row, including PaliGemma-3B and Qwen2-VL, is scored this way from its cached answers. This replaces the earlier extractor, which scored Gemma-4-E4B's markdown answers ("The object inside box 4 is a **zebra**.") as 0.0%; Qwen2-VL-7B gains 0.8 points on each dataset from "appears to be" answers. Answers that name a part or a broader category ("a piece of broccoli", "a vehicle") still count as wrong. Ministral 3 answers POPE with "No" about twice as often as "Yes" on balanced questions, which lowers its POPE accuracy. THRONE F1 uses a FLAN-T5-XL evaluator and 38 distractor categories per ground-truth object for every row (1,000 images for the PaliGemma-3B and Qwen2-VL rows, rescored from their cached captions); it supersedes the earlier FLAN-T5-base THRONE values. The THRONE paper counts an object only when nine votes agree (FLAN-T5 Base, Large and XL, three phrasings each); these tables use the XL evaluator alone with its three phrasings. The evaluator reads up to 1,024 tokens, so captions are judged whole. At the earlier 512-token limit, long captions were trimmed from the end (up to 95% of Gemma-4-E4B's COCO captions), which lowered recall; the COCO rows and the PaliGemma-3B and Gemma-4-12B Visual Genome rows were re-judged at 1,024 tokens, and Muse Glimmer's Visual Genome row was judged at 1,024 tokens only. The other Visual Genome rows had no trimmed captions at 512 tokens, and untrimmed captions receive identical judgments at both limits, so their values are unchanged. The exception is one Qwen2-VL-7B Visual Genome caption of 1,000, still trimmed and not re-judged (effect at most 0.1 points). To reproduce it, pass `--throne-evaluator google/flan-t5-xl --throne-distractor-ratio 38 --throne-evaluator-max-tokens 1024`. Muse Glimmer has no switch to turn reasoning off, so its prompt is prefilled with the answer channel; it answers ROPE with full descriptive phrases, which strict matching counts as wrong. Metrics measure different behaviors and should be compared within each metric, not combined into a single model ranking. The source values are in the [model results CSV](https://github.com/Advanced-Vision-and-Learning-Lab/mirage/blob/main/docs/model-results.csv). A new model belongs in these tables only after its results have been measured and reviewed. |
Collaborator
Author
There was a problem hiding this comment.
Does this need to be so wordy..... I hate verbosity
Collaborator
Author
There was a problem hiding this comment.
Cut. The extra note is gone. The paragraph is main's wording, updated only for the ninth model and the window: "...THRONE F1 uses a FLAN-T5-XL evaluator with a 1,024-token input."
SalimAlkharsa
commented
Oct 16, 2026
Collaborator
Author
There was a problem hiding this comment.
we dont need this do we really need it?
Collaborator
Author
There was a problem hiding this comment.
Deleted. The 1,024-token window is covered by repro.py --check (every final THRONE value) and by the cache replay, where the THRONE cells are byte-identical.
…note, drop the new test
# Conflicts: # run.py
The cache identity reads the dtype from a static name map that gives the same str(torch.dtype) strings, so evaluator keys do not change. The device is picked when the weights load.
ThroneMetric(evaluator=fn, tokenizer=tok) answers with fn instead of FLAN-T5, keyed as InjectedEvaluator. cache_only=True opens the evaluator cache read-only and never loads the weights, so a miss raises. run.py passes --cache-only through, so a THRONE replay no longer writes to the evaluator cache.
classwise is metrics/judged.py's throne_classwise, byte for byte, and completed_labels is build_table's is_present lambda. ThroneMetric.score returns the raw (paper-style) and head class-wise scores; support() counts annotated images per category with build_table's formula. ThroneMetric.for_dataset picks the categories (COCO 80, or the VG pool's object names) and min_support (VG 5, COCO none); run.py builds the metric through it. format_results is unchanged.
A plain run now uses the published protocol. Each knob resolves when the metric is built: the argument, then its MIRAGE_THRONE_* variable, then the constant. run.py's --throne-evaluator defaults to None so the metric resolves it. Runs that pass the flags explicitly keep their cache keys and outputs.
…tor; slurm note on --cache-only
compute_f_beta(p, r, beta=0.5) evaluated the same expression as fbeta(p, r) with beta**2 == 0.25 exactly, so the f0.5 value is bit-identical. One F-beta implementation remains.
Every other metric's run returns what its score takes; THRONE returned (image_id, caption) pairs, so callers reached into format_results(). The output JSON (format_results) is unchanged.
# Conflicts: # run.py
SalimAlkharsa
force-pushed
the
eval/throne-window
branch
from
October 17, 2026 20:46
ab26d49 to
822a3a3
Compare
SalimAlkharsa
force-pushed
the
eval/muse-glimmer
branch
from
October 17, 2026 20:46
f162349 to
0c1d68e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The THRONE evaluator (FLAN-T5-XL) read at most 512 tokens per question. The prompt is
Text: {caption}, then the instruction and the question, so any caption longer than 474 FLAN-T5 tokens was trimmed from its end, and the evaluator never saw objects named late in the caption. On COCO, this affected 53–95% of the captions from the chat models (Gemma 4, Ministral 3, Aya Vision, Muse Glimmer) against 7–25% for PaliGemma and Qwen2-VL. On Visual Genome, captions are short and almost none were trimmed.This PR makes the window configurable and re-judges every affected THRONE row with a 1024-token window, which fits every caption up to the 512-token generation cap. Stacked on #4.
--throne-evaluator-max-tokens/MIRAGE_THRONE_EVALUATOR_MAX_TOKENSsets the evaluator window. It is used for caption trimming, the tokenizermax_length(previously hardcoded to 512), and the evaluator cache key, and it is reported in the results JSON asevaluator_max_input_tokens.Re-judged rows
COCO THRONE for all 9 models, plus Visual Genome THRONE for the two models with trimmed VG captions (PaliGemma-3B: 41 of 1,000; Gemma-4-12B: 9 of 500) and for Muse Glimmer 30B. The other VG rows had no trimmed captions, so their scores are the same at either window.
Validation
tests/test_throne_window.py(trimming, environment default, generationmax_length, cache identity) and thedemo.pyflag pass-through.Results
THRONE-XL F1 (percent) at the 512- and 1,024-token windows. Every 1,024 job read all captions from cache (0 misses) and trimmed none.
setup_logs/throne_w_compare.pyon Grace, which joins the two evaluator caches on prompt): on every re-judged COCO row and on PaliGemma-3B VG (1,763,775 questions), 0 answers differ for captions that were not trimmed at 512. On Gemma-4-12B VG, 6 of 981,981 such answers differ, which is bf16 numerical noise from different batch padding; F1 is unchanged to four decimals.docs/model-results.csv, the homepage and the README, and updates the methodology note.scripts/check_results_tables.pypasses.Provenance
eval/throne-window@ 82c5510; 1bf77c4 adds the table updates. Jobs: 1× A100,setup_logs/throne1024_submit.shon Grace,--throne-evaluator-max-tokens 1024, caption caches read-only (--cache-only).mirage-prod/results/grid/<cell>_throne_w1024.json; evaluator cachesmirage-prod/cache/<cell>_throne_w1024_evaluator/. The 512-token JSONs and caches are kept unchanged next to them.