Skip to content

Re-judge THRONE with a 1024-token evaluator window - #5

Closed
SalimAlkharsa wants to merge 22 commits into
eval/muse-glimmerfrom
eval/throne-window
Closed

SalimAlkharsa wants to merge 22 commits into
eval/muse-glimmerfrom
eval/throne-window

Conversation

@SalimAlkharsa

@SalimAlkharsa SalimAlkharsa commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The THRONE evaluator (FLAN-T5-XL) read at most 512 tokens per question. The prompt is Text: {caption}, then the instruction and the question, so any caption longer than 474 FLAN-T5 tokens was trimmed from its end, and the evaluator never saw objects named late in the caption. On COCO, this affected 53–95% of the captions from the chat models (Gemma 4, Ministral 3, Aya Vision, Muse Glimmer) against 7–25% for PaliGemma and Qwen2-VL. On Visual Genome, captions are short and almost none were trimmed.

This PR makes the window configurable and re-judges every affected THRONE row with a 1024-token window, which fits every caption up to the 512-token generation cap. Stacked on #4.

  • --throne-evaluator-max-tokens / MIRAGE_THRONE_EVALUATOR_MAX_TOKENS sets the evaluator window. It is used for caption trimming, the tokenizer max_length (previously hardcoded to 512), and the evaluator cache key, and it is reported in the results JSON as evaluator_max_input_tokens.
  • The default stays 512, so earlier results reproduce from their caches.
  • T5 has no absolute position embeddings. Its relative attention bias puts every distance of 128 or more into one bucket, so inputs between 512 and 1024 tokens contain no position values unseen in training.

Re-judged rows

COCO THRONE for all 9 models, plus Visual Genome THRONE for the two models with trimmed VG captions (PaliGemma-3B: 41 of 1,000; Gemma-4-12B: 9 of 500) and for Muse Glimmer 30B. The other VG rows had no trimmed captions, so their scores are the same at either window.

Validation

  • CPU tests on Grace: tests/test_throne_window.py (trimming, environment default, generation max_length, cache identity) and the demo.py flag pass-through.
  • Sanity gate before the other 11 jobs ran (job 20168469, Gemma-4-12B COCO): the 453 captions that were not trimmed at 512 have identical answers on all 95,202 evaluator questions at 1024. Only the 47 trimmed captions were judged differently. No captions were trimmed at 1024, and all 500 captions were read from cache.

Results

THRONE-XL F1 (percent) at the 512- and 1,024-token windows. Every 1,024 job read all captions from cache (0 misses) and trimmed none.

Dataset Model Job Trimmed at 512 F1 at 512 F1 at 1,024 Δ
COCO Qwen2-VL-7B 20168948 250 / 1,000 84.40 84.48 +0.08
COCO Qwen2-VL-2B 20168947 213 / 1,000 83.83 83.99 +0.16
COCO Muse-Glimmer-30B 20168944 435 / 500 83.56 83.94 +0.38
COCO Aya-Vision-8B 20168942 337 / 500 82.25 83.20 +0.95
COCO Ministral-3-14B 20168943 369 / 500 81.63 82.28 +0.65
COCO Ministral-3-8B 20168941 265 / 500 80.79 81.01 +0.22
COCO PaliGemma-3B 20168945 70 / 1,000 80.90 80.90 0.00
COCO Gemma-4-12B 20168469 47 / 500 78.36 78.95 +0.59
COCO Gemma-4-E4B 20168940 477 / 500 76.96 78.73 +1.77
Visual Genome Muse-Glimmer-30B 20168951 — — 27.90 —
Visual Genome PaliGemma-3B 20168950 41 / 1,000 27.63 27.61 −0.02
Visual Genome Gemma-4-12B 20168949 9 / 500 27.09 27.09 0.00
  • COCO ranking: the only change is Ministral-3-8B (81.01) moving above PaliGemma-3B (80.90). Qwen2-VL-7B, Qwen2-VL-2B and Muse Glimmer stay first to third, with Qwen2-VL-2B and Muse Glimmer now 0.05 apart. Gains are largest for the models whose captions were trimmed most (Gemma-4-E4B +1.77, Aya +0.95).
  • Muse Glimmer VG THRONE was only judged at 1,024 tokens; its 512-token job was cancelled before it ran.
  • Rows not re-judged (Visual Genome: Qwen2-VL-2B, Gemma-4-E4B, Ministral-3-8B, Ministral-3-14B, Aya-Vision-8B) had no trimmed captions at 512 tokens. The gate showed that untrimmed captions get identical judgments at both windows, so their values equal the 1,024 result.
  • Qwen2-VL-7B Visual Genome had 1 of 1,000 captions trimmed at 512 tokens. Per Salim, it was not re-judged and is footnoted (effect at most 0.1 points).
  • Identity checks (setup_logs/throne_w_compare.py on Grace, which joins the two evaluator caches on prompt): on every re-judged COCO row and on PaliGemma-3B VG (1,763,775 questions), 0 answers differ for captions that were not trimmed at 512. On Gemma-4-12B VG, 6 of 981,981 such answers differ, which is bf16 numerical noise from different batch padding; F1 is unchanged to four decimals.
  • This PR also adds the Muse Glimmer rows (Add Muse Glimmer 30B #4) to docs/model-results.csv, the homepage and the README, and updates the methodology note. scripts/check_results_tables.py passes.

Provenance

  • Code: the jobs ran on eval/throne-window @ 82c5510; 1bf77c4 adds the table updates. Jobs: 1× A100, setup_logs/throne1024_submit.sh on Grace, --throne-evaluator-max-tokens 1024, caption caches read-only (--cache-only).
  • Results: mirage-prod/results/grid/<cell>_throne_w1024.json; evaluator caches mirage-prod/cache/<cell>_throne_w1024_evaluator/. The 512-token JSONs and caches are kept unchanged next to them.

Captions longer than the 512-token window minus the question prompt were
trimmed from the end, so the evaluator never saw objects named late in
long captions. --throne-evaluator-max-tokens (or
MIRAGE_THRONE_EVALUATOR_MAX_TOKENS) sets the window; the default stays 512
so earlier results reproduce from their caches. The window is part of the
evaluator cache key and is reported in the results JSON.
… rows

Re-judged COCO for all rows and Visual Genome for PaliGemma-3B and
Gemma-4-12B with --throne-evaluator-max-tokens 1024, so no caption is
trimmed. Other Visual Genome rows had no trimmed captions at 512 tokens;
one Qwen2-VL-7B Visual Genome caption is footnoted.
@SalimAlkharsa SalimAlkharsa mentioned this pull request Oct 16, 2026
Comment thread docs/index.mdx Outdated
</Tabs>

ROPE strict accuracy requires the extracted answer to match the label exactly. Before matching, the extractor removes answer formatting: markdown, surrounding quotes, a parenthetical aside, and lead-ins such as "appears to be" or "is pointing to". Every row, including PaliGemma-3B and Qwen2-VL, is scored this way from its cached answers. This replaces the earlier extractor, which scored Gemma-4-E4B's markdown answers ("The object inside box 4 is a **zebra**.") as 0.0%; Qwen2-VL-7B gains 0.8 points on each dataset from "appears to be" answers. Answers that name a part or a broader category ("a piece of broccoli", "a vehicle") still count as wrong. Ministral 3 answers POPE with "No" about twice as often as "Yes" on balanced questions, which lowers its POPE accuracy. THRONE F1 uses a FLAN-T5-XL evaluator and 38 distractor categories per ground-truth object for every row (1,000 images for the PaliGemma-3B and Qwen2-VL rows, rescored from their cached captions); it supersedes the earlier FLAN-T5-base THRONE values. To reproduce it, pass `--throne-evaluator google/flan-t5-xl --throne-distractor-ratio 38`. Metrics measure different behaviors and should be compared within each metric, not combined into a single model ranking. The source values are in the [model results CSV](https://github.com/Advanced-Vision-and-Learning-Lab/mirage/blob/main/docs/model-results.csv). A new model belongs in these tables only after its results have been measured and reviewed.
ROPE strict accuracy requires the extracted answer to match the label exactly. Before matching, the extractor removes answer formatting: markdown, surrounding quotes, a parenthetical aside, and lead-ins such as "appears to be" or "is pointing to". Every row, including PaliGemma-3B and Qwen2-VL, is scored this way from its cached answers. This replaces the earlier extractor, which scored Gemma-4-E4B's markdown answers ("The object inside box 4 is a **zebra**.") as 0.0%; Qwen2-VL-7B gains 0.8 points on each dataset from "appears to be" answers. Answers that name a part or a broader category ("a piece of broccoli", "a vehicle") still count as wrong. Ministral 3 answers POPE with "No" about twice as often as "Yes" on balanced questions, which lowers its POPE accuracy. THRONE F1 uses a FLAN-T5-XL evaluator and 38 distractor categories per ground-truth object for every row (1,000 images for the PaliGemma-3B and Qwen2-VL rows, rescored from their cached captions); it supersedes the earlier FLAN-T5-base THRONE values. The THRONE paper counts an object only when nine votes agree (FLAN-T5 Base, Large and XL, three phrasings each); these tables use the XL evaluator alone with its three phrasings. The evaluator reads up to 1,024 tokens, so captions are judged whole. At the earlier 512-token limit, long captions were trimmed from the end (up to 95% of Gemma-4-E4B's COCO captions), which lowered recall; the COCO rows and the PaliGemma-3B and Gemma-4-12B Visual Genome rows were re-judged at 1,024 tokens, and Muse Glimmer's Visual Genome row was judged at 1,024 tokens only. The other Visual Genome rows had no trimmed captions at 512 tokens, and untrimmed captions receive identical judgments at both limits, so their values are unchanged. The exception is one Qwen2-VL-7B Visual Genome caption of 1,000, still trimmed and not re-judged (effect at most 0.1 points). To reproduce it, pass `--throne-evaluator google/flan-t5-xl --throne-distractor-ratio 38 --throne-evaluator-max-tokens 1024`. Muse Glimmer has no switch to turn reasoning off, so its prompt is prefilled with the answer channel; it answers ROPE with full descriptive phrases, which strict matching counts as wrong. Metrics measure different behaviors and should be compared within each metric, not combined into a single model ranking. The source values are in the [model results CSV](https://github.com/Advanced-Vision-and-Learning-Lab/mirage/blob/main/docs/model-results.csv). A new model belongs in these tables only after its results have been measured and reviewed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be so wordy..... I hate verbosity

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cut. The extra note is gone. The paragraph is main's wording, updated only for the ninth model and the window: "...THRONE F1 uses a FLAN-T5-XL evaluator with a 1,024-token input."

Comment thread tests/test_throne_window.py Outdated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont need this do we really need it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted. The 1,024-token window is covered by repro.py --check (every final THRONE value) and by the cache replay, where the THRONE cells are byte-identical.

# Conflicts:
#	run.py
The cache identity reads the dtype from a static name map that gives the same
str(torch.dtype) strings, so evaluator keys do not change. The device is picked
when the weights load.
ThroneMetric(evaluator=fn, tokenizer=tok) answers with fn instead of FLAN-T5,
keyed as InjectedEvaluator. cache_only=True opens the evaluator cache read-only
and never loads the weights, so a miss raises. run.py passes --cache-only
through, so a THRONE replay no longer writes to the evaluator cache.
classwise is metrics/judged.py's throne_classwise, byte for byte, and
completed_labels is build_table's is_present lambda. ThroneMetric.score returns
the raw (paper-style) and head class-wise scores; support() counts annotated
images per category with build_table's formula. ThroneMetric.for_dataset picks
the categories (COCO 80, or the VG pool's object names) and min_support (VG 5,
COCO none); run.py builds the metric through it. format_results is unchanged.
A plain run now uses the published protocol. Each knob resolves when the metric
is built: the argument, then its MIRAGE_THRONE_* variable, then the constant.
run.py's --throne-evaluator defaults to None so the metric resolves it. Runs that
pass the flags explicitly keep their cache keys and outputs.
compute_f_beta(p, r, beta=0.5) evaluated the same expression as fbeta(p, r)
with beta**2 == 0.25 exactly, so the f0.5 value is bit-identical. One F-beta
implementation remains.
Every other metric's run returns what its score takes; THRONE returned
(image_id, caption) pairs, so callers reached into format_results(). The
output JSON (format_results) is unchanged.
# Conflicts:
#	run.py
@SalimAlkharsa
SalimAlkharsa deleted the eval/throne-window branch October 19, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant