Skip to content

Report Gemma 4, Ministral 3 and Aya Vision results; rescore THRONE with FLAN-T5-XL - #3

Closed
SalimAlkharsa wants to merge 20 commits into
eval/new-modelsfrom
eval/results-v1
Closed

SalimAlkharsa wants to merge 20 commits into
eval/new-modelsfrom
eval/results-v1

Conversation

@SalimAlkharsa

@SalimAlkharsa SalimAlkharsa commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds COCO and Visual Genome results for the five models from #2 (500 images each), and re-scores the THRONE column for every row with a FLAN-T5-XL evaluator at 38 distractors per ground-truth object.

  • New rows: Gemma-4-E4B, Gemma-4-12B, Ministral-3-8B, Ministral-3-14B, Aya-Vision-8B. 500 images per dataset, a prefix of the same fixed candidate pool used for the 1,000-image rows.
  • THRONE: PaliGemma-3B and Qwen2-VL rows are re-scored from their cached 1,000-image captions (--cache-only, 0 misses). The new values supersede the FLAN-T5-base THRONE values on the homepage.
  • ROPE extractor: the strict extractor now removes answer formatting before the exact match, and every row (old and new) is rescored from cache. See "ROPE extractor change" below.
  • Footnotes cover the sample sizes, the THRONE redefinition, the ROPE extractor, and Ministral's POPE bias.
  • gemma4-e4b-think / gemma4-12b-think keys support the thinking ablation (separate class, so cache keys never mix modes; 1,024 extra tokens of thinking budget on top of each metric's cap).

Notes on the numbers

  • ROPE strict is an exact label match after formatting is removed. Answers naming a part, a broader category or two alternatives ("a piece of broccoli", "a vehicle", "a tent or canopy") still count as wrong. That is why the chat models stay below Gemma-4-12B.
  • Ministral 3 POPE: answers are clean "Yes."/"No.", but it says "No" about twice as often as "Yes" on balanced questions (COCO: 1,790 vs 834).
  • VG CHAIR-I is high (77–88%) for the chat models: their 512-token captions mention many more VG object names than PaliGemma/Qwen2-VL captions do.

ROPE extractor change

Salim approved this change in the morning (relayed through the overseer session). It is two commits, e103a7d (extractor + tests) and f532e7a (rescored values and docs), and can be dropped on its own.

What changed (formatting only): ROPEMetric.parse_single_response now

  • strips markdown (**, *, backticks), a leading header or list marker, and surrounding quotes;
  • reads the answer after "appears to be", "seems to be", "is pointing to" and "is the", as well as "is a/an";
  • ends the answer at a parenthetical aside;
  • does not treat an echoed question ("I am not sure what is the object inside box 1") as an answer.

Matching is unchanged: still exact, with no synonyms, no fuzzy match and no picking one of several alternatives. tests/test_rope_parse.py uses real cached answers from every model, including regression cases where clean answers must parse identically.

Before/after (percent; rescored from cache, CPU job 20159560, 0 misses in all 16 cells; "before" equals the previous CSV for every row; last column = answers newly correct / newly wrong under strict):

Dataset Model n strict before strict after Δ lenient before lenient after Δ strict +/−
COCO PaliGemma-3B 5000 21.6 21.6 +0.0 23.6 23.6 +0.0 +0/−0
COCO Qwen2-VL-2B 5000 3.6 3.7 +0.0 12.0 11.9 -0.1 +1/−0
COCO Qwen2-VL-7B 5000 13.1 13.8 +0.8 47.4 47.1 -0.3 +38/−0
COCO Gemma-4-E4B 2500 0.0 30.9 +30.9 40.4 38.7 -1.7 +773/−0
COCO Gemma-4-12B 2500 44.4 44.5 +0.1 50.1 50.1 +0.0 +3/−0
COCO Ministral-3-8B 2500 5.1 11.0 +5.9 33.5 26.0 -7.4 +147/−0
COCO Ministral-3-14B 2500 3.0 12.7 +9.7 32.6 25.1 -7.5 +242/−0
COCO Aya-Vision-8B 2500 24.5 25.0 +0.5 40.0 39.6 -0.4 +12/−0
VG PaliGemma-3B 5000 8.2 8.2 +0.0 9.2 9.2 +0.0 +0/−0
VG Qwen2-VL-2B 5000 8.6 8.6 +0.1 15.8 15.8 -0.0 +4/−0
VG Qwen2-VL-7B 5000 10.3 11.1 +0.8 19.6 18.7 -0.9 +39/−0
VG Gemma-4-E4B 2500 0.0 9.5 +9.5 21.8 18.7 -3.1 +237/−0
VG Gemma-4-12B 2500 21.1 21.4 +0.2 28.4 28.3 -0.2 +6/−0
VG Ministral-3-8B 2500 0.3 3.6 +3.3 19.2 13.8 -5.4 +83/−0
VG Ministral-3-14B 2500 0.1 4.2 +4.1 19.5 13.7 -5.8 +102/−0
VG Aya-Vision-8B 2500 8.9 9.1 +0.2 17.2 16.8 -0.4 +6/−0
  • Old rows: PaliGemma-3B is unchanged. Qwen2-VL-7B moves +0.8 pp on both datasets: 38 COCO + 39 VG answers like "The object inside box 2 appears to be a fire hydrant." or "…is the license plate of the car." that the old extractor missed. Qwen2-VL-2B moves ≤0.1 pp.
  • Lenient drops (Ministral −5 to −7.5, E4B −1.7/−3.1): when the old extractor failed, lenient matched against the whole first line, so it credited objects mentioned anywhere in the sentence ("…a small, folded blanket placed on top of a backpack", "a sofa (or couch)"). Lenient now sees the extracted answer, the same as answers that always parsed. An earlier rescore (job 20159377) showed −12 pp lenient for Qwen2-VL-7B. The cause was that echoed-question bug ("what is the object inside box N"), which was fixed before commit; lenient is now −0.3.
  • Spot check: 24 random newly-accepted strict answers were read by eye; all name exactly the boxed object (e.g. "is a zebra.", "appears to be a bicycle.", "is an apple (or a bowl containing an apple)"). No false accepts.
  • No answer lost strict credit in any of the 16 cells (about 40,000 answers).

Validation

  • scripts/check_results_tables.py passes.
  • Replay check: every POPE, ROPE and CHAIR cell (30) was re-run from its cache on CPU with --cache-only. All had 0 misses and identical values.
  • Captions: every THRONE cell read its captions from cache with 0 misses.
  • ROPE rescore: all 16 ROPE cells re-run from cache with the new extractor (job 20159560), 0 misses. Results in mirage-prod/results/rope_rescore/.

Provenance

  • Code: eval/new-models @ 041d762 (eval/protocol-parity @ face8b7), transformers 5.19.0, torch 2.9.1+cu128, mistral-common 1.12.0. TAMU HPRC Grace, A100 40 GB, bf16, greedy.
  • Checkpoints (HF snapshot SHAs):
    • Gemma-4-E4B ee0ef6023621cff504d758262d4e04895a5af4a2
    • Gemma-4-12B 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7
    • Ministral-3-8B-Instruct-2512-BF16 f6fae9795746f63c9be8344932f01275f3c63734
    • Ministral-3-14B-Instruct-2512-BF16 3cea74c1ebaf5ce5f5a2553de470e2ceab825142
    • Aya-Vision-8B d7601ac8fd3baf6f59c5fc9514c618ef0f24268f
    • THRONE evaluator FLAN-T5-XL 7d6315df2c2fb742f0f5b556879d730926ca9001
    • Old models (captions replayed): PaliGemma d1d8734c, Qwen2-VL-2B 895c3a49, Qwen2-VL-7B eed13092.
  • SLURM: 46 cell jobs 20153024–20153086 (POPE, captions+CHAIR → THRONE afterok, ROPE, old-model THRONE). Replay check 20153088. Manifest: mirage-prod/setup_logs/grid_manifest.tsv.
  • Caches: mirage-prod/cache/, one directory per cell named <model key>_<dataset>_<stage> (THRONE evaluator answers in <key>_<dataset>_throne_evaluator). 175 MB total; dev caches copied read-only to cache/old/.
  • Compute: about 5.3k SU for smoke + grid + replay.

Thinking ablation (COCO POPE, n=500)

This table is reported separately from the main results and is not in the CSV or the homepage tables. Both modes use the same 500 images and the same 2,641 questions. Thinking-on adds 1,024 tokens to the 10-token POPE cap.

Model Thinking Accuracy Precision Recall F1 Predicted "yes" Job
Gemma-4-12B off 93.0% 97.8% 85.8% 91.4% 37.9% 20153042
Gemma-4-12B on 94.1% 97.7% 88.4% 92.8% 39.1% 20153856 (8 h 12 min)
Gemma-4-E4B off 93.5% 94.2% 90.6% 92.4% 41.6% 20153024
Gemma-4-E4B on 93.8% 95.6% 89.7% 92.6% 40.6% 20153855 (11 h 27 min)

With thinking on, Gemma-4-12B gains +1.1 pp: 30 fewer false negatives (162 → 132) at about the same false-positive count (22 → 24). Gemma-4-E4B gains only +0.3 pp, and it becomes more conservative: false positives fall 64 → 47 while false negatives rise 107 → 117.

Takeaway: thinking buys +0.3 pp (E4B) and +1.1 pp (12B) on COCO POPE. It costs 8–11 GPU-hours per 2,641 questions, against about 40 minutes per thinking-off job (including model load).

Unclosed thinking blocks: E4B closed every thought (0 warnings). In 11 of the 2,641 12B answers (0.4%), the thought did not close within 1,034 tokens. The wrapper logs Thinking block not closed and returns the truncated thought as the response. The unchanged POPE parser checks negative words first, so all 11 were scored as "no": each truncated thought contains at least one negative word ("I don't see…", "No spoon", "I'll go with no"). That is the result in the table above. Since all 11 count as "no", the worst case is that every one was a missed "yes": scoring them correctly would move accuracy by at most +0.4 pp (≤ 94.5%). This handling is now documented in docs/metrics-and-models.mdx (commit in this PR).

Thinking-on captions/CHAIR/THRONE were cut: 130–145 s per caption, about 6× the thinking-off cost.

Follow-up

  • Publish the per-cell caches as Hugging Face dataset artifacts. The layout is already one directory per cell.
  • Optional: make the ROPE strict extractor robust to markdown and phrase answers, then re-score all rows from cache (CPU only).

gemma4-e4b-think and gemma4-12b-think load the same checkpoints through
HFChatThinkingVLM, a separate class so cache keys never mix modes. They
add a 1,024-token thinking budget on top of each metric's max_new_tokens
and cut the thought at the end-of-thinking token before scoring.
…th FLAN-T5-XL

Add COCO and Visual Genome rows for gemma4-e4b, gemma4-12b,
ministral3-8b, ministral3-14b and aya-vision-8b (500 images each).
THRONE F1 for every row now uses a FLAN-T5-XL evaluator with 38
distractors per ground-truth object; the PaliGemma and Qwen2-VL values
are rescored from their cached 1,000-image captions and supersede the
FLAN-T5-base values.
Strip markdown emphasis and code marks, a leading header or list marker,
and surrounding quotes, and read the answer after "appears to be",
"is pointing to" and "is the" as well as "is a/an". A parenthetical aside
ends the answer. Matching stays exact: no synonyms and no picking one of
several alternatives. Tests use real cached answers from every model.
All 16 rows were rescored from their cached answers (0 cache misses).
Gemma-4-E4B rises from 0.0% to 30.9% (COCO) and 9.5% (VG); Ministral 3
gains 3-10 points; Qwen2-VL-7B gains 0.8 points on each dataset from
"appears to be" answers. PaliGemma-3B is unchanged. The homepage and
README notes now describe the extractor instead of a per-model caveat.
@SalimAlkharsa SalimAlkharsa mentioned this pull request Oct 15, 2026
Comment thread tests/test_rope_parse.py Outdated

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we absolutely need this? seems overly complex and not end to end?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted; this PR adds no tests. The end-to-end coverage is python repro.py --root TREE --check, which rebuilds the published ROPE numbers from cached answers. Also new in this PR: matching/ is deleted. Only ROPE used it, and matching/judge.py was dead code. Its test tests/test_synonym_match.py goes with it.

Conflict resolutions:
- models/hf_chat.py: batch_generate stays removed; HFChatThinkingVLM
  stays in hf_chat.py and pins cache_name = "HFChatThinkingVLM" (it
  would otherwise inherit "HFChatVLM" and key as the non-thinking model).
- predict/factory.py: the gemma4-*-think rows move into REGISTRY as
  models.hf_chat:HFChatThinkingVLM with the HFChatThinkingVLM mirror.
They raise NotImplementedError instead, so a metric that scores through
its own entry point (ROPE's evaluate_predictions) still constructs.
ROPEMetric.strict(pred, gt) is pred.strip().lower() == gt.strip().lower(),
the body of matching's ExactMatcher. evaluate_predictions and format_results
score with it only; the result JSON keeps overall_accuracy_strict,
per_position_accuracy_strict, num_images and num_objects_per_image, and
drops the lenient fields (match_mode, overall_accuracy,
per_position_accuracy, precision, recall, f1_score).

Also removed: the matching imports and sys.path hack, the matchers and
unused class_list parameters, the multi-box generate_prompt,
parse_response and evaluate. Prompts and box drawing are unchanged.
Nothing imports it any more: the strict check lives in ROPEMetric.strict,
and the lenient, synonym, WordNet and judge layers never produced a
published number.
Same body as HFChatThinkingVLM; cache_name stays "HFChatThinkingVLM", so
the gemma4-*-think cache keys do not change. The registry rows point at
models.gemma4:Gemma4Thinking.
…strings

Nothing in BaseMetric is abstract any more, so it no longer subclasses ABC.
The compute_* helpers keep their bodies; their docstrings shrink to one line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant