Repository navigation
Report Gemma 4, Ministral 3 and Aya Vision results; rescore THRONE with FLAN-T5-XL - #3
Closed
SalimAlkharsa wants to merge 20 commits into
Closed
SalimAlkharsa wants to merge 20 commits into
SalimAlkharsa wants to merge 20 commits into
Conversation
gemma4-e4b-think and gemma4-12b-think load the same checkpoints through HFChatThinkingVLM, a separate class so cache keys never mix modes. They add a 1,024-token thinking budget on top of each metric's max_new_tokens and cut the thought at the end-of-thinking token before scoring.
…th FLAN-T5-XL Add COCO and Visual Genome rows for gemma4-e4b, gemma4-12b, ministral3-8b, ministral3-14b and aya-vision-8b (500 images each). THRONE F1 for every row now uses a FLAN-T5-XL evaluator with 38 distractors per ground-truth object; the PaliGemma and Qwen2-VL values are rescored from their cached 1,000-image captions and supersede the FLAN-T5-base values.
SalimAlkharsa
force-pushed
the
eval/new-models
branch
from
October 15, 2026 12:57
041d762 to
22a54e2
Compare
SalimAlkharsa
force-pushed
the
eval/results-v1
branch
from
October 15, 2026 12:57
b8cba14 to
bbe9e1c
Compare
Strip markdown emphasis and code marks, a leading header or list marker, and surrounding quotes, and read the answer after "appears to be", "is pointing to" and "is the" as well as "is a/an". A parenthetical aside ends the answer. Matching stays exact: no synonyms and no picking one of several alternatives. Tests use real cached answers from every model.
All 16 rows were rescored from their cached answers (0 cache misses). Gemma-4-E4B rises from 0.0% to 30.9% (COCO) and 9.5% (VG); Ministral 3 gains 3-10 points; Qwen2-VL-7B gains 0.8 points on each dataset from "appears to be" answers. PaliGemma-3B is unchanged. The homepage and README notes now describe the extractor instead of a per-model caveat.
Closed
SalimAlkharsa
commented
Oct 16, 2026
Collaborator
Author
There was a problem hiding this comment.
do we absolutely need this? seems overly complex and not end to end?
Collaborator
Author
There was a problem hiding this comment.
Deleted; this PR adds no tests. The end-to-end coverage is python repro.py --root TREE --check, which rebuilds the published ROPE numbers from cached answers. Also new in this PR: matching/ is deleted. Only ROPE used it, and matching/judge.py was dead code. Its test tests/test_synonym_match.py goes with it.
…te, drop the new test
Conflict resolutions: - models/hf_chat.py: batch_generate stays removed; HFChatThinkingVLM stays in hf_chat.py and pins cache_name = "HFChatThinkingVLM" (it would otherwise inherit "HFChatVLM" and key as the non-thinking model). - predict/factory.py: the gemma4-*-think rows move into REGISTRY as models.hf_chat:HFChatThinkingVLM with the HFChatThinkingVLM mirror.
They raise NotImplementedError instead, so a metric that scores through its own entry point (ROPE's evaluate_predictions) still constructs.
ROPEMetric.strict(pred, gt) is pred.strip().lower() == gt.strip().lower(), the body of matching's ExactMatcher. evaluate_predictions and format_results score with it only; the result JSON keeps overall_accuracy_strict, per_position_accuracy_strict, num_images and num_objects_per_image, and drops the lenient fields (match_mode, overall_accuracy, per_position_accuracy, precision, recall, f1_score). Also removed: the matching imports and sys.path hack, the matchers and unused class_list parameters, the multi-box generate_prompt, parse_response and evaluate. Prompts and box drawing are unchanged.
Nothing imports it any more: the strict check lives in ROPEMetric.strict, and the lenient, synonym, WordNet and judge layers never produced a published number.
Same body as HFChatThinkingVLM; cache_name stays "HFChatThinkingVLM", so the gemma4-*-think cache keys do not change. The registry rows point at models.gemma4:Gemma4Thinking.
…strings Nothing in BaseMetric is abstract any more, so it no longer subclasses ABC. The compute_* helpers keep their bodies; their docstrings shrink to one line.
SalimAlkharsa
force-pushed
the
eval/new-models
branch
from
October 17, 2026 20:46
8017cb2 to
c5a86df
Compare
SalimAlkharsa
force-pushed
the
eval/results-v1
branch
from
October 17, 2026 20:46
455fce3 to
c01a80a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds COCO and Visual Genome results for the five models from #2 (500 images each), and re-scores the THRONE column for every row with a FLAN-T5-XL evaluator at 38 distractors per ground-truth object.
--cache-only, 0 misses). The new values supersede the FLAN-T5-base THRONE values on the homepage.gemma4-e4b-think/gemma4-12b-thinkkeys support the thinking ablation (separate class, so cache keys never mix modes; 1,024 extra tokens of thinking budget on top of each metric's cap).Notes on the numbers
ROPE extractor change
Salim approved this change in the morning (relayed through the overseer session). It is two commits, e103a7d (extractor + tests) and f532e7a (rescored values and docs), and can be dropped on its own.
What changed (formatting only):
ROPEMetric.parse_single_responsenow**,*, backticks), a leading header or list marker, and surrounding quotes;Matching is unchanged: still exact, with no synonyms, no fuzzy match and no picking one of several alternatives.
tests/test_rope_parse.pyuses real cached answers from every model, including regression cases where clean answers must parse identically.Before/after (percent; rescored from cache, CPU job 20159560, 0 misses in all 16 cells; "before" equals the previous CSV for every row; last column = answers newly correct / newly wrong under strict):
Validation
scripts/check_results_tables.pypasses.--cache-only. All had 0 misses and identical values.mirage-prod/results/rope_rescore/.Provenance
eval/new-models@ 041d762 (eval/protocol-parity@ face8b7), transformers 5.19.0, torch 2.9.1+cu128, mistral-common 1.12.0. TAMU HPRC Grace, A100 40 GB, bf16, greedy.ee0ef6023621cff504d758262d4e04895a5af4a2707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7f6fae9795746f63c9be8344932f01275f3c637343cea74c1ebaf5ce5f5a2553de470e2ceab825142d7601ac8fd3baf6f59c5fc9514c618ef0f24268f7d6315df2c2fb742f0f5b556879d730926ca9001d1d8734c, Qwen2-VL-2B895c3a49, Qwen2-VL-7Beed13092.afterok, ROPE, old-model THRONE). Replay check 20153088. Manifest:mirage-prod/setup_logs/grid_manifest.tsv.mirage-prod/cache/, one directory per cell named<model key>_<dataset>_<stage>(THRONE evaluator answers in<key>_<dataset>_throne_evaluator). 175 MB total; dev caches copied read-only tocache/old/.Thinking ablation (COCO POPE, n=500)
This table is reported separately from the main results and is not in the CSV or the homepage tables. Both modes use the same 500 images and the same 2,641 questions. Thinking-on adds 1,024 tokens to the 10-token POPE cap.
With thinking on, Gemma-4-12B gains +1.1 pp: 30 fewer false negatives (162 → 132) at about the same false-positive count (22 → 24). Gemma-4-E4B gains only +0.3 pp, and it becomes more conservative: false positives fall 64 → 47 while false negatives rise 107 → 117.
Takeaway: thinking buys +0.3 pp (E4B) and +1.1 pp (12B) on COCO POPE. It costs 8–11 GPU-hours per 2,641 questions, against about 40 minutes per thinking-off job (including model load).
Unclosed thinking blocks: E4B closed every thought (0 warnings). In 11 of the 2,641 12B answers (0.4%), the thought did not close within 1,034 tokens. The wrapper logs
Thinking block not closedand returns the truncated thought as the response. The unchanged POPE parser checks negative words first, so all 11 were scored as "no": each truncated thought contains at least one negative word ("I don't see…", "No spoon", "I'll go with no"). That is the result in the table above. Since all 11 count as "no", the worst case is that every one was a missed "yes": scoring them correctly would move accuracy by at most +0.4 pp (≤ 94.5%). This handling is now documented indocs/metrics-and-models.mdx(commit in this PR).Thinking-on captions/CHAIR/THRONE were cut: 130–145 s per caption, about 6× the thinking-off cost.
Follow-up