Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a795e8b
Add thinking-on Gemma 4 keys for the thinking ablation
SalimAlkharsa Oct 15, 2026
2a9453e
Report Gemma 4, Ministral 3 and Aya Vision results; rescore THRONE wi…
SalimAlkharsa Oct 15, 2026
3b3c350
Document how an unclosed thinking block is scored
SalimAlkharsa Oct 15, 2026
96a2561
Normalize answer formatting in the ROPE single-box extractor
SalimAlkharsa Oct 15, 2026
b4cfb7b
Rescore ROPE strict for every row with the formatting-robust extractor
SalimAlkharsa Oct 15, 2026
2ca4ebd
Merge eval/new-models; keep only the results tables and a one-line no…
SalimAlkharsa Oct 16, 2026
0093f01
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
75c1774
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
1dd4d7e
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
ca02f58
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
26f1f5d
Make BaseMetric.evaluate and format_results non-abstract
SalimAlkharsa Oct 17, 2026
abb839a
ROPE: plain ROPE is one exact-match comparison
SalimAlkharsa Oct 17, 2026
176a749
Delete matching/ and its test
SalimAlkharsa Oct 17, 2026
81fc4a7
predict/identity: drop the reference to the deleted matching package
SalimAlkharsa Oct 17, 2026
69e3356
Move the Gemma 4 thinking variant into gemma4.py as Gemma4Thinking
SalimAlkharsa Oct 17, 2026
975ea59
ROPE: docstrings describe the single-box protocol
SalimAlkharsa Oct 17, 2026
149c772
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
b317f24
base_metric: BaseMetric is a plain base class; one-line compute_* doc…
SalimAlkharsa Oct 17, 2026
93b78d8
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
c01a80a
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,25 @@ Mirage supports `paligemma`, `qwen2vl`, `gemma4-e4b`, `gemma4-12b`, `ministral3-

## Reference results

Scores are percentages from 1,000 images per model and dataset. Higher is better for POPE, ROPE, and THRONE; lower is better for CHAIR-I.
Scores are percentages from 1,000 images per model and dataset (500 for Gemma 4, Ministral 3 and Aya Vision); THRONE F1 uses a FLAN-T5-XL evaluator. Higher is better for POPE, ROPE, and THRONE; lower is better for CHAIR-I.

| Dataset | Model | POPE accuracy ↑ | ROPE strict accuracy ↑ | CHAIR-I hallucination rate ↓ | THRONE F1 ↑ |
| --- | --- | ---: | ---: | ---: | ---: |
| COCO | PaliGemma-3B | 94.2% | 21.6% | 9.1% | 72.9% |
| COCO | Qwen2-VL-2B | 93.6% | 3.6% | 12.8% | 73.4% |
| COCO | Qwen2-VL-7B | 93.5% | 13.1% | 12.7% | 72.9% |
| Visual Genome | PaliGemma-3B | 88.0% | 8.2% | 67.8% | 21.5% |
| Visual Genome | Qwen2-VL-2B | 88.7% | 8.6% | 75.6% | 24.0% |
| Visual Genome | Qwen2-VL-7B | 89.5% | 10.3% | 69.3% | 23.6% |
| COCO | PaliGemma-3B | 94.2% | 21.6% | 9.1% | 80.9% |
| COCO | Qwen2-VL-2B | 93.6% | 3.7% | 12.8% | 83.8% |
| COCO | Qwen2-VL-7B | 93.5% | 13.8% | 12.7% | 84.4% |
| COCO | Gemma-4-E4B | 93.5% | 30.9% | 12.5% | 77.0% |
| COCO | Ministral-3-8B | 88.1% | 11.0% | 13.7% | 80.8% |
| COCO | Aya-Vision-8B | 95.5% | 25.0% | 16.9% | 82.2% |
| COCO | Gemma-4-12B | 93.0% | 44.5% | 15.2% | 78.4% |
| COCO | Ministral-3-14B | 88.2% | 12.7% | 14.1% | 81.6% |
| Visual Genome | PaliGemma-3B | 88.0% | 8.2% | 67.8% | 27.6% |
| Visual Genome | Qwen2-VL-2B | 88.7% | 8.6% | 75.6% | 27.4% |
| Visual Genome | Qwen2-VL-7B | 89.5% | 11.1% | 69.3% | 27.2% |
| Visual Genome | Gemma-4-E4B | 89.6% | 9.5% | 84.1% | 25.5% |
| Visual Genome | Ministral-3-8B | 84.6% | 3.6% | 87.7% | 26.4% |
| Visual Genome | Aya-Vision-8B | 88.1% | 9.1% | 85.6% | 26.8% |
| Visual Genome | Gemma-4-12B | 90.4% | 21.4% | 76.8% | 27.1% |
| Visual Genome | Ministral-3-14B | 83.9% | 4.2% | 86.6% | 26.6% |

The [results CSV](docs/model-results.csv) is the source for these values.
24 changes: 17 additions & 7 deletions docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,24 +49,34 @@ Mirage turns dataset examples into model responses, scores those responses, and

## Reference model results

These reference results compare three model configurations across COCO and Visual Genome. Values are percentages from 1,000 images per model and dataset. Higher is better for POPE accuracy, ROPE strict accuracy, and THRONE F1. Lower is better for CHAIR-I hallucination rate. The tables use Mintlify's native tabs and tables so the values remain readable on every screen size.
These reference results compare eight model configurations across COCO and Visual Genome. Values are percentages from 1,000 images per model and dataset (500 for Gemma 4, Ministral 3 and Aya Vision); THRONE F1 uses a FLAN-T5-XL evaluator. Higher is better for POPE accuracy, ROPE strict accuracy, and THRONE F1. Lower is better for CHAIR-I hallucination rate. The tables use Mintlify's native tabs and tables so the values remain readable on every screen size.

<Tabs>
<Tab title="COCO">

| Model | POPE accuracy ↑ | ROPE strict accuracy ↑ | CHAIR-I hallucination rate ↓ | THRONE F1 ↑ |
| --- | ---: | ---: | ---: | ---: |
| PaliGemma-3B | 94.2% | 21.6% | 9.1% | 72.9% |
| Qwen2-VL-2B | 93.6% | 3.6% | 12.8% | 73.4% |
| Qwen2-VL-7B | 93.5% | 13.1% | 12.7% | 72.9% |
| PaliGemma-3B | 94.2% | 21.6% | 9.1% | 80.9% |
| Qwen2-VL-2B | 93.6% | 3.7% | 12.8% | 83.8% |
| Qwen2-VL-7B | 93.5% | 13.8% | 12.7% | 84.4% |
| Gemma-4-E4B | 93.5% | 30.9% | 12.5% | 77.0% |
| Ministral-3-8B | 88.1% | 11.0% | 13.7% | 80.8% |
| Aya-Vision-8B | 95.5% | 25.0% | 16.9% | 82.2% |
| Gemma-4-12B | 93.0% | 44.5% | 15.2% | 78.4% |
| Ministral-3-14B | 88.2% | 12.7% | 14.1% | 81.6% |
</Tab>
<Tab title="Visual Genome">

| Model | POPE accuracy ↑ | ROPE strict accuracy ↑ | CHAIR-I hallucination rate ↓ | THRONE F1 ↑ |
| --- | ---: | ---: | ---: | ---: |
| PaliGemma-3B | 88.0% | 8.2% | 67.8% | 21.5% |
| Qwen2-VL-2B | 88.7% | 8.6% | 75.6% | 24.0% |
| Qwen2-VL-7B | 89.5% | 10.3% | 69.3% | 23.6% |
| PaliGemma-3B | 88.0% | 8.2% | 67.8% | 27.6% |
| Qwen2-VL-2B | 88.7% | 8.6% | 75.6% | 27.4% |
| Qwen2-VL-7B | 89.5% | 11.1% | 69.3% | 27.2% |
| Gemma-4-E4B | 89.6% | 9.5% | 84.1% | 25.5% |
| Ministral-3-8B | 84.6% | 3.6% | 87.7% | 26.4% |
| Aya-Vision-8B | 88.1% | 9.1% | 85.6% | 26.8% |
| Gemma-4-12B | 90.4% | 21.4% | 76.8% | 27.1% |
| Ministral-3-14B | 83.9% | 4.2% | 86.6% | 26.6% |
</Tab>
</Tabs>

Expand Down
22 changes: 16 additions & 6 deletions docs/model-results.csv
Original file line number Diff line number Diff line change
@@ -1,7 +1,17 @@
model,dataset,pope_accuracy,rope_strict_accuracy,chair_instance_hallucination_rate,throne_f1
PaliGemma-3B,COCO,0.942,0.216,0.091,0.729
PaliGemma-3B,Visual Genome,0.880,0.082,0.678,0.215
Qwen2-VL-2B,COCO,0.936,0.036,0.128,0.734
Qwen2-VL-2B,Visual Genome,0.887,0.086,0.756,0.240
Qwen2-VL-7B,COCO,0.935,0.131,0.127,0.729
Qwen2-VL-7B,Visual Genome,0.895,0.103,0.693,0.236
PaliGemma-3B,COCO,0.942,0.216,0.091,0.809
PaliGemma-3B,Visual Genome,0.880,0.082,0.678,0.276
Qwen2-VL-2B,COCO,0.936,0.037,0.128,0.838
Qwen2-VL-2B,Visual Genome,0.887,0.086,0.756,0.274
Qwen2-VL-7B,COCO,0.935,0.138,0.127,0.844
Qwen2-VL-7B,Visual Genome,0.895,0.111,0.693,0.272
Gemma-4-E4B,COCO,0.935,0.309,0.125,0.770
Gemma-4-E4B,Visual Genome,0.896,0.095,0.841,0.255
Ministral-3-8B,COCO,0.881,0.110,0.137,0.808
Ministral-3-8B,Visual Genome,0.846,0.036,0.877,0.264
Aya-Vision-8B,COCO,0.955,0.250,0.169,0.822
Aya-Vision-8B,Visual Genome,0.881,0.091,0.856,0.268
Gemma-4-12B,COCO,0.930,0.445,0.152,0.784
Gemma-4-12B,Visual Genome,0.904,0.214,0.768,0.271
Ministral-3-14B,COCO,0.882,0.127,0.141,0.816
Ministral-3-14B,Visual Genome,0.839,0.042,0.866,0.266
27 changes: 0 additions & 27 deletions matching/__init__.py

This file was deleted.

47 changes: 0 additions & 47 deletions matching/base.py

This file was deleted.

57 changes: 0 additions & 57 deletions matching/cache.py

This file was deleted.

39 changes: 0 additions & 39 deletions matching/exact.py

This file was deleted.

100 changes: 0 additions & 100 deletions matching/judge.py

This file was deleted.

Loading
Loading