Skip to content

Docs rewrite, judge package and final results - #7

Merged
SalimAlkharsa merged 145 commits into
mainfrom
docs/rewrite
Oct 19, 2026
Merged

SalimAlkharsa merged 145 commits into
mainfrom
docs/rewrite

Conversation

@SalimAlkharsa

Copy link
Copy Markdown
Collaborator

Rewrites the docs and brings the refactor, the frozen judge verdicts and the final results to main.

Code

  • judge/ package: frozen, sha256-checked verdict store, with optional live grading (run.py --live-judge).
  • run.py is the single CLI. Every result carries a config block and a judged block, or lists missing verdicts.
  • repro.py --check rebuilds every published value from the records.
  • Removed matching/, NOPE/FaithScore and the table-check scripts. slurm/eval_cell.slurm is a fill-in-the-blank template.

Judge

  • The POPE "no"-question instructions were re-tuned: vague Visual Genome names are read generously, and "unsure" is kept for objects that truly can't be checked. The drop list, the Mod-POPE values in judgments/final_tables.json, the manifest hashes and the visual_genome-pope wiring cell are updated.
  • Human audit of all three judges (caption mentions, ROPE answers, POPE "no" questions) is on the Judge page.

Docs

  • New pages: overview, quickstart, models, prediction, metrics (POPE, ROPE, CHAIR, THRONE, and why they were modified) and judge.
  • The Gemma 4 thinking ablation on COCO POPE is on the Models page.

License

  • PolyForm Noncommercial 1.0.0, CITATION.cff, and attribution for the COCO/Visual Genome test images.

Checks

  • python repro.py --records tests/data/records.json.gz --check: 612 reference values, 612 rebuilt, 0 differ.
  • pytest: 7 passed, 1 skipped. The skip is the THRONE wiring cell, which needs the google/flan-t5-xl tokenizer in the local HF cache.
  • Docs tables match judgments/final_tables.json: 162 values and their ranks.

Fix the candidate pool at 2000 images (3000 for COCO ROPE) so the VG
vocabulary and image prefix do not depend on --samples, skip images
without annotations, stop POPE at --samples, cap captions at 512 tokens,
use the VG caption prompt and ROPE class list, and filter non-object
strings from the VG CHAIR vocabulary.

Add --throne-evaluator, --throne-distractor-ratio and --cache-only,
report prediction-cache stats in the result JSON, and add
slurm/eval_cell.slurm for one-cell jobs on a SLURM cluster.
An apostrophe inside the ${MIRAGE_CACHE_DIR:?...} message opened a
single quote, so bash rejected the whole job script.
An unquoted colon in a step name made the workflow file invalid, so
every run failed before starting. The check scripts live in scripts/,
not docs/scripts/, relative to the docs working directory.
…pper

models/hf_chat_vlm.py loads any processor/auto-class pair whose chat
template takes one image and one text turn, decodes only new tokens,
strips reasoning blocks and keeps thinking off by default. Register
gemma4-e4b, gemma4-12b, ministral3-8b, ministral3-14b and aya-vision-8b
with MIRAGE_<KEY>_MODEL_PATH overrides.
gemma4-e4b-think and gemma4-12b-think load the same checkpoints through
HFChatThinkingVLM, a separate class so cache keys never mix modes. They
add a 1,024-token thinking budget on top of each metric's max_new_tokens
and cut the thought at the end-of-thinking token before scoring.
…th FLAN-T5-XL

Add COCO and Visual Genome rows for gemma4-e4b, gemma4-12b,
ministral3-8b, ministral3-14b and aya-vision-8b (500 images each).
THRONE F1 for every row now uses a FLAN-T5-XL evaluator with 38
distractors per ground-truth object; the PaliGemma and Qwen2-VL values
are rescored from their cached 1,000-image captions and supersede the
FLAN-T5-base values.
Strip markdown emphasis and code marks, a leading header or list marker,
and surrounding quotes, and read the answer after "appears to be",
"is pointing to" and "is the" as well as "is a/an". A parenthetical aside
ends the answer. Matching stays exact: no synonyms and no picking one of
several alternatives. Tests use real cached answers from every model.
All 16 rows were rescored from their cached answers (0 cache misses).
Gemma-4-E4B rises from 0.0% to 30.9% (COCO) and 9.5% (VG); Ministral 3
gains 3-10 points; Qwen2-VL-7B gains 0.8 points on each dataset from
"appears to be" answers. PaliGemma-3B is unchanged. The homepage and
README notes now describe the extractor instead of a per-model caveat.
Muse Glimmer has no thinking-off switch, so MuseGlimmerVLM prefills the
template's answer channel (" to=user<|message|>") with reasoning strength
low, pins the template's current date, splits the bf16 weights across
both GPUs (device_map="auto"), and resizes images with PIL LANCZOS to the
processor's target size, since the processor falls back to BICUBIC with
torchvision < 0.27. The generic wrapper gains a device_map option, a
_build_inputs hook, and per-GPU peak memory logging.
Drives generate() with a fake processor and model and checks the exact
messages and apply_chat_template kwargs, thinking off and on. The same
assertions pass against the wrapper before the hook was added. Also
re-wraps the module docstring.
Captions longer than the 512-token window minus the question prompt were
trimmed from the end, so the evaluator never saw objects named late in
long captions. --throne-evaluator-max-tokens (or
MIRAGE_THRONE_EVALUATOR_MAX_TOKENS) sets the window; the default stays 512
so earlier results reproduce from their caches. The window is part of the
evaluator cache key and is reported in the results JSON.
… rows

Re-judged COCO for all rows and Visual Genome for PaliGemma-3B and
Gemma-4-12B with --throne-evaluator-max-tokens 1024, so no caption is
trimmed. Other Visual Genome rows had no trimmed captions at 512 tokens;
one Qwen2-VL-7B Visual Genome caption is footnoted.
…NE F0.5

Scores cached model outputs against static judge verdicts in judgments/
(never calls a judge; missing verdicts raise). Reproduces the reference
table in tests/data/final_tables.json exactly from the results tree.
The judge ships as tools/judge (driver and the three skills). Homepage
and README tables become a headline-only placeholder; docs/methodology.mdx
documents each modification with a raw vs Mod appendix.
demo.py keeps every flag, default, prompt and generation call; only the
code is condensed (no behavior change, verified by cache replay). Docs
pages and tests/test_demo_smoke.py return to main, apart from the
cache_stats key the demo now reports.
Caption prompts, CAPTION_GEN and SAMPLES move to metrics/base_metric.py,
VG_NON_OBJECTS and the VG CHAIR vocabulary to metrics/chair.py, and the
pool sizes and POPE/ROPE generation kwargs to the metric classes.
run.py imports them; values and prompt bytes are unchanged.
…inux clusters

The account, partition, GPU type and module lines are placeholders to fill in for
your cluster; the run.py invocation and the MIRAGE_* interface are unchanged.
# Conflicts:
#	slurm/eval_cell.slurm
# Conflicts:
#	run.py
# Conflicts:
#	run.py
# Conflicts:
#	run.py
… results tree

load_records reads the model replies off the tree (the step that needs images); rebuild scores
them and builds committee.json. --check gives the same 612 values either way.
…the cache

test_grading: the frozen replies rescored must give all 612 values of final_tables.json and
committee.json. test_wiring: the first COCO and Visual Genome images through run.py --cache-only,
with gemma4-e4b's published rows; every request must hit and the results match expected.json.
tests/data/freeze.py rebuilds the fixtures from a results tree. The smoke test is folded in.
…g; loaders name their dataset

Loaders iterate in file order (as every caller already did), carry KEY (coco / vg) for dataset_key, and record how many images were skipped before the pool filled. One helper resolves Visual Genome object names. No evaluated image changes.
…pty denominators; THRONE settings come from flags only

THRONE no longer reads the distractor ratio, window or dtype from the environment (the evaluator path still can). No published value changes.
… identity_for and half-wired quantization

A miss under --cache-only names the model revision and the cache file. Unpinned model paths log a warning; an empty model-path variable counts as unset. PaliGemma and Qwen2-VL raise like the other wrappers. Cache identities are unchanged for every registry key.
… and checks the rubric against the manifest

Absolute image paths, a clear error on a torn live shard, workdirs created and resumable, the judge model read from judge/manifest.json, and Judge.load can read a live shard without the claude binary.
… cover it, and a config block in every result

--live-judge grades a new model's missing caption mentions (CHAIR, Visual Genome THRONE). Visual Genome THRONE now reports missing_verdicts instead of scoring unjudged mentions as absent, and POPE is judged only at the published 6 questions per image. Results record the model identity and settings; tests/data/wiring/expected.json gains only that block. The docs-coverage and old results-table checks are removed ahead of the docs rewrite.
Deletes the README files, the Mintlify pages and the results CSV; docs/docs.json keeps the
site settings with an empty navigation.
New docs pages for the overview, quickstart, models, prediction, metrics and
judge, with ranking charts and the judge/human agreement table.

The POPE judge instructions now read vague object names generously and keep
"unsure" for objects that truly can't be checked. The drop list, the Mod-POPE
entries of final_tables.json, the manifest hashes and the visual_genome-pope
wiring cell are updated to match.
@mintlify

mintlify Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
mirage-f432d87b 🟢 Ready View Preview Oct 19, 2026, 4:06 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@SalimAlkharsa
SalimAlkharsa merged commit ba93738 into main Oct 19, 2026
2 checks passed
@SalimAlkharsa
SalimAlkharsa deleted the docs/rewrite branch October 19, 2026 04:27

This branch was successfully deployed

1 active deployment
staging - docs — 21d6f1aa Deployed Oct 19, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant