Repository navigation
Docs rewrite, judge package and final results - #7
Merged
Merged
Conversation
Fix the candidate pool at 2000 images (3000 for COCO ROPE) so the VG vocabulary and image prefix do not depend on --samples, skip images without annotations, stop POPE at --samples, cap captions at 512 tokens, use the VG caption prompt and ROPE class list, and filter non-object strings from the VG CHAIR vocabulary. Add --throne-evaluator, --throne-distractor-ratio and --cache-only, report prediction-cache stats in the result JSON, and add slurm/eval_cell.slurm for one-cell jobs on a SLURM cluster.
An apostrophe inside the ${MIRAGE_CACHE_DIR:?...} message opened a
single quote, so bash rejected the whole job script.
An unquoted colon in a step name made the workflow file invalid, so every run failed before starting. The check scripts live in scripts/, not docs/scripts/, relative to the docs working directory.
…pper models/hf_chat_vlm.py loads any processor/auto-class pair whose chat template takes one image and one text turn, decodes only new tokens, strips reasoning blocks and keeps thinking off by default. Register gemma4-e4b, gemma4-12b, ministral3-8b, ministral3-14b and aya-vision-8b with MIRAGE_<KEY>_MODEL_PATH overrides.
gemma4-e4b-think and gemma4-12b-think load the same checkpoints through HFChatThinkingVLM, a separate class so cache keys never mix modes. They add a 1,024-token thinking budget on top of each metric's max_new_tokens and cut the thought at the end-of-thinking token before scoring.
…th FLAN-T5-XL Add COCO and Visual Genome rows for gemma4-e4b, gemma4-12b, ministral3-8b, ministral3-14b and aya-vision-8b (500 images each). THRONE F1 for every row now uses a FLAN-T5-XL evaluator with 38 distractors per ground-truth object; the PaliGemma and Qwen2-VL values are rescored from their cached 1,000-image captions and supersede the FLAN-T5-base values.
Strip markdown emphasis and code marks, a leading header or list marker, and surrounding quotes, and read the answer after "appears to be", "is pointing to" and "is the" as well as "is a/an". A parenthetical aside ends the answer. Matching stays exact: no synonyms and no picking one of several alternatives. Tests use real cached answers from every model.
All 16 rows were rescored from their cached answers (0 cache misses). Gemma-4-E4B rises from 0.0% to 30.9% (COCO) and 9.5% (VG); Ministral 3 gains 3-10 points; Qwen2-VL-7B gains 0.8 points on each dataset from "appears to be" answers. PaliGemma-3B is unchanged. The homepage and README notes now describe the extractor instead of a per-model caveat.
Muse Glimmer has no thinking-off switch, so MuseGlimmerVLM prefills the
template's answer channel (" to=user<|message|>") with reasoning strength
low, pins the template's current date, splits the bf16 weights across
both GPUs (device_map="auto"), and resizes images with PIL LANCZOS to the
processor's target size, since the processor falls back to BICUBIC with
torchvision < 0.27. The generic wrapper gains a device_map option, a
_build_inputs hook, and per-GPU peak memory logging.
Drives generate() with a fake processor and model and checks the exact messages and apply_chat_template kwargs, thinking off and on. The same assertions pass against the wrapper before the hook was added. Also re-wraps the module docstring.
Captions longer than the 512-token window minus the question prompt were trimmed from the end, so the evaluator never saw objects named late in long captions. --throne-evaluator-max-tokens (or MIRAGE_THRONE_EVALUATOR_MAX_TOKENS) sets the window; the default stays 512 so earlier results reproduce from their caches. The window is part of the evaluator cache key and is reported in the results JSON.
… rows Re-judged COCO for all rows and Visual Genome for PaliGemma-3B and Gemma-4-12B with --throne-evaluator-max-tokens 1024, so no caption is trimmed. Other Visual Genome rows had no trimmed captions at 512 tokens; one Qwen2-VL-7B Visual Genome caption is footnoted.
…NE F0.5 Scores cached model outputs against static judge verdicts in judgments/ (never calls a judge; missing verdicts raise). Reproduces the reference table in tests/data/final_tables.json exactly from the results tree. The judge ships as tools/judge (driver and the three skills). Homepage and README tables become a headline-only placeholder; docs/methodology.mdx documents each modification with a raw vs Mod appendix.
demo.py keeps every flag, default, prompt and generation call; only the code is condensed (no behavior change, verified by cache replay). Docs pages and tests/test_demo_smoke.py return to main, apart from the cache_stats key the demo now reports.
…te, drop the new test
…note, drop the new test
…sts and the judge README
Caption prompts, CAPTION_GEN and SAMPLES move to metrics/base_metric.py, VG_NON_OBJECTS and the VG CHAIR vocabulary to metrics/chair.py, and the pool sizes and POPE/ROPE generation kwargs to the metric classes. run.py imports them; values and prompt bytes are unchanged.
…inux clusters The account, partition, GPU type and module lines are placeholders to fill in for your cluster; the run.py invocation and the MIRAGE_* interface are unchanged.
# Conflicts: # slurm/eval_cell.slurm
…ks the judged block
# Conflicts: # run.py
# Conflicts: # run.py
# Conflicts: # run.py
… results tree load_records reads the model replies off the tree (the step that needs images); rebuild scores them and builds committee.json. --check gives the same 612 values either way.
…the cache test_grading: the frozen replies rescored must give all 612 values of final_tables.json and committee.json. test_wiring: the first COCO and Visual Genome images through run.py --cache-only, with gemma4-e4b's published rows; every request must hit and the results match expected.json. tests/data/freeze.py rebuilds the fixtures from a results tree. The smoke test is folded in.
…g; loaders name their dataset Loaders iterate in file order (as every caller already did), carry KEY (coco / vg) for dataset_key, and record how many images were skipped before the pool filled. One helper resolves Visual Genome object names. No evaluated image changes.
…pty denominators; THRONE settings come from flags only THRONE no longer reads the distractor ratio, window or dtype from the environment (the evaluator path still can). No published value changes.
… identity_for and half-wired quantization A miss under --cache-only names the model revision and the cache file. Unpinned model paths log a warning; an empty model-path variable counts as unset. PaliGemma and Qwen2-VL raise like the other wrappers. Cache identities are unchanged for every registry key.
… and checks the rubric against the manifest Absolute image paths, a clear error on a torn live shard, workdirs created and resumable, the judge model read from judge/manifest.json, and Judge.load can read a live shard without the claude binary.
… cover it, and a config block in every result --live-judge grades a new model's missing caption mentions (CHAIR, Visual Genome THRONE). Visual Genome THRONE now reports missing_verdicts instead of scoring unjudged mentions as absent, and POPE is judged only at the published 6 questions per image. Results record the model identity and settings; tests/data/wiring/expected.json gains only that block. The docs-coverage and old results-table checks are removed ahead of the docs rewrite.
Deletes the README files, the Mintlify pages and the results CSV; docs/docs.json keeps the site settings with an empty navigation.
New docs pages for the overview, quickstart, models, prediction, metrics and judge, with ranking charts and the judge/human agreement table. The POPE judge instructions now read vague object names generously and keep "unsure" for objects that truly can't be checked. The drop list, the Mod-POPE entries of final_tables.json, the manifest hashes and the visual_genome-pope wiring cell are updated to match.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites the docs and brings the refactor, the frozen judge verdicts and the final results to main.
Code
judge/package: frozen, sha256-checked verdict store, with optional live grading (run.py --live-judge).run.pyis the single CLI. Every result carries a config block and a judged block, or lists missing verdicts.repro.py --checkrebuilds every published value from the records.matching/, NOPE/FaithScore and the table-check scripts.slurm/eval_cell.slurmis a fill-in-the-blank template.Judge
judgments/final_tables.json, the manifest hashes and the visual_genome-pope wiring cell are updated.Docs
License
CITATION.cff, and attribution for the COCO/Visual Genome test images.Checks
python repro.py --records tests/data/records.json.gz --check: 612 reference values, 612 rebuilt, 0 differ.pytest: 7 passed, 1 skipped. The skip is the THRONE wiring cell, which needs the google/flan-t5-xl tokenizer in the local HF cache.judgments/final_tables.json: 162 values and their ranks.