Skip to content

Match the reference evaluation protocol in demo.py - #1

Closed
SalimAlkharsa wants to merge 21 commits into
mainfrom
eval/protocol-parity
Closed

SalimAlkharsa wants to merge 21 commits into
mainfrom
eval/protocol-parity

Conversation

@SalimAlkharsa

@SalimAlkharsa SalimAlkharsa commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Brings demo.py in line with the protocol behind the homepage numbers (dev repo unit_test/* drivers), so new models can be run and old caches replayed through the prod code.

  • Image selection: fixed candidate pool (--pool-size default 2000, 3000 for COCO ROPE) instead of a multiple of --samples. Images without annotations are skipped, and POPE now stops at --samples (before this it evaluated the whole pool).

  • Captions: --max-new-tokens default 512, plus the VG caption prompt (VG_CAPTION_PROMPT).

  • VG CHAIR: stopword/colour/adjective filter on the object vocabulary.

  • VG ROPE: prompts list the classes present in the pool's ROPE boxes.

  • VG THRONE: categories come from the loaded pool (4,330 names at pool 2000), not all of objects.json.

  • New flags: --throne-evaluator and --throne-distractor-ratio. --cache-only never loads the model and fails on the first cache miss.

  • The result JSON includes cache_stats (hits/misses).

  • slurm/eval_cell.slurm runs one model × dataset × metric cell on TAMU HPRC Grace (A100, offline HF, per-cell cache dir).

  • demo.py imports the loaders from their submodules. This fixes a pre-existing test-order failure: tests/test_rope_sampling.py stubs the dataloader package, which broke test_demo_smoke.py in a full pytest run.

  • Docs CI repair: .github/workflows/docs.yml never ran on main. An unquoted colon in a step name made the file invalid, and the check scripts were referenced relative to docs/. Both are fixed, and the docs-coverage, results-table and broken-links checks now run and pass.

  • slurm/eval_cell.slurm: fixed a quoting bug (an apostrophe in a ${VAR:?message}) that made bash reject the script.

Validation

  • pytest -q (Grace login node, tf 5.19 venv): 23 passed.
  • Zero-GPU parity replay: the dev caches for PaliGemma, Qwen2-VL-2B and Qwen2-VL-7B ran through this demo.py --cache-only at 1,000 images, with 0 misses on every cell. Reproduced values:
Model Dataset POPE ROPE strict CHAIR-I CSV
PaliGemma-3B COCO 0.9416 0.2164 0.0911 0.942 / 0.216 / 0.091
PaliGemma-3B VG 0.8800 0.0824 0.6784 0.880 / 0.082 / 0.678
Qwen2-VL-2B COCO 0.9363 0.0364 0.1275 0.936 / 0.036 / 0.128
Qwen2-VL-2B VG 0.8875 0.0856 0.7563 0.887 / 0.086 / 0.756
Qwen2-VL-7B COCO 0.9350 0.1308 0.1273 0.935 / 0.131 / 0.127
Qwen2-VL-7B VG 0.8949 0.1034 0.6928 0.895 / 0.103 / 0.693

All 18 cells: 0 cache misses (CPU-only job on Grace, --cache-only, model paths set to the dev HF snapshot paths so cache keys pin the revision).

Fix the candidate pool at 2000 images (3000 for COCO ROPE) so the VG
vocabulary and image prefix do not depend on --samples, skip images
without annotations, stop POPE at --samples, cap captions at 512 tokens,
use the VG caption prompt and ROPE class list, and filter non-object
strings from the VG CHAIR vocabulary.

Add --throne-evaluator, --throne-distractor-ratio and --cache-only,
report prediction-cache stats in the result JSON, and add
slurm/eval_cell.slurm for one-cell jobs on a SLURM cluster.
@mintlify

mintlify Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
mirage-f432d87b 🟢 Ready View Preview Oct 17, 2026, 8:47 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

An apostrophe inside the ${MIRAGE_CACHE_DIR:?...} message opened a
single quote, so bash rejected the whole job script.
An unquoted colon in a step name made the workflow file invalid, so
every run failed before starting. The check scripts live in scripts/,
not docs/scripts/, relative to the docs working directory.
Comment thread .github/workflows/docs.yml Resolved
Comment thread docs/hpc.mdx Outdated

For custom dataset paths, extra model arguments, partitions, account codes, or module commands, edit the script's `python demo.py` command and `#SBATCH` block. Then submit it from the Mirage repository root with `sbatch slurm/demo.slurm`.

## TAMU HPRC Grace cells

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont make doc changes in general about non-operational / use stuff I want to rework those because we made a lot of changes

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted. This PR no longer touches any .mdx page: hpc, maintaining-docs, metrics-and-models and running-evaluations are back to main. Across the stack, the only docs changes are the numeric tables, docs/index.mdx and docs/model-results.csv, in #3 and #5.

Comment thread demo.py Outdated
model.unload_model()


def _vg_chair_vocabulary(all_objects: List[str]) -> List[str]:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we still need this with the judge changes having been made?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still needed, but it's no longer in demo.py. VG CHAIR needs a vocabulary: VG object names minus stopwords and VG_NON_OBJECTS. Raw CHAIR uses it, and so does mention tagging before the judge. It now lives in the metric that owns it: VG_NON_OBJECTS and vg_vocabulary() in metrics/chair.py. No code imports from demo.py anymore.

Comment thread demo.py Resolved
demo.py keeps every flag, default, prompt and generation call; only the
code is condensed (no behavior change, verified by cache replay). Docs
pages and tests/test_demo_smoke.py return to main, apart from the
cache_stats key the demo now reports.
Caption prompts, CAPTION_GEN and SAMPLES move to metrics/base_metric.py,
VG_NON_OBJECTS and the VG CHAIR vocabulary to metrics/chair.py, and the
pool sizes and POPE/ROPE generation kwargs to the metric classes.
run.py imports them; values and prompt bytes are unchanged.
A replay never writes to the cache (no new rows, no index creation on
open), and a miss still raises because the model is never loaded.
Caption prompts, CAPTION_GEN, POOL and the POPE/ROPE GEN settings are
MappingProxyType, so an in-place change (which would silently change
cache keys) raises. The two caption prompts live only in CAPTION_PROMPTS.
pope.QUESTIONS_PER_IMAGE is the one default for POPEMetric and
--pope-questions. run.py maps --dataset to the metric key with DS and
imports metrics.rope only for ROPE.
POPEMetric.run, ROPEMetric.run and BaseMetric.run (captions, used by
CHAIR and THRONE) hold the loops run.py._evaluate had, moved as they
were: the ground-truth filter before the slice, the seeded yes-picks,
the COCO vs VG ROPE class list, and the VG THRONE pool union
(pool_objects). run.py builds each metric with its CLI settings and
calls metric.run(loader, model, n); FaithScore reuses caption_images.
Model calls and format_results() are unchanged (checked against the
old demo.py with a recording fake model on COCO and VG).
The example now calls pope.run / rope.run / chair.run, the same protocol
run.py uses, instead of a hand-written copy of the loops. It takes no
flags (old 'demo.py --dataset ...' commands would otherwise run the
example and write to the cache); run.py is the CLI.
…s generation settings

NOPE opened a writable cache and loaded the model even under --cache-only.
It now opens the cache read-only and never loads weights (a miss raises), like
the dataset metrics. FaithScore always loads its verifier and Llama, so
--cache-only is refused for it. NOPEMetric.GEN holds the 20-token greedy
settings; the cache key is unchanged.
metrics/chair.py downloaded punkt_tab on import when it was missing, writing to
the first writable NLTK data directory. It now raises LookupError instead.
Comment thread slurm/eval_cell.slurm

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis whole file needs to be a demo/fill in the blank file because this here is exposing my info also this will be a public repo so handle rewriting the git history on this file so it doesnt show the info in the history

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done for the current branches (9430964). slurm/eval_cell.slurm is now a fill-in-the-blank template: --account=<your-account>, --partition=<gpu-partition>, --gres=gpu:1, and commented # module load <python> <cuda> lines. The run.py call and the MIRAGE_* variables are unchanged. The "(HPRC)" comment in metrics/rope.py now says "Linux clusters". No branch tip contains the account number or TAMU/HPRC/Grace anywhere.

History is not rewritten yet, so older commits still have the account number. A dry run is ready: the tip trees are byte-identical and the account disappears from every rewritten ref. It needs your go for a force-push of the six eval/* branches and the stable tags. Open choice: "(HPRC)" has been in main's rope.py since init, so a fully clean history means rewriting main too, and that deploys the site. Also, GitHub keeps the old commits reachable from the PR timelines, so for a clean public release, push a fresh repo or ask GitHub Support to purge.

…inux clusters

The account, partition, GPU type and module lines are placeholders to fill in for
your cluster; the run.py invocation and the MIRAGE_* interface are unchanged.

This branch was successfully deployed

1 active deployment
staging - docs — 61c5701f Deployed Oct 17, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant