Repository navigation
Match the reference evaluation protocol in demo.py - #1
SalimAlkharsa wants to merge 21 commits into
Conversation
Fix the candidate pool at 2000 images (3000 for COCO ROPE) so the VG vocabulary and image prefix do not depend on --samples, skip images without annotations, stop POPE at --samples, cap captions at 512 tokens, use the VG caption prompt and ROPE class list, and filter non-object strings from the VG CHAIR vocabulary. Add --throne-evaluator, --throne-distractor-ratio and --cache-only, report prediction-cache stats in the result JSON, and add slurm/eval_cell.slurm for one-cell jobs on a SLURM cluster.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
An apostrophe inside the ${MIRAGE_CACHE_DIR:?...} message opened a
single quote, so bash rejected the whole job script.
An unquoted colon in a step name made the workflow file invalid, so every run failed before starting. The check scripts live in scripts/, not docs/scripts/, relative to the docs working directory.
|
|
||
| For custom dataset paths, extra model arguments, partitions, account codes, or module commands, edit the script's `python demo.py` command and `#SBATCH` block. Then submit it from the Mirage repository root with `sbatch slurm/demo.slurm`. | ||
|
|
||
| ## TAMU HPRC Grace cells |
There was a problem hiding this comment.
Dont make doc changes in general about non-operational / use stuff I want to rework those because we made a lot of changes
| model.unload_model() | ||
|
|
||
|
|
||
| def _vg_chair_vocabulary(all_objects: List[str]) -> List[str]: |
There was a problem hiding this comment.
do we still need this with the judge changes having been made?
There was a problem hiding this comment.
Still needed, but it's no longer in demo.py. VG CHAIR needs a vocabulary: VG object names minus stopwords and VG_NON_OBJECTS. Raw CHAIR uses it, and so does mention tagging before the judge. It now lives in the metric that owns it: VG_NON_OBJECTS and vg_vocabulary() in metrics/chair.py. No code imports from demo.py anymore.
demo.py keeps every flag, default, prompt and generation call; only the code is condensed (no behavior change, verified by cache replay). Docs pages and tests/test_demo_smoke.py return to main, apart from the cache_stats key the demo now reports.
Caption prompts, CAPTION_GEN and SAMPLES move to metrics/base_metric.py, VG_NON_OBJECTS and the VG CHAIR vocabulary to metrics/chair.py, and the pool sizes and POPE/ROPE generation kwargs to the metric classes. run.py imports them; values and prompt bytes are unchanged.
A replay never writes to the cache (no new rows, no index creation on open), and a miss still raises because the model is never loaded.
Caption prompts, CAPTION_GEN, POOL and the POPE/ROPE GEN settings are MappingProxyType, so an in-place change (which would silently change cache keys) raises. The two caption prompts live only in CAPTION_PROMPTS. pope.QUESTIONS_PER_IMAGE is the one default for POPEMetric and --pope-questions. run.py maps --dataset to the metric key with DS and imports metrics.rope only for ROPE.
POPEMetric.run, ROPEMetric.run and BaseMetric.run (captions, used by CHAIR and THRONE) hold the loops run.py._evaluate had, moved as they were: the ground-truth filter before the slice, the seeded yes-picks, the COCO vs VG ROPE class list, and the VG THRONE pool union (pool_objects). run.py builds each metric with its CLI settings and calls metric.run(loader, model, n); FaithScore reuses caption_images. Model calls and format_results() are unchanged (checked against the old demo.py with a recording fake model on COCO and VG).
The example now calls pope.run / rope.run / chair.run, the same protocol run.py uses, instead of a hand-written copy of the loops. It takes no flags (old 'demo.py --dataset ...' commands would otherwise run the example and write to the cache); run.py is the CLI.
…s generation settings NOPE opened a writable cache and loaded the model even under --cache-only. It now opens the cache read-only and never loads weights (a miss raises), like the dataset metrics. FaithScore always loads its verifier and Llama, so --cache-only is refused for it. NOPEMetric.GEN holds the 20-token greedy settings; the cache key is unchanged.
metrics/chair.py downloaded punkt_tab on import when it was missing, writing to the first writable NLTK data directory. It now raises LookupError instead.
There was a problem hiding this comment.
THis whole file needs to be a demo/fill in the blank file because this here is exposing my info also this will be a public repo so handle rewriting the git history on this file so it doesnt show the info in the history
There was a problem hiding this comment.
Done for the current branches (9430964). slurm/eval_cell.slurm is now a fill-in-the-blank template: --account=<your-account>, --partition=<gpu-partition>, --gres=gpu:1, and commented # module load <python> <cuda> lines. The run.py call and the MIRAGE_* variables are unchanged. The "(HPRC)" comment in metrics/rope.py now says "Linux clusters". No branch tip contains the account number or TAMU/HPRC/Grace anywhere.
History is not rewritten yet, so older commits still have the account number. A dry run is ready: the tip trees are byte-identical and the account disappears from every rewritten ref. It needs your go for a force-push of the six eval/* branches and the stable tags. Open choice: "(HPRC)" has been in main's rope.py since init, so a fully clean history means rewriting main too, and that deploys the site. Also, GitHub keeps the old commits reachable from the PR timelines, so for a clean public release, push a fresh repo or ask GitHub Support to purge.
…inux clusters The account, partition, GPU type and module lines are placeholders to fill in for your cluster; the run.py invocation and the MIRAGE_* interface are unchanged.
c67045b to
61c5701
Compare
Summary
Brings
demo.pyin line with the protocol behind the homepage numbers (dev repounit_test/*drivers), so new models can be run and old caches replayed through the prod code.Image selection: fixed candidate pool (
--pool-sizedefault 2000, 3000 for COCO ROPE) instead of a multiple of--samples. Images without annotations are skipped, and POPE now stops at--samples(before this it evaluated the whole pool).Captions:
--max-new-tokensdefault 512, plus the VG caption prompt (VG_CAPTION_PROMPT).VG CHAIR: stopword/colour/adjective filter on the object vocabulary.
VG ROPE: prompts list the classes present in the pool's ROPE boxes.
VG THRONE: categories come from the loaded pool (4,330 names at pool 2000), not all of
objects.json.New flags:
--throne-evaluatorand--throne-distractor-ratio.--cache-onlynever loads the model and fails on the first cache miss.The result JSON includes
cache_stats(hits/misses).slurm/eval_cell.slurmruns one model × dataset × metric cell on TAMU HPRC Grace (A100, offline HF, per-cell cache dir).demo.pyimports the loaders from their submodules. This fixes a pre-existing test-order failure:tests/test_rope_sampling.pystubs thedataloaderpackage, which broketest_demo_smoke.pyin a fullpytestrun.Docs CI repair:
.github/workflows/docs.ymlnever ran onmain. An unquoted colon in a step name made the file invalid, and the check scripts were referenced relative todocs/. Both are fixed, and the docs-coverage, results-table and broken-links checks now run and pass.slurm/eval_cell.slurm: fixed a quoting bug (an apostrophe in a${VAR:?message}) that made bash reject the script.Validation
pytest -q(Grace login node, tf 5.19 venv): 23 passed.demo.py --cache-onlyat 1,000 images, with 0 misses on every cell. Reproduced values:All 18 cells: 0 cache misses (CPU-only job on Grace,
--cache-only, model paths set to the dev HF snapshot paths so cache keys pin the revision).