Skip to content

Add Muse Glimmer 30B - #4

Closed
SalimAlkharsa wants to merge 14 commits into
eval/results-v1from
eval/muse-glimmer
Closed

SalimAlkharsa wants to merge 14 commits into
eval/results-v1from
eval/muse-glimmer

Conversation

@SalimAlkharsa

@SalimAlkharsa SalimAlkharsa commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds Meta's Muse Glimmer 30B (meta-models/Muse-Glimmer-30B, Apache-2.0) as muse-glimmer-30b, evaluated with the same protocol as the other new rows: 500 images, the same subsets and caps, thinking off, and THRONE with FLAN-T5-XL at ratio 38. Stacked on #3.

  • Thinking: Muse Glimmer has no switch that turns thinking off. Its chat template only sets a reasoning strength (low/medium/high/xhigh, default high), and after <|start|>assistant the model picks a reasoning channel (to=self) or an answer channel (to=user). Here, "thinking off" means MuseGlimmerVLM prefills the answer channel to=user<|message|> (the same turn the template writes for an answer), with Reasoning strength: low kept in the system turn. The smoke compared this with "reasoning strength low" alone: the model still reasoned on every item, taking 5–11 s per POPE question against 0.35 s.
  • Pinned prompt date: the template writes the current date into the system turn. It is pinned to 2026-10-15 so prompts and cache keys don't change from day to day.
  • Weights: revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68 in bf16 (no quantization). About 60 GB, split over both A100 40 GB GPUs of a Grace node (device_map="auto"), with a peak of 29.7 / 30.0 GB per card.
  • Preprocessing: the image processor asks for LANCZOS but falls back to BICUBIC with torchvision < 0.27 (the venv has 0.24.1, and the venv is unchanged). The wrapper therefore resizes with PIL LANCZOS to the exact size the processor would choose (smart_resize), which makes the processor's own resize a same-size pass. Verified on 4 COCO images, 2 VG images and one over-cap 4000×3000 image: pixel_values are bit-identical to the do_resize=False path, and the patch grid matches the default path.
  • Generic wrapper: gains a device_map option, a _build_inputs hook, and per-GPU peak-memory logging. Nothing changes for the existing keys (see Validation).
  • Launch: eval_cell.slurm with --gres=gpu:a100:2 --cpus-per-task=8 --mem=96G (documented in docs/hpc.mdx).

Validation

  • CPU tests on Grace: 59 passed, including tests/test_muse_glimmer_inputs.py (answer prefill, template kwargs, default device map, LANCZOS target size) and tests/test_hf_chat_inputs.py, which pins the exact apply_chat_template messages and kwargs of the generic wrapper (thinking off and on). The same assertions also pass against the wrapper before the _build_inputs hook, so the existing keys build identical inputs.
  • Raw smoke (job 20160812): loads in 73 s; POPE 0.35 s/question, ROPE 3.0 s/box, captions 49 s each (they always reach the 512-token cap).
  • demo.py smoke (job 20161174, 5 images): POPE answers are clean Yes/No, and ROPE sentences parse. Captions contain no reasoning text. CHAIR returns numbers, and THRONE-XL reads all captions from cache (0 misses).

Results

Muse Glimmer 30B, 500 images per dataset (percent; ↑ better except CHAIR-I ↓). The CSV, homepage and README rows are added in #5, because its THRONE values use the 1,024-token evaluator window that #5 introduces.

Dataset POPE ↑ ROPE strict ↑ ROPE lenient ↑ CHAIR-I ↓ THRONE-XL F1 ↑
COCO 92.7 12.1 42.4 17.1 83.9
Visual Genome 88.6 6.3 29.9 81.7 27.9
  • POPE (jobs 20161351 / 20161352): COCO 92.69 (yes-accuracy 83.9, no-accuracy 99.4), VG 88.65 (82.1 / 95.1). The model rarely says "yes" to an absent object; most errors are missed objects.
  • ROPE (jobs 20161355 / 20161356): Glimmer answers with full descriptive phrases ("a person standing in the snow", "a portion of …"). Strict matching requires the exact category name, so strict accuracy is low; lenient matching scores the phrase match. No extractor change.
  • Captions / CHAIR: COCO 20161353 (5:36:54, 40.4 s/caption; CHAIR-I 17.08, CHAIR-S 71.6, recall 79.5). VG 20168115 (CHAIR-I 81.69, CHAIR-S 100, recall 32.9).
  • THRONE (1,024-token window, 0 trimmed captions, 0 caption-cache misses): COCO 20168944 (F1 83.94, P 82.2 / R 85.8), VG 20168951 (F1 27.90, P 17.1 / R 75.6). At the old 512-token window, COCO was 83.56 (job 20161354), with 435 of 500 captions trimmed.

Provenance

  • Code: eval/muse-glimmer @ 6c798cc (the jobs ran on cd323d1; 6c798cc only re-wraps a docstring and adds a test); model revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68, bf16, 2× A100 40 GB.
  • Results: mirage-prod/results/grid/muse-glimmer-30b_*.json on Grace; caches mirage-prod/cache/muse-glimmer-30b_<dataset>_<stage>/ (THRONE 1,024: ..._throne_w1024_evaluator).

Muse Glimmer has no thinking-off switch, so MuseGlimmerVLM prefills the
template's answer channel (" to=user<|message|>") with reasoning strength
low, pins the template's current date, splits the bf16 weights across
both GPUs (device_map="auto"), and resizes images with PIL LANCZOS to the
processor's target size, since the processor falls back to BICUBIC with
torchvision < 0.27. The generic wrapper gains a device_map option, a
_build_inputs hook, and per-GPU peak memory logging.
Drives generate() with a fake processor and model and checks the exact
messages and apply_chat_template kwargs, thinking off and on. The same
assertions pass against the wrapper before the hook was added. Also
re-wraps the module docstring.
Conflict resolutions:
- models/hf_chat.py: keeps _build_inputs, drops batch_generate.
  MuseGlimmerVLM stays in hf_chat.py and pins cache_name =
  "MuseGlimmerVLM" (it would otherwise inherit "HFChatVLM").
- predict/factory.py: the muse-glimmer-30b row moves into REGISTRY as
  models.hf_chat:MuseGlimmerVLM with the MuseGlimmerVLM mirror.
# Conflicts:
#	models/hf_chat.py
Class body unchanged (cache_name stays MuseGlimmerVLM); the registry row and
the lazy models exports point at the new module.
cache_name stays MuseGlimmerVLM, so cache keys do not change.
@SalimAlkharsa
SalimAlkharsa deleted the eval/muse-glimmer branch October 19, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant