Repository navigation
Add Muse Glimmer 30B - #4
Closed
SalimAlkharsa wants to merge 14 commits into
Closed
SalimAlkharsa wants to merge 14 commits into
SalimAlkharsa wants to merge 14 commits into
Conversation
Muse Glimmer has no thinking-off switch, so MuseGlimmerVLM prefills the
template's answer channel (" to=user<|message|>") with reasoning strength
low, pins the template's current date, splits the bf16 weights across
both GPUs (device_map="auto"), and resizes images with PIL LANCZOS to the
processor's target size, since the processor falls back to BICUBIC with
torchvision < 0.27. The generic wrapper gains a device_map option, a
_build_inputs hook, and per-GPU peak memory logging.
Drives generate() with a fake processor and model and checks the exact messages and apply_chat_template kwargs, thinking off and on. The same assertions pass against the wrapper before the hook was added. Also re-wraps the module docstring.
Conflict resolutions: - models/hf_chat.py: keeps _build_inputs, drops batch_generate. MuseGlimmerVLM stays in hf_chat.py and pins cache_name = "MuseGlimmerVLM" (it would otherwise inherit "HFChatVLM"). - predict/factory.py: the muse-glimmer-30b row moves into REGISTRY as models.hf_chat:MuseGlimmerVLM with the MuseGlimmerVLM mirror.
# Conflicts: # models/hf_chat.py
Class body unchanged (cache_name stays MuseGlimmerVLM); the registry row and the lazy models exports point at the new module.
cache_name stays MuseGlimmerVLM, so cache keys do not change.
SalimAlkharsa
force-pushed
the
eval/results-v1
branch
from
October 17, 2026 20:46
455fce3 to
c01a80a
Compare
SalimAlkharsa
force-pushed
the
eval/muse-glimmer
branch
from
October 17, 2026 20:46
f162349 to
0c1d68e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Meta's Muse Glimmer 30B (
meta-models/Muse-Glimmer-30B, Apache-2.0) asmuse-glimmer-30b, evaluated with the same protocol as the other new rows: 500 images, the same subsets and caps, thinking off, and THRONE with FLAN-T5-XL at ratio 38. Stacked on #3.<|start|>assistantthe model picks a reasoning channel (to=self) or an answer channel (to=user). Here, "thinking off" meansMuseGlimmerVLMprefills the answer channelto=user<|message|>(the same turn the template writes for an answer), withReasoning strength: lowkept in the system turn. The smoke compared this with "reasoning strength low" alone: the model still reasoned on every item, taking 5–11 s per POPE question against 0.35 s.2026-10-15so prompts and cache keys don't change from day to day.a4e59da52a7bc87ae7251dd5545c0dd437c44b68in bf16 (no quantization). About 60 GB, split over both A100 40 GB GPUs of a Grace node (device_map="auto"), with a peak of 29.7 / 30.0 GB per card.smart_resize), which makes the processor's own resize a same-size pass. Verified on 4 COCO images, 2 VG images and one over-cap 4000×3000 image:pixel_valuesare bit-identical to thedo_resize=Falsepath, and the patch grid matches the default path.device_mapoption, a_build_inputshook, and per-GPU peak-memory logging. Nothing changes for the existing keys (see Validation).eval_cell.slurmwith--gres=gpu:a100:2 --cpus-per-task=8 --mem=96G(documented indocs/hpc.mdx).Validation
tests/test_muse_glimmer_inputs.py(answer prefill, template kwargs, default device map, LANCZOS target size) andtests/test_hf_chat_inputs.py, which pins the exactapply_chat_templatemessages and kwargs of the generic wrapper (thinking off and on). The same assertions also pass against the wrapper before the_build_inputshook, so the existing keys build identical inputs.demo.pysmoke (job 20161174, 5 images): POPE answers are cleanYes/No, and ROPE sentences parse. Captions contain no reasoning text. CHAIR returns numbers, and THRONE-XL reads all captions from cache (0 misses).Results
Muse Glimmer 30B, 500 images per dataset (percent; ↑ better except CHAIR-I ↓). The CSV, homepage and README rows are added in #5, because its THRONE values use the 1,024-token evaluator window that #5 introduces.
Provenance
eval/muse-glimmer@ 6c798cc (the jobs ran on cd323d1; 6c798cc only re-wraps a docstring and adds a test); model revisiona4e59da52a7bc87ae7251dd5545c0dd437c44b68, bf16, 2× A100 40 GB.mirage-prod/results/grid/muse-glimmer-30b_*.jsonon Grace; cachesmirage-prod/cache/muse-glimmer-30b_<dataset>_<stage>/(THRONE 1,024:..._throne_w1024_evaluator).