Skip to content

About

ReaLVR: grounding latent visual reasoning in visual evidence. Training and evaluation code.

Topics

Resources

Stars

50 stars

Watchers

4 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

ReaLVR robot with a magnifying glass ReaLVR

Rethinking Latent Visual Reasoning
Grounding Latent Reasoning in Visual Evidence

“Reasoning like humans. Grounded in what you see.”

Paper Project Models License

Quick Start · Training · Evaluation · Data · Citation

ReaLVR walkthrough: image and question become tokens, latent reasoning unfolds, and answer contrast selects tokens for visual supervision.

From images and questions to latent reasoning grounded in visual evidence.

Overview

ReaLVR grounds continuous latent reasoning in the visual evidence needed to answer a question. It supervises the model’s own free-running latent trajectory, learning both what evidence to preserve and where to apply supervision alongside the reinforcement learning objective.

What to Preserve Where to Supervise At Inference
Contrast relevant and mismatched visual evidence. Use correct and wrong answers to assign credit to latent tokens. Keep the original architecture and inference procedure.

Note

Visual supervision is used during training only. Pretrained model weights are coming soon.

Method

One latent trajectory, two complementary contrasts
Step What Happens
Generate the trajectory Roll out a differentiable autoregressive latent span with the current model, before supplying answer tokens.
Identify relevant evidence Contrast a relevant visual prototype with mismatched visual prototypes using a cosine margin.
Assign latent credit Compare attention from correct and model-generated wrong answer readouts over the same latent span.
Train with visual supervision Apply detached credit weights to the visual margin and backpropagate through latent generation.

The two supervision branches are used during training only. The project page includes the method figures, benchmark results, and evidence sensitivity analysis.

Quick Start

Python 3.11 · PyTorch 2.6 · Qwen2.5-VL · DeepSpeed ZeRO-3

git clone https://github.com/xixiaouab/ReaLVR-code.git
cd ReaLVR-code
conda create -n realvr python=3.11 -y
conda activate realvr
pip install -r requirements.txt

Prepare the training data, then run the two stages below. The launch scripts default to eight GPUs; set GPUS_PER_NODE for your machine.

Training

Both stages use DeepSpeed ZeRO-3 (scripts/zero3.json). Set NNODES, NODE_RANK, MASTER_ADDR and GPUS_PER_NODE for multi-node runs.

Stage 1 · Latent Visual SFT

Start from Qwen/Qwen2.5-VL-7B-Instruct:

MODEL=Qwen/Qwen2.5-VL-7B-Instruct DATA=data/stage1.json IMAGE_FOLDER=data/images \
OUTPUT=checkpoints/stage1 bash scripts/stage1_sft.sh

Stage 2 · Evidence-Grounded GRPO

Continue from your Stage-1 checkpoint:

MODEL=checkpoints/stage1 DATA=data/stage2.json IMAGE_FOLDER=data/images \
OUTPUT=checkpoints/stage2 bash scripts/stage2_realvr.sh
ReaLVR Training Parameters
Argument Default Meaning
--lvr_steps 8 latent length K (training and inference)
--evidence_weight 0.2 weight of the evidence loss (0 gives plain GRPO)
--evidence_margin 0.5 target cosine margin
--credit_eta 0.3 uniform share of the position weights
--num_negatives 16 negative prototypes per example

Evaluation

python -m eval.evaluate --checkpoint checkpoints/stage2 --benchmark mmvp \
    --data-dir data/benchmarks --output-dir results/mmvp

Supported benchmarks: mmvp, blink, hrbench4k, hrbench8k, mme_realworld_lite.

Evaluation Options and Sharded Runs

The expected layout of --data-dir is listed in python -m eval.evaluate --help. By default each question gets one greedy answer with K = 8 latent steps at the processor's image-size limit. --max-pixels changes that limit, and --num-samples N samples N answers per question and selects one (--selection); the settings of every run are written to its summary.json. --start-index and --max-samples split a benchmark into shards; python -m eval.merge_results combines their summaries.

Data

Stage 1 · Visual Reconstruction Data

A JSON list of LLaVA-style records with one box per <lvr> placeholder:

{
  "image": "flickr30k/2618322793.jpg",
  "conversations": [
    {"from": "human", "value": "<image>\nWhat is the child on the swing wearing?"},
    {"from": "gpt", "value": "<lvr>\n<answer> Dark blue denim shorts. </answer>"}
  ],
  "bboxes": [[0.382, 0.456, 0.718, 0.656]]
}

Each <lvr> becomes <|lvr_start|>, one <|lvr|> per visual token inside its box, and <|lvr_end|>; the model is trained to reconstruct those visual tokens. --data_path can also be a JSON list of {"ds_name", "data_path", "image_folder"} entries to mix several datasets.

Stage 2 · Question–Answer Data

A JSON list of records:

{
  "image": "relative/path.jpg",
  "conversations": [
    {"from": "human", "value": "<image>\nQuestion ... Options: A. ... B. ..."},
    {"from": "gpt", "value": "<answer>B</answer>"}
  ],
  "bboxes": [[0.2, 0.3, 0.8, 0.9]]
}

bboxes (pixels or normalized) mark the visual evidence; without it the whole image is used. The paper uses a mixture of ViRL39K and Visual-CoT.

Repository Structure

Files and Entry Points
Path Purpose
realvr/model/ Qwen2.5-VL with latent token generation.
realvr/dataset/ SFT and GRPO data loading.
realvr/trainer/evidence.py Visual evidence loss and latent credit assignment.
realvr/trainer/ Stage-1 SFT and Stage-2 GRPO trainers.
realvr/train/ Training entry points and reward functions.
scripts/ Launch scripts and DeepSpeed configuration.
eval/ Benchmark evaluation and shard merging.
tests/ Model, training, and evaluation checks.
assets/animation/ Animation sources and GIF export scripts.
Running the Tests
pip install pytest
pytest tests/

tests/test_trainer.py runs Stage-2 steps end to end on CPU with a tiny random model; it needs the Qwen2.5-VL processor files locally (REALVR_TEST_PROCESSOR=/path/to/Qwen2.5-VL-7B-Instruct).

Authors

Xi Xiao · Tianchen Zhao · Youngeun Kim · Zhuowei Li · Linghan Xu · Jiaye Wu · Zheng Zhang · Xiang Xu · Xuanbai Chen · Farhan Tejani · Jakub Zablocki · Julia Xu · Yifan Xing

University of Alabama at Birmingham · Amazon AGI

Work done during an internship at Amazon AGI.

Contact: xxiao@uab.edu · tianchz@amazon.com

Citation

@misc{xiao2026realvr,
  title={Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence},
  author={Xiao, Xi and Zhao, Tianchen and Kim, Youngeun and Li, Zhuowei and Xu, Linghan and Wu, Jiaye and Zhang, Zheng and Xu, Xiang and Chen, Xuanbai and Tejani, Farhan and Zablocki, Jakub and Xu, Julia and Xing, Yifan},
  year={2026},
  eprint={2609.34563},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2609.34563}
}

Acknowledgements

This code builds on Latent Visual Reasoning, Qwen2-VL-Finetune, TRL and InternVL.

License

Copyright (c) 2026 Xi Xiao and ReaLVR contributors.

Licensed under CC BY-NC 4.0. Third-party components retain their original licenses; see THIRD_PARTY_NOTICES.md.

About

ReaLVR: grounding latent visual reasoning in visual evidence. Training and evaluation code.

Topics

Resources

Stars

50 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

Languages