VideoRealBench: A Chain-of-Thought Realism Evaluation Benchmark for Generated Human-Centric Videos

Our paper is available in VideoRealBench

News

[2026.8.6] The code is updated.

[2026.3.28] Our VideoRealBench has been accepted by CVPR2026. The code will be updated soon.

Directory

.
├── checkpoint/                     # VideoRealBench LoRA weight
│   ├── adapter_config.json
│   ├── adapter_model.safetensors
│   └── additional_config.json
├── data/
│   └── videorealbench_test.csv     # Annotation
├── eval/
│   ├── infer_qwen25vl.py           
│   ├── com_pearson.py              
│   └── prompt.py                  
├── model/
│   └── Qwen2.5-VL-7B-Instruct/    
├── video/                          # Video path
├── infer.sh                        # Inference script
├── environment.yml
└── requirements.txt

model/ and video/ are the expected paths; simply create or populate them after downloading the corresponding files.

Model File Description

The repository already contains the trained VideoRealBench model weights:

checkpoint/adapter_model.safetensors

This is a LoRA adapter of approximately 39 MB, configured with a LoRA rank of 8 and an alpha of 16, based on the Qwen2.5-VL-7B-Instruct model. It is not a standalone model; the Qwen2.5-VL-7B-Instruct base model must be loaded alongside it for inference.

Environment

It is recommended to use Linux, Python 3.11, and an NVIDIA GPU with CUDA support.

conda env create -f environment.yml
conda activate videorealbench_test
pip install -r requirements.txt

Key dependencies include PyTorch, Transformers, PEFT, qwen-vl-utils, Pandas, and SciPy. If the PyTorch version specified in requirements.txt is incompatible with your local CUDA setup, please install the appropriate version by following the instructions on the official PyTorch installation page.

Download Qwen2.5-VL-7B-Instruct

First, install the Hugging Face command-line tool:

pip install -U huggingface_hub

Then, download the base model in the project root directory:

hf download Qwen/Qwen2.5-VL-7B-Instruct \
  --local-dir model/Qwen2.5-VL-7B-Instruct

After the download is complete, you should see files similar to the following:

model/Qwen2.5-VL-7B-Instruct/
├── config.json
├── model-00001-of-00005.safetensors
├── ...
├── tokenizer.json
└── preprocessor_config.json

Model homepage: Qwen/Qwen2.5-VL-7B-Instruct. For detailed usage of hf download, please refer to the Hugging Face download documentation.

If accessing Hugging Face requires logging in, you can first run:

hf auth login

Download VideoRealDataset

Dataset download link is here VideoRealDataset

After downloading and extracting the files, ensure that the test CSV and the video directory are located in the project root directory, and keep the relative paths specified in the videopath column of the CSV unchanged.

data/videorealbench_test.csv
video/<video-id>.mp4

For example, if the CSV contains the following record:

video/807c1d0a5c5344f48fdcda695c91f987.mp4

The corresponding video must be located at the same path under the project root directory. You can perform a quick check before running inference:

test -f data/videorealbench_test.csv && echo "CSV ready"
test -d video && echo "Video directory ready"

Inference

Ensure the current working directory is the project root, then execute:

conda activate videorealbench_test
bash infer.sh

infer.sh uses the following default paths:

• Dataset：data/videorealbench_test.csv
• Qwen base model：model/Qwen2.5-VL-7B-Instruct
• VideoRealBench LoRA：checkpoint
• Results：result/infer_test/eval_test.csv

The script first completes inference on the test set and then calculates metrics such as Pearson, Spearman, MSE, and Accuracy. The key new fields in the inference results are:

• pred_score：Model-predicted scores of 1–5
• pred_text：Full analysis text generated by the model

Credits

We especially thank the contributors of the VideoPhy-2 for providing helpful code and videos.