Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
e1c9a2c
Match the reference evaluation protocol in demo.py
SalimAlkharsa Oct 15, 2026
c7a0e23
Fix quoting in eval_cell.slurm
SalimAlkharsa Oct 15, 2026
187be62
Fix the docs workflow so it parses and finds its scripts
SalimAlkharsa Oct 15, 2026
38eb972
Add Gemma 4, Ministral 3 and Aya Vision through one chat-template wra…
SalimAlkharsa Oct 15, 2026
a795e8b
Add thinking-on Gemma 4 keys for the thinking ablation
SalimAlkharsa Oct 15, 2026
2a9453e
Report Gemma 4, Ministral 3 and Aya Vision results; rescore THRONE wi…
SalimAlkharsa Oct 15, 2026
3b3c350
Document how an unclosed thinking block is scored
SalimAlkharsa Oct 15, 2026
96a2561
Normalize answer formatting in the ROPE single-box extractor
SalimAlkharsa Oct 15, 2026
b4cfb7b
Rescore ROPE strict for every row with the formatting-robust extractor
SalimAlkharsa Oct 15, 2026
a24dcb3
Add Muse Glimmer 30B through the chat wrapper
SalimAlkharsa Oct 15, 2026
f6e1aaa
Pin the HFChatVLM chat-template call across the _build_inputs hook
SalimAlkharsa Oct 16, 2026
80c26fd
Make the THRONE evaluator input window configurable
SalimAlkharsa Oct 16, 2026
9706d6e
Report THRONE at the 1024-token evaluator window and add Muse Glimmer…
SalimAlkharsa Oct 16, 2026
e444bbf
Note that THRONE uses the XL evaluator alone, not the paper's 9-vote …
SalimAlkharsa Oct 16, 2026
daebc84
Add judged metrics: Mod-POPE, Mod-ROPE, Mod-CHAIR and class-wise THRO…
SalimAlkharsa Oct 16, 2026
e6a063d
Simplify demo.py; drop doc prose and new tests from this PR
SalimAlkharsa Oct 16, 2026
8798d09
Merge eval/protocol-parity: simplified demo.py, no doc prose
SalimAlkharsa Oct 16, 2026
2ca4ebd
Merge eval/new-models; keep only the results tables and a one-line no…
SalimAlkharsa Oct 16, 2026
c7b4c60
Merge eval/results-v1; drop doc prose and the new tests from this PR
SalimAlkharsa Oct 16, 2026
0c480c5
Merge eval/muse-glimmer; keep only the results tables and a one-line …
SalimAlkharsa Oct 16, 2026
b8dce90
Merge eval/throne-window; no docs changes in this PR, drop the new te…
SalimAlkharsa Oct 16, 2026
3dff27c
Rename demo.py to run.py (the CLI slurm calls)
SalimAlkharsa Oct 17, 2026
720bc48
Import metric classes lazily so importing metrics needs no torch
SalimAlkharsa Oct 17, 2026
5827812
Metrics own their prompts, pool sizes and generation settings
SalimAlkharsa Oct 17, 2026
fda836a
Default to the published protocol: 500 images, 6 POPE questions per i…
SalimAlkharsa Oct 17, 2026
aeca8d4
slurm: eval_cell calls run.py; drop the generic demo.slurm
SalimAlkharsa Oct 17, 2026
96e2d7d
Add a 50-line demo.py: dataloader, model and metrics wired by hand
SalimAlkharsa Oct 17, 2026
96e199d
Merge eval/protocol-parity
SalimAlkharsa Oct 17, 2026
0093f01
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
d2ac788
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
07d922e
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
ec20b08
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
753cc1d
Open the prediction cache read-only under --cache-only
SalimAlkharsa Oct 17, 2026
95a8b33
Make the protocol constants read-only and give each one a single name
SalimAlkharsa Oct 17, 2026
76cc842
Move the POPE, ROPE and caption loops into the metrics (Metric.run)
SalimAlkharsa Oct 17, 2026
67732c5
demo.py: wire the metrics through Metric.run and refuse flags
SalimAlkharsa Oct 17, 2026
3c3e83d
Merge eval/protocol-parity
SalimAlkharsa Oct 17, 2026
75c1774
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
dd60539
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
e778716
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
61dedaf
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
8c92e45
Rename models/hf_chat_vlm.py to models/hf_chat.py
SalimAlkharsa Oct 17, 2026
4aa7c62
Remove batch_generate everywhere
SalimAlkharsa Oct 17, 2026
1b93e86
models: drop dead BaseModel helpers; import families lazily
SalimAlkharsa Oct 17, 2026
cdc85f5
Pin each model family's cache name on its class
SalimAlkharsa Oct 17, 2026
2bb65ff
One file per model family and a public, torch-free model registry
SalimAlkharsa Oct 17, 2026
d3f133b
build_predictor accepts a model instance
SalimAlkharsa Oct 17, 2026
1dd4d7e
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
49bb353
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
2038f8c
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
b38894f
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
35e6440
Guard the model registry against drift and mismatched keys
SalimAlkharsa Oct 17, 2026
ca02f58
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
8063137
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
9c8bf28
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
a2f1a51
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
26f1f5d
Make BaseMetric.evaluate and format_results non-abstract
SalimAlkharsa Oct 17, 2026
abb839a
ROPE: plain ROPE is one exact-match comparison
SalimAlkharsa Oct 17, 2026
176a749
Delete matching/ and its test
SalimAlkharsa Oct 17, 2026
81fc4a7
predict/identity: drop the reference to the deleted matching package
SalimAlkharsa Oct 17, 2026
69e3356
Move the Gemma 4 thinking variant into gemma4.py as Gemma4Thinking
SalimAlkharsa Oct 17, 2026
b0f9fb0
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
e063689
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
e242649
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
3e423c8
judged: score plain ROPE with ROPEMetric.strict
SalimAlkharsa Oct 17, 2026
975ea59
ROPE: docstrings describe the single-box protocol
SalimAlkharsa Oct 17, 2026
6f1c91b
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
fe1cfc6
Move MuseGlimmerVLM into models/muse_glimmer.py
SalimAlkharsa Oct 17, 2026
eb83b96
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
4d401a5
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
1bce26e
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
1442066
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
cc9a3bf
Name the Muse Glimmer class MuseGlimmer, like the other families
SalimAlkharsa Oct 17, 2026
e49e35e
THRONE: import torch only when the evaluator weights load
SalimAlkharsa Oct 17, 2026
1c65dc4
THRONE: injectable evaluator and tokenizer; read-only replay mode
SalimAlkharsa Oct 17, 2026
28d0d2e
base_metric: fbeta, BETA2 and MIN_WORDS for the judged scores
SalimAlkharsa Oct 17, 2026
4a97696
THRONE: class-wise F0.5 scoring and per-dataset setup
SalimAlkharsa Oct 17, 2026
931f337
THRONE defaults: FLAN-T5-XL, distractor ratio 38, window 1024
SalimAlkharsa Oct 17, 2026
784b776
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
31849ae
judged: score THRONE with metrics.throne.classwise
SalimAlkharsa Oct 17, 2026
a091c4d
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
723ca0b
THRONE: reject loader plus object_categories, warn on unpinned evalua…
SalimAlkharsa Oct 17, 2026
f4fb231
Move tools/judge into the judge package (run_judge.py becomes judge/c…
SalimAlkharsa Oct 17, 2026
a282337
Rename metrics/judged.py to repro.py, the manual 612-value check
SalimAlkharsa Oct 17, 2026
98e6ccd
Metrics own their judged scorers; verdict files load from judge/store.py
SalimAlkharsa Oct 17, 2026
7c5b420
repro: CHAIR setup, reference objects and THRONE scoring come from th…
SalimAlkharsa Oct 17, 2026
bedfff7
judge package: frozen VerdictStore, Judge, live ClaudeCLI (off by def…
SalimAlkharsa Oct 17, 2026
7cabc07
repro.py: verdicts through VerdictStore; --check and --export-committee
SalimAlkharsa Oct 17, 2026
afa5c28
Metric.run returns the records each metric's score takes
SalimAlkharsa Oct 17, 2026
2196311
run.py: judged block (or missing_verdicts) for POPE, ROPE, CHAIR and …
SalimAlkharsa Oct 17, 2026
9587946
demo.py: load each component and score with the judge (50 lines)
SalimAlkharsa Oct 17, 2026
6d1d3d8
predict/identity: document the cache-identity gaps (D5)
SalimAlkharsa Oct 17, 2026
e0bd36f
Point the judgments card and the CHAIR color_guard note at the curren…
SalimAlkharsa Oct 17, 2026
a0593fa
repro --check: find WordNet through the corpus loader (it ships as a …
SalimAlkharsa Oct 17, 2026
d2d7711
judge store: open checks only the verdict files; final_tables.json is…
SalimAlkharsa Oct 17, 2026
bedb06d
Judge: live grading only for mentions; open the store once
SalimAlkharsa Oct 17, 2026
d3591ab
judge/claude.py: one lookup for the claude binary
SalimAlkharsa Oct 17, 2026
7be785b
metrics: drop unused numpy and typing imports from base_metric and pope
SalimAlkharsa Oct 17, 2026
1908c3d
run.py: check the judged scores' inputs before generation; one verdic…
SalimAlkharsa Oct 17, 2026
a759271
run.py: --cache-only covers NOPE and refuses FaithScore; NOPE owns it…
SalimAlkharsa Oct 17, 2026
1b32de5
run.py: main and _parse_args take argv
SalimAlkharsa Oct 17, 2026
84d3039
CHAIR: never download NLTK data at import
SalimAlkharsa Oct 17, 2026
db8743e
POPE: drop the unused get_question_stats; document yes_objects
SalimAlkharsa Oct 17, 2026
77dd214
Merge eval/protocol-parity
SalimAlkharsa Oct 17, 2026
6086748
run.py: NOPE opens its cache through build_predictor(cache_only=...)
SalimAlkharsa Oct 17, 2026
149c772
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
b317f24
base_metric: BaseMetric is a plain base class; one-line compute_* doc…
SalimAlkharsa Oct 17, 2026
9e8bfb6
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
7546d7a
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
4eb5b47
THRONE: plain F0.5 through base_metric.fbeta
SalimAlkharsa Oct 17, 2026
7c6f2c8
THRONE: run returns per_image_details, the records score takes
SalimAlkharsa Oct 17, 2026
de8b1d5
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
b5573d1
One rule for the images a caption metric scores (base_metric.annotated)
SalimAlkharsa Oct 17, 2026
efb7e7b
run.py: main takes the model, judge and THRONE evaluator to use
SalimAlkharsa Oct 17, 2026
de22f3a
demo.py: print a note instead of aborting when verdicts are missing; …
SalimAlkharsa Oct 17, 2026
abb0319
POPE evaluate reads the object with QUESTION_RE; rope.py: two blank l…
SalimAlkharsa Oct 17, 2026
19bee94
Judge: a clear error when live mention grading has no image_paths
SalimAlkharsa Oct 17, 2026
ceffea5
slurm: list the judged block's preconditions in the eval_cell header
SalimAlkharsa Oct 17, 2026
80ec3a1
Ship the frozen committee: judgments/committee.json
SalimAlkharsa Oct 17, 2026
f626d28
slurm: eval_cell is a fill-in-the-blank template; rope comment says L…
SalimAlkharsa Oct 17, 2026
50512e7
Registry errors say to add or update the row in predict.factory.REGISTRY
SalimAlkharsa Oct 17, 2026
21f3128
tests: the one-image smoke test drives run.py (demo.py is now a wirin…
SalimAlkharsa Oct 17, 2026
7232626
Merge eval/protocol-parity
SalimAlkharsa Oct 17, 2026
93b78d8
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
e8ff5a5
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
b666dcf
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
e981c30
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
57e2208
tests: the smoke test passes the model and judge to run.main and chec…
SalimAlkharsa Oct 17, 2026
61c5701
Drop NOPE and FaithScore (no published numbers); they come back in th…
SalimAlkharsa Oct 17, 2026
c5a86df
Merge eval/protocol-parity
SalimAlkharsa Oct 17, 2026
c01a80a
Merge eval/new-models
SalimAlkharsa Oct 17, 2026
0c1d68e
Merge eval/results-v1
SalimAlkharsa Oct 17, 2026
822a3a3
Merge eval/muse-glimmer
SalimAlkharsa Oct 17, 2026
d07724c
Merge eval/throne-window
SalimAlkharsa Oct 17, 2026
4eb2c36
repro.py: --export-records and --records, so --check runs without the…
SalimAlkharsa Oct 17, 2026
a63b8ba
tests: grading against the published table, and run.py replayed from …
SalimAlkharsa Oct 17, 2026
4f7c6b3
dataloader: drop the unused downloaders, filters, shuffle and batchin…
SalimAlkharsa Oct 17, 2026
e80ac6d
metrics: drop POPE popular mode and CHAIR's unused switches; guard em…
SalimAlkharsa Oct 17, 2026
96a191f
predict, models: clear cache-only miss error; drop the qwen2vl alias,…
SalimAlkharsa Oct 17, 2026
8766510
judge: live grading keeps only requested verdicts, survives timeouts,…
SalimAlkharsa Oct 17, 2026
d8cf02b
run.py: --live-judge, no judged score where the frozen verdicts don't…
SalimAlkharsa Oct 17, 2026
ff698bd
Remove documentation text ahead of the docs rewrite
SalimAlkharsa Oct 18, 2026
e2d8883
Rewrite the docs and re-judge POPE "no" questions
SalimAlkharsa Oct 18, 2026
a9a5788
docs: add the Gemma 4 thinking ablation on COCO POPE to the models page
SalimAlkharsa Oct 19, 2026
21d6f1a
Add the PolyForm Noncommercial license, a citation file and test data…
SalimAlkharsa Oct 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 0 additions & 31 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,11 @@ on:
pull_request:
paths:
- "docs/**"
- "README.md"
- "demo.py"
- "dataloader/**"
- "metrics/**"
- "models/**"
- "predict/**"
- "scripts/**"
- "slurm/**"
- "tests/**"
- "requirements*.txt"
- "pyproject.toml"
- ".github/workflows/docs.yml"
push:
branches: [main]
paths:
- "docs/**"
- "README.md"
- "demo.py"
- "dataloader/**"
- "metrics/**"
- "models/**"
- "predict/**"
- "scripts/**"
- "slurm/**"
- "tests/**"
- "requirements*.txt"
- "pyproject.toml"
- ".github/workflows/docs.yml"

jobs:
Expand All @@ -41,15 +19,6 @@ jobs:
working-directory: docs
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: First test: code changes include documentation
env:
BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
HEAD_SHA: ${{ github.sha }}
run: python3 scripts/check_docs_coverage.py "$BASE_SHA" "$HEAD_SHA"
- name: Check reference results against CSV
run: python3 scripts/check_results_tables.py
- uses: actions/setup-node@v4
with:
node-version: 20
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,6 @@ slurm/runs/
.env
.DS_Store
.vscode/

# Live judge verdicts (judge.Judge.load(live=True))
judgments/live/
11 changes: 11 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
cff-version: 1.2.0
message: "If you use Mirage, please cite it as below."
title: "Mirage: object hallucination metrics for vision language models on sparsely labelled datasets"
type: software
authors:
- family-names: Alkharsa
given-names: Salim
repository-code: "https://github.com/Advanced-Vision-and-Learning-Lab/mirage"
url: "https://mirage-eval.com"
license: LicenseRef-PolyForm-Noncommercial-1.0.0
date-released: 2026-10-19
75 changes: 75 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# PolyForm Noncommercial License 1.0.0

<https://polyformproject.org/licenses/noncommercial/1.0.0>

## Acceptance

In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.

## Copyright License

The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose. However, you may only distribute the software according to [Distribution License](#distribution-license) and make changes or new works based on the software according to [Changes and New Works License](#changes-and-new-works-license).

## Distribution License

The licensor grants you an additional copyright license to distribute copies of the software. Your license to distribute covers distributing the software with changes and new works permitted by [Changes and New Works License](#changes-and-new-works-license).

## Notices

You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms or the URL for them above, as well as copies of any plain-text lines beginning with `Required Notice:` that the licensor provided with the software. For example:

> Required Notice: Copyright Yoyodyne, Inc. (http://example.com)

## Changes and New Works License

The licensor grants you an additional copyright license to make changes and new works based on the software for any permitted purpose.

## Patent License

The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.

## Noncommercial Purposes

Any noncommercial purpose is a permitted purpose.

## Personal Uses

Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, is use for a permitted purpose.

## Noncommercial Organizations

Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution is use for a permitted purpose regardless of the source of funding or obligations resulting from the funding.

## Fair Use

You may have "fair use" rights for the software under the law. These terms do not limit them.

## No Other Rights

These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else. These terms do not imply any other licenses.

## Patent Defense

If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.

## Violations

The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice. Otherwise, all your licenses end immediately.

## No Liability

***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***

## Definitions

The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.

**You** refers to the individual or entity agreeing to these terms.

**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization. **Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise. Control can be direct or indirect.

**Your licenses** are all the licenses granted to you for the software under these terms.

**Use** means anything you do with the software requiring one of your licenses.

Required Notice: Copyright 2026 Salim Alkharsa (https://github.com/Advanced-Vision-and-Learning-Lab/mirage)
39 changes: 7 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,14 @@
# Mirage

Mirage evaluates object hallucination in vision-language models using COCO and Visual Genome. It includes metric implementations, local model wrappers, and prediction caching.

**Documentation:** [mirage-f432d87b.mintlify.app](https://mirage-f432d87b.mintlify.app)

## Quick start

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt -r requirements-dev.txt
pytest -q tests/test_demo_smoke.py
```

## Run an evaluation

Prepare the dataset files and model weights locally, then run:
Mirage is an evaluation framework for vision language models. It measures object hallucination with POPE, ROPE, CHAIR and THRONE, each modified so it still works on datasets with sparse labels such as Visual Genome. Results, methods and guides are at [mirage-eval.com](https://mirage-eval.com).

```bash
python demo.py --dataset coco --metric chair --model paligemma --samples 5 --output results/chair-coco.json
pip install -r requirements.txt
python -m nltk.downloader punkt_tab averaged_perceptron_tagger_eng stopwords wordnet
python run.py --dataset coco --metric pope --model gemma4-e4b --samples 10
python repro.py --records tests/data/records.json.gz --check # reproduce the results table
```

Mirage currently supports `paligemma` and `qwen2vl`. See the [documentation](https://mirage-f432d87b.mintlify.app) for dataset setup, metrics, model configuration, and HPC instructions.

## Reference results

Scores are percentages from 1,000 images per model and dataset. Higher is better for POPE, ROPE, and THRONE; lower is better for CHAIR-I.

| Dataset | Model | POPE accuracy ↑ | ROPE strict accuracy ↑ | CHAIR-I hallucination rate ↓ | THRONE F1 ↑ |
| --- | --- | ---: | ---: | ---: | ---: |
| COCO | PaliGemma-3B | 94.2% | 21.6% | 9.1% | 72.9% |
| COCO | Qwen2-VL-2B | 93.6% | 3.6% | 12.8% | 73.4% |
| COCO | Qwen2-VL-7B | 93.5% | 13.1% | 12.7% | 72.9% |
| Visual Genome | PaliGemma-3B | 88.0% | 8.2% | 67.8% | 21.5% |
| Visual Genome | Qwen2-VL-2B | 88.7% | 8.6% | 75.6% | 24.0% |
| Visual Genome | Qwen2-VL-7B | 89.5% | 10.3% | 69.3% | 23.6% |
## License and citation

The [results CSV](docs/model-results.csv) is the source for these values.
Free for non-commercial use under the [PolyForm Noncommercial License 1.0.0](LICENSE). Commercial use needs a separate license from the copyright holder. If you use Mirage, please cite it with [CITATION.cff](CITATION.cff). Test images and annotations in `tests/data/wiring` keep their own licenses ([ATTRIBUTION.md](tests/data/wiring/ATTRIBUTION.md)).
56 changes: 26 additions & 30 deletions dataloader/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@

import random
from abc import ABC, abstractmethod
from typing import Any, Dict, Iterator, Optional, Union, List, Tuple
from typing import Any, Dict, Iterator, Optional, Union, List
from pathlib import Path
from PIL import Image

ROPE_SAMPLING_SEED = 42

Expand Down Expand Up @@ -42,28 +43,19 @@ class BaseDataLoader(ABC):

Attributes:
data_path (Path): Path to the dataset
batch_size (int): Number of samples per batch
shuffle (bool): Whether to shuffle the data
skipped (int): Images skipped by load_data (file missing or unreadable) before the pool filled
_loaded (bool): Internal flag to track if data has been loaded
"""

def __init__(
self,
data_path: Union[str, Path],
batch_size: int = 1,
shuffle: bool = False
):
def __init__(self, data_path: Union[str, Path]):
"""
Initialize the dataloader.

Args:
data_path: Path to the dataset directory or file
batch_size: Number of samples to load per batch
shuffle: Whether to shuffle the data during iteration
"""
self.data_path = Path(data_path)
self.batch_size = batch_size
self.shuffle = shuffle
self.skipped = 0
self._loaded = False

@abstractmethod
Expand Down Expand Up @@ -130,32 +122,36 @@ def validate_data_path(self) -> bool:
"""
return self.data_path.exists()

def get_info(self) -> Dict[str, Any]:
@abstractmethod
def load_image(self, sample: Dict[str, Any]) -> Image.Image:
"""
Get information about the dataset.
Load the image associated with a sample.

Args:
sample: Sample dictionary from __getitem__

Returns:
Dict containing dataset metadata
PIL.Image: Loaded RGB image
"""
return {
'data_path': str(self.data_path),
'batch_size': self.batch_size,
'shuffle': self.shuffle,
'loaded': self._loaded,
'total_samples': len(self) if self._loaded else None
}
pass

@abstractmethod
def extract_objects(self, sample_indices: Union[List[int], None] = None) -> Tuple[Dict[str, List[str]], List[str]]:
def extract_bboxes_for_rope(
self,
num_objects: int = 5,
sample_indices: Union[List[int], None] = None,
seed: Optional[int] = ROPE_SAMPLING_SEED,
) -> Dict[str, List[Dict[str, Any]]]:
"""
Extract objects or classes from dataset samples.
Extract bounding boxes suitable for ROPE evaluation.

Args:
sample_indices: Optional list of indices to process (None for all)
num_objects: Required number of bounding boxes per image
sample_indices: Optional list of image indices to consider (None for all)
seed: Base seed for the per-image sampling RNG (None for process-global random)

Returns:
Tuple containing:
- Dict mapping sample_id -> list of objects in that sample
- List of all unique objects across all samples
Dict mapping image_id (str) to a list of exactly num_objects bbox
dicts, each with keys: xmin, ymin, xmax, ymax, name
"""
raise NotImplementedError("Each dataloader must implement extract_objects method")
pass
Loading
Loading