Search a photo collection with a sentence or with another photo, and ask questions that are answered from the photos themselves. Two dual encoders (Google's SigLIP 2 and Meta's Perception Encoder) are fused into one embedding space behind a FAISS index, and a vision-language model (Qwen3-VL) reranks the best candidates and writes answers that cite the photos they come from.
5,000 COCO test photos on a laptop. A sentence, reranked by Qwen3-VL-4B; a photo; and a question answered from the four photos the search found, each one cited and outlined.flowchart LR
Q[text or photo query] --> E[SigLIP 2 + Perception Encoder<br/>fused embedding]
E --> I[FAISS HNSW<br/>float or int8]
I --> K[top 12]
K --> R[Qwen3-VL reranker<br/>p yes / p no]
R --> S[ranked photos]
S --> A[Qwen3-VL answer<br/>with photo citations]
Highlights
- Zero-shot retrieval above every published single model it is compared with: Recall@1 for text→image is 89.7 on Flickr30k and 65.1 on COCO 5K, against 86.0 and 58.6 for the best numbers reported for SigLIP 2, Perception Encoder and InternVL-C. Nothing is trained or tuned on the benchmarks.
- Two encoder families, one index. Each image is stored as the concatenation of its normalised SigLIP 2 and Perception Encoder embeddings scaled by 1/√2, so a single inner product is the mean of the two cosine similarities. The two models make different mistakes, and the fusion gains 2 to 3 points of text→image Recall@1 over the better of them on both benchmarks and on a 123k-image collection.
- A vision-language reranker that needs no tuning. Qwen3-VL-8B reads each of the top 12 photos with the query and scores log p(yes) - log p(no). Its scores and the embedding similarities are z-normalised within the candidate list and added, which lifts COCO text→image Recall@1 from 60.4 to 65.1.
- Measured at collection scale. Exact, HNSW, int8 HNSW and IVF-PQ indexes over all 123k COCO 2017 images, with recall against exact search, single-core latency and memory per image.
- Multimodal RAG with a measured benefit. On MRAG-Bench, SigLIP 2 retrieval finds 45 % of the ground-truth reference images, against 34 % for a CLIP retriever on the same images, and raises Qwen3-VL-8B's accuracy from 55.9 % to 61.9 %, more than half of what the ground-truth images give.
pip install -e ".[index,app]"
python -m imsearch.collection build --images ~/Pictures/trip --out runs/collections/trip \
--encoder siglip2-so400m pe-core-l
IMSEARCH_COLLECTION=runs/collections/trip IMSEARCH_RERANK=12 uvicorn imsearch.app:app --port 8090The web app at localhost:8090 searches by text or photo and answers questions. The same
service as an API:
curl "localhost:8090/api/search?q=a+red+boat+in+fog&k=24&rerank=12"
curl -F file=@query.jpg localhost:8090/api/search-image
curl -X POST localhost:8090/api/ask -H "Content-Type: application/json" \
-d '{"question": "What colour was the tent?", "k": 4}'IMSEARCH_VLM picks the vision-language model (default Qwen/Qwen3-VL-4B-Instruct, which runs
on an Apple-silicon laptop); one loaded model serves both reranking and answers. Collections
above 2,000 images get an HNSW index over int8 vectors, smaller ones exact search.
Recall@1 on the Flickr30k 1K and COCO 5K test sets (Karpathy splits), zero-shot. In parentheses the numbers reported by the SigLIP 2 and Perception Encoder papers for the same checkpoints.
| Model | Flickr30k text→image | Flickr30k image→text | COCO text→image | COCO image→text |
|---|---|---|---|---|
| OpenCLIP ViT-L/14, LAION-2B | 75.5 (75.0) | 89.5 (88.7) | 46.5 (46.1) | 63.3 (62.1) |
| SigLIP 2 So400m/16, 384 px | 83.3 (85.3) | 94.5 (95.9) | 56.2 (56.0) | 70.8 (71.2) |
| SigLIP 2 g-opt/16, 384 px | 83.3 (86.0) | 95.1 (95.4) | 56.4 (56.1) | 72.4 (72.8) |
| Perception Encoder Core L/14, 336 px | 85.3 (85.5) | 96.3 (96.6) | 56.6 (57.1) | 75.7 (75.9) |
| Perception Encoder Core G/14, 448 px | 85.9 (85.7) | 96.8 (96.2) | 57.8 (58.1) | 75.1 (75.4) |
| InternVL-C, 5.5B (published) | (85.0) | (95.7) | (58.6) | (74.9) |
| Fusion: SigLIP 2 So400m + PE-Core L | 87.3 | 97.1 | 59.8 | 77.5 |
| Fusion: SigLIP 2 g + PE-Core G | 87.1 | 97.7 | 60.4 | 77.4 |
| Fusion g + G, reranked by Qwen3-VL-8B | 89.7 | 96.8 | 65.1 | 79.2 |
With reranking, text→image Recall@5 / @10 is 98.4 / 99.2 on Flickr30k and 84.9 / 89.3 on COCO.
- The evaluation reproduces the papers. OpenCLIP and Perception Encoder land within about one point of their published numbers, and SigLIP 2 matches on COCO. On Flickr30k, SigLIP 2 with the Hugging Face checkpoints and processor scores 1.4 to 2.7 points lower.
- Fusion beats scale. SigLIP 2 So400m + PE-Core L (0.7B image-encoder parameters together) is better on all four columns than PE-Core G (1.9B) or InternVL-C (5.5B) alone. Fusing the two largest models adds little on top: the gain comes from combining two families, not from size.
- The reranker and the embeddings are complementary. On its own, the reranker orders the 12 candidates about as well as the embeddings (text→image 61.0 vs. 60.4 on COCO, 86.3 vs. 87.1 on Flickr30k), but the sum of both scores gains 4.7 and 2.6 points over the first stage. For image→text the reranker helps on COCO and costs 0.9 points on Flickr30k, where the first stage is already at 97.7.
- Cost. Qwen3-VL-8B scores about 55 image-caption pairs per second on one H100 (bfloat16, batch 16), so reranking the top 12 adds about 0.2 s per query.
All 123,287 COCO 2017 train and val images as the collection, the 25,010 COCO 5K test captions as queries. A query counts as found when the one image its caption was written for comes back; among 123k photos there are often several that fit a caption equally well, so the absolute numbers are lower than on the 5K benchmark. Fused SigLIP 2 So400m + PE-Core L (2,176 dimensions); latency for one query on one CPU core.
| Index | R@1 | R@10 | Exact top 10 found | ms per query | Bytes per image |
|---|---|---|---|---|---|
| Exact search | 28.4 | 55.8 | 100 % | 23.1 | 8,704 |
| HNSW, ef 64 | 27.5 | 54.0 | 95.3 % | 0.56 | 8,976 |
| HNSW, ef 256 | 28.2 | 55.4 | 99.1 % | 1.59 | 8,976 |
| HNSW over int8, ef 64 | 27.5 | 54.0 | 94.6 % | 0.29 | 2,448 |
| HNSW over int8, ef 256 | 28.2 | 55.4 | 98.2 % | 0.64 | 2,448 |
| IVF-PQ, 64 B codes | 13.6 | 35.1 | 35.9 % | 0.94 | 380 |
| IVF-PQ, 64 B codes, exact re-scoring of 16 × k | 26.7 | 51.3 | 85.3 % | 1.06 | 9,084 |
| IVF-PQ, 128 B codes | 18.4 | 42.9 | 47.4 % | 1.15 | 444 |
- HNSW over 8-bit vectors is the best trade-off. It keeps the graph and quantises each dimension to one byte: 3.7 times less memory than float HNSW, the same Recall@1, and 2.5 times faster at ef 256, because each distance reads a quarter of the bytes. Collections use it by default.
- Product quantisation is too coarse for these embeddings. 64 or 128 bytes for 2,176 dimensions lose more than half of the exact top 10. Re-scoring with the exact vectors brings the recall back but needs the vectors in memory again.
- Fusion holds at scale: exact search with the fused model finds 28.4 % of the targets at rank 1, against 25.4 % (SigLIP 2) and 25.2 % (PE-Core L) alone.
MRAG-Bench (ICLR 2025) has 1,353 multiple-choice questions about a query photo in nine scenarios where other photos of the same object help: other viewpoints, occlusion, partial views, changes over time and more. Each retriever searches, with the query photo, the pool of all 3,631 distinct reference images in the dataset release (the ground-truth images and the images the benchmark's own retriever found, for every question) and passes the five nearest to Qwen3-VL-8B, which answers from them and the query photo. As in the benchmark's protocol, the "incomplete" scenario gets one reference image. The benchmark's own retriever searched a larger corpus, so its row is not like-for-like; OpenCLIP L/14 on the same pool is the CLIP baseline.
| Reference images | Ground-truth images retrieved | Accuracy |
|---|---|---|
| None, query photo only | - | 55.9 % |
| Released with the benchmark (CLIP, larger corpus) | 22.7 % | 53.9 % |
| OpenCLIP ViT-L/14 | 33.7 % | 57.7 % |
| PE-Core L | 42.3 % | 59.1 % |
| SigLIP 2 So400m | 42.6 % | 60.6 % |
| PE-Core G | 45.2 % | 61.3 % |
| SigLIP 2 g | 44.7 % | 61.9 % |
| Fusion SigLIP 2 g + PE-Core G | 46.9 % | 60.5 % |
| Ground-truth images (upper bound) | 100 % | 67.2 % |
- Better retrieval, better answers. Accuracy follows the share of ground-truth images retrieved: 57.7 % with CLIP (34 %), 61.9 % with SigLIP 2 g (45 %), which is more than half of the 11.3 points the ground-truth images give.
- Largest gains where the query photo hides the object: obstruction 59.3 → 73.1 % and partial views 56.5 → 68.7 % (SigLIP 2 g).
- Above about 42 % retrieval recall the encoders are statistically tied: they lie within 2.8 points, about two standard errors on 1,353 questions, so the fusion's higher retrieval recall does not show in the answers here.
bash scripts/setup_cluster.sh # once, on a login node
bash scripts/download_models.sh # encoders and Qwen3-VL; jobs run with HF_HUB_OFFLINE=1
bash scripts/download_data.sh --coco # Flickr30k / COCO test sets, MRAG-Bench, COCO 2017
python3 slurm/generate.py benchmarks fusion --submit # retrieval, every encoder and the fusions
python3 slurm/generate.py rerank --submit # Qwen3-VL-8B, 32 shards
python3 slurm/generate.py rerank-merge --submit
python3 slurm/generate.py corpus index-bench --submit # 123k-image index benchmark
python3 slurm/generate.py rag-embed rag --submit # MRAG-BenchLocal evaluation of one benchmark: imsearch-eval --benchmark flickr30k --encoders siglip2-so400m pe-core-l --fuse siglip2-so400m pe-core-l. Embeddings are cached under runs/embeddings, so
fusions and reranking reuse them.
src/imsearch/
encoders.py SigLIP 2 (transformers) and open_clip models, incl. Perception Encoder, behind one interface
data.py Flickr30k / COCO test sets: images, five captions each
metrics.py text→image and image→text ranks, Recall@K
evaluate.py cached embeddings, late fusion, imsearch-eval
rerank.py Qwen3-VL yes/no reranker, z-score fusion, sharded runs and merge
index.py exact, HNSW, int8 HNSW and IVF-PQ indexes and their benchmark
corpus.py 123k-image embedding with parallel decoding, index benchmark
collection.py a searchable folder of photos, fused query encoder
rag.py MRAG-Bench: reference pool, retrieval, answers, accuracy per scenario
app.py FastAPI service: text and photo search, reranking, answers with citations
static/ the web app
slurm/ job definitions (cluster.toml) and the script generator
tests/ ranks, recall, fusion, indexes, answer parsing
- The encoders were pretrained on web-scale image-text data that may include COCO and Flickr photos, the usual caveat of zero-shot retrieval benchmarks.
- Reranking costs one VLM pass per candidate: about 0.2 s per query for 12 candidates on an H100, about 5 s with Qwen3-VL-4B on an Apple-silicon laptop.
- In MRAG-Bench's "incomplete" scenario any reference image lowers the accuracy, even the ground-truth one (40.2 → 25.5 %).
- Answers can only be as good as the retrieved photos; a question about something the collection does not show still gets the nearest photos.
Code: MIT. Models: SigLIP 2, Perception Encoder and Qwen3-VL are Apache-2.0, OpenCLIP weights MIT. Data, not redistributed: COCO images under their Flickr licences and annotations CC BY 4.0; Flickr30k for non-commercial research; MRAG-Bench CC BY 4.0.
- M. Tschannen et al., SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, arXiv:2502.14786, 2025.
- D. Bolya et al., Perception Encoder: The best visual embeddings are not at the output of the network, arXiv:2504.13181, 2025.
- Qwen Team, Qwen3-VL Technical Report, 2025.
- W. Hu et al., MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models, ICLR 2025.
- M. Cherti et al., Reproducible scaling laws for contrastive language-image learning, CVPR 2023.
- Y. Malkov, D. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, TPAMI 2018.
- H. Jégou, M. Douze, C. Schmid, Product quantization for nearest neighbor search, TPAMI 2011.
- M. Douze et al., The Faiss library, arXiv:2401.08281, 2024.
- A. Karpathy, L. Fei-Fei, Deep Visual-Semantic Alignments for Generating Image Descriptions, CVPR 2015.
