CHIME

Code for reproducing the experiments in Contrastive Hallucination Impact Mitigation and Evaluation for Zero-Shot Dense Retrieval Systems.

Environment setup

Python 3.10 and an NVIDIA GPU are recommended. Create a Conda environment and install the dependencies:

conda create -n chime python=3.10 -y
conda activate chime
python -m pip install --upgrade pip
python -m pip install -e '.[all]'
python -m spacy download en_core_web_sm

Reproduction

1. Download SciFact

chime-download --output-dir data/beir --datasets scifact

2. Prepare contrastive training data

chime-prepare \
  --dataset-dir data/beir/scifact \
  --split train \
  --output data/processed/scifact_chime_train.jsonl \
  --hard-negatives 3 \
  --random-negatives 2 \
  --seed 42

3. Train CHIME

chime-train \
  --data data/processed/scifact_chime_train.jsonl \
  --output-dir outputs/chime-scifact \
  --base-model Qwen/Qwen2.5-7B-Instruct \
  --epochs 3 \
  --batch-size 1 \
  --gradient-accumulation-steps 8 \
  --learning-rate 2e-5 \
  --temperature 0.1 \
  --lora-r 8 \
  --lora-alpha 32 \
  --lora-dropout 0.05 \
  --target-modules q_proj,k_proj,v_proj,o_proj \
  --max-length 1024 \
  --use-4bit \
  --fp16 \
  --gradient-checkpointing

4. Evaluate

Original-query baseline:

chime-evaluate \
  --dataset-dir data/beir/scifact \
  --mode query \
  --output results/scifact_query.json

Vanilla HyDE:

chime-evaluate \
  --dataset-dir data/beir/scifact \
  --mode hyde \
  --base-model Qwen/Qwen2.5-7B-Instruct \
  --use-4bit \
  --generation-cache results/scifact_hyde_generations.jsonl \
  --output results/scifact_hyde.json

CHIME with HIA:

chime-evaluate \
  --dataset-dir data/beir/scifact \
  --mode chime \
  --base-model Qwen/Qwen2.5-7B-Instruct \
  --adapter outputs/chime-scifact/final \
  --use-4bit \
  --with-hia \
  --generation-cache results/scifact_chime_generations.jsonl \
  --output results/scifact_chime.json

The complete SciFact workflow can also be run with:

bash scripts/reproduce_scifact.sh

To evaluate another BEIR dataset, download it with chime-download and replace data/beir/scifact in the evaluation command with the corresponding directory.