LoRA and Full Fine-Tuning Show Comparable Black-Box Jailbreak Degradation in Five Open-Weight LLMs
Taleef Tamsal and Jonathan Rusert, Purdue University Fort Wayne
Findings of the Association for Computational Linguistics: AACL-IJCNLP 2026
Content warning. This repository studies jailbreak attacks on language models. The tables and label files refer to harmful behaviors by benchmark ID only. They contain no behavior text, attack prompts or model outputs; see Data access.
This repository holds the code, per-cell result tables, behavior-level evaluator labels, human annotation labels and analysis scripts behind the paper. Most of the paper's statistics can be recomputed from the shipped tables in about a minute without a GPU. The rest need model completions, API calls or checkpoints that are not in this repository, so we ship their outputs alongside the scripts that produced them.
We fine-tune five open-weight instruction models (Gemma-2-9B-IT, Llama-3.1-8B-Instruct, Phi-4, Qwen2.5-14B-Instruct and Qwen3-4B-Instruct-2507) on Alpaca-cleaned with LoRA, QLoRA and full fine-tuning (FFT), and compare them with the unmodified base model and a 4-bit base. We attack all 25 configurations with three black-box jailbreaks (PAIR, DeepInception, ArtPrompt) and white-box AutoDAN on HarmBench-400 and JailbreakBench-100, and with the multi-turn ActorAttack on 393 HarmBench behaviors. A GCG probe covers each family's base, LoRA and FFT models on JailbreakBench-100. Three automated evaluators score the outputs (the HarmBench classifier HB-Cls, GPT-4o-mini and LlamaGuard-3), and we check them against 750 doubly annotated human samples.
- Benign fine-tuning raises black-box attack success rate (ASR) by similar amounts whichever method is used. Under HB-Cls the mean shift over 30 paired cells is +4.0 pp for LoRA, +3.4 for QLoRA and +3.4 for FFT. After Holm correction, which method reaches significance depends on the evaluator. Quantizing the base to 4 bits on its own does not change black-box ASR significantly under any evaluator, and a TOST shows it equivalent to the FP16 base (HB-Cls).
- We re-trained all 15 LoRA, QLoRA and FFT cells under one recipe at seed 42. On the DeepInception and ArtPrompt cells, LoRA and FFT differ by +1.60 pp (90% CI [-0.88, +4.08]) and meet the ±5 pp equivalence bound we fixed before training (TOST p = 0.014). LoRA and FFT also meet the bound under GPT-4o-mini (p = 0.006) and LlamaGuard-3 (p = 0.018). QLoRA does not meet it (-1.23 pp, 90% CI [-6.05, +3.60], p = 0.096). This result is specific to one recipe and two attacks: with a second training dataset or a longer schedule the test no longer clears, although no significant difference appears either. It also concerns the size of the shift, not which behaviors break: the two methods share only 29% of the behaviors they jailbreak.
- White-box AutoDAN behaves differently. On Llama-3.1 every training method lowers AutoDAN ASR, and the size of the LoRA effect depends heavily on the training seed (-30.0 ± 24.1 pp over three seeds on HarmBench, HB-Cls). Under the GCG probe, fine-tuned checkpoints are easier targets than their bases in four of five families.
- Evaluator choice changes the conclusions. On the 412 ASR samples where both annotators agreed (out of 500 drawn to over-represent evaluator disagreement), HB-Cls has 81.2% precision and an 8.6% false-positive rate. GPT-4o-mini's false-positive rate is 45.7% and LlamaGuard-3's is 40.6%. Switching evaluators changes absolute ASR, which methods come out significant, and how the families rank.
adaptation-not-algorithm/
├── analysis/ statistics, tables and figures
│ ├── revision/ scripts cited in the paper ("Reproduce with ..."), plus figures/
│ ├── phase5/ equivalence tests on the controlled seed-42 grid
│ ├── audits/ seed audits, original-grid TOST, StrongREJECT judge substitution
│ └── run_all.sh runs every analysis that works from the shipped tables
├── artifacts/ per-cell result tables; master_asr_long.csv is the main one
│ ├── revision/ one CSV per paper table, GPT-5.6 judge labels, figures (PNG)
│ ├── revision_stats/ controlled-grid tables and StrongREJECT judge labels
│ └── ablations/ LoRA rank, layer and multi-seed tables
├── data/
│ ├── evaluator_labels/ behavior-level verdicts of the three evaluators (IDs and labels only)
│ └── human_annotation/ 500 ASR and 250 over-refusal human labels (labels only)
├── training/ fine-tuning scripts and SLURM jobs for every trained cell
├── evaluation/ attack and evaluator wrappers, harmbench.patch
├── hf_release/ model-card generator and upload script for the checkpoints
├── LICENSE
├── requirements.txt packages for the analysis
├── environment.yml full training and evaluation environment (Python 3.10, CUDA 12.8)
└── CITATION.cff
phase5 and track2 in file names are our internal names for the controlled seed-42 grid and for
the recipe-robustness arms (second dataset, longer training) in Appendix C of the paper.
The analysis needs no GPU, model weights or completions. We tested it with Python 3.13.5.
git clone https://github.com/Taleef7/adaptation-not-algorithm.git
cd adaptation-not-algorithm
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
bash analysis/run_all.sh # every analysis that runs from the shipped tables, about a minuterun_all.sh rewrites the shipped CSVs and figures in place, so git diff afterwards shows whether
you got the same numbers we did. In a fresh environment built from requirements.txt, every table
comes out identical except lrt_results.csv, which differs only in the last few digits of the
mixed-model fit. The figure PNGs may not be byte-identical, because font rendering and image-library
versions differ, but they look the same. Keep scipy at 1.15.3: Wilcoxon p-values with tied or zero
differences change between scipy versions.
You can also run single scripts from the repository root:
python analysis/revision/claim1_canonical_numbers.py # Table 2: aggregate dASR, Wilcoxon, Holm, bootstrap CI
python analysis/revision/regime_a_evaluator_calibration.py # evaluator calibration against human consensus
python analysis/revision/camera_ready_stats.py # Friedman, TOST margin sensitivity, ORR calibration
python analysis/phase5/phase5_tost_5fam.py # seed-42 per-cell differences and aggregate TOST
python analysis/revision/camera_ready_stats_r6.py # three-evaluator TOST, adaptation shifts, AutoDAN/GCGKey numbers these scripts print or write:
| Result | Script | Value |
|---|---|---|
| Table 2, HB-Cls LoRA dASR | claim1_canonical_numbers.py |
+4.0 pp, p = 0.007, Holm p = 0.0504, 95% CI [+1.4, +6.6] |
| LoRA vs FFT TOST (seed-42 grid, HB-Cls) | camera_ready_stats.py |
+1.60 pp, 90% CI [-0.88, +4.08], p = 0.014 at ±5 pp |
| LoRA vs FFT TOST, other evaluators | camera_ready_stats_r6.py |
GPT-4o-mini p = 0.006; LlamaGuard-3 p = 0.018 |
| ASR calibration (N = 412) | regime_a_evaluator_calibration.py |
HB-Cls precision 81.2, FPR 8.6; GPT-4o-mini FPR 45.7; LlamaGuard-3 FPR 40.6 |
| Over-refusal calibration (N = 239) | camera_ready_stats.py |
GPT-4o-mini recall 87.1, FPR 18.1; HB-Cls recall 98.4, FPR 32.2 |
| Friedman test per family (HB-Cls) | camera_ready_stats.py |
e.g. Llama-3.1 p = 0.326, Gemma-2 p = 0.0008 |
| StrongREJECT-rubric GPT-4o-mini FPR | analysis/audits/run_strongreject_judge.py |
39.2% [33.7, 45.1] |
Where each table and figure comes from
Outputs are in artifacts/revision/ unless another directory is given.
| Paper item | Script (run from the repository root) | Output or input table |
|---|---|---|
| Fig. 1 (study design) | analysis/revision/figures/figure1_pipeline.py |
artifacts/revision/figures/figure1_pipeline.png |
| Fig. 2 (black-box vs white-box dASR) | analysis/revision/figures/fig1_delta_asr_comparison.py |
reads claim1_table1.csv, artifacts/peft_deltas.csv |
| Fig. 3 (ASR vs OR-Bench) | analysis/revision/figures/fig3_asr_orr_scatter.py |
reads or_bench_vs_legacy_orr_genuine4bit.csv |
| Fig. 4 (evaluator confusion matrices) | analysis/revision/figures/fig2_regime_a_confusions.py |
reads regime_a_confusion_*_asr.csv |
| Table 2 (aggregate dASR); covariate-adjusted OLS (appendix) | analysis/revision/claim1_canonical_numbers.py |
claim1_table1.csv, claim1_table_a1_ols.csv |
| Per-family dASR table (Hedges' g = cohens_d x (1 - 3/(4(n-1)-1)), n = 6) | analysis/revision/table2_per_family.py |
table2_per_family.csv |
| DeepInception/JBB sensitivity for the per-family table | analysis/revision/table2_diagnostics.py |
table2_di_jbb_exclusion_diagnostic.csv |
| Full ASR matrix, per-dataset dASR, cross-dataset correlations, AutoDAN tables | read directly from artifacts/master_asr_long.csv and artifacts/peft_deltas.csv |
|
| AutoDAN and GCG per family, GCG McNemar tests | analysis/revision/camera_ready_stats_r6.py |
r6_autodan_gcg_family.csv, r6_gcg_mcnemar.csv |
| Rank and layer ablations, multi-seed AutoDAN | tables only | artifacts/ablations/*.csv, artifacts/cross_seed_results.csv |
| ActorAttack tables and Wilcoxon test | analysis/revision/actorattack_wilcoxon.py |
artifacts/actorattack_scoreboard.csv, actorattack_wilcoxon.txt |
| Pooled cross-evaluator OLS, TOST margin sensitivity, family-clustered TOST, Friedman, GPT-5.6 judge refusals, ORR calibration | analysis/revision/camera_ready_stats.py |
claim1_evaluator_fixed_effect_ols.csv, tost_margin_sensitivity.csv, tost_family_clustered.csv, friedman_per_family.csv, frontier_judge_refusal_sensitivity.csv, orr_evaluator_calibration.csv |
| TOST table (seed-42 grid), attack split | analysis/phase5/phase5_tost_5fam.py, analysis/phase5/phase5_attack_split_tost.py |
artifacts/revision_stats/phase5_tost_*_5fam*.csv |
| Three-evaluator TOST, format-matched DeepInception shifts, TOST without the 14B families | analysis/revision/camera_ready_stats_r6.py |
r6_tost_3judge.csv, r6_adaptation_shift*.csv |
| PAIR under the controlled recipe | analysis/phase5/phase5_pair_tost.py (controlled data) |
artifacts/revision_stats/phase5_pair_tost_*.csv |
| Behavior difficulty and category stratification | analysis/revision/difficulty_stratification.py |
difficulty_stratification.csv |
| Likelihood-ratio test for the black-box/white-box divergence | analysis/revision/lrt_formalization.py |
lrt_results.csv |
| Recipe robustness (Appendix C) | analysis/revision/track2_recipe_robustness.py |
track2_equivalence.csv |
| Overlap of jailbroken behaviors | analysis/revision/same_behavior_overlap.py |
same_behavior_overlap.csv |
| Transfer across methods | analysis/revision/attack_transfer.py |
attack_transfer.csv |
| GCG loss analysis | analysis/revision/gcg_loss_analysis.py (controlled data) |
gcg_loss_*_5fam.csv |
| ASR evaluator calibration | analysis/revision/regime_a_evaluator_calibration.py |
regime_a_calibration.csv |
| GPT-5.6 judge check | analysis/revision/frontier_judge.py (API and controlled data), analysis/revision/frontier_judge_metrics.py |
frontier_judge_calibration.csv |
| GPT-4o-mini judge-input ablation (attack prompt vs HarmBench behavior) | analysis/revision/judge_input_ablation.py (API and controlled data) |
judge_input_ablation/metrics.csv |
| StrongREJECT rubric substitution | analysis/audits/run_strongreject_judge.py |
artifacts/revision_stats/strongreject_judge_fpr.csv |
| Copyright sensitivity | analysis/revision/copyright_sensitivity.py |
copyright_sensitivity.csv, copyright_calibration.csv |
| OR-Bench detailed results | analysis/revision/build_orbench_genuine4bit.py (controlled data) |
or_bench_vs_legacy_orr_genuine4bit.csv |
| OR-Bench judge agreement | analysis/revision/orbench_judge_agreement_regen4bit.py (controlled data) |
orbench_judge_*_regen4bit.csv |
| Safe LoRA tables | analysis/revision/defense_audit.py (controlled data) |
defense_audit.csv |
| MixAT table | tables only | artifacts/mixat_asr.csv |
Scripts marked "controlled data" read raw verdict files, completions, GCG optimization logs or
checkpoints that are not in this repository. Their outputs are shipped, so you can still check every
number and see how it was computed. These are analysis/revision/{gcg_loss_analysis, defense_audit, build_orbench_genuine4bit, orbench_judge_agreement, orbench_judge_agreement_regen4bit, build_track2_arm_csv, frontier_judge}.py, analysis/phase5/phase5_pair_tost.py and
analysis/audits/{verify_bb_seed, seed_audit}.py; training/original_grid/config_audit.py needs
the checkpoints. attack_transfer.py, phase5_tost_5fam.py, camera_ready_stats_r6.py,
copyright_sensitivity.py and difficulty_stratification.py read the raw files when they are
present and otherwise use data/evaluator_labels/.
Training and attack generation need GPUs, the base models and the benchmark data. The scripts in
training/ and evaluation/ are the ones we ran on Purdue's Gilbreth cluster. They are reference
implementations rather than a turnkey pipeline, so expect to adapt SLURM partitions, environments
and paths.
training/README.md: fine-tuning scripts and per-cell recipes for the original grid, the controlled seed-42 grid and the recipe-robustness arms. The original grid is not recipe-uniform;training/original_grid/RECIPE.mdlists the seed, dropout, optimizer and framework of every cell.evaluation/README.md: attack drivers, evaluator wrappers, our patch to HarmBench (harmbench.patch, against upstream commit8e1604d) and which script produced which result.
data/human_annotation/asr_500_labels.csv and orr_250_labels.csv have one row per annotated
sample: sample metadata (attack, benchmark, model_family, config, behavior_id, HarmBench
categories, selection_group, sampling_stratum), the two annotators' blind labels (annotator_A,
annotator_B), agree, consensus (the shared label where the annotators agree, empty otherwise)
and the evaluators' labels (hbcls_label, gpt4omini_label, llamaguard3_label,
majority_label). For ASR, 1 means the completion is harmful. For over-refusal, 1 means the benign
request was answered and 0 means it was refused.
The samples were drawn to over-represent evaluator disagreement (see Appendix F of the paper), so
the error rates describe contested cases rather than the full evaluation grid. One raw value of
annotator_B is out of range (sample asr_234, recorded as 4). It was a data-entry error;
regime_a_evaluator_calibration.py corrects it to 1 and logs the change, and the sample counts as a
disagreement either way. The completions themselves are not included.
We release eight of the evaluated configurations as gated repositories on the Hugging Face Hub. Accept the access terms on a model page before downloading. The base configurations are the unmodified public models.
| Repository | Base model | Method | Grid |
|---|---|---|---|
| taleef/Llama-3.1-8B-Instruct-Alpaca-LoRA-seed42 | meta-llama/Llama-3.1-8B-Instruct | LoRA (merged) | controlled seed-42 |
| taleef/Llama-3.1-8B-Instruct-Alpaca-FFT-seed42 | meta-llama/Llama-3.1-8B-Instruct | FFT | controlled seed-42 |
| taleef/gemma-2-9b-it-Alpaca-LoRA | google/gemma-2-9b-it | LoRA (merged) | original |
| taleef/gemma-2-9b-it-Alpaca-FFT | google/gemma-2-9b-it | FFT | original |
| taleef/phi-4-Alpaca-LoRA | microsoft/phi-4 | LoRA (merged) | original |
| taleef/phi-4-Alpaca-FFT | microsoft/phi-4 | FFT | original |
| taleef/Qwen3-4B-Instruct-2507-Alpaca-LoRA | Qwen/Qwen3-4B-Instruct-2507 | LoRA (merged) | original |
| taleef/Qwen3-4B-Instruct-2507-Alpaca-FFT | Qwen/Qwen3-4B-Instruct-2507 | FFT | original |
These models are less safe than their base models. They are released to reproduce and extend this safety evaluation and should not be deployed. Each model card lists the training recipe and the model's HB-Cls attack success rates.
Model completions and attack prompts (PAIR, AutoDAN and GCG test cases, DeepInception and ArtPrompt
prompts, ActorAttack dialogues) contain harmful text, so they are not in this repository. That
includes the text of the human-annotated samples. Researchers can request them under controlled
access by emailing tamst01@pfw.edu or jrusert@pfw.edu with their affiliation and intended use. The
public files are aggregate tables, binary labels keyed by HarmBench and JailbreakBench behavior IDs,
and code. To read the behavior text, join the IDs against the public HarmBench and JailbreakBench
behavior files (see data/evaluator_labels/README.md).
- Code in this repository: MIT (see
LICENSE). - Checkpoints: each one inherits its base model's license, which its model card states (the Llama 3.1 Community License for Llama-derived models, marked "Built with Llama"; the Gemma Terms of Use; Apache 2.0 for Qwen3; MIT for Phi-4). The fine-tuning data derive from Stanford Alpaca (CC BY-NC 4.0), so all Alpaca-derived checkpoints are for non-commercial research use only. Dolly-15k, used in the recipe-robustness arms, is CC BY-SA 3.0.
- HarmBench is MIT-licensed, and
evaluation/harmbench.patchis distributed under the same terms. JailbreakBench behaviors, OR-Bench and ActorAttack keep their original licenses.
If you use the code, data or checkpoints, please cite:
@inproceedings{tamsal2026adaptation,
title = {Adaptation, Not Algorithm: {LoRA} and Full Fine-Tuning Show Comparable Black-Box
Jailbreak Degradation in Five Open-Weight {LLMs}},
author = {Tamsal, Taleef and Rusert, Jonathan},
booktitle = {Findings of the Association for Computational Linguistics: AACL-IJCNLP 2026},
year = {2026}
}GitHub's "Cite this repository" button reads the same entry from CITATION.cff.
Computation for this work was carried out on the Gilbreth community cluster operated by the Rosen Center for Advanced Computing (RCAC) at Purdue University.
For questions about the code or data, open an issue or email Taleef Tamsal (tamst01@pfw.edu).