Does fine-tuning actually improve text-to-SQL? I measured it instead of assuming.
strict execution accuracy on 453 held-out questions
Same prompt, same schema, same database, same executor. Only the weights changed.
| # | configuration | strict accuracy | executable SQL | hallucination |
|---|---|---|---|---|
| 1 | Base Qwen3-8B | 10.82 % | 98.90 % | 0.66 % |
| 2 | Base + schema retrieval | 9.27 % | 92.27 % | 3.31 % |
| 3 | Fine-tuned (QLoRA) | 50.99 % | 95.81 % | 1.99 % |
| 4 | Fine-tuned + retrieval | 41.72 % | 94.92 % | 3.53 % |
| 5 | Fine-tuned + self-correction | 52.10 % | 98.23 % | 0.66 % |
Trained on one free Kaggle T4 in 4 h 53 m. 43.6 M trainable parameters — 0.917 % of the model.
Important
Most of the +40 points is the model learning column conventions, not better SQL reasoning.
"Right rows, wrong columns" went 159 → 0, while genuinely wrong row sets only fell 240 → 203.
I could only see that because the harness tracks projection-tolerant accuracy separately.
The hard tier (+81.2 pp) is where reasoning genuinely improved.
flowchart TD
Q["Business question<br/><i>Who are the top 15 customers by revenue?</i>"] --> P[Prompt + full schema<br/>frozen template]
P --> M["Qwen3-8B + QLoRA adapter"]
M --> S[Generated SQL]
S --> V{"Static validation<br/>parseable? one statement?<br/>read-only? real tables?"}
V -->|rejected| R
V -->|passes| X{"PostgreSQL<br/>read-only, 30s timeout"}
X -->|error| R["Self-correction<br/>database error fed back"]
X -->|success| OUT([Rows])
R --> M2["Qwen3-8B + adapter<br/>repair prompt"]
M2 --> X2{"PostgreSQL"}
X2 -->|success| OUT
X2 -->|still fails| FAIL([Reported as failed])
style M fill:#c8623a,color:#fff
style M2 fill:#c8623a,color:#fff
style OUT fill:#2f7d52,color:#fff
style FAIL fill:#b3402f,color:#fff
|
All 453 predictions. Filter to the ones fine-tuning fixed — or broke. |
Copy-paste loading snippet.
|
load_dataset(
"hari-krishna-ai/"
"enterprise-text-to-sql-benchmark") |
Correctness is judged by execution, never by string similarity
These are different strings and identical in meaning. Any honest benchmark must count both correct:
SELECT COUNT(*) FROM customers WHERE country = 'India';
SELECT COUNT(customer_id) FROM customers WHERE country = 'India';Every prediction is executed against PostgreSQL and its result set fingerprinted (MD5 over
sorted, stringified rows). Row order is compared only where the gold query has an ORDER BY.
Guardrails that keep the number honest
| guardrail | what it prevents |
|---|---|
| Splits by template equivalence group — 74 train / 52 val / 48 test, zero overlap | a paraphrase of a test question appearing in training |
Prompt fingerprint 8288e41a496531a9 asserted at every stage |
measuring prompt engineering instead of fine-tuning |
Schema fingerprint d03619e711661bc5 asserted at every stage |
comparing runs that saw different databases |
| The GPU host that generates never receives gold SQL | a "model" that copies the answer |
| Oracle self-test: gold SQL through the scorer must return 100 % | a broken evaluator silently deflating every result |
DATA_AS_OF constant instead of NOW() |
a benchmark whose answers change overnight |
| Retrieval tuned on validation, measured once on test | hyperparameters fitted to the test set |
Two results that went against expectations — and are reported anyway
Schema retrieval made things worse. Twice.
Showing only the retrieved tables cost the base model 1.55 pts and the fine-tuned model 9.27.
The adapter only ever saw full-schema prompts, so subsets are out of distribution for it — and the
damage lands on hard (−20.8) and enterprise (−13.1), the tiers that need joins. Drop a table a
join needs and the model invents one.
With 12 tables the whole schema is 5,455 characters. Retrieval solves a problem this database does not have.
Self-correction: two rates, not one.
| repair success — now executes | 11 / 19 | 57.9 % |
| repair correctness — returns the gold rows | 5 / 19 | 26.3 % |
Six of the eleven "successes" turned a crash into a confident wrong answer. Reporting one rate
would have claimed 57.9 % and hidden that. Repair works when the error names the fix
(column "method" does not exist → rename to payment_method) and fails when it names only the
symptom (ambiguous column reference).
Accuracy by difficulty — where reasoning actually improved
| tier | n | base | fine-tuned | Δ |
|---|---|---|---|---|
| easy | 126 | 0.0 % | 23.0 % | +23.0 |
| medium | 63 | 52.4 % | 52.4 % | +0.0 |
| hard | 144 | 3.5 % | 84.7 % | +81.2 |
| very_hard | 36 | — | 8.3 % | — |
| enterprise | 84 | 13.1 % | 52.4 % | +39.3 |
easy was 0 % at baseline entirely because of the projection problem — "show orders in 2023"
never says which columns to return. medium did not move at all, and that is unexplained rather
than glossed over.
QLoRA on one free T4 — and the detail that mattered most
| base | Qwen/Qwen3-8B @ b968826d9c46, 4-bit NF4 + double quant |
| adapter | LoRA r=16, α=32, dropout 0.05, all 7 attention + MLP projections |
| trainable | 43,646,976 of 4,761,498,624 — 0.917 % |
| loss | completion-only, prompt masked to -100 |
| supervised token share | 3.01 % |
| train / eval loss | 4.10 → 0.006 / 0.475 → 0.335 |
| hardware | one free Kaggle T4, 1 epoch, 4 h 53 m |
Why completion-only loss is the whole game here. Each example is ~1,490 prompt tokens and ~46 completion tokens, and the prompt is ~97 % schema — byte-identical across all 2,133 examples. Training on the full sequence sends ~97 % of the gradient into memorising a schema the model is handed at inference. The loss curve would look excellent throughout, because predicting a constant is easy, while the ability you care about barely moved.
Three safety layers — none of which trust the model
The service executes text a language model wrote. These hold regardless of what it writes:
- The database role is not a superuser. It owns one database and nothing else.
- Every session is read-only with a statement timeout, applied on pool checkout rather than once at creation — a reset session cannot silently lose the guarantee.
- SQL is statically rejected unless it is a single read-only statement over tables that exist.
32 adversarial tests attack them. Two attacks get past the validator — pg_read_file (a single
read-only SELECT over no tables) and writes smuggled through a CTE (parses as SELECT). The role
permissions and the read-only transaction stop them respectively. That is what defence-in-depth
means, and it is now evidence rather than an argument.
A release gate, not just a benchmark
scripts/release_gate.py exits non-zero and blocks a release. Improvement alone is not enough:
[PASS] accuracy gain clears the bar +40.17 pp (need >= 5.0)
[PASS] executable SQL did not regress too far -3.09 pp (tolerance 4.0)
[PASS] schema hallucination did not rise too far +1.33 pp (tolerance 2.0)
[PASS] no difficulty tier collapsed 5 tiers compared
RELEASE APPROVED
Verified in both directions — it blocks configuration 4 on hallucination (+2.87 pp against a
2.0 tolerance) despite its +30.90 pp accuracy gain, and caught a medium-tier collapse of
−7.9 pp that the mean alone would have hidden.
python -m venv env && env\Scripts\activate # Windows
pip install -r requirements.txt
cp .env.example .env # then fill in credentials
python scripts/init_db.py # database + non-superuser role
python scripts/load_schema.py
python scripts/generate_data.py # ~328k rows, fixed seed, ~1 min
python -m pytest tests/ -q # 149 tests
python -m uvicorn src.api.main:app --reload # http://localhost:8000Reproducing the experiment
# harness self-test — no model calls, must score 100 %
python scripts/run_baseline.py --oracle --sample 60 --tag oracle
# the frozen baseline
python scripts/run_baseline.py --provider featherless-ai --workers 6
# export what a GPU host needs (questions only; gold SQL never leaves)
python scripts/export_eval_pack.py
python scripts/export_eval_pack.py --retrieval keyword
# score predictions generated on the GPU host
python scripts/score_finetuned.py --predictions predictions.jsonl
python scripts/score_repair.py --repairs repairs.jsonl
# assemble the ablation and gate the release
python scripts/ablation_report.py
python scripts/release_gate.pyGeneration needs a GPU; scoring needs PostgreSQL. They live on different machines, so the eval pack
carries {id, question} only — the GPU host never sees a gold answer, asserted at both ends.
src/
├── api/ FastAPI service + demo UI
├── evaluation/ benchmark runner, 11-class outcome taxonomy, reporting
├── model/ frozen prompts, schema context, model backends
├── retrieval/ lexical schema retriever
└── sql/ validator, read-only executor, repair loop
database/ schema.sql, ERD.md
dataset/ generation, validation, SFT formatting, splits
training/ QLoRA training code + Kaggle notebooks
experiments/ frozen results for all 5 configurations
scripts/ operational entry points
deploy/ docker/ Space, Neon + Render, container images
tests/ 149 tests (93 database, 24 API, 32 adversarial)
Stated plainly, because they bound what the number means.
- Every test question came from the same generator as training. Real users write abbreviations, typos and genuinely ambiguous requests. This is the biggest caveat on 52.10 %.
- Roughly half the answers are still wrong.
- One epoch, one seed, one run. No variance estimate.
- Schema-specific. It learned this database's conventions — which is most of the gain.
- Silent wrong answers are the real risk, and self-correction cannot help: a query that runs and returns the wrong rows raises no error to feed back.
docker buildhas never been run — Docker is not installed on the development machine.scripts/verify_docker_image.pyverifies the image contents and entrypoint instead (27 checks).