Skip to content

About

An evaluation framework for Text to SQL tasks

Resources

Stars

14 stars

Watchers

1 watching

Forks

Repository files navigation

NL2SQL Benchmark

Overview

This repository contains the code for the LLM NL2SQL Benchmark.

Installation

Install the runtime dependencies with Python 3.10+:

python -m pip install -r requirements.txt
cp .env.example .env

Set OPENAI_API_KEY and, when needed, OPENAI_API_BASE_URL in .env.

Running

Our evaluation framework supports almost all models that can be downloaded from HuggingFace and Spider dataset and Bird dataset. You can run the benchmark with the following command and specify the model and dataset.

python main.py -m [method_name] -d [dataset_name] -l [language_model_name] -s [schema_linking_strategy]

Supported datasets are bird and spider. Final results use output/<dataset>/<method>/<vendor>/<model>/<schema_linker>/. Git allows dev_pred.sql, dev_pred_gt.sql, eval_result_dev.json, and summary.json at that path; intermediate traces, checkpoints, and logs remain ignored.

Export a completed full BIRD run from the batch runner without running inference or evaluation again:

python scripts/export_benchmark_results.py --run-dir output/bird/qwen3.8-27b-full-20261006

The exporter checks the dataset and saved evaluations, preserves the existing eval_result_dev.json format, and includes generation settings and metric limitations in each method's summary.json. It refuses to overwrite different results already saved at the destination.

Repository layout

  • model/, language_model/, schema_linking/: benchmark strategies and model adapters
  • tools/: SQL execution and analysis helpers
  • evaluation/: official-style BIRD and Spider evaluators
  • scripts/: standalone data preparation and inspection utilities
  • tests/: dependency-light regression tests

Run the test suite with python -m unittest discover -v. Historical and standalone areas, plus proposed deletion candidates, are documented in docs/REPOSITORY_AUDIT.md.

Evaluation Result

Spider Dataset

Spider easy medium hard extra all
count 248 446 174 166 1034
ACCURACY
Code Qwen2.5 7B 0.935 0.877 0.759 0.584 0.824
Qwen3 7B 0.879 0.812 0.672 0.524 0.758
Code Qwen2.5 32B 0.827 0.841 0.718 0.548 0.770
DeepSeek-R1 70B 0.774 0.796 0.603 0.482 0.708
Qwen2.5 72B 0.915 0.863 0.724 0.633 0.815
DeepSeek-R1 0.863 0.787 0.736 0.578 0.763

Bird Dataset

Bird simple moderate challenging total
count 925 465 144 1534
ACCURACY
Qwen2.5 Code 7B 56.11 34.48 33.79 47.46
Qwen3 7B 59.89 40.30 29.66 51.11
DeepSeek-V2 Code 16B 50.38 30.82 17.24 41.33
DeepSeek-R1 32B 42.27 18.10 17.93 32.66
QwQ 32B 58.92 40.30 35.86 51.11
Qwen2.5 32B 61.08 39.87 33.10 52.02
Qwen2.5 Code 32B 64.76 47.84 41.38 57.43
DeepSeek-R1 70B 48.97 27.37 23.45 40.03
Qwen2.5 72B 60.00 39.44 35.86 51.50
DeepSeek-R1 671B 60.11 42.24 37.93 52.61

The Impact of Schema Linking on NL2SQL

The performance of LLMs on NL2SQL is highly sensitive to the schema linking strategy. We provide the performance of different schema linking strategies on the Bird dataset with Qwen2.5-Coder-32B-Instruct model. The performance is calculated by the accuracy of the SQL query.

Bird simple moderate challenging total
count 925 465 144 1534
ACCURACY
Vanilla 64.76 47.84 41.38 57.43
Gold Linker 68.54 48.28 42.07 59.91

The Impact of Different Methods on NL2SQL

Qwen2 72B

Bird simple moderate challenging total
count 925 465 144 1534
ACCURACY
Base 59.89 40.95 38.62 52.15
Base (gold) 65.84 46.12 40.69 57.50
Vote 60.76 46.34 40.00 54.43
Rule 36.43 18.53 15.17 29.01
CoT 39.89 19.18 24.83 32.20
Column Desc. 28.76 13.58 19.31 23.27
Column Desc. (gold) 65.41 43.32 36.55 56.00
Sketch 9.19 4.53 8.28 7.69
Sketch (gold) 60.76 37.50 30.34 50.85

Qwen2.5 Coder 7B

Spider easy medium hard extra all
count 248 446 174 166 1034
ACCURACY
Base 93.5 87.7 75.9 58.4 82.4
Vote 94.4 89.7 82.8 63.9 85.5
Rule 92.7 83.9 70.1 57.2 79.4
CoT 90.7 84.3 69.5 49.4 77.8
Sketch 89.9 80.5 67.8 40.4 74.2

Qwen2.5 Coder 7B

Bird simple moderate challenging total
count 925 465 144 1534
ACCURACY
Base 56.11 34.48 33.79 47.46
Vote 61.30 44.18 38.62 53.98
Base (gold) 66.16 42.67 38.62 56.45
Rule 54.27 37.72 31.03 47.07
CoT 55.89 36.85 29.66 47.65
Column Desc. 49.84 25.65 24.83 40.16
Column Desc. (gold) 62.27 36.85 30.34 51.56
Sketch 53.51 34.05 30.34 45.44
Sketch (gold) 62.38 35.99 35.86 51.89

Qwen3.8-27B

BIRD dev execution accuracy (EX, %) with VanillaLinker, temperature 0, thinking disabled, and a maximum of 4,096 output tokens. These results evaluate the methods implemented in this repository, including correction and selection components. Counts reflect the difficulty labels in this run's dev.json.

Bird simple moderate challenging total
count 925 464 145 1534
ACCURACY (EX)
Base 70.27 57.11 49.66 64.34
CoTVerifier 70.49 58.19 52.41 65.06
CoVe 70.59 57.33 52.41 64.86
DACCorrection 70.59 57.11 52.41 64.80
CHESSUnitTester 70.92 56.68 51.03 64.73
DINSQLCorrection 70.49 58.84 53.10 65.32
SelfDebugging 72.86 59.70 55.17 67.21
MCSSelector 70.70 55.82 54.48 64.67
SQLPromptMix 71.24 57.33 49.66 64.99

Methods Comparison

Bird simple moderate challenging total
count 925 465 144 1534
ACCURACY
Base (gpt-4o) 60.22 40.09 31.72 51.43
Prompt (gpt-4o) 57.51 38.36 29.66 49.09
Hero-S (gpt-4o) 60.22 43.97 37.93 53.19
Hero-D (gpt-4o) 59.24 42.03 40.69 52.28
Hero-L (gpt-4o) 58.38 39.87 32.41 50.33
ACCURACY
Base (qwen-plus) 66.16 50.65 49.66 59.91
Prompt (qwen-plus) 64.00 48.28 49.66 57.89
Hero-S (qwen-plus) 65.84 53.02 51.72 60.63
Hero-D (qwen-plus) 66.81 54.31 51.34 61.47
Hero-L (qwen-plus) 64.97 50.22 48.28 58.93

About

An evaluation framework for Text to SQL tasks

Resources

Stars

14 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages