We introduce DiagDBench, a large-scale benchmark for training and evaluating database agents on interactive diagnosis, anomaly remediation, and optional workload performance optimization across diverse, realistic anomaly scenarios.
To construct these samples at scale, we propose an LLM-based framework with two complementary pipelines: Manual-Guided Anomaly Synthesis, which sequentially synthesizes user requests, database modifications, and anomaly-injection scripts grounded in DBA manuals and official DBMS documentation; and Incident-Guided Anomaly Reproduction, which reconstructs anomaly samples end to end from real-world database incident records.
📄 Research Paper · arXiv:2611.07339
🗄️ Dataset, Database Instances & Knowledge Base · DiagDBench on Hugging Face
- Repository Layout
- Manual-Guided Anomaly Synthesis
- Incident-Guided Anomaly Reproduction
- DiagDBench
- Database Agents for Diagnosis and Remediation
- Evaluation
This repository supports a four-stage workflow: anomaly sample generation, dataset construction, database agent execution, and evaluation of diagnostic reports and remediation scripts.
Database instances, sample archives, and the synthesis knowledge base are
distributed through the Hugging Face dataset rather than included in a fresh
GitHub checkout. Downloaded knowledge assets belong under
Anomaly_Def/Knowledge_Base/ and are ignored by Git.
DiagDBench/
├── README.md
├── Overview.png
├── Anomaly_Def/ # Anomaly definitions and synthesis support
│ └── Knowledge_Base/ # Download separately from Hugging Face; not tracked by Git
├── Baselines/ # Database diagnosis agents and shared tools
│ ├── ReAct/ # Agent that repeats reasoning, actions, and observations
│ ├── Codex/ # Codex CLI integration, database tools, and runtime checks
│ ├── Claude_Code/ # Claude Agent SDK integration using shared Codex tools
│ ├── DB-GPT/ # Multi-expert diagnosis agent adapted for this benchmark
│ ├── DBAIOps/ # Knowledge-based diagnosis service and browser extension
│ ├── input_contract.py
│ ├── preparation.py
│ ├── container_boundary.py
│ ├── native_runtime.py
│ ├── recovery_artifacts.py
│ └── script.sh # Starts a diagnosis agent from a configuration file
├── Evaluation/ # Runs diagnosis and measures recovery quality
│ ├── test_baseline_eval.py # Runs an agent and scores its diagnosis and recovery
│ ├── test_eval.py # Scores existing reports, recovery scripts, and workloads
│ ├── method_adapters.py
│ ├── measurement.py
│ ├── namespace.py
│ ├── recovery_reassessment.py
│ └── offline_rescore.py
├── Configs/ # Example configurations to copy and customize before running
├── Filter/ # Filters samples using rules or an LLM agent; includes configurations
├── Revision/ # Revises samples based on human review comments
├── Docker/ # Builds and starts PostgreSQL and MySQL containers
├── Scripts/ # Initializes/resets databases, checks anomalies, and reports database health
├── Skills/ # Instructions for generating, reproducing, and revising anomalies
├── DiagDBench/ # Sample download guide; archives hosted on Hugging Face
├── Data/ # Local schema directory (not tracked by Git)
├── Database_Instances/ # Database download guide; instances hosted on Hugging Face
├── test_syn_*.py # LLM-based anomaly synthesis scripts
├── test_extract_*.py # Collects generated samples, sample indexes, and diagnosis outputs
├── test_retri*.py # Retrieve relevant knowledge for anomaly definitions
├── test_stat_key_info.py # Summarizes sample counts and characteristics
├── docker_func.py # Manages containers, runs database commands, and limits heavy I/O tasks
├── llm_api.py # LLM API function call wrapper
├── utils.py # Common helpers, including splitting SQL into executable statements
├── prompt_template.py # Prompt templates for the generation and diagnosis workflows
└── requirements.txt # Python packages required by the core workflows
See the agent documentation, evaluation documentation, and revision documentation for setup and usage instructions.
Use Linux, a compatible Python environment, Docker, Bash, tmux, and the
nsenter/unshare/setpriv tools required by the isolation paths. Prepare the
necessary host permissions; do not disable isolation checks to bypass a failed
preflight. From the repository root:
python -m pip install -r requirements.txt
docker build -t pg_diag \
-f Docker/Ubuntu_PG/Dockerfile.postgresql Docker/Ubuntu_PG
docker build -t mysql_diag \
-f Docker/Ubuntu_MySQL/Dockerfile.mysql Docker/Ubuntu_MySQLThe image definitions target PostgreSQL 16 and MySQL 8.4. Database instances,
training/evaluation archives, and the synthesis knowledge base are distributed
through Hugging Face.
Database_Instances/ and DiagDBench/ contain download guides and also serve
as local download destinations. From the repository root:
python -m pip install -U huggingface_hub
# Download complete PostgreSQL dumps and MySQL loading packages.
hf download zhaoxinxin798/DiagDBench --repo-type dataset \
--include "Database_Instances/**" --local-dir .
# Download the training and evaluation archives.
hf download zhaoxinxin798/DiagDBench DiagDBench/Train.zip DiagDBench/Eval.zip \
--repo-type dataset --local-dir .These commands preserve the paths expected by the database initialization code and do not overwrite this repository's README.
For knowledge retrieval and manual-guided synthesis, also download the
knowledge base. Run the following from the repository root; Hugging Face
preserves the Knowledge_Base/ directory name under Anomaly_Def/:
hf download zhaoxinxin798/DiagDBench --repo-type dataset \
--include "Knowledge_Base/**" --local-dir Anomaly_DefThe resulting directory is Anomaly_Def/Knowledge_Base/, containing
error_code_list.json, timeout_list.json, PG_Retrieval/, and
MySQL_Retrieval/.
The error-code and timeout corpora are collected and structured from the
official PostgreSQL and MySQL websites and documentation. We make these
curated resources openly available as a database-diagnosis knowledge base
for community use, supporting knowledge retrieval, anomaly analysis, and
further research. PG_Retrieval/ and MySQL_Retrieval/ contain retrieval
results generated by this project from those corpora.
See database instance downloads and dataset downloads for details. Allow approximately 13.61 GB for the database instances and sample archives, plus 8.76 GB (8.15 GiB) for the full knowledge base: about 22.37 GB in total before sample extraction and database initialization. Schema files and sufficient CPU and memory must also be prepared.
Core requirements do not install every third-party baseline runtime:
- Codex needs its native CLI and the runtime checked by its preflight.
- Claude Code uses its Agent SDK and an endpoint compatible with that integration.
- DB-GPT and DBAIOps need their own configured Conda environments; DB-GPT also needs its native knowledge/embedding dependencies and resources.
| Workflow | Configuration | Entry |
|---|---|---|
| Standalone diagnosis | Configs/baselines/<method>/example.json |
Baselines/script.sh or the method's Python entry |
| Diagnosis + scoring | Configs/evaluation/<method>/example.json |
Evaluation/test_baseline_eval.py |
| Score existing artifacts | Configs/evaluation/scoring/example.json |
Evaluation/test_eval.py |
| User request / data modification / anomaly synthesis | Configs/syn_int/, Configs/syn_data/, Configs/syn_ano/ |
Corresponding root test_syn_*.py |
| Incident reproduction | Configs/syn_ano_rep/example.json |
test_syn_ano_rep.py |
| Extract synthesized samples | Configs/extract_key_info/example.json |
test_extract_key_info.py |
| Human-review revision | Configs/revision/example.json, example_mysql.json |
Revision/test_ref_anomaly.py |
| Rule-based / agentic filtering | Filter/Configs/example_rule.json, example_agentic.json |
Corresponding Filter/test_*.py |
This pipeline turns abstract anomaly definitions into executable samples through three sequential stages, using error-code and timeout information from official DBMS documentation. Each stage uses database execution feedback to refine its outputs.
| Stage | Entry point | Configuration | Main outputs |
|---|---|---|---|
| User request | test_syn_int.py |
Configs/syn_int/example.json |
User issue description and verification workload |
| Database modification | test_syn_data.py |
Configs/syn_data/example.json |
Data-modification SQL and database feature summary |
| Anomaly injection | test_syn_ano.py |
Configs/syn_ano/example.json |
Injection script and anomaly-validation evidence |
| Sample extraction | test_extract_key_info.py |
Configs/extract_key_info/example.json |
Sample directories and key_information.json |
Prepare database schemas, seed data, anomaly definitions, and a private model
configuration. Anomaly_Def/ contains the single/composite definitions and
syn_comp_anomaly.py, which constructs composite definitions. Download the
separately distributed Knowledge_Base/, curated from official DBMS
documentation and shared for community use, as described in
Environment Preparation before running retrieval.
test_retri.py and test_retri_top_n.py retrieve and enrich anomaly knowledge.
From the repository root, run retrieval first, then expand the definitions with
the retained matches. Set ANOMALY_DIR to your prepared input directory: each
<id>.json should contain one anomaly definition, either as an object or wrapped
in a single-element list. The scripts' default Anomaly_Def/anomaly_def_formatted
directory is not included in this checkout.
ANOMALY_DIR="/absolute/path/to/formatted_anomalies"
# PostgreSQL: retrieve error-code and timeout knowledge, then expand definitions.
python -B test_retri.py \
--dbms_name "PostgreSQL 16.11" \
--anomaly_dir "$ANOMALY_DIR" \
--output_dir Output/retrieval/PG_Retrieval
python -B test_retri_top_n.py \
--n 4 \
--anomaly_dir "$ANOMALY_DIR" \
--retrieve_dir Output/retrieval/PG_Retrieval \
--output_dir Output/retrieval/anomaly_def_pg_rag
# MySQL: keep its retrieval results and expanded definitions in separate directories.
python -B test_retri.py \
--dbms_name "MySQL 8.4" \
--anomaly_dir "$ANOMALY_DIR" \
--output_dir Output/retrieval/MySQL_Retrieval
python -B test_retri_top_n.py \
--n 4 \
--anomaly_dir "$ANOMALY_DIR" \
--retrieve_dir Output/retrieval/MySQL_Retrieval \
--output_dir Output/retrieval/anomaly_def_mysql_ragRun the pair for your target DBMS and use definitions applicable to that system.
Retrieval reads Anomaly_Def/Knowledge_Base/error_code_list.json and
Anomaly_Def/Knowledge_Base/timeout_list.json, which contain structured
error-code and timeout knowledge for PostgreSQL 16.11 and MySQL 8.4.
The download also includes precomputed PG_Retrieval/ and MySQL_Retrieval/
results and their retrieval-count files. Reuse these results only when their
anomaly definitions and filename mapping match your inputs; otherwise run
retrieval again, keeping new results under Output/ as shown above.
The expansion script matches input and retrieval filenames,
takes up to --n retained items (error-code matches followed by timeout matches),
and writes one expanded definition per item. Use the resulting directory as the
anomaly-definition input in the synthesis configuration.
cp Configs/syn_int/example.json syn-int.local.json
cp Configs/syn_data/example.json syn-data.local.json
cp Configs/syn_ano/example.json syn-ano.local.json
cp Configs/extract_key_info/example.json extract-key-info.local.json
# Fill schema, anomaly-definition, database, and model settings before starting.
python -B test_syn_int.py --config syn-int.local.json
# Set syn_int_directory to the generated user-request run directory.
python -B test_syn_data.py --config syn-data.local.json
# Set syn_int_directory and syn_data_directory to the corresponding run directories.
python -B test_syn_ano.py --config syn-ano.local.json
# Set synthesized_anomaly_directory to the generated anomaly run directory.
python -B test_extract_key_info.py --config extract-key-info.local.jsonReplace each REPLACE_WITH_GENERATED_RUN_DIRECTORY placeholder with the actual
output of the preceding stage. Set the extraction output_directory to a location
outside the repository when the samples will be used for isolated diagnosis or
evaluation. The resulting sample contract is described in DiagDBench.
This pipeline reconstructs an anomaly directly from a real-world database
incident record, using the database schema, system and user configuration,
and reproduction guidance.
test_syn_ano_rep.py runs an interactive construction loop to produce the anomaly
definition, user issue description, anomaly-verified workloads, database modifications, and anomaly
injection script.
Create a private copy of Configs/syn_ano_rep/example.json. Supply
incident_record_path, database_schema_directory, database/model settings, and
an output directory. Guidance is loaded from skills_directory, including
the incident-category and reproduction playbook files in Skills/.
Private incident datasets differ in their record structure and available evidence.
Adapt test_syn_ano_rep.py to the attributes of each source before reproduction:
- File format and record layout:
read_incident_record_csvcurrently reads only the first data row of a CSV file.build_incident_sample_listdiscovers numeric<id>.csvfilenames in batch mode. For multi-record CSV, JSON, or other exports, normalize the data to this layout or adjust these readers and the sample-ID mapping. - Field names and nested content: update
INCIDENT_FIELD_LABELSandformat_incident_record_infoto map the source's symptoms, impact, investigation notes, conclusions, and classification fields. Adapt the parsing of nested values such assummary,category_by_customer, andfault_locator_msg, including missing fields and encoding differences. - Incident meaning and reproduction criteria: align source-specific category
labels with the guidance in
Skills/, configure the corresponding schema and database environment, and select aSuccessCriterionsupported by the recorded evidence: performance degradation, blocking/timeouts, or execution errors. Extend the relevant validation logic if the source requires another criterion.
Changing the input path alone does not adapt an unfamiliar record schema. Inspect the formatted evidence for representative records before running a batch; retain the exclusion of tracking identifiers and raw snapshots when updating the field mapping.
cp Configs/syn_ano_rep/example.json syn-ano-rep.local.json
# Fill the incident record, schema, database, and model settings.
python -B test_syn_ano_rep.py --config syn-ano-rep.local.jsonFinal validation uses Scripts/anomaly_eval_rep.py. The declared
SuccessCriterion distinguishes performance degradation, blocking/timeouts, and
execution errors so that validation can match the incident's manifestation.
# Replace the sample path with an actual <id>_<database> reproduction directory.
python -B test_extract_key_info_rep.py \
--sample-dir /absolute/path/to/reproduction_run/1_tpch_1 \
--output-directory /absolute/path/outside/repository/reproduced_samplesThe extractor writes the sample directory and updates key_information.json.
It reads the run configuration from the sample's parent directory by default;
use --run-config to provide another path. Reproduced samples can then follow the
same curation and evaluation workflow as manually guided samples.
DiagDBench organizes samples from both construction pipelines into a common interface for interactive diagnosis and remediation. Each anomaly sample contains the database environment, user issue description, anomaly-verified workloads, database modifications, and anomaly-injection script.
The training and evaluation samples are distributed as DiagDBench/Train.zip
and DiagDBench/Eval.zip in the
Hugging Face dataset.
The archives contain 6,940 training samples and 454 evaluation samples.
Download them using the commands in
Environment Preparation.
Extract samples to a directory outside this repository when preparing isolated diagnosis or evaluation inputs, and configure the sample/index paths accordingly. Downloading or extracting the archives does not initialize databases or execute anomaly-injection scripts.
Train.zip contains Train/MySQL/ and Train/PostgreSQL/ sample directories.
Eval.zip contains an Eval/ directory and Eval/key_information.json,
with per-sample filenames such as:
anomaly_definition.jsonanomaly_verified_workload.sqluser_issue_description.txt.inject_entry, invoked throughanomaly_inject.sh
Use Filter/ to check sample validity and recoverability:
| Filter | Entry point | Configuration | Behavior |
|---|---|---|---|
| Rule-based | Filter/test_filter_syn_data.py |
Filter/Configs/example_rule.json |
Measures anomaly validity without an LLM |
| Agentic | Filter/test_agentic_llm_filter.py |
Filter/Configs/example_agentic.json |
Filters out recoverable samples (failed_ids) and retains unrecovered samples (pass_ids) |
Both entries accept --config and support --dry-run. Real filtering performs
database operations; the agentic filter also calls a model.
Revision/test_ref_anomaly.py refines samples using human review notes. It expects
wrapped Raw_Data/<dataset>/<id>/raw_sample/ inputs and
Human_Review/<dataset>/<three-digit-id>.txt, rather than the flat evaluation
layout. See the revision documentation.
cp Configs/revision/example.json revision.local.json
# Fill model/environment settings and prepare the review tree before running.
python -B Revision/test_ref_anomaly.py \
--config revision.local.json \
--input-root /absolute/path/to/review_input \
--dataset PG_Syn \
--output-root /absolute/path/to/revised_output \
--workers 1Use Configs/revision/example_mysql.json for MySQL. Only reviewed samples enter
the model revision loop. Successful revisions require anomaly evaluation and a
valid finish action; unreviewed or failed revisions keep the original sample
content. Keep revision output separate from the input tree.
Baselines/ implements five database agents with a shared input and output
contract. Agents use the user-observed issue, workloads, database account
information, and runtime observations to diagnose and remediate anomalies.
Method (workflow_type) |
Standalone entry | Diagnosis approach | Integrated scoring |
|---|---|---|---|
react |
Baselines/ReAct/react.py |
Explicit reasoning/action/observation loop | Yes |
codex |
Baselines/Codex/codex.py |
Native Codex CLI with controlled MCP tools | Yes |
claude_code |
Baselines/Claude_Code/claude_code.py |
Native Claude Agent SDK with controlled MCP tools | Not registered in the integrated dispatcher |
db_gpt |
Baselines/DB-GPT/db_gpt.py |
Native multi-expert diagnosis and recovery-artifact adapter | Yes; PostgreSQL only |
dbaiops |
Baselines/DBAIOps/dbaiops.py |
Knowledge-Q&A diagnosis and recovery-artifact adapter | Yes |
DB-GPT excludes unsupported non-PostgreSQL samples.
cp Configs/baselines/react/example.json react-baseline.local.json
# Fill the local configuration first.
python -B Baselines/ReAct/react.py --config react-baseline.local.json --dry-run
bash Baselines/script.sh react-baseline.local.jsonBaselines/script.sh performs configuration/port/container checks, saves a local
configuration snapshot and locks its output directory. It runs diagnosis only.
Use the corresponding Configs/baselines/<method>/example.json for another agent.
The Evaluation section covers scoring the generated artifacts.
Baseline completion writes:
diagnosis_report.md
solution_script.sh
optimized_verified_workload.sql
The final action's workload field is verified_workload.sql; the saver writes
optimized_verified_workload.sql. An empty optional rewrite retains the original
verification SQL as the effective artifact.
To collect existing standalone artifacts without running a model:
python -B test_extract_solution.py \
--solution-directory Output/baselines/react \
--output-directory Output/extracted_solutions/reactEvaluation/ implements two complementary evaluation protocols:
Execution-Grounded Remediation Evaluation and LLM-Judged Diagnostic
Report Evaluation. It measures recovery in a reconstructed anomalous
environment and uses an LLM judge to assess diagnostic reports and workload
equivalence.
ReAct, Codex, DB-GPT, and DBAIOps use Evaluation/test_baseline_eval.py. Select
the method with baseline.workflow_type; optional --method checks that this
selection matches the configuration. Integrated configs have baseline and
eval sections.
cp Configs/evaluation/react/example.json react.local.json
# Edit paths/model settings; supply OPENAI_API_KEY and JUDGE_API_KEY securely.
# No DB/model calls; requires a valid config and accessible sample inputs.
python -B Evaluation/test_baseline_eval.py \
--config react.local.json --method react --dry-run
# Check local runtime/isolation/native prerequisites.
python -B Evaluation/test_baseline_eval.py \
--config react.local.json --method react --preflight
# Real execution: creates containers and makes model calls.
python -B Evaluation/test_baseline_eval.py \
--config react.local.json --method react --sample-ids 0 1Use the corresponding example for another method. --dry-run and --preflight
are mutually exclusive; omit --sample-ids to process the configured index.
Use distinct worker port ranges, container prefixes, and output locations for
concurrent runs. Start with one worker and increase concurrency only after
checking resources and isolation prerequisites. Run DB-GPT benchmark jobs
sequentially because its native implementation uses shared intermediate files.
cp Configs/evaluation/scoring/example.json scoring.local.json
# Set the external sample index, solution_script_directory, output, and Judge.
python -B Evaluation/test_eval.py --config scoring.local.json --dry-run
python -B Evaluation/test_eval.py --config scoring.local.jsonStandalone scoring uses a flat configuration and does not start a diagnosis
agent. It still performs database operations and calls the Judge. Configure
ground_truth_diagnosis to match the per-sample reference filename.
Integrated outputs use <output_root>/<id>/baseline/ and <output_root>/<id>/eval/;
standalone scoring uses <output_base>/<id>/. Evaluation produces per-sample
score_detail.json, Judge/measurement evidence and logs, plus the aggregate
evaluation_result.json.
The evaluation workflow is as follows:
- Measure the original verified workload on a clean database.
- Reconstruct the database, apply the anomaly and verify injection.
- Execute the submitted recovery script and observe the configured settling time.
- Measure the submitted effective verification workload.
- Judge the diagnosis and ordered workload semantics using the shared evaluator.
The sample score is 0.3 * diagnosis_score + 0.7 * recovery_score.
