Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 29 of the 39 sources its claims cite were checked for support.

Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation

Open access
Gao, Dawei, Wang, Haibin, Li, Yaliang, Sun, Xiuyu, Qian, Yichen, Ding, Bolin, Zhou, Jingren · 2024 · Proceedings of the VLDB Endowment

Abstract

Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.

Overview

CoChat Summary

TL;DR

This paper systematically benchmarks prompt engineering for LLM-based Text-to-SQL across question representation, example selection, and example organization, and proposes DAIL-SQL, which reaches 86.6% execution accuracy on the Spider leaderboard. It also explores open-source LLMs and supervised fine-tuning, finding SFT substantially improves them but can degrade their in-context learning ability.

Key findings

  • DAIL-SQL refreshes the Spider leaderboard with 86.6% execution accuracy; without self-consistency it achieves 86.2%, and self-consistency adds 0.4% but is “very time consuming” and much more costly. The paper reports it surpasses the previous state of the art of 85.3% by 1.3% with much less token cost.
  • For question representation, OpenAI Demonstration Prompt (OD_P) fits all four evaluated LLMs and achieves 75.5% execution accuracy with GPT-3.5-TURBO; GPT-4 unexpectedly prefers the simple Basic Prompt (BS_P). For open-source LLMs, Code Representation Prompt (CR_P) achieves the best zero-shot result, 68.5% execution accuracy on Spider-dev.
  • For example selection, DAIL Selection generally outperforms other strategies; with GPT-4 in 5-shot it reaches 82.4% execution accuracy. For example organization, DAIL Organization with GPT-4 achieves 83.5% on Spider-dev and 76.0% on Spider-Realistic, while weaker LLMs often need Full-Information Organization.
  • Supervised fine-tuning greatly improves open-source LLMs: LLaMA-13B with Alpaca SFT Prompt reaches 68.6% execution accuracy on Spider, and LLaMA-33B with CR_P reaches 69.1%. However, after fine-tuning, adding contextual examples causes sudden decreases in accuracy, suggesting the models overfit to zero-shot prompts.

Methods

  • The paper conducts a systematic benchmark of prompt engineering for LLM-based Text-to-SQL: five question representations (BS_P, TR_P, OD_P, CR_P, AS_P), two prompt components (foreign key information and rule implication), four example selection strategies plus DAIL Selection and an Upper Limit reference, and three example organizations (Full-Information, SQL-Only, DAIL Organization) on four LLMs: GPT-4, GPT-3.5-TURBO, TEXT-DAVINCI-003, and Vicuna-33B.
  • Evaluation uses Spider and Spider-Realistic, with exact-set-match accuracy (EM) and execution accuracy (EX) as metrics; Spider-dev is used because the test split is not released. Few-shot examples are drawn from the Spider training split.
  • DAIL-SQL combines CR_P as question representation, DAIL Selection based on masked question similarity and query similarity with a threshold, and DAIL Organization, which keeps question-SQL pairs while removing token-cost database schema. The paper also fine-tunes open-source LLMs on the Spider training split (8659 samples) and evaluates zero-shot and few-shot performance.

Limitations

  • Paper-stated: only two rule implications are tested; open-source LLMs are fine-tuned with only the Spider training set; the databases in Spider and Spider-Realistic may not be large enough; the current evaluation metric prioritizes correctness over efficiency, leaving generation of efficient SQL among correct alternatives unexplored; LLMs larger than 33B are left as future work due to limited resources.
  • Text-evident: OpenAI LLMs are not fine-tuned because of unaffordable cost, so SFT conclusions are limited to open-source models. The paper also notes that fine-tuned LLMs fail to learn from contextual examples, and preserving in-context learning ability after fine-tuning remains an open question.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.14778/3641204.3641221

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

18 Valid · 9 Not valid · 8 Not checked · 140 Own results and interpretations

90 more claims · 5 with their cited sources checked

Create a free account to see all 175 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 53
  • It may state two separate claims as one. 38
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 20 references

Research works printed but not linked: 4

  1. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The 10th International Conference on Learning Representations.
  2. Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In 37th AAAI Conference on Artificial Intelligence. 13067--13075.
  3. Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-T5: Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing. In 37th AAAI Conference on Artificial Intelligence. 13076--13084.
  4. Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. LLAMA2: Open Foundation and Fine-Tuned Chat Models. CoRR (2023).

Cited by

See all 342

Metadata

Authors
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J.
Affiliations
Alibaba Group (China) (CN)
Venue
Proceedings of the VLDB Endowment
Published
2024-01
Volume
17
Issue
5
Pages
1132-1145
ISSN
2150-8097 (ISSN-L)
OpenAlex
W4396571402
Cited by
342
References
20
Citations by year
2026: 160 · 2025: 143 · 2024: 38
FWCI
50.11
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the VLDB Endowment (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
Keywords
text-to-SQL, large language models, prompt engineering, Spider benchmark, example selection, question representation, open-source LLMs, supervised fine-tuning, token efficiency, benchmark evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. https://app.cochat.ai/papers/10.14778/3641204.3641221
The paper
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J. (2024). Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment. https://doi.org/10.14778/3641204.3641221

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.