The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 29 of the 39 sources its claims cite were checked for support.
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
Open accessAbstract
Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions. To address this challenge, in this paper, we first conduct a systematical and extensive comparison over existing prompt engineering methods, including question representation, example selection and example organization, and with these experimental results, we elaborate their pros and cons. Based on these findings, we propose a new integrated solution, named DAIL-SQL, which refreshes the Spider leaderboard with 86.6% execution accuracy and sets a new bar. To explore the potential of open-source LLM, we investigate them in various scenarios, and further enhance their performance with supervised fine-tuning. Our explorations highlight open-source LLMs' potential in Text-to-SQL, as well as the advantages and disadvantages of the supervised fine-tuning. Additionally, towards an efficient and economic LLM-based Text-to-SQL solution, we emphasize the token efficiency in prompt engineering and compare the prior studies under this metric. We hope that our work provides a deeper understanding of Text-to-SQL with LLMs, and inspires further investigations and broad applications.
Overview
CoChat Summary
TL;DR
This paper systematically benchmarks prompt engineering for LLM-based Text-to-SQL across question representation, example selection, and example organization, and proposes DAIL-SQL, which reaches 86.6% execution accuracy on the Spider leaderboard. It also explores open-source LLMs and supervised fine-tuning, finding SFT substantially improves them but can degrade their in-context learning ability.
Key findings
- DAIL-SQL refreshes the Spider leaderboard with 86.6% execution accuracy; without self-consistency it achieves 86.2%, and self-consistency adds 0.4% but is “very time consuming” and much more costly. The paper reports it surpasses the previous state of the art of 85.3% by 1.3% with much less token cost.
- For question representation, OpenAI Demonstration Prompt (OD_P) fits all four evaluated LLMs and achieves 75.5% execution accuracy with GPT-3.5-TURBO; GPT-4 unexpectedly prefers the simple Basic Prompt (BS_P). For open-source LLMs, Code Representation Prompt (CR_P) achieves the best zero-shot result, 68.5% execution accuracy on Spider-dev.
- For example selection, DAIL Selection generally outperforms other strategies; with GPT-4 in 5-shot it reaches 82.4% execution accuracy. For example organization, DAIL Organization with GPT-4 achieves 83.5% on Spider-dev and 76.0% on Spider-Realistic, while weaker LLMs often need Full-Information Organization.
- Supervised fine-tuning greatly improves open-source LLMs: LLaMA-13B with Alpaca SFT Prompt reaches 68.6% execution accuracy on Spider, and LLaMA-33B with CR_P reaches 69.1%. However, after fine-tuning, adding contextual examples causes sudden decreases in accuracy, suggesting the models overfit to zero-shot prompts.
Methods
- The paper conducts a systematic benchmark of prompt engineering for LLM-based Text-to-SQL: five question representations (BS_P, TR_P, OD_P, CR_P, AS_P), two prompt components (foreign key information and rule implication), four example selection strategies plus DAIL Selection and an Upper Limit reference, and three example organizations (Full-Information, SQL-Only, DAIL Organization) on four LLMs: GPT-4, GPT-3.5-TURBO, TEXT-DAVINCI-003, and Vicuna-33B.
- Evaluation uses Spider and Spider-Realistic, with exact-set-match accuracy (EM) and execution accuracy (EX) as metrics; Spider-dev is used because the test split is not released. Few-shot examples are drawn from the Spider training split.
- DAIL-SQL combines CR_P as question representation, DAIL Selection based on masked question similarity and query similarity with a threshold, and DAIL Organization, which keeps question-SQL pairs while removing token-cost database schema. The paper also fine-tunes open-source LLMs on the Spider training split (8659 samples) and evaluates zero-shot and few-shot performance.
Limitations
- Paper-stated: only two rule implications are tested; open-source LLMs are fine-tuned with only the Spider training set; the databases in Spider and Spider-Realistic may not be large enough; the current evaluation metric prioritizes correctness over efficiency, leaving generation of efficient SQL among correct alternatives unexplored; LLMs larger than 33B are left as future work due to limited resources.
- Text-evident: OpenAI LLMs are not fine-tuned because of unaffordable cost, so SFT conclusions are limited to open-source models. The paper also notes that fine-tuned LLMs fail to learn from contextual examples, and preserving in-context learning ability after fine-tuning remains an open question.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
18 Valid · 9 Not valid · 8 Not checked · 140 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Different from prior studies, the core problem in LLM-based Text-to-SQL solution is how to prompt LLM to generate correct SQL queries, namely prompt engineering.”
- Also stated in: § 2 PRELIMINARY · p. 2 · plain
“Therefore, to tackle Text-to-SQL task with LLMs, the core is to find the optimal prompt, also known as prompt engineering [26, 29].”
What the sources say
- Partly supportedcites [26] [26] Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A Comprehensive Evaluation of ChatGPT's Zero-Shot Text-to-SQL Capability. CoRR abs/2303.13547 (2023). - sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says prompt design is required for models such as GPT-3 to generate target outputs, and the Method describes a prompt used to obtain SQL from ChatGPT. But the paper does not identify prompt engineering as the core problem in LLM-based Text-to-SQL; it adopted an existing prompt without further exploration. (LLM, conf=0.83)
- Partly supportedcites [29] [29] Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies. CoRR abs/2305.12586 (2023). - sourceExperimental — accuracy not yet measured
The Introduction says the paper investigates prompt configurations for LLM-based Text-to-SQL and that demonstration-prompt structure significantly affects performance. It also distinguishes prompting from fine-tuning, so its focus on prompt design does not establish that prompt engineering is the core problem for all LLM-based Text-to-SQL solutions. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2 PRELIMINARY · p. 2 · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Notably, equipped with GPT-4 [30], Pourreza et al. [37] achieved the first place in Spider leaderboard [3] with 85 . 3% execution accuracy.”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Before DAILSQL, the state-of-the-art performance in the Spider leaderboard is 85 . 3% [37].”
What the sources say
- Supportedcites [37] [37] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs/2304.11015 (2023). - source
Table 2 reports 85.3% execution accuracy for DIN-SQL with GPT-4 on the Spider holdout test set. Section 5.5.1 says this was the highest execution accuracy among officially published results at the time of writing, supporting the first-place claim in that context. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, LLMs can perform better for Text-to-SQL through in-context learning, in which only a few examples are provided in the input prompts.”
What the sources say
- Supportedcites [14] [14] Chunxi Guo, Zhiliang Tian, Jintao Tang, Pancheng Wang, Zhihua Wen, Kang Yang, and Ting Wang. 2023. A Case-Based Reasoning Framework for Adaptive Prompting in Cross-Domain Text-to-SQL. CoRR abs/2304.13301 (2023). - source
The “LLM-based SQL Generation” section says LLMs can achieve good Text-to-SQL performance with a few in-context examples and that retrieved demonstrations improve SQL generation. The Introduction also describes prompting the LLM with a few demonstration examples. (LLM, conf=0.92)
- Partly supportedcites [28] [28] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What Makes Good In-Context Examples for GPT-3?. In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures . 100-114. - sourceExperimental — accuracy not yet measured
The Trial section reports that selected in-context examples improved GPT-3’s performance on several tasks, including table-to-text generation and question answering. The Experimental Setup section lists the tasks studied; Text-to-SQL is not among them, so the source does not establish the claim for that task. Page numbers are unavailable in our copy. (LLM, conf=0.85)
- Supportedcites [37] [37] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs/2304.11015 (2023). - source
The Abstract reports that the paper’s in-context-learning approach improved LLMs’ simple few-shot text-to-SQL performance. The Introduction describes few-shot prompting as using a few demonstrations without fine-tuning. The reported improvement is specifically for the paper’s decomposed approach, which is consistent with the claim that LLMs can perform better through in-context learning. (LLM, conf=0.91)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“there is a scarcity of studies exploring the supervised fine-tuning of LLMs for Text-to-SQL [46], leaving this area as an open question.”
What the sources say
- Supportedcites [46] [46] Ruoxi Sun, Sercan Ö. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. 2023. SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL. CoRR abs/2306.00739 (2023). - source
Section 3.2, “Text-to-SQL with LLMs” (page not provided), says tuning approaches for Text-to-SQL have been relatively under-explored compared with prompting approaches. This supports the cited claim that studies of LLM fine-tuning for Text-to-SQL are scarce. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Unexpectedly, GPT-4 exhibits a preference for the simple BS 𝑃 derived from Din-SQL [37], indicating that a powerful LLM can mitigate the complexities associated with representation design.”
What the sources say
- Supportedcites [37] [37] Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. CoRR abs/2304.11015 (2023). - source
The citation identifies the source of the simple prompting approach, rather than attributing the citing paper’s GPT-4 preference finding to DIN-SQL. Section 4.3 describes simple few-shot prompting for easy queries, and the 5.6 ablation study includes a simple few-shot prompting variant. The source does not itself report the stated preference. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
90 more claims · 5 with their cited sources checked
Create a free account to see all 175 claims and to verify the 8 references still open.
- Probably not a substantial claim. 53
- It may state two separate claims as one. 38
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.3 DAIL-SQL · p. 5
“It then ranks the candidate examples based on the Euclidean distance between the embeddings of masked 𝑞 and 𝑞 𝑖 .”
§ A.1 Pseudocode · p. 15
“14 sort Q by cosine_simlarity( 𝑒 ( 𝑞 ′ ) , 𝑒 ( 𝑞 ′ 𝑖 ) )”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Supervised Fine-Tuning for Text-to-SQL · p. 10
“Full-Information Organization outperforms other strategies in different k-shot scenarios, which achieves 51 . 5% execution accuracy with Vicuna-33B.”
§ D.3 Experiments in Few-shot Scenario for Different Model Sizes and Training Corpus of Open-source LLMs · p. 19
“| Full-Information Organization | Vicuna-33B | 6.9 | 43.7 | 22.1 | 49.2 | 27.4 | 49.9 | 28.0 | 51.1 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Supervised Fine-Tuning for Text-to-SQL · p. 11
“In this experiment, the best performance on Spider is achieved by the combination of LLaMA13B and Alpaca SFT Prompt, whose exact-set-match and execution accuracy are 65 . 1% and 68 . 6%.”
§ D.5 Detailed Performance of Fine-tuned Open-source LLM on Spider and Spider-Realistic · p. 20
“| LLaMA-2-CHAT-7B | EX | 62.9 | 62.9 | 64.2 | 65.7 | 69.6 | 65.1 | 51.4 | 44.5 | 47.4 | 49.6 | 54.9 | 49.6 |”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Wang, B., Shin, R., Liu, X., Polozov, O., & Richardson, M. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1911.04942 · Cited by 964
Research works printed but not linked: 4
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The 10th International Conference on Learning Representations.
- Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL. In 37th AAAI Conference on Artificial Intelligence. 13067--13075.
- Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023. Graphix-T5: Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing. In 37th AAAI Conference on Artificial Intelligence. 13076--13084.
- Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. LLAMA2: Open Foundation and Fine-Tuned Chat Models. CoRR (2023).
Cited by
The Dawn of Natural Language to SQL: Are We Fully Ready?
Li, B., Luo, Y., Chai, C., Li, G., & Tang, N. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3682003 · Cited by 273
A Survey of Text-to-SQL in the Era of LLMs: Where Are We, and Where Are We Going?
Liu, X., Shen, S., Li, B., Ma, P., Jiang, R., Zhang, Y., Fan, J., Li, G., Tang, N., & Luo, Y. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3592032 · Cited by 347
OmniSQL: Synthesizing High-Quality Text-to-SQL Data at Scale
Li, H., Wu, S., Zhang, X., Huang, X., Zhang, J., Jiang, F., Wang, S., Zhang, T., Chen, J., Shi, R., et al. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749723 · Cited by 56
Enhancing knowledge retrieval with in-context learning and semantic search through generative AI
Ghali, M.-K., Farrag, A., Won, D., & Jin, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113047 · Cited by 77
PURPLE: Making a Large Language Model a Better SQL Writer
Ren, T., Fan, Y., He, Z., Huang, R., Dai, J., Huang, C., Jing, Y., Zhang, K., Yang, Y., & Wang, X. S. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00009 · Cited by 113
Metadata
- Authors
- Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J.
- Affiliations
- Alibaba Group (China) (CN)
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2024-01
- Volume
- 17
- Issue
- 5
- Pages
- 1132-1145
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W4396571402
- Cited by
- 342
- References
- 20
- Citations by year
- 2026: 160 · 2025: 143 · 2024: 38
- FWCI
- 50.11
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the VLDB Endowment (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Keywords
- text-to-SQL, large language models, prompt engineering, Spider benchmark, example selection, question representation, open-source LLMs, supervised fine-tuning, token efficiency, benchmark evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. https://app.cochat.ai/papers/10.14778/3641204.3641221
- The paper
- Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J. (2024). Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment. https://doi.org/10.14778/3641204.3641221
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.