CoChat
Sign Up
Cited by 8Natural Language Processing TechniquesTopic ModelingAdvanced Database Systems and Queries

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency

Open access
Yang, Yicun, Wang, Zhaoguo, Xia, Yu, Wei, Zhuoran, Ding, Haoran, Piskac, Ruzica, Chen, Haibo, Li, Jinyang·2025·Proceedings of the ACM on Management of Data
DOI 10.1145/3725271
Open this paper in CoChat — free accountOpen PDF

Abstract

State-of-the-art Text-to-SQL models rely on fine-tuning or few-shot prompting to help LLMs learn from training datasets containing mappings from natural language (NL) queries to SQL statements. Consequently, the quality of the dataset can greatly affect the accuracy of these Text-to-SQL models. Unlike other NL tasks, Text-to-SQL datasets are prone to errors despite extensive manual efforts due to the subtle semantics of SQL. Our study has found a non-negligible (>30%) portion of incorrect NL to SQL mapping cases exists in popular datasets Spider and BIRD. This paper aims to improve the quality of Text-to-SQL training datasets and thereby increase the accuracy of the resulting models. To do so, we propose a necessary correctness condition called execution consistency. For a given database instance, an NL to SQL mapping satisfies execution consistency if the execution result of an NL query matches that of the corresponding SQL. We develop SQLDriller to detect incorrect NL to SQL mappings based on execution consistency in a best-effort manner by crafting database instances that likely result in violations of execution consistency. It generates multiple candidate SQL predictions that differ in their syntax structures. Using a SQL equivalence checker, SQLDriller obtains counterexample database instances that can distinguish non-equivalent candidate SQLs. It then checks the execution consistency of an NL to SQL mapping under this set of counterexamples. The evaluation shows SQLDriller effectively detects and fixes incorrect mappings in the Text-to-SQL dataset, and it improves the model accuracy by up to 13.6%.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3725271

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 54

  1. Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Ré. 2023. Ask Me Anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1--5, 2023. OpenReview.net. https://openreview.net/forum?id=bhUPJnS2g0X
  2. 10.1007/978-3-030-99524-9_24 · DOI
  3. 10.18653/v1/2023.emnlp-main.436 · DOI
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. CoRR, Vol. abs/2107.03374 (2021). showeprint[arXiv]2107.03374 https://arxiv.org/abs/2107.03374 · arXiv
  5. 10.48550/arXiv.2409.02038 · DOI
  6. 10.14778/3236187.3236200 · DOI
  7. Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017. Cosette: An Automated Prover for SQL. In 8th Biennial Conference on Innovative Data Systems Research, CIDR 2017, Chaminade, CA, USA, January 8--11, 2017, Online Proceedings. www.cidrdb.org. http://cidrdb.org/cidr2017/papers/p51-chu-cidr17.pdf
  8. E. F. Codd. 1974. Seven Steps to Rendezvous with the Casual User. In Data Base Management, Proceeding of the IFIP Working Conference Data Base Management, Cargèse, Corsica, France, April 1--5, 1974. North-Holland, 179-200.
  9. 10.1007/978--3--540--78800--3_24 · DOI
  10. 10.18653/v1/n19--1423 · DOI
  11. 10.1145/3626768 · DOI
  12. 10.48550/arXiv.2307.07306 · DOI
  13. 10.18653/v1/2020.acl-main.187 · DOI
  14. Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir R. Radev. 2018. Improving Text-to-SQL Evaluation Methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15--20, 2018, Volume 1: Long Papers. Association for Computational Linguistics, 351-360. https://aclanthology.org/P18--1033/
  15. 10.3390/ai2040043 · DOI
  16. 10.1145/3649849 · DOI
  17. 10.1145/320251.320253 · DOI
  18. 10.18653/v1/P17--1089 · DOI
  19. 10.14778/2735461.2735468 · DOI
  20. 10.1609/aaai.v37i11.26535 · DOI
  21. 10.1145/3654930 · DOI
  22. Jinyang Li Binyuan Hui Reynold Cheng Bowen Qin Chenhao Ma Nan Huo Fei Huang Wenyu Du Luo Si and Yongbin Li. 2023a. Graphix-T5: mixing pre-trained transformers with graph-aware layers for text-to-SQL parsing. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence (AAAI'23/IAAI'23/EAAI'23). AAAI Press Article 1467 9 pages. https://doi.org/10.1609/aaai.v37i11.26536 · DOI
  23. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023b. Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to-SQLs. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 1835, 28 pages.
  24. 10.1145/3654979 · DOI
  25. 10.1162/dint_a_00194 · DOI
  26. 10.48550/arXiv.2408.05109 · DOI
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, Vol. abs/1907.11692 (2019). showeprint[arXiv]1907.11692 http://arxiv.org/abs/1907.11692 · arXiv
  28. 10.1007/s11704-024--40763--6 · DOI
  29. OpenAI. 2023. GPT-4 Technical Report. CoRR Vol. abs/2303.08774 (2023). showeprint[arXiv]2303.08774 https://doi.org/10.48550/arXiv.2303.08774 · DOI · Paper page
  30. Simone Papicchio, Paolo Papotti, and Luca Cagliero. 2023. QATCH: benchmarking SQL-centric tasks with table representation learning models on your data. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 1348, 20 pages.
  31. 10.1109/EDUCON52537.2022.9766617 · DOI
  32. 10.1145/604045.604120 · DOI
  33. Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. 2023. Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023). Association for Computational Linguistics, Toronto, Canada, 326-362. https://aclanthology.org/2023.trustnlp-1.28
  34. 10.52202/075280-1577 · DOI
  35. 10.14778/3415478.3415557 · DOI
  36. 10.18653/v1/2023.acl-short.15 · DOI
  37. 10.48550/arXiv.2406.19073 · DOI
  38. 10.18653/v1/2021.emnlp-main.779 · DOI
  39. 10.48550/arXiv.2405.16755 · DOI · Paper page
  40. 10.5555/3295222.3295349 · DOI
  41. 10.18653/v1/2020.acl-main.677 · DOI
  42. 10.1145/3448016.3452822 · DOI
  43. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1--5, 2023. OpenReview.net. https://openreview.net/forum?id=1PL1NIMMrw
  44. 10.1145/3514221.3526125 · DOI
  45. 10.1145/3133887 · DOI
  46. 10.14778/3659437.3659452 · DOI · Paper page
  47. 10.48550/arXiv.2312.10321 · DOI
  48. 10.18653/v1/2020.emnlp-main.29 · DOI · Paper page
  49. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. CoRR, Vol. abs/1709.00103 (2017). showeprint[arXiv]1709.00103 http://arxiv.org/abs/1709.00103 · arXiv
  50. 10.18653/v1/2022.findings-emnlp.192 · DOI
  51. 10.14778/3342263.3342267 · DOI
  52. 10.1109/ICDE53745.2022.00250 · DOI
  53. 10.1007/s41019-023-00235--6 · DOI
  54. 10.14778/3685800.3685816 · DOI

Metadata

Authors
Yang, Y., Wang, Z., Xia, Y., Wei, Z., Ding, H., Piskac, R., Chen, H., & Li, J.
Affiliations
  • Shanghai Jiao Tong University (CN)
  • Department of Computer Science, Yale University, New Haven, Connecticut, USA
  • New York University (US)
ORCID
Venue
Proceedings of the ACM on Management of Data
Publisher
Association for Computing Machinery (ACM)
Published
2025-06-17
Volume
3
Issue
3
Pages
1-28
ISSN
2836-6573 (ISSN-L)
DOI
10.1145/3725271
OpenAlex
W4411403493
Cited by
8
References
8
Citations by year
2026: 6 · 2025: 2
FWCI
2.61
Type
journal article
Language
en
Open access
Locations
Proceedings of the ACM on Management of Data (publishedVersion)
Funders
  • National Natural Science Foundation of China (62422209,62132014,62272304)
  • National Science Foundation (2220407)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
Keywords
text-to-SQL, data validation, Spider dataset, training data quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review