Semantic Evaluation for Text-to-SQL with Distilled Test Suites
Open accessAbstract
We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models.Our method distills a small test suite of databases that achieves high code coverage for the gold query from a large number of randomly generated databases.At evaluation time, it computes the denotation accuracy of the predicted queries on the distilled test suite, hence calculating a tight upper-bound for semantic accuracy efficiently.We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples.In contrast, the current Spider metric leads to a 2.5% false negative rate on average and 8.1% in the worst case, indicating that test suite accuracy is needed.Our implementation, along with distilled test suites for eleven Textto-SQL datasets, is publicly available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zhong et al. found that by evaluating on an increased number of databases, false positives become negligible.
“Zhong et al. [90] found that by evaluating on an increased number of databases, false positives become negligible.”
From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [90]
To make a fair comparison, the study followed prior work to use exact-set-match accuracy and execution accuracy.
“To make a fair comparison, we follow prior study [61] to use exact-set-match accuracy ( EM ) and execution accuracy ( EX ).”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034 · see citing works
Self-citationGreen, T. J., Karvounarakis, G., & Tannen, V. (2007) · Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/1265530.1265535 · Cited by 812 · see citing works
Learning to mine aligned code and natural language pairs from stack overflow
Yin, P., Deng, B., Chen, E., Vasilescu, B., & Neubig, G. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196408 · Cited by 218 · see citing works
Flagged first, then by citations
Research works printed but not linked: 37
- Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. In Pro- ceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Pro- ceedings of Machine Learning Research, pages 130- 140.
- PMLR.
- P Ammann and J Offutt. Introduction to software test- ing. cambridge university press, 2008.
- Ben Bogin, Jonathan Berant, and Matt Gardner. 2019a. Representing schema structure with graph neural networks for text-to-sql parsing. In ACL.
- Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b. Representing schema structure with graph neural networks for text-to-sql parsing. arXiv preprint arXiv:1905.06241. · arXiv
- DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. Ryan- sql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases. https://arxiv.org/abs/2004.03125. · arXiv
- Shumo Chu, Brendan Murphy, Jared Roesch, Alvin Cheung, and Dan Suciu. 2018. Axiomatic founda- tions and algorithms for deciding semantic equiva- lences of sql queries. Proceedings of the VLDB En- dowment, 11(11):1482-1495.
- Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994. Expanding the scope of the atis task: The atis-3 corpus. In HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
- Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018a. Improving text-to-sql evaluation methodology. arXiv preprint arXiv:1806.09029. · arXiv
- Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018b. Improving text-to-SQL evaluation methodology. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 351-360, Melbourne, Australia. Asso- ciation for Computational Linguistics. · DOI
- Hongyu Guo. 2017. A deep network with visual text composition behavior. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pages 372-377, Vancouver, Canada. Association for Com- putational Linguistics. · DOI
- Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. arXiv preprint arXiv:1905.08205. · arXiv
- Pengcheng He, Yi Mao, K. Chakrabarti, and W. Chen. 2019. X-sql: reinforce schema representation with context. ArXiv, abs/1908.08113. · arXiv
- Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. ArXiv, abs/1902.01069. · arXiv
- Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learn- ing a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 963-973, Vancouver, Canada. Association for Computational Linguistics. · DOI
- Caroline Lemieux, Rohan Padhye, Koushik Sen, and Dawn Song. 2018. Perffuzz: Automatically gen- erating pathological inputs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 254-265.
- Fei Li and HV Jagadish. 2014. Constructing an in- teractive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73-84.
- Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and se- mantic parsing. In Advances in Neural Information Processing Systems, pages 9994-10006.
- Percy Liang. 2013. Lambda dependency-based compo- sitional semantics. arXiv preprint arXiv:1309.4408. · arXiv
- Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3243-3253, Brussels, Belgium. Association for Computational Linguistics. · DOI
- Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020. Hy- brid ranking network for text-to-sql. arXiv preprint arXiv:2008.04759. · arXiv
- B. McCann, N. Keskar, Caiming Xiong, and R. Socher. 2018. The natural language decathlon: Mul- titask learning as question answering. ArXiv, abs/1806.08730. · arXiv
- Joan C Miller and Clifford J Maloney. 1963. System- atic mistake analysis of digital computer programs. Communications of the ACM, 6(2):58-63.
- Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A discrete hard EM ap- proach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 2851- 2864, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
- Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical ma- chine translation (t). In 2015 30th IEEE/ACM In- ternational Conference on Automated Software En- gineering (ASE), pages 574-584. IEEE.
- Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. 2019. Semantic fuzzing with zest. In Proceedings of the 28th ACM SIGSOFT International Symposium on Soft- ware Testing and Analysis, pages 329-340.
- Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a theory of natural language inter- faces to databases. In Proceedings of the 8th Inter- national Conference on Intelligent User Interfaces, IUI '03, page 149-157, New York, NY, USA. Asso- ciation for Computing Machinery. · DOI
- Tianze Shi, Kedar Tatwawadi, K. Chakrabarti, Yi Mao, Oleksandr Polozov, and W. Chen. 2018. Incsql: Training incremental text-to-sql parsers with non- deterministic oracles. ArXiv, abs/1809.05054. · arXiv
- Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to ex- ecutable formal queries. In Proceedings of the Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 2238-2249. Association for Computational Linguistics.
- Alane Laughlin Suhr, Kenton Lee, Ming-Wei Chang, and Pete Shaw. 2020. Exploring unexplored general- ization challenges for cross-database semantic pars- ing. In ACL 2020.
- Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for ma- chine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1810-1822, Florence, Italy. Associa- tion for Computational Linguistics. · DOI · Paper page
- Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436. · arXiv
- Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Pro- gramming Languages, 1(OOPSLA):1-26.
- Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin- cent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019a. CoSQL: A conversational text-to-SQL challenge towards cross- domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 1962- 1979, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
- Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vin- cent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. SParC: Cross-domain se- mantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4511-4523, Florence, Italy. Association for Computational Linguistics. · DOI
- John M. Zelle and Raymond J. Mooney. 1996. Learn- ing to parse database queries using inductive logic programming. In Proceedings of the Thirteenth Na- tional Conference on Artificial Intelligence -Volume 2, AAAI'96, page 1050-1055. AAAI Press.
- Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103. · arXiv
Cited by
Sorted by citations
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation
Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3641204.3641221 · Cited by 342 · see citing works
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · see citing works
“Zhong et al. [90] found that by evaluating on an increased number of databases, false positives become negligible.”
- Authors
- Zhong, R., Yu, T., & Klein, D.
- Affiliations
- University of California, Berkeley (US)
- Yale University (US)
- Corresponding author
- Zhong, Ruiqi
- Venue
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Published
- 2020
- Pages
- 396-411
- OpenAlex
- W3102020135
- Cited by
- 112
- References
- 3
- Citations by year
- 2026: 27 · 2025: 31 · 2024: 17 · 2023: 21 · 2022: 7 · 2021: 8 · 2020: 1
- FWCI
- 3.53
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- text-to-SQL, semantic evaluation, code coverage, Spider benchmark, query processing, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
