CoChat
Sign Up
PreprintCited by 5Topic ModelingNatural Language Processing TechniquesAdvanced Database Systems and Queries

The DOI matches its DataCite record.

RSL-SQL: Robust Schema Linking in Text-to-SQL Generation

Open access
Cao, Zhenbiao, Zheng, Yuanlei, Fan, Zhihao, Zhang, Xiaojin, Chen, Wei, Bai, Xiang·2024·arXiv
DOI 10.48550/arxiv.2411.00073arXiv 2411.00073
Open this paper in CoChat — free accountOpen PDF

Abstract

Text-to-SQL generation aims to translate natural language questions into SQL statements. In Text-to-SQL based on large language models, schema linking is a widely adopted strategy to streamline the input for LLMs by selecting only relevant schema elements, therefore reducing noise and computational overhead. However, schema linking faces risks that require caution, including the potential omission of necessary elements and disruption of database structural integrity. To address these challenges, we propose a novel framework called RSL-SQL that combines bidirectional schema linking, contextual information augmentation, binary selection strategy, and multi-turn self-correction. We improve the recall of pattern linking using forward and backward pruning methods, achieving a strict recall of 94% while reducing the number of input columns by 83%. Furthermore, it hedges the risk by voting between a full mode and a simplified mode enhanced with contextual information. Experiments on the BIRD and Spider benchmarks demonstrate that our approach achieves SOTA execution accuracy among open-source solutions, with 67.2% on BIRD and 87.9% on Spider using GPT-4o. Furthermore, our approach outperforms a series of GPT-4 based Text-to-SQL systems when adopting DeepSeek (much cheaper) with same intact prompts. Extensive analysis and ablation studies confirm the effectiveness of each component in our framework. The codes are available at https://github.com/Laqcce-cao/RSL-SQL.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2411.00073

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 37

  1. B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, "Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers," arXiv preprint arXiv:1911.04942, 2019. · arXiv
  2. B. Qin, B. Hui, L. Wang, M. Yang, J. Li, B. Li, R. Geng, R. Cao, J. Sun, L. Si et al., "A survey on text-to-sql parsing: Concepts, methods, and future directions," arXiv preprint arXiv:2208.13629, 2022. · arXiv
  3. G. Katsogiannis-Meimarakis and G. Koutrika, "A survey on deep learning approaches for text-to-sql," The VLDB Journal, vol. 32, no. 4, pp. 905-936, 2023.
  4. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  5. D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou, "Text- to-sql empowered by large language models: A benchmark evaluation," arXiv preprint arXiv:2308.15363, 2023. · arXiv
  6. M. Pourreza and D. Rafiei, "Din-sql: Decomposed in-context learning of text-to-sql with self-correction," Advances in Neural Information Processing Systems, vol. 36, 2024.
  7. Z. Li, X. Wang, J. Zhao, S. Yang, G. Du, X. Hu, B. Zhang, Y. Ye, Z. Li, R. Zhao et al., "Pet-sql: A prompt-enhanced two-stage text-to-sql framework with cross-consistency," arXiv preprint arXiv:2403.09732, 2024. · arXiv
  8. D. Lee, C. Park, J. Kim, and H. Park, "Mcs-sql: Leveraging multiple prompts and multiple-choice selection for text-to-sql generation," arXiv preprint arXiv:2405.07467, 2024. · arXiv
  9. S. Talaei, M. Pourreza, Y.-C. Chang, A. Mirhoseini, and A. Saberi, "Chess: Contextual harnessing for efficient sql synthesis," arXiv preprint arXiv:2405.16755, 2024. · arXiv
  10. W. Lei, W. Wang, Z. Ma, T. Gan, W. Lu, M.-Y. Kan, and T.-S. Chua, "Re-examining the role of schema linking in text-to-sql," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6943-6954.
  11. J. M. Zelle and R. J. Mooney, "Learning to parse database queries using inductive logic programming," in Proceedings of the national conference on artificial intelligence, 1996, pp. 1050-1055.
  12. A. Vaswani, "Attention is all you need," Advances in Neural Information Processing Systems, 2017.
  13. I. Sutskever, "Sequence to sequence learning with neural networks," arXiv preprint arXiv:1409.3215, 2014. · arXiv
  14. J. Guo, Z. Zhan, Y. Gao, Y. Xiao, J.-G. Lou, T. Liu, and D. Zhang, "To- wards complex text-to-sql in cross-domain database with intermediate representation," arXiv preprint arXiv:1905.08205, 2019. · arXiv
  15. C.-Y. Tai, Z. Chen, T. Zhang, X. Deng, and H. Sun, "Explor- ing chain-of-thought style prompting for text-to-sql," arXiv preprint arXiv:2305.14215, 2023. · arXiv
  16. X. Dong, C. Zhang, Y. Ge, Y. Mao, Y. Gao, J. Lin, D. Lou et al., "C3: Zero-shot text-to-sql with chatgpt," arXiv preprint arXiv:2307.07306, 2023. · arXiv
  17. T. Wolfson, D. Deutch, and J. Berant, "Weakly supervised text- to-sql parsing through question decomposition," arXiv preprint arXiv:2112.06311, 2021. · arXiv
  18. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24 824-24 837, 2022.
  19. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  20. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al., "Competition- level code generation with alphacode," Science, vol. 378, no. 6624, pp. 1092-1097, 2022.
  21. A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. Wang, and X. V. Lin, "Lever: Learning to verify language-to-code generation with execution," in International Conference on Machine Learning. PMLR, 2023, pp. 26 106-26 128.
  22. M. Pourreza, H. Li, R. Sun, Y. Chung, S. Talaei, G. T. Kakkar, Y. Gan, A. Saberi, F. Ozcan, and S. O. Arik, "Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql," arXiv preprint arXiv:2410.01943, 2024. · arXiv
  23. B. Bogin, M. Gardner, and J. Berant, "Representing schema structure with graph neural networks for text-to-sql parsing," arXiv preprint arXiv:1905.06241, 2019. · arXiv
  24. Z. Chen, L. Chen, Y. Zhao, R. Cao, Z. Xu, S. Zhu, and K. Yu, "Shadowgnn: Graph projection neural network for text-to-sql parser," arXiv preprint arXiv:2104.04689, 2021. · arXiv
  25. R. Cai, J. Yuan, B. Xu, and Z. Hao, "Sadga: Structure-aware dual graph aggregation network for text-to-sql," Advances in Neural Information Processing Systems, vol. 34, pp. 7664-7676, 2021.
  26. G. Qu, J. Li, B. Li, B. Qin, N. Huo, C. Ma, and R. Cheng, "Before generation, align it! a novel and effective strategy for mitigating hal- lucinations in text-to-sql generation," arXiv preprint arXiv:2405.15307, 2024. · arXiv
  27. M. Pourreza and D. Rafiei, "Dts-sql: Decomposed text-to-sql with small large language models," arXiv preprint arXiv:2402.01117, 2024. · arXiv
  28. H. Li, J. Zhang, H. Liu, J. Fan, X. Zhang, J. Zhu, R. Wei, H. Pan, C. Li, and H. Chen, "Codes: Towards building open-source language models for text-to-sql," Proceedings of the ACM on Management of Data, vol. 2, no. 3, pp. 1-28, 2024.
  29. R. Sun, S. Ö. Arik, A. Muzio, L. Miculicich, S. Gundabathula, P. Yin, H. Dai, H. Nakhost, R. Sinha, Z. Wang et al., "Sql-palm: Improved large language model adaptation for text-to-sql (extended)," arXiv preprint arXiv:2306.00739, 2023. · arXiv
  30. B. Li, Y. Luo, C. Chai, G. Li, and N. Tang, "The dawn of natural language to sql: Are we fully ready?" arXiv preprint arXiv:2406.01265, 2024. · arXiv
  31. W. Xie, G. Wu, and B. Zhou, "Mag-sql: Multi-agent generative approach with soft schema linking and iterative sub-sql refinement for text-to-sql," arXiv preprint arXiv:2408.07930, 2024. · arXiv
  32. B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, Q.-W. Zhang, Z. Yan, and Z. Li, "Mac-sql: Multi-agent collaboration for text-to-sql," arXiv preprint arXiv:2312.11242, 2023. · arXiv
  33. H. A. Caferoglu and Ö. Ulusoy, "E-sql: Direct schema linking via question enrichment in text-to-sql," arXiv preprint arXiv:2409.16751, 2024. · arXiv
  34. S. K. Gorti, I. Gofman, Z. Liu, J. Wu, N. Vouitsis, G. Yu, J. C. Cresswell, and R. Hosseinzadeh, "Msc-sql: Multi-sample critiquing small language models for text-to-sql translation," arXiv preprint arXiv:2410.12916, 2024. · arXiv
  35. J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo et al., "Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls," Advances in Neural Information Processing Systems, vol. 36, 2024.
  36. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman et al., "Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task," arXiv preprint arXiv:1809.08887, 2018. · arXiv
  37. K. Maamari, F. Abubaker, D. Jaroslawicz, and A. Mhedhbi, "The death of schema linking? text-to-sql in the age of well-reasoned language models," arXiv preprint arXiv:2408.07702, 2024. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Cao, Z., Zheng, Y., Fan, Z., Zhang, X., Chen, W., & Bai, X.
Affiliations
  • Huazhong University of Science and Technology (CN)
  • Alibaba Group (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2411.00073
arXiv
2411.00073
OpenAlex
W4404344612
Cited by
5
References
0
Citations by year
2026: 3 · 2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
Keywords
text-to-SQL, schema linking, large language models, database question answering, BIRD benchmark, Spider benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review