CoChat
Sign Up
PreprintCited by 22Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval

Open access
Guo, Chunxi, Tian, Zhiliang, Tang, Jintao, Wang, Pancheng, Wen, Zhihua, Yang, Kang, Wang, Ting·2023·arXiv
DOI 10.48550/arxiv.2304.13301arXiv 2304.13301
Open this paper in CoChat — free accountOpen PDF

Abstract

Text-to-SQL is a task that converts a natural language question into a structured query language (SQL) to retrieve information from a database. Large language models (LLMs) work well in natural language generation tasks, but they are not specifically pre-trained to understand the syntax and semantics of SQL commands. In this paper, we propose an LLM-based framework for Text-to-SQL which retrieves helpful demonstration examples to prompt LLMs. However, questions with different database schemes can vary widely, even if the intentions behind them are similar and the corresponding SQL queries exhibit similarities. Consequently, it becomes crucial to identify the appropriate SQL demonstrations that align with our requirements. We design a de-semanticization mechanism that extracts question skeletons, allowing us to retrieve similar examples based on their structural similarity. We also model the relationships between question tokens and database schema items (i.e., tables and columns) to filter out scheme-related information. Our framework adapts the range of the database schema in prompts to balance length and valuable information. A fallback mechanism allows for a more detailed schema to be provided if the generated SQL query fails. Ours outperforms state-of-the-art models and demonstrates strong generalization ability on three cross-domain Text-to-SQL benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.13301

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can perform better for Text-to-SQL through in-context learning.

    “However, LLMs can perform better for Text-to-SQL through in-context learning, in which only a few examples are provided in the input prompts.”

    From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [14]

  • CBR-ApSQL achieves 78.2% accuracy with TEXT-DAVINCI-003, but is still lower than the optimal performance achieved by DAIL_S + FI_O.

    “While for CBR-ApSQL, it achieves 78 . 2% accuracy with TEXT-DAVINCI-003, but still lower than the optimal performance achieved by DAIL 𝑆 + FI 𝑂 .”

    From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [14]

    Table 2, reproduced in the copy under section 5, reports 78.2 as the Spider test-suite accuracy for the paper’s “-SQL Revision” ablation, not for CBR-ApSQL. Section 4.1 says the authors used text-davinci-003, but the passages found for CBR-ApSQL and DAIL_S + FI_O do not report either method or the claimed comparison. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. Wang, B., Shin, R., Liu, X., Polozov, O., Richardson, M.: Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. ACL (2020)
  2. Cai, R., Xu, B., Zhang, Z., Yang, X., Li, Z., Liang, Z.: An encoder-decoder frame- work translating natural language to database queries. In: IJCAI (Jul 2018)
  3. Li, H., Zhang, J., Li, C., Chen, H.: Decoupling the skeleton parsing and schema linking for text-to-sql. arXiv:2302.05965 (2023) · arXiv
  4. Li, J., Hui, B., et al.: Graphix-t5: Mixing pre-trained transformers with graph- aware layers for text-to-sql parsing. arXiv:2301.07507 (2023) · arXiv
  5. Zhao, W.X., Zhou, K., Li, J., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023) · arXiv
  6. Brown, T., Mann, B., Ryder, N., et al.: Language models are few-shot learners. NIPS 33, 1877-1901 (2020)
  7. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., et al.: Evaluating large language models trained on code. arXiv:2107.03374 (2021) · arXiv
  8. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. NAACL (2018)
  9. Zhuang, L., Wayne, L., Ya, S., Jun, Z.: A robustly optimized bert pre-training approach with post-training. In: CCL. pp. 1218-1227 (2021)
  10. Lewis, M., Liu, Y., et al.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL (Jul 2020)
  11. Raffel, C., Shazeer, N., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 21, 5485-5551 (2020)
  12. Rajkumar, N., Li, R., Bahdanau, D.: Evaluating the text-to-sql capabilities of large language models. arXiv:2204.00498 (2022) · arXiv
  13. Liu, A., Hu, X., Wen, L., Yu, P.S.: A comprehensive evaluation of chatgpt's zero- shot text-to-sql capability. arXiv:2303.13547 (2023) · arXiv
  14. Cheng, Z., Xie, T., Shi, P., et al.: Binding language models in symbolic languages. In: ICLR (2023)
  15. Scholak, T., Schucher, N., Bahdanau, D.: Picard: Parsing incrementally for con- strained auto-regressive decoding from language models. EMNLP (2021)
  16. Qi, J., Tang, J., He, Z., et al.: RASAT: integrating relational structures into pre- trained seq2seq model for text-to-sql. In: EMNLP. pp. 3215-3229 (2022)
  17. Lee, Y.J., Lim, C.G., Choi, H.J.: Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation. In: COLING. pp. 669-683 (2022)
  18. Su, H., Kasai, J., et al.: Selective annotation makes language models better few- shot learners. arXiv:2209.01975 (2022) · arXiv
  19. Rubin, O., Herzig, J., Berant, J.: Learning to retrieve prompts for in-context learn- ing. In: NAACL. pp. 2655-2671 (2022)
  20. Guo, J., et al.: Towards complex text-to-sql in cross-domain database with inter- mediate representation. In: ACL. pp. 4524-4535 (2019)
  21. Bogin, B., Berant, J., Gardner, M.: Representing schema structure with graph neural networks for text-to-sql parsing. In: ACL (Sep 2019)
  22. Chen, Z., Chen, L., Zhao, Y., Cao, R., Xu, Z., Zhu, S., Yu, K.: Shadowgnn: Graph projection neural network for text-to-sql parser. In: NAACL (Jun 2021)
  23. Cao, R., Chen, L., et al.: Lgesql: Line graph enhanced text-to-sql model with mixed local and non-local relations. In: ACL (Jul 2021)
  24. Hui, B., Geng, R., Ren, Q., et al.: Dynamic hybrid relation exploration network for cross-domain context-dependent semantic parsing. AAAI (May 2021)
  25. Hui, B., Geng, R., Wang, L., et al.: S2sql: Injecting syntax to question-schema interaction graph encoder for text-to-sql parsers. In: ACL. pp. 1254-1262 (2022)
  26. Cai, Z., Li, X., Hui, B., Yang, M., Li, B., et al.: Star: Sql guided pre-training for context-dependent text-to-sql parsing. EMNLP (Oct 2022)
  27. Lin, X.V., Socher, R., Xiong, C.: Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. In: EMNLP. pp. 4870-4888 (2020)
  28. He, P., Mao, Y., Chakrabarti, K., Chen, W.: X-sql: reinforce schema representation with context. arXiv:1908.08113 (2019) · arXiv
  29. Lyu, Q., Chakrabarti, K., Hathi, S., Kundu, S., Zhang, J., Chen, Z.: Hybrid ranking network for text-to-sql. arXiv preprint arXiv:2008.04759 (2020) · arXiv
  30. Zhong, V., Lewis, M., Wang, S.I., Zettlemoyer, L.: Grounded adaptation for zero- shot executable semantic parsing. In: EMNLP. pp. 6869-6882 (2020)
  31. Yu, W., et al.: Similar questions correspond to similar sql queries: A case-based reasoning approach for text-to-sql translation. In: ICCBR. pp. 294-308. Springer (2021)
  32. Tian, Z., Bi, W., Li, X., Zhang, N.L.: Learning to abstract for memory-augmented conversational response generation. In: ACL. pp. 3816-3825 (2019)
  33. Song, Y., et al.: Retrieval bias aware ensemble model for conditional sentence generation. In: ICASSP. pp. 6602-6606. IEEE (2022)
  34. Wen, Z., et al.: Grace: Gradient-guided controllable retrieval for augmenting attribute-based text generation. In: Findings of ACL 2023. pp. 8377-8398 (2023)
  35. Ganea, O., Bécigneul, G., Hofmann, T.: Hyperbolic neural networks. NIPS (2018)
  36. Chen, B., et al.: Probing bert in hyperbolic spaces. arXiv:2104.03869 (2021) · arXiv
  37. Yu, T., Zhang, R., et al.: Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In: EMNLP (Jun 2019)
  38. Gan, Y., Chen, X., Huang, Q., Purver, M., et al: Towards robustness of text-to-sql models against synonym substitution. In: ACL (Jul 2021)
  39. Gan, Y., Chen, X., Purver, M.: Exploring underexplored limitations of cross- domain text-to-sql generalization. In: EMNLP (Dec 2021)
  40. Zhong, R., Yu, T., Klein, D.: Semantic evaluation for text-to-sql with distilled test suites. In: EMNLP. pp. 396-411 (2020)
  41. Johnson, J., Douze, M., Jegou, H.: Billion-scale similarity search with gpus. IEEE Transactions on Big Data p. 535-547 (Jun 2019)

Cited by

Sorted by citations

Metadata

Authors
Guo, C., Tian, Z., Tang, J., Wang, P., Wen, Z., Yang, K., & Wang, T.
Affiliations
National University of Defense Technology (CN)
Corresponding author
  • Tian, Zhiliang
  • Tang, Jintao
  • Wang, Ting
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
262-274
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2304.13301
arXiv
2304.13301
OpenAlex
W4388551928
Cited by
22
References
1
Citations by year
2026: 5 · 2025: 12 · 2024: 3 · 2023: 2
FWCI
3.49
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
text-to-SQL, large language models, GPT-3.5, database schema, prompt engineering, structural similarity, fallback mechanisms, domain generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review