The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Open accessAbstract
Recent years have witnessed the burgeoning of pretrained language models (LMs) for textbased natural language (NL) understanding tasks.Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables).In this paper we present TABERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables.TABERT is trained on a large corpus of 26 million tables and their English contexts.In experiments, neural semantic parsers using TABERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WIKITABLEQUESTIONS, while performing competitively on the text-to-SQL dataset SPIDER. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Transformer with global receptive field could effectively learn global representation and longrange dependency without absolute position information, achieving excellent performance on nonsequential data (i.e., images, tables) 24,27 .”
From scBERT as a Large-scale Pretrained Deep Language Model for Cell Type Annotation of Single-cell RNA-seq Data (2021) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827
Semantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement LearningPreprint
Zhong, V., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00103 · Cited by 1,384
ERNIE: Enhanced Representation through Knowledge IntegrationPreprint
Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., & Wu, H. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09223 · Cited by 1,149
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034
K-BERT: Enabling Language Representation with Knowledge Graph
Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5681 · Cited by 894 · cited 2×
“…contextual word representations, either by fusing vector representations of entities and relations on KBs into word representations of LMs (Peters et al., 2019; Zhang et al., 2019b,c), or by encouraging the LM to recover KB entities and relations from text (Sun et al., 2019; Liu et al., 2019a).”
BackgroundMethodSemantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base
Yih, W.-T., Chang, M.-W., He, X., & Gao, J. (2015) · doi:10.3115/v1/p15-1128 · Cited by 784
Self-citationRAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
Wang, B., Shin, R., Liu, X., Polozov, O., & Richardson, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.677 · Cited by 736
Knowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661
Learning to parse database queries using inductive logic programming
Zelle, J. M., & Mooney, R. J. (1996) · Cited by 642
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 2×
“These models allow us to capture the syntax and semantics of text via representations learned in an unsupervised manner, before fine-tuning the model to downstream tasks (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Liu et al., 2019b; Yang et al., 2019; Goldberg, 2019).”
Methodcontext2vec: Learning Generic Context Embedding with Bidirectional LSTM
Melamud, O., Goldberger, J., & Dagan, I. (2016) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1006 · Cited by 541
Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
Guo, J., Zhan, Z., Gao, Y., Xiao, Y., Lou, J., Liu, T., & Zhang, D. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1444 · Cited by 514
Compositional Semantic Parsing on Semi-Structured Tables
Pasupat, P., & Liang, P. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1142 · Cited by 514
Semantic Parsing via Paraphrasing
Berant, J., & Liang, P. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1133 · Cited by 504
SQLNet: Generating Structured Queries From Natural Language Without Reinforcement LearningPreprint
Xu, X., Liu, C., & Song, D. X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04436 · Cited by 477
Coarse-to-Fine Decoding for Neural Semantic Parsing
Dong, L., & Lapata, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1068 · Cited by 410
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385
Semantics-Aware BERT for Language Understanding
Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., & Zhou, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6510 · Cited by 364
Building a Semantic Parser Overnight
Wang, Y., Berant, J., & Liang, P. (2015) · doi:10.3115/v1/p15-1129 · Cited by 330
TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Generation
Tao, Y., Li, Z., Zhang, Z., Zhang, R., & Radev, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2093 · Cited by 314
A Comprehensive Exploration on WikiSQL with Table-Aware Word ContextualizationPreprint
Hwang, W., Yim, J., Park, S., & Seo, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01069 · Cited by 259
TabFact: A Large-scale Dataset for Table-based Fact Verification
Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., & Wang, W. Y. (2019) · arXiv · doi:10.48550/arxiv.1909.02164 · Cited by 239
SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-Domain Text-to-SQL Task
Yu, T., Yasunaga, M., Yang, K., Zhang, R., Wang, D., Li, Z., & Radev, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1193 · Cited by 238
Neural Semantic Parsing with Type Constraints for Semi-Structured Tables
Krishnamurthy, J., Dasigi, P., & Gardner, M. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1160 · Cited by 231
Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
Bogin, B., Berant, J., & Gardner, M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1448 · Cited by 207
TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation
Yin, P., & Neubig, G. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2002 · Cited by 203
Self-citationYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.08887 · Cited by 193
A Large Public Corpus of Web Tables containing Time and Context Metadata
Lehmberg, O., Ritze, D., Meusel, R., & Bizer, C. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872518.2889386 · Cited by 183
Neural Programmer: Inducing Latent Programs with Gradient DescentPreprint
Neelakantan, A., Le, Q. V., & Sutskever, I. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04834 · Cited by 171
Choi, D., Shin, M. C., Kim, E., & Shin, D. R. (2021) · Computational Linguistics · doi:10.1162/coli_a_00403 · Cited by 134
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
Zhang, R., Yu, T., Er, H., Shim, S., Xue, E. R., Lin, X., Shi, T., Xiong, C., Socher, R., & Radev, D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1537 · Cited by 132
Global Reasoning over Database Structures for Text-to-SQL Parsing
Bogin, B., Gardner, M., & Berant, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1378 · Cited by 115
Table Cell Search for Question Answering
Sun, H., Ma, H., He, X., Yih, W.-T., Su, Y., & Yan, X. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2883080 · Cited by 110
Self-citationMemory Augmented Policy Optimization for Program Synthesis and Semantic ParsingPreprint
Chen, L., Norouzi, M. H., Berant, J., Le, Q. V., & Lao, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02322 · Cited by 96
X-SQL: reinforce schema representation with contextPreprint
He, P., Mao, Y., Chakrabarti, K., & Chen, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08113 · Cited by 95
K-BERT: Enabling Language Representation with Knowledge GraphPreprint
Liu, W., Zhou, P., Zhao, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07606 · Cited by 84
Semantic Parsing with Syntax- and Table-Aware SQL Generation
Sun, Y., Tang, D., Duan, N., Ji, J., Cao, G., Feng, X., Qin, B., Liu, T., & Zhou, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1034 · Cited by 83
The Zephyr abstract syntax description language
Wang, D. C., Appel, A. W., Korn, J. L., & Serra, C. S. (1997) · Conference on Domain-Specific Languages · Cited by 81
Learning to Generalize from Sparse and Underspecified RewardsPreprint
Agarwal, R., Chen, L., Schuurmans, D., & Norouzi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07198 · Cited by 66
IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic OraclesPreprint
Shi, T., Tatwawadi, K., Chakrabarti, K., Mao, Y., Polozov, O., & Chen, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.05054 · Cited by 65
Robust Text-to-SQL Generation with Execution-Guided Decoding
Wang, C., Tatwawadi, K., Brockschmidt, M., Huang, P.-S., Mao, Y., Polozov, O., & Singh, R. (2018) · arXiv · Cited by 64
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent QuestionsPreprint
Zhang, R., Tao, Y., Er, H. Y., Shim, S., Xue, E., Lin, X., Shi, T., Xiong, C., Socher, R., & Radev, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00786 · Cited by 55
Macro Grammars and Holistic Triggering for Efficient Semantic Parsing
Zhang, Y., Pasupat, P., & Liang, P. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1125 · Cited by 48
Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
Wang, B., Titov, I., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1391 · Cited by 45
Iterative Search for Weakly Supervised Semantic Parsing
Dasigi, P., Gardner, M., Murty, S., Zettlemoyer, L., & Hovy, E. H. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1273 · Cited by 44
Coarse-to-Fine Decoding for Neural Semantic ParsingPreprint
Dong, L., & Lapata, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04793 · Cited by 42
Semantics-aware BERT for Language UnderstandingPreprint
Zhang, Z., Wu, Y., Zhao, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02209 · Cited by 23
Global Reasoning over Database Structures for Text-to-SQL ParsingPreprint
Bogin, B., Gardner, M., & Berant, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11214 · Cited by 10
Research works printed but not linked: 3
- Jonathan Berant and Percy Liang. 2014. Semantic pars- ing via paraphrasing. In Proceedings of ACL.
- Jindrich Libovický and Jindrich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019b. Ernie: En- hanced language representation with informative en- tities. In Proceedings of ACL.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Deep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,400
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., & Yao, J. (2021) · doi:10.1101/2021.12.05.471261 · Cited by 617
Deep entity matching with pre-trained language models
Li, Y., Li, J., Suhara, Y., Doan, A., & Tan, W.-C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3421424.3421431 · Cited by 544
Metadata
- Authors
- Yin, P., Neubig, G., Yih, W.-T., & Riedel, S.
- Affiliations
- Meta (United States) (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 8413-8426
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3035231859
- Cited by
- 487
- References
- 58
- Citations by year
- 2026: 68 · 2025: 86 · 2024: 72 · 2023: 89 · 2022: 76 · 2021: 84 · 2020: 12
- FWCI
- 24.93
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- pre-trained language models, tabular data, semantic parsing, structured data, natural language understanding, text-to-SQL, spiders, database tables
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
