The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study
Open accessAbstract
Large language models (LLMs) are becoming attractive as few-shot reasoners to solve Natural Language (NL)-related tasks. However, there is still much to learn about how well LLMs understand structured data, such as tables. Although tables can be used as input to LLMs with serialization, there is a lack of comprehensive studies that examine whether LLMs can truly comprehend such data. In this paper, we try to understand this by designing a benchmark to evaluate the structural understanding capabilities (SUC) of LLMs. The benchmark we create includes seven tasks, each with its own unique challenges, \eg, cell lookup, row retrieval, and size detection. We perform a series of evaluations on GPT-3.5 and GPT-4. We find that performance varied depending on several input choices, including table input format, content order, role prompting, and partition marks. Drawing from the insights gained through the benchmark evaluations, we proposeself-augmentation for effective structural prompting, such as critical value / range identification using internal knowledge of LLMs. When combined with carefully chosen input choices, these structural prompting methods lead to promising improvements in LLM performance on a variety of tabular tasks, \eg, TabFact(\uparrow2.31%), HybridQA(\uparrow2.13%), SQA(\uparrow2.72%), Feverous(\uparrow0.84%), and ToTTo(\uparrow5.68%). We believe that our open-source (please find code and data at https://github.com/microsoft/TableProvider) benchmark and proposed prompting methods can serve as a simple yet generic selection for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A qualitative examination of responses with unsatisfied constraints suggests that LLMs often struggle with constraints involving numbers, negations, or long instructions with large number of constraints , which is consistent with findings from previous works (Sui et al., 2024; García-Ferrero et al., 2023; Truong et al., 2023; Jiang et al., 2024).”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Sui et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-T., & Riedel, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.745 · Cited by 487
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., & Wang, W. Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.91 · Cited by 290
Planning with Large Language Models for Code GenerationPreprint
Zhang, S., Chen, Z., Shen, Y., Ding, M., Tenenbaum, J., & Gan, C. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2303.05510 · Cited by 277
ToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.89 · Cited by 252
TabFact: A Large-scale Dataset for Table-based Fact Verification
Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., & Wang, W. Y. (2019) · arXiv · doi:10.48550/arxiv.1909.02164 · Cited by 239
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
Search-based Neural Structured Learning for Sequential Question Answering
Iyyer, M., Yih, W.-T., & Chang, M.-W. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1167 · Cited by 213
Large Language Models are few(1)-shot Table Reasoners
Chen, W. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.83 · Cited by 170
Wang, Z., Dong, H., Jia, R., Li, J., Fu, Z., Han, S., & Zhang, D. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467434 · Cited by 153
Self-citationTAPEX: Table Pre-training via Learning a Neural SQL ExecutorPreprint
Liu, Q., Chen, B., Guo, J., Ziyadi, M., Lin, Z., Chen, W., & Lou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07653 · Cited by 145
Generated Knowledge Prompting for Commonsense ReasoningPreprint
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08387 · Cited by 145
TableFormer: Table Structure Understanding with Transformers
Nassar, A., Livathinos, N., Lysak, M., & Staar, P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00457 · Cited by 129
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 106
GitTables: A Large-Scale Corpus of Relational Tables
Hulsebos, M., Demiralp, Ç., & Groth, P. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588710 · Cited by 89
TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching
Gong, H., Sun, Y., Feng, X., Qin, B., Bi, W., Liu, X., & Liu, T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.179 · Cited by 76
Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks
Dong, H., Cheng, Z., He, X., Zhou, M., Zhou, A., Zhou, F., Liu, A., Han, S., & Zhang, D. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/761 · Cited by 73
Self-citationStruBERT: Structure-aware BERT for Table Search and Matching
Trabelsi, M., Chen, Z., Zhang, S., Davison, B. D., & Heflin, J. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511972 · Cited by 47
Web Table Extraction, Retrieval and Augmentation: A SurveyPreprint
Zhang, S., & Balog, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00207 · Cited by 25
MATE: Multi-view Attention for Table Transformer EfficiencyPreprint
Eisenschlos, J. M., Gor, M., Müller, T., & Cohen, W. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04312 · Cited by 14
Xie, T., Wu, C. H., Shi, P. J., Zhong, R., Scholak, T., Yasunaga, M., Wu, C.-S., Zhong, M., Yin, P., Wang, S. I., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.05966 · Cited by 11
Simulating Users in Interactive Web Table Retrieval
Engelmann, B., Breuer, T., & Schaer, P. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615187 · Cited by 6
(2025) · doi:10.1088/1751-8121/ae2076 · Cited by 1
Shao, Y., Zhou, M., Zhong, Y., Wu, T., Han, H., Shi, H., Huang, G., & Zhang, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.05284 · Cited by 1
Self-citationFEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information
Aly, R., Guo, Z., Schlichtkrull, M., Thorne, J., Vlachos, A., Christodoulopoulos, C., Cocarascu, O., & Mittal, A. (2021) · arXiv
Research works printed but not linked: 16
- Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-Text Pre-Training and Prompting of Language Models. arXiv:2107.06955 [cs] http://arxiv.org/abs/2107.06955 · arXiv
- 10.48550/arXiv.2106.05707 · DOI · Paper page
- 10.48550/arXiv · DOI
- 10.48550/arXiv.2201.09745 · DOI
- Longxu Dou Yan Gao Mingyang Pan Dingzirui Wang Wanxiang Che Dechen Zhan and Jian-Guang Lou. 2022. UniSAr: A Unified Structure-Aware Autoregressive Language Model for Text-to-SQL. arXiv:2203.07781 [cs] http: //arxiv.org/abs/2203.07781 · arXiv
- Xinyi He Mengyu Zhou Jialiang Xu Xiao Lv Tianle Li Yijia Shao Shi Han Zejian Yuan and Dongmei Zhang. 2022. Inferring Tabular Analysis Metadata by Infusing Distribution and Knowledge Information. https://doi.org/10.48550/ arXiv.2209.00946 arXiv:2209.00946 [cs] · arXiv
- 10.18653/v1/2020.acl-main.398 · DOI
- Madelon Hulsebos and Paul Groth. 2022. GitTables: A LargeScale Corpus of Relational Tables. https://doi.org/10.48550/arXiv.2106.07258 arXiv:2106.07258 [cs] · DOI
- 10.48550/arXiv.2105 · DOI
- 10.48550/arXiv.2205.11916 · DOI · Paper page
- Stephanie Lin Jacob Hilton and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] http://arxiv.org/ abs/2109.07958 · arXiv
- OpenAI. 2023. GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs] · DOI · Paper page
- Ankur P. Parikh Xuezhi Wang Sebastian Gehrmann Manaal Faruqui Bhuwan Dhingra Diyi Yang and Dipanjan Das. 2020. ToTTo: A Controlled TableTo-Text Generation Dataset. https://doi.org/10.48550/arXiv.2004.14373 arXiv:2004.14373 [cs] · DOI · Paper page
- Alon Talmor Ori Yoran Amnon Catav Dan Lahav Yizhong Wang Akari Asai Gabriel Ilharco Hannaneh Hajishirzi and Jonathan Berant. 2021. MultiModalQA: Complex Question Answering over Text Tables and Images. https://doi.org/10. 48550/arXiv.2104.06039 arXiv:2104.06039 [cs] · arXiv
- 10.48550/arXiv.2009.06732 · DOI · Paper page
- 10.48550/arXiv.2203 · DOI
Cited by
Sorted by citations
Kresevic, S., Giuffrè, M., Ajcevic, M., Accardo, A., Crocè, L. S., & Shung, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01091-y · Cited by 207
From text to insight: large language models for chemical data extraction
Schilling-Wilhelmi, M., Ríos-García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Marquez, J. A., & Jablonka, K. M. (2024) · Chemical Society Reviews · doi:10.1039/d4cs00913d · Cited by 178
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
Chang, C., Wang, W., Peng, W.-C., & Chen, T.-F. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3719207 · Cited by 335
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
Chang, C., Wang, W.-Y., Peng, W.-C., & Chen, T.-F. (2023) · arXiv · doi:10.48550/arxiv.2308.08469 · Cited by 146
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Metadata
- Authors
- Sui, Y., Zhou, M., Zhou, M., Han, S., & Zhang, D.
- Affiliations
- National University of Singapore (SG)
- Microsoft Research Asia (China) (CN)
- Venue
- Proceedings of the 17th ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2024-03-04
- Pages
- 645-654
- OpenAlex
- W4392366650
- Cited by
- 155
- References
- 33
- Citations by year
- 2026: 59 · 2025: 72 · 2024: 24
- FWCI
- 45.74
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, self-augmentation, GPT-3.5, GPT-4
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
