CoChat
Sign Up
Cited by 157Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

Open access
Sui, Yuan, Zhou, Mengyu, Zhou, Mingjie, Han, Shi, Zhang, Dongmei·2024·Proceedings of the 17th ACM International Conference on Web Search and Data Mining
DOI 10.1145/3616855.3635752
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are becoming attractive as few-shot reasoners to solve Natural Language (NL)-related tasks. However, there is still much to learn about how well LLMs understand structured data, such as tables. Although tables can be used as input to LLMs with serialization, there is a lack of comprehensive studies that examine whether LLMs can truly comprehend such data. In this paper, we try to understand this by designing a benchmark to evaluate the structural understanding capabilities (SUC) of LLMs. The benchmark we create includes seven tasks, each with its own unique challenges, \eg, cell lookup, row retrieval, and size detection. We perform a series of evaluations on GPT-3.5 and GPT-4. We find that performance varied depending on several input choices, including table input format, content order, role prompting, and partition marks. Drawing from the insights gained through the benchmark evaluations, we proposeself-augmentation for effective structural prompting, such as critical value / range identification using internal knowledge of LLMs. When combined with carefully chosen input choices, these structural prompting methods lead to promising improvements in LLM performance on a variety of tabular tasks, \eg, TabFact(\uparrow2.31%), HybridQA(\uparrow2.13%), SQA(\uparrow2.72%), Feverous(\uparrow0.84%), and ToTTo(\uparrow5.68%). We believe that our open-source (please find code and data at https://github.com/microsoft/TableProvider) benchmark and proposed prompting methods can serve as a simple yet generic selection for future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3616855.3635752

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 33 references

Research works printed but not linked: 16

  1. Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-Text Pre-Training and Prompting of Language Models. arXiv:2107.06955 [cs] http://arxiv.org/abs/2107.06955 · arXiv
  2. 10.48550/arXiv.2106.05707 · DOI · Paper page
  3. 10.48550/arXiv · DOI
  4. 10.48550/arXiv.2201.09745 · DOI
  5. Longxu Dou Yan Gao Mingyang Pan Dingzirui Wang Wanxiang Che Dechen Zhan and Jian-Guang Lou. 2022. UniSAr: A Unified Structure-Aware Autoregressive Language Model for Text-to-SQL. arXiv:2203.07781 [cs] http: //arxiv.org/abs/2203.07781 · arXiv
  6. Xinyi He Mengyu Zhou Jialiang Xu Xiao Lv Tianle Li Yijia Shao Shi Han Zejian Yuan and Dongmei Zhang. 2022. Inferring Tabular Analysis Metadata by Infusing Distribution and Knowledge Information. https://doi.org/10.48550/ arXiv.2209.00946 arXiv:2209.00946 [cs] · arXiv
  7. 10.18653/v1/2020.acl-main.398 · DOI
  8. Madelon Hulsebos and Paul Groth. 2022. GitTables: A LargeScale Corpus of Relational Tables. https://doi.org/10.48550/arXiv.2106.07258 arXiv:2106.07258 [cs] · DOI
  9. 10.48550/arXiv.2105 · DOI
  10. 10.48550/arXiv.2205.11916 · DOI · Paper page
  11. Stephanie Lin Jacob Hilton and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs] http://arxiv.org/ abs/2109.07958 · arXiv
  12. OpenAI. 2023. GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs] · DOI · Paper page
  13. Ankur P. Parikh Xuezhi Wang Sebastian Gehrmann Manaal Faruqui Bhuwan Dhingra Diyi Yang and Dipanjan Das. 2020. ToTTo: A Controlled TableTo-Text Generation Dataset. https://doi.org/10.48550/arXiv.2004.14373 arXiv:2004.14373 [cs] · DOI · Paper page
  14. Alon Talmor Ori Yoran Amnon Catav Dan Lahav Yizhong Wang Akari Asai Gabriel Ilharco Hannaneh Hajishirzi and Jonathan Berant. 2021. MultiModalQA: Complex Question Answering over Text Tables and Images. https://doi.org/10. 48550/arXiv.2104.06039 arXiv:2104.06039 [cs] · arXiv
  15. 10.48550/arXiv.2009.06732 · DOI · Paper page
  16. 10.48550/arXiv.2203 · DOI

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Sui, Y., Zhou, M., Zhou, M., Han, S., & Zhang, D.
Affiliations
  • National University of Singapore (SG)
  • Microsoft Research Asia (China) (CN)
ORCID
Venue
Proceedings of the 17th ACM International Conference on Web Search and Data Mining
Publisher
ACM
Published
2024-03-04
Pages
645-654
DOI
10.1145/3616855.3635752
OpenAlex
W4392366650
Cited by
155
References
33
Citations by year
2026: 59 · 2025: 72 · 2024: 24
FWCI
45.74
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, self-augmentation, GPT-3.5, GPT-4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review