Question Answering over Tabular Data with DataBench: A Large-Scale Empirical Evaluation of LLMs
Open accessAbstract
Large Language Models (LLMs) are showing emerging abilities, and one of the latest recognized ones deals with their ability to reason and answer questions from tabular data.Although there are some available datasets to assess question answering systems on tabular data, they are not large and diverse enough to properly assess the capabilities of LLMs.To this end, we propose DataBench, a benchmark composed of 65 real-world datasets over several domains, including 20 human-generated questions per dataset, totaling 1300 questions and answers overall.Using this benchmark, we perform a large-scale empirical comparison of several open and closed source models, including both code-generating and in-context learning models.The results highlight the current gap between open-source and closed-source models, with all types of model having room for improvement even in simple boolean questions or involving a single column.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
FinQA: A Dataset of Numerical Reasoning over Financial Data
Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Langdon, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.300 · Cited by 258 · see citing works
Flagged first, then by citations
- Authors
- Osés Grijalba, J., Ureña-López, L. A., Martínez Cámara, E., & Camacho-Collados, J.
- Venue
- Proceedings of the Language Resources and Evaluation Conference
- Published
- 2024-05
- Pages
- 13471-13488
- ISSN
- 2522-2686
- OpenAlex
- W7131878502
- Cited by
- 3
- References
- 1
- Citations by year
- 2026: 3
- FWCI
- 0.73
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Language Resources and Evaluation Conference (LREC) (publishedVersion)
- Funders
- UK Research and Innovation (MCIN/AEI/10.13039/501100011033)
- Agencia Estatal de Investigación (PID2020-116118GA-I00)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Database Systems and Queries
- Keywords
- question answering, tabular data, large language models, empirical evaluation, benchmark dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
