CoChat
Sign Up
Cited by 193Topic ModelingMultimodal Machine Learning ApplicationsInformation Retrieval and Search Behavior

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

Open access
Zhu, Fengbin, Lei, Wenqiang, Huang, Youcheng, Wang, Chao, Zhang, Shuo, Lv, Jiancheng, Feng, Fuli, Chua, Tat-Seng·2021·Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
DOI 10.18653/v1/2021.acl-long.254
Open this paper in CoChat — free accountOpen PDF

Abstract

Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chua. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.acl-long.254

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most existing financial benchmarks focus on static question-answering, such as FinQA, TAT-QA, and FinBench.

    “However, most existing benchmarks focus on static question-answering, such as FinQA, TAT-QA, and FinBench (Chen et al., 2021; Zhu et al., 2021; Yin et al., 2023).”

    From StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? (2025) · cites this paper as Zhu et al., 2021

  • Most existing financial benchmarks for LLM agents focus on static question-answering tasks and fail to capture the dynamics of realistic trading scenarios.

    “Most existing financial benchmarks focus on static question-answering, failing to capture the dynamics of real-market trading.”

    From StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? (2025) · cites this paper as Zhu et al., 2021

    The Introduction describes existing QA systems by the kinds of data they use, and the Abstract describes TAT-QA as a benchmark built from financial reports; neither assesses most financial benchmarks. Searches for static financial benchmarks and real-market trading dynamics found no passage saying that existing benchmarks fail to capture trading dynamics. Page numbers were not available in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. ACL.
  2. Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xi- aochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, and Wei Chu. 2020a. Question directed graph attention network for numerical reasoning over text. In EMNLP-IJCNLP, pages 6759-6768.
  3. ACL.
  4. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In ICLR.
  5. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020b. HybridQA: A dataset of multi-hop question answer- ing over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP, pages 1026-1036. ACL.
  6. Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answering on knowl- edge bases and text using universal schema and memory networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics, pages 358-365. ACL.
  7. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proc. of NAACL.
  8. Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaus- sian error linear units. CoRR, abs/1606.08415. · arXiv
  9. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th Inter- national Conference on Neural Information Process- ing Systems, pages 1693-1701. MIT Press.
  10. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4320-4333. ACL.
  11. Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017. Learning fine-grained expressions to solve math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 805-814. ACL.
  12. Xisen Jin, Wenqiang Lei, Zhaochun Ren, Hongshen Chen, Shangsong Liang, Yihong Zhao, and Dawei Yin. 2018. Explicit state tracking with semi- supervisionfor neural dialogue generation. In Pro- ceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 1403-1412.
  13. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics, pages 271-281. ACL.
  14. Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with sin- gle sequence-to-sequence architectures. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 1437-1447.
  15. Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. 2020. Re-examining the role of schema linking in text- to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943-6954.
  16. Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu, and Bing Qin. 2020. Molweni: A challenge multiparty dialogues-based machine reading comprehension dataset with dis- course structure. CoRR, abs/2004.05080. · arXiv
  17. Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In Proceedings of the 54th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2144- 2153. ACL.
  18. Liqiang Nie, Yongqi Li, Fuli Feng, Xuemeng Song, Meng Wang, and Yinglong Wang. 2020. Large- scale question tagging via joint question-topic em- bedding learning. ACM Transactions on Informa- tion Systems (TOIS), 38(2):1-23.
  19. Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Asso- ciation for Computational Linguistics and the 7th In- ternational Joint Conference on Natural Language Processing, pages 1470-1480. ACL.
  20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP- IJCNLP, pages 2383-2392. ACL.
  21. Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using transformation-based learning. In Third Workshop on Very Large Corpora.
  22. Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, and Jonathan Berant. 2020. A simple and effec- tive model for answering multi-span questions. In EMNLP-IJCNLP, pages 3074-3080. ACL.
  23. Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In EMNLP-IJCNLP, pages 2380-2390. ACL.
  24. Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for ques- tion answering. In Proceedings of the 25th Interna- tional Conference on World Wide Web, WWW '16, page 771-782. International World Wide Web Con- ferences Steering Committee.
  25. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transac- tions of the Association for Computational Linguis- tics, pages 287-302.
  26. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing. ACL.
  27. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Natural Language Processing, pages 1321-1331.
  28. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 3911-3921.
  29. Shuo Zhang and Krisztian Balog. 2019. Auto- completion for data cells in relational tables. In Pro- ceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19, pages 761-770.
  30. Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Trans. Intell. Syst. Technol., 11(2):13:1-13:35.
  31. Shuo Zhang, Zhuyun Dai, Krisztian Balog, and Jamie Callan. 2020. Summarizing and exploring tabular data in conversational search. SIGIR '20, pages 1537-1540.
  32. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103. · arXiv
  33. Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive sur- vey on open-domain question answering. CoRR, abs/2101.00774. · arXiv

Cited by

Sorted by citations

  • StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?

    Chen, Y., Yao, Z., Liu, Y., Xin, A., Ye, J., Yu, J., Hou, L., & Li, J. (2025) · arXiv · Cited by 1 · see citing works

    “However, existing financial benchmarks for LLM agents largely focus on static question-answering tasks (Chen et al., 2021; Zhu et al., 2021; Yin et al., 2023), which are designed to test the financial knowledge of LLMs but fail to capture the dynamics of realistic trading scenarios.”

Authors
Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., & Chua, T.-S.
Affiliations
  • National University of Singapore (SG)
  • Sichuan University (CN)
  • Bloomberg (United States) (US)
ORCID
Venue
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2021
Pages
3277-3287
DOI
10.18653/v1/2021.acl-long.254
OpenAlex
W3174986053
Cited by
193
References
7
Citations by year
2026: 46 · 2025: 48 · 2024: 25 · 2023: 42 · 2022: 28 · 2021: 4
FWCI
11.43
Type
conference paper
Language
en
Open access
Licence
cc-by
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
question answering, benchmark, tabular data, textual content, hybrid data, financial records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review