CoChat
Sign Up
Cited by 34Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale dataset

Open access
Rasool, Zafaryab, Kurniawan, Stefanus, Balugo, Sherwin, Barnett, Scott, Vasa, Rajesh, Chesser, Courtney, Hampstead, Benjamin M., Belleville, Sylvie +2 more·2024·Natural Language Processing Journal
DOI 10.1016/j.nlp.2024.100083
Open this paper in CoChat — free account

Abstract

Document-based Question-Answering (QA) tasks are crucial for precise information retrieval. While some existing work focus on evaluating large language model’s (LLMs) performance on retrieving and answering questions from documents, assessing the LLMs performance on QA types that require exact answer selection from predefined options and numerical extraction is yet to be fully assessed. In this paper, we specifically focus on this underexplored context and conduct empirical analysis of LLMs (GPT-4 and GPT-3.5) on question types, including single-choice, yes–no, multiple-choice, and number extraction questions from documents. We use the CogTale dataset for evaluation, which provide human expert-tagged responses, offering a robust benchmark for precision and factual grounding. We found that LLMs, particularly GPT-4, can precisely answer many single-choice and yes–no questions given relevant context, demonstrating their efficacy in information retrieval tasks. However, their performance diminishes when confronted with multiple-choice and number extraction formats, lowering the overall performance of the models on this task, indicating that these models may not yet be sufficiently reliable for the task. This limits the applications of LLMs on applications demanding precise information extraction and inference from documents, such as meta-analysis tasks. Our work offers a framework for ongoing dataset evaluation, ensuring that LLM applications for information retrieval and document analysis continue to meet evolving standards.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.nlp.2024.100083

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Large language models have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks.

    “On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks.”

    From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Rasool et al., 2024

    The Introduction (p. 1) says LLMs can solve various natural-language-processing tasks, and the Abstract (p. 1) reports success on some document-based question-answering tasks. Our copy does not support the additional examples of drawing or a significant reduction in time spent on routine tasks. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 21

  1. Acharya, A., Singh, B., Onoe, N., 2023. Llm based generation of itemdescription for recommendation system, in: Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1204-1207.
  2. Aher, G.V., Arriaga, R.I., Kalai, A.T., 2023. Using large language models to simulate multiple humans and replicate human subject studies, in: International Conference on Machine Learning, PMLR. pp. 337-371.
  3. Bai, Y., Ying, J., Cao, Y., Lv, X., He, Y ., Wang, X., Yu, J., Zeng, K., Xiao, Y ., Lyu, H., Zhang, J., Li, J., Hou, L., 2023. Benchmarking foundation models with language-model-as-an-examiner. arXiv:2306.04181 . · arXiv
  4. Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q.V., Xu, Y., Fung, P., 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv:2302.04023 . · arXiv
  5. Bian, N., Han, X., Sun, L., Lin, H., Lu, Y., He, B., 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv:2303.16421 . · arXiv
  6. Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N.A., Gardner, M., 2021. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011 . · arXiv
  7. Espejel, J.L., Ettifouri, E.H., Alassan, M.S.Y., Chouham, E.M., Dahhane, W., 2023. Gpt-3.5, gpt-4, or bard? evaluating llms reasoning ability in zeroshot setting and performance boosting through prompts. Natural Language Processing Journal 5, 100032.
  8. Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., Dasigi, P., 2020. Iirc: A dataset of incomplete information reading comprehension questions. arXiv preprint arXiv:2011.07127 . · arXiv
  9. Jin, Q., Dhingra, B., Liu, Z., Cohen, W., Lu, X., 2019. PubMedQA: A dataset for biomedical research question answering, in: Inui, K., Jiang, J., Ng, V ., Wan, X. (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China. pp. 2567-2577. URL: https: //aclanthology.org/D19-1259 , doi: 10.18653/v1/D19-1259 . · DOI · Paper page
  10. Kamalloo, E., Dziri, N., Clarke, C.L., Rafiei, D., 2023. Evaluating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06984 . · arXiv
  11. Kurz, A., Pohl, C., Ramsenthaler, M., Sorg, C., 2009. Cognitive rehabilitation in patients with mild cognitive impairment. International Journal of Geriatric Psychiatry: A journal of the psychiatry of late life and allied sciences 24, 163-168.
  12. Levine, Y., Ram, O., Jannai, D., Lenz, B., Shalev-Shwartz, S., Shashua, A., Leyton-Brown, K., Shoham, Y., 2022. Huge frozen language models as readers for open-domain question answering, in: ICML 2022 Workshop on Knowledge Retrieval and Language Models. URL: https:// openreview.net/forum?id=z3Bxu8xNJaF .
  13. Liu, C., Li, X., Shang, L., Jiang, X., Liu, Q., Lam, E., Wong, N., 2023. Gradually excavating external knowledge for implicit complex question answering, in: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14405-14417.
  14. OpenAI, 2023. Gpt-4 technical report. arXiv:2303.08774 . · arXiv
  15. Pereira, J., Fidalgo, R., Lotufo, R., Nogueira, R., 2023. Visconde: Multidocument qa with gpt-3 and neural reranking, in: European Conference on Information Retrieval, Springer. pp. 534-543.
  16. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D., 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv:2302.06476 . · arXiv
  17. Singh, C., Morris, J., Rush, A.M., Gao, J., Deng, Y., 2023. Tree prompting: Efficient task adaptation without fine-tuning, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6253-6267.
  18. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., et al., 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 . · arXiv
  19. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al., 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 . · arXiv
  20. Vidovich, M.R., Lautenschlager, N.T., Flicker, L., Clare, L., McCaul, K., Almeida, O.P., 2015. The pace study: a randomized clinical trial of cognitive activity strategy training for older people with mild cognitive impairment. The American Journal of Geriatric Psychiatry 23, 360-372.
  21. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al., 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824-24837.

Cited by

Sorted by citations

Authors
Rasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., & Bahar-Fuchs, A.
Affiliations
  • Deakin University (AU)
  • University of Michigan (US)
  • Institut Universitaire de Gériatrie de Montréal (CA)
Corresponding author
Rasool, Zafaryab
ORCID
Venue
Natural Language Processing Journal
Publisher
Elsevier BV
Published
2024-09
Volume
8
Pages
100083
ISSN
2949-7191 (ISSN-L)
DOI
10.1016/j.nlp.2024.100083
OpenAlex
W4399466552
Cited by
34
References
10
Citations by year
2026: 9 · 2025: 20 · 2024: 5
FWCI
5.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Natural Language Processing Journal (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, answer selection, GPT-4, GPT-3.5, multiple-choice questions, yes-no questions, single-choice questions, factual grounding, information retrieval, benchmark evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review