Performance of two large language models for data extraction in evidence synthesis
Open accessAbstract
Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT‐4) for extracting pre‐specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT‐4. GPT‐4 required use of a third‐party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT‐4 with the plug‐in was lower (68.8%); however, most of the errors were due to the plug‐in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT‐4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open‐source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most evaluations of LLMs for data extraction have been limited to single models, small or restricted datasets, or single extraction runs.
“Although these proof-of-concept studies have reported encouraging results and the potential to substantially reduce manual workload and human error, most evaluations have been limited to single models, small or restricted datasets, or single extraction runs (15–20).”
From Performance of large language models and prompt engineering strategies for data extraction in systematic reviews (2026) · cites this paper as [20]
Claude 2 achieved an accuracy of approximately 96% for extracting predefined data elements from systematic review articles.
“Claude 2 achieving an accuracy of approximately 96% (20)”
From Performance of large language models and prompt engineering strategies for data extraction in systematic reviews (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reich, K., Armstrong, A. W., Foley, P. A., Song, M., Wasfi, Y. S., Randazzo, B. P., Li, S., Shen, Y., & Gordon, K. B. (2017) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2016.11.042 · Cited by 762 · see citing works
Thaçi, D., Blauvelt, A., Reich, K., Tsai, T., Vanaclocha, F., Kingo, K., Ziv, M., Pinter, A., Hugot, S., You, R., et al. (2015) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2015.05.013 · Cited by 548 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
Mathes, T., Klaßen, P., & Pieper, D. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0431-4 · Cited by 240 · see citing works
Warren, R. B., Blauvelt, A., Poulin, Y. P., Beeck, S., Kelly, M., Wu, T., Geng, Z., & Paul, C. F. (2020) · British Journal of Dermatology · doi:10.1111/bjd.19341 · Cited by 210 · see citing works
Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E. E., & Zayed, T. (2023) · Systems · doi:10.3390/systems11070351 · Cited by 202 · see citing works
Reich, K., Gooderham, M., Green, L., Bewley, A., Zhang, Z., Khanskaya, I., Day, R. M., Goncalves, J., Shah, K., Piguet, V., et al. (2016) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.14015 · Cited by 200 · see citing works
Reich, K., Pinter, A., Lacour, J. ‐., Ferrándiz, C., Micali, G., French, L. E., Lomaga, M. A., Dutronc, Y., Henneges, C., Wilhelm, S., et al. (2017) · British Journal of Dermatology · doi:10.1111/bjd.15666 · Cited by 190 · see citing works
Papp, K. A., Merola, J. F., Gottlieb, A. B., Griffiths, C. E. M., Cross, N., Peterson, L., Cioffi, C., & Blauvelt, A. (2018) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2018.03.037 · Cited by 188 · see citing works
Blauvelt, A., Papp, K. A., Gottlieb, A. B., Jarell, A., Reich, K., Maari, C., Gordon, K. B., Ferris, L. K., Langley, R. G., Tada, Y., et al. (2019) · British Journal of Dermatology · doi:10.1111/bjd.18851 · Cited by 175 · see citing works
Data extraction for evidence synthesis using a large language model: A proof-of-concept study
Gartlehner, G., Kahwati, L., Hilscher, R., Thomas, I., Kugley, S., Crotty, K., Viswanathan, M., Nussbaumer‐Streit, B., Booth, G., Erskine, N., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1710 · Cited by 164 · cited 1× · see citing works
“9 Few articles examine zero-shot prompting of LLMs for data extraction for systematic reviews, 9 – 14 which involves extracting and summarizing data elements from text using a plain language query without additional model training.”
Self-citationGlatt, S., Helmer, E., Haier, B., Strimenopoulou, F., Price, G., Vajjah, P., Harari, O., Lambert, J., & Shaw, S. (2016) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.13185 · Cited by 142 · see citing works
Lebwohl, M., Blauvelt, A., Paul, C. F., et al. (2018) · Journal of the American Academy of Dermatology · doi:10.1016/j.jaad.2018.04.013 · Cited by 131 · see citing works
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106 · see citing works
Resource use during systematic review production varies widely: a scoping review
Nussbaumer-Streit, B., Ellen, M., Klerings, I., Sfetcu, R., Riva, N., Mahmić-Kaknjo, M., Poulentzas, G., Martinez, P., Baladia, E., Ziganshina, L. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.019 · Cited by 98 · see citing works
Self-citationSchopow, N., Osterhoff, G., & Baur, D. (2023) · JMIR Medical Informatics · doi:10.2196/48933 · Cited by 82 · see citing works
Bagel, J., Nia, J. K., Hashim, P. W., Patekar, M., de Vera, A., Hugot, S., Sheng, K., Xia, S., Gilloteau, I., Muscianisi, E., et al. (2018) · Dermatology and Therapy · doi:10.1007/s13555-018-0265-y · Cited by 75 · see citing works
Application ChatGPT in conducting systematic reviews and meta-analyses
Mahuli, S. A., Rai, A., Mahuli, A. V., & Kumar, A. (2023) · British Dental Journal · doi:10.1038/s41415-023-6132-y · Cited by 61 · see citing works
Li, T., Saldanha, I. J., Jap, J., Smith, B. T., Canner, J., Hutfless, S. M., Branch, V., Carini, S., Chan, W., de Bruijn, B., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.07.005 · Cited by 54 · see citing works
Zero-Shot Information Extraction for Clinical Meta-Analysis using Large Language Models
Kartchner, D., Ramalingam, S. S., Al-Hussaini, I., et al. (2023) · Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/2023.bionlp-1.37 · Cited by 32 · see citing works
How good are large language models for automated data extraction from randomized trials?Preprint
Sun, Z., Zhang, R., Doi, S. A. R., Furuya‐Kanamori, L., Yu, T., Lin, L., & Xu, C. (2024) · medRxiv · doi:10.1101/2024.02.20.24303083 · Cited by 15 · see citing works
Barbero (2014) · doi:10.1016/j · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [7]7. OpenAI R. GPT-4 technical report. arXiv, 2303-08774. 2023.
Cited by
Sorted by citations
Lieberum, J.-L., Toews, M., Metzendorf, M.-I., Heilmeyer, F., Siemens, W., Haverkamp, C., Böhringer, D., Meerpohl, J. J., & Eisele-Metzger, A. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111746 · Cited by 148 · see citing works
Accelerating clinical evidence synthesis with large language models
Wang, Z., Cao, L., Danek, B., Jin, Q., Lu, Z., & Sun, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01840-7 · Cited by 69 · see citing works
Language models for data extraction and risk of bias assessment in complementary medicine
Lai, H., Liu, J., Bai, C., Liu, H., Pan, B., Luo, X., Hou, L., Zhao, W., Xia, D., Tian, J., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01457-w · Cited by 59 · see citing works
Mahmoudi, H., Chang, D., Lee, H., Ghaffarzadegan, N., & Jalali, M. S. (2025) · JMIR AI · doi:10.2196/68097 · Cited by 35 · see citing works
Automation of Systematic Reviews with Large Language Models
Cao, C., Arora, R., Cento, P., Budak, A., Manta, K., Farahani, E., Cecere, M., Selemon, A., Sang, J., Gong, L. X., et al. (2025) · medRxiv · doi:10.1101/2025.06.13.25329541 · Cited by 33 · see citing works
Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews
Gartlehner, G., Kugley, S., Crotty, K., Viswanathan, M., Dobrescu, A., Nussbaumer-Streit, B., Booth, G., Treadwell, J. R., Han, J. M., Wagner, J., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-25-00739 · Cited by 36 · see citing works
Self-citationCollaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zou, S., He, H., Hossein, S. A., Hasan, B., et al. (2024) · doi:10.1101/2024.09.20.24314108 · Cited by 54 · see citing works
“[36] However, this study focused on a few data elements, limiting the generalizability of their approach.”
BackgroundOami, T., Okada, Y., Maeda, K., & Nakada, T.-A. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1799623 · Cited by 4 · see citing works
“Although these proof-of-concept studies have reported encouraging results and the potential to substantially reduce manual workload and human error, most evaluations have been limited to single models, small or restricted datasets, or single extraction runs (15–20).”
- Authors
- Konet, A., Thomas, I., Gartlehner, G., Kahwati, L., Hilscher, R., Kugley, S., Crotty, K., Viswanathan, M., & Chew, R.
- Affiliations
- RTI International (US)
- Universität für Weiterbildung Krems (AT)
- Corresponding author
- Konet, Amanda
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2024-06-19
- Volume
- 15
- Issue
- 5
- Pages
- 818-824
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W4399812563
- Semantic Scholar
- b719f3b74f5025aeaa0ed3ed8707e8557bce6654
- Cited by
- 79
- References
- 25
- Citations by year
- 2026: 34 · 2025: 41 · 2024: 4
- FWCI
- 22.05
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Research Synthesis Methods (publishedVersion, other-oa)
- PubMed (publishedVersion)
- Funders
- RTI International
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- large language models, data extraction, evidence synthesis, Claude 2, GPT-4, PDF parsing, extraction accuracy, LLM comparison
- MeSH terms
- Systematic Reviews as Topic, Algorithms, Humans, Language, Natural Language Processing, Programming Languages, Research Design, Software, Reproducibility of Results, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
