Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography . Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Self-citationHarnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
Automatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214
Revisiting Relation Extraction in the era of Large Language Models
Wadhwa, S., Amir, S., & Wallace, B. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.868 · Cited by 207
On behalf of the founding members of the ICASR group, Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund, H., Ouzzani, M., Thayer, K., Thomas, J., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0740-7 · Cited by 191
Self-citationTools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188
The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: a living systematic review
John, A., Okolie, C., Eyles, E., Webb, R. T., Schmidt, L., McGuinness, L. A., Olorisade, B. K., Arensman, E., Hawton, K., Kapur, N., et al. (2020) · F1000Research · doi:10.12688/f1000research.25522.1 · Cited by 178
Self-citationThree pitfalls to avoid in machine learning.
Riley, P. (2019) · Nature · doi:10.1038/d41586-019-02307-y · Cited by 142
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
Trialstreamer: A living, automatically updated database of clinical trial reports
Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa163 · Cited by 83
Self-citationSentence retrieval for abstracts of randomized controlled trials.
Chung, G. Y. (2009) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-9-10 · Cited by 83
Bui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70
Machine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65
Self-citationAfzal, M., Alam, F., Malik, K. M., & Malik, G. M. (2020) · Journal of Medical Internet Research · doi:10.2196/19810 · Cited by 65
Campillos-Llanos, L., Valverde-Mateos, A., Capllonch-Carrión, A., & Moreno-Sandoval, A. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01395-z · Cited by 65
Identifying scientific artefacts in biomedical literature: The Evidence Based Medicine use case.
Hassanzadeh, H., Groza, T., & Hunter, J. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.006 · Cited by 62
Improving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62
PICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks.
Jin, D., & Szolovits, P. (2018) · Proceedings of the BioNLP 2018 workshop · doi:10.18653/v1/w18-2308 · Cited by 61
Olorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
Self-citationPICO element detection in medical text without metadata: are first sentences enough?
Huang, K.-C., Chiang, I.-J., Xiao, F., Liao, C.-C., Liu, C. C.-H., & Wong, J.-M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.009 · Cited by 52
Evidence inference 2.0: More data, better models.
DeYoung, J., Lehman, E., Nye, B., Marshall, I., & Wallace, B. C. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.13 · Cited by 49
Kahale, L. A., Elkhoury, R., El Mikati, I., Pardo-Hernandez, H., Khamis, A. M., Schünemann, H. J., Haddaway, N. R., & Akl, E. A. (2022) · F1000Research · doi:10.12688/f1000research.51723.3 · Cited by 48
Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature.
Tian, K., Shirui, Z., & Chunhua, W. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190209 · Cited by 45
Advancing PICO element detection in biomedical text via deep neural networks.
Jin, D., & Szolovits, P. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa256 · Cited by 41
Enhancing evidence-based medicine with natural language argumentative analysis of clinical trials.
Mayer, T., Marro, S., Cabrio, E., & Villata, S. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102098 · Cited by 36
Automatically Identifying Health Outcome Information in MEDLINE Records
Demner-Fushman, D., Few, B., Hauser, S. E., & Thoma, G. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1911 · Cited by 35
Chung, G. Y.-C. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.12.011 · Cited by 34
Toward assessing clinical trial publications for reporting transparency
Kilicoglu, H., Rosemblat, G., Hoang, L., Wadhwa, S., Peng, Z., Malički, M., Schneider, J., & ter Riet, G. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103717 · Cited by 34
Karystianis, G., Thayer, K., Wolfe, M., & Tsafnat, G. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.04.004 · Cited by 31
Schmidt, L., Sinyor, M., Webb, R. T., Marshall, C., Knipe, D., Eyles, E. C., John, A., Gunnell, D., & Higgins, J. P. T. (2023) · Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2023.06.007 · Cited by 31
Self-citationData extraction methods for systematic review (semi)automation: A living review protocol.
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2020) · F1000Research · doi:10.12688/f1000research.22781.2 · Cited by 30
Self-citationPICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
Classification of PICO elements by text features systematically extracted from PubMed abstracts.
Huang, K.-C., Liu, C. C.-H., Yang, S.-S., Xiao, F., Wong, J.-M., Liao, C.-C., & Chiang, I.-J. (2011) · 2011 IEEE International Conference on Granular Computing · doi:10.1109/grc.2011.6122608 · Cited by 26
Aceso: PICO-Guided Evidence Summarization on Medical Literature.
Zhang, X., Geng, P., Zhang, T., Lu, Q., Gao, P., & Mei, J. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2984704 · Cited by 25
Mining characteristics of epidemiological studies from Medline: a case study in obesity.
Karystianis, G., Buchan, I., & Nenadic, G. (2014) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-5-22 · Cited by 23
Stylianou, N., Razis, G., Goulis, D. G., & Vlahavas, I. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101949 · Cited by 22
Automatic endpoint detection to support the systematic review process.
Blake, C., & Lucic, A. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.004 · Cited by 22
Brassey, J., Price, C., Edwards, J., Zlabinger, M., Bampoulidis, A., & Hanbury, A. (2021) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-111126 · Cited by 22
Improving Medical Information Retrieval with PICO Element Detection.
Boudin, F., Shi, L., & Nie, J.-Y. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-12275-0_8 · Cited by 21
Extracting Clinical Trial Design Information from MEDLINE Abstracts.
Hara, K., & Matsumoto, Y. (2007) · New Generation Computing · doi:10.1007/s00354-007-0017-5 · Cited by 21
Combination of conditional random field with a rule based method in the extraction of PICO elements.
Chabou, S., & Iglewski, M. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0699-2 · Cited by 19
Saiz, F. S., Sanders, C., Stevens, R., Nielsen, R., Britt, M., Yuravlivker, L., Preininger, A. M., & Jackson, G. P. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00087 · Cited by 17
Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations
Nye, B. E., DeYoung, J., Lehman, E., Nenkova, A., Marshall, I. J., & Wallace, B. C. (2020) · arXiv · doi:10.48550/arxiv.2010.03550 · Cited by 15
Tsubota, T., Bollegala, D., Zhao, Y., Jin, Y., & Kozu, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104185 · Cited by 11
A neuro-symbolic method for understanding free-text medical evidence.
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab077 · Cited by 11
Extracting PICO elements from RCT abstracts using 1-2gram analysis and multitask classification.
Yuan, X., Xiaoli, L., Shilei, L., Qinwen, S., & Ke, L. (2019) · Proceedings of the third International Conference on Medical and Health Informatics 2019 - ICMHI 2019 · doi:10.1145/3340037.3340043 · Cited by 10
Al-Hussaini, I., Nakajima An, D., Lee, A. J., Bi, S., & Mitchell, C. S. (2022) · 2022 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata55660.2022.10020807 · Cited by 10
MS2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · arXiv · doi:10.48550/arxiv.2104.06486 · Cited by 10
Sanchez-Graillet, O., Witte, C., Grimm, F., & Cimiano, P. (2022) · Journal of Biomedical Semantics · doi:10.1186/s13326-022-00271-7 · Cited by 9
A distantly supervised dataset for automated data extraction from diagnostic studies.
Norman, C., Leeflang, M., Spijker, R., Kanoulas, E., & Névéol, A. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5012 · Cited by 9
Assessment of contextualised representations in detecting outcome phrases in clinical trials
Abaho, M., Bollegala, D., Williamson, P. R., & Dodd, S. (2022) · arXiv · doi:10.48550/arxiv.2203.03547 · Cited by 9
Evaluating automated entity extraction with respect to drug and non-drug treatment strategies.
Guo, J., Blake, C., & Guan, Y. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103177 · Cited by 8
Basu, T., Kumar, S., Kalyan, A., Jayaswal, P., Goyal, P., Pettifer, S., & Jonnalagadda, S. R. (2016) · arXiv · doi:10.48550/arxiv.1606.06424 · Cited by 7
Singh, G., Sabet, Z., Shawe-Taylor, J., & Thomas, J. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-53352-6_12 · Cited by 7
Self-citationBarnett, A. (2022) · F1000Research · doi:10.12688/f1000research.123002.1 · Cited by 7
Tool for filtering PubMed search results by sample size.
Baladrón, C., Santos-Lozano, A., Aguiar, J. M., Lucia, A., & Martín-Hernández, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx155 · Cited by 6
Schmidt, L. (2020) · Harvard Dataverse · doi:10.7910/dvn/lngcoq · Cited by 3
Self-citationAvailable datasets for SR automation.
Schmidt, L. (2021) · Harvard Dataverse · doi:10.7910/dvn/0xtv25 · Cited by 3
Self-citation
Research works printed but not linked: 34
- D Fukumi Tsunoda, 2020, Machine learning e revisão sistemática de literatura automatizada: uma revisão sistemática., Revista Tecnologia e Sociedade., 16
- R Ruiz, 2021, Automation in Healthcare Systematic Review., Late Breaking Papers: HCI Applications in Health, Transport, and Industry. Cham.
- G Sundaram, 2022, Automating Systematic Literature Reviews with Natural Language Processing and Text Mining: a Systematic Literature Review., arXiv preprint arXiv.
- T Zhang, 2022, Information Extraction from the Text Data on Traditional Chinese Medicine: A Review on Tasks, Challenges, and Methods from 2010 to 2021., Evid Based Complement Alternat Med., 2022, 1679589, 10.1155/2022/1679589 · DOI
- L McGuinness, 2020, medrxivr: Accessing and searching medRxiv and bioRxivpreprint data in R., JOSS., 10.21105/joss.02651 · DOI · Paper page
- C Norman, 2018, Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process., AMIA Annu Symp Proc., 2018, 817-826
- F Boudin, 2010, Clinical Information Retrieval using Document and PICO Structure., Assoc. Compu. Linguist., 822-830
- Z Luo, 2011, Extracting temporal constraints from clinical research eligibility criteria using conditional random fields., AMIA Annu Symp Proc., 2011, 843-852
- J Rathbone, 2017, Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews., Syst Rev., 6, 233, 10.1186/s13643-017-0629-x · DOI · Paper page
- A Dhrangadhariya, 2023, Not so weak PICO: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation., JAMIA Open., 6, ooac107, 10.1093/jamiaopen/ooac107 · DOI · Paper page
- T Zhang, 2005, Unlocking the power of deep pico extraction: Step-wise medical ner identification., arXiv preprint arXiv.
- A Lucic, 2016, Improving Endpoint Detection to Support Automated Systematic Reviews., AMIA Annu Symp Proc., 2016, 1900-1909
- B Wallace, 2016, Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision., J Mach Learn Res., 17
- J Whitton, 2112, Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations., arXiv preprint arXiv.
- B Nye, 2018, A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature., Proc Conf Assoc Comput Linguist Meet., 2018, 197-207
- B de Bruijn, 2008, Automated information extraction of key trial design elements from clinical trial publications., AMIA Annu Symp Proc., 141-5
- G Singh, 2017, A Neural Candidate-Selector Architecture for Automatic Structured Clinical Text Annotation., Proc ACM Int Conf Inf Knowl Manag., 2017, 1519-1528, 10.1145/3132847.3132989 · DOI · Paper page
- I Amini, 2012, Overview of the ALTA., Shared Task., 2012, 124-129
- K Raja, 2016, A Hybrid Citation Retrieval Algorithm for Evidence-based Clinical Knowledge Summarization: Combining Concept Extraction, Vector Similarity and Query Expansion for High Precision., CoRR., abs/1609.01597 · arXiv
- H Xu, 2010, Mining Biomedical Literature for Terms related to Epidemiologic Exposures., AMIA Annu Symp Proc., 2010, 897-901
- K Raja, 2016, Towards Evidence-based Precision Medicine: Extracting Population Information from Biomedical Text using Binary Classifiers and Syntactic Patterns., AMIA Jt Summits Transl Sci Proc., 2016, 203-212
- H Zhu, 2012, Automatic extracting of patient-related attributes: disease, age, gender and race., Stud Health Technol Inform., 180, 589-593
- D Demner-Fushman, 2018, Finding medication doses in the liteature., AMIA Annu Symp Proc., 2018, 368-376
- B Nye, 2020, Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time., CoRR., abs/2005.10865 · arXiv
- K Burnham, 2002, Model Selection and Multimodel Inference (2nd ed.).
- E Lehman, 2019, Inferring which medical treatments work from reports of clinical trials., arXiv preprint arXiv.
- A Trenta, 2015, Extraction of evidence tables from abstracts of randomized clinical trials using a maximum entropy classifier and global constraints., CoRR, abs.
- M Hansen, A method of extracting the number of trial participants from abstracts describing randomized controlled trials., (1758-1109 (Electronic)).
- S Chabou, 2015, PICO Extraction by combining the robustness of machine-learning methods with the rule-based methods., 2015 World Congress on Information Technology and Computer Applications., 10.1109/WCITCA.2015.7367038 · DOI · Paper page
- M Dawes, 2007, The identification of clinically important elements within medical journal abstracts: Patient-Population-Problem, Exposure-Intervention, Comparison, Outcome, Duration and Results (PECODR)., Inform Prim Care., 15, 9-16
- S Amir, 2021, On the impact of random seeds on the fairness of clinical classifiers., 10.18653/v1/2021.naacl-main.299 · DOI
- OpenAI., 2023, GPT-4 Technical Report., ArXiv.
- C Shaib, 2023, Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)., 10.18653/v1/2023.acl-short.119 · DOI · Paper page
- S Wadhwa, 2023, Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs., arXiv preprint arXiv.
Cited by
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Data extraction for evidence synthesis using a large language model: A proof-of-concept study
Gartlehner, G., Kahwati, L., Hilscher, R., Thomas, I., Kugley, S., Crotty, K., Viswanathan, M., Nussbaumer‐Streit, B., Booth, G., Erskine, N., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1710 · Cited by 164
Mahmoudi, H., Chang, D., Lee, H., Ghaffarzadegan, N., & Jalali, M. S. (2025) · JMIR AI · doi:10.2196/68097 · Cited by 35
Liu, J., Lai, H., Zhao, W., Huang, J., Xia, D., Liu, H., Luo, X., Wang, B., Pan, B., Hou, L., et al. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002215 · Cited by 19
Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zou, S., He, H., Hossein, S. A., Hasan, B., et al. (2024) · doi:10.1101/2024.09.20.24314108 · Cited by 54
Metadata
- Authors
- Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T.
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2023-10-09
- Volume
- 10
- Pages
- 401
- ISSN
- 2046-1402 (ISSN-L)
- OpenAlex
- W4387450484
- Cited by
- 44
- References
- 70
- FWCI
- 3.37
- Type
- journal article
- Language
- en
- Update notices
- new_version (2025-04-08)
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (tdm)
- Funders
- National Institute for Health Research (RM-SR-2017-09-028, DRF-2018-11-ST2-048)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.