Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Open accessThe DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Abstract
Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography . Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Self-citationHarnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472
Research works printed but not linked: 35
- D Fukumi Tsunoda, 2020, Machine learning e revisão sistemática de literatura automatizada: uma revisão sistemática., Revista Tecnologia e Sociedade., 16
- R Ruiz, 2021, Automation in Healthcare Systematic Review., Late Breaking Papers: HCI Applications in Health, Transport, and Industry. Cham.
- G Sundaram, 2022, Automating Systematic Literature Reviews with Natural Language Processing and Text Mining: a Systematic Literature Review., arXiv preprint arXiv.
- T Zhang, 2022, Information Extraction from the Text Data on Traditional Chinese Medicine: A Review on Tasks, Challenges, and Methods from 2010 to 2021., Evid Based Complement Alternat Med., 2022, 1679589, 10.1155/2022/1679589 · DOI
- L McGuinness, 2020, medrxivr: Accessing and searching medRxiv and bioRxivpreprint data in R., JOSS., 10.21105/joss.02651 · DOI · Paper page
- C Norman, 2018, Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process., AMIA Annu Symp Proc., 2018, 817-826
- F Boudin, 2010, Clinical Information Retrieval using Document and PICO Structure., Assoc. Compu. Linguist., 822-830
- Z Luo, 2011, Extracting temporal constraints from clinical research eligibility criteria using conditional random fields., AMIA Annu Symp Proc., 2011, 843-852
- J Rathbone, 2017, Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews., Syst Rev., 6, 233, 10.1186/s13643-017-0629-x · DOI · Paper page
- A Dhrangadhariya, 2023, Not so weak PICO: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation., JAMIA Open., 6, ooac107, 10.1093/jamiaopen/ooac107 · DOI · Paper page
- T Zhang, 2005, Unlocking the power of deep pico extraction: Step-wise medical ner identification., arXiv preprint arXiv.
- A Lucic, 2016, Improving Endpoint Detection to Support Automated Systematic Reviews., AMIA Annu Symp Proc., 2016, 1900-1909
- B Wallace, 2016, Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision., J Mach Learn Res., 17
- J Whitton, 2112, Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations., arXiv preprint arXiv.
- B Nye, 2018, A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature., Proc Conf Assoc Comput Linguist Meet., 2018, 197-207
- B de Bruijn, 2008, Automated information extraction of key trial design elements from clinical trial publications., AMIA Annu Symp Proc., 141-5
- G Singh, 2017, A Neural Candidate-Selector Architecture for Automatic Structured Clinical Text Annotation., Proc ACM Int Conf Inf Knowl Manag., 2017, 1519-1528, 10.1145/3132847.3132989 · DOI · Paper page
- I Amini, 2012, Overview of the ALTA., Shared Task., 2012, 124-129
- K Raja, 2016, A Hybrid Citation Retrieval Algorithm for Evidence-based Clinical Knowledge Summarization: Combining Concept Extraction, Vector Similarity and Query Expansion for High Precision., CoRR., abs/1609.01597 · arXiv
- H Xu, 2010, Mining Biomedical Literature for Terms related to Epidemiologic Exposures., AMIA Annu Symp Proc., 2010, 897-901
- K Raja, 2016, Towards Evidence-based Precision Medicine: Extracting Population Information from Biomedical Text using Binary Classifiers and Syntactic Patterns., AMIA Jt Summits Transl Sci Proc., 2016, 203-212
- H Zhu, 2012, Automatic extracting of patient-related attributes: disease, age, gender and race., Stud Health Technol Inform., 180, 589-593
- D Demner-Fushman, 2018, Finding medication doses in the liteature., AMIA Annu Symp Proc., 2018, 368-376
- X Zhang, 2020, Aceso: PICO-Guided Evidence Summarization on Medical Literature., IEEE J Biomed Health Inform., 24, 2663-2670, 10.1109/JBHI.2020.2984704 · DOI · Paper page
- B Nye, 2020, Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time., CoRR., abs/2005.10865 · arXiv
- K Burnham, 2002, Model Selection and Multimodel Inference (2nd ed.).
- E Lehman, 2019, Inferring which medical treatments work from reports of clinical trials., arXiv preprint arXiv.
- A Trenta, 2015, Extraction of evidence tables from abstracts of randomized clinical trials using a maximum entropy classifier and global constraints., CoRR, abs.
- M Hansen, A method of extracting the number of trial participants from abstracts describing randomized controlled trials., (1758-1109 (Electronic)).
- S Chabou, 2015, PICO Extraction by combining the robustness of machine-learning methods with the rule-based methods., 2015 World Congress on Information Technology and Computer Applications., 10.1109/WCITCA.2015.7367038 · DOI · Paper page
- M Dawes, 2007, The identification of clinically important elements within medical journal abstracts: Patient-Population-Problem, Exposure-Intervention, Comparison, Outcome, Duration and Results (PECODR)., Inform Prim Care., 15, 9-16
- S Amir, 2021, On the impact of random seeds on the fairness of clinical classifiers., 10.18653/v1/2021.naacl-main.299 · DOI
- OpenAI., 2023, GPT-4 Technical Report., ArXiv.
- C Shaib, 2023, Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)., 10.18653/v1/2023.acl-short.119 · DOI · Paper page
- S Wadhwa, 2023, Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs., arXiv preprint arXiv.
Cited by
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Data extraction for evidence synthesis using a large language model: A proof-of-concept study
Gartlehner, G., Kahwati, L., Hilscher, R., Thomas, I., Kugley, S., Crotty, K., Viswanathan, M., Nussbaumer‐Streit, B., Booth, G., Erskine, N., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1710 · Cited by 164
Mahmoudi, H., Chang, D., Lee, H., Ghaffarzadegan, N., & Jalali, M. S. (2025) · JMIR AI · doi:10.2196/68097 · Cited by 35
Liu, J., Lai, H., Zhao, W., Huang, J., Xia, D., Liu, H., Luo, X., Wang, B., Pan, B., Hou, L., et al. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002215 · Cited by 19
Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zou, S., He, H., Hossein, S. A., Hasan, B., et al. (2024) · doi:10.1101/2024.09.20.24314108 · Cited by 54
Metadata
- Authors
- Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T.
- Venue
- F1000Research
- Publisher
- F1000 Research Ltd
- Published
- 2023-10-09
- Volume
- 10
- Pages
- 401
- ISSN
- 2046-1402 (ISSN-L)
- OpenAlex
- W4387450484
- Cited by
- 44
- References
- 68
- FWCI
- 3.37
- Type
- journal article
- Language
- en
- Update notices
- new_version (2025-04-08)
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (tdm)
- Funders
- National Institute for Health Research (RM-SR-2017-09-028, DRF-2018-11-ST2-048)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Data extraction methods for systematic review (semi)automation: Update of a living systematic review. https://app.cochat.ai/papers/10.12688/f1000research.51117.2
- The paper
- Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023). Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Research. https://doi.org/10.12688/f1000research.51117.2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.