CoChat

Data extraction methods for systematic review (semi)automation: Update of a living systematic review

Open access
Schmidt, Lena, Finnerty Mutlu, Ailbhe N., Elmore, Rebecca, Olorisade, Babatunde K., Thomas, James, Higgins, Julian P. T. · 2023 · F1000Research
Cited by 44

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

Abstract

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography . Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.12688/f1000research.51117.2

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 68 references

Research works printed but not linked: 35

  1. D Fukumi Tsunoda, 2020, Machine learning e revisão sistemática de literatura automatizada: uma revisão sistemática., Revista Tecnologia e Sociedade., 16
  2. R Ruiz, 2021, Automation in Healthcare Systematic Review., Late Breaking Papers: HCI Applications in Health, Transport, and Industry. Cham.
  3. G Sundaram, 2022, Automating Systematic Literature Reviews with Natural Language Processing and Text Mining: a Systematic Literature Review., arXiv preprint arXiv.
  4. T Zhang, 2022, Information Extraction from the Text Data on Traditional Chinese Medicine: A Review on Tasks, Challenges, and Methods from 2010 to 2021., Evid Based Complement Alternat Med., 2022, 1679589, 10.1155/2022/1679589 · DOI
  5. L McGuinness, 2020, medrxivr: Accessing and searching medRxiv and bioRxivpreprint data in R., JOSS., 10.21105/joss.02651 · DOI · Paper page
  6. C Norman, 2018, Data Extraction and Synthesis in Systematic Reviews of Diagnostic Test Accuracy: A Corpus for Automating and Evaluating the Process., AMIA Annu Symp Proc., 2018, 817-826
  7. F Boudin, 2010, Clinical Information Retrieval using Document and PICO Structure., Assoc. Compu. Linguist., 822-830
  8. Z Luo, 2011, Extracting temporal constraints from clinical research eligibility criteria using conditional random fields., AMIA Annu Symp Proc., 2011, 843-852
  9. J Rathbone, 2017, Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews., Syst Rev., 6, 233, 10.1186/s13643-017-0629-x · DOI · Paper page
  10. A Dhrangadhariya, 2023, Not so weak PICO: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation., JAMIA Open., 6, ooac107, 10.1093/jamiaopen/ooac107 · DOI · Paper page
  11. T Zhang, 2005, Unlocking the power of deep pico extraction: Step-wise medical ner identification., arXiv preprint arXiv.
  12. A Lucic, 2016, Improving Endpoint Detection to Support Automated Systematic Reviews., AMIA Annu Symp Proc., 2016, 1900-1909
  13. B Wallace, 2016, Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision., J Mach Learn Res., 17
  14. J Whitton, 2112, Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations., arXiv preprint arXiv.
  15. B Nye, 2018, A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature., Proc Conf Assoc Comput Linguist Meet., 2018, 197-207
  16. B de Bruijn, 2008, Automated information extraction of key trial design elements from clinical trial publications., AMIA Annu Symp Proc., 141-5
  17. G Singh, 2017, A Neural Candidate-Selector Architecture for Automatic Structured Clinical Text Annotation., Proc ACM Int Conf Inf Knowl Manag., 2017, 1519-1528, 10.1145/3132847.3132989 · DOI · Paper page
  18. I Amini, 2012, Overview of the ALTA., Shared Task., 2012, 124-129
  19. K Raja, 2016, A Hybrid Citation Retrieval Algorithm for Evidence-based Clinical Knowledge Summarization: Combining Concept Extraction, Vector Similarity and Query Expansion for High Precision., CoRR., abs/1609.01597 · arXiv
  20. H Xu, 2010, Mining Biomedical Literature for Terms related to Epidemiologic Exposures., AMIA Annu Symp Proc., 2010, 897-901
  21. K Raja, 2016, Towards Evidence-based Precision Medicine: Extracting Population Information from Biomedical Text using Binary Classifiers and Syntactic Patterns., AMIA Jt Summits Transl Sci Proc., 2016, 203-212
  22. H Zhu, 2012, Automatic extracting of patient-related attributes: disease, age, gender and race., Stud Health Technol Inform., 180, 589-593
  23. D Demner-Fushman, 2018, Finding medication doses in the liteature., AMIA Annu Symp Proc., 2018, 368-376
  24. X Zhang, 2020, Aceso: PICO-Guided Evidence Summarization on Medical Literature., IEEE J Biomed Health Inform., 24, 2663-2670, 10.1109/JBHI.2020.2984704 · DOI · Paper page
  25. B Nye, 2020, Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time., CoRR., abs/2005.10865 · arXiv
  26. K Burnham, 2002, Model Selection and Multimodel Inference (2nd ed.).
  27. E Lehman, 2019, Inferring which medical treatments work from reports of clinical trials., arXiv preprint arXiv.
  28. A Trenta, 2015, Extraction of evidence tables from abstracts of randomized clinical trials using a maximum entropy classifier and global constraints., CoRR, abs.
  29. M Hansen, A method of extracting the number of trial participants from abstracts describing randomized controlled trials., (1758-1109 (Electronic)).
  30. S Chabou, 2015, PICO Extraction by combining the robustness of machine-learning methods with the rule-based methods., 2015 World Congress on Information Technology and Computer Applications., 10.1109/WCITCA.2015.7367038 · DOI · Paper page
  31. M Dawes, 2007, The identification of clinically important elements within medical journal abstracts: Patient-Population-Problem, Exposure-Intervention, Comparison, Outcome, Duration and Results (PECODR)., Inform Prim Care., 15, 9-16
  32. S Amir, 2021, On the impact of random seeds on the fairness of clinical classifiers., 10.18653/v1/2021.naacl-main.299 · DOI
  33. OpenAI., 2023, GPT-4 Technical Report., ArXiv.
  34. C Shaib, 2023, Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)., 10.18653/v1/2023.acl-short.119 · DOI · Paper page
  35. S Wadhwa, 2023, Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs., arXiv preprint arXiv.

Cited by

See all 44

Metadata

Authors
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T.
Venue
F1000Research
Published
2023-10-09
Volume
10
Pages
401
ISSN
2046-1402 (ISSN-L)
OpenAlex
W4387450484
Cited by
44
References
68
FWCI
3.37
Type
journal article
Language
en
Open access
gold
Licence
creativecommons.org/licenses/by/4.0/ (tdm)
Funders
National Institute for Health Research (RM-SR-2017-09-028, DRF-2018-11-ST2-048)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Data extraction methods for systematic review (semi)automation: Update of a living systematic review. https://app.cochat.ai/papers/10.12688/f1000research.51117.2
The paper
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023). Data extraction methods for systematic review (semi)automation: Update of a living systematic review. F1000Research. https://doi.org/10.12688/f1000research.51117.2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.