A new iterative method to reduce workload in systematic review process
Open accessAbstract
High cost for systematic review of biomedical literature has generated interest in decreasing overall workload. This can be done by applying natural language processing techniques to 'automate' the classification of publications that are potentially relevant for a given question. Existing solutions need training using a specific supervised machine-learning algorithm and feature-extraction system separately for each systematic review. We propose a system that only uses the input and feedback of human reviewers during the course of review. As the reviewers classify articles, the query is modified using a simple relevance feedback algorithm, and the semantically closest document to the query is presented. An evaluation of our approach was performed using a set of 15 published drug systematic reviews. The number of articles that needed to be reviewed was substantially reduced (ranging from 6% to 30% for a 95% recall).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other data mining algorithms have achieved similar (40%) workload savings but recall (identifying relevant records) was lower (90 to 95%).
“Other data mining algorithms have achieved similar (40 %) workload savings [16] but recall (identifying relevant records) was lower (90 to 95 %), partly because testing was performed on datasets without a specifically associated research question.”
From Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers (2015) · cites this paper as [16]
The Results (page not supplied) report workload reductions of 0–68% for a supervised system at 95% recall, so a 40% saving is possible within the reported range, but the median was 18%. The Discussion says the authors’ own system could not reliably combine 40% savings with 90–95% recall. The Results say the dataset’s research questions were unknown, but do not establish that this caused lower recall. (LLM, conf=0.86)
Experimental — accuracy not yet measuredActive learning for systematic review screening can reduce the human effort required to identify relevant documents.
“Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review.”
From Statistical stopping criteria for automated screening in systematic reviews (2020) · cites this paper as [6]
This study and others that have evaluated semi-automated screening with support vector models, semantic vector models, and complement naïve Bayes models indicate that considerable workload savings can be achieved.
“This study and others that have evaluated semi-automated screening with support vector models [14, 15], semantic vector models [16], and complement naïve Bayes models [17] indicate that considerable workload savings can be achieved.”
From Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers (2015) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 24 · Cited by 61
Research works printed but not linked: 4
- ASHP Foundation Demonstrating Pharmacists' Value: A Systematic Evidence Review Request for Proposals Demonstrating Pharmacists' Value: A Systematic Evidence Review Request for Proposals 2 2010 Retrieved May 17, 2011, from http://www.ashpfoundation.org/MainMenuCategories/ResearchResourceCenter/FundingOpportunities/DemonstratingPharmacistsValueASystematicEvidenceReviewRequestforPro posals.aspx
- Higgins JPT Sally Green P Cochrane handbook for systematic reviews of interventions 2011 Vol. 4 John Wiley & Sons
- Jonnalagadda S An effective approach to biomedical information extraction with limited training data (PhD Dissertation, Arizona State University) (PhD) 2011 Arizona State University Phoenix
- Widdows D Cohen T The Semantic Vectors Package: New Algorithms and Public Tools for Distributional Semantics 2010 Vol. 1 43 Fourth IEEE International Conference on Semantic Computing
- Authors
- Jonnalagadda, S., & Petitti, D.
- Affiliations
- Mayo Clinic (US)
- Arizona State University (US)
- Venue
- International Journal of Computational Biology and Drug Design
- Publisher
- Inderscience Publishers
- Published
- 2013
- Volume
- 6
- Issue
- 1/2
- Pages
- 5
- ISSN
- 1756-0756 (ISSN-L)
- 1756-0764
- PMC
- PMC3787693
- OpenAlex
- W1982663268
- Cited by
- 61
- References
- 24
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 9 · 2023: 3 · 2022: 5 · 2021: 3 · 2020: 8 · 2019: 4 · 2018: 3 · 2017: 4 · 2016: 4 · 2015: 8 · 2014: 3
- FWCI
- 14.51
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- International Journal of Computational Biology and Drug Design (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Kaiser Permanente
- U.S. National Library of Medicine (K99LM011389)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- systematic reviews, relevance feedback, iterative search, workload reduction, natural language processing, drug review, literature search automation, query modification
- MeSH terms
- Abstracting and Indexing, Systematic Reviews as Topic, Algorithms, Feedback, Humans, Models, Theoretical, Natural Language Processing, Publications, Semantics, Drug Design, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
