CoChat
Sign Up
Cited by 70Meta-analysis and systematic reviewsText and Document Classification TechnologiesInformation Retrieval and Search Behavior

Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers

Open access
Bekhuis, Tanja, Demner-Fushman, Dina·2012·Artificial Intelligence in Medicine
DOI 10.1016/j.artmed.2012.05.002PMC PMC3393813
Open this paper in CoChat — free account

Abstract

Objectives To investigate whether (1) machine learning classifiers can help identify nonrandomized studies eligible for full-text screening by systematic reviewers; (2) classifier performance varies with optimization; and (3) the number of citations to screen can be reduced.Methods We used an open-source, data-mining suite to process and classify biomedical citations that point to mostly nonrandomized studies from 2 systematic reviews. We built training and test sets for citation portions and compared classifier performance by considering the value of indexing, various feature sets, and optimization. We conducted our experiments in 2 phases. The design of phase I with no optimization was: 4 classifiers × 3 feature sets × 3 citation portions. Classifiers included k-nearest neighbor, naïve Bayes, complement naïve Bayes, and evolutionary support vector machine. Feature sets included bag of words, and 2- and 3-term n-grams. Citation portions included titles, titles and abstracts, and full citations with metadata. Phase II with optimization involved a subset of the classifiers, as well as features extracted from full citations, and full citations with overweighted titles. We optimized features and classifier parameters by manually setting information gain thresholds outside of a process for iterative grid optimization with 10-fold cross-validations. We independently tested models on data reserved for that purpose and statistically compared classifier performance on 2 types of feature sets. We estimated the number of citations needed to screen by reviewers during a second pass through a reduced set of citations.Results In phase I, the evolutionary support vector machine returned the best recall for bag of words extracted from full citations; the best classifier with respect to overall performance was k-nearest neighbor. No classifier attained good enough recall for this task without optimization. In phase II, we boosted performance with optimization for evolutionary support vector machine and complement naïve Bayes classifiers. Generalization performance was better for the latter in the independent tests. For evolutionary support vector machine and complement naïve Bayes classifiers, the initial retrieval set was reduced by 46% and 35%, respectively.Conclusions Machine learning classifiers can help identify nonrandomized studies eligible for full-text screening by systematic reviewers. Optimization can markedly improve performance of classifiers. However, generalizability varies with the classifier. The number of citations to screen during a second independent pass through the citations can be substantially reduced.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.artmed.2012.05.002

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Employing machine learning tools as the second screener in a reviewer pair could overcome concerns about reliability.

    “Bekhuis et al. [24–26] found that employing machine learning tools as the second screener in a reviewer pair could overcome concerns about reliability.”

    From Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool (2018) · cites this paper as [24]

    The Introduction describes a machine-learning system that reduces the citations for a second, independent screening pass and says this could help control reviewer bias and error. But that second screening is performed by different human teammates; the paper does not propose the tool as the second screener in a reviewer pair or say it would overcome reliability concerns on its own (Introduction; no page number available). (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach.

    “When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”

    From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [61]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

Research works printed but not linked: 3

  1. [27]Cochrane Collaboration Selecting Studies for your Review: Practicalities of Sifting Accessed: 7 August 2011 http://www.cochrane-net.org/openlearning/HTML/mod8-3.htm
  2. [29]PubMed.gov: US National Library of Medicine, National Institutes of Health Accessed: 3 August 2011 http://www.ncbi.nlm.nih.gov/pubmed/
  3. [32]Mierswa I Wurst M Klinkenberg R Scholz M Euler T YALE (now RapidMiner): rapid prototyping for complex data mining tasks Proc ACM SIGKDD Int Conf on Knowl Discov Data Mining 2006

Cited by

Sorted by citations

Related

Authors
Bekhuis, T., & Demner-Fushman, D.
Affiliations
University of Pittsburgh (US)
Corresponding author
Bekhuis, Tanja
ORCID
Bekhuis, Tanja
Venue
Artificial Intelligence in Medicine
Publisher
Elsevier BV
Published
2012-07
Volume
55
Issue
3
Pages
197-207
ISSN
0933-3657 (ISSN-L)
DOI
10.1016/j.artmed.2012.05.002
PMC
PMC3393813
OpenAlex
W2063198586
Cited by
70
References
50
Citations by year
2026: 1 · 2025: 4 · 2024: 7 · 2023: 3 · 2022: 4 · 2021: 6 · 2020: 7 · 2019: 3 · 2018: 5 · 2017: 5 · 2016: 9 · 2015: 11 · 2014: 2 · 2013: 3
FWCI
1.41
Type
journal article
Language
en
Open access
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Artificial Intelligence in Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institute of Dental and Craniofacial Research (5 T15 LM/DE07059)
  • U.S. National Library of Medicine (R00LM010943, K99LM010943)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
non-randomized studies, full-text screening, systematic reviews, citation classification, machine learning classifiers, complement naive Bayes, k-nearest neighbors, n-gram features, information gain feature selection, classifier optimization
MeSH terms
Algorithms, Artificial Intelligence, Bayes Theorem, Humans, Medical Informatics, Review Literature as Topic, Biomedical Research, Data Mining, Support Vector Machine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review