CoChat
Sign Up
Cited by 100Meta-analysis and systematic reviewsText and Document Classification TechnologiesInformation Retrieval and Search Behavior

A new algorithm for reducing the workload of experts in performing systematic reviews

Open access
Matwin, Stan, Kouznetsov, Alexandre, Inkpen, Diana, Frunza, Oana, O'Blenis, Peter·2010·Journal of the American Medical Informatics Association
DOI 10.1136/jamia.2010.004325PMC PMC2995653
Open this paper in CoChat — free accountOpen PDF

Abstract

OBJECTIVE: To determine whether a factorized version of the complement naïve Bayes (FCNB) classifier can reduce the time spent by experts reviewing journal articles for inclusion in systematic reviews of drug class efficacy for disease treatment. DESIGN: The proposed classifier was evaluated on a test collection built from 15 systematic drug class reviews used in previous work. The FCNB classifier was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Weight engineering (WE) techniques were added to reduce underestimation for Medical Subject Headings (MeSH)-based and Publication Type (PubType)-based features. Cross-validation experiments were performed to evaluate the classifier's parameters and performance. MEASUREMENTS: Work saved over sampling (WSS) at no less than a 95% recall was used as the main measure of performance. RESULTS: The minimum workload reduction for a systematic review for one topic, achieved with a FCNB/WE classifier, was 8.5%; the maximum was 62.2% and the average over the 15 topics was 33.5%. This is 15.0% higher than the average workload reduction obtained using a voting perceptron-based automated citation classification system. CONCLUSION: The FCNB/WE classifier is simple, easy to implement, and produces significantly better results in reducing the workload than previously achieved. The results support it being a useful algorithm for machine-learning-based automation of systematic reviews of drug class efficacy for disease treatment.

Integrity signals

Registry

Crossref · 10.1136/jamia.2010.004325

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Another text mining algorithm achieved workload savings ranging from 8.5 to 62% with 15 test datasets, similar to our findings with Abstrackr (9 to 80%), but their results were based on a threshold of a minimum 95% recall of relevant studies, which is too low for systematic reviews.

    “Whilst another text mining algorithm [17] achieved workload savings ranging from 8.5 to 62 % with 15 test datasets, which are similar to our findings with Abstrackr (9 to 80 %), their results were based on a threshold of a minimum 95 % recall of relevant studies, which is too low for systematic reviews.”

    From Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers (2015) · cites this paper as [17]

  • This study and others that have evaluated semi-automated screening with support vector models, semantic vector models, and complement naïve Bayes models indicate that considerable workload savings can be achieved.

    “This study and others that have evaluated semi-automated screening with support vector models [14, 15], semantic vector models [16], and complement naïve Bayes models [17] indicate that considerable workload savings can be achieved.”

    From Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers (2015) · cites this paper as [17]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P.
Affiliations
  • University of Ottawa (CA)
  • University of New Brunswick (CA)
Corresponding author
Kouznetsov, Alexandre
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2010-07
Volume
17
Issue
4
Pages
446-453
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1136/jamia.2010.004325
PMC
PMC2995653
OpenAlex
W2154352790
Cited by
100
References
3
Citations by year
2026: 3 · 2025: 3 · 2024: 8 · 2023: 13 · 2022: 2 · 2021: 6 · 2020: 8 · 2019: 4 · 2018: 4 · 2017: 8 · 2016: 3 · 2015: 16 · 2014: 5 · 2013: 5 · 2012: 7
FWCI
1.50
Type
journal article
Language
en
Open access
Licence
public-domain
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, public-domain)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • Ontario Centres of Excellence
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
systematic reviews, citation classification, Medical Subject Headings
MeSH terms
Algorithms, Automation, Bayes Theorem, Decision Support Techniques, Drug Therapy, Efficiency, Humans, Review Literature as Topic, Information Storage and Retrieval, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review