CoChat

Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@95 measure: Table 1

Open access
Cohen, Aaron M · 2011 · Journal of the American Medical Informatics Association
Cited by 32

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Abstract

In the July 2010 issue of JAMIA, Matwin et al published an article entitled ‘A new algorithm for reducing the workload of experts in performing systematic reviews.’1 Briefly, the work proposes a factorized variant of the complement Naïve Bayes classifier as an improvement, using weight engineering on the features (FCNB/WE). The prior work of Cohen et al in this area is cited, and the data set made public along with this prior work is used for the evaluation.2 The Matwin et al article compares the authors' proposed system against the early Cohen et al published voting perceptron (VP) classifier results, using the ‘work saved over sampling at 95% recall’ (WSS@95) measure proposed in that paper. However, the article notes that WSS@95 figures were not published in Cohen's later work based on the support vector machine (SVM) classifier,3,4 and states that these figures were not available for comparison. Our team has not published the WSS@95 figures for the SVM classifier previously because our current investigations focus on using classification for a number of different use cases within the systematic review process. Each of these use cases, and potentially each review team and user, may prefer different recall-precision trade offs, and therefore different classification thresholds. Furthermore, this may be affected by the size of the literature base in a given review domain and/or the prevalence of articles meeting the inclusion criteria for that review. Therefore, we have chosen to use the area under the receiver-operating curve (AUC) as our measure of classifier accuracy across the range of possible operating points, instead of the single point measure WSS@95. We are currently conducting a study of systematic reviewer preferences for classification performance within these use cases to better understand how to optimize for these trade-offs. Although not previously published, in fact, we did collect WSS@95 performance figures for the 15 systematic review topic data set used by Matwin et al. The system used is identical to that published in our prior work.4 These results were obtained using five repetitions of twofold crossvalidation, as in Matwin as well as our prior work, and are shown in table 1. As can be seen in the table, the performance of our previously published SVM-based system is generally superior to that of Matwin, outperforming the FCNB/WE system for 12 of 15 topics on WSS@95. On average, our previously published system increases WSS@95 by 0.073 over FCNB/WE. This represents a mean improvement of almost 22%. WSS@95 Comparison of Cohen 20084 and Matwin 20101 Systems across 15 systematic review topics using 5×2 cross-validation FNCB/WE, Matwin et al 20101 factorized variant of the complement Naïve Bayes classifier as an improvement, using weight engineering on the features. WSS@95 Comparison of Cohen 20084 and Matwin 20101 Systems across 15 systematic review topics using 5×2 cross-validation FNCB/WE, Matwin et al 20101 factorized variant of the complement Naïve Bayes classifier as an improvement, using weight engineering on the features. It is interesting that the SVM approach is inferior to our prior VP results for the attention deficit hyperactivity disorder (ADHD) topic, and that FCNB/WE is superior to both SVM and VP for the opioids topic, especially given that the SVM AUC measure is about 0.90 for both of these topics. Both the ADHD and opioids topics have very low article inclusion rates (2.4% and 0.8% respectively) and a relatively small number of positive samples (20 and 15 respectively). Clearly, there is an opportunity for future research in enhancing classifier performance at the very high recall end of the receiver-operating curve, especially with very small numbers of positive samples in the training data. We are pleased to see other researchers investigating the important area of improving the efficiency of the systematic review process. We encourage Matwin et al to continue to improve their system, as we are, and to push the field forward, making these and other potentially effective tools available to support the systematic review process and evidence-based medicine. National Institutes of Health; National Library of Medicine. Not commissioned; not externally peer reviewed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/jamia.2010.008177

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 32

Metadata

Authors
Cohen, A. M.
Venue
Journal of the American Medical Informatics Association
Published
2011-01
Volume
18
Issue
1
Pages
104.1-104
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W2138118601
Cited by
32
References
3
FWCI
0.10
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.