Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@95 measure: Table 1
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
In the July 2010 issue of JAMIA, Matwin et al published an article entitled ‘A new algorithm for reducing the workload of experts in performing systematic reviews.’1 Briefly, the work proposes a factorized variant of the complement Naïve Bayes classifier as an improvement, using weight engineering on the features (FCNB/WE). The prior work of Cohen et al in this area is cited, and the data set made public along with this prior work is used for the evaluation.2 The Matwin et al article compares the authors' proposed system against the early Cohen et al published voting perceptron (VP) classifier results, using the ‘work saved over sampling at 95% recall’ (WSS@95) measure proposed in that paper. However, the article notes that WSS@95 figures were not published in Cohen's later work based on the support vector machine (SVM) classifier,3,4 and states that these figures were not available for comparison. Our team has not published the WSS@95 figures for the SVM classifier previously because our current investigations focus on using classification for a number of different use cases within the systematic review process. Each of these use cases, and potentially each review team and user, may prefer different recall-precision trade offs, and therefore different classification thresholds. Furthermore, this may be affected by the size of the literature base in a given review domain and/or the prevalence of articles meeting the inclusion criteria for that review. Therefore, we have chosen to use the area under the receiver-operating curve (AUC) as our measure of classifier accuracy across the range of possible operating points, instead of the single point measure WSS@95. We are currently conducting a study of systematic reviewer preferences for classification performance within these use cases to better understand how to optimize for these trade-offs. Although not previously published, in fact, we did collect WSS@95 performance figures for the 15 systematic review topic data set used by Matwin et al. The system used is identical to that published in our prior work.4 These results were obtained using five repetitions of twofold crossvalidation, as in Matwin as well as our prior work, and are shown in table 1. As can be seen in the table, the performance of our previously published SVM-based system is generally superior to that of Matwin, outperforming the FCNB/WE system for 12 of 15 topics on WSS@95. On average, our previously published system increases WSS@95 by 0.073 over FCNB/WE. This represents a mean improvement of almost 22%. WSS@95 Comparison of Cohen 20084 and Matwin 20101 Systems across 15 systematic review topics using 5×2 cross-validation FNCB/WE, Matwin et al 20101 factorized variant of the complement Naïve Bayes classifier as an improvement, using weight engineering on the features. WSS@95 Comparison of Cohen 20084 and Matwin 20101 Systems across 15 systematic review topics using 5×2 cross-validation FNCB/WE, Matwin et al 20101 factorized variant of the complement Naïve Bayes classifier as an improvement, using weight engineering on the features. It is interesting that the SVM approach is inferior to our prior VP results for the attention deficit hyperactivity disorder (ADHD) topic, and that FCNB/WE is superior to both SVM and VP for the opioids topic, especially given that the SVM AUC measure is about 0.90 for both of these topics. Both the ADHD and opioids topics have very low article inclusion rates (2.4% and 0.8% respectively) and a relatively small number of positive samples (20 and 15 respectively). Clearly, there is an opportunity for future research in enhancing classifier performance at the very high recall end of the receiver-operating curve, especially with very small numbers of positive samples in the training data. We are pleased to see other researchers investigating the important area of improving the efficiency of the systematic review process. We encourage Matwin et al to continue to improve their system, as we are, and to push the field forward, making these and other potentially effective tools available to support the systematic review process and evidence-based medicine. National Institutes of Health; National Library of Medicine. Not commissioned; not externally peer reviewed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Self-citationA new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100 · cited 1×
“published an article entitled ‘A new algorithm for reducing the workload of experts in performing systematic reviews.’”
BackgroundCross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 1×
“published voting perceptron (VP) classifier results, using the ‘work saved over sampling at 95% recall’ (WSS@95) measure proposed in that paper. However, the article notes that WSS@95 figures were not published in Cohen's later work based on the support vector machine (SVM) classifier,”
MethodSelf-citation
Cited by
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
SWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196
“Cohen (2006) [6] Matwin (2010) [28] Cohen (2008/11) [29, 30] SWIFT-Review (25 trials)”
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
“…including our own recent work (Cohen, 2006; Cohen et al., 2006; Martinez et al., 2008; Cohen et al., 2010; Matwin et al., 2010; Wallace et al., 2010a, 2010b; Cohen, 2011; Wallace and Small, 2011; Kim and Choi, 2012; Jonnalagadda and Petitti, 2013; Miwa et al., 2014; and Howard et al., 2016).”
Background
Metadata
- Authors
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-01
- Volume
- 18
- Issue
- 1
- Pages
- 104.1-104
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2138118601
- Cited by
- 32
- References
- 3
- FWCI
- 0.10
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.