A Pilot Study Using Machine Learning and Domain Knowledge to Facilitate Comparative Effectiveness Review Updating
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Background. Comparative effectiveness and systematic reviews require frequent and time-consuming updating. Results of earlier screening should be useful in reducing the effort needed to screen relevant articles. Methods. We collected 16,707 PubMed citation classification decisions from 2 comparative effectiveness reviews: interventions to prevent fractures in low bone density (LBD) and off-label uses of atypical antipsychotic drugs (AAP). We used previously written search strategies to guide extraction of a limited number of explanatory variables pertaining to the intervention, outcome, and study design. We empirically derived statistical models (based on a sparse generalized linear model with convex penalties [GLMnet] and a gradient boosting machine [GBM]) that predicted article relevance. We evaluated model sensitivity, positive predictive value (PPV), and screening workload reductions using 11,003 PubMed citations retrieved for the LBD and AAP updates. Results. GLMnet-based models performed slightly better than GBM-based models. When attempting to maximize sensitivity for all relevant articles, GLMnet-based models achieved high sensitivities (0.99 and 1.0 for AAP and LBD, respectively) while reducing projected screening by 55.4% and 63.2%. The GLMnet-based model yielded sensitivities of 0.921 and 0.905 and PPVs of 0.185 and 0.102 when predicting articles relevant to the AAP and LBD efficacy/effectiveness analyses, respectively (using a threshold of P ≥ 0.02). GLMnet performed better when identifying adverse effect relevant articles for the AAP review (sensitivity = 0.981) than for the LBD review (0.685). The system currently requires MEDLINE-indexed articles. Conclusions. We evaluated statistical classifiers that used previous classification decisions and explanatory variables derived from MEDLINE indexing terms to predict inclusion decisions. This pilot system reduced workload associated with screening 2 simulated comparative effectiveness review updates by more than 50% with minimal loss of relevant articles.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Domain adaptation with structural correspondence learning
Blitzer, J., McDonald, R., & Pereira, F. (2006) · Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06 · doi:10.3115/1610075.1610094 · Cited by 1,543
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838
Maher, A. R., Maglione, M., Bagley, S., Suttorp, M., Hu, J.-H., Ewing, B., Wang, Z., Timmer, M., Sultzer, D., & Shekelle, P. G. (2011) · JAMA · doi:10.1001/jama.2011.1360 · Cited by 558
Self-citationJadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P., Moher, M., & Moher, D. (1998) · JAMA · doi:10.1001/jama.280.3.278 · Cited by 534
MacLean, C., Newberry, S., Maglione, M., McMahon, M., Ranganath, V., Suttorp, M., Mojica, W., Timmer, M., Alexander, A., McNamara, M., et al. (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-3-200802050-00198 · Cited by 529
Self-citationSaag, K. G., Zanchetta, J. R., Devogelaer, J., Adler, R. A., Eastell, R., See, K., Krege, J. H., Krohn, K., & Warner, M. R. (2009) · Arthritis & Rheumatism · doi:10.1002/art.24879 · Cited by 493 · cited 1×
“Figure 1 shows a stylized example adapted from the LBD search strategy and 1 citation.(9,28) Data related to generic study characteristics (such as randomized controlled trial [RCT] or human study) were extracted from all studies.”
MethodReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 3×
“Our results are in accord with prior attempts using machine learning to facilitate comparative effectiveness review data collection.(12,13,18) For example, an active learning model achieved 50% workload reductions and 100% sensitivity in several cases.”
MethodResultSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 4×
“For example, an active learning model achieved 50% workload reductions and 100% sensitivity in several cases.(18) In contrast to prior studies, we adopted a more parsimonious approach that focused on relatively few study design characteristics (publication type, demographic groups, and statistical design), interventio…”
ResultInfluentialText Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177
Towards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101 · cited 2×
“Some studies added domain knowledge when classifying features using, for example, United Medical Language System (UMLS).(17) Researchers have also used numerous algorithms to model relevance as a function of these many thousands of potential explanatory variables, including variants of a support vector machine (SVM),(…”
Method
Cited by
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialHealth Information Technology: An Updated Systematic Review With a Focus on Meaningful Use
Jones, S. S., Rudin, R. S., Perry, T., & Shekelle, P. G. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1531 · Cited by 475
Self-citationO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Crandall, C. J., Newberry, S. J., Diamant, A., Lim, Y.-W., Gellad, W. F., Booth, M. J., Motala, A., & Shekelle, P. G. (2014) · Annals of Internal Medicine · doi:10.7326/m14-0317 · Cited by 254
Self-citationPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
“52 Subsequent studies 15 - 25 , 26 - 29 , 32 , 33 , 36 have explored different feature spaces ( words or Active learning 54 - 56 is the process of using a classification model's predictions to iteratively select training data.”
Background
Metadata
- Authors
- Dalal, S. R., Shekelle, P. G., Hempel, S., Newberry, S. J., Motala, A., & Shetty, K. D.
- Venue
- Medical Decision Making
- Publisher
- SAGE Publications
- Published
- 2012-09-07
- Volume
- 33
- Issue
- 3
- Pages
- 343-355
- ISSN
- 0272-989X (ISSN-L)
- 1552-681X
- OpenAlex
- W2121932021
- Cited by
- 32
- References
- 11
- FWCI
- 2.48
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Pilot Study Using Machine Learning and Domain Knowledge to Facilitate Comparative Effectiveness Review Updating. https://app.cochat.ai/papers/10.1177/0272989x12457243
- The paper
- Dalal, S. R., Shekelle, P. G., Hempel, S., Newberry, S. J., Motala, A., & Shetty, K. D. (2013). A Pilot Study Using Machine Learning and Domain Knowledge to Facilitate Comparative Effectiveness Review Updating. Medical Decision Making. https://doi.org/10.1177/0272989x12457243
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.