The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
A systematic review of machine learning models for predicting outcomes of stroke with structured data
Open accessAbstract
BACKGROUND AND PURPOSE: Machine learning (ML) has attracted much attention with the hope that it could make use of large, routinely collected datasets and deliver accurate personalised prognosis. The aim of this systematic review is to identify and critically appraise the reporting and developing of ML models for predicting outcomes after stroke. METHODS: We searched PubMed and Web of Science from 1990 to March 2019, using previously published search filters for stroke, ML, and prediction models. We focused on structured clinical data, excluding image and text analysis. This review was registered with PROSPERO (CRD42019127154). RESULTS: Eighteen studies were eligible for inclusion. Most studies reported less than half of the terms in the reporting quality checklist. The most frequently predicted stroke outcomes were mortality (7 studies) and functional outcome (5 studies). The most commonly used ML methods were random forests (9 studies), support vector machines (8 studies), decision trees (6 studies), and neural networks (6 studies). The median sample size was 475 (range 70-3184), with a median of 22 predictors (range 4-152) considered. All studies evaluated discrimination with thirteen using area under the ROC curve whilst calibration was assessed in three. Two studies performed external validation. None described the final model sufficiently well to reproduce it. CONCLUSIONS: The use of ML for predicting stroke outcomes is increasing. However, few met basic reporting standards for clinical prediction tools and none made their models available in a way which could be used or evaluated. Major improvements in ML study conduct and reporting are needed before it can meaningfully be considered for practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2010) · International Journal of Surgery · doi:10.1016/j.ijsu.2010.02.007 · Cited by 12,156
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147 · cited 5×
“The reporting quality was assessed according to TRIPOD [7] with a few terms adjusted to fit ML methods (See Table 1 for explanations of ML terms).”
BackgroundMethodInfluentialMultiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“Potential ethical challenges of implementing ML models was also addressed in recent studies [42].”
BackgroundThe prevention and handling of the missing data
Kang, H. (2013) · Korean Journal of Anesthesiology · doi:10.4097/kjae.2013.64.5.402 · Cited by 1,728 · cited 1×
“Previous studies have shown that more complicated imputation methods such as multiple imputation [29,30] are better at restoring the natural variability of the missing values than single imputation and retain more useful information than complete case analysis [31].”
BackgroundCollins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · cited 1×
“External validation is an invaluable part of implementing the model in routine clinical practice–it assesses the transportability of the predictions to new data (and hence the generalisability of the model) and should be undertaken before clinical use [21,38].”
Machine learning and neurosurgical outcome prediction: a systematic review
Senders, J. T., Staples, P. C., Karhade, A. V., Zaki, M. M., Gormley, W. B., Broekman, M. L. D., Smith, T. R., & Arnaout, O. (2018) · World Neurosurgery · doi:10.1016/j.wneu.2017.09.149 · Cited by 486 · cited 1×
“We combined published PubMed search filters for stroke [8], ML [9], and prediction models [10].”
Method
Research works printed but not linked: 25
- CO Johnson, 2019, Global, regional, and national burden of stroke, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016, The Lancet Neurology, 18, 439, 10.1016/S1474-4422(19)30034-1 · DOI · Paper page
- H Kamal, 2018, Machine Learning in Acute Ischemic Stroke Neuroimaging, Front Neurol, 9, 945, 10.3389/fneur.2018.00945 · DOI
- R Feng, 2018, Deep learning guided stroke management: a review of clinical applications, J NeuroIntervent Surg, 10, 358, 10.1136/neurintsurg-2017-013355 · DOI
- E-J Lee, 2017, Deep into the Brain: Artificial Intelligence in Stroke Imaging, J Stroke, 19, 277, 10.5853/jos.2017.02054 · DOI · Paper page
- HJA van Os, 2018, Predicting Outcome of Endovascular Treatment for Acute Ischemic Stroke: Potential Value of Machine Learning Algorithms, Front Neurol, 9, 784, 10.3389/fneur.2018.00784 · DOI · Paper page
- M Monteiro, 2018, Using Machine Learning to Improve the Prediction of Functional Outcome in Ischemic Stroke Patients, IEEE/ACM Trans Comput Biol Bioinform, 15, 1953, 10.1109/TCBB.2018.2811471 · DOI
- J Lin, 2018, Prediction of neurologic deterioration based on support vector machine algorithms and serum osmolarity equations, Brain Behav, 8, e01023, 10.1002/brb3.1023 · DOI
- Y Liang, 2019, Comparative Study of Back Propagation Artificial Neural Networks and Logistic Regression Model in Predicting Poor Prognosis after Acute Ischemic Stroke, Open Med (Wars), 14, 324, 10.1515/med-2019-0030 · DOI
- J Heo, 2019, Machine Learning-Based Model for Prediction of Outcomes in Acute Stroke, Stroke, 50, 1263, 10.1161/STROKEAHA.118.024293 · DOI · Paper page
- S Tanioka, 2019, Machine Learning Analysis of Matricellular Proteins and Clinical Variables for Early Prediction of Delayed Cerebral Ischemia After Aneurysmal Subarachnoid Hemorrhage, Mol Neurobiol
- AP Cox, 2016, Predictive analysis for identifying potentially undiagnosed post-stroke spasticity patients in United Kingdom, J Biomed Inform, 60, 328, 10.1016/j.jbi.2016.02.012 · doi:10.1016/j.jbi.2016.02.012
- JF Easton, 2014, Risk factors and prediction of very short term versus short/intermediate term post-stroke mortality: A data mining approach, Computers in Biology and Medicine, 54, 199, 10.1016/j.compbiomed.2014.09.003 · DOI
- C Tjortjis, 2007, Using T3, an improved decision tree classifier, for mining stroke-related medical data, Methods Inf Med, 46, 523, 10.1160/ME0317 · DOI
- J Kruppa, 2014, Probability estimation with machine learning methods for dichotomous and multicategory outcome: applications, Biom J, 56, 564, 10.1002/bimj.201300077 · DOI
- IR Konig, 2007, Practical experiences on the necessity of external validation, Stat Med, 26, 5499, 10.1002/sim.3069 · DOI
- G Celik, 2014, Predicting 10-day mortality in patients with strokes using neural networks and multivariate statistical methods, J Stroke Cerebrovasc Dis, 23, 1506, 10.1016/j.jstrokecerebrovasdis.2013.12.018 · DOI
- M Tokmakci, 2008, The reevaluate statistical results of quality of life in patients with cerebrovascular disease using adaptive network-based fuzzy inference system, Expert Systems with Applications, 34, 958, 10.1016/j.eswa.2006.10.026 · doi:10.1016/j.eswa.2006.10.026
- AR Al Taleb, 2017, Application of Data Mining Techniques to Predict Length of Stay of Stroke Patients
- KC Ho, 2014, Predicting discharge mortality after acute ischemic stroke using balanced data, AMIA Annu Symp Proc, 2014, 1787
- UB Mogensen, 2013, A random forest approach for competing risks based on pseudo-values, Statistics in Medicine, 32, 3102, 10.1002/sim.5775 · DOI
- S-Y Peng, 2010, Random forest can predict 30-day mortality of spontaneous intracerebral hemorrhage with remarkable discrimination, Eur J Neurol, 17, 945, 10.1111/j.1468-1331.2010.02955.x · DOI
- T Li, 2018, HyperTuner: Visual Analytics for Hyperparameter Tuning by Professionals, 11
- S Senanayake, 2019, Machine learning in predicting graft failure following kidney transplantation: A systematic review of published predictive models, International Journal of Medical Informatics, 130, 103957, 10.1016/j.ijmedinf.2019.103957 · DOI
- ND Shah, 2018, Big Data and Predictive Analytics: Recalibrating Expectations, JAMA, 320, 27, 10.1001/jama.2018.5602 · DOI · Paper page
- EW Steyerberg, 2014, Towards better clinical prediction models: seven steps for development and an ABCD for validation, European Heart Journal, 35, 1925, 10.1093/eurheartj/ehu207 · DOI · Paper page
Cited by
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069
Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review
Yin, J., Ngiam, K. Y., & Teo, H. H. (2021) · Journal of Medical Internet Research · doi:10.2196/25759 · Cited by 419
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140
Metadata
- Authors
- Wang, W., Kiik, M., Peek, N., Curcin, V., Marshall, I. J., Rudd, A. G., Wang, Y., Douiri, A., Wolfe, C. D., & Bray, B.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-06-12
- Volume
- 15
- Issue
- 6
- Pages
- e0234722
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W3034655770
- Cited by
- 195
- References
- 16
- FWCI
- 6.62
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A systematic review of machine learning models for predicting outcomes of stroke with structured data. https://app.cochat.ai/papers/10.1371/journal.pone.0234722
- The paper
- Wang, W., Kiik, M., Peek, N., Curcin, V., Marshall, I. J., Rudd, A. G., Wang, Y., Douiri, A., Wolfe, C. D., & Bray, B. (2020). A systematic review of machine learning models for predicting outcomes of stroke with structured data. PLOS ONE. https://doi.org/10.1371/journal.pone.0234722
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.