The need to approximate the use-case in clinical machine learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, data leakage due to identity confounding is not limited to deep neural networks, and has been uncovered using random forests (Saeb et al., 2017; Chaibub Neto et al., 2019; Tougui et al., 2021) and support vector machines (Tougui et al., 2021).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“For example, if a study trains a classifier to identify individual subjects with a disease, then there may be stronger bias when the study involves fewer participants (Saeb et al., 2017).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“This conclusion mirrors results from studies examining a variety of other types of data and classifier models (Saeb et al., 2017; Wen et al., 2018, 2020; Chaibub Neto et al., 2019; Tougui et al., 2021; Tampu et al., 2022).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 2×
“The number of trees for the random forest classifier was set to 50, which was optimized based on the out of bag error [22].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“Machine learning has grown massively over the past decades, with countless applications in technology, marketing, and science [1].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Inappropriatevalidationprocedures,ontheotherhand,canlead to unreliable results, which might contribute to the problem of irreproducibility of research findings [31, 32] and thereby undermine the trust in both medicine and data science.”
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · cited 1×
“Splitting data into training and test subsets can be done using various methods, such as leave-one-out, leave-p-out, k-fold, and Monte-Carlo sampling [18].”
The Rise of Consumer Health Wearables: Promises and Barriers
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001953 · Cited by 1,235 · cited 1×
“A driving factor is that people carry these devices with them most of the time, and thus a growing amount of daily life data, such as physical activities [5, 6], is becoming available.”
Research works printed but not linked: 1
- Google Scholar. Google 2016 https://scholar.google.com (6 April 2017, date last accessed).
Cited by
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802
Self-citationClassification of Brain Tumors from MRI Images Using a Convolutional Neural Network
Badža, M. M., & Barjaktarović, M. Č. (2020) · Applied Sciences · doi:10.3390/app10061999 · Cited by 686
“Figure 3 shows confusion matrices for the record-wise 10 -fold cross-validation approach for testing data obtained from the original dataset.”
MethodMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
“Another example of leakage arises when repeated measures of an individual are split across train and test set, the algorithm then learning to recognize the individual patient rather than markers of a condition [Saeb et al., 2017].”
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes
Mekruksavanich, S., & Jitpattanakul, A. (2021) · Sensors · doi:10.3390/s21051636 · Cited by 335
“Various approaches, such as leave-one-out, leave-p-out, and k-fold cross validation, can be used to separate the data for training and testing [49,50].”
BackgroundMethodRobust EEG-based cross-site and cross-protocol classification of states of consciousness
Engemann, D. A., Raimondo, F., King, J.-R., Rohaut, B., Louppe, G., Faugeras, F., Annen, J., Cassol, H., Gosseries, O., Fernandez-Slezak, D., et al. (2018) · Brain · doi:10.1093/brain/awy251 · Cited by 329
Machine Learning for Healthcare Wearable Devices: The Big Picture
Sabry, F., Eltaras, T., Labda, W., Alzoubi, K., & Malluhi, Q. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/4653923 · Cited by 321
A Guide to Cross-Validation for Artificial Intelligence in Medical Imaging
Bradshaw, T. J., Huemann, Z., Hu, J., & Rahmim, A. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220232 · Cited by 250
Medical deep learning—A systematic meta-review
Egger, J., Gsaxner, C., Pepe, A., Pomykala, K. L., Jonske, F., Kurz, M., Li, J., & Kleesiek, J. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106874 · Cited by 251
Zhang, K., Zhong, S., & Zhang, H. (2020) · Environmental Science & Technology · doi:10.1021/acs.est.0c02526 · Cited by 215
Tougui, I., Jilbab, A., & Mhamdi, J. E. (2021) · Healthcare Informatics Research · doi:10.4258/hir.2021.27.3.189 · Cited by 199
“[18], who used the Human Activity Recognition Dataset to assess how subject-wise and record-wise CV techniques estimated the RF classifier’s performance.”
MethodWilimitis, D., & Walsh, C. G. (2023) · JMIR AI · doi:10.2196/49023 · Cited by 185
“In the former example, record-wise cross-validation might be adopted for diagnosis at a given clinical encounter, and subject-wise cross-validation would be favorable for prognosis over time [19].”
MethodDetecting driver fatigue using heart rate variability: A systematic review
Lu, K., Sjörs Dahlman, A., Karlsson, J., & Candefjord, S. (2022) · Accident Analysis & Prevention · doi:10.1016/j.aap.2022.106830 · Cited by 164
Mundt, M., Koeppe, A., David, S., Witter, T., Bamer, F., Potthast, W., & Markert, B. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.00041 · Cited by 165
“This leads to an improved accuracy (Saeb et al., 2017).”
MethodInfluentialUsing and understanding cross-validation strategies. Perspectives on Saeb et al.
Little, M. A., Varoquaux, G., Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017) · GigaScience · doi:10.1093/gigascience/gix020 · Cited by 158
“[1], fit the data, and cause identity confounding-introduces a new problem that undermines its consistency as an estimator of out-of-sample prediction accuracy, either in theory or empirically, or both.”
BackgroundMethodSelf-citationKehl, K. L., Elmarakeby, H., Nishino, M., Van Allen, E. M., Lepisto, E. M., Hassett, M. J., Johnson, B. E., & Schrag, D. (2019) · JAMA Oncology · doi:10.1001/jamaoncol.2019.1800 · Cited by 159
Personality Research and Assessment in the Era of Machine Learning
Stachl, C., Pargent, F., Hilbert, S., Harari, G. M., Schoedel, R., Vaid, S. S., Gosling, S. D., & Bühner, M. (2020) · European Journal of Personality · doi:10.1002/per.2257 · Cited by 201
Automated detection of ADHD: Current trends and future perspective
Loh, H. W., Ooi, C. P., Barua, P. D., Palmer, E. E., Molinari, F., & Acharya, U. R. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105525 · Cited by 192
Hillel, T., Bierlaire, M., Elshafie, M., & Jin, Y. (2020) · Journal of Choice Modelling · doi:10.1016/j.jocm.2020.100221 · Cited by 204
Cross-Validation Approaches for Replicability in Psychology
Koul, A., Becchio, C., & Cavallo, A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01117 · Cited by 143
“The choice of the most appropriate method depends on multiple factors including the sample size and the experimental design, the research question and the application (Gong, 1986; Borra and Di Ciaccio, 2010; Saeb et al., 2017; Varoquaux et al., 2017).”
MethodRemote smartphone monitoring of Parkinson’s disease and individual response to therapy
Omberg, L., Neto, E. C., Perumal, T. M., Pratap, A., Tediarjo, A., Adams, J., Bloem, B. R., Bot, B. M., Elson, M. J., Goldman, S. M., et al. (2021) · Nature Biotechnology · doi:10.1038/s41587-021-00974-9 · Cited by 151
Cognitive-Behavioral Therapy in the Digital Age: Presidential Address
Wilhelm, S., Weingarden, H., Ladis, I., Braddick, V., Shin, J., & Jacobson, N. C. (2020) · Behavior Therapy · doi:10.1016/j.beth.2019.08.001 · Cited by 120
The accuracy of passive phone sensors in predicting daily mood
Pratap, A., Atkins, D. C., Renn, B. N., Tanana, M. J., Mooney, S. D., Anguera, J. A., & Areán, P. A. (2019) · Depression and Anxiety · doi:10.1002/da.22822 · Cited by 119
“Care is needed in running and evaluatingmachine learningmodels to avoid learning idiosyncratic digital fingerprints rather than broadly applicable associations of the passive features with mood fluctuations (Neto et al., 2017, Saeb et al, 2017).”
Emotion Recognition from ECG Signals Using Wavelet Scattering and Machine Learning
Sepúlveda, A., Castillo, F., Palma, C., & Rodriguez-Fernández, M. (2021) · Applied Sciences · doi:10.3390/app11114945 · Cited by 116
“However, if we want to build a model that can generalize to new subjects, k-fold cross validation can be this overly optimistic since records from the same subject are present in both training and test sets [50].”
BackgroundBoyle, R., Jollans, L., Rueda-Delgado, L. M., Rizzo, R., Yener, G. G., McMorrow, J. P., Knight, S. P., Carey, D., Robertson, I. H., Emek-Savaş, D. D., et al. (2021) · Brain Imaging and Behavior · doi:10.1007/s11682-020-00260-3 · Cited by 104
The inclusion of augmented intelligence in medicine: A framework for successful implementation
Bazoukis, G., Hall, J., Loscalzo, J., Antman, E. M., Fuster, V., & Armoundas, A. A. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2021.100485 · Cited by 102
AI in Rehabilitation Medicine: Opportunities and Challenges
Lanotte, F., O’Brien, M. K., & Jayaraman, A. (2023) · Annals of Rehabilitation Medicine · doi:10.5535/arm.23131 · Cited by 146
“Improper CV can produce overly optimistic or even completely invalid models [92].”
BackgroundMethodSelf-citationLabel‐Free Identification of White Blood Cells Using Machine Learning
Nassar, M., Doan, M., Filby, A. J., Wolkenhauer, O., Fogg, D. K., Piasecka, J., Thornton, C. A., Carpenter, A. E., Summers, H. D., Rees, P. C. T., et al. (2019) · Cytometry Part A · doi:10.1002/cyto.a.23794 · Cited by 96
Predicting age from cortical structure across the lifespan
Madan, C. R., & Kensinger, E. A. (2018) · European Journal of Neuroscience · doi:10.1111/ejn.13835 · Cited by 103
Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment
Salekin, M. S., Zamzmi, G., Goldgof, D., Kasturi, R., Ho, T., & Sun, Y. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104150 · Cited by 94
“[32, 18]) as it allows to capture the differences between patients.”
BackgroundNatural Language Processing to Ascertain Cancer Outcomes from Medical Oncologist Notes
Kehl, K. L., Xu, W., Lepisto, E., Elmarakeby, H., Hassett, M. J., Van Allen, E. M., Johnson, B. E., & Schrag, D. (2020) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00020 · Cited by 90
Detecting the impact of subject characteristics on machine learning-based diagnostic applications
Chaibub Neto, E., Pratap, A., Perumal, T. M., Tummalacherla, M., Snyder, P., Bot, B. M., Trister, A. D., Friend, S. H., Mangravite, L., & Omberg, L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0178-x · Cited by 83
Data leakage in deep learning studies of translational EEG
Brookshire, G., Kasper, J., Blauch, N. M., Wu, Y. C., Glatt, R., Merrill, D. A., Gerrol, S., Yoder, K. J., Quirk, C., & Lucero, C. (2024) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1373515 · Cited by 103
“…anatomical (Wen et al., 2020) MRI, optical coherence tomography (OCT) (Tampu et al., 2022), accelerometer and gyroscope recordings from smartphones (Saeb et al., 2017), audio voice recordings (Chaibub Neto et al., 2019; Tougui et al., 2021), and performance on motor tasks (Chaibub Neto et al.,…”
BackgroundResultInfluentialHigh-Resolution Motor State Detection in Parkinson’s Disease Using Convolutional Neural Networks
Pfister, F., Um, T. T., Pichler, D., Goschenhofer, J., Abedinpour, K., Lang, M., Endo, S., Ceballos-Baumann, A., Hirche, S., Bischl, B., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-61789-3 · Cited by 88
Heo, S., Kwon, S., & Lee, J. (2021) · IEEE Access · doi:10.1109/access.2021.3060441 · Cited by 77
“In addition, Leave-One-Subject-Out cross validation was used to verify the generalization performance [26], [27].”
MethodAlcantara, R. S., Edwards, W. B., Millet, G. Y., & Grabowski, A. M. (2022) · PeerJ · doi:10.7717/peerj.12752 · Cited by 83
Adler, D. A., Wang, F., Mohr, D. C., & Choudhury, T. (2022) · PLoS ONE · doi:10.1371/journal.pone.0266516 · Cited by 105
“LOSO-CV simulates the prediction error of applying models to participants unseen during model training [43].”
Self-citationLombardi, A., Diacono, D., Amoroso, N., Biecek, P., Monaco, A., Bellantuono, L., Pantaleo, E., Logroscino, G., De Blasi, R., Tangaro, S., et al. (2022) · Brain Informatics · doi:10.1186/s40708-022-00165-5 · Cited by 74
“According to this validation scheme, the dataset is split into as many sets as the number of subjects: one subject is randomly selected for testing while the others are used to train the model, and the procedure is repeated until all the subjects have been used as test [56, 57].”
MethodKong, Y., Posada-Quintero, H. F., & Chon, K. H. (2021) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3065218 · Cited by 73
A skin-conformable wireless sensor to objectively quantify symptoms of pruritus
Chun, K. S., Kang, Y. J., Lee, J. Y., Nguyen, M., Lee, B., Lee, R., Jo, H. H., Allen, E. D., Chen, H., Kim, J., et al. (2021) · Science Advances · doi:10.1126/sciadv.abf9405 · Cited by 72
Acoustic and language analysis of speech for suicidal ideation among US veterans
Belouali, A., Gupta, S., Sourirajan, V., Yu, J., Allen, N. J., Alaoui, A., Dutton, M. A., & Reinhard, M. J. (2021) · BioData Mining · doi:10.1186/s13040-021-00245-y · Cited by 87
How to remove or control confounds in predictive models, with applications to brain biomarkers
Chyzhyk, D., Varoquaux, G., Milham, M., & Thirion, B. (2022) · GigaScience · doi:10.1093/gigascience/giac014 · Cited by 65
“Similarly, in multi-participant prediction with repeated measurements from the same participant, participant-wise cross-validation can rule out that prediction is driven by participant identification [22,27].”
Ahmadi, A., Davoudi, S., & Daliri, M. R. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.11.006 · Cited by 67
Meaningless comparisons lead to false optimism in medical machine learning
DeMasi, O., Kording, K., & Recht, B. (2017) · PLoS ONE · doi:10.1371/journal.pone.0184604 · Cited by 63
Self-citationCos, H., Li, D., Williams, G., Chininis, J., Dai, R., Zhang, J., Srivastava, R., Raper, L., Sanford, D., Hawkins, W., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/23595 · Cited by 63
“Moreover, the application of machine learning methodologies and feature engineering techniques on patient activity data have shown that human biobehavioral rhythms, semantic features, and second-order statistical features are predictors of clinical outcomes [18-23].”
InfluentialFold-stratified cross-validation for unbiased and privacy-preserving federated learning
Bey, R., Goussault, R., Grolleau, F., Benchoufi, M., & Porcher, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa096 · Cited by 70
CT- and MRI-Based 3D Reconstruction of Knee Joint to Assess Cartilage and Bone
Ciliberti, F. K., Guerrini, L., Gunnarsson, A. E., Recenti, M., Jacob, D., Cangiano, V., Tesfahunegn, Y. A., Íslind, A. S., Tortorella, F., Tsirilaki, M., et al. (2022) · Diagnostics · doi:10.3390/diagnostics12020279 · Cited by 63
Cross-Validation for Correlated Data
Rabinowicz, A., & Rosset, S. (2020) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1801451 · Cited by 92
“While the fundamental concerns that this presents are widely acknowledged, a clear understanding of when adjustments are needed, and what adjustments are appropriate, seems lacking in much of the literature (Roberts et al. 2017; Saeb et al. 2017; Anderson et al. 2018).”
BackgroundResting-state EEG measures cognitive impairment in Parkinson’s disease
Anjum, M. F., Espinoza, A. I., Cole, R. C., Singh, A. P., May, P., Uc, E. Y., Dasgupta, S., & Narayanan, N. S. (2024) · npj Parkinson s Disease · doi:10.1038/s41531-023-00602-0 · Cited by 75
“Lastly, we had a moderate-sized dataset and thus evaluated the performance of LEAPD scores using cross-validation schemes, commonly utilized in measuring diagnostic performances 51,52 .”
MethodSleep Apnea Detection From Variational Mode Decomposed EEG Signal Using a Hybrid CNN-BiLSTM
Mahmud, T., Khan, I. A., Mahmud, T. I., Fattah, S. A., Zhu, W., & Ahmad, M. O. (2021) · IEEE Access · doi:10.1109/access.2021.3097090 · Cited by 61
Farzan, F., Atluri, S., Frehlich, M., Dhami, P., Kleffner, K., Price, R., Lam, R. W., Frey, B. N., Milev, R., Ravindran, A., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-07613-x · Cited by 54
Estimation of Parkinson’s disease severity using speech features and extreme gradient boosting
Tunç, H. C., Sakar, C. O., Apaydın, H., Serbes, G., Gündüz, A., Tütüncü, M., & Gürgen, F. S. (2020) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-020-02250-5 · Cited by 57
“LOSO CV also resembles the use-case scenario of the model [62].”
MethodAutomatic motion artifact detection in electrodermal activity data using machine learning
Hossain, M.-B., Posada-Quintero, H. F., Kong, Y., McNaboe, R. Q., & Chon, K. H. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103483 · Cited by 63
Integrated data-driven modeling and experimental optimization of granular hydrogel matrices
Verheyen, C. A., Uzel, S. G. M., Kurum, A., Roche, E. T., & Lewis, J. A. (2023) · Matter · doi:10.1016/j.matt.2023.01.011 · Cited by 56
How Validation Methodology Influences Human Activity Recognition Mobile Systems
Bragança, H. L. D. S., Colonna, J. G., Oliveira, H. A. B. F., & Souto, E. (2022) · Sensors · doi:10.3390/s22062360 · Cited by 65
Tsanas, A., Little, M. A., & Ramig, L. O. (2021) · IEEE Access · doi:10.1109/access.2021.3050524 · Cited by 60
Lonini, L., Shawen, N., Botonis, O. K., Fanton, M., Jayaraman, C., Mummidisetty, C. K., Shin, S. Y., Rushin, C. C., Jenz, S. T., Xu, S., et al. (2021) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2021.3058841 · Cited by 54
“…Cadence – Healthy : 106.9 steps per minute , IQR = [102 Wevalidated the model using a leave-one-subject-out cross validation, so to mimic the use-case of the paradigm [25], i.e. training on snapshots from a cohort with known diagnosis and testing on snapshots of a new with unknown diagnosis.”
InfluentialSelf-citationLiang, Y., Liu, B., & Zhang, H. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3120024 · Cited by 56
Machine Learning Models for Parkinson Disease: Systematic Review
Tabashum, T., Snyder, R. C., O’Brien, M. K., & Albert, M. V. (2024) · JMIR Medical Informatics · doi:10.2196/50117 · Cited by 73
“Although Saeb et al [37] concluded that subject-wise methods are more clinically relevant compared to record-wise methods, Little et al [38] argued that subject-wise methods might not be the best in all use cases.”
MethodCreagh, A. P., Simillion, C., Bourke, A. K., Scotland, A., Lipsmeier, F., Bernasconi, C., van Beek, J., Baker, M., Gossens, C., Lindemann, M., et al. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2998187 · Cited by 53
“However, in subject-wise CV it is argued that there is no “leakage” of subject information between training and testing partitions where models can learn individual subjects’ characteristics [51].”
BackgroundUsing EEG to Distinguish Between Writing and Typing for the Same Cognitive Task
Qu, X., Mei, Q., Liu, P., & Hickey, T. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60735-7_7 · Cited by 55
K ‐fold cross‐validation for complex sample surveys
Wieczorek, J., Guerin, C., & McMahon, T. (2022) · Stat · doi:10.1002/sta4.454 · Cited by 76
Baeza-Delgado, C., Alberich, L. C., Sierra, J. M. C., Veiga-Canuto, D., Heras, B. M. D. L., Raza, B., & Martí‐Bonmatí, L. (2022) · European Radiology Experimental · doi:10.1186/s41747-022-00276-y · Cited by 55
“Other mitigation strategies may include subject-wise cross-validation [32], resampling techniques [33], or data augmentation methods for medical images [34], and even exploring data imputation solutions for clinical data as suggested by Pezoulas et al.”
Azarmi, F., Ashtiani, S. N. M., Shalbaf, A., Behnam, H., & Daliri, M. R. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.103495 · Cited by 47
Vajs, I. A., Ković, V., Jakovljević, T., et al. (2022) · Sensors · doi:10.3390/s22134900 · Cited by 43
Antimicrobial Resistance and Machine Learning: Challenges and Opportunities
Elyan, E., Hussain, A., Sheikh, A., Elmanama, A. A., Vuttipittayamongkol, P., & Hijazi, K. (2022) · IEEE Access · doi:10.1109/access.2022.3160213 · Cited by 49
“A technical discussion on different model’s validation methods can be found at [94].”
BackgroundMarotta, L., Buurke, J. H., van Beijnum, B.-J. F., & Reenalda, J. (2021) · Sensors · doi:10.3390/s21103451 · Cited by 53
Multi-Level Functional Connectivity Fusion Classification Framework for Brain Disease Diagnosis
Liang, Y., & Xu, G. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3159031 · Cited by 46
Filtjens, B., Ginis, P., Nieuwboer, A., Slaets, P., & Vanrumste, B. (2022) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-022-01025-3 · Cited by 52
“This approach mirrors the clinically relevant scenario of FOG assessment in newly recruited subjects [63], where the model is tasked to assess FOG in unseen subjects.”
Identifying the signature of prospective motor control in children with autism
Cavallo, A., Romeo, L., Ansuini, C., Battaglia, F. M., Nobili, L., Pontil, M., Panzeri, S., & Becchio, C. (2021) · Scientific Reports · doi:10.1038/s41598-021-82374-2 · Cited by 47
A comparative study of machine learning techniques for suicide attempts predictive model
Nordin, N., Zainol, Z. B., Noor, M. H. M., & Fong, C. L. (2021) · Health Informatics Journal · doi:10.1177/1460458221989395 · Cited by 42
Arani, M. S. A., Costa, D. E., & Shihab, E. (2021) · Sensors · doi:10.3390/s21216997 · Cited by 44
“To evaluate the performance of our cross-subject model, we use the Leave-One-SubjectOut (LOSO) evaluation method [57,58].”
MethodHaque, F., Reaz, M. B. I., Chowdhury, M. E. H., Ezeddin, M., Serkan Kiranyaz, B., Alhatou, M. I., Ali, S. H. M., Bakar, A. A. A., & Srivastava, G. (2022) · Sensors · doi:10.3390/s22093507 · Cited by 40
“A previous study showed that splitting the data into training and test sets in a record-wise fashion can lead to a massive underestimation of prediction error achieved by the machine learning algorithm [48].”
Cook, F., Lobo, D., Martin, M., Imbert, N., Grati, H., Daami, N., Cherait, C., Saidi, N.-E., Abbay, K., Jaubert, J., et al. (2018) · British Journal of Anaesthesia · doi:10.1016/j.bja.2018.09.021 · Cited by 41
Empirical investigation of multi-source cross-validation in clinical ECG classification
Leinonen, T., Wong, D., Vasankari, A., Wahab, A., Nadarajah, R., Kaisti, M., & Airola, A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109271 · Cited by 66
“In these cases, subject-level CV methods, such as leave-subject-out [22, 23], are frequently used to ensure robust model performance across di ff erent individuals.”
A Framework for AI-Assisted Detection of Patent Ductus Arteriosus from Neonatal Phonocardiogram
Gómez-Quintana, S., Schwarz, C. E., Shelevytsky, I., Shelevytska, V., Semenova, O., Factor, A., Popovici, E. M., & Temko, A. (2021) · Healthcare · doi:10.3390/healthcare9020169 · Cited by 47
Carrle, F. P., Hollenbenders, Y., & Reichenbach, A. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1219133 · Cited by 53
“This procedure resulted in subject-wise cross-validation (Saeb et al., 2017).”
How Not To Drown in Data: A Guide for Biomaterial Engineers
Vasilevich, A. S., Carlier, A., de Boer, J., & Singh, S. (2017) · Trends in biotechnology · doi:10.1016/j.tibtech.2017.05.007 · Cited by 38
Classification of Alpine Skiing Styles Using GNSS and Inertial Measurement Units
Neuwirth, C., Snyder, C., Kremser, W., Brunauer, R., Holzer, H., & STÖGGL, T. L. (2020) · Sensors · doi:10.3390/s20154232 · Cited by 42
How Are Personality States Associated with Smartphone Data?
Rüegger, D., Stieger, M., Nißen, M., Allemand, M., Fleisch, E., & Kowatsch, T. (2020) · European Journal of Personality · doi:10.1002/per.2309 · Cited by 36
“To ensure that models generalize to out ‐ of ‐ sample participants, and in line with previous work (Chittaranjan et al., 2011; Kalimeri et al., 2013), we used participant ‐ wise splitting (Saeb, Lonini, Jayaraman, Mohr, & Kording, 2017).”
A machine learning perspective on the emotional content of Parkinsonian speech
Sechidis, K., Fusaroli, R., Orozco‐Arroyave, J. R., Wolf, D., & Zhang, Y. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102061 · Cited by 37
A Computer Vision Approach for Estimating Lifting Load Contributors to Injury Risk
Zhou, G., Aggarwal, V., Yin, M., & Yu, D. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2022.3148339 · Cited by 39
Doan, M., Barnes, C. M., McQuin, C., Caicedo, J. C., Goodman, A., Carpenter, A. E., & Rees, P. C. T. (2021) · Nature Protocols · doi:10.1038/s41596-021-00549-7 · Cited by 40
Multi-class Time Continuity Voting for EEG Classification
Qu, X., Liu, P., Li, Z., & Hickey, T. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60735-7_3 · Cited by 41
Combining voice and language features improves automated autism detection
MacFarlane, H., Salem, A. C., Chen, L., Asgari, M., & Fombonne, E. J. (2022) · Autism Research · doi:10.1002/aur.2733 · Cited by 39
Advances in Machine Learning Approaches to Heart Failure with Preserved Ejection Fraction
Ahmad, F. S., Luo, Y., Wehbe, R. M., Thomas, J. D., & Shah, S. J. (2022) · Heart Failure Clinics · doi:10.1016/j.hfc.2021.12.002 · Cited by 34
Machine Learning and Health Science Research: Tutorial
Cho, H., She, J., de Marchi, D., El-Zaatari, H., Barnes, E. L., Kahkoska, A. R., Kosorok, M. R., & Virkud, A. (2024) · Journal of Medical Internet Research · doi:10.2196/50890 · Cited by 43
Smartwatches Can Detect Walker and Cane Use in Older Adults
Antos, S. A., Danilovich, M., Eisenstein, A., Gordon, K. E., & Kording, K. P. (2019) · Innovation in Aging · doi:10.1093/geroni/igz008 · Cited by 36
Self-citationIHC Color Histograms for Unsupervised Ki67 Proliferation Index Calculation
Geread, R. S., Morreale, P., Dony, R. D., Brouwer, E. R., Wood, G. A., Androutsos, D., & Khademi, A. (2019) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00226 · Cited by 33
“To minimize issues with small dataset sizes, cross validation (CV) (Efron and Tibshirani, 1997; Saeb et al., 2017; Willis and Riley, 2017) is used.”
MethodSegmentation of Brain Tumors from MRI Images Using Convolutional Autoencoder
Atanasijević, M. B., & Barjaktarović, M. Č. (2021) · Applied Sciences · doi:10.3390/app11094317 · Cited by 39
“A second approach has been implemented to test the network’s generalization ability in medical diagnostics [27].”
MethodAutomated Detection of Epileptic Biomarkers in Resting-State Interictal MEG Data
Soriano, M. C., Niso, G., Clements, J. M., Ortín, S., Carrasco, S., Gudín, M., Mirasso, C. R., & Pereda, E. (2017) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00043 · Cited by 38
Estimating 3D ground reaction forces in running using three inertial measurement units
Scheltinga, B. L., Kok, J. N., Buurke, J. H., & Reenalda, J. (2023) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2023.1176466 · Cited by 31
“This leave-onesubject-out cross-validation was used to test how well the models work for the data of subjects that the model was never trained or validated for (33).”
MethodYang, P.-K., Filtjens, B., Ginis, P., Goris, M., Nieuwboer, A., Gilat, M., Slaets, P., & Vanrumste, B. (2024) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-024-01320-1 · Cited by 39
“This approach mirrored the clinically relevant scenario of FOG assessment in newly recruited subjects (56), where the model assesses FOG in unseen subjects.”
ResultHe, Q., Wang, Q., Wu, Y., Yi, L., & Wei, K. (2021) · PsyCh Journal · doi:10.1002/pchj.447 · Cited by 33
Veetil, I. K., Chowdary, D. E., Chowdary, P. N., Sowmya, V., & Gopalakrishnan, E. A. (2024) · Digital Signal Processing · doi:10.1016/j.dsp.2024.104407 · Cited by 37
Padovano, D., Martínez‐Rodrigo, A., Pastor, J. M., Rieta, J. J., & Alcaraz, R. (2022) · IEEE Access · doi:10.1109/access.2022.3201911 · Cited by 35
“In this setting, complex classifiers can pick up a confounding relationship between a subject-specific feature (e.g., the heart rate) and diagnostic status, and so produce unrealistically high prediction accuracy [23], [81].”
BackgroundBahameish, M., Stockman, T., & Carrión, J. R. (2024) · Sensors · doi:10.3390/s24103210 · Cited by 53
“Consistent with the literature [23,84,85], LOGO CV and, particularly, nested LOGO CV methods provided lower accuracy rates compared to standard and nested K-fold CV methods, with a mean difference of 5%, across the investigated ML models.”
MethodLiu, Q., Mo, S., Cheung, V. C. K., Cheung, B. M. F., Wang, S., Chan, P. P. K., Malhotra, A., Cheung, R. T. H., & Chan, R. H. M. (2020) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2020.110072 · Cited by 35
Data Leakage in Notebooks: Static Detection and Better Processes
Yang, C., Brower-Sinning, R. A., Lewis, G. A., & Kästner, C. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556918 · Cited by 38
GPT-Driven Radiology Report Generation with Fine-Tuned Llama 3
Voinea, Ș.-V., Mămuleanu, M., Teică, R. V., Florescu, L. M., Selișteanu, D., & Gheonea, I. A. (2024) · Bioengineering · doi:10.3390/bioengineering11101043 · Cited by 38
“The dataset was divided into training, validation, and test sets using an 80/10/10 to ensure that there was a robust training process and that there were sufficient data model evaluation [19].”
BackgroundMethod
Metadata
- Authors
- Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Saeb, Sohrab
- Lonini, Luca
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-03-15
- Volume
- 6
- Issue
- 5
- Pages
- 1-9
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC5441397
- OpenAlex
- W2595557058
- Cited by
- 367
- References
- 36
- Citations by year
- 2026: 87 · 2025: 24 · 2024: 35 · 2023: 39 · 2022: 57 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 11 · 2017: 10
- FWCI
- 28.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (P20MH090318, R01MH100482)
- National Institute of Neurological Disorders and Stroke (R01NS063399)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- cross-validation, machine learning in healthcare, wearable sensors, accelerometer, smartphone health data, overfitting, clinical decision support
- MeSH terms
- Smartphone, Machine Learning, Wearable Electronic Devices, Algorithms, Humans, Reproducibility of Results, Exercise, Monitoring, Ambulatory, Accelerometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.