The need to approximate the use-case in clinical machine learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, data leakage due to identity confounding is not limited to deep neural networks, and has been uncovered using random forests (Saeb et al., 2017; Chaibub Neto et al., 2019; Tougui et al., 2021) and support vector machines (Tougui et al., 2021).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“For example, if a study trains a classifier to identify individual subjects with a disease, then there may be stronger bias when the study involves fewer participants (Saeb et al., 2017).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“This conclusion mirrors results from studies examining a variety of other types of data and classifier models (Saeb et al., 2017; Wen et al., 2018, 2020; Chaibub Neto et al., 2019; Tougui et al., 2021; Tampu et al., 2022).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 2×
“The number of trees for the random forest classifier was set to 50, which was optimized based on the out of bag error [22].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“Machine learning has grown massively over the past decades, with countless applications in technology, marketing, and science [1].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Inappropriatevalidationprocedures,ontheotherhand,canlead to unreliable results, which might contribute to the problem of irreproducibility of research findings [31, 32] and thereby undermine the trust in both medicine and data science.”
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · cited 1×
“Splitting data into training and test subsets can be done using various methods, such as leave-one-out, leave-p-out, k-fold, and Monte-Carlo sampling [18].”
The Rise of Consumer Health Wearables: Promises and Barriers
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001953 · Cited by 1,235 · cited 1×
“A driving factor is that people carry these devices with them most of the time, and thus a growing amount of daily life data, such as physical activities [5, 6], is becoming available.”
Research works printed but not linked: 1
- Google Scholar. Google 2016 https://scholar.google.com (6 April 2017, date last accessed).
Cited by
Items 101–200 of 357
Wang, J., Li, Y., Ya, W., & Huang, W. (2018) · Frontiers in Neurology · doi:10.3389/fneur.2018.01038 · Cited by 30
“In addition, previous studies have also shown that this method mirrors the clinically relevant use-case scenario of diagnosis and is therefore especially useful in clinical diagnostic applications (31, 33).”
MethodQu, X., Sun, Y., Sekuler, R., & Hickey, T. J. (2018) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8659031 · Cited by 36
Parekh, A., Tolbert, T. M., Mooney, A. M., Ramos‐Cejudo, J., Osorio, R. S., Treml, M., Herkenrath, S.-D., Randerath, W. J., Ayappa, I. A., & Rapoport, D. M. (2021) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202011-4055oc · Cited by 34
Discrimination of multiple sclerosis using OCT images from two different centers
Khodabandeh, Z., Rabbani, H., Ashtari, F., Zimmermann, H. G., Motamedi, S., Brandt, A. U., Paul, F., & Kafieh, R. (2023) · Multiple Sclerosis and Related Disorders · doi:10.1016/j.msard.2023.104846 · Cited by 34
“This approach reduces the over-estimation of prediction accuracy (Saeb et al., 2017) in instance-wise CV with leakage of information between training and testing phases.”
BackgroundGarcía-Moreno, F. M., Bermúdez-Edo, M., Rodríguez-García, E., Pérez‐Mármol, J. M., Garrido, J. L. G., & Rodríguez‐Fórtiz, M. J. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104625 · Cited by 33
Jang, E. H., Choi, K. W., Kim, A. Y., Yu, H. Y., Jeon, H. J., & Byun, S. (2022) · ETRI Journal · doi:10.4218/etrij.2021-0299 · Cited by 29
“To prevent data leakage during CV, we adopted a subject-wise, rather than sample-wise, data split [44].”
MethodGuo, C. C., Chiesa, P. A., de Moor, C., Fazeli, M. S., Schofield, T., Hofer, K., Belachew, S. M., & Scotland, A. (2022) · Journal of Medical Internet Research · doi:10.2196/37683 · Cited by 35
“One illustration from this review is the critical role of thresholds [132], which require researchers to decide between manual versus automatic thresholds [133] and global versus person specific [134].”
Altai, Z., Boukhennoufa, I., Zhai, X., Phillips, A., Moran, J., & Liew, B. X. W. (2023) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2023.1215770 · Cited by 34
“…ndings were observed in previous studies for classi fi cations (e.g., healthy and diseased subjects) based on voice measurements (Neto et al., 2019) and activity recognition (Saeb et al., 2017), where model classi fi cation ability was reduced when ignoring subjects variability during model training.”
BackgroundThe Use of Deep Learning Software in the Detection of Voice Disorders: A Systematic Review
Barlow, J., Sragi, Z., Rivera‐Rivera, G., Al‐Awady, A., Daşdöğen, Ü., Courey, M. S., & Kirke, D. N. (2024) · Otolaryngology–Head and Neck Surgery · doi:10.1002/ohn.636 · Cited by 25
McCabe, M. V., Citters, D. W. V., & Chapman, R. M. (2022) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2022.2044028 · Cited by 29
“…techniques (e.g. scaling, rotation etc.) to induce additional training data variability (Mundt et al., 2020a; Rashid and Louis, 2019), designing ANNs for specific subject categories (e.g. age range, gender, pathology) (Saeb et al., 2017), and optimizing sensor types/quantities/placement.”
BackgroundMachine Learning for Detection and Risk Assessment of Lifting Action
Thomas, B., Lu, M., Jha, R., & Bertrand, J. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2022.3212666 · Cited by 30
“relatively small dataset, a leave-one-subject-out (LOSO) crossvalidation (CV) was considered for evaluation of the model [23].”
MethodExamining Implicit Discretization in Spectral Schemes
Quinan, P. S., Padilla, L., Creem-Regehr, S. H., & Meyer, M. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13695 · Cited by 24
Huang, J., Li, Y., Xie, H., Yang, S., Jiang, C., Sun, W., Li, D., Liao, Y., Ba, X., & Xiao, L. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.532110 · Cited by 26
Comparing emotion feature extraction approaches for predicting depression and anxiety
Burkhardt, H. A., Pullmann, M. D., Hull, T., Aren, P., & Cohen, T. (2022) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.18653/v1/2022.clpsych-1.9 · Cited by 37
“To avoid information leakage due to within-patient effects (Saeb et al., 2017), data were split into training and test sets such that all observations from an individual patient were kept within the same fold.”
MethodClassification of Human Motion Data Based on Inertial Measurement Units in Sports: A Scoping Review
Kranzinger, C., Bernhart, S., Kremser, W., Venek, V., Rieser, H., Mayr, S., & Kranzinger, S. (2023) · Applied Sciences · doi:10.3390/app13158684 · Cited by 30
Wearable-Based Parkinson’s Disease Severity Monitoring Using Deep Learning
Goschenhofer, J., Pfister, F., Yüksel, K. A., Bischl, B., Fietzek, U. M., & Thomas, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46133-1_24 · Cited by 30
“Using regular cross validation (CV) strategies, subject-specific information could be exploited resulting in an overly optimistic estimate of the generalization performance [37].”
BackgroundMethodA.Santoyo-Ramón, J., Casilari, E., & Cano, J. M. B. (2021) · Scientific Reports · doi:10.1038/s41598-021-02537-z · Cited by 24
Najafi, T. A., Affanni, A., Rinaldo, R., & Zontone, P. (2023) · Sensors · doi:10.3390/s23177346 · Cited by 24
“This approach prevents potential biases and overestimation of the model’s performance, which might occur with other subject-dependent methods [59,60].”
BackgroundFinger movements recognition using minimally redundant features of wavelet denoised EMG
Phukan, N., Kakoty, N. M., Shivam, P., & Gan, J. Q. (2019) · Health and Technology · doi:10.1007/s12553-019-00338-z · Cited by 25
“This is repeated for each subject as test subject and the average of all the test results has been estimated as the average recognition rates [36].”
BackgroundHart, A., Reis, D., Prestele, E., & Jacobson, N. C. (2022) · Journal of Medical Internet Research · doi:10.2196/34015 · Cited by 22
“One of the most popular methods in the studies reviewed by Yim et al [22] and Saeb et al [23] is training and cross-validating the models on a random subset of the entire sample.”
BackgroundMethodInfluentialValidation Methods to Promote Real-world Applicability of Machine Learning in Medicine
Rafiq, R. B., Modave, F. P., Guha, S., & Albert, M. V. (2020) · International Conference on Digital Medicine and Image Processing (DMIP) · doi:10.1145/3441369.3441372 · Cited by 31
“One of the most prevalent issues is the lack of proper testing of cross-validation results [23], so in this section, we briefly step through some common ways in which this limitation and others previously mentioned can be properly addressed.”
BackgroundBalanced Leave-One-Subject-Out Cross- Validation for Microsleep Classification
Pauli, M. P., Pohl, C., & Gölz, M. (2021) · Current Directions in Biomedical Engineering · doi:10.1515/cdbme-2021-2038 · Cited by 22
“Because of hierarchy argument, it has been recommended to use leave-one-subject-out CV (LOSO CV) instead of standard CV for multi-subject clinical diagnostic studies [2].”
MethodMarotta, L., Scheltinga, B. L., van Middelaar, R. P., Bramer, W. M., van Beijnum, B.-J. F., Reenalda, J., & Buurke, J. H. (2022) · Sensors · doi:10.3390/s22083008 · Cited by 34
Constable, P. A., Pinzon-Arenas, J. O., Díaz, L. R. M., Lee, I., Marmolejo‐Ramos, F., Loh, L., Zhdanov, A. E., Kulyabin, M., Brabec, M., Skuse, D., et al. (2024) · Bioengineering · doi:10.3390/bioengineering12010015 · Cited by 26
“This helps to improve the generalization and result interpretation of the model [47].”
Gao, Y., Song, F., Zhang, P., Liu, J., Cui, J., Ma, Y., Zhang, G., & Luo, J. (2021) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-021-00455-0 · Cited by 22
Retinal vessel multifractals predict pial collateral status in patients with acute ischemic stroke
Khan, A., Boever, P. D., Gerrits, N., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0267837 · Cited by 21
Generalizability of deep learning models for predicting outdoor irregular walking surfaces
Shah, V. R., Flood, M. W., Grimm, B. P., & Dixon, P. C. (2022) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2022.111159 · Cited by 21
Mammogram segmentation using multi-atlas deformable registration
Sharma, M. K., Jas, M., Karale, V., Sadhu, A. K., & Mukhopadhyay, S. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.06.001 · Cited by 23
Mansourian, M., Khademi, S., & Marateb, H. R. (2021) · Diagnostics · doi:10.3390/diagnostics11030393 · Cited by 22
Ueno, T., Ichikawa, D., Shimizu, Y., Narisawa, T., Tsuji, K., Ochi, E., Sakurai, N., Iwata, H., & Matsuoka, Y. (2021) · Japanese Journal of Clinical Oncology · doi:10.1093/jjco/hyab169 · Cited by 26
“For crossvalidation, we used subject-wise data splitting rather than recordwise data splitting because identity confounding has been reported with the latter (25,26).”
MethodJlassi, O., & Dixon, P. C. (2024) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2024.112116 · Cited by 20
Artificial intelligence in cardiology: the debate continues
Asselbergs, F. W., & Fraser, A. G. (2021) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztab090 · Cited by 23
“26 Unless all relevant data are used as inputs, then phenotypic characterization by unsupervised ML may be uninformative.”
Kabir, M. M., Shin, J., & Mridha, M. F. (2023) · IEEE Access · doi:10.1109/access.2023.3289402 · Cited by 23
“In addition, as the sensor data contains a time component, it is crucial to handle the data split to preserve the temporal order of the sensor readings, ensuring the generalizability of the proposed approach [49].”
Primativo, S., Clark, C. N., Yong, K. X. X., Firth, N. C., Nicholas, J. M., Alexander, D. C., Warren, J. D., Rohrer, J. D., & Crutch, S. J. (2017) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2017.10.014 · Cited by 22
Moore, S. R., Kranzinger, C., Fritz, J., STÖGGL, T. L., Kröll, J., & Schwameder, H. (2020) · Sensors · doi:10.3390/s20236737 · Cited by 22
Tang, R. H., Zhang, Z., Chen, X., Waller, L., Zhang, A. C., Chen, J., Han, Y., An, C., Cho, S. H., & Lo, Y. (2020) · APL Photonics · doi:10.1063/5.0024151 · Cited by 22
An Intraoral Non-Occlusal MEMS Sensor for Bruxism Detection
O’Hare, E., Cogan, J., Dillon, F., et al. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3128246 · Cited by 19
Ahmed, M. S., & Ahmed, N. (2023) · JMIR Formative Research · doi:10.2196/28848 · Cited by 41
“This introduces the potential for biased models and overestimated performance [93].”
BackgroundBeyond Detection: Towards Actionable Sensing Research in Clinical Mental Healthcare
Adler, D. A., Yang, Y., Viranda, T., Xu, X., Mohr, D. C., Meter, A. V., Tartaglia, J., Jacobson, N. C., Wang, F., Estrin, D. L., et al. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699755 · Cited by 36
“…measure, for example, a self-reported active sensing measure such as the PHQ-9 [ 97, The performance of detection models are measured using the generalization accuracy, estimating how well the model approximates a ground truth mental health measure within data not included in model training [138].”
Self-citationWorkflow Development for the Functional Characterization of ncRNAs
Wolfien, M., Brauer, D., Bagnacani, A., & Wolkenhauer, O. (2019) · Methods in molecular biology · doi:10.1007/978-1-4939-8982-9_5 · Cited by 19
Ferrari, E., Retico, A., & Bacciu, D. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101804 · Cited by 22
“Studies on biometric sensor data, instead, have shown that the relationship between features and disease class label learned by the classifier is confounded by the identity of the subjects, because the easier task of subject identification replaces the harder task of disease recognition [6, 2].”
BackgroundSNAPS: Sensor Analytics Point Solutions for Detection and Decision Support Systems
McLamore, E. S., Datta, S., Morgan, V., Cavallaro, N. D., Kiker, G. A., Jenkins, D. M., Rong, Y., Gomes, C. L., Claussen, J. C., Vanegas, D. C., et al. (2019) · Sensors · doi:10.3390/s19224935 · Cited by 57
Kehl, K. L., Groha, S., Lepisto, E. M., Elmarakeby, H., Lindsay, J. R., Gusev, A., Allen, E. M. V., Hassett, M. J., & Schrag, D. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.20.00184 · Cited by 23
Attention-based multi-semantic dynamical graph convolutional network for eeg-based fatigue detection
Liu, H., Liu, Q., Cai, M., et al. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1275065 · Cited by 17
Identifying cognitive deficits in cocaine dependence using standard tests and machine learning
Jiménez, S., Ángeles-Valdéz, D., Villicaña, V., Reyes-Zamorano, E., Alcalá‐Lozano, R., Gonzalez-Olvera, J. J., & Garza‐Villarreal, E. A. (2019) · Progress in Neuro-Psychopharmacology and Biological Psychiatry · doi:10.1016/j.pnpbp.2019.109709 · Cited by 16
Rellum, S. R., Schuurmans, J., van der Ven, W. H., Eberl, S., Driessen, A. H. G., Vlaar, A. P. J., & Veelo, D. P. (2021) · Journal of Thoracic Disease · doi:10.21037/jtd-21-765 · Cited by 20
“Still, traditional linear statistics may be more suitable for risk prediction models (85), as advanced models perform similarly but are more complex to develop.”
BackgroundOrtuño-Miró, S., Molina‐Rodríguez, S., Belmonte, C., & Ibáñez-Ballesteros, J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/acad2b · Cited by 20
“Some researchers have suggested that LOO may be more useful in a diagnostic scenario [98, 117], whereas others recommend k -fold or repeated random splits for more stable estimates [97, 118].”
BackgroundMethodFeature Selection Methods for Robust Decoding of Finger Movements in a Non-human Primate
Padmanaban, S., Baker, J. J., & Greger, B. E. (2018) · Frontiers in Neuroscience · doi:10.3389/fnins.2018.00022 · Cited by 17
“The appropriate approach is needed to approximate the use-case in machine learning (Saeb et al., 2017).”
BackgroundEvolutionary Denoising-Based Machine Learning for Detecting Knee Disorders
Parisi, L., & RaviChandran, N. (2020) · Neural Processing Letters · doi:10.1007/s11063-020-10361-1 · Cited by 15
“Moreover, the leave-one-subject-out (LOSO) cross-validation approach [37] was adopted, as it closely mirrors the use case of medical diagnosis, wherein one subject is tested, whilst the others are used for training the classifier [38, 39].”
MethodHip Joint Angles and Moments during Stair Ascent Using Neural Networks and Wearable Sensors
McCabe, M. V., Citters, D. W. V., & Chapman, D. M. F. (2023) · Bioengineering · doi:10.3390/bioengineering10070784 · Cited by 16
Gupta, R. K., & Sinha, R. (2023) · IEEE Access · doi:10.1109/access.2023.3297091 · Cited by 16
“For the first experiment, the leave-one-subject-out method [43] is performed on the training partition to find the optimal threshold boundary.”
Computational kinematics of dance: distinguishing hip hop genres
Baker, B., Liu, T., Matelsky, J., Parodi, F., Mensh, B. D., Krakauer, J. W., & Kording, K. P. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1295308 · Cited by 17
Self-citationSilva-Acosta, V. D. C., Román-Godínez, I., Torres-Ramos, S., & Salido-Ruiz, R. A. (2021) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2021.102950 · Cited by 16
Noninvasive Monitoring of Simulated Hemorrhage and Whole Blood Resuscitation
Gupta, J. F., Arshad, S. H., Telfer, B. A., Snider, E. J., & Convertino, V. A. (2022) · Biosensors · doi:10.3390/bios12121168 · Cited by 16
Edavally, S., Miller, D. D., & Youssef, N. A. (2021) · Annals of Clinical Psychiatry · doi:10.12788/acp.0041 · Cited by 14
“28 This helps generalize the data across complex clinical sites and scenarios.”
Arian, R., Aghababaei, A., Soltanipour, A., Khodabandeh, Z., Rakhshani, S., Iyer, S. B., Ashtari, F., Rabbani, H., & Kafieh, R. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.7.13 · Cited by 13
A template and tutorial for preregistering studies using passive smartphone measures
Langener, A. M., Siepe, B. S., Elsherif, M. M., Niemeijer, K., Andresen, P. K., Akre, S., Bringmann, L. F., Cohen, Z. D., Choukas, N. R., Drexl, K., et al. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02474-5 · Cited by 20
Ashtiani, S. N. M., Behnam, H., & Daliri, M. R. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3081605 · Cited by 13
“A leave-one-subject-out cross-validation method [38] was applied to evaluate the performance of classification.”
MethodVolleyball Data Analysis System and Method Based on Machine Learning
Dai, X., & Li, S. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/9943067 · Cited by 18
“The radial basis kernel function is very dependent on the parameters, and the training is very time-consuming [29, 30].”
BackgroundPosada-Quintero, H. F., Vergara, P. N. M., Parra-Hernández, R. M., et al. (2020) · Social Sciences · doi:10.3390/socsci9030030 · Cited by 13
“In this study, we have used leave-one-subject-out cross-validation (Saeb et al. 2017) for the evaluation of the models.”
BackgroundMethodParticipant-Aware Model Validation for Repeated-Measures Data: Comparative Cross-Validation Study
Karbalaie, A., Abtahi, F., & Häger, C. K. (2026) · JMIR AI · doi:10.2196/87728 · Cited by 27
“More recent applied studies in neuroimaging, biomechanics, and mobile health have adopted LOPOCV or group CV to control for participant-level leakage (eg, [8,9]).”
A multimodal sensing ring for quantification of scratch intensity
Padmanabha, A., Choudhary, S., Majidi, C., & Erickson, Z. (2023) · Communications Medicine · doi:10.1038/s43856-023-00345-2 · Cited by 14
“We utilize LOSO-CV as it mirrors the clinically relevant scenario of tracking scratching in new subjects whose data is not in the training set 29 .”
MethodRumala, D. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45249-9_23 · Cited by 39
“Our findings align with previous research advocating for the use of subject-wise splitting approach [14,16,2] and early-split [19] to avoid data leakage and identity confounding.”
ResultA smartphone- and wearable-based biomarker for the estimation of unipolar depression severity
Zhuparris, A., Maleki, G., van Londen, L., Koopmans, I. W., Aalten, V., Yocarini, I. E., Exadaktylos, V., van Hemert, A. M., Cohen, A. F., Gal, P., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-46075-2 · Cited by 17
Predicting Individual Hearing-Aid Preference From Self-Reported Listening Experiences in Daily Life
Christensen, J. H., Rumley, J., Gil-Carvajal, J. C., Whiston, H., Lough, M., & Saunders, G. H. (2024) · Ear and Hearing · doi:10.1097/aud.0000000000001520 · Cited by 16
“The main aim of the study was to determine how well EMAs, and retrospective questionnaire responses could predict hearing-aid preference using both diagnostic and prognostic analytical approaches (Saeb et al. 2017).”
BackgroundMethodGalimzhanov, A. M., Matetić, A., Tenekecioğlu, E., & Mamas, M. A. (2023) · International Journal of Cardiology · doi:10.1016/j.ijcard.2023.131339 · Cited by 17
Upper limb movement profiles during spontaneous motion in acute stroke
Datta, S., Karmakar, C. K., Rao, A. S., Yan, B., & Palaniswami, M. S. (2021) · Physiological Measurement · doi:10.1088/1361-6579/abf01e · Cited by 13
“As our dataset comprises a single recording from every subject, hence this cross-validation ensures training and test sets comprise data fromdifferent subjects in each fold, ensuring correct and generalized learning (Saeb et al 2017).”
MethodCone Penetration Test Prediction Based on Random Forest Models and Deep Neural Networks
Pacheco, V. L., Bragagnolo, L., Rosa, F. D., & Thomé, A. (2023) · Geotechnical and Geological Engineering · doi:10.1007/s10706-023-02535-0 · Cited by 12
Machine learning algorithms improve MODIS GPP estimates in United States croplands
Menefee, D. S., Lee, T. O., Flynn, K. C., Chen, J., Abraha, M., Baker, J. M., & Suyker, A. (2023) · Frontiers in Remote Sensing · doi:10.3389/frsen.2023.1240895 · Cited by 15
Herráiz, Á. H., Martínez‐Rodrigo, A., Bertomeu‐González, V., Martín, Ó. A., Rieta, J. J., & Alcaraz, R. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105920 · Cited by 14
Zhao, H., Wang, Z., Ding, C., Xue, B., Xu, D., Wang, Z., Hong, H., & Zhu, X. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3476130 · Cited by 15
“We have conducted subject-wise cross-validation based on the leave-one-subject-out strategy [46].”
Guan, Z., Zhang, X., Huang, W., Li, K., Chen, D., Li, W., Sun, J., Chen, L., Mao, Y., Sun, H., et al. (2024) · Neuroscience Bulletin · doi:10.1007/s12264-024-01319-7 · Cited by 14
“For training and validation, 5-fold subject-wise cross-validation was applied to the dataset of 60 participants (DEPs/HCs = 30/30) [58].”
Marsch, L. A., Chen, C.-H., Adams, S. R., Asyyed, A. H., Does, M. B., Hassanpour, S., Hichborn, E. G., Jackson-Morris, M., Jacobson, N. C., Jones, H. K., et al. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.871916 · Cited by 12
“Previous work has shown that whether training data is split by record or by patient can significantly affect model performance (62), with better performance typically being achieved when splitting data by record rather than by participant.”
BackgroundNold, V., Junghans, L., Bayer, B., Bisgen, L., Duerkop, M., Drerup, R., Presser, B., Schwab, T., Bluhmki, E., Wieschalka, S., et al. (2023) · Frontiers in Chemical Engineering · doi:10.3389/fceng.2022.1044245 · Cited by 13
“dependent on the leverage of this data point, disproportionately distort the resulting model (Næs and Ellekjær, 1993; Meloun et al., 2011; Saeb et al., 2017).”
Digital Visualization of Environmental Risk Indicators in the Territory of the Urban Industrial Zone
Safarov, R. Z., Shomanova, Z., Nossenko, Y., et al. (2024) · Sustainability · doi:10.3390/su16125190 · Cited by 10
Soreq, E., Kolanko, M., CRT group, Ravindran, K. K. G., della Monica, C., Revell, V. L., Daniels, S., Joffe, A., Lai, H., Golemme, M., et al. (2025) · Alzheimer s & Dementia · doi:10.1002/alz.70758 · Cited by 14
S2 Heart Sound Detects Aortic Valve Calcification Independent of Hemodynamic Changes in Mice
Dargam, V., Ng, H. H., Nasim, S., Chaparro, D., Irion, C. I., Seshadri, S. R., Barreto, A. B., Danziger, Z. C., Shehadeh, L. A., & Hutcheson, J. D. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.809301 · Cited by 11
“Although used extensively in digital health (12, 51, 52), record-wise splits causes identity confounding, where the classifier is learning from the identity of the subject rather than the identity of the disease (53).”
MethodFeature Importance Analysis for Compensatory Reserve to Predict Hemorrhagic Shock
Gupta, J. F., Telfer, B. A., & Convertino, V. A. (2022) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc48229.2022.9871661 · Cited by 10
“Given that samples from the same individual are highly correlated, folds were split by subject in order to avoid data leakage [28].”
BackgroundLeitner, J., Behnke, A., Chiang, P.-H., Ritter, M. L., Millen, M. M., & Dey, S. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3239366 · Cited by 12
“We carry out LOSO CV to mirror the clinically relevant use-case scenario of diagnosis for newly infected subjects [43].”
MethodSynthesising 2D Video from 3D Motion Data for Machine Learning Applications
Mundt, M., Oberlack, H., Goldacre, M., Powles, J., Funken, J., Morris, C., Potthast, W., & Alderson, J. A. (2022) · Sensors · doi:10.3390/s22176522 · Cited by 9
“The dataset was split into a 70% training, 15% validation and 15% test set with no participant being part of more than one dataset (subject-wise split) [43].”
A survey of autonomous monitoring systems in mental health
Gopalakrishnan, A., Gururajan, R., Zhou, X., Venkataraman, R., Chan, K. C., & Higgins, N. (2024) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1527 · Cited by 18
Melshin, G., DiMaggio, A., Zeramdini, N., Mackinley, M., Palaniyappan, L. K., & Voppel, A. (2025) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-025-00040-1 · Cited by 12
Yuan, G., He, W., & Liu, G. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.830820 · Cited by 9
“In the outer loop and using the selected subset of features, the RFC was evaluated by unseen test data set via a subject-wise 10-fold cross-validation (Saeb et al., 2017).”
MethodDoheny, E. P., Flood, M. W., Ryan, S., McCarthy, C., O’Carroll, O., O’Seaghdha, C. M., Mallon, P. W. G., Feeney, E. R., Keatings, V. M., Wilson, M., et al. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104911 · Cited by 11
“Tenfold cross-validation was used to develop the predictive model, and to assess model performance, with feature selection and model optimisation performed within each fold [26,27].”
Food Choices after Cognitive Load: An Affective Computing Approach
Kappattanavar, A. M., Hecker, P., Moontaha, S., Steckhan, N., & Bert Arnrich, A. (2023) · Sensors · doi:10.3390/s23146597 · Cited by 13
Virtual reality for assessing stereopsis performance and eye characteristics in Post-COVID
Mehringer, W. A., Stoeve, M., Krauss, D., Ring, M., Steußloff, F., Güttes, M., Zott, J., Hohberger, B., Michelson, G., & Eskofier, B. M. (2023) · Scientific Reports · doi:10.1038/s41598-023-40263-w · Cited by 11
“The LOOCV leads to the least biased performance estimation and should be applied when working with clinical data since it mimics the clinical use-case scenario of diagnosis 89 .”
MethodRathakrishnan, K., Min, S. N., & Park, S. J. (2020) · Applied Sciences · doi:10.3390/app11010192 · Cited by 14
Payne, V. L., Sattar, U., Wright, M. C., Hill, E., Butler, J. M., Macpherson, B. C., Jeppesen, A., Fiol, G. D., & Madaras‐Kelly, K. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae089 · Cited by 10
Goffart, S., Delingette, H., Chierici, A. P., Guzzi, L., Nasr, B., Lareyre, F., & Raffort, J. (2025) · Angiology · doi:10.1177/00033197241310572 · Cited by 15
“42 The k-fold CV is used in most of the publications however the initial data splitting and the number of folds in the training is various.”
Lampe, J. W., Padmanaban, S., Becker, L. B., et al. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2934682 · Cited by 8
“The need to approximate use-case in developing machine learning algorithms for clinical applications has been examined before [17].”
BackgroundChopra, S., Moroni, M., Sanjak, J. S., MacMillan, L., Hritzo, B. A., Martello, S. E., Bylicky, M. A., May, J. M., Coleman, C. N., & Aryankalayil, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-95120-5 · Cited by 8
Chalkidis, G., McPherson, J. P., Beck, A. C., Newman, M. G., Yui, S., & Staes, C. J. (2022) · JCO Clinical Cancer Informatics · doi:10.1200/cci.21.00163 · Cited by 12
“22 23-25, We used repeated Monte Carlo record-wise cross-validation simulations 34,35,39 to train and evaluate the classi fi er.”
MethodHerráiz, Á. H., Martínez‐Rodrigo, A., Carneiro, D. R., Bertomeu‐González, V., Rieta, J. J., & Alcaraz, R. (2023) · IEEE Access · doi:10.1109/access.2023.3317793 · Cited by 9
“Unfortunately, patient-specific training and testing of common supervised learning algorithms, i.e., considering samples from the same patients for training and testing, often prevents them from optimal generalization [53], [54].”
BackgroundBaek, H.-S., Kim, J., Jeong, C., Lee, J., Ha, J., Jo, K., Kim, M., Sohn, T. S., Lee, I. S., Lee, J. M., et al. (2024) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/clinem/dgae254 · Cited by 8
Acoustic Features for Identifying Suicide Risk in Crisis Hotline Callers: Machine Learning Approach
Su, Z., Jiang, H., Yang, Y., Hou, X., Su, Y., & Yang, L. (2025) · Journal of Medical Internet Research · doi:10.2196/67772 · Cited by 12
“In the training set, we employed a grouped 5-fold cross-validation approach with a grid search strategy to optimize the model parameters [49].”
Automatic cough detection via a multi-sensor smart garment using machine learning
Dixon, P. C., Dubeau, S., Roy, J.-F., & Fournier, P.-A. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110192 · Cited by 8
Liu, L., Jing, B., Liu, X., Li, R., Wan, Z., Zhang, J., Ouyang, X., Kong, Q.-N., Xiao-ling, K., Wang, D., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102099 · Cited by 11
“26 By implementing 5-fold cross-validation, the model is exposed to a diverse range of histologic patterns, which helps mitigate the impact of slide-to-slide variability and focal heterogeneity.”
Jiang, B., Dorosan, M., Leong, J. W. H., Ong, M. E. H., Lam, S. S. W., & Ang, T. L. (2024) · Singapore Medical Journal · doi:10.4103/singaporemedj.smj-2023-187 · Cited by 7
“[34] For the relatively more balanced categories of poor and good bowel preparation, BPMs were validated with a five‑fold patient‑level and class‑stratified CV strategy.”
MethodData Science Approaches for Effective Use of Mobile Device–Based Collection of Real‐World Data
Omberg, L., Neto, E. C., & Mangravite, L. M. (2020) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1781 · Cited by 7
“A recent literature review of mobile health classification studies demonstrated that 47% had artificially inflated the performance of their measures through failure to account for the identity of individual data points.(8) Our own quantification of this effect across three studies showed that identity confounding can…”
Background
Metadata
- Authors
- Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Saeb, Sohrab
- Lonini, Luca
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-03-15
- Volume
- 6
- Issue
- 5
- Pages
- 1-9
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC5441397
- OpenAlex
- W2595557058
- Cited by
- 367
- References
- 36
- Citations by year
- 2026: 87 · 2025: 24 · 2024: 35 · 2023: 39 · 2022: 57 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 11 · 2017: 10
- FWCI
- 28.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (P20MH090318, R01MH100482)
- National Institute of Neurological Disorders and Stroke (R01NS063399)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- cross-validation, machine learning in healthcare, wearable sensors, accelerometer, smartphone health data, overfitting, clinical decision support
- MeSH terms
- Smartphone, Machine Learning, Wearable Electronic Devices, Algorithms, Humans, Reproducibility of Results, Exercise, Monitoring, Ambulatory, Accelerometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.