The need to approximate the use-case in clinical machine learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, data leakage due to identity confounding is not limited to deep neural networks, and has been uncovered using random forests (Saeb et al., 2017; Chaibub Neto et al., 2019; Tougui et al., 2021) and support vector machines (Tougui et al., 2021).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“For example, if a study trains a classifier to identify individual subjects with a disease, then there may be stronger bias when the study involves fewer participants (Saeb et al., 2017).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“This conclusion mirrors results from studies examining a variety of other types of data and classifier models (Saeb et al., 2017; Wen et al., 2018, 2020; Chaibub Neto et al., 2019; Tougui et al., 2021; Tampu et al., 2022).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 2×
“The number of trees for the random forest classifier was set to 50, which was optimized based on the out of bag error [22].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“Machine learning has grown massively over the past decades, with countless applications in technology, marketing, and science [1].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Inappropriatevalidationprocedures,ontheotherhand,canlead to unreliable results, which might contribute to the problem of irreproducibility of research findings [31, 32] and thereby undermine the trust in both medicine and data science.”
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · cited 1×
“Splitting data into training and test subsets can be done using various methods, such as leave-one-out, leave-p-out, k-fold, and Monte-Carlo sampling [18].”
The Rise of Consumer Health Wearables: Promises and Barriers
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001953 · Cited by 1,235 · cited 1×
“A driving factor is that people carry these devices with them most of the time, and thus a growing amount of daily life data, such as physical activities [5, 6], is becoming available.”
Research works printed but not linked: 1
- Google Scholar. Google 2016 https://scholar.google.com (6 April 2017, date last accessed).
Cited by
Items 301–357 of 357
Arian, R., Far, M. B., Sadeghi, R., et al. (2026) · Translational Vision Science & Technology · doi:10.1167/tvst.15.4.7
“The remaining data were partitioned into training and validation sets using a “subject-wise” method, 13–15 ensuring that all B-scans from a single patient were allocated exclusively to either the validation or training dataset to avoid any overestimation of performance.”
Migda, B., Madetko‐Alster, N., Migda, A. M., Kutyłowski, M., & Alster, P. (2026) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2026.1816084
“As discussed by Saeb et al. (2017) validation strategies must reflect the intended clinical use-case.”
Screening anxiety via contrastive autobiographical recall
Shahid, S. I., Najaran, M. H. T., Förster, F., & Steuber, V. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1798100
“…with limited sample sizes, predictive performance estimates can be highly variable and may become overly optimistic when evaluation permits leakage between training and test data (16), or uses inappropriate cross-validation schemes such as sample-level rather than participant-level splitting (17).”
Kim, B., Chae, M., Seo, H., Jeong, J., Park, S., Jeong, I., Kim, Y., Kim, Y., Jung, T., Lee, T. H., et al. (2026) · Journal of Affective Disorders · doi:10.1016/j.jad.2026.122076 · Cited by 2
Oesterle, T., Xiao, Y., Bormann, N. L., Bergen‐Cico, D., Lai, B., Sharma, H., & Salekin, A. (2026) · Drug and Alcohol Dependence Reports · doi:10.1016/j.dadr.2026.100457 · Cited by 1
“Model performance was estimated using 5-fold user-disjoint cross-validation (Gorriz et al., 2024; Saeb et al., 2017), such that each participant ’ s data appeared in only one fold.”
Szponar, M., Gmaj, B., Orłowski, J., & Kamiński, J. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.110779
Hattori, M., & Hasunuma, N. (2025) · Methods of Information in Medicine · doi:10.1055/a-2895-5222
Manshouri, N. (2026) · Healthcare Analytics · doi:10.1016/j.health.2026.100477 · Cited by 4
Chen, T., & Huo, G. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00943-9 · Cited by 1
Prediction of cognitive impairment in epilepsy with spike-and-wave activation in sleep based on EEG
Han, R., Li, J., Liang, C., Ma, J. Y., Luo, J., & Peng, B. (2026) · Therapeutic Advances in Neurological Disorders · doi:10.1177/17562864261453784
“24 Additionally, we also ensured that the distribution of data in the training and test sets was as similar as possible.”
Geometry-informed correction of projection bias in browser-based monocular squat assessment
Iizuka, R., Yamada, K., Sato, M., Gotoda, N., & Akagi, R. (2026) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2026.1831625 · Cited by 1
“We demonstrated that a simple linear correction model validated through LOSO-CV effectively mitigated this systematic underestimation, reducing the RMSE to less than 4.0°.”
InfluentialMartínez-Reyes, D. C., Navas, O.-M., Rey, J. S. R., Borras, C. P., & Villegas, D. F. (2026) · Biomechanics · doi:10.3390/biomechanics6030062
“To ensure generalization, the dataset was split using a strict subject-held-out protocol, with 70% of participants used for training, 15% for validation, and 15% for testing [33,34].”
Yousefi, S., & Daliri, M. R. (2026) · Brain Informatics · doi:10.1186/s40708-026-00324-y
“…performance, a leave-one-subject-out (LOSO) cross-validation strategy was employed, in which one subject was held out as the test sample while all remaining subjects were used for model development, and this procedure was repeated until each subject had served once as the test case [45].”
Herath, H. M. K. K. M. B., Madusanka, N., Hewage, C. T. E. R., & Lee, B.-I. (2026) · Bioengineering · doi:10.3390/bioengineering13080860 · Cited by 1
“Subject identifiers were preserved end-to-end so that 5-fold cross-validation operated at the subject level rather than the window level [35], eliminating leakage from the same-subject windows appearing in both the training and test sets.”
Best, R., Gao, S. S., MacDonell, S.-Y., et al. (2026) · BJA Open · doi:10.1016/j.bjao.2026.100644
“38 A patient-based approach was chosen to prevent data leakage (i.e. a patient ’ s measurements could potentially be present in both the training and validation sets), and record-wise cross-validation may overestimate prediction accuracy compared with subject-wise cross-validation.”
InfluentialCheng, C.-Y., Hsieh, C.-Y., Chou, M.-Y., Su, Y.-E., & Chang, I. (2026) · doi:10.2196/preprints.107578
Almuhini, A., Ahanathapillai, V., Raza, Z., Aslam, I., Patel, A., Evans, C., Pecchia, L., & Piaggio, D. (2026) · Sensors · doi:10.3390/s26154822
“In contrast, subject-wise validation provides a more realistic measure of generalisation [11,12].”
Hierarchically structured data can undermine machine learning results in geoscience
Scharf, T. W., Daggitt, M. L., & Kirkland, C. L. (2026) · Earth and Planetary Science Letters · doi:10.1016/j.epsl.2026.120250
Roinson, R., Karamanoukian, A., Boucher, J.-P., et al. (2026) · Journal of Applied Biomechanics · doi:10.1123/jab.2025-0313
Zhang, S., Liang, D., Lei, Y., Luo, Y., & Shi, R. (2026) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2026.1898243 · Cited by 2
“Grouped validation should match the level at which the model is expected to generalize (Saeb et al., 2017; Little et al., 2017; Bradshaw et al., 2023).”
Ferreira, B. J. M., Andrade, C. L. D., Bresolin, T., de Araújo, J. S., do Nascimento, E. M., de Moraes, S. A., Rodrigues, M. C. F., Melo, S. A. F., Menezes, D. R., & Queiróz, M. A. Á. (2026) · AgriEngineering · doi:10.3390/agriengineering8080338
“By contrast, the nested leave-one-animal-out framework adopted in the present study completely isolated each test animal during preprocessing, feature selection, model optimization, and final evaluation, thereby providing a substantially more rigorous assessment of model generalization [25,26].”
InfluentialElectrodermal signatures of stress during tactical law enforcement training
Bigliassi, M., Curtis, J., Yasutake, V. V., Acosta, K., Bagherian, D., Laing, R., & Balban, M. Y. (2026) · Stress · doi:10.1080/10253890.2026.2723354
Firdaus, M. N. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7331360
Choi, A., Choi, D., & Park, J. (2026) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813261484491
Segment-wise split leakage in hierarchical multimodal prediction: a case study on DAIC-WOZ
Lee, S., & Kim, J. (2026) · Korean Journal of Applied Statistics · doi:10.5351/kjas.2026.39.4.385
Chemical Fingerprints of New vs Weathered Plastics: A Machine Learning Approach
Nguyen, H., Fries, E., Jones, D., Megson, D., Zonderman, A., Hildebrandt, L., Coffin, S., Pröfrock, D., & Sühring, R. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.6c03575 · Cited by 1
Level-Aware Residual Mixup for Defect Classification in Material Extrusion with Limited Print Jobs
Kim, Y., & Lee, S. (2026) · Materials · doi:10.3390/ma19173769
“Since segments from the same job may share job-specific characteristics, random splitting can lead to overly optimistic performance estimates by allowing the model to exploit print-job information rather than learn defect-related patterns [14,38].”
Spanodimitriou, Y., Ejaz, K. T., Ciampi, G., Scorpio, M., Masullo, M., Rosato, A., Maffei, L., & Sibilio, S. (2026) · Sensors · doi:10.3390/s26175640 · Cited by 1
Can Humor Reveal Dark Personalities?
Glinkina, L. S., Доптан, Э. С.-О., & Kosonogov, V. (2026) · Journal of Individual Differences · doi:10.1027/1614-0001/a000465
Shi, C., Kang, X., Nie, L., et al. (2026) · Applied Sciences · doi:10.3390/app16188998
Facial classification of five Huacaya alpaca biotypes using deep learningPreprint
Ticona, J., Quenaya, A., Mamani-Pari, D., & Maquera, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7441892
Li, H., Wang, C., Xu, Y., Wang, X., & Zhao, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7459680
Szczepański, T., Jasiński, M., Koźba‐Gosztyła, M., Więcław, J., Szymczyk, P., Bladowska, J., & Czapiga, B. (2026) · Diagnostics · doi:10.3390/diagnostics16203291
“…that ignore this clustering overstate precision [51]; likewise, segment-level leave-one-out cross-validation allows segments from the same patient to inform both training and testing, so leave-one-patient-out validation, which better approximates prospective use, was the primary scheme [52].”
Bose, S., Chakraborthy, A., Maulik, G., Nag, A., Sukkuea, A., Alahi, M. E. E., & Altinsoy, M. E. (2026) · Sensors and Actuators A Physical · doi:10.1016/j.sna.2026.118531
Sabiku, I., Kwofie, E. M., Allotey, D. K., Kwofie, E. A., Iheonye, A., Asavajaru, P., Klassen, D., Das, P. C., Bhagwat, A., Rajagopalan, N., et al. (2026) · Food and Bioproducts Processing · doi:10.1016/j.fbp.2026.09.008
Lin, J., & Wu, Z. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.111435
Boylu, D., Polat, S., Daldal, N., & Önder, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7493770
Toward Multimodal Detection of Body-Focused Repetitive Behaviors Using Vision and Wearable Sensors
Sasikumar, N., Bhansali, J., & Tiglao, N. M. C. (2026) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc70543.2026.11693919
“…validation, macro-F1 drops by 8.3 points to 0.610 ± 0.115 (95% CI [0.585, 0.636], Student-t with df =80 ); the stratified and LOSO confidence intervals do not overlap, so the participant-leakage gap is statistically meaningful and consistent in magnitude with the wearable-HAR literature [8], [14].”
InfluentialMultimodal Evaluation of Exoskeleton Assistance Using Optimized Machine Learning
Buxman, S., & Kakhki, F. D. (2026) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813261485895
Jofilsan, N. C., de Lima, S. M. L., & dos Santos, W. P. (2026) · IEEE Access · doi:10.1109/access.2026.3737456
“Saeb et al. showed this clearly: by-record cross-validation, which mixes a subject’s records across folds, can report almost perfect accuracy where by-subject testing shows results near chance [9].”
InfluentialVoxNeuro: Leakage-Safe Parkinson’s Screening Support via Multi-View Grassmannian Speech AnalysisPreprint
Khan, L., & Waghe, M. (2026) · medRxiv · doi:10.64898/2026.09.27.26364114
Cross-Modal Robust EEG Biomarkers for Sustained Attention in Multimodal Learning
Ullamgunta, H., Kumar, S., Bablani, A., & Sarma, H. (2026) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct68978.2026.11700712
“To respect the hierarchical structure of the EEG data, statistical inference was performed at the subject level rather than by pooling epochs across participants [14].”
Graph-Based Auditory Brainstem Response Threshold Estimation from Incomplete Intensity Series
Tao, Y., Ren, P., & Han, P. (2026) · Bioengineering · doi:10.3390/bioengineering13101128
“This design reduces the optimistic bias that can arise from splitting correlated records independently [30–32].”
Jeong, J., & Jung, D. (2026) · Gels · doi:10.3390/gels12100883
Em, P. P., & Teoh, T. S. (2026) · Sensors · doi:10.3390/s26196202
A Two-Stage Hierarchical Classifier for Pain Detection and Localisation from Physiological Signals
Jim, S. P., Rifat, A. K., Hossain, T., Alsherfawi, A., Uddin, M. Z., Yonezawa, T., & Ahad, M. A. R. (2026) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw71585.2026.11712101
“When subject counts are small, fixed-split accuracy has wide confidence intervals [10], and subject-level leakage or selection bias inflates reported scores [21].”
Li, B., Chen, Z., Sun, H., Jiang, X., Gu, J., Liang, C., Hu, S., & Li, K. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831182
“Furthermore, regarding the validation paradigm for clinical small-sample machine learning, avoiding subject leakage (i.e., videos of the same subject being scattered across training and testing sets) is a current scholarly consensus[17].”
A dual-stream multi-scale cross-attention network for pericardial effusion detection
Gao, J., Ju, P., Chen, L., 李玉赞, Huang, J., & Wu, J. (2026) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2026.2736736
Gonabadi, A. M., & Burnfield, J. M. (2026) · Applied Sciences · doi:10.3390/app16199891
Evaluation Integrity in Spatially Grouped Machine Learning: Lessons from Drillhole Grade EstimationPreprint
Maldeh, S. A., Mohammadzadeh, J., & Harifi, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7576916
Evaluation Bias in Cross-Context Generalization of EEG-Based Affective Computing: A Simulation Study
Kahlan, M. F. T., Amini, M. M. R., Gharehveran, S. S., Shirini, K., Abdolahi, A., & Shahhoseini, A. (2026) · Algorithms · doi:10.3390/a19100882
Korostylova, M., Kobylin, O., & Kobylin, I. (2026) · ГРААЛЬ НАУКИ · doi:10.36074/grail-of-science.16.10.2026.034
“…sample into training and test sets without taking into account which participant performed each repetition leads to information leakage: the model may ‘memorise’ the motor characteristics of a specific individual rather than learning to distinguish between correct and incorrect performance [12].”
Divergent Failure Modes of Two Intrusion Detection Benchmarks Under A Common Audit
S, A. K., & Selvavinayaki (2026) · Eduschool International Journal of Data Science and Machine learning (EIJDSML) · doi:10.63090/eijdsml/3139.7344.0012
Okuyucu, K., Şenel, A., Gülünay, A., & Yücel, M. (2026) · Diagnostics · doi:10.3390/diagnostics16203403
“This person-level split is methodologically more rigorous than a random observation-level split in repeated-measures designs, as the latter would allow the model to learn participant-specific patterns during training and artificially inflate test performance [33].”
Marin-Rodriguez, W., Andrade-Girón, D., Vellon-Flores, V., Díaz-Vega, E., & Solano-Armas, T. (2026) · Research Square · doi:10.21203/rs.3.rs-11235253/v1
Metadata
- Authors
- Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Saeb, Sohrab
- Lonini, Luca
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-03-15
- Volume
- 6
- Issue
- 5
- Pages
- 1-9
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC5441397
- OpenAlex
- W2595557058
- Cited by
- 367
- References
- 36
- Citations by year
- 2026: 87 · 2025: 24 · 2024: 35 · 2023: 39 · 2022: 57 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 11 · 2017: 10
- FWCI
- 28.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (P20MH090318, R01MH100482)
- National Institute of Neurological Disorders and Stroke (R01NS063399)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- cross-validation, machine learning in healthcare, wearable sensors, accelerometer, smartphone health data, overfitting, clinical decision support
- MeSH terms
- Smartphone, Machine Learning, Wearable Electronic Devices, Algorithms, Humans, Reproducibility of Results, Exercise, Monitoring, Ambulatory, Accelerometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.