The need to approximate the use-case in clinical machine learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, data leakage due to identity confounding is not limited to deep neural networks, and has been uncovered using random forests (Saeb et al., 2017; Chaibub Neto et al., 2019; Tougui et al., 2021) and support vector machines (Tougui et al., 2021).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“For example, if a study trains a classifier to identify individual subjects with a disease, then there may be stronger bias when the study involves fewer participants (Saeb et al., 2017).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“This conclusion mirrors results from studies examining a variety of other types of data and classifier models (Saeb et al., 2017; Wen et al., 2018, 2020; Chaibub Neto et al., 2019; Tougui et al., 2021; Tampu et al., 2022).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 2×
“The number of trees for the random forest classifier was set to 50, which was optimized based on the out of bag error [22].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“Machine learning has grown massively over the past decades, with countless applications in technology, marketing, and science [1].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Inappropriatevalidationprocedures,ontheotherhand,canlead to unreliable results, which might contribute to the problem of irreproducibility of research findings [31, 32] and thereby undermine the trust in both medicine and data science.”
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · cited 1×
“Splitting data into training and test subsets can be done using various methods, such as leave-one-out, leave-p-out, k-fold, and Monte-Carlo sampling [18].”
The Rise of Consumer Health Wearables: Promises and Barriers
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001953 · Cited by 1,235 · cited 1×
“A driving factor is that people carry these devices with them most of the time, and thus a growing amount of daily life data, such as physical activities [5, 6], is becoming available.”
Research works printed but not linked: 1
- Google Scholar. Google 2016 https://scholar.google.com (6 April 2017, date last accessed).
Cited by
Items 201–300 of 357
Ocana-Santero, G., Warming, H., Munday, V., MacKay, H. A., Gibeily, C., Hemingway, C., Stacey, J. A., Saha, A., Lazarte, I. P., Bachetta, A., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-59659-5 · Cited by 9
“A 60/40 random train/test split was performed across individual animals to ensure subject-wise cross-validation 103 .”
Image Segmentation and Classification Using CNN Model to Detect Brain Tumors
Hilles, S. M. S., & Saleh, N. S. (2021) · International Informatics and Software Engineering Conference (IISEC) · doi:10.1109/iisec54230.2021.9672428 · Cited by 8
Modeling Social Network of Professional Sports Athletes Based on Machine Learning Algorithms
Ying, M. (2022) · International Transactions on Electrical Energy Systems · doi:10.1155/2022/6283618 · Cited by 7
RetractedChen, H., Xu, L., Gu, J., et al. (2022) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2022.104547 · Cited by 6
Khodabandeh, Z., Rabbani, H., Ashtari, F., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-1547669/v1 · Cited by 6
Lipps, C., Frère, J., & Weissland, T. (2024) · Journal of Experimental Orthopaedics · doi:10.1002/jeo2.70081 · Cited by 6
“In other words, athletes in the training set were not included in the test set [29].”
Enhanced Alzheimer’s detection with EEG source imaging and multi-branch joint attention
Sun, Y., Feng, L., Xu, B., Jia, S., Li, D., Ni, W., & Jia, Z. (2025) · Journal of Neural Engineering · doi:10.1088/1741-2552/add8bb · Cited by 6
“The performance of the model is evaluated using the LOSO method, which ensures that the model identifies pathological features rather than individual patient characteristics (Saeb et al 2017).”
Salas, E. A. L., & Kumaran, S. S. (2023) · Geo-spatial Information Science · doi:10.1080/10095020.2023.2211612 · Cited by 6
“We assessed the performance of the model by using leave-one-out cross-validation instead of the train/test split validation method since the former is more robust to confounding model effects (Saeb et al. 2017).”
Žulj, S., Carvalho, P., Ribeiro, R. T., Andrade, R., & Magjarević, R. (2021) · Journal of Applied Biomedicine · doi:10.1016/j.bbe.2021.04.013 · Cited by 9
Addressing robust travel mode identification with individual trip‐chain trajectory noise reduction
Zeng, J., Zhang, G., Hu, Y., & Wang, D. (2022) · IET Intelligent Transport Systems · doi:10.1049/itr2.12243 · Cited by 5
Technology Management for Accelerated Recovery during COVID-19
Morandé, S., & Tewari, V. (2020) · SEISENSE Journal of Management · doi:10.33215/sjom.v3i5.445 · Cited by 5
Enhanced diagnostic interpretation of the MoCA using machine learning
Gourdeau, C., Gourdeau, C. L., Bernier, P. J., & Laforce, R. J. (2026) · Frontiers in Neuroscience · doi:10.3389/fnins.2026.1679649 · Cited by 4
“According to Saeb et al. (2017), failing to account for this hierarchical structure introduces a high risk of “data leakage.””
Xu, R., Cao, S. C., Barnett‐Cowan, M., Irving, E. L., Niechwiej‐Szwedo, E., & Kearns, S. K. (2026) · Sensors · doi:10.3390/s26123627 · Cited by 6
“The primary evaluation employed leave-one-subject-out (LOSO) cross-validation, in which all trials from one pilot were held out as the test set, while the remaining pilots formed the training set [29].”
Lam, G., Rish, I., & Dixon, P. C. (2023) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2023.111606 · Cited by 4
Vocal expression of emotions discriminates dementia with Lewy bodies from Alzheimer's disease
Kobayashi, M., Yamada, Y., Shinkawa, K., Nemoto, M., Ota, M., Nemoto, K., & Arai, T. (2024) · Alzheimer s & Dementia Diagnosis Assessment & Disease Monitoring · doi:10.1002/dad2.12594 · Cited by 8
“30 Wealsocomparedperformance againstabaselinemodelusingclinico-demographic variables (i.e., Mini-Mental State Examination [MMSE] score, age, sex, and years of education).”
BackgroundChallenges in Resampling-Based Performance Estimation
Schulz-Kümpel, H., Boulesteix, A., Fischer, S., & Hornung, R. (2025) · Society, environment and statistics · doi:10.1007/978-3-032-10004-7_3 · Cited by 5
MRI image segmentation using machine learning networks and level set approaches
Almajmaie, L. K. A., Raheem, A. R., Mahmood, W. A., & Albawi, S. (2021) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v12i1.pp793-801 · Cited by 8
Parsarad, S., Saeedizadeh, N., Soufi, G. J., Shafieyoon, S., Hekmatnia, F., Zarei, A. P., Soleimany, S., Yousefi, A., Nazari, H., Torabi, P., et al. (2023) · Journal of Imaging · doi:10.3390/jimaging9080159 · Cited by 4
Sarwar, A., & Agu, E. (2021) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh52753.2021.00020 · Cited by 7
Development of inertial sensor-based methods to assess physical fatigue in running applications
Marotta, L. (2022) · doi:10.3990/1.9789036555067 · Cited by 4
Efficient Human Activity Recognition Using Machine Learning and Wearable Sensor Data
Zhong, Z., & Liu, B. (2025) · Applied Sciences · doi:10.3390/app15084075 · Cited by 4
“The were split according to subjects, ensuring that the training and test folds The data were split according to subjects, ensuring that the training and test folds consist of records from different subjects [29]. consist of records from different subjects [29].”
Vongjarudech, T., Dosne, A., Remmerie, B. M. M., et al. (2025) · CPT Pharmacometrics & Systems Pharmacology · doi:10.1002/psp4.70047 · Cited by 3
“It provides insight into how well a model fits the data and assesses its ability to predict data not used for model fitting, ensuring model generalizability when applied to an independent dataset [23–26].”
Mula, S., Zangróniz, R., Martín, Ó. A., Rieta, J. J., & Alcaraz, R. (2025) · IEEE Access · doi:10.1109/access.2025.3604652 · Cited by 5
“Indeed, training and testing on the same patients, although using different segments extracted from different time points, often lead classification model to memorize patient-specific features, resulting in poor generalization, and overly optimistic results [58], [59].”
Narrowing the gap: expected versus deployment performance
Zhou, A. X., Aczon, M., Laksana, E., Ledbetter, D., & Wetzel, R. C. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad100 · Cited by 3
Moukaddam, N. J., Lamichhane, B., Salas, R., Goodman, W. K., & Sabharwal, A. (2023) · Behavioural Neurology · doi:10.1155/2023/8552180 · Cited by 4
“We chose LOPO for cross-validation because LOPO approximates clinical deployment [44, 45] and is commonly used for evaluations with a limited number of participants [46–50], as is the case for our study.”
MethodSensor-Based Evaluation of Physical Therapy Exercises
Whitford, A. S., Kim, E., Halilaj, E., Enseki, K. R., Popchak, A., & Hodgins, J. K. (2021) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc46164.2021.9629773 · Cited by 3
Contactless longitudinal monitoring of ageing and dementia-related sleep trajectories in the homePreprint
Soreq, E., Kolanko, M., Ravindran, K. K. G., della Monica, C., Revell, V. L., Daniels, S., Lai, H., Golemme, M., Giovane, M. D., Walsh, C., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4523167/v1 · Cited by 3
Plain, B., Pielage, H., Kramer, S. E., Richter, M. M., Saunders, G. H., Versfeld, N. J., Zekveld, A. A., & Bhuiyan, T. A. (2024) · Trends in Hearing · doi:10.1177/23312165241232551 · Cited by 4
“…the classi fi er using this technique meant it was likely that different trials from the same participant were appearing in both the training and test datasets, which can falsely enhance the results of the classi fi ers (Gholamiangonabadi et al., 2020; Miltiadous et al., 2021; Saeb et al., 2017).”
MethodA portable affective computing system for identifying mate preference
Yuan, G., Wang, T., Ju, W., & Fu, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-68772-2 · Cited by 2
“The calculation formulas are as follows: RFC is a commonly used classification algorithm, and it needs to adjust fewer parameters, making it less easy to overfit and more suitable for generalization to invisible data 29 .”
BackgroundMethodMachine Learning VLSI CAD Experiments Should Consider Atomic Data Groups
Gunter, A. D., & Wilton, S. J. E. (2024) · doi:10.1109/mlcad62225.2024.10740199 · Cited by 2
Kim, S. Y., Shakeri, A., Darabi, S. S., Kim, E., & Kim, K. (2025) · Sensors · doi:10.3390/s25154566 · Cited by 2
“In this study, each subject contributes multiple recordings, so sample-level partitioning of the dataset would allow information from the same individual to appear in both training and in evaluation sets, leading to data leakage and overly optimistic performance estimates [20].”
Gan, G., Xu, Y., Wang, Y. H., et al. (2025) · Medical Physics · doi:10.1002/mp.18058 · Cited by 2
Monitoring of hemorrhage and whole blood resuscitation in non-human primates
Arshad, S. H., Hinojosa‐Laborde, C., Snider, E. J., Telfer, B. A., & Convertino, V. A. (2025) · npj Biosensing · doi:10.1038/s44328-025-00049-8 · Cited by 2
“We split training folds by subject in order to avoid data leakage 31 since samples from a single subject are highly correlated.”
PDGV Dataset: Investigating Deep Learning CNN for Gait-Based Parkinson’s Disease Recognition
Mahfouf, Z., Jarraya, I., Dhieb, T., Néji, M., Farhat, N., Smaoui, E., Hamdani, T. M., Damak, M., Mhiri, C., & Alimi, A. M. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-025-10921-4 · Cited by 3
Outdoor Walking Classification Based on Inertial Measurement Unit and Foot Pressure Sensor Data
Jlassi, O., Emmerzaal, J., Vinco, G., Garcia, F., Ley, C., Grimm, B. P., & Dixon, P. C. (2025) · Sensors · doi:10.3390/s26010232 · Cited by 2
“%) sets using a subject-wise splitting approach to ensure that all data from a single participant were exclusively in either the training or the test set [20].”
Huggins, C. J., Clarke, R., Abásolo, D., Gil‐Rey, E., Tobias, J. H., Deere, K. C., & Allison, S. J. (2022) · Sensors · doi:10.3390/s22239176 · Cited by 2
Wei, J. W. T., Pheng, H. S., & Ong, K. H. (2024) · International Conference on Artificial Intelligence and Data Sciences (AiDAS) · doi:10.1109/aidas63860.2024.10730307 · Cited by 2
“…which divides data into training and test sets without regard for subject identity, enables the algorithm to learn not only the outcome variable of interest but also individual-specific characteristics, potentially resulting in overfitting and limiting generalization to other datasets [18][19].”
BackgroundThe Binomial Block Bootstrap Estimator for Evaluating Loss on Dependent Clusters.
Barnes, M., & Dubrawski, A. W. (2017) · Uncertainty in Artificial Intelligence · Cited by 2
Pattern recognition in data as a diagnosis tool
Carpio, A., Simón, A., Torres, A., & Villa, L. F. (2022) · Journal of Mathematics in Industry · doi:10.1186/s13362-022-00119-w · Cited by 3
Video-based AI Decision Support System for Lifting Risk Assessment
Zhou, G., Aggarwal, V., Yin, M., & Yu, D. (2021) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc52423.2021.9659025 · Cited by 4
“The leave-one-subject-out cross-validation strategy [31], which is capable of evaluating the model’s performance when facing completely unseen people, was utilized to test the model.”
MethodComparison of Two Deep Learning Models for the Recognition of Parkinson’s Disease Gait Patterns
Bobic, V. N., Djuric-Jovicic, M. D., Mišković, N., Kostić, V., & Kvaščev, G. (2023) · International Conference on Electrical, Electronic and Computing Engineering (IcETRAN) · doi:10.1109/icetran59631.2023.10192156 · Cited by 2
“DL models can learn complex predictors, and if data division is performed at a record level rather than a subject level, models could capture features related to the identity of subjects, thus providing high but misleading classification accuracy [25].”
BackgroundKim, D.-Y., Mandalapu, V., Hart, J. M., et al. (2019) · IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE) · doi:10.1109/chase48038.2019.00019 · Cited by 2
The NACOB multi-surface walking dataset
Jlassi, O., Shah, V. R., & Dixon, P. C. (2024) · Scientific Data · doi:10.1038/s41597-024-03683-5 · Cited by 2
Soltanipour, A., Arian, R., Aghababaei, A., Ashtari, F., Zhou, Y., Keane, P. A., & Kafieh, R. (2025) · Bioengineering · doi:10.3390/bioengineering12080847 · Cited by 1
Machine learning-based classification of ice hockey skating tasks using kinematic data
Jlassi, O., Wilkie, E. W. C., Kelly, M., et al. (2025) · Sports Biomechanics · doi:10.1080/14763141.2025.2569580 · Cited by 1
“Feature importance was calculated for the top performing model in each category using features derived from all segments to demonstrate the relevance of each feature in differentiating tasks (Chen & Guestrin, 2016; Saeb et al., 2017).”
Few-Shot Contrastive Learning for Cross-Task Stroke Prognosis Prediction With Multimodal Data
Peng, H., Dai, Y., Zheng, R., Wang, M., Wang, C., Luo, Y., & Wang, H. (2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2026.3661971 · Cited by 2
“Traditional machine learning methods, usually trained with large datasets, tend to overfit with limited data, leading to poor generalization and performance [8], [9], [10].”
Hughes, C. M. L., & Zhang, Y. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1700489 · Cited by 5
Patrick, B., Kanjo, E., & Kaiwartya, O. (2026) · Smart Agricultural Technology · doi:10.1016/j.atech.2026.101986 · Cited by 5
Azam, M. B., & Singh, S. I. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03455-8 · Cited by 2
“Saeb et al. (2017) emphasised that clinical machine learning requires validation approximating the intended use case, typically generalisation to new, unseen subjects [33].”
Jacobucci, R., Shao, W., Kobrinsky, V., & Ammerman, B. A. (2026) · JMIR Mental Health · doi:10.2196/90581 · Cited by 4
“…split the probe to see whether person-specific digital patterns persist over time and are therefore suitable for personalized JITAIs; subject holdouts ask whether learned indicators transfer across heterogeneous digital habits—a harder, but necessary, condition for broader screening [31,39].”
Systematic review of machine learning and deep learning models for EEG-based detection of depression
Luna-Guevara, G. R., Vargas-Huicochea, I., & Rodríguez-Machain, A. C. (2026) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2026.04.030 · Cited by 1
Pe, S., Dagliati, A., Parimbelli, E., Guerra, B. M. V., Sozzi, S., Bellazzi, R., & Nicora, G. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106464 · Cited by 2
Automated Abnormal Vessel Detection in Re-Endothelialized Mouse Lungs: A Proof of Concept
Paciocco, J., Hasan, A., Chan, J. K. D., Taniguchi, D., Kamata, S., Amon, C., Karoubi, G., & Bazylak, A. (2026) · Tissue Engineering Part C Methods · doi:10.1177/19373384261458003 · Cited by 1
“To avoid biasing this diagnostic model, 46 the data were organized into 10 folds manually since some images were taken of different regions of the same lung.”
Ammar, B. B., & Karamti, W. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1881829 · Cited by 1
Xu, R., Zhang, C., Cao, S. C., Barnett‐Cowan, M., Irving, E. L., Niechwiej‐Szwedo, E., & Kearns, S. K. (2026) · IEEE Sensors Journal · doi:10.1109/jsen.2026.3708589 · Cited by 1
“Many EEG classification papers use segment-level k-fold cross-validation, which can place data from the same person in both training and test sets [31], [32], [33].”
Artificial intelligence-enabled strategies for addressing bottlenecks in probiotic development
Im, J., Haque, M. A., Kim, H., Jang, M. J., Razzak, M. A., Song, S., San-Cristóbal, R., Talcott, S., & Ku, S. (2026) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2026.106072 · Cited by 1
Objective assessments of human motor ability of the upper limb: A systematic review
Duque, E., Trefftz, H., & Srivastava, S. (2021) · Technology and Disability · doi:10.3233/tad-200263 · Cited by 3
“Recently, Saeb et al. [18] evaluate reliability of re-ported accuracy in ML models where data comes from wearable sensors or smartphones, pointing out that the on the rise use of ML for clinical predictions brings also relevant challenges to be considered such as generalization and proper cross…”
Que, S., van Esch, R., Cramer, I. C., Verstappen, C. C. A. G., Kloeze, C., Veer, M. V. ’., Dierick, A., Hommerson, S., van Meulen, F., Overeem, S., et al. (2023) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.2653585 · Cited by 1
“We built training datasets and test datasets based on the leave-one-subject-out cross-validation.(19) One data sample of COF was composed of three respiratory features extracted from a 30-second sliding window without overlap, including average inter-breath intervals (IBI), coefficient of variance (CoV) of breathing p…”
MethodA preregistered, open pipeline for early cerebral palsy risk assessment from infant videos
Segado, M., Prosser, L. A., Duncan, A. F., Johnson, M. J., & Kording, K. P. (2026) · GigaScience · doi:10.1093/gigascience/giag003 · Cited by 3
Self-citationMuzychenko, I., Apollonova, I. A., Evans, D., & Zhang, L. (2019) · International Conference on Medical and Health Informatics (ICMHI) · doi:10.1145/3340037.3340062 · Cited by 1
Finger Movement Recognition based on Muscle Synergy using Electromyogram
Shivam, P., Kakoty, N. M., Malarvili, M. B., & Widiyanti, P. (2019) · IEEE Region Humanitarian Technology Conference · doi:10.1109/r10-htc47129.2019.9042432 · Cited by 1
“This is repeated for each subject as test subject and the average of all the test results have been estimated as the average recognition rates [37].”
BackgroundChallenges and Trends in Clinical Data Analytics
Kaddi, S. S., & M.Patil, M. (2020) · Journal of Science and Technology · doi:10.46243/jst.2020.v5.i4.pp348-360 · Cited by 1
Theilig, M.-M., Knapp, A. A., Nicholas, J., Zarnekow, R., & Mohr, D. C. (2020) · Research Square · doi:10.21203/rs.3.rs-68076/v1 · Cited by 1
“Methods for the validation of various inputs, such as accelerometer data, have also shown to have systematic shortcomings within applications for medical machine learning [21].”
BackgroundSelf-citationTheilig, M.-M., Knapp, A. A., Nicholas, J., Zarnekow, R., & Mohr, D. C. (2020) · Research Square · doi:10.21203/rs.3.rs-68076/v2 · Cited by 1
Self-citationIHC Color Histograms for Unsupervised Ki67 Proliferation Index Calculation
Geread, S., Morreale, P., Dony, R. D., et al. (2022) · doi:10.32920/21287937.v1 · Cited by 1
Arian, R., Aghababaei, A., Soltanipour, A., et al. (2023) · medRxiv · doi:10.1101/2023.09.03.23294985 · Cited by 1
“Therefore, the images of the same participant cannot be used in both training and validation datasets concurrently, preventing an overestimation of the model performance (37).”
BackgroundA Template and Tutorial for Preregistering Studies Using Passive Smartphone MeasuresPreprint
Langener, A. M., Siepe, B. S., Elsherif, M. M., Niemeijer, K., Andresen, P. K., Akre, S., Bringmann, L. F., Cohen, Z. D., Choukas, N. R., Drexl, K., et al. (2024) · doi:10.31234/osf.io/p4xf8 · Cited by 1
Karbalaie, A., Abtahi, F., & Häger, C. K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5352124 · Cited by 1
Feng, M., Zhang, Q., & Fang, H. (2026) · Scientific Data · doi:10.1038/s41597-026-06710-9 · Cited by 4
DeMasi, O., & Recht, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05635
“While there are examples of researchers attempting to reproduce results on related datasets [3, 30], reproducing results on different datasets (and thus populations) is rarely done.”
BackgroundResultWeiner, M. G. (2018) · CHEST Journal · doi:10.1016/j.chest.2018.01.027 · Cited by 1
Dependency Leakage: Analysis and Scalable Estimators.Preprint
Barnes, M., & Dubrawski, A. W. (2018) · arXiv (Cornell University)
Classification of in-bed movements using a mattress-based sensor array
Lee, Y. J., Sakr, M., Wu, L. C., Ipsiroglu, O., & der Loos, H. F. M. V. (2019) · Sleep Medicine · doi:10.1016/j.sleep.2019.11.1107
Evaluation of Methods for Data-Driven Tools that Empower Mental Health Professionals
DeMasi, O. (2019) · UC Berkeley
Granular Motor State Monitoring of Free Living Parkinson's Disease Patients via Deep LearningPreprint
Yüksel, K. A., Goschenhofer, J., Varma, H. V., Fietzek, U. M., & Pfister, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.06913
“Moreover, only a minority of the literature used a LOSO validation scheme, which is crucial for real-life medical applications to ensure generalizability [23]; especially in this problem where there is a significant variety in how these symptoms affects patient motions.”
BackgroundClustering methods and Bayesian inference for the analysis of the evolution of immune disordersPreprint
Carpio, A., Simón, A., & Villa, L. F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11531
“Developing tools to assess the reliability of such automatic procedures and to choose the best method for different situations and clinical environments has become essential [18].”
MethodFiltjens, B., Ginis, P., Nieuwboer, A., Slaets, P., & Vanrumste, B. (2021) · arXiv (Cornell University)
Guo, C. C., Chiesa, P. A., de Moor, C., Fazeli, M. S., Schofield, T., Hofer, K., Belachew, S. M., & Scotland, A. (2022) · doi:10.2196/preprints.37683
Theilig, M.-M., Knapp, A. A., Nicholas, J., Zarnekow, R., & Mohr, D. C. (2021) · doi:10.2196/preprints.28014
Self-citationMuzychenko, Y. N., Apollonova, I. A., & Evans, D. (2022) · Вестник Российского университета дружбы народов. Серия: Психология и педагогика · doi:10.22363/2313-1683-2022-19-1-128-145 · Cited by 2
IHC Color Histograms for Unsupervised Ki67 Proliferation Index Calculation
Geread, S., Morreale, P., Dony, R. D., et al. (2022) · doi:10.32920/21287937
Postgraduate Teaching Reform Based on Machine Learning and Improved SVM Algorithm
(2021) · Machine Learning Theory and Practice · doi:10.38007/ml.2021.020205
Gao, Y., DeSouza, G. N., Berardi, M. L., & Dietrich, M. (2023) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec53210.2023.10253983
Analyzing morphological alternations of vessels in multiple Sclerosis using SLO images of the eyesPreprint
Soltanipour, A., Arian, R., Aghababaei, A., Kafieh, R., & Ashtari, F. (2023) · medRxiv · doi:10.1101/2023.12.12.23299846
“To prevent any leakage among test, validation, and training samples, a subject-wise approach was adopted, wherein all images associated with a specific subject were exclusively designated for either the test, validation, or training set [20].”
MethodLangener, A. M. (2024) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.1069770227 · Cited by 1
A Template and Tutorial for Preregistering Studies Using Passive Smartphone MeasuresPreprint
Langener, A. M., Siepe, B. S., Elsherif, M. M., Niemeijer, K., Andresen, P. K., Akre, S., Bringmann, L. F., Cohen, Z. D., Choukas, N. R., Drexl, K., et al. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/p4xf8_v1
Deep learning-based physical exercise assessment of older adults using single-camera videos
Stefanova, V., Maeyens, E., Brughmans, J., Vargemidis, D., Lebleu, J., Stulens, L., Broeder, S., Gilis, K., Deschodt, M., Vanwanseele, B., et al. (2025) · medRxiv · doi:10.1371/journal.pdig.0001576
Chatziioannou, M., Kougiagka, E., & Klaoudatos, D. (2025) · Fishes · doi:10.3390/fishes10090466
Ren, Y., Mulukutla, R., Mankoff, J. C., & Dey, A. K. (2025) · JMIR Formative Research · doi:10.2196/78657 · Cited by 2
“First, the scarcity of labeled data makes it difficult to build a model that generalizes well to unseen users [61,93,96,102-105], though identifying broad patterns and key features remains essential in imbalanced settings [106].”
Monitoring Parkinson’s Disease In-the-Wild
Mvomo, C. E., Bedime, J. S. N., Perfetto, S., Leibovich, D., Guedes, C., Potvin-Desrochers, A., Dixon, P. C., Easthope, C. A., & Paquette, C. (2025) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi67747.2025.11269453
“%) sets (regardless of data source or sensor type) using an inter-subject split scheme [39].”
Langener, A. M., Haddox, D., Mackin, D. M., Price, G. D., Lekkas, D., Collins, A. C., Griffin, T. Z., Heinz, M. V., Nemesure, M. D., Nepal, S. K., et al. (2026) · Addictive Behaviors · doi:10.1016/j.addbeh.2026.108624
“While these results are impressive, the study used a cross-validation strategy that split data by time points rather than by individuals, potentially inflating performance estimates and limiting the model’s generalizability to new users (Langener & Jacobson, 2025; Saeb et al., 2017).”
de Freitas, L. A., Ferreira, R. E. P., Alves, A. A. C., Dorea, J. R. R., Paz, C. C. P., & Rosa, G. J. M. (2026) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2026.106843
Gui, Y., Fan, Y., Liu, L., Cao, J., Zhang, L., Yu, Y., Adams, R. D., Han, J., & Lyu, J. (2026) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2026.1790452
“Although LOSO-CV and Bootstrap resampling were employed to ensure statistical reliability, the model’s generalization capability requires further verification in larger, multi-center clinical cohorts.”
Kim, J., & Jo, O. (2026) · Electronics · doi:10.3390/electronics15071390
Díaz, L. R. M., Kong, Y., Kundrát, J., & Posada-Quintero, H. F. (2026) · Applied Sciences · doi:10.3390/app16073240 · Cited by 3
Satyanshu, K., Gupta, D., Chandrasekar, N., Bokka, R., Abdulhussein, H. M., & Jwad, N. A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14186-6_20
Methodology for Design and Analysis of Machine Learning Competitions
Pavão, A. (2023) · theses.fr (ABES) · doi:10.70675/9234d2f0zebc5z4c1czaf3bzb0ad7a481e04 · Cited by 5
Metadata
- Authors
- Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Saeb, Sohrab
- Lonini, Luca
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-03-15
- Volume
- 6
- Issue
- 5
- Pages
- 1-9
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC5441397
- OpenAlex
- W2595557058
- Cited by
- 367
- References
- 36
- Citations by year
- 2026: 87 · 2025: 24 · 2024: 35 · 2023: 39 · 2022: 57 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 11 · 2017: 10
- FWCI
- 28.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (P20MH090318, R01MH100482)
- National Institute of Neurological Disorders and Stroke (R01NS063399)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- cross-validation, machine learning in healthcare, wearable sensors, accelerometer, smartphone health data, overfitting, clinical decision support
- MeSH terms
- Smartphone, Machine Learning, Wearable Electronic Devices, Algorithms, Humans, Reproducibility of Results, Exercise, Monitoring, Ambulatory, Accelerometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.