The need to approximate the use-case in clinical machine learning
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, data leakage due to identity confounding is not limited to deep neural networks, and has been uncovered using random forests (Saeb et al., 2017; Chaibub Neto et al., 2019; Tougui et al., 2021) and support vector machines (Tougui et al., 2021).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“For example, if a study trains a classifier to identify individual subjects with a disease, then there may be stronger bias when the study involves fewer participants (Saeb et al., 2017).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
The cited source says what the claim attributes to it.
In the citing paper
“This conclusion mirrors results from studies examining a variety of other types of data and classifier models (Saeb et al., 2017; Wen et al., 2018, 2020; Chaibub Neto et al., 2019; Tougui et al., 2021; Tampu et al., 2022).”
From Data leakage in deep learning studies of translational EEG (2024) · cites this paper as Saeb et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 2×
“The number of trees for the random forest classifier was set to 50, which was optimized based on the out of bag error [22].”
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“Machine learning has grown massively over the past decades, with countless applications in technology, marketing, and science [1].”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Inappropriatevalidationprocedures,ontheotherhand,canlead to unreliable results, which might contribute to the problem of irreproducibility of research findings [31, 32] and thereby undermine the trust in both medicine and data science.”
A survey of cross-validation procedures for model selection
Arlot, S., & Célisse, A. (2010) · LillOA (Université de Lille (University Of Lille)) · Cited by 3,669
A survey of cross-validation procedures for model selection
Arlot, S., & Celisse, A. (2010) · Statistics Surveys · doi:10.1214/09-ss054 · Cited by 3,522 · cited 1×
“Splitting data into training and test subsets can be done using various methods, such as leave-one-out, leave-p-out, k-fold, and Monte-Carlo sampling [18].”
The Rise of Consumer Health Wearables: Promises and Barriers
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001953 · Cited by 1,235 · cited 1×
“A driving factor is that people carry these devices with them most of the time, and thus a growing amount of daily life data, such as physical activities [5, 6], is becoming available.”
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906 · cited 1×
“In such scenarios, complementary approaches such as active learning or interactive machine learning [26, 27] could be beneficial.”
Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines
Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.038 · Cited by 792 · cited 1×
“For CV to be valid, training and test sets need to be independent [30].”
Saeb, S., Zhang, M., Karr, C. J., Schueller, S. M., Corden, M. E., Kording, K. P., & Mohr, D. C. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4273 · Cited by 782 · cited 1×
“Examples include detecting cardiovascular diseases [7], falls [8], measuring rehabilitation outcomes in stroke and amputees [9–11], monitoring Parkinson’s disease (PD) symptoms [12–14], and detecting depression [15, 16].”
Self-citationRotation-invariant convolutional neural networks for galaxy morphology prediction
Dieleman, S., Willett, K. W., & Dambre, J. (2015) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stv632 · Cited by 781 · cited 1×
“Social media and e-commerce websites filter contents and recommend products based on the user’s interests, and scientific data, from astrophysics [2] to neurophysiology [3] and medicine [4], are analyzed using machine learning algorithms.”
Transition-Aware Human Activity Recognition Using Smartphones
Reyes-Ortiz, J.-L., Oneto, L., Samà, A., Parra, X., & Anguita, D. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.07.085 · Cited by 740 · cited 1×
“We chose a publicly available human activity recognition dataset [20, 21], which contained recordings of 30 subjects performing 6 activities: sitting, standing, walking, stair climbing up/down, and laying down.”
Deriving reproducible biomarkers from multi-site resting-state data: An Autism-based example
Abraham, A., Milham, M. P., Di Martino, A., Craddock, R. C., Samaras, D., Thirion, B., & Varoquaux, G. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.045 · Cited by 701
Wearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom
Bachlin, M., Plotnik, M., Roggen, D., Maidan, I., Hausdorff, J. M., Giladi, N., & Troster, G. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036165 · Cited by 693
Monitoring Motor Fluctuations in Patients With Parkinson's Disease Using Wearable Sensors
Patel, S., Lorincz, K., Hughes, R., Huggins, N., Growdon, J., Standaert, D., Akay, M., Dy, J., Welsh, M., & Bonato, P. (2009) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2033471 · Cited by 623 · cited 1×
“Examples include detecting cardiovascular diseases [7], falls [8], measuring rehabilitation outcomes in stroke and amputees [9–11], monitoring Parkinson’s disease (PD) symptoms [12–14], and detecting depression [15, 16].”
Canzian, L., & Musolesi, M. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2805845 · Cited by 477 · cited 1×
“Examples include detecting cardiovascular diseases [7], falls [8], measuring rehabilitation outcomes in stroke and amputees [9–11], monitoring Parkinson’s disease (PD) symptoms [12–14], and detecting depression [15, 16].”
Oresko, J. J., Jin, Z., Cheng, J., Huang, S., Sun, Y., Duschl, H., & Cheng, A. C. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2010.2047865 · Cited by 446
Detecting and monitoring the symptoms of Parkinson's disease using smartphones: A pilot study
Arora, S., Venkataraman, V., Zhan, A., Donohue, S., Biglan, K. M., Dorsey, E. R., & Little, M. A. (2015) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2015.02.026 · Cited by 387
Mining large-scale smartphone data for personality studies
Chittaranjan, G., Blom, J., & Gatica-Perez, D. (2013) · Personal and Ubiquitous Computing · doi:10.1007/s00779-011-0490-1 · Cited by 381
High-resolution CMOS MEA platform to study neurons at subcellular, cellular, and network levels
Müller, J., Ballini, M., Livi, P., Chen, Y., Radivojevic, M., Shadmani, A., Viswam, V., Jones, I. L., Fiscella, M., Diggelmann, R., et al. (2015) · Lab on a Chip · doi:10.1039/c5lc00133a · Cited by 326 · cited 1×
“, [28, 29]) requires the use of techniques that can analyze such large amounts of highdimensional data, and in those applications, machine learning is replacing traditional data analysis.”
Fall Classification by Machine Learning Using Mobile Phones
Albert, M. V., Kording, K., Herrmann, M., & Jayaraman, A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0036556 · Cited by 207 · cited 1×
“Examples include detecting cardiovascular diseases [7], falls [8], measuring rehabilitation outcomes in stroke and amputees [9–11], monitoring Parkinson’s disease (PD) symptoms [12–14], and detecting depression [15, 16].”
Self-citationA Novel Approach to Monitor Rehabilitation Outcomes in Stroke Survivors Using Wearable Technology
Patel, S., Hughes, R., Hester, T., Stein, J., Akay, M., Dy, J. G., & Bonato, P. (2010) · Proceedings of the IEEE · doi:10.1109/jproc.2009.2038727 · Cited by 170
Using and understanding cross-validation strategies. Perspectives on Saeb et al.
Little, M. A., Varoquaux, G., Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P. (2017) · GigaScience · doi:10.1093/gigascience/gix020 · Cited by 158
Self-citationWearable motion sensors to continuously measure real-world physical activities
Dobkin, B. H. (2013) · Current Opinion in Neurology · doi:10.1097/wco.0000000000000026 · Cited by 142 · cited 1×
“A driving factor is that people carry these devices with them most of the time, and thus a growing amount of daily life data, such as physical activities [5, 6], is becoming available.”
Dobkin, B. H., Xu, X., Batalin, M., Thomas, S., & Kaiser, W. (2011) · Stroke · doi:10.1161/strokeaha.110.611095 · Cited by 129
Holzinger, A., Plass, M., Holzinger, K., Crişan, G. C., Pintea, C.-M., & Palade, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-45507-5_6 · Cited by 97
Donsa, K., Spat, S., Beck, P., Pieber, T. R., & Holzinger, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16226-3_10 · Cited by 60 · cited 1×
“Social media and e-commerce websites filter contents and recommend products based on the user’s interests, and scientific data, from astrophysics [2] to neurophysiology [3] and medicine [4], are analyzed using machine learning algorithms.”
Automatic discovery of cell types and microcircuitry from neural connectomics
Jonas, E., & Kording, K. (2015) · eLife · doi:10.7554/elife.04250 · Cited by 51 · cited 1×
“Social media and e-commerce websites filter contents and recommend products based on the user’s interests, and scientific data, from astrophysics [2] to neurophysiology [3] and medicine [4], are analyzed using machine learning algorithms.”
Self-citationCanzian, L., & Musolesi, M. (2015) · Cited by 39
Interactive Machine Learning (iML)
Holzinger, A. (2015) · Informatik-Spektrum · doi:10.1007/s00287-015-0941-6 · Cited by 36
Opportunities for Smartphones in Clinical Care: The Future of Mobile Mood Monitoring: (Commentary)
Sandstrom, G. M., Lathia, N., Mascolo, C., & Rentfrow, P. J. (2016) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.15com10054 · Cited by 33 · cited 1×
“The use of machine learning for clinical predictions is growing in popularity [25], although significant challenges lie ahead.”
Albert, M. V., Deeny, S., McCarthy, C., Valentin, J., & Jayaraman, A. (2014) · PM&R · doi:10.1016/j.pmrj.2014.06.006 · Cited by 30 · cited 1×
“Examples include detecting cardiovascular diseases [7], falls [8], measuring rehabilitation outcomes in stroke and amputees [9–11], monitoring Parkinson’s disease (PD) symptoms [12–14], and detecting depression [15, 16].”
Self-citationMaking Activity Recognition Robust against Deceptive Behavior
Saeb, S., Kording, K. P., & Mohr, D. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144795 · Cited by 12 · cited 1×
“We also found random forests to perform well in our previous activity recognition study [23].”
Self-citationDeriving robust biomarkers from multi-site resting-state data: An Autism-based examplePreprint
Abraham, A., Milham, M. P., Martino, A. D., Craddock, R. C., Samaras, D., Thirion, B., & Varoquaux, G. P. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/075853 · Cited by 7 · cited 1×
“In more complex scenarios, we may want to generalize from one clinical site to another, in which case CV needs to be across these sites [19].”
Supporting data for "The need to approximate the use-case in clinical machine learning"
Saeb, S., Lonini, L., Jayaraman, A., David, C. M., & Konrad, P. K. (2017) · GigaScience Database · doi:10.5524/100283
Self-citation
Research works printed but not linked: 1
- Google Scholar. Google 2016 https://scholar.google.com (6 April 2017, date last accessed).
Cited by
Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning
Mohr, D. C., Zhang, M., & Schueller, S. M. (2017) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-044949 · Cited by 802
Self-citationClassification of Brain Tumors from MRI Images Using a Convolutional Neural Network
Badža, M. M., & Barjaktarović, M. Č. (2020) · Applied Sciences · doi:10.3390/app10061999 · Cited by 686
“Figure 3 shows confusion matrices for the record-wise 10 -fold cross-validation approach for testing data obtained from the original dataset.”
MethodMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
“Another example of leakage arises when repeated measures of an individual are split across train and test set, the algorithm then learning to recognize the individual patient rather than markers of a condition [Saeb et al., 2017].”
Thieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes
Mekruksavanich, S., & Jitpattanakul, A. (2021) · Sensors · doi:10.3390/s21051636 · Cited by 335
“Various approaches, such as leave-one-out, leave-p-out, and k-fold cross validation, can be used to separate the data for training and testing [49,50].”
BackgroundMethod
Metadata
- Authors
- Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P.
- Affiliations
- Northwestern University (US)
- Corresponding author
- Saeb, Sohrab
- Lonini, Luca
- Venue
- GigaScience
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-03-15
- Volume
- 6
- Issue
- 5
- Pages
- 1-9
- ISSN
- 2047-217X (ISSN-L)
- PMC
- PMC5441397
- OpenAlex
- W2595557058
- Cited by
- 367
- References
- 36
- Citations by year
- 2026: 87 · 2025: 24 · 2024: 35 · 2023: 39 · 2022: 57 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 11 · 2017: 10
- FWCI
- 28.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- GigaScience (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (P20MH090318, R01MH100482)
- National Institute of Neurological Disorders and Stroke (R01NS063399)
- Topics
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- cross-validation, machine learning in healthcare, wearable sensors, accelerometer, smartphone health data, overfitting, clinical decision support
- MeSH terms
- Smartphone, Machine Learning, Wearable Electronic Devices, Algorithms, Humans, Reproducibility of Results, Exercise, Monitoring, Ambulatory, Accelerometry
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.