CoChat

The need to approximate the use-case in clinical machine learning

Open access

Published version of a preprint

The DOI matches its Crossref record; 36 of the 38 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/gigascience/gix019

Cites

36 of 38 references verified1 citation errors · 0 not checkable

Statistics

Figures

Figure 1: A visualization of subject-wise (A) and record-wise (B) CV for a diagnosis scenario. Data is split into training (blue) and test (yellow) sets to evaluate the performance of machine learning algorithms. Each box represents a record from each subject. While subject-wise CV only uses data from other subjects (green arrows), record-wise in addition uses data from the same test subject (red arrows) to predict its state. The problem is, that if the algorithm can (implicitly) detect the identity of the person based on the features, it can automatically also “diagnose” the disease.
Figure 1: A visualization of subject-wise (A) and record-wise (B) CV for a diagnosis scenario. Data is split into training (blue) and test (yellow) sets to evaluate the performance of machine learning algorithms. Each box represents a record from each subject. While subject-wise CV only uses data from other subjects (green arrows), record-wise in addition uses data from the same test subject (red arrows) to predict its state. The problem is, that if the algorithm can (implicitly) detect the identity of the person based on the features, it can automatically also “diagnose” the disease.CC BY · PubMed Central
Figure 2: Effect of subject-wise and record-wise CV on the classification error for the UCI activity recognition dataset. As the number of folds (x-axis) used to perform CV increases, the error tends to decrease. Similarly, performance improves when the number of subjects increases (symbols denote total number of subjects used to train and test the classifier). Record-wise CV significantly inflates the predicted accuracy (orange) as compared to subject-wise CV (green). Error bars indicate 95% CIs.
Figure 2: Effect of subject-wise and record-wise CV on the classification error for the UCI activity recognition dataset. As the number of folds (x-axis) used to perform CV increases, the error tends to decrease. Similarly, performance improves when the number of subjects increases (symbols denote total number of subjects used to train and test the classifier). Record-wise CV significantly inflates the predicted accuracy (orange) as compared to subject-wise CV (green). Error bars indicate 95% CIs.CC BY · PubMed Central
Figure 3: Classification error in a simulated dataset for record-wise and subject-wise CV methods, as a function of cross-subject (b) and within-subject (c) variability (A), and number of subjects (B). The number of features is set to 10. (A) Each column shows the classification error for both CV methods as a function of b and c for a given number of subjects (4, 12, and 32). Brighter colors indicate higher classification error values (black = 0; white = 1). (B) The mean and the standard deviation of classification error, across the values of b and c, for subject-wise and record-wise methods, as a function of number of subjects.
Figure 3: Classification error in a simulated dataset for record-wise and subject-wise CV methods, as a function of cross-subject (b) and within-subject (c) variability (A), and number of subjects (B). The number of features is set to 10. (A) Each column shows the classification error for both CV methods as a function of b and c for a given number of subjects (4, 12, and 32). Brighter colors indicate higher classification error values (black = 0; white = 1). (B) The mean and the standard deviation of classification error, across the values of b and c, for subject-wise and record-wise methods, as a function of number of subjects.CC BY · PubMed Central
Figure 4: PRISMA flowchart.
Figure 4: PRISMA flowchart.CC BY · PubMed Central
Figure 5: The difference between subject-wise and record-wise CV papers in (A) reported classification errors, and (B) number of citations. Each dot represents one paper, and boxes show the interquartile range. The horizontal lines inside the boxes indicate the medians.
Figure 5: The difference between subject-wise and record-wise CV papers in (A) reported classification errors, and (B) number of citations. Each dot represents one paper, and boxes show the interquartile range. The horizontal lines inside the boxes indicate the medians.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 38 references

Research works printed but not linked: 1

  1. Google Scholar. Google 2016 https://scholar.google.com (6 April 2017, date last accessed).

Cited by

Items 101–200 of 357

Metadata

Authors
Saeb, S., Lonini, L., Jayaraman, A., Mohr, D. C., & Kording, K. P.
Affiliations
Northwestern University (US)
Corresponding author
  • Saeb, Sohrab
  • Lonini, Luca
Venue
GigaScience
Published
2017-03-15
Volume
6
Issue
5
Pages
1-9
ISSN
2047-217X (ISSN-L)
OpenAlex
W2595557058
Cited by
367
References
36
Citations by year
2026: 87 · 2025: 24 · 2024: 35 · 2023: 39 · 2022: 57 · 2021: 48 · 2020: 35 · 2019: 21 · 2018: 11 · 2017: 10
FWCI
28.72
Type
journal article
Language
en
Open access
Locations
  • GigaScience (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Institute of Mental Health (P20MH090318, R01MH100482)
  • National Institute of Neurological Disorders and Stroke (R01NS063399)
Topics
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
cross-validation, machine learning in healthcare, wearable sensors, accelerometer, smartphone health data, overfitting, clinical decision support
MeSH terms
Smartphone, Machine Learning, Wearable Electronic Devices, Algorithms, Humans, Reproducibility of Results, Exercise, Monitoring, Ambulatory, Accelerometry

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.