CoChat
Sign Up
Cited by 240Non-Invasive Vital Sign MonitoringContext-Aware Activity Recognition SystemsSleep and related disorders

The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.

Sleep stage prediction with raw acceleration and photoplethysmography heart rate data derived from a consumer wearable device

Open access
Walch, Olivia, Huang, Yitong, Forger, Daniel, Goldstein, Cathy·2019·Sleep
DOI 10.1093/sleep/zsz180PMC PMC6930135
Open this paper in CoChat — free accountOpen PDF

Abstract

Wearable, multisensor, consumer devices that estimate sleep are now commonplace, but the algorithms used by these devices to score sleep are not open source, and the raw sensor data is rarely accessible for external use. As a result, these devices are limited in their usefulness for clinical and research applications, despite holding much promise. We used a mobile application of our own creation to collect raw acceleration data and heart rate from the Apple Watch worn by participants undergoing polysomnography, as well as during the ambulatory period preceding in lab testing. Using this data, we compared the contributions of multiple features (motion, local standard deviation in heart rate, and “clock proxy”) to performance across several classifiers. Best performance was achieved using neural nets, though the differences across classifiers were generally small. For sleep-wake classification, our method scored 90% of epochs correctly, with 59.6% of true wake epochs (specificity) and 93% of true sleep epochs (sensitivity) scored correctly. Accuracy for differentiating wake, NREM sleep, and REM sleep was approximately 72% when all features were used. We generalized our results by testing the models trained on Apple Watch data using data from the Multi-ethnic Study of Atherosclerosis (MESA), and found that we were able to predict sleep with performance comparable to testing on our own dataset. This study demonstrates, for the first time, the ability to analyze raw acceleration and heart rate data from a ubiquitous wearable device with accepted, disclosed mathematical methods to improve accuracy of sleep and sleep stage prediction.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/sleep/zsz180

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Sample data from one subject’s night of sleep. From top: Motion, raw acceleration from the Apple Watch microelectromechanical system accelerometer (in x, y, and z directions); Counts, acceleration processed as activity counts using code from [34]; Heart rate, heart rate from Apple Watch photoplethysmography; “Clock proxy,” predicted from ambulatory recording with Apple Watch; Stages, hypnogram from scored polysomnography. Insets on the right show zoomed-in version of the data on the left, with selected windows marked with gray overlays.
Figure 1. Sample data from one subject’s night of sleep. From top: Motion, raw acceleration from the Apple Watch microelectromechanical system accelerometer (in x, y, and z directions); Counts, acceleration processed as activity counts using code from [34]; Heart rate, heart rate from Apple Watch photoplethysmography; “Clock proxy,” predicted from ambulatory recording with Apple Watch; Stages, hypnogram from scored polysomnography. Insets on the right show zoomed-in version of the data on the left, with selected windows marked with gray overlays.CC BY-NC · PubMed Central
Figure 2. ROC curves across multiple classifiers and features for differentiating sleep and wake. The x-axis represents the fraction of true wake epochs incorrectly classified as sleep and the y-axis represents the fraction of true sleep epochs correctly classified as sleep. ROC curves are generated by applying the full range of possible thresholds to the class probabilities assigned to each epoch by the classifiers.
Figure 2. ROC curves across multiple classifiers and features for differentiating sleep and wake. The x-axis represents the fraction of true wake epochs incorrectly classified as sleep and the y-axis represents the fraction of true sleep epochs correctly classified as sleep. ROC curves are generated by applying the full range of possible thresholds to the class probabilities assigned to each epoch by the classifiers.CC BY-NC · PubMed Central
Figure 3. Precision-recall curves across multiple classifiers and features for differentiating sleep and wake. The x-axis represents the fraction of true wake epochs correctly classified as wake and the y-axis represents the fraction of all epochs labeled as wake by the classifier that were correct.
Figure 3. Precision-recall curves across multiple classifiers and features for differentiating sleep and wake. The x-axis represents the fraction of true wake epochs correctly classified as wake and the y-axis represents the fraction of all epochs labeled as wake by the classifier that were correct.CC BY-NC · PubMed Central
Figure 4. ROC curves across multiple classifiers and features for classifying wake/NREM/REM. Each point on the x-axis represents the fraction of wake epochs classified incorrectly, with any wake epoch classified either as NREM or REM sleep counting as a false positive. The y-axis summarizes REM and NREM accuracy rates. To choose a threshold for distinguishing REM and NREM sleep, a binary search was performed to find the value that minimized the difference between REM accuracy and NREM accuracy. In the case of the motion-only feature set, the dashed and dotted lines correspond to the REM and NREM accuracies (respectively), and the solid line is their average. For all others, the solid line represents the average of the REM and NREM accuracies, which could be made nearly identical through the choice of the appropriate threshold. NREM, non-rapid eye movement; REM, rapid eye movement.
Figure 4. ROC curves across multiple classifiers and features for classifying wake/NREM/REM. Each point on the x-axis represents the fraction of wake epochs classified incorrectly, with any wake epoch classified either as NREM or REM sleep counting as a false positive. The y-axis summarizes REM and NREM accuracy rates. To choose a threshold for distinguishing REM and NREM sleep, a binary search was performed to find the value that minimized the difference between REM accuracy and NREM accuracy. In the case of the motion-only feature set, the dashed and dotted lines correspond to the REM and NREM accuracies (respectively), and the solid line is their average. For all others, the solid line represents the average of the REM and NREM accuracies, which could be made nearly identical through the choice of the appropriate threshold. NREM, non-rapid eye movement; REM, rapid eye movement.CC BY-NC · PubMed Central
Figure 5. Bland-Altman plots for TST (minutes), SOL (minutes), WASO (minutes), SE (fraction), stage REM sleep (minutes), and NREM sleep (minutes) as predicted by the neural net classifier. The differences in classifier-produced values versus PSG values are plotted on the y-axis and the corresponding ground truth PSG values are plotted on the x-axis. Sleep metrics were computed using the same fixed thresholds for wake (θW=0.3)and REM (θREM=0.35) for all subjects.
Figure 5. Bland-Altman plots for TST (minutes), SOL (minutes), WASO (minutes), SE (fraction), stage REM sleep (minutes), and NREM sleep (minutes) as predicted by the neural net classifier. The differences in classifier-produced values versus PSG values are plotted on the y-axis and the corresponding ground truth PSG values are plotted on the x-axis. Sleep metrics were computed using the same fixed thresholds for wake (θW=0.3)and REM (θREM=0.35) for all subjects.CC BY-NC · PubMed Central
Figure 6. Histograms of performance when training on all subjects but one, and testing on the omitted subject using the neural net classifier. For all tested subjects’ sleep nights, an epoch was counted as wake if its probability exceeded 0.3 (θW=0.3).
Figure 6. Histograms of performance when training on all subjects but one, and testing on the omitted subject using the neural net classifier. For all tested subjects’ sleep nights, an epoch was counted as wake if its probability exceeded 0.3 (θW=0.3).CC BY-NC · PubMed Central
Figure 7. Histograms of performance when training on all subjects and testing on one subject who was omitted from training. The two performance measures plotted are accuracy and specificity. Here, specificity refers to the fraction of wake epochs scored correctly as wake. Each row corresponds to a fixed “true positive rate” (TPR), referring to the fraction of sleep epochs scored correctly as sleep. For each row, the threshold separating sleep and wake was chosen to match the fixed true positive rate. As the required true positive rate increases, the likelihood of a wake epoch being scored as sleep increases; hence, there is a skew towards lower values in the specificity histograms as TPR increases.
Figure 7. Histograms of performance when training on all subjects and testing on one subject who was omitted from training. The two performance measures plotted are accuracy and specificity. Here, specificity refers to the fraction of wake epochs scored correctly as wake. Each row corresponds to a fixed “true positive rate” (TPR), referring to the fraction of sleep epochs scored correctly as sleep. For each row, the threshold separating sleep and wake was chosen to match the fixed true positive rate. As the required true positive rate increases, the likelihood of a wake epoch being scored as sleep increases; hence, there is a skew towards lower values in the specificity histograms as TPR increases.CC BY-NC · PubMed Central
Figure 8. MESA dataset (n = 188) validation of neural net models trained on the Apple Watch-PSG dataset. Left) Sleep/wake differentiation performance in MESA dataset Right) Wake/NREM/REM classifier performance in MESA dataset. The dashed lines represent REM accuracy for the motion-only feature set; the dotted lines represent NREM accuracy for the motion-only feature set. For all, solid lines represent the average of the NREM and REM accuracies, chosen to be as close to equal as possible via threshold selection. NREM, non-rapid eye movement; REM, rapid eye movement.
Figure 8. MESA dataset (n = 188) validation of neural net models trained on the Apple Watch-PSG dataset. Left) Sleep/wake differentiation performance in MESA dataset Right) Wake/NREM/REM classifier performance in MESA dataset. The dashed lines represent REM accuracy for the motion-only feature set; the dotted lines represent NREM accuracy for the motion-only feature set. For all, solid lines represent the average of the NREM and REM accuracies, chosen to be as close to equal as possible via threshold selection. NREM, non-rapid eye movement; REM, rapid eye movement.CC BY-NC · PubMed Central
Figure 9. Quantifying sleep metrics performance in the MESA dataset (n = 188) using a model trained on the Apple Watch-PSG dataset. Bland-Altman plots for TST (minutes), SOL (minutes), WASO (minutes), SE (fraction), stage REM sleep (minutes), and NREM sleep (minutes) as classified by a neural net classifier. The differences in classifier values versus PSG values are plotted on the y-axis (actual - predicted) and the corresponding ground truth PSG values are plotted on the x-axis. Sleep metrics were computed using the same fixed thresholds for wake (θW=0.3) and REM (θREM=0.35) for all subjects.
Figure 9. Quantifying sleep metrics performance in the MESA dataset (n = 188) using a model trained on the Apple Watch-PSG dataset. Bland-Altman plots for TST (minutes), SOL (minutes), WASO (minutes), SE (fraction), stage REM sleep (minutes), and NREM sleep (minutes) as classified by a neural net classifier. The differences in classifier values versus PSG values are plotted on the y-axis (actual - predicted) and the corresponding ground truth PSG values are plotted on the x-axis. Sleep metrics were computed using the same fixed thresholds for wake (θW=0.3) and REM (θREM=0.35) for all subjects.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 56 references

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Walch, O., Huang, Y., Forger, D., & Goldstein, C.
Affiliations
  • University of Michigan (US)
  • Dartmouth College (US)
Corresponding author
Walch, Olivia
ORCID
Venue
Sleep
Publisher
Oxford University Press (OUP)
Published
2019-08-13
Volume
42
Issue
12
ISSN
  • 0161-8105 (ISSN-L)
  • 1550-9109
DOI
10.1093/sleep/zsz180
PMC
PMC6930135
OpenAlex
W2937191485
Cited by
240
References
56
Citations by year
2026: 20 · 2025: 42 · 2024: 33 · 2023: 38 · 2022: 39 · 2021: 40 · 2020: 28
FWCI
9.43
Type
journal article
Language
en
Open access
Licence
Locations
  • SLEEP (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Smith ScholarWorks (Smith College) (submittedVersion, cc-by-nc)
Funders
Exercise & Sport Science Initiative University of Michigan (U056400, U049702, NSF DMS 1714094)
Topics
  • Physical Sciences › Engineering › Biomedical Engineering › Non-Invasive Vital Sign Monitoring
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Sleep and related disorders
Keywords
wearable devices, Apple Watch, accelerometry, photoplethysmography, heart rate, sleep-wake classification, NREM sleep, REM sleep, polysomnography, artificial neural networks, classifier comparison, MESA, cross-dataset generalization
MeSH terms
Acceleration, Wearable Electronic Devices, Adult, Algorithms, Data Interpretation, Statistical, Electroencephalography, Female, Forecasting, Heart Rate, Humans, Male, Middle Aged, Sleep Stages, Photoplethysmography, Polysomnography, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review