A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Sleep-tracking devices, particularly within the consumer sleep technology (CST) space, are increasingly used in both research and clinical settings, providing new opportunities for large-scale data collection in highly ecological conditions. Due to the fast pace of the CST industry combined with the lack of a standardized framework to evaluate the performance of sleep trackers, their accuracy and reliability in measuring sleep remains largely unknown. Here, we provide a step-by-step analytical framework for evaluating the performance of sleep trackers (including standard actigraphy), as compared with gold-standard polysomnography (PSG) or other reference methods. The analytical guidelines are based on recent recommendations for evaluating and using CST from our group and others (de Zambotti and colleagues; Depner and colleagues), and include raw data organization as well as critical analytical procedures, including discrepancy analysis, Bland–Altman plots, and epoch-by-epoch analysis. Analytical steps are accompanied by open-source R functions (depicted at https://sri-human-sleep.github.io/sleep-trackers-performance/AnalyticalPipeline_v1.0.0.html). In addition, an empirical sample dataset is used to describe and discuss the main outcomes of the proposed pipeline. The guidelines and the accompanying functions are aimed at standardizing the testing of CSTs performance, to not only increase the replicability of validation studies, but also to provide ready-to-use tools to researchers and clinicians. All in all, this work can help to increase the efficiency, interpretation, and quality of validation studies, and to improve the informed adoption of CST in research and clinical settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Measuring agreement in method comparison studies
Bland, J. M., & Altman, D. G. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800204 · Cited by 9,619 · cited 10×
“Alternatively, when log transformation does not remove heteroscedasticity, the SD of the differences can be expressed as a function of SM, using the coefficients estimated with equation (3) [16].”
BackgroundMethodInfluentialMeasuring agreement in method comparison studies
Bland, J. M., & Altman, D. G. (1999) · Statistical Methods in Medical Research · doi:10.1191/096228099673819272 · Cited by 9,619
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754
Measurement in Medicine: The Analysis of Method Comparison Studies
Altman, D. G., & Bland, J. M. (1983) · The Statistician · doi:10.2307/2987937 · Cited by 4,300
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · cited 1×
“In sleep detection, sleep epochs are usually more prevalent than wake epochs, and this would result in cases of “high agreement but low kappa” [36].”
BackgroundByrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179 · cited 2×
“Individual-level EBE metrics Table 3 shows an overview of the most widely reported EBE metrics used to evaluate the performance of CST devices (see also [32, 33]), which can be computed by applying the functions included in our pipeline [11] (sections 3.”
MethodMarino, M., Li, Y., Rueschman, M. N., Winkelman, J. W., Ellenbogen, J. M., Solet, J. M., Dulin, H., Berkman, L. F., & Buxton, O. M. (2013) · Sleep · doi:10.5665/sleep.3142 · Cited by 1,056
Wearable Sleep Technology in Clinical and Research Settings
DE ZAMBOTTI, M., CELLINI, N., GOLDSTONE, A., COLRAIN, I. M., & BAKER, F. C. (2019) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000001947 · Cited by 564 · cited 10×
“Whereas in the original Bland–Altman plot SM is represented by the mean of the two measurements [22], referencederived measures (and particularly PSG measures) have been recommended [1, 5] and frequently adopted [30, 31] to represent SM (i.”
BackgroundMethodResultInfluentialSelf-citationAccuracy of Wristband Fitbit Models in Assessing Sleep: Systematic Review and Meta-Analysis
Haghayegh, S., Khoshnevis, S., Smolensky, M. H., Diller, K. R., & Castriotta, R. J. (2019) · Journal of Medical Internet Research · doi:10.2196/16273 · Cited by 446 · cited 1×
“For instance, Haghayegh and colleagues [9] excluded 50 of the 72 identified articles from their review on the accuracy of Fitbit devices also due to ineligible or insufficiently described method or outcomes, and they reported inconsistencies across multiple aspects such as recording setting, method of reference, and s…”
BackgroundBaron, K. G., Duffecy, J., Berendsen, M. A., Cheung Mason, I., Lattie, E. G., & Manalo, N. C. (2018) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2017.12.002 · Cited by 186 · cited 2×
“A similar degree of heterogeneity has been reported by other reviews of CST validation studies [1, 3], including those evaluating the performance of clinical-grade actigraphy [10].”
BackgroundResultHaghayegh, S., Kang, H.-A., Khoshnevis, S., Smolensky, M. H., & Diller, K. R. (2020) · Physiological Measurement · doi:10.1088/1361-6579/ab86d6 · Cited by 152
Cited by
Performance of seven consumer sleep-tracking devices compared with polysomnography
Chinoy, E. D., Cuellar, J. A., Huwa, K. E., Jameson, J. T., Watson, C. H., Bessman, S. C., Hirsch, D. A., Cooper, A. D., Drummond, S. P. A., & Markwald, R. R. (2021) · Sleep · doi:10.1093/sleep/zsaa291 · Cited by 433
An open-source, high-performance tool for automated sleep staging
Vallat, R., & Walker, M. P. (2021) · eLife · doi:10.7554/elife.70092 · Cited by 363
Miller, D. J., Sargent, C., & Roach, G. D. (2022) · Sensors · doi:10.3390/s22166317 · Cited by 328
Ghorbani, S., Golkashani, H. A., Chee, N. I., Teo, T. B., Dicom, A. R., Yilmaz, G., Leong, R. L., Ong, J. L., & Chee, M. W. (2022) · Nature and Science of Sleep · doi:10.2147/nss.s359789 · Cited by 36
“Discrepancy Analyses To visualise discrepancies between Gen2, Gen3 and PSG measurements, Bland–Altman plots were generated for each night based on a standardized framework.(32) A negative bias represents underestimation by the device compared to PSG.”
MethodIto-Masui, A., Sakamoto, R., Matsuo, E., Kawamoto, E., Motomura, E., Tanii, H., Yu, H., Sano, A., Imai, H., & Shimaoka, M. (2023) · Journal of Medical Internet Research · doi:10.2196/45834 · Cited by 20
Metadata
- Authors
- Menghini, L., Cellini, N., Goldstone, A., Baker, F. C., & de Zambotti, M.
- Venue
- Sleep
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-10-07
- Volume
- 44
- Issue
- 2
- ISSN
- 0161-8105 (ISSN-L)
- 1550-9109
- OpenAlex
- W3083632288
- Cited by
- 141
- References
- 13
- FWCI
- 16.89
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.