Validation of Zulu Watch against Polysomnography and Actigraphy for On-Wrist Sleep-Wake Determination and Sleep-Depth Estimation
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Traditional measures of sleep or commercial wearables may not be ideal for use in operational environments. The Zulu watch is a commercial sleep-tracking device designed to collect longitudinal sleep data in real-world environments. Laboratory testing is the initial step towards validating a device for real-world sleep evaluation; therefore, the Zulu watch was tested against the gold-standard polysomnography (PSG) and actigraphy. Eight healthy, young adult participants wore a Zulu watch and Actiwatch simultaneously over a 3-day laboratory PSG sleep study. The accuracy, sensitivity, and specificity of epoch-by-epoch data were tested against PSG and actigraphy. Sleep summary statistics were compared using paired samples t-tests, intraclass correlation coefficients, and Bland–Altman plots. Compared with either PSG or actigraphy, both the accuracy and sensitivity for Zulu watch sleep-wake determination were >90%, while the specificity was low (~26% vs. PSG, ~33% vs. actigraphy). The accuracy for sleep scoring vs. PSG was ~87% for interrupted sleep, ~52% for light sleep, and ~49% for deep sleep. The Zulu watch showed mixed results but performed well in determining total sleep time, sleep efficiency, sleep onset, and final awakening in healthy adults compared with PSG or actigraphy. The next step will be to test the Zulu watch’s ability to evaluate sleep in industrial operations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“sleep assessment algorithms are frequently proprietary and validated in small populations, making their measurement validity unclear5–8”
From Self-supervised learning of accelerometer data provides new insights for sleep and its association with mortality (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT
Martin Bland, J., & Altman, D. (1986) · The Lancet · doi:10.1016/s0140-6736(86)90837-8 · Cited by 47,965
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709 · cited 2×
“Based on the guidelines set forth by Koo and Li [26], ICCs for time of sleep onset and time of final awakening were good (0.”
MethodSleep duration and all-cause mortality: a systematic review and meta-analysis of prospective studies
Cappuccio, F. P., D'Elia, L., Strazzullo, P., & Miller, M. A. (2010) · Sleep · doi:10.1093/sleep/33.5.585 · Cited by 2,224
The Rise of Consumer Health Wearables: Promises and Barriers
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001953 · Cited by 1,235 · cited 1×
“This necessity restricts the ability of devices to log sleep episodes in the absence of wireless connectivity to download data, and it can generate concerns about privacy among researchers and consumers alike [23,24].”
BackgroundMarino, M., Li, Y., Rueschman, M. N., Winkelman, J. W., Ellenbogen, J. M., Solet, J. M., Dulin, H., Berkman, L. F., & Buxton, O. M. (2013) · Sleep · doi:10.5665/sleep.3142 · Cited by 1,056 · cited 1×
“Low specificity is an issue across research actigraphy devices and scoring algorithms [30,31], with longer epoch lengths relating to lower specificity [9].”
BackgroundNormal Human Sleep: An Overview
Carskadon, M. A., & Dement, W. C. (2005) · Principles and Practice of Sleep Medicine · doi:10.1016/b0-72-160797-7/50009-4 · Cited by 995 · cited 1×
“Starting with NREM, the sleep stages cycle throughout the night, each playing a critical role in the maintenance of the brain and body to promote optimal health and functioning [7].”
BackgroundLittner, M., Kushida, C. A., Anderson, W. M., Bailey, D., Berry, R. B., Davila, D. G., Hirshkowitz, M., Kapen, S., Kramer, M., Loube, D., et al. (2003) · Sleep · doi:10.1093/sleep/26.3.337 · Cited by 691
Wearable Sleep Technology in Clinical and Research Settings
DE ZAMBOTTI, M., CELLINI, N., GOLDSTONE, A., COLRAIN, I. M., & BAKER, F. C. (2019) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0000000000001947 · Cited by 564
The SBSM Guide to Actigraphy Monitoring: Clinical and Research Applications
Ancoli-Israel, S., Martin, J. L., Blackwell, T., Buenaver, L., Liu, L., Meltzer, L. J., Sadeh, A., Spira, A. P., & Taylor, D. J. (2015) · Behavioral Sleep Medicine · doi:10.1080/15402002.2015.1046356 · Cited by 559 · cited 2×
“Low specificity is an issue across research actigraphy devices and scoring algorithms [30,31], with longer epoch lengths relating to lower specificity [9].”
BackgroundMethodPeake, J. M., Kerr, G., & Sullivan, J. P. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00743 · Cited by 523 · cited 1×
“However, while the devices collect accelerometry and other biometric data, sleep scoring is done using algorithms on a companion mobile app or software platform [14,22].”
MethodSmith, M. T., McCrae, C. S., Cheung, J., Martin, J. L., Harrod, C. G., Heald, J. L., & Carden, K. A. (2018) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.7230 · Cited by 483
Performance of seven consumer sleep-tracking devices compared with polysomnography
Chinoy, E. D., Cuellar, J. A., Huwa, K. E., Jameson, J. T., Watson, C. H., Bessman, S. C., Hirsch, D. A., Cooper, A. D., Drummond, S. P. A., & Markwald, R. R. (2021) · Sleep · doi:10.1093/sleep/zsaa291 · Cited by 433 · cited 2×
“for additional study participant screening details [18].”
Self-citationFatigue risk management in the workplace
Lerman, S. E., Eskin, E., Flower, D. J., George, E. C., Gerson, B., Hartenbaum, N., Hursh, S. R., & Moore-Ede, M. (2012) · Journal of Occupational & Environmental Medicine · doi:10.1097/jom.0b013e318247a3b0 · Cited by 398
Self-citationWake Detection Capacity of Actigraphy During Sleep
Paquet, J., Kawinska, A., & Carrier, J. (2007) · Sleep · doi:10.1093/sleep/30.10.1362 · Cited by 352 · cited 2×
“Epochs longer than 60 s are not recommended when using Actiwatch devices and likely affect the specificity of the device [12,31,32].”
BackgroundMantua, J., Gravel, N., & Spencer, R. (2016) · Sensors · doi:10.3390/s16050646 · Cited by 314
Konjarski, M., Murray, G., Lee, V. V., & Jackson, M. L. (2018) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2018.05.005 · Cited by 259
Depner, C. M., Cheng, P. C., Devine, J. K., Khosla, S., de Zambotti, M., Robillard, R., Vakulin, A., & Drummond, S. P. A. (2020) · Sleep · doi:10.1093/sleep/zsz254 · Cited by 245 · cited 3×
“While the Zulu watch is intended for longitudinal measurement of sleep in real-world environments, laboratory testing against the gold standards under controlled conditions is an important initial step towards establishing the performance of a sleep-tracking device [5].”
BackgroundMethodSelf-citationConsumer Sleep Technology: An American Academy of Sleep Medicine Position Statement
Khosla, S., Deak, M. C., Gault, D., Goldstein, C. A., Hwang, D., Kwon, Y., O'Hearn, D., Schutte-Rodin, S., Yurcheshen, M., Rosen, I. M., et al. (2018) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.7128 · Cited by 242
Methodological Challenges When Using Actigraphy in Research
Berger, A. M., Wielgus, K. K., Young-McCaughan, S., Fischer, P., Farr, L., & Lee, K. A. (2008) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2007.10.008 · Cited by 231 · cited 3×
“Epochs longer than 60 s are not recommended when using Actiwatch devices and likely affect the specificity of the device [12,31,32].”
BackgroundConsumer Sleep Technologies: A Review of the Landscape
Ko, P.-R. T., Kientz, J. A., Choe, E. K., Kay, M., Landis, C. A., & Watson, N. F. (2015) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.5288 · Cited by 211 · cited 1×
“the future of sleep research and sleep medicine [28,29] and have the capacity to surpass gold-standard actigraphy in terms of functionality and accuracy of sleep measurement in the real world.”
BackgroundBaron, K. G., Duffecy, J., Berendsen, M. A., Cheung Mason, I., Lattie, E. G., & Manalo, N. C. (2018) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2017.12.002 · Cited by 186
Wrist actigraphic measures of sleep and rhythms
Kripke, D. F., Mullaney, D. J., Messin, S., & Wyborney, V. G. (1978) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(78)90133-5 · Cited by 181 · cited 1×
“Sleep researchers have been tracking sleep using wrist-worn actigraphy since the late 1970s [4]; recently, commercial wrist-worn wearable devices have begun to feature sleep-tracking technology as well.”
BackgroundSleep Staging Based on Autonomic Signals: A Multi-Center Validation Study
Hedner, J. A., White, D. P., Malhotra, A., Herscovici, S., Pittman, S. D., Zou, D., Grote, L., & Pillar, G. (2011) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.1078 · Cited by 170 · cited 1×
“Many wearables also incorporate other sensors, such as heart rate, oxygen saturation, or peripheral arterial tone, to supplement wrist activity data [20,21].”
BackgroundDeterminants for Sustained Use of an Activity Tracker: Observational Study
Hermsen, S., Moons, J., Kerkhof, P., Wiekens, C., & De Groot, M. (2017) · JMIR mHealth and uHealth · doi:10.2196/mhealth.7311 · Cited by 150 · cited 1×
“Battery life, therefore, is an important limiting factor when choosing a device for research, clinical, or personal measurement of sleep [12,25].”
BackgroundBhat, S., Ferraris, A., Gupta, D., Mozafarian, M., DeBari, V. A., Gushway-Henry, N., Gowda, S. P., Polos, P. G., Rubinstein, M., Seidu, H., et al. (2015) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.4840 · Cited by 144
Consumer sleep tracking devices: a review of mechanisms, validity and utility
Kolla, B. P., Mansukhani, S., & Mansukhani, M. P. (2016) · Expert Review of Medical Devices · doi:10.1586/17434440.2016.1171708 · Cited by 144 · cited 1×
“However, while the devices collect accelerometry and other biometric data, sleep scoring is done using algorithms on a companion mobile app or software platform [14,22].”
MethodLiang, Z., & Martell, M. A. C. (2018) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-018-0013-1 · Cited by 121
Kahawage, P., Jumabhoy, R., Hamill, K., de Zambotti, M., & Drummond, S. P. A. (2019) · Journal of Sleep Research · doi:10.1111/jsr.12931 · Cited by 91
Defining the rest interval associated with the main sleep period in actigraph scoring
Chow, C. M., Wong, S. N., Shin, M., Maddox, R. G., Feilds, K.-L., Paxton, K., Hawke, C. I., Hazell, P., & Steinbeck, K. S. (2016) · Nature and Science of Sleep · doi:10.2147/nss.s114969 · Cited by 85 · cited 1×
“While the practice parameters for the clinical or research use of actigraphy in healthy individuals, sleep-disordered populations, or military operational contexts have been published [8–10], these guidelines still encourage hand scoring of data to achieve the highest levels of accuracy, and there is no resolute conse…”
BackgroundLiang, Z., & Chapa-Martell, M. A. (2019) · JMIR mHealth and uHealth · doi:10.2196/13384 · Cited by 76
Dynamics and Ultradian Structure of Human Sleep in Real Life
Winnebeck, E. C., Fischer, D. R., Leise, T., & Roenneberg, T. (2017) · Current Biology · doi:10.1016/j.cub.2017.11.063 · Cited by 69 · cited 9×
“Moreover, deep sleep could represent any stage of sleep or a combination of NREM and REM epochs, as indicated in Figure 1 by the arrows explaining the conversion between the PSG numerical scores and range of averages for the PSG scores.”
BackgroundInfluentialMuzet, A., Werner, S., Fuchs, G., Roth, T., Saoud, J. B., Viola, A. U., Schaffhauser, J.-Y., & Luthringer, R. (2016) · Sleep Medicine · doi:10.1016/j.sleep.2016.01.015 · Cited by 53 · cited 10×
“Moreover, deep sleep could represent any stage of sleep or a combination of NREM and REM epochs, as indicated in Figure 1 by the arrows explaining the conversion between the PSG numerical scores and range of averages for the PSG scores.”
BackgroundInfluentialKubala, A. G., Gibbs, B. B., Buysse, D. J., Patel, S. R., Hall, M. H., & Kline, C. E. (2019) · Behavioral Sleep Medicine · doi:10.1080/15402002.2019.1651316 · Cited by 37
Wearables and Personal Health Data: Putting a Premium on Your Privacy
Troiano, A. T. (2017) · bepress Legal Repository · Cited by 17
Devine, J. K., Choynowski, J. J., Burke, T. M., Carlsson, K., Capaldi, V. F., McKeon, A. B., & Sowden, W. J. (2020) · Military Medical Research · doi:10.1186/s40779-020-00255-7 · Cited by 17 · cited 1×
“Supplementary measures may not always be practical to collect in operational research settings like the military, aviation, or shift work because they may increase the burden of data collection in a safety-sensitive environment [10].”
BackgroundSelf-citationPaul, M. A., Hursh, S. R., & Love, R. J. (2020) · Military Medicine · doi:10.1093/milmed/usaa210 · Cited by 12 · cited 1×
“The next step after laboratory validation testing of the Zulu watch will be to test its utility in real-world operations, not only to measure sleep compared to field actigraphy or self-report, but also to see how the Zulu watch measurements of sleep or sleep depth can be used to inform assumptions about sleep behavior…”
BackgroundSelf-citationDevine, J. K., Schwartz, L. P., Hursh, S. R., Mosher, E., Schumacher, S., Boyle, L., Davis, J. E., Stafford‐Smith, M., & Fitzgibbons, S. (2020) · Journal of surgical education · doi:10.1016/j.jsurg.2020.11.010 · Cited by 6 · cited 1×
“The next step after laboratory validation testing of the Zulu watch will be to test its utility in real-world operations, not only to measure sleep compared to field actigraphy or self-report, but also to see how the Zulu watch measurements of sleep or sleep depth can be used to inform assumptions about sleep behavior…”
BackgroundSelf-citation
Research works printed but not linked: 1
- Berry R.B. Brooks R. Gamaldo C.E. Harding S.M. Marcus C. Vaughn B.V. The AASM manual for the scoring of sleep and associated events Rules Terminol. Techn. Specif. Darien Ill. Am. Acad. Sleep Med. 2012 176 2012
Cited by
A Systematic Review of Sensing Technologies for Wearable Sleep Staging
Imtiaz, S. A. (2021) · Sensors · doi:10.3390/s21051562 · Cited by 162
Evaluating reliability in wearable devices for sleep staging
Birrer, V., Elgendi, M., Lambercy, O., & Menon, C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01016-9 · Cited by 164
“Anadditionaltwodevices 40,43 utilized input from accelerometers and additionally other sensors, such as ambient light, bio-impedance, or skin temperature, but did not include a PPG sensor.”
Svensson, T., Madhawa, K., Nt, H., Chung, U., & Svensson, A. K. (2024) · Sleep Medicine · doi:10.1016/j.sleep.2024.01.020 · Cited by 139
Ecological and social pressures interfere with homeostatic sleep regulation in the wild
Loftus, J., Harel, R., Nuñez, C., & Crofoot, M. (2022) · eLife · doi:10.7554/elife.73695 · Cited by 63
A systematic review of the performance of actigraphy in measuring sleep stages
Yuan, H., Hill, E. A., Kyle, S. D., & Doherty, A. (2024) · Journal of Sleep Research · doi:10.1111/jsr.14143 · Cited by 56
“Studies that used commercial150 grade devices relied on the proprietary algorithms from the vendors which do not 151 reveal the process for sleep stage classification [18, 19].”
BackgroundMethodInfluential
Metadata
- Authors
- Devine, J. K., Chinoy, E. D., Markwald, R. R., Schwartz, L. P., & Hursh, S. R.
- Affiliations
- Institutes for Behavior Resources (US)
- Leidos (United States) (US)
- Naval Health Research Center (US)
- Corresponding author
- Devine, Jaime K.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2020-12-25
- Volume
- 21
- Issue
- 1
- Pages
- 76
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC7796293
- OpenAlex
- W3113553538
- Cited by
- 32
- References
- 37
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 6 · 2023: 8 · 2022: 7 · 2021: 5 · 2020: 1
- FWCI
- 5.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Medical Technology Enterprise Consortium (MTEC-17-08-Multi-Topic-0104)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Sleep and related disorders
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Sleep and Work-Related Fatigue
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- wearable sleep trackers, sleep-wake detection, polysomnography, actigraphy, device validation, total sleep time, sleep efficiency
- MeSH terms
- Humans, Sleep, Wrist, Reproducibility of Results, Polysomnography, Young Adult, Actigraphy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.