Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Unobtrusive and accurate ambulatory methods are needed to monitor long-term sleep patterns for improving health. Previously developed ambulatory sleep detection methods rely either in whole or in part on self-reported diary data as ground truth, which is a problem, since people often do not fill them out accurately. This paper presents an algorithm that uses multimodal data from smart-phones and wearable technologies to detect sleep/wake state and sleep onset/offset using a type of recurrent neural network with long-short-term memory (LSTM) cells for synthesizing temporal information. We collected 5580 days of multimodal data from 186 participants and compared the new method for sleep/wake classification and sleep onset/offset detection to, first, nontemporal machine learning methods and, second, a state-of-the-art actigraphy software. The new LSTM method achieved a sleep/wake classification accuracy of 96.5%, and sleep onset/offset detection F1 scores of 0.86 and 0.84, respectively, with mean absolute errors of 5.0 and 5.5 min, respectively, when compared with sleep/wake state and sleep onset/offset assessed using actigraphy and sleep diaries. The LSTM results were statistically superior to those from non-temporal machine learning algorithms and the actigraphy software. We show good generalization of the new algorithm by comparing participant-dependent and participant-independent models, and we show how to make the model nearly realtime with slightly reduced performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
The Role of Actigraphy in the Study of Sleep and Circadian Rhythms
Ancoli-Israel, S., Cole, R., Alessi, C., Chambers, M., Moorcroft, W., & Pollak, C. P. (2003) · Sleep · doi:10.1093/sleep/26.3.342 · Cited by 2,751 · cited 1×
“Actigraphy [19], [20] is based on the observation that there is less movement during sleep and more movement during wake [21], [22].”
The consensus sleep diary: standardizing prospective sleep self-monitoring
Carney, C. E., Buysse, D. J., Ancoli-Israel, S., Edinger, J. D., Krystal, A. D., Lichstein, K. L., & Morin, C. M. (2012) · Sleep · doi:10.5665/sleep.1642 · Cited by 2,172 · cited 1×
“Sleep diaries and questionnaires [23] also have several drawbacks, namely users’ adherence and reporting bias [23], [24].”
The role and validity of actigraphy in sleep medicine: an update
Sadeh, A. (2011) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2010.10.001 · Cited by 1,415 · cited 1×
“ment disorders [37] and shift workers [38], [39]).”
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
Sleep Deprivation and Vigilant Attention
Lim, J., & Dinges, D. F. (2008) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1417.002 · Cited by 1,213 · cited 1×
“disturbs mood [5], leads to slowed reaction time [6] and attentional failures [7], [8], and compromises memory and learning [9].”
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 1×
“Finally, compared with implicit knowledge learned in LSTM, RNNs with explicit memory [57], [58] can memorize facts and common sense, which are potentially useful for the task of sleep detection.”
Metabolic consequences of sleep and sleep loss
Van Cauter, E., Spiegel, K., Tasali, E., & Leproult, R. (2008) · Sleep Medicine · doi:10.1016/s1389-9457(08)70013-3 · Cited by 812 · cited 1×
“increases appetite and food intake [2], decreases insulin sensitivity and glucose tolerance [3], impairs immune function [4],”
Impact of insufficient sleep on total daily energy expenditure, food intake, and weight gain
Markwald, R. R., Melanson, E. L., Smith, M. R., Higgins, J., Perreault, L., Eckel, R. H., & Wright, K. P. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1216951110 · Cited by 810 · cited 1×
“increases appetite and food intake [2], decreases insulin sensitivity and glucose tolerance [3], impairs immune function [4],”
Visual discrimination learning requires sleep after training
Stickgold, R., James, L., & Hobson, J. A. (2000) · Nature Neuroscience · doi:10.1038/81756 · Cited by 715 · cited 1×
“disturbs mood [5], leads to slowed reaction time [6] and attentional failures [7], [8], and compromises memory and learning [9].”
Sleep habits and susceptibility to the common cold
Cohen, S., Doyle, W. J., Alper, C. M., Janicki-Deverts, D., & Turner, R. B. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2008.505 · Cited by 605 · cited 1×
“increases appetite and food intake [2], decreases insulin sensitivity and glucose tolerance [3], impairs immune function [4],”
Objectively Measured Sleep Characteristics among Early-Middle-Aged Adults
Lauderdale, D. S., Knutson, K. L., Yan, L. L., Rathouz, P. J., Hulley, S. B., Sidney, S., & Liu, K. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj199 · Cited by 581
Unobtrusive Sleep Monitoring using Smartphones
Chen, Z., Lin, M., Chen, F., Lane, N., Cardone, G., Wang, R., Li, T., Chen, Y., Choudhury, T., & Cambell, A. (2013) · Proceedings of the ICTs for improving Patients Rehabilitation Research Techniques · doi:10.4108/icst.pervasivehealth.2013.252148 · Cited by 300 · cited 4×
“automatically inferred sleep duration with ±42 minute error using smartphone data (light sensor, phone lock, off and charge logs, activity and audio features) and regression models [27].”
InfluentialBarger, L. K., Flynn-Evans, E. E., Kubey, A., Walsh, L., Ronda, J. M., Wang, W., Wright, K. P., & Czeisler, C. A. (2014) · The Lancet Neurology · doi:10.1016/s1474-4422(14)70122-x · Cited by 277
Circadian rhythm of wrist temperature in normal-living subjects
Sarabia, J. A., Rol, M. A., Mendiola, P., & Madrid, J. A. (2008) · Physiology & Behavior · doi:10.1016/j.physbeh.2008.08.005 · Cited by 222 · cited 1×
“ing environments similar to those in our experiment [53].”
DoppleSleep: a contactless unobtrusive sleep sensing system using short-range Doppler radar
Rahman, T., Adams, A. T., Ravichandran, R. V., Zhang, M., Patel, S. N., Kientz, J. A., & Choudhury, T. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2804280 · Cited by 175 · cited 1×
“smartphone placed on the bed to track movement via accelerometers [14], using contact [15] and non-contact microphones [16], short-range doppler radar [17] and WiFi [18].”
Arora, T., Broglia, E., Pushpakumar, D., Lodhi, T., & Taheri, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0072406 · Cited by 160
Quantitative analysis of wrist electrodermal activity during sleep
Sano, A., Picard, R. W., & Stickgold, R. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.09.011 · Cited by 141
Self-citationMonitoring sleep with a smartphone accelerometer
NATALE, V., DREJAK, M., ERBACCI, A., TONETTI, L., FABBRI, M., & MARTONI, M. (2012) · Sleep and Biological Rhythms · doi:10.1111/j.1479-8425.2012.00575.x · Cited by 73 · cited 1×
“smartphone placed on the bed to track movement via accelerometers [14], using contact [15] and non-contact microphones [16], short-range doppler radar [17] and WiFi [18].”
Sleep as a vital sign: why medical practitioners need to routinely ask their patients about sleep
Grandner, M. A., & Malhotra, A. (2015) · Sleep Health · doi:10.1016/j.sleh.2014.12.011 · Cited by 53
Sano, A., & Picard, R. W. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6943744 · Cited by 25 · cited 2×
“1 Hz intervals), and three storm features according to [52],”
Self-citation
Research works printed but not linked: 32
- 10.1093/sleep/22.3.378 · DOI · Paper page
- lee, 2015, Consumer sleep tracking devices: A critical review, Stud Health Technol Inform, 210, 458
- 10.5664/jcsm.5498 · DOI · Paper page
- 10.1109/BHI.2017.7897306 · DOI
- 10.1093/sleep/15.4.293 · DOI · Paper page
- 10.1016/S1389-9457(00)00098-8 · DOI · Paper page
- 10.5665/sleep.4918 · DOI · Paper page
- 10.1145/2556288.2557220 · DOI · Paper page
- 10.1016/j.sleh.2014.12.008 · DOI · Paper page
- 10.1053/smrv.2001.0182 · DOI · Paper page
- 10.1016/j.brainresbull.2003.12.012 · doi:10.1016/j.brainresbull.2003.12.012
- 10.1016/0014-4886(68)90025-3 · DOI
- 10.1093/sleep/15.5.461 · DOI · Paper page
- 10.2170/jjphysiol.14.365 · DOI
- 10.1371/journal.pone.0061142 · DOI · Paper page
- hinton, 0, Rmsprop: Divide the gradient by a running average of its recent magnitude, Neural Netw Mach Learning
- 10.1002/0471751723.ch10 · DOI
- sadeh, 1989, Actigraphically-based automatic bedtime sleep-wake scoring: Validity and clinical applications, J Ambulatory Monitoring, 2, 209
- 10.1378/chest.121.3.753 · DOI
- 10.1007/s10439-015-1290-y · doi:10.1007/s10439-015-1290-y
- 10.1371/journal.pone.0084139 · DOI
- 10.1145/2746285.2746303 · DOI · Paper page
- 10.1037/h0033615 · DOI
- dinges, 1997, Cumulative sleepiness, mood disturbance, and psychomotor vigilance performance decrements during a week of sleep restricted to 4-5 hours per night, Sleep, 20, 267
- doran, 2001, Sustained attention performance during sleep deprivation: Evidence of state instability, Archives Italiennes de Biologie, 139, 253
- 10.1109/TNSRE.2017.2733220 · DOI · Paper page
- sathyanarayana, 2016, Sleep quality prediction from wearable data using deep learning, JMIR Mhealth Uhealth, 4
- 10.1016/j.pmcj.2011.09.004 · DOI · Paper page
- graves, 0, Towards end-to-end speech recognition with recurrent neural networks, Proc Int Conf Mach Learn, 32, 1764
- 10.2196/jmir.6821 · DOI
- 10.3115/v1/W14-4012 · DOI · Paper page
- sutskever, 0, Sequence to sequence learning with neural networks, Proc Neural Inf Process Syst, 3104
Cited by
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456
Deep multi-view learning methods: A review
Yan, X., Hu, S., Mao, Y., Ye, Y., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.090 · Cited by 322
Deep learning based multimodal biomedical data fusion: an overview and comparative review
Duan, J., Xiong, J., Li, Y., & Ding, W. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102536 · Cited by 284
Sleep stage classification from heart-rate variability using long short-term memory neural networks
Radha, M., Fonseca, P., Moreau, A., Ross, M., Cerny, A., Anderer, P., Long, X., & Aarts, R. M. (2019) · Scientific Reports · doi:10.1038/s41598-019-49703-y · Cited by 186
“With non-EEG data, Sano et al .”
Metadata
- Authors
- Sano, A., Chen, W., Lopez-Martinez, D., Taylor, S., & Picard, R. W.
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2019-07
- Volume
- 23
- Issue
- 4
- Pages
- 1607-1617
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W2889162782
- Cited by
- 70
- References
- 21
- FWCI
- 2.87
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Institutes of Health (R01GM105018)
- Samsung Electronics
- NEC
- MIT Media Lab Consortium
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.