CoChat
Sign Up
PreprintCited by 11EEG and Brain-Computer InterfacesSleep and related disordersObstructive Sleep Apnea Research

SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals

Open access
Thapa, Rahul, He, Bryan, Kjaer, Magnus Ruud, Moore, Hyatt, Ganjoo, Gauri, Mignot, Emmanuel, Zou, James·2024·arXiv
DOI 10.48550/arxiv.2405.17766arXiv 2405.17766
Open this paper in CoChat — free accountOpen PDF

Abstract

Sleep is a complex physiological process evaluated through various modalities recording electrical brain, cardiac, and respiratory activities. We curate a large polysomnography dataset from over 14,000 participants comprising over 100,000 hours of multi-modal sleep recordings. Leveraging this extensive dataset, we developed SleepFM, the first multi-modal foundation model for sleep analysis. We show that a novel leave-one-out approach for contrastive learning significantly improves downstream task performance compared to representations from standard pairwise contrastive learning. A logistic regression model trained on SleepFM's learned embeddings outperforms an end-to-end trained convolutional neural network (CNN) on sleep stage classification (macro AUROC 0.88 vs 0.72 and macro AUPRC 0.72 vs 0.48) and sleep disordered breathing detection (AUROC 0.85 vs 0.69 and AUPRC 0.77 vs 0.61). Notably, the learned embeddings achieve 48% top-1 average accuracy in retrieving the corresponding recording clips of other modalities from 90,000 candidates. This work demonstrates the value of holistic multi-modal sleep modeling to fully capture the richness of sleep recordings. SleepFM is open source and available at https://github.com/rthapa84/sleepfm-codebase.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.17766

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 40

  1. Bannur, S., Hyland, S., Liu, Q., Perez-Garcia, F., Ilse, M., Castro, D. C., Boecking, B., Sharma, H., Bouzid, K., Thieme, A., et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15016-15027, 2023.
  2. Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. Making the most of text semantics to improve biomedical vision-language processing. In European conference on computer vision, pp. 1-21. Springer, 2022.
  3. Brink-Kjaer, A., Leary, E. B., Sun, H., Westover, M. B., Stone, K. L., Peppard, P. E., Lane, N. E., Cawthon, P. M., Redline, S., Jennum, P., et al. Age estimation from sleep studies using deep learning predicts life expectancy. NPJ digital medicine, 5(1):103, 2022.
  4. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
  5. Ghassemi, M. M., Moody, B. E., Lehman, L.-W. H., Song, C., Li, Q., Sun, H., Mark, R. G., Westover, M. B., and Clifford, G. D. You snooze, you win: the phys- ionet/computing in cardiology challenge 2018. In 2018 Computing in Cardiology Conference (CinC), volume 45, pp. 1-4. IEEE, 2018.
  6. Gopal, B., Han, R., Raghupathi, G., Ng, A., Tison, G., and Rajpurkar, P. 3kg: Contrastive learning of 12-lead elec- trocardiograms using physiologically-inspired augmen- tations. In Machine Learning for Health, pp. 156-167.
  7. PMLR, 2021.
  8. Haidar, R., McCloskey, S., Koprinska, I., and Jeffries, B. Convolutional neural networks on multiple respiratory channels to detect hypopnea and obstructive apnea events. In 2018 International Joint Conference on Neural Net- works (IJCNN), pp. 1-7. IEEE, 2018.
  9. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
  10. Huang, S.-C., Shen, L., Lungren, M. P., and Yeung, S. Glo- ria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 3942-3951, 2021.
  11. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
  12. Kiyasseh, D., Zhu, T., and Clifton, D. A. Clocs: Contrastive learning of cardiac signals across space, time, and pa- tients. In International Conference on Machine Learning, pp. 5606-5615. PMLR, 2021.
  13. Kryger, M. H., Roth, T., and Dement, W. C. Principles and practice of sleep medicine fifth edition, 2010.
  14. Lalam, S. K., Kunderu, H. K., Ghosh, S., Kumar, H., Awasthi, S., Prasad, A., Lopez-Jimenez, F., Attia, Z. I., Asirvatham, S., Friedman, P., et al. ECG representation learning with multi-modal EHR data. Transactions on Machine Learning Research, 2023.
  15. Leary, E. B., Stone, K. L., and Mignot, E. Living to dream-reply. JAMA neurology, 78(4):495-496, 2021.
  16. Li, C., Qi, Y., Ding, X., Zhao, J., Sang, T., and Lee, M. A deep learning method approach for sleep stage classifi- cation with eeg spectrogram. International Journal of Environmental Research and Public Health, 19(10):6322, 2022.
  17. Lu, M. Y., Chen, B., Zhang, A., Williamson, D. F., Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., and Mahmood, F. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19764-19775, 2023.
  18. Mostafa, S. S., Mendonca, F., Ravelo-Garcia, A. G., Juliá- Serdá, G. G., and Morgado-Dias, F. Multi-objective hy- perparameter optimization of convolutional neural net- work for obstructive sleep apnea detection. IEEE Access, 8:129586-129599, 2020.
  19. Mousavi, S., Afghah, F., and Acharya, U. R. Sleepeegnet: Automated sleep stage scoring with sequence to sequence deep learning approach. PloS one, 14(5):e0216456, 2019.
  20. Nassi, T. E., Ganglberger, W., Sun, H., Bucklin, A. A., Biswal, S., van Putten, M. J., Thomas, R. J., and West- over, M. B. Automated scoring of respiratory events in sleep with a single effort belt and deep neural net- works. IEEE transactions on biomedical engineering, 69 (6):2094-2104, 2021.
  21. Nørskov, A. V., Zahid, A. N., and Mørup, M. CSLP-AE: A contrastive split-latent permutation autoencoder frame- work for zero-shot electroencephalography signal conver- sion. arXiv preprint arXiv:2311.07788, 2023. · arXiv
  22. Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. · arXiv
  23. Ouyang, D., Theurer, J., Stein, N. R., Hughes, J. W., Elias, P., He, B., Yuan, N., Duffy, G., Sandhu, R. K., Ebinger, J., et al. Electrocardiographic deep learning for predicting post-procedural mortality. arXiv preprint arXiv:2205.03242, 2022. · arXiv
  24. Perslev, M., Darkner, S., Kempfner, L., Nikolic, M., Jennum, P. J., and Igel, C. U-sleep: Resilient high-frequency sleep staging. NPJ digital medicine, 4 (1), 72, 2021.
  25. Phan, H., Andreotti, F., Cooray, N., Chén, O. Y., and De Vos, M. Seqsleepnet: end-to-end hierarchical recurrent neu- ral network for sequence-to-sequence automatic sleep staging. IEEE Transactions on Neural Systems and Reha- bilitation Engineering, 27(3):400-410, 2019.
  26. Phan, H., Chén, O. Y., Tran, M. C., Koch, P., Mertins, A., and De Vos, M. Xsleepnet: Multi-view sequential model for automatic sleep staging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5903- 5915, 2021.
  27. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
  28. Raghu, A., Chandak, P., Alam, R., Guttag, J., and Stultz, C. Contrastive pre-training for multimodal medical time series. In NeurIPS 2022 Workshop on Learning from Time Series for Health, 2022.
  29. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
  30. Stephansen, J. B., Olesen, A. N., Olsen, M., Ambati, A., Leary, E. B., Moore, H. E., Carrillo, O., Lin, L., Han, F., Yan, H., et al. Neural network analysis of sleep stages enables efficient diagnosis of narcolepsy. Nature commu- nications, 9(1):5229, 2018.
  31. Supratak, A., Dong, H., Wu, C., and Guo, Y. Deepsleepnet: A model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Transactions on Neural Sys- tems and Rehabilitation Engineering, 25(11):1998-2008, 2017.
  32. Tan, M. and Le, Q. Efficientnet: Rethinking model scal- ing for convolutional neural networks. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
  33. Tsinalis, O., Matthews, P., Guo, Y., and Zafeiriou, S. Auto- matic sleep stage scoring with single-channel EEG using convolutional neural networks. arxiv 2016. arXiv preprint arXiv:1610.01683. · arXiv
  34. Tsinalis, O., Matthews, P. M., and Guo, Y. Automatic sleep stage scoring using time-frequency analysis and stacked sparse autoencoders. Annals of biomedical engineering, 44:1587-1597, 2016.
  35. Worley, S. L. The extraordinary importance of sleep: the detrimental effects of inadequate sleep on health and pub- lic safety drive an explosion of sleep research. Pharmacy and Therapeutics, 43(12):758, 2018.
  36. Yeo, M., Byun, H., Lee, J., Byun, J., Rhee, H.-Y., Shin, W., and Yoon, H. Respiratory event detection during sleep using electrocardiogram and respiratory related signals: Using polysomnogram and patch-type wearable device data. IEEE Journal of Biomedical and Health Informatics, 26(2):550-560, 2021.
  37. Yildirim, O., Baloglu, U. B., and Acharya, U. R. A deep learning model for automated sleep stages classification using PSG signals. International journal of environmental research and public health, 16(4):599, 2019.
  38. Zahid, A. N., Jennum, P., Mignot, E., and Sorensen, H. B. MSED: A multi-modal sleep event detection model for clinical sleep analysis. IEEE Transactions on Biomedical Engineering, 2023.
  39. Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Lan- glotz, C. P. Contrastive learning of medical visual rep- resentations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022.
  40. Zhao, X., Wang, X., Yang, T., Ji, S., Wang, H., Wang, J., Wang, Y., and Wu, Q. Classification of sleep apnea based on EEG sub-band signal characteristics. Scientific Reports, 11(1):5824, 2021.

Cited by

Sorted by citations

Related

Authors
Thapa, R., He, B., Kjaer, M. R., Moore, H., Ganjoo, G., Mignot, E., & Zou, J.
Affiliations
  • Stanford Medicine (US)
  • Stanford University (US)
Corresponding author
Thapa, Rahul
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2405.17766
arXiv
2405.17766
OpenAlex
W4399151533
Cited by
10
References
0
Citations by year
2026: 3 · 2025: 6 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Sleep and related disorders
  • Health Sciences › Medicine › Physiology › Obstructive Sleep Apnea Research
Keywords
sleep analysis, polysomnography, multimodal representation learning, sleep stage classification, brain activity, electrocardiogram, respiratory signals, contrastive learning, foundation models, cross-modal retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review