CoChat
Sign Up
PreprintCited by 51Non-Invasive Vital Sign MonitoringECG Monitoring and AnalysisEmotion and Mood Recognition

Large-scale Training of Foundation Models for Wearable Biosignals

Open access
Abbaspourazad, Salar, Elachqar, Oussama, Miller, Andrew C., Emrani, Saba, Nallasamy, Udhyakumar, Shapiro, Ian·2023·arXiv
DOI 10.48550/arxiv.2312.05409arXiv 2312.05409
Open this paper in CoChat — free accountOpen PDF

Abstract

Tracking biosignals is crucial for monitoring wellness and preempting the development of severe medical conditions. Today, wearable devices can conveniently record various biosignals, creating the opportunity to monitor health status without disruption to one's daily routine. Despite widespread use of wearable devices and existing digital biomarkers, the absence of curated data with annotated medical labels hinders the development of new biomarkers to measure common health conditions. In fact, medical datasets are usually small in comparison to other domains, which is an obstacle for developing neural network models for biosignals. To address this challenge, we have employed self-supervised learning using the unlabeled sensor data collected under informed consent from the large longitudinal Apple Heart and Movement Study (AHMS) to train foundation models for two common biosignals: photoplethysmography (PPG) and electrocardiogram (ECG) recorded on Apple Watch. We curated PPG and ECG datasets from AHMS that include data from ~141K participants spanning ~3 years. Our self-supervised learning framework includes participant level positive pair selection, stochastic augmentation module and a regularized contrastive loss optimized with momentum training, and generalizes well to both PPG and ECG modalities. We show that the pre-trained foundation models readily encode information regarding participants' demographics and health conditions. To the best of our knowledge, this is the first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices $\unicode{x2013}$ prior works have commonly used smaller-size datasets collected in clinical and experimental settings. We believe PPG and ECG foundation models can enhance future wearable devices by reducing the reliance on labeled data and hold the potential to help the users improve their health.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.05409

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 33

  1. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, October 2020. URL http://arxiv. org/abs/2006.11477. arXiv:2006.11477 [cs, eess]. · arXiv
  2. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language, April 2022. URL http://arxiv.org/abs/2202.03555. arXiv:2202.03555 [cs. · arXiv
  3. Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-Invariance-Covariance Regular- ization for Self-Supervised Learning, January 2022. URL http://arxiv.org/abs/2105. 04906. arXiv:2105.04906 [cs. · arXiv
  4. Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards Democratizing Joint-Embedding Self-Supervised Learning, March 2023. URL http://arxiv.org/abs/2303.01986. arXiv:2303.01986 [cs. · arXiv
  5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, June 2020. URL http://arxiv.org/abs/ 2002.05709. arXiv:2002.05709 [cs, stat]. · arXiv
  6. Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self-Supervised Vision Transformers, August 2021. URL http://arxiv.org/abs/2104.02057. arXiv:2104.02057 · arXiv
  7. Joseph Y. Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi. Subject-Aware Contrastive Learning for Biosignals, June 2020. URL http://arxiv.org/abs/2007. 04871. arXiv:2007.04871 [cs, eess, stat]. · arXiv
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL http://arxiv. org/abs/1810.04805. arXiv:1810.04805 [cs. · arXiv
  9. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL http://arxiv.org/abs/2010.11929. arXiv:2010.11929 [cs. · arXiv
  10. Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank, February 2023. URL http://arxiv.org/abs/2210.02885. arXiv:2210.02885 [cs. · arXiv
  11. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One Embedding Space To Bind Them All, May 2023. URL http://arxiv.org/abs/2305.05665. arXiv:2305.05665 [cs. · arXiv
  12. Bryan Gopal, Ryan Han, Gautham Raghupathi, Andrew Ng, Geoff Tison, and Pranav Rajpurkar. 3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Aug- mentations. In Proceedings of Machine Learning for Health, pp. 156-167. PMLR, November 2021. URL https://proceedings.mlr.press/v158/gopal21a.html. ISSN: 2640-3498.
  13. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised Learning, September 2020. URL http://arxiv.org/ abs/2006.07733. arXiv:2006.07733 [cs, stat]. · arXiv
  14. Haraldur T. Hallgrímsson, Filip Jankovic, Tim Althoff, and Luca Foschini. Learning Individualized Cardiovascular Responses from Large-scale Wearable Sensors Data, December 2018. URL http://arxiv.org/abs/1812.01696. arXiv:1812.01696 [cs, stat]. · arXiv
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recog- nition, December 2015. URL http://arxiv.org/abs/1512.03385. arXiv:1512.03385 · arXiv
  16. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http://arxiv.org/abs/2106.09685. arXiv:2106.09685 [cs. · arXiv
  17. Dani Kiyasseh, Tingting Zhu, and David A. Clifton. CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients, May 2021. URL http://arxiv.org/abs/2005.13249. arXiv:2005.13249 [cs, eess, stat]. · arXiv
  18. Demetres Kostas, Stéphane Aroca-Ouellette, and Frank Rudzicz. BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data. Frontiers in Human Neuroscience, 15:653659, June 2021. ISSN 1662-5161. doi: 10.3389/fnhum. 2021.653659. URL https://www.frontiersin.org/articles/10.3389/fnhum. 2021.653659/full. · DOI
  19. Xiang Lan, Dianwen Ng, Shenda Hong, and Mengling Feng. Intra-Inter Subject Self-supervised Learning for Multivariate Cardiac Signals, September 2021. URL http://arxiv.org/abs/ 2109.08908. arXiv:2109.08908 [cs, eess]. · arXiv
  20. Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):2579-2605, 2008. ISSN 1533-7928. URL http://jmlr.org/ papers/v9/vandermaaten08a.html.
  21. Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Contrastive Representa- tion Learning for Electroencephalogram Classification. In Proceedings of the Machine Learn- ing for Health NeurIPS Workshop, pp. 238-253. PMLR, November 2020. URL https: //proceedings.mlr.press/v136/mohsenvand20a.html. ISSN: 2640-3498.
  22. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Con- trastive Predictive Coding, January 2019. URL http://arxiv.org/abs/1807.03748. arXiv:1807.03748 [cs, stat]. · arXiv
  23. OpenAI. GPT-4 Technical Report, March 2023. URL http://arxiv.org/abs/2303.08774. arXiv:2303.08774 [cs. · arXiv
  24. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, April 2023. URL http://arxiv.org/abs/2304.07193. arXiv:2304.07193 · arXiv
  25. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv.org/abs/2103.00020. arXiv:2103.00020 [cs. · arXiv
  26. Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pp. 606-610, September 2007.
  27. Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vec- tors for similarity search, August 2019. URL http://arxiv.org/abs/1806.03198. arXiv:1806.03198 [cs, stat]. · arXiv
  28. Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Net- works, September 2020. URL http://arxiv.org/abs/1905.11946. arXiv:1905.11946 [cs, stat]. · arXiv
  29. Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, and Cecilia Mascolo. Exploring Contrastive Learning in Human Activity Recognition for Healthcare, February 2021. URL http://arxiv. org/abs/2011.11542. arXiv:2011.11542 [cs, eess]. · arXiv
  30. Xian Wu, Chao Huang, Pablo Roblesgranda, and Nitesh Chawla. Representation Learning on Variable Length and Incomplete Wearable-Sensory Time Series, May 2020. URL http://arxiv.org/ abs/2002.03595. arXiv:2002.03595 [cs, eess, stat]. · arXiv
  31. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On Layer Normalization in the Transformer Archi- tecture, June 2020. URL http://arxiv.org/abs/2002.04745. arXiv:2002.04745 [cs, stat]. · arXiv
  32. Hang Yuan, Tatiana Plekhanova, Rosemary Walmsley, Amy C. Reynolds, Kathleen J. Maddi- son, Maja Bucan, Philip Gehrman, Alex Rowlands, David W. Ray, Derrick Bennett, Joanne McVeigh, Leon Straker, Peter Eastwood, Simon D. Kyle, and Aiden Doherty. Self-supervised learning of accelerometer data provides new insights for sleep and its association with mortal- ity, July 2023. URL https://www.medrxiv.org/content/10.1101/2023.07.07. 23292251v1. Pages: 2023.07.07.23292251. · DOI
  33. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction, June 2021. URL http://arxiv.org/abs/2103. 03230. arXiv:2103.03230 [cs, q-bio]. · arXiv

Cited by

Sorted by citations

Related

Authors
Abbaspourazad, S., Elachqar, O., Miller, A. C., Emrani, S., Nallasamy, U., & Shapiro, I.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2312.05409
arXiv
2312.05409
OpenAlex
W4389649986
Cited by
28
References
0
Citations by year
2026: 5 · 2025: 10 · 2024: 13
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Brigham and Women's Hospital
Topics
  • Physical Sciences › Engineering › Biomedical Engineering › Non-Invasive Vital Sign Monitoring
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
foundation models, wearable biosignals, photoplethysmography, electrocardiogram, self-supervised learning, Apple Watch, consumer wearable devices, contrastive learning, digital biomarkers, health monitoring

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review