Self-Supervised Learning for ECG-Based Emotion Recognition
Open accessAbstract
We present an electrocardiogram (ECG) -based emotion recognition system using self-supervised learning. Our proposed architecture consists of two main networks, a signal transformation recognition network and an emotion recognition network. First, unlabelled data are used to successfully train the former network to detect specific pre-determined signal transformations in the self-supervised learning step. Next, the weights of the convolutional layers of this network are transferred to the emotion recognition network, and two dense layers are trained in order to classify arousal and valence scores. We show that our self-supervised approach helps the model learn the ECG feature manifold required for emotion recognition, performing equal or better than the fully-supervised version of the model. Our proposed method outperforms the state-of-the-art in ECG-based emotion recognition with two publicly available datasets, SWELL and AMIGOS. Further analysis highlights the advantage of our self-supervised approach in requiring significantly less data to achieve acceptable results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work using self-supervised learning has shown promise in the same data modalities such as ECG data.
“Recent work using self-supervised learning has shown promise in the same data modalities such as ECG data [142,143].”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [142]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Toward machine emotional intelligence: analysis of affective physiological state
Picard, R. W., Vyzas, E., & Healey, J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.954607 · Cited by 2,334 · see citing works
Detecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207 · see citing works
Raina, R., Battle, A. J., Lee, H., Packer, B., & Ng, A. Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273592 · Cited by 1,514 · see citing works
AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups
Miranda-Correa, J. A., Abadi, M. K., Sebe, N., & Patras, I. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2884461 · Cited by 740 · see citing works
Um, T. T. (n.d.) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 602 · see citing works
Open Information Extraction Using Wikipedia
Wu, F. Q., & Weld, D. S. (2010) · Cited by 547 · see citing works
Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction
Xu, D., Xiao, J., Zhao, Z., Shao, J., Xie, D., & Zhuang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01058 · Cited by 417 · see citing works
Multi-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346 · see citing works
The SWELL Knowledge Work Dataset for Stress and User Modeling Research
Koldijk, S., Sappelli, M., Verberne, S., Neerincx, M. A., & Kraaij, W. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663257 · Cited by 297 · see citing works
Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry
Kocabas, M., Karagoz, S., & Akbaş, E. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00117 · Cited by 297 · see citing works
Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019) · IEEE Access · doi:10.1109/access.2018.2883213 · Cited by 289 · see citing works
Dynamic Difficulty Adjustment in Computer Games Through Real-Time Anxiety-Based Affective Feedback
Liu, C., Agrawal, P., Sarkar, N., & Chen, S. (2009) · International Journal of Human-Computer Interaction · doi:10.1080/10447310902963944 · Cited by 259 · see citing works
Reliability of Ultra‐Short ECG Indices for Heart Rate Variability
Nussinovitch, U., Elishkevitz, K. P., Katz, K., Nussinovitch, M., Segev, S., Volovitz, B., & Nussinovitch, N. (2011) · Annals of Noninvasive Electrocardiology · doi:10.1111/j.1542-474x.2011.00417.x · Cited by 250 · see citing works
Recognizing Emotions Induced by Affective Sounds through Heart Rate Variability
Nardelli, M., Valenza, G., Greco, A., Lanatà, A., & Scilingo, E. P. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2432810 · Cited by 219 · see citing works
Detecting Work Stress in Offices by Combining Unobtrusive Sensors
Koldijk, S., Neerincx, M. A., & Kraaij, W. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2610975 · Cited by 184 · see citing works
Stress Detection in Working People
Sriramprakash, S., Prasanna, V. D., & Murthy, O. V. R. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.09.090 · Cited by 174 · see citing works
Transitive Invariance for Self-Supervised Visual Representation Learning
Wang, X., He, K., & Gupta, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.149 · Cited by 173 · see citing works
A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion From Heartbeat
Harper, R., & Southern, J. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981610 · Cited by 71 · see citing works
Using smart offices to predict occupational stress
Alberdi, A., Aztiria, A., Basarab, A., & Cook, D. J. (2018) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2018.04.005 · Cited by 47 · see citing works
Verma, P., & Sood, S. K. (2018) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-018-1877-1 · Cited by 39 · see citing works
Classification of Cognitive Load and Expertise for Adaptive Simulation using Deep Multitask Learning
Sarkar, P., Ross, K., Ruberto, A. J., Rodenburg, D., Hungler, P., & Etemad, A. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925507 · Cited by 36 · see citing works
Self-citationSelf-supervised audio representation learning for mobile devicesPreprint
Tagliasacchi, M., Gfeller, B., Quitry, F. D. C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11796 · Cited by 31 · see citing works
Ross, K., Sarkar, P., Rodenburg, D., Ruberto, A. J., Hungler, P. C., Szulewski, A., Howes, D. W., & Etemad, A. (2019) · Sensors · doi:10.3390/s19194270 · Cited by 27 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Subject-Aware Contrastive Learning for Biosignals
Cheng, J. Y., Goh, H., Dogrusoz, K., Tuzel, O., & Azemi, E. (2020) · arXiv · doi:10.48550/arxiv.2007.04871 · Cited by 74 · see citing works
Self-supervised transfer learning of physiological representations from free-living wearable data
Spathis, D., Perez-Pozuelo, I., Brage, S., Wareham, N. J., & Mascolo, C. (2021) · Proceedings of the Conference on Health, Inference, and Learning · doi:10.1145/3450439.3451863 · Cited by 40 · see citing works
Human-centred artificial intelligence for mobile health sensing: challenges and opportunities
Dang, T., Spathis, D., Ghosh, A., & Mascolo, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.230806 · Cited by 30 · see citing works
Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion
Rahimi Taghanaki, S., Rainbow, M. J., & Etemad, A. (2021) · 2021 International Symposium on Wearable Computers · doi:10.1145/3460421.3480417 · Cited by 22 · see citing works
“Self-supervised learning has been previously explored in time-series representation learning applications such as computer vision [10, 12], bio-signal analysis (e.g. ECG [21, 22] and EEG [3]), speech recognition [17], and others.”
BackgroundSelf-citation
- Authors
- Sarkar, P., & Etemad, A.
- Affiliations
- Queen's University (CA)
- Venue
- ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 3217-3221
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2980927909
- Cited by
- 146
- References
- 24
- Citations by year
- 2026: 6 · 2025: 17 · 2024: 28 · 2023: 35 · 2022: 29 · 2021: 24 · 2020: 7
- FWCI
- 23.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
- Keywords
- self-supervised learning, electrocardiogram, valence and arousal classification, transfer learning, convolutional layers, AMIGOS dataset, physiological signal classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
