The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation
Open accessAbstract
Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HRVIPL-HR is available at: ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Remote plethysmographic imaging using ambient light
Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008) · Optics Express · doi:10.1364/oe.16.021434 · Cited by 1,719
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
Non-contact, automated cardiac pulse measurements using video imaging and blind source separation
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010) · Optics Express · doi:10.1364/oe.18.010762 · Cited by 1,643
Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2086456 · Cited by 1,486
Robust pulse rate from chrominance-based rPPG
de Haan, G., & Jeanne, V. (2013) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2013.2266196 · Cited by 1,378
Eulerian video magnification for revealing subtle changes in the world
Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., & Freeman, W. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185561 · Cited by 1,357
Algorithmic principles of remote PPG
Wang, W., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2609282 · Cited by 1,234
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Chen, W., & McDuff, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_22 · Cited by 716
Detecting Pulse from Head Motions in Video
Balakrishnan, G., Durand, F., & Guttag, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.440 · Cited by 553
Remote Heart Rate Measurement from Face Videos under Realistic Situations
Li, X., Chen, J., Zhao, G., & Pietikainen, M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.543 · Cited by 550
Improved motion robustness of remote-PPG by using the blood volume pulse signature
de Haan, G., & van Leest, A. (2014) · Physiological Measurement · doi:10.1088/0967-3334/35/9/1913 · Cited by 491
Non-contact video-based pulse rate measurement on a mobile service robot
Stricker, R., Muller, S., & Gross, H.-M. (2014) · The 23rd IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2014.6926392 · Cited by 456
Measuring pulse rate with a webcam — A non-contact method for evaluating cardiac activity
Lewandowska, M., Rumiński, J., Kocejko, T., & Nowak, J. (2011) · Cited by 420
DistancePPG: Robust non-contact vital signs monitoring using a camera
Kumar, M., Veeraraghavan, A., & Sabharwal, A. (2015) · Biomedical Optics Express · doi:10.1364/boe.6.001565 · Cited by 417 · cited 1×
“[21] Webcam G Patch level signals combination FFT using weighted average”
MethodA Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation
Wang, W., Stuijk, S., & de Haan, G. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2508602 · Cited by 370
Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J. F., & Sebe, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.263 · Cited by 340
Improvements in Remote Cardiopulmonary Measurement Using a Five Band Digital Camera
McDuff, D., Gontarek, S., & Picard, R. W. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2323695 · Cited by 287
Exploiting Spatial Redundancy of Image Sensor for Motion Robust rPPG
Wang, W., Stuijk, S., & de Haan, G. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2356291 · Cited by 284
Mean-Variance Loss for Deep Age Estimation from a Face
Pan, H., Han, H., Shan, S., & Chen, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00554 · Cited by 235
Self-citationVIPL-HR: A Multi-modal Database for Pulse Estimation from Less-Constrained Face Video
Niu, X., Han, H., Shan, S., & Chen, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-20873-8_36 · Cited by 213
Self-citationMotion-Resistant Remote Imaging Photoplethysmography Based on the Optical Properties of Skin
Feng, L., Po, L.-M., Xu, X., Li, Y., & Ma, R. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2014.2364415 · Cited by 186
Robust Heart Rate Measurement from Video Using Select Random Patches
Lam, A., & Kuno, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.415 · Cited by 176
Motion Robust Remote-PPG in Infrared
van Gastel, M., Stuijk, S., & de Haan, G. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2390261 · Cited by 142
Li, X., Alikhani, I., Shi, J., Seppänen, T., Junttila, J. M., Majamaa‐Voltti, K., Tulppo, M. P., & Zhao, G. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00043 · Cited by 125
Deep learning with time-frequency representation for pulse estimation from facial videos
Hsu, G.-S. J., Ambikapathi, A., & Chen, M.-S. (2017) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2017.8272721 · Cited by 122
McDuff, D., Blackford, E. B., & Estepp, J. R. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.17 · Cited by 108
Improved pulse detection from head motions using DCT
Irani, R., Nasrollahi, K., & Moeslund, T. (2014) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0004669001180124 · Cited by 84
Tsouri, G. R., & Zheng, L. (2015) · Journal of Biomedical Optics · doi:10.1117/1.jbo.20.4.048002 · Cited by 74
Learning-based heart rate detection from remote photoplethysmography features
Hsu, Y., Lin, Y.-L., & Hsu, W. H. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854440 · Cited by 71
Automatic Engagement Prediction with GAP Feature
Niu, X., Han, H., Zeng, J., Sun, X., Shan, S., Huang, Y., Yang, S., & Chen, X. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3264982 · Cited by 69
Self-citationImproving 2D Face Recognition via Discriminative Face Depth Estimation
Cui, J., Zhang, H., Han, H., Shan, S., & Chen, X. (2018) · International Conference on Biometrics · doi:10.1109/icb2018.2018.00031 · Cited by 64
Self-citationConstrained independent component analysis approach to nonobtrusive pulse rate measurements
Tsouri, G. R., Kyal, S., Dianat, S. A., & Mestha, L. K. (2012) · Journal of Biomedical Optics · doi:10.1117/1.jbo.17.7.077011 · Cited by 62
Motion robust remote photoplethysmography in CIELab color space
Yang, Y., Liu, C., Yu, H., Shao, D., Tsow, F., & Tao, N. J. (2016) · Journal of Biomedical Optics · doi:10.1117/1.jbo.21.11.117001 · Cited by 57
Amplitude-selective filtering for remote-PPG
Wang, W., Brinker, A. C. D., Stuijk, S., & de Haan, G. (2017) · Biomedical Optics Express · doi:10.1364/boe.8.001965 · Cited by 57
Estimating Heart Rate and Rhythm via 3D Motion Tracking in Depth Video
Yang, C., Cheung, G., & Stanković, V. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2672198 · Cited by 52
Continuous heart rate measurement from face: A robust rPPG approach with distribution learning
Niu, X., Han, H., Shan, S., & Chen, X. (2017) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2017.8272752 · Cited by 52
Self-citationDeep Multi-Task Learning for Joint Prediction of Heterogeneous Face Attributes
Wang, F., Han, H., Shan, S., & Chen, X. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.30 · Cited by 42
Self-citation
Cited by
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., & Tao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3415112 · Cited by 552
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Yu, Z., Shen, Y., Shi, J., Zhao, H., Torr, P., & Zhao, G. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00415 · Cited by 300
DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
Qi, H., Guo, Q., Juefei-Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., & Zhao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.07634 · Cited by 280
DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
Qi, H., Guo, Q., Juefei-Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., & Zhao, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413707 · Cited by 220
Countering Spoof: Towards Detecting Deepfake with Multidimensional Biological Signals
Jin, X., Ye, D., & Chen, C. (2021) · Security and Communication Networks · doi:10.1155/2021/6626974 · Cited by 17
Metadata
- Authors
- Niu, X., Shan, S., Han, H., & Chen, X.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2020
- Volume
- 29
- Pages
- 2409-2423
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W2982196965
- Cited by
- 420
- References
- 41
- FWCI
- 11.28
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.