DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Non-contact video-based physiological measurement has many applications in health care and human-computer interaction. Practical applications require measurements to be accurate even in the presence of large head rotations. We propose the first end-to-end system for video-based measurement of heart and breathing rate using a deep convolutional network. The system features a new motion representation based on a skin reflection model and a new attention mechanism using appearance information to guide motion estimation, both of which enable robust measurement under heterogeneous lighting and major motions. Our approach significantly outperforms all current state-of-the-art methods on both RGB and infrared video datasets. Furthermore, it allows spatial-temporal distributions of physiological signals to be visualized via the attention mechanism.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“CV also offers potential as a tool for remote physiological measurements.For instance, systems could use video 141 to analyze heart and breathing rates 141 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [141]
Citations
Cites
(2014) · Cited by 33,419
(2014) · Climate Change 2013 – The Physical Science Basis · doi:10.1017/cbo9781107415324.004 · Cited by 23,880
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
Beyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
Remote plethysmographic imaging using ambient light
Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008) · Optics Express · doi:10.1364/oe.16.021434 · Cited by 1,719
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
Non-contact, automated cardiac pulse measurements using video imaging and blind source separation
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010) · Optics Express · doi:10.1364/oe.18.010762 · Cited by 1,643
Self-citationAdvancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2086456 · Cited by 1,486
Self-citationRobust pulse rate from chrominance-based rPPG
de Haan, G., & Jeanne, V. (2013) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2013.2266196 · Cited by 1,378
Eulerian video magnification for revealing subtle changes in the world
Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., & Freeman, W. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185561 · Cited by 1,357
Algorithmic principles of remote PPG
Wang, W., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2609282 · Cited by 1,234
Unsupervised Learning for Physical Interaction through Video Prediction
Finn, C., Goodfellow, I., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1605.07157 · Cited by 759
FlowNet: Learning Optical Flow with Convolutional Networks
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 562
Detecting Pulse from Head Motions in Video
Balakrishnan, G., Durand, F., & Guttag, J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.440 · Cited by 553
Remote Heart Rate Measurement from Face Videos under Realistic Situations
Li, X., Chen, J., Zhao, G., & Pietikainen, M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.543 · Cited by 550
Improved motion robustness of remote-PPG by using the blood volume pulse signature
de Haan, G., & van Leest, A. (2014) · Physiological Measurement · doi:10.1088/0967-3334/35/9/1913 · Cited by 491
VideoLSTM convolves, attends and flows for action recognition
Li, Z., Gavrilyuk, K., Gavves, E., Jain, M., & Snoek, C. G. M. (2018) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.10.011 · Cited by 466
Non-contact video-based vital sign monitoring using ambient light and auto-regressive models
Tarassenko, L., Villarroel, M., Guazzi, A., Jorge, J., Clifton, D. A., & Pugh, C. (2014) · Physiological Measurement · doi:10.1088/0967-3334/35/5/807 · Cited by 423
Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2854726 · Cited by 385
A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation
Wang, W., Stuijk, S., & de Haan, G. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2508602 · Cited by 370
Heart rate measurement based on a time-lapse image.
Takano, C., & Ohta, Y. (2007) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2006.09.006 · Cited by 345
Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J. F., & Sebe, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.263 · Cited by 340
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02586 · Cited by 320
Improvements in Remote Cardiopulmonary Measurement Using a Five Band Digital Camera
McDuff, D., Gontarek, S., & Picard, R. W. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2323695 · Cited by 287
Self-citationExploiting Spatial Redundancy of Image Sensor for Motion Robust rPPG
Wang, W., Stuijk, S., & de Haan, G. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2356291 · Cited by 284
Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions
Agarwal, A., Negahban, S., & Wainwright, M. J. (2012) · The Annals of Statistics · doi:10.1214/12-aos1000 · Cited by 280
Aarts, L. A. M., Jeanne, V., Cleary, J. P., Lieber, C., Nelson, J. S., Bambang Oetomo, S., & Verkruysse, W. (2013) · Early Human Development · doi:10.1016/j.earlhumdev.2013.09.016 · Cited by 241
A survey of remote optical photoplethysmographic imaging methods
McDuff, D. J., Estepp, J. R., Piasecki, A. M., & Blackford, E. B. (2015) · 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2015.7319857 · Cited by 204
Self-citationA Machine Learning Approach to Improve Contactless Heart Rate Monitoring Using a Webcam
Monkaresi, H., Calvo, R. A., & Yan, H. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2291900 · Cited by 179
Robust Heart Rate Measurement from Video Using Select Random Patches
Lam, A., & Kuno, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.415 · Cited by 176
Learning Spatiotemporal Features with 3D Convolutional NetworksPreprint
Tran, D., Bourdev, L., Fergus, R., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.0767 · Cited by 155
Delving Deeper into Convolutional Networks for Learning Video Representations
Ballas, N., Yao, L., Pal, C., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06432 · Cited by 126
Estepp, J. R., Blackford, E. B., & Meier, C. M. (2014) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2014.6974121 · Cited by 118
Robust efficient estimation of heart rate pulse from video
Xu, S., Sun, L., & Rohde, G. K. (2014) · Biomedical Optics Express · doi:10.1364/boe.5.001124 · Cited by 83
Chaichulee, S., Villarroel, M., Jorge, J., Arteta, C., Green, G., McCormick, K. P., Zisserman, A., & Tarassenko, L. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.41 · Cited by 78
Two-Stream Flow-Guided Convolutional Attention Networks for Action Recognition
Tran, A., & Cheong, L. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.368 · Cited by 74
VideoLSTM Convolves, Attends and Flows for Action RecognitionPreprint
Li, Z., Gavves, E., Jain, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.01794 · Cited by 63
Video Description Generation Incorporating Spatio-Temporal Features and a Soft-Attention MechanismPreprint
Li, Y., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. C. (2015) · arXiv (Cornell University) · Cited by 46
Supervised learning approach to remote heart rate estimation from facial videos
Osman, A., Turcot, J., & el Kaliouby, R. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163150 · Cited by 38
Cardiolens: remote physiological monitoring in a mixed reality environment
Hurter, C., & McDuff, D. (2017) · ACM SIGGRAPH Emerging Technologies · doi:10.1145/3084822.3084834 · Cited by 37
Self-citationEliminating Physiological Information from Facial Videos
Chen, W. '., & Picard, R. W. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.15 · Cited by 27
Self-citation
Research works printed but not linked: 2
Cited by
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal NetworksPreprint
Yu, Z., Li, X., & Zhao, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02419 · Cited by 505
FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
Ciftci, U. A., Demir, I., & Yin, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3009287 · Cited by 499
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals MeasurementPreprint
Liu, X., Fromm, J., Patel, S., & McDuff, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03790 · Cited by 498
Self-citationRhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation
Niu, X., Shan, S., Han, H., & Chen, X. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2947204 · Cited by 420
Metadata
- Authors
- Chen, W., & McDuff, D.
- Affiliations
- MIT Media Lab (US)
- Microsoft (United States) (US)
- Corresponding author
- Chen, Weixuan
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2018
- Pages
- 356-373
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2963433879
- Cited by
- 716
- References
- 49
- Citations by year
- 2026: 75 · 2025: 163 · 2024: 132 · 2023: 109 · 2022: 93 · 2021: 75 · 2020: 43 · 2019: 25 · 2018: 1
- FWCI
- 78.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Biomedical Engineering › Non-Invasive Vital Sign Monitoring
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Heart Rate Variability and Autonomic Control
- Keywords
- heart rate estimation, respiratory rate estimation, non-contact monitoring, convolutional attention network, motion estimation, infrared video
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.