The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
Open accessAbstract
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications. Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose the PhysFormer, an end-to-end video transformer based architecture, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. Furthermore, we also propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. One highlight is that, unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community. The codes are available at https://github.com/ZitongYu/PhysFormer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Curriculum learning [2], as a major machine learning regime with philosophy of easy-to-hard curriculum, is utilized to train PhysFormer.”
MethodPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 1×
“Many other ViT variants [8, 13, 21, 22, 25, 37, 53, 59, 69] are proposed from then, which achieve promising performance compared with its counterpart CNNs for image analysis tasks [6, 23, 73].”
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1×
“Many other ViT variants [8, 13, 21, 22, 25, 37, 53, 59, 69] are proposed from then, which achieve promising performance compared with its counterpart CNNs for image analysis tasks [6, 23, 73].”
Self-citationTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021) · arXiv · doi:10.48550/arxiv.2106.13230 · Cited by 2,294
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,074
TSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,055
Remote plethysmographic imaging using ambient light
Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008) · Optics Express · doi:10.1364/oe.16.021434 · Cited by 1,719
A Multimodal Database for Affect Recognition and Implicit Tagging
Soleymani, M., Lichtenauer, J., Pun, T., & Pantic, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.25 · Cited by 1,664
Is Space-Time Attention All You Need for Video Understanding?
Bertasius, G., Wang, H., & Torresani, L. (2021) · arXiv · doi:10.48550/arxiv.2102.05095 · Cited by 1,663
Non-contact, automated cardiac pulse measurements using video imaging and blind source separation
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010) · Optics Express · doi:10.1364/oe.18.010762 · Cited by 1,643
Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2086456 · Cited by 1,486
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407 · cited 2×
“Some works [20, 41] conduct temporal contextual modeling with transformer based on single-frame features from pretrained 2D networks, while other works [1, 3, 4, 15,38] mine the spatio-temporal attentions via video trans-former directly.”
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394
Robust pulse rate from chrominance-based rPPG
de Haan, G., & Jeanne, V. (2013) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2013.2266196 · Cited by 1,378
TransReID: Transformer-based Object Re-Identification
He, S., Luo, H., Wang, P., Wang, F., Li, H., & Jiang, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01474 · Cited by 1,244
Algorithmic principles of remote PPG
Wang, W., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2609282 · Cited by 1,234
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Chen, W., & McDuff, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_22 · Cited by 716
Facial Age Estimation by Learning from Label Distributions
Geng, X., Yin, C., & Zhou, Z.-H. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.51 · Cited by 618
Neimark, D., Bar, O., Zohar, M., & Asselmann, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00355 · Cited by 551
Remote Heart Rate Measurement from Face Videos under Realistic Situations
Li, X., Chen, J., Zhao, G., & Pietikainen, M. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.543 · Cited by 550
Self-citationDeep Label Distribution Learning With Label Ambiguity
Gao, B.-B., Xing, C., Xie, C.-W., Wu, J., & Geng, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2689998 · Cited by 508
RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation
Niu, X., Shan, S., Han, H., & Chen, X. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2947204 · Cited by 420
Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J. F., & Sebe, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.263 · Cited by 340
Yu, Z., Peng, W., Li, X., Hong, X., & Zhao, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00024 · Cited by 340
Self-citationEarly Convolutions Help Transformers See Better
Xiao, T., Singh, M., Mintun, E., Darrell, T. J., Dollár, P., & Girshick, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14881 · Cited by 339
End-to-End Temporal Action Detection With Transformer
Liu, X., Wang, Q., Hu, Y., Tang, X., Zhang, S., Bai, S., & Bai, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3195321 · Cited by 282
Learning Joint Spatial-Temporal Transformations for Video Inpainting
Zeng, Y., Fu, J., & Chao, H. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_31 · Cited by 281
Video Super-Resolution TransformerPreprint
Cao, J., Li, Y., Zhang, K., & Gool, L. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06847 · Cited by 254
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals MeasurementPreprint
Liu, X., Fromm, J., Patel, S., & McDuff, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03790 · Cited by 237
Video-Based Heart Rate Measurement: Recent Advances and Future Prospects
Chen, X., Cheng, J., Song, R., Liu, Y., Ward, R., & Wang, Z. J. (2019) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2018.2879706 · Cited by 227
Deep learning for face anti-spoofing: A survey
Yu, Z., Qin, Y., Li, X., Zhao, C., Lei, Z., & Zhao, G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3215850 · Cited by 227
Self-citationVideo-Based Remote Physiological Measurement via Cross-Verified Feature Disentangling
Niu, X., Yu, Z., Han, H., Li, X., Shan, S., & Zhao, G. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_18 · Cited by 213
Self-citationDual-GAN: Joint BVP and Noise Modeling for Remote Physiological Measurement
Lu, H., Han, H., & Zhou, S. K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01222 · Cited by 196
Meta-rPPG: Remote Heart Rate Estimation Using a Transductive Meta-learner
Lee, E., Chen, E., & Lee, C.-Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58583-9_24 · Cited by 181
EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video
Qiu, Y., Liu, Y., Arteaga-Falconi, J., Dong, H., & El Saddik, A. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2883866 · Cited by 180
Robust Heart Rate Measurement from Video Using Select Random Patches
Lam, A., & Kuno, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.415 · Cited by 176
SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific
Niu, X., Han, H., Shan, S., & Chen, X. (2018) · 2018 24th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.2018.8546321 · Cited by 174
Age Estimation Using Expectation of Label Distribution Learning
Gao, B.-B., Zhou, H.-Y., Wu, J., & Geng, X. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/99 · Cited by 170
Gideon, J., & Stent, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00396 · Cited by 166
AutoHR: A Strong End-to-End Baseline for Remote Heart Rate Measurement With Neural Searching
Yu, Z., Li, X., Niu, X., Shi, J., & Zhao, G. (2020) · IEEE Signal Processing Letters · doi:10.1109/lsp.2020.3007086 · Cited by 165
Self-citationFuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting
Liu, R., Deng, H., Huang, Y., Shi, X., Lu, L., Sun, W., Wang, X., Dai, J., & Li, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01378 · Cited by 162
TransRPPG: Remote Photoplethysmography Transformer for 3D Mask Face Presentation Attack Detection
Yu, Z., Li, X., Wang, P., & Zhao, G. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3089908 · Cited by 144
Self-citationSearching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
Yu, Z., Zhou, B., Wan, J., Wang, P., Chen, H., Liu, X., Li, S. Z., & Zhao, G. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3087348 · Cited by 140
Self-citationSpace-time Mixing Attention for Video TransformerPreprint
Bulat, A., Pérez-Rúa, J.-M., Sudhakaran, S., Martínez, B., & Tzimiropoulos, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05968 · Cited by 127
Li, X., Alikhani, I., Shi, J., Seppänen, T., Junttila, J. M., Majamaa‐Voltti, K., Tulppo, M. P., & Zhao, G. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00043 · Cited by 125
Self-citationDeep learning with time-frequency representation for pulse estimation from facial videos
Hsu, G.-S. J., Ambikapathi, A., & Chen, M.-S. (2017) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2017.8272721 · Cited by 122
Robust Remote Heart Rate Estimation from Face Utilizing Spatial-temporal Attention
Niu, X., Zhao, X., Han, H., Das, A., Dantcheva, A., Shan, S., & Chen, X. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756554 · Cited by 116
Facial-Video-Based Physiological Signal Measurement: Recent advances and affective applications
Yu, Z., Li, X., & Zhao, G. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3106285 · Cited by 110
Self-citationHR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
Ding, M., Lian, X., Yang, L., Wang, P., Jin, X., Lu, Z., & Luo, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00300 · Cited by 90
Nowara, E. M., McDuff, D., & Veeraraghavan, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00491 · Cited by 68
Continuous heart rate measurement from face: A robust rPPG approach with distribution learning
Niu, X., Han, H., Shan, S., & Chen, X. (2017) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/btas.2017.8272752 · Cited by 52
Temporal Action Proposal Generation with TransformersPreprint
Wang, L., Yang, H., Wu, W., Yao, H., & Huang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12043 · Cited by 37
NAS-HR: Neural architecture search for heart rate estimation from face videos
Lü, H., & Han, H. (2021) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2020.10.002 · Cited by 33
EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals MeasurementPreprint
Liu, X., Hill, B. L., Jiang, Z., Patel, S., & McDuff, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04447 · Cited by 30
Geometry-Contrastive Transformer for Generalized 3D Pose Transfer
Chen, H., Tang, H., Yu, Z., Sebe, N., & Zhao, G. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19901 · Cited by 18
Self-citationCamera-Based Physiological Sensing: Challenges and Future DirectionsPreprint
Liu, X., Patel, S., & McDuff, D. (2021) · arXiv (Cornell University) · Cited by 6
AniFormer: Data-driven 3D Animation with Transformer
Chen, H., Tang, H., Sebe, N., & Zhao, G. (2021) · Iris (University of Trento) · doi:10.5281/zenodo.5901777 · Cited by 2
Self-citation
Metadata
- Authors
- Yu, Z., Shen, Y., Shi, J., Zhao, H., Torr, P., & Zhao, G.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 4176-4186
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312358294
- Cited by
- 300
- References
- 71
- FWCI
- 129.39
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.