CoChat
Sign Up
PreprintCited by 53Emotion and Mood RecognitionFace recognition and analysisFace and Expression Recognition

The DOI matches its DataCite record.

Prompting Visual-Language Models for Dynamic Facial Expression Recognition

Open access
Zhao, Zengqun, Patras, Ioannis·2023·arXiv
DOI 10.48550/arxiv.2308.13382arXiv 2308.13382
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper presents a novel visual-language model called DFER-CLIP, which is based on the CLIP model and designed for in-the-wild Dynamic Facial Expression Recognition (DFER). Specifically, the proposed DFER-CLIP consists of a visual part and a textual part. For the visual part, based on the CLIP image encoder, a temporal model consisting of several Transformer encoders is introduced for extracting temporal facial expression features, and the final feature embedding is obtained as a learnable "class" token. For the textual part, we use as inputs textual descriptions of the facial behaviour that is related to the classes (facial expressions) that we are interested in recognising -- those descriptions are generated using large language models, like ChatGPT. This, in contrast to works that use only the class names and more accurately captures the relationship between them. Alongside the textual description, we introduce a learnable token which helps the model learn relevant context information for each expression during training. Extensive experiments demonstrate the effectiveness of the proposed method and show that our DFER-CLIP also achieves state-of-the-art results compared with the current supervised DFER methods on the DFEW, FERV39k, and MAFW benchmarks. Code is publicly available at https://github.com/zengqunzhao/DFER-CLIP.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.13382

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • DFER-CLIP's use of linguistic descriptions is limited because it only uses emotion category descriptions and does not take full advantage of linguistic features as a new modality.

    “While DFER-CLIP employs multimodal modelling using CLIP, it is argued that its use of linguistic descriptions is limited because it only utilises emotion category descriptions and does not take full advantage of linguistic features as a new modality.”

    From Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition (2025) · cites this paper as [10]

    The Abstract says DFER-CLIP uses textual descriptions of facial behaviour generated with language models, in contrast to approaches that use only class names. The Introduction likewise says its text encoder uses facial-behaviour descriptions instead of class names and learns context for each expression. The paper therefore contradicts the claim that it only uses emotion-category descriptions. (LLM, conf=0.92)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • DFER-CLIP's fine-tuning approach fails to take into account fusion of spatio-temporal features at the initial stage, instead extracting features at the end through a time-series module.

    “Moreover, the fine-tuning approach under consideration fails to take into account the fusion of spatio-temporal features at the initial stage. Instead, the approach extracts features at the end of the process through the use of a time-series module.”

    From Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition (2025) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 51

  1. Wissam J Baddar and Yong Man Ro. Mode variational lstm robust to unseen modes of variation: Application to facial expression recognition. In AAAI, pages 3215-3223, 2019.
  2. Mina Bishay, Petar Palasek, Stefan Priebe, and Ioannis Patras. Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis. IEEE Trans- actions on Affective Computing, 12(4):949-961, 2019.
  3. Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.
  4. Junkai Chen, Zenghai Chen, Zheru Chi, and Hong Fu. Facial expression recognition in video with multiple feature fusion. IEEE Transactions on Affective Computing, 9(1): 38-50, 2016.
  5. Ciprian Adrian Corneanu, Marc Oliu Simón, Jeffrey F Cohn, and Sergio Escalera Guer- rero. Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8):1548-1568, 2016.
  6. Charles Darwin and Phillip Prodger. The expression of the emotions in man and ani- mals. Oxford University Press, 1998.
  7. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  8. Zoran Duric, Wayne D Gray, Ric Heishman, Fayin Li, Azriel Rosenfeld, Michael J Schoelles, Christian Schunn, and Harry Wechsler. Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction. Proceeding of the IEEE, 90(7):1272-1289, 2002.
  9. Paul Ekman. Universals and cultural differences in facial expressions of emotion. In Nebraska symposium on motivation. University of Nebraska Press, 1971.
  10. Niki Maria Foteinopoulou and Ioannis Patras. Learning from label relationships in human affect. In MM, pages 80-89, 2022.
  11. Niki Maria Foteinopoulou, Christos Tzelepis, and Ioannis Patras. Estimating continu- ous affect with label uncertainty. In ACII, pages 1-8, 2021.
  12. Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns re- trace the history of 2d cnns and imagenet? In CVPR, pages 6546-6555, 2018.
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
  14. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compu- tation, 9(8):1735-1780, 1997.
  15. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904-4916, 2021.
  16. Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In MM, pages 2881-2889, 2020.
  17. Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual- language models for efficient video understanding. In ECCV, pages 105-124, 2022.
  18. Bokyeung Lee, Hyunuk Shin, Bonhwa Ku, and Hanseok Ko. Frame level emotion guided dynamic facial expression recognition with emotion grouping. In CVPRW, pages 5680-5690, 2023.
  19. Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition networks. In ICCV, pages 10143-10152, 2019.
  20. Hanting Li, Mingzhe Sui, Zhaoqing Zhu, and Feng Zhao. Nr-dfernet: Noise-robust network for dynamic facial expression recognition. arXiv preprint arXiv:2206.04975, 2022. · arXiv
  21. Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. Cliper: A unified vision- language framework for in-the-wild facial expression recognition. arXiv preprint arXiv:2303.00193, 2023. · arXiv
  22. Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. Intensity-aware loss for dynamic facial expression recognition in the wild. In AAAI, pages 67-75, 2023.
  23. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In ICML, pages 12888-12900, 2022.
  24. Shan Li and Weihong Deng. Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing, 13(3):1195-1215, 2020.
  25. Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In CVPR, 2023.
  26. Daizong Liu, Hongting Zhang, and Pan Zhou. Video-based facial expression recogni- tion using graph convolutional networks. In ICPR, pages 607-614, 2021.
  27. Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. Open- vocabulary semantic segmentation with frozen vision-language models. In BMVC, pages 1-15, 2022.
  28. Fuyan Ma, Bin Sun, and Shutao Li. Logo-former: Local-global spatio-temporal trans- former for dynamic facial expression recognition. In ICASSP, pages 1-5, 2023.
  29. Sachit Menon and Carl Vondrick. Visual classification via description from large lan- guage models. In ICLR, 2023.
  30. Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In AAAI, pages 1359-1367, 2020.
  31. Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang. Zero-shot temporal action detection via vision-language prompting. In ECCV, pages 681-697, 2022.
  32. Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In ECCV, pages 1-18, 2022.
  33. James Oldfield, Christos Tzelepis, Yannis Panagakis, Mihalis A Nicolaou, and Ioannis Patras. Parts of speech-grounded subspaces in vision-language models. arXiv preprint arXiv:2305.14053, 2023. · arXiv
  34. Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In CVPR, pages 2085-2094, 2021.
  35. Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, pages 5533-5541, 2017.
  36. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748- 8763, 2021.
  37. Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In CVPR, pages 6545-6554, 2023.
  38. Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In CVPR, pages 4489-4497, 2015.
  39. Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018.
  40. Hanyang Wang, Bo Li, Shuang Wu, Siyuan Shen, Feng Liu, Shouhong Ding, and Aimin Zhou. Rethinking the learning paradigm for dynamic facial expression recogni- tion. In CVPR, pages 17958-17968, 2023.
  41. Yan Wang, Wei Song, Wei Tao, Antonio Liotta, Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge, Wei Zhang, et al. A systematic review on affective comput- ing: Emotion models, databases, and recent advances. Information Fusion, 2022.
  42. Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Ferv39k: a large-scale multi-scene dataset for facial expression recognition in videos. In CVPR, pages 20922-20931, 2022.
  43. Yan Wang, Yixuan Sun, Wei Song, Shuyong Gao, Yiwen Huang, Zhaoyu Chen, Weifeng Ge, and Wenqiang Zhang. Dpcnet: Dual path multi-excitation collaborative network for facial expression representation learning in videos. In MM, pages 101-110, 2022.
  44. Torsten Wilhelm. Towards facial expression analysis in a driver assistance system. In FG, pages 1-4, 2019.
  45. Zengqun Zhao and Qingshan Liu. Former-dfer: Dynamic facial expression recognition transformer. In MM, pages 1553-1561, 2021.
  46. Zengqun Zhao, Qingshan Liu, and Shanmin Wang. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Trans- actions on Image Processing, 30:6544-6556, 2021.
  47. Zengqun Zhao, Qingshan Liu, and Feng Zhou. Robust lightweight facial expression recognition network with label distribution training. In AAAI, pages 3510-3519, 2021.
  48. Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learn- ing in a visual-linguistic manner. In CVPR, pages 18697-18709, 2022.
  49. Lin Zhong, Qingshan Liu, Peng Yang, Bo Liu, Junzhou Huang, and Dimitris N Metaxas. Learning active facial patches for expression analysis. In CVPR, pages 2562- 2569, 2012.
  50. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816-16825, 2022.
  51. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337-2348, 2022.

Cited by

Sorted by citations

Metadata

Authors
Zhao, Z., & Patras, I.
Affiliations
Queen Mary University of London (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2308.13382
arXiv
2308.13382
OpenAlex
W4386228569
Cited by
53
References
0
Citations by year
2026: 12 · 2025: 24 · 2024: 17
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Queen Mary University of London
  • Horizon 2020 Framework Programme (951911)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
Keywords
dynamic facial expression recognition, vision-language models, CLIP, class token, ChatGPT

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review