Multimodal Dynamic Face Emotion Verification Model based on Priori Prompt Words for Dynamic Facial Expression Recognition
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index; 11 of the 11 sources its claims cite were checked for support.
Abstract
This paper proposes MPPW-DFER, a multimodal dynamic facial emotion verification model that adds language-like prior prompt words to dynamic facial videos and introduces Temporal Deformable Attention Embedding. It reports outperforming existing methods on DFER and MAFW.
Overview
CoChat Summary
TL;DR
The paper proposes MPPW-DFER, a multimodal dynamic facial expression recognition model that uses a large multimodal model to generate Priori Prompt Word descriptions from video and adds Temporal Deformable Attention Embedding for early temporal modeling. It reports best UAR/WAR on DFER and improved WAR on MAFW over compared methods.
Key findings
- On DFER, MPPW-DFER achieved UAR 68.23 and WAR 77.94, outperforming HiCMAE (63.76 UAR, 75.0 WAR) and MMA-DFER (67.01 UAR, 77.51 WAR).
- On MAFW, MPPW-DFER achieved UAR 43.87 and WAR 60.04; UAR was only 0.8% lower than MMA-DFER, while WAR improved by 2.6% over MMA-DFER’s 58.45.
- Ablation showed that adding PPW alone to a visual-only backbone reached UAR 56.95 and WAR 60.32 on DFER, and UAR 55.68 and WAR 60.57 on MAFW; adding TDAE produced the full model results.
- The audio-only baseline performed sub-optimally on both datasets, which the paper says underscores the importance of visual cues for emotion recognition.
Methods
- The backbone follows MMA-DFER for video and audio, using Face MAE and Video MAE for visual/audio backbones, an AudioMAE pre-trained on AudioSet, six transformers, MLP dimension adjustment, and feature-fusion bottlenecks between Transformer layers.
- Priori Prompt Words are generated with Qwen2-VL-7B using prompts focused on facial expressions, body language, and gestures, limited to concise descriptions of no more than 15 words and excluding background information; CLIP Text Encoder and CLIP Image Encoder are used with only their first two Transformer layers, frozen for feature extraction and fusion.
- TDAE is introduced in the embedding layer: video is divided into 16 segments, keyframes are randomly sampled from each segment, and attention is shifted to the spatial dimension with global pooling over the channel dimension.
- Experiments use DFER/DFEW (16,000 video segments, 7 emotion categories) and MAFW (10,045 segments, 11 emotion categories), with 5-fold cross-validation, UAR and WAR metrics, 16 images per video at 224x224, learning rate 1e-4, batch size 8, AdamW, 50 rounds, and two Nvidia 4090 24G GPUs for about 20 hours.
Limitations
- The paper states that in domains such as education, merely recognizing emotions is insufficient; models must also provide interpretable explanations and richer contextual descriptions, which is left for future work.
- The paper reports that MAFW UAR was slightly lower than MMA-DFER by 0.8%, even though WAR improved.
- The text makes evident that PPW generation was initially time-consuming: generating a five-second video glossary took about eight to ten seconds on an Nvidia H800 40G GPU, so it was moved to a preliminary processing step.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 5 Not valid · 0 Not checked · 25 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Ensuring data diversity is one aspect of this challenge, but it is also necessary to take into account complex scene variations, environmental disturbances, and individual expression differences. These factors contribute to the difficulty and expense of data collection and labelling [7].”
What the sources say
- Partly supportedcites [7] [7] Wiles, O., Koepke, A., & Zisserman, A. (2018). Self-supervised learning of a facial attribute embedding from video. arXiv preprint arXiv:1808.06882. Google Scholar a [...] expense of data collection and labelling b [...] advent of self-supervised learning (SSL) sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says supervised facial-analysis methods need large amounts of labelled data that may be expensive or difficult to obtain in realistic, unconstrained settings; it also notes disagreement among annotators of emotions. It does not identify data diversity, scene variations, environmental disturbances, or differences between individuals’ expressions as causes of that difficulty and expense. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the primary focus of this module is the classification of features derived from already fused video and audio data, where a substantial extraction of features has been conducted.”
§ 3 Method > 3.3 Temporal Deformable Attention Embedding
What the sources say
- Partly supportedcites [19] [19] Chumachenko, K., Iosifidis, A., & Gabbouj, M. (2024). MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4673-4682). Crossref Google Scholar a [...] and achieved significant results. MMA-DFER b [...] methods as those implemented in MMA-DFER c [...] ] and MMA-DFER sourceExperimental — accuracy not yet measured
Section 3.4 places the Multimodal Temporal Transformer after the video and audio representations are fused, so the claim is right about its position after feature extraction. But the section describes its purpose as modeling between-frame temporal dependencies; its output is then passed to a separate classifier. Calling classification the module’s primary focus overstates what the paper says. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method, MPPW-DFER, achieved the best results across both datasets in terms of UAR and WAR.”
- Also stated in: § 5.2 Ablation Experiment · plain
“Our proposed model, V-base+PPW+TDAE, outperformed all baseline methods, attaining 68.23 (UAR) and 77.94 (WAR) on DFER and 43.87 (UAR) and 60.04 (WAR) on MAFW.”
§ 5 Result and discussion > 5.1 Baseline Model Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 5.2 Ablation Experiment · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our approach outperforms existing methods on two public datasets, DFER and MAFW, demonstrating its effectiveness and innovation in the field.”
- Also stated in: § 1 Introduction · boosted
“We validate our method through experiments on the DFER [12] and MAFW [13] datasets. The results demonstrate that our approach achieves the best modeling performance to date.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforming previous methods like HiCMAE and MMA-DFER, which recorded 63.76, 75.0, and 67.01, 77.51, respectively.”
§ 5 Result and discussion > 5.1 Baseline Model Experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
10 more claims
Create a free account to see all 37 claims and every source verdict.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 12
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5 Result and discussion > 5.2 Ablation Experiment
“Further gains were observed when both PPW and TDAE were integrated. Our proposed model, V-base+PPW+TDAE, outperformed all baseline methods, attaining 68.23 (UAR) and 77.94 (WAR) on DFER and 43.87 (UAR) and 60.04 (WAR) on MAFW.”
§ 5 Result and discussion > 5.2 Ablation Experiment
“The inclusion of PPW alone significantly improved performance, achieving a UAR of 56.95 and a WAR of 60.32 on DFER, and 55.68 (UAR) and 60.57 (WAR) on MAFW.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5 Result and discussion > 5.1 Baseline Model Experiments
“Our method, MPPW-DFER, achieved the best results across both datasets in terms of UAR and WAR.”
§ 5 Result and discussion > 5.1 Baseline Model Experiments
“On MAFW, our method achieved 43.87 for UAR, which is only slightly lower than MMA-DFER, a difference of just 0.8%.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4 Experiment setting > 4.2 Implementation Details and Metrics
“A selection of seven representative methods for computing facial emotion recognition using deep learning models was made”
§ 4 Experiment setting > 4.2 Implementation Details and Metrics
“including Wav2Vec2.0 [24], HuBERT [26], WavLM Plus [25], C3D+LSTM [28], NR DFERNet [27], TMEP [21], DFER-CLIP [13], SVFAP [22], MAE-DFER [8], HiCMAE [10], S2D [25] and MMA-DFER [19] for DFER and MAFW experiments.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1 Introduction
“We validate our method through experiments on the DFER [12] and MAFW [13] datasets.”
§ 4 Experiment setting > 4.1 Dataset Details
“We use two publicly available dynamic facial emotion datasets: DFEW [14] and MAFW [15].”
Citations
Cites
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
Agriculture Development, Pesticide Application and Its Impact on the Environment
Tudi, M., Daniel Ruan, H., Wang, L., Lyu, J., Sadler, R., Connell, D., Chu, C., & Phung, D. T. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18031112 · Cited by 2,658 · cited 1×
“The field of facial expression recognition has evolved from addressing the static facial expression recognition (SFER) problem in controlled image environments [4] to dynamic facial expression recognition.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926 · cited 2×
“Recent FER technologies have been implemented in educational institutions [25, 26], facilitating applications such as the monitoring of student engagement and the assessment of learning outcomes through sentiment analysis.”
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.12191 · Cited by 1,302
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · cited 1×
“For example, Sun et al. [8] proposed a model based on a self-supervised architecture (MAE-DFER) based on the previous VideoMAE [9] and introduced an efficient local-global interaction converter (LGI Former) as an encoder.”
Facial recognition technology in schools: critical questions and concerns
Andrejevic, M., & Selwyn, N. (2020) · Learning, Media and Technology · doi:10.1080/17439884.2020.1686014 · Cited by 284 · cited 1×
“In the context of educational environments, particularly those re-liant upon video-based analytics, Dynamic Facial Emotion Recognition (DFER) assumes significant importance [23].”
DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild
Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., & Liu, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413620 · Cited by 207
MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognition
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612365 · Cited by 172
Contrastive Audio-Visual Masked AutoencoderPreprint
Gong, Y., Rouditchenko, A., Liu, A. H., Harwath, D. F., Karlinsky, L., Kuehne, H., & Glass, J. R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07839 · Cited by 169
Spatio-temporal convolutional features with nested LSTM for facial expression recognition
Yu, Z., Liu, G., Liu, Q., & Deng, J. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.028 · Cited by 162 · cited 2×
“Yu et al. [14] proposed SAANet, which introduces a sparse self-attention mechanism to augment the perceptual action unit (AU) features, and also combines a dual-sampling strategy and metric learning.”
CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition
Li, H., Niu, H., Zhu, Z., & Zhao, F. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687508 · Cited by 130
Self-supervised learning of a facial attribute embedding from video
Wiles, O., Koepke, A. S., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1808.06882 · Cited by 111 · cited 2×
“The advent of self-supervised learning (SSL) [7] methodologies has presented a number of potentially efficacious solutions to this predicament.”
Transformer-based Multimodal Information Fusion for Facial Expression Analysis
Zhang, W., Qiu, F., Wang, S., Zeng, H., Zhang, Z., An, R., Ma, B., & Ding, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00271 · Cited by 103 · cited 1×
“This evolution has been further fuelled by the availability of large-scale datasets spanning static, dynamic, and multimodal domains [5].”
Clip-aware expressive feature learning for video-based facial expression recognition
Liu, Y., Feng, C., Yuan, X., Zhou, L., Wang, W., Qin, J., & Luo, Z. (2022) · Information Sciences · doi:10.1016/j.ins.2022.03.062 · Cited by 99 · cited 2×
“In their seminal work, Liu et al. [15] introduced CE-FLNet, an innovative framework that prioritises the identification of the most informative frames in the context of Facial Expression Recognition (FER).”
Liu, Y., Dai, W., Feng, C., Wang, W., Yin, G., Zeng, J., & Shan, S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3548190 · Cited by 86 · cited 2×
“We validate our method through experiments on the DFER [12] and MAFW [13] datasets.”
NR-DFERNet: Noise-Robust Network for Dynamic Facial Expression RecognitionPreprint
Li, H., Sui, M., Zhu, Z., & Zhao, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.04975 · Cited by 79
Zhang, X., Li, M., Lin, S., Xu, H., & Xiao, G. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3312858 · Cited by 78
Chen, Y., Li, J., Shan, S., Wang, M., & Hong, R. (2025) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3453443 · Cited by 78 · cited 1×
“0 [24], HuBERT [26], WavLM Plus [25], C3D+LSTM [28], NR DFERNet [27], TMEP [21], DFER-CLIP [13], SVFAP [22], MAE-DFER [8], HiCMAE [10], S2D [25] and MMA-DFER [19] for DFER and MAFW experiments.”
Sun, L., Lian, Z., Liu, B., & Tao, J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102382 · Cited by 74
Prompting Visual-Language Models for Dynamic Facial Expression Recognition
Zhao, Z., & Patras, I. (2023) · arXiv · doi:10.48550/arxiv.2308.13382 · Cited by 53
Altameem, T. A., & Altameem, A. (2020) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104044 · Cited by 42 · cited 1×
“…techniques for automatically understanding human emotional states by means of facial expressions, speech features, or linguistic semantics have made significant progress and have been widely used in various domains such as collaborative teaching, healthcare [1], and human-computer interaction [2].”
Classen, C. (2023) · Routledge eBooks · doi:10.4324/9781003369769 · Cited by 35
SER30K: A Large-Scale Dataset for Sticker Emotion Recognition
Liu, S., Zhang, X., & Yang, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548407 · Cited by 35
Chumachenko, K., Iosifidis, A., & Gabbouj, M. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00470 · Cited by 34 · cited 3×
“The methodology employed for the backbone comprised the utilisation of identical processing and training methods as those implemented in MMA-DFER [19] for video and audio, as illustrated in Figure 1.”
DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the WildPreprint
Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05924 · Cited by 20
Real-life Dynamic Facial Expression Recognition: A Review
Saleem, S. M., Zeebaree, S. R. M., & Abdulrazzaq, M. B. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1963/1/012010 · Cited by 14 · cited 1×
“…techniques for automatically understanding human emotional states by means of facial expressions, speech features, or linguistic semantics have made significant progress and have been widely used in various domains such as collaborative teaching, healthcare [1], and human-computer interaction [2].”
The concept of semantic field in linguistics
Ataboyev, I. M., & Turgunova, F. R. (2022) · ACADEMICIA An International Multidisciplinary Research Journal · doi:10.5958/2249-7137.2022.00223.3 · Cited by 8
MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression RecognitionPreprint
Sun, L., Lian, Z., Liu, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02227 · Cited by 7 · cited 2×
“0 [24], HuBERT [26], WavLM Plus [25], C3D+LSTM [28], NR DFERNet [27], TMEP [21], DFER-CLIP [13], SVFAP [22], MAE-DFER [8], HiCMAE [10], S2D [25] and MMA-DFER [19] for DFER and MAFW experiments.”
CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression RecognitionPreprint
Li, H., Niu, H., Zhu, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.00193 · Cited by 4 · cited 1×
“CLIPER [17] introduced Multi-Expression Text Descriptors (METD) to learn fine-grained expression representations and designed an early layer fusion module to generate feature maps with better spatial consistency, thus improving interpretability.”
EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression RecognitionPreprint
Foteinopoulou, N. M., & Patras, I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16640 · Cited by 1
Metadata
- Authors
- Xie, H.
- Affiliations
- Shenzhen Entropy Future Education Technology Co., Shenzhen, Guangdong, China
- Corresponding author
- Xie, Huiling
- ORCID
- Xie, Huiling
- Venue
- Proceedings of the 2025 International Conference on Artificial Intelligence and Educational Systems
- Publisher
- ACM
- Published
- 2025-04-25
- Pages
- 297-303
- OpenAlex
- W4412923181
- Semantic Scholar
- fd170a256d9c8be653099568496cc0cb34d9d4db
- Cited by
- 0
- References
- 22
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Artificial Intelligence and Educational Systems (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- dynamic facial expression recognition, multimodal fusion, temporal attention, large multimodal models, video-based emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.