Speech Emotion Recognition via an Attentive Time–Frequency Neural Network
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 3 of the 28 sources its claims cite were checked for support.
Abstract
Spectrogram is commonly used as the input feature of deep neural networks to learn the high(er)-level time–frequency pattern of speech signal for speech emotion recognition (SER). Generally, different emotions correspond to specific energy activations both within frequency bands and time frames on spectrogram, which indicates the frequency and time domains are both essential to represent the emotion for SER. However, recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain, which makes these methods suffer from the following issues: 1) neglecting to model the emotion-related correlations within frequency domain during the time–frequency joint learning and 2) ignoring to capture the specific frequency bands associated with emotions. To cope with the issues, we propose an attentive time–frequency neural network (ATFNN) for SER, including a time–frequency neural network (TFNN) and time–frequency attention. Specifically, aiming at the first issue, we design a TFNN with a frequency-domain encoder (F-Encoder) based on the Transformer encoder and a time-domain encoder (T-Encoder) based on the bidirectional long short-term memory (Bi-LSTM). The F-Encoder and T-Encoder model the correlations within frequency bands and time frames, respectively, and they are embedded into a time–frequency joint learning strategy to obtain the time–frequency patterns of speech emotions. Moreover, to handle the second issue, we adopt the time–frequency attention with a frequency-attention network (F-Attention) and a time-attention network (T-Attention) to focus on the emotion-related long-range dependencies between frequency bands and across time frames, which can enhance the emotional discrimination of speech features. Extensive experimental results on three public emotional databases, i.e., IEMOCAP, ABC, and CASIA, show that our proposed ATFNN outperforms the state-of-the-art methods.
Overview
CoChat Summary
TL;DR
The paper proposes an attentive time–frequency neural network (ATFNN) for speech emotion recognition, combining frequency-domain and time-domain encoders with frequency and time attention. It reports state-of-the-art results on IEMOCAP, ABC, and CASIA.
Key findings
- ATFNN achieved the best WAR on IEMOCAP at 73.81%, with UAR 64.48%; ADARL had higher UAR at 65.86%.
- On ABC, ATFNN obtained WAR 68.84% and UAR 57.57%, outperforming LLDs + HMM/GMM, RGSR-LPDA, LLDs + SVM, and GerDA.
- On CASIA, ATFNN reached WAR and UAR of 48.75%, higher than LLD + DR, DNN + ELM, HuWSF, and DTPM.
- Ablation showed ATFNN (73.81% WAR on IEMOCAP) outperformed TFNN (72.99%), ATNN (71.76%), and AFNN (70.71%); attention visualizations indicated happy has increased high-frequency energy and sad decreased high-frequency energy.
Methods
- The ATFNN embeds time–frequency attention into a time–frequency neural network: an F-Encoder based on a Transformer encoder without position encoding, a T-Encoder based on a three-layer Bi-LSTM with 128 hidden nodes, plus F-Attention and T-Attention convolution blocks.
- Experiments used IEMOCAP (2,280 utterances, 4 emotions), ABC (430 utterances, 6 emotions), and CASIA (1,200 utterances, 6 emotions), with leave-one-speaker-out cross-validation and WAR/UAR metrics.
- Inputs were log-Mel-spectrograms extracted by STFT with 20 ms Hamming window, 50% overlap, 512-point FFT, and 80 Mel bands; utterances were divided into 128-frame segments. The model was trained from scratch in PyTorch on RTX3090 GPUs with batch size 128 and Adam learning rate 0.0005.
Limitations
- The paper states that precise correspondence between different emotions and frequency bands should be deeply explored in future research.
- The paper notes that IEMOCAP and ABC are class-imbalanced, causing WAR/UAR discrepancy and confusion, e.g., 58.27% of happy samples recognized as neutral in IEMOCAP and 45.45% of intoxicate samples recognized as cheerful in ABC.
- The paper states CASIA performance is much lower than IEMOCAP and ABC, possibly because CASIA forces speakers to express different emotions in the same sentence and is a tonal language; it also notes ADARL’s domain adaptation gives better UAR on IEMOCAP than ATFNN.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 13 Not checked · 46 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Another issue faced by the time-frequency representation of speech emotion is that the distribution of emotion information in an utterance is sparse in both frequency domain and time domain [19], [23].”
What the sources say
- Not checked yetcites [19] source
- Supportedcites [23] source
The paper says emotion may be detectable only at specific moments in a long utterance (opening summary, p. 1). Its Experiment results and analysis (p. 4) also describe low attention weights for some voice frames and high-frequency regions, supporting the claim that emotion-relevant information is unevenly distributed across time and frequency. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning”
What the sources say
- Could not verifycites [8] sourceExperimental — accuracy not yet measured
Two searches of Mao et al. (2014) returned no paper-scoped full-text snippets, so they cannot establish whether it makes this criticism. The wording appears in the abstract of a different SER paper, “Speech Emotion Recognition via an Attentive Time–Frequency Neural Network” (unscoped score 0.484). That match identifies a plausible origin for the wording, but it falls below the rubric’s misattribution threshold and does not rule out Mao et al. as a source for the underlying point. (LLM, conf=0.72)
- Not checked yetcites [13] source
Checked before — its earlier verdict is held back until a new check runs.
- Could not verifycites [17] sourceExperimental — accuracy not yet measured
No open-access snippets were returned for Li et al. (2015), “LSTM time and frequency recurrence for automatic speech recognition,” so its support for the claim cannot be checked. The quoted wording appears in “Speech Emotion Recognition via an Attentive Time–Frequency Neural Network” (Semantic Scholar score 0.484), a distinct SER paper. That match raises an attribution concern, but the cited paper’s unavailable text and the match’s score do not meet the threshold for a misattribution verdict. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“there are frequent energy activations of happy above 50 at the Mel-scaled frequency (corresponding to the frequency band around 2 700Hz), which demonstrates that happy has an increased in high-frequency energy.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“it also obtains suboptimal results on UAR (64.48 % ) than ADARL (65.86 % ) [32].”
What the sources say
- Not checked yetcites [32] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From these results, it is obvious that our proposed ATFNN achieves the state-of-the-art performance.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
30 more claims
Create a free account to see all 60 claims and to verify the 25 references still open.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 9
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Currently, a common approach involves using self-supervised training methods to learn high-quality representations from unlabeled data, and then using these representations to replace hand-crafted features to complete downstream tasks. This is a research hotspot in the field of speech emotion [49,50,51,52,53].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [50]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ D. Results and Analysis
“where AFNN and TFNN represent the FNN with F-Attention and the TNN with T-Attention respectively”
§ A. TFNN
“The TFNN aims to learn the high(er)-level time-frequency representation from the input spectrogram feature for SER, which includes two modules, i. e., F-Encoder and T-Encoder.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ IV. CONCLUSION
“Experimental results on the two public databases, i. g., IEMOCAP and ABC, prove that our ATFNN outperforms the state-of-the-art methods.”
§ Speech Emotion Recognition via an Attentive Time-Frequency Neural Network
“Extensive experimental results on three public emotional databases, i. e., IEMOCAP, ABC, and CASIA, show that the proposed ATFNN outperforms the state-of-the-art methods.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ C. Experimental Setting
“speech utterances are all divided into small segments with 128 frames (i. e., 20ms)”
§ C. Experimental Setting
“20 ms Hamming window size with 50% frame overlapping is adopted”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A. Experimental Databases
“Notably, in the paper, we adopt 1200 utterances of ABC released publicly for the experiment section.”
§ B. Experimental Protocol
“| ABC | German | aggressive (95), cheerful (105), intoxicate (33), nervous (93), neutral (79), tired (25) | 8 (4 males) | 430 | 16000 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Notably, in the paper, we adopt 1200 utterances of ABC released publicly for the experiment section. 4 volunteers with 2 males and 2 females are required to simulate these 6 emotions to produce 300 utterances for each emotion.”
Reported 1200 · recomputed 1800
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“…kernel size is 1 × 5 and the channel number is 8) and add-norm operations (i.e., residual connection operations followed by layer normalization [24]), the frequency-domain encoding ˆ x is obtained by integrating the learning of correlations within frequency bands into the time–frequency…”
BackgroundIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 1×
“Three public emotional speech databases, i.e., the interactive emotional dyadic motion capture database (IEMOCAP) [25], the airplane behavior corpus (ABC) [26], and China emotional database (CASIA) [27], [28], are adopted in our experiments to prove the effectiveness of the proposed ATFNN method.”
MethodThe emotion probe: Studies of motivation and attention.
Lang, P. J. (1995) · American Psychologist · doi:10.1037//0003-066x.50.5.372 · Cited by 2,746 · cited 1×
“For this purpose, high-arousal and positive-valence emotions, e.g., happy , cheerful , and low-arousal and negative-valence emotions, e.g., sad , tired , are selected as comparison for attention visualization [37], [38].”
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 2×
“For instance, Schuller et al. [11] integrated 6552 hand-crafted features into an open-source toolkit, i.e., open-EAR, for four affective recognition tasks and then extended the openEAR to a multifunctional and convenient toolkit for diverse speech-related tasks, i.e., openSMILE [12].”
BackgroundMethodEmotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678 · cited 1×
“The extraction of deep features is increasingly embedded in a time–frequency joint learning strategy to generate more effective time–frequency patterns of speech emotions [16], [17], [18], [19].”
BackgroundEmotion recognition based on physiological changes in music listening
Kim, J., & Andre, E. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.26 · Cited by 1,082 · cited 1×
“For this purpose, high-arousal and positive-valence emotions, e.g., happy , cheerful , and low-arousal and negative-valence emotions, e.g., sad , tired , are selected as comparison for attention visualization [37], [38].”
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794 · cited 1×
“…the experiments, including LLDs with a dimension reduction combining PCA and LDA (LLD + DR) [34], DNNs with the extreme learning machine (DNN_ELM) [35], weighted spectral feature learning based on local Hu moment (HuWSF) [36], and deep convolutional neural network with discriminant temporal…”
MethodAkçay, M. B., & Oğuz, K. (2020) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 721 · cited 2×
“RNNs, especially LSTM, have demonstrated their powerful capabilities in modeling the long-range dependence of the time series signal, and have been widely used in speech signal processing [13], [14], [15].”
BackgroundMao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568 · cited 6×
“…the extraction of deep features, generally, spectrogram features (e.g., MFCC, magnitude spectrogram, and Mel-spectrogram) with rich time– frequency information are often used as input features of DNNs to learn the high(er)-level time–frequency patterns of speech signal for SER [7], [8], [9], [13].”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Zhang, Design of speech corpus for Mandarin text to speech, 1
- 10.1037/0003-066x.50.5.372 · DOI · Paper page
Cited by
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
Self-citationWang, Y., Lu, C., Lian, H., Yan, Z., Schuller, B. W., Zong, Y., & Zheng, W. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447726 · Cited by 59
“This phenomenon is consistent with previous studies [17], that is low arousal emotions (e.g., sad ) commonly exhibit activations in the mid-low frequency range, demonstrating that our method can indeed obtain hierarchical speech emotion representations.”
MethodResultSelf-citationModeling Speech Emotion Recognition via Attention-Oriented Parallel CNN Encoders
Makhmudov, F., Kutlimuratov, A., Akhmedov, F., Abdallah, M. S., & Cho, Y. I. (2022) · Electronics · doi:10.3390/electronics11234047 · Cited by 35
AMGCN: An adaptive multi-graph convolutional network for speech emotion recognition
Lian, H., Lu, C., Chang, H., Yan, Z., Li, S., Li, Y., & Zong, Y. (2025) · Speech Communication · doi:10.1016/j.specom.2024.103184 · Cited by 19
Self-citationSTIDNet: Identity-Aware Face Forgery Detection With Spatiotemporal Knowledge Distillation
Fang, M., Yu, L., Xie, H., Tan, Q., Tan, Z., Hussain, A., Wang, Z., Li, J., & Tian, Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3356549 · Cited by 16
“W ITH the development of deep learning and generative adversarial networks [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], recent advancements in face manipulation techniques enable the creation of incredibly realistic fake videos (so-called deepfakes) [3], [4],…”
Background
Metadata
- Authors
- Lu, C., Zheng, W., Lian, H., Zong, Y., Tang, C., Li, S., & Zhao, Y.
- Affiliations
- Southeast University (CN)
- Venue
- IEEE Transactions on Computational Social Systems
- Published
- 2023-12
- Volume
- 10
- Issue
- 6
- Pages
- 3159-3168
- ISSN
- 2329-924X (ISSN-L)
- 2373-7476
- OpenAlex
- W4312857458
- Cited by
- 51
- References
- 38
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 10 · 2023: 6
- FWCI
- 8.98
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Computational Social Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSFC (U2003207, 61921004, 61902064)
- Jiangsu Frontier Technology Basic Research Project (BK20192004)
- Zhishan Young Scholarship of Southeast University
- Scientific Research Foundation of Graduate School of Southeast University (YBPY1955)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech emotion recognition, spectrogram, time-frequency attention, Transformer encoder, bidirectional LSTM, long-range dependencies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Speech Emotion Recognition via an Attentive Time–Frequency Neural Network. https://app.cochat.ai/papers/10.1109/tcss.2022.3219825
- The paper
- Lu, C., Zheng, W., Lian, H., Zong, Y., Tang, C., Li, S., & Zhao, Y. (2023). Speech Emotion Recognition via an Attentive Time–Frequency Neural Network. IEEE Transactions on Computational Social Systems. https://doi.org/10.1109/tcss.2022.3219825
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.