Spatial–Temporal Recurrent Neural Network for Emotion Recognition
Open accessAbstract
In this paper, we propose a novel deep learning framework, called spatial-temporal recurrent neural network (STRNN), to integrate the feature learning from both spatial and temporal information of signal sources into a unified spatial-temporal dependency model. In STRNN, to capture those spatially co-occurrent variations of human emotions, a multidirectional recurrent neural network (RNN) layer is employed to capture long-range contextual cues by traversing the spatial regions of each temporal slice along different directions. Then a bi-directional temporal RNN layer is further used to learn the discriminative features characterizing the temporal dependencies of the sequences, where sequences are produced from the spatial RNN layer. To further select those salient regions with more discriminative ability for emotion recognition, we impose sparse projection onto those hidden states of spatial and temporal domains to improve the model discriminant ability. Consequently, the proposed two-layer RNN model provides an effective way to make use of both spatial and temporal dependencies of the input signals for emotion recognition. Experimental results on the public emotion datasets of electroencephalogram and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RNNs have been used to learn temporal context from EEG features to improve recognition accuracies.
“RNNs have been used to learn temporal context from EEG features to improve recognition accuracies [146], [55].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [146]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303 · see citing works
Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition
Jung, H., Lee, S., Yim, J., Park, S., & Kim, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.341 · Cited by 781 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
EEG Based Emotion Recognition: A Tutorial and Review
Li, X., Zhang, Y., Tiwari, P., Song, D., Hu, B., Yang, M., Zhao, Z., Kumar, N., & Marttinen, P. (2023) · ACM Computing Surveys · doi:10.1145/3524499 · Cited by 508 · see citing works
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240 · see citing works
Pan, B., Hirota, K., Jia, Z., & Dai, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126866 · Cited by 142 · see citing works
Chen, L., Wang, K., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3150097 · Cited by 113 · see citing works
A Multi-Domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition
Li, R., Wang, Y., & Lu, B.-L. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475697 · Cited by 73 · see citing works
Core-Brain-Network-Based Multilayer Convolutional Neural Network for Emotion Recognition
Gao, Z., Li, R., Ma, C., Rui, L., & Sun, X. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3090164 · Cited by 61 · see citing works
Chen, L., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3236320 · Cited by 53 · see citing works
IENet: a robust convolutional neural network for EEG based brain-computer interfaces
Du, Y., & Liu, J. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac7257 · Cited by 34 · see citing works
“RNNs have achieved great results in detecting SSVEP response [50], identifying motor imagery [33], predicting epileptic seizures [51], detecting sleep disorders [52] and recognizing emotion [53].”
BackgroundKim, H., Chae, Y., Kim, S., & Im, C.-H. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2022.3200394 · Cited by 17 · see citing works
- Authors
- Zhang, T., Zheng, W., Cui, Z., Zong, Y., & Li, Y.
- Affiliations
- Southeast University (CN)
- Nanjing University of Science and Technology (CN)
- Venue
- IEEE Transactions on Cybernetics
- Published
- 2019-03
- Volume
- 49
- Issue
- 3
- Pages
- 839-847
- ISSN
- 2168-2267 (ISSN-L)
- 2168-2275
- OpenAlex
- W2613375858
- Cited by
- 595
- References
- 5
- Citations by year
- 2026: 41 · 2025: 80 · 2024: 85 · 2023: 119 · 2022: 94 · 2021: 78 · 2020: 62 · 2019: 29 · 2018: 6 · 2017: 1
- FWCI
- 57.31
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Cybernetics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Basic Research Program of China (2015CB351704)
- National Natural Science Foundation of China (61231002, 61602244, 61772276)
- Key Research and Development Program of Jiangsu Province (BE2016616)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- emotion recognition, sparse projection, EEG, facial expressions
- MeSH terms
- Deep Learning, Electroencephalography, Emotions, Facial Expression, Female, Humans, Image Processing, Computer-Assisted, Male, Signal Processing, Computer-Assisted, Video Recording, Databases, Factual
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
