The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Emotion recognition using deep learning approach from audio–visual emotional big data
Abstract
This paper proposes an emotion recognition system using a deep learning approach from emotional Big Data. The Big Data comprises of speech and video. In the proposed system, a speech signal is first processed in the frequency domain to obtain a Mel-spectrogram, which can be treated as an image. Then this Mel-spectrogram is fed to a convolutional neural network (CNN). For video signals, some representative frames from a video segment are extracted and fed to the CNN. The outputs of the two CNNs are fused using two consecutive extreme learning machines (ELMs). The output of the fusion is given to a support vector machine (SVM) for final classification of the emotions. The proposed system is evaluated using two audio–visual emotional databases, one of which is Big Data. Experimental results confirm the effectiveness of the proposed system involving the CNNs and the ELMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Extreme learning machine: Theory and applications
Huang, G.-B., Zhu, Q.-Y., & Siew, C.-K. (2006) · Neurocomputing · doi:10.1016/j.neucom.2005.12.126 · Cited by 13,285
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,290
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365
Multisensor data fusion: A review of the state-of-the-art
Khaleghi, B., Khamis, A., Karray, F. O., & Razavi, S. N. (2013) · Information Fusion · doi:10.1016/j.inffus.2011.08.001 · Cited by 2,074
Proceedings IEEE International Conference on Acoustics Speech and Signal Processing
Zheng, F., McLaughlin, S., & Mulgrew, B. (1991) · Cited by 1,905
Disease Prediction by Machine Learning Over Big Data From Healthcare Communities
Chen, M., Hao, Y., Hwang, K., Wang, L., & Wang, L. (2017) · IEEE Access · doi:10.1109/access.2017.2694446 · Cited by 1,311
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477450 · Cited by 1,086
Proceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003
Speech emotion recognition using deep neural network and extreme learning machine
Han, K., Yu, D., & Tashev, I. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-57 · Cited by 794
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615
Evaluating deep learning architectures for Speech Emotion Recognition
Fayek, H. M., Lech, M., & Cavedon, L. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.02.013 · Cited by 554
Facial expression recognition via learning deep sparse autoencoders
Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., & Dobaie, A. M. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.08.043 · Cited by 489
FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition
Ding, H., Zhou, S. K., & Chellappa, R. (2017) · 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) · doi:10.1109/fg.2017.23 · Cited by 416
EmoNets: Multimodal deep learning approaches for emotion recognition in video
Kahou, S. E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. (2016) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0195-2 · Cited by 412
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401
Sparse Autoencoder-Based Feature Transfer Learning for Speech Emotion Recognition
Deng, J., Zhang, Z., Marchi, E., & Schuller, B. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.90 · Cited by 366
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346
Emotion-aware connected healthcare big data towards 5G
Hossain, M. S., & Muhammad, G. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2017.2772959 · Cited by 273
Self-citationSpeech emotion recognition based on feature selection and extreme learning machine decision tree
Liu, Z.-T., Wu, M., Cao, W.-H., Mao, J.-W., Xu, J.-P., & Tan, G.-Z. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.07.050 · Cited by 267
Multimodal emotion recognition using deep learning architectures
Ranganathan, H., Chakraborty, S., & Panchanathan, S. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477679 · Cited by 249
Zhang, Y., Chen, M., Huang, D., Wu, D., & Li, Y. (2017) · Future Generation Computer Systems · doi:10.1016/j.future.2015.12.001 · Cited by 243
Cognitive Computing: Architecture, Technologies and Intelligent Applications
Chen, M., Herrera, F., & Hwang, K. (2018) · IEEE Access · doi:10.1109/access.2018.2791469 · Cited by 242
Cognitive Internet of Vehicles
Chen, M., Tian, Y., Fortino, G., Zhang, J., & Humar, I. (2018) · Computer Communications · doi:10.1016/j.comcom.2018.02.006 · Cited by 237
Video-based emotion recognition in the wild using deep transfer learning and score fusion
Kaya, H., Gürpınar, F., & Salah, A. A. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.01.012 · Cited by 235
Data-Driven Computing and Caching in 5G Networks: Architecture and Delay Analysis
Chen, M., Qian, Y., Hao, Y., Li, Y., & Song, J. (2018) · IEEE Wireless Communications · doi:10.1109/mwc.2018.1700216 · Cited by 195
Context-aware intelligent recommendation system for tourism
Meehan, K., Lunney, T., Curran, K., & McCaughey, A. (2013) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2013.6529508 · Cited by 159
Chen, M., Zhou, P., & Fortino, G. (2016) · IEEE Access · doi:10.1109/access.2016.2641480 · Cited by 159
A Facial-Expression Monitoring System for Improved Healthcare in Smart Cities
Muhammad, G., Alsulaiman, M., Amin, S. U., Ghoneim, A., & Alhamid, M. F. (2017) · IEEE Access · doi:10.1109/access.2017.2712788 · Cited by 149
Self-citationSPHA: Smart Personal Health Advisor Based on Deep Analytics
Chen, M., Zhang⋆, Y., Qiu, M., Guizani, N., & Hao, Y. (2018) · IEEE Communications Magazine · doi:10.1109/mcom.2018.1700274 · Cited by 142
AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and ChallengePreprint
Valstar, M., Gratch, J. S., Schuller, B. W., Ringeval, F., Lalanne, D., Torres, M. T., Scherer, S. J., Stratou, G., Cowie, R., & Pantić, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01600 · Cited by 133
Yogesh, C. K., Hariharan, M., Ngadiran, R., Adom, A. H., Yaacob, S., Berkai, C., & Polat, K. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.10.035 · Cited by 124
Multi-modal emotion analysis from facial expressions and electroencephalogram
Huang, X., Kortelainen, J., Zhao, G., Li, X., Moilanen, A., Seppänen, T., & Pietikäinen, M. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2015.09.015 · Cited by 122
Smart Home 2.0: Innovative Smart Home System Powered by Botanical IoT and Emotion Detection
Chen, M., Yang, J., Zhu, X., Wang, X., Liu, M., & Song, J. (2017) · Mobile Networks and Applications · doi:10.1007/s11036-017-0866-1 · Cited by 121
Menezes, M. L. R., Samara, A., Galway, L., Sant’Anna, A., Verikas, A., Alonso‐Fernandez, F., Wang, H., & Bond, R. (2017) · Personal and Ubiquitous Computing · doi:10.1007/s00779-017-1072-7 · Cited by 117
Audio-Visual Emotion Recognition Using Big Data Towards 5G
Hossain, M. S., Muhammad, G., Alhamid, M. F., Song, B., & Al-Mutib, K. N. (2016) · Mobile Networks and Applications · doi:10.1007/s11036-016-0685-9 · Cited by 100
Self-citationAn Emotion Recognition System for Mobile Applications
Hossain, M. S., & Muhammad, G. (2017) · IEEE Access · doi:10.1109/access.2017.2672829 · Cited by 100
Self-citationDeep Neural Networks with Relativity Learning for facial expression recognition
Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., & Tao, D. (2016) · 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew.2016.7574736 · Cited by 93
System Design for Big Data Application in Emotion-Aware Healthcare
Lin, K., Xia, F., Wang, W., Tian, D., & Song, J. (2016) · IEEE Access · doi:10.1109/access.2016.2616643 · Cited by 91
Learning emotion-based acoustic features with deep belief networks
Schmidt, E. M., & Kim, Y. E. (2011) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/aspaa.2011.6082328 · Cited by 89
Audio–Visual Emotion-Aware Cloud Gaming Framework
Hossain, M. S., Muhammad, G., Song, B., Hassan, M. M., Alelaiwi, A. A., & Alamri, A. M. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2444731 · Cited by 89
Self-citationBejani, M., Gharavian, D., & Charkari, N. M. (2012) · Neural Computing and Applications · doi:10.1007/s00521-012-1228-3 · Cited by 80
New approach in quantification of emotional intensity from the speech signal: emotional temperature
Alonso, J. B., Cabrera, J., Medina, M. L., & Travieso, C. M. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.07.062 · Cited by 75
Cloud-Based Collaborative Media Service Framework for HealthCare
Hossain, M. S., & Muhammad, G. (2014) · International Journal of Distributed Sensor Networks · doi:10.1155/2014/858712 · Cited by 72
Self-citationAudio-visual emotion recognition using multi-directional regression and Ridgelet transform
Hossain, M. S., & Muhammad, G. (2015) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-015-0207-2 · Cited by 69
Self-citationEmotion recognition from speech signals via a probabilistic echo-state network
Trentin, E., Scherer, S. J., & Schwenker, F. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.10.015 · Cited by 65
Deep learning and SVM‐based emotion recognition from Chinese speech for smart affective services
Zhang, W., Zhao, D., Chai, Z., Yang, L. T., Liu, X., Gong, F., & Yang, S. (2017) · Software Practice and Experience · doi:10.1002/spe.2487 · Cited by 50
A breakthrough in Speech emotion recognition using Deep Retinal Convolution Neural NetworksPreprint
Niu, Y., Zou, D., Niu, Y., He, Z., & Tan, H. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09917 · Cited by 48
Audio Visual Emotion Recognition Based on Triple-Stream Dynamic Bayesian Network Models
Jiang, D., Cui, Y., Zhang, X., Fan, P., Ganzalez, I., & Hichem Sahli, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_64 · Cited by 44
Second- and High-Order Graph Matching for Correspondence Problems
Zhang, R., & Wang, W. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2718225 · Cited by 38
User emotion recognition from a larger pool of social network data using active learning
Muhammad, G., & Alhamid, M. F. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3912-2 · Cited by 31
Self-citationGender-dependent emotion recognition based on HMMs and SPHMMs
Shahin, I. M. A. (2012) · International Journal of Speech Technology · doi:10.1007/s10772-012-9170-4 · Cited by 27
Face Expression Recognition Using Histograms of Oriented Gradients with Reduced Features
Kar, N. B., Babu, K. S., & Jena, S. K. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-981-10-2107-7_19 · Cited by 25
An intelligent framework for emotion aware e-healthcare support systems
Doctor, F., Karyotis, C., Iqbal, R., & James, A. E. (2016) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2016.7850044 · Cited by 23
An Integrated Emotion-Aware Framework for Intelligent Tutoring Systems
Harley, J. M., Lajoie, S. P., Frasson, C., & Hall, N. C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_75 · Cited by 22
Hossain, M. S., Muhammad, G., & Qurishi, M. A. (2017) · Mobile Networks and Applications · doi:10.1007/s11036-017-0928-4 · Cited by 17
Self-citation
Cited by
Convolutional neural network: a review of models, methodologies and applications to object detection
Dhillon, A., & Verma, G. K. (2020) · Progress in Artificial Intelligence · doi:10.1007/s13748-019-00203-0 · Cited by 1,119
Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.01.011 · Cited by 806
Artificial intelligence in the creative industries: a review
Anantrasirichai, N., & Bull, D. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10039-7 · Cited by 793
Speech Emotion Recognition Using Deep Learning Techniques: A Review
Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019) · IEEE Access · doi:10.1109/access.2019.2936124 · Cited by 702
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Metadata
- Authors
- Hossain, M. S., & Muhammad, G.
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2019-09
- Volume
- 49
- Pages
- 69-78
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W2918087949
- Cited by
- 453
- References
- 62
- FWCI
- 55.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.