The DOI matches its Crossref record, with a small difference in title or year; 69 of the 69 references it cites were found in a registry or an index.
Deep Pain: Exploiting Long Short-Term Memory Networks for Facial Expression Classification
Open accessAbstract
Pain is an unpleasant feeling that has been shown to be an important factor for the recovery of patients. Since this is costly in human resources and difficult to do objectively, there is the need for automatic systems to measure it. In this paper, contrary to current state-of-the-art techniques in pain assessment, which are based on facial features only, we suggest that the performance can be enhanced by feeding the raw frames to deep learning models, outperforming the latest state-of-the-art results while also directly facing the problem of imbalanced data. As a baseline, our approach first uses convolutional neural networks (CNNs) to learn facial features from VGG_Faces, which are then linked to a long short-term memory to exploit the temporal relation between video frames. We further compare the performances of using the so popular schema based on the canonically normalized appearance versus taking into account the whole image. As a result, we outperform current state-of-the-art area under the curve performance in the UNBC-McMaster Shoulder Pain Expression Archive Database. In addition, to evaluate the generalization properties of our proposed methodology on facial motion recognition, we also report competitive results in the Cohn Kanade+ facial expression database.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“More advanced face frontalization involving the approximation of 3D shape is useful when dealing with 3D head pose variation [104], [105].”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [105]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,330
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,313
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 6,122
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Principal warps: thin-plate splines and the decomposition of deformations
Bookstein, F. L. (1989) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.24792 · Cited by 4,906
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
Multi-column deep neural networks for image classification
Ciresan, D. C., Meier, U., & Schmidhuber, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248110 · Cited by 3,880
Generalized procrustes analysis
Gower, J. C. (1975) · Psychometrika · doi:10.1007/bf02291478 · Cited by 3,267
Comprehensive database for facial expression analysis
Takeo Kanade, A., Cohn, J. F., & Tian, Y. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840611 · Cited by 2,644
Benjamin, S. (1994) · Journal of Psychosomatic Research · doi:10.1016/0022-3999(94)90012-4 · Cited by 2,001
Gradient Flow in Recurrent Nets: The Difficulty of Learning LongTerm Dependencies
(2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544037.ch14 · Cited by 1,312
The Checklist Manifesto: How to Get Things Right
Gawande, A. A. (2011) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(15)30310-0 · Cited by 1,213
Going deeper in facial expression recognition using deep neural networks
Mollahosseini, A., Chan, D., & Mahoor, M. H. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477450 · Cited by 1,155
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & de la Torre, F. N. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.47 · Cited by 828
Facial Expression Recognition via a Boosted Deep Belief Network
Liu, P., Han, S., Meng, Z., & Tong, Y. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.233 · Cited by 667
Painful data: The UNBC-McMaster shoulder pain expression archive database
Lucey, P., Cohn, J. F., Prkachin, K. M., Solomon, P. E., & Matthews, I. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771462 · Cited by 634
Consequences of Inadequate Postoperative Pain Relief and Chronic Persistent Postoperative Pain
Joshi, G. P., & Ogunnaike, B. O. (2005) · Anesthesiology Clinics of North America · doi:10.1016/j.atc.2004.11.013 · Cited by 620
Multi-view Face Detection Using Deep Convolutional Neural Networks
Farfade, S., Saberian, M. J., & Li, L.-J. (2015) · International Conference on Multimedia Retrieval · doi:10.1145/2671188.2749408 · Cited by 591
Prkachin, K. M., & Solomon, P. E. (2008) · Pain · doi:10.1016/j.pain.2008.04.010 · Cited by 472
Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition
Liu, M., Shan, S., Wang, R., & Chen, X. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.226 · Cited by 392
Learning active facial patches for expression analysis
Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., & Metaxas, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247974 · Cited by 380
The painful face – Pain expression recognition using active appearance models
Ashraf, A. B., Lucey, S., Cohn, J. F., Chen, T., Ambadar, Z., Prkachin, K. M., & Solomon, P. E. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2009.05.007 · Cited by 355
Deeply Learning Deformable Facial Action Parts Model for Dynamic Expression Analysis
Liu, M., Li, S., Shan, S., Wang, R., & Chen, X. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_10 · Cited by 333
Minority cancer patients and their providers
Anderson, K. O., Mendoza, T. R., Valero, V., Richman, S. P., Russell, C. A., Hurley, J. E., DeLeon, C., Washington, P., Palos, G. R., Payne, R., et al. (2000) · Cancer · doi:10.1002/(sici)1097-0142(20000415)88:8 · Cited by 333
Peak-Piloted Deep Network for Facial Expression Recognition
Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N. M., & Yan, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_27 · Cited by 317
Fully Automatic Recognition of the Temporal Phases of Facial Actions
Valstar, M., & Pantić, M. (2011) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2011.2163710 · Cited by 307
Automatically Detecting Pain in Video Through Facial Action Units
Lucey, P., Cohn, J. F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., & Prkachin, K. M. (2011) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2082525 · Cited by 267
The facial expression of pain.
Craig, K. D., Prkachin, K. M., & GRUNAU, R. E. (2001) · Cited by 259
Continuous Pain Intensity Estimation from Facial Expressions
Kaltwang, S., Rudovic, O., & Pantic, M. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33191-6_36 · Cited by 245
AU-aware Deep Networks for facial expression recognition
Liu, M., Li, S., Shan, S., & Chen, X. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553734 · Cited by 229
Automatic coding of facial expressions displayed during posed and genuine pain
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.12.010 · Cited by 216
Brown, J. E., Chatterjee, N. R., Younger, J., & Mackey, S. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024124 · Cited by 199
Automatic detection of pain intensity
Hammal, Z., & Cohn, J. F. (2012) · Proceedings of the 14th ACM international conference on Multimodal interaction · doi:10.1145/2388676.2388688 · Cited by 185
Automatically detecting pain using facial actions
Lucey, P., Cohn, J. F., Lucey, S., Matthews, I., Sridharan, S., & Prkachin, K. M. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349321 · Cited by 126
Recurrent Convolutional Neural Network Regression for Continuous Pain Intensity Estimation in Video
Zhou, J. L., Hong, X., Su, F., & Zhao, G. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.191 · Cited by 125
Facial Expression Intensity Estimation Using Ordinal Information
Zhao, R., Gan, Q., Wang, S., & Ji, Q. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.377 · Cited by 113
Context-Sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units
Rudovic, O. O., Pavlović, V., & Pantić, M. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2356192 · Cited by 111
LOMo: Latent Ordinal Model for Facial Analysis in Videos
Sikka, K., Sharma, G., & Bartlett, M. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.602 · Cited by 94
Rudovic, O. O., Pavlović, V., & Pantić, M. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247983 · Cited by 88
Pain detection through shape and appearance features
Khan, R. A., Meyer, A., Konik, H., & Bouakaz, S. (2013) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2013.6607608 · Cited by 72
Deep Learning of Representations
Bengio, Y., & Courville, A. C. (2013) · Intelligent systems reference library · doi:10.1007/978-3-642-36657-4_1 · Cited by 66
Learning Pain from Emotion: Transferred HoT Data Representation for Pain Intensity Estimation
Florea, C., Florea, L., & Vertan, C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16199-0_54 · Cited by 65
Weakly supervised pain localization using multiple instance learning
Sikka, K., Dhall, A., & Bartlett, M. S. (2013) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2013.6553762 · Cited by 62
Classification and weakly supervised pain localization using multiple segment representation
Sikka, K., Dhall, A., & Bartlett, M. S. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.02.008 · Cited by 58
Automatic Pain Intensity Estimation with Heteroscedastic Conditional Ordinal Random Fields
Rudovic, O. O., Pavlović, V., & Pantić, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41939-3_23 · Cited by 57
Structured Output Ordinal Regression for Dynamic Facial Emotion Intensity Prediction
Kim, M., & Pavlović, V. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15558-1_47 · Cited by 55
Deep Temporal Appearance-Geometry Network for Facial Expression RecognitionPreprint
Jung, H., Lee, S., Park, S., Lee, I., Ahn, C., & Kim, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01532 · Cited by 48
Social science and the study of pain since Zborowski: A need for a new agenda
Encandela, J. (1993) · Social Science & Medicine · doi:10.1016/0277-9536(93)90039-7 · Cited by 47
A novel approach for pain intensity detection based on facial feature deformations
Rathee, N., & Ganotra, D. (2015) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2015.09.007 · Cited by 46
Spatiotemporal analysis of RGB-D-T facial images for multimodal pain level recognition
Irani, R., Nasrollahi, K., Simon, M. O., Corneanu, C. A., Escalera, S., Bahnsen, C., Lundtoft, D. H., Moeslund, T. B., Pedersen, T. L., Klitgaard, M.-L., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301341 · Cited by 42
Self-citationPain Intensity Evaluation through Facial Action Units
Zafar, Z., & Khan, N. A. (2014) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2014.803 · Cited by 41
Agitation and pain assessment using digital imaging
Gholami, B., Haddad, W. M., & Tannenbaum, A. (2009) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2009.5332437 · Cited by 39
Pain recognition using spatiotemporal oriented energy of facial muscles
Irani, R., Nasrollahi, K., & Moeslund, T. B. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301340 · Cited by 38
Self-citationPain detection from facial characteristics using supervised descent method
Neshov, N. N., & Manolova, A. H. (2015) · IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS) · doi:10.1109/idaacs.2015.7340738 · Cited by 36
Ming, Z., Bugeau, A., Rouas, J.-L., & Shochi, T. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7284870 · Cited by 35
Pedersen, H. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20904-3_12 · Cited by 31
Using Hankel matrices for dynamics-based facial emotion recognition and pain detection
Presti, L. L., & Cascia, M. L. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301351 · Cited by 24
Automatically detecting action units from faces of pain: Comparing shape and appearance features
Lucey, P., Cohn, J. F., Lucey, S., Sridharan, S., & Prkachin, K. M. (2009) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2009.5204279 · Cited by 22
Flagged first, then by citations
Research works printed but not linked: 1
- Ekman, 2002, Facial Action Coding System—The Manual on CD-ROM
Cited by
Sorted by citations
Facial Expression Analysis under Partial Occlusion
Zhang, L., Verma, B., Tjondronegoro, D., & Chandran, V. (2018) · ACM Computing Surveys · doi:10.1145/3158369 · Cited by 1,338
Khare, S. K., Blanes‐Vidal, V., Nadimi, E. S., & Acharya, U. R. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102019 · Cited by 508
Attention mechanism-based CNN for facial expression recognition
Li, J., Jin, K., Zhou, D., Kubota, N., & Ju, Z. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.014 · Cited by 368
Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
Ahmedt‐Aristizabal, D., Armin, M. A., Denman, S., Fookes, C., & Petersson, L. (2021) · Sensors · doi:10.3390/s21144758 · Cited by 298
Automatic Recognition Methods Supporting Pain Assessment: A Survey
Werner, P., Lopez-Martinez, D., Walter, S., Al-Hamadi, A. K., Gruss, S., & Picard, R. W. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2946774 · Cited by 262
Metadata
- Authors
- Rodriguez, P., Cucurull, G., Gonzalez, J., Gonfaus, J. M., Nasrollahi, K., Moeslund, T. B., & Roca, F. X.
- Affiliations
- Computer Vision Center, Universitat Autònoma de Barcelona, Barcelona, Spain
- Universitat Autònoma de Barcelona (ES)
- Aalborg University (DK)
- Venue
- IEEE Transactions on Cybernetics
- Published
- 2022-05
- Volume
- 52
- Issue
- 5
- Pages
- 3314-3324
- ISSN
- 2168-2267 (ISSN-L)
- 2168-2275
- OpenAlex
- W2587128043
- Cited by
- 297
- References
- 69
- Citations by year
- 2026: 12 · 2025: 28 · 2024: 33 · 2023: 32 · 2022: 26
- FWCI
- 43.48
- Type
- journal article
- Language
- en
- Open access
- green
- vbn.aau.dk
- Licence
- Locations
- IEEE Transactions on Cybernetics (publishedVersion)
- PubMed (publishedVersion)
- VBN Forskningsportal (Aalborg Universitet) (acceptedVersion, other-oa)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, public-domain)
- Funders
- Spanish Project (TIN2015-65464-R)
- Generalitat de Catalunya (2016FI_B 01163)
- COST Action IC1307 iV&L Net (European Network on Integrating Vision and Language) through COST
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Keywords
- pain assessment, facial expression recognition, long short-term memory networks, deep pain, video sequences, temporal modeling, class imbalance
- MeSH terms
- Emotions, Facial Expression, Humans, Memory, Short-Term, Pain, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
