The DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Emotion Recognition Using Eye-Tracking: Taxonomy, Review and Current Challenges
Open accessAbstract
The ability to detect users’ emotions for the purpose of emotion engineering is currently one of the main endeavors of machine learning in affective computing. Among the more common approaches to emotion detection are methods that rely on electroencephalography (EEG), facial image processing and speech inflections. Although eye-tracking is fast in becoming one of the most commonly used sensor modalities in affective computing, it is still a relatively new approach for emotion detection, especially when it is used exclusively. In this survey paper, we present a review on emotion recognition using eye-tracking technology, including a brief introductory background on emotion modeling, eye-tracking devices and approaches, emotion stimulation methods, the emotional-relevant features extractable from eye-tracking data, and most importantly, a categorical summary and taxonomy of the current literature which relates to emotion recognition using eye-tracking. This review concludes with a discussion on the current open research problems and prospective future research directions that will be beneficial for expanding the body of knowledge in emotion detection using eye-tracking as the primary sensor modality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
Active Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190
PsychoPy—Psychophysics software in Python
Peirce, J. W. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2006.11.017 · Cited by 4,603
Baum, L. E., Petrie, T., Soules, G., & Weiss, N. (1970) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177697196 · Cited by 4,297
What are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264
Pagani, M., Lombardi, F., Guzzetti, S., Rimoldi, O., Furlan, R., Pizzinelli, P., Sandrone, G., Malfatto, G., Dell'Orto, S., & Piccaluga, E. (1986) · Circulation Research · doi:10.1161/01.res.59.2.178 · Cited by 4,130
Land, E. H., & McCann, J. J. (1971) · Journal of the Optical Society of America · doi:10.1364/josa.61.000001 · Cited by 4,095
Cohen, J., MacWhinney, B., Flatt, M., & Provost, J. (1993) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03204507 · Cited by 2,802
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Statistical Inference for Probabilistic Functions of Finite State Markov Chains
Baum, L. E., & Petrie, T. (1966) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177699147 · Cited by 2,775
Rayner, K. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210902816461 · Cited by 2,682
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Mauss, I. B., & Robinson, M. D. (2009) · Cognition and Emotion · doi:10.1080/02699930802204677 · Cited by 2,078
Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
White, B. W., & Rosenblatt, F. (1963) · The American Journal of Psychology · doi:10.2307/1419730 · Cited by 1,792
Selecting and interpreting measures of thematic classification accuracy
Stehman, S. V. (1997) · Remote Sensing of Environment · doi:10.1016/s0034-4257(97)00083-7 · Cited by 1,729
PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS
ROSENBLATT, F. (1961) · doi:10.21236/ad0256582 · Cited by 1,691
Eye Tracking in Human-Computer Interaction and Usability Research
Jacob, R. J. K., & Karn, K. S. (2003) · The Mind's Eye · doi:10.1016/b978-044451020-4/50031-1 · Cited by 1,567
EEG analysis based on time domain properties
Hjorth, B. (1970) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(70)90143-4 · Cited by 1,504
Plutchik, R. (2001) · American Scientist · doi:10.1511/2001.4.344 · Cited by 1,419
Contributions to Neuropsychological Assessment: A Clinical Manual
Benton, A. L. (1983) · Medical Entomology and Zoology · Cited by 1,409
Baum, L. E. (1972) · Medical Entomology and Zoology · Cited by 1,405
The keystroke-level model for user performance time with interactive systems
Card, S. K., Moran, T. P., & Newell, A. (1980) · Communications of the ACM · doi:10.1145/358886.358895 · Cited by 1,174
Neural Networks: A Review from a Statistical Perspective
Cheng, B., & Titterington, D. M. (1994) · Statistical Science · doi:10.1214/ss/1177010638 · Cited by 1,048
Baum, L. E., & Eagon, J. A. (1967) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1967-11751-8 · Cited by 1,036
EEG-Based Emotion Recognition in Music Listening
Lin, Y.-P., Wang, C.-H., Jung, T.-P., Wu, T.-L., Jeng, S.-K., Duann, J.-R., & Chen, J.-H. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2048568 · Cited by 1,021
A review of emotion recognition using physiological signals
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018) · Sensors · doi:10.3390/s18072074 · Cited by 923
Pupil Size as Related to Interest Value of Visual Stimuli
Hess, E. H., & Polt, J. M. (1960) · Science · doi:10.1126/science.132.3423.349 · Cited by 888
Pupil size variation as an indication of affective processing
Partala, T., & Surakka, V. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00017-x · Cited by 836
Oxytocin, vasopressin, and human social behavior
Heinrichs, M., von Dawans, B., & Domes, G. (2009) · Frontiers in Neuroendocrinology · doi:10.1016/j.yfrne.2009.05.005 · Cited by 799
Gilzenrat, M. S., Nieuwenhuis, S., Jepma, M., & Cohen, J. D. (2010) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.10.2.252 · Cited by 798
User Modeling in Human–Computer Interaction
Fischer, G. (2001) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1011145532042 · Cited by 732
Multimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722
Eye Movement Analysis for Activity Recognition Using Electrooculography
Bulling, A., Ward, J. A., Gellersen, H., & Tröster, G. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.86 · Cited by 718
Cabanac, M. (2002) · Behavioural Processes · doi:10.1016/s0376-6357(02)00078-5 · Cited by 652 · cited 1×
“Emotions are a mental state that are experienced by a human and is associated with feelings and a degree of pleasure or displeasure [11]. a degree of pleasure or displeasure [11].”
BackgroundBal, E., Harden, E., Lamb, D., Van Hecke, A. V., Denver, J. W., & Porges, S. W. (2010) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-009-0884-3 · Cited by 575
Chen, W., Er, M. J., & Wu, S. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.857353 · Cited by 560
Interpreting face images using active appearance models
Edwards, G. J., Taylor, C. J., & Cootes, T. F. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670965 · Cited by 484
Belin, P., Fillion-Bilodeau, S., & Gosselin, F. (2008) · Behavior Research Methods · doi:10.3758/brm.40.2.531 · Cited by 473
Prediction as a candidate for learning deep hierarchical models of data
Palm, R. (2012) · Cited by 444
Selective preference in visual fixation away from negative images in old age? An eye-tracking study.
Isaacowitz, D. M., Wadlinger, H. A., Goren, D., & Wilson, H. R. (2006) · Psychology and Aging · doi:10.1037/0882-7974.21.1.40 · Cited by 430
A comparison of three quantification methods for estimation of respiratory sinus arrhythmia.
Grossman, P., van Beek, J., & Wientjes, C. (1990) · Psychophysiology · doi:10.1111/j.1469-8986.1990.tb03198.x · Cited by 381
Growth transformations for functions on manifolds
Baum, L. E., & Sell, G. (1968) · Pacific Journal of Mathematics · doi:10.2140/pjm.1968.27.211 · Cited by 363
An empirical study of machine learning techniques for affect recognition in human–robot interaction
Rani, P., Liu, C., Sarkar, N., & Vanman, E. (2006) · Pattern Analysis and Applications · doi:10.1007/s10044-006-0025-y · Cited by 313
The application of eye-tracking technology in the study of autism
Boraston, Z., & Blakemore, S. (2007) · The Journal of Physiology · doi:10.1113/jphysiol.2007.133587 · Cited by 287
Zhang, Y.-D., Yang, Z.-J., Lu, H.-M., Zhou, X.-X., Phillips, P., Liu, Q.-M., & Wang, S.-H. (2016) · IEEE Access · doi:10.1109/access.2016.2628407 · Cited by 279
A Comparison of Two Process Tracing Methods for Choice Tasks
Lohse, G. L., & Johnson, E. J. (1996) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1996.0087 · Cited by 272
Multimodal emotion recognition using EEG and eye tracking data
Zheng, W.-L., Dong, B.-N., & Lu, B.-L. (2014) · 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/embc.2014.6944757 · Cited by 230
Lischke, A., Berger, C., Prehn, K., Heinrichs, M., Herpertz, S. C., & Domes, G. (2012) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2011.07.015 · Cited by 221
Impaired facial emotion recognition in early-onset right mesial temporal lobe epilepsy
Meletti, S., Benuzzi, F., Rubboli, G., Cantalupo, G., Stanzani Maserati, M., Nichelli, P., & Tassinari, C. A. (2003) · Neurology · doi:10.1212/wnl.60.3.426 · Cited by 198
Patterns of Visual Scanning as Predictors of Emotion Identification in Normal Aging.
Wong, B., Cronin-Golomb, A., & Neargarder, S. (2005) · Neuropsychology · doi:10.1037/0894-4105.19.6.739 · Cited by 196
Respiratory Sinus Arrhythmia: Physiological Basis, Quantitative Methods, and Clinical Implications
Porges, S. W. (1986) · Cardiorespiratory and Cardiosomatic Psychophysiology · doi:10.1007/978-1-4757-0360-3_7 · Cited by 189
Combining eye movements and EEG to enhance emotion recognition
Lu, Y., Zheng, W., Li, B., & Lu, B. (2015) · Cited by 184
Understanding eye movements in face recognition using hidden Markov models
Chuk, T., Chan, A. B., & Hsiao, J. H. (2014) · Journal of Vision · doi:10.1167/14.11.8 · Cited by 165
Choi, K.-H., Kim, J., Kwon, O. S., Kim, M. J., Ryu, Y. H., & Park, J.-E. (2017) · Psychiatry Research · doi:10.1016/j.psychres.2017.02.025 · Cited by 164
The contribution of different facial regions to the recognition of conversational expressions
Nusseck, M., Cunningham, D. W., Wallraven, C., & Bulthoff, H. H. (2008) · Journal of Vision · doi:10.1167/8.8.1 · Cited by 162
Rubin, D. C., & Talarico, J. M. (2009) · Memory · doi:10.1080/09658210903130764 · Cited by 161
A Deep Learning-Based Approach for Multi-Label Emotion Classification in Tweets
Jabreel, M., & Moreno, A. (2019) · Applied Sciences · doi:10.3390/app9061123 · Cited by 159
Pupil dilation reflects the time course of emotion recognition in human vocalizations
Oliva, M., & Anikin, A. (2018) · Scientific Reports · doi:10.1038/s41598-018-23265-x · Cited by 154
Diurnal variations in vagal and sympathetic cardiac control
Hayano, J., Sakakibara, Y., Yamada, M., Kamiya, T., Fujinami, T., Yokoyama, K., Watanabe, Y., & Takata, K. (1990) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.1990.258.3.h642 · Cited by 146
Gaze and Eye Tracking: Techniques and Applications in ADAS
Khan, M. Q., & Lee, S. (2019) · Sensors · doi:10.3390/s19245540 · Cited by 135
From affect programs to dynamical discrete emotions
Colombetti, G. (2009) · Philosophical Psychology · doi:10.1080/09515080903153600 · Cited by 131
An android for enhancing social skills and emotion recognition in people with autism
Pioggia, G., Igliozzi, R., Ferro, M., Ahluwalia, A. D., Muratori, F., & Rossi, D. D. (2005) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2005.856076 · Cited by 119
Autism, fever, epigenetics and the locus coeruleus
Mehler, M. F., & Purpura, D. P. (2008) · Brain Research Reviews · doi:10.1016/j.brainresrev.2008.11.001 · Cited by 118
Age-Related Deficits in Face Recognition are Related to Underlying Changes in Scanning Behavior
Firestone, A., Turk‐Browne, N. B., & Ryan, J. D. (2006) · Aging Neuropsychology and Cognition · doi:10.1080/13825580600899717 · Cited by 117
Bissoli, A. L. C., Lavino-Júnior, D., Sime, M. M., Encarnação, L. F., & Bastos-Filho, T. (2019) · Sensors · doi:10.3390/s19040859 · Cited by 116
Human Eye Tracking and Related Issues: A Review
Singh, H., & Singh, J. (2012) · Cited by 111
Sodhro, A. H., Sangaiah, A. K., Sodhro, G. H., Lohano, S., & Pirbhulal, S. (2018) · Sensors · doi:10.3390/s18030923 · Cited by 97
Visual scanning patterns and executive function in relation to facial emotion recognition in aging
Circelli, K. S., Clark, U. S., & Cronin‐Golomb, A. (2013) · Aging Neuropsychology and Cognition · doi:10.1080/13825585.2012.675427 · Cited by 95
Neural Networks for Emotion Recognition Based on Eye Tracking Data
Aracena, C., Basterrech, S., Snåšel, V., & Velásquez, J. D. (2015) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2015.460 · Cited by 69
Malcolm, G. L., Lanyon, L., Fugard, A. J. B., & Barton, J. J. S. (2008) · Journal of Vision · doi:10.1167/8.8.2 · Cited by 65
Tsang, K. L. V. 曾. (2016) · Autism · doi:10.1177/1362361316667830 · Cited by 63
Gómez‐Ibáñez, A., Urrestarazu, E., & Viteri, C. (2014) · Seizure · doi:10.1016/j.seizure.2014.08.012 · Cited by 62
Exploring Eye Activity as an Indication of Emotional States Using an Eye-Tracking Sensor
Alghowinem, S., Alshehri, M., Goecke, R., & Wagner, M. (2014) · Studies in computational intelligence · doi:10.1007/978-3-319-04702-7_15 · Cited by 59
Eye tracking and pupil size variation as response to affective stimuli: a preliminary study
Lanatà, A., Armato, A., Valenza, G., & Scilingo, E. P. (2011) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.4108/icst.pervasivehealth.2011.246056 · Cited by 55
Yannakakis, G. N., Karpouzis, K. C., Paiva, A., & Hudlická, E. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_62 · Cited by 54
Begum, S., Barua, S., & Ahmed, M. U. (2014) · Sensors · doi:10.3390/s140711770 · Cited by 54
Ulutas, B. H., Özkan, N. F., & Michalski, R. (2019) · Central European Journal of Operations Research · doi:10.1007/s10100-019-00628-x · Cited by 42
Automatic Emotion Perception Using Eye Movement Information for E-Healthcare Systems
Wang, Y., Lv, Z., & Zheng, Y. (2018) · Sensors · doi:10.3390/s18092826 · Cited by 37
Development of an intelligent e-healthcare system for the domestic care industry
Wong, B. C.-B., Ho, G. T. S., & Tsui, E. (2017) · Industrial Management & Data Systems · doi:10.1108/imds-08-2016-0342 · Cited by 34
Psychopathy and Physiological Detection of Concealed Information: A review
Verschuère, B., Crombez, G., Koster, E. H. W., & Uzieblo, K. (2006) · Psychologica Belgica · doi:10.5334/pb-46-1-2-99 · Cited by 33
Alhargan, A., Cooke, N. J., & Binjammaz, T. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3137016 · Cited by 33
Affect recognition in an interactive gaming environment using eye tracking
Alhargan, A., Cooke, N. J., & Binjammaz, T. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273614 · Cited by 28
Jänig, W. (2013) · Springer eBooks · doi:10.1007/978-3-642-10769-6_10 · Cited by 25
Using Eye Tracking to Assess Gaze Concentration in Meditation
Chang, K.-M., & Chueh, M.-T. W. (2019) · Sensors · doi:10.3390/s19071612 · Cited by 12
de Melo, C. M., Paiva, A., & Gratch, J. S. (2014) · Wiley eBooks · doi:10.1002/9781118796443.ch21 · Cited by 8
Cited by
Review of Eye Tracking Metrics Involved in Emotional and Cognitive Processes
Skaramagkas, V., Giannakakis, G., Ktistakis, E., Manousos, D., Karatzanis, I., Tachos, N., Tripoliti, E., Marias, K., Fotiadis, D. I., & Tsiknakis, M. (2023) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2021.3066072 · Cited by 419
Advances in the use of virtual reality to treat mental health conditions
Bell, I. H., Pot-Kolder, R., Rizzo, A., Rus-Calafell, M., Cardi, V., Cella, M., Ward, T., Riches, S., Reinoso, M., Thompson, A., et al. (2024) · Nature Reviews Psychology · doi:10.1038/s44159-024-00334-9 · Cited by 140
State-of-the-art Multimodal Emotion Recognition: A comprehensive survey and taxonomy
Yazici, A., Kucukyilmaz, T., Dokeroglu, T., Sharipbay, A., Lee, M.-H., & Tyler, B. (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200642 · Cited by 11
Metadata
- Authors
- Lim, J. Z., Mountstephens, J., & Teo, J.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2020-04-22
- Volume
- 20
- Issue
- 8
- Pages
- 2384
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W3020487153
- Cited by
- 293
- References
- 88
- FWCI
- 30.30
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.