Cognitive Heat: Exploring the Usage of Thermal Imaging to Unobtrusively Estimate Cognitive Load
Open accessThe DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
Abstract
Current digital systems are largely blind to users’ cognitive states. Systems that adapt to users’ states show great potential for augmenting cognition and for creating novel user experiences. However, most approaches for sensing cognitive states, and cognitive load specifically, involve obtrusive technologies, such as physiological sensors attached to users’ bodies. This paper present an unobtrusive indicator of the users’ cognitive load based on thermal imaging that is applicable in real-world. We use a commercial thermal camera to monitor a person’s forehead and nose temperature changes to estimate their cognitive load. To assess the effect of different levels of cognitive load on facial temperature we conducted a user study with 12 participants. The study showed that different levels of the Stroop test and the complexity of reading texts affect facial temperature patterns, thereby giving a measure of cognitive load. To validate the feasibility for real-time assessments of cognitive load, we conducted a second study with 24 participants, we analyzed the temporal latency of temperature changes. Our system detected temperature changes with an average latency of 0.7 seconds after users were exposed to a stimulus, outperforming latency in related work that used other thermal imaging techniques. We provide empirical evidence showing how to unobtrusively detect changes in cognitive load in real-time. Our exploration of exposing users to different content types gives rise to thermal-based activity tracking, which facilitates new applications in the field of cognition-aware computing.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Research using thermal imaging found promising results in distinguishing cognitive load levels using standardized cognitive load tests [2].”
From A Survey on Measuring Cognitive Workload in Human-Computer Interaction (2023) · cites this paper as [2]
Citations
Cites
Studies of interference in serial verbal reactions
Stroop, J. R. (1935) · Journal of Experimental Psychology · doi:10.1037/h0054651 · Cited by 18,622
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
The Computer for the 21st Century
Weiser, M. (1991) · Scientific American · doi:10.1038/scientificamerican0991-94 · Cited by 10,017
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Detecting Stress During Real-World Driving Tasks Using Physiological Sensors
Healey, J. A., & Picard, R. W. (2005) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2005.848368 · Cited by 2,207
Task-evoked pupillary responses, processing load, and the structure of processing resources.
Beatty, J. (1982) · Psychological Bulletin · doi:10.1037/0033-2909.91.2.276 · Cited by 1,959 · cited 1×
“Different approaches have been introduced to infer cognitive states ranging from using facial expressions [65], eye movements [18] and pupil size [6, 46], skin conductance [20, 32, 56], brain signals [8, 57, 58], Electrodermal activity (EDA) [17, 41, 42], heart and respiration rate [32].”
BackgroundLicklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
AUGMENTING HUMAN INTELLECT: A CONCEPTUAL FRAMEWORK
ENGELBART, D. C. (1962) · doi:10.21236/ad0289565 · Cited by 1,082
Studies of interference in serial verbal reactions.
Stroop, J. R. (1992) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.121.1.15 · Cited by 918
Thermal cameras and applications: A survey
Gade, R., & Moeslund, T. B. (2014) · Machine Vision and Applications · doi:10.1007/s00138-013-0570-5 · Cited by 780
Fundamentals of physiological computing
Fairclough, S. H. (2009) · Interacting with Computers · doi:10.1016/j.intcom.2008.10.011 · Cited by 608
Classification of factors influencing the use of infrared thermography in humans: A review
Fernández-Cuevas, I., Bouzas Marins, J. C., Arnáiz Lastras, J., Gómez Carmona, P. M., Piñonosa Cano, S., García-Concepción, M. Á., & Sillero-Quintana, M. (2015) · Infrared Physics & Technology · doi:10.1016/j.infrared.2015.02.007 · Cited by 586
Sharma, N., & Gedeon, T. (2012) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2012.07.003 · Cited by 510
Feeling and facial efference: Implications of the vascular theory of emotion.
Zajonc, R. B., Murphy, S. T., & Inglehart, M. (1989) · Psychological Review · doi:10.1037//0033-295x.96.3.395 · Cited by 439
Galvanic skin response (GSR) as an index of cognitive load
Shi, Y., Ruiz, N., Taib, R., Choi, E., & Chen, F. (2007) · CHI '07 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1240866.1241057 · Cited by 423
Coping with Stress: The Effectiveness of Different Types of Music
Labbé, E., Schmidt, N., Babin, J., & Pharr, M. (2007) · Applied Psychophysiology and Biofeedback · doi:10.1007/s10484-007-9043-9 · Cited by 382 · cited 2×
“used heart rate, respiration rate and GSR as an indicator [32].”
BackgroundMethodThermal infrared imaging in psychophysiology: Potentialities and limits
Ioannou, S., Gallese, V., & Merla, A. (2014) · Psychophysiology · doi:10.1111/psyp.12243 · Cited by 367
Cardiovascular differentiation of emotions.
Sinha, R., Lovallo, W. R., & Parsons, O. A. (1992) · Psychosomatic Medicine · doi:10.1097/00006842-199207000-00005 · Cited by 280
Mechanisms and Mediators of Psychological Stress-Induced Rise in Core Temperature
Oka, T., Oka, K., & Hori, T. (2001) · Psychosomatic Medicine · doi:10.1097/00006842-200105000-00018 · Cited by 271
Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity.
LEVENSON, R. W. (1988) · Cited by 236
StressCam: non-contact measurement of users' emotional states through thermal imaging
Puri, C., Olson, L., Pavlidis, I., Levine, J., & Starren, J. (2005) · CHI '05 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1056808.1057007 · Cited by 230
Stress detection from speech and Galvanic Skin Response signals
Kurniawan, H., Maslov, A. V., & Pechenizkiy, M. (2013) · Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2013.6627790 · Cited by 227
The use of nasal skin temperature measurements in studying emotion in macaque monkeys
Kuraoka, K., & Nakamura, K. (2010) · Physiology & Behavior · doi:10.1016/j.physbeh.2010.11.029 · Cited by 197
Evaluating a Brain-Computer Interface to Categorise Human Emotional Response
Crowley, K., Sliney, A., Pitt, I., & Murphy, D. (2010) · 2010 10th IEEE International Conference on Advanced Learning Technologies · doi:10.1109/icalt.2010.81 · Cited by 192
Sweller, J., Ayres, P., & Kalyuga, S. (2011) · Cognitive Load Theory · doi:10.1007/978-1-4419-8126-4_6 · Cited by 180
A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions
Pfleging, B., Fekety, D. K., Schmidt, A., & Kun, A. L. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858117 · Cited by 179
Self-citationThermal image analysis for polygraph testing
Pavlidis, I., & Levine, J. (2002) · IEEE Engineering in Medicine and Biology Magazine · doi:10.1109/memb.2002.1175139 · Cited by 177
Under pressure: sensing stress of computer users
Hernandez, J., Paredes, P., Roseway, A., & Czerwinski, M. (2014) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557165 · Cited by 167
Exploring the Use of Thermal Infrared Imaging in Human Stress Research
Engert, V., Merla, A., Grant, J. A., Cardone, D., Tusche, A., & Singer, T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090782 · Cited by 155
Or, C. K. L., & Duffy, V. G. (2007) · Occupational Ergonomics · doi:10.3233/oer-2007-7202 · Cited by 154
Assessing Cognitive Load with Physiological Sensors
Ikehara, C. S., & Crosby, M. E. (2005) · Proceedings of the 38th Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2005.103 · Cited by 150
Imaging Facial Signs of Neurophysiological Responses
Shastri, D., Merla, A., Tsiamyrtzis, P., & Pavlidis, I. (2009) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2008.2003265 · Cited by 147
A data set of real world driving to assess driver workload
Schneegass, S., Pfleging, B., Broy, N., Heinrich, F., & Schmidt, A. (2013) · Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications · doi:10.1145/2516540.2516561 · Cited by 135
Self-citationStay Cool! Understanding Thermal Attacks on Mobile-based User Authentication
Abdelrahman, Y., Khamis, M., Schneegaß, S., & Alt, F. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025461 · Cited by 134
Self-citationSweller, J. (2018) · Perspectives on Medical Education · doi:10.1007/s40037-017-0395-4 · Cited by 127
Thermal Signatures of Emotional Arousal: A Functional Infrared Imaging Study
Merla, A., & Romani, G. L. (2007) · 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2007.4352270 · Cited by 122
Manini, B., Cardone, D., Ebisch, S. J. H., Bafunno, D., Aureli, T., & Merla, A. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00299 · Cited by 121
The Autonomic Signature of Guilt in Children: A Thermal Infrared Imaging Study
Ioannou, S., Ebisch, S., Aureli, T., Bafunno, D., Ioannides, H. A., Cardone, D., Manini, B., Romani, G. L., Gallese, V., & Merla, A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0079440 · Cited by 117
Fast by Nature - How Stress Patterns Define Human Experience and Performance in Dexterous Tasks
Pavlidis, I., Tsiamyrtzis, P., Shastri, D., Wesley, A., Zhou, Y., Lindner, P., Buddharaju, P., Joseph, R., Mandapati, A., Dunkin, B., et al. (2012) · Scientific Reports · doi:10.1038/srep00305 · Cited by 111
Effect of head skin temperature on tympanic and oral temperature in man
McCaffrey, T. V., McCook, R. D., & Wurster, R. D. (1975) · Journal of Applied Physiology · doi:10.1152/jappl.1975.39.1.114 · Cited by 111
Thermal Facial Analysis for Deception Detection
Rajoub, B. A., & Reyer Zwiggelaar, A. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2317309 · Cited by 111
Symbiotic Interaction: A Critical Definition and Comparison to other Human-Computer Paradigms
Jacucci, G., Spagnolli, A., Freeman, J., & Gamberini, L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-13500-7_1 · Cited by 89
Perinasal Imaging of Physiological Stress and Its Affective Potential
Shastri, D., Papadakis, M., Tsiamyrtzis, P., Bass, B., & Pavlidis, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.13 · Cited by 89 · cited 1×
“We particularly build upon previous work that assessed stress based on the variations in forehead and nose temperature [11, 44, 55].”
BackgroundClassification of EEG signals using different feature extraction techniques for mental-task BCI
Hosni, S. I., Gadallah, M., Bahgat, S. F., & AbdelWahab, M. S. (2007) · International Conference on Computer Engineering & Systems (ICCES) · doi:10.1109/icces.2007.4447052 · Cited by 78
HeatWave: thermal imaging for surface user interaction
Larson, E. C., Cohn, G. A., Gupta, S., Ren, X., Harrison, B. L., Fox, D., & Patel, S. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979317 · Cited by 78
Augmenting Human Intellect and Amplifying Perception and Cognition
Schmidt, A. (2017) · IEEE Pervasive Computing · doi:10.1109/mprv.2017.8 · Cited by 71
Self-citationNew technological windows into mind: there is more in eyes and brains for human-computer interaction
Velichkovsky, B. M., & Hansen, J. P. (1996) · Proceedings of the SIGCHI conference on Human factors in computing systems common ground - CHI '96 · doi:10.1145/238386.238619 · Cited by 70
Forehead Thermal Signature Extraction in Lie Detection
Zhu, Z., Tsiamyrtzis, P., & Pavlidis, I. T. (2007) · Conference proceedings · doi:10.1109/iembs.2007.4352269 · Cited by 68
Chill-Out: Relaxation Training through Respiratory Biofeedback in a Mobile Casual Game
Parnandi, A., Ahmed, B., Shipp, E. M., & Gutiérrez‐Osuna, R. (2014) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-319-05452-0_18 · Cited by 59
Development of a skin temperature measuring system for non-contact stress evaluation
Kataoka, H., Kano, H., Yoshida, H., Saijo, A., Yasuda, M., & Osumi, M. (2002) · Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Vol.20 Biomedical Engineering Towards the Year 2000 and Beyond (Cat. No.98CH36286) · doi:10.1109/iembs.1998.745598 · Cited by 56
Smartphones Get Emotional: Mind Reading Images and Reconstructing the Neural Sources
Petersen, M. K., Stahlhut, C., Stopczynski, A., Larsen, J. E., & Hansen, L. K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_72 · Cited by 51
Thermal Imaging as a Way to Classify Cognitive Workload
Stemberger, J., Allison, R. S., & Schnell, T. (2010) · 2010 Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2010.37 · Cited by 46
Goyal, N., & Fussell, S. R. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130917 · Cited by 46
Stop helping me - I'm bored!: why assembly assistance needs to be adaptive
Funk, M., Dingler, T., Cooper, J., & Schmidt, A. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers - UbiComp '15 · doi:10.1145/2800835.2807942 · Cited by 45
Self-citationA Control-Theoretic Approach to Adaptive Physiological Games
Parnandi, A., Son, Y., & Gutiérrez‐Osuna, R. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.8 · Cited by 43
Exploiting thermal reflection for interactive systems
Shirazi, A. S., Abdelrahman, Y., Henze, N., Schneegaß, S., Khalilbeigi, M., & Schmidt, A. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557208 · Cited by 38
Self-citationInvestigation of Material Properties for Thermal Imaging-Based Interaction
Abdelrahman, Y., Shirazi, A. S., Henze, N., & Schmidt, A. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702290 · Cited by 34
Self-citationElKomy, M., Abdelrahman, Y., Funk, M., Dingler, T., Schmidt, A., & Abdennadher, S. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053179 · Cited by 32
Self-citationAbdelrahman, Y., Knierim, P., Woźniak, P. W., Henze, N., & Schmidt, A. (2017) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3123024.3129269 · Cited by 28
Self-citationSpagnolli, A., Freeman, J., Gamberini, L., & Jacucci, G. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-13500-7 · Cited by 25
The effect of face fanning during recovery from exercise hyperthermia
Germain, M., Jobin, M., & Cabanac, M. (1987) · Canadian Journal of Physiology and Pharmacology · doi:10.1139/y87-017 · Cited by 20
DETERMINING LEARNING LEVEL AND EFFECTIVE TRAINING TIMES USING THERMOGRAPHY
Kang, J., McGinley, J., McFadyen, G., & Babski-Reeves, K. L. (2006) · Cited by 18
Shirazi, A. S., Hassib, M. E., Henze, N., Schmidt, A., & Kunze, K. S. (2014) · Augmented Human International Conference · doi:10.1145/2582051.2582096 · Cited by 17
Self-citationJenkins, S. D., & Brown, R. D. H. (2014) · International Conference on Quantitative InfraRed Thermography (QIRT) · doi:10.21611/qirt.2014.131 · Cited by 17
MediaBrain: Annotating Videos based on Brain-Computer Interaction
Shirazi, A. S., Funk, M., Pfleiderer, F., Glück, H., & Schmidt, A. (2012) · Oldenbourg Wissenschaftsverlag eBooks · doi:10.1524/9783486718782.263 · Cited by 15
Self-citationSnake view: exploring thermal imaging as a vision extender in mountains
Abdelrahman, Y., Schmidt, A., & Knierim, P. (2017) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3123024.3124450 · Cited by 15
Self-citationMobile Computing, Applications, and Services
Sigg, S., Nurmi, P., & Salim, F. D. (2015) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-319-29003-4 · Cited by 14
A framework of personal assistant for computer users by analyzing video stream
Wang, Z., Yan, J., & Aghajan, H. K. (2012) · Workshop on Eye Gaze in Intelligent Human Machine Interaction (GazeIn) · doi:10.1145/2401836.2401850 · Cited by 7
Detecting Mental Workload Fluctuation during Learning of a Novel Task Using Thermography
Kang, J., & Babski-Reeves, K. L. (2008) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120805201947 · Cited by 5
Detecting mental workload fluctuation during learning of a novel task using thermography
Kang, J., & Babski-Reeves, K. L. (2008) · PsycEXTRA Dataset · doi:10.1037/e578262012-047 · Cited by 2
Research works printed but not linked: 4
- Yomna Abdelrahman. 2013. Thermal imaging for interactive surfaces. Master’s thesis. Yomna Abdelrahman. 2013. Thermal imaging for interactive surfaces. Master’s thesis.
- Hosni Sarah M, 2007, International Conference on. IEEE, 220--226
- Kang J, 2006, Proceedings of Army Science Conference
- 10.5555/2062850.2062929 · doi:10.5555/2062850.2062929
Cited by
A Review of Psychophysiological Measures to Assess Cognitive States in Real-World Driving
Lohani, M., Payne, B. R., & Strayer, D. L. (2019) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2019.00057 · Cited by 348
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
Self-citationNext Steps for Human-Computer Integration
Mueller, F. F., Lopes, P., Strohmeier, P., Ju, W., Seim, C., Weigel, M., Nanayakkara, S., Obrist, M., Li, Z., Delfa, J., et al. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376242 · Cited by 260
“Furthermore, technology has the potential to enhance our sensory system, for example, by extending the abilities of the visual sensation through wearing thermal imaging glasses [1] or by a brain-computer interface that allows accessing other people’s indicated cognitive load [26].”
BackgroundMultimodal Data Fusion in Learning Analytics: A Systematic Review
Mu, S., Cui, M., & Huang, X. (2020) · Sensors · doi:10.3390/s20236856 · Cited by 154
Multimodal data as a means to understand the learning experience
Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.02.003 · Cited by 202
Self-citation
Metadata
- Authors
- Abdelrahman, Y., Velloso, E., Dingler, T., Schmidt, A., & Vetere, F.
- Affiliations
- University of Stuttgart (DE)
- The University of Melbourne (AU)
- Osaka Prefecture University (JP)
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2017-09-11
- Volume
- 1
- Issue
- 3
- Pages
- 1-20
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W2755506150
- Cited by
- 154
- References
- 70
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 14 · 2023: 16 · 2022: 16 · 2021: 21 · 2020: 21 · 2019: 28 · 2018: 15 · 2017: 4
- FWCI
- 22.99
- Type
- journal article
- Language
- en
- Open access
- Locations
- Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion)
- Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, other-oa)
- Funders
- German Research Foundation within the SimTech Cluster of Excellence (EXC 310/2)
- Amplify project which received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (683008)
- Victorian State Government and Microsoft through their contributions to the Microsoft Research Centre for Social Natural User Interfaces
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- cognitive load, infrared thermography, unobtrusive sensing, forehead temperature, nose temperature, Stroop task, activity monitoring, thermal camera
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.