AffectAura: an intelligent system for emotional memory
Abstract
We present AffectAura, an emotional prosthetic that allows users to reflect on their emotional states over long periods of time. We designed a multimodal sensor set-up for continuous logging of audio, visual, physiological and contextual data, a classification scheme for predicting user affective state and an interface for user reflection. The system continuously predicts a user's valence, arousal and engage-ment, and correlates this with information on events, communications and data interactions. We evaluate the interface through a user study consisting of six users and over 240 hours of data, and demonstrate the utility of such a reflection tool. We show that users could reason forward and backward in time about their emotional experiences using the interface, and found this useful.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This is the first study, that the authors are aware of, that makes use of wearable, mobile sensors for detecting emotions.
“While implicit emotion detection has been done in the past [13], this is the first study, that we are aware of, that makes use of wearable, mobile sensors for detecting emotions.”
From Food and Mood: Just-in-Time Support for Emotional Eating (2013) · cites this paper as [13]
McDuff et al. found that the electrodermal activity (EDA) signal was the most beneficial for emotion detection.
“They found that the electrodermal activity (EDA) signal was the most beneficial, which is a measure of the eccrine sweat glands [14,15].”
From Food and Mood: Just-in-Time Support for Emotional Eating (2013) · cites this paper as [13]
The recognition accuracy achieved here is at par with other affect recognition systems.
“Furthermore, we would like to highlight that the recognition accuracy achieved here is at par with other affect recognition systems [13].”
From Food and Mood: Just-in-Time Support for Emotional Eating (2013) · cites this paper as [13]
Cites
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · see citing works
Active Shape Models-Their Training and Application
Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1004 · Cited by 7,190 · see citing works
Higgins, E. T. (1997) · American Psychologist · doi:10.1037/0003-066x.52.12.1280 · Cited by 6,775 · see citing works
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409 · see citing works
Handbook of Affective Sciences
(2002) · doi:10.1093/oso/9780195126013.001.0001 · Cited by 5,104 · see citing works
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A. W., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995316 · Cited by 3,545 · see citing works
The Massachusetts Institute of Technology
Endowed, S. C. F. (1999) · Anthropology News · doi:10.1111/an.1999.40.2.6 · Cited by 3,147 · see citing works
Memory: A Contribution to Experimental Psychology
Ebbinghaus, H. (2013) · Annals of Neurosciences · doi:10.5214/ans.0972.7531.200408 · Cited by 2,827 · see citing works
Neighbourhood Components Analysis
Goldberger, J., Hinton, G. E., Roweis, S. T., & Salakhutdinov, R. (2004) · Cited by 1,855 · see citing works
The Role of Affect in Decision Making
Loewenstein, G. F., & Lerner, J. S. (2002) · doi:10.1093/oso/9780195126013.003.0031 · Cited by 1,348 · see citing works
Boucsein, W. (1992) · doi:10.1007/978-1-4757-5093-5 · Cited by 818 · see citing works
A Wearable Sensor for Unobtrusive, Long-Term Assessment of Electrodermal Activity
Poh, M.-Z., Swenson, N. C., & Picard, R. W. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2009.2038487 · Cited by 693 · see citing works
Visualization of Time-Oriented Data
Aigner, W., Miksch, S., Schumann, H., & Tominski, C. (2011) · Human-Computer Interaction Series · doi:10.1007/978-0-85729-079-3 · Cited by 688 · see citing works
Life is Pleasant—and Memory Helps to Keep it that Way!
Walker, W. R., Skowronski, J. J., & Thompson, C. P. (2003) · Review of General Psychology · doi:10.1037/1089-2680.7.2.203 · Cited by 625 · see citing works
A novel demonstration of enhanced memory associated with emotional arousal
Cahill, L., & McGaugh, J. L. (1995) · Consciousness and Cognition · doi:10.1006/ccog.1995.1048 · Cited by 616 · see citing works
Nicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · see citing works
Emotional Reactivity to Everyday Problems, Affective Inertia, and Neuroticism
Suls, J. M., Green, P., & Hillis, S. L. (1998) · Personality and Social Psychology Bulletin · doi:10.1177/0146167298242002 · Cited by 421 · see citing works
Multimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101300 · Cited by 342 · see citing works
Self-citationBoehner, K., DePaula, R., Dourish, P., & Sengers, P. (2005) · Decennial Aarhus Conference · doi:10.1145/1094562.1094570 · Cited by 239 · see citing works
StartleCam: a cybernetic wearable camera
Healey, J. A., & Picard, R. W. (2002) · International Symposium on Wearable Computers · doi:10.1109/iswc.1998.729528 · Cited by 212 · see citing works
Viégas, F., Golder, S. A., & Donath, J. S. (2006) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1124772.1124919 · Cited by 207 · see citing works
Experiencing the Affective Diary
Ståhl, A., Höök, K., Svensson, M. G. A., Taylor, A. S., & Combetto, M. (2008) · Personal and Ubiquitous Computing · doi:10.1007/s00779-008-0202-7 · Cited by 183 · see citing works
On artificial bandwidth extension of telephone speech
Jax, P., & Vary, P. (2003) · Signal Processing · doi:10.1016/s0165-1684(03)00082-3 · Cited by 161 · see citing works
Examining task engagement in sensor-based statistical models of human interruptibility
Fogarty, J., Ko, A. J., Aung, H. H., Golden, E., Tang, K. P., & Hudson, S. E. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055018 · Cited by 111 · see citing works
A computational model for the automatic recognition of affect in speech
Fernandez, R. C., & Picard, R. W. (2004) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 72 · see citing works
Invited Talk: An Exploratory Social-Emotional Prosthetic for Autism Spectrum Disorders
el Kaliouby, R., Teeters, A., & Picard, R. W. (2006) · Wearable and Implantable Body Sensor Networks · doi:10.1109/bsn.2006.34 · Cited by 34 · see citing works
A personal, mobile system for understanding stress and interruptions
Liu, K. K.-L. (2004) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 27 · see citing works
Donath, J. S., Dragulescu, A., Zinman, A., et al. (2010) · doi:10.1145/1836786.1836793 · Cited by 25 · see citing works
Understanding memory triggers for task tracking
Brush, A. J. B., Meyers, B. R., Tan, D. S., & Czerwinski, M. P. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240768 · Cited by 21 · see citing works
Self-citationla Torre, F. D., & Agell, C. (2007) · doi:10.1109/icme.2007.4284781 · Cited by 9 · see citing works
Daily, S. B., & Picard, R. W. (2007) · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Using psycho-physiological measures to assess task difficulty in software development
Fritz, T., Begel, A., Müller, S. C., Yigit-Elliott, S., & Züger, M. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568266 · Cited by 262 · see citing works
Reflection Companion: A Conversational System for Engaging Users in Reflection on Physical Activity
Kocielnik, R., Xiao, L., Avrahami, D., & Hsieh, G. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3214273 · Cited by 221 · see citing works
Smart technologies for long-term stress monitoring at work
Kocielnik, R., Sidorova, N., Maggi, F. M., Ouwerkerk, M., & Westerink, J. H. D. M. (2013) · Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2013.6627764 · Cited by 154 · see citing works
Designing emotionally sentient agents
McDuff, D., & Czerwinski, M. (2018) · Communications of the ACM · doi:10.1145/3186591 · Cited by 132 · see citing works
Self-citationEMMA: an emotion-aware wellbeing chatbot
Ghandeharioun, A., McDuff, D., Czerwinski, M., & Rowan, K. (2019) · 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2019.8925455 · Cited by 126 · see citing works
Self-citationSpringer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115 · see citing works
“Our focus is on how users interact with intelligent personal informatics (PI) systems which are being increasingly deployed within commercial [86–88] and research domains [7,25,37,54,85].”
BackgroundProgressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information?
Springer, A., & Whittaker, S. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3374218 · Cited by 84 · see citing works
Food and Mood: Just-in-Time Support for Emotional Eating
Carroll, E. A., Czerwinski, M., Roseway, A., Kapoor, A., Johns, P., Rowan, K., & Schraefel, M. C. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.48 · Cited by 64 · see citing works
Self-citationDesign Recommendations for Self-Monitoring in the Workplace: Studies in Software Development
Meyer, A. N., Murphy, G. C., Zimmermann, T., & Fritz, T. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134714 · Cited by 60 · see citing works
On Being Told How We Feel: How Algorithmic Sensor Feedback Influences Emotion Perception
Hollis, V., Pekurovsky, A., Wu, E., & Whittaker, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264924 · Cited by 52 · see citing works
Brainatwork: logging cognitive engagement and tasks in the workplace using electroencephalography
Hassib, M., Khamis, M., Friedl, S., Schneegass, S., & Alt, F. (2017) · Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia · doi:10.1145/3152832.3152865 · Cited by 24 · see citing works
- Authors
- McDuff, D., Karlson, A., Kapoor, A., Roseway, A., & Czerwinski, M.
- Affiliations
- MIT Media Lab (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2012-05-05
- Pages
- 849-858
- OpenAlex
- W2121557515
- Cited by
- 242
- References
- 31
- Citations by year
- 2026: 6 · 2025: 11 · 2024: 13 · 2023: 3 · 2022: 14 · 2021: 21 · 2020: 20 · 2019: 22 · 2018: 20 · 2017: 17 · 2016: 23 · 2015: 27 · 2014: 22 · 2013: 21 · 2012: 2
- FWCI
- 44.53
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Keywords
- multimodal sensing, valence and arousal, engagement, user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
