Understanding Human Reactions Looking at Facial Microexpressions With an Event Camera
Abstract
With the establishment ofIndustry 4.0, machines are now required to interact with workers. By observing biometrics they can assess if humans are authorized, or mentally and physically fit to work. Understanding body language, makes human–machine interaction more natural, secure, and effective. Nonetheless, traditional cameras have limitations; low frame rate and dynamic range hinder a comprehensive human understanding. This poses a challenge, since faces undergo frequent instantaneous microexpressions. In addition, this is privacy-sensitive information that must be protected. We propose to model expressions with event cameras, bio-inspired vision sensors that have found application within the Industry 4.0 scope. They capture motion at millisecond rates and work under challenging conditions like low illumination and highly dynamic scenes. Such cameras are also privacy-preserving, making them extremely interesting for industry. We show that using event cameras, we can understand human reactions by only observing facial expressions. Comparison with red-green-blue (RGB)-based modeling demonstrates improved effectiveness and robustness.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Event cameras capture only brightness changes in the scene, not detailed visual images of people, and are often considered privacy-preserving because they eliminate detailed visual private data such as face details.
“Event cameras are often considered privacy-preserving [7], [35] because they inherently eliminate detailed visual private data such as face details.”
From Event Anonymization: Privacy-Preserving Person Re-Identification and Pose Estimation in Event-Based Vision (2024) · cites this paper as [35]
Cites
A Multi-Modal Distributed Real-Time IoT System for Urban Traffic Control (Invited Paper)
Khanam, Z., Achari, V. P. S., Boukhennoufa, I., et al. (2024) · Durham Research Online (Durham University) · doi:10.4230/oasics.ng-res.2024.2 · Cited by 14,048 · see citing works
A discriminatively trained, multiscale, deformable part model
Felzenszwalb, P. F., McAllester, D., & Ramanan, D. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587597 · Cited by 2,911 · see citing works
Gallego, G., Delbruck, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutenegger, S., Davison, A. J., Conradt, J., Daniilidis, K., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3008413 · Cited by 2,566 · see citing works
An HOG-LBP human detector with partial occlusion handling
Wang, X., Han, T. X., & Yan, S. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459207 · Cited by 1,727 · see citing works
Bulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464 · see citing works
Bulat, A., & Tzimiropoulos, G. (2017) · Nottingham ePrints (University of Nottingham) · Cited by 1,277 · see citing works
Ghosh‐Dastidar, S., & Adeli, H. (2009) · International Journal of Neural Systems · doi:10.1142/s0129065709002002 · Cited by 1,066 · see citing works
HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition
Lagorce, X., Orchard, G., Galluppi, F., Shi, B. E., & Benosman, R. B. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2574707 · Cited by 617 · see citing works
ESIM: an Open Event Camera Simulator
Rebecq, H., Gehrig, D., & Scaramuzza, D. (2018) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-175993 · Cited by 351 · see citing works
Riek, L. D. (2017) · Communications of the ACM · doi:10.1145/3127874 · Cited by 271 · see citing works
EEG-Based Estimation and Classification of Mental Fatigue
Trejo, L. J., Kubitz, K. A., Rosipal, R., Kochavi, R., & Montgomery, L. D. (2015) · Psychology · doi:10.4236/psych.2015.65055 · Cited by 236 · see citing works
Deep learning-based methods for person re-identification: A comprehensive review
Wu, D. J., Zheng, S.-J., Zhang, X., Yuan, C., Cheng, F., Yang, Z., Lin, Y.-J., Zhao, Z., Jiang, Y.-L., & Huang, D.-S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.079 · Cited by 229 · see citing works
Fast Event-based Corner Detection
Mueggler, E., Bartolozzi, C., & Scaramuzza, D. (2017) · British Machine Vision Conference (BMVC) · doi:10.5244/c.31.33 · Cited by 195 · see citing works
A facial expression emotion recognition based human-robot interaction system
Liu, Z., Wu, M. H., Cao, W., Chen, L., Xu, J., Zhang, R., Zhou, M., & Mao, J.-W. (2017) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2017.7510622 · Cited by 185 · see citing works
A Comprehensive Database for Benchmarking Imaging Systems
Panetta, K. A., Wan, Q., Agaian, S. С., et al. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2884458 · Cited by 167 · see citing works
Akalın, N., Kristoffersson, A., & Loutfi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102744 · Cited by 166 · see citing works
Hong, A., Lunscher, N., Hu, T., Tsuboi, Y., Zhang, X., Alves, S. F. D. R., Nejat, G., & Benhabib, B. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2020.2974688 · Cited by 119 · see citing works
Filippini, C., Perpetuini, D., Cardone, D., Chiarelli, A. M., & Merla, A. (2020) · Applied Sciences · doi:10.3390/app10082924 · Cited by 119 · see citing works
Real-time face & eye tracking and blink detection using event cameras
Ryan, C., O’Sullivan, B., Elrasad, A., Cahill, A., Lemley, J., Kielty, P., Posch, C., & Perot, E. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.019 · Cited by 118 · see citing works
Real-Time 6DOF Pose Relocalization for Event Cameras With Stacked Spatial LSTM Networks
Nguyen, A., Do, T.-T., Caldwell, D. G., & Tsagarakis, N. G. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00207 · Cited by 92 · see citing works
Song, X., Han, X., Li, Y., et al. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350956 · Cited by 88 · see citing works
Temporal binary representation for event-based action recognition
Innocenti, S. U., Becattini, F., Pernici, F., & Bimbo, A. D. (2020) · Florence Research (University of Florence) · doi:10.1109/icpr48806.2021.9412991 · Cited by 88 · see citing works
Self-citationEvent-Based Face Detection and Tracking Using the Dynamics of Eye Blinks
Lenz, G., Ieng, S.-H., & Benosman, R. B. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.00587 · Cited by 63 · see citing works
Speech emotion recognition in emotional feedback for Human-Robot Interaction
Javier, G., Sundgren, D., Rahmani, R., Larsson, A., Moran, A., & Bonet, I. (2015) · INTERNATIONAL JOURNAL OF ADVANCED RESEARCH IN ARTIFICIAL INTELLIGENCE · doi:10.14569/ijarai.2015.040204 · Cited by 62 · see citing works
Event-driven Re-Id: A New Benchmark and Method Towards Privacy-Preserving Person Re-Identification
Ahmad, S., Scarpellini, G., Morerio, P., & Bue, A. D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw54805.2022.00052 · Cited by 54 · see citing works
Biometrics and Privacy-Preservation: How Do They Evolve?
Tran, Q. N., Turnbull, B., & Hu, J. (2021) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2021.3068385 · Cited by 47 · see citing works
A Review on Facial Micro-Expressions Analysis: Datasets, Features and MetricsPreprint
Merghani, W., Davison, A. K., & Yap, M. H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.02397 · Cited by 46 · see citing works
Emotion Analysis in Human–Robot Interaction
Szabóová, M., Sarnovský, M., Krešňáková, V., & Machová, K. (2020) · Electronics · doi:10.3390/electronics9111761 · Cited by 42 · see citing works
Toward a Taxonomy of Affective Computing
Schwark, J. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1064638 · Cited by 41 · see citing works
Human–Robot Interactions and Affective Computing: The Ethical Implications
Devillers, L. Y. (2021) · Springer eBooks · doi:10.1007/978-3-030-54173-6_17 · Cited by 34 · see citing works
Chen, G., Wang, F., Yuan, X., et al. (2021) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003483 · Cited by 24 · see citing works
Mental State Assessment and Validation Using Personalized Physiological Biometrics
Patel, A. N., Howard, M. D., Roach, S. M., Jones, A. P., Bryant, N. B., Robinson, C. S. H., Clark, V. P., & Pilly, P. K. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00221 · Cited by 18 · see citing works
Boosted Kernelized Correlation Filters for Event-based Face Detection
Ramesh, B., & Yang, H. (2020) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw50321.2020.9096944 · Cited by 17 · see citing works
N-ROD: a Neuromorphic Dataset for Synthetic-to-Real Domain Adaptation
Cannici, M., Plizzari, C., Planamente, M., Ciccone, M., Bottino, A., Barbara Caputo, B., & Matteucci, M. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00148 · Cited by 15 · see citing works
PLM-IPE: A Pixel-Landmark Mutual Enhanced Framework for Implicit Preference Estimation
Becattini, F., Song, X., Baecchi, C., et al. (2021) · ACM Multimedia Asia (MMAsia) · doi:10.1145/3469877.3490621 · Cited by 15 · see citing works
Self-citationSegmentation Free Object Discovery in Video
Cuffaro, G., Becattini, F., Baecchi, C., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_4 · Cited by 13 · see citing works
Self-citationIndexing quantized ensembles of exemplar-SVMs with rejecting taxonomies
Becattini, F., Seidenari, L., & Bimbo, A. D. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-017-4794-7 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Human-in-the-Loop Robot Learning for Smart Manufacturing: A Human-Centric Perspective
Chen, H., Li, S., Fan, J., Duan, A., Yang, C., Navarro-Alarcon, D., & Zheng, P. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3528051 · Cited by 67 · see citing works
Ahmad, S., Morerio, P., & Del Bue, A. (2024) · IEEE Access · doi:10.1109/access.2024.3399539 · Cited by 16 · see citing works
- Authors
- Becattini, F., Palai, F., & Bimbo, A. D.
- Affiliations
- University of Florence (IT)
- Venue
- IEEE Transactions on Industrial Informatics
- Published
- 2022-12
- Volume
- 18
- Issue
- 12
- Pages
- 9112-9121
- ISSN
- 1551-3203 (ISSN-L)
- 1941-0050
- OpenAlex
- W4288770705
- Cited by
- 50
- References
- 37
- Citations by year
- 2026: 10 · 2025: 15 · 2024: 15 · 2023: 9 · 2022: 1
- FWCI
- 9.20
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Industrial Informatics (publishedVersion)
- Florence Research (University of Florence) (submittedVersion)
- Use Siena air (University of Siena) (submittedVersion)
- Funders
- Italian MIUR
- PRIN 2017 (20172BH297)
- intelligent computer vision
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- facial micro-expressions, event cameras, human-machine interaction, Industry 4.0, biometric monitoring, privacy-preserving sensing, low frame rate video, dynamic range
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
