Human Gaze Assisted Artificial Intelligence: A Review
Open accessAbstract
Human gaze reveals a wealth of information about internal cognitive state. Thus, gaze-related research has significantly increased in computer vision, natural language processing, decision learning, and robotics in recent years. We provide a high-level overview of the research efforts in these fields, including collecting human gaze data sets, modeling gaze behaviors, and utilizing gaze information in various applications, with the goal of enhancing communication between these research areas. We discuss future challenges and potential applications that work towards a common goal of human-centered artificial intelligence.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Christof Koch, A., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,289 · see citing works
The attention system of the human brain
Posner, M. I., & Petersen, S. E. (1990) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.13.030190.000325 · Cited by 8,070 · cited 1× · see citing works
“Because humans actively seek the information they need, gaze can reveal the underlying attentional patterns [Posner and Petersen, 1990].”
BackgroundRecurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
The eyes have it: the neuroethology, function and evolution of social gaze
Emery, N. J. (2000) · Neuroscience & Biobehavioral Reviews · doi:10.1016/s0149-7634(00)00025-7 · Cited by 2,037 · see citing works
Learning from Simulated and Unsupervised Images through Adversarial Training
Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.241 · Cited by 1,831 · see citing works
Salient Object Detection: A Benchmark
Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2487833 · Cited by 1,550 · see citing works
The Secrets of Salient Object Segmentation
Li, Y., Hou, X., Christof Koch, A., Rehg, J. M., & Yuille, A. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.43 · Cited by 1,401 · see citing works
Appearance-based gaze estimation in the wild
Zhang, X., Sugano, Y., Fritz, M., & Bulling, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299081 · Cited by 844 · see citing works
(2016) · Cited by 844 · see citing works
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A. B., & Durand, F. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2815601 · Cited by 835 · see citing works
Jiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729 · see citing works
The development of gaze following and its relation to language
Brooks, R., & Meltzoff, A. N. (2005) · Developmental Science · doi:10.1111/j.1467-7687.2005.00445.x · Cited by 659 · see citing works
MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation
Zhang, X., Sugano, Y., Fritz, M., & Bulling, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2778103 · Cited by 596 · see citing works
Social eye gaze in human-robot interaction
Admoni, H., & Scassellati, B. (2017) · J. Hum. Robot Interact. · doi:10.5898/jhri.6.1.admoni · Cited by 587 · see citing works
DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations
Kruthiventi, S. S. S., Ayush, K., & Babu, R. V. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2710620 · Cited by 517 · see citing works
Human Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?
Das, A., Agrawal, H., Zitnick, C. L., Parikh, D., & Batra, D. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1092 · Cited by 431 · see citing works
Appearance-Based Gaze Estimation in the Wild
Zhang, X., Sugano, Y., Fritz, M., & Bulling, A. (2015) · Cited by 411 · see citing works
Predicting human gaze beyond pixels
Xu, J., Jiang, M. X., Wang, S., Kankanhalli, M., & Zhao, Q. (2014) · Journal of Vision · doi:10.1167/14.1.28 · Cited by 353 · see citing works
Rendering of Eyes for Eye-Shape Registration and Gaze Estimation
Wood, E., Baltruaitis, T., Zhang, X., Sugano, Y., Robinson, P., & Bulling, A. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.428 · Cited by 320 · see citing works
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
Li, Y., Liu, M., & Rehg, J. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01228-1_38 · Cited by 317 · see citing works
Predicting the Driver's Focus of Attention: The DR(eye)VE Project
Palazzi, A., Abati, D., Calderara, S., Solera, F., & Cucchiara, R. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2845370 · Cited by 315 · see citing works
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model
Wang, W., Shen, J., Guo, F., Cheng, M., & Borji, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00514 · Cited by 281 · see citing works
Intrinsic and extrinsic effects on image memorability
Bylinskii, Z., Isola, P., Bainbridge, C. M., Torralba, A. B., & Oliva, A. (2015) · Vision Research · doi:10.1016/j.visres.2015.03.005 · Cited by 264 · see citing works
DeepGaze II: Reading fixations from deep features trained on object recognitionPreprint
Kümmerer, M., Wallis, T. S. A., & Bethge, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01563 · Cited by 259 · see citing works
Defending Yarbus: Eye movements reveal observers' task
Borji, A., & Itti, L. (2014) · Journal of Vision · doi:10.1167/14.3.29 · Cited by 246 · see citing works
Anticipatory robot control for efficient human-robot collaboration
Huang, C.-M., & Mutlu, B. (2016) · 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2016.7451737 · Cited by 227 · see citing works
Ruhland, K., Peters, C., Andrist, S., Badler, J. B., Badler, N. I., Gleicher, M. L., Mutlu, B., & McDonnell, R. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12603 · Cited by 218 · see citing works
Conversational gaze aversion for humanlike robots
Andrist, S., Tan, X. Z., Gleicher, M. L., & Mutlu, B. (2014) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2559636.2559666 · Cited by 217 · see citing works
Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition
Mathe, S., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366154 · Cited by 217 · see citing works
Deep Pictorial Gaze Estimation
Park, S., Spurr, A., & Hilliges, O. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_44 · Cited by 212 · see citing works
Fiore, S. M., Wiltshire, T. J., Lobato, E. J. C., Jentsch, F. G., Huang, W. H., & Axelrod, B. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00859 · Cited by 172 · see citing works
Huang, Q., Veeraraghavan, A., & Sabharwal, A. (2017) · Machine Vision and Applications · doi:10.1007/s00138-017-0852-4 · Cited by 171 · see citing works
LSTA: Long Short-Term Attention for Egocentric Action Recognition
Sudhakaran, S., Escalera, S., & Lanz, O. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01019 · Cited by 165 · see citing works
What you learn is what you see: using eye movements to study infant cross‐situational word learning
Yu, C., & Smith, L. B. (2010) · Developmental Science · doi:10.1111/j.1467-7687.2010.00958.x · Cited by 161 · see citing works
Predicting Driver Attention in Critical Situations
Xia, Y., Zhang, D., Kim, J., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20873-8_42 · Cited by 161 · see citing works
Hayhoe, M. (2017) · Annual Review of Vision Science · doi:10.1146/annurev-vision-102016-061437 · Cited by 156 · see citing works
Self-citationEnd-to-End Saliency Mapping via Probability Distribution Prediction
Jetley, S., Murray, N., & Vig, E. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.620 · Cited by 149 · see citing works
Where Should Saliency Models Look Next?
Bylinskii, Z., Recasens, A., Borji, A., Oliva, A., Torralba, A. B., & Durand, F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_49 · Cited by 137 · see citing works
Predicting Affect from Gaze Data during Interaction with an Intelligent Tutoring System
Jaques, N., Conati, C., Harley, J. M., & Azevedo, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_4 · Cited by 135 · see citing works
Computational Human-Robot Interaction
Thomaz, A. L., Hoffman, G., & Çakmak, M. (2016) · Foundations and Trends in Robotics · doi:10.1561/2300000049 · Cited by 133 · see citing works
Training Object Class Detectors from Eye Tracking Data
Papadopoulos, D. P., Clarke, A. D. F., Keller, F., & Ferrari, V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_24 · Cited by 131 · see citing works
Deng, T., Yan, H., Qin, L., Ngo, T., & Manjunath, B. S. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2915540 · Cited by 130 · see citing works
Palinko, O., Rea, F., Sandini, G., et al. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759741 · Cited by 108 · see citing works
Anticipatory Robot Control for Efficient Human-Robot Collaboration
Huang, C., & Mutlu, B. (2016) · doi:10.5555/2906831.2906846 · Cited by 105 · see citing works
Gaze Embeddings for Zero-Shot Image Classification
Karessli, N., Akata, Z., Schiele, B., & Bulling, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.679 · Cited by 105 · see citing works
Exploring Human-Like Attention Supervision in Visual Question Answering
Qiao, T., Dong, J., & Xu, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12272 · Cited by 100 · see citing works
Paying Attention to Descriptions Generated by Image Captioning Models
Tavakoli, H. R., Shetty, R., Borji, A., & Laaksonen, J. T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.272 · Cited by 82 · see citing works
Studying Relationships between Human Gaze, Description, and Computer Vision
Yun, K., Peng, Y., Samaras, D., Zelinsky, G. J., & Berg, T. L. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.101 · Cited by 79 · see citing works
Eye-Hand Behavior in Human-Robot Shared Manipulation
Aronson, R. M., Santini, T., Kübler, T., Kasneci, E., Srinivasa, S. S., & Admoni, H. (2018) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3171221.3171287 · Cited by 79 · see citing works
Gaze-based, Context-aware Robotic System for Assisted Reaching and Grasping
Shafti, A., Orlov, P. A., & Faisal, A. A. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793804 · Cited by 77 · see citing works
Supervising Neural Attention Models for Video Captioning by Human Gaze Data
Yu, Y., Choi, J., Kim, Y., et al. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.648 · Cited by 75 · see citing works
AGIL: Learning Attention from Human for Visuomotor Tasks
Zhang, R., Liu, Z., Zhang, L., Whritner, J. A., Muller, K., Hayhoe, M., & Ballard, D. H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_41 · Cited by 71 · see citing works
Self-citationMeaning Guides Attention during Real-World Scene Description
Henderson, J. M., Hayes, T. R., Rehrig, G., & Ferreira, F. (2018) · Scientific Reports · doi:10.1038/s41598-018-31894-5 · Cited by 69 · see citing works
Leveraging Human Guidance for Deep Reinforcement Learning Tasks
Zhang, R., Torabi, F., Guan, L., Ballard, D. H., & Stone, P. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/884 · Cited by 69 · see citing works
Self-citationHutt, S. J., Mills, C., White, S., et al. (2016) · Cited by 67 · see citing works
Recognizing engagement in human-robot interaction
Rich, C., Ponsleur, B., Holroyd, A., et al. (2010) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/1734454.1734580 · Cited by 65 · see citing works
Human Gaze Following for Human-Robot Interaction
Saran, A., Majumdar, S., Short, E. S., Thomaz, A. L., & Niekum, S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593580 · Cited by 64 · see citing works
Self-citationComputational Human-Robot Interaction
Thomaz, A. L., Hoffman, G., & Çakmak, M. (2016) · now publishers, Inc. eBooks · doi:10.1561/9781680832099 · Cited by 63 · see citing works
Nguyen, T., Zhao, Q., & Yan, S. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1042-6 · Cited by 63 · see citing works
Object Referring in Videos with Language and Human Gaze
Vasudevan, A. B., Dai, D., & Gool, L. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00434 · Cited by 62 · see citing works
Embodied Collaborative Referring Expression Generation in Situated Human-Robot Interaction
Fang, R., Doering, M., & Chai, J. Y. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696467 · Cited by 59 · see citing works
Boosted Attention: Leveraging Human Attention for Image Captioning
Chen, S., & Zhao, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_5 · Cited by 53 · see citing works
Seeing with Humans: Gaze-Assisted Neural Image Captioning
Sugano, Y., & Bulling, A. (2016) · arXiv · doi:10.48550/arxiv.1608.05203 · Cited by 47 · see citing works
Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset
Zhang, R., Walshe, C., Liu, Z., Guan, L., Muller, K., Whritner, J. A., Zhang, L., Hayhoe, M., & Ballard, D. H. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6161 · Cited by 46 · see citing works
Self-citationRevisiting Video Saliency: A Large-scale Benchmark and a New ModelPreprint
Wang, W., Shen, J., Guo, F., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07424 · Cited by 42 · see citing works
Understanding and Visualizing Deep Visual Saliency Models
He, S., Tavakoli, H. R., Borji, A., et al. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01045 · Cited by 42 · see citing works
Human Attention in Image Captioning: Dataset and Analysis
He, S., Tavakoli, H. R., Borji, A., et al. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00862 · Cited by 42 · see citing works
Egocentric Activity Prediction via Event Modulated Attention
Shen, Y., Ni, B., Li, Z., & Zhuang, N. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_13 · Cited by 40 · see citing works
Human-Robot Interaction Based on Gaze Gestures for the Drone Teleoperation
Yu, M., Lin, Y., Schmidt, D. M., et al. (2014) · Journal of Eye Movement Research · doi:10.16910/jemr.7.4.4 · Cited by 39 · see citing works
A gaze model improves autonomous driving
Liu, C., Chen, Y., Tai, L., et al. (2019) · ACM Symposium on Eye Tracking Research & Applications (ETRA) · doi:10.1145/3314111.3319846 · Cited by 34 · see citing works
Recognizing engagement in human-robot interaction
Rich, C., Ponsleur, B., Holroyd, A., & Sidner, C. L. (2010) · doi:10.5555/1734454.1734580 · Cited by 31 · see citing works
Gaze-Informed Egocentric Action Recognition for Memory Aid Systems
Zuo, Z., Yang, L., Peng, Y., Chao, F., & Qu, Y. (2018) · IEEE Access · doi:10.1109/access.2018.2808486 · Cited by 30 · see citing works
Modeling communicative behaviors for object references in human-robot interaction
Admoni, H., Weng, T., & Scassellati, B. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487510 · Cited by 22 · see citing works
Digging Deeper Into Egocentric Gaze Prediction
Tavakoli, H. R., Rahtu, E., Kannala, J., et al. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00035 · Cited by 22 · see citing works
Gaze and motion information fusion for human intention inference
Ravichandar, H. C., Kumar, A., & Dani, A. P. (2018) · International Journal of Intelligent Robotics and Applications · doi:10.1007/s41315-018-0051-0 · Cited by 21 · see citing works
DIDEC: The Dutch Image Description and Eye-tracking Corpus
van Miltenburg, E., Kádár, Á., Koolen, R., et al. (2018) · Research portal (Tilburg University) · Cited by 17 · see citing works
Intelligent Interfaces to Empower People with Disabilities
Betke, M. (2009) · Springer eBooks · doi:10.1007/978-0-387-93808-0_15 · Cited by 16 · see citing works
Physical symbol grounding and instance learning through demonstration and eye tracking
Penkov, S., Bordallo, A., & Ramamoorthy, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989697 · Cited by 16 · see citing works
Understanding Teacher Gaze Patterns for Robot LearningPreprint
Saran, A., Short, E. S., Thomaz, A. L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07202 · Cited by 13 · see citing works
Self-citationAtari-HEAD: Atari Human Eye-Tracking and Demonstration DatasetPreprint
Zhang, R., Walshe, C., Liu, Z., Guan, L., Muller, K., Whritner, J. A., Zhang, L., Hayhoe, M., & Ballard, D. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.06754 · Cited by 13 · see citing works
Self-citationSNAG: Spoken Narratives and Gaze Dataset
Vaidyanathan, P., Prud’hommeaux, E. T., Pelz, J. B., et al. (2018) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p18-2022 · Cited by 12 · see citing works
Nonverbal Robot Feedback for Human TeachersPreprint
Huang, S. H., Huang, I., Pandya, R., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02320 · Cited by 7 · see citing works
Supervising Neural Attention Models for Video Captioning by Human Gaze DataPreprint
Yu, Y., Choi, J., Kim, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06029 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- [3]3. Henny Admoni, Anca Dragan, Siddhartha S Srinivasa, and Brian Scassellati. Deliberate delays during robot- to-human handovers improve compliance with gaze communica- tion. In Proceedings of the 2014 ACM/IEEE international con- ference on Human-robot interaction, pages 49-56, 2014.
- [11]11. Sean Andrist, Michael Gleicher, and Bilge Mutlu. Looking coordinated: Bidirectional gaze mechanisms for collaborative interaction with virtual characters. In Proceedings of the 2017 CHI conference on human factors in computing sys- tems, pages 2571-2582, 2017.
- [32]32. Cornia et al., 2018a] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Paying more attention to saliency: Im- age captioning with saliency and context attention. ACM Trans- actions on Multimedia Computing, Communications, and Appli- cations (TOMM), 14(2):1-21, 2018.
- [33]33. Cornia et al., 2018b] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Predicting human eye fixations via an lstm-based saliency attentive model. IEEE Transactions on Image Processing, 27(10):5142-5154, 2018.
- [68]68. Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech Matusik, and Antonio Torralba. Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2176-2184, 2016.
- [82]82. Moon et al., 2014] AJung Moon, Daniel M Troniak, Brian Glee- son, Matthew KXJ Pan, Minhua Zheng, Benjamin A Blumer, Karon MacLean, and Elizabeth A Croft. Meet me where i'm gazing: how shared attention gaze affects human-robot handover timing. In Proceedings of the 2014 ACM/IEEE international con- ference on Human-robot interaction, pages 334-341, 2014.
- [90]90. Papoutsaki et al., 2016] Alexandra Papoutsaki, Patsorn Sangkloy, James Laskey, Nediyana Daskalova, Jeff Huang, and James Hays. Webgazer: Scalable webcam eye tracking using user in- teractions. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence-IJCAI 2016, 2016.
- [101]101. Adria Recasens, Aditya Khosla, Carl Von- drick, and Antonio Torralba. Where are they looking? In Ad- vances in Neural Information Processing Systems, pages 199- 207, 2015.
- [103]103. Adria Recasens, Carl Vondrick, Aditya Khosla, and Antonio Torralba. Following gaze in video. In Pro- ceedings of the IEEE International Conference on Computer Vi- sion, pages 1435-1443, 2017.
- [107]107. Viktor Richter, Birte Carlmeyer, Florian Lier, Sebastian Meyer zu Borgsen, David Schlangen, Franz Kummert, Sven Wachsmuth, and Britta Wrede. Are you talking to me? im- proving the robustness of dialogue systems in a multi party hri scenario by incorporating gaze direction and lip movement of at- tendees. In Proceedings of the Fourth International Conference on Human Agent Interaction, pages 43-50, 2016.
- [133]133. Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in neural information processing systems, pages 5998- 6008, 2017.
- [141]141. Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, and Shenghua Gao. Gaze prediction in dynamic 360 immersive videos. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5333-5342, 2018.
- [153]153. Zhang et al., 2018b] Ziheng Zhang, Yanyu Xu, Jingyi Yu, and Shenghua Gao. Saliency detection in 360 videos. In Proceed- ings of the European Conference on Computer Vision (ECCV), pages 488-503, 2018.
Cited by
Sorted by citations
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
- Authors
- Zhang, R., Saran, A., Liu, B., Zhu, Y., Guo, S., Niekum, S., Ballard, D., & Hayhoe, M.
- Affiliations
- The University of Texas at Austin (US)
- Corresponding author
- Zhang, Ruohan
- Venue
- Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
- Published
- 2020-07
- Volume
- 2020
- Pages
- 4951-4958
- PMC
- PMC7476326
- OpenAlex
- W3041498386
- Cited by
- 48
- References
- 85
- Citations by year
- 2026: 5 · 2025: 4 · 2024: 10 · 2023: 15 · 2022: 7 · 2021: 7
- FWCI
- 5.30
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Science Foundation (1617639, 1749204, 1638107, 1724157)
- University of Texas at Austin (N00014-18-2243)
- National Institutes of Health
- Office of Naval Research (N00014-18-2243, N00014)
- National Eye Institute (R01EY005729)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- human gaze, gaze-based interaction, human-centered AI, cognitive states
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
