The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Predicting human gaze beyond pixels
Open accessAbstract
A large body of previous models to predict where people look in natural scenes focused on pixel-level image attributes. To bridge the semantic gap between the predictive power of computational saliency models and human behavior, we propose a new saliency architecture that incorporates information at three layers: pixel-level image attributes, object-level attributes, and semantic-level attributes. Object- and semantic-level information is frequently ignored, or only a few sample object categories are discussed where scaling to a large number of object categories is not feasible nor neurally plausible. To address this problem, this work constructs a principled vocabulary of basic attributes to describe object- and semantic-level information thus not restricting to a limited number of object categories. We build a new dataset of 700 images with eye-tracking data of 15 viewers and annotation data of 5,551 segmented objects with fine contours and 12 semantic attributes (publicly available with the paper). Experimental results demonstrate the importance of the object- and semantic-level information in the prediction of visual attention.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738
Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry
Koch, C., & Ullman, S. (1987) · Matters of Intelligence · doi:10.1007/978-94-009-3833-5_5 · Cited by 3,722
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
Multiple significance tests: the Bonferroni method
Bland, J. M., & Altman, D. G. (1995) · BMJ · doi:10.1136/bmj.310.6973.170 · Cited by 3,341
(1987) · Medical Entomology and Zoology · Cited by 2,512
Mechanisms of visual attention in the human cortex
Ungerleider, S. K. A. L. G. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.315 · Cited by 2,159
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
Emotion, cognition, and behavior
Dolan, R. J. (2002) · Science · doi:10.1126/science.1076358 · Cited by 1,979
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206772 · Cited by 1,917
The fusiform face area: a cortical region specialized for the perception of faces
Kanwisher, N., & Yovel, G. (2006) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2006.1934 · Cited by 1,718
Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006) · Psychological Review · doi:10.1037/0033-295x.113.4.766 · Cited by 1,639
Modeling the role of salience in the allocation of overt visual attention
Parkhurst, D., Law, K., & Niebur, E. (2002) · Vision Research · doi:10.1016/s0042-6989(01)00250-4 · Cited by 1,388
The eyes have it! Reflexive orienting is triggered by nonpredictive gaze
Friesen, C. K., & Kingstone, A. (1998) · Psychonomic Bulletin & Review · doi:10.3758/bf03208827 · Cited by 1,325
Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects
Chao, L. L., Haxby, J. V., & Martin, A. (1999) · Nature Neuroscience · doi:10.1038/13217 · Cited by 1,242
Hoffman, E. A., & Haxby, J. V. (2000) · Nature Neuroscience · doi:10.1038/71152 · Cited by 1,235
SUN: A Bayesian framework for saliency using natural statistics
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008) · Journal of Vision · doi:10.1167/8.7.32 · Cited by 1,227
Modeling visual attention via selective tuning
Tsotsos, J. K., Culhane, S. M., Kei Wai, W. Y., Lai, Y., Davis, N., & Nuflo, F. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(95)00025-9 · Cited by 1,143
A Cortical Region Consisting Entirely of Face-Selective Cells
Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H., & Livingstone, M. S. (2006) · Science · doi:10.1126/science.1119983 · Cited by 1,141
Tatler, B. W. (2007) · Journal of Vision · doi:10.1167/7.14.4 · Cited by 1,028
Huth, A. G., Nishimoto, S., Vu, A. T., & Gallant, J. L. (2012) · Neuron · doi:10.1016/j.neuron.2012.10.014 · Cited by 896
Saliency, attention, and visual search: An information theoretic approach
Bruce, N. D. B., & Tsotsos, J. K. (2009) · Journal of Vision · doi:10.1167/9.3.5 · Cited by 855
What does the amygdala contribute to social cognition
Adolphs, R. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05445.x · Cited by 836
Image Signature: Highlighting Sparse Salient Regions
Hou, X., Harel, J., & Koch, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.146 · Cited by 834
Eye guidance in natural vision: reinterpreting salience
Tatler, B. W., Hayhoe, M. M., Land, M. F., & Ballard, D. H. (2011) · Journal of Vision · doi:10.1167/11.5.5 · Cited by 775
Human Amygdala Responsivity to Masked Fearful Eye Whites
Whalen, P. J., Kagan, J., Cook, R. G., Davis, F. C., Kim, H., Polis, S., McLaren, D. G., Somerville, L. H., McLean, A. A., Maxwell, J. S., et al. (2004) · Science · doi:10.1126/science.1103617 · Cited by 766
Activation in Human MT/MST by Static Images with Implied Motion
Kourtzi, Z., & Kanwisher, N. (2000) · Journal of Cognitive Neuroscience · doi:10.1162/08989290051137594 · Cited by 765
Visual correlates of fixation selection: effects of scale and time
Tatler, B. W., Baddeley, R. J., & Gilchrist, I. D. (2005) · Vision Research · doi:10.1016/j.visres.2004.09.017 · Cited by 700
Static and space-time visual saliency detection by self-resemblance
Seo, H. J., & Milanfar, P. (2009) · Journal of Vision · doi:10.1167/9.12.15 · Cited by 633
Cree, G. S., & McRae, K. (2003) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.132.2.163 · Cited by 607
Bayesian Surprise Attracts Human Attention
Itti, L., & Baldi, P. (2005) · neural information processing systems · Cited by 533
When Strangers Pass: Processing of Mutual and Averted Social Gaze in the Superior Temporal Sulcus
Pelphrey, K. A., Viola, R. J., & McCarthy, G. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00726.x · Cited by 464
Objects predict fixations better than early saliency
Einhauser, W., Spain, M., & Perona, P. (2008) · Journal of Vision · doi:10.1167/8.14.18 · Cited by 451
fMRI Responses to Video and Point-Light Displays of Moving Humans and Manipulable Objects
Beauchamp, M. S., Lee, K. E., Haxby, J. V., & Martin, A. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903770007380 · Cited by 445
Natural scene statistics at the centre of gaze
Reinagel, P., & Zador, A. M. (1999) · Network Computation in Neural Systems · doi:10.1088/0954-898x/10/4/304 · Cited by 434
Neural mechanisms of social attention
Nummenmaa, L., & Calder, A. J. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.12.006 · Cited by 428
Neural correlates of attention in primate visual cortex
Treue, S. (2001) · Trends in Neurosciences · doi:10.1016/s0166-2236(00)01814-2 · Cited by 427
Context-free attentional operators: The generalized symmetry transform
Reisfeld, D., Wolfson, H., & Yeshurun, Y. (1995) · International Journal of Computer Vision · doi:10.1007/bf01418978 · Cited by 414
Patches with Links: A Unified System for Processing Faces in the Macaque Temporal Lobe
Moeller, S., Freiwald, W. A., & Tsao, D. Y. (2008) · Science · doi:10.1126/science.1157436 · Cited by 403
Foulsham, T., & Underwood, G. (2008) · Journal of Vision · doi:10.1167/8.2.6 · Cited by 395
Faces and text attract gaze independent of the task: Experimental data and computer model
Cerf, M., Frady, E. P., & Koch, C. (2009) · Journal of Vision · doi:10.1167/9.12.10 · Cited by 377
Changing faces: a detection advantage in the flicker paradigm
Ro, T., Russell, C., & Lavie, N. (2001) · Psychological Science · doi:10.1111/1467-9280.00317 · Cited by 348
LIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328
A comparative evaluation of interactive segmentation algorithms
McGuinness, K., & O’Connor, N. E. (2010) · Pattern Recognition · doi:10.1016/j.patcog.2009.03.008 · Cited by 308
Garrard, P., Ralph, M. A. L., Hodges, J. R., & Patterson, K. E. (2001) · Cognitive Neuropsychology · doi:10.1080/02643290042000053 · Cited by 298
Modelling search for people in 900 scenes: A combined source model of eye guidance
Ehinger, K. A., Hidalgo-Sotelo, B., Torralba, A., & Oliva, A. (2009) · Visual Cognition · doi:10.1080/13506280902834720 · Cited by 290
Value-dependent selection in the brain: Simulation in a synthetic neural model
Friston, K. J., Tononi, G., Reeke, G. N., Sporns, O., & Edelman, G. M. (1994) · Neuroscience · doi:10.1016/0306-4522(94)90592-4 · Cited by 268
Object-based attentional selection in scene viewing
Nuthmann, A., & Henderson, J. M. (2010) · Journal of Vision · doi:10.1167/10.8.20 · Cited by 264
SUN: Top-down saliency using natural statistics
Kanan, C., Tong, M. H., Zhang, L., & Cottrell, G. W. (2009) · Visual Cognition · doi:10.1080/13506280902771138 · Cited by 262
A Neural Model of Figure–Ground Organization
Craft, E., Schütze, H., Niebur, E., & von der Heydt, R. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00203.2007 · Cited by 260
Learning a saliency map using fixated locations in natural scenes
Zhao, Q., & Koch, C. (2011) · Journal of Vision · doi:10.1167/11.3.9 · Cited by 256
Self-citationA simple saliency model predicts a number of motion popout phenomena
Rosenholtz, R. (1999) · Vision Research · doi:10.1016/s0042-6989(99)00077-2 · Cited by 235
Brain networks for analyzing eye gaze
Hooker, C. I., Paller, K. A., Gitelman, D. R., Parrish, T. B., Mesulam, M.-M., & Reber, P. J. (2003) · Cognitive Brain Research · doi:10.1016/s0926-6410(03)00143-5 · Cited by 222
Bindemann, M., Burton, A. M., Hooge, I. T. C., Jenkins, R., & de Haan, E. H. F. (2005) · Psychonomic Bulletin & Review · doi:10.3758/bf03206442 · Cited by 209
An Eye Fixation Database for Saliency Detection in Images
Ramanathan, S., Katti, H., Sebe, N., Kankanhalli, M., & Chua, T.-S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_3 · Cited by 209
Self-citationWhat and where: A Bayesian inference theory of attention
Chikkerur, S., Serre, T., Tan, C., & Poggio, T. (2010) · Vision Research · doi:10.1016/j.visres.2010.05.013 · Cited by 197
The discriminant center-surround hypothesis for bottom-up saliency
Gao, D., Mahadevan, V., & Vasconcelos, N. M. (2007) · neural information processing systems · Cited by 195
Measuring visual saliency by Site Entropy Rate
Wang, W., Wang, Y., Huang, Q., & Gao, W. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539927 · Cited by 190
Object and scene analysis by saccadic eye-movements: an investigation with higher-order statistics
Zetzsche, C., Schill, K., Krieger, G., Hauske, G., & Rentschler, I. (2000) · Spatial Vision · doi:10.1163/156856800741216 · Cited by 189
Faces call for attention: evidence from patients with visual extinction
Vuilleumier, P. (2000) · Neuropsychologia · doi:10.1016/s0028-3932(99)00107-4 · Cited by 178
Effects of Salience Are Short-Lived
Donk, M., & van Zoest, W. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02149.x · Cited by 178
Local figure–ground cues are valid for natural images
Fowlkes, C. C., Martin, D. R., & Malik, J. (2007) · Journal of Vision · doi:10.1167/7.8.2 · Cited by 168
Semantic guidance of eye movements in real-world scenes
Hwang, A. D., Wang, H.-C., & Pomplun, M. (2011) · Vision Research · doi:10.1016/j.visres.2011.03.010 · Cited by 164
Baddeley, R. J., & Tatler, B. W. (2006) · Vision Research · doi:10.1016/j.visres.2006.02.024 · Cited by 159
Surface color perception and equivalent illumination models
Brainard, D. H., & Maloney, L. T. (2011) · Journal of Vision · doi:10.1167/11.5.1 · Cited by 153
The Different Functions of Gaze
Argyle, M., Ingham, R., Alkema, F., & McCALLIN, M. (1973) · Semiotica · doi:10.1515/semi.1973.7.1.19 · Cited by 138
The control of attention to faces
Bindemann, M., Burton, A. M., Langton, S., Schweinberger, S. R., & DOHERTY, M. J. (2007) · Journal of Vision · doi:10.1167/7.10.15 · Cited by 119
Learning visual saliency by combining feature maps in a nonlinear manner using AdaBoost
Zhao, Q., & Koch, C. (2012) · Journal of Vision · doi:10.1167/12.6.22 · Cited by 111
Self-citationEye Movements for Reward Maximization
Sprague, N. R., & Ballard, D. H. (2003) · Cited by 106
A model of top-down attentional control during visual search in complex scenes
Hwang, A. D., Higgins, E. C., & Pomplun, M. (2009) · Journal of Vision · doi:10.1167/9.5.25 · Cited by 105
Learning saliency-based visual attention: A review
Zhao, Q., & Koch, C. (2012) · Signal Processing · doi:10.1016/j.sigpro.2012.06.014 · Cited by 103
Self-citationA Motion Aftereffect From Still Photographs Depicting Motion
Winawer, J. A., Huk, A. C., & Boroditsky, L. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02080.x · Cited by 99
Wischnewski, M., Belardinelli, A., Schneider, W. X., & Steil, J. J. (2010) · Cognitive Computation · doi:10.1007/s12559-010-9080-1 · Cited by 97
Schirmer, A., Escoffier, N., Zysset, S., Koester, D., Striano, T., & Friederici, A. D. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.01.018 · Cited by 95
Everyone knows what is interesting: Salient locations which should be fixated
Masciocchi, C. M., Mihalaş, Ş., Parkhurst, D. J., & Niebur, E. (2009) · Journal of Vision · doi:10.1167/9.11.25 · Cited by 93
Schirmer, A., Teh, K. S., Wang, S., Vijayakumar, R., Ching, A. S. M., Nithianantham, D., Escoffier, N., & Cheok, A. D. (2010) · Social Neuroscience · doi:10.1080/17470919.2010.507958 · Cited by 73
Self-citationDelayed Response to Animate Implied Motion in Human Motion Processing Areas
Lorteije, J. A. M., Kenemans, J. L., Jellema, T., Lubbe, R. H. J. V. D., de Heer, F., & Wezel, R. J. A. V. (2006) · Journal of Cognitive Neuroscience · doi:10.1162/089892906775783732 · Cited by 72
Neurophysiological evidence for the influence of past experience on figure–ground perception
Trujillo, L. T., Allen, J. J. B., Schnyer, D. M., & Peterson, M. A. (2010) · Journal of Vision · doi:10.1167/10.2.5 · Cited by 67
Contrast statistics for foveated visual systems: fixation selection by minimizing contrast entropy
Raj, R. G., Geisler, W. S., Bovik, A. C., & Frazor, R. A. (2005) · Journal of the Optical Society of America A · doi:10.1364/josaa.22.002039 · Cited by 58
Integrating audiovisual information for the control of overt attention
Onat, S., Libertus, K., & Peter König, B. (2007) · Journal of Vision · doi:10.1167/7.10.11 · Cited by 51
Açık, A., Onat, S., Schumann, F., Einhäuser, W., & Peter König, B. (2009) · Vision Research · doi:10.1016/j.visres.2009.03.011 · Cited by 47
Saliency on a natural scene background: Effects of color and luminance contrast add linearly
Engmann, S., Hart, B. M. ’., Sieren, T., Onat, S., Peter König, B., & Einhäuser, W. (2009) · Attention Perception & Psychophysics · doi:10.3758/app.71.6.1337 · Cited by 47
Optimal and Preferred Eye Landing Positions in Objects and Scenes
Foulsham, T., & Kingstone, A. F. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.762798 · Cited by 44
Cited by
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., & Durand, F. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2815601 · Cited by 802
Jiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729
Self-citationSALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks
Huang, X., Shen, C., Boix, X., & Zhao, Q. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.38 · Cited by 597
Self-citationDeep Visual Attention Prediction
Wang, W., & Shen, J. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2787612 · Cited by 585
Salient object detection: A survey
Borji, A., Cheng, M.-M., Hou, Q., Jiang, H., & Li, J. (2019) · Computational Visual Media · doi:10.1007/s41095-019-0149-9 · Cited by 524
Metadata
- Authors
- Xu, J., Jiang, M., Wang, S., Kankanhalli, M. S., & Zhao, Q.
- Venue
- Journal of Vision
- Published
- 2014-01-28
- Volume
- 14
- Issue
- 1
- Pages
- 28-28
- ISSN
- 1534-7362 (ISSN-L)
- OpenAlex
- W2007571554
- Cited by
- 353
- References
- 83
- FWCI
- 24.04
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.