The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks
Abstract
Saliency in Context (SALICON) is an ongoing effort that aims at understanding and predicting visual attention. Conventional saliency models typically rely on low-level image statistics to predict human fixations. While these models perform significantly better than chance, there is still a large gap between model prediction and human behavior. This gap is largely due to the limited capability of models in predicting eye fixations with strong semantic content, the so-called semantic gap. This paper presents a focused study to narrow the semantic gap with an architecture based on Deep Neural Network (DNN). It leverages the representational power of high-level semantics encoded in DNNs pretrained for object recognition. Two key components are fine-tuning the DNNs fully convolutionally with an objective function based on the saliency evaluation metrics, and integrating information at different image scales. We compare our method with 14 saliency models on 6 public eye tracking benchmark datasets. Results demonstrate that our DNNs can automatically learn features particularly for saliency prediction that surpass by a big margin the state-of-the-art. In addition, our model ranks top to date under all seven metrics on the MIT300 challenge set.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
A feature-integration theory of attention.
Treisman, A. M., & Gelade, G. (1980) · Cognitive Psychology · doi:10.1016/0010-0285(80)90005-5 · Cited by 12,366
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271
(1966) · Medical Entomology and Zoology · Cited by 10,293
Signal detection theory and psychophysics
Ingleby, J. D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,509
Computational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733
Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry
Koch, C., & Ullman, S. (1987) · Matters of Intelligence · doi:10.1007/978-94-009-3833-5_5 · Cited by 3,722
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
A saliency-based search mechanism for overt and covert shifts of visual attention
Itti, L., & Koch, C. (2000) · Vision Research · doi:10.1016/s0042-6989(99)00163-7 · Cited by 3,156
Learning Hierarchical Features for Scene Labeling
Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.231 · Cited by 2,620
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
Context-Aware Saliency Detection
Goferman, S., Zelnik-Manor, L., & Tal, A. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.272 · Cited by 1,567
The Secrets of Salient Object Segmentation
Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.43 · Cited by 1,338
Saliency Based on Information Maximization
Bruce, N. D. B., & Tsotsos, J. K. (2005) · neural information processing systems · Cited by 1,307
SUN: A Bayesian framework for saliency using natural statistics
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008) · Journal of Vision · doi:10.1167/8.7.32 · Cited by 1,227
Image Signature: Highlighting Sparse Salient Regions
Hou, X., Harel, J., & Koch, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.146 · Cited by 834
Components of bottom-up gaze allocation in natural images
Peters, R. J., Iyer, A., Itti, L., & Koch, C. (2005) · Vision Research · doi:10.1016/j.visres.2005.03.019 · Cited by 739
Visual correlates of fixation selection: effects of scale and time
Tatler, B. W., Baddeley, R. J., & Gilchrist, I. D. (2005) · Vision Research · doi:10.1016/j.visres.2004.09.017 · Cited by 700
A Benchmark of Computational Models of Saliency to Predict Human Fixations
Judd, T., Durand, F., & Torralba, A. B. (2012) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 511
Saliency Detection: A Boolean Map Approach
Zhang, J., & Sclaroff, S. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.26 · Cited by 495
Predicting human gaze using low-level saliency combined with face detection
Cerf, M., Harel, J., Einhäuser, W., & Koch, C. (2008) · CaltechAUTHORS (California Institute of Technology) · Cited by 465
Objects predict fixations better than early saliency
Einhauser, W., Spain, M., & Perona, P. (2008) · Journal of Vision · doi:10.1167/8.14.18 · Cited by 451
Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images
Vig, E., Dorr, M., & Cox, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.358 · Cited by 428
Visual saliency estimation by nonlinearly integrating features using region covariances
Erdem, E., & Erdem, A. (2013) · Journal of Vision · doi:10.1167/13.4.11 · Cited by 396
Exploiting local and global patch rarities for saliency detection
Borji, A., & Itti, L. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247711 · Cited by 376
Predicting human gaze beyond pixels
Xu, J., Jiang, M., Wang, S., Kankanhalli, M. S., & Zhao, Q. (2014) · Journal of Vision · doi:10.1167/14.1.28 · Cited by 353
Self-citationDeep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
Kümmerer, M., Theis, L., & Bethge, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.1045 · Cited by 231
An Eye Fixation Database for Saliency Detection in Images
Ramanathan, S., Katti, H., Sebe, N., Kankanhalli, M., & Chua, T.-S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_3 · Cited by 209
Predicting eye fixations using convolutional neural networks
Liu, N., Han, J., Zhang, D., Wen, S., & Liu, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298633 · Cited by 201
Saliency from hierarchical adaptation through decorrelation and variance normalization
Garcia-Diaz, A., Fdez-Vidal, X. R., Pardo, X. M., & Dosil, R. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2011.11.007 · Cited by 194
RARE2012: A multi-scale rarity-based saliency detection with its comparative statistical analysis
Riche, N., Mancas, M., Duvinage, M., Mibulumukini, M., Gosselin, B., & Dutoit, T. (2013) · Signal Processing: Image Communication · doi:10.1016/j.image.2013.03.009 · Cited by 189
Learning visual saliency by combining feature maps in a nonlinear manner using AdaBoost
Zhao, Q., & Koch, C. (2012) · Journal of Vision · doi:10.1167/12.6.22 · Cited by 111
Self-citationLearning to predict eye fixations for semantic contents using multi-layer sparse network
Shen, C., & Zhao, Q. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.09.053 · Cited by 52
Self-citationShort Papers A Model of Saliency-Based Visual Attention for Rapid Scene Analysis
Itti, L., Koch, C., & Niebur, E. (1998) · Cited by 25
Cited by
Object Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545
BASNet: Boundary-Aware Salient Object Detection
Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., & Jagersand, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00766 · Cited by 1,565
Deep Visual Attention Prediction
Wang, W., & Shen, J. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2787612 · Cited by 585
DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection
Li, X., Zhao, L., Wei, L., Yang, M.-H., Wu, F., Zhuang, Y., Ling, H., & Wang, J. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2579306 · Cited by 538
Saliency Detection with Recurrent Fully Convolutional Networks
Wang, L., Wang, L., Lu, H., Zhang, P., & Ruan, X. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_50 · Cited by 509
Metadata
- Authors
- Huang, X., Shen, C., Boix, X., & Zhao, Q.
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 262-270
- OpenAlex
- W2212216676
- Cited by
- 597
- References
- 43
- FWCI
- 20.78
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.