SALICON: Saliency in Context
The DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index.
Abstract
Saliency in Context (SALICON) is an ongoing effort that aims at understanding and predicting visual attention. This paper presents a new method to collect large-scale human data during natural explorations on images. While current datasets present a rich set of images and task-specific annotations such as category labels and object segments, this work focuses on recording and logging how humans shift their attention during visual exploration. The goal is to offer new possibilities to (1) complement task-specific annotations to advance the ultimate goal in visual understanding, and (2) understand visual attention and learn saliency models, all with human attentional data at a much larger scale. We designed a mouse-contingent multi-resolutional paradigm based on neurophysiological and psychophysical studies of peripheral vision, to simulate the natural viewing behavior of humans. The new paradigm allowed using a general-purpose mouse instead of an eye tracker to record viewing behaviors, thus enabling large-scale data collection. The paradigm was validated with controlled laboratory as well as large-scale online data. We report in this paper a proof-of-concept SALICON dataset of human “free-viewing” data on 10,000 images from the Microsoft COCO (MS COCO) dataset with rich contextual information. We evaluated the use of the collected data in the context of saliency prediction, and demonstrated them a good source as ground truth for the evaluation of saliency algorithms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To solve this problem, supervised attention methods tried to use additional loss functions to learn to predict gold attention on a manually labeled corpus [31,49,102].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [49]
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,702
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271
The VideoToolbox software for visual psychophysics: transforming numbers into movies
Pelli, D. G. (1997) · Spatial Vision · doi:10.1163/156856897x00366 · Cited by 11,098
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
In the Eye of the Beholder: A Survey of Models for Eyes and Gaze
Hansen, D. W., & Ji, Q. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.30 · Cited by 1,546
Saliency Based on Information Maximization
Bruce, N. D. B., & Tsotsos, J. K. (2005) · neural information processing systems · Cited by 1,307
SUN: A Bayesian framework for saliency using natural statistics
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008) · Journal of Vision · doi:10.1167/8.7.32 · Cited by 1,227
Tatler, B. W. (2007) · Journal of Vision · doi:10.1167/7.14.4 · Cited by 1,028
Saliency, attention, and visual search: An information theoretic approach
Bruce, N. D. B., & Tsotsos, J. K. (2009) · Journal of Vision · doi:10.1167/9.3.5 · Cited by 855
Image Signature: Highlighting Sparse Salient Regions
Hou, X., Harel, J., & Koch, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.146 · Cited by 834
The multidimensional wisdom of crowds
Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010) · CaltechAUTHORS (California Institute of Technology) · Cited by 742
Evaluation of eye gaze interaction
Sibert, L. E., & Jacob, R. J. K. (2000) · Proceedings of the SIGCHI conference on Human Factors in Computing Systems · doi:10.1145/332040.332445 · Cited by 539
What guides a reader's eye movements?
Rayner, K., & McConkie, G. W. (1976) · Vision Research · doi:10.1016/0042-6989(76)90143-7 · Cited by 507
Saliency Detection: A Boolean Map Approach
Zhang, J., & Sclaroff, S. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.26 · Cited by 495
Objects predict fixations better than early saliency
Einhauser, W., Spain, M., & Perona, P. (2008) · Journal of Vision · doi:10.1167/8.14.18 · Cited by 451
Faces and text attract gaze independent of the task: Experimental data and computer model
Cerf, M., Frady, E. P., & Koch, C. (2009) · Journal of Vision · doi:10.1167/9.12.10 · Cited by 377
Predicting human gaze beyond pixels
Xu, J., Jiang, M., Wang, S., Kankanhalli, M. S., & Zhao, Q. (2014) · Journal of Vision · doi:10.1167/14.1.28 · Cited by 353
Self-citationLearning a saliency map using fixated locations in natural scenes
Zhao, Q., & Koch, C. (2011) · Journal of Vision · doi:10.1167/11.3.9 · Cited by 256
Self-citationFine-Grained Crowdsourcing for Fine-Grained Recognition
Deng, J., Krause, J., & Fei-Fei, L. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.81 · Cited by 249
An Eye Fixation Database for Saliency Detection in Images
Ramanathan, S., Katti, H., Sebe, N., Kankanhalli, M., & Chua, T.-S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_3 · Cited by 209
Saliency from hierarchical adaptation through decorrelation and variance normalization
Garcia-Diaz, A., Fdez-Vidal, X. R., Pardo, X. M., & Dosil, R. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2011.11.007 · Cited by 194
Pointer, J. S., & Hess, R. F. (1989) · Vision Research · doi:10.1016/0042-6989(89)90061-8 · Cited by 184
Training Object Class Detectors from Eye Tracking Data
Papadopoulos, D. P., Clarke, A. D. F., Keller, F., & Ferrari, V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_24 · Cited by 160
Pomplun, M., Reingold, E. M., & Shen, J. (2001) · Cognition · doi:10.1016/s0010-0277(01)00123-8 · Cited by 118
Soranus, Q. V. (1993) · Oxford University Press eBooks · doi:10.1093/oseo/instance.00076674 · Cited by 118
Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts
Navalpakkam, V., Jentzsch, L., Sayres, R., Ravi, S., Ahmed, A. A., & Smola, A. J. (2013) · International World Wide Web Conference (WWW) · doi:10.1145/2488388.2488471 · Cited by 107
Fixations on low-resolution images
Judd, T., Durand, F., & Torralba, A. B. (2011) · Journal of Vision · doi:10.1167/11.4.14 · Cited by 104
Improving searcher models using mouse cursor activity
Huang, J. J. S., White, R. W., Buscher, G., & Wang, K. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2348283.2348313 · Cited by 102
Gaze-contingent real-time simulation of arbitrary visual fields
Perry, J. S., & Geisler, W. S. (2002) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.469554 · Cited by 99
Jiang, M. X., Xu, J., & Zhao, Q. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10584-0_2 · Cited by 73
Self-citationImage invariance with changes in size: the role of peripheral contrast thresholds
Peli, E., Yang, J., & Goldstein, R. B. (1991) · Journal of the Optical Society of America A · doi:10.1364/josaa.8.001762 · Cited by 69
Acuity Perimetry and the Sampling Theory of Visual Resolution
Thibos, L. N. (1998) · Optometry and Vision Science · doi:10.1002/j.1538-9235.1998.tb05052.x · Cited by 50
Research works printed but not linked: 3
- 10.1097/00006324-199806000-00024 · doi:10.1097/00006324-199806000-00024
- hunziker, 2006, Im Auge des Lesers: foveale und periphere Wahrnehmung - vom Buchstabieren zur Lesefreude, Trans-media Verlag
- shen, 2014, Webpage saliency, ECCV, 33
Cited by
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879
“For experiments, it uses SALICON dataset [60] and achieves good results.”
MethodSalient Object Detection in the Deep Learning Era: An In-Depth Survey
Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H., & Yang, R. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3051099 · Cited by 811
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., & Durand, F. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2815601 · Cited by 802
Deep Visual Attention Prediction
Wang, W., & Shen, J. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2787612 · Cited by 585
Metadata
- Authors
- Jiang, M., Huang, S., Duan, J., & Zhao, Q.
- Affiliations
- National University of Singapore (SG)
- ORCID
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 1072-1080
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1934890906
- Cited by
- 729
- References
- 40
- Citations by year
- 2026: 30 · 2025: 58 · 2024: 75 · 2023: 76 · 2022: 75 · 2021: 100 · 2020: 93 · 2019: 84 · 2018: 60 · 2017: 44 · 2016: 27 · 2015: 7
- FWCI
- 18.87
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Keywords
- visual attention, saliency prediction, peripheral vision
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.