The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index.
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Open accessAbstract
How best to evaluate a saliency model's ability to predict where humans look in images is an open research question. The choice of evaluation metric depends on how saliency is defined and how the ground truth is represented. Metrics differ in how they rank saliency models, and this results from how false positives and false negatives are treated, whether viewing biases are accounted for, whether spatial deviations are factored in, and how the saliency maps are pre-processed. In this paper, we provide an analysis of 8 different evaluation metrics and their properties. With the help of systematic experiments and visualizations of metric computations, we add interpretability to saliency scores and more transparency to the evaluation of saliency models. Building off the differences in metric properties and behaviors, we make recommendations for metric selections under specific assumptions and for specific applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
(1966) · Medical Entomology and Zoology · Cited by 10,293
Image registration methods: a survey
Zitová, B., & Flusser, J. (2003) · Image and Vision Computing · doi:10.1016/s0262-8856(03)00137-9 · Cited by 6,423
Content-based image retrieval at the end of the early years
Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895972 · Cited by 6,056
Swain, M. J., & Ballard, D. H. (1991) · International Journal of Computer Vision · doi:10.1007/bf00130487 · Cited by 5,623
The Earth Mover's Distance as a Metric for Image Retrieval
Rubner, Y., Tomasi, C., & Guibas, L. J. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026543900054 · Cited by 4,568
Frequency-tuned salient region detection
Achanta, R., Hemami, S., Estrada, F., & Susstrunk, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206596 · Cited by 4,167
Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry
Koch, C., & Ullman, S. (1987) · Matters of Intelligence · doi:10.1007/978-94-009-3833-5_5 · Cited by 3,722
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
IARBUS, A. L. (1967) · Medical Entomology and Zoology · Cited by 3,319
Yarbus, A. L. (1967) · doi:10.1007/978-1-4899-5379-7 · Cited by 3,304
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
Self-citationState-of-the-Art in Visual Attention Modeling
Borji, A., & Itti, L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.89 · Cited by 1,817
A survey of content-based image retrieval with high-level semantics
Liu, Y., Zhang, D., Lu, G., & Ma, W.-Y. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.04.045 · Cited by 1,696
Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006) · Psychological Review · doi:10.1037/0033-295x.113.4.766 · Cited by 1,639
Self-citationModeling the role of salience in the allocation of overt visual attention
Parkhurst, D., Law, K., & Niebur, E. (2002) · Vision Research · doi:10.1016/s0042-6989(01)00250-4 · Cited by 1,388
Seam carving for content-aware image resizing
Avidan, S., & Shamir, A. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276390 · Cited by 1,362
Modeling attention to salient proto-objects
Walther, D., & Koch, C. (2006) · Neural Networks · doi:10.1016/j.neunet.2006.10.001 · Cited by 1,249
SUN: A Bayesian framework for saliency using natural statistics
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008) · Journal of Vision · doi:10.1167/8.7.32 · Cited by 1,227
Saliency Based on Information Maximization
Bruce, N. D. B., & Tsotsos, J. K. (2005) · neural information processing systems · Cited by 1,134
How to Evaluate Foreground Maps
Margolin, R., Zelnik-Manor, L., & Tal, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.39 · Cited by 1,106
Seam carving for content-aware image resizing
Avidan, S., & Shamir, A. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276390 · Cited by 1,092
Tatler, B. W. (2007) · Journal of Vision · doi:10.1167/7.14.4 · Cited by 1,028
Learning to Detect A Salient Object
Liu, T., Sun, J., Zheng, N.-N., Tang, X., & Shum, H.-Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383047 · Cited by 946
Saliency, attention, and visual search: An information theoretic approach
Bruce, N. D. B., & Tsotsos, J. K. (2009) · Journal of Vision · doi:10.1167/9.3.5 · Cited by 855
Salient Region Detection and Segmentation
Achanta, R., Estrada, F., Wils, P., & Süsstrunk, S. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-79547-6_7 · Cited by 844
Fast and robust Earth Mover's Distances
Pele, O., & Werman, M. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459199 · Cited by 805
Automatic Foveation for Video Compression Using a Neurobiological Model of Visual Attention
Itti, L. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.834657 · Cited by 759
Improved seam carving for video retargeting
Rubinstein, M., Shamir, A., & Avidan, S. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360615 · Cited by 742
Components of bottom-up gaze allocation in natural images
Peters, R. J., Iyer, A., Itti, L., & Koch, C. (2005) · Vision Research · doi:10.1016/j.visres.2005.03.019 · Cited by 739
Jiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729
Visual correlates of fixation selection: effects of scale and time
Tatler, B. W., Baddeley, R. J., & Gilchrist, I. D. (2005) · Vision Research · doi:10.1016/j.visres.2004.09.017 · Cited by 700
A coherent computational approach to model bottom-up visual attention
Le Meur, O., Le Callet, P., Barba, D., & Thoreau, D. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.86 · Cited by 642
Static and space-time visual saliency detection by self-resemblance
Seo, H. J., & Milanfar, P. (2009) · Journal of Vision · doi:10.1167/9.12.15 · Cited by 633
Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study
Borji, A., Sihite, D. N., & Itti, L. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2210727 · Cited by 613
The gaze selects informative details within pictures
Mackworth, N. H., & Morandi, A. J. (1967) · Perception & Psychophysics · doi:10.3758/bf03210264 · Cited by 595
Visual Saliency Based on Scale-Space Analysis in the Frequency Domain
Li, J., Levine, M. D., An, X., Xu, X., & He, H. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.147 · Cited by 574
Scanpaths in saccadic eye movements while viewing and recognizing patterns
Noton, D., & Stark, L. (1971) · Vision Research · doi:10.1016/0042-6989(71)90213-6 · Cited by 539
A generic framework of user attention model and its application in video summarization
Ma, Y., Hua, X., Lu, L., & Zhang, H.-J. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.854410 · Cited by 516
A Benchmark of Computational Models of Saliency to Predict Human Fixations
Judd, T., Durand, F., & Torralba, A. B. (2012) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 511
Self-citationObjects predict fixations better than early saliency
Einhauser, W., Spain, M., & Perona, P. (2008) · Journal of Vision · doi:10.1167/8.14.18 · Cited by 451
Stylization and abstraction of photographs
DeCarlo, D., & Santella, A. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566650 · Cited by 432
Automatic thumbnail cropping and its effectiveness
Suh, B., Ling, H., Bederson, B. B., & Jacobs, D. W. (2003) · Proceedings of the 16th annual ACM symposium on User interface software and technology · doi:10.1145/964696.964707 · Cited by 421
Visual saliency estimation by nonlinearly integrating features using region covariances
Erdem, E., & Erdem, A. (2013) · Journal of Vision · doi:10.1167/13.4.11 · Cited by 396
Center-surround divergence of feature statistics for salient object detection
Klein, D. A., & Frintrop, S. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126499 · Cited by 369
Real-time foveated multiresolution system for low-bandwidth video communication:
Geisler, W. S., & Perry, J. S. (1998) · SPIE Proceedings · doi:10.1117/12.320120 · Cited by 369
An Integrated Model of Top-Down and Bottom-Up Attention for Optimizing Detection Speed
Navalpakkam, V., & Itti, L. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.54 · Cited by 369
Quantifying center bias of observers in free viewing of dynamic natural scenes
Tseng, P. H., Carmi, R., Cameron, I. G. M., Munoz, D. P., & Itti, L. (2009) · Journal of Vision · doi:10.1167/9.7.4 · Cited by 362
Bayesian Surprise Attracts Human Attention
Itti, L., & Baldi, P. (2005) · neural information processing systems · Cited by 362
Predicting human gaze beyond pixels
Xu, J., Jiang, M., Wang, S., Kankanhalli, M. S., & Zhao, Q. (2014) · Journal of Vision · doi:10.1167/14.1.28 · Cited by 353
Gaze-based interaction for semi-automatic photo cropping
Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., & Cohen, M. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124886 · Cited by 352
A framework for visual saliency detection with applications to image thumbnailing
Marchesotti, L., Cifarelli, C., & Csurka, G. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459467 · Cited by 322
Scene content selected by active vision
Parkhurst, D., & Niebur, E. (2003) · Spatial Vision · doi:10.1163/15685680360511645 · Cited by 318
Methods for comparing scanpaths and saliency maps: strengths and weaknesses
Le Meur, O., & Baccino, T. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0226-9 · Cited by 314
A Linear Time Histogram Metric for Improved SIFT Matching
Pele, O., & Werman, M. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88690-7_37 · Cited by 305
Predicting visual fixations on video based on low-level visual features
Le Meur, O., Le Callet, P., & Barba, D. (2007) · Vision Research · doi:10.1016/j.visres.2007.06.015 · Cited by 305
Perceptual Metrics for Image Database Navigation
Rubner, Y., & Tomasi, C. (2001) · Springer eBooks · doi:10.1007/978-1-4757-3343-3 · Cited by 296
Learning a saliency map using fixated locations in natural scenes
Zhao, Q., & Koch, C. (2011) · Journal of Vision · doi:10.1167/11.3.9 · Cited by 256
Non-parametric similarity measures for unsupervised texture segmentation and image retrieval
Puzicha, J., Hofmann, T., & Buhmann, J. M. (2002) · Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1997.609331 · Cited by 254
Does luminance‐contrast contribute to a saliency map for overt visual attention?
Einhäuser, W., & König, P. (2003) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.2003.02508.x · Cited by 238
Stylization and abstraction of photographs
DeCarlo, D., & Santella, A. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566650 · Cited by 231
Analysis of Scores, Datasets, and Models in Visual Saliency Prediction
Borji, A., Tavakoli, H. R., Sihite, D. N., & Itti, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.118 · Cited by 221
Computational versus Psychophysical Bottom-Up Image Saliency: A Comparative Evaluation Study
Toet, A. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.53 · Cited by 214
Saliency detection for content-aware image resizing
Achanta, R., & Susstrunk, S. E. (2009) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2009.5413815 · Cited by 205
Foveation scalable video coding with automatic fixation selection
Wang, Z., Lu, L., & Bovik, A. C. (2003) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.809015 · Cited by 196
The discriminant center-surround hypothesis for bottom-up saliency
Gao, D., Mahadevan, V., & Vasconcelos, N. M. (2007) · neural information processing systems · Cited by 195
Information-theoretic model comparison unifies saliency metrics
Kümmerer, M., Wallis, T. S. A., & Bethge, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510393112 · Cited by 158
Supervised Evaluation of Image Segmentation and Object Proposal Techniques
Pont-Tuset, J., & Marqués, F. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2481406 · Cited by 141
Where Should Saliency Models Look Next?
Bylinskii, Z., Recasens, A., Borji, A., Oliva, A., Torralba, A. B., & Durand, F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_49 · Cited by 138
Self-citationImproved seam carving for video retargeting
Rubinstein, M., Shamir, A., & Avidan, S. (2008) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1399504.1360615 · Cited by 123
An information maximization model of eye movements.
Renninger, L. W., Coughlan, J. M., Verghese, P., & Malik, J. (2005) · PubMed · Cited by 108
Image Matching via Saliency Region Correspondences
Toshev, A., Shi, J., & Daniilidis, K. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.382973 · Cited by 95
Fixations on low-resolution images
Judd, T., Durand, F., & Torralba, A. B. (2011) · Journal of Vision · doi:10.1167/11.4.14 · Cited by 94
Self-citationOn the Efficient Evaluation of Probabilistic Similarity Functions for Image Retrieval
Vasconcelos, N. M. (2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.830760 · Cited by 90
On computational modeling of visual saliency: Examining what’s right, and what’s left
Bruce, N. D. B., Wloka, C., Frosst, N., Rahman, S., & Tsotsos, J. K. (2015) · Vision Research · doi:10.1016/j.visres.2015.01.010 · Cited by 80
Studying Relationships between Human Gaze, Description, and Computer Vision
Yun, K., Peng, Y., Samaras, D., Zelinsky, G. J., & Berg, T. L. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.101 · Cited by 79
Goferman, S., Tal, A., & Zelnik‐Manor, L. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01615.x · Cited by 73
Mobile robot vision navigation & localization using Gist and Saliency
Chang, C.-K., Siagian, C., & Itti, L. (2010) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2010.5649136 · Cited by 73
Measures and Limits of Models of Fixation Selection
Wilming, N., Betz, T., Kietzmann, T. C., & Peter König, B. (2011) · PLoS ONE · doi:10.1371/journal.pone.0024038 · Cited by 69
Deriving an appropriate baseline for describing fixation behaviour
Clarke, A. D. F., & Tatler, B. W. (2014) · Vision Research · doi:10.1016/j.visres.2014.06.016 · Cited by 59
Towards the quantitative evaluation of visual attention models
Bylinskii, Z., DeGennaro, E. M., Rajalingham, R., Ruda, H., Zhang, J., & Tsotsos, J. K. (2015) · Vision Research · doi:10.1016/j.visres.2015.04.007 · Cited by 58
Self-citationComparative Study of Fixation Density Maps
Engelke, U., Liu, H., Wang, J., Callet, P. L., Heynderickx, I. E. J., Zepernick, H., & Maeder, A. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2227767 · Cited by 45
General object tracking with a component-based target descriptor
Frintrop, S. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509638 · Cited by 41
López‐García, F., Ramon, X., Manuel, X., et al. (2011) · InTech eBooks · doi:10.5772/14343 · Cited by 19
Simultaneous Robot Localization and Mapping Based on a Visual Attention System
Frintrop, S., Jensfelt, P., & Christensen, H. I. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-77343-6_27 · Cited by 17
Selection of a best metric and evaluation of bottom-up visual saliency models
Emami, M., & Hoberock, L. L. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2013.08.004 · Cited by 17
High-level aspects of oculomotor control during viewing of natural-task images
Canosa, R. L., Pelz, J. B., Mennie, N. R., & Peak, J. (2003) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.477375 · Cited by 11
Spatially Binned ROC: A Comprehensive Saliency Metric
Wloka, C., & Tsotsos, J. K. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.63 · Cited by 9
How close are we to understanding image-based saliency?Preprint
Kümmerer, M., Wallis, T. S. A., & Bethge, M. (2014) · arXiv (Cornell University) · doi:10.15496/publikation-4858 · Cited by 6
Cited by
Predicting the Driver's Focus of Attention: The DR(eye)VE Project
Palazzi, A., Abati, D., Calderara, S., Solera, F., & Cucchiara, R. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2845370 · Cited by 313
Evaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163
Grounded Human-Object Interaction Hotspots From Video
Nagarajan, T., Feichtenhofer, C., & Grauman, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00878 · Cited by 147
Self-Supervised Representation Learning for Ultrasound Video
Jiao, J., Droste, R., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098666 · Cited by 56
Human Gaze Assisted Artificial Intelligence: A Review
Zhang, R., Saran, A., Liu, B., Zhu, Y., Guo, S., Niekum, S., Ballard, D., & Hayhoe, M. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/689 · Cited by 48
Metadata
- Authors
- Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., & Durand, F.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-03-01
- Volume
- 41
- Issue
- 3
- Pages
- 740-757
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2336525740
- Cited by
- 802
- References
- 90
- FWCI
- 44.36
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.