The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model
Open accessAbstract
In this work, we contribute to video saliency research in two ways. First, we introduce a new benchmark for predicting human eye movements during dynamic scene free-viewing, which is long-time urged in this field. Our dataset, named DHF1K (Dynamic Human Fixation), consists of 1K high-quality, elaborately selected video sequences spanning a large range of scenes, motions, object types and background complexity. Existing video saliency datasets lack variety and generality of common dynamic scenes and fall short in covering challenging situations in unconstrained environments. In contrast, DHF1K makes a significant leap in terms of scalability, diversity and difficulty, and is expected to boost video saliency modeling. Second, we propose a novel video saliency model that augments the CNN-LSTM network architecture with an attention mechanism to enable fast, end-to-end saliency learning. The attention mechanism explicitly encodes static saliency information, thus allowing LSTM to focus on learning more flexible temporal saliency representation across successive frames. Such a design fully leverages existing large-scale static fixation datasets, avoids overfitting, and significantly improves training efficiency and testing performance. We thoroughly examine the performance of our model, with respect to state-of-the-art saliency models, on three large-scale datasets (i.e., DHF1K, Hollywood2, UCF sports). Experimental results over more than 1.2K testing videos containing 400K frames demonstrate that our model outperforms other competitors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Computational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733
Frequency-tuned salient region detection
Achanta, R., Hemami, S., Estrada, F., & Susstrunk, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206596 · Cited by 4,167
Residual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
Saliency Detection: A Spectral Residual Approach
Hou, X., & Zhang, L. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383267 · Cited by 3,422
Global contrast based salient region detection
Cheng, M.-M., Zhang, G.-X., Mitra, N. J., Huang, X., & Hu, S.-M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995344 · Cited by 2,976
Self-citationA Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,656
Global Contrast Based Salient Region Detection
Cheng, M.-M., Mitra, N. J., Huang, X., Torr, P. H. S., & Hu, S.-M. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2345401 · Cited by 2,463
Self-citationLearning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998
State-of-the-Art in Visual Attention Modeling
Borji, A., & Itti, L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.89 · Cited by 1,817
Self-citationSalient Object Detection: A Benchmark
Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2487833 · Cited by 1,364
Self-citationThe Secrets of Salient Object Segmentation
Li, Y., Hou, X., Koch, C., Rehg, J. M., & Yuille, A. L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.43 · Cited by 1,338
A Decomposable Attention Model for Natural Language Inference
Parikh, A., Täckström, O., Das, D., & Uszkoreit, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1244 · Cited by 1,296
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., & Xiao, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03365 · Cited by 1,245
SUN: A Bayesian framework for saliency using natural statistics
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008) · Journal of Vision · doi:10.1167/8.7.32 · Cited by 1,227
Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition
Rodriguez, M. D., Ahmed, J., & Shah, M. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587727 · Cited by 1,223
Marszalek, M., Laptev, I., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206557 · Cited by 1,168
Deeply Supervised Salient Object Detection with Short Connections
Hou, Q., Cheng, M.-M., Hu, X., Borji, A., Tu, Z., & Torr, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.563 · Cited by 1,143
Self-citationSaliency Based on Information Maximization
Bruce, N. D. B., & Tsotsos, J. K. (2005) · neural information processing systems · Cited by 1,134
Learning to Detect A Salient Object
Liu, T., Sun, J., Zheng, N.-N., Tang, X., & Shum, H.-Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383047 · Cited by 946
Guo, C., & Zhang, L. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2030969 · Cited by 934
Automatic Foveation for Video Compression Using a Neurobiological Model of Visual Attention
Itti, L. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.834657 · Cited by 759
Wolfe, J. M. (2007) · Integrated Models of Cognitive Systems · doi:10.1093/acprof:oso/9780195189193.003.0008 · Cited by 740
Jiang, M., Huang, S., Duan, J., & Zhao, Q. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298710 · Cited by 729
A coherent computational approach to model bottom-up visual attention
Le Meur, O., Le Callet, P., Barba, D., & Thoreau, D. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.86 · Cited by 642
Video Salient Object Detection via Fully Convolutional Networks
Wang, W., Shen, J., & Shao, L. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2754941 · Cited by 642
Self-citationStatic and space-time visual saliency detection by self-resemblance
Seo, H. J., & Milanfar, P. (2009) · Journal of Vision · doi:10.1167/9.12.15 · Cited by 633
Quantitative Analysis of Human-Model Agreement in Visual Saliency Modeling: A Comparative Study
Borji, A., Sihite, D. N., & Itti, L. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2210727 · Cited by 613
Self-citationSALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks
Huang, X., Shen, C., Boix, X., & Zhao, Q. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.38 · Cited by 597
Deep Visual Attention Prediction
Wang, W., & Shen, J. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2787612 · Cited by 585
Self-citationSaliency-aware geodesic video object segmentation
Wang, W., Shen, J., & Porikli, F. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298961 · Cited by 524
Self-citationA Benchmark of Computational Models of Saliency to Predict Human Fixations
Judd, T., Durand, F., & Torralba, A. B. (2012) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 511
Neural correlates of attentive selection for color or luminance in extrastriate area V4
Motter, B. (1994) · The Journal of Neuroscience · doi:10.1523/jneurosci.14-04-02178.1994 · Cited by 482
DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations
Kruthiventi, S. S. S., Ayush, K., & Babu, R. V. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2710620 · Cited by 478
Predicting human gaze using low-level saliency combined with face detection
Cerf, M., Harel, J., Wolfgang Einhäuser, A., & Christof Koch, A. (2008) · CaltechAUTHORS (California Institute of Technology) · Cited by 473
Saliency-Aware Video Object Segmentation
Wang, W., Shen, J., Yang, R., & Porikli, F. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2662005 · Cited by 472
Self-citationShallow and Deep Convolutional Networks for Saliency Prediction
Pan, J., Sayrol, E., Giro-I-Nieto, X., McGuinness, K., & OConnor, N. E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.71 · Cited by 471
Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion
Mital, P. K., Smith, T. J., Hill, R. L., & Henderson, J. M. (2011) · Cognitive Computation · doi:10.1007/s12559-010-9074-z · Cited by 468
Dynamic visual attention: searching for coding length increments
Hou, X., & Zhang, L. (2008) · CaltechAUTHORS (California Institute of Technology) · Cited by 431
Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images
Vig, E., Dorr, M., & Cox, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.358 · Cited by 428
Spatiotemporal Saliency in Dynamic Scenes
Mahadevan, V., & Vasconcelos, N. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.112 · Cited by 409
Cao, C., Liu, X., Yang, Y., Yu, Y., Wang, J., Wang, Z., Huang, Y., Wang, L., Huang, C., Xu, W., et al. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.338 · Cited by 392
Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction
Noh, H., Seo, P. H., & Han, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.11 · Cited by 345
Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes
Itti, L. (2005) · Visual Cognition · doi:10.1080/13506280444000661 · Cited by 331
Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition
Mathe, S., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366154 · Cited by 217
Discriminant Saliency for Visual Recognition from Cluttered Scenes
Gao, D., & Vasconcelos, N. M. (2004) · Cited by 196
The discriminant center-surround hypothesis for bottom-up saliency
Gao, D., Mahadevan, V., & Vasconcelos, N. M. (2007) · neural information processing systems · Cited by 195
Video Saliency Incorporating Spatiotemporal Cues and Uncertainty Weighting
Fang, Y., Wang, Z., Lin, W., & Fang, Z. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2336549 · Cited by 193
Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction
Bak, Ç., Kocak, A., Erdem, E., & Erdem, A. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2777665 · Cited by 176
What is a Salient Object? A Dataset and a Baseline Model for Salient Object Detection
Borji, A. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2383320 · Cited by 175
Self-citationLearning Video Saliency from Human Gaze Using Candidate Selection
Rudoy, D., Goldman, D. B., Shechtman, E., & Zelnik-Manor, L. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.152 · Cited by 159
Correspondence Driven Saliency Transfer
Wang, W., Shen, J., Shao, L., et al. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2601784 · Cited by 143
Self-citationEnd-to-End Saliency Mapping via Probability Distribution Prediction
Jetley, S., Murray, N., & Vig, E. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.620 · Cited by 140
Where Should Saliency Models Look Next?
Bylinskii, Z., Recasens, A., Borji, A., Oliva, A., Torralba, A. B., & Durand, F. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_49 · Cited by 138
Self-citationDeep Cropping via Attention Box Prediction and Aesthetics Assessment
Wang, W., & Shen, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.240 · Cited by 126
Self-citationLearning to Predict Eye Fixations via Multiresolution Convolutional Neural Networks
Liu, N., Han, J., Liu, T., & Li, X. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2628878 · Cited by 112
How many bits does it take for a stimulus to be salient?
Khatoonabadi, S. H., Vasconcelos, N. M., Bajić, I. V., & Shan, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299189 · Cited by 110
Wischnewski, M., Belardinelli, A., Schneider, W. X., & Steil, J. J. (2010) · Cognitive Computation · doi:10.1007/s12559-010-9080-1 · Cited by 97
Leborán, V., García-Díaz, A., Fdez-Vidal, X. R., & Pardo, X. M. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2567391 · Cited by 93
Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTMPreprint
Jiang, L. W., Xu, M., & Wang, Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06316 · Cited by 86
Cited by
A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping
Wang, W., Shen, J., & Ling, H. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2840724 · Cited by 348
Self-citationHuman Gaze Assisted Artificial Intelligence: A Review
Zhang, R., Saran, A., Liu, B., Zhu, Y., Guo, S., Niekum, S., Ballard, D., & Hayhoe, M. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/689 · Cited by 48
Metadata
- Authors
- Wang, W., Shen, J., Guo, F., Cheng, M.-M., & Borji, A.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 4894-4903
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962965915
- Cited by
- 268
- References
- 67
- FWCI
- 12.08
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
