The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
A Multimodal Saliency Model for Videos With High Audio-Visual Correspondence
Abstract
Audio information has been bypassed by most of current visual attention prediction studies. However, sound could have influence on visual attention and such influence has been widely investigated and proofed by many psychological studies. In this paper, we propose a novel multi-modal saliency (MMS) model for videos containing scenes with high audio-visual correspondence. In such scenes, humans tend to be attracted by the sound sources and it is also possible to localize the sound sources via cross-modal analysis. Specifically, we first detect the spatial and temporal saliency maps from the visual modality by using a novel free energy principle. Then we propose to detect the audio saliency map from both audio and visual modalities by localizing the moving-sounding objects using cross-modal kernel canonical correlation analysis, which is first of its kind in the literature. Finally we propose a new two-stage adaptive audiovisual saliency fusion method to integrate the spatial, temporal and audio saliency maps to our audio-visual saliency map. The proposed MMS model has captured the influence of audio, which is not considered in the latest deep learning based saliency models. To take advantages of both deep saliency modeling and audio-visual saliency modeling, we propose to combine deep saliency models and the MMS model via a later fusion, and we find that an average of 5% performance gain is obtained. Experimental results on audio-visual attention databases show that the introduced models incorporating audio cues have significant superiority over state-of-the-art image and video saliency models which utilize a single visual modality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A model of saliency-based visual attention for rapid scene analysis
Itti, L., Koch, C., & Niebur, E. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.730558 · Cited by 11,271
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 4×
“1) Free Energy Principle: The free energy-based brain theory assumes that the cognitive process is governed by an internal generative model (IGM) G [35], [42].”
BackgroundInfluentialKernel Methods for Pattern Analysis
Shawe-Taylor, J., & Cristianini, N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809682 · Cited by 6,587
RELATIONS BETWEEN TWO SETS OF VARIATES
HOTELLING, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,263
Harel, J., Koch, C., & Perona, P. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0073 · Cited by 3,439
Saliency Detection: A Spectral Residual Approach
Hou, X., & Zhang, L. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383267 · Cited by 3,422
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,202
Learning to predict where humans look
Judd, T., Ehinger, K., Durand, F., & Torralba, A. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459462 · Cited by 2,016
State-of-the-Art in Visual Attention Modeling
Borji, A., & Itti, L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.89 · Cited by 1,817
A free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369
Salient Object Detection: A Benchmark
Borji, A., Cheng, M.-M., Jiang, H., & Li, J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2487833 · Cited by 1,364
SUN: A Bayesian framework for saliency using natural statistics
Zhang, L., Tong, M. H., Marks, T. K., Shan, H., & Cottrell, G. W. (2008) · Journal of Vision · doi:10.1167/8.7.32 · Cited by 1,227
Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.889 · Cited by 1,207
Information Retrieval for Music and Motion
Müller, M. (2007) · doi:10.1007/978-3-540-74048-3 · Cited by 818
Beyond pixels: exploring new representations and applications for motion analysis
Freeman, W. T., Adelson, E. H., & Liu, C. (2009) · Cited by 805
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., & Durand, F. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2815601 · Cited by 802
Spatio-temporal Saliency detection using phase spectrum of quaternion fourier transform
Guo, C., Ma, Q., & Zhang, L. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587715 · Cited by 754
Static and space-time visual saliency detection by self-resemblance
Seo, H. J., & Milanfar, P. (2009) · Journal of Vision · doi:10.1167/9.12.15 · Cited by 633
Using Free Energy Principle For Blind Image Quality Assessment
Gu, K., Zhai, G., Yang, X., & Zhang, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2373812 · Cited by 611
Self-citationVisual Saliency Based on Scale-Space Analysis in the Frequency Domain
Li, J., Levine, M. D., An, X., Xu, X., & He, H. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.147 · Cited by 574
A generic framework of user attention model and its application in video summarization
Ma, Y., Hua, X., Lu, L., & Zhang, H.-J. (2005) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2005.854410 · Cited by 545
Blind Image Quality Estimation via Distortion Aggravation
Min, X., Zhai, G., Gu, K., Liu, Y., & Yang, X. (2018) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2018.2816783 · Cited by 508
Self-citationSaliency Detection: A Boolean Map Approach
Zhang, J., & Sclaroff, S. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.26 · Cited by 495
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model
Cornia, M., Baraldi, L., Serra, G., & Cucchiara, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2851672 · Cited by 474
Predicting human gaze using low-level saliency combined with face detection
Cerf, M., Harel, J., Einhäuser, W., & Koch, C. (2008) · CaltechAUTHORS (California Institute of Technology) · Cited by 465
Blind Quality Assessment Based on Pseudo-Reference Image
Min, X., Gu, K., Zhai, G., Liu, J., Yang, X., & Chen, C. W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2788206 · Cited by 450
Self-citationSound enhances visual perception: Cross-modal effects of auditory organization on vision.
Vroomen, J., & Gelder, B. D. (2000) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.26.5.1583 · Cited by 434
Dynamic visual attention: searching for coding length increments
Hou, X., & Zhang, L. (2008) · CaltechAUTHORS (California Institute of Technology) · Cited by 431
Unified Blind Quality Assessment of Compressed Natural, Graphic, and Screen Content Images
Min, X., Ma, K., Gu, K., Zhai, G., Wang, Z., & Lin, W. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2735192 · Cited by 337
Self-citationSalGAN: Visual Saliency Prediction with Generative Adversarial Networks
Pan, J., Canton-Ferrer, C., McGuinness, K., O'Connor, N. E., Torres, J., Sayrol, E., & Giró-i-Nieto, X. (2017) · Dublin City University Open Access Institutional Repository (Dublin City University) · doi:10.48550/arxiv.1701.01081 · Cited by 315
Understanding Low- and High-Level Contributions to Fixation Prediction
Kummerer, M., Wallis, T. S. A., Gatys, L. A., & Bethge, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.513 · Cited by 282
A Psychovisual Quality Metric in Free-Energy Principle
Zhai, G., Wu, X., Yang, X., Lin, W., & Zhang, W. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2161092 · Cited by 276
Self-citationObjective Quality Evaluation of Dehazed Images
Min, X., Zhai, G., Gu, K., Yang, X., & Guan, X. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2018.2868771 · Cited by 269
Self-citationRevisiting Video Saliency: A Large-Scale Benchmark and a New Model
Wang, W., Shen, J., Guo, F., Cheng, M.-M., & Borji, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00514 · Cited by 268
Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention
Evangelopoulos, G., Zlatintsi, A., Potamianos, A., Maragos, P., Rapantzikos, K., Skoumas, G., & Avrithis, Y. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2267205 · Cited by 263
Quality Evaluation of Image Dehazing Methods Using Synthetic Hazy Images
Min, X., Zhai, G., Gu, K., Zhu, Y., Zhou, J., Guo, G., Yang, X., Guan, X., & Zhang, W. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2902097 · Cited by 252
Self-citationAuditory psychomotor coordination and visual search performance
Perrott, D. R., Saberi, K., Brown, K., & Strybel, T. Z. (1990) · Perception & Psychophysics · doi:10.3758/bf03211521 · Cited by 178
A Tutorial on Canonical Correlation Methods
Uurtio, V., Monteiro, J. M., Kandola, J., Shawe-Taylor, J. S., Fernández-Reyes, D., & Rousu, J. (2017) · ACM Computing Surveys · doi:10.1145/3136624 · Cited by 171
How saliency, faces, and sound influence gaze in dynamic social scenes
Coutrot, A., & Guyader, N. (2014) · Journal of Vision · doi:10.1167/14.8.5 · Cited by 158
Information-theoretic model comparison unifies saliency metrics
Kümmerer, M., Wallis, T. S. A., & Bethge, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1510393112 · Cited by 158
Fixation Prediction through Multimodal Analysis
Min, X., Zhai, G., Gu, K., & Yang, X. (2016) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/2996463 · Cited by 151
Self-citationSpatiotemporal Saliency Detection for Video Sequences Based on Random Walk With Restart
Kim, H., Kim, Y., Sim, J.-Y., & Kim, C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2425544 · Cited by 146
Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTMPreprint
Jiang, L. W., Xu, M., & Wang, Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06316 · Cited by 96
Multimodal Analysis for Identification and Segmentation of Moving-Sounding Objects
Izadinia, H., Saleemi, I., & Shah, M. A. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2228476 · Cited by 90
Influence of soundtrack on eye movements during video exploration
Coutrot, A., Guyader, N., Ionescu, G., & Caplier, A. (2012) · Journal of Eye Movement Research · doi:10.16910/jemr.5.4.2 · Cited by 90
Different types of sounds influence gaze differently in videos
Song, G., Pellerin, D., & Granjon, L. (2013) · Journal of Eye Movement Research · doi:10.16910/jemr.6.4.1 · Cited by 84
Visual Saliency Detection With Free Energy Theory
Gu, K., Zhai, G., Lin, W., et al. (2015) · IEEE Signal Processing Letters · doi:10.1109/lsp.2015.2413944 · Cited by 83
Self-citationA Saliency Prior Context Model for Real-Time Object Tracking
Ma, C., Miao, Z., Zhang, X.-P. S., & Li, M. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2694219 · Cited by 83
Self-citationDAVE: A Deep Audio-Visual Embedding for Dynamic Saliency PredictionPreprint
Tavakoli, H. R., Borji, A., Rahtu, E., & Kannala, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10693 · Cited by 54
A Tutorial on Canonical Correlation Methods
Uurtio, V, Monteiro, J., Kandola, J., Shawe-Taylor, J., Fernández-Reyes, D., & Rousu, J. (2018) · UCL Discovery (University College London) · Cited by 54
Chang, B. H. W., Krüger, U., Kustra, R., & Zhang, J. (2014) · Cited by 38
Free-energy principle inspired visual quality assessment: An overview
Zhai, G., Min, X., & Liu, N. (2019) · Digital Signal Processing · doi:10.1016/j.dsp.2019.02.017 · Cited by 38
Self-citationAudio Matters in Visual Attention
Chen, Y., Nguyen, T., Kankanhalli, M., Yuan, J., Yan, S., & Wang, M. (2014) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2014.2329380 · Cited by 34
Sound influences visual attention discriminately in videos
Min, X., Zhai, G., Gao, Z., Hu, C., & Yang, X. (2014) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2014.6982312 · Cited by 28
Self-citationSubjective Quality Evaluation of Foveated Video Coding Using Audio-Visual Focus of Attention
Lee, J., Francesca De Simone, A., & Ebrahimi, T. (2011) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2011.2165199 · Cited by 27
Cited by
Perceptual image quality assessment: a survey
Zhai, G., & Min, X. (2020) · Science China Information Sciences · doi:10.1007/s11432-019-2757-1 · Cited by 655
Self-citationTOPIQ: A Top-Down Approach From Semantics to Distortions for Image Quality Assessment
Chen, C., Mo, J., Hou, J., Wu, H., Liao, L., Sun, W., Yan, Q., & Lin, W. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3378466 · Cited by 276
Perceptual video quality assessment: a survey
Min, X., Duan, H., Sun, W., Zhu, Y., & Zhai, G. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4133-3 · Cited by 218
Self-citation
Metadata
- Authors
- Min, X., Zhai, G., Zhou, J., Zhang, X.-P., Yang, X., & Guan, X.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2020
- Volume
- 29
- Pages
- 3805-3819
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W3000351820
- Cited by
- 295
- References
- 55
- FWCI
- 11.39
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.