Learning to Localize Sound Source in Visual Scenes
Open accessAbstract
Visual events are usually accompanied by sounds in our daily lives. We pose the question: Can the machine learn the correspondence between visual scene and the sound, and localize the sound source only by observing sound and visual scene pairs like human? In this paper, we propose a novel unsupervised algorithm to address the problem of localizing the sound source in visual scenes. A two-stream network structure which handles each modality, with attention mechanism is developed for sound source localization. Moreover, although our network is formulated within the unsupervised learning framework, it can be extended to a unified architecture with a simple modification for the supervised and semi-supervised learning settings as well. Meanwhile, a new sound source dataset is developed for performance evaluation. Our empirical evaluation shows that the unsupervised method eventually go through false conclusion in some cases. We also show that even with a few supervision, i.e., semi-supervised setup, false conclusion is able to be corrected effectively.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
L1 loss and binary classification loss are widely used for audio-visual representation learning.
“Except for marginbased loss, L1 loss and binary classification loss [48, 149, 150, 151] are also widely used for A-V representations learning.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [149]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Control of goal-directed and stimulus-driven attention in the brain
Corbetta, M., & Shulman, G. L. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn755 · Cited by 12,881 · see citing works
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Understanding Machine Learning: From Theory to Algorithms
Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115 · see citing works
(2015) · Cited by 2,857 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Deep Metric Learning Using Triplet Network
Hoffer, E., & Ailon, N. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24261-3_7 · Cited by 2,134 · see citing works
Optimum Array Processing: Part IV of Detection, Estimationand Modulation Theory
Trees, H. L. V. (2013) · Cited by 1,684 · see citing works
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,632 · see citing works
Skinner, B. F. (1948) · Journal of Experimental Psychology · doi:10.1037/h0055873 · Cited by 1,456 · cited 1× · see citing works
“1It is an experiment [26] about delivering food to hungry pigeons in a cage at regular time intervals regardless of the bird behavior.”
BackgroundDeep content-based music recommendation
van den Oord, A., Dieleman, S., & Schrauwen, B. (2013) · Ghent University Academic Bibliography (Ghent University) · Cited by 953 · see citing works
What in the World Do We Hear?: An Ecological Approach to Auditory Event Perception
Gaver, W. W. (1993) · Ecological Psychology · doi:10.1207/s15326969eco0501_1 · Cited by 903 · see citing works
Arandjelović, R., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 836 · see citing works
SoundNet: Learning Sound Representations from Unlabeled VideoPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09001 · Cited by 658 · see citing works
Ambient Sound Provides Supervision for Visual Learning
Owens, A., Wu, J., McDermott, J. H., Freeman, W. T., & Torralba, A. B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_48 · Cited by 385 · see citing works
Owens, A., Isola, P., McDermott, J. H., Torralba, A. B., Adelson, E. H., & Freeman, W. T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.264 · Cited by 352 · see citing works
Visual question answering: Datasets, algorithms, and future challenges
Kafle, K., & Kanan, C. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.005 · Cited by 245 · see citing works
Learning Image Representations Tied to Ego-Motion
Jayaraman, D., & Grauman, K. L. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.166 · Cited by 229 · see citing works
Audio Vision: Using Audio-Visual Synchrony to Locate Sounds
Hershey, J. R., & Movellan, J. R. (1999) · Cited by 227 · see citing works
Kidron, E., Schechner, Y. Y., & Elad, M. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.274 · Cited by 183 · see citing works
Learning Joint Statistical Models for Audio-Visual Fusion and Segregation
Fisher, J. S., Darrell, T., Freeman, W. T., & Viola, P. A. (2000) · Cited by 176 · see citing works
Majdak, P., Goupell, M. J., & Laback, B. (2010) · Attention Perception & Psychophysics · doi:10.3758/app.72.2.454 · Cited by 151 · see citing works
See, Hear, and Read: Deep Aligned RepresentationsPreprint
Aytar, Y., Vondrick, C., & Torralba, A. B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00932 · Cited by 117 · see citing works
Aurally Aided Visual Search in Three-Dimensional Space
Bolia, R. S., D’Angelo, W. R., & McKinley, R. L. (1999) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872099779656789 · Cited by 112 · see citing works
The influence of vision on the absolute identification of sound-source position
Shelton, B. R., & Searle, C. L. (1980) · Perception & Psychophysics · doi:10.3758/bf03198830 · Cited by 104 · see citing works
Barzelay, Z., & Schechner, Y. Y. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383344 · Cited by 92 · see citing works
Multimodal Analysis for Identification and Segmentation of Moving-Sounding Objects
Izadinia, H., Saleemi, I., & Shah, M. A. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2228476 · Cited by 90 · see citing works
Aurally Aided Visual Search under Virtual and Free-Field Listening Conditions
Perrott, D. R., Cisneros, J., McKinley, R. L., & D’Angelo, W. R. (1996) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872096778827260 · Cited by 81 · see citing works
Eye movements in auditory space perception
Jones, B., & Kabanoff, B. (1975) · Perception & Psychophysics · doi:10.3758/bf03203206 · Cited by 58 · see citing works
Seeing the Sound: A New Multimodal Imaging Device for Computer Vision
Zunino, A., Crocco, M., Martelli, S., Trucco, A., Bue, A. D., & Murino, V. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.95 · Cited by 29 · see citing works
Suggesting Sounds for Images from Video Collections
Solèr, M., Bazin, J., Wang, O., Krause, A., & Sorkine‐Hornung, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48881-3_59 · Cited by 11 · see citing works
Flagged first, then by citations
2 of the 37 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704 · see citing works
Not only Look, But Also Listen: Learning Multimodal Violence Detection Under Weak Supervision
Wu, P., Liu, J., Shi, Y., Sun, Y., Shao, F., Wu, Z., & Yang, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_20 · Cited by 521 · see citing works
Audio-Visual Event Localization in Unconstrained Videos
Tian, Y., Shi, J., Li, B., Duan, Z., & Xu, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_16 · Cited by 506 · see citing works
Arandjelović, R., & Zisserman, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_27 · Cited by 448 · see citing works
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J. H., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_35 · Cited by 435 · see citing works
EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition
Kazakos, E., Nagrani, A., Zisserman, A., & Damen, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00559 · Cited by 10 · see citing works
Self-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_13 · Cited by 221 · see citing works
Dual Attention Matching for Audio-Visual Event Localization
Wu, Y., Zhu, L., Yan, Y., & Yang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00639 · Cited by 220 · see citing works
Audio-Visual Instance Discrimination with Cross-Modal Agreement
Morgado, P. M., Vasconcelos, N. M., & Misra, I. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01229 · Cited by 218 · see citing works
Zhao, H., Gan, C., Ma, W.-C., & Torralba, A. B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00182 · Cited by 218 · see citing works
Deep Multimodal Clustering for Unsupervised Audiovisual Learning
Hu, D., Nie, F., & Li, X. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00947 · Cited by 202 · see citing works
Co-Separating Sounds of Visual Objects
Gao, R., & Grauman, K. L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00398 · Cited by 199 · see citing works
SoundSpaces: Audio-Visual Navigation in 3D Environments
Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al-Halah, Z., Ithapu, V. K., Robinson, P., & Grauman, K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_2 · Cited by 202 · see citing works
Music Gesture for Visual Sound Separation
Gan, C., Huang, D. Y., Zhao, H., Tenenbaum, J. B., & Torralba, A. B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01049 · Cited by 197 · see citing works
Deep Audio-visual Learning: A Survey
Zhu, H., Luo, M., Wang, R., Zheng, A., & He, R. (2021) · Machine Intelligence Research · doi:10.1007/s11633-021-1293-0 · Cited by 185 · see citing works
Localizing Visual Sounds the Hard Way
Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., & Zisserman, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01659 · Cited by 178 · see citing works
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Gao, R., & Grauman, K. L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01524 · Cited by 193 · see citing works
Speech2Face: Learning the Face Behind a Voice
Oh, T.-H., Dekel, T., Kim, C., Mosseri, I., Freeman, W. T., Rubinstein, M., & Matusik, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00772 · Cited by 167 · see citing works
Self-citationUnified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
Tian, Y., Li, D., & Xu, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_26 · Cited by 159 · see citing works
Audiovisual SlowFast Networks for Video RecognitionPreprint
Xiao, F., Lee, Y. J., Grauman, K. L., Malik, J., & Feichtenhofer, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.08740 · Cited by 195 · see citing works
Multiple Sound Sources Localization from Coarse to Fine
Qian, R., Hu, D., Dinkel, H., Wu, M., Xu, N., & Lin, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58565-5_18 · Cited by 156 · see citing works
Gao, R., & Grauman, K. L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00041 · Cited by 151 · see citing works
Self-Supervised Moving Vehicle Tracking With Stereo Sound
Gan, C., Zhao, H., Chen, P., Cox, D., & Torralba, A. B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00715 · Cited by 150 · see citing works
Zhou, J., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S. T., Guo, D., Kong, L., Wang, M., & Zhong, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_22 · Cited by 147 · see citing works
Dual-modality Seq2Seq Network for Audio-visual Event Localization
Lin, Y.-B., Li, Y.-J., & Wang, Y.-C. F. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683226 · Cited by 129 · see citing works
Foley Music: Learning to Generate Music from Videos
Gan, C., Huang, D. Y., Chen, P., Tenenbaum, J. B., & Torralba, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58621-8_44 · Cited by 126 · see citing works
Positive Sample Propagation along the Audio-Visual Event Line
Zhou, J., Zheng, L., Zhong, Y., Hao, S., & Wang, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00833 · Cited by 126 · see citing works
Look, Listen, and Act: Towards Audio-Visual Embodied Navigation
Gan, C., Zhang, Y., Wu, J., Gong, B., & Tenenbaum, J. B. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197008 · Cited by 124 · see citing works
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
Li, G., Wei, Y., Tian, Y., Xu, C., Wen, J.-R., & Hu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01852 · Cited by 124 · see citing works
Weakly Supervised Audio-Visual Violence Detection
Wu, P., Liu, X., & Liu, J. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3147369 · Cited by 115 · see citing works
Contrastive self-supervised learning: review, progress, challenges and future research directions
Kumar, P., Rawat, P., & Chauhan, S. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-022-00245-6 · Cited by 118 · see citing works
Robust Audio-Visual Instance Discrimination
Morgado, P. M., Misra, I., & Vasconcelos, N. M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01274 · Cited by 108 · see citing works
Vision Transformers are Parameter-Efficient Audio-Visual Learners
Lin, Y.-B., Sung, Y.-L., Lei, J., Bansal, M., & Bertasius, G. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00228 · Cited by 113 · see citing works
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682524 · Cited by 108 · see citing works
ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction
Jain, S., Yarlagadda, P., Jyoti, S., Karthik, S., Subramanian, R., & Gandhi, V. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635989 · Cited by 103 · see citing works
Acoustic Non-Line-Of-Sight Imaging
Lindell, D. B., Wetzstein, G., & Koltun, V. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00694 · Cited by 102 · see citing works
Self-supervised Audio-visual Co-segmentation
Rouditchenko, A., Zhao, H., Gan, C., McDermott, J. H., & Torralba, A. B. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682467 · Cited by 99 · see citing works
AudioCaps: Generating Captions for Audios in The Wild
Kim, C. D., Kim, B., Lee, H., & Kim, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1011 · Cited by 96 · see citing works
Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning
Rahman, T., Xu, B., & Sigal, L. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00900 · Cited by 95 · see citing works
Exploring Heterogeneous Clues for Weakly-Supervised Audio-Visual Video Parsing
Wu, Y., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00138 · Cited by 95 · see citing works
A Survey of Sound Source Localization and Detection Methods and Their Applications
Jekateryńczuk, G., & Piotrowski, Z. (2023) · Sensors · doi:10.3390/s24010068 · Cited by 95 · see citing works
Labelling unlabelled videos from scratch with multi-modal self-supervision
Asano, Y. M., Patrick, M., Rupprecht, C., & Vedaldi, A. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2006.13662 · Cited by 94 · see citing works
Broaden Your Views for Self-Supervised Video Learning
Recasens, A., Luc, P., Alayrac, J.-B., Wang, L., Strub, F., Tallec, C., Malinowski, M., Pătrăucean, V., Altché, F., Valko, M., et al. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00129 · Cited by 90 · see citing works
Audiovisual Transformer with Instance Attention for Audio-Visual Event Localization
Lin, Y.-B., & Wang, Y.-C. F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69544-6_17 · Cited by 89 · see citing works
On Learning Associations of Faces and Voices
Kim, C., Shin, H. V., Oh, T.-H., Kaspar, A., Elgharib, M. A., & Matusik, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20873-8_18 · Cited by 88 · see citing works
Self-citationDiscriminative Sounding Objects Localization via Self-supervised Audiovisual MatchingPreprint
Hu, D., Qian, R., Jiang, M., Tan, X., Wen, S., Ding, E., Lin, W., & Dou, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05466 · Cited by 88 · see citing works
STAViS: Spatio-Temporal AudioVisual Saliency Network
Tsiami, A., Koutras, P., & Maragos, P. A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00482 · Cited by 86 · see citing works
Recursive Visual Sound Separation Using Minus-Plus Net
Xu, X., Dai, B., & Lin, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00097 · Cited by 82 · see citing works
Vision-Infused Deep Audio Inpainting
Zhou, H., Liu, Z., Xu, X., Luo, P., & Wang, X. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00037 · Cited by 80 · see citing works
Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
Zhou, H., Xu, X., Lin, D., Wang, X., & Liu, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_4 · Cited by 80 · see citing works
VisualEchoes: Spatial Image Representation Learning Through Echolocation
Gao, R., Chen, C., Al-Halah, Z., Schissler, C., & Grauman, K. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_38 · Cited by 79 · see citing works
See the Sound, Hear the Pixels
Ramaswamy, J., & Das, S. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093616 · Cited by 78 · see citing works
Valverde, F. R., Hurtado, J. V., & Valada, A. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01144 · Cited by 78 · see citing works
Telling Left From Right: Learning Spatial Correspondence of Sight and Sound
Yang, K., Russell, B. C., & Salamon, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00995 · Cited by 76 · see citing works
Audio-Visual Segmentation with Semantics
Zhou, J., Shen, X., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S. T., Guo, D., Kong, L., Wang, M., et al. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02261-x · Cited by 74 · see citing works
Learning Representations from Audio-Visual Spatial AlignmentPreprint
Morgado, P. M., Li, Y., & Vasconcelos, N. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01819 · Cited by 74 · see citing works
Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention
Duan, B., Tang, H., Wang, W., Zong, Z., Yang, G., & Yan, Y. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00406 · Cited by 70 · see citing works
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
Gao, R., Si, Z., Chang, Y., Clarke, S., Bohg, J., Fei-Fei, L., Yuan, W., & Wu, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01034 · Cited by 69 · see citing works
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
Tian, Y., Hu, D., & Xu, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00277 · Cited by 67 · see citing works
Localizing Visual Sounds the Easy Way
Mo, S., & Morgado, P. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_13 · Cited by 67 · see citing works
Lessons from infant learning for unsupervised machine learning
Zaadnoordijk, L., Besold, T. R., & Cusack, R. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00488-2 · Cited by 65 · see citing works
What Makes the Sound?: A Dual-Modality Interacting Network for Audio-Visual Event Localization
Ramaswamy, J. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053895 · Cited by 65 · see citing works
Deep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64 · see citing works
Adversarial-Metric Learning for Audio-Visual Cross-Modal Matching
Zheng, A., Hu, M., Jiang, B., Huang, Y., Yan, Y., & Luo, B. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3050089 · Cited by 62 · see citing works
Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications
Senocak, A., Oh, T.-H., Kim, J., Yang, M., & Kweon, I. S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2952095 · Cited by 61 · see citing works
Self-citationBatVision: Learning to See 3D Spatial Layout with Two Ears
Christensen, J. H., Hornauer, S., & Yu, S. X. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196934 · Cited by 56 · see citing works
Self-supervised object detection from audio-visual correspondence
Afouras, T., Asano, Y. M., Fagan, F., Vedaldi, A., & Metze, F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01032 · Cited by 56 · see citing works
Visually Informed Binaural Audio Generation without Binaural Audios
Xu, X., Zhou, H., Liu, Z., Dai, B., Wang, X., & Lin, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01523 · Cited by 55 · see citing works
Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning
Sun, W., Zhang, J., Wang, J., Liu, Z., Zhong, Y., Feng, T., Guo, Y., Zhang, Y., & Barnes, N. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00621 · Cited by 53 · see citing works
Learning to Set Waypoints for Audio-Visual Navigation
Chen, C., Majumder, S., Al-Halah, Z., Gao, R., Ramakrishnan, S. K., & Grauman, K. (2020) · arXiv · doi:10.48550/arxiv.2008.09622 · Cited by 53 · see citing works
AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos With Deep Learning
Ghose, S., & Prevost, J. J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3005033 · Cited by 51 · see citing works
Mix and Localize: Localizing Sound Sources in Mixtures
Hu, X., Chen, Z., & Owens, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01023 · Cited by 53 · see citing works
Disentangled Representation Learning for Cross-Modal Biometric Matching
Ning, H., Zheng, X., Lu, X., & Yuan, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3071243 · Cited by 51 · see citing works
Parida, K. K., Matiyali, N., Guha, T., & Sharma, G. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093438 · Cited by 49 · see citing works
Self-supervised audiovisual representation learning for remote sensing data
Heidler, K., Mou, L., Hu, D., Jin, P., Li, G., Gan, C., Wen, J.-R., & Zhu, X. X. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.103130 · Cited by 52 · see citing works
Audio-Visual Event Localization by Learning Spatial and Semantic Co-Attention
Xue, C. Q., Zhong, X., Cai, M., Chen, H., & Wang, W. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3127029 · Cited by 49 · see citing works
Multi-Target DoA Estimation with an Audio-Visual Fusion Mechanism
Qian, X., Madhavi, M. C., Pan, Z., Wang, J., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413776 · Cited by 49 · see citing works
Audio-Visual Grouping Network for Sound Localization from Mixtures
Mo, S., & Tian, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01018 · Cited by 48 · see citing works
Chen, C., Song, M., Song, W., Guo, L., & Jian, M. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3203421 · Cited by 47 · see citing works
Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds
Vasudevan, A. B., Dai, D., & Gool, L. V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_37 · Cited by 47 · see citing works
Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen SoundsPreprint
Tzinis, E., Wisdom, S., Jansen, A., Hershey, S., Remez, T., Ellis, D. P. W., & Hershey, J. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01143 · Cited by 47 · see citing works
Class-Aware Sounding Objects Localization via Audiovisual Correspondence
Hu, D., Wei, Y., Qian, R., Lin, W., Song, R., & Wen, J.-R. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3137988 · Cited by 46 · see citing works
Shon, S., Oh, T.-H., & Glass, J. R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683477 · Cited by 46 · see citing works
Self-citationEgocentric Deep Multi-Channel Audio-Visual Active Speaker Localization
Jiang, H., Murdock, C., & Ithapu, V. K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01029 · Cited by 46 · see citing works
Semantic and Relation Modulation for Audio-Visual Event Localization
Wang, H., Zha, Z.-J., Li, L., Chen, X., & Luo, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3226328 · Cited by 47 · see citing works
Gao, J., Chen, M., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01805 · Cited by 44 · see citing works
Vision + X: A Survey on Multimodal Learning in the Light of Data
Zhu, Y., Wu, Y., Sebe, N., & Yan, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3420239 · Cited by 46 · see citing works
Progressive Spatio-temporal Perception for Audio-Visual Question Answering
Li, G., Hou, W., & Hu, D. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612293 · Cited by 43 · see citing works
Planamente, M., Plizzari, C., Alberti, E., & Barbara Caputo, B. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00024 · Cited by 43 · see citing works
Chung, S.-W., Kang, H.-G., & Chung, J. S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1113 · Cited by 38 · see citing works
Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment
Kim, S.-B., Senocak, A., Ha, H., Owens, A., & Oh, T.-H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00622 · Cited by 41 · see citing works
Self-citationBAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge
Liu, C., Li, P. P., Zhang, H., Li, L., Huang, Z. H., Wang, D., & Yu, X. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3405622 · Cited by 41 · see citing works
Annotation-free Audio-Visual Segmentation
Liu, J., Wang, Y., Ju, C., Ma, C., Zhang, Y., & Xie, W. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00551 · Cited by 42 · see citing works
Read and Attend: Temporal Localisation in Sign Language Videos
Varol, G., Momeni, L., Albanie, S., Afouras, T., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01658 · Cited by 39 · see citing works
LipLearner: Customizable Silent Speech Interactions on Mobile Devices
Su, Z., Fang, S., & Rekimoto, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581465 · Cited by 40 · see citing works
EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving
Lin, J., Chen, J., Peng, K., He, X., Li, Z., Stiefelhagen, R., & Yang, K. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3437645 · Cited by 39 · see citing works
The Right to Talk: An Audio-Visual Transformer Approach
Truong, T.-D., Duong, C. N., De Vu, T., Pham, H. A., Raj, B., Le, N., & Luu, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00114 · Cited by 38 · see citing works
Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
Liu, C., Li, P. P., Qi, X., Zhang, H., Li, L., Wang, D., & Yu, X. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612373 · Cited by 38 · see citing works
Audio-Visual Class-Incremental Learning
Pian, W., Mo, S., Guo, Y., & Tian, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00717 · Cited by 39 · see citing works
- Authors
- Senocak, A., Oh, T.-H., Kim, J., Yang, M.-H., & Kweon, I. S.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- University of California, Merced (US)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 4358-4366
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2792330714
- Cited by
- 328
- References
- 37
- Citations by year
- 2026: 17 · 2025: 16 · 2024: 53 · 2023: 48 · 2022: 47 · 2021: 62 · 2020: 45 · 2019: 31 · 2018: 9
- FWCI
- 27.73
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Open Access System for Information Sharing (Pohang University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- sound localization, visual scenes, audio-visual correspondence, unsupervised learning, two-stream network, attention mechanism, semi-supervised learning, audiovisual integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
