Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Both ViT and its variants have been widely applied to various computer vision tasks and work well for both supervised and self-supervised visual learning.
“Both ViT and its variants have been widely applied to various computer vision tasks, including low-level tasks [92], recognition [93], detection [94], segmentation [95], etc, and also work well for both supervised [93] and self-supervised [96], [97], [98] visual learning.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [92]
Korbar et al. [148] broadened the scope of the audio-visual synchronization study to encompass arbitrary human activities rather than lip movements.
“With the same training strategy, Korbar et al . [148] broadened the scope of the study to encompass arbitrary human activities rather than lip movements.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [148]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · see citing works
Self-citationAdvances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,089 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300 · see citing works
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · see citing works
Some Experiments on the Recognition of Speech, with One and with Two Ears
Cherry, E. C. (1953) · The Journal of the Acoustical Society of America · doi:10.1121/1.1907229 · Cited by 4,554 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 3,711 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
Auditory Scene Analysis: The Perceptual Organization of Sound
Bregman, A. S., & McAdams, S. E. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.408434 · Cited by 3,067 · see citing works
Performance measurement in blind audio source separation
Vincent, E., Gribonval, R., & Févotte, C. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tsa.2005.858005 · Cited by 2,985 · see citing works
(2011) · Cited by 2,752 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Self-citationVoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256 · see citing works
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · see citing works
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,588 · see citing works
Deep clustering: Discriminative embeddings for segmentation and separation
Hershey, J. R., Chen, Z., Roux, J. L., & Watanabe, S. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7471631 · Cited by 1,389 · see citing works
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,266 · see citing works
Factorial Hidden Markov Models
Ghahramani, Z., & Jordan, M. I. (1997) · Machine Learning · doi:10.1023/a:1007425814087 · Cited by 1,157 · see citing works
An audio-visual corpus for speech perception and automatic speech recognition
Cooke, M. P., Barker, J., Cunningham, S. P., & Shao, X. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2229005 · Cited by 1,146 · see citing works
Virtanen, T. I. (2007) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.885253 · Cited by 1,025 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Permutation invariant training of deep models for speaker-independent multi-talker speech separation
Yu, D., Kolbæk, M., Tan, Z., & Jensen, J. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952154 · Cited by 857 · see citing works
Arandjelović, R., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 836 · see citing works
Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · see citing works
Lip reading sentences in the wild
Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.367 · Cited by 702 · see citing works
Sound alters visual motion perception
Sekuler, R., Sekuler, A. B., & Lau, R. (1997) · Nature · doi:10.1038/385308a0 · Cited by 695 · see citing works
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · see citing works
Auditory Scene Analysis: The Perceptual Organization of Sound
Kramer, G., & Bregman, A. S. (1992) · Leonardo Music Journal · doi:10.2307/1513223 · Cited by 681 · see citing works
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · see citing works
Sensory modalities are not separate modalities: plasticity and interactions
Shimojo, S. (2001) · Current Opinion in Neurobiology · doi:10.1016/s0959-4388(00)00241-5 · Cited by 603 · see citing works
The Development of Embodied Cognition: Six Lessons from Babies
Smith, L., & Gasser, M. (2005) · Artificial Life · doi:10.1162/1064546053278973 · Cited by 597 · see citing works
Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation
Huang, P.-S., Kim, M., Hasegawa‐Johnson, M., & Smaragdis, P. (2015) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2015.2468583 · Cited by 460 · see citing works
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J. H., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_35 · Cited by 435 · see citing works
Self-Supervised Video Representation Learning with Odd-One-Out Networks
Fernando, B., Bilen, H., Gavves, E., & Gould, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.607 · Cited by 421 · see citing works
Deep attractor network for single-microphone speaker separation
Chen, Z., Luo, Y., & Mesgarani, N. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952155 · Cited by 399 · see citing works
Ambient Sound Provides Supervision for Visual Learning
Owens, A., Wu, J., McDermott, J. H., Freeman, W. T., & Torralba, A. B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_48 · Cited by 385 · see citing works
Self-citationOwens, A., Isola, P., McDermott, J. H., Torralba, A. B., Adelson, E. H., & Freeman, W. T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.264 · Cited by 352 · see citing works
Self-citationLearning to Localize Sound Source in Visual Scenes
Senocak, A., Oh, T.-H., Kim, J., Yang, M.-H., & Kweon, I. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00458 · Cited by 328 · see citing works
C3D: Generic Features for Video Analysis.
Tran, D., Bourdev, L. D., Fergus, R., Torresani, L., & Paluri, M. (2014) · Cited by 321 · see citing works
Learning and Using the Arrow of Time
Wei, D., Lim, J. J., Zisserman, A., & Freeman, W. T. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00840 · Cited by 318 · see citing works
Learning to Separate Object Sounds by Watching Unlabeled Video
Gao, R., Feris, R. S., & Grauman, K. L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_3 · Cited by 257 · see citing works
Factorial Hidden Markov Models.
Zoubin Ghahramani, A., & Jordan, M. I. (1996) · doi:10.21236/ada307097 · Cited by 256 · see citing works
Lin, M., Chen, Q., & Yan, S. (2013) · arXiv · doi:10.48550/arxiv.1312.4400 · Cited by 252 · see citing works
Michelsanti, D., & Tan, Z. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1620 · Cited by 239 · see citing works
Audio Vision: Using Audio-Visual Synchrony to Locate Sounds
Hershey, J. R., & Movellan, J. R. (1999) · Cited by 227 · see citing works
Monaural speech separation and recognition challenge
Cooke, M. P., Hershey, J. R., & Rennie, S. J. (2009) · Computer Speech & Language · doi:10.1016/j.csl.2009.02.006 · Cited by 197 · see citing works
Unsupervised Learning of Visual Representations using VideosPreprint
Wang, X., & Gupta, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00687 · Cited by 194 · see citing works
Kidron, E., Schechner, Y. Y., & Elad, M. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.274 · Cited by 183 · see citing works
Learning Joint Statistical Models for Audio-Visual Fusion and Segregation
Fisher, J. S., Darrell, T., Freeman, W. T., & Viola, P. A. (2000) · Cited by 176 · see citing works
Learning Spatiotemporal Features with 3D Convolutional NetworksPreprint
Tran, D., Bourdev, L., Fergus, R., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.0767 · Cited by 155 · see citing works
Gabbay, A., Shamir, A., & Peleg, S. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1955 · Cited by 145 · see citing works
Barzelay, Z., & Schechner, Y. Y. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383344 · Cited by 92 · see citing works
Learning visual groups from co-occurrences in space and time
Isola, P., Zoran, D., Krishnan, D., & Adelson, E. H. (2015) · arXiv · doi:10.48550/arxiv.1511.06811 · Cited by 90 · see citing works
Seeing Through Noise: Visually Driven Speaker Separation And Enhancement
Gabbay, A., Ephrat, A., Halperin, T., & Peleg, S. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462527 · Cited by 87 · see citing works
Blind Audiovisual Source Separation Based on Sparse Redundant Representations
Casanovas, A. L., Monaci, G., Vandergheynst, P., & Gribonval, R. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2050650 · Cited by 79 · see citing works
Audiovisual Speech Source Separation: An overview of key methodologies
Rivet, B., Wang, W., Naqvi, S. M., & Chambers, J. A. (2014) · IEEE Signal Processing Magazine · doi:10.1109/msp.2013.2296173 · Cited by 76 · see citing works
Binding and unbinding the auditory and visual streams in the McGurk effect
Nahorna, O., Berthommier, F., & Schwartz, J. (2012) · The Journal of the Acoustical Society of America · doi:10.1121/1.4728187 · Cited by 75 · see citing works
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
Owens, A., Wu, J., McDermott, J. H., Freeman, W. T., & Torralba, A. B. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1083-5 · Cited by 56 · see citing works
Self-citationThe Conversation: Deep Audio-Visual Speech Enhancement
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1400 · Cited by 54 · see citing works
Audio-visual object localization and separation using low-rank and sparsity
Pu, J. X., Panagakis, Y., Petridis, S., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952687 · Cited by 44 · see citing works
Audio-visual Segmentation and “The Cocktail Party Effect”
Darrell, T. J., Fisher, J. W., & Viola, P. A. (2000) · Lecture notes in computer science · doi:10.1007/3-540-40063-x_5 · Cited by 40 · see citing works
Nahorna, O., Berthommier, F., & Schwartz, J. (2015) · The Journal of the Acoustical Society of America · doi:10.1121/1.4904536 · Cited by 35 · see citing works
Audio-visual speech enhancement using deep neural networks
Hou, J.-C., Wang, S., Lai, Y.-H., Lin, J., Tsao, Y., Chang, H.-W., & Wang, H. (2016) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2016.7820732 · Cited by 30 · see citing works
Schwartz, J., Berthommier, F., & Savariaux, C. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-437 · Cited by 29 · see citing works
Learning to Separate Object Sounds by Watching Unlabeled VideoPreprint
Gao, R., Feris, R. S., & Grauman, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01665 · Cited by 28 · see citing works
Detecting audio-visual synchrony using deep neural networks
Marcheret, E., Potamianos, G., Vopicka, J., & Goel, V. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-201 · Cited by 25 · see citing works
Lip synchronization for animation
McAllister, D. F., Rodman, R. D., Bitzer, D. L., & Freeman, A. S. (1997) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/259081.259312 · Cited by 22 · see citing works
Lip synchronization of speech.
McAllister, D. F., Rodman, R. D., Bitzer, D. L., & Freeman, A. S. (1997) · AVSP · Cited by 18 · see citing works
Speaker separation using visually-derived binary masks.
Khan, F. U., & Milner, B. P. (2013) · AVSP · Cited by 15 · see citing works
Pose from Action: Unsupervised Learning of Pose Features based on MotionPreprint
Purushwalkam, S., & Gupta, A. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.05420 · Cited by 14 · see citing works
Objects that SoundPreprint
Arandjelović, R., & Zisserman, A. (2017) · arXiv (Cornell University) · Cited by 14 · see citing works
Audio-visual graphical models for speech processing
Hershey, J., Attias, H. T., Jojić, N., et al. (2004) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2004.1327194 · Cited by 13 · see citing works
Fusion and combination in audio-visual integration
Omata, K., & Mogi, K. (2007) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2007.1910 · Cited by 10 · see citing works
Objects that SoundPreprint
Arandjelović, R., & Zisserman, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06651 · Cited by 2 · see citing works
The Sound of PixelsPreprint
Zhao, H., Gan, C., Rouditchenko, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03160 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [11]11. Barker, J.P., Berthommier, F., Schwartz, J.L.: Is primitive av coherence an aid to segment the scene? In: AVSP'98 International Conference on Auditory-Visual Speech Processing. (1998) 3
- [31]31. Roweis, S.T.: One microphone source separation. In: Advances in neural information processing systems. (2001) 793-799 3
- [45]45. Gabbay, A., Shamir, A., Peleg, S.: Visual speech enhancement using noise-invariant training. arXiv preprint arXiv:1711.08789 (2017) 4, 13, 14 · arXiv
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · see citing works
Revisiting Self-Supervised Visual Representation Learning
Kolesnikov, A., Zhai, X., & Beyer, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00202 · Cited by 675 · see citing works
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · see citing works
What Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588 · see citing works
Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349 · see citing works
Multi-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346 · see citing works
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
Alwassel, H., Mahajan, D., Korbar, B., Torresani, L., Ghanem, B., & Tran, D. (2019) · arXiv · doi:10.48550/arxiv.1911.12667 · Cited by 298 · see citing works
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works
MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition
Lin, L., Song, S., Yang, W., & Liu, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413548 · Cited by 223 · see citing works
SoundSpaces: Audio-Visual Navigation in 3D Environments
Chen, C., Jain, U., Schissler, C., Gari, S. V. A., Al-Halah, Z., Ithapu, V. K., Robinson, P., & Grauman, K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_2 · Cited by 202 · see citing works
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H. (2021) · arXiv · doi:10.48550/arxiv.2107.06592 · Cited by 156 · see citing works
Self-supervised learning for videos: A survey
Schiappa, M. C., Rawat, Y. S., & Shah, M. (2023) · ACM Computing Surveys · doi:10.1145/3577925 · Cited by 155 · see citing works
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101 · see citing works
Self-citationCosmo: contrastive fusion learning with small data for multimodal human activity recognition
Ouyang, X., Shuai, X., Zhou, J., Shi, I. W., Xie, Z., Xing, G., & Huang, J. (2022) · Proceedings of the 28th Annual International Conference on Mobile Computing And Networking · doi:10.1145/3495243.3560519 · Cited by 94 · see citing works
A survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90 · see citing works
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference
Subedar, M., Krishnan, R., Meyer, P. L., Tickoo, O., & Huang, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00640 · Cited by 85 · see citing works
ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics
Taleb, A., Kirchler, M., Monti, R., & Lippert, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02024 · Cited by 83 · see citing works
Deep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64 · see citing works
An Improved Inter-Intra Contrastive Learning Framework on Self-Supervised Video Representation
Tao, L., Wang, X., & Yamasaki, T. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3141051 · Cited by 50 · see citing works
Parida, K. K., Matiyali, N., Guha, T., & Sharma, G. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093438 · Cited by 49 · see citing works
The Right to Talk: An Audio-Visual Transformer Approach
Truong, T.-D., Duong, C. N., De Vu, T., Pham, H. A., Raj, B., Le, N., & Luu, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00114 · Cited by 38 · see citing works
Self-Supervised Contrastive Video-Speech Representation Learning for Ultrasound
Jiao, J., Cai, Y., Alsharid, M., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59716-0_51 · Cited by 38 · see citing works
Parida, K. K., Srivastava, S., & Sharma, G. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00221 · Cited by 29 · see citing works
LT-GAN: Self-Supervised GAN with Latent Transformation Detection
Patel, P., Kumari, N., Singh, M., & Krishnamurthy, B. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00323 · Cited by 18 · see citing works
Speech2Action: Cross-Modal Supervision for Action Recognition
Nagrani, A., Sun, C., Ross, D., Sukthankar, R., Schmid, C., & Zisserman, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01033 · Cited by 12 · see citing works
- Authors
- Owens, A., & Efros, A. A.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Owens, Andrew
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2018
- Pages
- 639-658
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2796992393
- Cited by
- 704
- References
- 87
- Citations by year
- 2026: 21 · 2025: 42 · 2024: 86 · 2023: 89 · 2022: 128 · 2021: 146 · 2020: 110 · 2019: 68 · 2018: 13 · 2017: 1
- FWCI
- 63.63
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency (FA8750-16-C-0166)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- multisensory fusion, self-supervised learning, sound localization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
