Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention
Open accessAbstract
Binaural audio gives the listener an immersive experience and can enhance augmented and virtual reality. However, recording binaural audio requires specialized setup with a dummy human head having microphones in left and right ears. Such a recording setup is difficult to build and setup, therefore mono audio has become the preferred choice in common devices. To obtain the same impact as binaural audio, recent efforts have been directed towards lifting mono audio to binaural audio conditioned on the visual input from the scene. Such approaches have not used an important cue for the task: the distance of different sound producing objects from the microphones. In this work, we argue that depth map of the scene can act as a proxy for inducing distance information of different objects in the scene, for the task of audio binauralization. We propose a novel encoder-decoder architecture with a hierarchical attention mechanism to encode image, depth and audio feature jointly. We design the network on top of state-of-the-art transformer networks for image and depth representation. We show empirically that the proposed method outperforms state-of-the-art methods comfortably for two challenging public datasets FAIR-Play and MUSIC-Sereo. We also demonstrate with qualitative results that the method is able to focus on the right information required for the task. The qualitative results are available at our project page https://krantiparida.github.io/projects/bmonobinaural.html
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ViL-BERT fails to exploit linguistic structure.
“Akula et al. [292] empirically demonstrate that ViL-BERT fails to exploit linguistic structure”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [292]
Some multimodal datasets have been proposed for evaluating robustness.
“Thus, some multimodal datasets [130], [292] are proposed for evaluating the robustness.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [292]
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · cited 2× · see citing works
“Following [26], we obtain features from four different layers of the transformer network, i.”
MethodTowards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., & Koltun, V. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000462024 · Cited by 1,093 · see citing works
Arandjelović, R., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 836 · see citing works
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704 · see citing works
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · see citing works
(2021) · International Conference on Learning Representations · Cited by 449 · see citing works
Arandjelović, R., & Zisserman, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_27 · Cited by 448 · see citing works
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J. H., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_35 · Cited by 435 · see citing works
Cooperative Learning of Audio and Video Models from Self-Supervised SynchronizationPreprint
Korbar, B., Tran, D., & Torresani, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00230 · Cited by 331 · see citing works
Learning to Localize Sound Source in Visual Scenes
Senocak, A., Oh, T.-H., Kim, J., Yang, M.-H., & Kweon, I. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00458 · Cited by 328 · see citing works
Creating Interactive Virtual Acoustic Environments
Savioja, L., Huopaniemi, J., Lokki, T., et al. (1999) · Journal of the Audio Engineering Society · Cited by 273 · see citing works
Learning to Separate Object Sounds by Watching Unlabeled Video
Gao, R., Feris, R. S., & Grauman, K. L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_3 · Cited by 257 · see citing works
Self-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_13 · Cited by 221 · see citing works
Audio-Visual Instance Discrimination with Cross-Modal Agreement
Morgado, P. M., Vasconcelos, N. M., & Misra, I. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01229 · Cited by 218 · see citing works
Music Gesture for Visual Sound Separation
Gan, C., Huang, D. Y., Zhao, H., Tenenbaum, J. B., & Torralba, A. B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01049 · Cited by 197 · see citing works
Rendering Localized Spatial Audio in a Virtual Auditory Space
Zotkin, D. N., Ramani Duraiswami, B., & Davis, L. S. (2004) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2004.827516 · Cited by 173 · see citing works
Self-Supervised Generation of Spatial Audio for 360 VideoPreprint
Morgado, P. M., Vasconcelos, N. M., Langlois, T., & Wang, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.02587 · Cited by 88 · see citing works
Discriminative Sounding Objects Localization via Self-supervised Audiovisual MatchingPreprint
Hu, D., Qian, R., Jiang, M., Tan, X., Wen, S., Ding, E., Lin, W., & Dou, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05466 · Cited by 88 · see citing works
STAViS: Spatio-Temporal AudioVisual Saliency Network
Tsiami, A., Koutras, P., & Maragos, P. A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00482 · Cited by 86 · see citing works
Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
Zhou, H., Xu, X., Lin, D., Wang, X., & Liu, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_4 · Cited by 80 · see citing works
VisualEchoes: Spatial Image Representation Learning Through Echolocation
Gao, R., Chen, C., Al-Halah, Z., Schissler, C., & Grauman, K. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_38 · Cited by 79 · see citing works
Telling Left From Right: Learning Spatial Correspondence of Sight and Sound
Yang, K., Russell, B. C., & Salamon, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00995 · Cited by 76 · see citing works
Learning Representations from Audio-Visual Spatial AlignmentPreprint
Morgado, P. M., Li, Y., & Vasconcelos, N. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01819 · Cited by 74 · see citing works
Wu, E., & Koike, H. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00152 · Cited by 68 · see citing works
On Our Perception of the Direction of a Source of Sound
Rayleigh, L. (1875) · Proceedings of the Musical Association · doi:10.1093/jrma/2.1.75 · Cited by 61 · see citing works
Natural Sound Rendering for Headphones: Integration of signal processing techniques
Sunder, K., He, J., Tan, E.-L., et al. (2015) · IEEE Signal Processing Magazine · doi:10.1109/msp.2014.2372062 · Cited by 59 · see citing works
BatVision: Learning to See 3D Spatial Layout with Two Ears
Christensen, J. H., Hornauer, S., & Yu, S. X. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196934 · Cited by 56 · see citing works
Visually Informed Binaural Audio Generation without Binaural Audios
Xu, X., Zhou, H., Liu, Z., Dai, B., Wang, X., & Lin, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01523 · Cited by 55 · see citing works
Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
Hu, D., Qian, R., Jiang, M., et al. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4079386 · Cited by 50 · see citing works
Parida, K. K., Matiyali, N., Guha, T., & Sharma, G. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093438 · Cited by 49 · see citing works
Self-citationSemantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds
Vasudevan, A. B., Dai, D., & Gool, L. V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_37 · Cited by 47 · see citing works
Our Perception of the Direction of a Source of Sound
Strutt, J. W. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511703966.041 · Cited by 46 · see citing works
Beyond Image to Depth: Improving Depth Prediction using Echoes
Parida, K. K., Srivastava, S., & Sharma, G. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00817 · Cited by 41 · see citing works
Self-citationMazumder, P., Singh, P. P., Parida, K. K., et al. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00313 · Cited by 39 · see citing works
Self-citationNeural Synthesis of Binaural Speech From Mono Audio
Richard, A., Marković, D., Gebru, I. D., Krenn, S., Butler, G. A., la Torre, F. D., & Sheikh, Y. A. (2020) · International Conference on Learning Representations · Cited by 37 · see citing works
Implicit HRTF Modeling Using Temporal Convolutional Networks
Gebru, I. D., Markovic, D., Richard, A., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414750 · Cited by 28 · see citing works
Localize to Binauralize: Audio Spatialization from Visual Sound Source Localization
Rachavarapu, K. K., Aakanksha, A., Sundaresha, V., & Rajagopalan, A. N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00194 · Cited by 27 · see citing works
Self-Supervised Audio Spatialization with Correspondence Classifier
Lu, Y.-D., Lee, H.-Y., Tseng, H.-Y., & Yang, M. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803494 · Cited by 26 · see citing works
Relighting Images in the Wild with a Self-Supervised Siamese Auto-Encoder
Liu, Y., Neophytou, A., Sengupta, S., et al. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00008 · Cited by 15 · see citing works
Discriminative semantic transitive consistency for cross-modal learning
Parida, K. K., & Sharma, G. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103404 · Cited by 1 · see citing works
Self-citationDepth Infused Binaural Audio Generation using Hierarchical Cross-Modal AttentionPreprint
Parida, K. K., Srivastava, S., Matiyali, N., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04906 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
- Authors
- Parida, K. K., Srivastava, S., & Sharma, G.
- Affiliations
- Indian Institute of Technology Kanpur (IN)
- CDAC,Noida
- Venue
- 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2022-01
- Pages
- 2151-2160
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W3213191779
- Cited by
- 29
- References
- 44
- Citations by year
- 2026: 6 · 2025: 5 · 2024: 9 · 2023: 5 · 2022: 4
- FWCI
- 4.09
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Music Technology and Sound Studies
- Keywords
- binaural audio, depth images, cross-modal attention, hierarchical attention, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
