Audiovisual Masked Autoencoders
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Can we leverage the audiovisual information already present in video to improve self-supervised representation learning? To answer this question, we study various pre-training architectures and objectives within the masked autoencoding framework, motivated by the success of similar methods in natural language and image understanding. We show that we can achieve significant improvements on audiovisual downstream classification tasks, surpassing the state-of-the-art on VGGSound and AudioSet. Furthermore, we can leverage our audiovisual pretraining scheme for multiple unimodal downstream tasks using a single audiovisual pretrained model. We additionally demonstrate the transferability of our representations, achieving state-of-the-art audiovisual results on Epic Kitchens without pre-training specifically for this dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 1×
“Subsequent works converged on contrastive [17, 36, 38, 53], self-distillation [15,33,47] or clustering-based [8,14] objectives that encouraged a neural network to learn feature representations that were invariant to a predefined set of data transformations.”
BackgroundSpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767 · cited 8×
“In addition, we also achieve 65.0% Top-1 accuracy on audiovisual finetuning, improving upon [52] For video-only finetuning, our architecture corresponds to an unfactorised ViViT [6].”
MethodResultInfluentialSelf-citation
Research works printed but not linked: 43
- Akbari, VATT: Transformers for multimodal self-supervised learning from raw video, audio and text
- Alayrac, Self-supervised multimodal versatile networks
- 10.1109/ICCV.2017.73 · DOI · Paper page
- 10.1007/978-3-030-01246-5_27 · DOI · Paper page
- Asano, Labelling unlabelled videos from scratch with multi-modal self-supervision
- Asano, Self-labelling via simultaneous clustering and representation learning
- Bao, BEiT: BERT pre-training of image transformers
- 10.1109/ICASSP40776.2020.9053174 · DOI · Paper page
- Chen, A simple framework for contrastive learning of visual representations
- 10.1109/ICASSP49357.2023.10095691 · DOI
- 10.1109/CVPR52688.2022.02070 · DOI
- Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale
- 10.24963/ijcai.2020/78 · DOI · Paper page
- 10.1109/CVPR52729.2023.01003 · DOI
- 10.1109/CVPR52688.2022.01563 · DOI · Paper page
- 10.1109/ICPR48806.2021.9413035 · DOI · Paper page
- 10.2478/eletel-2014-0042 · DOI · Paper page
- 10.1109/CVPR42600.2020.00975 · DOI · Paper page
- Jaegle, Perceiver IO: A general architecture for structured inputs & outputs
- 10.1109/CVPR.2014.223 · DOI · Paper page
- 10.1109/ICASSP39728.2021.9413376 · DOI
- Korbar, Cooperative learning of audio and video models from self-supervised synchronization
- Lee, Compressive visual representations
- 10.1109/ICCV.2019.00718 · DOI · Paper page
- 10.1007/978-3-030-01216-8_12 · DOI · Paper page
- 10.1109/CVPR42600.2020.00990 · DOI · Paper page
- Nagrani, Attention bottlenecks for multimodal fusion
- 10.1109/CVPR.2016.278 · DOI · Paper page
- 10.1109/ICCV48922.2021.00944 · DOI
- Patrick, Keeping your eye on the ball: Trajectory attention in video transformers
- Radford, Learning transferable visual models from natural language super-vision
- Ramazanova, OWL (observe, watch, listen): Localizing actions in egocentric video via audiovisual temporal context, 10.1109/CVPRW59228.2023.00516 · DOI
- 10.1016/j.plrev.2010.04.006 · DOI · Paper page
- Simonyan, Two-stream convolutional networks for action recognition in videos
- 10.1109/CVPR52688.2022.01519 · DOI · Paper page
- 10.1109/ICASSP49357.2023.10096532 · DOI
- 10.1109/CVPR.2016.308 · DOI · Paper page
- 10.1007/978-3-030-01216-8_16 · DOI · Paper page
- Vaswani, Attention is all you need
- 10.1145/1390156.1390294 · DOI · Paper page
- 10.1109/CVPR42600.2020.01271 · DOI · Paper page
- Zellers, MERLOT: Multimodal neural script knowledge models
- 10.1007/978-3-319-46487-9_40 · DOI · Paper page
Cited by
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
Global birdsong embeddings enable superior transfer learning for bioacoustic classification
Ghani, B., Denton, T., Kahl, S., & Klinck, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-49989-z · Cited by 167
Efficient neural decoding based on multimodal training
Wang, Y. (2024) · Brain Sciences · doi:10.3390/brainsci14100988 · Cited by 3
“Here, we followed Georgescu et al. [33] to use a mid-fusion multimodal MAE with a shared decoder trans-former, and the standard transformer architecture was used [34,35].”
Method
Metadata
- Authors
- Georgescu, M.-I., Fonseca, E., Ionescu, R. T., Lucic, M., Schmid, C., & Arnab, A.
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 16098-16108
- OpenAlex
- W4390874283
- Cited by
- 44
- References
- 33
- FWCI
- 9.73
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Audiovisual Masked Autoencoders. https://app.cochat.ai/papers/10.1109/iccv51070.2023.01479
- The paper
- Georgescu, M.-I., Fonseca, E., Ionescu, R. T., Lucic, M., Schmid, C., & Arnab, A. (2023). Audiovisual Masked Autoencoders. 2023 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/iccv51070.2023.01479
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.