The DOI matches its Crossref record; 109 of the 111 references it cites were found in a registry or an index, and 2 are not checked yet; 19 of the 38 sources its claims cite were checked for support.
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Open accessVerified 2026-10-27 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
MultiMAE extends masked autoencoding to optionally accept multiple input modalities and predict multiple output tasks, using masking across patches and modalities to learn cross-modal predictive coding. Trained entirely with pseudo labels on ImageNet-1K, it matches or improves transfer over MAE across classification, segmentation, and depth tasks, especially when depth or pseudo-labeled modalities are available.
Key findings
- RGB-only transfer: MultiMAE matches MAE on ImageNet-1K classification (83.3) and ADE20K segmentation (46.2), while improving Hypersim segmentation (37.0 vs. 36.5), NYUv2 segmentation (52.0 vs. 50.8), and NYUv2 depth δ1 (86.4 vs. 85.1) (Table 1).
- With ground-truth depth: MultiMAE RGB-D reaches 47.6 mIoU on Hypersim vs. MAE’s 36.9, and 56.0 on NYUv2 vs. MAE’s 49.3; depth-only MultiMAE also works well (38.5 Hypersim, 41.4 NYUv2) while MAE does not (32.5, 23.4) (Table 2).
- With pseudo labels: MultiMAE RGB-pD-pS improves ADE20K segmentation to 47.1 vs. 46.2 RGB-only, Hypersim to 40.1 vs. 37.0, and NYUv2 to 54.0 vs. 52.0; MAE benefits much less (Table 3).
- Ablation: RGB-D-S MultiMAE has best average rank 1.5 on Taskonomy, with NYUv2 segmentation 50.6, NYUv2 depth 85.4, and ImageNet-1K 83.0; specialized models can outperform it on individual tasks but perform poorly on others (Table 4).
Methods
- Architecture: ViT-B encoder with per-modality linear patch projections; only 98 visible tokens are encoded (1/6 of all tokens for three 224×224 modalities with 16×16 patches). Task-specific shallow decoders use cross-attention, an MLP, and two Transformer blocks at 256 dimensions.
- Pre-training: 1.28M ImageNet images with pseudo-labeled depth (DPT-Hybrid trained on Omnidata) and semantic segmentation (Mask2Former with Swin-S trained on COCO, 133 classes); RGB and per-patch standardized RGB outputs; losses are MSE for RGB, L1 for depth, cross-entropy for segmentation. Models are trained for 400 or 1600 epochs with AdamW, batch size 2048, and 8 A100 GPUs.
- Transfer: downstream tasks include ImageNet-1K classification, ADE20K/Hypersim/NYUv2 semantic segmentation, NYUv2 depth estimation, and Taskonomy dense regression; pre-trained decoders are replaced by task-specific heads, with metrics top-1 accuracy, mIoU, δ1, and L1 loss.
Limitations
- Stated: MultiMAE is pre-trained on only three visual modalities; scaling to more modalities and tasks (e.g., videos, text, bounding boxes, sparse depth, feature maps) is left as future work. It is trained on pseudo-labeled ImageNet-1K, though the method could use larger RGB datasets. Pixelwise L1/L2 losses produce blurry predictions in ambiguous regions, and probabilistic or generative modeling is suggested as future work. The masking strategy is simple unbiased random sampling; biasing masking toward certain modalities or locations may be better.
- Evident from text: multi-modal transfer without masking scales with the full number of modalities and tokens, making more than two modalities computationally expensive. Semantic segmentation pseudo labels come from a COCO-trained model, so the authors do not evaluate semantic segmentation transfer on COCO.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 12 Not valid · 5 Not checked · 76 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“This is supported by previous findings [58, 70, 99] showing that there is usually no single visual pre-training task that transfers well to any arbitrary other task, and instead, a set is required.”
What the sources say
- Not checked — no full text of the sourcecites [58] Smith, L., Gasser, M.: The development of embodied cognition: six lessons from babies. Artif. Life 11(1–2), 13–29 (2005) source
- Disputedcites [70] Wei, C., Fan, H., Xie, S., Wu, C.Y., Yuille, A., Feichtenhofer, C.: Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133 (2021) sourceExperimental — accuracy not yet measured
Section 4.2 (page unavailable) reports that a single MaskFeat pre-training approach transfers successfully to action detection and human-object interaction classification, though it does not test every possible downstream task. More directly, the paper's Multi-tasking discussion (page unavailable) finds that combining pixel and HOG objectives performs worse than either objective alone, contrary to the sentence's assertion that a set of pre-training tasks is required. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
- Not matchedcites [99]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“This is supported by previous findings [58, 70, 99] showing that there is usually no single visual pre-training task that transfers well to any arbitrary other task, and instead, a set is required.”
What the sources say
- Not checked — no full text of the sourcecites [58] Smith, L., Gasser, M.: The development of embodied cognition: six lessons from babies. Artif. Life 11(1–2), 13–29 (2005) source
- Disputedcites [70] Wei, C., Fan, H., Xie, S., Wu, C.Y., Yuille, A., Feichtenhofer, C.: Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133 (2021) sourceExperimental — accuracy not yet measured
Section 4.2 (Transfer Learning; page not available) reports that a single MaskFeat pre-training approach transfers well to both action detection and human-object interaction classification. The paper does not test transfer to every arbitrary task, but its multi-tasking analysis says combining two prediction objectives performed worse than either objective alone, contrary to the claim that a set of pre-training tasks is required. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Not matchedcites [99]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Instead, pre-training with a diverse set of tasks [11,83] has been observed to improve the performance on downstream tasks [31,80] and potentially learn a better representation.”
What the sources say
- Not checked — no full text of the sourcecites [11] Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., Torralba, A.: Learning aligned cross-modal representations from weakly aligned data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2940–2949 (2016) source
- Not matchedcites [83]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [31] Kaiser, L., et al.: One model to learn them all. arXiv preprint arXiv:1706.05137 (2017) sourceExperimental — accuracy not yet measured
The Abstract (p. 1) and Experiments (p. 7) report that joint training across diverse tasks improves performance on some tasks with less data. The paper studies concurrent multi-task training, however, not pre-training followed by evaluation on downstream tasks. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [80] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5122–5130 (2017) source
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“it has been shown that there is usually no single pretraining objective that transfers best to all possible downstream tasks [58,70,99].”
What the sources say
- Not checked — no full text of the sourcecites [58] Smith, L., Gasser, M.: The development of embodied cognition: six lessons from babies. Artif. Life 11(1–2), 13–29 (2005) source
- Partly supportedcites [70] Wei, C., Fan, H., Xie, S., Wu, C.Y., Yuille, A., Feichtenhofer, C.: Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133 (2021) sourceExperimental — accuracy not yet measured
The “Linear probing” section (page unavailable) reports that MaskFeat lags behind contrastive methods on linear probes while learning features useful under fine-tuning. This supports the narrower point that pretraining objectives can perform differently under different evaluations, but the paper does not establish that no objective transfers best to all possible downstream tasks. (LLM, conf=0.76)
- Not matchedcites [99]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“It also performs better than MAE on ImageNet-C, but falls behind DINO [12] and MoCo-v3 [17].”
What the sources say
- Could not verifycites [12] Chen, L.-C., et al.: Naive-student: leveraging semi-supervised learning in video sequences for urban scene segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 695–714. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_40 sourceExperimental — accuracy not yet measured
The Results (pp. 7–8) compare urban-scene segmentation methods on Cityscapes, while the ImageNet-1K results (p. 12) compare WR-38 and WR-41. Searches for MultiMAE, DINO, MoCo-v3, and ImageNet-C found no passage comparing their ImageNet-C performance. (LLM, conf=0.76)
- Not matchedcites [17] De Sa, V.R.: Sensory modality segregation. In: NIPS, pp. 913–920. Citeseer (2003)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
52 more claims · 3 with their cited sources checked
Create a free account to see all 97 claims and every source verdict.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 25
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4. Influence of pre-training task choices and masking on transfer performance · p. 7
“On Taskonomy, we report the ranking of different pre-trained models, averaged over all nine tasks.”
§ D. Detailed Taskonomy transfer results · p. 18
“Table 4 compared several baselines by their average rank on eight different Taskonomy [99] downstream tasks.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3. Transfers with multiple modalities · p. 7
“| MAE [35] | 83.3 | 46.2 | 36.5 | 50.8 | 85.1 |”
§ F. Comparison of MAE variants · p. 19
“| MAE (D8, PyTorch) | 83.3 | 46.2 | 36.5 | 50.1 | 85.1 |”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Research works printed but not linked: 8
- Atito, S., Awais, M., Kittler, J.: Sit: self-supervised vision transformer. arXiv preprint arXiv:2104.03602 (2021) · arXiv
- De Sa, V.R.: Sensory modality segregation. In: NIPS, pp. 913–920. Citeseer (2003)
- Lee, D.H., et al.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning. ICML (2013)
- Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
- Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.: Multimodal deep learning. In: ICML (2011)
- Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., J’egou, H.: Training data-efficient image transformers & distillation through attention. In: ICML (2021)
- Tripuraneni, N., Jordan, M., Jin, C.: On the theory of transfer learning: the importance of task diversity. Adv. Neural Inf. Process. Syst. 33, 7852–7862 (2020)
- Wu, M., Goodman, N.D.: Multimodal generative models for scalable weakly-supervised learning. In: NeurIPS (2018)
Cited by
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731
SatMAE: Pre-Training Transformers for Temporal and Multi-Spectral Satellite Imagery
Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., & Ermon, S. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-0015 · Cited by 303
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02240 · Cited by 247
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
Wang, X., Wang, W., Cao, Y., Shen, C., & Huang, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00660 · Cited by 197
Metadata
- Authors
- Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Corresponding author
- Bachmann, Roman
- Venue
- Lecture notes in computer science
- Publisher
- Springer Nature Switzerland
- Published
- 2022
- Pages
- 348-367
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4312685069
- Cited by
- 268
- References
- 63
- Citations by year
- 2026: 37 · 2025: 85 · 2024: 95 · 2023: 49 · 2022: 2
- FWCI
- 31.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- masked autoencoder, multimodal pre-training, multi-task learning, predictive coding, pseudo-labeling, image classification, semantic segmentation, depth estimation, transfer learning, cross-modal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: MultiMAE: Multi-modal Multi-task Masked Autoencoders. https://app.cochat.ai/papers/10.1007/978-3-031-19836-6_20
- The paper
- Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022). MultiMAE: Multi-modal Multi-task Masked Autoencoders. Lecture notes in computer science. https://doi.org/10.1007/978-3-031-19836-6_20
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.