CoChat
Sign Up
Cited by 456Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

Open access
Assran, Mahmoud, Duval, Quentin, Misra, Ishan, Bojanowski, Piotr, Vincent, Pascal, Rabbat, Michael, LeCun, Yann, Ballas, Nicolas·2023·2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52729.2023.01499
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample target blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transformers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01499

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous video foundation model works tend to focus on understanding evaluation using probe-based evaluation or visual question answering tasks after aligning with a large-language model.

    “Previous works, however, tend to focus on understanding evaluation using probe-based evaluation or visual question answering tasks after aligning with a large-language model.”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Assran et al., 2023

    Section 6, “Local Prediction Tasks” (page unavailable), reports evaluating I-JEPA image representations with a linear probe, supporting the probe-based evaluation portion. The cited paper concerns images; the passages do not establish a trend among video foundation models or discuss visual question answering after alignment with a large language model. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • JEPA-Anything predicts target representations from observed context, building on joint-embedding predictive architectures.

    “Building on joint-embedding predictive architectures [10-12], it predicts target representations from observed context.”

    From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [10]

  • Joint-embedding predictive architectures learn latent targets without reconstructing every observation detail.

    “Joint-embedding predictive architectures learn latent targets without reconstructing every observation detail [10-12]”

    From RSI-Anything: Unlocking Recursive Self-Improvement with World Models (2026) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 60

  1. Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and rep- resentation learning. Internatinoal Conference on Learning Representations, 2020. 1
  2. Mahmoud Assran, Nicolas Ballas, Lluis Castrejon, and Michael Rabbat. Supervision accelerates pre-training in con- trastive semi-supervised learning of visual representations. NeurIPS Workshop on Self-Supervised Learning, 2020. 13
  3. Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bo- janowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese net- works for label-efficient learning. European Conference on Computer Vision, 2022. 1, 2, 3, 5, 6, 12, 13, 16, 17
  4. Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bo- janowski, Armand Joulin, Nicolas Ballas, and Michael Rab- bat. Semi-supervised learning of visual features by non- parametrically predicting view assignments with support samples. IEEE/CVF International Conference on Computer Vision, 2021. 3, 13
  5. Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019. 13
  6. Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. arXiv preprint arXiv:2212.07525, 2022. 5 · arXiv
  7. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Ji- atao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022. 1, 3, 4, 5, 6, 13 · arXiv
  8. Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 1, 3, 4, 13 · arXiv
  9. Adrien Bardes, Jean Ponce, and Yann LeCun. Vi- creg: Variance-invariance-covariance regularization for self- supervised learning. arXiv preprint arXiv:2105.04906, 2021. 1, 3, 13 · arXiv
  10. Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self- supervised learning of local visual features. arXiv preprint arXiv:2210.01571, 2022. 1, 13 · arXiv
  11. Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regulariza- tion: Improving deep networks generalization by removing their head. arXiv preprint arXiv:2206.13378, 2022. 13 · arXiv
  12. Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised rep- resentation knows about. Transactions on Machine Learning Research, 2022. 7, 16
  13. John Bridle, Anthony Heading, and David MacKay. Un- supervised classifiers, mutual information and'phantom tar- gets. Advances in neural information processing systems, 4, 1991. 13
  14. Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669-688, 1993. 1, 3
  15. Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. arXiv preprint arXiv:2208.05688, 2022. 13 · arXiv
  16. Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020. 1, 6 · arXiv
  17. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In International Conference on Ma- chine Learning, pages 1691-1703. PMLR, 2020. 13
  18. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. preprint arXiv:2002.05709, 2020. 1, 2, 13 · arXiv
  19. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised mod- els are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020. 5 · arXiv
  20. Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026, 2022. 5 · arXiv
  21. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020. 12, 13 · arXiv
  22. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566, 2020. 1, 3, 13 · arXiv
  23. Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021. 4 · arXiv
  24. Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.08954, 2022. 13 · arXiv
  25. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3, 4, 12, 13 · arXiv
  26. Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740, 2021. 13 · arXiv
  27. Karl Friston. A theory of cortical responses. Philosophi- cal transactions of the Royal Society B: Biological sciences, 360(1456):815-836, 2005. 1
  28. Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6928-6938, 2020. 13
  29. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016. 13
  30. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Do- ersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Moham- mad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020. 1, 3, 5, 12, 13 · arXiv
  31. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual repre- sentation learning. arXiv preprint arXiv:1911.05722, 2019. 1, 3, 12, 13 · arXiv
  32. Olivier Henaff. Data-efficient image recognition with con- trastive predictive coding. In International conference on machine learning, pages 4182-4192. PMLR, 2020. 13
  33. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual in- formation estimation and maximization. arXiv preprint arXiv:1808.06670, 2018. 13 · arXiv
  34. Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In In- ternational conference on machine learning, pages 1558- 1567. PMLR, 2017. 13
  35. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017. 12
  36. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning. In CVPR, 2017. 6
  37. Andreas Krause, Pietro Perona, and Ryan Gomes. Dis- criminative clustering by regularized information maximiza- tion. Advances in neural information processing systems, 23, 2010. 13
  38. Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006. 2
  39. Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):105-117, 1988. 13
  40. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 12 · arXiv
  41. Yi Ma, Doris Tsao, and Heung-Yeung Shum. On the prin- ciples of parsimony and self-consistency for the emergence of intelligence. Frontiers of Information Technology & Elec- tronic Engineering, pages 1-26, 2022. 13
  42. Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020. 13
  43. Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. International Conference on Learning Representations, 2021. 13
  44. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems, 32, 2019. 12
  45. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2536-2544, 2016. 1, 4
  46. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Confer- ence on Machine Learning, pages 8821-8831. PMLR, 2021.
  47. Antti Tarvainen and Harri Valpola. Mean teachers are bet- ter role models: Weight-averaged consistency targets im- prove semi-supervised deep learning results. arXiv preprint arXiv:1703.01780, 2017. 12 · arXiv
  48. Yuandong Tian, Xinlei Chen, and Surya Ganguli. Un- derstanding self-supervised learning dynamics without con- trastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021. 13
  49. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through at- tention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021. 3
  50. Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625, 2019. 13 · arXiv
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017. 3
  52. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful represen- tations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010. 1, 4, 13
  53. Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature predic- tion for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133, 2021. 1, 13 · arXiv
  54. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint arXiv:1904.12848, 2019. 13 · arXiv
  55. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A sim- ple framework for masked image modeling. arXiv preprint arXiv:2111.09886, 2021. 1, 4 · arXiv
  56. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regular- ization strategy to train strong classifiers with localizable fea- tures. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 6023-6032, 2019. 16
  57. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021. 1, 3, 13 · arXiv
  58. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. Internatinoal Conference on Learning Representations, 2018. 16
  59. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Tor- ralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural in- formation processing systems, 27, 2014. 12
  60. Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Ibot: Image bert pre- training with online tokenizer. International Conference on Learning Representations, 2022. 2, 4, 5, 6, 12, 13

Cited by

Sorted by citations

Authors
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., & Ballas, N.
Affiliations
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • Meta AI (FAIR)
  • New York University (US)
ORCID
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
15619-15629
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52729.2023.01499
OpenAlex
W4386076428
Cited by
456
References
11
Citations by year
2026: 198 · 2025: 148 · 2024: 92 · 2023: 18
FWCI
84.34
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
self-supervised learning, image representation, JEPA, vision transformer, masking strategy, semantic representation, object counting, depth estimation, linear classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review