The DOI matches its Crossref record; the references it cites are not checked yet.
Diffusion Model as Representation Learner
Open accessAbstract
Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks. Despite its promises, the learned representations of pre-trained DPMs, however, have not been fully understood. In this paper, we conduct an in-depth investigation of the representation power of DPMs, and propose a novel knowledge transfer method that leverages the knowledge acquired by generative DPMs for recognition tasks. Our study begins by examining the feature space of DPMs, revealing that DPMs are inherently denoising autoencoders that balance the representation learning with regularizing model capacity. To this end, we introduce a novel knowledge transfer paradigm named RepFusion. Our paradigm extracts representations at different time steps from off-the-shelf DPMs and dynamically employs them as supervision for student networks, in which the optimal time is determined through reinforcement learning. We evaluate our approach on several image classification, semantic segmentation, and landmark detection benchmarks, and demonstrate that it outperforms state-of-the-art methods. Our results uncover the potential of DPMs as a powerful tool for representation learning and provide insights into the usefulness of generative models beyond sample generation. The code is available at https://github.com/Adamdad/Repfusion.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work has shown that diffusion models are an effective way to do representation learning, particularly for image and video-based tasks.
“Recent work has shown that diffusion models are an effective way to do so, particularly for image and video-based tasks.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Yang & Wang (2023)
The Experiments and Experimental Setting sections report effective diffusion-model representation learning for image classification, semantic segmentation, and facial landmark detection. The paper mentions video synthesis, but our copy does not show representation-learning results for video-based tasks; that part of the claim is unsupported. Page numbers were not provided. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists haven’t been fetched yet
References 89 · Cited by 58
Research works printed but not linked: 75
- Abstreiter, 2021, Diffusion-based representation learning
- Alain, 2014, What regularized auto-encoders learn from the data-generating distribution, The Journal of Machine Learning Research, 15, 3563
- Alain, Regularized auto-encoders estimate local statistics
- Antoniou, 2017, Data augmentation generative adversarial networks
- Baranchuk, 2021, Label-efficient semantic segmentation with diffusion models
- Bengio, 2012, Implicit density estimation by local moment matching to sample from auto-encoders
- Bengio, 2013, Generalized denoising auto-encoders as generative models, 26
- 10.1007/BF00332918 · DOI
- Caron, 2020, Unsupervised learning of visual features by contrasting cluster assignments, Advances in neural information processing systems, 33, 9912
- Cayton, 2005, Algorithms for manifold learning, 12, 1
- 10.1007/978-3-319-12637-1_67 · DOI
- Chen, 2020, Improved baselines with momentum contrastive learning
- 10.1109/CVPR52729.2023.01544 · DOI
- 10.1109/CVPR52729.2023.01544 · DOI
- Geras, 2014, Scheduled denoising autoencoders
- 10.1109/CVPR42600.2020.00975 · DOI · Paper page
- He, Is synthetic data from generative models ready for image recognition?
- Hinton, 2015, Distilling the knowledge in a neural network
- Hjelm, Learning deep representations by mutual information estimation and maximization
- Ho, 2020, Denoising diffusion probabilistic models, Advances in Neural Information Processing Systems, 33, 6840
- Ho, 2022, Video diffusion models
- Hyvarinen, 2005, Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6
- 10.1016/0047-259X(75)90042-1 · DOI
- 10.1007/978-3-031-20071-7_7 · DOI
- 10.1109/CVPR46437.2021.01545 · DOI
- Kingma, 2010, Regularized estimation of image statistics by score matching, Advances in neural information processing systems, 23
- Krizhevsky, 2009, Learning multiple layers of features from tiny images
- Kwon, Diffusion models already have a semantic latent space
- Le, 2015, Tiny imagenet visual recognition challenge, CS 231N, 7, 3
- LeCun, 2010, Mnist handwritten digit database, 2
- 10.1109/CVPR42600.2020.00559 · DOI · Paper page
- 10.1109/ICCV51070.2023.01531 · DOI
- 10.1109/ICCV51070.2023.01471 · DOI
- Li, 2023, Is synthetic data from diffusion models ready for knowledge distillation?
- Liu, 2022, Dataset distillation via factorization, Advances in neural information processing systems
- 10.1109/CVPR52729.2023.00366 · DOI
- 10.1109/CVPR.2015.7298965 · DOI · Paper page
- Luo, 2023, Diffusion hyperfeatures: Searching through time and space for semantic correspondence
- Ma, 2023, LLM-Pruner: On the Structural Pruning of Large Language Models
- 10.1109/CVPR52729.2023.01374 · DOI
- Mittal, 2022, From points to functions: Infinite-dimensional representations in diffusion models
- Miyato, 2018, Spectral normalization for generative adversarial networks
- Mnih, Asynchronous methods for deep reinforcement learning, 1928
- 10.1109/CVPR.2019.00409 · DOI
- Poole, Dreamfusion: Text-to-3d using 2d diffusion
- 10.1109/CVPR52688.2022.01036 · DOI · Paper page
- 10.1109/ICCV51070.2023.00552 · DOI
- Romero, 2014, Fitnets: Hints for thin deep nets
- Roy, The effective rank: A measure of effective dimensionality, 606
- 10.1109/CVPR.2018.00474 · DOI · Paper page
- Shi, Towards understanding and mitigating dimensional collapse in heterogeneous federated learning
- Shi, 2023, Dragdiffusion: Harnessing diffusion models for interactive point-based image editing
- Song, 2019, Generative modeling by estimating gradients of the data distribution, Advances in neural information processing systems, 32
- Tang, 2023, Emergent correspondence from image diffusion
- Tian, 2019, Contrastive representation distillation
- 10.1145/1390156.1390294 · DOI · Paper page
- Vincent, 2010, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Journal of machine learning research, 11
- Wang, Learning to grow pretrained models for efficient transformer training
- Wang, Data efficient neural scaling law via model reusing, 36193
- 10.1007/978-1-4615-3618-5_2 · DOI
- 10.1109/CVPR.2018.00227 · DOI
- 10.1109/ICRA48891.2023.10161384 · DOI
- 10.1007/978-3-030-01231-1_29 · DOI
- 10.1109/CVPR52729.2023.00289 · DOI
- 10.1007/978-3-031-19830-4_5 · DOI
- 10.1109/CVPR52729.2023.02160 · DOI
- Yang, 2022, Deep model reassembly, Advances in neural information processing systems, 35, 25739
- 10.1109/CVPR42600.2020.00710 · DOI
- 10.1007/978-3-030-01261-8_20 · DOI · Paper page
- 10.1109/CVPR52729.2023.00350 · DOI
- 10.5244/C.30.87 · DOI · Paper page
- Zagoruyko, 2017, Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer
- 10.1007/s11704-016-6107-0 · DOI
- 10.1109/CVPR46437.2021.01001 · DOI · Paper page
- 10.1109/ICCV51070.2023.00527 · DOI
Metadata
- Authors
- Yang, X., & Wang, X.
- Affiliations
- National University of Singapore (SG)
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 18892-18903
- OpenAlex
- W4390872396
- Cited by
- 58
- References
- 89
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 27 · 2023: 2
- FWCI
- 6.21
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministry of Education
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- diffusion probabilistic models, representation learning, knowledge transfer, denoising autoencoder, image classification, semantic segmentation, landmark detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
