The DOI matches its Crossref record; 88 of the 91 references it cites were found in a registry or an index.
Adding Conditional Control to Text-to-Image Diffusion Models
Abstract
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Image annotations, including prompts and spatial cues [92, 121], are essential for training and evaluating image generation models as they provide localized and semantically rich supervision that help models learn meaningful visual representations.”
From Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation (2026) · cites this paper as [121]
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
A Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881
Research works printed but not linked: 13
- 2022, Disco diffusion
- 2022, Protogen x3.4 (photorealism) official release
- Ha, Hypernetworks
- 2022, Hypernetwork style training, a tiny guide, stable-diffusion-webui
- 10.1109/TPAMI.2020.2970919 · DOI · Paper page
- 2022, Novelai improvements on stable diffusion
- Lehtinen, Noise2noise: Learning image restoration without clean data
- 2023, Midjourney
- 2022, Comic-diffusion v2, trained on 6 styles at once
- 2023, Dall-e-2
- Schuhmann, LAION-5b: An open large-scale dataset for training next generation image-text models
- 2022, Stable diffusion v1.5 model card
- 2022, Stable diffusion v2 model card, stable-diffusion-2-depth
Cited by
Sorted by citations
Items 1,001–1,012 of 1,012
CLIPtone: Unsupervised Learning for Text-Based Image Tone Adjustment
Lee, H., Kang, K., Ok, J., & Cho, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00284 · Cited by 11
Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models
Cai, S., Ceylan, D., Gadelha, M., Huang, C.-H. P., Wang, T. Y., & Wetzstein, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00727 · Cited by 20
MaxFusion: Plug&Play Multi-modal Generation in Text-to-Image Diffusion Models
Nair, N. G., Valanarasu, J. M. J., & Patel, V. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72920-1_6 · Cited by 11
PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00426 · Cited by 30
“Following the general design of ControlNet (Zhang et al., 2023), we freeze each DiT Block and create a trainable copy, augmenting with two zero linear layers before and after it.”
MethodInfluentialCosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control
NVIDIA, :, Alhaija, H. A., Alvarez, J., Bala, M., Cai, T., Cao, T., Cha, L., Chen, J., Chen, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14492 · Cited by 121
Unbounded: A Generative Infinite Game of Character Life Simulation
Li, J., Li, Y., Wadhwa, N., Pritch, Y., Jacobs, D. E., Rubinstein, M., Bansal, M., & Ruiz, N. (2024) · arXiv · doi:10.48550/arxiv.2410.18975 · Cited by 2
Engaging Communities Meaningfully in Defining Disability Representation for AI Image Generation
Thieme, A., Faia Marques, R., Grayson, M., Balachandar, S., Tyler Cassidy, C., Choksi, M. Z., Longden, C., Huda, R. S., Ileve Kalovwe, N., Mallon, C., et al. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790768 · Cited by 1
Glob-Diffusion: A Global Consistent Diffusion Model for Large-Scale Image Generation
Kang, Y., Shi, H., Liu, H., Xie, W., Fang, L., & Bruzzone, L. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3643743 · Cited by 1
Chen, Z., & Wang, H. (2026) · Displays · doi:10.1016/j.displa.2026.103566 · Cited by 1
Kwon, P., & Chen, C. (2025) · arXiv · doi:10.48550/arxiv.2512.01686 · Cited by 1
“Story visualization [11, 33, 65, 79], the task of generating coherent visual sequences from textual narratives and character identities, has gained increasing attention with the advancement of generation models [2, 14, 40, 43] and controllable image customization methods [34, 35, 63, 69, 74].”
Manga Generation via Layout-controllable Diffusion
Chen, S., Li, D., Bao, Z., Zhou, Y., Tan, L., Zhong, Y., & Zhao, Z. (2024) · arXiv · doi:10.48550/arxiv.2412.19303
DiffSim: Taming Diffusion Models for Evaluating Visual Similarity
Song, Y., Liu, X., & Shou, M. Z. (2024) · arXiv · Cited by 1
“Enhancements in controllable image generation have been driven by methods such as ControlNet [82] and T2I-adapter [46], which significantly improve controllability over generated images by employing multimodal inputs such as depth maps and segmentation maps.”
Metadata
- Authors
- Zhang, L., Rao, A., & Agrawala, M.
- Affiliations
- Stanford University (US)
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 3813-3824
- OpenAlex
- W4390873054
- Cited by
- 4,547
- References
- 91
- Citations by year
- 2026: 1290 · 2025: 1847 · 2024: 1255 · 2023: 155
- FWCI
- 486.28
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Brown Institute for Media Innovation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- ControlNet, text-to-image diffusion models, spatial conditioning, conditional image generation, Stable Diffusion, parameter-efficient fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
