The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation
Open accessAbstract
Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learning has led to a breakthrough. The impressive progress of monocular depth estimators has mirrored the growth in model capacity, from relatively modest CNNs to large Transformer architectures. Still, monocular depth estimators tend to struggle when presented with images with unfamiliar content and layout, since their knowledge of the visual world is restricted by the data seen during training, and challenged by zero-shot generalization to new domains. This motivates us to explore whether the extensive priors captured in recent generative diffusion models can enable better, more generalizable depth estimation. We introduce Marigold, a method for affine-invariant monocular depth estimation that is derived from Stable Diffusion and retains its rich prior knowledge. The estimator can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases. Project page: https://marigoldmonodepth.github.io.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · cited 4×
“The step from CNNs to vision transformers has further boosted performance, as evidenced by DPT [36] and Omnidata [13].”
InfluentialDreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
Deep Ordinal Regression Network for Monocular Depth Estimation
Fu, H., Gong, M., Wang, C., Batmanghelich, K., & Tao, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00214 · Cited by 1,991
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer
Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., & Koltun, V. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3019967 · Cited by 1,618
Self-citationCLIP-Adapter: Better Vision-Language Models with Feature Adapters
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,197
MegaDepth: Learning Single-View Depth Prediction from Internet Photos
Li, Z., & Snavely, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00218 · Cited by 1,135
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., & Koltun, V. (2022) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000462024 · Cited by 1,093
Self-citationLAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos
Schops, T., Schonberger, J. L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., & Geiger, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.272 · Cited by 915
Wagner, W., Ullrich, A., Ducic, V., Melzer, T., & Studnicka, N. (2006) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2005.12.001 · Cited by 762
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
AdaBins: Depth Estimation Using Adaptive Bins
Farooq Bhat, S., Alhashim, I., & Wonka, P. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00400 · Cited by 726
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language ModelingPreprint
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03930 · Cited by 586
Extract Free Dense Labels from CLIP
Zhou, C., Loy, C. C., & Dai, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_40 · Cited by 513
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
Lee, J. H., Han, M.-K., Ko, D. W., & Suh, I. H. (2019) · arXiv · doi:10.48550/arxiv.1907.10326 · Cited by 507
Neural Window Fully-connected CRFs for Monocular Depth Estimation
Yuan, W., Gu, X., Dai, Z., Zhu, S., & Tan, P. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00389 · Cited by 361
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
Roberts, M., Ramapuram, J., Ranjan, A., Kumar, A., Bautista, M. A., Paczan, N., Webb, R., & Susskind, J. M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01073 · Cited by 358
DDP: Diffusion Model for Dense Visual Prediction
Ji, Y., Chen, Z., Xie, E., Hong, L., Liu, X., Liu, Z., Lü, T., Li, Z., & Luo, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01987 · Cited by 288
Progressive Distillation for Fast Sampling of Diffusion Models
Salimans, T., & Ho, J. (2022) · arXiv · doi:10.48550/arxiv.2202.00512 · Cited by 287
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans
Eftekhar, A., Sax, A., Malik, J., & Zamir, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01061 · Cited by 258
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization
Yao, H., Zhang, R., & Xu, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00653 · Cited by 252
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image
Yin, W., Zhang, C., Chen, H., Cai, Z., Yu, G., Wang, K., Chen, X., & Shen, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00830 · Cited by 233
Li, Z., Chen, Z., Liu, X., & Jiang, J. (2023) · Machine Intelligence Research · doi:10.1007/s11633-023-1458-0 · Cited by 228
Learning to Recover 3D Scene Shape from a Single Image
Yin, W., Zhang, J., Wang, O., Niklaus, S., Mai, L., Chen, S., & Shen, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00027 · Cited by 218
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation
Li, Z., Wang, X., Liu, X., & Jiang, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3416065 · Cited by 213
Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction
Yang, G., Tang, H., Ding, M., Sebe, N., & Ricci, E. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01596 · Cited by 210
The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
Saxena, S., Herrmann, C., Hur, J., Kar, A., Norouzi, M., Sun, D., & Fleet, D. J. (2023) · neural information processing systems · doi:10.52202/075280-1713 · Cited by 179
Unleashing Text-to-Image Diffusion Models for Visual Perception
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., & Lu, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00527 · Cited by 175
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners
Zhang, R., Hu, X., Li, B., Huang, S., Deng, H., Qiao, Y., Gao, P., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01460 · Cited by 171
Self-citationP3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior
Patil, V., Sakaridis, C., Liniger, A., & Van Gool, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00166 · Cited by 159
DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimation
Duan, Y., Guo, X., & Zhu, Z. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73247-8_25 · Cited by 159
Monocular Depth Estimation using Diffusion ModelsPreprint
Saxena, S., Kar, A., Norouzi, M., & Fleet, D. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.14816 · Cited by 149
Exploring Visual Prompts for Adapting Large-Scale Models
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022) · arXiv · doi:10.48550/arxiv.2203.17274 · Cited by 129
Towards Zero-Shot Scale-Aware Monocular Depth Estimation
Guizilini, V., Vasiljevic, I., Chen, D., Ambruş, R., & Gaidon, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00847 · Cited by 122
Neural LiDAR Fields for Novel View Synthesis
Huang, S., Gojčič, Ž., Wang, Z., Williams, F., Kasten, Y., Fidler, S., Konrad Schindler, B., & Litany, O. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01672 · Cited by 115
Self-citationDiverseDepth: Affine-invariant Depth Prediction Using Diverse DataPreprint
Yin, W., Wang, X., Shen, C., Liu, Y., Tian, Z., Xu, S., Sun, C., & Renyin, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00569 · Cited by 82
All in Tokens: Unifying Output Space of Visual Tasks via Soft Token
Ning, J., Li, C., Zhang, Z., Wang, C., Geng, Z., Dai, Q., He, K., & Hu, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01822 · Cited by 80
Bidirectional Attention Network for Monocular Depth Estimation
Aich, S., Vianney, J. M. U., Islam, M. A., & Liu, M. K. B. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560885 · Cited by 78
OASIS: A Large-Scale Dataset for Single Image 3D in the Wild
Chen, W., Qian, S., Fan, D., Kojima, N., Hamilton, M., & Deng, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00076 · Cited by 57
Generative Models: What Do They Know? Do They Know Things? Let's Find Out!Preprint
Du, X., Kolkin, N., Shakhnarovich, G., & Bhattad, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.17137 · Cited by 57
SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained ModelsPreprint
Pantazis, O., Brostow, G., Jones, K. E., & Aodha, O. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03794 · Cited by 42
3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features
Xu, C., Ling, H., Fidler, S., & Litany, O. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01010 · Cited by 36
Cited by
Hu, M., Yin, W., Zhang, C., Cai, Z., Long, X., Chen, H., Wang, K., Yu, G., Shen, C., & Shen, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3444912 · Cited by 260
Probing the 3D Awareness of Visual Foundation Models
Banani, M. E., Raj, A., Maninis, K.-K., Kar, A., Li, Y., Rubinstein, M., Sun, D., Guibas, L., Johnson, J., & Jampani, V. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02059 · Cited by 52
WorldScore: A Unified Evaluation Benchmark for World Generation
Duan, H., Yu, H., Chen, S., Fei–Fei, L., & Wu, J. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02573 · Cited by 17
Metadata
- Authors
- Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R. C., & Schindler, K.
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 9492-9502
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402753888
- Cited by
- 412
- References
- 57
- FWCI
- 89.96
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.