CoChat
Sign Up
Cited by 5Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control

Open access
Gupta, Gunshi, Yadav, Karmesh, Gal, Yarin, Kira, Zsolt, Batra, Dhruv, Lu, Cong, Rudner, Tim·2024·Advances in Neural Information Processing Systems 37
DOI 10.52202/079017-2360
Open this paper in CoChat — free accountOpen PDF

Abstract

Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs. Such capabilities are difficult to learn solely from task-specific data. This has led to the emergence of pre-trained vision-language models as a tool for transferring representations learned from internet-scale data to downstream tasks and new domains. However, commonly used contrastively trained representations such as in CLIP have been shown to fail at enabling embodied agents to gain a sufficiently fine-grained scene understanding -- a capability vital for control. To address this shortcoming, we consider representations from pre-trained text-to-image diffusion models, which are explicitly optimized to generate images from text prompts and as such, contain text-conditioned representations that reflect highly fine-grained visuo-spatial information. Using pre-trained text-to-image diffusion models, we construct Stable Control Representations which allow learning downstream control policies that generalize to complex, open-ended environments. We show that policies learned using Stable Control Representations are competitive with state-of-the-art representation learning approaches across a broad range of simulated control settings, encompassing challenging manipulation and navigation tasks. Most notably, we show that Stable Control Representations enable learning policies that exhibit state-of-the-art performance on OVMM, a difficult open-vocabulary navigation benchmark.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/079017-2360

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent work has shown that diffusion models are an effective way to do representation learning, particularly for image and video-based tasks.

    “Recent work has shown that diffusion models are an effective way to do so, particularly for image and video-based tasks.”

    From Video Diffusion Models: A Survey (2024) · cites this paper as Gupta et al. (2024)

    The Related Work section describes diffusion models as flexible representation learners for image-based computer-vision tasks, and the Introduction reports strong results for visual control tasks. The paper also uses frames sampled from videos for fine-tuning (Appendix B.1), but the cited passages do not establish effectiveness specifically for video-based representation-learning tasks. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. A. Ajay, S. Han, Y. Du, S. Li, A. Gupta, T. Jaakkola, J. Tenenbaum, L. Kaelbling, A. Srivastava, and P. Agrawal. "Compositional foundation models for hierarchical planning". arXiv preprint arXiv:2309.08587 (2023) (cit. on p. 3). · arXiv
  2. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. "Flamingo: a visual language model for few-shot learning". Advances in Neural Information Processing Systems (2022) (cit. on p. 1).
  3. A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakr- ishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. San- keti, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. "RT-1: Robotics Transformer for Real-World Control at Scale". arXiv e-prints (2022) (cit. on p. 1).
  4. X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. "Pali: a jointly-scaled multilin- gual language-image model". arXiv preprint arXiv:2209.06794 (2022) (cit. on p. 1). · arXiv
  5. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. "Diffu- sion policy: visuomotor policy learning via action diffusion". arXiv preprint arXiv:2303.04137 (2023) (cit. on p. 3). · arXiv
  6. D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray. "Scaling egocentric vision: the epic-kitchens dataset". In: European Conference on Computer Vision (ECCV). 2018 (cit. on pp. 6, 21).
  7. Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. "Learning universal policies via text-guided video generation". URL https://arxiv. org/abs/2302.00111 (2023) (cit. on p. 3).
  8. P. Esser, R. Rombach, and B. Ommer. "Taming transformers for high-resolution image synthesis". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021 (cit. on p. 4).
  9. R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al. "The" something something" video database for learning and evaluating visual common sense". In: Proceedings of the IEEE international conference on computer vision. 2017 (cit. on p. 6).
  10. P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. "Idql: implicit q-learning as an actor-critic method with diffusion policies". arXiv preprint arXiv:2304.10573 (2023) (cit. on p. 3). · arXiv
  11. J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: high definition video generation with diffusion models. 2022. arXiv: 2210.02303 [cs.CV] (cit. on p. 2). · arXiv
  12. J. Ho, A. Jain, and P. Abbeel. "Denoising diffusion probabilistic models". In: Advances in Neural Information Processing Systems. 2020 (cit. on p. 3).
  13. J. Ho and T. Salimans. "Classifier-free diffusion guidance". In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications. 2021 (cit. on p. 4).
  14. M. T. Jackson, M. T. Matthews, C. Lu, B. Ellis, S. Whiteson, and J. Foerster. Policy-guided diffusion. 2024. arXiv: 2404.06356 [cs.LG] (cit. on p. 3). · arXiv
  15. M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. "Planning with diffusion for flexible behavior synthesis". In: International Conference on Machine Learning. 2022 (cit. on p. 3).
  16. I. Kapelyukh, V. Vosylius, and E. Johns. "Dall-e-bot: introducing web-scale diffusion models to robotics". IEEE Robotics and Automation Letters (2023) (cit. on p. 3).
  17. S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. "Language-driven representation learning for robotics". In: Robotics: Science and Systems (RSS). 2023 (cit. on pp. 3, 7, 9, 12, 20, 23).
  18. A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. "Simple but effective: clip embeddings for embodied ai". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022 (cit. on pp. 3, 9).
  19. M. Khanna, Y. Mao, H. Jiang, S. Haresh, B. Shacklett, D. Batra, A. Clegg, E. Undersander, A. X. Chang, and M. Savva. "Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for Object- Goal Navigation". arXiv preprint (2023) (cit. on p. 24).
  20. V. Kumar, R. Shah, G. Zhou, V. Moens, V. Caggiano, J. Vakil, A. Gupta, and A. Rajeswaran. "Robohive -a unified framework for robot learning". In: NeurIPS: Conference on Neural Information Processing Systems. 2023 (cit. on pp. 7, 24).
  21. H. Liu, C. Li, Q. Wu, and Y. J. Lee. "Visual instruction tuning". arXiv preprint arXiv:2304.08485 (2023) (cit. on p. 1). · arXiv
  22. C. Lu, P. J. Ball, Y. W. Teh, and J. Parker-Holder. "Synthetic experience replay". In: Thirty-seventh Conference on Neural Information Processing Systems. 2023 (cit. on p. 3).
  23. Y. J. Ma, W. Liang, V. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. "LIV: Language-Image Representations and Rewards for Robotic Control". arXiv e-prints (2023) (cit. on pp. 1, 21).
  24. A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra. "Zson: zero- shot object-goal navigation using multimodal goal embeddings". Advances in Neural Information Processing Systems (2022) (cit. on p. 9).
  25. A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, P. Abbeel, J. Malik, et al. "Where are we in the search for an artificial visual cortex for embodied intelligence?" arXiv preprint arXiv:2303.18240 (2023) (cit. on pp. 3, 6-8, 23-25). · arXiv
  26. Z. Mandi, H. Bharadhwaj, V. Moens, S. Song, A. Rajeswaran, and V. Kumar. "Cacti: a framework for scalable multi-task multi-scene visual imitation learn- ing". arXiv preprint arXiv:2212.05711 (2022) (cit. on p. 3). · arXiv
  27. L. Mezghani, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. "Memory-augmented reinforcement learning for image-goal navigation". arXiv preprint arXiv:2101.05181 (2021) (cit. on p. 25). · arXiv
  28. S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. "R3m: a universal visual representation for robot manipulation". In: 6th Annual Conference on Robot Learning. 2022 (cit. on pp. 1, 3, 7, 23).
  29. OpenAI. Gpt-4 technical report. 2023. arXiv: 2303.08774 [cs.CL] (cit. on p. 1). · arXiv
  30. T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. "Imitating human behaviour with diffusion models". arXiv preprint arXiv:2301.10677 (2023) (cit. on p. 3). · arXiv
  31. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. "Learning transferable visual models from natural language supervision". In: Proceedings of the 38th International Conference on Machine Learning. 2021 (cit. on pp. 1-4, 7, 13, 23).
  32. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. "High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022 (cit. on pp. 1, 2, 4, 7, 23).
  33. O. Ronneberger, P. Fischer, and T. Brox. "U-net: convolutional networks for biomedical image segmentation". In: Medical Image Computing and Computer- Assisted Intervention -MICCAI 2015 -18th International Conference Munich, Germany, October 5 -9, 2015, Proceedings, Part III. 2015 (cit. on p. 4).
  34. C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. "Photorealistic text- to-image diffusion models with deep language understanding". Advances in Neural Information Processing Systems (2022) (cit. on p. 4).
  35. M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. "Habitat: A Platform for Em- bodied AI Research". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2019 (cit. on p. 25).
  36. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. "LAION-5B: an open large-scale dataset for training next generation image-text models". Advances in Neural Information Processing Systems (2022) (cit. on p. 4).
  37. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. "Proximal Policy Optimization Algorithms". arXiv e-prints (2017) (cit. on p. 4).
  38. D. Shah, B. Osiński, b. ichter, and S. Levine. "Lm-nav: robotic navigation with large pre-trained models of language, vision, and action". In: Proceedings of The 6th Conference on Robot Learning. 2023 (cit. on p. 1).
  39. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. "Deep unsu- pervised learning using nonequilibrium thermodynamics". In: International conference on machine learning. PMLR. 2015 (cit. on p. 3).
  40. A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. "Habitat 2.0: training home assistants to rearrange their habitat". In: Advances in Neural Information Processing Systems (NeurIPS). 2021 (cit. on p. 24).
  41. R. Tang, L. Liu, A. Pandey, Z. Jiang, G. Yang, K. Kumar, P. Stenetorp, J. Lin, and F. Ture. "What the DAAM: interpreting stable diffusion using cross attention". In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023 (cit. on pp. 2, 3, 6).
  42. J. Tian, L. Aggarwal, A. Colaco, Z. Kira, and M. Gonzalez-Franco. "Diffuse, attend, and segment: unsupervised zero-shot segmentation using stable diffu- sion". arXiv preprint arXiv:2308.12469 (2023) (cit. on pp. 2, 3). · arXiv
  43. J. Traub. "Representation learning with diffusion models". arXiv preprint arXiv:2210.11058 (2022) (cit. on pp. 3, 4). · arXiv
  44. H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen- Estruch, Q. Vuong, A. He, et al. "Bridgedata v2: a dataset for robot learning at scale". arXiv preprint arXiv:2308.12952 (2023) (cit. on p. 6). · arXiv
  45. J. Wang, X. Li, J. Zhang, Q. Xu, Q. Zhou, Q. Yu, L. Sheng, and D. Xu. "Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter". arXiv e-prints (2023) (cit. on pp. 2-4, 6).
  46. K.-J. Wang, Y.-H. Liu, H.-T. Su, J.-W. Wang, Y.-S. Wang, W. Hsu, and W.-C. Chen. "OCID-ref: a 3D robotic dataset with embodied language for clutter scene grounding". In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021 (cit. on p. 9).
  47. E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. "Dd-ppo: learning near-perfect pointgoal navigators from 2.5 billion frames". arXiv preprint arXiv:1911.00357 (2019) (cit. on pp. 8, 25). · arXiv
  48. F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. "Gibson env: real- world perception for embodied agents". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018 (cit. on p. 25).
  49. T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. "Masked visual pre-training for motor control". arXiv:2203.06173 (2022) (cit. on pp. 3, 6, 7, 23). · arXiv
  50. J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello. "ODISE: Open- Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models". arXiv preprint arXiv: 2303.04803 (2023) (cit. on p. 5). · arXiv
  51. K. Yadav, A. Majumdar, R. Ramrakhya, N. Yokoyama, A. Baevski, Z. Kira, O. Maksymets, and D. Batra. Ovrl-v2: a simple state-of-art baseline for imagenav and objectnav. 2023. arXiv: 2303.07798 [cs.CV] (cit. on pp. 3, 12, 25). · arXiv
  52. X. Yang and X. Wang. "Diffusion model as representation learner". In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision. 2023 (cit. on pp. 3-5).
  53. S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, T.-Y. Yang, V. Jain, A. W. Clegg, J. Turner, et al. "Homerobot: open-vocabulary mobile manipulation". arXiv preprint arXiv:2306.11565 (2023) (cit. on pp. 2, 9, 24). · arXiv
  54. T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. "Meta- world: a benchmark and evaluation for multi-task and meta reinforcement learning". In: Conference on Robot Learning (CoRL). 2019. arXiv: 1910.10897 [cs.LG] (cit. on pp. 7, 24). · arXiv
  55. T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. "Scaling robot learning with semantically imagined experience". arXiv preprint arXiv:2302.11550 (2023) (cit. on p. 3). · arXiv
  56. J. Zhao, H. Zheng, C. Wang, L. Lan, and W. Yang. "Magicfusion: boosting text- to-image generation performance by fusing diffusion models". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023 (cit. on p. 12).
  57. X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. "Detecting twenty- thousand classes using image-level supervision". In: ECCV. 2022 (cit. on p. 9).
  58. Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. K. Gupta, L. Fei-Fei, and A. Farhadi. "Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning". ICRA (2017) (cit. on p. 8).
  59. B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu, S. Levine, L. Lee, T.-W. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, brian ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, C. Finn, K. A. Dubey, D. Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, J. Carbajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han. "RT-2: vision-language-action models transfer web knowledge to robotic control". In: 7th Annual Conference on Robot Learning. 2023 (cit. on p. 1).

Cited by

Sorted by citations

Metadata

Authors
Gupta, G., Yadav, K., Gal, Y., Kira, Z., Batra, D., Lu, C., & Rudner, T.
ORCID
Venue
Advances in Neural Information Processing Systems 37
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2024
Pages
74182-74210
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/079017-2360
OpenAlex
W4415797177
Cited by
5
References
2
Citations by year
2026: 3 · 2025: 2
FWCI
1.04
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Engineering and Physical Sciences Research Council (EP/S024050/1)
  • Office of Naval Research
  • Army Research Office
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
text-to-image diffusion models, pre-trained diffusion models, representation learning, diffusion control, image generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review