CoChat
Sign Up
PreprintCited by 68Multimodal Machine Learning ApplicationsAdvanced Neural Network ApplicationsTopic Modeling

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Open access
Tong, Shengbang, Brown, Ellis, Wu, Penghao, Woo, Sanghyun, Middepogu, Manoj, Akula, Sai Charitha, Yang, Jihan, Yang, Shusheng +6 more·2024·arXiv
DOI 10.48550/arxiv.2406.16860arXiv 2406.16860
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.16860

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A video encoder trained without language supervision can outperform encoders trained with language supervision, in contrast to conventional wisdom.

    “Additionally, since we only change the vision encoder, we provide evidence that a video encoder trained without language supervision can outperform encoders trained with language supervision, in contrast to conventional wisdom (Tong et al., 2024; Li et al., 2024b; Liu et al., 2024d; Yuan et al., 2025).”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Tong et al., 2024

    Section 3.4 reports that the self-supervised DINOv2 vision model outperformed some language-supervised models on vision-centric benchmarks. Cambrian-1 evaluates image-oriented vision encoders, not a video encoder, and it reports an advantage on particular benchmarks rather than across encoders generally. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 134

  1. M. Acharya, K. Kafle, and C. Kanan. "TallyQA: Answering complex counting questions". In: AAAI. 2019.
  2. A. Agrawal et al. "Don't just assume; look and answer: Overcoming priors for visual question answering". In: CVPR. 2018.
  3. A. Ahmadyan et al. "Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations". In: CVPR (2021).
  4. H. A. Alawwad et al. "Enhancing Textbook Question Answering Task with Large Lan- guage Models and Retrieval Augmented Generation". In: arXiv preprint arXiv:2402.05128 (2024). · arXiv
  5. J.-B. Alayrac et al. "Flamingo: a visual language model for few-shot learning". In: NeurIPS. 2022.
  6. M. Assran et al. "Self-supervised learning from images with a joint-embedding predictive architecture". In: CVPR. 2023.
  7. J. Bai et al. "Qwen Technical Report". In: arXiv preprint arXiv:2309.16609 (2023). · arXiv
  8. M. E. Banani et al. "Probing the 3D Awareness of Visual Foundation Models". In: arXiv preprint arXiv:2404.08636 (2024). · arXiv
  9. G. Baruch et al. "ARKitScenes -A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data". In: NeurIPS Datasets and Benchmarks Track (Round 1). 2021.
  10. J. Belouadi, A. Lauscher, and S. Eger. "Automatikz: Text-guided synthesis of scientific vector graphics with tikz". In: ICLR. 2024.
  11. R. Birkl, D. Wofk, and M. Müller. "Midas v3. 1-a model zoo for robust monocular relative depth estimation". In: arXiv preprint arXiv:2307.14460 (2023). · arXiv
  12. A. F. Biten et al. "Latr: Layout-aware transformer for scene-text vqa". In: CVPR. 2022.
  13. A. F. Biten et al. "Scene text visual question answering". In: ICCV. 2019.
  14. G. Brazil et al. "Omni3d: A large benchmark and model for 3d object detection in the wild". In: CVPR. 2023.
  15. H. Caesar et al. "nuscenes: A multimodal dataset for autonomous driving". In: CVPR. 2020.
  16. J. Cha et al. "Honeybee: Locality-enhanced projector for multimodal llm". In: CVPR. 2024.
  17. S. Cha et al. "Visually Dehallucinative Instruction Generation: Know What You Don't Know". In: arXiv preprint arXiv:2402.09717 (2024). · arXiv
  18. D. J. Chalmers. "Does Thought Require Sensory Grounding? From Pure Thinkers to Large Language Models". In: Proceedings and Addresses of the American Philosophical Association 97 (2023), pp. 22-45.
  19. Y. Chang et al. "A survey on evaluation of large language models". In: ACM Transactions on Intelligent Systems and Technology 15.3 (2024), pp. 1-45.
  20. G. H. Chen et al. "ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision- Language Model". In: arXiv preprint arXiv:2402.11684 (2024). · arXiv
  21. L. Chen et al. "Are We on the Right Way for Evaluating Large Vision-Language Models?" In: arXiv preprint arXiv:2403.20330 (2024). · arXiv
  22. L. Chen et al. "Sharegpt4v: Improving large multi-modal models with better captions". In: arXiv preprint arXiv:2311.12793 (2023). · arXiv
  23. X. Chen et al. "Pali: A jointly-scaled multilingual language-image model". In: ICLR. 2023.
  24. X. Chen, S. Xie, and K. He. "An empirical study of training self-supervised vision transformers". In: ICCV. 2021.
  25. Z. Chen et al. "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites". In: arXiv preprint arXiv:2404.16821 (2024). · arXiv
  26. Z. Cheng et al. "HiTab: A hierarchical table dataset for question answering and natural language generation". In: ACL. 2022.
  27. M. Cherti et al. "Reproducible scaling laws for contrastive language-image learning". In: CVPR. 2023.
  28. W.-L. Chiang et al. "Chatbot arena: An open platform for evaluating llms by human preference". In: arXiv preprint arXiv:2403.04132 (2024). · arXiv
  29. X. Chu et al. "Mobilevlm v2: Faster and stronger baseline for vision language model". In: arXiv preprint arXiv:2402.03766 (2024). · arXiv
  30. W. Dai et al. "Instructblip: Towards general-purpose vision-language models with instruction tuning". In: NeurIPS. 2024.
  31. H. Dong et al. "Rlhf workflow: From reward modeling to online rlhf". In: arXiv preprint arXiv:2405.07863 (2024). · arXiv
  32. A. Dosovitskiy et al. "An image is worth 16x16 words: Transformers for image recogni- tion at scale". In: ICLR. 2021.
  33. A. Fang et al. "Data filtering networks". In: ICLR. 2024.
  34. X. Fu et al. "BLINK: Multimodal Large Language Models Can See but Not Perceive". In: arXiv preprint arXiv:2404.12390 (2024). · arXiv
  35. S. Y. Gadre et al. "Datacomp: In search of the next generation of multimodal datasets". In: vol. 36. 2024.
  36. J. Gao et al. "G-llava: Solving geometric problem with multi-modal large language model". In: arXiv preprint arXiv:2312.11370 (2023). · arXiv
  37. P. Gao et al. "Llama-adapter v2: Parameter-efficient visual instruction model". In: arXiv preprint arXiv:2304.15010 (2023). · arXiv
  38. P. Gao et al. "SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models". In: arXiv preprint arXiv:2402.05935 (2024). · arXiv
  39. Y. Ge et al. "Planting a seed of vision in large language model". In: arXiv preprint arXiv:2307.08041 (2023). · arXiv
  40. A. Geiger, P. Lenz, and R. Urtasun. "Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite". In: CVPR. 2012.
  41. R. Girshick et al. "Rich feature hierarchies for accurate object detection and semantic segmentation". In: CVPR. 2014.
  42. Y. Goyal et al. "Making the v in vqa matter: Elevating the role of image understanding in visual question answering". In: CVPR. 2017.
  43. D. Gurari et al. "Vizwiz grand challenge: Answering visual questions from blind people". In: CVPR. 2018.
  44. K. He et al. "Masked autoencoders are scalable vision learners". In: CVPR. 2022.
  45. X. He et al. "PathVQA: 30000+ Questions for Medical Visual Question Answering". In: CoRR abs/2003.10286 (2020). · arXiv
  46. T. Hiippala et al. "AI2D-RST: A multimodal corpus of 1000 primary school science diagrams". In: Language Resources and Evaluation 55 (2021), pp. 661-688.
  47. J. Hoffmann et al. "Training compute-optimal large language models". In: NeurIPS (2023).
  48. Y.-C. Hsiao, F. Zubach, M. Wang, et al. "Screenqa: Large-scale question-answer pairs over mobile app screenshots". In: arXiv preprint arXiv:2209.08199 (2022). · arXiv
  49. D. A. Hudson and C. D. Manning. "GQA: A New Dataset for Real-World Visual Reason- ing and Compositional Question Answering". In: CVPR. 2019.
  50. A. Jaegle et al. "Perceiver: General perception with iterative attention". In: ICML. 2021.
  51. J. Johnson et al. "Clevr: A diagnostic dataset for compositional language and elementary visual reasoning". In: CVPR. 2017.
  52. N. Jouppi et al. "Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings". In: Proceedings of the 50th Annual International Symposium on Computer Architecture. 2023.
  53. K. Kafle et al. "Dvqa: Understanding data visualizations via question answering". In: CVPR. 2018.
  54. S. Kantharaj et al. "Chart-to-text: A large-scale benchmark for chart summarization". In: ACL. 2022.
  55. S. Karamcheti et al. "Prismatic vlms: Investigating the design space of visually-conditioned language models". In: arXiv preprint arXiv:2402.07865 (2024). · arXiv
  56. A. Kembhavi et al. "A diagram is worth a dozen images". In: ECCV. 2016.
  57. D. Kiela et al. "The hateful memes challenge: Detecting hate speech in multimodal memes". In: NeurIPS. 2020.
  58. G. Kim et al. "Donut: Document understanding transformer without ocr". In: ECCV. 2022.
  59. A. Kirillov et al. "Segment anything". In: ICCV. 2023.
  60. R. Krishna et al. "Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations". In: IJCV (2016).
  61. H. Laurençon, L. Tronchon, and V. Sanh. "Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset". In: arXiv preprint arXiv:2403.09029 (2024). · arXiv
  62. H. Laurençon et al. "What matters when building vision-language models?" In: arXiv preprint arXiv:2405.02246 (2024). · arXiv
  63. A. C. Li et al. "Internet Explorer: Targeted Representation Learning on the Open Web". In: ICML. 2023.
  64. A. C. Li et al. "Your diffusion model is secretly a zero-shot classifier". In: ICCV. 2023.
  65. L. Li et al. "Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models". In: arXiv preprint arXiv:2403.00231 (2024). · arXiv
  66. Y. Li et al. "Mini-gemini: Mining the potential of multi-modality vision language models". In: arXiv preprint arXiv:2403.18814 (2024). · arXiv
  67. T.-Y. Lin et al. "Microsoft coco: Common objects in context". In: ECCV. 2014.
  68. H. Liu et al. "Improved baselines with visual instruction tuning". In: arXiv preprint arXiv:2310.03744 (2023). · arXiv
  69. H. Liu et al. "Visual Instruction Tuning". In: NeurIPS. 2023.
  70. Y. Liu et al. "Mmbench: Is your multi-modal model an all-around player?" In: arXiv preprint arXiv:2307.06281 (2023). · arXiv
  71. Y. Liu et al. "On the hidden mystery of ocr in large multimodal models". In: arXiv preprint arXiv:2305.07895 (2023). · arXiv
  72. Z. Liu and K. He. "A Decade's Battle on Dataset Bias: Are We There Yet?" In: arXiv preprint arXiv:2403.08632 (2024). · arXiv
  73. Z. Liu et al. "A convnet for the 2020s". In: CVPR. 2022.
  74. H. Lu et al. "DeepSeek-VL: towards real-world vision-language understanding". In: arXiv preprint arXiv:2403.05525 (2024). · arXiv
  75. P. Lu et al. "Dynamic prompt learning via policy gradient for semi-structured mathe- matical reasoning". In: ICLR. 2023.
  76. P. Lu et al. "Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning". In: NeurIPS. 2021.
  77. P. Lu et al. "Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning". In: ACL. 2021.
  78. P. Lu et al. "Learn to explain: Multimodal reasoning via thought chains for science question answering". In: NeurIPS. 2022.
  79. P. Lu et al. "Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts". In: ICLR (2023).
  80. Z. Luo et al. "Wizardcoder: Empowering code large language models with evol-instruct". In: ICLR. 2024.
  81. A. Majumdar et al. "OpenEQA: Embodied Question Answering in the Era of Foundation Models". In: 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024. 2024.
  82. K. Marino et al. "OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge". In: CVPR. 2019.
  83. A. Masry et al. "Chartqa: A benchmark for question answering about charts with visual and logical reasoning". In: ACL. 2022.
  84. M. Mathew, D. Karatzas, and C. Jawahar. "Docvqa: A dataset for vqa on document images". In: WACV. 2021.
  85. B. McKinzie et al. "Mm1: Methods, analysis & insights from multimodal llm pre- training". In: arXiv preprint arXiv:2403.09611 (2024). · arXiv
  86. A. Mitra et al. Orca-Math: Unlocking the potential of SLMs in Grade School Math. 2024. arXiv: 2402.14830 [cs.CL]. · arXiv
  87. M. Oquab et al. "Dinov2: Learning robust visual features without supervision". In: TMLR (2023).
  88. L. Ouyang et al. "Training language models to follow instructions with human feedback". In: NeurIPS. 2022.
  89. P. Pasupat and P. Liang. "Compositional semantic parsing on semi-structured tables". In: ACL. 2015.
  90. J. Piaget, M. Cook, et al. The origins of intelligence in children. Vol. 8. 5. International Universities Press New York, 1952.
  91. J. Pont-Tuset et al. "Connecting Vision and Language with Localized Narratives". In: ECCV. 2020.
  92. A. Radford et al. "Learning transferable visual models from natural language supervi- sion". In: ICML. 2021.
  93. R. Rafailov et al. "Direct preference optimization: Your language model is secretly a reward model". In: NeurIPS. 2024.
  94. M. Roberts et al. "Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding". In: ICCV. 2021.
  95. R. Rombach et al. "High-Resolution Image Synthesis With Latent Diffusion Models". In: CVPR. 2022.
  96. O. Russakovsky et al. "Imagenet large scale visual recognition challenge". In: IJCV (2015).
  97. C. Schuhmann et al. "Laion-5b: An open large-scale dataset for training next generation image-text models". In: NeurIPS. 2022.
  98. D. Schwenk et al. "A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge". In: ECCV. 2022.
  99. M. Shridhar et al. "ALFWorld: Aligning Text and Embodied Environments for Interactive Learning". In: ICLR. 2021.
  100. C. Si et al. "Design2Code: How Far Are We From Automating Front-End Engineering?" In: arXiv preprint arXiv:2403.03163 (2024). · arXiv
  101. O. Sidorov et al. TextCaps: a Dataset for Image Captioning with Reading Comprehension. 2020. arXiv: 2003.12462 [cs.CV]. · arXiv
  102. A. Singh et al. "Towards vqa models that can read". In: CVPR. 2019.
  103. S. Song, S. P. Lichtenberg, and J. Xiao. "Sun rgb-d: A rgb-d scene understanding bench- mark suite". In: CVPR. 2015.
  104. Q. Sun et al. "Eva-clip: Improved training techniques for clip at scale". In: arXiv preprint arXiv:2303.15389 (2023). · arXiv
  105. R. Tanaka, K. Nishida, and S. Yoshida. "VisualMRC: Machine Reading Comprehension on Document Images". In: AAAI. 2021.
  106. B. J. Tang, A. Boggust, and A. Satyanarayan. "Vistext: A benchmark for semantically rich chart captioning". In: arXiv preprint arXiv:2307.05356 (2023). · arXiv
  107. S. Tong, E. Jones, and J. Steinhardt. "Mass-producing failures of multimodal systems with language models". In: NeurIPS. 2024.
  108. S. Tong et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms". In: CVPR. 2024.
  109. H. Touvron et al. "LLaMA: Open and efficient foundation language models". In: arXiv preprint arXiv:2302.13971 (2023). · arXiv
  110. H. Tu et al. "How many unicorns are in this image? a safety evaluation benchmark for vision llms". In: arXiv preprint arXiv:2311.16101 (2023). · arXiv
  111. K. Vishniakov, Z. Shen, and Z. Liu. "ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy". In: ICML. 2024.
  112. J. Wang et al. "To see is to believe: Prompting gpt-4v for better visual instruction tuning". In: arXiv preprint arXiv:2311.07574 (2023). · arXiv
  113. K. Wang et al. "Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset". In: arXiv preprint arXiv:2402.14804 (2024). · arXiv
  114. J. Wei et al. "Chain-of-thought prompting elicits reasoning in large language models". In: NeurIPS. 2022.
  115. H. Wu et al. "Q-instruct: Improving low-level visual abilities for multi-modality founda- tion models". In: arXiv preprint arXiv:2311.06783 (2023). · arXiv
  116. P. Wu and S. Xie. "V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs". In: CVPR. 2024. [138]
  117. H. Xu et al. "Demystifying clip data". In: ICLR. 2024.
  118. A. Young et al. "Yi: Open foundation models by 01. ai". In: arXiv preprint arXiv:2403.04652 (2024). · arXiv
  119. L. Yu et al. Modeling Context in Referring Expressions. 2016. arXiv: 1608.00272 [cs.CV]. · arXiv
  120. T. Yu et al. "Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback". In: arXiv preprint arXiv:2312.00849 (2023). · arXiv
  121. X. Yue et al. "Mammoth: Building math generalist models through hybrid instruction tuning". In: ICLR. 2024.
  122. X. Yue et al. "Mmmu: A massive multi-discipline multimodal understanding and rea- soning benchmark for expert agi". In: CVPR. 2024.
  123. M. Yuksekgonul et al. "When and why vision-language models behave like bags-of- words, and what to do about it?" In: ICLR. 2022.
  124. X. Zhai et al. "Sigmoid loss for language image pre-training". In: ICCV. 2023.
  125. Y. Zhai et al. "Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning". In: arXiv preprint arXiv:2405.10292 (2024). · arXiv
  126. C. Zhang et al. "Raven: A dataset for relational and analogical visual reasoning". In: CVPR. 2019.
  127. Y. Zhang et al. "Llavar: Enhanced visual instruction tuning for text-rich image under- standing". In: arXiv preprint arXiv:2306.17107 (2023). · arXiv
  128. Y. Zhao et al. "Pytorch fsdp: experiences on scaling fully sharded data parallel". In: arXiv preprint arXiv:2304.11277 (2023). · arXiv
  129. L. Zheng et al. "Judging llm-as-a-judge with mt-bench and chatbot arena". In: NeurIPS. 2024.
  130. T. Zheng et al. "OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement". In: arXiv preprint arXiv:2402.14658 (2024). · arXiv
  131. B. Zhou et al. "Semantic understanding of scenes through the ade20k dataset". In: IJCV (2019).
  132. K. Zhou et al. "Don't Make Your LLM an Evaluation Benchmark Cheater". In: arXiv preprint arXiv:2311.01964 (2023). · arXiv
  133. D. Zhu et al. "Minigpt-4: Enhancing vision-language understanding with advanced large language models". In: arXiv preprint arXiv:2304.10592 (2023). · arXiv
  134. F. Zhu et al. "TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance". In: ACL. 2021.

Cited by

Sorted by citations

Authors
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., Wang, Z., Fergus, R., LeCun, Y., & Xie, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
87310-87356
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2406.16860
arXiv
2406.16860
OpenAlex
W4415796658
Cited by
68
References
1
Citations by year
2026: 52 · 2025: 13 · 2024: 3
FWCI
14.08
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Defense Science and Engineering Graduate
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
multimodal large language models, open-source AI, multimodal understanding, vision-language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review