Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

“GQA evaluates spatial reasoning, while As such, we evaluate bounding box prediction accuracy on RefCOCO, RefCOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Yu et al., 2016), and on OCID-Ref (Wang et al., 2021).”

“Visually-conditioned language models inherit all of the risks and biases associated with language models (Touvron et al., 2023; Brown et al., 2020), as well as with underlying vision models and corresponding pretraining datasets (Schuhmann et al., 2021; Lin et al., 2014).”

“4, right) and (2) a finetuning stage, where only the visual representation is frozen while both the projection and LM are trained.”

“For TextVQA, we also run a variant of the evaluation where VLMs are additionally prompted with input tokens parsed by an OCR-system.”

“Referring expression grounding (bounding box prediction) and region captioning data sourced from RefCOCO (Kazemzadeh et al., 2014; Yu et al., 2016) and Visual Genome (Krishna et al., 2017).”

“5, and those defined in Dai et al. (2023) for InstructBLIP.”

“GQA evaluates spatial reasoning, while As such, we evaluate bounding box prediction accuracy on RefCOCO, RefCOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Yu et al., 2016), and on OCID-Ref (Wang et al., 2021).”

“We leverage TIMM (Wightman, 2019) and Hugging Face Transformers (Wolf et al., 2019) to provide pretrained models.”

“Unfortunately, instruct tuning has drawbacks, introducing bias and regressions in performance (Ouyang et al., 2022).”

“This “patch-as-token” approach enables training with a simple objective – next-token prediction – and allows us to harness the ecosystem of powerful LMs such as Llama-2 and Mistral (Touvron et al., 2023; Jiang et al., 2023), along with the tools to efficiently train them (e.g., FSDP; Zhao et al., 2023).”

“…adoption is fueled by a recent paradigm shift in how we develop VLMs; es-chewing the complex architectures and training objectives of prior work (Tan & Bansal, 2019; Li et al., 2022; 2023b), new VLMs adopt a simple approach, treating patch features from pretrained visual backbones (e.g., CLIP;…”

“We evaluate on Visual Spatial Reasoning (VSR; Liu et al., 2022), TallyQA (Acharya et al., 2018), POPE (Li et al., 2023d), and AI2 Diagrams (AI2D; Kembhavi et al., 2016).”

“The first subset consists of a 558K sample mixture of examples sourced from various captioning datasets (e.g., Conceptual Captions, LAION Sharma et al., 2018; Schuhmann et al., 2021), while the second consists of 665K multimodal instruct tuning examples comprised of synthetic data generated in Liu et al. (2023c), as w…”

“5, and PALI-3 that adopt the same underlying recipe, while varying individual ingredients such as the choice of pretrained components, data, or optimization procedure (Liu et al., 2023b; Chen et al., 2023b).”

“…by a recent paradigm shift in how we develop VLMs; es-chewing the complex architectures and training objectives of prior work (Tan & Bansal, 2019; Li et al., 2022; 2023b), new VLMs adopt a simple approach, treating patch features from pretrained visual backbones (e.g., CLIP; Radford et al.,…”

“Base vs. Instruct-Tuned LMs. Instruct tuning (or chat tuning; Ouyang et al., 2022; Chung et al., 2022) is a way to finetune base LMs (trained for next-token prediction) to behave as dialogue agents, offering a natural input/output interface for a wide spectrum of applications.”

“We investigate how LM performance on language-only benchmarks translates to downstream VLM performance, training VLMs from Mis-tral v1 7B and Mistral Instruct v1 7B (Jiang et al., 2023), recent LMs that outperform Llama-2 on language and code benchmarks (Hendrycks et al., 2021; Chen et al., 2021).”

“One of the prevalent design choices adopted by many VLMs (Chen et al., 2023a; Ye et al., 2023) is the inclusion of a two-stage training pipeline: (1) an alignment stage to align vision and language features by training the randomly initialized projector F ψ in isolation, freezing all other…”

“…datasets: LVIS-Instruct-4V (Wang et al., 2023), obtained by prompting GPT-4V to generate rich synthetic examples from images sourced from LVIS (Gupta et al., 2019), and LRV-Instruct (Liu et al., 2023a) that specifically optimizes for image diversity relative to existing datasets (adding e.g.,…”

“GQA evaluates spatial reasoning, while As such, we evaluate bounding box prediction accuracy on RefCOCO, RefCOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Yu et al., 2016), and on OCID-Ref (Wang et al., 2021).”

“…et al., 2010; Goyal et al., 2017; Hudson & Manning, 2019; Singh et al., 2019), four tasks spanning object localization (Kazemzadeh et al., 2014; Yu et al., 2016; Wang et al., 2021), and four challenge tasks evaluating fine-grained spatial reasoning, hallucination, and diagram understanding…”

“We also find that fused visual backbones that merge features from different backbones such as CLIP (Radford et al., 2021) and DINOv2 (Oquab et al., 2023) lead to more per-formant VLMs across the board.”

“Of primary concern is the generality of our model architecture; while the three component architecture we define in §2 is reflective of the majority of existing VLMs, there are other architecture innovations and optimization procedures that our study does not currently capture; as a notable example, we do not study ar…”

“5 training implementation: when benchmarked on the same hardware (an AWS p4de.24xlarge node with 8 A100 GPUs), we observe 20% faster step times with our FSDP-backed implementation, a notable gain given LLaVa leverages the well-optimized DeepSpeed ZeRO library (Rasley et al., 2020).”

“For TextVQA, we also run a variant of the evaluation where VLMs are additionally prompted with input tokens parsed by an OCR-system.”

“For TextVQA, we also run a variant of the evaluation where VLMs are additionally prompted with input tokens parsed by an OCR-system.”

“A synthetically generated dataset of conversations, fine-grained descriptions, and question-answering data from Liu et al. (2023c), sourced by prompting GPT-4 (OpenAI et al., 2023) with image captions and object bounding boxes from COCO (Lin et al., 2014).”

“For this reason, we include both VizWiz (Bigham et al., 2010) and POPE (Li et al., 2023d) in our evaluation suite; VizWiz has a series of commonsense questions and unanswerable questions that are explicitly used to probe model reliability.”

“…established position that building visually-conditioned language models models in the open – with open data, open (and efficient) training code, and open evaluation code – is strictly beneficial for the broader machine learning community and the public (Zellers et al., 2019; Touvron et al., 2023).”

“A, different “trigger prompts” induce models to produce outputs of a specific structure (e.g., short phrases for visual question answering evaluations such as VQAv2).”

“…of a given VLM. Recent work in evaluating VLMs tends to rely on automated evaluations that use powerful LMs such as GPT-4 (OpenAI et al., 2023) to judge relative and subjective performance(Liu et al., 2023e; Yu et al., 2023), making it hard to measure the absolute impact of a given design change.”

“A rich body of prior work in vision identifies that different types of visual representations trained with different inductive biases can lead to improved performance for a broad spectrum of applications (Kobayashi et al., 2022; Karamcheti et al., 2023).”

“…responses from image input and text prompts, providing a general, expressive interface for a growing spectrum of applications – grounded chat (Li et al., 2023c; Gong et al., 2023), visual programming (Sur’is et al., 2023; Sub-ramanian et al., 2023), robotic control (Driess et al., 2023;…”

“Following the hypotheses in Kerr et al. (2023) and similar work, we believe that DINOv2 features provide features that capture low-level spatial properties of an image, augmenting the higher-level “semantic” properties captured by vision-language contrastive models.”

“Visually-conditioned language models inherit all of the risks and biases associated with language models (Touvron et al., 2023; Brown et al., 2020), as well as with underlying vision models and corresponding pretraining datasets (Schuhmann et al., 2021; Lin et al., 2014).”

“We investigate how LM performance on language-only benchmarks translates to downstream VLM performance, training VLMs from Mis-tral v1 7B and Mistral Instruct v1 7B (Jiang et al., 2023), recent LMs that outperform Llama-2 on language and code benchmarks (Hendrycks et al., 2021; Chen et al., 2021).”

“…comparison between CLIP, SigLIP (Zhai et al., 2023), DINOv2 (Oquab et al., 2023), and a standard Vision Transformer pretrained for classification (on ImageNet-21K, finetuned on ImageNet-1K; Dosovitskiy et al., 2021; Steiner et al., 2021); for fair comparison, we use the ViT-Large model variant.”

“We investigate how LM performance on language-only benchmarks translates to downstream VLM performance, training VLMs from Mis-tral v1 7B and Mistral Instruct v1 7B (Jiang et al., 2023), recent LMs that outperform Llama-2 on language and code benchmarks (Hendrycks et al., 2021; Chen et al., 2021).”

“…four tasks spanning object localization (Kazemzadeh et al., 2014; Yu et al., 2016; Wang et al., 2021), and four challenge tasks evaluating fine-grained spatial reasoning, hallucination, and diagram understanding (Acharya et al., 2018; Liu et al., 2022; Li et al., 2023d; Kembhavi et al., 2016).”

“…image input and text prompts, providing a general, expressive interface for a growing spectrum of applications – grounded chat (Li et al., 2023c; Gong et al., 2023), visual programming (Sur’is et al., 2023; Sub-ramanian et al., 2023), robotic control (Driess et al., 2023; Brohan et al., 2023), etc.”

“…image input and text prompts, providing a general, expressive interface for a growing spectrum of applications – grounded chat (Li et al., 2023c; Gong et al., 2023), visual programming (Sur’is et al., 2023; Sub-ramanian et al., 2023), robotic control (Driess et al., 2023; Brohan et al., 2023), etc.”

“Images and captions sourced from TextCaps (images with text/logos; Sidorov et al., 2020).”

“A rich body of prior work in vision identifies that different types of visual representations trained with different inductive biases can lead to improved performance for a broad spectrum of applications (Kobayashi et al., 2022; Karamcheti et al., 2023).”

“With these requirements, we implement our training codebase in PyTorch, using Fully Sharded Data Parallel (FSDP; Zhao et al., 2023) and BF16 mixed precision.”

“We adopt the general architecture used by many recent VLMs, such as LLaVa, Qwen-VL, and PaLI-3 (Liu et al., 2023c; Bai et al., 2023; Chen et al., 2023b).”

“5, and PALI-3 that adopt the same underlying recipe, while varying individual ingredients such as the choice of pretrained components, data, or optimization procedure (Liu et al., 2023b; Chen et al., 2023b).”

“Following reproducibility practices from prior work (Karamcheti et al., 2021; Biderman et al., 2023), we fix initialization randomness and fix batch order during training.”

“VLMs are especially vulnerable to adversarial or even out-of-distribution image inputs that may inadvertently trigger unsafe output (Qi et al., 2023; Liu et al., 2023d).”

“…al., 2023), obtained by prompting GPT-4V to generate rich synthetic examples from images sourced from LVIS (Gupta et al., 2019), and LRV-Instruct (Liu et al., 2023a) that specifically optimizes for image diversity relative to existing datasets (adding e.g., charts, scientific diagrams, and news…”

“…four tasks spanning object localization (Kazemzadeh et al., 2014; Yu et al., 2016; Wang et al., 2021), and four challenge tasks evaluating fine-grained spatial reasoning, hallucination, and diagram understanding (Acharya et al., 2018; Liu et al., 2022; Li et al., 2023d; Kembhavi et al., 2016).”

“In this experiment, we perform a head-to-head comparison between CLIP, SigLIP (Zhai et al., 2023), DINOv2 (Oquab et al., 2023), and a standard Vision Transformer pretrained for classification (on ImageNet-21K, finetuned on ImageNet-1K; Dosovitskiy et al., 2021; Steiner et al., 2021); for fair comparison, we use the Vi…”

“Another trend in recent VLMs is increasing input image resolution with the hope of capturing fine-grained details that improve downstream performance (Liu et al., 2023b; Li et al., 2023a).”

“…of a given VLM. Recent work in evaluating VLMs tends to rely on automated evaluations that use powerful LMs such as GPT-4 (OpenAI et al., 2023) to judge relative and subjective performance(Liu et al., 2023e; Yu et al., 2023), making it hard to measure the absolute impact of a given design change.”

“…that our study does not currently capture; as a notable example, we do not study architectures that learn to downsample image patches, such as the Perceiver-based architectures used by Flamingo and IDEFICS (Alayrac et al., 2022; Laurenc¸on et al., 2023) for interleaved image-text training.”

“…image input and text prompts, providing a general, expressive interface for a growing spectrum of applications – grounded chat (Li et al., 2023c; Gong et al., 2023), visual programming (Sur’is et al., 2023; Sub-ramanian et al., 2023), robotic control (Driess et al., 2023; Brohan et al., 2023),…”

“Finally, for challenge tasks, we format each example as a multiple choice question and report accuracy; for VSR and POPE this means two options (for True/False and Yes/No, respectively), for AI2D this means the four provided multiple choice options, and for TallyQA, this means sixteen options (the numbers 0 - 15, incl…”

“VLMs are especially vulnerable to adversarial or even out-of-distribution image inputs that may inadvertently trigger unsafe output (Qi et al., 2023; Liu et al., 2023d).”

“GQA evaluates spatial reasoning, while As such, we evaluate bounding box prediction accuracy on RefCOCO, RefCOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Yu et al., 2016), and on OCID-Ref (Wang et al., 2021).”

“We swept the learning rate across multiple orders of magnitude for VLA training, and achieved the best results using a fixed learning rate of 2e-5 (the same learning rate used during VLM pretraining [44]).”

“These models are available on HuggingFace, as provided by [61].”

“To better visualize the performance differences across metrics of different scales, we use Z -score normalization (using the mean and standard deviation across all models) to scale the plots 45 .”

“Although some systematic studies have been made for the 7B/13B LMMs [37], [48], there is no clear evidence that these empirical design choices can be directly transferred to the LMMs of much smaller sizes.”