What Matters When Building Vision Language Models for Product Image Analysis?
Abstract
This paper investigates multi-modal large language models (MLLMs) for predicting product features from images, comparing fine-tuned versus proprietary models. We introduce two domain-specific benchmarks: (1) Inductive Bias vs. Image Evidence (IBIE) Benchmark, which evaluates MLLMs' ability to distinguish between image-derived features and latent knowledge, and (2) Catalog-bench, which assesses feature prediction using Catalog terminology. Our fine-tuned model outperforms proprietary models like Gemini by 9.4% and 29.13% on these benchmarks respectively. We address the crucial aspect of computational efficiency, exploring cost effective deployment solutions under limited hardware resources. The significance of this work extends beyond ecommerce to physical retail, where efficient MLLMs are essential for real-time processing of visual data from store cameras and shelf sensors. These models enable automated inventory management, produce quality monitoring, and planogram compliance while operating within in-store computing constraints. This capability is particularly valuable for physical retail environments where immediate decisions about restocking and quality control are critical, while also enabling real-time assistance to customers seeking information about product details, ingredients, and nutritional content.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent research suggests that VLMs require fine-tuning for good performance in recognizing products and their attributes.
“For recognizing products and their attributes, recent research suggests that VLMs require fi ne-tuning for good performance [104, 125].”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [125]
Cites
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · see citing works
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · see citing works
Sigmoid Loss for Language Image Pre-Training
Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01100 · Cited by 1,127 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 777 · see citing works
SqueezeLLM: Dense-and-Sparse QuantizationPreprint
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., & Keutzer, K. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2306.07629 · Cited by 410 · see citing works
VILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P. A., Shoeybi, M., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02520 · Cited by 284 · see citing works
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration
Lin, J., Tang, J., Tang, H., Yang, S., Xiao, G., & Han, S. (2025) · GetMobile: Mobile Computing and Communications · doi:10.1145/3714983.3714987 · Cited by 274 · see citing works
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022) · arXiv · doi:10.48550/arxiv.2210.17323 · Cited by 197 · see citing works
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023) · neural information processing systems · doi:10.52202/075280-3138 · Cited by 72 · see citing works
Fang, C., Li, X., Fan, Z., Xu, J., Nag, K., Körpeoğlu, E., Kumar, S., & Achan, K. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661357 · Cited by 53 · see citing works
PAM: Understanding Product Images in Cross Product Category Attribute Extraction
Lin, R., He, X., Feng, J., Zalmout, N., Liang, Y., Xiong, L., & Dong, X. L. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467164 · Cited by 28 · see citing works
Understanding Guided Image Captioning Performance across Domains
Ng, E. G., Pang, B., Sharma, P. K., & Soricut, R. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.14 · Cited by 19 · see citing works
EIVEN: Efficient Implicit Attribute Value Extraction using Multimodal LLM
Zou, H. P., Yu, G., Fan, Z., Bu, D., Liu, H., Dai, P., Jia, D., & Caragea, C. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-industry.40 · Cited by 9 · see citing works
Generative Models for Product Attribute Extraction
Blume, A., Zalmout, N., Ji, H., & Li, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.55 · Cited by 7 · see citing works
MIVC: Multiple Instance Visual Component for Visual-Language Models
Wu, W., Li, Q., Zhong, W., & Huang, J. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00793 · Cited by 2 · see citing works
Flagged first, then by citations
1 of the 20 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Garg, K., Tang, X., Heo, J., Morgan, D. R., Gergle, D., Sudderth, E. B., & Piper, A. M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791309 · Cited by 3 · see citing works
“For recognizing products and their attributes, recent research suggests that VLMs require fine-tuning for good performance [104, 125].”
Barskaya, G. B., Chernysheva, T., Sbrodov, S. O., & Tretyak, A. O. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-16105-5_14
No Code, No Cloud: On-Device Mockup-to-Code with Lightweight Vision-Language AI
Kuganathan, A., Purandare, M., & Stolze, M. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789144
Cheng, Z., Wang, Z., Tang, Y., Liu, D., Zhang, B., Νικολακόπουλος, Α. Ν., Itabada, P. S., Zhang, J., Chou, C.-C., Gao, P., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2759-4_9
Self-citation
- Authors
- Trabelsi, A., Zontak, M., Qian, Y., Jackson, B., Khan, S., & Batur, U.
- Affiliations
- Amazon (United States) (US)
- Venue
- 2025 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)
- Publisher
- IEEE
- Published
- 2025-02-28
- Pages
- 1282-1291
- OpenAlex
- W4409917617
- Cited by
- 3
- References
- 20
- Citations by year
- 2026: 3
- FWCI
- 2.27
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- multimodal large language models, physical stores, inventory management, product quality monitoring
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
