The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Object Hallucination in Image Captioning
Open accessAbstract
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.One problem is that standard metrics only measure similarity to ground truth captions and may not fully capture image relevance.In this work, we propose a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination.We analyze how captioning model architectures and learning objectives contribute to object hallucination, explore when hallucination is likely due to image misclassification or language priors, and assess how well current sentence metrics capture object hallucination.We investigate these questions on the standard image captioning benchmark, MSCOCO, using a diverse set of models.Our analysis yields several interesting findings, including that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors. * Denotes equal contribution.NBT: A woman talking on a cell phone while sitting on a bench.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language model (LLM) systems, such as ChatGPT1 or Gemini2, can show impressive reasoning and question-answering capabilities but often ‘hallucinate’ false outputs and unsubstantiated answers3,4.”
From Detecting hallucinations in large language models using semantic entropy (2024) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This is because there is another contributor of hallucinations - training and modeling choices of neural models [121, 215, 222, 263].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [222]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“All the baseline models employ features extracted from the fourth layer of ResNet-101 (He et al., 2016), except for the GAN model which employs ResNet-152.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 9×
“We also consider LRCN (Donahue et al., 2015) which considers visual information at each time step, as opposed to just initializing the LSTM hidden state with extracted features.”
BackgroundMethodInfluentialMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“For attention based models, we consider Att2In (Rennie et al., 2017), which is similar to the original attention based model proposed by (Xu et al., 2015), except the image feature is only input into the cell gate as this was shown to lead to better performance.”
MethodResultLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“We also consider LRCN (Donahue et al., 2015) which considers visual information at each time step, as opposed to just initializing the LSTM hidden state with extracted features.”
MethodSelf-citationVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 2×
“his is perhaps because bounding box proposals come from the MSCOCO detection task and are thus “in-domain” as opposed to the TopDown-BB model which relies on proposals learned from the Visual Genome (Krishna et al., 2017) dataset. Second, frequently training models with the self-critical loss actually increases the am…”
MethodCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 6×
“Likewise, our work is related to other work which aims to build better evaluation tools ((Vedantam et al., 2015), (Anderson et al., 2016), (Cui et al., 2018)).”
BackgroundMethodInfluential(2005) · Cited by 4,152
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,231 · cited 5×
“For non-attention based methods we consider the FC model from Rennie et al. (2017) which incorporates visual information by initializing the LSTM hidden state with high level image features.”
MethodInfluentialSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 3×
“Likewise, our work is related to other work which aims to build better evaluation tools ((Vedantam et al., 2015), (Anderson et al., 2016), (Cui et al., 2018)).”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 3
- Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137.
- Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2016. Re-evaluating automatic metrics for image captioning. In European Chapter of the Association for Computational Linguistics.
- Anna Rohrbach, Makarand Tapaswi, Atousa Torabi, Tegan Maharaj, Marcus Rohrbach, Sanja Fi- dler Christopher Pal, and Bernt Schiele. 2017a. The Joint Video and Language Understand- ing Workshop: MovieQA and The Large Scale Movie Description Challenge (LSMDC). https://sites.google.com/site/ describingmovies/lsmdc-2017.
Cited by
Sorted by citations
Items 401–427 of 427
Nguyen, T., Yoon, H. S., Luu, T. M., Yoon, E., & Yoo, C. D. (2026) · IEEE Access · doi:10.1109/access.2026.3723028
“…as our baseline to evaluate the effectiveness of our method for mitigating object hallucination in LVLMs. Object Hallucination in Vision Language Models: Object hallucination refers to the generation of descriptions containing objects absent from the visual modality (i.e., the given image) [72].”
Huang, C.-Y., Chen, P.-Z., & Yang, H. (2026) · Proceedings - International Conference on Image Processing · doi:10.1109/icip61757.2026.11630515
Cohesion: Consensus-Based Hallucination Subspace Estimation for Multimodal Large Language Models
Chen, W.-H., Chen, Y.-F., & Chen, J.-C. (2026) · Proceedings - International Conference on Image Processing · doi:10.1109/icip61757.2026.11630364
Improving Conversational Artificial Intelligence by Anti-Hallucination Mechanism
Faza, M. N., Priyatna, K. D., Junaidi, M. A. D., & Sari, R. F. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-3-032-25259-3_27
Efficient Vision-Language Fusion-Cascade for Explainable Manufacturing Workplace Safety MonitoringPreprint
Dongkyu, Y., & Choi, S.-H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7379938
Toward trustworthy construction safety hazard detection with visual-language models
Orooje, M. S., Cecconi, F. R., Lee, G., Mei, Q., & Adil, M. (2026) · Journal of Information Technology in Construction · doi:10.36680/j.itcon.2026.045
Fang, J., Hou, Y., Han, B., Zhang, X., Liu, H., Hou, J., & Jia, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37252-9_8
Gao, Y., You, L. H., Xie, J., Wang, C., Fu, K., Liu, J., Xu, R., Kou, Z., Xu, H., Gao, L., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37023-5_2
Wang, Z., Gu, Z., Zhang, Z., Ma, Z., Li, B., Xu, C., Dai, S., Wang, Y., & Liu, H. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116162
Song, Z., Wang, C., & He, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37383-0_36
Graph-based Multi-Agent LLM Framework for OCR-based Visual Question Answering
Dang, T. L.-T., Nguyen, H. M., & Nguyen, K. V. (2026) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr72750.2026.11685818
“Large VLMs such as Flamingo [13] and BLIP-2 [14] likewise rely on implicit inter-979-8-3195-0175-2 nal representations, leaving hallucination a persistent concern in text-rich scenes [15], [16].”
Qin, T., Chen, J., Zhang, T., & Wen, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37422-6_19
kuppani, C., & Halavar, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38062-3_51
Grounding vs. Prompt Engineering for Compute-Constrained Vision-Language Models in AI Product Demos
Nk, J., Gaur, L., Meenakshi, R., Arya, V., & Raman, R. (2026) · International Journal of Software Science and Computational Intelligence · doi:10.4018/ijssci.423390
Direct Preference Optimization for Perceptual Alignment via Vision-Language Consistency
Lee, S., & Lee, D.-G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37520-9_18
MapQA: A Map-Question-Answering Benchmark for Visual Language Model Reasoning
Arnold, C., Alini, A., Alabdulkareem, A., Wang, J., Feenstra, P., Arnold, C., DeWitt, J. K., Ritsema, N. C., Yae, J. H., Katz, B., et al. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831218
“In addition, prior work [15, 45] shows that LMMs frequently underutilize visual evidence during reasoning, allowing language inputs and learned world knowledge to dominate their predictions.”
Mitigating Object Hallucination in Lvlms Via Attention Imbalance Rectification
Sun, H., Li, Q., Wang, P., & Zhang, M. (2026) · doi:10.1109/cvprf71909.2026.00857
“However, the issue of object hallucination , i.e., the generation of non-existent objects in visual inputs, remains a major challenge for researchers and significantly compromises the reliability and trustworthiness of LVLMs [31, 37, 41, 44].”
Attention-Space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
Jo, Y., Bae, S.-P., & Kim, T. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00929 · Cited by 1
“Specifically, we use POPE [19] and CHAIR [26], where the alignment between model responses and ground-truth answers is evaluated using GPT-4.”
Do Audio-Visual Large Language Models Really See and Hear?
Selvakumar, R. K., Jayakumar, K., Sakshi, S., Ghosh, S., Gao, R., & Manocha, D. (2026) · doi:10.1109/cvprf71909.2026.00566
“On the other hand, meth-ods for evaluating static image captions [27, 49] do not transfer to videos, which involve temporal dynamics, state changes, and audio-visual correspondence that cannot be evaluated using highly-engineered approaches.”
Wang, Z., Xu, B., Xia, Y., & Li, P. (2026) · doi:10.1109/cvprf71909.2026.00869
“Relevance (CHAIR) [31] is a widely used benchmark for evaluating object hallucinations.”
Alleviating Hallucinations in Large Vision-Language Models via Decoding-Time Perturbation Adaptation
Bai, J.-Q., Guo, H.-C., Liu, J.-H., Zhou, Z.-B., Yang, J., & Huang, F.-R. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00944
“Existing studies typically attribute hallucinations to several underlying causes: Over-reliance on language priors [6, 22, 37]; Lack of focus on critical visual cues [8, 48, 53], including the impact from attention sinks and an-9869 chor tokens [19, 36, 44, 54, 61]; Learning spurious correlations…”
Attention Never Lie: Visual Attention Defocus Reveals and Rectifies Hallucinations in MLLMs
Zhao, C.-X., Zhou, Y., & Yang, J.-F. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00854 · Cited by 2
“Notably, we choose GPT-4o as the primary hallucination evaluation tool because CHAIR’s [37] instance-level detection method sometimes fails to accurately identify hallucinations.”
Yong, C., Li, L., Xu, C., Huang, J., & Shen, F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7590072
Xu, J., Chen, Y., Hu, X., 谢远昌, Zhang, X., Yin, J., & Wang, C. (2026) · Accident Analysis & Prevention · doi:10.1016/j.aap.2026.108813
Grounded Semantic Quality and Diversity Diagnostics for Image Caption Evaluation
Kharat, S., & Barve, S. (2026) · International Conference on Networks, Computers and Communications (ICNCCom) · doi:10.1109/icnccom69117.2026.11736290
“For evaluation entity awareness mostly Entity precision, entity recall and entity F1 score is used which help measure how accurately the entity information is mentioned in the generated description by extracting named entities from generated and reference descriptions [11,12].”
Mitigating Hallucinations in Multimodal Large Language Models via Multi-view Consistency DecodingPreprint
Yang, Z., Song, X., Pan, Q., & Shi, G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7665640
RUBi: Reducing Unimodal Biases for Visual Question Answering
(2019) · HAL · Cited by 50
Metadata
- Authors
- Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K.
- Affiliations
- Max Planck Society (DE)
- University of California, Berkeley (US)
- Boston University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 4035-4045
- OpenAlex
- W2892087592
- Cited by
- 420
- References
- 19
- Citations by year
- 2026: 103 · 2025: 80 · 2024: 70 · 2023: 42 · 2022: 32 · 2021: 45 · 2020: 32 · 2019: 15 · 2018: 1
- FWCI
- 7.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- object hallucination, image captioning, hallucination evaluation, language priors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
