The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Object Hallucination in Image Captioning
Open accessAbstract
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.One problem is that standard metrics only measure similarity to ground truth captions and may not fully capture image relevance.In this work, we propose a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination.We analyze how captioning model architectures and learning objectives contribute to object hallucination, explore when hallucination is likely due to image misclassification or language priors, and assess how well current sentence metrics capture object hallucination.We investigate these questions on the standard image captioning benchmark, MSCOCO, using a diverse set of models.Our analysis yields several interesting findings, including that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors. * Denotes equal contribution.NBT: A woman talking on a cell phone while sitting on a bench.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language model (LLM) systems, such as ChatGPT1 or Gemini2, can show impressive reasoning and question-answering capabilities but often ‘hallucinate’ false outputs and unsubstantiated answers3,4.”
From Detecting hallucinations in large language models using semantic entropy (2024) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This is because there is another contributor of hallucinations - training and modeling choices of neural models [121, 215, 222, 263].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [222]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“All the baseline models employ features extracted from the fourth layer of ResNet-101 (He et al., 2016), except for the GAN model which employs ResNet-152.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 9×
“We also consider LRCN (Donahue et al., 2015) which considers visual information at each time step, as opposed to just initializing the LSTM hidden state with extracted features.”
BackgroundMethodInfluentialMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“For attention based models, we consider Att2In (Rennie et al., 2017), which is similar to the original attention based model proposed by (Xu et al., 2015), except the image feature is only input into the cell gate as this was shown to lead to better performance.”
MethodResultLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“We also consider LRCN (Donahue et al., 2015) which considers visual information at each time step, as opposed to just initializing the LSTM hidden state with extracted features.”
MethodSelf-citationVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 2×
“his is perhaps because bounding box proposals come from the MSCOCO detection task and are thus “in-domain” as opposed to the TopDown-BB model which relies on proposals learned from the Visual Genome (Krishna et al., 2017) dataset. Second, frequently training models with the self-critical loss actually increases the am…”
MethodCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 6×
“Likewise, our work is related to other work which aims to build better evaluation tools ((Vedantam et al., 2015), (Anderson et al., 2016), (Cui et al., 2018)).”
BackgroundMethodInfluential(2005) · Cited by 4,152
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,231 · cited 5×
“For non-attention based methods we consider the FC model from Rennie et al. (2017) which incorporates visual information by initializing the LSTM hidden state with high level image features.”
MethodInfluentialSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 3×
“Likewise, our work is related to other work which aims to build better evaluation tools ((Vedantam et al., 2015), (Anderson et al., 2016), (Cui et al., 2018)).”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 3
- Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137.
- Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2016. Re-evaluating automatic metrics for image captioning. In European Chapter of the Association for Computational Linguistics.
- Anna Rohrbach, Makarand Tapaswi, Atousa Torabi, Tegan Maharaj, Marcus Rohrbach, Sanja Fi- dler Christopher Pal, and Bernt Schiele. 2017a. The Joint Video and Language Understand- ing Workshop: MovieQA and The Large Scale Movie Description Challenge (LSMDC). https://sites.google.com/site/ describingmovies/lsmdc-2017.
Cited by
Sorted by citations
Items 201–300 of 427
Vision-Language Pretraining: Current Trends and the Future
Agrawal, A., Teney, D., & Nematzadeh, A. (2022) · doi:10.18653/v1/2022.acl-tutorials.7 · Cited by 5
“• Shortcomings of vision-language models (Agrawal et al., 2016; Rohrbach et al., 2018; Gan et al., 2020; Ross et al., 2020; van Mil-tenburg, 2016; Misra et al., 2015; Raji et al., 2020; Zhao et al., 2017a).”
Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
Li, R., Wu, Y., & He, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01275 · Cited by 5
“• Additionally, we assess the effectiveness of ICCC on another VLM (LLAVA [22]) and various vision-language reasoning datasets, including ScienceQA-IMG [31], MM-VET [37], and hallucinations [30].”
MethodHuang, P.-H., Li, J.-L., Chen, C.-P., Chang, M., & Chen, W.-C. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00597 · Cited by 7
“We set up the causal effect metric δ ( P, P ′ ) between a distribution P and the distribution after intervention P ′ is defined as the fewer hallucinations after the intervention; that is, δ = I ( H ( A ) > H ( A ′ )) , where A ′ denotes the resulting answer after the intervention and H is the…”
Commentary generation for financial markets
Zhu, D., Lappas, T., & Rachidi, T. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118364 · Cited by 4
Zaman, I.-R., & Trăușan-Matu, Ș. (2024) · Information · doi:10.3390/info15030123 · Cited by 5
Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of ExplainabilityPreprint
Shu, D., Zhao, H., Hu, J., Liu, W., Lu, C. Y., & Du, M. (2025) · Qeios · doi:10.32388/ooam1q · Cited by 4
Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
Sarkar, S., Che, Y., Gavin, A., Beerel, P. A., & Kundu, S. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.631 · Cited by 4
Improve the image caption generation on out-domain dataset by external knowledge augmented
Cao, D., Zhou, H., & Wang, Y. (2025) · Multimedia Systems · doi:10.1007/s00530-024-01645-w · Cited by 3
“Reference [26] proposes the CHAIR metric, which reflects the degree of object hallucination, enabling the evaluation of hallucination issues in models.”
InfluentialExploring Multimodal Knowledge for Image Compression via Large Foundation Models
Gao, J., Huang, Z., Mao, Q., Ma, S., & Jia, C. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3607616 · Cited by 3
“However, despite their promise, current large multi-modal models face significant challenges with hallucination in the I2T task, which deteriorates the generation of accurate text [36], [37].”
Visual Evidence-Aware for Object Hallucinations Rectification in LLM-Based Video Captioning
Wang, Y., Zhou, J., Liu, Q., Hu, F., & Wang, G. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3613993 · Cited by 3
“The former involves constructing more balanced and comprehensive training data to avoid hallucinations caused by data biases [13], while the latter focuses on mitigating hallucinations through model design [12], [14] and post-hoc correction methods [15], [16], [17], [18].”
InfluentialAsking Questions to Alleviate Object Hallucination in Large Vision-Language Models
Bi, C., Dang, T., Wang, S., Cao, F., & Huang, Q. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3618949 · Cited by 3
“Wang et al. [11] and Rohrbach et al. [55] checked the output text against the ground-truth to decide if the hallucination happens.”
Reasoning Segmentation for Images and Videos: A Survey
Shen, Y., Li, C., Xiong, F., Jeong, J.-O., Wang, T., Latman, M., & Unberath, M. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02903-2 · Cited by 15
“CHAIR The CHAIR (Caption Hallucination Assessment with Image Rele-vance) metric [109, 68] evaluates hallucination in RS by measuring the proportion of objects mentioned in the generated text response that are absent from the ground truth.”
Communication breakdown: On the low mutual intelligibility between human and neural captioning
Dessì, R., Gualdoni, E., Franzon, F., Boleda, G., & Baroni, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.546 · Cited by 3
CircuitVQA: A Visual Question Answering Dataset for Electrical Circuit Images
Mehta, R., Singh, B., Varma, V., & Gupta, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70341-6_26 · Cited by 3
BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-Language Models
Ye-Bin, M., Hyeon-Woo, N., Choi, W., & Oh, T.-H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73247-8_14 · Cited by 3
A goal-oriented document-grounded dialogue based on evidence generation
Song, Y. I., Fan, H., Liu, J., Liu, Y., Ye, X., & Ouyang, Y. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102378 · Cited by 3
Multiview Language Bias Reduction for Visual Question Answering
Li, P., Tan, Z., & Bao, B. (2022) · IEEE Multimedia · doi:10.1109/mmul.2022.3217307 · Cited by 3
Visual Semantic Relatedness Dataset for Image Captioning
Sabir, A., Moreno-Noguer, F., & Padró, L. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00592 · Cited by 4
“wrong visual, object hallucination (Rohrbach et al., 2018), etc.”
Chen, W., Wu, C.-K., Chen, H., & Chen, C.-C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.54 · Cited by 4
Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models
Kim, M., Kim, M., Bae, J., Choi, S., Kim, S., & Chang, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73016-0_14 · Cited by 5
Gabín, J., Ares, M. E., & Parapar, J. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01379-9 · Cited by 4
“This issue is well-known in several NLG areas, such as image captioning Rohrbach et al. (2018).”
Yan, H., Chang, Z., Tang, B., Lin, B., Luo, M., Bi, Y., & Li, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128392 · Cited by 3
MH-PEFT: Mitigating Hallucinations in Large Vision-Language Models through the PEFT Method
Li, F. (2025) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1145/3728725.3728747 · Cited by 3
“Early studies primarily focused on identifying and measuring hallucinations in image captioning tasks [24, 36].”
Xia, S.-J., Zhang, H., Jiang, Y., Chen, X., Chen, Y., Li, Z., Wan, Z., & Sun, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5731427 · Cited by 3
Advancing Causal Intervention in Image Captioning With Causal Prompt
Yu, Y., Kim, Y., & Ro, Y. M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3487200 · Cited by 2
“…the object bias in each generated image caption, we adopt two metrics, sentence-level CHAIRs and object instance level CHAIRi, respectively [62] CHAIR s = # of sentence with biased object # of all sentences (12) 3) Implementation Details: We first extracted object features using the…”
Aerial Mirage: Unmasking Hallucinations in Large Vision Language Models
Basak, D., Bhatt, S., Kanduri, S., & Desarkar, M. S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00537 · Cited by 4
“Hallucination Evaluation: Early hallucination evaluation methods, such as CHAIR [15], focused on identifying object hallucinations in image captioning by quantifying differences of objects from model-generated captions with groundtruth captions.”
Holtzman, A., West, P., & Zettlemoyer, L. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0009 · Cited by 32
He, Y., Sun, H., Qi, Q., Zhuang, Z., Ren, P., Wang, H., Nan, Y., & Wang, J. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209127 · Cited by 2
“Object HalBench [7] is a widely used benchmark for evaluating object hallucination.”
Kou, J., Lu, X., Yin, A., Zhang, W., Fang, X., Li, Y., Pan, T., Zhang, L., Cheng, G., Yuan, H., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189154 · Cited by 2
Boosting Faithful Multi-Modal LLMs via Complementary Visual Grounding
Fu, Z., Mao, Z.-D., Zhang, L., & Zhang, Y.-D. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3644140 · Cited by 6
“• CHAIR [26] is designed as the image captioning task.”
InfluentialLearned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate
Moslonka, C., Randrianarivo, H., Garnier, A., & Malherbe, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_8 · Cited by 4
Wang, Z., Ma, T., & Wang, H. (2026) · Die Radiologie · doi:10.1007/s00117-026-01608-4 · Cited by 2
LIRA: Inferring Segmentation in Large Multi-Modal Models with Local Interleaved Region Assistance
Li, Z. L., Yang, B., Liu, Q., Zhang, X., Zx, M., Yin, L., Deng, L., Sun, Y., Liu, Y., & Bai, X. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02230 · Cited by 4
“Specifically, the hallucinations of ChairS [45] decreased by 3.0% and 4.8% for the 1.8B and 7B LLMs, respectively.”
InfluentialAre Current Decoding Strategies Capable of Facing the Challenges of Visual Dialogue?
Chaudhary, A., Lucassen, A. J., Tsani, I., & Testoni, A. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.24 · Cited by 2
VLIS: Unimodal Language Models Guide Multimodal Language Generation
Chung, J., & Yu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.46 · Cited by 2
Sridhar, A. K., Guo, Y., Visser, E., & Mahfuz, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448154 · Cited by 4
“In recent years, there has been extensive research on pushing the boundaries of multimodal-to-text generation tasks like image captioning [1], audio captioning ([2], [3]) etc.”
BackgroundImproving Faithfulness and Factuality with Contrastive Learning in Explainable Recommendation
Zhuang, H., Zhang, W. E., Chen, W., Yang, J. Y., & Sheng, Q. Z. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3653984 · Cited by 5
“Explanations for recommendation can be generated in various forms or styles from different information sources [47], such as: — Relevant Users or Items : It mostly presents the nearest-neighbor users or items as the explanations, which could be used in the collaborative filtering recommendation…”
The Optimal Choice of the Encoder–Decoder Model Components for Image Captioning
Bartosiewicz, M., & Iwanowski, M. (2024) · Information · doi:10.3390/info15080504 · Cited by 2
Mitigating Hallucination in Visual-Language Models via Re-balancing Contrastive Decoding
Liang, X., Yu, J., Mu, L., Zhuang, J., Hu, J., Yang, Y., Ye, J., Lu, L., Chen, J., & Hu, H. R. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8620-6_33 · Cited by 8
Continual SFT Matches Multimodal RLHF with Negative Supervision
Zhu, K., Wang, Y., Sun, Y., Chen, Q., Liu, J., Zhang, G., & Wang, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01362 · Cited by 2
“Specifically, for a given image input I , it went through an H (combination of vision encoder and connector [43]) to get the latent embeddings v : v = H ( I ) , which are concatenated with the question embeddings q : x = ( v , q ) , and are fed into a large language model π θ ( · ) that…”
Xie, J., Gao, Y., You, L. H., Xu, X. H., Xu, H., Kou, Z., Fu, K., Qu, Y., Yang, W., Guo, J., et al. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755320 · Cited by 2
Weakly Supervised Content Selection for Improved Image Captioning
Chandu, K. R., Sharma, P. K., Changpinyo, S., Thapliyal, A., & Soricut, R. (2020) · Cited by 2
Unsupervised Opinion Summarization Using Approximate Geodesics
Chowdhury, S. B. R., Monath, N., Dubey, K. A., Ahmed, A. A., & Chaturvedi, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.8 · Cited by 2
Natural Language Generation in Dialogue Systems for Customer Care
Lascio, M. F. D., Sanguinetti, M., Luca Anselma, A., Mana, D., Mazzei, A., Patti, V., & Simeoni, R. (2020) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.8450 · Cited by 2
“The well-known problem of hallucinations in neural networks deters their use in real-world NLG (Rohrbach et al., 2018).”
BackgroundVideoHallu: Evaluating and Mitigating Multi-modal Hallucinations for Synthetic VideosPreprint
Li, Z., Wu, X., Qin, Y., Shi, G., Du, H., Manocha, D., Zhou, T., & Boyd-Graber, J. L. (2025) · Qeios · doi:10.32388/bxc6x1 · Cited by 2
Albert Gatt, A. (2025) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199384655.013.896 · Cited by 2
Vision-and-Dialog Navigation by Fusing Cross-modal features
Nie, H., Dong, M., & Bi, S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533776 · Cited by 1
“For example, neural captioning models are found grounding object mentions into incorrect objects and can hallucinating non-existing objects [32].”
BackgroundTrust but Verify: Programmatic VLM Evaluation in the Wild
Prabhu, V., Purushwalkam, S., Yan, A., Xiong, C., & Xu, R. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00312 · Cited by 1
“While the last category better reflects real-world usage, it faces two critical challenges: generating natural questions (right example: GAVIE’s [26] unnatural coordinate-based questions) and reliable evaluation (second from right: GPT-4 incorrectly penalizing a response with additional correct…”
Li, X., Lin, L., Wang, S., & Wu, H. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730081 · Cited by 3
“Visual Object hallucinations refer to MLLMs generating plausible outputs that include visual object information not present in the image [5, 32, 46], which impact the accuracy of information retrieved and extracted from multimodal data sources [8, 59].”
Stop learning it all to mitigate visual hallucination, Focus on the hallucination target
Yoon, D., Song, Y., & Park, W. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00397 · Cited by 1
Mitigating Hallucinations in Large Vision-Language Models via Visual-Enhanced Contrastive Decoding
Qiang, P., Tan, H., Zhang, H., Li, X., Li, R., & Liang, J. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3651099 · Cited by 1
“CHAIR: Caption Hallucination Assessment with Image Rel-evance (CHAIR) [53] is a widely adopted metric for evaluating object hallucinations in image captioning tasks.”
Fuzzy Contrastive Decoding to Alleviate Object Hallucination in Large Vision-Language Models
Kim, J., Kim, J., Kim, Y., & 조성배 (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01913 · Cited by 1
“To address the mitigation of hallucination in LLMs, extensive research has focused on hallucination detection [5, 16, 23, 26, 37, 39, 40].”
Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension
Wang, X., Yang, Z., Li, L., Lu, H., Xu, Y., Lin, C.-C., Lin, K., Huang, F., & Wang, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00117 · Cited by 1
“For hallucination evaluation, we benchmark on CHAIR [38] and MMHal [43].”
ChartCap: Mitigating Hallucination of Dense Chart Captioning
Lim, J., Ahn, J., & Kim, G. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01224 · Cited by 5
“Hallucination in VLMs refers to the instances where the model generates text that does not align with the visual content [51].”
Zhou, Y., Yang, M., Zhang, J., Li, J., & Feng, S. L. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104869 · Cited by 1
Li, S., Wang, F. H., Qin, S., Duan, R., Cheng, H., & Ye, L. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3796715 · Cited by 1
“To evaluate the contribution of each component to the overall effectiveness of our method, we conduct comprehensive ablation studies on Object HalBench [27] and AMBER [35].”
Inaccuracies, omissions, and bullshit: 2024 U.S. elections through the eyes of ChatGPT
Jang, H., McGregor, S. C., & Neill, L. (2026) · Information Communication & Society · doi:10.1080/1369118x.2026.2685121 · Cited by 1
Mitigating hallucination in Multimodal Large Language Models via cross-layer visual anchors
yang, C., Cai, S., Yuan, J., & Hu, C. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114380 · Cited by 1
Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucination
Li, X., Juvekar, A. S., Zhang, J., Liu, X., Wahed, M., Nguyen, K. A., Shen, Y., Yu, T., & Lourentzou, I. (2026) · doi:10.1109/cvprf71909.2026.00715 · Cited by 2
“…can manifest in several ways, such as pixel-grounding hallucinations [22, 49, 50], where a model may generate spatially plausible but semantically incorrect masks, or object-level hallucinations [9, 10, 19, 37], when a model incorrectly mentions or labels objects that do not exist in a scene.”
LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation
Zhou, Y., Zhao, S., Chen, Y., Wang, Z., Jin, C., Zhao, M., & Metaxas, D. N. (2026) · doi:10.1109/cvprf71909.2026.00923 · Cited by 1
“Specifically, as shown in Figure 1(a), existing approaches typically generate new data with specific goals, such as generating negative samples to address hallucination issues [29, 36, 37, 41, 46] or generating more detailed descriptions for spatial and logical relationships [15], which in turn…”
Long-hao, W., Ding-jie, L., & Li, X. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.08.012 · Cited by 1
Hallucination Detection for Grounded Instruction Generation
Zhao, L., Nguyen, K., & Daumé, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.266 · Cited by 2
Automated Comment Generation Based on the Large Language Model
Cai, K., Zhou, J., Kong, L., Liang, D., & Li, X. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0730-0_25 · Cited by 1
Who Are All The Stochastic Parrots Imitating? They Should Tell Us!
Shaier, S., Hunter, L., & Kann, K. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.13 · Cited by 2
Feinglass, J., & Yang, Y. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00434 · Cited by 1
“This shift represents a dangerous trend in VL pipelines, which are known to be sensitive to language [34] and dataset [13] priors.”
BackgroundObject Hallucination Detection in Large Vision Language Models via Evidential Conflict
Liu, Z., Huang, T., Wang, R., & Jing, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-67977-3_7 · Cited by 1
TRINS: Towards Multimodal Language Models that Can Read
Zhang, R., Zhang, Y., Chen, J., Zhou, Y., Gu, J., Chen, C., & Sun, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02131 · Cited by 1
“Additionally, captioning models may be susceptible to hallucinations [42], a concern mitigated in TRINS, which provides more comprehensive and reliable visual descriptions.”
BackgroundZero-Shot Image Caption Inference System Based on Pretrained Models
Zhang, X., Shen, J., Wang, Y., Xiao, J., & Li, J. (2024) · Electronics · doi:10.3390/electronics13193854 · Cited by 1
Detailed Image Captioning and Hashtag Generation
Shetty, N., & Li, Y. (2024) · Future Internet · doi:10.3390/fi16120444 · Cited by 1
OPCap:Object-aware Prompting Captioning
Huang, F., Cao, Y., & Zhong, J. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770956 · Cited by 1
“• Hallucination: CHAIRs (sentence-level) and CHAIRi (object-level) [20], quantifying the frequency of irrelevant or incorrect object mentions.”
Unsupervised Opinion Summarization Using Approximate Geodesics
Chowdhury, S. B. R., Monath, N., Dubey, K. A., Ahmed, A. A., & Chaturvedi, S. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.11 · Cited by 1
Grounded Video DescriptionPreprint
Zhou, L., Kalantidis, Y., Chen, X., Corso, J. J., & Rohrbach, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06587 · Cited by 1
“While researchers have started to discover and study these problems for image description [14, 9, 24, 20],1 they are even more pronounced”
BackgroundInfluentialLearning Compact Reward for Image CaptioningPreprint
Li, N., & Chen, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10925 · Cited by 1
“, 2016), CHAIRs and CHAIRi (Rohrbach et al., 2018).”
InfluentialDenoising Large-Scale Image Captioning from Alt-text Data using Content Selection ModelsPreprint
Chandu, K. R., Sharma, P. K., Changpinyo, S., Thapliyal, A., & Soricut, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05175 · Cited by 3
“2017) in both stages of the image captioning model, exploring various design choices while maintaining simplicity; and 3) measuring the progress via the most reliable metrics possible through human evaluation (Rohrbach et al. 2018).”
BackgroundInfluentialCounterfactual Maximum Likelihood Estimation for Training Deep NetworksPreprint
Wang, X., Chen, W., Saxon, M., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03831 · Cited by 1
“However, image captioning models also suffer severely from spurious correlations [65] that lead to hallucinations.”
BackgroundMethodNeural Abstructions: Abstractions that Support Construction for Grounded Language LearningPreprint
Burns, K., Manning, C. D., & Fei-Fei, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09285 · Cited by 2
“Significant ablations don’t result in significant performance drops [16, 34], vision is “ignored” in favor of language cues that are well correlated with prediction [51, 63], and models overfit to spurious correlations [13] that undermine generalization [1].”
Self-citationGoal-driven text descriptions for imagesPreprint
Luo, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12575 · Cited by 4
Wang, H., & Li, Z. (2026) · arXiv · doi:10.48550/arxiv.2605.20469
“In the general domain, CHAIR (Caption Hallucination Assessment with Image Relevance) [11] checks whether cap-tione d objects appear in ground truth.”
REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image CaptioningPreprint
Jiang, M., Hu, J., Huang, Q., Zhang, L., Diesner, J., & Gao, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02217
Iterative Edit-Based Unsupervised Sentence SimplificationPreprint
Kumar, D., Mou, L., Golab, L., & Vechtomova, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09639 · Cited by 5
“We did this because neural models are known to hallucinate information (Rohrbach et al., 2018).”
BackgroundEnhanced Modality Transition for Image CaptioningPreprint
Wang, Z., Luo, Y., & Huang, Z. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11526
“The details of each component in the framework are explained in this section.”
BackgroundAGGGEN: Ordering and Aggregating while GeneratingPreprint
Xu, X., Dušek, O., Rieser, V., & Konstas, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05580
“, 2020) as well as other natural language generation (NLG) tasks (Cao et al., 2018; Rohrbach et al., 2018).”
BackgroundTowards controllable neural generation of arguments
Hua, X. (2021) · Digital Repository Service (Northeastern University) · doi:10.17760/d20412319
Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning ModelsPreprint
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01037
“[27] points out that hallucinating less does not necessarily render higher sentence-level evaluation metrics [29], [31], [99], which is also in line with our observations in Table IV.”
BackgroundResultIncorporating additional knowledge into image captioners
Yang, X. (2021) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/151726
On factuality in neural language models
Nadeem, M. (2021)
Accessing Diverse Web Knowledge with Natural Language Interface
chen, W. (2021) · eScholarship (California Digital Library)
Towards robust inference against distribution shifts in computer vision
Tang, K. (2021) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/154119
A geometrical deep learning model for the lexicalisation of “unseen” RDF triples
Cremaschi, M., Saleri, S., & Maurino, A. (2021) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys53884.2021.00334
Image Captioning with External Knowledge
Nikiforova, S. (2023) · Utrecht University Repository (Utrecht University) · doi:10.33540/1709
Automated Image Captioning for the Visually Impaired
Bode, N., & Sukhai, M. A. (2023) · Encyclopedia of Computer Graphics and Games · doi:10.1007/978-3-319-08234-9_501-1
More Than Spoken Words: Nonverbal Message Extraction and Generation
Yu, D., Wang, X., Chen, W., Du, N., Wang, L., Mi, H., & Yu, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1021
Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions
Noble, B., & Ilinykh, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.580
Automated Image Captioning for the Visually Impaired
Bode, N., & Sukhai, M. A. (2024) · Encyclopedia of Computer Graphics and Games · doi:10.1007/978-3-031-23161-2_501
RICOA: Rich Captioning with Object Attributes
Şahin, E. M., & Akar, G. B. (2024) · International Conference on Consumer Electronics · doi:10.1109/icce59016.2024.10444202
“…evaluation of these state-of-the-art methods results in high scores, it is shown that image captioning models have a tendency to replicate phrases from the training dataset, to mention imaginary objects (hallucination) or to fail to recognize significant details in the image [1], [2], [9], [10].”
BackgroundMmaa: An Attribute-Aware Collaborative Learning Network for Radiology Report GenerationPreprint
guo, Y., Li, S., Song, W., Hao, A., & Qin, H. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862971
Unbiased Image Caption Generation Based on Dynamic Counterfactual Inference
Zhou, J., Yang, C. Z., Zhu, Y., & Zhang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-3623-2_11
Deconfounded Fashion Image Caption with Transformer and RetrievalPreprint
Peng, T., Yin, W., Liu, J., Li, L., & Hu, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4871168
Metadata
- Authors
- Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K.
- Affiliations
- Max Planck Society (DE)
- University of California, Berkeley (US)
- Boston University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 4035-4045
- OpenAlex
- W2892087592
- Cited by
- 420
- References
- 19
- Citations by year
- 2026: 103 · 2025: 80 · 2024: 70 · 2023: 42 · 2022: 32 · 2021: 45 · 2020: 32 · 2019: 15 · 2018: 1
- FWCI
- 7.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- object hallucination, image captioning, hallucination evaluation, language priors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
