The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Object Hallucination in Image Captioning
Open accessAbstract
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene.One problem is that standard metrics only measure similarity to ground truth captions and may not fully capture image relevance.In this work, we propose a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination.We analyze how captioning model architectures and learning objectives contribute to object hallucination, explore when hallucination is likely due to image misclassification or language priors, and assess how well current sentence metrics capture object hallucination.We investigate these questions on the standard image captioning benchmark, MSCOCO, using a diverse set of models.Our analysis yields several interesting findings, including that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors. * Denotes equal contribution.NBT: A woman talking on a cell phone while sitting on a bench.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, it is also apparent that deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [222]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language model (LLM) systems, such as ChatGPT1 or Gemini2, can show impressive reasoning and question-answering capabilities but often ‘hallucinate’ false outputs and unsubstantiated answers3,4.”
From Detecting hallucinations in large language models using semantic entropy (2024) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This is because there is another contributor of hallucinations - training and modeling choices of neural models [121, 215, 222, 263].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [222]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“All the baseline models employ features extracted from the fourth layer of ResNet-101 (He et al., 2016), except for the GAN model which employs ResNet-152.”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 9×
“We also consider LRCN (Donahue et al., 2015) which considers visual information at each time step, as opposed to just initializing the LSTM hidden state with extracted features.”
BackgroundMethodInfluentialMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“For attention based models, we consider Att2In (Rennie et al., 2017), which is similar to the original attention based model proposed by (Xu et al., 2015), except the image feature is only input into the cell gate as this was shown to lead to better performance.”
MethodResultLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“We also consider LRCN (Donahue et al., 2015) which considers visual information at each time step, as opposed to just initializing the LSTM hidden state with extracted features.”
MethodSelf-citationVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 2×
“his is perhaps because bounding box proposals come from the MSCOCO detection task and are thus “in-domain” as opposed to the TopDown-BB model which relies on proposals learned from the Visual Genome (Krishna et al., 2017) dataset. Second, frequently training models with the self-critical loss actually increases the am…”
MethodCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 6×
“Likewise, our work is related to other work which aims to build better evaluation tools ((Vedantam et al., 2015), (Anderson et al., 2016), (Cui et al., 2018)).”
BackgroundMethodInfluential(2005) · Cited by 4,152
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,231 · cited 5×
“For non-attention based methods we consider the FC model from Rennie et al. (2017) which incorporates visual information by initializing the LSTM hidden state with high level image features.”
MethodInfluentialSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 3×
“Likewise, our work is related to other work which aims to build better evaluation tools ((Vedantam et al., 2015), (Anderson et al., 2016), (Cui et al., 2018)).”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 3
- Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137.
- Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2016. Re-evaluating automatic metrics for image captioning. In European Chapter of the Association for Computational Linguistics.
- Anna Rohrbach, Makarand Tapaswi, Atousa Torabi, Tegan Maharaj, Marcus Rohrbach, Sanja Fi- dler Christopher Pal, and Bernt Schiele. 2017a. The Joint Video and Language Understand- ing Workshop: MovieQA and The Large Scale Movie Description Challenge (LSMDC). https://sites.google.com/site/ describingmovies/lsmdc-2017.
Cited by
Sorted by citations
Items 101–200 of 427
Image captioning based on scene graphs: A survey
Jia, J., Ding, X., Pang, S., Gao, X., Xin, X., Hu, R., & Nie, J. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120698 · Cited by 20
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
Peng, H. M., Wang, X., Hu, S., Jin, H., Hou, L., Li, J., Liu, Z., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.335 · Cited by 34
DIME-FM : DIstilling Multimodal and Efficient Foundation Models
Sun, X., Zhang, P., Zhang, P., Shah, H., Saenko, K., & Xia, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01423 · Cited by 24
“Visual Question Answering (VQA) [4, 81, 34], Image Captioning [2, 44, 59, 27] etc .”
BackgroundSelf-citationKaul, P., Li, Z., Yang, H., Dukler, Y., Swaminathan, A. D., Taylor, C. J., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02571 · Cited by 37
“There are two major established works which specifically develop a benchmark for evaluating hallucinations in vision-language models: POPE [26] and CHAIR [40], which we discuss in detail in Sec.”
InfluentialREO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning
Jiang, M., Hu, J., Huang, Q., Zhang, L., Diesner, J., & Gao, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1156 · Cited by 17
Fact-based Content Weighting for Evaluating Abstractive Summarisation
Xu, X., Dušek, O., Li, J., Rieser, V., & Konstas, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.455 · Cited by 18
“…an open challenge (Schluter, 2017; Kryściński et al., 2019): First, decoders are amenable to pathogeniessuch as hallucination and/or omission of important information, which are hard to capture using existing evaluation metrics (Cao et al., 2018; Rohrbach et al., 2018; Dušek et al., 2020).”
BackgroundMethodReasoning like Humans: On Dynamic Attention Prior in Image Captioning
Wang, Y., Sun, X., Li, X., Zhang, W., & Gao, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107313 · Cited by 19
On Guiding Visual Attention with Language Specification
Petryk, S., Dunlap, L., Nasseri, K., Gonzalez, J. E., Darrell, T., & Rohrbach, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01756 · Cited by 23
“However, proper grounding within vision-and-language models is a research question on its own [23, 34].”
BackgroundSelf-citationJamil, A., Rehman, S. U., Mahmood, K., Villar, M. G., Prola, T. A., Díez, I. D. L. T., Samad, M. A., & Ashraf, I. (2024) · IEEE Access · doi:10.1109/access.2024.3365528 · Cited by 23
RetractedAugmented Human and Human-Machine Co-evolution: Efficiency and Ethics
Dengel, A. R., Devillers, L. Y., & Schaal, L. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69128-8_13 · Cited by 18
CAT+: Investigating and Enhancing Audio-Visual Understanding in Large Language Models
Ye, Q., Yu, Z., Shao, R., Cui, Y., Kang, X., Liu, X., Torr, P. H. S., & Cao, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3582389 · Cited by 21
IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding
Zhu, L., Ji, D., Chen, T., Xu, P., Ye, J., & Liu, J. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00150 · Cited by 96
“Different from θ , which theoretically balances the use of both image v and text { t, y i } for prediction, ˆ θ exhibits a higher dependency and places greater emphasis on the information contained within the image v .”
Bottom-up and Top-down Object Inference Networks for Image Captioning
Pan, Y., Li, Y., Yao, T., & Mei, T. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3580366 · Cited by 16
“To better understand the impact of top-down signals in our BTO-Net, we conduct hallucination analysis [45] to assess the rate of object hallucination (i.”
MethodOn Hallucination and Predictive Uncertainty in Conditional Language GenerationPreprint
Xiao, Y., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15025 · Cited by 16
“A smaller value of CHAIRi indicates the model is less likely to generate hallucinated objects, and a higher CIDEr indicates better caption quality.”
InfluentialLanguage Models As or For Knowledge Bases
Razniewski, S., Yates, A., Kassner, N., & Weikum, G. (2021) · arXiv · doi:10.48550/arxiv.2110.04888 · Cited by 16
“This is a case of LM hallucination [25, 26].”
BackgroundLLM-CDM: A Large Language Model Enhanced Cognitive Diagnosis for Intelligent Education
Chen, X., Zhang, J., Zhou, T., & Zhang, F. (2025) · IEEE Access · doi:10.1109/access.2025.3549309 · Cited by 17
“Although LLMs have broad application prospects in education, they also face the ‘‘hallucination’’ problem [20], [21], [22], [23], [24], in which the model may generate information that sounds reasonable but is inaccurate or fabricated.”
Knowledge-Augmented Language Model Verification
Baek, J., Jeong, S., Kang, M., Park, J. C., & Hwang, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.107 · Cited by 33
Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization
Chrysostomou, G., Zhao, Z., Williams, M., & Αλέτρας, Ν. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00695 · Cited by 24
“…work observed that models tend to generate hallucinatory content that is not based on or cannot be entailed from the source document (Vinyals and Le, 2015; Rohrbach et al., 2018; Cao et al., 2018; Maynez et al., 2020; Raunak et al., 2021; Falke et al., 2019; Maynez et al., 2020; Chen et al., 2022).”
BackgroundFine-tuning Language Models for Factuality
Tian, K., Mitchell, E., Yao, H., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2311.08401 · Cited by 14
Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding
Tang, F., Liu, C., Xu, Z., Hu, M., Huang, Z., Xue, H., Chen, Z., Peng, Z., Yang, Z., Zhou, S., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02435 · Cited by 19
“Effect of Positional Awareness Encoding.”
Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., & He, C. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209377 · Cited by 14
“Lastly, in scrutinizing current benchmarks for multimodal hallucination evaluation, we’ve identified significant shortcomings in existing assessment systems [5, 13, 14].”
GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?
Taesiri, M. R., Feng, T., Bezemer, C., & Nguyen, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02118 · Cited by 15
“Prevalent hallucination in open-source models: Hallu-cination typically refers to situations in which the model’s generated text contains information not present in the image [16, 58, 81, 85].”
BackgroundStrengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
Pi, R., Han, T., Xiong, W. H., Zhang, J., Liu, R., Pan, R., & Zhang, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73414-4_22 · Cited by 43
Explicit Image Caption Editing
Wang, Z., Chen, L., Ma, W., Han, G., Niu, Y., Shao, J., & Xiao, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_7 · Cited by 14
Penguins Don’t Fly: Reasoning about Generics through Instantiations and Exceptions
Allaway, E., Hwang, J. D., Bhagavatula, C., McKeown, K. R., Downey, D., & Choi, Y. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.192 · Cited by 26
Transformer-Empowered Invariant Grounding for Video Question Answering
Li, Y., Wang, X., Xiao, J., Ji, W., & Chua, T. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3303451 · Cited by 17
KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection
Choi, S.-H., Fang, T., Wang, Z., & Song, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.867 · Cited by 46
The Pitfalls of Defining Hallucination
Deemter, K. V. (2024) · Computational Linguistics · doi:10.1162/coli_a_00509 · Cited by 13
“…taken for granted; with the ascent of neural architectures, researchers in various areas of NLG have emphasized that this is often not justified (Vinyals and Le 2015; Koehn and Knowles 2017; Rohrbach et al. 2018; Maynez et al. 2020; Duˇsek and Kasner 2020; Raunak, Menezes, and Junczys-Dowmunt 2021).”
BackgroundPiao, Y., Chen, H., Wu, S., Li, X., Li, Z., & Yang, D. (2024) · Digital Health · doi:10.1177/20552076241284771 · Cited by 13
“32,33 This is a phenomenon that is also common in the English environment.”
BackgroundCrossing the Format Boundary of Text and Boxes: Towards Unified Vision-Language ModelingPreprint
Yang, Z., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, Z., Lu, Y., & Wang, L. (2021) · arXiv (Cornell University) · Cited by 13
CLIPScore: A Reference-free Evaluation Metric for Image CaptioningPreprint
Hessel, J., Holtzman, A., Forbes, M., Bras, R. J. L., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08718 · Cited by 1,051
“Prior work has demonstrated that, for many literal description tasks, humans often prefer correctness in captions over specificity (Rohrbach et al., 2018, 2017).”
BackgroundProceedings of the 4th New Frontiers in Summarization Workshop
(2023) · doi:10.18653/v1/2023.newsum-1 · Cited by 12
Evaluating Performance of ChatGPT on MKSAP Cardiology Board Review Questions
Milutinovic, S., Petrovic, M. T., Begosh-Mayne, D., Lopez‐Mattei, J., Chazal, R. A., Wood, M. J., & Escarcega, R. O. (2024) · International Journal of Cardiology · doi:10.1016/j.ijcard.2024.132576 · Cited by 11
Improving image captioning descriptiveness by ranking and LLM-based fusion
Celona, L., Bianco, S., Donzella, M., & Napoletano, P. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11672-x · Cited by 41
“This could be achieved by enhancing the caption ranking policy, moving beyond solely selecting captions that closely align with the image, and instead prioritizing captions that are diverse from one another.”
Building a Persuasive Virtual Dietitian
Luca Anselma, A., & Mazzei, A. (2020) · Informatics · doi:10.3390/informatics7030027 · Cited by 10
“We use a commonly adopted architecture for NLG in data-to-text systems, especially in application contexts (the well-known problem of hallucinations in neural networks deter their use in real-world NLG [37], especially in the domain of health and wellness), which is a pipeline composed of four distinct modules: data a…”
MethodGenerating Intelligible Plumitifs Descriptions: Use Case Application with Ethical Considerations
Beauchemin, D., Garneau, N., Gaumond, E., Déziel, P.-L., Khoury, R., & Lamontagne, L. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.3 · Cited by 11
Generative adversarial network for Table-to-Text generation
Zhao, J., Zhan, Z., Li, T., Li, R., Hu, C., Wang, S., & Zhang, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.036 · Cited by 11
Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles
Ye, S., Xie, Y., Chen, D., Yi‐chong, X., Yuan, L., Zhu, C., & Liao, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00259 · Cited by 16
“Our image splitting for COCO follows a popular split [3,12,20,33,54,70,73,74,80].”
BackgroundRelational Reasoning using Prior Knowledge for Visual CaptioningPreprint
Hou, J., Wu, X., Qi, Y., Zhao, W., Luo, J., & Jia, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01290 · Cited by 11
“…from these methods that directly adds explicit high-level semantic concepts from external knowledge, our method use external knowledge to reason relationships between semantic concepts via joint commonsense and relation reasoning, without facing the “hallucinating” problem as stated in [25].”
A comprehensive survey on deep learning approaches for image captioning: a systematic review
Al-Malla, M. A., Hamdoun, O., & Ghneim, N. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01377-w · Cited by 11
Hyperbolic Learning with Synthetic Captions for Open-World Detection
Kong, F., Chen, Y., Cai, J., & Modolo, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01586 · Cited by 11
“However, synthetic cap-tions suffer from the hallucination problem [44], where concepts unrelated to the image are added to the generated caption; this simple contrastive loss is not equipped to handle them and its learning suffers from the generated noise.”
BackgroundSQ-LLaVA: Self-Questioning for Large Vision-Language Assistant
Sun, G., Qin, C., Wang, J., Chen, Z., Xu, R., & Tao, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72673-6_9 · Cited by 13
XAINES: Explaining AI with Narratives
Hartmann, M., Du, H., Feldhus, N., Kruijff‐Korbayová, I., & Sonntag, D. (2022) · Künstliche Intell. · doi:10.1007/s13218-022-00780-8 · Cited by 13
“However, Rohrbach et al. [81] raised awareness for the phenomenon of object hallucinations , i.e. the description of objects that are not actually visible in an image.”
BackgroundAggGen: Ordering and Aggregating while Generating
Xu, X., Dušek, O., Rieser, V., & Konstas, I. (2021) · doi:10.18653/v1/2021.acl-long.113 · Cited by 17
“, 2020) as well as other natural language generation (NLG) tasks (Cao et al., 2018; Rohrbach et al., 2018).”
BackgroundNullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection
Yang, L., Zheng, Z., Chen, B., Zhao, Z., Lin, C., & Shen, C. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.01364 · Cited by 9
“Sub-sequently, the words are mapped to MSCOCO objects using the synonym and double-word lists provided in [36].”
InfluentialAttention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs
Tu, C., Ye, P. D., Zhou, D., Bai, L., Yu, G., Chen, T., & Ouyang, W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02607-z · Cited by 24
“Benchmarks like CHAIR (Rohrbach et al., 2018), MMHal-Bench (Sun et al., 2023), and AMBER (Wang et al., 2023a) assess hallucinations by comparing the generated content to the visual inputs and consider metrics like recall and precision to further measure the overall performance.”
InfluentialKaffee, L.-A., Vougiouklis, P., & Simperl, E. P. B. (2021) · Semantic Web · doi:10.3233/sw-210431 · Cited by 12
“Other problems of neural language generation, such as factually wrong sentences or hallucinations [74], were excluded from the study as”
BackgroundVision to Language: Methods, Metrics and Datasets
Sharif, N., Nadeem, U., Shah, S. A. A., Bennamoun, M., & Liu, W. (2020) · Learning and Analytics in Intelligent Systems · doi:10.1007/978-3-030-49724-8_2 · Cited by 9
Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning
Yang, X., Zhang, H., Gao, C., & Cai, J. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01692-8 · Cited by 9
“(2) We exploited CHAIRs and CHAIRi (Rohrbach et al., 2018) to quantitatively measure the bias degree of the generated captions to some objects.”
BackgroundMethodInfluentialHaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
Wang, Z., Bingham, G., Yu, A. W., Le, Q. V., Luong, M.-T., & Ghiasi, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72980-5_17 · Cited by 20
Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key
Yang, Z.-H., Luo, X., Han, D.-Q., Xu, Y.-J., & Li, D.-S. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.00992 · Cited by 93
“We focuses on mitigating hallucinations in LVLMs, with experiments conducted on four HalBench [31]: A widely used benchmark for assessing object hallucination.”
Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models
Yoshikawa, H., & Okazaki, N. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.150 · Cited by 12
“This is because large pre-trained language models are known to fluently generate “facts” that they have never seen (Cao et al., 2018; Rohrbach et al., 2018; Müller et al., 2020).”
BackgroundFood-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
Ma, Z., Pan, M., Wu, W., Cheng, K., Zhang, J., Huang, S., & Chen, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611994 · Cited by 16
“Moreover, we find that OFA tends to generate hallucinations [4, 36], and GIT prefers to append meaningless words to the end of the sentence.”
BackgroundPositive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training
Sarto, S., Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02535-y · Cited by 18
Cunha, R., Ferreira, T. C., Pagano, A. S., & Alves, F. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.205 · Cited by 11
“Hallucination and repetition often present on neural models (Rohrbach et al., 2018; Moryossef et al., 2019; Holtzman et al., 2018) can also account for OnlyNames good performance which does not suffer from this problem.”
BackgroundUnderstanding Image Captioning Models beyond Visualizing AttentionPreprint
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2020) · arXiv (Cornell University) · Cited by 8
Allaway, E., Bhagavatula, C., Hwang, J. D., McKeown, K. R., & Leslie, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00530 · Cited by 9
Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding
Suo, W., Zhang, L., Sun, M., Wu, L. Y., Wang, P., & Zhang, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02783 · Cited by 20
“For generative task, we establish experiments on the three widely used datasets: AMBER [48], Object-HalBench [36] with the language prompt “Please describe this image in detail”, as well as MMHalbench [40] with the original instructions as the prompts.”
InfluentialDu, J., Wu, J., Kong, D., Liang, W., Hao, F., Xu, J., Wang, B., Wang, G., & Li, P. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3599101 · Cited by 7
“To remedy the hallucination [27], [28], [29] (LVLMs produce incorrect statements to the input images and prompts), compared to getting “hard” prompts (“hard” means prompts from LVLMs are directly input to SAM without any interaction with images), several methods concentrate on deriving “soft”…”
A Survey of Multimodal Hallucination Evaluation and Detection
Chen, Z., Min, Y., Zhang, J., Yan, B., Wang, J., Wang, X., & Shan, S. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02756-9 · Cited by 31
“Compared to discriminative tasks, generative tasks typically elicit longer and open-ended responses, posing challenges for traditional accuracy-based metrics and necessitating dedicated hallucination detection methods.”
InfluentialThinking Hallucination for Video Captioning
Ullah, N., & Mohanta, P. P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26316-3_37 · Cited by 7
Simple Token-Level Confidence Improves Caption Correctness
Petryk, S., Whitehead, S., Gonzalez, J. E., Darrell, T., Rohrbach, A., & Rohrbach, M. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00564 · Cited by 8
“This effect was also observed in prior works [12, 45], which describe how standard metrics can often fail to penalize hallucinations.”
BackgroundMethodResultInfluentialSelf-citationGENEVIC: GENetic data Exploration and Visualization via Intelligent interactive Console
Nath, A., Mwesigwa, S., Dai, Y., Jiang, X., & Zhao, Z. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae500 · Cited by 7
“In addition, these AI models, while capable of generating seemingly accurate outputs, can sometimes produce fictitious information, known as the “hallucination effect” (Rohrbach et al. 2018, Xiao and Wang 2021), complicating their reliability in specialized fields.”
BackgroundMulti-Step Critiquing User Interface for Recommender Systems
Petrescu, D., Antognini, D., & Faltings, B. V. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3478886 · Cited by 8
“In addition, as is often the case in natural language applications, hallucinations can occur [9, 19].”
BackgroundLooking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy
Testoni, A., & Raffaella Bernardi, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.736 · Cited by 7
“Rohrbach et al. (2018) propose another variant of the CHAIR metric called CHAIR-i (per-instance), defined as the number of hallucinated objects in a sequence divided by the”
BackgroundMethodInfluential‘Beach’ to ‘Bitch’: Inadvertent Unsafe Transcription of Kids’ Content on YouTube
Ramesh, K., KhudaBukhsh, A. R., & Kumar, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21470 · Cited by 8
Keyword Localisation in Untranscribed Speech Using Visually Grounded Speech Models
Olaleye, K., Oneaţă, D., & Kamper, H. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3180220 · Cited by 7
Neural Methods for Data-to-text Generation
Sharma, M., Gogineni, A. K., & Ramakrishnan, N. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3660639 · Cited by 23
“Often, the divergence present in benchmark training datasets, wherein the narrative may contain data absent from the source or not cover the entirety of the data instance, is the culprit behind hallucination tendencies in the model [280].”
InfluentialLearning to Rank Visual Stories From Human Ranking Data
Hsu, C.-Y., Chu, Y.-W., Chen, V., Lo, K.-C., Chen, C., Huang, T.-H., & Ku, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.441 · Cited by 9
Evaluating the Performance of Large Language Models on a Neurology Board-Style ExaminationPreprint
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · medRxiv · doi:10.1101/2023.07.13.23292598 · Cited by 7
“This trait is a recognized limitation of LLMs 29,30 and originates from the training objective of these models, which is to predict the most likely sequence of words following an input.”
BackgroundMiniCPM-V LLaMA Model for Image Recognition: A Case Study on Satellite Datasets
Kömürcü, K., & Petkevičius, L. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3547144 · Cited by 10
Visual hallucination detection in large vision-language models via evidential conflict
Huang, T., Liu, Z., Wang, R., Zhang, Y., & Jing, L. (2025) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2025.109507 · Cited by 10
Mimic In-Context Learning for Multimodal Tasks
Jiang, Y.-C., Fu, J., Hao, C.-D., Hu, X., Peng, Y., Geng, X., & Yang, X. (2025) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52734.2025.02776 · Cited by 21
“ICL performance is highly dependent on the number of demonstrations; when demonstrations are insufficient, the LMMs may misalign query representations.”
LENS: Layers of Evaluation of Hallucination in GenAI Systems
Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189150 · Cited by 32
See or Guess: Counterfactually Regularized Image Captioning
Cao, Q., Chen, X., Song, R., Wang, X., Huang, X., & Ren, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681458 · Cited by 6
Informative Attention Supervision for Grounded Video Description
Wan, B., Jiang, W., & Fang, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746751 · Cited by 6
Models See Hallucinations: Evaluating the Factuality in Video Captioning
Liu, H., & Wan, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.723 · Cited by 6
Chae, J., & Kim, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892787 · Cited by 7
“The proportion of hallucinated objects of generated captions is calculated according to a synonym criteria of [41].”
MethodRebuffel, C., Soulier, L., Scoutheeten, G., & Gallinari, P. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.18 · Cited by 7
Agarwal, P., Betancourt, A., Panagiotou, V., & Díaz-Rodríguez, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11743 · Cited by 6
“We show on pre-trained models that, despite being able to successfully generate grammatically correct sentences, their captions are often misaligned with the objects present in the scene or hallucinate objects (Rohrbach et al., 2018).”
Berger, U., Stanovsky, G., Abend, O., & Frermann, L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.52 · Cited by 13
Tang, Y., Fang, Y., Wang, T., Sun, L., & Chen, L. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747597 · Cited by 7
“…an imbalance between the visual and linguistic modalities within LVLMs [2, 37], as the former manifests as LVLMs using prior textual knowledge to "infer" image content without understanding [71], while the latter manifests as LVLMs overly focusing on images without fully considering text queries.”
Yang, L., He, L., Hu, D., Liu, Y., Peng, Y., Chen, H., & Zhou, M. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3440872 · Cited by 5
“To further assess the quality of the generated diverse captions, we employ the widely used CHAIR metric [33] to determine whether our VaT model produces diverse and well-grounded captions or exhibits a higher degree of “hallucinations.””
InfluentialVisually Dehallucinative Instruction Generation
Cha, S., Lee, J., Lee, Y., & Yang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446658 · Cited by 5
“, proposed for sentence-level hallucination evaluation [9, 17].”
MethodQuestion-Answering Dense Video Events
Qin, H., Xiao, J., & Yao, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729945 · Cited by 11
“However, optimizing for holistic sentence generation often results in over-fitting (Chen, Li, and Hu 2020) and object hallucination (Rohrbach et al. 2018).”
“I’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in GuessWhat?!
Testoni, A., & Bernardi, R. (2021) · doi:10.18653/v1/2021.acl-srw.11 · Cited by 5
“The fine-grained visual input representation of these two models leads to a consistent reduction in hallucinations, confirming that a strong visual processing is critical for avoiding hallucination (Rohrbach et al., 2018).”
BackgroundMethodResultInfluentialFerguson, W., Batra, D., Mooney, R. J., Parikh, D., Torralba, A. B., Bau, D., Diller, D., Fasching, J., Fiotto-Kaufman, J., Goyal, Y., et al. (2021) · Applied AI Letters · doi:10.1002/ail2.60 · Cited by 6
Multilingual Sequence-to-Sequence Models for Hebrew NLP
Eyal, M., Noga, H., Aharoni, R., Szpektor, I., & Tsarfaty, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.487 · Cited by 6
Deep Generative Model of Individual Variability in fMRI Images of Psychiatric Patients
Matsubara, T., Kusano, K., Tashiro, T., Ukai, K., & Uehara, K. (2020) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2020.3008707 · Cited by 6
“Our motivation comes from recent machine learning studies demonstrating that data-driven methods including deep learning tend to depend on spuriously correlated information, and ignore true causality [30], [31].”
BackgroundExploring Uncertainty Measures for Image-caption Embedding-and-retrieval Task
Hama, K., Matsubara, T., Uehara, K., & Cai, J. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3425663 · Cited by 8
“These issues can be caused by insufficient training samples, dataset bias, and dataset shift [21, 27, 34, 49].”
BackgroundCausal Attention for Vision-Language TasksPreprint
Yang, X., Zhang, H., Qi, G.-J., & Cai, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03493 · Cited by 5
What are AI hallucinations? Why AIs sometimes make things up
Choi, A., & Mei, K. (2025) · doi:10.64628/aai.etwucf49j · Cited by 5
Li, S., Xu, X., Meng, W., Song, J., Peng, C. S., & Shen, H. T. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3599076 · Cited by 4
“Hallucination, defined as the generation of information that is irrelevant, incorrect, or nonsensical relative to the input context, hasemergedasasignificantchallengeinLargeVision-Language Models(LVLMs)[30].”
Zhang, X., Zhu, Y., Gu, C., Cao, J., Cheng, H., & Wu, K. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104431 · Cited by 4
Yuan, K., Lan, H., Gao, Y., & Wang, K. (2026) · Land · doi:10.3390/land15030449 · Cited by 10
Mitigating Object Hallucinations via Sentence-Level Early Intervention
Peng, S., Yang, S., Li, J., & Tian, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00067 · Cited by 22
“6, it effectively mitigates object hallucinations when evaluated on the Object Halbench [53] benchmark.”
Influential3D-GENERALIST: Vision-Language-Action Models for Crafting 3D Worlds
Sun, F.-Y., Wu, S., Jacobsen, C., Yim, T., Zou, H., Zook, A., Li, S., Chou, Y., Can, E., Wu, X., et al. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00177 · Cited by 4
“Refer to Figure 7 and the supplementary material for qualitative comparisons that showcase the importance of leveraging semantic knowledge in the VLM (i.e., Molmo) to predict asset placement.”
Large Vision-Language Model Security: A Survey
Wang, T., Fang, Z., Xue, H., Zhang, C., Jin, M., Xu, W., Shu, D., Yang, S. J., Wang, Z. G., & Liu, D. (2024) · Communications in computer and information science · doi:10.1007/978-981-96-0151-6_1 · Cited by 4
Wu, Y., Zhang, L., Yao, H., Du, J., Yan, K., Ding, S., Wu, Y., & Li, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01365 · Cited by 7
“They have manifested effectiveness in alleviating object hallucinations on popular benchmarks, such as POPE [18] and CHAIR [32].”
Wang, Y., Liu, Q., Fan, J., Hong, J., Chu, H., Tian, M., Gao, B., & Chen, H. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3673470 · Cited by 13
“The new corner case image and the generated answer can be encoded into the cross-modal database with the updating stage, with human correction optionally introduced when severe hallucinations occur. decisions [23], [24], [25], [26], [27].”
Metadata
- Authors
- Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K.
- Affiliations
- Max Planck Society (DE)
- University of California, Berkeley (US)
- Boston University (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 4035-4045
- OpenAlex
- W2892087592
- Cited by
- 420
- References
- 19
- Citations by year
- 2026: 103 · 2025: 80 · 2024: 70 · 2023: 42 · 2022: 32 · 2021: 45 · 2020: 32 · 2019: 15 · 2018: 1
- FWCI
- 7.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- object hallucination, image captioning, hallucination evaluation, language priors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
