Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning
Open accessAbstract
Explaining an image with missing or non-existent objects is known as object bias (hallucination) in image captioning. This behaviour is quite common in the state-of-the-art captioning models which is not desirable by humans. To decrease the object hallucination in captioning, we propose three simple yet efficient training augmentation method for sentences which requires no new training data or increase in the model size. By extensive analysis, we show that the proposed methods can significantly diminish our models’ object bias on hallucination metrics. Moreover, we experimentally demonstrate that our methods decrease the dependency on the visual features. All of our code, configuration files and model weights are available online1.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Biten et al.'s introduced method can reduce object hallucination without changing model architectures.
“Results show that their introduced method can reduce object hallucination without changing model architectures.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092 · cited 3× · see citing works
“We use Adam [22] optimizer with batch size 10 and learning rate 0.0002 and 0.0005 for UpDown [3] and AoA [19], respectively.”
InfluentialObject Hallucination in Image Captioning
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1437 · Cited by 420 · see citing works
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · see citing works
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291 · see citing works
Good News, Everyone! Context Driven Entity-Aware Captioning for News Images
Biten, A. F., Gomez, L., Rusiñol, M., & Karatzas, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01275 · Cited by 136 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 35
- Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9690-9698, 2020.
- Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, 2016. 4.1
- Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and VQA. arXiv preprint arXiv:1707.07998, 2017. 2, 3.1, 4.1, 4.2, 4.3, 1 · arXiv
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. pages 1-15, 2014. 2
- Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 456-473, 2018. 1
- Shizhe Chen, Qin Jin, Peng Wang, and Qi Wu. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 9962-9971, 2020. 2
- Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Show, control and tell: A framework for generating con- trollable and grounded captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8307-8316, 2019. 2
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009. 1
- Michael Denkowski and Alon Lavie. Meteor universal: Lan- guage specific translation evaluation for any target language. In Workshop on Statistical Machine Translation, 2014. 4.1
- Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional net- works for visual recognition and description. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2625-2634, 2015. 1
- Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017. 1
- Alon Halevy, Peter Norvig, and Fernando Pereira. The un- reasonable effectiveness of data. IEEE Intelligent Systems, 24(2):8-12, 2009. 1
- Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997. 3.1, 4.3
- Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016. 4.2 · arXiv
- Andrej Karpathy and Li Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):664-676, 2017. 4.1
- Thomas S Kuhn. The structure of scientific revolutions. Uni- versity of Chicago press, 2012. 1
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444, 2015. 1
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014. 1, 4.1
- Hugo Liu and Push Singh. Conceptnet-a practical com- monsense reasoning tool-kit. BT technology journal, 22(4):211-226, 2004. 2, 4.3
- Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7219-7228, 2018. 1
- Haley MacLeod, Cynthia L Bennett, Meredith Ringel Mor- ris, and Edward Cutrell. Understanding blind people's ex- periences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Hu- man Factors in Computing Systems, pages 5988-5999, 2017.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wj Zhu. BLEU: a method for automatic evaluation of machine trans- lation. Annual Meeting on Association for Computational Linguistics, 2002. 4.1
- Judea Pearl. Causality. Cambridge university press, 2009. 2
- Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Conference on Computer Vision and Pattern Recognition, 2017. 1, 2, 4.2, 4.3, 1, 1
- Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial train- ing. In Proceedings of the IEEE International Conference on Computer Vision, pages 4135-4144, 2017. 1
- Alasdair Tran, Alexander Mathews, and Lexing Xie. Trans- form and tell: Entity-aware news image captioning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13035-13045, 2020. 2
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Il- lia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. 2, 3.1, 4.1 · arXiv
- Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In IEEE Conference on Computer Vision and Pattern Recognition, 2015. 4.1, 4.2, 4.3
- Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, and Kate Saenko. Captioning images with diverse objects. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 5753-5761, 2017. 2
- Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In IEEE Conference on Computer Vision and Pattern Recognition, 2015. 2
- Zeyu Wang, Berthy Feng, Karthik Narasimhan, and Olga Russakovsky. Towards unique and informative captioning of images. arXiv preprint arXiv:2009.03949, 2020. 4.1, 1, 1, 1, 4.3 · arXiv
- Ronald J Williams. Simple statistical gradient-following al- gorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992. 2
- Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, and Qi Wu. Towards accurate text-based image captioning with content diversity exploration. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12637-12646, 2021. 2
- Xu Yang, Hanwang Zhang, and Jianfei Cai. Deconfounded image captioning: A causal retrospect. arXiv preprint arXiv:2003.03923, 2020. 1, 2, 1 · arXiv
- Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph de- composition. In European Conference on Computer Vision, pages 211-229. Springer, 2020.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
AI Hallucinations: A Misnomer Worth Clarifying
Maleki, N., Padmanabhan, B., & Dutta, K. (2024) · 2024 IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00033 · Cited by 194 · see citing works
GenAssist: Making Image Generation Accessible
Huh, M., Peng, Y.-H., & Pavel, A. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606735 · Cited by 67 · see citing works
“…to text-to-image models ( e.g., better aligning with human preferences [78]) and vision language models ( e.g., better composition reasoning [38], reducing hallucinations [7]), GenAssist could also learn what prompts are prone to generation errors and guide BLV creators in creating strong prompts.”
BackgroundVolcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
Lee, S., Park, S. H., Jo, Y., & Seo, M. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.23 · Cited by 20 · see citing works
HallE-Control: Controlling Object Hallucination in Large Multimodal Models
Zhai, B., Yang, S., Xu, C., Shen, S., Keutzer, K., Li, C., & Li, M. (2023) · arXiv · doi:10.48550/arxiv.2310.01779 · Cited by 2 · see citing works
- Authors
- Biten, A. F., Gomez, L., & Karatzas, D.
- Affiliations
- Universitat Autònoma de Barcelona (ES)
- UAB,Computer Vision Center,Spain
- ORCID
- Biten, Ali Furkan
- Venue
- 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2022-01
- Pages
- 2473-2482
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W3202965415
- Cited by
- 43
- References
- 8
- Citations by year
- 2026: 6 · 2025: 17 · 2024: 12 · 2023: 7 · 2022: 1
- FWCI
- 3.35
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, public-domain)
- arXiv (Cornell University) (cc-by)
- Funders
- Generalitat de Catalunya
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- object hallucination, image captioning, hallucination mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
