The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Open accessAbstract
We propose to use the visual denotations of linguistic expressions (i.e. the set of images they describe) to define novel denotational similarity metrics, which we show to be at least as beneficial as distributional similarities for two tasks that require semantic inference. To compute these denotational similarities, we construct a denotation graph, i.e. a subsumption hierarchy over constituents and their denotations, based on a large corpus of 30K images and 150K descriptive captions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
An Information-Theoretic Definition of Similarity
Lin, D. (1998) · International Conference on Machine Learning · Cited by 3,790
Word association norms, mutual information, and lexicography
Church, K., & Hanks, P. (1990) · doi:10.5555/89086.89095 · Cited by 3,590
Using Information Content to Evaluate Semantic Similarity in a Taxonomy
Resnik, P. (1995) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9511007 · Cited by 3,193
Barwise, J., & Perry, J. (1981) · The Journal of Philosophy · doi:10.2307/2026578 · Cited by 2,352
Computing semantic relatedness using Wikipedia-based explicit semantic analysis
Gabrilovich, E., & Markovitch, S. (2007) · Cited by 1,981
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · neural information processing systems · Cited by 1,356
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,314
Self-citationComputational linguistics in 1990
Karlgren, H. (1990) · Proceedings of the 13th conference on Computational linguistics - · doi:10.3115/992507.992531 · Cited by 1,243
Cited by
Items 101–162 of 162
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
Rohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C., Larochelle, H., Courville, A., & Schiele, B. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0987-1 · Cited by 300
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0965-7 · Cited by 298
Self-citationOrder-Embeddings of Images and Language
Vendrov, I., Kiros, R., Fidler, S., & Urtasun, R. (2015) · arXiv · doi:10.48550/arxiv.1511.06361 · Cited by 296
Learning the Best Pooling Strategy for Visual Semantic Embedding
Chen, J., Hu, H., Wu, H., Jiang, Y., & Wang, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01553 · Cited by 293
Multi-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 291
HICO: A Benchmark for Recognizing Human-Object Interactions in Images
Chao, Y.-W., Wang, Z., He, Y., Wang, J., & Deng, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.122 · Cited by 290
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., & Yuan, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00461 · Cited by 290
Image Captioning with Deep Bidirectional LSTMs
Wang, C., Yang, H., Bartz, C., & Meinel, C. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2964299 · Cited by 287
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R., Saenko, K., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.8 · Cited by 286
Clotho: an Audio Captioning Dataset
Drossos, K., Lipping, S., & Virtanen, T. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052990 · Cited by 285
Context-Aware Attention Network for Image-Text Retrieval
Zhang, Q., Lei, Z., Zhang, Z., & Li, S. Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00359 · Cited by 285
StyleNet: Generating Attractive Visual Captions with Styles
Gan, C., Gan, Z., He, X., Gao, J., & Deng, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.108 · Cited by 283
Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections
Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., & Lazebnik, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_35 · Cited by 281
Self-citationAssociating neural word embeddings with deep image representations using Fisher Vectors
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299073 · Cited by 281
Cross-Modal Attention With Semantic Consistence for Image–Text Matching
Xu, X., Wang, T., Yang, Y., Zuo, L., Shen, F., & Shen, H. T. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2967597 · Cited by 279
VLP: A Survey on Vision-language Pre-training
Chen, F.-L., Zhang, D.-Z., Han, M.-L., Chen, X.-Y., Shi, J., Xu, S., & Xu, B. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1369-5 · Cited by 279
Identity-Aware Textual-Visual Matching with Latent Co-attention
Li, S., Xiao, T., Li, H., Yang, W., & Wang, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.209 · Cited by 277
A Survey of Cross-lingual Word Embedding Models
Ruder, S., Vulić, I., & Søgaard, A. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11640 · Cited by 266
EVA-02: A visual representation for neon genesis
Fang, Y., Sun, Q., Wang, X., Huang, T., Wang, X., & Cao, Y. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105171 · Cited by 264
Probabilistic Embeddings for Cross-Modal Retrieval
Chun, S., Oh, S. J., Sampaio de Rezende, R., Kalantidis, Y., & Larlus, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00831 · Cited by 256
CCL: Cross-modal Correlation Learning With Multigrained Fusion by Hierarchical Network
Peng, Y., Qi, J., Huang, X., & Yuan, Y. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2742704 · Cited by 228
CommonGen: A constrained text generation challenge for generative commonsense reasoning
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.165 · Cited by 239
Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., & Najork, M. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463257 · Cited by 214
Unsupervised Contrastive Cross-modal Hashing
Hu, P., Zhu, H., Lin, J., Peng, D., Zhao, Y.-P., & Peng, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3177356 · Cited by 199
Attention Correctness in Neural Image Captioning
Liu, C., Mao, J., Sha, F., & Yuille, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11197 · Cited by 210
RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
Zhan, Y., Xiong, Z., & Yuan, Y. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2023.3250471 · Cited by 179
“Over the years, various visual grounding datasets [18], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65] based on real-world and computer-generated images have been proposed to study visual grounding.”
BackgroundMethodDE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models
Sha, Z., Li, Z., Yu, N., & Zhang, Y. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3616588 · Cited by 175
ArtEmis: Affective Language for Visual Art
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01140 · Cited by 161
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundMei, X., Meng, C., Liu, H., Kong, Q., Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., & Wang, W. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2024.3419446 · Cited by 150
Multimodal Federated Learning: A Survey
Che, L., Wang, J., Zhou, Y., & Ma, F. (2023) · doi:10.20944/preprints202307.1420.v1 · Cited by 127
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100
AudioCaps: Generating Captions for Audios in The Wild
Kim, C. D., Kim, B., Lee, H., & Kim, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1011 · Cited by 96
Interleaved text/image Deep Mining on a large-scale radiology database
Shin, H.-C., Lu, L., Kim, L., Seff, A., Yao, J., & Summers, R. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298712 · Cited by 94
HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment
Vulić, I., Gerz, D., Kiela, D., Hill, F., & Korhonen, A. (2016) · arXiv · doi:10.48550/arxiv.1608.02117 · Cited by 90
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“Flickr30k (Young et al., 2014) comprises 31,783 images, depicting diverse everyday activities, events, and scenes.”
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76
Hu, J., Small, H., Kean, H., Takahashi, A., Zekelman, L., Kleinman, D., Ryan, E., Nieto-Castañón, A., Ferreira, V., & Fedorenko, E. (2023) · Cerebral Cortex · doi:10.1093/cercor/bhac350 · Cited by 73
XGPT: Cross-modal Generative Pre-Training for Image Captioning
Xia, Q., Huang, H., Duan, N., Zhang, D., Ji, L., Sui, Z., Cui, E., Bharti, T., & Zhou, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-88480-2_63 · Cited by 70
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68
Probing Image-Language Transformers for Verb Understanding
Hendricks, L. A., & Nematzadeh, A. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.318 · Cited by 63
Building an Evaluation Scale using Item Response Theory
Lalor, J., Wu, H., & yu, H. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1062 · Cited by 60
M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
Ni, M., Huang, H., Su, L., Cui, E., Bharti, T., Wang, L., Zhang, D., & Duan, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00397 · Cited by 57
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2016) · arXiv · doi:10.48550/arxiv.1612.06890 · Cited by 40
GilBERT: Generative Vision-Language Pre-Training for Image-Text Retrieval
Hong, W., Ji, K., Liu, J., Wang, J., Chen, J., & Chu, W. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462838 · Cited by 37
Guo, W., Huang, H., Kong, X., & He, R. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3351053 · Cited by 36
Learning to Predict Denotational Probabilities For Modeling Entailment
Lai, A., & Hockenmaier, J. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1068 · Cited by 33
Self-citationSimons, R. N., Gurari, D., & Fleischmann, K. R. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415176 · Cited by 27
Alsharid, M., Cai, Y., Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102630 · Cited by 25
Abdulgalil, H. D., & Basir, O. A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100159 · Cited by 20
Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
Cui, W., Zheng, G., & Wang, W. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.444 · Cited by 17
Multi-stage Pre-training over Simplified Multimodal Pre-training Models
Liu, T., Feng, F., & Wang, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.199 · Cited by 11
Look, Read and Enrich - Learning from Scientific Figures and their Captions
Gomez-Perez, J. M., & Ortega, R. (2019) · Proceedings of the 10th International Conference on Knowledge Capture · doi:10.1145/3360901.3364420 · Cited by 8
Quality and Relevance Metrics for Selection of Multimodal Pretraining Data
Rao, R., Rao, S., Nouri, E., Dey, D., Celikyilmaz, A., & Dolan, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00486 · Cited by 6
Crowdsourcing Thumbnail Captions: Data Collection and Validation
Aguirre, C., Cao, S., Mahmood, A., & Huang, C.-M. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3589346 · Cited by 6
A Descriptive Review of Image Datasets for Accessible Alternative Descriptions in STEM Domains
Cardia, M., Buzzi, M., Galesi, G., & Leporini, B. (2025) · Proceedings of the 16th Biannual Conference of the Italian SIGCHI Chapter · doi:10.1145/3750069.3750122 · Cited by 5
Peri, D., Sah, S., & Ptucha, R. (2019) · 2019 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2019.8802922 · Cited by 4
Tu, R.-C., Ma, Z.-A., Lan, T., Zhao, Y., Huang, H., & Mao, X.-L. (2024) · arXiv · doi:10.48550/arxiv.2411.15488 · Cited by 4
FormalAlign: Automated Alignment Evaluation for Autoformalization
Lu, J., Wan, Y., Huang, Y., Xiong, J., Liu, Z., & Guo, Z. (2024) · arXiv · doi:10.48550/arxiv.2410.10135 · Cited by 1
Metadata
- Authors
- Young, P., Lai, A., Hodosh, M., & Hockenmaier, J.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2014-12
- Volume
- 2
- Pages
- 67-78
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2185175083
- Cited by
- 2,426
- References
- 34
- FWCI
- 93.47
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.