The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Probabilistic Embeddings for Cross-Modal Retrieval
Open accessAbstract
Cross-modal retrieval methods build a common representation space for samples from multiple modalities, typically from the vision and the language domains. For images and their captions, the multiplicity of the correspondences makes the task particularly challenging. Given an image (respectively a caption), there are multiple captions (respectively images) that equally make sense. In this paper, we argue that deterministic functions are not sufficiently powerful to capture such one-to-many correspondences. Instead, we propose to use Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space. Since common benchmarks such as COCO suffer from non-exhaustive annotations for cross-modal matches, we propose to additionally evaluate retrieval on the CUB dataset, a smaller yet clean database where all possible image-caption pairs are annotated. We extensively ablate PCME and demonstrate that it not only improves the retrieval performance over its deterministic counterpart but also provides uncertainty estimates that render the embeddings more interpretable. Code is available at https://github.com/naver-ai/pcme.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
(2012) · Cited by 9,062
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400
Self-citationDimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Canonical Correlation Analysis: An Overview with Application to Learning Methods
Hardoon, D. R., Szedmak, S., & Shawe-Taylor, J. (2004) · Neural Computation · doi:10.1162/0899766042321814 · Cited by 3,202
Improved regularization of convolutional neural networks with cutout
DeVries, T., & Taylor, G. W. (2017) · arXiv · doi:10.48550/arxiv.1708.04552 · Cited by 2,776
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Generating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
Zero-Shot Learning — The Good, the Bad and the Ugly
Xian, Y., Schiele, B., & Akata, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.328 · Cited by 891
Learning Deep Representations of Fine-Grained Visual Descriptions
Reed, S., Akata, Z., Lee, H., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.13 · Cited by 760
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.541 · Cited by 760
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00844 · Cited by 747 · cited 1×
“[43] utilized Variational Autoencoders [23] for zero-shot recognition.”
MethodDeep Fragment Embeddings for Bidirectional Image Sentence Mapping
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 739
Dual Attention Networks for Multimodal Reasoning and Matching
Nam, H., Ha, J.-W., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.232 · Cited by 726
Visual Semantic Reasoning for Image-Text Matching
Li, K., Zhang, Y., Li, K., Li, Y., & Fu, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00475 · Cited by 599
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05612 · Cited by 543
Multi-Modality Cross Attention Network for Image and Sentence Matching
Wei, X., Zhang, T., Li, Y., Zhang, Y., & Wu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01095 · Cited by 451
Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space
Neelakantan, A., Shankar, J., Passos, A., & McCallum, A. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1113 · Cited by 442
A Metric Learning Reality Check
Musgrave, K., Belongie, S., & Lim, S.-N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58595-2_41 · Cited by 414
12-in-1: Multi-Task Vision and Language Representation Learning
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., & Lee, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01045 · Cited by 410
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
Wang, Z., Liu, X., Li, H., Sheng, L., Yan, J., Wang, X., & Shao, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00586 · Cited by 359
Shi, Y., & Jain, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00700 · Cited by 335
Data Uncertainty Learning in Face Recognition
Chang, J., Lan, Z., Cheng, C., & Wei, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00575 · Cited by 320
Context-Aware Attention Network for Image-Text Retrieval
Zhang, Q., Lei, Z., Zhang, Z., & Li, S. Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00359 · Cited by 285
Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections
Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., & Lazebnik, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_35 · Cited by 281
Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
Song, Y., & Soleymani, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00208 · Cited by 275
Word Representations via Gaussian Embedding
Vilnis, L., & McCallum, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6623 · Cited by 267
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM
Huang, Y., Wang, W., & Wang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.767 · Cited by 233
AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant WeightsPreprint
Heo, B., Chun, S., Oh, S. J., Han, D., Yun, S., Kim, G., Uh, Y., & Ha, J.-W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08217 · Cited by 201
Self-citationVSE++: Improving Visual-Semantic Embeddings with Hard Negatives.
Faghri, F., Fleet, D. J., Kiros, J., & Fidler, S. (2018) · Cited by 184
Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
Liu, C., Mao, Z., Liu, A.-A., Zhang, T., Wang, B., & Zhang, Y. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350869 · Cited by 182
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image AnnotationPreprint
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7399 · Cited by 107
View-Invariant Probabilistic Embedding for Human Pose
Sun, J. J., Zhao, J., Chen, L.-C., Schroff, F., Adam, H., & Liu, T. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58558-7_4 · Cited by 103
NSML: Meet the MLaaS platform with a real-world case study
Kim, H., Kim, M., Seo, D., Kim, J., Park, H., Park, S., Jo, H., Kim, K., Yang, Y., Kim, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.09957 · Cited by 91 · cited 1×
“All experiments were conducted on NAVER Smart Machine Learning (NSML) [19] platform.”
MethodAdaptive Offline Quintuplet Loss for Image-Text Matching
Chen, T., Deng, J., & Luo, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_33 · Cited by 77
Smoothing the Geometry of Probabilistic Box Embeddings
Li, X., Vilnis, L., Zhang, D., Boratko, M., & McCallum, A. (2018) · International Conference on Learning Representations · Cited by 56
Modeling Uncertainty with Hedged Instance Embedding
Oh, S. J., Murphy, K., Pan, J., Roth, J., Schroff, F., & Gallagher, A. (2018) · arXiv · doi:10.48550/arxiv.1810.00319 · Cited by 55
Self-citationPreserving Semantic Neighborhoods for Robust Cross-Modal Retrieval
Thomas, C., & Kovashka, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_19 · Cited by 42
A Mixture Model for Learning Multi-Sense Word Embeddings
Nguyen, D. Q., Nguyen, D. Q., Modi, A., Thater, S., & Pinkal, M. (2017) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s17-1015 · Cited by 39
Separating Self-Expression and Visual Content in Hashtag Supervision
Veit, A., Nickel, M., Belongie, S., & van der Maaten, L. J. P. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00620 · Cited by 35
Probabilistic Embeddings for Speaker Diarization
Silnova, A., Brümmer, N., Rohdin, J., Stafylakis, T., & Burget, L. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-4 · Cited by 16
Stochastic Prototype EmbeddingsPreprint
Scott, T. R., Ridgeway, K., & Mozer, M. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11702 · Cited by 10
Chen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Cited by
Qian, S., Xue, D., Fang, Q., & Xu, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3188547 · Cited by 68
AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation
Yang, J.-H., Lassance, C., Sampaio De Rezende, R., Srinivasan, K., Redi, M., Clinchant, S., & Lin, J. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591903 · Cited by 9
Self-citation
Metadata
- Authors
- Chun, S., Oh, S. J., Sampaio de Rezende, R., Kalantidis, Y., & Larlus, D.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 8411-8420
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3168433561
- Cited by
- 256
- References
- 56
- FWCI
- 14.03
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.