CoChat
Sign Up
PreprintCited by 5Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesAdvanced Image and Video Retrieval Techniques

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Exploring Uncertainty Measures for Image-Caption Embedding-and-Retrieval Task

Open access
Hama, Kenta, Matsubara, Takashi, Uehara, Kuniaki, Cai, Jianfei·2019·arXiv
DOI 10.48550/arxiv.1904.08504arXiv 1904.08504
Open this paper in CoChat — free accountOpen PDF

Abstract

With the wide development of black-box machine learning algorithms, particularly deep neural network (DNN), the practical demand for the reliability assessment is rapidly rising. On the basis of the concept that `Bayesian deep learning knows what it does not know,' the uncertainty of DNN outputs has been investigated as a reliability measure for the classification and regression tasks. However, in the image-caption retrieval task, well-known samples are not always easy-to-retrieve samples. This study investigates two aspects of image-caption embedding-and-retrieval systems. On one hand, we quantify feature uncertainty by considering image-caption embedding as a regression task, and use it for model averaging, which can improve the retrieval performance. On the other hand, we further quantify posterior uncertainty by considering the retrieval as a classification task, and use it as a reliability measure, which can greatly improve the retrieval performance by rejecting uncertain queries. The consistent performance of two uncertainty measures is observed with different datasets (MS COCO and Flickr30k), different deep learning architectures (dropout and batch normalization), and different similarity functions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1904.08504

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Ghosh et al.'s model selection technique over the number of nodes did not come at the expense of computational or predictive performance.

    “Their model selection technique over the number of nodes did not come at the expense of computational or predictive performance.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [445]

    The cited paper’s Section 4.2 (p. 4) and retrieval-results discussion (p. 5) concern the number of models used for image-caption retrieval, not selection of the number of nodes. Searches for Ghosh, node selection, and computational or predictive performance found no passage addressing the attributed claim. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Ghosh et al. exploited the horseshoe, continuous shrinkage priors and the regularized horseshoe distributions for selection of model in BNNs.

    “Ghosh et al. [445] exploited the horseshoe, continuous shrinkage priors and the regularized horseshoe distributions for selection of model in BNNs (see Fig. 22).”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [445]

    Section 2.1 (p. 2) discusses Bayesian neural networks and model averaging, but does not address horseshoe priors or model selection. Searches for horseshoe, continuous shrinkage, regularized horseshoe, and Ghosh found no passage supporting the attributed claim. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • The strong shrinkage provided by the horseshoe was effective in turning off nodes that did not help explaining data when placed over node pre-activations and coupled with appropriate variational approximations.

    “The strong shrinkage provided by the horseshoe was effective in turning of nodes that did not help explaining data when placed over node pre-activations and coupled with appropriate variational approximations.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [445]

    The Experimental Settings (p. 4) describe dropout in an image-caption retrieval model, not a horseshoe prior. Searches for horseshoe shrinkage, node pre-activations, and variational approximations found no passage in our copy addressing the claimed effect. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. [1][1] Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, and Dmitry Vetrov. 2018. Uncertainty Estimation via Stochastic Batch Normalization. In International Conference on Learning Representations Workhosp (ICLRW) . arXiv:1802.04893 - · arXiv
  2. [2][2] David Barber and Christopher M Bishopt. 1997. Ensemble Learning for MultiLayer Networks. In Advances in Neural Information Processing Systems (NIPS) . -
  3. [3][3] Yoshua Bengio. 2012. Deep Learning of Representations for Unsupervised and Transfer Learning. In ICML Workshop on Unsupervised and Transfer Learning . 17-36. https://doi.org/10.1007/978-3-642-36657-4\_1 arXiv:1305.0445 - · DOI
  4. [4][4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (aug 2013), 1798-1828. https://doi.org/10.1109/ TPAMI.2013.50 arXiv:1206.5538 - · arXiv
  5. [5][5] CM Christopher M CM Bishop. 2006. Pattern recognition and machine learning. In Springer , Vol. 4. 738. https://doi.org/10.1117/1.2819119 arXiv:0-387-31073-8 - · DOI · Paper page
  6. [6][6] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. 2019. A Closer Look at Few-shot Classification. In International Conference on Learning Representations (ICLR) . -
  7. [7][7] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 1724-1734. https://doi.org/10.3115/ v1/D14-1179 arXiv:1406.1078 - · arXiv
  8. [8][8] Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 248-255. https://doi.org/10.1109/CVPRW. 2009.5206848 - · DOI
  9. [9][9] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In British Machine Vision Conference (BMVC) . arXiv:1707.05612 - · arXiv
  10. [10][10] Andrea Frome, Greg S. Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. DeViSE: A Deep Visual-Semantic Embedding Model. Advances in Neural Information Processing Systems (NIPS) (2013), 21212129. https://doi.org/10.1016/0921-4534(95)00110-7 arXiv:arXiv:1312.5650v3 - · DOI
  11. [11][11] Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In International Conference on Machine Learning (ICML) . arXiv:1506.02142 - · arXiv
  12. [12][12] Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised Domain Adaptation by Backpropagation. In International Conference on Machine Learning (ICML) . 1180-1189. arXiv:1409.7495 - · arXiv
  13. [13][13] Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, and Gang Wang. 2018. Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . https://doi.org/10.1109/CVPR.2018.00750 arXiv:1711.06420 - · DOI
  14. [15][15] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. 2019. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . arXiv:1812.05720 - · arXiv
  15. [16][16] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. β -VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. In International Conference on Learning Representations (ICLR) . 1-14. -
  16. [17][17] Geoffrey E. Hinton and Drew van Camp. 1993. Keeping the neural networks simple by minimizing the description length of the weights. In Annual Conference on Computational Learning Theory (COLT) . ACM Press, New York, New York, USA, 5-13. https://doi.org/10.1145/168304.168306 arXiv:1408.5093 - · DOI · Paper page
  17. [18][18] Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning (ICML) . arXiv:1502.03167 - · arXiv
  18. [19][19] Himanshu Jain, Yashoteja Prabhu, and Manik Varma. 2016. Extreme Multi-label Loss Functions for Recommendation, Tagging, Ranking & Other Missing Label Applications. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) . 935-944. https://doi.org/10.1145/2939672.2939756 - · DOI
  19. [20][20] Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Vol. 39. IEEE, 3128-3137. https://doi.org/10.1109/CVPR.2015. 7298932 arXiv:1503.08909v2 - · DOI
  20. [21][21] Alex Kendall and Yarin Gal. 2017. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. In Advances in Neural Information Processing Systems (NIPS) . arXiv:1703.04977 - · arXiv
  21. [22][22] Alex Kendall, Vijay Badrinarayanan, Roberto Cipolla, Vijay Badrinarayanan, and Roberto Cipolla. 2017. Bayesian SegNet: model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In British Machine Vision Conference (BMVC) . arXiv:1511.02680 - · arXiv
  22. [23][23] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR) . 1-15. arXiv:1412.6980 - · arXiv
  23. [24][24] Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR) . 1-14. arXiv:arXiv:1312.6114v10 - · arXiv
  24. [25][25] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. 1-13. arXiv:1411.2539 - · arXiv
  25. [26][26] Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or epistemic? Does it matter? Structural Safety 31, 2 (2009), 105-112. https://doi.org/10.1016/j.strusafe. 2008.06.020 - · DOI
  26. [27][27] Christian Leibig and Siegfried Wahl. 2016. Discriminative Bayesian neural networks know what they do not know. In NIPS Bayesian Deep Learning Workshop . 1-4. -
  27. [28][28] Yuncheng Li, Yale Song, and Jiebo Luo. 2017. Improving pairwise ranking for multi-label image classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 1837-1845. https://doi.org/10.1109/CVPR.2017.199 arXiv:1704.03135v3 - · DOI
  28. [29][29] Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV) , Vol. 8693 LNCS. 740-755. https://doi.org/10.1007/978-3-319-10602-1\_48 arXiv:1405.0312 - · DOI · Paper page
  29. [30][30] David J. C. MacKay. 1992. A Practical Bayesian Framework for Backpropagation Networks. Neural Computation 4, 3 (may 1992), 448-472. https://doi.org/10. 1162/neco.1992.4.3.448 -
  30. [31][31] Takashi Matsubara, Ryosuke Tachibana, and Kuniaki Uehara. 2018. Anomaly Machine Component Detection by Deep Generative Model with Unregularized Score. In International Joint Conference on Neural Networks (IJCNN) . -
  31. [32][32] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems (NIPS) , Vol. 1. https://doi.org/10.1162/ jmlr.2003.3.4-5.951 arXiv:1310.4546 - · arXiv
  32. [33][33] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. 2015. Distributional Smoothing with Virtual Adversarial Training. In International Conference on Learning Representations (ICLR) , Vol. 30. 1-18. arXiv:1507.00677 - · arXiv
  33. [34][34] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object Hallucination in Image Captioning. In Conference on Empirical Methods in Natural Language Processing (EMNLP) . arXiv:arXiv:1809.02156v1 - · arXiv
  34. [35][35] Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. 2004. Are Loss Functions All the Same? Neural Computation 16, 5 (may 2004), 1063-1076. https://doi.org/10.1162/089976604773135104 - · DOI
  35. [36][36] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems (NIPS) . 1-19. https://doi.org/arXiv: 1504.01391 arXiv:1704.00028 - · arXiv
  36. [37][37] Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations (ICLR) . 1-14. arXiv:1409.1556 - · arXiv
  37. [38][38] Lewis Smith and Yarin Gal. 2018. Understanding Measures of Uncertainty for Adversarial Example Detection. Uncertainty in Artificial Intelligence (UAI) (2018). https://doi.org/arXiv:1803.08533v1 arXiv:arXiv:1803.08533v1 - · arXiv
  38. [39][39] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15 (2014), 1929-1958. https: //doi.org/10.1214/12-AOS1000 arXiv:1102.4807 - · DOI
  39. [40][40] Ahmed Taha, Yi-Ting Chen, Xitong Yang, Teruhisa Misu, and Larry Davis. 2019. Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems. arXiv (2019). arXiv:1901.07702 - · arXiv
  40. [41][41] Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. 2018. RICAP : Random Image Cropping and Patching Data Augmentation for Deep CNNs. In Asian Conference on Machine Learning (ACML) . -
  41. [42][42] Sumio Watanabe. 2010. Equations of states in singular statistical estimation. Neural Networks 23, 1 (2010), 20-34. https://doi.org/10.1016/j.neunet.2009.08.002 arXiv:0712.0653 - · DOI
  42. [43][43] Jason Weston, Samy Bengio, and Nicolas Usunier. 2010. Large scale image annotation: Learning to rank with joint word-image embeddings. European Conference on Machine Learning (ECML) (2010). -
  43. [44][44] Yijun Xiao and William Yang Wang. 2019. Quantifying Uncertainties in Natural Language Processing Tasks. AAAI Conference on Artificial Intelligence (AAAI) (2019). arXiv:1811.07253 - · arXiv
  44. [45][45] Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes. In Conference on Empirical Methods in Natural Language Processing (EMNLP) . arXiv:1809.00812 - · arXiv
  45. [46][46] M. H. Peter Young, Alice Lai, and J. Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67-78. -
  46. [47][47] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations (ICLR) . arXiv:1710.09412 - · arXiv
  47. [48][48] Li Zhang, Tao Xiang, and Shaogang Gong. 2017. Learning a deep embedding model for zero-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Vol. 2017-Janua. 3010-3019. https://doi.org/10.1109/CVPR. 2017.321 arXiv:1611.05088 - · DOI
  48. [49][49] Quanshi Zhang, Wenguan Wang, and Song-Chun Zhu. 2018. Examining CNN Representations with respect to Dataset Bias. In AAAI Conference on Artificial Intelligence (AAAI) . arXiv:1710.10577 - · arXiv
  49. [50][50] Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, and Yi-Dong Shen. 2017. Dual-Path Convolutional Image-Text Embedding with Instance Loss. arXiv 14, 8 (2017), 1-15. arXiv:1711.05535 · arXiv

Cited by

Sorted by citations

  • A review of uncertainty quantification in deep learning: Techniques, applications and challenges

    Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111

    “Natural language processing (NLP): There are few NLPbased subjects such as neural machine translation [508] and some other interdisciplinary subjects such as image captioning [509], [510], [511], [512], visual question answering (VQA) [513], [514], [515], [516], [517] which are closely associated with NLP.”
    Background

Metadata

Authors
Hama, K., Matsubara, T., Uehara, K., & Cai, J.
Affiliations
  • The University of Osaka (JP)
  • Osaka Gakuin University (JP)
  • Monash University (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
17
Issue
2
Pages
1-19
ISSN
1551-6857 (ISSN-L)
DOI
10.48550/arxiv.1904.08504
arXiv
1904.08504
OpenAlex
W3157539197
Cited by
5
References
1
Citations by year
2023: 1 · 2022: 1 · 2021: 1 · 2020: 1 · 2019: 1
FWCI
0.27
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • ACM Transactions on Multimedia Computing Communications and Applications (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Japan Society for the Promotion of Science
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
uncertainty estimation, image captioning, posterior uncertainty, Bayesian deep learning, embedding retrieval, model averaging, dropout, batch normalization, similarity measures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review