CoChat
Sign Up
PreprintCited by 6Advanced Neural Network ApplicationsMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning

Elucidating image-to-set prediction: An analysis of models, losses and datasets

Open access
Pineda, Luis, Salvador, Amaia, Drozdzal, Michal, Romero, Adriana·2019·arXiv
DOI 10.48550/arxiv.1904.05709arXiv 1904.05709
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we identify an important reproducibility challenge in the image-to-set prediction literature that impedes proper comparisons among published methods, namely, researchers use different evaluation protocols to assess their contributions. To alleviate this issue, we introduce an image-to-set prediction benchmark suite built on top of five public datasets of increasing task complexity that are suitable for multi-label classification (VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an in-depth analysis where we study the key components of current models, namely the choice of the image representation backbone as well as the set predictor design. Our results show that (1) exploiting better image representation backbones leads to higher performance boosts than enhancing set predictors, and (2) modeling both the label co-occurrences and ordering has a slight positive impact in terms of performance, whereas explicit cardinality prediction only helps when training on complex datasets, such as Recipe1M. To facilitate future image-to-set prediction research, we make the code, best models and dataset splits publicly available at: https://github.com/facebookresearch/image-to-set.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1904.05709

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Inference code for DETR can be implemented in less than 50 lines in PyTorch.

    “Inference code for DETR can be implemented in less than 50 lines in PyTorch [32].”

    From End-to-End Object Detection with Transformers (2020) · cites this paper as [32]

    The cited paper’s B.1 Data pre-processing section (p. 12) says its own models were implemented in PyTorch, but does not discuss DETR inference code or its length. Searches for DETR and for a 50-line inference implementation found no supporting passage in our copy. (LLM, conf=0.77)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 66

  1. [1][1] Alessandro Antonucci, Giorgio Corani, Denis Deratani Mau´ a, and Sandra Gabaglio. An ensemble of bayesian networks for multilabel classification. In IJCAI , 2013. 3 -
  2. [2][2] Wei Bi and James T. Kwok. Efficient multi-label classification with many labels. In ICML , 2013. 3 -
  3. [3][3] Xavier Bouthillier, C´ esar Laurent, and Pascal Vincent. Unreproducible research is reproducible. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 725734, Long Beach, California, USA, 09-15 Jun 2019. PMLR. 1 -
  4. [4][4] Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. BMVC , 2014. 3 -
  5. [6][6] Shang-Fu Chen, Yi-Chen Chen, Chih-Kuan Yeh, and YuChiang Frank Wang. Order-free rnn with visual attention for multi-label classification. In AAAI , 2018. 2, 3, 6 -
  6. [7][7] Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrent attentional reinforcement learning for multi-label image recognition. In AAAI , 2018. 2, 3, 5 -
  7. [8][8] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In CVPR , 2019. 2, 3 -
  8. [9][9] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus-wide: a real-world web image database from national university of singapore. In ACM Conference on Image and Video Retrieval , 2009. 2, 12 -
  9. [10][10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 1 -
  10. [11][11] Krzysztof Dembczy´ nski, Weiwei Cheng, and Eyke H¨ ullermeier. Bayes optimal multilabel classification via probabilistic classifier chains. In ICML , 2010. 3 -
  11. [12][12] Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. The importance of skip connections in biomedical image segmentation. In Gustavo Carneiro, Diana Mateus, Lo¨ ıc Peter, Andrew Bradley, Jo˜ ao Manuel R. S. Tavares, Vasileios Belagiannis, Jo˜ ao Paulo Papa, Jacinto C. Nascimento, Marco Loog, Zhi Lu, Jaime S. Cardoso, and Julien Cornebise, editors, Deep Learning and Data Labeling for Medical Applications , pages 179-187, Cham, 2016. Springer International Publishing. 4 -
  12. [13][13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 2, 12 -
  13. [14][14] Weifeng Ge, Sibei Yang, and Yizhou Yu. Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning. In CVPR , 2018. 2, 3 -
  14. [15][15] Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. Deep convolutional ranking for multilabel image annotation. CoRR , abs/1312.4894, 2013. 3, 4 - · arXiv
  15. [16][16] Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. In CVPR , 2019. 2, 3 -
  16. [17][17] Kaiming He, Georgia Gkioxari, Piotr Doll´ ar, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 1 -
  17. [21][21] Daniel J Hsu, Sham M Kakade, John Langford, and Tong Zhang. Multi-label prediction via compressed sensing. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, NeurIPS . Curran Associates, Inc., 2009. 3 -
  18. [22][22] Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In AISTATS , pages 240-248, 2016. 5 -
  19. [23][23] Simon J´ egou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In CVPR-W , 2017. 1 -
  20. [24][24] Ashish Kapoor, Raajay Viswanathan, and Prateek Jain. Multilabel classification using bayesian compressed sensing. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, NeurIPS . Curran Associates, Inc., 2012. 3 -
  21. [25][25] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014. 12 - · arXiv
  22. [26][26] Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The GAN landscape: Losses, architectures, regularization, and normalization. CoRR , abs/1807.04720, 2018. 1 - · arXiv
  23. [27][27] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. JMLR , 18(185):1-52, 2018. 2, 5 -
  24. [28][28] Liang Li, Shuhui Wang, Shuqiang Jiang, and Qingming Huang. Attentive recurrent neural network for weaksupervised multi-label image classification. In ACM Multimedia , 2018. 2, 3 -
  25. [29][29] Yuncheng Li, Yale Song, and Jiebo Luo. Improving pairwise ranking for multi-label image classification. In CVPR , 2017. 2, 3 -
  26. [30][30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 1, 2, 12 -
  27. [31][31] Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang. Multi-label classification via feature-aware implicit label space encoding. In ICML , 2014. 3 -
  28. [32][32] Feng Liu, Tao Xiang, Timothy M Hospedales, Wankou Yang, and Changyin Sun. Semantic regularisation for recurrent image annotation. In CVPR , 2017. 2, 3 -
  29. [33][33] Luchen Liu, Sheng Guo, Weilin Huang, and Matthew R Scott. Decoupling category-wise independence and relevance with self-attention for multi-label image classification. In ICASSP , 2019. 2, 3 -
  30. [34][34] Weiwei Liu and Ivor W. Tsang. Large margin metric learning for multi-label prediction. In AAAI , 2015. 3 -
  31. [35][35] Yongcheng Liu, Lu Sheng, Jing Shao, Junjie Yan, Shiming Xiang, and Chunhong Pan. Multi-label image classification via knowledge distillation from weakly-supervised detection. In ACM Multimedia , 2018. 2, 3 -
  32. [36][36] Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Sch¨ olkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. CoRR , abs/1811.12359, 2018. 1 - · arXiv
  33. [37][37] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015. 1 -
  34. [38][38] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR , 2017. 4 -
  35. [39][39] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 700-709. Curran Associates, Inc., 2018. 1 -
  36. [40][40] Yan Luo, Ming Jiang, and Qi Zhao. Visual attention in multilabel image classification. In CVPR-W , 2019. 2 -
  37. [41][41] Fan Lyu, Qi Wu, Fuyuan Hu, Qingyao Wu, and Mingkui Tan. Attend and imagine: Multi-label image classification with visual attention and recurrent neural networks. IEEE Transactions on Multimedia , 2019. 3 -
  38. [42][42] Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. CoRR , abs/1805.00932, 2018. 3, 4 - · arXiv
  39. [43][43] G´ abor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. CoRR , abs/1707.05589, 2017. 1 - · arXiv
  40. [45][45] Jinseok Nam, Eneldo Loza Menc´ ıa, Hyunwoo J Kim, and Johannes F¨ urnkranz. Maximizing subset accuracy with recurrent neural networks in multi-label classification. In NeurIPS , 2017. 3, 6 -
  41. [46][46] Nelson Nauata, Hexiang Hu, Guang-Tong Zhou, Zhiwei Deng, Zicheng Liao, and Greg Mori. Structured label inference for visual understanding. TPAMI , 2019. 3 -
  42. [47][47] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semisupervised learning algorithms. CoRR , abs/1804.09170, 2018. 1 - · arXiv
  43. [48][48] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurIPS-W , 2017. 13 -
  44. [50][50] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NeurIPS , 2015. 1 -
  45. [52][52] S Hamid Rezatofighi, Anton Milan, Qinfeng Shi, Anthony Dick, and Ian Reid. Joint learning of set cardinality and state distribution. AAAI , 2018. 2, 3, 4 -
  46. [53][53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV , 2015. 1, 3 -
  47. [54][54] Amaia Salvador, Michal Drozdzal, Xavier Gir´ o i Nieto, and Adriana Romero. Inverse cooking: Recipe generation from food images. In CVPR , 2019. 3, 4, 5, 12 -
  48. [55][55] Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Torralba. Learning cross-modal embeddings for cooking recipes and food images. In CVPR , 2017. 2, 5, 12 -
  49. [57][57] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2015. 1 -
  50. [58][58] Yale Song, Daniel McDuff, Deepak Vasisht, and Ashish Kapoor. Exploiting sparsity and co-occurrence structure for action unit recognition. In FG . IEEE Computer Society, 2015. 3 -
  51. [59][59] Grigorios Tsoumakas and Ioannis Vlahavas. Random klabelsets: An ensemble method for multilabel classification. In Joost N. Kok, Jacek Koronacki, Raomon Lopez de Mantaras, Stan Matwin, Dunja Mladeniˇ c, and Andrzej Skowron, editors, ECML , 2007. 3 -
  52. [60][60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. 4 -
  53. [61][61] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. CNN-RNN: A unified framework for multi-label image classification. In CVPR , 2016. 2, 3, 5 -
  54. [62][62] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recognition by recurrently discovering attentional regions. In ICCV , 2017. 2, 3, 5 -
  55. [63][63] Yunchao Wei, Wei Xia, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan. CNN: single-label to multi-label. CoRR , abs/1406.5726, 2014. 3 - · arXiv
  56. [64][64] Yunchao Wei, Wei Xia, Min Lin, Junshi Huang, Bingbing Ni, Jian Dong, Yao Zhao, and Shuicheng Yan. Hcp: A flexible cnn framework for multi-label image classification. TPAMI , 2016. 3 -
  57. [65][65] Sean Welleck, Zixin Yao, Yu Gai, Jialin Mao, Zheng Zhang, and Kyunghyun Cho. Loss functions for multiset prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 5783-5792. Curran Associates, Inc., 2018. 3 -
  58. [66][66] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI , 2011. 3 -
  59. [67][67] Baoyuan Wu, Fan Jia, Wei Liu, Bernard Ghanem, and Siwei Lyu. Multi-label learning with missing labels using mixed dependency graphs. IJCV , 2018. 3 -
  60. [68][68] Saining Xie, Ross Girshick, Piotr Doll´ ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , 2017. 3, 7 -
  61. [69][69] Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, and Jianfei Cai. Exploit bounding box annotations for multi-label object recognition. In CVPR , 2016. 3 -
  62. [70][70] Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and YuChiang Frank Wang. Learning deep latent spaces for multilabel classification. CoRR , abs/1707.00418, 2017. 3 - · arXiv
  63. [71][71] Junjie Zhang, Qi Wu, Chunhua Shen, Jian Zhang, and Jianfeng Lu. Multilabel image classification with regional latent semantic dependencies. IEEE Transactions on Multimedia , 2018. 2, 3 -
  64. [73][73] Feipeng Zhao and Yuhong Guo. Semi-supervised multi-label learning with incomplete labels. In IJCAI , 2015. 3 -
  65. [74][74] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , 2017. 2, 5, 12 -
  66. [75][75] Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, and Xiaogang Wang. Learning spatial regularization with imagelevel supervisions for multi-label image classification. In CVPR , 2017. 2, 3

Cited by

Sorted by citations

Authors
Pineda, L., Salvador, A., Drozdzal, M., & Romero, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1904.05709
arXiv
1904.05709
OpenAlex
W2939912336
Cited by
6
References
4
Citations by year
2025: 1 · 2021: 2 · 2020: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
image captioning, multi-label classification, reproducibility, benchmark, label ordering, volatile organic compounds, COCO, NUS-WIDE, ADE20K

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review