The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Deep Learning for Content-Based Image Retrieval: A Comprehensive Study
Open accessAbstract
Learning effective feature representations and similarity measures are crucial to the retrieval performance of a content-based image retrieval (CBIR) system. Despite extensive research efforts for decades, it remains one of the most challenging open problems that considerably hinders the successes of real-world CBIR systems. The key challenge has been attributed to the well-known ``semantic gap'' issue that exists between low-level image pixels captured by machines and high-level semantic concepts perceived by human. Among various techniques, machine learning has been actively investigated as a possible direction to bridge the semantic gap in the long term. Inspired by recent successes of deep learning techniques for computer vision and other applications, in this paper, we attempt to address an open problem: if deep learning is a hope for bridging the semantic gap in CBIR and how much improvements in CBIR tasks can be achieved by exploring the state-of-the-art deep learning techniques for learning feature representations and similarity measures. Specifically, we investigate a framework of deep learning with application to CBIR tasks with an extensive set of empirical studies by examining a state-of-the-art deep learning method (Convolutional Neural Networks) for CBIR tasks under varied settings. From our empirical studies, we find some encouraging results and summarize some important insights for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
SURF: Speeded Up Robust Features
Bay, H., Tuytelaars, T., & Van Gool, L. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_32 · Cited by 14,955
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,319
Content-based image retrieval at the end of the early years
Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., & Jain, R. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.895972 · Cited by 6,056
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,822
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., & Saul, L. K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1577078 · Cited by 4,044
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · cited 4×
“We do not evaluate features from lower convolutional layers in the network since the lower layers are unlikely to contain richer semantic representations than the later features which form higher-level hypothesis from the low-level to mid-level local information [13].”
BackgroundMethodResultInfluentialTexture features for browsing and retrieval of image data
Manjunath, B. S., & Ma, W. Y. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.531803 · Cited by 3,727
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Object retrieval with large vocabularies and fast spatial matching
Philbin, J., Chum, O., Isard, M., Sivic, J., & Zisserman, A. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383172 · Cited by 2,929
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680
Salakhutdinov, R., & Hinton, G. E. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 2,605
Distance Metric Learning for Large Margin Nearest Neighbor Classification
Weinberger, K. Q., Blitzer, J. C., & Saul, L. K. (2005) · neural information processing systems · Cited by 2,132
Restricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Online Passive-Aggressive Algorithms
Crammer, K., Dekel, O., Keshet, J., Shalev‐Shwartz, S., & Singer, Y. (2003) · Cited by 1,665
Content-based multimedia information retrieval: State of the art and challenges
Lew, M. S., Sebe, N., Djeraba, C., & Jain, R. (2006) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/1126004.1126005 · Cited by 1,528
Aggregating Local Image Descriptors into Compact Codes
Jegou, H., Perronnin, F., Douze, M., Sanchez, J., Perez, P., & Schmid, C. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.235 · Cited by 1,503
Attribute and simile classifiers for face verification
Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459250 · Cited by 1,361
(2012) · neural information processing systems · Cited by 1,291
Salakhutdinov, R., & Hinton, G. (2009) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2008.11.006 · Cited by 1,246 · cited 1×
“Among these techniques, some works have focused on learning to hashing or compact codes [41, 35, 23, 57, 58].”
BackgroundImproving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,073
Discovering objects and their location in images
Sivic, J., Russell, B. C., Efros, A. A., Zisserman, A., & Freeman, W. T. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.77 · Cited by 975
Image retrieval using color and shape
Jain, A. K., & Vailaya, A. (1996) · Pattern Recognition · doi:10.1016/0031-3203(95)00160-3 · Cited by 912
Online Passive-Aggressive Algorithms
CrammerKoby, DekelOfer, KeshetJoseph, Shalev-ShwartzShai, & SingerYoram (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248566 · Cited by 894
Evaluating bag-of-visual-words representations in scene classification
Yang, J., Jiang, Y.-G., Hauptmann, A. G., & Ngo, C.-W. (2007) · Proceedings of the international workshop on Workshop on multimedia information retrieval · doi:10.1145/1290082.1290111 · Cited by 825
Is that you? Metric learning approaches for face identification
Guillaumin, M., Verbeek, J., & Schmid, C. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459197 · Cited by 757
A tutorial survey of architectures, algorithms, and applications for deep learning
Deng, L. (2014) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2013.9 · Cited by 725
Large Scale Online Learning of Image Similarity through Ranking
Chechik, G., Sharma, V., Shalit, U., & Bengio, S. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02172-5_2 · Cited by 722
CENTRIST: A Visual Descriptor for Scene Categorization
Wu, J., & Rehg, J. M. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.224 · Cited by 709
Hamming Distance Metric Learning
Norouzi, M., Fleet, D. J., & Salakhutdinov, R. (2012) · neural information processing systems · Cited by 701
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives
Bengio, Y., Courville, A. C., & Vincent, P. (2012) · Cited by 581
VisualRank: Applying PageRank to Large-Scale Image Search
Jing, Y., & Baluja, S. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.121 · Cited by 441
Learning distance functions using equivalence relations
Bar-Hillel, A., Hertz, T., Shental, N., & Weinshall, D. (2003) · Cited by 433
Locally adaptive metric nearest-neighbor classification
Domeniconi, C., Peng, J., & Gunopulos, D. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1033219 · Cited by 307
Learning Distance Metrics with Contextual Constraints for Image Retrieval
Hoi, S. C. H., Liu, W., Lyu, M. R., & Ma, W.-Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.167 · Cited by 307
Self-citationOnline Metric Learning and Fast Similarity Search
Jain, P., Kulis, B., Dhillon, I. S., et al. (2008) · Cited by 193
Semantics-Preserving Bag-of-Words Models and Applications
Wu, L., Hoi, S. C. H., & Yu, N. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2045169 · Cited by 191
Self-citationRegularized Distance Metric Learning:Theory and Algorithm
Jin, R., Wang, S., & Zhou, Y. (2009) · Cited by 177
Online multimodal deep similarity learning with application to image retrieval
Wu, P., Hoi, S. C. H., Xia, H., Zhao, P., Wang, D., & Miao, C. (2013) · Proceedings of the 21st ACM international conference on Multimedia · doi:10.1145/2502081.2502112 · Cited by 172
Self-citationScene-Centered Description from Spatial Envelope Properties
Oliva, A., & Torralba, A. B. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36181-2_26 · Cited by 68
Contextual Query Expansion for Image Retrieval
Xie, H., Zhang, Y., Tan, J., Guo, L., & Li, J. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2305909 · Cited by 59
Self-citationMian, A. S., Hu, Y., Hartley, R. I., & Owens, R. A. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2282996 · Cited by 52
Evaluating Open-Universe Face Identification on the Web
Becker, B. C., & Ortiz, E. G. (2013) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2013.133 · Cited by 52
Scalable Similarity Search With Topology Preserving Hashing
Zhang, L., Zhang, Y., Gu, X., Tang, J., & Tian, Q. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2326010 · Cited by 51
Self-citationEnhancing Bag-of-Words Models with Semantics-Preserving Metric Learning
Wu, L., & Hoi, S. C. H. (2011) · IEEE Multimedia · doi:10.1109/mmul.2011.7 · Cited by 31
Self-citationLearning to name faces: a multimodal learning scheme for search-based face annotation
Wang, D., Hoi, S. C. H., Wu, P., Zhu, J., He, Y., & Miao, C. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484040 · Cited by 18
Self-citationA Prior-Free Weighting Scheme for Binary Code Ranking
Zhang, Y., Zhang, L., & Tian, Q. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2306392 · Cited by 17
Self-citationLearning image-to-class distance metric for image classification
Wang, Z., Hu, Y., & Chia, L.-T. (2013) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2438653.2438669 · Cited by 11
Cited by
Object Detection With Deep Learning: A Review
Zhao, Z.-Q., Zheng, P., Xu, S.-T., & Wu, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2876865 · Cited by 5,545
Deep learning in agriculture: A survey
Kamilaris, A., & Prenafeta-Boldú, F. X. (2018) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2018.02.016 · Cited by 5,133
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
“Given the recent successes that deep learning techniques have achieved, the research presented in [166] attempts to evaluate if deep learning can bridge the semantic gap in content-based image retrieval (CBIR).”
BackgroundMethodInfluentialA survey on deep learning techniques for image and video semantic segmentation
Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., & Garcia-Rodriguez, J. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2018.05.018 · Cited by 1,490
CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images
Khan, A. I., Shah, J. L., & Bhat, M. M. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105581 · Cited by 1,312
Metadata
- Authors
- Wan, J., Wang, D., Hoi, S. C. H., Wu, P., Zhu, J., Zhang, Y., & Li, J.
- Venue
- Proceedings of the 22nd ACM international conference on Multimedia
- Publisher
- ACM
- Published
- 2014-11-03
- Pages
- 157-166
- OpenAlex
- W2123229215
- Cited by
- 811
- References
- 57
- FWCI
- 37.66
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.