The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Bilinear Convolutional Neural Networks for Fine-Grained Visual Recognition
Abstract
We present a simple and effective architecture for fine-grained recognition called Bilinear Convolutional Neural Networks (B-CNNs). These networks represent an image as a pooled outer product of features derived from two CNNs and capture localized feature interactions in a translationally invariant manner. B-CNNs are related to orderless texture representations built on deep features but can be trained in an end-to-end manner. Our most accurate model obtains 84.1%, 79.4%, 84.5% and 91.3% per-image accuracy on the Caltech-UCSD birds [66], NABirds [63], FGVC aircraft [42], and Stanford cars [33] dataset respectively and runs at 30 frames-per-second on a NVIDIA Titan X GPU. We then present a systematic analysis of these networks and show that (1) the bilinear features are highly redundant and can be reduced by an order of magnitude in size without significant loss in accuracy, (2) are also effective for other image classification tasks such as texture and scene recognition, and (3) can be trained from scratch on the ImageNet dataset offering consistent improvements over the baseline architecture. Finally, we present visualizations of these models on various datasets using top activations of neural units and gradient-based inversion techniques. The source code for the complete system is available at http://vis-www.cs.umass.edu/bcnn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
(2016) · Cited by 31,590
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Schölkopf, B., & Smola, A. J. (2001) · The MIT Press eBooks · doi:10.7551/mitpress/4175.001.0001 · Cited by 11,703
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
Visual categorization with bags of keypoints
Csurka, G. (2004) · Cited by 4,379
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Caltech-UCSD Birds-200-2011 Dataset
Wah, C. (2024) · CaltechAUTHORS (California Institute of Technology) · doi:10.57702/i9oeewf6 · Cited by 3,525
3D Object Representations for Fine-Grained Categorization
Krause, J., Stark, M., Deng, J., & Fei-Fei, L. (2013) · 2013 IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.77 · Cited by 3,496
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Aggregating local descriptors into a compact image representation
Jegou, H., Douze, M., Schmid, C., & Perez, P. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540039 · Cited by 2,692
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.213 · Cited by 2,684
Vlfeat: an open and portable library of computer vision algorithms
Vedaldi, A., & Fulkerson, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874249 · Cited by 2,664
Improving the Fisher Kernel for Large-Scale Image Classification
Perronnin, F., Sánchez, J., & Mensink, T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_11 · Cited by 2,588
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition
Arandjelovic, R., Gronat, P., Torii, A., Pajdla, T., & Sivic, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.572 · Cited by 2,584
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438
Self-citationLearning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.848998 · Cited by 2,225
Bilinear CNN Models for Fine-Grained Visual Recognition
Lin, T.-Y., RoyChowdhury, A., & Maji, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.170 · Cited by 2,135
Self-citationA Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients
Portilla, J., & Simoncelli, E. P. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026553619983 · Cited by 1,806
NetVLAD: CNN architecture for weakly supervised place recognitionPreprint
Arandjelović, R., Gronát, P., Torii, A., Pajdla, T., & Šivic, J. (2015) · arXiv (Cornell University) · Cited by 1,790
Fisher Kernels on Visual Vocabularies for Image Categorization
Perronnin, F., & Dance, C. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383266 · Cited by 1,641
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582
Representing and Recognizing the Visual Appearance of Materials Using Three-Dimensional Textons
Leung, T., & Malik, J. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011126920638 · Cited by 1,498 · cited 1×
“Early work [37] represents the texture by computing the statistics of linear filter-bank responses (e.”
BackgroundQuattoni, A., & Torralba, A. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206537 · Cited by 1,463
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461
Part-Based R-CNNs for Fine-Grained Category Detection
Zhang, N., Donahue, J., Girshick, R., & Darrell, T. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_54 · Cited by 1,150
Fine-Grained Visual Classification of Aircraft
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., & Vedaldi, A. (2013) · INRIA a CCSD electronic archive server · doi:10.48550/arxiv.1306.5151 · Cited by 1,084
Self-citationMulti-scale Orderless Pooling of Deep Convolutional Activation Features
Gong, Y., Wang, L., Guo, R., & Lazebnik, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_26 · Cited by 1,030
Gao, Y., Beijbom, O., Zhang, N., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.41 · Cited by 871
Separating Style and Content with Bilinear Models
Tenenbaum, J. B., & Freeman, W. T. (2000) · Neural Computation · doi:10.1162/089976600300015349 · Cited by 840 · cited 2×
“Bilinear models were used by Tanenbaum and Freeman [48] to model twofactor variations such as “style” and “content” for images.”
MethodMultiple Object Recognition with Visual Attention
Ba, J., Mnih, V., & Kavukcuoglu, K. (2014) · arXiv · doi:10.48550/arxiv.1412.7755 · Cited by 808
CNN Features off-the-shelf: an Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · arXiv · doi:10.48550/arxiv.1403.6382 · Cited by 607
Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., & Belongie, S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298658 · Cited by 592
Bird Species Categorization Using Pose Normalized Deep Convolutional Nets
Branson, S., Horn, G. V., Belongie, S., & Perona, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2952 · Cited by 559
Visualizing Deep Convolutional Neural Networks Using Natural Pre-images
Mahendran, A., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0911-8 · Cited by 496
Fine-grained recognition without part annotations
Krause, J., Jin, H., Yang, J., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299194 · Cited by 466
Semantic Segmentation with Second-Order Pooling
Carreira, J., Caseiro, R., Batista, J., & Sminchisescu, C. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33786-4_32 · Cited by 458
Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks
Simon, M., & Rodner, E. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.136 · Cited by 394
Deep Filter Banks for Texture Recognition, Description, and Segmentation
Cimpoi, M., Maji, S., Kokkinos, I., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-015-0872-3 · Cited by 335
Self-citationDescribing people: A poselet-based approach to attribute classification
Bourdev, L., Maji, S., & Malik, J. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126413 · Cited by 335
Self-citationPicking Deep Filter Responses for Fine-Grained Image Recognition
Zhang, X., Xiong, H., Zhou, W., Lin, W., & Tian, Q. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.128 · Cited by 321
Fast and scalable polynomial kernels via explicit feature maps
Pham, N., & Pagh, R. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487591 · Cited by 306
SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-Grained Recognition
Zhang, H., Xu, T., Elhoseiny, M., Huang, X., Zhang, S., Elgammal, A., & Metaxas, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.129 · Cited by 297
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
Krause, J., Sapp, B., Howard, A., Zhou, H., Toshev, A., Duerig, T., Philbin, J., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_19 · Cited by 296
Class-specific material categorisation
Caputo, B., Hayman, E., & Mallikarjuna, P. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.54 · Cited by 286
Matrix Backpropagation for Deep Networks with Structured Layers
Ionescu, C., Vantzos, O., & Sminchisescu, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.339 · Cited by 281
Birdlets: Subordinate categorization using volumetric primitives and pose-normalized appearance
Farrell, R. J., Oza, O., Zhang, N., Morariu, V. I., Darrell, T., & Davis, L. S. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126238 · Cited by 196
Learning to segment moving objects in videos
Fragkiadaki, K., Arbeláez, P., Felsen, P., & Malik, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299035 · Cited by 195
Visualizing and Understanding Deep Texture Representations
Lin, T., & Maji, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.305 · Cited by 151
Self-citationBilinear classifiers for visual recognition
Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2009) · doi:10.13016/m2uvdv-qzxg · Cited by 141
Boosted Convolutional Neural Networks
Moghimi, M. K., Belongie, S., Saberian, M., Yang, J. W., Vasconcelos, N. M., & Li, L.-J. (2016) · British Machine Vision Conference (BMVC) · doi:10.5244/c.30.24 · Cited by 136
Mining Discriminative Triplets of Patches for Fine-Grained Classification
Wang, Y., Choi, J., Morariu, V. I., & Davis, L. S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.131 · Cited by 129
Pose pooling kernels for sub-category recognition
Zhang, N., Farrell, R. J., & Darrell, T. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248364 · Cited by 124
A Compact and Discriminative Face Track Descriptor
Parkhi, O., Simonyan, K., Vedaldi, A., et al. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.219 · Cited by 120
Revisiting the Fisher vector for fine-grained classification
Gosselin, P.-H., Murray, N., Jeǵou, H., & Perronnin, F. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.06.011 · Cited by 119
Gatys, L. A., Ecker, A. S., & Bethge, M. (2015) · arXiv (Cornell University) · doi:10.12751/nncn.bc2015.0220 · Cited by 104
Cross-Convolutional-Layer Pooling for Image Recognition
Liu, L., Shen, C., & van den Hengel, A. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2637921 · Cited by 95
Fine-grained pose prediction, normalization, and recognitionPreprint
Zhang, N., Shelhamer, E., Gao, Y., & Darrell, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07063 · Cited by 49
Texture Synthesis Using Shallow Convolutional Networks with Random FiltersPreprint
Ustyuzhaninov, I., Brendel, W., Gatys, L. A., & Bethge, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00021 · Cited by 37
Cited by
Machine Learning Methods for Histopathological Image Analysis
Komura, D., & Ishikawa, S. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.01.001 · Cited by 937
Mutual Attention Inception Network for Remote Sensing Visual Question Answering
Zheng, X., Wang, B., Du, X., & Lu, X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2021.3079918 · Cited by 159
Metadata
- Authors
- Lin, T.-Y., RoyChowdhury, A., & Maji, S.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2018-06-01
- Volume
- 40
- Issue
- 6
- Pages
- 1309-1322
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2731821979
- Cited by
- 328
- References
- 73
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
