The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Why is Real-World Visual Object Recognition Hard?
Open accessAbstract
Progress in understanding the brain mechanisms underlying vision requires the construction of computational models that not only emulate the brain's anatomy and physiology, but ultimately match its performance on visual tasks. In recent years, "natural" images have become popular in the study of vision and have been used to show apparently impressive progress in building such models. Here, we challenge the use of uncontrolled "natural" images in guiding that progress. In particular, we show that a simple V1-like model--a neuroscientist's "null" model, which should perform poorly at real-world visual object recognition tasks--outperforms state-of-the-art object recognition systems (biologically inspired and otherwise) on a standard, ostensibly natural image recognition test. As a counterpoint, we designed a "simpler" recognition test to better span the real-world variation in object pose, position, and scale, and we show that this test correctly exposes the inadequacy of the V1-like model. Taken together, these results demonstrate that tests based on uncontrolled natural images can be seriously misleading, potentially guiding progress in the wrong direction. Instead, we reexamine what it means for images to be natural and argue for a renewed focus on the core problem of object recognition--real-world image variation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex
Hubel, D. H., & Wiesel, T. N. (1962) · The Journal of Physiology · doi:10.1113/jphysiol.1962.sp006837 · Cited by 13,680
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916
Receptive fields and functional architecture of monkey striate cortex
Hubel, D. H., & Wiesel, T. N. (1968) · The Journal of Physiology · doi:10.1113/jphysiol.1968.sp008455 · Cited by 6,568
LabelMe: A Database and Web-Based Tool for Image Annotation
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0090-8 · Cited by 4,210
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · 2004 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvpr.2004.383 · Cited by 2,502
Natural Image Statistics and Neural Representation
Simoncelli, E. P., & Olshausen, B. A. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.1193 · Cited by 2,460 · cited 1×
“Although controversial ([7,8]), a popular recent approach in the study of vision is the use of ‘‘ natural ’’ images [7,9–12], in part because they ostensibly capture the essence of problems encountered in the real world.”
Background(2007) · CaltechAUTHORS (California Institute of Technology) · Cited by 2,274
The “independent components” of natural scenes are edge filters
Bell, A. J., & Sejnowski, T. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00121-1 · Cited by 2,201
Cited by
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
“In a recent paper, Pinto et al (2008) criticised the use of “natural” images altogether, arguing for the use of synthetic data (e.g. rendered 3D models) for which one has better control over the variability in the data – parameter settings can be sampled at will and annotation is not needed, as…”
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Convolutional networks and applications in vision
LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537907 · Cited by 2,213
“[4] have identified the most appropriate non-linearities and normaliza-”
BackgroundWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153
“A common choice for the filter bank of the first stage is Gabor Wavelets [28, 22, 24].”
BackgroundMethodInfluentialPerformance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Self-citationHow does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790
Self-citationBuilding high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695
“This exper-iment shows that patches, being detected as faces by the OpenCV face detector, account for less than 3% of the 100,000 sampled patches”
BackgroundMethodUsing goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Self-citationMan vs. computer: Benchmarking machine learning algorithms for traffic sign recognition
Stallkamp, J., Schlipsing, M., Salmen, J., & Igel, C. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.016 · Cited by 1,537
Lillicrap, T. P., Santoro, A., Marris, L., Akerman, C. J., & Hinton, G. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0277-3 · Cited by 908
Cadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824
Self-citationOn feature combination for multiclass object classification
Gehler, P., & Nowozin, S. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459169 · Cited by 797
“V1S+ In [16] a population of locally normalized, thresholded Gabor functions spanning a range of orientations and spatial frequencies are derived and advocated as particular simple features.”
BackgroundTraffic sign recognition with multi-scale Convolutional Networks
Sermanet, P., & LeCun, Y. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033589 · Cited by 759
Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition
Sariyanidi, E., Gunes, H., & Cavallaro, A. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2366127 · Cited by 685
Time Series Classification Using Multi-Channels Deep Convolutional Neural Networks
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-08010-9_33 · Cited by 652
Convolutional Learning of Spatio-temporal Features
Taylor, G. W., Fergus, R., LeCun, Y., & Bregler, C. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15567-3_11 · Cited by 629
Deep feature based rice leaf disease identification using support vector machine
Sethy, P. K., Barpanda, N. K., Rath, A. K., & Behera, S. K. (2020) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2020.105527 · Cited by 598
Cruz-Roa, A., Basavanhally, A., González, F., Gilmore, H., Feldman, M., Ganesan, S., Shih, N., Tomaszewski, J., & Madabhushi, A. (2014) · SPIE Proceedings · doi:10.1117/12.2043872 · Cited by 573
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Lindsay, G. W. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01544 · Cited by 557
Diagnosing Error in Object Detectors
Hoiem, D., Chodpathumwan, Y., & Dai, Q. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33712-3_25 · Cited by 520
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0392-5 · Cited by 496
“…synthetic objects with high view variation (scale, position and rotation) on cluttered natural backgrounds (similar to the ones used in Majaj et al., 2015; Pinto et al., 2008), images with occlusion, deformation, missing object-parts, and colored photographs (MS COCO dataset; Lin et al., 2014).”
BackgroundMethodSelf-citationArevalo, J., González, F. A., Ramos-Pollán, R., Oliveira, J. L., & Guevara Lopez, M. A. (2016) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2015.12.014 · Cited by 476
Multi-spectral SIFT for scene category recognition
Brown, M., & Susstrunk, S. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995637 · Cited by 469
Towards a mathematical theory of cortical micro-circuits
George, D., & Hawkins, J. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000532 · Cited by 404
A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction
Wiatowski, T., & Bolcskei, H. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2776228 · Cited by 399
Vehicle Type Classification Using a Semisupervised Convolutional Neural Network
Dong, Z., Wu, Y., Pei, M., & Jia, Y. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2402438 · Cited by 387
Bullot, N. J., & Reber, R. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000489 · Cited by 381
NeuFlow: A runtime reconfigurable dataflow processor for vision
Farabet, C., Martini, B., Corda, B., Akselrod, P., Culurciello, E., & LeCun, Y. (2011) · CVPR 2011 WORKSHOPS · doi:10.1109/cvprw.2011.5981829 · Cited by 369
Rajalingham, R., Issa, E. B., Bashivan, P., Kar, K., Schmidt, K., & DiCarlo, J. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0388-18.2018 · Cited by 358
Self-citation50 Years of object recognition: Directions forward
Andreopoulos, A., & Tsotsos, J. K. (2013) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2013.04.005 · Cited by 347
“[193] indicate, an improvement in evaluation methodologies might entail simulating environments and testing recognition systems on these environments.”
BackgroundSeeing it all: Convolutional network layers map the function of the human visual system
Eickenberg, M., Gramfort, A., Varoquaux, G., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.001 · Cited by 338
“15 edge detectors, this is almost certainly untrue for the mechanism by which mammalian 295 brains process faces due to the strong invariance and selectivity properties with respect to 296 nontrivial transformations that they can undergo (see [42] for a discussion).”
BackgroundExplicit information for category-orthogonal object properties increases along the ventral stream
Hong, H., Yamins, D. L. K., Majaj, N. J., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4247 · Cited by 337
Self-citationDeep learning in robotics: A review of recent research
Pierson, H. A., & Gashler, M. S. (2017) · Advanced Robotics · doi:10.1080/01691864.2017.1365009 · Cited by 331
Sparse Modeling for Image and Vision Processing
Mairal, J. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781680830095 · Cited by 322
From faces to hands: Changing visual input in the first two years
Fausey, C. M., Jayaraman, S., & Smith, L. B. (2016) · Cognition · doi:10.1016/j.cognition.2016.03.005 · Cited by 280
Multiple Kernel Learning for Visual Object Recognition: A Review
(2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.212 · Cited by 274
Learning invariant features through topographic filter maps
Kavukcuoglu, K., Ranzato, M., Fergus, R., & LeCun, Y. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206545 · Cited by 274
Ferguson, M., Ak, R., Lee, Y.-T. T., & Law, K. H. (2018) · Smart and Sustainable Manufacturing Systems · doi:10.1520/ssms20180033 · Cited by 274
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2016) · Frontiers of Computer Science · doi:10.1007/s11704-015-4478-2 · Cited by 273
Unsupervised pre-trained filter learning approach for efficient convolution neural network
Rehman, S. U., Tu, S., Waqas, M., Huang, Y., Rehman, O. U., Ahmad, B., & Ahmad, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.084 · Cited by 270
Real-world visual statistics and infants' first-learned object names
Clerkin, E. M., Hart, E., Rehg, J. M., Yu, C., & Smith, L. B. (2017) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2016.0055 · Cited by 268
A hybrid deep learning CNN–ELM for age and gender classification
Duan, M., Li, K., Yang, C., & Li, K. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.08.062 · Cited by 258
Pinto, N., Doukhan, D., DiCarlo, J. J., & Cox, D. D. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000579 · Cited by 248
Self-citationHuber, L. S., Geirhos, R., & Wichmann, F. A. (2023) · Journal of Vision · doi:10.1167/jov.23.7.4 · Cited by 42
Multiplicative mixing of object identity and image attributes in single inferior temporal neurons
Ratan Murty, N. A., & Arun, S. P. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714287115 · Cited by 15
Metadata
- Authors
- Pinto, N., Cox, D. D., & DiCarlo, J. J.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2008-01-25
- Volume
- 4
- Issue
- 1
- Pages
- e27
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2026942141
- Cited by
- 537
- References
- 32
- FWCI
- 17.69
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.