The Limits of Feedforward Vision: Recurrent Processing Promotes Robust Object Recognition when Objects Are Degraded
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Everyday vision requires robustness to a myriad of environmental factors that degrade stimuli. Foreground clutter can occlude objects of interest, and complex lighting and shadows can decrease the contrast of items. How does the brain recognize visual objects despite these low-quality inputs? On the basis of predictions from a model of object recognition that contains excitatory feedback, we hypothesized that recurrent processing would promote robust recognition when objects were degraded by strengthening bottom–up signals that were weakened because of occlusion and contrast reduction. To test this hypothesis, we used backward masking to interrupt the processing of partially occluded and contrast reduced images during a categorization experiment. As predicted by the model, we found significant interactions between the mask and occlusion and the mask and contrast, such that the recognition of heavily degraded stimuli was differentially impaired by masking. The model provided a close fit of these results in an isomorphic version of the experiment with identical stimuli. The model also provided an intuitive explanation of the interactions between the mask and degradations, indicating that masking interfered specifically with the extensive recurrent processing necessary to amplify and resolve highly degraded inputs, whereas less degraded inputs did not require much amplification and could be rapidly resolved, making them less susceptible to masking. Together, the results of the experiment and the accompanying model simulations illustrate the limits of feedforward vision and suggest that object recognition is better characterized as a highly interactive, dynamic process that depends on the coordination of multiple brain areas.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 2×
“The model of recurrent processing advocated here that is centered around the amplification of signals can be contrasted with “predictive coding” models of recurrent feedback (e.g., Rao, 1999; Rao & Ballard, 1997; see also Friston, 2009, 2010 ).”
MethodRatcliff, R. (1978) · Psychological Review · doi:10.1037/0033-295x.85.2.59 · Cited by 4,513 · cited 1×
“This voting procedure reflects variability because of fixation error and head position during the experiment, which can be used in an aggregate manner during the recognition process (Ratcliff & McKoon, 2008; Bradski & Grossberg, 1995; Ratcliff, 1978).”
MethodSpeed of processing in the human visual system
Thorpe, S., Fize, D., & Marlot, C. (1996) · Nature · doi:10.1038/381520a0 · Cited by 3,538
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · cited 6×
“The results of this filtering are further processed by a chain of V1-like operations—Gabor filtering followed by a spatial " max " operation (e.g., Riesenhuber & Poggio, 1999)—and then used as inputs to the model proper.”
BackgroundMethodInfluentialThe Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks
Ratcliff, R., & McKoon, G. (2008) · Neural Computation · doi:10.1162/neco.2008.12-06-420 · Cited by 3,059 · cited 1×
“This voting procedure reflects variability because of fixation error and head position during the experiment, which can be used in an aggregate manner during the recognition process (Ratcliff & McKoon, 2008; Bradski & Grossberg, 1995; Ratcliff, 1978).”
MethodThe distinct modes of vision offered by feedforward and recurrent processing
Lamme, V. A. F., & Roelfsema, P. R. (2000) · Trends in Neurosciences · doi:10.1016/s0166-2236(00)01657-x · Cited by 2,338 · cited 5×
“Backward masking has been suggested to selectively disrupt feedback (see Di Lollo, Enns, & Rensink, 2000; Lamme & Roelfsema, 2000, for reviews) and has been used in a variety of experiments to dissociate the effects of feedforward and recurrent processing (e.g., Boehler, Schoenfeld, Heinze, & Hopf,…”
BackgroundInfluentialThe free-energy principle: a rough guide to the brain?
Friston, K. (2009) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2009.04.005 · Cited by 1,831
How does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790 · cited 1×
“Visual object recognition in cortex has traditionally been conceptualized as the net computational result of feedforward processing performed in the ventral visual stream (DiCarlo et al., 2012; Serre, Kreiman, et al., 2007; Serre, Oliva, & Poggio, 2007; VanRullen, 2007; Riesenhuber & Poggio, 1999).”
BackgroundTop-down facilitation of visual recognition
Bar, M., Kassam, K. S., Ghuman, A. S., Boshyan, J., Schmid, A. M., Dale, A. M., Hämäläinen, M. S., Marinkovic, K., Schacter, D. L., Rosen, B. R., et al. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507062103 · Cited by 1,489 · cited 1×
“In accordance with this latter view, recent reports have cited recurrent processing effects occurring as early as 100– 150 msec (Koivisto, Railo, Revonsuo, Vanni, & Salminen- Vaparanta, 2011; Roland, 2010; Bar et al., 2006; Foxe & Simpson, 2002; Lamme & Roelfsema, 2000).”
BackgroundHochstein (2002) · doi:10.1016/s0896-6273(02)01091-7 · Cited by 1,397
Cerebral mechanisms of word masking and unconscious repetition priming
Dehaene, S., Naccache, L., Cohen, L., Bihan, D. L., Mangin, J.-F., Poline, J.-B., & Rivière, D. (2001) · Nature Neuroscience · doi:10.1038/89551 · Cited by 1,305 · cited 1×
“Consistent with this view, masking has been shown to cause a decoupling in the functional connectivity (i.e., coactivation) between low-level and high-level visual areas, which has the psychological effect of greatly reduced perceptual visibility (Haynes, Driver, & Rees, 2005; Dehaene et al., 2001).”
BackgroundAn Extension of Box's Results on the Use of the $F$ Distribution in Multivariate Analysis
Geisser, S., & Greenhouse, S. W. (1958) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706545 · Cited by 1,098
Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons
Hupé, J. M., James, A. C., Payne, B. R., Lomber, S. G., Girard, P., & Bullier, J. (1998) · Nature · doi:10.1038/29537 · Cited by 864 · cited 3×
“, V5/MT), which has been shown to strengthen the responses in the lower-level areas and improve discriminability in figure-ground segregation tasks (Hupe et al., 1998; Lamme, Super, & Spekreijse, 1998).”
BackgroundWhole brain resting-state analysis reveals decreased functional connectivity in major depression
Veer, I. M. (2010) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2010.00041 · Cited by 753
Enns, J. T., & Di Lollo, V. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01520-5 · Cited by 684
Flow of activation from V1 to frontal cortex in humans
Foxe, J., & Simpson, G. (2002) · Experimental Brain Research · doi:10.1007/s00221-001-0906-7 · Cited by 670
Effects of Spatial Attention on Contrast Response Functions in Macaque Area V4
Williford, T., & Maunsell, J. H. R. (2006) · Journal of Neurophysiology · doi:10.1152/jn.01207.2005 · Cited by 353
O'Reilly, R. C. (1996) · Neural Computation · doi:10.1162/neco.1996.8.5.895 · Cited by 352 · cited 3×
“As all connections are excitatory, layer activations are controlled using a k-winners-take-all (kWTA) inhibitory competition rule (OʼReilly & Munakata, 2000; OʼReilly, 1996) that ensures only the k most active units remain active over time.”
MethodSelf-citationMasking Disrupts Reentrant Processing in Human Visual Cortex
Fahrenfort, J. J., Scholte, H. S., & Lamme, V. A. F. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.9.1488 · Cited by 344 · cited 3×
“The exact same effect has been reported for the figure-ground segregation tasks, which have been shown to crucially depend on recurrent processing for successful segmentation (Fahrenfort et al., 2007, 2008 ).”
BackgroundDynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex
Rao, R. P. N., & Ballard, D. H. (1997) · Neural Computation · doi:10.1162/neco.1997.9.4.721 · Cited by 290 · cited 2×
“The model of recurrent processing advocated here that is centered around the amplification of signals can be contrasted with “predictive coding” models of recurrent feedback (e.g., Rao, 1999; Rao & Ballard, 1997; see also Friston, 2009, 2010).”
MethodMasking Interrupts Figure-Ground Signals in V1
Lamme, V. A. F., Zipser, K., & Spekreijse, H. (2002) · Journal of Cognitive Neuroscience · doi:10.1162/089892902320474490 · Cited by 285 · cited 1×
“By using backward masking to selectively disrupt influence from recurrent processing mechanisms (Boehler et al., 2008; Fahrenfort et al., 2007, 2008; Lamme et al., 2002), we were able to infer its function during the recognition of degraded visual stimuli.”
The time course of visual processing: Backward masking and natural scene categorisation
Bacon-Macé, N., Macé, M. J.-M., Fabre-Thorpe, M., & Thorpe, S. J. (2005) · Vision Research · doi:10.1016/j.visres.2005.01.004 · Cited by 236
Reconciling predictive coding and biased competition models of cortical function
Spratling, M. W. (2008) · Frontiers in Computational Neuroscience · doi:10.3389/neuro.10.004.2008 · Cited by 153
Visual Selective Behavior Can Be Triggered by a Feed-Forward Process
VanRullen, R., & Koch, C. (2003) · Journal of Cognitive Neuroscience · doi:10.1162/089892903321208141 · Cited by 140 · cited 2×
“…as 100–150 msec on the computations that can take place before categorytuned brain areas become active, providing strong support for models based on a single feedforward sweep of responses (VanRullen & Koch, 2003; Li, VanRullen, Koch, & Perona, 2002; VanRullen & Thorpe, 2001; Thorpe et al., 1996).”
BackgroundMashour, G. A. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.09.009 · Cited by 92
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Human Pose Estimation with Iterative Error Feedback
Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.512 · Cited by 805
“Neuroscience models of the human brain suggest that feedforward connections act as information carriers while numerous feedback connections act as modulators or competitive inhibitors to aid feature grouping [14], figure-ground segregation [16] and object recognition [46].”
BackgroundCognitive computational neuroscience
Kriegeskorte, N., & Douglas, P. K. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0210-5 · Cited by 535
Recurrence is required to capture the representational dynamics of the human visual system
Kietzmann, T. C., Spoerer, C. J., Sörensen, L. K. A., Cichy, R. M., Hauk, O., & Kriegeskorte, N. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905544116 · Cited by 406
Deep neural networks in computational neuroscience
Kietzmann, T. C., McClure, P., & Kriegeskorte, N. (2019) · Oxford Research Encyclopedia of Neuroscience · doi:10.1093/acrefore/9780190264086.013.46 · Cited by 186
Metadata
- Authors
- Wyatte, D., Curran, T., & O'Reilly, R.
- Venue
- Journal of Cognitive Neuroscience
- Publisher
- MIT Press - Journals
- Published
- 2012-11-01
- Volume
- 24
- Issue
- 11
- Pages
- 2248-2261
- ISSN
- 0898-929X (ISSN-L)
- 1530-8898
- OpenAlex
- W2022638524
- Cited by
- 119
- References
- 26
- FWCI
- 3.53
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.