Cited by 324Multisensory perception and integrationVisual perception and processing mechanismsTactile and Sensory Interactions

Decoding sound and imagery content in early visual cortex

Open access
Vetter, Petra, Smith, Fraser W., Muckli, Lars·2014·Current Biology

Abstract

Human early visual cortex was traditionally thought to process simple visual features such as orientation, contrast, and spatial frequency via feedforward input from the lateral geniculate nucleus (e.g., [1]). However, the role of nonretinal influence on early visual cortex is so far insufficiently investigated despite much evidence that feedback connections greatly outnumber feedforward connections [2-5]. Here, we explored in five fMRI experiments how information originating from audition and imagery affects the brain activity patterns in early visual cortex in the absence of any feedforward visual stimulation. We show that category-specific information from both complex natural sounds and imagery can be read out from early visual cortex activity in blindfolded participants. The coding of nonretinal information in the activity patterns of early visual cortex is common across actual auditory perception and imagery and may be mediated by higher-level multisensory areas. Furthermore, this coding is robust to mild manipulations of attention and working memory but affected by orthogonal, cognitively demanding visuospatial processing. Crucially, the information fed down to early visual cortex is category specific and generalizes to sound exemplars of the same category, providing evidence for abstract information feedback rather than precise pictorial feedback. Our results suggest that early visual cortex receives nonretinal input from other brain areas when it is generated by auditory perception and/or imagery, and this input carries common abstract information. Our findings are compatible with feedback of predictive information to the earliest visual input level (e.g., [6]), in line with predictive coding models [7-10].

AI snapshot

Written from the paper’s full text.

TL;DR

Using fMRI and multivariate pattern analysis in blindfolded participants, this paper shows that category-specific information from natural sounds and auditory imagery can be decoded from early visual cortex activity without feedforward visual stimulation. The feedback appears abstract/categorical rather than pictorial, is robust to mild attention/working-memory interference, and is disrupted by demanding visuospatial processing.

Key findings

  • In experiment 1, a linear SVM discriminated three natural sounds (bird singing, traffic noise, talking crowd) from early visual cortex, particularly V2 and V3, at about 42%; classification also succeeded in auditory cortex but not motor cortex.
  • In experiment 2, imagined sounds were decoded from V1 and auditory cortex, and cross-classification between sound perception and imagery succeeded in V1 and V2, indicating shared activity patterns across real and imagined sounds.
  • In experiment 3, decoding during natural sound stimulation remained successful mainly in peripheral regions and consistently in V2 despite an orthogonal auditory working-memory task; in experiment 4, a demanding visuospatial imagery task left only residual above-chance classification in far-peripheral V2.
  • In experiment 5, “human” versus “inanimate” sound categories were classified in several early visual areas for eight out of nine exemplar combinations, and cross-classification across different exemplars within a category succeeded in V2 and V3 in two out of three combinations, supporting abstract/categorical feedback rather than precise pictorial reinstatement.

Methods

  • Five fMRI experiments with blindfolded participants omitted visual stimulation while subjects listened to natural sounds, imagined sounds from word cues, or performed orthogonal cognitive tasks during sound stimulation.
  • Activity patterns were extracted from retinotopically mapped V1, V2, and V3 and analyzed with multivariate pattern classification, primarily a linear support vector machine; whole-brain searchlight analyses identified other regions carrying sound/imagery content.
  • Univariate GLM analyses tested whether decoding was driven by positive activation or deactivation; cross-classification tested generalization between perception and imagery and across sound exemplars within categories.

Limitations

  • The paper states that classification accuracies were relatively low, describing the results as proof of principle and robust across subject and stimulus samples.
  • The paper notes that without feedforward stimulation, it is difficult to study the functional role of nonretinal feedback in actual visual perception.
  • The paper states that whether the proposed pathways play a causal role in inducing differential activity patterns remains to be investigated; the provided text also does not report exact sample sizes per experiment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.cub.2014.04.020

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 28 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Cross-classification between different sound exemplars within a category succeeds in V2 and V3 in two out of three combinations.

    “Crucially, cross-classification succeeded in V2 and V3 in two out of three combinations, i.e., training the classifier for the pair “traffic 1” versus “people 1” lead to successful classification of “traffic 2” versus “people 2,” and training the classifier for the pair “traffic 2” versus “people 2” lead to successful classification of “airplane” versus “kids” (Figure 3H; Table S1).”
    Quote in the paperSubstantial claim
  • Findingown data

    Cross-classification between auditory perception and imagery succeeds in V1 and V2.

    “Cross-classification succeeded in V1 and V2 (Figure 2G).”
    Quote in the paperSubstantial claim
  • Findingown data

    In auditory cortex, cross-classification between auditory perception and imagery does not succeed.

    “In auditory cortex, cross-classification did not succeed, indicating that activity patterns induced by feedforward auditory stimulation are coded differently than those induced by feedback through auditory imagery.”
    Quote in the paperSubstantial claim
  • Findingown data

    In experiments 3 and 4, the secondary tasks activate early visual areas consistent with an engagement of object attention.

    “In experiments 3 and 4, the secondary tasks activated early visual areas consistent with an engagement of object attention.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In experiments 1 and 2, real and imagined sounds can be decoded in parts of the precuneus and in posterior superior temporal sulcus (pSTS).

    “In experiments 1 and 2, real and imagined sounds could be decoded in parts of the precuneus and in posterior superior temporal sulcus (pSTS) (see overlapping regions in Figure 4).”
    Quote in the paperSubstantial claim
  • May combine two claims. 24
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 5
  • May combine two claims.

    Findingown data

    Listening to sounds in the absence of visual stimulation elicits no positive activation in early visual areas but instead elicits a weak deactivation.

    “Even at very liberal thresholds (p < 0.05 uncorrected; Figure S3), listening to sounds in the absence of visual stimulation elicited no positive activation in early visual areas but instead elicited a weak deactivation, consistent with previous findings (e.g.,[16]) and in contrast to classical findings for visual mental imagery [17, 18].”
    Quote in the paperSubstantial claim
  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Imagined sounds can be decoded from early visual cortex (V1) and auditory cortex, with successful discrimination in foveal and peripheral areas of early visual cortex but not far periphery, even in the absence of both visual and auditory stimulation.

    “Successful discrimination of imagined sounds was observed in both foveal and peripheral areas of early visual cortex (but not far periphery), in V1 and auditory cortex (Figures 2E and 2F; classification of real sounds replicated the results of experiment 1, cf. Figure S1B).”
    • Also stated in: § Decoding of Sound and Imagery Content in Early Visual Cortex · hedged
      “Therefore, even in the absence of both visual and auditory stimulation, the contents of mental imagery could be decoded from both V1 and auditory cortex.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Imagery (experiment 2) elicits no positive activity but exhibits weak deactivations in both early visual and auditory cortices.

    “Imagery (experiment 2) elicited no positive activity but exhibited weak deactivations in both early visual and auditory cortices.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    In experiment 4, the classifier failed to discriminate the three natural sounds in most of early visual cortex, but residual above-chance classification remained in the far periphery of V2 despite orthogonal engagement of attentionally demanding active visuospatial processing.

    “Although the classifier failed to discriminate the three natural sounds in most of early visual cortex, residual above-chance classification remained in the far periphery of V2 (Figures 3E and 3F) despite the orthogonal engagement of attentionally demanding active visuospatial processing.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Higher visual areas project back to V1 the associated color of grayscale objects or the predicted motion path of an apparent motion illusion.

    “Moreover, higher visual areas project back to V1 the associated color of grayscale objects [36] or the predicted motion path of an apparent motion illusion [37, 38].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [36] Bannert, 2013, Decoding the yellow of a gray banana, Curr. Biol. source

      Not checked in the free run.

    • cites [37] Muckli, 2005, Primary visual cortex activity along the apparent-motion trace reflects illusory perception, PLoS Biol. source

      Not checked in the free run.

    • cites [38] Vetter, 2013, TMS over V5 disrupts motion prediction, Cereb. Cortex

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The findings are compatible with feedback of predictive information to the earliest visual input level, in line with predictive coding models.

    “Our findings are compatible with feedback of predictive information to the earliest visual input level (e.g., [6]), in line with predictive coding models [7–10].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [6] Mumford, 1992, On the computational architecture of the neocortex. II. The role of cortico-cortical loops, Biol. Cybern. source

      Not checked in the free run.

    • cites [7] Friston, 2010, The free-energy principle: a unified brain theory?, Nat. Rev. Neurosci. source

      Not checked in the free run.

    • cites [8] Clark, 2013, Whatever next? Predictive brains, situated agents, and the future of cognitive science, Behav. Brain Sci. source

      Not checked in the free run.

    • cites [9] Bar, 2007, The proactive brain: using analogies and associations to generate predictions, Trends Cogn. Sci. source

      Not checked in the free run.

    • cites [10] Bastos, 2012, Canonical microcircuits for predictive coding, Neuron source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The information fed down to early visual cortex is category specific and generalizes to sound exemplars of the same category, providing evidence for abstract information feedback rather than precise pictorial feedback.

    “Crucially, the information fed down to early visual cortex is category specific and generalizes to sound exemplars of the same category, providing evidence for abstract information feedback rather than precise pictorial feedback.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    This coding is robust to mild manipulations of attention and working memory but affected by orthogonal, cognitively demanding visuospatial processing.

    “Furthermore, this coding is robust to mild manipulations of attention and working memory but affected by orthogonal, cognitively demanding visuospatial processing.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    Both sounds and imagery cues induce similar activity patterns in early visual cortex, and feedback information is coded consistently across imagery and auditory perception.

    “This demonstrates that both sounds and imagery cues induced similar activity patterns in early visual cortex and that feedback information is coded consistently across imagery and auditory perception.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    Despite relatively low classification accuracies, the series of experiments replicated the successful decoding of sounds in early visual areas several times, demonstrating proof of principle and robustness across different subject and stimulus samples.

    “Note that despite relatively low classification accuracies, our series of experiments replicated the successful decoding of sounds in early visual areas several times, demonstrating proof of principle and the robustness of our results across different subject and stimulus samples.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    The content-specific information from sounds, when they are heard and/or imagined, is relayed from auditory cortex to early visual cortex via pSTS and precuneus, eliciting differential activity patterns in both of these regions.

    “In the context of our findings, we suggest that the content-specific information from sounds, when they are heard and/or imagined, is relayed from auditory cortex to early visual cortex via pSTS and precuneus, eliciting differential activity patterns in both of these regions.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    The coding of nonretinal information in the activity patterns of early visual cortex is common across actual auditory perception and imagery and may be mediated by higher-level multisensory areas.

    “The coding of nonretinal information in the activity patterns of early visual cortex is common across actual auditory perception and imagery and may be mediated by higher-level multisensory areas.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    Early visual cortex receives nonretinal input from other brain areas when it is generated by auditory perception and/or imagery, and this input carries common abstract information.

    “Our results suggest that early visual cortex receives nonretinal input from other brain areas when it is generated by auditory perception and/or imagery, and this input carries common abstract information.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The precuneus has been identified as an area responding to both visual and auditory stimuli and possibly serving as an audiovisual convergence area.

    “The precuneus has been identified as an area responding to both visual and auditory stimuli and possibly serving as an audiovisual convergence area [25].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25] Hertz, 2010, Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis, Neuroimage source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The information that is fed down to early visual cortex is not only content specific but also category specific, i.e., related to the information shared by sound exemplars of the same category.

    “The results of experiment 5 suggest that the information that is fed down to early visual cortex is not only content specific but also category specific, i.e., related to the information shared by sound exemplars of the same category.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    The successful decoding in experiment 1 was not specific to the first sound exemplars used and could not be caused by differential low-level acoustic features of the sounds.

    “The findings of experiment 5 furthermore demonstrate that the successful decoding in experiment 1 was not specific to the first sound exemplars we used and could not be caused by differential low-level acoustic features of the sounds (e.g., frequency distribution).”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    Information feedback is unlikely to be caused by an exact pictorial representation but instead contains abstract and possibly semantic information.

    “This suggests that information feedback is unlikely to be caused by an exact pictorial representation but instead contains abstract and possibly semantic information.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Premisecitation

    There is evidence for multisensory integration on the subcortical level and for direct anatomical connections between early auditory and early visual areas, mostly reaching peripheral regions.

    “Apart from the route via multisensory areas, there is evidence for multisensory integration on the subcortical level [27] and for direct anatomical connections between early auditory and early visual areas [12, 28, 29], mostly reaching peripheral regions [12–14], consistent with both our eccentricity and searchlight results.”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [27] van den Brink, 2013, Subcortical, modality-specific pathways contribute to multisensory processing in humans, Cereb. Cortex

      Not checked in the free run.

    • cites [12] Eckert, 2008, A cross-modal system linking primary auditory and visual cortices: evidence from intrinsic fMRI connectivity analysis, Hum. Brain Mapp. source

      Not checked in the free run.

    • cites [28] Beer, 2013, Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing, Front Integr Neurosci source

      Not checked in the free run.

    • cites [29] Beer, 2011, Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex, Exp. Brain Res. source

      Not checked in the free run.

    • cites [13] Rockland, 2003, Multisensory convergence in calcarine visual areas in macaque monkey, Int. J. Psychophysiol. source

      Not checked in the free run.

    • cites [14] Cate, 2009, Auditory attention activates peripheral visual cortex, PLoS ONE source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The results were not caused by an unspecific attention effect or a simple reactivation of early visual cortex due to pictorial visual mental imagery.

    “This also means that our results were not caused by an unspecific attention effect or a simple reactivation of early visual cortex due to pictorial visual mental imagery.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationown data

    The information arriving in early visual cortex is categorical and independent of its exact source.

    “Our results demonstrate that the information arriving in early visual cortex is categorical and independent of its exact source.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Activity patterns in nonstimulated parts of early visual cortex contain content-specific information from the surrounding visual context, from objects presented in the periphery, and from visual stimuli solely held in working memory rather than being actively perceived.

    “For example, activity patterns in nonstimulated parts of early visual cortex contain content-specific information from the surrounding visual context [31, 32], from objects presented in the periphery [33], and from visual stimuli solely held in working memory rather than being actively perceived [34, 35].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [31] Smith, 2010, Nonstimulated early visual areas carry information about surrounding context, Proc. Natl. Acad. Sci. USA source

      Not checked in the free run.

    • cites [32] Ban, 2013, Topographic representation of an occluded object and the effects of spatiotemporal context in human early visual areas, J. Neurosci. source

      Not checked in the free run.

    • cites [33] Williams, 2008, Feedback of visual object information to foveal retinotopic cortex, Nat. Neurosci. source

      Not checked in the free run.

    • cites [34] Harrison, 2009, Decoding reveals the contents of visual working memory in early visual areas, Nature source

      Not checked in the free run.

    • cites [35] Albers, 2013, Shared representations for working memory and mental imagery in early visual cortex, Curr. Biol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Hippocampal projections to peripheral early visual regions have been demonstrated in the context of boundary extension for scene processing.

    “Also, hippocampal projections to peripheral early visual regions have been demonstrated in the context of boundary extension for scene processing [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] Chadwick, 2013, The hippocampus extrapolates beyond the view in scenes: an fMRI study of boundary extension, Cortex source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    pSTS is implicated in audiovisual integration and has been shown to feed down information to primary visual and auditory cortices.

    “pSTS is implicated in audiovisual integration and has been shown to feed down information to primary visual and auditory cortices [26].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26] Naumer, 2011, Investigating human audio-visual object perception with a combination of hypothesis-generating and hypothesis-testing fMRI analysis tools, Exp. Brain Res. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The information contained in these activity patterns is generalizable across different sound exemplars within a category, demonstrating that sounds trigger shared categorical information transfer to early visual cortex rather than a fine-grained pictorial representation.

    “That is, the information contained in these activity patterns is generalizable across different sound exemplars within a category, demonstrating that sounds trigger shared categorical information transfer to early visual cortex rather than a fine-grained pictorial representation.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Previous accounts suggested that prediction and mental imagery may involve overlapping brain mechanisms, and mental imagery might have evolved from predictive brain mechanisms.

    “In fact, previous accounts suggested that prediction and mental imagery may involve overlapping brain mechanisms [6, 43, 44], and mental imagery might have evolved from predictive brain mechanisms.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [6] Mumford, 1992, On the computational architecture of the neocortex. II. The role of cortico-cortical loops, Biol. Cybern. source

      Not checked in the free run.

    • cites [43] Maguire, 2013, The hippocampus: a manifesto for change, J. Exp. Psychol. Gen. source

      Not checked in the free run.

    • cites [44] Moulton, 2009, Imagining predictions: mental imagery as mental emulation, Philos. Trans. R. Soc. Lond. B Biol. Sci. source

      Not checked in the free run.

25 of 25
  • The paper relies on this without support.

    The word cues “forest,” “traffic,” and “people” reliably elicit the intended mental imagery of the corresponding sounds and natural scenes in subjects.

    “Subjects were instructed to engage in mental imagery of the sounds and a corresponding natural scene.”
    “runs in which subjects solely imagined the sounds upon hearing the word cues “forest,” “traffic,” or “people””
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Functional magnetic resonance imaging has shown that feedback from higher- to lower-level sensory cortices carries spatiotemporally precise and context-specific predictions.

    “Functional magnetic resonance imaging (fMRI) has shown that feedback from higher- to lower-level sensory cortices carries spatiotemporally precise and context-specific predictions (25–29).”

    From The Predictive Coding Account of Psychosis (2018) · cites this paper as [28]

    The Discussion (page not provided) interprets category-specific activity in early visual cortex as possible predictive feedback from higher areas. The Abstract (page not provided) describes the information as abstract rather than precise pictorial feedback; the paper does not establish that the predictions are spatiotemporally precise, and presents their predictive role as an interpretation rather than a demonstrated result. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

1 of the 52 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Vetter, P., Smith, F. W., & Muckli, L.
Affiliations
  • University of Geneva (CH)
  • University of Glasgow (GB)
Corresponding author
  • Vetter, Petra
  • Muckli, Lars
Venue
Current Biology
Publisher
Elsevier BV
Published
2014-06
Volume
24
Issue
11
Pages
1256-1262
ISSN
0960-9822 (ISSN-L)
OpenAlex
W2016034937
Cited by
323
References
52
Citations by year
2026: 12 · 2025: 23 · 2024: 38 · 2023: 75 · 2022: 20 · 2021: 21 · 2020: 21 · 2019: 25 · 2018: 18 · 2017: 19 · 2016: 20 · 2015: 25 · 2014: 6
FWCI
32.47
Type
journal article
Language
en
Open access
Locations
  • Current Biology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
  • Archive ouverte UNIGE (University of Geneva) (acceptedVersion, other-oa)
  • UEA Digital Repository (University of East Anglia) (submittedVersion, cc-by)
  • Archive ouverte UNIGE (University of Geneva) (submittedVersion, cc-by)
Funders
  • College of Science and Engineering, University of Minnesota
  • Biotechnology and Biological Sciences Research Council (BB/G005044/1)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
early visual cortex, auditory imagery, auditory perception, top-down feedback, multisensory integration, predictive coding, abstract information, visuospatial processing, attention, working memory, blindfolded participants
MeSH terms
Acoustic Stimulation, Auditory Perception, Humans, Magnetic Resonance Imaging, Memory, Short-Term, Photic Stimulation, Visual Cortex, Visual Perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review