Cited by 324Multisensory perception and integrationVisual perception and processing mechanismsTactile and Sensory Interactions

The DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index; none of the 28 sources its claims cite is checked for support yet.

Decoding Sound and Imagery Content in Early Visual Cortex

Open access

Abstract

Human early visual cortex was traditionally thought to process simple visual features such as orientation, contrast, and spatial frequency via feedforward input from the lateral geniculate nucleus (e.g., [1]). However, the role of nonretinal influence on early visual cortex is so far insufficiently investigated despite much evidence that feedback connections greatly outnumber feedforward connections [2-5]. Here, we explored in five fMRI experiments how information originating from audition and imagery affects the brain activity patterns in early visual cortex in the absence of any feedforward visual stimulation. We show that category-specific information from both complex natural sounds and imagery can be read out from early visual cortex activity in blindfolded participants. The coding of nonretinal information in the activity patterns of early visual cortex is common across actual auditory perception and imagery and may be mediated by higher-level multisensory areas. Furthermore, this coding is robust to mild manipulations of attention and working memory but affected by orthogonal, cognitively demanding visuospatial processing. Crucially, the information fed down to early visual cortex is category specific and generalizes to sound exemplars of the same category, providing evidence for abstract information feedback rather than precise pictorial feedback. Our results suggest that early visual cortex receives nonretinal input from other brain areas when it is generated by auditory perception and/or imagery, and this input carries common abstract information. Our findings are compatible with feedback of predictive information to the earliest visual input level (e.g., [6]), in line with predictive coding models [7-10].

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

Using fMRI and multivariate pattern analysis in blindfolded participants, this paper shows that category-specific information from natural sounds and auditory imagery can be decoded from early visual cortex activity without feedforward visual stimulation. The feedback appears abstract/categorical rather than pictorial, is robust to mild attention/working-memory interference, and is disrupted by demanding visuospatial processing.

Key findings

  • In experiment 1, a linear SVM discriminated three natural sounds (bird singing, traffic noise, talking crowd) from early visual cortex, particularly V2 and V3, at about 42%; classification also succeeded in auditory cortex but not motor cortex.
  • In experiment 2, imagined sounds were decoded from V1 and auditory cortex, and cross-classification between sound perception and imagery succeeded in V1 and V2, indicating shared activity patterns across real and imagined sounds.
  • In experiment 3, decoding during natural sound stimulation remained successful mainly in peripheral regions and consistently in V2 despite an orthogonal auditory working-memory task; in experiment 4, a demanding visuospatial imagery task left only residual above-chance classification in far-peripheral V2.
  • In experiment 5, “human” versus “inanimate” sound categories were classified in several early visual areas for eight out of nine exemplar combinations, and cross-classification across different exemplars within a category succeeded in V2 and V3 in two out of three combinations, supporting abstract/categorical feedback rather than precise pictorial reinstatement.

Methods

  • Five fMRI experiments with blindfolded participants omitted visual stimulation while subjects listened to natural sounds, imagined sounds from word cues, or performed orthogonal cognitive tasks during sound stimulation.
  • Activity patterns were extracted from retinotopically mapped V1, V2, and V3 and analyzed with multivariate pattern classification, primarily a linear support vector machine; whole-brain searchlight analyses identified other regions carrying sound/imagery content.
  • Univariate GLM analyses tested whether decoding was driven by positive activation or deactivation; cross-classification tested generalization between perception and imagery and across sound exemplars within categories.

Limitations

  • The paper states that classification accuracies were relatively low, describing the results as proof of principle and robust across subject and stimulus samples.
  • The paper notes that without feedforward stimulation, it is difficult to study the functional role of nonretinal feedback in actual visual perception.
  • The paper states that whether the proposed pathways play a causal role in inducing differential activity patterns remains to be investigated; the provided text also does not report exact sample sizes per experiment.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.cub.2014.04.020

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Experimental Setup and ROI Definition In each of the five experiments, ten healthy subjects were scanned with solely auditory stimulation in the absence of visual stimulation. Subjects wore a blindfold and were instructed to keep their eyes closed at all times, and room lights were switched off. In a separate session, retinotopic mapping was performed for all subjects in all experiments to define early visual areas V1, V2, and V3. We show probability maps from the retinotopic mapping data of experiment 1 (n = 10) as derived from functionally informed cortex-based alignment on a flattened Montreal Neurological Institute (MNI) template. White lines indicate mean eccentricity boundaries. Sound-induced blood-oxygen-level-dependent activation patterns from these regions of interest (ROIs) were fed into a multivariate pattern analysis.
Figure 1 Experimental Setup and ROI Definition In each of the five experiments, ten healthy subjects were scanned with solely auditory stimulation in the absence of visual stimulation. Subjects wore a blindfold and were instructed to keep their eyes closed at all times, and room lights were switched off. In a separate session, retinotopic mapping was performed for all subjects in all experiments to define early visual areas V1, V2, and V3. We show probability maps from the retinotopic mapping data of experiment 1 (n = 10) as derived from functionally informed cortex-based alignment on a flattened Montreal Neurological Institute (MNI) template. White lines indicate mean eccentricity boundaries. Sound-induced blood-oxygen-level-dependent activation patterns from these regions of interest (ROIs) were fed into a multivariate pattern analysis.CC BY · PubMed Central
Figure 2 Experimental Design and Classification Results of Experiments 1 and 2 (A) In experiment 1, subjects listened to one of three different natural sounds, interleaved with silence (apart from scanner noise). (B) Mean classification accuracy of the classifier distinguishing the three natural sounds in the different ROIs. Early visual cortex (EVC) contains V1, ventral V2, dorsal V2, ventral V3, and dorsal V3. Chance level is at one out of three. Error bars indicate SEM. All p values were derived from a permutation analysis (see Supplemental Experimental Procedures). Results for V1, V2, and V3 are single threshold corrected. ∗p < 0.05, ∗∗p = 0.001. For significant results, confusion matrices are displayed underneath the graphs to show that classification was not solely successful due to the difference between the activity patterns evoked by one sound versus all other patterns. Columns of the confusion matrices indicate the sound displayed (F, forest; p, people; T, traffic), and rows indicate which sound the classifier predicted. Classifier performance is represented by color hues, with warm colors for above-chance classification and cold colors for below-chance classification. (C) Mean classification accuracies for all visual ROIs divided into three eccentricities (fovea, periphery, and far periphery). ∗p < 0.05 (uncorrected), ∗∗p < 0.05 (false discovery rate corrected). (D) In experiment 2, subjects received a word cue to imagine the sounds and the associated visual scene. Four runs with word cues were alternated with four runs of actual sound stimulation. (E) Classification results are shown for imagined sounds. ∗p < 0.05, ∗∗p = 0.001. (F) Mean classification accuracies for different eccentricities of the visual ROIs. ∗p < 0.05 (uncorrected), ∗∗p < 0.05 (false discovery rate corrected). (G) Cross-classification results of experiment 2. The classifier was trained on real sounds and tested on imagined sounds and vice versa, and results were averaged. ∗p < 0.05, ∗∗p = 0.001.
Figure 2 Experimental Design and Classification Results of Experiments 1 and 2 (A) In experiment 1, subjects listened to one of three different natural sounds, interleaved with silence (apart from scanner noise). (B) Mean classification accuracy of the classifier distinguishing the three natural sounds in the different ROIs. Early visual cortex (EVC) contains V1, ventral V2, dorsal V2, ventral V3, and dorsal V3. Chance level is at one out of three. Error bars indicate SEM. All p values were derived from a permutation analysis (see Supplemental Experimental Procedures). Results for V1, V2, and V3 are single threshold corrected. ∗p < 0.05, ∗∗p = 0.001. For significant results, confusion matrices are displayed underneath the graphs to show that classification was not solely successful due to the difference between the activity patterns evoked by one sound versus all other patterns. Columns of the confusion matrices indicate the sound displayed (F, forest; p, people; T, traffic), and rows indicate which sound the classifier predicted. Classifier performance is represented by color hues, with warm colors for above-chance classification and cold colors for below-chance classification. (C) Mean classification accuracies for all visual ROIs divided into three eccentricities (fovea, periphery, and far periphery). ∗p < 0.05 (uncorrected), ∗∗p < 0.05 (false discovery rate corrected). (D) In experiment 2, subjects received a word cue to imagine the sounds and the associated visual scene. Four runs with word cues were alternated with four runs of actual sound stimulation. (E) Classification results are shown for imagined sounds. ∗p < 0.05, ∗∗p = 0.001. (F) Mean classification accuracies for different eccentricities of the visual ROIs. ∗p < 0.05 (uncorrected), ∗∗p < 0.05 (false discovery rate corrected). (G) Cross-classification results of experiment 2. The classifier was trained on real sounds and tested on imagined sounds and vice versa, and results were averaged. ∗p < 0.05, ∗∗p = 0.001.CC BY · PubMed Central
Figure 3 Experimental Design of Experiments 3 and 4 and Classification Results of Experiments 3, 4, and 5 (A) In experiment 3, subjects performed an orthogonal auditory working memory task while hearing natural sounds. They retained a word list of five animals or everyday objects in working memory and matched them with a second scrambled word list containing one different word in half of the trials. Match or mismatch was indicated with a button press during response time. (B) Classification results for the three different sounds during performance of the task. Significance levels and analysis parameters were the same as in experiments 1 and 2. Error bars indicate SEM. (C) Mean classification accuracies for all visual ROIs divided into three eccentricities (fovea, periphery, and far periphery). (D) In experiment 4, subjects performed a visuospatial imaginary cube-assembly task while hearing natural sounds. They mentally constructed an imaginary figure according to five assembly instructions, rotated the imaginary figure 90° clockwise, and indicated match or mismatch of the correct solution with the second list of instructions. (E) Classification results. ∗p < 0.05, ∗∗p = 0.001. (F) Classification results by eccentricity of visual ROIs. ∗p < 0.05 (uncorrected), ∗∗p < 0.05 (false discovery rate corrected). (G) In experiment 5, subjects listened to three different sound exemplars for each of the two categories, “human” (People 1, People 2, Kids) and “inanimate” (Traffic 1, Traffic 2, Airplane). Sounds were cut to 6 s, and interstimulus intervals were 6 s, otherwise the experimental design was the same as in experiment 1. The table shows early visual areas with significant above-chance classification for all combinations of “human” versus “inanimate” sounds. All p values were derived from permutation analyses. ∗p < .05, ∗∗p < 0.005, ∗∗∗p = 0.001. (H) Cross-classification of one pair of exemplars against another.
Figure 3 Experimental Design of Experiments 3 and 4 and Classification Results of Experiments 3, 4, and 5 (A) In experiment 3, subjects performed an orthogonal auditory working memory task while hearing natural sounds. They retained a word list of five animals or everyday objects in working memory and matched them with a second scrambled word list containing one different word in half of the trials. Match or mismatch was indicated with a button press during response time. (B) Classification results for the three different sounds during performance of the task. Significance levels and analysis parameters were the same as in experiments 1 and 2. Error bars indicate SEM. (C) Mean classification accuracies for all visual ROIs divided into three eccentricities (fovea, periphery, and far periphery). (D) In experiment 4, subjects performed a visuospatial imaginary cube-assembly task while hearing natural sounds. They mentally constructed an imaginary figure according to five assembly instructions, rotated the imaginary figure 90° clockwise, and indicated match or mismatch of the correct solution with the second list of instructions. (E) Classification results. ∗p < 0.05, ∗∗p = 0.001. (F) Classification results by eccentricity of visual ROIs. ∗p < 0.05 (uncorrected), ∗∗p < 0.05 (false discovery rate corrected). (G) In experiment 5, subjects listened to three different sound exemplars for each of the two categories, “human” (People 1, People 2, Kids) and “inanimate” (Traffic 1, Traffic 2, Airplane). Sounds were cut to 6 s, and interstimulus intervals were 6 s, otherwise the experimental design was the same as in experiment 1. The table shows early visual areas with significant above-chance classification for all combinations of “human” versus “inanimate” sounds. All p values were derived from permutation analyses. ∗p < .05, ∗∗p < 0.005, ∗∗∗p = 0.001. (H) Cross-classification of one pair of exemplars against another.CC BY · PubMed Central
Figure 4 Results of the Whole-Brain Searchlight Analysis for Experiments 1–4 Overlay of significant above-chance classification of the three heard or imagined sounds onto a flattened and inflated cortical surface reconstruction (MNI template) for experiments 1–4. Note that a searchlight analysis is less sensitive than an ROI analysis because (1) the searchlight volume is small, and, thus, the classifier is less able to pick out subtle differences in activity patterns and because (2) correction for multiple comparisons is necessary on the whole-brain level (see Supplemental Experimental Procedures). Significance level is p < 0.05 with cluster threshold correction. Searchlight size was 343 voxels. For results with increased statistical power and a bigger searchlight, see Figure S4. Early visual areas depict probability maps as in Figure 1. pSTS, posterior superior temporal sulcus; SFS, superior frontal sulcus; MFS, middle frontal sulcus.
Figure 4 Results of the Whole-Brain Searchlight Analysis for Experiments 1–4 Overlay of significant above-chance classification of the three heard or imagined sounds onto a flattened and inflated cortical surface reconstruction (MNI template) for experiments 1–4. Note that a searchlight analysis is less sensitive than an ROI analysis because (1) the searchlight volume is small, and, thus, the classifier is less able to pick out subtle differences in activity patterns and because (2) correction for multiple comparisons is necessary on the whole-brain level (see Supplemental Experimental Procedures). Significance level is p < 0.05 with cluster threshold correction. Searchlight size was 343 voxels. For results with increased statistical power and a bigger searchlight, see Figure S4. Early visual areas depict probability maps as in Figure 1. pSTS, posterior superior temporal sulcus; SFS, superior frontal sulcus; MFS, middle frontal sulcus.CC BY · PubMed Central

Verification

Claims the paper makes

20 Own interpretation · 15 Own result · 10 Not checked yet

Verify all 28 references — free account

Show all 20 claims

  • It may state two separate claims as one. 24
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 5
25 of 25
  • The paper relies on this without support.

    The word cues “forest,” “traffic,” and “people” reliably elicit the intended mental imagery of the corresponding sounds and natural scenes in subjects.

    “Subjects were instructed to engage in mental imagery of the sounds and a corresponding natural scene.”
    “runs in which subjects solely imagined the sounds upon hearing the word cues “forest,” “traffic,” or “people””
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 52 references

Cited by

Sorted by citations

Metadata

Authors
Vetter, P., Smith, F. W., & Muckli, L.
Affiliations
  • University of Geneva (CH)
  • University of Glasgow (GB)
Corresponding author
  • Vetter, Petra
  • Muckli, Lars
Venue
Current Biology
Publisher
Elsevier BV
Published
2014-06
Volume
24
Issue
11
Pages
1256-1262
ISSN
0960-9822 (ISSN-L)
OpenAlex
W2016034937
Cited by
323
References
52
Citations by year
2026: 12 · 2025: 23 · 2024: 38 · 2023: 75 · 2022: 20 · 2021: 21 · 2020: 21 · 2019: 25 · 2018: 18 · 2017: 19 · 2016: 20 · 2015: 25 · 2014: 6
FWCI
32.47
Type
journal article
Language
en
Open access
Locations
  • Current Biology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
  • Archive ouverte UNIGE (University of Geneva) (acceptedVersion, other-oa)
  • UEA Digital Repository (University of East Anglia) (submittedVersion, cc-by)
  • Archive ouverte UNIGE (University of Geneva) (submittedVersion, cc-by)
Funders
  • College of Science and Engineering, University of Minnesota
  • Biotechnology and Biological Sciences Research Council (BB/G005044/1)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
early visual cortex, auditory imagery, auditory perception, top-down feedback, multisensory integration, predictive coding, abstract information, visuospatial processing, attention, working memory, blindfolded participants
MeSH terms
Acoustic Stimulation, Auditory Perception, Humans, Magnetic Resonance Imaging, Memory, Short-Term, Photic Stimulation, Visual Cortex, Visual Perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review