Decoding Sound and Imagery Content in Early Visual Cortex

In the paper

“Even at very liberal thresholds (p < 0.05 uncorrected; Figure S3), listening to sounds in the absence of visual stimulation elicited no positive activation in early visual areas but instead elicited a weak deactivation, consistent with previous findings (e.g.,[16]) and in contrast to classical findings for visual mental imagery [17, 18].”

In the paper

“Successful discrimination of imagined sounds was observed in both foveal and peripheral areas of early visual cortex (but not far periphery), in V1 and auditory cortex (Figures 2E and 2F; classification of real sounds replicated the results of experiment 1, cf. Figure S1B).”

• Also stated in: § Decoding of Sound and Imagery Content in Early Visual Cortex · hedged

  “Therefore, even in the absence of both visual and auditory stimulation, the contents of mental imagery could be decoded from both V1 and auditory cortex.”

In the paper

“Imagery (experiment 2) elicited no positive activity but exhibited weak deactivations in both early visual and auditory cortices.”

In the paper

“Although the classifier failed to discriminate the three natural sounds in most of early visual cortex, residual above-chance classification remained in the far periphery of V2 (Figures 3E and 3F) despite the orthogonal engagement of attentionally demanding active visuospatial processing.”

In the paper

“Moreover, higher visual areas project back to V1 the associated color of grayscale objects [36] or the predicted motion path of an apparent motion illusion [37, 38].”

What the sources say

• Not checked yetcites [36] Bannert, 2013, Decoding the yellow of a gray banana, Curr. Biol. source
• Not checked yetcites [37] Muckli, 2005, Primary visual cortex activity along the apparent-motion trace reflects illusory perception, PLoS Biol. source
• Not checked yetcites [38] Vetter, 2013, TMS over V5 disrupts motion prediction, Cereb. Cortex

In the paper

“Our findings are compatible with feedback of predictive information to the earliest visual input level (e.g., [6]), in line with predictive coding models [7–10].”

What the sources say

• Not checked yetcites [6] Mumford, 1992, On the computational architecture of the neocortex. II. The role of cortico-cortical loops, Biol. Cybern. source
• Not checked yetcites [7] Friston, 2010, The free-energy principle: a unified brain theory?, Nat. Rev. Neurosci. source
• Not checked yetcites [8] Clark, 2013, Whatever next? Predictive brains, situated agents, and the future of cognitive science, Behav. Brain Sci. source
• Not checked yetcites [9] Bar, 2007, The proactive brain: using analogies and associations to generate predictions, Trends Cogn. Sci. source
• Not checked yetcites [10] Bastos, 2012, Canonical microcircuits for predictive coding, Neuron source

In the paper

“Crucially, the information fed down to early visual cortex is category specific and generalizes to sound exemplars of the same category, providing evidence for abstract information feedback rather than precise pictorial feedback.”

In the paper

“Furthermore, this coding is robust to mild manipulations of attention and working memory but affected by orthogonal, cognitively demanding visuospatial processing.”

In the paper

“This demonstrates that both sounds and imagery cues induced similar activity patterns in early visual cortex and that feedback information is coded consistently across imagery and auditory perception.”

In the paper

“Note that despite relatively low classification accuracies, our series of experiments replicated the successful decoding of sounds in early visual areas several times, demonstrating proof of principle and the robustness of our results across different subject and stimulus samples.”

In the paper

“In the context of our findings, we suggest that the content-specific information from sounds, when they are heard and/or imagined, is relayed from auditory cortex to early visual cortex via pSTS and precuneus, eliciting differential activity patterns in both of these regions.”

In the paper

“The coding of nonretinal information in the activity patterns of early visual cortex is common across actual auditory perception and imagery and may be mediated by higher-level multisensory areas.”

In the paper

“Our results suggest that early visual cortex receives nonretinal input from other brain areas when it is generated by auditory perception and/or imagery, and this input carries common abstract information.”

In the paper

“The precuneus has been identified as an area responding to both visual and auditory stimuli and possibly serving as an audiovisual convergence area [25].”

What the sources say

• Not checked yetcites [25] Hertz, 2010, Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis, Neuroimage source

In the paper

“The results of experiment 5 suggest that the information that is fed down to early visual cortex is not only content specific but also category specific, i.e., related to the information shared by sound exemplars of the same category.”

In the paper

“The findings of experiment 5 furthermore demonstrate that the successful decoding in experiment 1 was not specific to the first sound exemplars we used and could not be caused by differential low-level acoustic features of the sounds (e.g., frequency distribution).”

In the paper

“This suggests that information feedback is unlikely to be caused by an exact pictorial representation but instead contains abstract and possibly semantic information.”

In the paper

“Apart from the route via multisensory areas, there is evidence for multisensory integration on the subcortical level [27] and for direct anatomical connections between early auditory and early visual areas [12, 28, 29], mostly reaching peripheral regions [12–14], consistent with both our eccentricity and searchlight results.”

What the sources say

• Not checked yetcites [27] van den Brink, 2013, Subcortical, modality-specific pathways contribute to multisensory processing in humans, Cereb. Cortex
• Not checked yetcites [12] Eckert, 2008, A cross-modal system linking primary auditory and visual cortices: evidence from intrinsic fMRI connectivity analysis, Hum. Brain Mapp. source
• Not checked yetcites [28] Beer, 2013, Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing, Front Integr Neurosci source
• Not checked yetcites [29] Beer, 2011, Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex, Exp. Brain Res. source
• Not checked yetcites [13] Rockland, 2003, Multisensory convergence in calcarine visual areas in macaque monkey, Int. J. Psychophysiol. source
• Not checked yetcites [14] Cate, 2009, Auditory attention activates peripheral visual cortex, PLoS ONE source

In the paper

“This also means that our results were not caused by an unspecific attention effect or a simple reactivation of early visual cortex due to pictorial visual mental imagery.”

In the paper

“Our results demonstrate that the information arriving in early visual cortex is categorical and independent of its exact source.”

In the paper

“For example, activity patterns in nonstimulated parts of early visual cortex contain content-specific information from the surrounding visual context [31, 32], from objects presented in the periphery [33], and from visual stimuli solely held in working memory rather than being actively perceived [34, 35].”

What the sources say

• Not checked yetcites [31] Smith, 2010, Nonstimulated early visual areas carry information about surrounding context, Proc. Natl. Acad. Sci. USA source
• Not checked yetcites [32] Ban, 2013, Topographic representation of an occluded object and the effects of spatiotemporal context in human early visual areas, J. Neurosci. source
• Not checked yetcites [33] Williams, 2008, Feedback of visual object information to foveal retinotopic cortex, Nat. Neurosci. source
• Not checked yetcites [34] Harrison, 2009, Decoding reveals the contents of visual working memory in early visual areas, Nature source
• Not checked yetcites [35] Albers, 2013, Shared representations for working memory and mental imagery in early visual cortex, Curr. Biol. source

In the paper

“Also, hippocampal projections to peripheral early visual regions have been demonstrated in the context of boundary extension for scene processing [30].”

What the sources say

• Not checked yetcites [30] Chadwick, 2013, The hippocampus extrapolates beyond the view in scenes: an fMRI study of boundary extension, Cortex source

In the paper

“pSTS is implicated in audiovisual integration and has been shown to feed down information to primary visual and auditory cortices [26].”

What the sources say

• Not checked yetcites [26] Naumer, 2011, Investigating human audio-visual object perception with a combination of hypothesis-generating and hypothesis-testing fMRI analysis tools, Exp. Brain Res. source

In the paper

“That is, the information contained in these activity patterns is generalizable across different sound exemplars within a category, demonstrating that sounds trigger shared categorical information transfer to early visual cortex rather than a fine-grained pictorial representation.”

In the paper

“In fact, previous accounts suggested that prediction and mental imagery may involve overlapping brain mechanisms [6, 43, 44], and mental imagery might have evolved from predictive brain mechanisms.”

What the sources say

• Not checked yetcites [6] Mumford, 1992, On the computational architecture of the neocortex. II. The role of cortico-cortical loops, Biol. Cybern. source
• Not checked yetcites [43] Maguire, 2013, The hippocampus: a manifesto for change, J. Exp. Psychol. Gen. source
• Not checked yetcites [44] Moulton, 2009, Imagining predictions: mental imagery as mental emulation, Philos. Trans. R. Soc. Lond. B Biol. Sci. source