Crossmodal interactions in audiovisual emotion processing
Open accessAbstract
Emotion in daily life is often expressed in a multimodal fashion. Consequently emotional information from one modality can influence processing in another. In a previous fMRI study we assessed the neural correlates of audio-visual integration and found that activity in the left amygdala is significantly attenuated when a neutral stimulus is paired with an emotional one compared to conditions where emotional stimuli were present in both channels. Here we used dynamic causal modelling to investigate the effective connectivity in the neuronal network underlying this emotion presence congruence effect. Our results provided strong evidence in favor of a model family, differing only in the interhemispheric interactions. All winning models share a connection from the bilateral fusiform gyrus (FFG) into the left amygdala and a non-linear modulatory influence of bilateral posterior superior temporal sulcus (pSTS) on these connections. This result indicates that the pSTS not only integrates multi-modal information from visual and auditory regions (as reflected in our model by significant feed-forward connections) but also gates the influence of the sensory information on the left amygdala, leading to attenuation of amygdala activity when a neutral stimulus is integrated. Moreover, we found a significant lateralization of the FFG due to stronger driving input by the stimuli (faces) into the right hemisphere, whereas such lateralization was not present for sound-driven input into the superior temporal gyrus. In summary, our data provides further evidence for a rightward lateralization of the FFG and in particular for a key role of the pSTS in the integration and gating of audio-visual emotional information.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The VLPFC is anatomically connected to the STG, which processes higher order multi-modal integration and regulates amygdala activity.
“It is also anatomically connected to the STG (Ongür et al., 2003), which processes higher order multi-modal integration and also regulates amygdala activity (Müller et al., 2012).”
From Neural network of cognitive emotion regulation—an ALE meta-analysis and MACM analysis (2014) · cites this paper as Müller et al., 2012
The superior temporal gyrus performs higher-order multimodal integrative processes and also affects amygdala activity.
“It computes higher order multimodal integrative processes and also affects amygdala activity (Müller et al., 2012), which may reflect its association to social cognition.”
From Neural network of cognitive emotion regulation—an ALE meta-analysis and MACM analysis (2014) · cites this paper as Müller et al., 2012
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The assessment and analysis of handedness: The Edinburgh inventory
Oldfield, R. C. (1971) · Neuropsychologia · doi:10.1016/0028-3932(71)90067-4 · Cited by 35,900 · see citing works
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738 · see citing works
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · see citing works
A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data
Eickhoff, S. B., Stephan, K. E., Mohlberg, H., Grefkes, C., Fink, G. R., Amunts, K., & Zilles, K. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2004.12.034 · Cited by 4,033 · see citing works
Self-citationVoice-selective areas in human auditory cortex
Belin, P., Zatorre, R. J., Lafaille, P., Ahad, P. A., & Pike, G. B. (2000) · Nature · doi:10.1038/35002078 · Cited by 1,789 · see citing works
The fusiform face area: a cortical region specialized for the perception of faces
Kanwisher, N., & Yovel, G. (2006) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2006.1934 · Cited by 1,718 · see citing works
Abbott, L. F., & Regehr, W. G. (2004) · Nature · doi:10.1038/nature03010 · Cited by 1,684 · see citing works
Bayesian model selection for group studies
Stephan, K. E., Penny, W. D., Daunizeau, J., Moran, R. J., & Friston, K. J. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2009.03.025 · Cited by 1,423 · see citing works
Cortical pathways to the mammalian amygdala
McDonald, A. J. (1998) · Progress in Neurobiology · doi:10.1016/s0301-0082(98)00003-3 · Cited by 1,303 · see citing works
Assignment of functional activations to probabilistic cytoarchitectonic areas revisited
Eickhoff, S., Paus, T., Caspers, S., Grosbras, M.-H., Evans, A. C., Zilles, K., & Amunts, K. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.03.060 · Cited by 1,038 · see citing works
Self-citationThe fusiform face area subserves face perception, not generic within-category identification
Kalanit Grill-spector, A., Knouf, N. A., & Kanwisher, N. (2004) · Nature Neuroscience · doi:10.1038/nn1224 · Cited by 984 · see citing works
Calvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954 · see citing works
Thinking the voice: neural correlates of voice perception
Belin, P., Fecteau, S., & Bédard, C. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.01.008 · Cited by 908 · see citing works
What does the amygdala contribute to social cognition
Adolphs, R. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05445.x · Cited by 836 · see citing works
Neural responses to facial and vocal expressions of fear and disgust
Phillips, M. L., Young, A. W., Scott, S. K., Calder, A., Andrew, C., Giampietro, V., Williams, S., Bullmore, E. T., Brammer, M. J., & Gray, J. A. (1998) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1998.0506 · Cited by 817 · see citing works
Ten simple rules for dynamic causal modeling
Stephan, K. E., Penny, W. D., Moran, R. J., den Ouden, H. E. M., Daunizeau, J., & Friston, K. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.015 · Cited by 803 · see citing works
Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus
Beauchamp, M. S., Lee, K. E., Argall, B., & Martin, A. D. (2004) · Neuron · doi:10.1016/s0896-6273(04)00070-4 · Cited by 765 · see citing works
Emotional perception: Meta-analyses of face and natural scene processing
Sabatinelli, D., Fortune, E., Li, Q., Siddiqui, A., Krafft, C., Oliver, W. T., Beck, S. M., & Jeffries, J. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.10.011 · Cited by 704 · see citing works
Testing anatomically specified hypotheses in functional imaging using cytoarchitectonic maps
Eickhoff, S. B., Heim, S., Zilles, K., & Amunts, K. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.04.204 · Cited by 643 · see citing works
Self-citationSuperior Temporal Sulcus—It's My Area: Or Is It?
Hein, G., & Knight, R. T. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20148 · Cited by 486 · see citing works
Effective Connectivity within the Distributed Cortical Network for Face Perception
Fairhall, S. L., & Ishai, A. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhl148 · Cited by 471 · see citing works
Grefkes, C., Eickhoff, S. B., Nowak, D. A., Dafotakis, M., & Fink, G. R. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.03.048 · Cited by 436 · see citing works
Self-citationNonlinear dynamic causal models for fMRI
Stephan, K. E., Kasper, L., Harrison, L. M., Daunizeau, J., den Ouden, H. E. M., Breakspear, M., & Friston, K. J. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.04.262 · Cited by 416 · see citing works
Clarke, S., & Judith Miklossy, A. (1990) · The Journal of Comparative Neurology · doi:10.1002/cne.902980205 · Cited by 399 · see citing works
Face perception is mediated by a distributed cortical network
Ishai, A., Schmidt, C. F., & Boesiger, P. (2005) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2005.05.027 · Cited by 393 · see citing works
Dynamic causal modeling of cortical activity from the acute to the chronic stage after stroke
Rehme, A. K., Eickhoff, S. B., Wang, L. E., Fink, G. R., & Grefkes, C. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.01.014 · Cited by 334 · see citing works
Self-citationA revised view of sensory cortical parcellation
Wallace, M. T., Ramachandran, R., & Stein, B. E. (2004) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0305697101 · Cited by 329 · see citing works
Brain Activation during Facial Emotion Processing
Gur, R. C., Schroeder, L. F., Turner, T. H., McGrath, C., Chan, R. M., Turetsky, B. I., Alsop, D. C., Maldjian, J. A., & Gur, R. E. (2002) · NeuroImage · doi:10.1006/nimg.2002.1097 · Cited by 326 · see citing works
Self-citationAudiovisual integration of emotional signals in voice and face: An event-related fMRI study
Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., & Wildgruber, D. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.06.020 · Cited by 307 · see citing works
Crossmodal binding of fear in voice and face
Dolan, R. J., Morris, J. S., & de Gelder, B. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.171288598 · Cited by 304 · see citing works
Essen, D. V., Newsome, W., & Bixby, J. L. (1982) · Journal of Neuroscience · doi:10.1523/jneurosci.02-03-00265.1982 · Cited by 300 · see citing works
A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion
Nath, A. R., & Beauchamp, M. S. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.07.024 · Cited by 282 · see citing works
A systems perspective on the effective connectivity of overt speech production
Eickhoff, S. B., Heim, S., Zilles, K., & Amunts, K. (2009) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2008.0287 · Cited by 252 · see citing works
Self-citationAuditory-visual speech perception examined by fMRI and PET
Sekiyama, K., Kanno, I., Miura, S., & Sugita, Y. (2003) · Neuroscience Research · doi:10.1016/s0168-0102(03)00214-1 · Cited by 249 · see citing works
Binding crossmodal object features in perirhinal cortex
Taylor, K. I., Moss, H. E., Stamatakis, E. A., & TYLER, L. K. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0509704103 · Cited by 247 · see citing works
Beauchamp, M. S., Nath, A. R., & Pasalar, S. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4865-09.2010 · Cited by 240 · see citing works
Laterality effect on emotional faces processing: ALE meta-analysis of evidence
Fusar‐Poli, P., Placentino, A., Carletti, F., Allen, P., Landi, P., Abbamonte, M., Barale, F., Pérez, J., McGuire, P. K., & Politi, P. (2009) · Neuroscience Letters · doi:10.1016/j.neulet.2009.01.065 · Cited by 220 · see citing works
Amygdala responses to nonlinguistic emotional vocalizations
Fecteau, S., Belin, P., Joanette, Y., & Armony, J. L. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.02.043 · Cited by 214 · see citing works
Dissociating Reading Processes on the Basis of Neuronal Interactions
Mechelli, A., Crinion, J., de long, S., Friston, K. J., Ralph, M. A. L., Patterson, K. E., McClelland, J. L., & Price, C. J. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/089892905774589190 · Cited by 214 · see citing works
Perception of Facial Expressions and Voices and of their Combination in the Human Brain
Pourtois, G., DEGELDER, B., Bol, A., & Crommelinck, M. (2005) · Cortex · doi:10.1016/s0010-9452(08)70177-1 · Cited by 203 · see citing works
Habel, U., Windischberger, C., Derntl, B., Robinson, S., Kryspin‐Exner, I., Gur, R. C., & Moser, E. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.01.023 · Cited by 197 · see citing works
Is the Fusiform Face Area Specialized for Faces, Individuation, or Expert Individuation?
Rhodes, G., Byatt, G., Michie, P. T., & Puce, A. (2004) · Journal of Cognitive Neuroscience · doi:10.1162/089892904322984508 · Cited by 192 · see citing works
Auditory perception of laughing and crying activates human amygdala regardless of attentional state
Sander, K., & Scheich, H. (2001) · Cognitive Brain Research · doi:10.1016/s0926-6410(01)00045-3 · Cited by 183 · see citing works
Cerebral processing of emotional prosody—influence of acoustic parameters and arousal
Wiethoff, S., Wildgruber, D., Kreifelts, B., Becker, H. G. T., Herbert, C., Grodd, W., & Ethofer, T. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.09.028 · Cited by 175 · see citing works
A comparison of bound and unbound audio–visual information processing in the human cerebral cortex
Olson, I. R., Gatenby, J. C., & Gore, J. C. (2002) · Cognitive Brain Research · doi:10.1016/s0926-6410(02)00067-8 · Cited by 156 · see citing works
Superior temporal activation in response to dynamic audio-visual emotional cues
Robins, D. L., Hunyadi, E., & Schultz, R. T. (2008) · Brain and Cognition · doi:10.1016/j.bandc.2008.08.007 · Cited by 149 · see citing works
Bidirectional communication between amygdala and fusiform gyrus during facial recognition
Herrington, J. D., Taylor, J. M., Grupe, D. W., Curby, K. M., & Schultz, R. T. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.03.072 · Cited by 138 · see citing works
Kreifelts, B., Ethofer, T., Huberle, E., Grodd, W., & Wildgruber, D. (2010) · Human Brain Mapping · doi:10.1002/hbm.20913 · Cited by 132 · see citing works
Noradrenergic Enhancement Improves Motor Network Connectivity in Stroke Patients
Wang, L. E., Fink, G. R., Diekhoff, S., Rehme, A., Eickhoff, S., & Grefkes, C. (2010) · Annals of Neurology · doi:10.1002/ana.22237 · Cited by 130 · see citing works
Self-citationDien, J. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2008.08.024 · Cited by 122 · see citing works
Hunter, M. D., Eickhoff, S. B., Pheasant, R. J., Douglas, M. J., Watts, G. R., Farrow, T. F. D., Hyland, D., Kang, J., Wilkinson, I. D., & Horoshenkov, K. V. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.06.053 · Cited by 121 · see citing works
Self-citationDistinct fMRI responses to laughter, speech, and sounds along the human peri-sylvian cortex
Meyer, M., Zysset, S., von Cramon, D. Y., & Alter, K. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.02.008 · Cited by 120 · see citing works
The Role of the Posterior Superior Temporal Sulcus in Audiovisual Processing
Hocking, J., & Price, C. J. (2008) · Cerebral Cortex · doi:10.1093/cercor/bhn007 · Cited by 116 · see citing works
Effects of prosodic emotional intensity on activation of associative auditory cortex
Ethofer, T., Anders, S., Wiethoff, S., Erb, M., Herbert, C., Saur, R., Grodd, W., & Wildgruber, D. (2006) · Neuroreport · doi:10.1097/01.wnr.0000199466.32036.5d · Cited by 114 · see citing works
Supramodal Representation of Emotions
Klasen, M., Kenworthy, C. A., Mathiak, K. A., Kircher, T. T. J., & Mathiak, K. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.2833-11.2011 · Cited by 112 · see citing works
Investigating audiovisual integration of emotional signals in the human brain
Ethofer, T., Pourtois, G., & Wildgruber, D. (2006) · Progress in brain research · doi:10.1016/s0079-6123(06)56019-4 · Cited by 111 · see citing works
Incongruence effects in crossmodal emotional integration
Müller, V. I., Habel, U., Derntl, B., Schneider, F. W., Zilles, K., Turetsky, B. I., & Eickhoff, S. B. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.10.047 · Cited by 111 · see citing works
Self-citationKreifelts, B., Ethofer, T., Shiozawa, T., Grodd, W., & Wildgruber, D. (2009) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2009.07.001 · Cited by 105 · see citing works
Primary and multisensory cortical activity is correlated with audiovisual percepts
Benoit, M. M., Raij, T., Lin, F., Jääskeläinen, I. P., & Stufflebeam, S. M. (2009) · Human Brain Mapping · doi:10.1002/hbm.20884 · Cited by 88 · see citing works
Hagan, C. C., Woods, W., Johnson, S. R., Calder, A. J., Green, G. G. R., & Young, A. W. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0905792106 · Cited by 82 · see citing works
Connections between the amygdala and auditory cortical areas in the macaque monkey
Yukie, M. (2002) · Neuroscience Research · doi:10.1016/s0168-0102(01)00325-x · Cited by 76 · see citing works
Gainotti, G. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2011.04.027 · Cited by 72 · see citing works
Face processing without awareness in the right fusiform gyrus
Morris, J. P., Pelphrey, K. A., & McCarthy, G. (2007) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2007.05.020 · Cited by 70 · see citing works
Pandya, D. Ν., & Rosene, D. L. (1993) · Experimental Neurology · doi:10.1006/exnr.1993.1024 · Cited by 64 · see citing works
Cieslik, E. C., Zilles, K., Grefkes, C., & Eickhoff, S. B. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.05.089 · Cited by 43 · see citing works
Self-citationCallosal connections of the parabelt auditory cortex in macaque monkeys
Hackett, T. A., Stepniewska, I., & Kaas, J. H. (1999) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.1999.00492.x · Cited by 42 · see citing works
Multisensory perception of action in posterior temporal and parietal cortices
James, T. W., VanDerKlok, R. M., Stevenson, R. A., & James, K. H. (2011) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2010.10.030 · Cited by 38 · see citing works
Semiotic aspects of human nonverbal vocalizations: a functional imaging study
Dietrich, S., Hertrich, I., Alter, K., Ischebeck, A. K., & Ackermann, H. (2007) · Neuroreport · doi:10.1097/wnr.0b013e3282f290df · Cited by 18 · see citing works
Ban, T., Naito, J., & Kawamura, K. (1984) · Neuroscience Letters · doi:10.1016/0304-3940(84)90136-8 · Cited by 6 · see citing works
Worsley (1996) · doi:10.1002/(sici)1097-0193(1996)4:1
Flagged first, then by citations
Cited by
Sorted by citations
Neural network of cognitive emotion regulation—an ALE meta-analysis and MACM analysis
Kohn, N., Eickhoff, S. B., Scheller, M., Laird, A. R., Fox, P. T., & Habel, U. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.001 · Cited by 956 · see citing works
Self-citationMultisensory integration mechanisms during aging
Freiherr, J., Lundström, J. N., Habel, U., & Reetz, K. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00863 · Cited by 173 · see citing works
- Authors
- Müller, V. I., Cieslik, E. C., Turetsky, B. I., & Eickhoff, S. B.
- Affiliations
- Forschungszentrum Jülich (DE)
- University of Pennsylvania (US)
- Corresponding author
- Müller, Veronika I.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2012-03
- Volume
- 60
- Issue
- 1
- Pages
- 553-561
- ISSN
- 1053-8119 (ISSN-L)
- PMC
- PMC8022349
- OpenAlex
- W2136380346
- Cited by
- 65
- References
- 70
- Citations by year
- 2025: 5 · 2024: 4 · 2023: 5 · 2022: 2 · 2021: 5 · 2020: 1 · 2019: 7 · 2018: 4 · 2017: 5 · 2016: 3 · 2015: 4 · 2014: 7 · 2013: 10 · 2012: 3
- FWCI
- 8.25
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- NeuroImage (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- JuSER (Forschungszentrum Jülich) (submittedVersion)
- RWTH Publications (RWTH Aachen) (submittedVersion)
- Funders
- Deutsche Forschungsgemeinschaft (IRTG 1328)
- Human Brain Project (R01-MH074457-01A1)
- Helmholtz Initiative on systems biology
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neuroscience and Music Perception
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Keywords
- audiovisual integration, emotional processing, cross-modal interaction, effective connectivity, amygdala, fusiform gyrus, posterior superior temporal sulcus, interhemispheric interaction, hemispheric lateralization, emotional congruence, dynamic causal modeling
- MeSH terms
- Auditory Perception, Brain, Emotions, Female, Humans, Magnetic Resonance Imaging, Male, Visual Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
