Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
In everyday life, multiple sensory channels jointly trigger emotional experiences and one channel may alter processing in another channel. For example, seeing an emotional facial expression and hearing the voice's emotional tone will jointly create the emotional experience. This example, where auditory and visual input is related to social communication, has gained considerable attention by researchers. However, interactions of visual and auditory emotional information are not limited to social communication but can extend to much broader contexts including human, animal, and environmental cues. In this article, we review current research on audiovisual emotion processing beyond face-voice stimuli to develop a broader perspective on multimodal interactions in emotion processing. We argue that current concepts of multimodality should be extended in considering an ecologically valid variety of stimuli in audiovisual emotion processing. Therefore, we provide an overview of studies in which emotional sounds and interactions with complex pictures of scenes were investigated. In addition to behavioral studies, we focus on neuroimaging, electro- and peripher-physiological findings. Furthermore, we integrate these findings and identify similarities or differences. We conclude with suggestions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeDoux, J. E. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.155 · Cited by 8,444
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI
Phan, K. L., Wager, T., Taylor, S. F., & Liberzon, I. (2002) · NeuroImage · doi:10.1006/nimg.2002.1087 · Cited by 3,502
Multiple resources and performance prediction
Wickens, C. D. (2002) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220210123806 · Cited by 2,621 · cited 2×
“For a short overview of the here reviewed studies see Table 1.”
BackgroundResultEmotion, attention, and the startle reflex
Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1990) · Psychological Review · doi:10.1037/0033-295x.97.3.377 · Cited by 2,205 · cited 2×
“…data of an internet-based survey suggest that sound stimuli were significantly perceived as more horrible when they were accompanied by pictures that show associated information (e.g., the sound of a crying baby combined with a picture of a crying baby) compared to pictures with unassociated…”
BackgroundHow brains beware: neural mechanisms of emotional attention
Vuilleumier, P. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.10.011 · Cited by 2,162
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · cited 1×
“Generally, one can assume that multisensory integration of emotional information follows similar principles as multisensory integration of other types of complex information (see, e.g., Stein and Meredith, 1993; de Gelder and Bertelson, 2003; Ernst and Bülthoff, 2004; Spence, 2007).”
BackgroundNeural processing of emotional faces requires attention
Pessoa, L., McKenna, M., Gutierrez, E., & Ungerleider, L. G. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.172403899 · Cited by 1,155
The human amygdala and the emotional evaluation of sensory stimuli
Zald, D. H. (2003) · Brain Research Reviews · doi:10.1016/s0165-0173(02)00248-5 · Cited by 1,062
Emotion, motivation, and anxiety: brain mechanisms and psychophysiology
Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1998) · Biological Psychiatry · doi:10.1016/s0006-3223(98)00275-3 · Cited by 1,043 · cited 3×
“These modulatory effects are shown crossmodally, thus, there also seem to be independent of the stimulus modality (see, e.g., Bradley et al., 1990; Lang et al., 1998).”
BackgroundEmotion and the motivational brain
Lang, P. J., & Bradley, M. M. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2009.10.007 · Cited by 1,006 · cited 1×
“To sum up, emotional pictures and sounds mainly elicit similar reactions on the level of self-report, behavioral, physiological, and neuronal – both types of stimuli strongly activate appetitive and defensive motivational circuits (Bradley and Lang, 2000a; Lang and Bradley, 2010).”
BackgroundThe Human Amygdala: An Evolved System for Relevance Detection
Sander, D., Grafman, J., & Zalla, T. (2003) · Reviews in the Neurosciences · doi:10.1515/revneuro.2003.14.4.303 · Cited by 944 · cited 2×
“…December 2014 | Volume 5 | Article 1351 | 1 cues and has been documented to operate independently from the sensory modality which conveyed the information (Armony and LeDoux, 2000; Sander et al., 2003; Zald, 2003; Öhman, 2005; Stekelenburg and Vroomen, 2007; Scharpf et al., 2010; Armony, 2013).”
BackgroundPupil size variation as an indication of affective processing
Partala, T., & Surakka, V. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00017-x · Cited by 836 · cited 2×
“Specifically, voxels were identified that were sensitive to the valence of pictures within the visual modality, as well as voxels that were sensitive to the valence of sounds within the auditory modality, but no voxels that were sensitive to valence across the two modalities.”
BackgroundCostafreda, S. G., Brammer, M. J., David, A. S., & Fu, C. H. Y. (2008) · Brain Research Reviews · doi:10.1016/j.brainresrev.2007.10.012 · Cited by 772 · cited 2×
“On the neural level the amygdala has long been identified as a key structure of emotional detection both in humans and animals (for reviews and meta-analysis see LeDoux, 2000; Phan et al., 2002; Costafreda et al., 2008; Armony, 2013).”
BackgroundThe role of the amygdala in human fear: automatic detection of threat
Öhman, A. (2005) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2005.03.019 · Cited by 691
Brain mechanisms for emotional influences on perception and attention: What is magic and what is not
Pourtois, G., Schettino, A., & Vuilleumier, P. (2013) · Biological Psychology · doi:10.1016/j.biopsycho.2012.02.007 · Cited by 687
The amygdala, social behavior, and danger detection
AMARAL, D. G. (2003) · Annals of the New York Academy of Sciences · doi:10.1196/annals.1280.015 · Cited by 293 · cited 1×
“Relevant for the present context, via the thalamus the amygdala receives input not only from the visual modality but from all senses (Nishijo et al., 1988; Amaral et al., 1992; Amaral, 2003).”
BackgroundTest-retest reliability of evoked BOLD signals from a cognitive-emotive fMRI test battery.
Plichta, M. M., Schwarz, A. J., Grimm, O., Morgen, K., Mier, D., Haddad, L., Gerdes, A. B. M., Sauer, C., Tost, H., Esslinger, C., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.129 · Cited by 287
Self-citationDistinct fMRI responses to laughter, speech, and sounds along the human peri-sylvian cortex
Meyer, M., Zysset, S., von Cramon, D. Y., & Alter, K. (2005) · Cognitive Brain Research · doi:10.1016/j.cogbrainres.2005.02.008 · Cited by 120
Processing of body odor signals by the human brain
Pause, B. M. (2012) · Chemosensory Perception · doi:10.1007/s12078-011-9108-2 · Cited by 107
When spiders appear suddenly: Spider-phobic patients are distracted by task-irrelevant spiders
Gerdes, A. B. M., Alpers, G. W., & Pauli, P. (2008) · Behaviour Research and Therapy · doi:10.1016/j.brat.2007.10.010 · Cited by 99 · cited 1×
“Emotional pictures influence perceptual processing and attract enhanced attention (Öhman and Wiens, 2003; Alpers and Pauli, 2006; Alpers and Gerdes, 2007; Gerdes et al., 2008, 2009; Stienen et al., 2011; Pourtois et al., 2012; Gerdes and Alpers, 2014).”
BackgroundSelf-citationCross-modal attention capture by affective stimuli: Evidence from event-related potentials
KEIL, A., BRADLEY, M. M., JUNGHOFER, M., RUSSMANN, T., LOWENTHAL, W., & LANG, P. J. (2007) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.7.1.18 · Cited by 89 · cited 2×
“Using affective IADS sounds; Scherer and Larsen (2011) found significant cross-modal priming effects for negative sound primes on emotional visual word targets.”
BackgroundPerceiving blocks of emotional pictures and sounds: effects on physiological variables.
Brouwer, A.-M., van Wouwe, N., Mühl, C., van Erp, J., & Toet, A. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00295 · Cited by 85
Domínguez‐Borràs, J., Garcia‐Garcia, M., & Escera, C. (2008) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2008.06411.x · Cited by 45
Current emotion research in behavioral neuroscience: the role(s) of the amygdala
Armony, J. L. (2013) · Emotion Review · doi:10.1177/1754073912457208 · Cited by 44
Metadata
- Authors
- Gerdes, A. B. M., Wieser, M. J., & Alpers, G. W.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2014-12-01
- Volume
- 5
- Pages
- 1351-1351
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W2169842885
- Cited by
- 166
- References
- 25
- FWCI
- 8.43
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.