Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
In everyday life, multiple sensory channels jointly trigger emotional experiences and one channel may alter processing in another channel. For example, seeing an emotional facial expression and hearing the voice's emotional tone will jointly create the emotional experience. This example, where auditory and visual input is related to social communication, has gained considerable attention by researchers. However, interactions of visual and auditory emotional information are not limited to social communication but can extend to much broader contexts including human, animal, and environmental cues. In this article, we review current research on audiovisual emotion processing beyond face-voice stimuli to develop a broader perspective on multimodal interactions in emotion processing. We argue that current concepts of multimodality should be extended in considering an ecologically valid variety of stimuli in audiovisual emotion processing. Therefore, we provide an overview of studies in which emotional sounds and interactions with complex pictures of scenes were investigated. In addition to behavioral studies, we focus on neuroimaging, electro- and peripher-physiological findings. Furthermore, we integrate these findings and identify similarities or differences. We conclude with suggestions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeDoux, J. E. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.155 · Cited by 8,444
Hearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183
Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI
Phan, K. L., Wager, T., Taylor, S. F., & Liberzon, I. (2002) · NeuroImage · doi:10.1006/nimg.2002.1087 · Cited by 3,502
Multiple resources and performance prediction
Wickens, C. D. (2002) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220210123806 · Cited by 2,621 · cited 2×
“For a short overview of the here reviewed studies see Table 1.”
BackgroundResultEmotion, attention, and the startle reflex
Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1990) · Psychological Review · doi:10.1037/0033-295x.97.3.377 · Cited by 2,205 · cited 2×
“…data of an internet-based survey suggest that sound stimuli were significantly perceived as more horrible when they were accompanied by pictures that show associated information (e.g., the sound of a crying baby combined with a picture of a crying baby) compared to pictures with unassociated…”
BackgroundHow brains beware: neural mechanisms of emotional attention
Vuilleumier, P. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.10.011 · Cited by 2,162
Merging the senses into a robust percept
Ernst, M. O., & Bülthoff, H. H. (2004) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2004.02.002 · Cited by 1,906 · cited 1×
“Generally, one can assume that multisensory integration of emotional information follows similar principles as multisensory integration of other types of complex information (see, e.g., Stein and Meredith, 1993; de Gelder and Bertelson, 2003; Ernst and Bülthoff, 2004; Spence, 2007).”
BackgroundNeural processing of emotional faces requires attention
Pessoa, L., McKenna, M., Gutierrez, E., & Ungerleider, L. G. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.172403899 · Cited by 1,155
The human amygdala and the emotional evaluation of sensory stimuli
Zald, D. H. (2003) · Brain Research Reviews · doi:10.1016/s0165-0173(02)00248-5 · Cited by 1,062
Emotion, motivation, and anxiety: brain mechanisms and psychophysiology
Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1998) · Biological Psychiatry · doi:10.1016/s0006-3223(98)00275-3 · Cited by 1,043 · cited 3×
“These modulatory effects are shown crossmodally, thus, there also seem to be independent of the stimulus modality (see, e.g., Bradley et al., 1990; Lang et al., 1998).”
Background
Metadata
- Authors
- Gerdes, A. B. M., Wieser, M. J., & Alpers, G. W.
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2014-12-01
- Volume
- 5
- Pages
- 1351-1351
- ISSN
- 1664-1078 (ISSN-L)
- OpenAlex
- W2169842885
- Cited by
- 166
- References
- 25
- FWCI
- 8.43
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains. https://app.cochat.ai/papers/10.3389/fpsyg.2014.01351
- The paper
- Gerdes, A. B. M., Wieser, M. J., & Alpers, G. W. (2014). Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2014.01351
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.