Multisensory perception of action in posterior temporal and parietal cortices
Open accessAbstract
Environmental events produce many sensory cues for identifying the action that evoked the event, the agent that performed the action, and the object targeted by the action. The cues for identifying environmental events are usually distributed across multiple sensory systems. Thus, to understand how environmental events are recognized requires an understanding of the fundamental cognitive and neural processes involved in multisensory object and action recognition. Here, we investigated the neural substrates involved in auditory and visual recognition of object-directed actions. Consistent with previous work on visual recognition of isolated objects, visual recognition of actions, and recognition of environmental sounds, we found evidence for multisensory audiovisual event-selective activation bilaterally at the junction of the posterior middle temporal gyrus and the lateral occipital cortex, the left superior temporal sulcus, and bilaterally in the intraparietal sulcus. The results suggest that recognition of events through convergence of visual and auditory cues is accomplished through a network of brain regions that was previously implicated only in visual recognition of action.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The superior temporal sulcus and temporoparietal junction receive convergent input from multiple sensory domains and play important roles in integrating that information.
“Two of the presumptive brain regions mediating ToM function, the STS and TPJ (see above), receive convergent input from a variety of sensory domains and have been shown to play important roles in the integration of this information (Beauchamp et al., 2012; Downar et al., 2000, 2001; James et al., 2011; Shulman et al., 2003; Stevenson et al., 2007; Stevenson and James, 2009).”
From Behavioral, Perceptual, and Neural Alterations in Sensory and Multisensory Function in Autism Spectrum Disorder (2015) · cites this paper as James et al., 2011
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,308 · see citing works
The VideoToolbox software for visual psychophysics: transforming numbers into movies
Pelli, D. G. (1997) · Spatial Vision · doi:10.1163/156856897x00366 · Cited by 11,110 · see citing works
Forman, S. D., Cohen, J. D., Fitzgerald, M., Eddy, W. F., Mintun, M. A., & Noll, D. C. (1995) · Magnetic Resonance in Medicine · doi:10.1002/mrm.1910330508 · Cited by 3,202 · see citing works
Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex.
Malach, R., Reppas, J. B., Benson, R. R., Kwong, K. K., Jiang, H., Kennedy, W. A., Ledden, P. J., Brady, T. J., Rosen, B. R., & Tootell, R. B. H. (1995) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.92.18.8135 · Cited by 1,761 · see citing works
Imitation, empathy, and mirror neurons
Iacoboni, M. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163604 · Cited by 1,606 · see citing works
Deconvolution of impulse response in event-related BOLD fMRI.
Glover, G. (1999) · NeuroImage · doi:10.1006/nimg.1998.0419 · Cited by 1,532 · see citing works
Sensorimotor synchronization: A review of the tapping literature
Repp, B. H. (2005) · Psychonomic Bulletin & Review · doi:10.3758/bf03206433 · Cited by 1,469 · see citing works
The lateral occipital complex and its role in object recognition
Kalanit Grill-spector, A., Kourtzi, Z., & Kanwisher, N. (2001) · Vision Research · doi:10.1016/s0042-6989(01)00073-6 · Cited by 1,410 · see citing works
ALE meta-analysis of action observation and imitation in the human brain
Caspers, S., Zilles, K., Laird, A. R., & Eickhoff, S. B. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.12.112 · Cited by 1,386 · see citing works
Calvert, G. A., Campbell, R., & Brammer, M. J. (2000) · Current Biology · doi:10.1016/s0960-9822(00)00513-3 · Cited by 954 · see citing works
What in the World Do We Hear?: An Ecological Approach to Auditory Event Perception
Gaver, W. W. (1993) · Ecological Psychology · doi:10.1207/s15326969eco0501_1 · Cited by 903 · see citing works
Integration of Auditory and Visual Information about Objects in Superior Temporal Sulcus
Beauchamp, M. S., Lee, K. E., Argall, B., & Martin, A. D. (2004) · Neuron · doi:10.1016/s0896-6273(04)00070-4 · Cited by 765 · see citing works
A fronto‐parietal circuit for object manipulation in man: evidence from an fMRI‐study
Binkofski, F., Buccino, G., Posse, S., Seitz, R. J., Rizzolatti, G., & Freund, H.-J. (1999) · European Journal of Neuroscience · doi:10.1046/j.1460-9568.1999.00753.x · Cited by 720 · see citing works
Electrophysiology and brain imaging of biological motion
Puce, A., & Perrett, D. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1221 · Cited by 693 · see citing works
Brain Areas Active during Visual Perception of Biological Motion
Grossman, E. D., & Blake, R. (2002) · Neuron · doi:10.1016/s0896-6273(02)00897-8 · Cited by 686 · see citing works
Human parietal cortex in action
Culham, J. C., & Valyear, K. F. (2006) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2006.03.005 · Cited by 654 · see citing works
Visuo-haptic object-related activation in the ventral visual pathway
Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001) · Nature Neuroscience · doi:10.1038/85201 · Cited by 630 · see citing works
Ventral occipital lesions impair object recognition but not object‐directed grasping: an fMRI study
James, T. W., Culham, J. C., Humphrey, G. K., Milner, A. D., & Goodale, M. A. (2003) · Brain · doi:10.1093/brain/awg248 · Cited by 578 · see citing works
Self-citationAnalysis of a large fMRI cohort: statistical and methodological issues for group analyses
Thirion, B., Pinel, P., Mériaux, S., Roche, A., Dehaene, S., & Poline, J.-B. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.11.054 · Cited by 563 · see citing works
Convergence of Visual and Tactile Shape Processing in the Human Lateral Occipital Complex
Amedi, A. (2002) · Cerebral Cortex · doi:10.1093/cercor/12.11.1202 · Cited by 452 · see citing works
Haptic study of three-dimensional objects activates extrastriate visual areas
James, T. W., Humphrey, G. K., Gati, J. S., Servos, P., Menon, R. S., & Goodale, M. A. (2002) · Neuropsychologia · doi:10.1016/s0028-3932(02)00017-9 · Cited by 418 · see citing works
Self-citationMirror neurons and mirror systems in monkeys and humans.
Fabbri-Destro, M., & Rizzolatti, G. (2008) · Physiology · doi:10.1152/physiol.00004.2008 · Cited by 392 · see citing works
Multisensory integration, perception and ecological validity
de Gelder, B., & Bertelson, P. (2003) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2003.08.014 · Cited by 366 · see citing works
Distinct Cortical Pathways for Processing Tool versus Animal Sounds
Lewis, J. W., Brefczynski‐Lewis, J. A., Phinney, R. E., Janik, J. J., & DeYoe, E. A. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.0419-05.2005 · Cited by 331 · see citing works
Hierarchical Processing of Tactile Shape in the Human Brain
Bodegård, A., Geyer, S., Grefkes, C., Zilles, K., & Roland, P. E. (2001) · Neuron · doi:10.1016/s0896-6273(01)00362-2 · Cited by 265 · see citing works
Stevenson, R. A., & James, T. W. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.09.034 · Cited by 252 · see citing works
Self-citationHuman brain regions involved in recognizing environmental sounds
Lewis, J. W., Wightman, F. L., Brefczynski, J. A., Phinney, R. E., Binder, J. R., & DeYoe, E. A. (2004) · Cerebral Cortex · doi:10.1093/cercor/bhh061 · Cited by 230 · see citing works
Selective visuo‐haptic processing of shape and texture
Stilla, R. F., & Sathian, K. (2007) · Human Brain Mapping · doi:10.1002/hbm.20456 · Cited by 228 · see citing works
Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., & Rowland, B. A. (2009) · Experimental Brain Research · doi:10.1007/s00221-009-1880-8 · Cited by 221 · see citing works
Grounding Object Concepts in Perception and Action: Evidence from FMRI Studies of Tools
Beauchamp, M. S., & Martin, A. D. (2007) · Cortex · doi:10.1016/s0010-9452(08)70470-2 · Cited by 166 · see citing works
Binkofski, F., Kunesch, E., Claßen, J., Seitz, R. J., & Freund, H. (2001) · Brain · doi:10.1093/brain/124.1.132 · Cited by 147 · see citing works
Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects
Stevenson, R. A., Geoghegan, M. L., & James, T. W. (2006) · Experimental Brain Research · doi:10.1007/s00221-006-0770-6 · Cited by 104 · see citing works
Self-citationObserving Learned Object-specific Functional Grasps Preferentially Activates the Ventral Stream
Valyear, K. F., & Culham, J. C. (2009) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2009.21256 · Cited by 95 · see citing works
Bohlhalter, S. (2002) · Brain · doi:10.1093/brain/awf245 · Cited by 84 · see citing works
Stevenson, R. A., Kim, S., & James, T. W. (2009) · Experimental Brain Research · doi:10.1007/s00221-009-1783-8 · Cited by 68 · see citing works
Self-citationProbing category selectivity for environmental sounds in the human auditory brain
Doehrmann, O., Naumer, M. J., Volz, S., Kaiser, J., & Altmann, C. F. (2008) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2008.05.011 · Cited by 64 · see citing works
Kim, S., & James, T. W. (2009) · Human Brain Mapping · doi:10.1002/hbm.20897 · Cited by 60 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Baum, S. H., Stevenson, R. A., & Wallace, M. T. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.007 · Cited by 370 · see citing works
Self-citationStevenson, R. A., Zemtsov, R. K., & Wallace, M. T. (2012) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/a0027339 · Cited by 269 · see citing works
Crossmodal interactions in audiovisual emotion processing
Müller, V. I., Cieslik, E. C., Turetsky, B. I., & Eickhoff, S. B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.12.007 · Cited by 65 · see citing works
- Authors
- James, T. W., VanDerKlok, R. M., Stevenson, R. A., & James, K. H.
- Affiliations
- Indiana University (US)
- Corresponding author
- James, Thomas W.
- Venue
- Neuropsychologia
- Publisher
- Elsevier BV
- Published
- 2011-01
- Volume
- 49
- Issue
- 1
- Pages
- 108-114
- ISSN
- 0028-3932 (ISSN-L)
- PMC
- PMC3005003
- OpenAlex
- W2129332830
- Cited by
- 38
- References
- 37
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 4 · 2023: 1 · 2022: 1 · 2021: 1 · 2020: 1 · 2019: 1 · 2018: 2 · 2017: 1 · 2015: 4 · 2014: 5 · 2013: 5 · 2012: 3
- FWCI
- 5.02
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Neuropsychologia (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Lilly Endowment
- National Institutes of Health (5r01dc000111-38)
- National Institute on Deafness and Other Communication Disorders (T32DC00012, R01 DC-000111, T32DC000012)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Social Sciences › Psychology › Social Psychology › Action Observation and Synchronization
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Keywords
- multisensory perception, human action recognition, object-directed action, audiovisual integration, posterior middle temporal gyrus, lateral occipital cortex, superior temporal sulcus, intraparietal sulcus, event recognition, neural correlates
- MeSH terms
- Acoustic Stimulation, Auditory Perception, Brain Mapping, Electroencephalography, Female, Humans, Image Processing, Computer-Assisted, Functional Laterality, Magnetic Resonance Imaging, Male, Oxygen, Parietal Lobe, Photic Stimulation, Temporal Lobe, Visual Perception, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
