Multimodal Language Processing in Human Communication

Holler, Judith, Levinson, Stephen C.·2019·Trends in Cognitive Sciences

DOI 10.1016/j.tics.2019.05.006

Cited by (433)

Items 101–200 of 433

Leading voices: dialogue semantics, cognitive science and the polyphonic structure of multimodal interaction
Andy Lücking, Jonathan Ginzburg · 2022 · Language and Cognition
Multimodal mitigation: how facial and body cues index politeness in Catalan requests
Iris Hübscher, Cristina Sánchez-Conde, Joan Borràs-Comes, Laura Vincze, Pilar Prieto · 2022 · Journal of Politeness Research
Efficient trade-offs as explanations in functional linguistics: some problems and an alternative proposal
Natalia Levshina · 2020 · ABRALIN
Facial clues to conversational intentions
Judith Holler · 2025 · Trends in Cognitive Sciences
The ‘design features’ of language revisited
Michael Pleyer, Marcus Perlman, Gary Lupyan, Koen de Reus, Limor Raviv · 2025 · Trends in Cognitive Sciences
Attentional Relevance Modulates Nonverbal Attractiveness Perception in Multimodal Display
Yanbing Hu, Zhen Mou, Xiaoming Jiang · 2023 · Journal of Nonverbal Behavior
Combining EEG and 3D-eye-tracking to study the prediction of upcoming speech in naturalistic virtual environments: A proof of principle
Eleanor Huizeling, Phillip M. Alday, David Peeters, Peter Hagoort A · 2023 · Neuropsychologia
Interactional Infrastructure across Modalities: A Comparison of Repair Initiators and Continuers in British Sign Language and British English
Hannah Lutzenberger, Lierin de Wael, Rehana Omardeen, Mark Dingemanse · 2024 · Sign language studies
Action-Depicting Gestures and Morphosyntax: The Function of Gesture-Speech Alignment in the Conversational Turn
Paweł Urbanik, Jan Svennevig · 2021 · Frontiers in Psychology
Multisensory Input Modulates P200 and L2 Sentence Comprehension: A One-Week Consolidation Phase
Nasim Boustani, Reza Pishghadam, Shaghayegh Shayesteh · 2021 · Frontiers in Psychology
Audiovisual Perception of Lexical Stress: Beat Gestures and Articulatory Cues
Ronny Bujok, Antje S. Meyer, Hans Rutger Bosker · 2024 · Language and Speech
Visual Information in Computer-Mediated Interaction Matters: Investigating the Association Between the Availability of Gesture and Turn Transition Timing in Conversation
James P. Trujillo, Stephen C. Levinson, Judith Holler · 2021 · Lecture notes in computer science
Conversational eyebrow frowns facilitate question identification: An online VR study
Naomi Nota, James P. Trujillo, Judith Holler · 2022
Specific learning disorders in young adults: Investigating pragmatic abilities and their relationship with theory of mind, executive functions and quality of life.
Michela Camia, Erika Benassi, Sara Giovagnoli, Maristella Scorza · 2022 · Research in Developmental Disabilities
Musical communication among parents and their children: An analysis tool to study their interaction
Daniela Lerma-Arregocés, Jèssica Pérez–Moreno · 2023 · International Journal of Music Education
Development of communicative-pragmatic abilities in children with early cochlear implants
Alberto Parola, Dize Hilviu, S. Vivaldo, Andrea Marini, Diego Di Lisi, Patrizia Consolino, Francesca M. Bosco · 2023 · Journal of Child Language
The fuzzy boundaries of the social (pragmatic) communication disorder (SPCD): Why the picture is still so confusing?
Ilaria Gabbatore, Agata Marchetti Guerrini, Francesca M. Bosco · 2023 · Heliyon
Evaluating the Relative Perceptual Salience of Linguistic and Emotional Prosody in Quiet and Noisy Contexts
Minyue Zhang, Hui Zhang, Enze Tang, Hongwei Ding, Yang Zhang · 2023 · Behavioral Sciences
The role of general cognitive skills in integrating visual and linguistic information during sentence comprehension: individual differences across the lifespan
Florian Hintz, Cesko C. Voeten, Dorottya Dobó, Krisztina Sára Lukics, Ágnes Lukács · 2024 · Scientific Reports
Towards a novel conceptualization of prosody that accounts for spoken and visual signals
Pilar Prieto, Núria Esteve‐Gibert, Stefanie Shattuck‐Hufnagel · 2024 · Gesture
Gesture, spatial cognition and the evolution of language
Stephen C. Levinson · 2023 · Philosophical Transactions of the Royal Society B Biological Sciences
Models of Possibilities Instead of Logic as the Basis of Human Reasoning
Philip Nicholas Johnson-Laird, Ruth M. J. Byrne, Sangeet Khemlani · 2024 · Minds and Machines
The ECOLANG Multimodal Corpus of adult-child and adult-adult Language
Yan Gu, Ed Donnellan, Beata J. Grzyb, Gwen Brekelmans, Margherita Murgiano, Ricarda Brieke, Pamela Perniss, Gabriella Vigliocco · 2025 · Scientific Data
Toddler negative emotion expression and parent-toddler verbal conversation: Evidence from daylong recordings
Margaret A. Fields‐Olivieri, Pamela M. Cole · 2022 · Infant Behavior and Development
Multimodal Attention Dynamic Fusion Network for Facial Micro-Expression Recognition
Hongling Yang, Lun Xie, Hang Pan, Chiqin Li, Zhiliang Wang, Jialiang Zhong · 2023 · Entropy
Using networks to visualize, analyse and interpret multimodal communication
Severine B. S. W. Hex, Daniel I. Rubenstein · 2023 · Animal Behaviour
Are Communication Skills, Emotion Regulation and Theory of Mind Skills Impaired in Adolescents with Developmental Dyslexia?
Gonca Özyurt, Yusuf Öztürk, Serkan Turan, Remzi Oğulcan Çıray, Ezgi Karagöz Tanıgör, Çağatay Ermiş, Ali Evren Tufan, Aynur Pekcanlar Akay · 2024 · Developmental Neuropsychology
Event related spectral perturbations of gesture congruity: Visuospatial resources are recruited for multimodal discourse comprehension
Jacob Momsen, Jared Gordon, Ying Choon Wu, Seana Coulson A · 2021 · Brain and Language
What's your point? Insights from virtual reality on the relation between intention and action in the production of pointing gestures
Renuka Raghavan, Limor Raviv, David Peeters · 2023 · Cognition
Language specific constraints on conversation: Evidence from Danish and Norwegian
Christina Dideriksen, Morten H. Christiansen, Mark Dingemanse, Malte Højmark‐Bertelsen, Christer Johansson, Kristian Tylén, Riccardo Fusaroli · 2022
Experimental Semiotics: past, present, and future
Jonas Nölle, Bruno Galantucci · 2021
Multimodality matters in numerical communication
Bodo Winter, Tyler Marghetis · 2023 · Frontiers in Psychology
The role of co-speech gestures in retrieval and prediction during naturalistic multimodal narrative processing
Sergio Osorio, Benjamin Straube, Lars Meyer, Yifei He · 2023 · Language Cognition and Neuroscience
Visual bodily signals and conversational context benefit the anticipation of turn ends
Marlijn ter Bekke, Stephen C. Levinson, Lina van Otterdijk, Michelle Kühn, Judith Holler · 2024 · Cognition
Longitudinal assessment of pragmatic and cognitive decay in healthy aging, and interplay with subjective cognitive decline and cognitive reserve
Ilaria Gabbatore, Riccardo Conterio, Giorgia Vegna, Francesca M. Bosco · 2025 · Scientific Reports
Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia
Yifei He, Miriam Steines, Gebhard Sammer, Arne Nagels, Tilo T. J. Kircher, Benjamin Straube · 2021 · NeuroImage Clinical
Planning when to say: Dissociating cue use in utterance initiation using cross-validation.
Laurel Ellen Brehm, Antje S. Meyer · 2021 · Journal of Experimental Psychology General
Towards a temporospatial framework for measurements of disorganization in speech using semantic vectors
Terje B. Holmlund, Chelsea Chandler, Peter W. Foltz, Catherine M. Diaz-Asper, Alex S. Cohen, Zachary B. Rodriguez, Brita Elvevåg · 2022 · Schizophrenia Research
The prediction potential indexes the meaning and communicative function of upcoming utterances
Salomé Antoine, Luigi Grisoni, Rosario Tomasello, Friedemann Pulvermüller A · 2024 · Cortex
A cross-species framework for classifying sound-movement couplings
Silvia Leonetti, Andrea Ravignani, Wim T. J. L. Pouw · 2024 · Neuroscience & Biobehavioral Reviews
The effect of gesture expressivity on emotional resonance in storytelling interaction
Christoph Rühlemann, James P. Trujillo · 2024 · Frontiers in Psychology
Looking for social pragmatic communication disorder in the complex world of Italian special needs: an exploratory study
Ilaria Gabbatore, Agata Marchetti Guerrini, Francesca M. Bosco · 2025 · Scientific Reports
Co-Speech Hand Gestures Are Used to Predict Upcoming Meaning
Marlijn ter Bekke, Linda Drijvers, Judith Holler · 2025 · Psychological Science
Can Large Language Models Simulate Spoken Human Conversations?
E. Mayor, L. Bietti, Adrian Bangerter · 2025 · Cognitive Sciences
Brexit means Brexit: a constructionist analysis
Jose A. Mompeán, Javier Valenzuela · 2019 · Estudios Ingleses de la Universidad Complutense
Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension
Florian Hintz, Yung Han Khoe, Antje Strauß, Adam Johannes Alfredo Psomakas, Judith Holler · 2023 · Cognitive Affective & Behavioral Neuroscience
Beyond the tried and true: How virtual reality, dialog setups, and a focus on multimodality can take bilingual language production research forward
Alex Titus, Ton Dijkstra, Roel M. Willems, David Peeters · 2023 · Neuropsychologia
The electrophysiology of lexical prediction of emoji and text
Benjamin Weissman, Neil Cohn, Darren Tanner · 2024 · Neuropsychologia
Establishing conversational engagement and being effective: The role of body movement in mediated communication
Jens F. Binder · 2023 · Acta Psychologica
Emotion dysregulation and social communication problems but not ToM properties may predict obsessive-compulsive disorder symptom severity
Yusuf Öztürk, Gonca Özyurt, Serkan Turan, Ali Evren Tufan, Aynur Pekcanlar Akay · 2023 · Nordic Journal of Psychiatry
Cognitive Linguistics meets Interactional Linguistics: Language development in the arena of language use
Heike Behrens, Stefan Pfänder · 2022 · Yearbook of the German Cognitive Linguistics Association
Forms and Functions of Gestures in Preverbal 12‐ to 15‐Months Old Infants
Shreejata Gupta, Eulalie Pequay, François Clément, Isabelle Dautriche · 2024 · Infancy
Assessing communicative-pragmatic ability in telehealth: e-ABaCo in autistic individuals
Ilaria Traetta, Ilaria Gabbatore, Alessandra Aimar, Giuseppe Maurizio Arduino, Francesca M. Bosco · 2025 · Frontiers in Psychiatry
Pragmatic and narrative decline in healthy aging: a multilevel assessment of their interplay
Francesco Petriglia, Andrea Marini, Ilaria Gabbatore, Francesca M. Bosco · 2025 · Journal of Communication Disorders
The multimodal construction of political personae through the strategic management of semiotic resources of emotion expression
Nicolae-Sorin Drăgan, Gheorghe-Ilie Fârte · 2022 · Social Semiotics
The impact of face masks on face-to-face neural tracking of speech: Auditory and visual obstacles
Marta Fantoni, Alessandra Federici, Ivan Camponogara, Giacomo Handjaras, Alice Martinelli, Evgenia Bednaya, Emiliano Ricciardi, Francesco Pavani, Davide Bottari · 2024 · Heliyon
Evidence For Selective Adaptation and Recalibration in the Perception of Lexical Stress
Hans Rutger Bosker · 2021 · Language and Speech
“Chunking” spoken language: Introducing weak cesuras
Dagmar Barth-Weingarten, Richard Ogden · 2021 · Open Linguistics
Visual Linguistics with R
Christoph Rühlemann · 2020 · John Benjamins Publishing Company eBooks
The Processing of Semantic Complexity and Cospeech Gestures in Schizophrenia: A Naturalistic, Multimodal fMRI Study
Paulina Cuevas, Yifei He, Miriam Steines, Benjamin Straube · 2022 · Schizophrenia Bulletin Open
Multimodal communication in instructional settings: An investigation of the functional roles of gestures and arrows
Cengiz Acartürk, Melda Coskun, Serap Emil · 2021 · Revista signos
Translation, Cross-Cultural Adaptation, and Validation of a Dutch Version of the Actions and Feelings Questionnaire in Autistic and Neurotypical Adults
Hedwig A. van der Meer, Irina Simanova, Cornelis C. Kan, James P. Trujillo · 2021 · Journal of Autism and Developmental Disorders
tipos de interacción musical en el contexto familiar y cotidiano
Daniela Lerma-Arregocés · 2023 · Epistemus Revista de Estudios en Música Cognición y Cultura
Civil inattention—On the sources of relational segregation
Ilkka Arminen, Anna S. M. Heino · 2023 · Frontiers in Sociology
Facilitating question identification through natural intensity eyebrow movements in virtual avatars
Naomi Nota, James P. Trujillo, Vere Jacobs, Judith Holler · 2023 · Scientific Reports
Flexible use of multimodal communicative strategies in adult chimpanzees
Angèle Lombrey, Adriana Luna Martinez, Nick Dannenmann, Katerina Harvati, Ulrich Knief, Marlen Fröhlich · 2025 · Scientific Reports
Simulating the “cultural other”: the impact of GenAI interlocutors on EFL learners’ intercultural communicative competence, speaking anxiety, and learning engagement
Zhanbo Qu, Liyun Chen · 2026 · Frontiers in Psychology
When Language Evolution Meets Multimodality: Current Status and Challenges Toward Multimodal Computational Models
Patrizia Grifoni, Arianna D’Ulizia, Fernando Ferri · 2021 · IEEE Access
Word frequency and cognitive effort in turns-at-talk: turn structure affects processing load in natural conversation
Christoph Rühlemann, Mathias Barthel · 2024 · Frontiers in Psychology
Beat gestures and prosodic prominence interactively influence language comprehension
Ambra Ferrari, Peter Hagoort A · 2024 · Cognition
How Children With and Without Developmental Language Disorder Use Prosody and Gestures to Process Phrasal Ambiguities
Albert Giberga, Ernesto Guerra, Nadia Ahufinger, Alfonso Igualada, Mari Aguilera, Núria Esteve‐Gibert · 2025 · Languages
Multi-modal emotion prediction system using convergence media and active contents
Kyungyong Chung, Jin-Su Kim · 2021 · Personal and Ubiquitous Computing
Predicting Conversational Turns: Signers' and Nonsigners' Sensitivity to Language-Specific and Globally Accessible Cues
Connie De Vos, Marisa Casillas, Tom Uittenbogert, Onno A. Crasborn, Stephen C. Levinson · 2022 · Language
An AI-powered Hierarchical Communication Framework for Robust Human-Robot Collaboration in Industrial Settings
Debasmita Mukherjee, Kashish Gupta, Homayoun Najjaran · 2022 · IEEE RO-MAN
An Outlook for AI Innovation in Multimodal Communication Research
Alexander Henlein, Anastasia Bauer, Reetu Bhattacharjee, Aleksandra Ćwiek, Alina Gregori, Frank Kügler, Jens Lemanski, Andy Lücking, Alexander Mehler, Pilar Prieto et al. · 2024 · Lecture notes in computer science
Gesturing during disfluent speech: A pragmatic account
Yağmur Deniz Kısa, Susan Goldin-meadow A, Daniel J. Casasanto · 2024 · Cognition
From neural matter to rapid symbolic learning in brains and artificial neural networks: a brief overview and perspective
Rosario Tomasello · 2025 · Linguistics Vanguard
METFORC: Classification with Meta-Learning and Multimodal Stratified Time Series Forest
Pouya Hosseinzadeh, Omar Bahri, Peiyu Li, Soukaïna Filali Boubrahimi, Shah Muhammad Hamdi · 2023 · International Conference on Machine Learning and Applications
Iconic Words Are Associated With Iconic Gestures
Ell Wilding, Bodo Winter, Jeannette Littlemore, Marcus Perlman · 2025 · Cognitive Science
Combining Vision and Language Representations for Patch-based Identification of Lexico-Semantic Relations
Prince Jha, Gaël Harry Dias, Alexis Lechervy, José G. Moreno, Anubhav Jangra, Sebastião Pais, Sriparna Saha · 2022 · ACM International Conference on Multimedia (ACM MM)
Communicative constraints affect oro-facial gestures and acoustics: Whispered vs normal speech
Marzena Żygis, Susanne I. Fuchs · 2023 · The Journal of the Acoustical Society of America
Developing an Artificial Neural Network-based model for predicting EFL achievement level
Amani Bouzayenne, Harizi Riadh · 2024 · Multimedia Tools and Applications
“We’ve lost you Ian”: Multi-modal corpus innovations in capturing, processing and analysing professional online spoken interactions
Anne O’Keeffe, Dawn Knight, Geraldine Mark, Christopher Fitzgerald, Justin McNamara, Svenja Adolphs, Benjamin R. Cowan, Tania Fahey Palma, Fiona Farr, Sandrine Peraldi · 2024 · Research in Corpus Linguistics
Facial signals shape predictions about the nature of upcoming conversational responses
Alexandra K. Emmendorfer, Judith Holler · 2025 · Scientific Reports
Pragmatics in the Health Sciences
Maria Garraffa, Greta Mazzaggio · 2025 · Cambridge University Press eBooks
The time course of speech act recognition conveyed by speech prosody: a gating study
Caterina Villani, Isabella Paola Boux, Friedemann Pulvermüller A, Rosario Tomasello · 2025 · Language Cognition and Neuroscience
Beyond Robotic Speech: Mutual Benefits to Cognitive Psychology and Artificial Intelligence from the Study of Multimodal Communication
Beata J. Grzyb, Gabriella Vigliocco · 2021 · Oxford University Press eBooks
TMS reveals a two-stage priming circuit of gesture-speech integration
Wanying Zhao · 2023 · Frontiers in Psychology
Conversation and the evolution of metacognition
Ronald J. Planer · 2023 · Evolutionary Linguistic Theory
Word predictability, prosody, gesture and mouth movements in face-to-face language comprehension
Ye Zhang, Diego Frassinelli, Jyrki Tuomainen, Jeremy I Skipper, Gabriella Vigliocco · 2020 · bioRxiv (Cold Spring Harbor Laboratory)
Turn-taking fluency in free conversations with individuals diagnosed with schizophrenia
Tifenn Fauviaux, Ghilès Mostafaoui, R. C. Schmidt, Mathilde Parisi, Victor Vattier, Dorra Mrabet, Delphine Capdevielle, Stéphane Raffard, Ludovic Marín · 2025 · Schizophrenia
Multimodal information density is highest in question beginnings, and early entropy is associated with fewer but longer visual signals
James P. Trujillo, Judith Holler · 2025 · Discourse Processes
Cross-modal co-occurrence analysis of nonverbal behavior and content words, vocal emotion and prosody in individuals with subclinical depression
Liusheng Wang, Fuying Wu, Haiyan Zhang, Dongfang Lin · 2025 · BMC Psychology
Rethinking linguistic feedback: A modality-agnostic and holistic approach to multimodal addressee signals in spoken and signed dyadic interaction
Anastasia Bauer, Sonja Gipper, Tobias-Alexander Herrmann, Jana Hosemann · 2025 · Glossa a journal of general linguistics
Social communication development in a contingent world: insights from autism
Kelsey L. West, Antonia M. H. Piergies, Camila Alviar, Miriam Diane Lense · 2026 · Philosophical Transactions of the Royal Society B Biological Sciences
Better early than late: the temporal dynamics of pointing cues during cross-situational word learning
Rachael W. Cheung, Calum Hartley, Padraic Monaghan B · 2024 · Language and Cognition
Language cognition and language computation — human and machine language understanding
少楠王, 鼐丁, 楠林, 家俊张, 成庆宗 · 2021 · Scientia Sinica Informationis
Differences in functional brain organization during gesture recognition between autistic and neurotypical individuals
James P. Trujillo, Aslı Özyürek, Cornelis C. Kan, Irina Simanova, Harold Bekkering A · 2022 · Social Cognitive and Affective Neuroscience
Symbolic and deictic gestures as a tool to promote parent-child communication in the context of hearing loss: A systematic review
Arianna Colombani, Amanda Saksida, Francesco Pavani, Eva Orzan · 2022 · International Journal of Pediatric Otorhinolaryngology
Design and Application of Multimedia Technology-Based Curriculum for Visual Communication Majors
Chibo Zhang, Yongli Zhao · 2023 · Advances in Multimedia