Multiple Temporalities of Language and Body in Interaction: Challenges for Transcribing Multimodality
The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
The article focuses on the principles of multimodal CA, the way they can be operationalized in a transcription system, and the analytical and conceptual consequences of transcription choices. Elaborating on the foundations of multimodal CA and on the basis of video recordings of French and Swiss German encounters, as well as animal interactions, the article discusses classic and contemporary challenges for transcription and analysis, such as beyond gesture and gaze, body arrangements in interactional spaces, larger groups, material environments, mobile settings, silent activities, and animal encounters. It also highlights the diversity of multimodal practices involved: mobilizing occasioned material resources, movements not only of the upper (head, gesture) but also the lower (feet, legs, posterior) parts of the body, haptic contacts touching objects and coparticipants, and camera movements. The precise transcription of relevant details reveals complex arrangements of multimodal resources and gestalts. Their fine-grained, distinct, multiple temporalities constitute the basis of their sequential order—for sequentiality as a fundamental organizational principle of action. Data are in French and Swiss German.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
(1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511665868 · Cited by 4,133
Gesture Visible Action as Utterance
Kendon, A. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511807572 · Cited by 3,891
Action and embodiment within situated human interaction
Goodwin, C. (2000) · Journal of Pragmatics · doi:10.1016/s0378-2166(99)00096-x · Cited by 3,073
Everyday Language: Studies in Ethnomethodology
Psathas, G. (1979) · Human Studies · Cited by 2,349
Conversational Organization: Interaction Between Speakers and Hearers
Goodwin, C. (1981) · Cited by 2,305
Hand and Mind: What Gestures Reveal about Thought
Feyereisen, P., & McNeill, D. (1994) · The American Journal of Psychology · doi:10.2307/1423299 · Cited by 1,688
Conducting Interaction: Patterns of Behavior in Focused Encounters.
Burgoon, J. K., & Kendon, A. (1992) · Contemporary Sociology · doi:10.2307/2075490 · Cited by 1,634
Stance, Alignment, and Affiliation During Storytelling: When Nodding Is a Token of Affiliation
Stivers, T. (2008) · Research on Language and Social Interaction · doi:10.1080/08351810701691123 · Cited by 1,408
Gesticulation and Speech: Two Aspects of the Process of Utterance
KENDON, A. (1980) · The Relationship of Verbal and Nonverbal Communication · doi:10.1515/9783110813098.207 · Cited by 1,205
The Interactive Construction of a Sentence in Natural Conversation
Goodwin, C. (1979) · Cited by 1,146
Embodied Interaction: Language And Body In The Material World
Streeck, J., Goodwin, C., & Lebaron, C. D. (2014) · Cited by 1,127
Body Movement and Speech in Medical Interaction
Heath, C., & Nicholls, K. (1986) · Cambridge University Press eBooks · doi:10.1017/cbo9780511628221 · Cited by 1,044
Challenges of multimodality: Language and the body in social interaction
Mondada, L. (2016) · Journal of Sociolinguistics · doi:10.1111/josl.1_12177 · Cited by 768 · cited 7×
“This has made it possible to speak of a “visual,” “embodied,” or “multimodal” turn in the discipline (Mondada, 2016a; Nevile, 2015).”
BackgroundInfluentialSelf-citationMultimodal resources for turn-taking: pointing and the emergence of possible next speakers
Mondada, L. (2007) · Discourse Studies · doi:10.1177/1461445607075346 · Cited by 635
Self-citationHeath, C., & Luff, P. (2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511489839 · Cited by 628
JEFFERSON, G. (1973) · Semiotica · doi:10.1515/semi.1973.9.1.47 · Cited by 600
Shatz, M., Ochs, E., & Schieffelin, B. B. (1981) · Language · doi:10.2307/413708 · Cited by 586
Seeing as situated activity: Formulating planes
Goodwin, C., & Goodwin, M. H. (1996) · Cognition and Communication at Work · doi:10.1017/cbo9781139174077.004 · Cited by 584
Mondada, L. (2009) · Journal of Pragmatics · doi:10.1016/j.pragma.2008.09.019 · Cited by 553
Self-citationSacks, H. (1985) · Structures of Social Action · doi:10.1017/cbo9780511665868.005 · Cited by 549
The local constitution of multimodal resources for social interaction
Mondada, L. (2014) · Journal of Pragmatics · doi:10.1016/j.pragma.2014.04.004 · Cited by 536
Self-citationConcurrent operations on talk: Notes on the interactive organization of assessments
Goodwin, C., & Goodwin, M. H. (2022) · IPrA Papers in Pragmatics · doi:10.1075/iprapip.1.1.01goo · Cited by 532
On some gestures' relation to talk
Schegloff, E. A. (1985) · Structures of Social Action · doi:10.1017/cbo9780511665868.018 · Cited by 493
Visualisation and Cognition: Drawing Things Together
Latour, B. (2012) · Cited by 429
Transcribing for Social Research
Hepburn, A., & Bolden, G. B. (2017) · SAGE Publications Ltd eBooks · doi:10.4135/9781473920460 · Cited by 410
Gesture as communication I: Its coordination with gaze and speech
Streeck, J. (1993) · Communication Monographs · doi:10.1080/03637759309376314 · Cited by 401
15 Environmentally Coupled Gestures
Goodwin, C. (2007) · Gesture Studies · doi:10.1075/gs.1.18goo · Cited by 321
Gesturecraft: The manu-facture of meaning
Streeck, J. (2009) · Gesture Studies · doi:10.1075/gs.2 · Cited by 280
The Embodied Turn in Research on Language and Social Interaction
Nevile, M. (2015) · Research on Language and Social Interaction · doi:10.1080/08351813.2015.1025499 · Cited by 272
A sketch of some orderly aspects of overlap in natural conversation
Jefferson, G. (2004) · Pragmatics & Beyond New Series · doi:10.1075/pbns.125.05jef · Cited by 271
Multimodal interaction from a conversation analytic perspective
Deppermann, A. (2013) · Journal of Pragmatics · doi:10.1016/j.pragma.2012.11.014 · Cited by 262
Multimodal transcription as academic practice: a social semiotic perspective
Bezemer, J., & Mavers, D. (2011) · International Journal of Social Research Methodology · doi:10.1080/13645579.2011.563616 · Cited by 258
Walking away: The embodied achievement of activity closings in mobile interaction
Broth, M., & Mondada, L. (2013) · Journal of Pragmatics · doi:10.1016/j.pragma.2012.11.016 · Cited by 199
Self-citationSocial relationships and ritualized greetings in adult male baboons (Papio cynocephalus anubis)
Smuts, B. B., & Watanabe, J. M. (1990) · International Journal of Primatology · doi:10.1007/bf02192786 · Cited by 198
Gaze behavior in face-to-face interaction
Rossano, F. (2012) · MPG.PuRe (Max Planck Society) · Cited by 179
Interacting with objects: Language, materiality, and social activity
Nevile, M., Haddington, P., Heinemann, T., & Rauniomaa, M. (2014) · Cited by 178
Multiactivity in Social Interaction: Beyond multitasking
(2014) · John Benjamins Publishing Company eBooks · doi:10.1075/z.187 · Cited by 176
The Interdependence of Bodily Demonstrations and Clausal Syntax
Keevallik, L. (2013) · Research on Language and Social Interaction · doi:10.1080/08351813.2013.753710 · Cited by 165
Conducting interaction patterns of behavior in focused encounters
Kendon, A. (1990) · Cited by 165
Haddington, P., Mondada, L., & Nevile, M. (2013) · De Gruyter eBooks · doi:10.1515/9783110291278 · Cited by 158
Self-citationBodies in action: Multimodal analysis of walking and talking
Mondada, L. (2014) · Language and Dialogue · doi:10.1075/ld.4.3.02mon · Cited by 153
Self-citationEnvironmentally Coupled Gestures
(2017) · Co-Operative Action · doi:10.1017/9781139016735.017 · Cited by 137
Facial and Verbal Expressions in Assessing Stories and Topics
Ruusuvuori, J., & Peräkylä, A. (2009) · Research on Language and Social Interaction · doi:10.1080/08351810903296499 · Cited by 121
Pointing, talk, and the bodies: Reference and joint attention as embodied interactional achievements
Mondada, L. (2014) · John Benjamins Publishing Company eBooks · doi:10.1075/z.188.06mon · Cited by 116
Self-citationSequence organization and timing of bonobo mother-infant interactions
Rossano, F. (2013) · Interaction Studies Social Behaviour and Communication in Biological and Artificial Systems · doi:10.1075/is.14.2.02ros · Cited by 113
Ryave, A. L., & Schenkein, J. N. (1974) · Cited by 111
Introduction: A body of resources – CA studies of social conduct
Hazel, S., Mortensen, K., & Rasmussen, G. (2013) · Journal of Pragmatics · doi:10.1016/j.pragma.2013.10.007 · Cited by 103
Writing-in-interaction: Studying writing as a multimodal phenomenon in social interaction
Mondada, L., & Svinhufvud, K. (2016) · Language and Dialogue · doi:10.1075/ld.6.1.01mon · Cited by 85
Self-citationThe Joy of Biking Together: Sharing Everyday Experiences of Vélomobility
McIlvenny, P. (2013) · Mobilities · doi:10.1080/17450101.2013.844950 · Cited by 84
(2014) · Routledge eBooks · doi:10.4324/9781315851709 · Cited by 79
Inclined to better understanding—The coordination of talk and ‘leaning forward’ in doing repair
Rasmussen, G. (2013) · Journal of Pragmatics · doi:10.1016/j.pragma.2013.10.001 · Cited by 67
Mondada, L. (2014) · Studies in language and social interaction · doi:10.1075/slsi.26.11mon · Cited by 59
Self-citationThe Dynamics of Auction: Social Interaction and the Sale of Fine Art and Antiques
Heath, C. (2012) · Cited by 56
Making multiple requests in French and Finnish convenience stores
Mondada, L., & Sorjonen, M. (2016) · Language in Society · doi:10.1017/s0047404516000646 · Cited by 56
Self-citationThe studio interaction as a contextual resource for TV-production
Broth, M. (2008) · Journal of Pragmatics · doi:10.1016/j.pragma.2007.10.010 · Cited by 46
The Organization of Turn-Taking in Pool Skate Sessions
Ivarsson, J., & Greiffenhagen, C. (2015) · Research on Language and Social Interaction · doi:10.1080/08351813.2015.1090114 · Cited by 33
Research works printed but not linked: 17
- 10.1177/1461445615590717 · doi:10.1177/1461445615590717
- Bergmann, J. (1985). Flüchtigkeit und methodische fixierung sozialer wirklichkeit: Aufzeichnungen als daten der interpretativen soziologie [Fleetingness and methodological fixation of social reality]. In W. Bonss & H. Hartmann (Eds.), Entzauberte Wissenschaft (pp. 299–320). Göttingen, Germany: Schwarz.
- Birdwhistell, R. L. (1970). Kinesics in context: Essays on body motion communication. Philadelphia, PA: University of Pennsylvania Press.
- Bulwer, J. (1644). Chirologia. London, England: Harper.
- Condon, W. S. (1971). Speech and body motion synchrony of the speaker-hearer. In D. L. Horton & J. J. Jenkins (Eds.), Perception of language (pp. 150–173). Columbus, OH: Merrill.
- De Stefani, E. (2010). Reference as an interactively and multimodally accomplished practice. In M. Pettorino, A. Giannini, I. Chiari, & F. Dovetto (Eds.), Spoken communication (pp. 137–170). Newcastle, England: Cambridge Scholars Publishing.
- 10.1525/aa.1994.96.issue-3 · doi:10.1525/aa.1994.96.issue-3
- Haddington, P., Keisanen, T., Mondada, L., & Nevile, M. (Eds.). (2015). Multiactivity in social interaction. Amsterdam, The Netherlands: John Benjamins.
- Jefferson, G. (1985). An exercise in the transcription and analysis of laughter. In T. A. Van Dijk (Ed.), Handbook of discourse analysis volume 3 (pp. 25–34). New York, NY: Academic Press.
- Lerner, G. H., & Zimmerman, D. H. (2002). Action and the appearance of action in the conduct of very young children. In P. Glenn, C. LeBaron, & J. Mandelbaum (Eds.), Studies in language and social interaction (pp. 441–457). Mahwah, NJ: Lawrence Erlbaum.
- 10.1002/9781118584194.ch17 · DOI
- McQuown, N. (1971). The natural history of an interview. Chicago, IL: Microfilm Collection, University of Chicago. (Original work published 1955)
- 10.1075/ld · doi:10.1075/ld
- Mondada, L. (2014d). Shooting as a research activity: The embodied production of video data. In M. Broth, E. Laurier, & L. Mondada (Eds.), Video at work (pp. 33–62). London, England: Routledge.
- Mondada, L. (2016b). Zwischen text und bild: Multimodale transkription [Between text and image: Multimodal transcriptions]. In H. Hausendorf, R. Schmitt, & W. Kesselheim (Eds.), Interaktionsarchitektur, sozialtopograhie und interaktionsraum (pp. 111–160). Tübingen, Germany: Narr.
- 10.1075/gest · doi:10.1075/gest
- Schegloff, E. A. (1998). Body torque. Social Research, 65(3), 535–586.
Cited by
Mondada, L. (2019) · Journal of Pragmatics · doi:10.1016/j.pragma.2019.01.016 · Cited by 490
Self-citationMultimodal Language Processing in Human Communication
Holler, J., & Levinson, S. C. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.05.006 · Cited by 441
“For example, lip movements tend to precede corresponding phonation by up to 100– 300 ms in natural speech [1], referential gestures tend to precede corresponding lexical information by several hundredmilliseconds to several seconds even [2–4] (see also [5,6]), with this timing being slightly tighter for pitch accents…”
“Pepper, what do you mean?” Miscommunication and repair in robot-led survey interaction
Stommel, W., de Rijk, L., & Boumans, R. (2022) · 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man53752.2022.9900528 · Cited by 19
“Transcription symbols, adapted from [35; 36]”
Tai, K. W. H., & Choi, K. C. K. (2026) · Journal of Multilingual and Multicultural Development · doi:10.1080/01434632.2025.2580563 · Cited by 11
Embodied-Visual Practices during Conversational Repair: Scoping Review
Saalasti, S., Pajo, K., Fox, B., Pekkala, S., & Laakso, M. (2023) · Research on Language and Social Interaction · doi:10.1080/08351813.2023.2272528 · Cited by 9
Metadata
- Authors
- Mondada, L.
- Venue
- Research on Language and Social Interaction
- Publisher
- Informa UK Limited
- Published
- 2018-01-02
- Volume
- 51
- Issue
- 1
- Pages
- 85-106
- ISSN
- 0835-1813 (ISSN-L)
- 1532-7973
- OpenAlex
- W2793998950
- Cited by
- 1,056
- References
- 57
- FWCI
- 377.35
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.