CoChat
Sign Up
Cited by 37Phonetics and Phonology ResearchHearing Impairment and CommunicationNeurobiology of Language and Bilingualism

Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language

Open access
Kelly, Spencer D., Hirata, Yukari, Manansala, Michael, Huang, Jessica·2014·Frontiers in Psychology
DOI 10.3389/fpsyg.2014.00673PMC PMC4077026
Open this paper in CoChat — free accountOpen PDF

Abstract

Co-speech hand gestures are a type of multimodal input that has received relatively little attention in the context of second language learning. The present study explored the role that observing and producing different types of gestures plays in learning novel speech sounds and word meanings in an L2. Naïve English-speakers were taught two components of Japanese-novel phonemic vowel length contrasts and vocabulary items comprised of those contrasts-in one of four different gesture conditions: Syllable Observe, Syllable Produce, Mora Observe, and Mora Produce. Half of the gestures conveyed intuitive information about syllable structure, and the other half, unintuitive information about Japanese mora structure. Within each Syllable and Mora condition, half of the participants only observed the gestures that accompanied speech during training, and the other half also produced the gestures that they observed along with the speech. The main finding was that participants across all four conditions had similar outcomes in two different types of auditory identification tasks and a vocabulary test. The results suggest that hand gestures may not be well suited for learning novel phonetic distinctions at the syllable level within a word, and thus, gesture-speech integration may break down at the lowest levels of language processing and learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fpsyg.2014.00673

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. Asher J. J. (1969). The total physical response approach to second language learning. Mod. Lang. J. 53, 3– 17 10.2307/322091 · DOI
  2. Bernardis P. Gentilucci M. (2006). Speech and gesture share the same communication system. Neuropsychologia 44, 178– 190 10.1016/j.neuropsychologia.2005.05.007 16005477 · DOI
  3. Besle J. Fort A. Delpuech C. Giard M. H. (2004). Bimodal speech: early suppressive visual effects in human auditory cortex. Eur. J. Neurosci. 20, 2225– 2234 10.1111/j.1460-9568.2004.03670.x 15450102 PMC1885424 · DOI
  4. Biau E. Soto-Faraco S. (2013). Beat gestures modulate auditory integration in speech perception. Brain Lang. 124, 143– 152 10.1016/j.bandl.2012.10.008 23333667 · DOI
  5. Bundgaard-Nielsen R. L. Best C. T. Tyler M. D. (2011). Vocabulary size is associated with second-language vowel perception performance in adult learners. Stud. Second Lang. Acquis. 33, 433– 461 10.1017/S0272263111000040 · DOI
  6. Calvert G. Spence C. Stein B. E. (eds.). (2004). The Handbook of Multisensory Processes. Cambridge, MA: The MIT Press
  7. Calvert G. A. Brammer M. J. Bullmore E. T. Campbell R. Iversen S. D. David A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10, 2619– 2623 10.1097/00001756-199908200-00033 10574380 · DOI
  8. Clark H. H. (1996). Using Language. Cambridge: Cambridge University Press
  9. Cohen R. L. (1989). Memory for action events: the power of enactment. Educ. Psychol. Rev. 1, 57– 80 10.1007/BF01326550 · DOI
  10. Feyereisen P. (2006). Further investigation on the mnemonic effect of gestures: their meaning matters. Eur. J. Cogn. Psychol. 18, 185– 205 10.1080/09541440540000158 · DOI
  11. Gentilucci M. (2003). Grasp observation influences speech production. Eur. J. Neurosci. 17, 179– 184 10.1046/j.1460-9568.2003.02438.x 12534983 · DOI
  12. Goldin-Meadow S. (2005). Hearing Gesture: How Our Hands Help Us Think. Cambridge, MA: Harvard University Press
  13. Goldin-Meadow S. (2014). How gesture works to change our minds. Trends Neurosci. Educ. 3, 4– 6 10.1016/j.tine.2014.01.002 PMC4225621 25396115 · DOI
  14. Goldin-Meadow S. Cook S. W. Mitchell Z. A. (2009). Gesturing gives children new ideas about math. Psychol. Sci. 20, 267– 272 10.1111/j.1467-9280.2009.02297.x 19222810 PMC2750886 · DOI
  15. Gullberg M. (2006). Some reasons for studying gesture and second language acquisition. Int. Rev. Appl. Linguist. Lang. Teach. 44, 103– 124 10.1515/IRAL.2006.004 · DOI
  16. Gullberg M. de Bot K. Volterra V. (2008). Gestures and some key issues in the study of language development. Gesture 8, 149– 179 10.1075/gest.8.2.03gul · DOI
  17. Han M. (1994). Acoustic manifestations of mora timing in Japanese. J. Acoust. Soc. Am. 96, 73– 82 10.1121/1.410376 · DOI
  18. Hardison D. M. (2003). Acquisition of second-language speech: effects of visual cues, context, and talker variability. Appl. Psycholinguist. 24, 495– 522 10.1017/S0142716403000250 · DOI
  19. Hardison D. M. (2005). Second-language spoken word identification: effects of perceptual training, visual cues, and phonetic environment. Appl. Psycholinguist. 26, 579– 596 10.1017/S0142716405050319 · DOI
  20. Hirata Y. (2004a). Effects of speaking rate on the vowel length distinction in Japanese. J. Phon. 32, 565– 589 10.1016/j.wocn.2004.02.004 · DOI
  21. Hirata Y. (2004b). Training native English speakers to perceive Japanese length contrasts in word versus sentence contexts. J. Acoust. Soc. Am. 116, 2384– 2394 10.1121/1.1783351 15532669 · DOI
  22. Hirata Y. Kelly S. (2010). Effects of lips and hands on auditory learning of second-language speech sounds. J. Speech Lang. Hear. Res. 53, 298– 310 10.1044/1092-4388(2009/08-0243) 20220023 · DOI
  23. Hirata Y. Whitehurst E. Cullings E. (2007). Training native English speakers to identify Japanese vowel length contrast with sentences at varied speaking rates. J. Acoust. Soc. Am. 121, 3837– 3845 10.1121/1.2734401 17552731 · DOI
  24. Holle H. Obermeier C. Schmidt-Kassow M. Friederici A. D. Ward J. Gunter T. C. (2012). Gesture facilitates the syntactic analysis of speech. Front. Psychol. 3: 74 10.3389/fpsyg.2012.00074 22457657 PMC3307377 · DOI · Paper page
  25. Hostetter A. B. (2011). When do gestures communicate? A meta-analysis. Psychol. Bull. 137, 297– 315 10.1037/a0022128 21355631 · DOI
  26. Hubbard A. L. Wilson S. M. Callan D. E. Dapretto M. (2008). Giving speech a hand: gesture modulates activity in auditory cortex during speech perception. Hum. Brain Mapp. 30, 1028– 1037 10.1002/hbm.20565 18412134 PMC2644740 · DOI
  27. Kelly S. D. Barr D. Church R. B. Lynch K. (1999). Offering a hand to pragmatic understanding: the role of speech and gesture in comprehension and memory. J. Mem. Lang. 40, 577– 592 10.1006/jmla.1999.2634 · DOI
  28. Kelly S. D. Kravitz C. Hopkins M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain Lang. 89, 253– 260 10.1016/S0093-934X(03)00335-3 15010257 · DOI
  29. Kelly S. D. Lee A. (2012). When actions speak too much louder than words: gesture disrupts word learning when phonetic demands are high. Lang. Cogn. Process. 27, 793– 807 10.1080/01690965.2011.581125 · DOI
  30. Kelly S. D. Manning S. Rodak S. (2008). Gesture gives a hand to language and learning: perspectives from cognitive neuroscience, developmental psychology and education. Lang. Linguist. Compass 2, 1– 20 10.1111/j.1749-818X.2008.00067.x19777128 · DOI
  31. Kelly S. D. McDevitt T. Esch M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Lang. Cogn. Process. 24, 313– 334 10.1080/01690960802365567 · DOI · Paper page
  32. Kelly S. D. Özyürek A. Maris E. (2010). Two sides of the same coin: speech and gesture mutually interact to enhance comprehension. Psychol. Sci. 21, 260– 267 10.1177/0956797609357327 20424055 · DOI
  33. Kendon A. (2004). Gesture: Visible Action as Utterance. Cambridge, MA: Cambridge University Press
  34. Klucharev V. Möttönen R. Sams M. (2003). Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cogn. Brain Res. 18, 65– 75 10.1016/j.cogbrainres.2003.09.004 14659498 · DOI
  35. Krahmer E. Swerts M. (2007). The effects of visual beats on prosodic prominence: acoustic analyses, auditory perception and visual perception. J. Mem. Lang. 57, 396– 414 10.1016/j.jml.2007.06.005 · DOI
  36. Macedonia M. Müller K. Friederici A. D. (2011). The impact of iconic gestures on foreign language word learning and its neural substrate. Hum. Brain Mapp. 32, 982– 998 10.1002/hbm.21084 20645312 PMC6870319 · DOI
  37. McNeill D. (1992). Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: University of Chicago Press
  38. McNeill D. (2005). Gesture and Thought. Chicago, IL: University of Chicago Press 10.7208/chicago/9780226514642.001.0001 · DOI · Paper page
  39. Montgomery K. J. Isenberg N. Haxby J. V. (2007). Communicative hand gestures and object-directed hand movements activated the mirror neuron system. Soc. Cogn. Affect. Neurosci. 2, 114– 122 10.1093/scan/nsm004 18985130 PMC2555455 · DOI
  40. Port R. F. Dalby J. O'Dell M. (1987). Evidence for mora timing in Japanese. J. Acoust. Soc. Am. 81, 1574– 1585 10.1121/1.394510 3584695 · DOI
  41. Quinn-Allen L. (1995). The effects of emblematic gestures on the development and access of mental representations of French expressions. Mod. Lang. J. 79, 521– 529 10.1111/j.1540-4781.1995.tb05454.x · DOI
  42. Sadakata M. Sekiyama K. (2011). Enhanced perception of various linguistic features by musicians: a cross-linguistics study. Acta Psychol. 138, 1– 10 10.1016/j.actpsy.2011.03.007 21726835 · DOI
  43. Saltz E. Donnenwerth-Nolan S. (1981). Does motoric imagery facilitate memory for sentences? A selective interference test. J. Verb. Learn. Verb. Behav. 20, 322– 332 10.1016/S0022-5371(81)90472-2 · DOI
  44. Straube B. Green A. Weis S. Chatterjee A. (2009). Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance. J. Cogn. Neurosci. 21, 821– 836 10.1162/jocn.2009.21053 18578601 · DOI
  45. Sueyoshi A. Hardison D. M. (2005). The role of gestures and facial cues in second language listening comprehension. Lang. Learn. 55, 661– 699 10.1111/j.0023-8333.2005.00320.x · DOI
  46. Tajima K. Kato H. Rothwell A. Akahane-Yamada R. Munhall K. (2008). Training English listeners to perceive phonemic length contrasts in Japanese. J. Acoust. Soc. Am. 123, 397– 413 10.1121/1.2804942 18177169 · DOI
  47. Thompson L. A. (1995). Encoding and memory for visible speech and gestures: a comparison between young and older adults. Psychol. Aging 10: 215 10.1037/0882-7974.10.2.215 7662181 · DOI
  48. Vance T. J. (1987). An Introduction to Japanese Phonology. Albany, NY: State University of New York Press
  49. Wang Y. Behne D. Jiang H. (2008). Linguistic experience and audio-visual perception of non-native fricatives. J. Acoust. Soc. Am. 124, 1716– 1726 10.1121/1.2956483 19045662 · DOI
  50. Willems R. M. Hagoort P. (2007). Neural evidence for the interplay between language, gesture, and action: a review. Brain Lang. 101, 278– 289 10.1016/j.bandl.2007.03.004 17416411 · DOI
  51. Willems R. M. Özyürek A. Hagoort P. (2007). When language meets action: the neural integration of gesture and speech. Cereb. Cortex 17, 2322– 2333 10.1093/cercor/bhl141 17159232 · DOI
  52. Wong P. C. M. Perrachione T. K. (2007). Learning pitch patterns in lexical identification by native English-speaking adults. Appl. Psycholinguist. 28, 565– 585 10.1017/S0142716407070312 · DOI

Cited by

Sorted by citations

Authors
Kelly, S. D., Hirata, Y., Manansala, M., & Huang, J.
Affiliations
Colgate University (US)
Corresponding author
Kelly, Spencer D.
ORCID
Venue
Frontiers in Psychology
Publisher
Frontiers Media SA
Published
2014-07-01
Volume
5
Pages
673-673
ISSN
1664-1078 (ISSN-L)
DOI
10.3389/fpsyg.2014.00673
PMC
PMC4077026
OpenAlex
W2054536910
Cited by
37
References
0
Citations by year
2026: 2 · 2025: 2 · 2024: 1 · 2023: 4 · 2022: 1 · 2021: 4 · 2020: 6 · 2019: 4 · 2018: 6 · 2017: 5 · 2015: 2
FWCI
3.25
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Frontiers in Psychology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Science Foundation (1052765)
  • Colgate University
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
  • Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
Keywords
second language acquisition, vowel duration, Japanese as a second language, gesture-speech integration, syllable structure, vocabulary acquisition, gesture production, auditory identification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review