Virtual character performance from speech
The DOI matches its Crossref record; 55 of the 59 references it cites were found in a registry or an index.
Abstract
We demonstrate a method for generating a 3D virtual character performance from the audio signal by inferring the acoustic and semantic properties of the utterance. Through a prosodic analysis of the acoustic signal, we perform an analysis for stress and pitch, relate it to the spoken words and identify the agitation state. Our rule-based system performs a shallow analysis of the utterance text to determine its semantic, pragmatic and rhetorical content. Based on these analyses, the system generates facial expressions and behaviors including head movements, eye saccades, gestures, blinks and gazes. Our technique is able to synthesize the performance and generate novel gesture animations based on coarticulation with other closely scheduled animations. Because our method utilizes semantics in addition to prosody, we are able to generate virtual character performances that are more appropriate than methods that use only prosody. We perform a study that shows that our technique outperforms methods that use prosody alone.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Rule-based approaches, however, often require laborious manual tuning and often struggle to achieve a natural motion quality.”
From Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents (2024) · cites this paper as [20]
Citations
Cites
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465 · cited 2×
“3.1.1 Mel frequency cepstral coef.cients (mf cc) We extract 12 coef.cients capturing the overall spectral information from the speech signal using Hanning windowed 32 ms frames with a 10 ms shift (i.e. 100Hz sample rate)[Davis and Mermelstein 1980].”
MethodInfluentialHand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding
EKMAN, P., & FRIESEN, W. V. (1969) · Semiotica · doi:10.1515/semi.1969.1.1.49 · Cited by 3,316
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
(1999) · Cited by 1,521
The phonetic description of voice quality
Laver, J. (1980) · Cited by 1,441
Argyle, M., Cook, M., & Cramer, D. (1994) · British Journal of Psychiatry · doi:10.1017/s0007125000073980 · Cited by 1,304
Proceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 761
BEAT: the Behavior Expression Animation Toolkit
Cassell, J., Vilhjálmsson, H. H., & Bickmore, T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383315 · Cited by 739 · cited 2×
“…on overall nonverbal behavior generation but using limited forms of analyses such as detecting individual keywords [Bergmann and Kopp 2009; Lee and Marsella 2010] or using textual analysis to derive factors like rheme (or focus) and theme (or topic) to determine emphasis [Cassell et al. 2001].”
MethodCassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., Douville, B., Prevost, S., & Stone, M. (1994) · Proceedings of the 21st annual conference on Computer graphics and interactive techniques - SIGGRAPH '94 · doi:10.1145/192161.192272 · Cited by 665
Perlin, K. (2002) · Proceedings of the 29th annual conference on Computer graphics and interactive techniques · doi:10.1145/566570.566636 · Cited by 610
Linguistic functions of head movements in the context of speech
McClave, E. Z. (2000) · Journal of Pragmatics · doi:10.1016/s0378-2166(99)00079-x · Cited by 500
Towards a common framework for multimodal generation: the behavior markup language,
Kopp, S., Krenn, B., Marsella, S., Marshall, A. N., Pelachaud, C., Pirker, H., Thórisson, K. R., & Vilhjálmsson, H. (2006) · Lecture Notes in Computer Science · doi:10.1007/11821830_17 · Cited by 417 · cited 2×
“…gesture point right, gaze left, gaze right Emphasis gesture rhythm, small nod, beat gesture Intensi.er brow frown, half nod 3.5 Animation Speci.cation Behaviors are mapped to the BML language, a high-level XML language that describes a behavior and an execution schedule [Kopp et al. 2006].”
BackgroundInfluentialSelf-citationBrand, M. (1999) · Proceedings of the 26th annual conference on Computer graphics and interactive techniques - SIGGRAPH '99 · doi:10.1145/311535.311537 · Cited by 411
2006 International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2006)
(2005) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.849938 · Cited by 313
Language and gesture: unity or duality?
Kendon, A. (2000) · Language and Gesture · doi:10.1017/cbo9780511620850.004 · Cited by 306
Elements of Meaning in Gesture
Calbris, G. (2011) · Gesture Studies · doi:10.1075/gs.5 · Cited by 294
Perlin, K. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566636 · Cited by 260
SmartBody: behavior realization for embodied conversational agents
Thiébaux, M., Marsella, S., Marshall, A. N., & Kallmann, M. E. (2008) · doi:10.5555/1402383.1402409 · Cited by 249
Self-citationNonverbal Behavior Generator for Embodied Conversational Agents
Lee, J., & Marsella, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11821830_20 · Cited by 238
Self-citationGesture modeling and animation based on a probabilistic re-creation of speaker style
Neff, M., Kipp, M., Albrecht, I., & Seidel, H.-P. (2008) · ACM Transactions on Graphics · doi:10.1145/1330511.1330516 · Cited by 232
Joint robust voicing detection and pitch estimation based on residual harmonics
Drugman, T., & Alwan, A. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-519 · Cited by 225
Kendon, A. (2002) · Gesture · doi:10.1075/gest.2.2.03ken · Cited by 197
Levine, S., Krähenbühl, P., Thrun, S., & Koltun, V. (2010) · ACM Transactions on Graphics · doi:10.1145/1833349.1778861 · Cited by 187
Gestures as Part of Speech: Methodological Implications
Bavelas, J. B. (1994) · Research on Language and Social Interaction · doi:10.1207/s15327973rlsi2703_3 · Cited by 181
Rigid Head Motion in Expressive Speech Animation: Analysis and Synthesis
Busso, C., Deng, Z., Grimm, M., Neumann, U., & Narayanan, S. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2006.885910 · Cited by 180
Max - A multimodal assistant in virtual reality construction
Kopp, S., Jung, B., Pfeiffer-Leßmann, N., & Wachsmuth, I. (2003) · PUB – Publications at Bielefeld University (Bielefeld University) · Cited by 158
Grammatical Trigrams: A Probabilistic Model of Link Grammar
Lafferty, J., Sleator, S., & Temperley, D. (1992) · Defense Technical Information Center (DTIC) · doi:10.21236/ada256365 · Cited by 140
An annotation scheme for conversational gestures: how to economically capture timing and form
Kipp, M., Neff, M., & Albrecht, I. (2007) · Computers and the Humanities · doi:10.1007/s10579-007-9053-5 · Cited by 137
Stone, M., DeCarlo, D., Oh, I., Rodriguez, C., Stere, A., Lees, A., & Bregler, C. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015753 · Cited by 137
Mood swings: expressive speech animation
Chuang, E., & Bregler, C. (2005) · TOGS · doi:10.1145/1061347.1061355 · Cited by 131
Levine, S., Krähenbühl, P., Thrun, S., & Koltun, V. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778861 · Cited by 129
Real-time prosody-driven synthesis of body language
Levine, S., Theobalt, C., & Koltun, V. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618518 · Cited by 120
Building a Character Animation System
Shapiro, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25090-3_9 · Cited by 111
Self-citationGreta: an interactive expressive ECA system
Niewiadomski, R., Bevacqua, E., Mancini, M., & Pelachaud, C. (2009) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/lpbe9373 · Cited by 104
Kipp, M., Neff, M., Kipp, K. H., & Albrecht, I. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74997-4_2 · Cited by 99
Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces
Deng, Z., Neumann, U., Lewis, J. P., Kim, T.-Y., Bulut, M., & Narayanan, S. B. (2006) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2006.90 · Cited by 98
Hand and Mind: What Gestures Reveal about Thought
Creider, C. A. (1994) · Journal of Linguistic Anthropology · doi:10.1525/jlin.1994.4.1.81 · Cited by 89
The CereVoice Characterful Speech Synthesiser SDK
Aylett, M. P., & Pidcock, C. J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74997-4_65 · Cited by 70
Scherer, S. J., Kane, J., Gobl, C., & Schwenker, F. (2012) · Computer Speech & Language · doi:10.1016/j.csl.2012.06.001 · Cited by 70
Self-citationElckerlyc: A BML Realizer for continuous, multimodal interaction with a Virtual Human
van Welbergen, H., Reidsma, D., Ruttkay, Z. M., & Zwiers, J. (2009) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-010-0051-3 · Cited by 63
Specifying and animating facial signals for discourse in embodied conversational agents
DeCarlo, D., Stone, M., Revilla, C., & Venditti, J. J. (2004) · Computer Animation and Virtual Worlds · doi:10.1002/cav.5 · Cited by 62
EMBR – A Realtime Animation Engine for Interactive Embodied Agents
Héloir, A., & Kipp, M. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04380-2_43 · Cited by 59
Automatic Generation of Non-Verbal Facial Expressions from Speech
Albrecht, I., Haber, J., & Seidel, H. (2002) · Springer eBooks · doi:10.1007/978-1-4471-0103-1_18 · Cited by 54
Bergmann, K., & Kopp, S. (2009) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/vtam7538 · Cited by 49
Learning dynamic audio-visual mapping with input-output Hidden Markov models
Li, Y., & Shum, H. (2006) · IEEE Trans. Multim. · doi:10.1109/tmm.2006.870732 · Cited by 44
Predicting Speaker Head Nods and the Effects of Affective Information
Lee, J., & Marsella, S. C. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2051874 · Cited by 37
Self-citationWord prominence detection using robust yet simple prosodic features
Mishra, T., Sridhar, V. R., & Conkie, A. D. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-408 · Cited by 33
Expressive Facial Gestures From Motion Capture Data
Ju, E., & Lee, J. (2008) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2008.01135.x · Cited by 27
Multi-party, multi-role comprehensive listening behavior
Wang, Z., Lee, J., & Marsella, S. (2013) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9215-8 · Cited by 19
Self-citationKane, J., Scherer, S. J., Aylett, M. P., Morency, L., & Gobl, C. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639219 · Cited by 19
Self-citationExpressive speech synthesis: synthesising ambiguity.
Aylett, M. P., Potard, B., & Pidcock, C. J. (2013) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 8
Research works printed but not linked: 9
- Bergmann K., Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, 361--368
- Cao Y., Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '03, 225--231
- 10.5555/974305.974323 · doi:10.5555/974305.974323
- 10.1002/cav.v15:1 · doi:10.1002/cav.v15:1
- Drugman T., 1973, Proceedings of Interspeech 2011
- Kane J., International Conference on Acoustics, Speech, and Signal Processing (ICASSP).
- Lafferty J., In Proceedings of the AAAI Fall Symposium on Probabilistic Approaches to Natural Language, 89--97
- Mishra T., 2012, Proceedings of Interspeech
- Niewiadomski R., Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems -, 2
Cited by
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434
“Several authors have tried alleviating the effect by deducing additional motion, such as head movements, eye saccades or blinks from audio (e.g., [Deng et al. 2004; Marsella et al. 2013]).”
BackgroundLearning Individual Styles of Conversational Gesture
Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., & Malik, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00361 · Cited by 335
“Later work [27] extended this approach to use reinforcement learning and speech recognition, combined acoustic analysis with text [29], created hybrid rule-based systems [36], and used restricted Boltzmann machines for inference [11].”
MethodSpeech gesture generation from the trimodal context of text, audio, and speaker identity
Yoon, Y., Cha, B., Lee, J.-H., Jang, M., Lee, J., Kim, J., & Lee, G. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417838 · Cited by 328
Ruhland, K., Peters, C. E., Andrist, S., Badler, J. B., Badler, N. I., Gleicher, M., Mutlu, B., & McDonnell, R. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12603 · Cited by 215
Generating Holistic 3D Human Motion from Speech
Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., & Black, M. J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00053 · Cited by 163
“Recent methods have shown rapid progress on modeling the translation from human speech to body motion, and can be roughly divided into rule-based [38] and learning-based [20, 21, 23, 32, 33, 55] methods.”
Method
Metadata
- Authors
- Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A.
- Affiliations
- University of Southern California (US)
- Venue
- Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation
- Publisher
- ACM
- Published
- 2013-07-19
- Pages
- 25-35
- OpenAlex
- W2024536104
- Cited by
- 168
- References
- 59
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 18 · 2023: 20 · 2022: 13 · 2021: 20 · 2020: 12 · 2019: 18 · 2018: 10 · 2017: 9 · 2016: 9 · 2015: 14 · 2014: 9 · 2013: 1
- FWCI
- 15.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech-driven animation, prosodic analysis, semantic analysis, pragmatic analysis, facial expression synthesis, head movement, eye saccades, gesture synthesis, coarticulation, rule-based systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.