Virtual character performance from speech
The DOI matches its Crossref record; 55 of the 59 references it cites were found in a registry or an index.
Abstract
We demonstrate a method for generating a 3D virtual character performance from the audio signal by inferring the acoustic and semantic properties of the utterance. Through a prosodic analysis of the acoustic signal, we perform an analysis for stress and pitch, relate it to the spoken words and identify the agitation state. Our rule-based system performs a shallow analysis of the utterance text to determine its semantic, pragmatic and rhetorical content. Based on these analyses, the system generates facial expressions and behaviors including head movements, eye saccades, gestures, blinks and gazes. Our technique is able to synthesize the performance and generate novel gesture animations based on coarticulation with other closely scheduled animations. Because our method utilizes semantics in addition to prosody, we are able to generate virtual character performances that are more appropriate than methods that use only prosody. We perform a study that shows that our technique outperforms methods that use prosody alone.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Rule-based approaches, however, often require laborious manual tuning and often struggle to achieve a natural motion quality.”
From Incorporating Spatial Awareness in Data-Driven Gesture Generation for Virtual Agents (2024) · cites this paper as [20]
Citations
Cites
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465 · cited 2×
“3.1.1 Mel frequency cepstral coef.cients (mf cc) We extract 12 coef.cients capturing the overall spectral information from the speech signal using Hanning windowed 32 ms frames with a 10 ms shift (i.e. 100Hz sample rate)[Davis and Mermelstein 1980].”
MethodInfluentialHand and mind: what gestures reveal about thought
(1993) · Choice Reviews Online · doi:10.5860/choice.30-2492 · Cited by 5,094
The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding
EKMAN, P., & FRIESEN, W. V. (1969) · Semiotica · doi:10.1515/semi.1969.1.1.49 · Cited by 3,316
Accurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
(1999) · Cited by 1,521
The phonetic description of voice quality
Laver, J. (1980) · Cited by 1,441
Argyle, M., Cook, M., & Cramer, D. (1994) · British Journal of Psychiatry · doi:10.1017/s0007125000073980 · Cited by 1,304
Proceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 761
Research works printed but not linked: 9
- Bergmann K., Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, 361--368
- Cao Y., Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SCA '03, 225--231
- 10.5555/974305.974323 · doi:10.5555/974305.974323
- 10.1002/cav.v15:1 · doi:10.1002/cav.v15:1
- Drugman T., 1973, Proceedings of Interspeech 2011
- Kane J., International Conference on Acoustics, Speech, and Signal Processing (ICASSP).
- Lafferty J., In Proceedings of the AAAI Fall Symposium on Probabilistic Approaches to Natural Language, 89--97
- Mishra T., 2012, Proceedings of Interspeech
- Niewiadomski R., Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems -, 2
Cited by
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434
“Several authors have tried alleviating the effect by deducing additional motion, such as head movements, eye saccades or blinks from audio (e.g., [Deng et al. 2004; Marsella et al. 2013]).”
BackgroundLearning Individual Styles of Conversational Gesture
Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., & Malik, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00361 · Cited by 335
“Later work [27] extended this approach to use reinforcement learning and speech recognition, combined acoustic analysis with text [29], created hybrid rule-based systems [36], and used restricted Boltzmann machines for inference [11].”
MethodSpeech gesture generation from the trimodal context of text, audio, and speaker identity
Yoon, Y., Cha, B., Lee, J.-H., Jang, M., Lee, J., Kim, J., & Lee, G. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417838 · Cited by 328
Ruhland, K., Peters, C. E., Andrist, S., Badler, J. B., Badler, N. I., Gleicher, M., Mutlu, B., & McDonnell, R. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12603 · Cited by 215
Generating Holistic 3D Human Motion from Speech
Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., & Black, M. J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00053 · Cited by 163
“Recent methods have shown rapid progress on modeling the translation from human speech to body motion, and can be roughly divided into rule-based [38] and learning-based [20, 21, 23, 32, 33, 55] methods.”
Method
Metadata
- Authors
- Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A.
- Affiliations
- University of Southern California (US)
- Venue
- Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation
- Publisher
- ACM
- Published
- 2013-07-19
- Pages
- 25-35
- OpenAlex
- W2024536104
- Cited by
- 168
- References
- 59
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 18 · 2023: 20 · 2022: 13 · 2021: 20 · 2020: 12 · 2019: 18 · 2018: 10 · 2017: 9 · 2016: 9 · 2015: 14 · 2014: 9 · 2013: 1
- FWCI
- 15.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech-driven animation, prosodic analysis, semantic analysis, pragmatic analysis, facial expression synthesis, head movement, eye saccades, gesture synthesis, coarticulation, rule-based systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Virtual character performance from speech. https://app.cochat.ai/papers/10.1145/2485895.2485900
- The paper
- Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A. (2013). Virtual character performance from speech. Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation. https://doi.org/10.1145/2485895.2485900
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.