Character-Centered Dialogue Generation from Scene-Level Prompts
Open accessAbstract
Recent advances in scene-based video generation enable coherent visual narratives from structured prompts, yet a key aspect of storytelling -- character-driven dialogue and speech -- remains underexplored. We present a modular pipeline that transforms action-level prompts into visually and auditorily grounded dialogue, enriching scene-based storytelling with natural voice and character expression. Our method takes a pair of prompts per scene, defining the setting and character behavior. While a story generation model such as Text2Story produces the visual scene, we focus on generating expressive, character-consistent utterances grounded in both the prompts and a representative scene image. A pretrained vision-language encoder extracts high-level visual semantics, which are combined with structured prompts to guide a large language model for dialogue synthesis. To maintain contextual and emotional consistency across scenes, we introduce a Recursive Narrative Bank, a speaker-aware, temporally structured memory that accumulates each character's dialogue history. Inspired by Script Theory, this design enables dialogue that reflects evolving goals, social context, and narrative roles. Finally, we render each utterance as expressive, character-conditioned speech, producing fully voiced, multimodal video narratives. Our training-free framework generalizes across diverse story settings, providing a scalable solution for coherent, character-grounded audiovisual storytelling.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When integrated into real game production pipelines, LLM-based approaches face persistent challenges: freeform outputs often violate structural constraints, produce inconsistent world knowledge or personas, and fail to satisfy engine-level parsing and execution requirements.
“However, when integrated into real game production pipelines, LLM-based approaches face persistent challenges; freeform outputs often violate structural constraints, produce inconsistent world knowledge or personas, and fail to satisfy engine-level parsing and execution requirements.”
From Game Knowledge Management System: Schema-Governed LLM Pipeline for Executable Narrative Generation in RPGs (2026) · cites this paper as [7]
The Detailed Ablation Study (p. 11) reports that removing the Recursive Narrative Bank reduces dialogue continuity, supporting the concern about character consistency. The paper studies scene-based audiovisual storytelling, not real game production pipelines; our copy does not support the claims about freeform output violating structural constraints, inconsistent world knowledge, or engine-level parsing and execution failures. (LLM, conf=0.78)
Experimental — accuracy not yet measuredMost LLM-for-game systems lack reproducible mechanisms for engine-level validation and deployment and are primarily evaluated through text-level analysis or subjective judgment rather than verified runtime execution.
“Consequently, most LLM-for-game systems lack reproducible mechanisms for engine-level validation and deployment and are primarily evaluated through text-level analysis or subjective judgment rather than verified runtime execution [5-7].”
From Game Knowledge Management System: Schema-Governed LLM Pipeline for Executable Narrative Generation in RPGs (2026) · cites this paper as [7]
The Quantitative Evaluation (p. 4) and Users Evaluation (p. 5) describe automated metrics and participant judgments for this paper’s multimodal storytelling system. They do not address LLM-for-game systems as a group or whether those systems lack reproducible engine-level validation, deployment, and verified runtime execution. Searches for game-engine validation and runtime evaluation found no passage establishing the attributed claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 17
- [1][1] Taewon Kang, Divya Kothandaraman, and Ming C Lin, 'Text2story: Advancing video storytelling with text guidance,' arXiv preprint arXiv:2503.06310 , 2025. - · arXiv
- [2][2] Roger C Schank and Robert P Abelson, Scripts, plans, goals, and understanding: An inquiry into human knowledge structures , Psychology press, 2013. -
- [3][3] Gordon H Bower, John B Black, and Terrence J Turner, 'Scripts in memory for text,' Cognitive psychology , vol. 11, no. 2, pp. 177-220, 1979. -
- [5][5] Xiaochuan Lin and Xiangyong Chen, 'Improving visual storytelling with multimodal large language models,' arXiv preprint arXiv:2407.02586 , 2024. - · arXiv
- [6][6] Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen, 'Seed-story: Multimodal long story generation with large language model,' arXiv preprint arXiv:2407.08683 , 2024. - · arXiv
- [7][7] Chuanqi Zang, Jiji Tang, Rongsheng Zhang, Zeng Zhao, Tangjie Lv, Mingtao Pei, and Wei Liang, 'Let storytelling tell vivid stories: An expressive and fluent multimodal storyteller,' arXiv preprint arXiv:2403.07301 , 2024. - · arXiv
- [8][8] Samee Arif, Taimoor Arif, Muhammad Saad Haroon, Aamina Jamal Khan, Agha Ali Raza, and Awais Athar, 'The art of storytelling: Multi-agent generative ai for dynamic multimodal narratives,' arXiv preprint arXiv:2409.11261 , 2024. - · arXiv
- [9][9] Jaeyeon Bae, Seokhoon Jeong, Seokun Kang, Namgi Han, Jae-Yon Lee, Hyounghun Kim, and Taehwan Kim, 'Sound of story: Multi-modal storytelling with audio,' arXiv preprint arXiv:2310.19264 , 2023. - · arXiv
- [10][10] Samuel S Sohn, Danrui Li, Sen Zhang, Che-Jui Chang, and Mubbasir Kapadia, 'From words to worlds: Transforming one-line prompt into immersive multi-modal digital stories with communicative llm agent,' arXiv preprint arXiv:2406.10478 , 2024. - · arXiv
- [11][11] Min Zhang, Zilin Wang, Liyan Chen, Kunhong Liu, and Juncong Lin, 'Dialogue director: Bridging the gap in dialogue visualization for multimodal storytelling,' arXiv preprint arXiv:2412.20725 , 2024. - · arXiv
- [12][12] Xuenan Xu, Jiahao Mei, Chenliang Li, Yuning Wu, Ming Yan, Shaopeng Lai, Ji Zhang, and Mengyue Wu, 'Mm-storyagent: Immersive narrated storybook video generation with a multi-agent paradigm across text, image and audio,' arXiv preprint arXiv:2503.05242 , 2025. - · arXiv
- [13][13] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo, 'Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions,' in European Conference on Computer Vision . Springer, 2024, pp. 244-260. -
- [14][14] Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, and Siyu Zhu, 'Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer,' in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 21086-21095. -
- [15][15] Gaojie Lin, Jianwen Jiang, Jiaqi Yang, Zerong Zheng, Chao Liang, Yuan Zhang, and Jingtuo Liu, 'Omnihuman-1: Rethinking the scaling-up of one-stage conditioned human animation models,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2025, pp. 13847-13858. -
- [18][18] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al., 'Cogvideox: Text-to-video diffusion models with an expert transformer,' arXiv preprint arXiv:2408.06072 , 2024. - · arXiv
- [19][19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi, 'Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,' in International conference on machine learning . PMLR, 2022, pp. 12888-12900. -
- [20][20] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al., 'Tacotron: Towards end-to-end speech synthesis,' arXiv preprint arXiv:1703.10135 , 2017. - · arXiv
Cited by
Sorted by citations
Text-Conditioned Background Generation for Editable Multi-layer Documents
Kang, T., Joseph, K. J., Tensmeyer, C., Kil, J., Zhu, W., Lin, M. C., & Morariu, V. I. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37255-0_7
Self-citation
- Authors
- Kang, T., & Lin, M. C.
- Affiliations
- University of Maryland, College Park (US)
- Venue
- 2026 IEEE International Conference on Image Processing (ICIP)
- Publisher
- IEEE
- Published
- 2026-09
- Pages
- 1-6
- ISSN
- 1522-4880 (ISSN-L)
- OpenAlex
- W6979264393
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - International Conference on Image Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- University of Maryland
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- dialogue generation, audiovisual narrative, speech synthesis, character expression, large language models, script theory
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
