LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Video creation has become increasingly popular, yet the expertise and effort required for editing often pose barriers to beginners. In this paper, we explore the integration of large language models (LLMs) into the video editing workflow to reduce these barriers. Our design vision is embodied in LAVE, a novel system that provides LLM-powered agent assistance and language-augmented editing features. LAVE automatically generates language descriptions for the user’s footage, serving as the foundation for enabling the LLM to process videos and assist in editing tasks. When the user provides editing objectives, the agent plans and executes relevant actions to fulfill them. Moreover, LAVE allows users to edit videos through either the agent or direct UI manipulation, providing flexibility and enabling manual refinement of agent actions. Our user study, which included eight participants ranging from novices to proficient editors, demonstrated LAVE’s effectiveness. The results also shed light on user perceptions of the proposed LLM-assisted editing paradigm and its impact on users’ creativity and sense of co-creation. Based on these findings, we propose design implications to inform the future development of agent-assisted content editing.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 1×
“Our work builds upon existing literature in human-AI co-creation [8, 14, 21, 24, 30, 32, 39, 54, 56, 72] and further contributes by developing a new AI system for video editing and studying its impact.”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
Quantifying the Creativity Support of Digital Tools through the Creativity Support Index
Cherry, E., & Latulipe, C. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2617588 · Cited by 498 · cited 2×
“Additionally, we adapted applicable questions from the Creative Support Index [18].”
BackgroundMethodHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Song, C. H., Sadler, B. M., Wu, J., Chao, W.-L., Washington, C., & Su, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00280 · Cited by 435
Make-A-Video: Text-to-Video Generation without Text-Video Data
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14792 · Cited by 395
Wordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374 · cited 4×
“Building upon previous work that integrates natural language with video editing, we propose to instrument video editing with LLM’s versatile linguistic capabilities, e.g., storytelling and reasoning, which have proven useful in assisting various creative tasks [15, 20, 41–43, 50, 75, 82].”
BackgroundMethodInfluentialLarge language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 1×
“Our work builds upon existing literature in human-AI co-creation [8, 14, 21, 24, 30, 32, 39, 54, 56, 72] and further contributes by developing a new AI system for video editing and studying its impact.”
BackgroundNovice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 3×
“This includes areas such as story writing [15, 20, 50, 82], music composition [26, 46, 47], comic creation [64], and game design [85].”
BackgroundText-based Editing of Talking-head Video
Fried, O., Tewari, A., Zollhöfer, M., Finkelstein, A., Shechtman, E., Goldman, D. B., Genova, K., Jin, Z., Theobalt, C., & Agrawala, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323028 · Cited by 270
Explainable AI for Designers: A Human-Centered Perspective on Mixed-Initiative Co-Creation
Zhu, J., Liapis, A., Risi, S., Bidarra, R., & Youngblood, G. M. (2018) · 2018 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2018.8490433 · Cited by 255
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240 · cited 4×
“Building upon previous work that integrates natural language with video editing, we propose to instrument video editing with LLM’s versatile linguistic capabilities, e.g., storytelling and reasoning, which have proven useful in assisting various creative tasks [15, 20, 41–43, 50, 75, 82].”
BackgroundMethodInfluentialLiu, B. (2021) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmab013 · Cited by 228
Autonomy, Authenticity, Authorship and Intention in Computer Generated Art
McCormack, J., Gifford, T., & Hutchings, P. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-16667-0_3 · Cited by 218
TaleBrush: Sketching Stories with Generative Pretrained Language Models
Chung, J. J. Y., Kim, W., Yoo, K. M., Lee, H., Adar, E., & Chang, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501819 · Cited by 206
Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J. T., & Bernstein, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415234 · Cited by 183 · cited 1×
“Our work builds upon existing literature in human-AI co-creation [8, 14, 21, 24, 30, 32, 39, 54, 56, 72] and further contributes by developing a new AI system for video editing and studying its impact.”
BackgroundEnabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156
Self-citationChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Arawjo, I., Swoopes, C., Vaithilingam, P., Wattenberg, M., & Glassman, E. (2023) · arXiv · doi:10.48550/arxiv.2309.09128 · Cited by 146
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Logan IV, R., Balazevic, I., Wallace, E., Petroni, F., Singh, S., & Riedel, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.222 · Cited by 133
ChemCrow: Augmenting large-language models with chemistry tools
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2023) · arXiv · doi:10.48550/arxiv.2304.05376 · Cited by 122
AI Song Contest: Human-AI Co-Creation in Songwriting
Huang, C.-Z. A., Koops, H. V., Newton-Rex, E., Dinculescu, M., & Cai, C. J. (2020) · arXiv · doi:10.48550/arxiv.2010.05388 · Cited by 90
Implicit Representations of Meaning in Neural Language Models
Li, B. Z., Nye, M., & Andreas, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.143 · Cited by 84 · cited 1×
“They have been shown to encode a wealth of human knowledge [27, 36, 57] and can perform sophisticated reasoning [34, 51, 76] and action planning [27].”
BackgroundEvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y.-H., & Kim, J. (2023) · arXiv · doi:10.48550/arxiv.2309.13633 · Cited by 82
WinoQueer: A community-in-the-loop benchmark for anti-LGBTQ+ bias in large language models
Felkner, V., Chang, H.-C. H., Jang, E., & May, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.507 · Cited by 74
Chakrabarty, T., Padmakumar, V., Brahman, F., & Muresan, S. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656201 · Cited by 61
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Shaw, P., Joshi, M., Cohan, J., Berant, J., Pasupat, P., Hu, H., Khandelwal, U., Lee, K., & Toutanova, K. N. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1490 · Cited by 12
Research works printed but not linked: 38
- 10.1080/19393555.2022.2060879 · DOI
- Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. arxiv:2304.09337 [cs.HC] · arXiv
- Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine potential pitfalls when designing human-ai co-creative systems. arXiv preprint arXiv:2104.00358 (2021). · arXiv
- 10.1145/3411764.3445131 · DOI
- 10.1145/3290605.3300931 · DOI
- 10.1038/s42256-020-0161-x · DOI
- Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. 2022. Imagen Video: High Definition Video Generation with Diffusion Models. arxiv:2210.02303 [cs.CV] · arXiv
- Wenlong Huang Pieter Abbeel Deepak Pathak and Igor Mordatch. 2022. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. arxiv:2201.07207 [cs.LG] · arXiv
- 10.1145/3290605.3300311 · DOI
- 10.1145/3544548.3581494 · DOI · Paper page
- 10.1093/jcmc/zmac014 · DOI · Paper page
- Ehud Karpas Omri Abend Yonatan Belinkov Barak Lenz Opher Lieber Nir Ratner Yoav Shoham Hofit Bata Yoav Levine Kevin Leyton-Brown Dor Muhlgay Noam Rozen Erez Schwartz Gal Shachaf Shai Shalev-Shwartz Amnon Shashua and Moshe Tenenholtz. 2022. MRKL Systems: A modular neuro-symbolic architecture that combines large language models external knowledge sources and discrete reasoning. arxiv:2205.00445 [cs.CL] · arXiv
- 10.1145/2470654.2481301 · DOI · Paper page
- 10.1145/3544548.3581006 · DOI
- Vivian Liu, Tao Long, Nathan Raw, and Lydia Chilton. 2023. Generative Disco: Text-to-Video Generation for Music Visualization. arXiv preprint arXiv:2304.08551 (2023). · arXiv
- 10.1145/3526113.3545621 · DOI · Paper page
- 10.1145/3563657.3596098 · DOI · Paper page
- 10.1145/3490099.3511159 · DOI · Paper page
- Federico Magni, Jiyoung Park, and Melody Manchi Chao. 2023. Humans as Creativity Gatekeepers: Are We Biased Against AI Creativity?Journal of Business and Psychology (2023), 1–14.
- Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arxiv:2112.00114 [cs.LG] · arXiv
- OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL] · arXiv
- 10.1145/3311957.3359433 · DOI
- 10.1145/3379337.3415864 · DOI · Paper page
- Jeba Rezwana and Mary Lou Maher. 2022. Identifying ethical issues in ai partners in human-ai co-creation. arXiv preprint arXiv:2204.07644 (2022). · arXiv
- 10.18653/v1/2020.emnlp-main.437 · DOI · Paper page
- 10.1145/3526113.3545617 · DOI · Paper page
- Derek Tam Anisha Mascarenhas Shiyue Zhang Sarah Kwan Mohit Bansal and Colin Raffel. 2022. Evaluating the Factual Consistency of Large Language Models Through Summarization. arxiv:2211.08412 [cs.CL] · DOI · Paper page
- 10.1145/2984511.2984569 · DOI · Paper page
- Maarten Van Someren, Yvonne F Barnard, and J Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. London: AcademicPress 11 (1994), 29–41.
- Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Shomir Wilson, 2023. Nationality Bias in Text Generation. arXiv preprint arXiv:2302.02463 (2023). · arXiv
- 10.1145/3526113.3545680 · DOI
- 10.1145/3355089.3356520 · DOI
- Sitong Wang, Samia Menon, Tao Long, Keren Henderson, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V Nickerson, and Lydia B Chilton. 2023. ReelFramer: Co-creating News Reels on Social Media with Generative AI. arXiv preprint arXiv:2304.09653 (2023). · arXiv
- 10.1145/3379337.3415845 · DOI · Paper page
- 10.1145/3379337.3415882 · DOI · Paper page
- Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9, 1 (2023). · arXiv
- Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. arXiv preprint arXiv:2306.15895 (2023). · arXiv
- 10.1145/3491102.3517479 · DOI · Paper page
Cited by
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Amuse: Human-AI Collaborative Songwriting with Multimodal Inspirations
Kim, Y., Lee, S.-J., & Donahue, C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713818 · Cited by 23
“Consequently, these systems are being applied across a growing range of creative pursuits, such as illustration [59], creative writing [18], and video editing [97].”
Data Playwright: Authoring Data Videos With Annotated Narration
Shen, L., Li, H., Wang, Y., Luo, T., Luo, Y., & Qu, H. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3477926 · Cited by 21
Cao, Y., Huang, Y., Truong, A., Shin, H. V., & Xia, H. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713401 · Cited by 16
Self-citation
Metadata
- Authors
- Wang, B., Li, Y., Lv, Z., Xia, H., Xu, Y., & Sodhi, R.
- Venue
- Proceedings of the 29th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2024-03-18
- Pages
- 699-714
- OpenAlex
- W4391940619
- Cited by
- 63
- References
- 40
- FWCI
- 25.17
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.