CoChat

LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

Open access
Wang, Bryan, Li, Yuliang, Lv, Zhaoyang, Xia, Haijun, Xu, Yan, Sodhi, Raj · 2024 · Proceedings of the 29th International Conference on Intelligent User Interfaces
Cited by 63

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Abstract

Video creation has become increasingly popular, yet the expertise and effort required for editing often pose barriers to beginners. In this paper, we explore the integration of large language models (LLMs) into the video editing workflow to reduce these barriers. Our design vision is embodied in LAVE, a novel system that provides LLM-powered agent assistance and language-augmented editing features. LAVE automatically generates language descriptions for the user’s footage, serving as the foundation for enabling the LLM to process videos and assist in editing tasks. When the user provides editing objectives, the agent plans and executes relevant actions to fulfill them. Moreover, LAVE allows users to edit videos through either the agent or direct UI manipulation, providing flexibility and enabling manual refinement of agent actions. Our user study, which included eight participants ranging from novices to proficient editors, demonstrated LAVE’s effectiveness. The results also shed light on user perceptions of the proposed LLM-assisted editing paradigm and its impact on users’ creativity and sense of co-creation. Based on these findings, we propose design implications to inform the future development of agent-assisted content editing.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3640543.3645143

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 40 references

Research works printed but not linked: 38

  1. 10.1080/19393555.2022.2060879 · DOI
  2. Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. arxiv:2304.09337 [cs.HC] · arXiv
  3. Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine potential pitfalls when designing human-ai co-creative systems. arXiv preprint arXiv:2104.00358 (2021). · arXiv
  4. 10.1145/3411764.3445131 · DOI
  5. 10.1145/3290605.3300931 · DOI
  6. 10.1038/s42256-020-0161-x · DOI
  7. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. 2022. Imagen Video: High Definition Video Generation with Diffusion Models. arxiv:2210.02303 [cs.CV] · arXiv
  8. Wenlong Huang Pieter Abbeel Deepak Pathak and Igor Mordatch. 2022. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. arxiv:2201.07207 [cs.LG] · arXiv
  9. 10.1145/3290605.3300311 · DOI
  10. 10.1145/3544548.3581494 · DOI · Paper page
  11. 10.1093/jcmc/zmac014 · DOI · Paper page
  12. Ehud Karpas Omri Abend Yonatan Belinkov Barak Lenz Opher Lieber Nir Ratner Yoav Shoham Hofit Bata Yoav Levine Kevin Leyton-Brown Dor Muhlgay Noam Rozen Erez Schwartz Gal Shachaf Shai Shalev-Shwartz Amnon Shashua and Moshe Tenenholtz. 2022. MRKL Systems: A modular neuro-symbolic architecture that combines large language models external knowledge sources and discrete reasoning. arxiv:2205.00445 [cs.CL] · arXiv
  13. 10.1145/2470654.2481301 · DOI · Paper page
  14. 10.1145/3544548.3581006 · DOI
  15. Vivian Liu, Tao Long, Nathan Raw, and Lydia Chilton. 2023. Generative Disco: Text-to-Video Generation for Music Visualization. arXiv preprint arXiv:2304.08551 (2023). · arXiv
  16. 10.1145/3526113.3545621 · DOI · Paper page
  17. 10.1145/3563657.3596098 · DOI · Paper page
  18. 10.1145/3490099.3511159 · DOI · Paper page
  19. Federico Magni, Jiyoung Park, and Melody Manchi Chao. 2023. Humans as Creativity Gatekeepers: Are We Biased Against AI Creativity?Journal of Business and Psychology (2023), 1–14.
  20. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arxiv:2112.00114 [cs.LG] · arXiv
  21. OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL] · arXiv
  22. 10.1145/3311957.3359433 · DOI
  23. 10.1145/3379337.3415864 · DOI · Paper page
  24. Jeba Rezwana and Mary Lou Maher. 2022. Identifying ethical issues in ai partners in human-ai co-creation. arXiv preprint arXiv:2204.07644 (2022). · arXiv
  25. 10.18653/v1/2020.emnlp-main.437 · DOI · Paper page
  26. 10.1145/3526113.3545617 · DOI · Paper page
  27. Derek Tam Anisha Mascarenhas Shiyue Zhang Sarah Kwan Mohit Bansal and Colin Raffel. 2022. Evaluating the Factual Consistency of Large Language Models Through Summarization. arxiv:2211.08412 [cs.CL] · DOI · Paper page
  28. 10.1145/2984511.2984569 · DOI · Paper page
  29. Maarten Van Someren, Yvonne F Barnard, and J Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. London: AcademicPress 11 (1994), 29–41.
  30. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Shomir Wilson, 2023. Nationality Bias in Text Generation. arXiv preprint arXiv:2302.02463 (2023). · arXiv
  31. 10.1145/3526113.3545680 · DOI
  32. 10.1145/3355089.3356520 · DOI
  33. Sitong Wang, Samia Menon, Tao Long, Keren Henderson, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V Nickerson, and Lydia B Chilton. 2023. ReelFramer: Co-creating News Reels on Social Media with Generative AI. arXiv preprint arXiv:2304.09653 (2023). · arXiv
  34. 10.1145/3379337.3415845 · DOI · Paper page
  35. 10.1145/3379337.3415882 · DOI · Paper page
  36. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421 9, 1 (2023). · arXiv
  37. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. arXiv preprint arXiv:2306.15895 (2023). · arXiv
  38. 10.1145/3491102.3517479 · DOI · Paper page

Cited by

See all 63

Metadata

Authors
Wang, B., Li, Y., Lv, Z., Xia, H., Xu, Y., & Sodhi, R.
Venue
Proceedings of the 29th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2024-03-18
Pages
699-714
OpenAlex
W4391940619
Cited by
63
References
40
FWCI
25.17
Type
conference paper
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing. https://app.cochat.ai/papers/10.1145/3640543.3645143
The paper
Wang, B., Li, Y., Lv, Z., Xia, H., Xu, Y., & Sodhi, R. (2024). LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing. Proceedings of the 29th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3640543.3645143

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.