CoChat
Sign Up
Cited by 39Usability and User Interface DesignSpeech and dialogue systemsSpeech Recognition and Synthesis

Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation

Open access
Lin, Susan, Warner, Jeremy, Zamfirescu-Pereira, J.D., Lee, Matthew G, Jain, Sauhard, Cai, Shanqing, Lertvittayakumjorn, Piyawat, Huang, Michael Xuelin +3 more·2024·Proceedings of the CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3613904.3642217
Open this paper in CoChat — free accountOpen PDF

Abstract

Dictation enables efficient text input on mobile devices. However, writing with speech can produce disfluent, wordy, and incoherent text and thus requires heavy post-processing. This paper presents Rambler, an LLM-powered graphical user interface that supports gist-level manipulation of dictated text with two main sets of functions: gist extraction and macro revision. Gist extraction generates keywords and summaries as anchors to support the review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. Together they pave the way for interactive dictation and revision that help close gaps between spontaneously spoken words and well-structured writing. In a comparative study with 12 participants performing verbal composition tasks, Rambler outperformed the baseline of a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over the content while supporting surprisingly diverse user strategies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642217

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Kenneth C Arnold, April M Volzer, and Noah G Madrid. 2021. Generative Models can Help Writers without Writing for Them.. In IUI Workshops.
  2. Oloff C. Biermann, Ning F. Ma, and Dongwook Yoon. 2022. From Tool to Compan- ion: Storywriters Want AI Writers to Respect Their Personal Values and Writing Strategies. Proceedings of the 2022 ACM Designing Interactive Systems Conference (2022). https://api.semanticscholar.org/CorpusID:249578888
  3. Oloff C. Biermann, Ning F. Ma, and Dongwook Yoon. 2022. From Tool to Com- panion: Storywriters Want AI Writers to Respect Their Personal Values and Writing Strategies. In Proceedings of the 2022 ACM Designing Interactive Systems Conference (Virtual Event) (DIS '22). Association for Computing Machinery, New York, NY, USA, 1209-1227. https://doi.org/10.1145/3532106.3533506 · DOI · Paper page
  4. Mary Jo Bitner, Amy L Ostrom, and Felicia N Morgan. 2008. Service blueprinting: a practical technique for service innovation. California management review 50, 3 (2008), 66-94.
  5. Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Explo- ration with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Association for Computing Machinery, New York, NY, USA, Article 96, 14 pages. https://doi.org/10.1145/3586183.3606725 · DOI
  6. Hai Dang, Karim Benharrak, Florian Lehmann, and Daniel Buschek. 2022. Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST '22). Association for Computing Machinery, New York, NY, USA, Article 98, 13 pages. https://doi.org/10.1145/3526113.3545672 · DOI
  7. Hai Dang, Frederik Brudy, George Fitzmaurice, and Fraser Anderson. 2023. WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23). Asso- ciation for Computing Machinery, New York, NY, USA, Article 63, 17 pages. https://doi.org/10.1145/3586183.3606772 · DOI
  8. Jiayue Fan, Chenning Xu, Chun Yu, and Yuanchun Shi. 2021. Just Speak It: Minimize Cognitive Load for Eyes-Free Text Editing with a Smart Voice Assistant. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 910-921. https://doi.org/10.1145/3472749.3474795 · DOI
  9. Debjyoti Ghosh, Can Liu, Shengdong Zhao, and Kotaro Hara. 2020. Commanding and Re-Dictation: Developing Eyes-Free Voice-Based Interaction for Editing Dictated Text. ACM Trans. Comput.-Hum. Interact. 27, 4, Article 28 (aug 2020), 31 pages. https://doi.org/10.1145/3390889 · DOI
  10. Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. 2022. Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers. ArXiv abs/2211.05030 (2022). https://api.semanticscholar.org/CorpusID: 253420678 · arXiv
  11. Peiling Jiang, Jude Rayan, Steven P Dow, and Haijun Xia. 2023. Graphologue: Exploring Large Language Model Responses with Interactive Diagrams. arXiv preprint arXiv:2305.11473 (2023). · arXiv
  12. Clare-Marie Karat, Christine Halverson, Daniel Horn, and John Karat. 1999. Patterns of Entry and Correction in Large Vocabulary Continuous Speech Recog- nition Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI '99). Association for Computing Machinery, New York, NY, USA, 568-575. https://doi.org/10.1145/ 302979.303160 · DOI
  13. Belinda Z. Li, Jason Eisner, Adam Pauls, and Sam Thomson. 2023. Toward Interactive Dictation. arXiv:2307.04008 [cs.CL] · arXiv
  14. Daniel Li, Thomas Chen, Albert Tung, and Lydia B Chilton. 2021. Hierar- chical Summarization for Longform Spoken Dialog. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 582-597. https://doi.org/10.1145/3472749.3474771 · DOI
  15. Daniel Li, Thomas Chen, Alec Zadikian, Albert Tung, and Lydia B Chilton. 2023. Improving Automatic Summarization for Browsing Longform Spoken Dialog. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1-20.
  16. Junwei Liao, Sefik Eskimez, Liyang Lu, Yu Shi, Ming Gong, Linjun Shou, Hong Qu, and Michael Zeng. 2023. Improving readability for automatic speech recognition transcription. ACM Transactions on Asian and Low-Resource Language Information Processing 22, 5 (2023), 1-23.
  17. Can Liu, Siying Hu, Li Feng, and Mingming Fan. 2022. Typist Experiment: An Investigation of Human-to-Human Dictation via Role-Play to Inform Voice-Based Text Authoring. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 338 (nov 2022), 33 pages. https://doi.org/10.1145/3555758 · DOI
  18. Paria Jamshid Lou and Mark Johnson. 2020. End-to-End Speech Recognition and Disfluency Removal. In Findings of the Association for Computational Linguistics: EMNLP 2020. 2051-2061.
  19. Brinda Mehra, Kejia Shen, Hen Chen Yen, and Can Liu. 2023. Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition. In Proceedings of the 5th International Conference on Conversational User Interfaces (Eindhoven, Netherlands) (CUI '23). Association for Computing Machinery, New York, NY, USA, Article 15, 11 pages. https://doi.org/10.1145/3571884.3597134 · DOI
  20. Bill Moggridge. 2006. Designing Interactions. The MIT Press.
  21. Christine Murad, Cosmin Munteanu, Benjamin R. Cowan, and Leigh Clark. 2019. Revolution or Evolution? Speech Interaction and HCI Design Guidelines. IEEE Pervasive Computing 18, 2 (2019), 33-45. https://doi.org/10.1109/MPRV.2019. 2906991 · DOI · Paper page
  22. Ken Perlin and David Fox. 1993. Pad: An Alternative Approach to the Computer Interface. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (Anaheim, CA) (SIGGRAPH '93). Association for Computing Machinery, New York, NY, USA, 57-64. https://doi.org/10.1145/166117.166125 · DOI
  23. Mizanur Rahman, Harold Jan Redonda Terano, Nafizur Rahman, Aidin Salamzadeh, and Saidur Rahaman. 2023. ChatGPT and Academic Research: A Review and Recommendations Based on Practical Examples. Journal of Educa- tion, Management and Development Studies (2023). https://api.semanticscholar. org/CorpusID:257845986
  24. Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Au- tomatic Keyword Extraction from Individual Documents. John Wiley & Sons, Ltd, Chapter 1, 1-20. https://doi.org/10.1002/9780470689646.ch1 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470689646.ch1 · DOI
  25. Sherry Ruan, Jacob O. Wobbrock, Kenny Liou, Andrew Ng, and James A. Landay. 2018. Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 159 (jan 2018), 23 pages. https://doi.org/10.1145/3161187 · DOI
  26. J Schler, M Koppel, S Argamon, and JW Pennebaker. 2006. Effects of Age and Gender on Blogging in Proceedings of 2006 AAAI Spring Symposium on Compu- tational Approaches for Analyzing Weblogs. In Proceedings of 2006 AAAI Spring Symposium on Computational Approaches for Analyzing Weblogs, Vol. 1.
  27. Carola Strobl, Emilie Ailhaud, Kalliopi Benetos, Ann Devitt, Otto Kruse, Antje Proske, and Christian Rapp. 2019. Digital support for academic writing: A review of technologies and pedagogies. Computers & education 131 (2019), 33-48.
  28. Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. arXiv preprint arXiv:2305.11483 (2023). · arXiv
  29. Tomohiro Tanaka, Ryo Masumura, Hirokazu Masataki, and Yushi Aono. 2018. Neural Error Corrective Language Models for Automatic Speech Recognition.. In INTERSPEECH. 401-405.
  30. Daniel Vogel and Patrick Baudisch. 2007. Shift: a technique for operating pen- based interfaces using touch. In Proceedings of the SIGCHI conference on Human factors in computing systems. 657-666.
  31. Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and LC Ray. 2022. AI as an Active Writer: Interaction Strategies with Generated Text in Human-AI Collaborative Fiction Writing 56-65. In IUI Workshops. https: //api.semanticscholar.org/CorpusID:248301902
  32. J.D. Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G Lee, Bjoern Hartmann, and Qian Yang. 2023. Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS '23). As- sociation for Computing Machinery, New York, NY, USA, 2206-2220. https: //doi.org/10.1145/3563657.3596138 · DOI
  33. J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/ 3544548.3581388 · DOI · Paper page
  34. Maozheng Zhao, Wenzhe Cui, IV Ramakrishnan, Shumin Zhai, and Xiaojun Bi. 2021. Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for Smartphones. In The 34th Annual ACM Symposium on User Interface Software and Technology. 162-178.
  35. Maozheng Zhao, Henry Huang, Zhi Li, Rui Liu, Wenzhe Cui, Kajal Toshniwal, Ananya Goel, Andrew Wang, Xia Zhao, Sina Rashidian, Furqan Baig, Khiem Phi, Shumin Zhai, IV Ramakrishnan, Fusheng Wang, and Xiaojun Bi. 2022. Eye- SayCorrect: Eye Gaze and Voice Based Hands-Free Text Correction for Mobile Devices. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI '22). Association for Computing Machinery, New York, NY, USA, 470-482. https://doi.org/10.1145/3490099.3511103 · DOI
  36. Wanzheng Zhu and Suma Bhat. 2020. GRUEN for Evaluating Linguistic Quality of Generated Text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 94-108.

Cited by

Sorted by citations

Authors
Lin, S., Warner, J., Zamfirescu-Pereira, J. D., Lee, M. G., Jain, S., Cai, S., Lertvittayakumjorn, P., Huang, M. X., Zhai, S., Hartmann, B., & Liu, C.
Affiliations
  • University of California, Berkeley (US)
  • Google (United States) (US)
  • EECS, UC Berkeley, United States
  • City University of Hong Kong (HK)
ORCID
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-19
DOI
10.1145/3613904.3642217
OpenAlex
W4396833729
Cited by
39
References
3
Citations by year
2026: 7 · 2025: 28 · 2024: 4
FWCI
12.57
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
Hong Kong Research Grants Council - ECS scheme (CityU 21209419)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
keyword extraction, textual revision, graphical user interface, disfluent speech, mobile writing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review