CoChat
Sign Up
Cited by 62Innovative Human-Technology InteractionDesign Education and PracticeRecommender Systems and Techniques

How AI Processing Delays Foster Creativity: Exploring Research Question Co-Creation with an LLM-based Agent

Liu, Yiren, Chen, Si, Cheng, Haocong, Yu, Mengxia, Ran, Xiao, Mo, Andrew, Tang, Yiliu, Huang, Yun·2024·Proceedings of the CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3613904.3642698
Open this paper in CoChat — free account

Abstract

Developing novel research questions (RQs) often requires extensive literature reviews, especially in interdisciplinary fields. To support RQ development through human-AI co-creation, we leveraged Large Language Models (LLMs) to build an LLM-based agent system named CoQuest. We conducted an experiment with 20 HCI researchers to examine the impact of two interaction designs: breadth-first and depth-first RQ generation. The findings revealed that participants perceived the breadth-first approach as more creative and trustworthy upon task completion. Conversely, during the task, participants considered the depth-first generated RQs as more creative. Additionally, we discovered that AI processing delays allowed users to reflect on multiple RQs simultaneously, leading to a higher quantity of generated RQs and an enhanced sense of control. Our work makes both theoretical and practical contributions by proposing and evaluating a mental model for human-AI co-creation of RQs. We also address potential ethical issues, such as biases and over-reliance on AI, advocating for using the system to improve human research creativity rather than automating scientific inquiry. The system’s source is available at: https://github.com/yiren-liu/coquest.

Integrity signals

Registry

Crossref · 10.1145/3613904.3642698

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The literature review that forms the basis of research conception may include facts or citations that are inaccurate, irrelevant, or fake, i.e., hallucinations.

    “First, the literature review that forms the basis of research conception may include facts or citations that are inaccurate, irrelevant, or fake, i.e., “hallucinations” [5, 34–36].”

    From Autonomous artificial intelligence, scientific research, and human values (2026) · cites this paper as [34]

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. Tim Dettmers and Luke Zettlemoyer. 2023. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning. PMLR, 7750–7774.
  2. Ziv Epstein, Hope Schroeder, and Dava Newman. 2022. When happy accidents spark creativity: Bringing collaborative speculation to life with generative AI. In International Conference on Innovative Computing and Cloud Computing.
  3. EunJeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon. 2023. Aligning Language Models to User Opinions. arXiv preprint arXiv:2305.14929 (2023). · arXiv
  4. Shahedul Huq Khandkar. 2009. Open coding. University of Calgary 23, 2009 (2009).
  5. Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating Mixed-Initiative Creative Interfaces via Expressive Range Coverage Analysis.. In IUI Workshops. 34–45.
  6. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning. PMLR, 19274–19286.
  7. Zhiyu Lin, Upol Ehsan, Rohan Agarwal, Samihan Dani, Vidushi Vashishth, and Mark Riedl. 2023. Beyond Prompts: Exploring the Design Space of Mixed-Initiative Co-Creativity Systems. arXiv preprint arXiv:2305.07465 (2023). · arXiv
  8. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. AgentBench: Evaluating LLMs as Agents. ArXiv abs/2308.03688 (2023). · arXiv
  9. Yiren Liu, Mengxia Yu, Meng-Long Jiang, and Yun Huang. 2023. Creative Research Question Generation for Human-Computer Interaction Research. In IUI Workshops.
  10. Michele Loi, Eleonora Viganò, and Lonneke van der Plas. 2020. The societal and ethical relevance of computational creativity. arXiv preprint arXiv:2007.11973 (2020). · arXiv
  11. Michael Muller, Justin D Weisz, and Werner Geyer. 2020. Mixed initiative generative AI interfaces: An analytic framework for generative AI applications. In Proceedings of the Workshop The Future of Co-Creative Systems-A Workshop on Human-Computer Co-Creativity of the 11th International Conference on Computational Creativity (ICCC 2020).
  12. Michael Sheinman Orenstrakh, Oscar Karnalim, Carlos Anibal Suarez, and Michael Liut. 2023. Detecting llm-generated text in computing education: A comparative study for chatgpt cases. arXiv preprint arXiv:2307.07411 (2023). · arXiv
  13. Ought. 2023. Elicit: The AI Research Assistant. https://elicit.org
  14. Carole L Palmer, Lauren C Teffeau, and Carrie M Pirmann. 2009. Scholarly information practices in the online environment. Report commissioned by OCLC Research. Published online at: www. oclc. org/programs/publications/reports/2009-02. pdf (2009).
  15. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924 (2023). · arXiv
  16. Riaz Qureshi, Daniel Shaughnessy, Kayden A. R. Gill, Karen A. Robinson, Tianjing Li, and Eitan S. Agai. 2023. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation?Systematic Reviews 12 (2023).
  17. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019). · arXiv
  18. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023. LaMP: When Large Language Models Meet Personalization. arXiv preprint arXiv:2304.11406 (2023). · arXiv
  19. Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 (2023). · arXiv
  20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). · arXiv
  21. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 (2023). · arXiv
  22. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.
  23. Yuqing Wang and Yun Zhao. 2023. Metacognitive Prompting Improves Understanding in Large Language Models. arXiv preprint arXiv:2308.05342 (2023). · arXiv
  24. Justin D Weisz, Michael Muller, Jessica He, and Stephanie Houde. 2023. Toward general design principles for generative AI applications. arXiv preprint arXiv:2301.05578 (2023). · arXiv
  25. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing Reasoning and Acting in Language Models. ArXiv abs/2210.03629 (2022). · arXiv
  26. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022). · arXiv
  27. Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311 (2023). · arXiv
  28. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, 2023. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint arXiv:2309.01219 (2023). · arXiv
  29. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The Eleventh International Conference on Learning Representations.
  30. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can Large Language Models Transform Computational Social Science?arXiv preprint arXiv:2305.03514 (2023). · arXiv

Cited by

Sorted by citations

Related

Authors
Liu, Y., Chen, S., Cheng, H., Yu, M., Ran, X., Mo, A., Tang, Y., & Huang, Y.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • University of Notre Dame (US)
ORCID
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-25
DOI
10.1145/3613904.3642698
OpenAlex
W4396833486
Semantic Scholar
b4db5c8d8f3f0cb3e37305314fb7edbabaca43f1
Cited by
61
References
39
Citations by year
2026: 25 · 2025: 32 · 2024: 4
FWCI
19.43
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Funders
NSF (2119589)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
research question formulation, human-AI co-creation, large language models, LLM agents, creativity, mental models, trust in AI, overreliance on AI, algorithmic bias, HCI research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review