The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.
AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation
Open accessAbstract
The growing availability of generative AI technologies such as large language models (LLMs) has significant implications for creative work. This paper explores twofold aspects of integrating LLMs into the creative process – the divergence stage of idea generation, and the convergence stage of evaluation and selection of ideas. We devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space. To assess the potential of using LLMs in the idea evaluation process, we design an evaluation engine and compared it to idea ratings assigned by three expert and six novice evaluators. Our findings suggest that integrating LLM in Brainwriting could enhance both the ideation process and its outcome. We also provide evidence that LLMs can support idea evaluation. We conclude by discussing implications for HCI education and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Conversely, other studies highlight LLMs under-performance in comparison to humans [118].”
From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [118]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we argue that AI pair programmers simultaneously increase expected benefits by enhancing ideation capabilities and enabling exploration of alternative solutions”
From The Impact of Generative AI on Collaborative Open-Source Software Development: Evidence from GitHub Copilot (2024) · cites this paper as Shaer et al. 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Empirical evidence on the creativity of LLM outputs is mixed.”
From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [118]
The cited source says what the claim attributes to it.
In the citing paper
“AI tools can serve as creative aids, offering suggestions that spark new ideas (Mollick, 2023;Shaer et al., 2024).”
From Fanfiction in the Age of AI: Community Perspectives on Creativity, Authenticity and Adoption (2026) · cites this paper as Shaer et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Conceptual Integration Networks
Fauconnier, G., & Turner, M. B. (1998) · Cognitive Science · doi:10.1016/s0364-0213(99)80038-x · Cited by 2,038 · cited 1×
“” Student teams combined these approaches during the ideation session.”
BackgroundSketchpad: a man-machine graphical communication system
Sutherland, I. E. (1963) · Proceedings of the May 21-23, 1963, spring joint computer conference on - AFIPS '63 (Spring) · doi:10.1145/1461551.1461591 · Cited by 1,821
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 1×
“The ideas chosen by each group for their fnal project were mostly created by combining an idea generated by team members and an idea suggested or enhanced by the LLM.”
MethodHow breakthroughs happen: the surprising truth about how companies innovate
(2004) · Choice Reviews Online · doi:10.5860/choice.41-6004 · Cited by 559 · cited 1×
“During a successful group brainstorming session, participants draw on each other’s ideas and pre-existing knowledge to combine ideas in new ways [26].”
BackgroundTeams in Organizations: Prevalence, Characteristics, and Effectiveness
Devine, D. J., Clayton, L. D., Philips, J. L., Dunford, B. B., & Melner, S. B. (1999) · Small Group Research · doi:10.1177/104649649903000602 · Cited by 493 · cited 1×
“In this paper, rather than focusing on large-scale ideation, we explore ways to enhance small groups (3-4 people) ideation through the use of LLMs. Brainstorming [54] is one of the most widely adopted techniques for generating creative ideas within groups [11].”
Identifying Quality, Novel, and Creative Ideas: Constructs and Scales for Idea Evaluation
Brigham Young University, Dean, D., Hender, J., Henley Management College, Rodgers, T., Consultant College Station, T., Santanen, E., & Bucknell University (2006) · Journal of the Association for Information Systems · doi:10.17705/1jais.00106 · Cited by 476 · cited 3×
“We chose to use two criteria from Dean et al.’s evaluation framework [10]: relevance and innovation .”
InfluentialEntrepreneur behaviors, opportunity recognition, and the origins of innovative ventures
Dyer, J. H., Gregersen, H. B., & Christensen, C. (2008) · Strategic Entrepreneurship Journal · doi:10.1002/sej.59 · Cited by 461 · cited 2×
“In addition, we chose a third criterion, insightfulness , based on Dyer et al.’s research on the origin of innovative ventures [16].”
UX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · cited 2×
“Considering the challenges interaction designers face when working with AI as a design material [14, 32, 69, 81, 84], this course aims to integrate co-creation and critical engagement with generative AI into its learning goals.”
BackgroundTechnology and More-Than-Human Design
Giaccardi, E., & Redström, J. (2020) · Design Issues · doi:10.1162/desi_a_00612 · Cited by 286 · cited 3×
“Emerging theories about posthumanism, post-human, and more-than-human interaction design [19, 30, 79, 80] provide further context for human-AI co-creation activities by highlighting possibilities to distribute agency in design among humans and non-humans.”
Research works printed but not linked: 22
- I Elaine Allen and Christopher A Seaman. 2007. Likert scales and data analyses. Quality progress 40, 7 (2007), 64–65.
- Virginia Braun and Victoria Clarke. 2012. Thematic analysis.American Psychological Association, Washington, D.C.
- David H. Cropley, Caroline Theurer, Sven Mathijssen, and Rebecca L. Marrone. 2023. Fit-for-Purpose Creativity Assessment: Using Machine Learning to Score a Figural Creativity Test. PsyArXiv Preprints N/A, N/A (2023), N/A. Available online at PsyArXiv.
- Edward De Bono. 1999. Six Thinking Hats. Back Bay Books, New York.
- Michael Diehl and Wolfgang Stroebe. 1987. Productivity loss in brainstorming groups: Toward the solution of a riddle.Journal of personality and social psychology 53, 3 (1987), 497.
- Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN N/A, N/A (2023), N/A.
- Google. 2023. Bard: Chat-Based AI Tool from Google, Powered by PaLM 2. https://bard.google.com/. Accessed: 14-09-2023.
- Daniel Kahneman, Olivier Sibony, and Cass R Sunstein. 2021. Noise: a flaw in human judgment. Hachette UK, London, UK.
- Lars Krupp, Steffen Steinert, Maximilian Kiefer-Emmanouilidis, Karina E Avila, Paul Lukowicz, Jochen Kuhn, Stefan Küchemann, and Jakob Karolus. 2023. Unreflected Acceptance–Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education. arXiv preprint arXiv:2309.03087 N/A, N/A (2023), N/A. · arXiv
- Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics 22, 1 (1951), 79–86.
- Harsh Kumar Ilya Musabirov Mohi Reza Jiakai Shi Anastasia Kuzminykh Joseph Jay Williams and Michael Liut. 2023. Impact of guidance and interaction strategies for LLM use on Learner Performance and perception. https://arxiv.org/abs/2310.13712 · arXiv
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- J McCormack and A Dorin. 2014. Generative Design: A Paradigm for Design Research. Futureground - DRS International Conference N/A, N/A (2014), 17–21.
- Alex F Osborn. 1953. Applied imagination.Charles Scribner’s Son’s, New York, USA.
- Jeongeon Park, Bryan Min, Xiaojuan Ma, and Juho Kim. 2023. Choicemates: Supporting unfamiliar online decision-making with multi-agent conversational interactions. https://arxiv.org/abs/2310.01331 · arXiv
- Paul B Paulus and Mary T Dzindolet. 1993. Social influence processes in group brainstorming.Journal of personality and social psychology 64, 4 (1993), 575.
- Paul B Paulus and Huei-Chuan Yang. 2000. Idea generation in groups: A basis for creativity in organizations. Organizational behavior and human decision processes 82, 1 (2000), 76–87.
- Vildan Salikutluk, Dorothea Koert, and Frank Jäkel. 2023. Interacting with Large Language Models: A Case Study on AI-Aided Brainstorming for Guesstimation Problems. In HHAI 2023: Augmenting Human Intellect. IOS Press, Amsterdam, Netherlands, 153–167.
- 10.48456/tr-574 · doi:10.48456/tr-574
- Barbara Tversky and Juliet Y. Chou. 2011. Creativity: Depth and Breadth. In Design Creativity 2010, Toshiharu Taura and Yukari Nagai (Eds.). Springer London, London, 209–214.
- Brygg Ullmer, Orit Shaer, Ali Mazalek, and Caroline Hummels. 2022. Weaving Fire into Form: Aspirations for Tangible and Embodied Interaction (1 ed.). Vol. 44. Association for Computing Machinery, New York, NY, USA.
- 10.1145/3393914.3395886 · DOI
Cited by
Hadan, H., Wang, D., Mogavi, R. H., Tu, J., Zhang-Kennedy, L., & Nacke, L. E. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100095 · Cited by 91
“Our RQ1 aims to validate this hypothesis by examining reviewers ’ awareness across various levels of AI involvement in research writing.”
Pang, R. Y., Schroeder, H., Smith, K. S., Barocas, S., Xiao, Z., Tseng, E., & Bragg, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713726 · Cited by 146
“We found 8 theoretical contributions (5.23%), ranging from a framework for collaborative group-AI brain-writing [139], a conceptual framework to bridge the gulf of envisioning [148], and a design space for intelligent writing assistants [89] (also a metareview contribution).”
InfluentialFrom Pen to Prompt: How Creative Writers Integrate AI into their Writing Practice
Guo, A., Sathyanarayanan, S., Wang, L., Heer, J., & Zhang, A. X. (2025) · Creativity and Cognition · doi:10.1145/3698061.3726910 · Cited by 77
Tang, M., Hofreiter, S., Werner, C. H., Zielińska, A., & Karwowski, M. (2025) · The Journal of Creative Behavior · doi:10.1002/jocb.1519 · Cited by 32
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Metadata
- Authors
- Shaer, O., Cooper, A., Mokryn, O., Kun, A. L., & Ben Shoshan, H.
- Affiliations
- Wellesley College (US)
- University of Haifa (IL)
- University of New Hampshire (US)
- Information and Knowledge Management, University of Haifa, Israel
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-17
- OpenAlex
- W4396833086
- Cited by
- 130
- References
- 48
- Citations by year
- 2026: 48 · 2025: 72 · 2024: 10
- FWCI
- 166.75
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1840085)
- Topics
- Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- brainwriting, large language models, idea generation, idea evaluation, concept selection, creative process, collaborative AI, solution space, human-AI collaboration, HCI education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. https://app.cochat.ai/papers/10.1145/3613904.3642414
- The paper
- Shaer, O., Cooper, A., Mokryn, O., Kun, A. L., & Ben Shoshan, H. (2024). AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642414
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.