Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 45 of the 48 references it cites were found in a registry or an index.

AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation

Open access
Shaer, Orit, Cooper, Angelora, Mokryn, Osnat, Kun, Andrew L, Ben Shoshan, Hagit · 2024 · Proceedings of the CHI Conference on Human Factors in Computing Systems

Abstract

The growing availability of generative AI technologies such as large language models (LLMs) has significant implications for creative work. This paper explores twofold aspects of integrating LLMs into the creative process – the divergence stage of idea generation, and the convergence stage of evaluation and selection of ideas. We devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space. To assess the potential of using LLMs in the idea evaluation process, we design an evaluation engine and compared it to idea ratings assigned by three expert and six novice evaluators. Our findings suggest that integrating LLM in Brainwriting could enhance both the ideation process and its outcome. We also provide evidence that LLMs can support idea evaluation. We conclude by discussing implications for HCI education and practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642414

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Research works printed but not linked: 22

  1. I Elaine Allen and Christopher A Seaman. 2007. Likert scales and data analyses. Quality progress 40, 7 (2007), 64–65.
  2. Virginia Braun and Victoria Clarke. 2012. Thematic analysis.American Psychological Association, Washington, D.C.
  3. David H. Cropley, Caroline Theurer, Sven Mathijssen, and Rebecca L. Marrone. 2023. Fit-for-Purpose Creativity Assessment: Using Machine Learning to Score a Figural Creativity Test. PsyArXiv Preprints N/A, N/A (2023), N/A. Available online at PsyArXiv.
  4. Edward De Bono. 1999. Six Thinking Hats. Back Bay Books, New York.
  5. Michael Diehl and Wolfgang Stroebe. 1987. Productivity loss in brainstorming groups: Toward the solution of a riddle.Journal of personality and social psychology 53, 3 (1987), 497.
  6. Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN N/A, N/A (2023), N/A.
  7. Google. 2023. Bard: Chat-Based AI Tool from Google, Powered by PaLM 2. https://bard.google.com/. Accessed: 14-09-2023.
  8. Daniel Kahneman, Olivier Sibony, and Cass R Sunstein. 2021. Noise: a flaw in human judgment. Hachette UK, London, UK.
  9. Lars Krupp, Steffen Steinert, Maximilian Kiefer-Emmanouilidis, Karina E Avila, Paul Lukowicz, Jochen Kuhn, Stefan Küchemann, and Jakob Karolus. 2023. Unreflected Acceptance–Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education. arXiv preprint arXiv:2309.03087 N/A, N/A (2023), N/A. · arXiv
  10. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics 22, 1 (1951), 79–86.
  11. Harsh Kumar Ilya Musabirov Mohi Reza Jiakai Shi Anastasia Kuzminykh Joseph Jay Williams and Michael Liut. 2023. Impact of guidance and interaction strategies for LLM use on Learner Performance and perception. https://arxiv.org/abs/2310.13712 · arXiv
  12. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
  13. J McCormack and A Dorin. 2014. Generative Design: A Paradigm for Design Research. Futureground - DRS International Conference N/A, N/A (2014), 17–21.
  14. Alex F Osborn. 1953. Applied imagination.Charles Scribner’s Son’s, New York, USA.
  15. Jeongeon Park, Bryan Min, Xiaojuan Ma, and Juho Kim. 2023. Choicemates: Supporting unfamiliar online decision-making with multi-agent conversational interactions. https://arxiv.org/abs/2310.01331 · arXiv
  16. Paul B Paulus and Mary T Dzindolet. 1993. Social influence processes in group brainstorming.Journal of personality and social psychology 64, 4 (1993), 575.
  17. Paul B Paulus and Huei-Chuan Yang. 2000. Idea generation in groups: A basis for creativity in organizations. Organizational behavior and human decision processes 82, 1 (2000), 76–87.
  18. Vildan Salikutluk, Dorothea Koert, and Frank Jäkel. 2023. Interacting with Large Language Models: A Case Study on AI-Aided Brainstorming for Guesstimation Problems. In HHAI 2023: Augmenting Human Intellect. IOS Press, Amsterdam, Netherlands, 153–167.
  19. 10.48456/tr-574 · doi:10.48456/tr-574
  20. Barbara Tversky and Juliet Y. Chou. 2011. Creativity: Depth and Breadth. In Design Creativity 2010, Toshiharu Taura and Yukari Nagai (Eds.). Springer London, London, 209–214.
  21. Brygg Ullmer, Orit Shaer, Ali Mazalek, and Caroline Hummels. 2022. Weaving Fire into Form: Aspirations for Tangible and Embodied Interaction (1 ed.). Vol. 44. Association for Computing Machinery, New York, NY, USA.
  22. 10.1145/3393914.3395886 · DOI

Cited by

See all 133

Metadata

Authors
Shaer, O., Cooper, A., Mokryn, O., Kun, A. L., & Ben Shoshan, H.
Affiliations
  • Wellesley College (US)
  • University of Haifa (IL)
  • University of New Hampshire (US)
  • Information and Knowledge Management, University of Haifa, Israel
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-17
OpenAlex
W4396833086
Cited by
130
References
48
Citations by year
2026: 48 · 2025: 72 · 2024: 10
FWCI
166.75
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1840085)
Topics
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
brainwriting, large language models, idea generation, idea evaluation, concept selection, creative process, collaborative AI, solution space, human-AI collaboration, HCI education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. https://app.cochat.ai/papers/10.1145/3613904.3642414
The paper
Shaer, O., Cooper, A., Mokryn, O., Kun, A. L., & Ben Shoshan, H. (2024). AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642414

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.