CoChat
Sign Up
Cited by 60Ethics and Social Impacts of AIInnovative Human-Technology InteractionDigital Media and Visual Art

User Perspectives on Ethical Challenges in Human-AI Co-Creativity: A Design Fiction Study

Open access
Rezwana, Jeba, Maher, Mary Lou·2023·Creativity and Cognition
DOI 10.1145/3591196.3593364
Open this paper in CoChat — free accountOpen PDF

Abstract

In a human-AI co-creation, AI not only categorizes, evaluates and interprets data but also generates new content and interacts with humans. As co-creative AI is a form of intelligent technology that directly involves humans, it is critical to anticipate and address ethical issues during all design stages. The open-ended nature of human-AI interactions in co-creation poses many challenges for designing ethical co-creative AI systems. Researchers have been exploring ethical issues associated with autonomous AI in recent years, but ethics in human-AI co-creativity is a relatively new research area. In order to design human-centered ethical AI, it is important to understand the perspectives, expectations, and ethical concerns of potential users. In this paper, we present a study with 18 participants to explore several ethical dilemmas and challenges in human-AI co-creation from the perspective of potential users using a design fiction (DF) methodology. DF is a speculative research method that depicts a new concept or technology through stories as an intangible prototype. We present the findings from the study as potential users’ perspectives, stances, and expectations around ethical challenges in human-AI co-creativity as a basis for designing human-centered ethical AI partners for human-AI co-creation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3591196.3593364

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Robertson et al. found that autocomplete suggestions could threaten users' autonomy.

    “Robertson et al. [106] found that autocomplete suggestions could threaten users' autonomy.”

    From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [106]

    Section 5.3.1 reports that participants disliked AI taking control of a creative process, but it does not discuss autocomplete suggestions or report a finding by Robertson et al. Searches for “Robertson,” “autocomplete,” and autonomy-related terms found no passage addressing the attributed finding. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 38 · Cited by 60

Research works printed but not linked: 49

  1. Theo Araujo, Natali Helberger, Sanne Kruikemeier, and Claes H De Vreese. 2020. In AI we trust? Perceptions about automated decision-making by artificial intelligence. AI & SOCIETY 35, 3 (2020), 611-623.
  2. Gary Bente, Sabine Rüggenberg, and Nicole C Krämer. 2004. Social presence and interpersonal trust in avatar-based, collaborative net-communications. In Proceedings of the Seventh Annual International Workshop on Presence. 54-61.
  3. Mark Blythe. 2014. Research through design fiction: narrative in real and imag- inary abstracts. In Proceedings of the SIGCHI conference on human factors in computing systems. 703-712.
  4. Michał Boni. 2021. The ethical dimension of human-artificial intelligence collab- oration. European View 20, 2 (2021), 182-190.
  5. Barry Brown, Julian Bleecker, Marco D'adamo, Pedro Ferreira, Joakim Formo, Mareike Glöss, Maria Holm, Kristina Höök, Eva-Carin Banka Johnson, Emil Kaburuan, et al. 2016. The IKEA Catalogue: Design fiction in academic and industrial collaborations. In Proceedings of the 19th International Conference on Supporting Group Work. 335-344.
  6. Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. arXiv preprint arXiv:2104.00358 (2021). · arXiv
  7. Amit K Chopra and Munindar P Singh. 2018. Sociotechnical systems and ethics in the large. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 48-53.
  8. James A Crowder, John Carbone, and Shelli Friess. 2020. Human-AI Collaboration. In Artificial Psychology. Springer, 35-50.
  9. Nicholas Davis. 2013. Human-computer co-creativity: Blending human and computational creativity. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 9.
  10. Nicholas Davis, Chih-Pin Hsiao, Yanna Popova, and Brian Magerko. 2015. An enactive model of creativity for computational collaboration and co-creation. In Creativity in the Digital Age. Springer, 109-133.
  11. Nicholas Davis, Chih-PIn Hsiao, Kunwar Yashraj Singh, Lisa Li, and Brian Magerko. 2016. Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent. In Proceedings of the 21st International Conference on Intelligent User Interfaces. 196-207.
  12. Arne Dietrich. 2004. The cognitive neuroscience of creativity. Psychonomic bulletin & review 11, 6 (2004), 1011-1026.
  13. Paul Dourish and Genevieve Bell. 2014. "Resistance is futile": reading science fiction alongside ubiquitous computing. Personal and Ubiquitous Computing 18, 4 (2014), 769-778.
  14. Anthony Dunne and Fiona Raby. 2013. Speculative everything: design, fiction, and social dreaming. MIT press.
  15. Christopher Flathmann, Beau G Schelble, Rui Zhang, and Nathan J McNeese. 2021. Modeling and Guiding the Creation of Ethical Human-AI Teams. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 469-479.
  16. Niklas Gudowsky and Aaron Rosa. 2019. Bridging epistemologies-identifying uniqueness of lay and expert knowledge for agenda setting. Futures 109 (2019), 24-38.
  17. Guy Hoffman and Gil Weinberg. 2010. Shimon: an interactive improvisational robotic marimba player. In CHI'10 Extended Abstracts on Human Factors in Computing Systems. 3097-3102.
  18. Guy Hoffman and Gil Weinberg. 2011. Interactive improvisation with a robotic marimba player. Autonomous Robots 31, 2-3 (2011), 133-153.
  19. Stephanie Houde and Charles Hill. 1997. What do prototypes prototype? In Handbook of human-computer interaction. Elsevier, 367-381.
  20. Kyle E Jennings, Dean Keith Simonton, and Stephen E Palmer. 2011. Under- standing exploratory creativity in a visual domain. In Proceedings of the 8th ACM conference on Creativity and cognition. 223-232.
  21. Rex Eugene Jung, Brittany S Mead, Jessica Carrasco, and Ranee A Flores. 2013. The structure of creative cognition in the human brain. Frontiers in human neuroscience 7 (2013), 330.
  22. Anna Kantosalo and Hannu Toivonen. 2016. Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In Proceedings of the seventh international conference on computational creativity. 77-84.
  23. Pegah Karimi, Kazjon Grace, Mary Lou Maher, and Nicholas Davis. 2018. Evaluating creativity in computational co-creative systems. arXiv preprint arXiv:1807.09886 (2018). · arXiv
  24. Pegah Karimi, Jeba Rezwana, Safat Siddiqui, Mary Lou Maher, and Nasrin Dehbo- zorgi. 2020. Creative sketching partner: an analysis of human-AI co-creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 221-230.
  25. Tom Kelley. 2001. Prototyping is the shorthand of innovation. Design Management Journal (Former Series) 12, 3 (2001), 35-42.
  26. Q Vera Liao, Matthew Davis, Werner Geyer, Michael Muller, and N Sadat Shami. 2016. What can you do? Studying social-agent orientation and agent proactive interactions with an agent for employees. In Proceedings of the 2016 acm conference on designing interactive systems. 264-275.
  27. Joseph Lindley and Robert Potts. 2014. A machine learning: an example of HCI prototyping with design fiction. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. 1081-1084.
  28. Maria Teresa Llano and Jon McCormack. 2020. Existential Risks of Co-Creative Systems. In Workshop on the Future of Co-creative Systems 2020. Association for Computational Creativity (ACC).
  29. Duri Long, Mikhail Jacob, Nicholas Davis, and Brian Magerko. 2017. Designing for socially interactive systems. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. 39-50.
  30. Ezio Manzini. 2015. Design, when everybody designs: An introduction to design for social innovation. MIT press.
  31. Thomas Markussen and Eva Knutz. 2013. The Poetics of Design Fiction. In Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces (Newcastle upon Tyne, United Kingdom) (DPPI '13). Association for Computing Machinery, New York, NY, USA, 231-240. https://doi.org/10. 1145/2513506.2513531 · DOI
  32. Thomas Memmel, Fredrik Gundelsweiler, and Harald Reiterer. 2007. Agile human- centered software engineering. In BCS-HCI'07: 21st British HCI Group Annual Conference on People and Computers. 167-175.
  33. Masahiro Mori, Karl F MacDorman, and Norri Kageki. 2012. The uncanny valley [from the field]. IEEE Robotics & automation magazine 19, 2 (2012), 98-100.
  34. Larissa Vivian Nägele, Merja Ryöppy, and Danielle Wilde. 2018. PDFi: partic- ipatory design fiction with vulnerable users. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction. 819-831.
  35. SANTIAGO NEGRETE YANKELEVICH and NORA ANGELICA MORALES ZARAGOZA. 2014. The apprentice framework: planning and assessing creativity. Fifth International Conference on Computational Creativity.
  36. Hao Nie, Xianpei Han, Ben He, Le Sun, Bo Chen, Wei Zhang, Suhui Wu, and Hao Kong. 2019. Deep sequence-to-sequence entity matching for heterogeneous entity resolution. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 629-638.
  37. Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.
  38. Sarvapali D Ramchurn, Sebastian Stein, and Nicholas R Jennings. 2021. Trust- worthy human-AI partnerships. Iscience 24, 8 (2021), 102891.
  39. Jeba Rezwana and Mary Lou Maher. 2022. Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems. ACM Transactions on Computer-Human Interaction (2022).
  40. Jeba Rezwana and Mary Lou Maher. 2022. Identifying Ethical Issues in AI Partners in Human-AI Co-Creation. arXiv preprint arXiv:2204.07644 (2022). · arXiv
  41. Stuart Russell, Sabine Hauert, Russ Altman, and Manuela Veloso. 2015. Ethics of artificial intelligence. Nature 521, 7553 (2015), 415-416.
  42. E B-N Sanders. 2000. Generative tools for co-designing. In Collaborative design: Proceedings of codesigning 2000. Springer, 3-12.
  43. Keng Siau and Weiyu Wang. 2020. Artificial intelligence (AI) ethics: ethics of AI and ethical AI. Journal of Database Management (JDM) 31, 2 (2020), 74-87.
  44. Theresa Jean Tanenbaum, Marcel Pufal, and Karen Tanenbaum. 2016. The limits of our imagination: design fiction as a strategy for engaging with dystopian futures. In Proceedings of the Second Workshop on Computing within Limits. 1-9.
  45. Neil Thurman, Judith Moeller, Natali Helberger, and Damian Trilling. 2019. My friends, editors, algorithms, and I: Examining audience attitudes to news selection. Digital journalism 7, 4 (2019), 447-469.
  46. Suzanne Tolmeijer, Markus Christen, Serhiy Kandul, Markus Kneer, and Abra- ham Bernstein. 2022. Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making. In CHI Conference on Human Factors in Computing Systems. 1-17.
  47. Wendell Wallach and Colin Allen. 2008. Moral machines: Teaching robots right from wrong. Oxford University Press.
  48. Dakuo Wang, Pattie Maes, Xiangshi Ren, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2021. Designing AI to Work with or for People?. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1-5.
  49. Wei Xu. 2019. Toward human-centered AI: a perspective from human-computer interaction. interactions 26, 4 (2019), 42-46.

Authors
Rezwana, J., & Maher, M. L.
Affiliations
University of North Carolina at Charlotte (US)
ORCID
Venue
Creativity and Cognition
Publisher
ACM
Published
2023-06-19
Pages
62-74
DOI
10.1145/3591196.3593364
OpenAlex
W4381104161
Cited by
60
References
38
Citations by year
2026: 23 · 2025: 24 · 2024: 13
FWCI
15.03
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
Creativity and Cognition (publishedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
Keywords
human-AI co-creation, ethical considerations, user perspective, design fiction, human-centered AI, ethical AI, human-AI interaction, user expectations, ethical dilemmas

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review