Jess+: AI and robotics with inclusive music-making
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
This paper discusses the findings from a cross-sector research project investigating how a digital score created using AI and robotics might stimulate new creative opportunities and relationships within the practices of an inclusive music ensemble. Through the concept of a digital score [65], AI and a robotic arm were introduced into an ensemble’s musical practice to evaluate the impact and benefits of using autonomous systems to challenge barriers around a disabled musician’s access to creative music-making. Throughout the development process we placed an emphasis on involvement and togetherness of not only the AI and robots’ contribution to shared creativity amongst the ensemble, but also to the social aspects of the creative process across the team of musicians, developers, researchers and supporting organisations. The findings were surprising with many aspects of the project exceeding the expectations of the original aims. In short, all the musicians benefited from the introduction of these unfamiliar technologies with practices enhanced and relationships transformed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The manifestation of AI agency can vary significantly, ranging from a passive resource provider to an active co-constructor [177], contingent upon the system's capabilities and the predefined collaborative workflow [206].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [177]
Citations
Cites
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Sloane-Seale, A. (2009) · Canadian Journal of University Continuing Education · doi:10.21225/d54s3d · Cited by 14,328
(2014) · Medical Entomology and Zoology · Cited by 6,828
Musicking: the meanings of performing and listening
(1999) · Choice Reviews Online · doi:10.5860/choice.36-3838 · Cited by 1,526
Ambiguity as a resource for design
Gaver, W. W., Beaver, J., & Benford, S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642653 · Cited by 1,474
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Proceedings of the International Conference on New Interfaces for Musical Expression
Jensenius, A. R., Skogstad, S. A. V. D., Nymoen, K., Tveit, A., Godøy, R. I., & Overholt, D. (2011) · NORA - Norwegian Open Research Archives · Cited by 445
Allen, J. E., Guinn, C. I., & Horvtz, E. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.796083 · Cited by 442 · cited 1×
“One prominent topic in this feld concerns human-machine collaboration, which foregrounds a broad range of concerns such as the negotiation of roles, autonomy and control [1, 5, 32], co-creativity [15, 37, 66], sense making [34, 52], and transparency, accountability and trust [4, 59] amongst others.”
BackgroundCreating creativity: user interfaces for supporting innovation
Shneiderman, B. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/344949.345077 · Cited by 432
Next Steps for Human-Computer Integration
Mueller, F. F., Lopes, P., Strohmeier, P., Ju, W., Seim, C., Weigel, M., Nanayakkara, S., Obrist, M., Li, Z., Delfa, J., et al. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376242 · Cited by 260 · cited 1×
“For example, Mueller et al. [47] foreground Human-Computer Integration distinct from matters of interaction, with a focus on tight integration between technologies situated on, or in the human-body.”
BackgroundAccessible Digital Musical Instruments—A Review of Musical Interfaces in Inclusive Music Practice
Frid, E. (2019) · Multimodal Technologies and Interaction · doi:10.3390/mti3030057 · Cited by 177
The Design of Implicit Interactions: Making Interactive Systems Less Obnoxious
Ju, W., & Leifer, L. (2008) · Design Issues · doi:10.1162/desi.2008.24.3.72 · Cited by 177 · cited 1×
“One prominent topic in this feld concerns human-machine collaboration, which foregrounds a broad range of concerns such as the negotiation of roles, autonomy and control [1, 5, 32], co-creativity [15, 37, 66], sense making [34, 52], and transparency, accountability and trust [4, 59] amongst others.”
BackgroundProceedings of the 2004 conference on New interfaces for musical expression
Nagashima, Y., & Lyons, M. J. (2004) · Cited by 146
Gottschalk, J. (2016) · Bloomsbury Academic eBooks · doi:10.5040/9781501396328 · Cited by 136
Thinking in Jazz: The Infinite Art of Improvisation
Spring, H., & Berliner, P. F. (1995) · Notes · doi:10.2307/899035 · Cited by 126
van der Schyff, D., Schiavio, A., Walton, A., Velardo, V., & Chemero, A. (2018) · Music & Science · doi:10.1177/2059204318792319 · Cited by 119
Emergence as a Feature of Practice-based Design Research
Gaver, W., Krogh, P. G., Boucher, A., & Chatting, D. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533524 · Cited by 107
Dimensions of Musical Creativity
Schiavio, A., & Benedek, M. (2020) · Frontiers in Neuroscience · doi:10.3389/fnins.2020.578932 · Cited by 105 · cited 1×
“Here, we also think that musicians’ creativity in the ensemble was helped by the social dimension and their previous playing relationships [57].”
BackgroundDesign Considerations for Real-Time Collaboration with Creative Artificial Intelligence
McCormack, J., Hutchings, P., Gifford, T., Yee-King, M., Llano, M. T., & D’inverno, M. (2020) · Organised Sound · doi:10.1017/s1355771819000451 · Cited by 101 · cited 1×
“Additionally, in Design Considerations for Real-Time Collaboration with Creative Artifcial Intelligence , Mc-Cormack [44] ofers a framework for maximizing the human–AI creative interaction, which can be migrated to human–robotic mu-sicking.”
MethodMusical friends and foes: The social cognition of affiliation and control in improvised interactions
Aucouturier, J., & Canonne, C. (2017) · Cognition · doi:10.1016/j.cognition.2017.01.019 · Cited by 94
Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Kaur, H., Adar, E., Gilbert, E., & Lampe, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533135 · Cited by 92
A survey of robotic musicianship
Bretan, M., & Weinberg, G. (2016) · Communications of the ACM · doi:10.1145/2818994 · Cited by 78 · cited 1×
“For example, research in computational creativity [42], AI and music [45], and robotic musicianship [9, 35, 68].”
BackgroundProceedings of the International Conference on New Interfaces for Musical Expression
Queiroz, M., & Xambó, A. (2019) · DMU Open Research Archive (De Montfort University) · Cited by 71
Correa, A. G. D., Ficheman, I. K., do Nascimento, M., & de Deus Lopes, R. (2009) · 2009 Ninth IEEE International Conference on Advanced Learning Technologies · doi:10.1109/icalt.2009.111 · Cited by 70 · cited 1×
“Examples are numerous engaging with a range of technologies, settings and diverse abilities, such as for music therapy [12, 31] or education [20] or multi-sensory sonic spaces [26].”
BackgroundBrain-Computer Music Interfacing (BCMI): From Basic Research to the Real World of Special Needs
Miranda, E. R., Magee, W. L., Wilson, J. J., Eaton, J., & Palaniappan, R. (2011) · Music and Medicine · doi:10.1177/1943862111399290 · Cited by 69
In a Silent Way: Communication Between AI and Improvising Musicians Beyond Sound
McCormack, J., Gifford, T., Hutchings, P., Llano Rodriguez, M. T., Yee-King, M., & d'Inverno, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300268 · Cited by 64
Affective brain–computer music interfacing
Daly, I., Williams, D., Kirke, A., Weaver, J., Malik, A., Hwang, F., Miranda, E., & Nasuto, S. J. (2016) · Journal of Neural Engineering · doi:10.1088/1741-2560/13/4/046022 · Cited by 59
Benford, S., Ramchurn, R., Marshall, J., Wilson, M. L., Pike, M., Martindale, S., Hazzard, A., Greenhalgh, C., Kallionpää, M., Tennent, P., et al. (2020) · Human-Computer Interaction · doi:10.1080/07370024.2020.1754214 · Cited by 49
Self-citationProceedings of the Seventh International Conference on Computational Creativity
Llano, M. T., Christian Guckelsberger, B., Hepworth, R., Gow, J., Corneli, J., & Colton, S. (2016) · Cited by 46
Perception of ‘Back-Channeling’ Nonverbal Feedback in Musical Duo Improvisation
Moran, N., Hadley, L. V., Bader, M., & Keller, P. E. (2015) · PLoS ONE · doi:10.1371/journal.pone.0130070 · Cited by 44
Message Ritual: A Posthuman Account of Living with Lamp
Rajcic, N., & McCormack, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581363 · Cited by 43 · cited 2×
“…specifcally that "ambiguity of relationship creates the condition for a deeply personal projection of imagination and values onto a design" [23], but also Rajic and McCormick’s observation of the importance to highlight the “the open nature of interpretation of machine generated” outputs [53].”
BackgroundResult#Scanners: Exploring the Control of Adaptive Films using Brain-Computer Interaction
Pike, M., Ramchurn, R., Benford, S., & Wilson, M. L. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858276 · Cited by 41
Supporting Rhythm Activities of Deaf Children using Music-Sensory-Substitution Systems
Petry, B., Illandara, T., Elvitigala, D. S., & Nanayakkara, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174060 · Cited by 41 · cited 1×
“Other work focuses on reimagining existing instruments, for example guitars for neurodiversity [28] or physical disability [27], some of which are focused on bespoke designs for specifc individuals [38] and hearing impairment [51].”
BackgroundMusical Interaction Reveals Music as Embodied Language
Dell’Anna, A., Leman, M., & Berti, A. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.667838 · Cited by 36
Santos, R. G., & Santos, R. B. (2015) · Evaluation Review · doi:10.1177/0193841x15602818 · Cited by 34
Musical Sounds, Motor Resonance, and Detectable Agency
Launay, J. (2015) · Empirical Musicology Review · doi:10.18061/emr.v10i1-2.4579 · Cited by 34 · cited 1×
“…audiences recognize social relationships and communicative behavior be-tween members of a performing ensemble [2, 46], and they infer human agency when hearing music–even without visual confrmation of a performer [36, 49], making sounded performances a means of interpersonal communication” [7].”
BackgroundGreenfield, G. R. (2013) · Journal of Mathematics and the Arts · doi:10.1080/17513472.2013.795882 · Cited by 24
Failing with Style: Designing for Aesthetic Failure in Interactive Performance
Hazzard, A., Greenhalgh, C., Kallionpää, M., Benford, S., Veinberg, A., Kanga, Z., & McPherson, A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300260 · Cited by 23
Self-citationAccessible Instruments in the Wild: Engaging with a Community of Learning-Disabled Musicians
Harrison, J., Chamberlain, A., & McPherson, A. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3313037 · Cited by 23
Playing Fast and Loose with Music Recognition
Greenhalgh, C., Benford, S., Hazzard, A., & Chamberlain, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025900 · Cited by 22 · cited 2×
“Greenhalgh et al. [25] proposes the concept of ‘looseness’ as a design consideration for human-autonomous machine interaction (in their case a music recognition system), where a tightly constrained system in turn constrains a performer’s ability to be expressive, whereas a ‘loose’ system in turn…”
BackgroundSelf-citationCreative AI and Musicking Robots
Vear, C. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.631752 · Cited by 20 · cited 3×
“On this matter, Vear [66] wrote: “Embodiment (in music) is the process in musicking of drawing the musician’s sound into their bodily sense of being.”
BackgroundSelf-citationPerformance without Barriers: Improvising with Inclusive and Accessible Digital Musical Instruments
Samuels, K., & Schroeder, F. (2019) · Contemporary Music Review · doi:10.1080/07494467.2019.1684061 · Cited by 17 · cited 1×
“We focused on improvisation as the vehicle of music making – something that has happened frequently in inclusive design research (see [55]), because improvisation presented in this way is a "co-created activity that endeavours continuously to question exclusionary identities and assumptions.””
BackgroundLoopBlocks: Design and Preliminary Evaluation of an Accessible Tangible Musical Step Sequencer
Förster, A., & Komesker, M. (2021) · New Interfaces for Musical Expression · doi:10.21428/92fbeb44.f45e1caf · Cited by 15
Robotic Musicianship - Musical Interactions Between Humans and Machines
Weinberg, G. (2007) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.5772/5206 · Cited by 12 · cited 1×
“For example, research in computational creativity [42], AI and music [45], and robotic musicianship [9, 35, 68].”
BackgroundFive Provocations for a More Creative TAS
Benford, S., Hazzard, A., Vear, C., Webb, H., Chamberlain, A., Greenhalgh, C., Ramchurn, R., & Marshall, J. (2023) · International Symposium on Trustworthy Autonomous Systems (TAS) · doi:10.1145/3597512.3599709 · Cited by 12
Self-citationLocating Creativity in Differing Approaches to Musical Robotics
Kemper, S. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.647028 · Cited by 10
Music generation based on emotional EEG
Luo, G., Chen, H., Li, Z., & Wang, M. (2022) · International Conference on Innovation in Artificial Intelligence (ICIAI) · doi:10.1145/3529466.3529492 · Cited by 10
The Digital Score: Musicianship, Creativity and Innovation
Feller, R. (2019) · Computer Music Journal · doi:10.1162/comj_r_00526 · Cited by 5
“We also like to be surprised”: Disruption, provocation and surprise in the music of Christian Wolff
Thomas, P., & Payne, E. (2020) · Circuit Musiques contemporaines · doi:10.7202/1071119ar · Cited by 3
Research works printed but not linked: 17
- Avron Barr, Edward A Feigenbaum, and Paul R Cohen. 1981. The handbook of artificial intelligence. Vol. 3. HeurisTech Press.
- Laura Bishop. 2018. Collaborative musical creativity: How ensembles coordinate spontaneity. Frontiers in psychology 9 (2018), 1285.
- Rodney A Brooks. 1991. Intelligence without representation. Artificial intelligence 47, 1-3 (1991), 139–159.
- Linda Candy, Ernest Edmonds, and Craig Vear. 2021. Practice-based research. In The Routledge International Handbook of Practice-Based Research. Routledge, 27–41.
- Kerstin Dautenhahn. 2007. Socially intelligent robots: dimensions of human–robot interaction. Philosophical transactions of the royal society B: Biological sciences 362, 1480 (2007), 679–704.
- Arne Dietrich. 2004. The cognitive neuroscience of creativity. Psychonomic bulletin & review 11 (2004), 1011–1026.
- Simone Gumtau, Paul Newland, Chris Creed, and Simon Kunath. 2005. MEDIATE–a responsive environment designed for children with autism. In Accessible Design in the Digital World Conference 2005. 1–8.
- Jacob Harrison. 2020. Supplementary Material for thesis ‘Instruments and access: The role of instruments in music and disability’. Ph. D. Dissertation. Queen Mary University of London.
- Todd Lubart, Dario Esposito, Alla Gubenko, and Claude Houssemand. 2021. Creativity in humans, robots, humbots. Creativity: Theories–Research–Applications 8, 1 (2021).
- Bahar Uddin Mahmud, Guan Yue Hong, and Bernard Fong. 2022. A Study of Human-AI Symbiosis for Creative Work: Recent Developments and Future Directions in Deep Learning. ACM Transactions on Multimidia Computing Communications and Applications (2022).
- P Russel Norvig and S Artificial Intelligence. 2002. A modern approach. Prentice Hall Upper Saddle River, NJ, USA: Rani, M., Nayak, R., & Vyas, OP (2015). An ontology-based adaptive personalized e-learning system, assisted by software agents on cloud storage. Knowledge-Based Systems 90 (2002), 33–48.
- Kirk N Olsen and Roger T Dean. 2016. Does perceived exertion influence perceived affect in response to music? Investigating the “FEELA” hypothesis.Psychomusicology: Music, Mind, and Brain 26, 3 (2016), 257.
- Jeba Rezwana and Mary Lou Maher. 2022. Designing creative AI partners with COFI: A framework for modeling interaction in human-AI co-creative systems. ACM Transactions on Computer-Human Interaction (2022).
- R Keith Sawyer. 1999. The emergence of creativity. Philosophical psychology 12, 4 (1999), 447–469.
- Tim Swingler. 1998. The invisible keyboard in the air: An overview of the educational, therapeutic and creative applications of the EMS Soundbeam. In 2nd European Conference for Disability, Virtual Reality & Associated Technology.
- Greg Turner, Alastair Weakley, Yun Zhang, and Ernest A Edmonds. 2005. Attuning: A Social and Technical Study of Artist-Programmer Collaborations.. In PPIG. 10.
- Aaron Williamon, Sam Thompson, Tânia Lisboa, and Charles Wiffen. 2006. Creativity, originality, and value in music performance. In Musical Creativity. Psychology Press, 177–196.
Cited by
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Togetherness in musical interaction
Bishop, L. (2024) · Routledge Open Research · doi:10.12688/routledgeopenres.18202.1 · Cited by 21
Morita, R., Watanabe, K., Zhou, J., Dengel, A. R., & Ishimaru, S. (2025) · Augmented Humans International Conference (AHs) · doi:10.1145/3745900.3746066 · Cited by 17
Towards Responsible AI Music: an Investigation of Trustworthy Features for Creative SystemsPreprint
de Berardinis, J., Porcaro, L., Meroño-Peñuela, A., Cangelosi, A., & Buckley, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18814 · Cited by 17
Morita, R., Watanabe, K., Zhou, J., Dengel, A., & Ishimaru, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.07463 · Cited by 13
Metadata
- Authors
- Vear, C., Hazzard, A., Moroz, S., & Benerradi, J.
- Affiliations
- University of Nottingham (GB)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-17
- OpenAlex
- W4396833450
- Semantic Scholar
- de6fe0b9dd78ba59b41c8e9a86364fad88bc2335
- Cited by
- 11
- References
- 48
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 2
- FWCI
- 2.28
- Type
- conference paper
- Language
- en
- Open access
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Repository@Nottingham (University of Nottingham) (submittedVersion)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
- Funders
- Trustworthy Autonomous Systems Network
- European Research Council (ERC-2020-COG ? 101002086)
- Topics
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Social Sciences › Education › Music Education and Analysis
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Art, Technology, and Culture
- Keywords
- inclusive music making, digital scores, AI in music, robot manipulators, ensemble performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.