Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions
The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Abstract
The rapid development and availability of low-cost technologies have created a wide interest in virtual reality. In the field of computer music, the term “virtual musical instruments” has been used for a long time to describe software simulations, extensions of existing musical instruments, and ways to control them with new interfaces for musical expression. Virtual reality musical instruments (VRMIs) that include a simulated visual component delivered via a head-mounted display or other forms of immersive visualization have not yet received much attention. In this article, we present a field overview of VRMIs from the viewpoint of the performer. We propose nine design guidelines, describe evaluation methods, analyze case studies, and consider future challenges.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Philosophical conceptions of the self: implications for cognitive science.
Gallagher, S. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01417-5 · Cited by 2,881 · cited 3×
“The resulting sensation is called virtual body ownership, and several studies have shown that it is possible in VR to generate perceptual illusions of ownership over a virtual body seen from the first-person perspective, in other words, of a body that visually substitutes the person’s real body (Gallagher 2000). Kilte…”
BackgroundInfluentialPlace illusion and plausibility can lead to realistic behaviour in immersive virtual environments
Slater, M. (2009) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2009.0138 · Cited by 2,691 · cited 5×
“Particularly, Slater (2009) describes sensory motor contingencies as the set of actions a user will know how to perform in order to perceive something (e.g., moving one’s head and eyes to change gaze direction, kneeling to get a closer look at something on the ground, or turning one’s head to…”
BackgroundInfluentialSurround-screen projection-based virtual reality: the design and implementation of the CAVE
Cruz-Neira, C., Sandin, D. J., & DeFanti, T. A. (1993) · Proceedings of the 20th annual conference on Computer graphics and interactive techniques · doi:10.1145/166117.166134 · Cited by 2,327
3D User Interfaces: Theory and Practice
Bowman, D. A., Kruijff, E., LaViola, J. J., & Poupyrev, I. (2004) · Cited by 2,016
A discussion of cybersickness in virtual environments
LaViola, J. J. (2000) · ACM SIGCHI Bulletin · doi:10.1145/333329.333344 · Cited by 1,426 · cited 1×
“More generally, it is crucial to reduce system latency, since it is believed to increase cybersickness (LaViola 2000), the topic of the next principle.”
BackgroundNowak, K. L., & Biocca, F. (2003) · Presence: Teleoperators and Virtual Environments · doi:10.1162/105474603322761289 · Cited by 1,141
Sutherland, I. E. (1965) · Cited by 1,073
When second wave HCI meets third wave challenges
Bødker, S. (2006) · Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles · doi:10.1145/1182475.1182476 · Cited by 917 · cited 1×
“This layer focuses on higher design goals, embracing experience and “meaning-making” (Bødker 2006).”
BackgroundThe go-go interaction technique: non-linear mapping for direct manipulation in VR
Poupyrev, I., Billinghurst, M., Weghorst, S., & Ichikawa, T. (1996) · Proceedings of the 9th annual ACM symposium on User interface software and technology - UIST '96 · doi:10.1145/237091.237102 · Cited by 872
3-D sound for virtual reality and multimedia
Madole, D., & Begault, D. (1995) · Computer Music Journal · doi:10.2307/3680997 · Cited by 868 · cited 1×
“For virtual reality, 3-D sound is an especially relevant issue, because the location and motion of the visual virtual objects need to match the location and motion of the auditory objects (Begault 1994).”
Background
Research works printed but not linked: 21
- Berthaut F., 2014, IEEE VR Workshop: Sonic Interaction in Virtual Environments, 19
- Cook P., 2001, Proceedings of the International Conference on New Interfaces for Musical Expression, 3
- Cook P. R., 2009, Proceedings of the International Conference on New Interfaces for Musical Expression, 9, 218
- Fernandes A. S., 2016, IEEE Symposium on 3D User Interfaces, 201
- Fillwalk J., 2015, IEEE Symposium on 3D User Interfaces, 201
- Hamilton R., 2009, Proceedings of the AES 35th International Conference: Audio for Games, 1
- Heilbrun A., 1989, Whole Earth Review (64):108-119.
- Johnston A., 2011, Proceedings of the International Conference on New Interfaces for Musical Expression, 280
- Kohlrausch A., 1999, International Society for Optics and Photonics (SPIE) Proceedings on Human Vision and Electronic Imaging, 34
- Maki-Patola T., 2004, Proceedings of the International Conference on Digital Audio Effects, 11
- Mäki-Patola T., 2005, Proceedings of the International Conference on New Interfaces for Musical Expression, 11
- Marshall M. T., 2011, Proceedings of the International Conference on New Interfaces for Musical Expression, 399
- Moore A. G., 2015, Proceedings of the IEEE Symposium on 3D User Interfaces, 205
- Morreale F., 2014, Proceedings of the International Conference on New Interfaces for Musical Expression, 467
- Poupyrev I., 1996, Proceedings of the ACM Symposium on User Interface Software and Technology, 79
- Rodet X., 2005, Proceedings of the International Conference on New Interfaces for Musical Expression, 109
- Serafin S., 2006, Proceedings of the International Conference on New Interfaces for Musical Expression, 240
- Serafin S., 2015, IEEE Symposium on 3D User Interfaces, 207
- Wang G., 2014, Proceedings of the Joint International Computer Music Conference and the Sound and Music Computing Conference, 391
- Xiao X., 2014, Proceedings of the International Conference on New Interfaces for Musical Expression, 629
- Zielasko D., 2015, IEEE Symposium on 3D User Interfaces, 209
Cited by
Items 101–143 of 143
Deploying Virtual Air Drum Tool to Detect Sound Frequency in Computer Vision
Valarmathi, N., Anbarasu, M., Ragul, E., Kumar, M. S., & Udayakumar, S. (2022) · International Conference on Applied Artificial Intelligence and Computing (ICAAIC) · doi:10.1109/icaaic53929.2022.9793172 · Cited by 1
Maneuvrier, A. (2020) · theses.fr (ABES) · doi:10.70675/3fbb4dbfzc553z459dzba8fz7ee7a2f5ec28 · Cited by 7
Perspectives in Education for Sound and Music Computing
Avanzini, F., Baratè, A., Haus, G. M., Ludovico, L. A., Ntalampiras, S., & Presti, G. (2018)
Avaliação qualitativa da engenharia de sistemas imersivos para edutenimento.
Rodrigues, F. (2018) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.3.2018.tde-03052018-102736
Playing Virtual Musical Drums by MEMS 3D Accelerometer Sensor Data and Machine Learning
Hossain, S. F., Hirose, K., Kanaya, S., & Altaf-Ul-Amin, M. (2020) · doi:10.5121/csit.2020.100306
On Digitizing the Greek Music Tradition: Designing the Cretan Lute for Mobile Devices
Tsotakos, G., Margounakis, D. G., Pachidis, T. P., & Politis, D. V. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-96296-8_36
Waveform Design for 5G and beyond Systems
Choi, K. (2022) · MDPI eBooks · doi:10.3390/books978-3-0365-3174-8
Perception of Guitar Strings on a Flat Visuo-Haptic Display
Rohou--Claquin, B., Auvray, M., Carrou, J. L., & Gueorguiev, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15019-7_13
“However, despite these advantages [16, 6], DMIs currently suffer from a large drawback compared to real instruments, which is the absence of haptic feedback.”
Image Virtual Viewpoint Generation Method under Hole Pixel Information Update
Leng, L., Gao, C., Zhang, F., Li, D., Zhang, W., Gao, T., Zeng, Z., Tang, L., Luo, Q., & Duan, Y. (2022) · Symmetry · doi:10.3390/sym15010034
Phenomenological approach on the role of sonic heritage as an intangible heritage
Lee, H. (2022) · Europub Journal of Social Sciences Research · doi:10.54746/ejssrv3n1-008
“For example, Serafin et al. (2016) introduced Virtual Reality Musical Instruments (VRMIs) that include a simulated visual component delivered via a head-mounted display or other forms of immersive visualization.”
BackgroundMusic is in the air. Sounding performances in hybrid and virtual space
Akkermann, M. (2023) · Array · doi:10.25370/array.v20223484
Playing the Malay Gamelan Bonang in the Air: A Pilot Study
Saffian, K. A., Norowi, N. M., Abdullah, L. N., Sulaiman, P. S., & Musib, A. F. (2023) · Asian HCI Symposium · doi:10.1145/3604571.3604577
“Thus, the subsequent experiment will need to consider some design and evaluation aspects for virtual reality musical instruments [11].”
BackgroundRhythmic Accuracy of 3D Spatial Interaction for Digital Music
Arterbury, T., Poor, G. M., & Terron, A. P. A. (2023) · Audio Mostly Conference (AM) · doi:10.1145/3616195.3616216
“Most existing examples are based on Virtual Reality (VR) technology, sometimes called VRMIs [26, 29, 31, 43].”
USE OF IMMERSIVE TECHNOLOGIES IN MODERN CONDUCTING ART
Качуринець, Л. В. (2023) · Humanities science current issues · doi:10.24919/2308-4863/66-1-14
Experimentálne prístupy k práci so zvukom vo virtuálnej realite
Cihanská, A. M. (2021) · ArteActa · doi:10.62804/aa.2021.002
A Virtual Reality Musical Instrument Integrated with a Remote Playing Robot System
Zhu, Z., Nai, W., Wang, X., & Liu, Y. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw62533.2024.00185
The Impact of Music and Metaverse on Education: Results of a Scoping Review
Paolo, A. D., Todino, M. D., & Sibilio, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57402-3_7 · Cited by 1
Facin, D. S., Corrêa, C. V., & Filho, F. D. S. A. (2025) · doi:10.54750/euroed.978-65-01-44078-1_2
Critical perspectives of society
Catapan, B. L. S. B. (2025) · doi:10.54750/euroed.978-65-01-44078-1
Evaluating Presence in Immersive Virtual Reality Concert Experiences
Clarke, J., & Nuanáin, C. Ó. (2025) · Immersive and 3D Audio: from Architecture to Automotive (I3DA) · doi:10.1109/i3da65421.2025.11202050
“The advent of virtual reality (VR) has significantly expanded the ways in which people experience all forms of media, including film - or so-called “Cinematic Virtual Reality” (CVR) - [1], [2], interactive games [3]–[5] (particularly serious games [6], [7]) and music [8]–[10].”
Perspectives on Practical Implementations in the Web-based Musical Metaverse
Dziwis, D. T., Boem, A., Nowakowski, M., & Turchet, L. (2025) · IEEE International Symposium on the Internet of Sounds (IS2) · doi:10.1109/is264627.2025.11284607
“Moreover, through hand tracking or input controllers user can directly manipulate such objects directly as they will do in their physical reality [45].”
Metaverse Music and Multimedia Storytelling
Tauber, S. (2026) · Springer eBooks · doi:10.1007/978-3-032-03296-6_13
RISCOS PROFISSIONAIS DO TRABALHO REMOTO NO PÓS PANDEMIA: UMA REVISÃO DE LITERATURA
Werle, T. S. D. S., dos Santos, R., dos Santos, F. L., Souza, M. O. D. S., Santos, M. I. S., Higa, G. J. D. O., Tonole, R., & Passos, J. P. (2025) · doi:10.54750/euroed.978-65-01-44078-1_3
Can We Touch the Real in Virtual Reality? An Empirical Study of Multisensory Interaction
Gupta, S., Pretty, E. J., Li, Z., Yang, X., Hamari, J., & Xi, N. (2026) · International GamiFIN Conference (GamiFIN) · doi:10.1145/3789624.3789644
“Musical instruments provide a particularly suitable context for this investigation, as prior work in musical haptics has shown that even simple instrumental actions rely on precise sensorimotor coordination, making deviations in tactile feedback readily perceptible [23, 26].”
Perception sonore des objets pour un contrôle intuitif de la synthèse : vers un langage des sons
Bourachot, A. (2021) · theses.fr (ABES) · doi:10.70675/638d9553z0fd3z482fzb122z8e61ac160c79 · Cited by 1
Problèmes liés aux jeux d’argent et usages excessifs liés aux écrans
Rémond, J.-J. (2021) · theses.fr (ABES) · doi:10.70675/2c180510zde5fz4b4az8bcaz631d1c92d177
Reynaert, V. (2021) · theses.fr (ABES) · doi:10.70675/120c9e5ez2e2cz4bffza1aez15826ebc2013
Orchestrating a software ecosystem for the creation, manipulation, and distribution of spatial audio
Carpentier, T. (2024) · theses.fr (ABES) · doi:10.70675/763081e4z94f5z45a7zb1e6z2562921a073b · Cited by 2
Looking inside the VR Music Scene: Mapping Platforms, Events and People
Ppali, S., Boem, A., Covaci, A., Constantinides, M., Liarokapis, F., & Turchet, L. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791497
“Research has focused on technical aims such as reducing latency, integrating physical instruments, or creating virtual instruments [8, 39, 80, 92, 93, 109].”
Blow based collaboration in a digital art virtual environment
Lomet, J., Gaugne, R., & Gouranton, V. (2026) · International Conference on Movement and Computing (MOCO) · doi:10.1145/3802842.3802861
“In addition to visual arts, breath, particularly blowing, has been extensively explored in the domain of new digital musical instruments (NIMEs), where airflow acts as an expressive input for sound synthesis [43].”
Empirical Analysis of 360° VR Documentary Production Pipeline
Nobaew, B., & Temdee, P. (2025) · Technology Innovation Management and Engineering Science International Conference (TIMES-iCON) · doi:10.1109/times-icon67125.2025.11488135
“Recent technical studies have examined specific aspects: Ozcinar et al. [6] investigated viewport-adaptive streaming for 360-degree video quality optimization, while research on video quality assessment [7], [8] has focused on perceptual quality rather than creation workflows.”
Charalambidou, C., & Mygdanis, Y. (2025) · International Journal of Music in Early Childhood · doi:10.1386/ijmec_00085_1
VRhythmSpace: Designing a Musical Game for Play, Learning, and Performance
Li, J., Jiang, Z., Chen, Y., Ren, Y., Li, J., Yao, K., Sun, X., Jia, B., & Xu, N. (2026) · International Conference on Educational and Information Technology · doi:10.1109/iceit68991.2026.11521823
“Due to the immersive nature of VR, spatial embodied interaction designs would ultimately make musical expression more intuitive and easy to access [16].”
Sound and Visual Objects (SVO) in N Dimensions in the Labyrinth of Time
de Gérando, S., & Baroin, G. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-27827-2_34
Rossi, C., Frisone, F., Corrò, C., Pagnoncelli, G., Sanna, M., CONFALONIERI, E. M., Oasi, O., Chittaro, L., & Riva, G. (2026) · The Arts in Psychotherapy · doi:10.1016/j.aip.2026.102505
Kulyabov, D. S., Fabbrocino, A., Franceschini, A., Dessì, P., Facchi, A., & Rodà, A. (2026) · Digital Applications in Archaeology and Cultural Heritage · doi:10.1016/j.daach.2026.e00588
Nobaew, B., & Aunsri, N. (2026) · IEEE Access · doi:10.1109/access.2026.3745392
Ayed, M. B., & Ayed, M. B. (2026) · Springer eBooks · doi:10.1007/978-3-032-29958-1_15
Metadata
- Authors
- Serafin, S., Erkut, C., Kojs, J., Nilsson, N. C., & Nordahl, R.
- Affiliations
- Aalborg University (DK)
- University of Miami (US)
- Venue
- Computer Music Journal
- Publisher
- MIT Press
- Published
- 2016-09
- Volume
- 40
- Issue
- 3
- Pages
- 22-40
- ISSN
- 0148-9267 (ISSN-L)
- 1531-5169
- OpenAlex
- W2563502399
- Cited by
- 139
- References
- 74
- Citations by year
- 2026: 11 · 2025: 13 · 2024: 14 · 2023: 17 · 2022: 25 · 2021: 16 · 2020: 15 · 2019: 10 · 2018: 13 · 2017: 4 · 2016: 1
- FWCI
- 6.34
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Computer Music Journal (publishedVersion)
- Project Muse (Johns Hopkins University) (submittedVersion)
- VBN Forskningsportal (Aalborg Universitet) (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Physical Sciences › Computer Science › Human-Computer Interaction › Virtual Reality Applications and Impacts
- Keywords
- virtual musical instruments, immersive visualization, head-mounted displays, musical instrument design, design guidelines, performer perspective, musical expression, assessment methods, computer music
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.