Smart Musical Instruments: Vision, Design Principles, and Future Directions

“For this purpose, one could employ deep reinforcement-learning techniques [75], where a musician can interactively update a model based on his/her personal behavior.”

“paradigm [63], [93], [96] considering IoMusT ecosystems.”

“paradigm [63], [93], [96] considering IoMusT ecosystems.”

“Design principles and considerations developed by the IoT research community include proposals for the characteristics and capabilities of the smart objects composing the IoT [18], [19], [50], on the services they can support [4], or on processes for industries [37].”

“paradigm [63], [93], [96] considering IoMusT ecosystems.”

“This means that the interactions that the musicians has with the instrument, or with the connected Musical Things through it, have to be predictable according to the expected behavior which was previously predefined.”

“Personalization has been advocated as a fundamental design component for IoT devices [50], [73], and this is no less true for SMIs.”

“, wireless sensor networks [23]), networked music performance systems [92], sensors and actuator augmentations typical of augmented instruments [72], and embedded intelligence tailored to support real-time audio and sensor processing.”

“One of the most noticeable examples of such DMIs is the ReacTable [48], [126].”

“The real-time operating system must ensure a round-trip latency of less than 10 ms to achieve credible musical interactions [66], [123] (i.”

“Therefore, Millimeter Waves represent a promising technology for enhancing musical instruments with ultra-low latency and highlyreliable wireless communication capabilities.”

“More often such methods are applied offline, especially by analyzing large datasets of audio files [5], while some authors have used real-time feature extraction for performance purposes [15], [43], [45], [74], especially employing machine learning techniques [29], [91].”

“This challenge falls in the agenda of the Tactile Internet community [1], [61].”

“This situation, which is rooted in the tenets of the autobiographical design [80], is the reason for which Jordá uses the term ‘‘digital luthier’’ for thosewho not only build but also perform with their instruments [47].”

“This challenge falls in the agenda of the Tactile Internet community [1], [61].”

“Some of these open challenges are common to those of the general field of IoMusT (the reader is referred to [112] and [113] for a discussion on this topic).”

“Personalization has been advocated as a fundamental design component for IoT devices [50], [73], and this is no less true for SMIs.”

“For instance, new protocols for the exchange of musical messages are needed to cope with the limitations of existing widely adopted protocols and standards (such as MIDI and OSC).”

“, wireless sensor networks [23]), networked music performance systems [92], sensors and actuator augmentations typical of augmented instruments [72], and embedded intelligence tailored to support real-time audio and sensor processing.”

“Other authors have explored forms of collaborative music making resulting from the interconnection of DMIs through local or remote networks [3], [122], by leveraging the so-called networked music performance (NMP) systems (see [92] and [32] for recent reviews).”

“Other authors have explored forms of collaborative music making resulting from the interconnection of DMIs through local or remote networks [3], [122], by leveraging the so-called networked music performance (NMP) systems (see [92] and [32] for recent reviews).”

“New DMIs may succeed where they contribute to expanding the creative possibilities of the player [77], [82].”

“For instance, while HCI often targets the improvement of human performance with interactive technologies, measured in terms of efficiency and accuracy, NIME focuses on concepts such as creativity [60], expressivity [24], engagement [16], constraints [58], appropriation [127], and”

“SMIs lie at the intersection between DMIs and smart objects [33], [50], as they encompass capabilities and features of both of them.”

“This amount is an accepted target for latency in real-time interactive audio systems [30], [31], [123].”

“Another set of authors have investigated computational methods for the extraction of information from the sound of musical instruments, within the fields of music information retrieval (MIR) [17] and semantic audio [99].”

“, sensor fusion, real-time music information retrieval, control of the sound engine from the sensors according to defined mapping strategies, adaptive digital audio effects [119]), as well as other types of computations performed on the internal memory (e.”

“Sensor-based augmentations consist of enhancing the instrument with an interface composed of sensors dedicated to the tracking of performer’s gestures, which are repurposed into electronically generated sounds via mapping techniques [41], [47], [57].”

“For instance, while HCI often targets the improvement of human performance with interactive technologies, measured in terms of efficiency and accuracy, NIME focuses on concepts such as creativity [60], expressivity [24], engagement [16], constraints [58], appropriation [127], and”

“, by means of hard real-time onset detectors [104]) and exploit sensor fusion and machine learning methods [29].”

“One of the emerging technologies for ultra-low latency and highly reliable wireless communication is Millimeter Waves [97], [98].”

“Some of these systems have been developed for technology-mediated audience participation (TMAP) purposes [39], where the audience of live music performances takes part to the creative process and interacts with the performers [20], [27], [89], [125].”

“Following these considerations, hereinafter we propose design principles that are specific to SMIs and that aim to complement other design guidelines presented by various authors in the NIME community [21], [22], [76], [77], [95], [126].”

“Examples of this category are the Magnetic Resonator Piano [64], the Feedback Resonance Guitar [85], or the Overtone Fiddle [84].”

“Smart instruments are instances of ‘‘Musical Things’’, within the ‘‘Internet of Musical Things’’ (IoMusT) paradigm [49], [112], [113], which refers to the network of interoperable devices dedicated to the production and/or reception of musical content.”

“For instance, while HCI often targets the improvement of human performance with interactive technologies, measured in terms of efficiency and accuracy, NIME focuses on concepts such as creativity [60], expressivity [24], engagement [16], constraints [58], appropriation [127], and”

“Examples of this category are the Magnetic Resonator Piano [64], the Feedback Resonance Guitar [85], or the Overtone Fiddle [84].”

“In 2008, Johnston and colleagues proposed design and evaluation criteria specific for DMIs based on acoustic instruments controlling virtual instruments simulated with physical modeling techniques [46].”

“, strummed, plucked, blowed, hit, or bowed) in order to control the electronically generated sounds [86].”

“To attract the musicians and favor the adoption of SMIs we suggest that the instrument should have ‘‘signature features’’ [79] (also called ‘‘unique identity’’ in [83]).”

“, [2], [90], [100]), which are structured, machine processable representations of audio analyses that facilitate automated data processing and knowledge-based reasoning [28].”

“Considering an organological perspective [59], one can draw the following relationships between SMIs and existing DMIs.”

“To attract the musicians and favor the adoption of SMIs we suggest that the instrument should have ‘‘signature features’’ [79] (also called ‘‘unique identity’’ in [83]).”

“These standards may include protocols for wireless transmission and for musical data exchange or synchronization over it, along with mechanisms for auto-configuration and auto-identification [25], [62].”

“, [38], [50]) and in particular the recent work of Cena et al.”

“The Sensus Smart Guitar has been used to wirelessly control visuals, digital audio workstations running on laptops, virtual reality headsets, to control or be controlled by smartphones, to record audio files and share them on social networks [111].”

“, [69], [70]) to indicate musical instruments with onboard acoustic actuation, which is only one component of a SMI in the usage proposed by the author and colleagues in [116].”

“For instance, aesthetic considerations should be made when designing haptic sensations for the musicians (both physically and digitally produced), as they may affect how pleasant the interaction of the musician with the instrument is perceived [36].”

“For instance, the Smart Mandolin has been used to perform concerts leveraging 4www.youtube.com/watch?v=BTTVOk-OpaA 5www.mindmusiclabs.com/wp-content/uploads/2018/01/DVMark- Smart-Multiamp.pdf real-time audio features extraction techniques.”

“developed the Carolan Guitar, an acoustic guitar augmentedwith decorations in form of QR codes that allow one to collect and access online digital records about the history of the instrument [6].”

“, [2], [90], [100]), which are structured, machine processable representations of audio analyses that facilitate automated data processing and knowledge-based reasoning [28].”

“This principle is in line with existing design frameworks that suggested that the execution and the identification of unique playing styles should be supported by means of personalization capabilities offered by a DMI [76], [120].”

“V. OPEN CHALLENGES AND FUTURE DIRECTIONS IoT technologies have provided new possibilities for exploration of musical interactions between the players and their instrument, between players, and between players and audiences, in both co-located and remote settings.”

“The Smart Mandolin and the Smart Cajón have been utilized to control haptic wearable devices [108] in possession of audience members, by exploiting real-time audio features extraction techniques [107], [109].”

“Some of these systems have been developed for technology-mediated audience participation (TMAP) purposes [39], where the audience of live music performances takes part to the creative process and interacts with the performers [20], [27], [89], [125].”

“[9] E. Berdahl, S. Salazar, and M. Borins, ‘‘Embedded networking and hardware-accelerated graphics with satellite CCRMA,’’ in Proc.”

“Among the numerous examples of this category, one can cite the Hyper Flute [87], the Augmented Violin [11], the Electrumpet [55], or the Hyper Mandolin [103].”

“More often such methods are applied offline, especially by analyzing large datasets of audio files [5], while some authors have used real-time feature extraction for performance purposes [15], [43], [45], [74], especially employing machine learning techniques [29], [91].”

“Design principles and considerations developed by the IoT research community include proposals for the characteristics and capabilities of the smart objects composing the IoT [18], [19], [50], on the services they can support [4], or on processes for industries [37].”

“developed a drum-based DMI that shares several features with SMIs: sensors and actuators enhancements, embedded sound processing, wireless connectivity for reception of OSC messages [35].”

“Current efforts towards this end are represented by the MPEG-A: Interactive Music Application Format [34], which combines multiple audio tracks and appropriate additional information and is expected to enable interoperability among the interactive music services recently emerged.”

“Such opportunities envisioned in [145] with respect to Smart VRMIs have yet to be explored for the most part.”

“The sounds produced by performers must be first digitized via a soundcard in case the instrument is not digital or smart [52].”

“These include smart musical instruments [45] and wearable devices for performers and audiences [46].”

“Other authors have proposed preliminary architectures based on Semantic Web technologies to foster interoperability across heterogeneous Musical Things.”

“Hence, for projects involving heterogeneous IoMusT hardware (such as smart instruments [46]), it is possible to build the whole software sys-tem upon Elk Audio OS and easily adopt existing libraries.”

“According to the vision proposed in Reference [33], the embedded intelligence of an SMI is characterized by five core capabilities: (1) knowledge management, i.e., the capability of maintaining knowledge about itself and the environment; (2) reasoning, i.e., the capability of making inferences on…”

“One of the building blocks of the IoMusT paradigm is the so-called smart musical instruments (SMIs), an emerging class of musical instruments characterised by embedded sensors, actuators, wireless connectivity, and on-board processing (Turchet, 2019).”

“One of the most prominent instances of Musical Things is the so-called Smart Musical Instruments, an emerging category of IoT devices characterized by sensors, actuators, wireless connectivity, and embedded intelligence [2].”

“Similarly, the IoMusT and SMI ontologies (responsible for semantic interoperability and discoverability) have already been evaluated in their respective studies [17, 11].”

“For instance, it is envisioned that musical instruments will equip with a communication interface, making them smart musical instruments (SMIs) [3].”

“Smart musical instruments [10] is an emerging class of digital musical instruments, which is envisioned to be aware of the emotions expressed by the performers in order to support various kinds of musical activities through dedicated services (such as the query of music repositories by playing excerpts with a given em…”

“Concep-tualizing music as multimodal, MHWs can communicate musical information by utilizing tactile sensations (e.g., pressure), with the vibrotactile feedback being one common input [46, 81, 82, 84, 85].”

“Alternatively, if these guitars were Smart Musical Instruments [36,37] then these digital resources might be physically stored in and accessed through the instrument.”

“[146] L. Turchet, ‘‘Smart musical instruments: Vision, design principles, and future directions,’’ IEEE Access, vol. 7, pp. 8944–8963, 2019.”

“Smart musical instruments (SMIs) [30], which are instruments characterized by the presence of sensors and actuators equipped with intelligence and that allow to connect to the network, also appear as a solution.”

“For example we can imagine smart instruments [18] embedding a metronome in a way to provide an efficient way to face the network latency.”

“Examples of commercial smart instruments include the Sensus Smart Guitar developed by MIND Music Labs [16], an electroacoustic guitar with on-board processing (ELK music operating system [54]), sensors and wireless communication, as well as the HyVibe guitar [55], an acoustic guitar with an active vibration control sy…”

“Examples of Musical Things are musical haptic wearables for the audience (MHWAs) [14] and smart musical instruments [12].”

“In cellular networks, smart musical instruments (SMIs) (as well as other kinds of Musical Things) can be seen as a completely new class of user equipments (UEs) [2].”

“Musical things, such as smart musical instruments [8] or musical haptic wearables [9], are connected by an infrastructure that enables multidirectional communication, both locally and remotely.”

“An example of this family of instruments is the Sensus Smart Guitar developed by MIND Music Labs [5].”

“Additionally, IoS facilitates the interconnection of smart musical instruments (SMI) [7] and supports the creation of innovative wearable musical equipment.”

“Smart Instruments are a family of musical instruments characterized by embedded computational intelligence, wireless connectivity, an embedded sound delivery system, and an onboard system for feedback to the player [23], [47].”

“Noticeable examples in this space are the so-called smart musical instruments [120] and the musical haptic wearables for performers or audience members [127].”

“Despite being in its early stages, practical applications of these concepts have already been explored, such as in networked music performance [1] and multimedia remixing practices [4] utilizing smart musical instruments (SMIs) [5].”

“It can also be used in live stage performances to record bands in separate studios (Turchet 2019).”

“These are an emerging category of musical instruments characterized by sensors, actuators, wireless connectivity, and embedded intelligence [69].”

“A prominent example within this realm is the Smart Musical Instrument (SMI) [6], [7].”

“The “smart musical instruments” are an emerging family of digital musical instruments [1], which draws upon different lines of existing research including augmented instruments [2], embedded acoustic instruments, [3], and networked music performance systems [4].”

“We believe the ideas behind 8https://www.ableton.com/en/live/max-for-live 9https://www.ableton.com/en/live 10https://ffont.github.io/freesound-loop-generator 11https://www.xlnaudio.com/products/xo 12https://www.loopcloud.com/cloud 13https://lesound.io/product/audiotexture-free 14https://output.com/products/arcade SOUR…”

“In music settings, studies have explored haptics as musical instruments [9, 48, 51], for conveying musical instruction [17, 21, 24, 25, 36], as a technical learning aid [10, 53], for communication [50] and for synchronization [3, 5, 15, 16].”

“In this context, we believe that any effort in reducing the gap between development and deployment for audio and music DL can foster the creation of new self-contained Digital Musical Instruments (DMIs), such as Smart Musical Instruments (SMIs) [12], as well as AI-equipped audio devices.”

“This is a classic Neapolitan mandolin smartified with different types of sensors, a microphone, a loudspeaker, wireless connectivity to both local networks and the Internet, and a low-latency audio processing board [28].”

“…querying online sounds repositories AM’20, September 15–17, 2020, Graz, Austria the Internet of Audio Things [37] and Internet of Musical Things [38] paradigms interface Audio Commons repositories with devices based on embedded systems such as wearables [29] or smart musical instruments [35].”

“Recent work on the Internet of Sounds (or the related areas of the Internet of Audio Things [10] and Internet of Musical Things [11]) has focused on the instrument: Internet-enabled smart instruments that combine sensors, computational capabilities, and network connectivity [8].”

“Taken together, these features make the Retrologue Hardware Synth an instance of the family of smart musical instruments proposed in [25].”

“This is particularly true for applications centered around the so-called smart musical instruments [30].”

“A prominent example is the family of smart musical instruments proposed in [14].”

“The majority of IoMusT ecosystems available today have mostly focused on the use of intelligent and connected musical instruments [15], [16], smart textiles [13], and networked music performance systems [17], [18].”

“For instance, IoMusT research has recently developed interactions between performers that leverage the connection of smart musical instruments [24] with smartphones [25].”

“Recently, Turchet proposed a vision where smart instruments have capabilities of advanced reasoning on, learning from, and adaptation to the player’s actions and data from connected devices, which will confer them with unprecedented context-awareness and proactivity features [61].”

“Turchet, L. Smart Musical Instruments: Vision, Design Principles, and Future Directions.”

“A reliable pianist identifier holds great potential for not only studying the styles of different performers, but also various applications in music education, music information retrieval and smart musical instruments [6].”

“Another possibility consists of integrating the XAI system directly into the musical instrument, as it has been explored recently in the literature on smart musical instruments [33].”

“Smart musical instruments (SMIs), such as the smart mandolin, were defined by Luca Turchet as ‘ devices dedicated to playing music, which are equipped with embedded intelligence and are able to communicate with external devices ’ (Turchet 2018: 8948).”

“To date the opportunities offered by the Internet of Musical Things paradigm for creating TMAP performances has been scarcely investigated [41], but there is some recent work on TMAP and IoMusT.”

“For example, in the case of the compressor, the ratio was set within the range [1, 4], while all other parameters were pre-tuned and kept fixed.”

“Smart Musical Instruments (SMIs) are a subset of Digital Musical Instruments (DMIs) that place additional emphasis on connectivity, leveraging of sensors and embedded intelligence [17].”

“We evaluate the others by relying on accurate measurements of proof-of-concept prototypes (§V) composed by two types of smart musical instruments [13] wirelessly connected via UWB to two types of musical haptic wearable.”

“They are exemplified as smart musical instruments [16], wearable equipment [17], or smartphones with music apps that publish audio resources.”

“The use of such methods is particularly relevant to some applications in the emerging field of smart musical instruments [6], which may repurpose the information extracted from the signal with imperceptible latency for the player (see, e.”

“Smart musical instruments [6] are envisioned to generate and leverage large quantities of data to enable a whole new set of proactive services based on the retrieval of the context surrounding the instrument itself (e.g., who is the player, which music is being played, in which musical activity,…”

“The present study is part of a larger project targeting the creation of self-contained instruments such as smart guitars [35].”

“…of the French ANR research project DOTS, supported by IRCAM, the Web Audio API has been reimplemented to run outside the browser in devices such as Raspberry PIs [22], and we've run some WAM instruments and effects on these devices, making it possible to create smart WAM-based instruments [21].”

“These include smart musical instrument [2], which are self-contained instruments embedding the ability of processing music-related information and communicating wirelessly with external devices.”

“Moreover, the AudioInsight may serve as an example for sound texture exploration and development of Smart Musical Instruments (SMIs) [38].”

“This application utilizes a smart musical instrument (SMI) [4], the RemixDrum [8], along with two electronic pedals as a control interface.”

“However, the advancement of embedded computing, wireless connectivity, and artificial intelligence has given rise to a class of Smart Musical Instruments (SMIs) [1], which represent intelligent, responsive musical platforms.”

“This article selects the data enhancement operation of zooming in and rotating, horizontal flipping and vertical flipping of the scene image as follows [16-18].”

“Turchet and colleagues presented a system called Sensus Smart Guitar in 2016 [20].”

“An emerging application for low-power battery operated digital simulations is on-board audio processing on Smart Musical Instruments [1].”

“The field of musical instrument design has seen a progression from solely acoustic instruments to fully digital and smart instruments [1].”

“Such implementations are a primary part in enabling the applications centered around the so-called smart musical instruments [9].”

“Smart Musical Instruments (SMIs) [21] are one type of DMI that places significant weight on connectivity, usage of sensors, and on-board intelligence and are one of the most common things found within the IoS.”

“A prominent example of a musical thing in the IoMusT is that of SMIs [7], a family of interoperable musical instruments equipped with sensing and actuation capabilities, along with embedded reasoning abilities and wireless interconnecting technology.”

“The PhysioDrum environment comprises a virtual drum kit accessed via a virtual reality (VR) headset, enabling immersive interaction through an smart musical instrument (SMI) [11] called RemixDrum [12], complemented by two foot pedals.”

“By providing discrete haptic cues at key thresholds, performers can build a more accurate mental model of the instrument’s state and thus reduce the cognitive load that Turchet identifies as a key design consideration for SMIs [21].”