Visualizing digital media interactions: providing feedback on jam2jam AV performances
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Instrumental music performance is a well-established case of real-time interaction with technology and, when extended to ensembles, of interaction with others. However, these interactions are fleeting and the opportunities to reflect on action is limited, even though audio and video recording has recently provided important opportunities in this regard. In this paper we report on research to further extend these reflective opportunities through the capture and visualization of gestural data collected during collaborative virtual performances; specifically using the digital media instrument Jam2jam AV and the specifically-developed visualization software Jam2jam AV Visualize. We discusses how such visualization may assist performance development and understanding. The discussion engages with issues of representation, authenticity of virtual experiences, intersubjectivity and wordless collaboration, and creativity support. Two usage scenarios are described showing that collaborative intent is evident in the data visualizations more clearly than in audio-visual recordings alone, indicating that the visualization of performance gestures can be an efficient way of identifying deliberate and co-operative performance behaviours.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Educating the reflective practitioner
Schön, D. A. (1987) · Cited by 14,166
Clark, A., & Chalmers, D. (1998) · Analysis · doi:10.1093/analys/58.1.7 · Cited by 6,333 · cited 3×
“Brisbane.”
BackgroundWhere the Action Is: The Foundations of Embodied Interaction
Dourish, P. (2001) · Cited by 3,148
Abstracting craft: the practiced digital hand
(1997) · Choice Reviews Online · doi:10.5860/choice.35-0332 · Cited by 451 · cited 2×
“It lets ideas become things (McCullough, 1996:52).”
A tour through the visualization zoo
Heer, J., Bostock, M., & Ogievetsky, V. (2010) · Communications of the ACM · doi:10.1145/1743546.1743567 · Cited by 436
A preliminary framework for description, analysis and comparison of creative systems
Wiggins, G. A. (2006) · Knowledge-Based Systems · doi:10.1016/j.knosys.2006.04.009 · Cited by 291 · cited 2×
“Innovations in visualisation continue today, including the use of vertical, horizontal and 45° diagonal lines for stylized public transportation maps from the 1930s and the use of web-based interactive data visualisation (Fry, 2008).”
BackgroundShneiderman, B. (2002) · Communications of the ACM · doi:10.1145/570907.570945 · Cited by 280
Creativity Support Tools: Report From a U.S. National Science Foundation Sponsored Workshop
Shneiderman, B., Fischer, G., Czerwinski, M., Resnick, M., Myers, B., Candy, L., Edmonds, E., Eisenberg, M., Giaccardi, E., Hewett, T., et al. (2006) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc2002_1 · Cited by 274
Representing musical structure
Howell, P., West, R., & Cross, I. R. M. (1991) · Cited by 194
Beyond MIDI: The Handbook of Musical Codes
Repetto, D. I., & Selfridge-Field, E. (1998) · Notes · doi:10.2307/900186 · Cited by 179 · cited 3×
“Nussbaum, C.”
BackgroundThe Back of the Napkin: Solving Problems and Selling Ideas with Pictures
Roam, D. (2008) · Cited by 155
The Musical Representation: Meaning, Ontology, and Emotion
Nussbaum, C. O. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7470.001.0001 · Cited by 114
The musical representation: meaning, ontology, and emotion
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0213 · Cited by 98 · cited 3×
“This paper presents new work on visualizing performance gestures from virtual media ensembles and explores its effectiveness as feedback that supports participant reflection, typically in music education settings.”
BackgroundFry, B. (2007) · Cited by 66
Performance visualization: a new challenge to music through visualization
Hiraga, R., Mizaki, R., & Fujishiro, I. (2002) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/641007.641054 · Cited by 31
Visualization of music performance as an aid to listener's comprehension
Hiraga, R., & Matsuda, N. (2004) · Working Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/989863.989878 · Cited by 17 · cited 1×
“Previous research has examined the visualization of music performance data; see, for example, articles by Hiraga and his colleagues (Hiraga & Matsuda, 2004; Hiraga, Mizaki, & Fujishiro, 2002).”
MethodVisualization in comparative music research
Toiviainen, P., & Eerola, T. (2007) · Physica-Verlag eBooks · doi:10.1007/978-3-7908-1709-6_16 · Cited by 13
Evolving interactions: agile design for networked media performance
Brown, A. R., Dillon, S. C., Kerr, T., & Sorensen, A. (2009) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/1738826.1738834 · Cited by 9
Self-citationThe social life of visualization
Macdonald, H., Yuille, J., Stanton, R., & Viller, S. (2009) · Australian Conference on Computer-Human Interaction (OzCHI) · doi:10.1145/1738826.1738858 · Cited by 5
Cited by
Reviewing reflection: on the use of reflection in interactive system design
Baumer, E. P. S., Khovanskaya, V., Matthews, M., Reynolds, L., Schwanda Sosik, V., & Gay, G. (2014) · Proceedings of the 2014 conference on Designing interactive systems · doi:10.1145/2598510.2598598 · Cited by 266
Kumar, H., Xiao, R., Lawson, B., Musabirov, I., Shi, J., Wang, X., Luo, H., Williams, J. J., Rafferty, A. N., Stamper, J., et al. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662042 · Cited by 53
“Many have previously argued for and demonstrated the value of reflection in education [5, 7, 9, 12–14, 18, 22, 24, 31, 32, 36, 37, 39, 42, 44, 46, 50].”
BackgroundCollaborative Digital Media Performance with Generative Music Systems
Brown, A. R., & Dillon, S. C. (2012) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199928019.013.0036 · Cited by 11
Self-citationAn Affective Video Generation System Supporting Impromptu Musical Performance
Kobayashi, A., & Fujishiro, I. (2016) · International Conference on Cyberworlds (CW) · doi:10.1109/cw.2016.11 · Cited by 2
“[1] and Brown [2], or to assist players in mastering musical instruments, including Rogers et al.”
BackgroundFactors of Coolness in Choreography Design Support System for Lock Dance Beginners
Yamamoto, K., TANAKA, I., Kuramoto, I., & Tsujino, Y. (2017) · Transactions of Japan Society of Kansei Engineering · doi:10.5057/jjske.tjske-d-17-00070 · Cited by 1
Metadata
- Authors
- Brown, A. R.
- Affiliations
- Queensland University of Technology (AU)
- Corresponding author
- Brown, Andrew R.
- ORCID
- Brown, Andrew R.
- Venue
- Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction
- Publisher
- ACM
- Published
- 2010-11-22
- Pages
- 196-199
- OpenAlex
- W2101888083
- Cited by
- 5
- References
- 19
- Citations by year
- 2024: 1 · 2020: 1 · 2017: 1 · 2016: 1 · 2014: 1
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Australian Conference on Computer-Human Interaction (OzCHI) (publishedVersion)
- Funders
- Australian Government
- Topics
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Art, Technology, and Culture
- Social Sciences › Arts and Humanities › Music › Diverse Music Education Insights
- Keywords
- collaborative music performance, virtual performance, performance development, intersubjectivity, creativity support, ensemble interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.