CoChat
Sign Up
Cited by 19Tactile and Sensory InteractionsTeleoperation and Haptic SystemsInteractive and Immersive Displays

The DOI matches its Crossref record; the references it cites are not checked yet.

Haptic feedback combined with movement sonification using a friction sound improves task performance in a virtual throwing task

Open access
Frid, Emma, Moll, Jonas, Bresin, Roberto, Sallnäs Pysander, Eva-Lotta·2019·Journal on Multimodal User Interfaces
DOI 10.1007/s12193-018-0264-4
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper we present a study on the effects of auditory- and haptic feedback in a virtual throwing task performed with a point-based haptic device. The main research objective was to investigate if and how task performance and perceived intuitiveness is affected when interactive sonification and/or haptic feedback is used to provide real-time feedback about a movement performed in a 3D virtual environment. Emphasis was put on task solving efficiency and subjective accounts of participants’ experiences of the multimodal interaction in different conditions. The experiment used a within-subjects design in which the participants solved the same task in different conditions: visual-only, visuohaptic, audiovisual and audiovisuohaptic. Two different sound models were implemented and compared. Significantly lower error rates were obtained in the audiovisuohaptic condition involving movement sonification based on a physical model of friction, compared to the visual-only condition. Moreover, a significant increase in perceived intuitiveness was observed for most conditions involving haptic and/or auditory feedback, compared to the visual-only condition. The main finding of this study is that multimodal feedback can not only improve perceived intuitiveness of an interface but that certain combinations of haptic feedback and movement sonification can also contribute with performance-enhancing properties. This highlights the importance of carefully designing feedback combinations for interactive applications.

Overview

Integrity signals

Registry

Crossref · 10.1007/s12193-018-0264-4

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In visual throwing tasks, point-based haptic devices and auditory feedback did not significantly contribute to the improvement of task performance.

    “point-based haptic devices and moreover auditory feedback did not signi cantly contribute to the improvement of task performance.”

    From Study of Multimodal Interfaces and the Improvements on Teleoperation (2020) · cites this paper as [45]

Citations

The citation lists haven’t been fetched yet

References 53 · Cited by 19

Research works printed but not linked: 48

  1. Avanzini F, Rocchesso D, Serafin S (2004) Friction sounds for sensory substitution. In: Proceedings of the 10th meeting of the international conference on auditory display (ICAD 2004), Sydney, pp 1–8
  2. Bencina R, Wilde D, Langley S (2008) Gesture $$\approx $$ ≈ sound experiments: process and mappings. In: Proceedings of the 2008 conference on new interfaces for musical expression (NIME08), pp 197–202
  3. Brewster SA, Wright PC, Edwards AD (1993) An evaluation of earcons for use in auditory human–computer interfaces. In: Proceedings of the INTERACT’93 and CHI’93 conference on human factors in computing systems, pp 222–227 · DOI
  4. Burke JL, Prewett MS, Gray AA, Yang L, Stilson FR, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: A meta-analysis. In: Proceedings of the 8th international conference on multimodal interfaces, pp 108–117 · DOI · Paper page
  5. Conti F, Barbagli F, Balaniuk R, Halg M, Lu C, Morris D, Sentis L, Warren J, Khatib O, Salisbury K (2003) The chai libraries. In: Proceedings of Eurohaptics 2003, Dublin, pp 496–500
  6. Crommentuijn K, Winberg F (2006) Designing auditory displays to facilitate object localization in virtual haptic 3d environments. In: Proceedings of the 8th international ACM SIGACCESS conference on computers and accessibility. ACM, pp 255–256 · DOI
  7. Cumming G (2014) The new statistics—why and how. Psychol Sci 25:7–29 · DOI · Paper page
  8. Delle Monache S, Polotti P, Rocchesso D (2010) A toolkit for explorations in sonic interaction design. In: Proceedings of the 5th audio mostly conference: a conference on interaction with sound, p. 1 · DOI
  9. Driver J, Spence C (1998) Crossmodal attention. Curr Opin Neurobiol 8(2):245–253 · DOI
  10. Dubus G, Bresin R (2013) A systematic review of mapping strategies for the sonification of physical quantities. PloS one 8(12):e82491 · DOI
  11. Effenberg AO, Fehse U, Schmitz G, Krueger B, Mechling H (2016) Movement sonification: effects on motor learning beyond rhythmic adjustments. Front Neurosci 10:219 · DOI
  12. Ekman I, Rinott M (2010) Using vocal sketching for designing sonic interactions. In: Proceedings of the 8th ACM conference on designing interactive systems (DIS), pp 123–131 · DOI
  13. Fernström M, McNamara C (2005) After direct manipulation—direct sonification. ACM Trans Appl Percept (TAP) 2(4):495–499 · DOI
  14. Gaver WW (1993) Synthesizing auditory icons. In: Proceedings of the INTERACT’93 and CHI’93 conference on human factors in computing systems. ACM, pp 228–235 · DOI
  15. Gescheider GA, Barton WG, Bruce MR, Goldberg JH, Greenspan MJ (1969) Effects of simultaneous auditory stimulation on the detection of tactile stimuli. J Exp Psychol 81(1):120 · DOI
  16. Grabowski NA, Barner KE (1998) Data visualisation methods for the blind using force feedback and sonification. In: Proceedings of Spie-the international society for optical engineering, vol 3524. SPIE-Int. Soc. Opt. Eng., pp 131–139
  17. Guest S, Catmur C, Lloyd D, Spence C (2002) Audiotactile interactions in roughness perception. Exp Brain Res 146(2):161–171 · DOI
  18. Hermann T, Hunt A, Neuhoff JG (2011) The sonification handbook. Logos Verlag, Berlin
  19. Hötting K, Röder B (2004) Hearing cheats touch, but less in congenitally blind than in sighted individuals. Psychol Sci 15(1):60–64 · DOI · Paper page
  20. Jousmäki V, Hari R (1998) Parchment-skin illusion: sound-biased touch. Curr Biol 8(6):R190–R191 · DOI
  21. Kaklanis N, Votis K, Tzovaras D (2013) Open touch/sound maps: a system to convey street data through haptic and auditory feedback. Comput Geosci 57:59–67 · DOI
  22. Kitagawa N, Spence C (2006) Audiotactile multisensory interactions in human information processing. Jpn Psychol Res 48(3):158–173 · DOI
  23. Koseleff P (1957) Studies in the perception of heaviness. i. 1.2: some relevant facts concerning the size-weight-effect. Acta Psychologica 13:242–252 · DOI
  24. Laurienti PJ, Kraft RA, Maldjian JA, Burdette JH, Wallace MT (2004) Semantic congruence is a critical factor in multisensory behavioral performance. Exp Brain Res 158(4):405–414 · DOI
  25. Lee S, Kim GJ (2008) Effects of haptic feedback, stereoscopy, and image resolution on performance and presence in remote navigation. Int J Hum Comput Stud 66(10):701–717 · DOI
  26. McDonald P, Van Emmerik R, Newell K (1989) The effects of practice on limb kinematics in a throwing task. J Mot Behav 21(3):245–264 · DOI
  27. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748 · DOI · Paper page
  28. Molholm S, Ritter W, Javitt DC, Foxe JJ (2004) Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cereb Cortex 14(4):452–465 · DOI
  29. Moll J, Sallnäs Pysander EL (2013) A haptic tool for group work on geometrical concepts engaging blind and sighted pupils. ACM Trans Access Comput (TACCESS) 4(4):14
  30. Nam CS, Shu J, Chung D (2008) The roles of sensory modalities in collaborative virtual environments (CVES). Comput Hum Behav 24(4):1404–1417 · DOI
  31. Moll J, Huang Y, Sallnäs EL (2010) Audio makes a difference in haptic collaborative virtual environments. Interact Comput 22(6):544–555 · DOI
  32. Moll J, Pysander ELS, Eklundh KS, Hellström SO (2014) The effects of audio and haptic feedback on collaborative scanning and placing. Interact Comput 26(3):177–195 · DOI
  33. Frid E, Bresin R, Moll J, Pysander ELS (2017) An exploratory study on the effect of auditory feedback on gaze behavior in a virtual throwing task with and without haptic feedback. In: Sound and Music Computing Conference 2017 (SMC2017)
  34. Rassmus-Gröhn CMK (2005) Audio haptic tools for navigation in non visual environments. In: ENACTIVE 2005, the 2nd international conference on enactive interfaces, Genoa-Italy, 2005, pp 17–18
  35. Repp BH, Penel A (2004) Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Psychol Res 68(4):252–270
  36. Ro T, Hsu J, Yasar NE, Elmore LC, Beauchamp MS (2009) Sound enhances touch perception. Exp Brain Res 195(1):135–143 · DOI
  37. Sallnäs EL (2004) The effect of modality on social presence, presence and performance in collaborative virtual environments. PhD Thesis. KTH Royal Institute of Technology, Stockholm
  38. Sallnäs EL, Rassmus-Gröhn K, Sjöström C (2000) Supporting presence in collaborative environments by haptic force feedback. ACM Trans Comput Hum Interact (TOCHI) 7(4):461–476 · DOI
  39. Schürmann M, Caetano G, Hlushchuk Y, Jousmäki V, Hari R (2006) Touch activates human auditory cortex. NeuroImage 30(4):1325–1331 · DOI
  40. Sherrick CE (1976) The antagonisms of hearing and touch. In: Hirsh SK, Eldredge DH, Hirsh IJ, Silverman SR (eds) Hearing and Davis: essays honoring Hallowell Davis, St. Washington University Press, Louis, pp 149–158
  41. Shimojo S, Shams L (2001) Sensory modalities are not separate modalities: plasticity and interactions. Curr Opin Neurobiol 11(4):505–509 · DOI
  42. Sigrist R, Rauter G, Marchal-Crespo L, Riener R, Wolf P (2015) Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233(3):909–925 · DOI
  43. Someren Mv, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical approach to modelling cognitive processes. Academic Press, Cambridge
  44. Srinivasan MA, Basdogan C (1997) Haptics in virtual environments: taxonomy, research status, and challenges. Comput Gr 21(4):393–404 · DOI
  45. Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9(4):255–266 · DOI · Paper page
  46. Suied C, Susini P, McAdams S, Patterson RD (2010) Why are natural sounds detected faster than pips? J Acoust Soc Am 127(3):EL105–EL110 · DOI
  47. Susini P, Misdariis N, Lemaitre G, Houix O (2012) Naturalness influences the perceived usability and pleasantness of an interfaces sonic feedback. J Multimodal User Interfaces 5(3–4):175–186 · DOI
  48. Wright M, Freed A et al (1997) Open Sound Control: A new protocol for communicating with sound synthesizers. In: Proceedings of the 1997 international computer music conference (ICMC1997), vol 2013, p 10

Metadata

Authors
Frid, E., Moll, J., Bresin, R., & Sallnäs Pysander, E.-L.
Affiliations
  • KTH Royal Institute of Technology (SE)
  • Uppsala University (SE)
Corresponding author
Frid, Emma
ORCID
Venue
Journal on Multimodal User Interfaces
Publisher
Springer Science and Business Media LLC
Published
2018-05-09
Volume
13
Issue
4
Pages
279-290
ISSN
  • 1783-7677 (ISSN-L)
  • 1783-8738
DOI
10.1007/s12193-018-0264-4
OpenAlex
W2803034103
Cited by
19
References
53
Citations by year
2026: 2 · 2025: 3 · 2024: 4 · 2022: 3 · 2021: 2 · 2020: 3 · 2019: 1 · 2018: 1
FWCI
1.04
Type
journal article
Language
en
Update notices
correction (2018-09-14)
Open access
Licence
Locations
  • Journal on Multimodal User Interfaces (publishedVersion, cc-by)
  • Uppsala University Publications (Uppsala University) (submittedVersion, other-oa)
Funders
  • Vetenskapsrådet (D0511301)
  • Kungliga Tekniska Högskolan (Personal Grant)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
  • Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
Keywords
haptic feedback, movement sonification, friction sound, multimodal feedback, task performance, sound models, virtual worlds, real-time feedback

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review