Pictures in Your Mind: Using Interactive Gesture-Controlled Reliefs to Explore Art
Open accessAbstract
Tactile reliefs offer many benefits over the more classic raised line drawings or tactile diagrams, as depth, 3D shape, and surface textures are directly perceivable. Although often created for blind and visually impaired (BVI) people, a wider range of people may benefit from such multimodal material. However, some reliefs are still difficult to understand without proper guidance or accompanying verbal descriptions, hindering autonomous exploration. In this work, we present a gesture-controlled interactive audio guide (IAG) based on recent low-cost depth cameras that can be operated directly with the hands on relief surfaces during tactile exploration. The interactively explorable, location-dependent verbal and captioned descriptions promise rapid tactile accessibility to 2.5D spatial information in a home or education setting, to online resources, or as a kiosk installation at public places. We present a working prototype, discuss design decisions, and present the results of two evaluation studies: the first with 13 BVI test users and the second follow-up study with 14 test users across a wide range of people with differences and difficulties associated with perception, memory, cognition, and communication. The participant-led research method of this latter study prompted new, significant and innovative developments.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot the international journal of robotics research and application · doi:10.1108/ir.1999.04926fae.001 · Cited by 39,228 · see citing works
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · see citing works
Object modelling by registration of multiple range images
Chen, Y., & Medioni, G. G. (1992) · Image and Vision Computing · doi:10.1016/0262-8856(92)90066-c · Cited by 3,209 · see citing works
User-defined gestures for surface computing
Wobbrock, J. O., Morris, M. R., & Wilson, A. D. (2009) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1518701.1518866 · Cited by 1,321 · see citing works
Model globally, match locally: Efficient and robust 3D object recognition
Drost, B., Ulrich, M., Navab, N., & Ilić, S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540108 · Cited by 980 · see citing works
Efficient model-based 3D tracking of hand articulations using Kinect
Oikonomidis, I., Kyriazis, N., & Argyros, A. (2011) · British Machine Vision Conference (BMVC) · doi:10.5244/c.25.101 · Cited by 895 · see citing works
The use of eye movements in human-computer interaction techniques: what you look at is what you get
Jacob, R. J. K. (1991) · ACM Transactions on Information Systems · doi:10.1145/123078.128728 · Cited by 845 · see citing works
3D Object Recognition in Cluttered Scenes with Local Surface Features: A Survey
Guo, Y., Bennamoun, M., Sohel, F. A., Lu, M., & Wan, J. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2316828 · Cited by 602 · see citing works
Inclusive Research with People with Learning Disabilities: Past, Present and Futures
Walmsley, J., & Johnson, K. (2003) · Thammasat University Digital Collections · Cited by 455 · see citing works
Accurate, Robust, and Flexible Real-time Hand Tracking
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., et al. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702179 · Cited by 431 · see citing works
Piper, B., Ratti, C., & Ishii, H. (2002) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/503376.503439 · Cited by 408 · see citing works
What a Difference a Decade Makes: Reflections on doing ‘emancipatory’ disability research
BARNES, C. (2003) · Disability & Society · doi:10.1080/713662197 · Cited by 406 · see citing works
Usable gestures for blind people: understanding preference and performance
Kane, S. K., Wobbrock, J. O., & Ladner, R. E. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979001 · Cited by 314 · see citing works
Using a depth camera as a touch sensor
Wilson, A. D. (2010) · ACM International Conference on Interactive Tabletops and Surfaces (ITS) · doi:10.1145/1936652.1936665 · Cited by 263 · see citing works
Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware
Yeo, H.-S., Lee, B.-G., & Lim, H. (2013) · Multimedia Tools and Applications · doi:10.1007/s11042-013-1501-1 · Cited by 215 · see citing works
O'Modhrain, S., Giudice, N. A., Gardner, J. A., & Legge, G. E. (2015) · IEEE Transactions on Haptics · doi:10.1109/toh.2015.2466231 · Cited by 100 · see citing works
D’Agnano, F., Balletti, C., Guerra, F., & Vernier, P. (2015) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprsarchives-xl-5-w4-207-2015 · Cited by 98 · see citing works
The Tactile Graphics Helper: Providing Audio Clarification for Tactile Graphics Using Machine Vision
Fusco, G., & Morash, V. S. (2015) · Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS '15 · doi:10.1145/2700648.2809868 · Cited by 91 · see citing works
Scalable 3D Tracking of Multiple Interacting Objects
Kyriazis, N., & Argyros, A. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.438 · Cited by 89 · see citing works
LucentMaps: 3D Printed Audiovisual Tactile Maps for Blind and Visually Impaired People
Götzelmann, T. (2016) · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982163 · Cited by 86 · see citing works
Customizable 3D Printed Tactile Maps as Interactive Overlays
Taylor, B., Dey, A., Siewiorek, D., & Smailagic, A. (2016) · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982167 · Cited by 84 · see citing works
Klatzky, R. L., Giudice, N. A., Bennett, C. R., & Loomis, J. M. (2014) · Multisensory Research · doi:10.1163/22134808-00002447 · Cited by 74 · see citing works
Art Beyond Sight: A Resource Guide to Art, Creativity, and Visual Impairment
Axel, E. S., & Levent, N. (2002) · Medical Entomology and Zoology · Cited by 70 · see citing works
Kane, S. K., Frey, B., & Wobbrock, J. O. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470704 · Cited by 63 · see citing works
High-quality tactile paintings
Reichinger, A., Maierhofer, S., & Purgathofer, W. (2011) · Journal on Computing and Cultural Heritage · doi:10.1145/2037820.2037822 · Cited by 63 · see citing works
Self-citationBerlá, E. P., & Butterfield, L. H. (1977) · The Journal of Special Education · doi:10.1177/002246697701100309 · Cited by 54 · see citing works
Gesture-Based Interactive Audio Guide on Tactile Reliefs
Reichinger, A., Fuhrmann, A., Maierhofer, S., & Purgathofer, W. (2016) · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982176 · Cited by 52 · see citing works
Self-citationTactile graphics with a voice: using QR codes to access text in tactile graphics
Baker, C. M., Milne, L. R., Scofield, J., Bennett, C. L., & Ladner, R. E. (2014) · Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility - ASSETS '14 · doi:10.1145/2661334.2661366 · Cited by 51 · see citing works
The 65th IFLA Council and General Conference
Division, C. N., & Bang, T. (1999) · Interlending & Document Supply · doi:10.1108/ilds.1999.12227bab.012 · Cited by 50 · see citing works
Scientific diagrams made easy with iveotm
Gardner, J. A., & Bulatov, V. (2006) · Lecture Notes in Computer Science · doi:10.1007/11788713_179 · Cited by 49 · see citing works
(2014) · doi:10.5040/9798881816100 · Cited by 48 · see citing works
(2001) · Information technology and disabilities · Cited by 47 · see citing works
From 2D to 2.5D i.e. from painting to tactile model
Furferi, R., Governi, L., Volpe, Y., Puggelli, L., Vanni, N., & Carfagni, M. (2014) · Graphical Models · doi:10.1016/j.gmod.2014.10.001 · Cited by 43 · see citing works
Evaluation of a Portable Artificial Vision Device Among Patients With Low Vision
Moisseiev, E., & Mannis, M. J. (2016) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2016.1000 · Cited by 36 · see citing works
Including people with learning difficulties in cultural and heritage sites
Rix, J., Lowe, T., & Forum, T. H. (2010) · International Journal of Heritage Studies · doi:10.1080/13527251003620743 · Cited by 34 · see citing works
Enlarged Skeleton Models of Plankton for Tactile Teaching
Teshima, Y., Matsuoka, A., Fujiyoshi, M., Ikegami, Y., Kaneko, T., Oouchi, S., Watanabe, Y., & Yamazawa, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14100-3_78 · Cited by 32 · see citing works
Computer-Aided Design of Tactile Models
Reichinger, A., Neumüller, M., Rist, F., Maierhofer, S., & Purgathofer, W. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31534-3_73 · Cited by 31 · see citing works
Self-citationEasy-to-Use and Accurate Calibration of RGB-D Cameras from Spheres
Staranowicz, A. N., Brown, G. R., Morbidi, F., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-53842-1_23 · Cited by 29 · see citing works
Shi, L., McLachlan, R., Zhao, Y., & Azenkot, S. (2016) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2982142.2982153 · Cited by 29 · see citing works
Shen, H., Edwards, O., Miele, J. A., & Coughlan, J. M. (2013) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2513383.2513423 · Cited by 28 · see citing works
Brock, A., Lebaz, S., Oriola, B., Picard, D., Jouffrais, C., & Truillet, P. (2012) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2212776.2223821 · Cited by 23 · see citing works
Calibration of Kinect-type RGB-D sensors for robotic applications
Karan, B. (2015) · FME Transaction · doi:10.5937/fmet1501047k · Cited by 20 · see citing works
Oouchi, S., Yamazawa, K., & Secchi, L. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14100-3_79 · Cited by 15 · see citing works
Making Blind People Autonomous in the Exploration of Tactile Models: A Feasibility Study
Buonamici, F., Furferi, R., Governi, L., & Volpe, Y. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-20681-3_8 · Cited by 14 · see citing works
Tactile 3D Bas-relief from Single-point Perspective Paintings: A Computer Based Method
Furferi, R. (2014) · Journal of Information and Computational Science · doi:10.12733/jics20104135 · Cited by 12 · see citing works
A Concept for Re-useable Interactive Tactile Reliefs
Reichinger, A., Fuhrmann, A. L., Maierhofer, S., & Purgathofer, W. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-41267-2_15 · Cited by 8 · see citing works
Self-citationYamazawa, K., Teshima, Y., Watanabe, Y., Ikegami, Y., Fujiyoshi, M., Oouchi, S., & Kaneko, T. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31534-3_74 · Cited by 7 · see citing works
Kunstvermittlung 2.0: Neue Medien und ihre Potenziale
Hausmann, A., & Frenzel, L. (2014) · Springer eBooks · doi:10.1007/978-3-658-08346-5 · Cited by 6 · see citing works
Museum ohne Grenzen – Multimediale Anwendungen und Barrierefreiheit in der Berlinischen Galerie
Brinkmeyer, D. (2014) · Springer eBooks · doi:10.1007/978-3-658-02869-5_7 · Cited by 2 · see citing works
Planar Object Tracking in the Wild: A BenchmarkPreprint
Liang, P., Wu, Y., Lu, H., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07938 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [5]5. Gary Bradski and Adrian Kaehler. 2008. Learning OpenCV: Computer Vision with the OpenCV Library. O'Reilly Media, Inc., Sebastopol, CA.
- [7]7. Anke Brock, Samuel Lebaz, Bernard Oriola, Delphine Picard, Christophe Jouffrais, and Philippe Truillet. 2012. Kin'touch: Understanding how visually impaired people explore tactile maps. In Extended Abstracts on Human Factors in Computing Systems (CHI'12). ACM, 2471-2476.
- [20]20. Paolo Gualandi and Loretta Secchi. 2000. Tecniche di rappresentazione plastica della realtà visiva. In Toccare L'arte: L'educazione Estetica Di Ipovedenti e Non Vedenti, Andreas Bellini (Ed.). Armando Editore, 49-98.
- [23]23. Shaun K. Kane, Brian Frey, and Jacob O. Wobbrock. 2013. Access lens: A gesture-based screen reader for real-world documents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 347-350.
- [47]47. Huiying Shen, Owen Edwards, Joshua Miele, and James M. Coughlan. 2013. Camio: A 3D computer vision system enabling audio/haptic interaction with physical objects by blind users. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS'13). ACM, 41.
Cited by
Sorted by citations
Cavazos Quero, L., Iranzo Bartolomé, J., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10030297 · Cited by 86 · see citing works
Review of Quantitative Empirical Evaluations of Technology for People with Visual Impairments
Brulé, E., Tomlinson, B. J., Metatla, O., Jouffrais, C., & Serrano, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376749 · Cited by 76 · see citing works
- Authors
- Reichinger, A., Carrizosa, H. G., Wood, J., Schröder, S., Löw, C., Luidolt, L. R., Schimkowitsch, M., Fuhrmann, A., Maierhofer, S., & Purgathofer, W.
- Affiliations
- VRVis GmbH (Austria) (AT)
- Open University, Milton Keynes, United Kingdom
- University of Sussex (GB)
- University of Vienna (AT)
- ORCID
- Venue
- ACM Transactions on Accessible Computing
- Published
- 2018-03-22
- Volume
- 11
- Issue
- 1
- Pages
- 1-39
- ISSN
- 1936-7228 (ISSN-L)
- 1936-7236
- OpenAlex
- W2790827359
- Cited by
- 34
- References
- 50
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 3 · 2023: 6 · 2022: 4 · 2021: 7 · 2020: 3 · 2019: 6 · 2018: 2
- FWCI
- 2.68
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Accessible Computing (publishedVersion)
- Open Research Online (The Open University) (acceptedVersion)
- Funders
- Erasmus+ (2014-1AT01-KA204-001014)
- European Union's Horizon 2020 research and innovation programme (693229)
- Austrian Science Fund (P24352-N23)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- gesture control, tactile accessibility, visually impaired users, multimodal accessibility, verbal description, assistive technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
