The Tangible Desktop: A Multimodal Approach to Nonvisual Computing
The DOI matches its Crossref record, with a small difference in title or year; 56 of the 58 references it cites were found in a registry or an index.
Abstract
Audio-only interfaces, facilitated through text-to-speech screen reading software, have been the primary mode of computer interaction for blind and low-vision computer users for more than four decades. During this time, the advances that have made visual interfaces faster and easier to use, from direct manipulation to skeuomorphic design, have not been paralleled in nonvisual computing environments. The screen reader–dependent community is left with no alternatives to engage with our rapidly advancing technological infrastructure. In this article, we describe our efforts to understand the problems that exist with audio-only interfaces. Based on observing screen reader use for 4 months at a computer training school for blind and low-vision adults, we identify three problem areas within audio-only interfaces: ephemerality, linear interaction, and unidirectional communication. We then evaluated a multimodal approach to computer interaction called the Tangible Desktop that addresses these problems by moving semantic information from the auditory to the tactile channel. Our evaluation demonstrated that among novice screen reader users, Tangible Desktop improved task completion times by an average of 6 minutes when compared to traditional audio-only computer systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As modern computer applications all have two-dimensional graphical user interfaces primarily designed for sighted interaction, there is an inherent mismatch with the one-dimensional navigation supported by a screen reader, thereby resulting in numerous usability issues as uncovered by the findings of prior studies [9, 30, 46, 49, 50].”
From Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models (2024) · cites this paper as [9]
In the citing paper
“Increasing tactile feedback and tangible interaction on touchscreens can improve their accessibility.”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
In the citing paper
“consisted of many hardware components that were difficult to organize or carry [4]”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
In the citing paper
“or not focused on mobile systems [4,20]”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
In the citing paper
“The absence of tactile cues such as keys and buttons makes touchscreens difficult to navigate for people with visual impairments.”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Tangible bits: towards seamless interfaces between people, bits and atoms
Ishii, H., & Ullmer, B. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258715 · Cited by 3,853 · cited 7×
“Individually, concepts such as positional constraints [32], haptics [9, 55], and physical icons [20] have been shown to be meaningful for nonvisual and sighted users alike.”
BackgroundMethodInfluentialHollan, J., Hutchins, E., & Kirsh, D. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/353485.353487 · Cited by 2,300
Nielsen, J. (1994) · Cited by 1,897
Earcons and Icons: Their Structure and Common Design Principles
Blattner, M., Sumikawa, D., & Greenberg, R. (1989) · Human-Computer Interaction · doi:10.1207/s15327051hci0401_1 · Cited by 980
"Constant, constant, multi-tasking craziness": managing multiple working spheres
González, V. M., & Mark, G. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985707 · Cited by 715
Tactons: structured tactile messages for non-visual information display
Brewster, S. A., & Brown, L. M. (2004) · Enlighten: Publications (University of Glasgow) · Cited by 697
As-rigid-as-possible shape manipulation
Igarashi, T., Moscovich, T., & Hughes, J. F. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073323 · Cited by 675
TeslaTouch: electrovibration for touch surfaces
Bau, O., Poupyrev, I., Israr, A., & Harrison, C. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866074 · Cited by 664
Tangible User Interfaces: Past, Present, and Future Directions
Shaer, O. (2009) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000026 · Cited by 581
Tangible User Interfaces: Past, Present, and Future Directions
Shaer, O. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601983299 · Cited by 414
Research works printed but not linked: 3
- Ängeslevä Jussi, 2003, Body mnemonics portable device interaction design concept, Proceedings of UIST, 3, 2
- Brewster Stephen, Proceedings of the Australasian User Interface Conference (AUIC’04)
- Vitense Holly S., Proceedings of the 5th International ACM Conference on Assistive Technologies. ACM, 49–56
Cited by
A Place to Play: The (Dis)Abled Embodied Experience for Autistic Children in Online Spaces
Ringland, K. E. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300518 · Cited by 71
Editing spatial layouts through tactile templates for people with visual impairments.
Li, J., Kim, S., Miele, J. A., Agrawala, M., & Follmer, S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300436 · Cited by 62
“They have employed 3D printing to let blind users access works of art [39], STEM education [11, 20], and traditional WIMP desktop interface elements [4].”
MethodFunctional programming for compiling and decompiling computer-aided design
Nandi, C., Wilcox, J. R., Panchekha, P., Blau, T., Grossman, D., & Tatlock, Z. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3236794 · Cited by 64
Li, F. M., Chen, D. L., Fan, M., & Truong, K. N. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445321 · Cited by 45
Examining Visual Semantic Understanding in Blind and Low-Vision Technology Users
Potluri, V., Grindeland, T. E., Froehlich, J. E., & Mankoff, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445040 · Cited by 42
“For example, studies of beginning users highlight challenges with understanding visual context, such as when the cursor has switched to a new application [8].”
BackgroundSelf-citationNonvisual Interaction Techniques at the Keyboard Surface
Khurana, R., McIsaac, D., Lockerman, E., & Mankoff, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173585 · Cited by 34
Self-citationConsumer-grade fabrication and its potential to revolutionize accessibility
Mankoff, J. C., Hofmann, M. K., Chen, X., Hudson, S. E., Hurst, A. K., & Kim, J. (2019) · Communications of the ACM · doi:10.1145/3339824 · Cited by 46
Self-citationZhang, X., Tran, T., Sun, Y., Culhane, I., Jain, S., Fogarty, J., & Mankoff, J. (2018) · ASSETS '18 · doi:10.1145/3234695.3236349 · Cited by 30
Self-citationTangibleGrid: Tangible Web Layout Design for Blind Users
Li, J., Yan, Z., Jarjue, E., Shetty, A., & Peng, H. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545627 · Cited by 35
“Tangible Desktop [8] further explores the concept by replacing the auditory channel with a set of tangible gadgets, which allows novice screen reader users to have a faster task completion time than audio-only systems.”
BackgroundJiang, C., Kuang, E., & Fan, M. (2025) · ACM Transactions on Accessible Computing · doi:10.1145/3711931 · Cited by 52
TIP-Toy: a tactile, open-source computational toolkit to support learning across visual abilities
Barbareschi, G., Costanza, E., & Holloway, C. (2020) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3417005 · Cited by 30
“This would be a similar strategy to that ofered by the Tangible Desktop for more basic navigation tasks [5].”
MethodEnabling Uniform Computer Interaction Experience for Blind Users through Large Language Models
Kodandaram, S. R., Uckun, U., Bi, X., Ramakrishnan, IV, & Ashok, V. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675605 · Cited by 24
Understanding and Designing for Deaf or Hard of Hearing Drivers on Uber
Lee, S., Hubert‐Wallander, B., Stevens, M., & Carroll, J. M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300759 · Cited by 26
The FlexiBoard: Tangible and Tactile Graphics for People with Vision Impairments
Raynal, M., Ducasse, J., Macé, M., Oriola, B., & Jouffrais, C. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8030017 · Cited by 12
“Baldwin et al. [34] proposed a tangible interface as an alternative to standard audio-only screen readers.”
Dolata, M., Agotai, D., Schubiger, S., & Schwabe, G. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359326 · Cited by 21
Auditory and haptic feedback to train basic mathematical skills of children with visual impairments
Marichal, S., Rosales, A., Perilli, F. G., Pires, A. C., & Blat, J. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2022.2060860 · Cited by 17
“Auditory information may be a crucial complement (Leuders 2016), although sequential auditory feedback cannot convey as much information as the visual channel (Ducasse 2017; Baldwin et al. 2017).”
BackgroundJafri, R., Althbiti, S. M. M., Alattas, N. A. A., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3228455 · Cited by 9
“access [36], exploring geographic maps [37], [38], [39] and navigation [40].”
BackgroundAn Activity Centered Approach to Nonvisual Computer Interaction
Baldwin, M. S., Mankoff, J. C., Nardi, B. A., & Hayes, G. R. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3374211 · Cited by 17
Self-citationA Tangible Block Editor for the Scratch Programming Language
(2021) · doi:10.1145/3411763.3451833 · Cited by 8
An investigation of dynamic crossmodal instantiation in TUIs
Feng, F., & Stockman, T. (2017) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3136755.3136782 · Cited by 6
“Tangible computing, however, has innate physical dimensions such as size, texture, colour, and acoustic features [5, 6, 20], which can also be leveraged to exploit multimodal information channels for natural, accessible and continuous interaction.”
BackgroundSharif, A., Potluri, V., Ang, J. R. X., Wobbrock, J. O., & Mankoff, J. C. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3688505 · Cited by 6
Self-citationLi, F. M., Chen, D. L., Fan, M., & Truong, K. N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09348 · Cited by 50
Accessible Multimodal Tool Support for Brainstorming Meetings
Koutny, R., Günther, S., Dhingra, N., Kunz, A. M., Miesenberger, K., & Mühlhäuser, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58805-2_2 · Cited by 5
“Toaddress these limitations, research has designed a variety of tools that allow for multimodal interfaces (e.g., [2,17,24,31]) and to support collaboration between BVIP and sighted people.”
BackgroundChheda-Kothary, A., Rios, D. A., Smith, K. S., Reyna, A., Zhang, C., & Smith, B. A. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3614490 · Cited by 7
“The importance of learning about visuals in more immersive, spatial ways has long been established by the accessibility research [4, 5, 8– 10, 12, 13, 16, 17, 20–23, 26, 28–31, 33, 41] and BLV communities [6, 40].”
BackgroundEmbodied Measurement: Tangible Interactions to Enhance the Validity of Self-Report Measures
Uhl, J. C., Regal, G., Koesten, L. M., Oppermann, M., Murtinger, M., & Tscheligi, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714055 · Cited by 4
Haptic Feedback to Overcome Barriers for Visually Impaired Users in Digital Audio Workstations
Karpodini, C., Michailidis, T., Creed, C., & Williams, I. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2464899 · Cited by 6
“Audio and haptics often work in synergy improving communication with computer applications when visual interaction is not feasible or preferred (Baldwin et al., 2017).”
Rataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies
Veuskens, T., Luyten, K., & Ramakers, R. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3397329 · Cited by 5
"It Brought Me Joy": Opportunities for Spatial Browsing in Desktop Screen Readers
Chheda-Kothary, A., Sharif, A., Rios, D. A., & Smith, B. A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714125 · Cited by 9
Neuro-Inclusive Design Considerations for Assistive and Accessible Technologies and Beyond
Boyd, L. E. (2024) · Synthesis lectures on technology and health · doi:10.1007/978-3-031-48843-6_8 · Cited by 2
Zhang, S., Masu, R., Bettega, M., & Fan, M. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746466 · Cited by 5
“This method is widely used, and big repositories of Braille music exist (e.g., Braille scores in the International Music Score Library Project (IMSLP) 13 or Braille music collection at the Library of Congress in the United States 14 .).”
Nayak, A. K., Prakash, Y., Jayarathna, S., Lee, H.-N., & Ashok, V. G. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791509 · Cited by 2
Vibration Or Voice? Enhancing EQ Information Accessibility for Blind Music Producers with Haptics
Karpodini, C., Michailidis, T., Creed, C., Stockman, T., & Williams, I. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746374 · Cited by 1
Touchpad Mapper: Exploring Non-Visual Touchpad Interactions for Screen-Reader Users
Sharif, A., Potluri, V., Ang, J. R. X., Wobbrock, J. O., & Mankoff, J. C. (2024) · International Web for All Conference (W4A) · doi:10.1145/3677846.3677867 · Cited by 2
Self-citationFeng, F., & Stockman, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06655
“Tangible interfaces, in a sense, have innate physical dimensions, such as position, colour and acoustic features, that can be leveraged to updating feedback in response to motor activities [31,5,8].”
BackgroundPervasive Desktop Computing by Direct Manipulation of an Augmented Lamp
Chen, Y. (2024) · theses.fr (ABES) · doi:10.70675/b7a45011zf192z4b6dza48az12ae01263fa5
Runtime Repair and Enhancement of Mobile App Accessibility
Zhang, X. (2019) · ResearchWorks at the University of Washington (University of Washington)
Metadata
- Authors
- Baldwin, M. S., Hayes, G. R., Haimson, O. L., Mankoff, J., & Hudson, S. E.
- Affiliations
- University of California, Irvine (US)
- Carnegie Mellon University (US)
- Venue
- ACM Transactions on Accessible Computing
- Published
- 2017-08-11
- Volume
- 10
- Issue
- 3
- Pages
- 1-28
- ISSN
- 1936-7228 (ISSN-L)
- 1936-7236
- OpenAlex
- W2744896498
- Cited by
- 37
- References
- 58
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 6 · 2023: 1 · 2022: 3 · 2021: 4 · 2020: 5 · 2019: 6 · 2018: 3 · 2017: 1
- FWCI
- 2.35
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Accessible Computing (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- screen readers, blind and visually impaired users, tactile feedback, multimodal interaction, transience, linear interaction, one-way communication, assistive technology, task completion time
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.