The Tangible Desktop: A Multimodal Approach to Nonvisual Computing
The DOI matches its Crossref record, with a small difference in title or year; 56 of the 58 references it cites were found in a registry or an index.
Abstract
Audio-only interfaces, facilitated through text-to-speech screen reading software, have been the primary mode of computer interaction for blind and low-vision computer users for more than four decades. During this time, the advances that have made visual interfaces faster and easier to use, from direct manipulation to skeuomorphic design, have not been paralleled in nonvisual computing environments. The screen reader–dependent community is left with no alternatives to engage with our rapidly advancing technological infrastructure. In this article, we describe our efforts to understand the problems that exist with audio-only interfaces. Based on observing screen reader use for 4 months at a computer training school for blind and low-vision adults, we identify three problem areas within audio-only interfaces: ephemerality, linear interaction, and unidirectional communication. We then evaluated a multimodal approach to computer interaction called the Tangible Desktop that addresses these problems by moving semantic information from the auditory to the tactile channel. Our evaluation demonstrated that among novice screen reader users, Tangible Desktop improved task completion times by an average of 6 minutes when compared to traditional audio-only computer systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As modern computer applications all have two-dimensional graphical user interfaces primarily designed for sighted interaction, there is an inherent mismatch with the one-dimensional navigation supported by a screen reader, thereby resulting in numerous usability issues as uncovered by the findings of prior studies [9, 30, 46, 49, 50].”
From Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models (2024) · cites this paper as [9]
In the citing paper
“Increasing tactile feedback and tangible interaction on touchscreens can improve their accessibility.”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
In the citing paper
“consisted of many hardware components that were difficult to organize or carry [4]”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
In the citing paper
“or not focused on mobile systems [4,20]”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
In the citing paper
“The absence of tactile cues such as keys and buttons makes touchscreens difficult to navigate for people with visual impairments.”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [4]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Tangible bits: towards seamless interfaces between people, bits and atoms
Ishii, H., & Ullmer, B. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258715 · Cited by 3,853 · cited 7×
“Individually, concepts such as positional constraints [32], haptics [9, 55], and physical icons [20] have been shown to be meaningful for nonvisual and sighted users alike.”
BackgroundMethodInfluentialHollan, J., Hutchins, E., & Kirsh, D. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/353485.353487 · Cited by 2,300
Nielsen, J. (1994) · Cited by 1,897
Earcons and Icons: Their Structure and Common Design Principles
Blattner, M., Sumikawa, D., & Greenberg, R. (1989) · Human-Computer Interaction · doi:10.1207/s15327051hci0401_1 · Cited by 980
"Constant, constant, multi-tasking craziness": managing multiple working spheres
González, V. M., & Mark, G. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985707 · Cited by 715
Tactons: structured tactile messages for non-visual information display
Brewster, S. A., & Brown, L. M. (2004) · Enlighten: Publications (University of Glasgow) · Cited by 697
As-rigid-as-possible shape manipulation
Igarashi, T., Moscovich, T., & Hughes, J. F. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073323 · Cited by 675
TeslaTouch: electrovibration for touch surfaces
Bau, O., Poupyrev, I., Israr, A., & Harrison, C. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866074 · Cited by 664
Tangible User Interfaces: Past, Present, and Future Directions
Shaer, O. (2009) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000026 · Cited by 581
Tangible User Interfaces: Past, Present, and Future Directions
Shaer, O. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601983299 · Cited by 414
Research works printed but not linked: 3
- Ängeslevä Jussi, 2003, Body mnemonics portable device interaction design concept, Proceedings of UIST, 3, 2
- Brewster Stephen, Proceedings of the Australasian User Interface Conference (AUIC’04)
- Vitense Holly S., Proceedings of the 5th International ACM Conference on Assistive Technologies. ACM, 49–56
Cited by
A Place to Play: The (Dis)Abled Embodied Experience for Autistic Children in Online Spaces
Ringland, K. E. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300518 · Cited by 71
Editing spatial layouts through tactile templates for people with visual impairments.
Li, J., Kim, S., Miele, J. A., Agrawala, M., & Follmer, S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300436 · Cited by 62
“They have employed 3D printing to let blind users access works of art [39], STEM education [11, 20], and traditional WIMP desktop interface elements [4].”
MethodFunctional programming for compiling and decompiling computer-aided design
Nandi, C., Wilcox, J. R., Panchekha, P., Blau, T., Grossman, D., & Tatlock, Z. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3236794 · Cited by 64
Li, F. M., Chen, D. L., Fan, M., & Truong, K. N. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445321 · Cited by 45
Examining Visual Semantic Understanding in Blind and Low-Vision Technology Users
Potluri, V., Grindeland, T. E., Froehlich, J. E., & Mankoff, J. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445040 · Cited by 42
“For example, studies of beginning users highlight challenges with understanding visual context, such as when the cursor has switched to a new application [8].”
BackgroundSelf-citation
Metadata
- Authors
- Baldwin, M. S., Hayes, G. R., Haimson, O. L., Mankoff, J., & Hudson, S. E.
- Affiliations
- University of California, Irvine (US)
- Carnegie Mellon University (US)
- Venue
- ACM Transactions on Accessible Computing
- Published
- 2017-08-11
- Volume
- 10
- Issue
- 3
- Pages
- 1-28
- ISSN
- 1936-7228 (ISSN-L)
- 1936-7236
- OpenAlex
- W2744896498
- Cited by
- 37
- References
- 58
- Citations by year
- 2026: 1 · 2025: 7 · 2024: 6 · 2023: 1 · 2022: 3 · 2021: 4 · 2020: 5 · 2019: 6 · 2018: 3 · 2017: 1
- FWCI
- 2.35
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Accessible Computing (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- screen readers, blind and visually impaired users, tactile feedback, multimodal interaction, transience, linear interaction, one-way communication, assistive technology, task completion time
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Tangible Desktop: A Multimodal Approach to Nonvisual Computing. https://app.cochat.ai/papers/10.1145/3075222
- The paper
- Baldwin, M. S., Hayes, G. R., Haimson, O. L., Mankoff, J., & Hudson, S. E. (2017). The Tangible Desktop: A Multimodal Approach to Nonvisual Computing. ACM Transactions on Accessible Computing. https://doi.org/10.1145/3075222
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.