CoChat
Sign Up
Cited by 24Digital Accessibility for DisabilitiesAssistive Technology in Communication and MobilityTactile and Sensory Interactions

Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models

Open access
Kodandaram, Satwik Ram, Uckun, Utku, Bi, Xiaojun, Ramakrishnan, IV, Ashok, Vikas·2024·The 26th International ACM SIGACCESS Conference on Computers and Accessibility
DOI 10.1145/3663548.3675605PMC PMC11707650
Open this paper in CoChat — free accountOpen PDF

Abstract

Blind individuals, who by necessity depend on screen readers to interact with computers, face considerable challenges in navigating the diverse and complex graphical user interfaces of different computer applications. The heterogeneity of various application interfaces often requires blind users to remember different keyboard combinations and navigation methods to use each application effectively. To alleviate this significant interaction burden imposed by heterogeneous application interfaces, we present Savant, a novel assistive technology powered by large language models (LLMs) that allows blind screen reader users to interact uniformly with any application interface through natural language. Novelly, Savant can automate a series of tedious screen reader actions on the control elements of the application when prompted by a natural language command from the user. These commands can be flexible in the sense that the user is not strictly required to specify the exact names of the control elements in the command. A user study evaluation of Savant with 11 blind participants demonstrated significant improvements in interaction efficiency and usability compared to current practices.

AI snapshot

Written from the paper’s full text.

TL;DR

Savant is an LLM-powered assistive layer that lets blind screen-reader users issue flexible natural-language commands to operate controls across desktop applications, automating the screen-reader action sequence. In a study with 11 blind participants, it improved completion, speed, key-press burden, perceived ease, and workload relative to JAWS alone and an AccessBolt pop-up interface.

Key findings

  • In the user study, task completion was highest with Savant: 18 of 22 tasks, versus 14 of 22 with AccessBolt and 6 of 22 with JAWS alone.
  • Average task time was 101 seconds with Savant, compared with 159 seconds for AccessBolt and 395 seconds for JAWS; average key presses were 44, 163, and 319, respectively.
  • Perceived ease was higher with Savant (mean SEQ 6.57) than AccessBolt (5.6) or JAWS (2), while NASA-TLX workload was lower with Savant (mean 16.16) than AccessBolt (28.93) or JAWS (49.9).
  • The Gen LLM mapped natural-language commands to control-value pairs with 78% accuracy (113 of 145 commands) on the Wizard-of-Oz dataset; the curated few-shot dataset contained about 1,200 examples across 11 applications.

Methods

  • The system uses a preprocessing stage to crawl application controls via Microsoft UI Automation, build an Application Control Tree, and generate a few-shot examples dataset with GPT-4; at runtime, GPT-4 with BERT-based retrieval interprets commands and an Action Relay executes screen-reader steps.
  • A Wizard-of-Oz study with 11 blind participants across Excel, Gmail, File Explorer, Word, and Zoom produced 145 natural-language commands annotated with control-value pairs for training and testing.
  • An IRB-approved within-subject user study with 11 blind JAWS users compared JAWS-only, AccessBolt, and Savant across six applications, using counterbalanced ordering, task completion, time, key presses, SEQ, NASA-TLX, and qualitative coding.

Limitations

  • Paper-stated limitations include inability to handle complex multi-instruction commands, pop-up and sub-window controls, tasks spanning multiple applications, and the lack of wake-word activation; multilingual support and chain-of-thought prompting are proposed as future work.
  • The paper also reports that errors arose from speech-to-text mistakes and ambiguous or composite utterances, and that some Savant failures were due to LLM misattribution of commands.
  • Evident from the text: the evaluation involved a small sample of 11 participants, a limited set of applications and tasks, and no overlap with the earlier WOZ sample, which may constrain generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3663548.3675605

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 32 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Blind users are frequently about ten times slower than sighted users on similar computer tasks.

    “with research indicating that they are frequently ten times slower than sighted individuals when performing similar computer tasks [13, 26, 46].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [26] source

      The Abstract says blind screen-reader users generally need significantly more time than sighted peers for simple application tasks. It gives no tenfold estimate. The Evaluation > Results compares two input methods used by blind participants, not blind and sighted participants. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [46] source
  • Premisecitation

    Blind people often have difficulty performing basic tasks in computer applications.

    “As a result, blind people often face difficulties in performing basic tasks (such as accessing ribbon commands) in applications [24].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • Rests on an assumption: Blind individuals rely on screen readers to use computers.
  • Premisecitation

    Assistive technology research for improving computer application usability has largely focused on web and mobile applications.

    “However, these works have largely centered on the web [6, 12, 15, 16, 43] and mobile applications [11, 22, 27–29].”
    Quote in the paperSubstantial claim6 of 10 sources checked
    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [12] source
    • cites [15] source
      Experimental — accuracy not yet measured
    • cites [16] source
      Experimental — accuracy not yet measured
    • cites [43] source
      Experimental — accuracy not yet measured
    • cites [11] source
      Experimental — accuracy not yet measured
    • cites [22] source
      Experimental — accuracy not yet measured
    • cites [27] source
    • cites [28] source
      Experimental — accuracy not yet measured
    • cites [29] source
  • Findingown data

    User experience improvements were statistically significant between JAWS and AccessBolt (Wilcoxon W=2.0, p=7.36e−5).

    “A Wilcoxon test between JAWS and AccessBolt showed that the improvements in user experience were statistically significant (W=2.0, p=7.36e−5).”
    Quote in the paperSubstantial claim
  • Findingown data

    In the JAWS condition, a key reason for task difficulty was invisible controls that required opening a container control first.

    “A closer analysis of the user behavior during the tasks revealed that in the JAWS condition, one of the main reasons the participants struggled to complete the tasks was because of invisible controls that could be accessed only by opening up the corresponding ‘container’ control.”
    Quote in the paperSubstantial claim
  • May combine two claims. 32
  • Classified as probably not a substantial claim. 11
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 3

  • May combine two claims.

    Premisecitation

    A prior study found that blind participants valued uniform access across applications, screen readers, and platforms, and all participants considered its absence a significant drawback.

    “In a previous study, blind participants emphasized the importance of having uniform and consistent access independent of the specific application, screen readers, and platforms [10], and the absence of this type of access in contemporary assistive technology was considered a significant drawback by all study participants.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] source

      The Discussion (Uniformity of interaction experience; page unavailable) identifies consistent access across screen readers and platforms as a recurring need, and the Conclusion extends that need to applications and operating systems. The paper says all participants wanted a portable screen reader, but it does not say that all participants judged the absence of uniform access across applications, screen readers, and platforms a significant drawback. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Billah et al. observed that differences in shortcut mappings between screen readers and between applications caused confusion and interaction problems for blind screen reader users.

    “In another study, Billah et al. [10] observed that differences in shortcut mappings between different screen readers and between different applications generated confusion and interaction problems for blind screen reader users.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] source

      The Findings (Switching Devices, Platforms, and Screen Readers; page unavailable) report that screen readers have differing shortcuts and that switching between them is disruptive. The Discussion (page unavailable) links such changes to navigational confusion. The paper calls some shortcuts application-specific, but the cited passages do not establish that differing shortcut mappings between applications caused the reported confusion. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • Borderline on S1 (asserted), S3 (load-bearing), S4 (based): unclear from the text.

    Findingnone given

    After cleaning, the FED contained 1,200 high-quality few-shot examples.

    “After cleaning the data set, we converged on 1, 200 high-quality few-shot examples in the FED.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The FED contains few-shot examples for 11 different applications and all controls of these applications.

    “The FED consists of few-shot examples for 11 different applications (see Appendix A.4) and for all the controls of these applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Gen LLM's accuracy on the WOZ dataset was 78%, correctly interpreting 113 of 145 commands.

    “The overall accuracy of our Gen LLM was 78% (113 out of 145 NLCs).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ground-truth <CE, Value(CE)> pairs in the WOZ study dataset are accurate.
    • Rests on an assumption: The BERT-based semantic similarity search retrieves few-shot examples relevant enough to ensure Gen LLM produces an accurate <CE, Value(CE)> pair.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Smaller applications like Notepad provided about 50 examples, while larger applications like Excel provided close to 400 examples.

    “Smaller applications, such as Notepad, provided roughly 50 examples, whereas larger applications, like Excel, offered close to 400 examples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Average key presses were 319 for JAWS, 163 for AccessBolt, and 44 for Savant.

    “On average, the participants pressed 319 keys under JAWS condition (median=279, SD=132.8), 163 keys under AccessBolt condition (median=115, SD=189.1) and 44 under Savant condition (median=20, SD=51.9).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Task completion counts were 6 of 22 for JAWS, 14 of 22 for AccessBolt, and 18 of 22 for Savant.

    “In the JAWS condition, the participants completed only 6 out of the 22 tasks. The completion rate in the AccessBolt condition was much higher (14 out of the 22 tasks), however it was the highest in the Savant condition (18 out of 22 tasks).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The tasks chosen for the user study are representative and useful for screen reader users.

  • May combine two claims.

    Findingown data

    In the Savant condition, 11 of 22 tasks succeeded on the first attempt, 7 succeeded after multiple tries, and 4 remained incomplete.

    “Out of 22 tasks assigned to the participants in the Savant condition, 11 were successfully completed on the frst attempt (i.e., the LLM correctly identified the <CE, Value(CE)> pair), while 7 were completed after multiple tries. This repeated interaction contributed to longer completion times. 4 tasks remained incomplete despite repeated attempts by the participants.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When commands were ambiguous, the system sometimes presented multiple options, and participants spent extra time choosing the correct control.

    “Occasionally, due to ambiguity in the given natural language command, the system presented multiple options to the user. In these instances, participants spent additional time determining the correct control element.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In a Wizard-of-Oz study with 11 blind participants, the authors collected 145 natural language commands, each manually annotated with the corresponding <CE, Value(CE)> pair.

    “Overall, from the WOZ study, we collected a total 145 NLCs, and each NLC was manually annotated with the <CE, Value(CE)> pair of the corresponding application.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The natural language commands collected in the Wizard-of-Oz study are representative of the commands blind users would give while interacting with computer applications.

  • May combine two claims.

    Findingown data

    Average SEQ ratings were 2 for JAWS, 5.6 for AccessBolt, and 6.57 for Savant.

    “On average, the tasks got a rating of 2 under JAWS condition (median=2, SD=1.19), 5.6 under AccessBolt (median=6, SD=1.61) and 6.57 under Savant (median=7, SD=0.80).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Basic navigation keys (Tab and Arrow) made up 91% of key presses under JAWS, 70% under AccessBolt, and 30% under Savant.

    “The analysis revealed that in the JAWS condition, a striking 91% of the total key presses were these basic navigation keys. In contrast, this percentage decreased to 70% in the AccessBolt condition and dropped even further to 30% in the Savant condition.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    The few-shot examples covered diverse applications and a broad range of command types.

    “These few-shot examples were general and varied in several aspects. First, they covered a number of diverse applications, such as file management (Explorer), text editing (Wordpad, Notepad, and Microsoft Word), video conferencing (Zoom), music streaming (Spotify), and email (Outlook).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The natural language commands collected in the Wizard-of-Oz study are representative of the commands blind users would give while interacting with computer applications.

  • May combine two claims.

    Findingown data

    Average task time was 395 seconds for JAWS, 159 seconds for AccessBolt, and 101 seconds for Savant.

    “On average, the participants spent 395 seconds under JAWS condition (median=420, SD=61.1), 159 seconds under AccessBolt condition (median=148.5, SD=69.6), and 101 seconds under Savant condition (median=98, SD=49.5) to do the tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Friedman's test for NASA-TLX gave chi-square 18.18 (df=2) and p=0.00011, significant at p < .05.

    “Friedman’s test for NASA-TLX produced a chi-square statistic of 18.18 (df = 2) and a p-value of 0.00011, implying statistical significance at p < .05.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    NASA-TLX scores were 16.16 for Savant, 28.93 for AccessBolt, and 49.9 for JAWS.

    “The TLX score for Savant(mean=16.16, median=15.0, SD=8.57)) condition was significantly lower than for AccessBolt (mean=28.93, median=29.16, SD=5.09) and JAWS (mean=49.9, median=45.8, SD=20.9) conditions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In JAWS, Effort and Frustration subscales drove higher workload; in AccessBolt, Effort was the discriminating subscale; in Savant, ratings were uniform across all six subscales.

    “A closer inspection of the responses to individual subscales of the TLX questionnaire revealed that the Effort and Frustration subscales contributed more toward the relatively higher workload scores in the JAWS condition, i.e., the ratings for these subscales were significantly higher than those for the other subscales (Temporal Demand, Physical Demand, Mental Demand, and Overall Performance). For”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In a user study with 11 blind participants, Savant produced significant improvements in perceived usability (about three times higher SEQ) and task workload (about three times lower NASA-TLX) compared with status-quo screen readers for typical computer tasks involving controls in diverse applications.

    “Savant showed significant improvements in both perceived usability (3 times higher SEQ score) and task workload (3 times lower NASA-TLX score) over status-quo screen readers, for typical computer tasks that involve accessing controls in a diverse set of applications.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Most Savant time went to LLM processing of the natural language command, with some time for pipeline execution.

    “Most of the time spent in the Savant condition was allocated to processing the natural language command by the LLM, with a portion dedicated to executing the entire pipeline.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Two causes of incorrect responses were identified: complex multi-instruction commands and automatic speech-to-text conversion errors.

    “Upon analyzing the cases where the system produced incorrect responses, two reasons were identified. Firstly, some user commands were complex (e.g., “Select this cell and add a note”), consisting of multiple instructions within a single command. These commands often required several steps and involved accessing multiple controls. Our system is presently not equipped to handle a complex command and”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participants P6 and P9 liked that Savant used the same command interface across all applications.

    “Participants P6 and P9 also pointed out that they liked how Savant had the same command interface for all applications, which they thought was a cool idea.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All participants agreed they had to keep practicing keyboard shortcuts or had forgotten how to do specific tasks.

    “One point all participants agreed upon was that they had to keep practicing keyboard shortcuts or that they had forgotten how to do specific tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Leveraging LLM capabilities in assistive technologies is relatively unexplored.

    “However, leveraging their capabilities in assistive technologies, as exemplified by the Savant system outlined in this paper, is a relatively unexplored topic.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Savant lets a user use the same command to adjust margins in other similar applications such as WordPad and Pages, removing the need to remember distinct shortcuts.

    “Finally, Savant enables Joe to use the same command to adjust margins in other similar applications (e.g., WordPad, Pages) as well, thereby obviating the need to remember distinct shortcuts for these applications.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Large language models are trained on large datasets and can understand, analyze, and generate natural language and other content for many tasks.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Condition and application order was counterbalanced across participants using a Latin square.

    “The ordering of the conditions and applications across participants was counterbalanced using the Latin square method6.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Savant shields users from low-level screen-reader operations and GUI-navigation intricacies, enabling more convenient and uniform interaction via flexible natural language commands.

    “In contrast, Savant shields the user from all these low-level screen-reader operations and related GUI-navigation intricacies, thereby enabling a more convenient and uniform interaction via flexible natural language commands for blind screen-reader users.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Large language models are trained on large datasets and can understand, analyze, and generate natural language and other content for many tasks.

  • May combine two claims.

    Interpretationown data

    In JAWS and AccessBolt, participants mainly used navigation keys to search for controls; in Savant, navigation key use was lower because natural language input was available.

    “In the JAWS and AccessBolt conditions, participants predominantly relied on navigation keys to search for the target control either within the application GUI or the AccessBolt’s popup interface. On the other hand, in the Savant condition, the use of navigation keys was naturally lesser due to the availability of natural language input alternatives.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Shinohara et al. found that a single accidental key press could completely disorient a blind user in an application UI, and recovery often required substantial mental effort and time.

    “Their study notably revealed that even a single accidental key press could utterly disorient a blind user in the application UI, and moreover, recovery from such errors often entailed significant mental effort and time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] source

  • The quote only partly states this claim.

    Interpretationreasoning

    The diversity of applications and command types ensured the model could generalize across various computer applications and functionalities.

    “The diversity of applications ensured the model’s ability to generalize across various computer applications and functionalities.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Large language models are trained on large datasets and can understand, analyze, and generate natural language and other content for many tasks.
    • Rests on an assumption: The natural language commands collected in the Wizard-of-Oz study are representative of the commands blind users would give while interacting with computer applications.

  • May combine two claims.

    Premisereasoning

    Application-specific assistants are tightly bound to their host application and closed source, making it difficult to replicate them for general use across all applications.

    “The obvious drawbacks of these assistants are that they are tightly bound to their host application and moreover the underlying technologies are closed source, therefore, it is difficult if not impossible to replicate these assistants for general use across all applications.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Research prototypes for natural language assistance support multiple applications but target either web or mobile applications.

    “A common aspect of these aforementioned research prototypes is that they target either web or mobile applications.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [4] source
    • cites [52] source

  • May combine two claims.

    Interpretationown data

    Savant removed the search process; participants could issue a natural language command and control access was automated.

    “In contrast, the Savant condition eliminated this search process; participants could simply issue a natural language command, and access to the control was automated.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Extant web and mobile prototypes cannot be readily adapted for desktop usage because of heterogeneity in underlying software implementations and technologies.

    “Extant prototypes for web and mobile cannot be readily adapted for desktop usage due to the inherent heterogeneity in the underlying software implementations and technologies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The earlier WOZ study and this user study had no participants in common, which supports external validity.

    “There was no overlap between the participant pools of the earlier WOZ study and this study, thereby promoting external validity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Modern computer applications have two-dimensional GUIs designed primarily for sighted interaction, which mismatches the one-dimensional navigation of screen readers and causes usability issues.

    “As modern computer applications all have two-dimensional graphical user interfaces primarily designed for sighted interaction, there is an inherent mismatch with the one-dimensional navigation supported by a screen reader, thereby resulting in numerous usability issues as uncovered by the findings of prior studies [9, 30, 46, 49, 50].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [9] source
    • cites [30] source
    • cites [46] source
    • cites [49] source
    • cites [50] source
    • Rests on an assumption: Modern computer applications have complex, densely packed graphical interfaces designed mainly for visual mouse interaction rather than keyboard shortcuts.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Each few-shot example was independently assessed by two research team members for accuracy, and erroneous examples were discarded.

    “Each example was independently assessed by two members of our research team for the accuracy of the generated few-shot examples. Examples found to be erroneous were discarded.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Using a design process with user feedback, the paper demonstrated notable improvements in computer interaction efficiency and desktop application usability.

    “By employing a design process that incorporated user feedback, we have successfully showcased notable improvements in the efficiency of computer interactions and the usability of desktop applications.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The user study participants' performance and feedback are representative of blind screen reader users generally.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This progress represents a favorable step toward a more inclusive digital environment for people with visual impairments.

    “This progress signifies a favorable advancement towards creating a digital environment that is more inclusive for individuals with visual impairments.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The user study participants' performance and feedback are representative of blind screen reader users generally.

  • May combine two claims.

    Interpretationown data

    Savant can automatically access and invoke all application controls on the user's behalf using LLMs, enabling uniform task performance across arbitrary desktop applications.

    “We proposed Savant interface that can automatically access and invoke all the application controls on the user’s behalf by leveraging large language models (LLM), thereby allowing users to perform tasks uniformly across arbitrary desktop applications.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The user study participants' performance and feedback are representative of blind screen reader users generally.
40 of 40

  • The paper relies on this without support.

    Blind individuals rely on screen readers to use computers.

    “People who are blind depend on assistive technology, specifically a screen reader (e.g., JAWS1, VoiceOver2, and NVDA3), to interact with computer applications.”

  • The paper relies on this without support.

    Modern computer applications have complex, densely packed graphical interfaces designed mainly for visual mouse interaction rather than keyboard shortcuts.

    “However, modern computer applications feature complex and densely packed graphical user interfaces (GUIs) that are mainly designed for visual point-and-click interactions using a mouse, rather than relying on keyboard shortcuts, the principal method of interaction for blind individuals.”

  • The paper relies on this without support.

    Large language models are trained on large datasets and can understand, analyze, and generate natural language and other content for many tasks.

    “LLMs belong to a class of foundational models that are trained on vast amounts of data, enabling them to comprehend, analyze, and produce natural language and various other forms of content for a wide variety of tasks.”

  • The paper relies on this without support.

    The natural language commands collected in the Wizard-of-Oz study are representative of the commands blind users would give while interacting with computer applications.

    “To understand the types of NLCs that Savant can expect to receive from blind users while they interact with computer applications, we conducted a Wizard-of-Oz (WOZ) study with 11 blind participants.”

  • The paper relies on this without support.

    The ground-truth <CE, Value(CE)> pairs in the WOZ study dataset are accurate.

    “For each example in the WOZ dataset, we fed the NLC as input to our model to generate the output of the <CE, Value(CE)> pair, which was then compared to the ground truth pair in the data set.”

  • The paper relies on this without support.

    The tasks chosen for the user study are representative and useful for screen reader users.

    “We selected these tasks as representative and useful for screen reader users.”

  • The paper relies on this without support.

    The BERT-based semantic similarity search retrieves few-shot examples relevant enough to ensure Gen LLM produces an accurate <CE, Value(CE)> pair.

    “Using word embedding to retrieve relevant few-shot examples is crucial to ensure that Gen LLM generates a <CE, Value(CE)> pair that is an accurate refection of the interpretation of user’s NLC.”

  • The paper relies on this without support.

    The user study participants' performance and feedback are representative of blind screen reader users generally.

    “We recruited 11 blind screen reader users to participate in the study.”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting accounts of when NASA-TLX was administered

    Minor

    The procedure says the questionnaire was administered after all tasks, whereas the measurements section says it was administered after each condition. Participants completed tasks under three conditions, so these describe different administration schedules.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 7

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Vikas Ganjigunte Ashok. 2018. Non-Visual Web Browsing: From Accessibility with Screen Readers to Usability with Assistants. Ph. D. Dissertation. State University of New York at Stony Brook.
  3. Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Giulio Mori. 2012. Designing e-learning collaborative tools for blind people. E-Learning-Long-Distance and Lifelong Perspectives (2012), 125–144.
  4. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183.
  5. 10.1109/SMC42975.2020.9283015 · DOI · Paper page
  6. Giulio Mori, Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Victor MR Penichet. 2011. Making “Google Docs” user interface more accessible for blind people. In Advances in New Technologies, Interactive Interfaces, and Communicability: First International Conference, ADNTIIC 2010, Huerta Grande, Argentina, October 20-22, 2010, Revised Selected Papers 1. Springer, 20–29.
  7. John Gerard Schoeberlein and Yuanqiong Wang. 2014. Usability Evaluation of an Accessible Collaborative Writing Prototype for Blind Users.Journal of Usability Studies 10, 1 (2014).

Cited by

Sorted by citations

Related

Authors
Kodandaram, S. R., Uckun, U., Bi, X., Ramakrishnan, IV, & Ashok, V.
Affiliations
  • Stony Brook University (US)
  • Old Dominion University (US)
ORCID
Venue
The 26th International ACM SIGACCESS Conference on Computers and Accessibility
Publisher
ACM
Published
2024-10-27
Volume
2024
Pages
1-14
DOI
10.1145/3663548.3675605
PMC
PMC11707650
OpenAlex
W4403576958
Semantic Scholar
610a6c4f560dc2d6472cd158ec916a2d0d64d29f
Cited by
24
References
34
Citations by year
2026: 13 · 2025: 11
FWCI
13.06
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2113485, 2153056, 2125147)
  • National Institutes of Health
  • National Eye Institute (R01EY030085, R01EY035688)
  • Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD097188)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
Keywords
blind users, screen readers, large language models, assistive technology, natural language commands, savants, keyboard navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review