CoChat
Sign Up
Cited by 52Digital Accessibility for DisabilitiesSoftware Testing and Debugging TechniquesUsability and User Interface Design

AXNav: Replaying Accessibility Tests from Natural Language

Open access
Taeb, Maryam, Swearngin, Amanda, Schoop, Eldon, Cheng, Ruijia, Jiang, Yue, Nichols, Jeffrey·2024·Proceedings of the CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3613904.3642777
Open this paper in CoChat — free account

Abstract

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642777

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most prior work on AI-supported UI evaluation-including the single-pass baseline condition-treats models as automated judges: systems that take a screen as input and output a fixed list of issues, with users interacting mainly through coarse-grained, post-hoc prompts to revise it.

    “Most prior work on AI-supported UI evaluation-including our single-pass baseline condition-treats models as automated judges : systems that take a screen as input and output a fi xed list of issues, with users interacting mainly through coarse-grained, post-hoc prompts to revise it [17, 85, 93].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [85]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Recent generative models can offload parts of UI evaluation: metrics quantify visual properties, predictive models simulate user behaviors, and LLMs detect heuristic violations from screenshots.

    “Metrics can quantify visual properties [71], predictive models can simulate user behaviors [17, 91], and Large Language Models (LLMs) can detect heuristic violations directly from screenshots [17, 85, 93].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [85]

    The Introduction (p. 2) says AXNav replays tasks through accessibility services so testers can see and hear how a user might perform them, supporting the user-behavior portion. But Test Heuristics (p. 9) describes screenshot-based issue detection using a UI-element detection model, OCR, and rule-based matching—not an LLM detecting heuristic violations directly from screenshots. The paper does not support the sentence’s visual-metrics portion, which is cited to a different source. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. [4][4] Abdulaziz Alshayban, Iftekhar Ahmed, and Sam Malek. 2020. Accessibility issues in Android apps: state of affairs, sentiments, and ways forward. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE '20) . Association for Computing Machinery, New York, NY, USA, 1323-1334. https://doi.org/10.1145/3377811.3380392 - · DOI
  2. [5][5] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for HumanAI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3290605.3300233 - · DOI · Paper page
  3. [7][7] Advait Bhat, Saaket Agashe, and Anirudha Joshi. 2021. How do people interact with biased text prediction models while writing?. In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing , Su Lin Blodgett, Michael Madaio, Brendan O'Connor, Hanna Wallach, and Qian Yang (Eds.). Association for Computational Linguistics, Online, 116-121. https://aclanthology.org/2021.hcinlp-1.18 -
  4. [9][9] Jeffrey P. Bigham, Jeremy T. Brudvik, and Bernie Zhang. 2010. Accessibility by demonstration: enabling end users to guide developers to web accessibility solutions. In Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (Orlando, Florida, USA) (ASSETS '10) . Association for Computing Machinery, New York, NY, USA, 35-42. https://doi.org/10.1145/ 1878803.1878812 -
  5. [11][11] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022. A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility. arXiv:2202.02312 [cs.CL] - · arXiv
  6. [13][13] John Canny. 1986. A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8, 6 (1986), 679-698. https://doi.org/10.1109/TPAMI.1986.4767851 - · DOI · Paper page
  7. [14][14] Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, and Xiaoyi Zhang. 2022. Towards Complete Icon Labeling in Mobile Applications. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 387, 14 pages. https://doi.org/10.1145/3491102.3502073 - · DOI
  8. [15][15] Sen Chen, Chunyang Chen, Lingling Fan, Mingming Fan, Xian Zhan, and Yang Liu. 2022. Accessible or Not? An Empirical Investigation of Android App Accessibility. IEEE Transactions on Software Engineering 48, 10 (2022), 3954-3968. https: //doi.org/10.1109/TSE.2021.3108162 - · DOI
  9. [17][17] Luis Cruz, Rui Abreu, and David Lo. 2019. To the attention of mobile software developers: guess what, test your app! Empirical Software Engineering 24, 4 (2019), 2438-2468. https://doi.org/10.1007/s10664-019-09701-0 - · DOI
  10. [18][18] Morgan Dixon and James Fogarty. 2010. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10) . Association for Computing Machinery, New York, NY, USA, 1525-1534. https://doi.org/10.1145/1753326.1753554 - · DOI
  11. [19][19] Richard O. Duda and Peter E. Hart. 1972. Use of the Hough Transformation to Detect Lines and Curves in Pictures. Commun. ACM 15, 1 (Jan 1972), 11-15. https://doi.org/10.1145/361237.361242 - · DOI · Paper page
  12. [20][20] Marcelo Medeiros Eler, Jose Miguel Rojas, Yan Ge, and Gordon Fraser. 2018. Automated Accessibility Testing of Mobile Apps. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) . IEEE, New York, NY, USA, 116-126. https://doi.org/10.1109/ICST.2018.00021 - · DOI
  13. [22][22] Sidong Feng and Chunyang Chen. 2023. Prompting Is All You Need: Automated Android Bug Replay with Large Language Models. arXiv:2306.01987 [cs.SE] - · arXiv
  14. [23][23] Raymond Fok, Mingyuan Zhong, Anne Spencer Ross, James Fogarty, and Jacob O. Wobbrock. 2022. A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 461, 16 pages. https://doi.org/10.1145/3491102.3502143 - · DOI
  15. [25][25] Greg Guest, Kathleen M MacQueen, and Emily E Namey. 2011. Applied thematic analysis . Sage Publications, Thousand Oaks, CA. https://doi.org/10.4135/ 9781483384436 -
  16. [26][26] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. arXiv:2307.12856 [cs.LG] - · arXiv
  17. [28][28] Yue Jiang, Eldon Schoop, Amanda Swearngin, and Jeffrey Nichols. 2023. ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations. arXiv:2310.04869 [cs.HC] - · arXiv
  18. [29][29] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli TranJohnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. arXiv:2207.05221 [cs.CL] - · arXiv
  19. [30][30] Pavneet Singh Kochhar, Ferdian Thung, Nachiappan Nagappan, Thomas Zimmermann, and David Lo. 2015. Understanding the Test Automation Culture of App Developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) (Graz, Austria). IEEE, New York, NY, USA, 1-10. https://doi.org/10.1109/ICST.2015.7102609 - · DOI
  20. [31][31] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., Red Hook, NY, USA, 22199-22213. https://proceedings.neurips.cc/paper\_files/paper/2022/ file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf -
  21. [32][32] Xiao Li, Nana Chang, Yan Wang, Haohua Huang, Yu Pei, Linzhang Wang, and Xuandong Li. 2017. ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST) . IEEE, New York, NY, USA, 161-171. https://doi.org/10.1109/ICST.2017.22 - · DOI
  22. [33][33] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping Natural Language Instructions to Mobile UI Action Sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Online, 8198-8210. https: //doi.org/10.18653/v1/2020.acl-main.729 - · DOI
  23. [34][34] Jun-Wei Lin, Navid Salehnamadi, and Sam Malek. 2021. Test automation in opensource Android apps: a large-scale empirical study. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (Virtual Event, Australia) (ASE '20) . Association for Computing Machinery, New York, NY, USA, 1078-1089. https://doi.org/10.1145/3324884.3416623 - · DOI
  24. [35][35] Mario Linares-Vásquez, Cárlos Bernal-Cardenas, Kevin Moran, and Denys Poshyvanyk. 2017. How do Developers Test Android Applications?. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, New York, NY, USA, 613-622. https://doi.org/10.1109/ICSME.2017.47 - · DOI
  25. [36][36] Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2023. Chatting with GPT-3 for Zero-Shot HumanLike Mobile Automated GUI Testing. arXiv:2305.09434 [cs.SE] - · arXiv
  26. [37][37] Jennifer Mankoff, Holly Fait, and Tu Tran. 2005. Is your web page accessible? a comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA) (CHI '05) . Association for Computing Machinery, New York, NY, USA, 41-50. https://doi.org/10.1145/1054972.1054979 - · DOI
  27. [38][38] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] - · arXiv
  28. [39][39] Nobuyuki Otsu. 1979. A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man, and Cybernetics 9, 1 (1979), 62-66. https: //doi.org/10.1109/TSMC.1979.4310076 - · DOI
  29. [40][40] Minxue Pan, Tongtong Xu, Yu Pei, Zhong Li, Tian Zhang, and Xuandong Li. 2022. GUI-Guided Test Script Repair for Mobile Apps. IEEE Transactions on Software Engineering 48, 3 (2022), 910-929. https://doi.org/10.1109/TSE.2020.3007664 - · DOI
  30. [41][41] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Android in the Wild: A Large-Scale Dataset for Android Device Control. arXiv:2307.10088 [cs.LG] - · arXiv
  31. [43][43] Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock. 2017. Epidemiology as a Framework for Large-Scale Mobile Application Accessibility Assessment. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS '17) . Association for Computing Machinery, New York, NY, USA, 2-11. https://doi.org/10.1145/3132525.3132547 - · DOI
  32. [44][44] Navid Salehnamadi, Abdulaziz Alshayban, Jun-Wei Lin, Iftekhar Ahmed, Stacy Branham, and Sam Malek. 2021. Latte: Use-Case and Assistive-Service Driven Automated Accessibility Testing Framework for Android. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, 11 pages. https://doi.org/10.1145/3411764.3445455 - · DOI
  33. [45][45] Navid Salehnamadi, Ziyao He, and Sam Malek. 2023. Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-Replay. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 73, 20 pages. https://doi.org/10.1145/3544548.3580679 - · DOI
  34. [46][46] Navid Salehnamadi, Forough Mehralian, and Sam Malek. 2023. Groundhog: An Automated Accessibility Crawler for Mobile Apps. In Proceedings of the 37th IEEE/ACMInternational Conference on Automated Software Engineering (Rochester, MI, USA) (ASE '22) . Association for Computing Machinery, New York, NY, USA, Article 50, 12 pages. https://doi.org/10.1145/3551349.3556905 - · DOI
  35. [47][47] Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. 2024. From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces. Advances in Neural Information Processing Systems 36 (2024). https://proceedings.neurips.cc/paper\_files/paper/2023/hash/ 6c52a8a4fadc9129c6e1d1745f2dfd0f-Abstract-Conference.html -
  36. [48][48] Camila Silva, Marcelo Medeiros Eler, and Gordon Fraser. 2018. A survey on the tool support for the automatic evaluation of mobile accessibility. In Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion (Thessaloniki, Greece) (DSAI '18) . Association for Computing Machinery, New York, NY, USA, 286-293. https://doi.org/10.1145/3218585.3218673 - · DOI
  37. [49][49] Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. 2023. UGIF: UI Grounded Instruction Following. arXiv:2211.07615 [cs.CL] - · arXiv
  38. [52][52] Bryan Wang, Gang Li, and Yang Li. 2023. Enabling Conversational Interaction with Mobile UI Using Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 432, 17 pages. https://doi.org/10.1145/3544548.3580895 - · DOI · Paper page
  39. [54][54] Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023. Empowering LLM to use Smartphone for Intelligent Task Automation. arXiv:2308.15272 [cs.AI] - · arXiv
  40. [57][57] Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 275, 15 pages. https: //doi.org/10.1145/3411764.3445186 - · DOI
  41. [58][58] Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, and Yan Lu. 2023. Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators. arXiv:2306.01242 [cs.AI] Received 14 September 2023; revised 12 December 2024; accepted 19 January 2024 · arXiv

Cited by

Sorted by citations

Authors
Taeb, M., Swearngin, A., Schoop, E., Cheng, R., Jiang, Y., & Nichols, J.
Affiliations
  • Florida State University (US)
  • Apple (United States) (US)
  • Aalto University (FI)
ORCID
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-16
DOI
10.1145/3613904.3642777
OpenAlex
W4396833734
Cited by
52
References
4
Citations by year
2026: 14 · 2025: 31 · 2024: 7
FWCI
28.41
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
accessibility testing, large language models, voice-over, test automation, quality assurance testing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review