Remote usability evaluations with disabled people.
The DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index.
Abstract
Finding participants for evaluations with specific demographics can be a problem for usability and user experience specialists. In particular, finding participants with disabilities is especially problematic, yet testing with disabled people is becoming increasingly important. Two case studies are presented that explore using asynchronous remote evaluation techniques with disabled participants. These show that while quantitative data are comparable, the amount and richness of qualitative data are not likely to be comparable. The implications for formative and summative evaluations are discussed and a set of principles for local and remote evaluations with disabled users is presented.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875
Guidelines for Designing User Interface Software
Smith, S. L., & Mosier, J. N. (1986) · doi:10.21236/ada177198 · Cited by 756 · cited 1×
“The HCI community debated this issue in the late 1980s, when guidelines for interface design, such as those by Smith and Mosier [23] were evaluated and found to be difficult to use [4, 6, 14, 15 ].”
Method“User sensitive inclusive design”--- in search of a new paradigm
Newell, A. F., & Gregor, P. (2000) · Proceedings on the 2000 conference on Universal Usability - CUU '00 · doi:10.1145/355460.355470 · Cited by 344
User-Sensitive Inclusive Design
Newell, A. F., Gregor, P., Morgan, M., Pullin, G., & Macaulay, C. (2011) · Universal Access in the Information Society · doi:10.1007/s10209-010-0203-y · Cited by 252
Remote evaluation: the network as an extension of the usability laboratory
Hartson, H. R., Castillo, J. C., Kelso, J., & Neale, W. C. (1996) · Proceedings of the SIGCHI conference on Human factors in computing systems common ground - CHI '96 · doi:10.1145/238386.238511 · Cited by 195
Fundamental principles and priority setting for universal usability
Vanderheiden, G. (2000) · Proceedings on the 2000 conference on Universal Usability - CUU '00 · doi:10.1145/355460.355469 · Cited by 129 · cited 1×
“As Vanderheiden and colleagues have noted [ 25 , 26], there are probably as many definitions of these concepts, all closely related, as there are people discussing them, and considerable misunderstanding.”
BackgroundWebQuilt: A Proxy-based Approach to Remote Web Usability Testing
Hong, J., Heer, J., Waterson, S. J., & Landay, J. A. (2001) · Cited by 112
Application of guidelines for designing user interface software
MOSIER, J. N., & SMITH, S. L. (1986) · Behaviour & Information Technology · doi:10.1080/01449298608914497 · Cited by 100
Applying cognitive psychology to user-interface design
Wilson, J. G. (1988) · Applied Ergonomics · doi:10.1016/0003-6870(88)90050-6 · Cited by 88
WebQuilt: A proxy-based approach to remote web usability testing
(2001) · ACM Transactions on Information Systems · doi:10.1145/502115.502118 · Cited by 82 · cited 1×
“This technique is now widely used, but practitioners are aware of the problems of analyzing and interpreting the large amounts of data that are produced [8, 9 , 11].”
Method
Research works printed but not linked: 8
- Bradner , E. ( 2004 ). Keeping your distance. User experience, 3(6), 11 . Bradner, E. (2004). Keeping your distance. User experience, 3(6), 11.
- Disability Rights Commission ( 2004 ). The web: access and inclusion for disabled people . London : The Stationery Office . Disability Rights Commission (2004). The web: access and inclusion for disabled people. London: The Stationery Office.
- 10.5555/302163.302177 · doi:10.5555/302163.302177
- 10.1080/13614560412331326017 · doi:10.1080/13614560412331326017
- Law , C. , Barnicle , K. and Henry , S.L . ( 2000 ). Usability screening techniques: evaluating for a wider range of environments, circumstances and abilities . Proc. UPA 2000 . Available at: http://trace.wisc.edu/docs/usability_screen_techs_UPA 2000/usability_screening_techs.htm Law, C., Barnicle, K. and Henry, S.L. (2000). Usability screening techniques: evaluating for a wider range of environments, circumstances and abilities. Proc. UPA 2000. Available at: http://trace.wisc.edu/docs/usability_screen_techs_UPA2000/usability_screening_techs.htm
- Petrie , H. ( 2001 ). Acessibility and usability requirements for ICTs for disabled and elderly people: a functional classification approach . In J.G. Abascal and C. Nicolle (Eds.), Inclusive guidelines for human computer interaction. London : Taylor and Francis . 0-748409-48-3. Petrie, H. (2001). Acessibility and usability requirements for ICTs for disabled and elderly people: a functional classification approach. In J.G. Abascal and C. Nicolle (Eds.), Inclusive guidelines for human computer interaction. London: Taylor and Francis. 0-748409-48-3.
- Petrie , H. , and Hamilton , F . ( 2004 ). The Disability Rights Commission Formal Investigation into Web Site Accessibility. In Dearden, A. and Watts, L. (Eds.) , Proceedings HCI 2004: Design For Life ( Leeds, UK) , pp. 157 -- 158 . Petrie, H., and Hamilton, F. (2004). The Disability Rights Commission Formal Investigation into Web Site Accessibility. In Dearden, A. and Watts, L. (Eds.), Proceedings HCI 2004: Design For Life (Leeds, UK), pp. 157--158.
- Poulson , D. , Ashby , M. and Richardson , S . ( 1996 ). USERfit , A practical handbook on user-centred design for Assistive Technology , Brussels- Luxembourg : ESC-EC-EAEC. Poulson, D., Ashby, M. and Richardson, S. (1996). USERfit, A practical handbook on user-centred design for Assistive Technology, Brussels-Luxembourg: ESC-EC-EAEC.
Cited by
Items 101–164 of 164
Supporting the Development of Accessible Web Applications
Arrue, M., Vigo, M., & Abascal, J. (2008) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.3217/jucs-014-16-2699 · Cited by 10
SignQUOTE: A Remote Testing Facility for Eliciting Signed Qualitative Feedback
Schnepp, J. C., Wolfe, R. J., Shiver, B., McDonald, J. C., & Toro, J. (2011) · ScholarWorks@BGSU (Bowling Green State University) · Cited by 8
Prior, S. (2011) · Discovery Research Portal (University of Dundee) · doi:10.13140/rg.2.2.14622.23367 · Cited by 12
A Framework for Remote User Evaluation of Accessibility and Usability of Websites
Power, C., Petrie, H., & Mitchell, R. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02707-9_67 · Cited by 7
“identified several other dimensions that can be used to further categorize the remote evaluation activities [8].”
BackgroundSelf-citationAspelund, A., Valkonen, P., Viitanen, J., & Rauta, V. (2024) · JMIR Human Factors · doi:10.2196/53691 · Cited by 8
“Researchers must pay attention to building trust and confidentiality [26], choose tools that are familiar and easily accessible for the participants [27], focus on body language and facial expressions to establish rapport [26,28], and provide the participants with technical support as needed [29].”
BackgroundEnhancing Wikipedia Editing with WAI-ARIA
Senette, C., Buzzi, M. C., Buzzi, M., & Leporini, B. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-10308-7_11 · Cited by 7
“[16] conducted two case studies with disabled users (including totally blind and visually-impaired persons) to explore asynchronous remote evaluation techniques, and showed that while quantitative data were comparable, local evaluations collected richer qualitative data.”
MethodWebTrax: Visualizing Non-visual Web Interactions
Bigham, J. P., & Murray, K. I. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14100-3_51 · Cited by 7
“Commonly, when blind users perform an evaluation they focus on the most problematic accessibility problems and sometimes are not able to fully evaluate a web site because the accessibility problems discovered prevent full access to the site [10].”
BackgroundPetrie, H., & Wakefield, M. (2020) · International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI) · doi:10.1145/3439231.3439248 · Cited by 7
“In line with the results of Petrie et al. [13], initial analysis of the accessibility and usability problems and user comments on their experience suggests that the quantity of information provided by the unmoderated sessions was considerably less than that provided by the moderated sessions.”
BackgroundResultSelf-citationAvaliação da acessibilidade de websites por usuários com deficiência
Freire, A. P., de Lara, S. M. A., & Fortes, R. P. D. M. (2013) · doi:10.5555/2577101.2577198 · Cited by 14
Remote Evaluation in Universal Design Using Video Conferencing Systems During the COVID-19 Pandemic
Simon-Liedtke, J. T., Bong, W. K., Schulz, T. W., & Fuglerud, K. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78092-0_8 · Cited by 10
Acessibilidade no desenvolvimento de sistemas web: um estudo sobre o cenário brasileiro
Freire, A. P. (2008) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.55.2008.tde-06052008-101644 · Cited by 45
Ramôa, G., Schmidt, V., Schwarz, T., Stiefelhagen, R., & König, P. (2024) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2024.1368983 · Cited by 5
Angerbauer, K., & Sedlmair, M. (2022) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv57783.2022.00007 · Cited by 7
“[50], this was especially useful for reaching people with disabilities in our case people with limited mobility or limited color vision.”
BackgroundInfluentialCheoh, J. L., Beigpourian, B., Wei, S., Ferguson, D. M., & Ohland, M. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9274056 · Cited by 7
“Researchers in the field of user experiences have conducted a wide range of studies from remote usability studies [6] to in-person usability studies [7] to examine the perspectives of people with disabilities when they navigate on web interfaces.”
BackgroundLessons Learned from Remote User-Centered Design with People with Dementia
Dixon, E., Shetty, A., Pimento, S., & Lazar, A. (2021) · Design for inclusion · doi:10.1007/978-3-030-70293-9_6 · Cited by 6
A diversity-sensitive evaluation method
Mourouzis, A., Antona, M., & Stephanidis, C. (2010) · Universal Access in the Information Society · doi:10.1007/s10209-010-0211-y · Cited by 5
“Methods for the assessment of usability for people with disability through remote evaluation and critical incident reporting have also been proposed [43], [45].”
BackgroundCarvalho, L. P., de Souza, J. G. R., Prates, R. O., & Freire, A. P. (2023) · Interacting with Computers · doi:10.1093/iwc/iwad023 · Cited by 5
The Challenge of Usability Evaluation of Online Social Networks with a Focus on Facebook
Alam, T., & Ali, M. İ. (2010) · DiVA (Blekinge Institute of Technology) · Cited by 7
ED&I and SE: Challenges, Progress, and Lessons
Grundy, J., Kanij, T., Hoda, R., Khalajzadeh, H., Madugalla, A. K., & McIntosh, J. (2024) · Apress eBooks · doi:10.1007/978-1-4842-9651-6_2 · Cited by 3
Ramôa, G., & Müller, K. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746398 · Cited by 4
“Recruiting BVI participants for user studies is challenging due to their "low representation in the general population and mobility challenges" [18], as seen in previous research [29, 32, 50, 64, 75, 81, 85].”
Touching Base on Children's Interactions with Tablet Games
Read, J. C., Clarke, S., Fitton, D., Joes, R., Horton, M., & Sim, G. (2017) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3130859.3131432 · Cited by 3
Okur, M. R., Kılınç, H., & Yıldız, G. (2021) · Sakarya University Journal of Education · doi:10.19126/suje.930534 · Cited by 3
Evaluation methods and measurement
Lazar, J. K., Goldstein, D., & Taylor, A. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-800646-7.00008-3 · Cited by 3
Persona-based expert review of an e-learning system for adults
Stojmenova, E., Lugmayr, A. R., & Dinevski, D. (2013) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew.2013.6618367 · Cited by 3
“However, relying solely on user interface design guidelines is not enough [13].”
MethodIntelligent interfaces enabling blind web users to build accessibility into the web
Ladner, R. E., & Bigham, J. P. (2009) · Cited by 3
Chaves, J., Salvador, Y., Braz, J., Piffer, M. M., Hostin, E., & Riegel, I. M. (2019) · Anais do Encontro Nacional de Computação dos Institutos Federais (ENCompIF) · doi:10.5753/encompif.2019.6355 · Cited by 2
Persona-Based Design for Personal Health Information Management Systems
Stojmenova, E., Dinevski, D., Guna, J., & PogaÄnik, M. ⁄. (2013) · Cited by 2
Reaching People Where They Are: Remote Macroergonomics Research
Hill, J. R., Abebe, E., Holden, R. J., McLaughlin, A. C., Rogers, C. C., & Werner, N. E. (2022) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181322661012 · Cited by 2
“…that requires them to use specialized equipment to complete study protocols, have responsibilities at home (e.g. caregiving) or work that prevent them from travelling, or are homebound (Andreasen et al., 2007; Dray & Siegel, 2004; Hammontree et al., 1994; LaMar, 2020; Petrie et al., 2006).”
BackgroundInvestigating the Acceptance of Voice User Interfaces for Users With Disabilities
Jin, D. H., Kim, Y. M., & Lee, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3520149 · Cited by 4
O acesso dos analfabetos funcionais ao conteúdo informacional dos sites governamentais brasileiros
Capra, E. P., Benjamin, K., Ferreira, S. B. L., Capelli, C., & da Silveira, D. S. (2013) · doi:10.5555/2577101.2577128 · Cited by 2
Expert review of the multimodal interaction model for Foot Reflexology VRST application
Sulaiman, S. B., Okere, H. C., Awang, D. R. B., & Foong, O.-M. (2016) · International Conference on Computer and Information Sciences · doi:10.1109/iccoins.2016.7783271 · Cited by 4
“Stojmenova, Lugmayr and Dinevski [23], mentioned the countless guidelines available for supporting user designs and system designs for all the categories of users, but relying solely on system design guidelines does not suffice as concurred by [24].”
BackgroundUnderstanding and Classifying Online Amputee Users on Reddit
Yu, X., & Brady, E. L. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3110037 · Cited by 3
“To address this, researchers have created remote settings to carry out experiments [3] and pools of representative users to allow for easy recruitment [7].”
BackgroundMethods of Usability Testing for Users with Cognitive Impairments
Schöne, C., Große, U., Wölfel, A., & Krömker, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78092-0_7 · Cited by 2
EXPLORING FORMER & MODERN VIEWS: A CATCH-ALL TO ASSISTIVE TECHNOLOGY APPLICATIONS
Berger, A. (2023) · E+M Ekonomie a Management · doi:10.15240/tul/001/2023-1-012 · Cited by 2
A Case Study of Simulating HCI for Special Needs
Biswas, P., & Robinson, P. (2008) · Springer eBooks · doi:10.1007/978-1-84800-211-1_17 · Cited by 2
An Experiential Comparative Analysis of Two Remote Usability Testing Methods
Varga, E. (2011) · RIT Scholar Works (Rochester Institute of Technology) · Cited by 2
Generating a domain-specific inspection method through an adaptive framework
Alrobaea, R. (2015) · UEA Digital Repository (University of East Anglia) · Cited by 2
Exploring the Future of Remote User ResearchPreprint
Martelaro, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.03349 · Cited by 2
“(Petrie et al., 2006) User researchers often rely on non-verbal cues, broadly observing the user’s context, and experiencing the physical sensations of a user’s environment to develop a full sense of user experience.”
BackgroundDevelopment and Evaluation of an mHealth App That Promotes Access to 3D Printable Assistive Devices
Bush, J., Benham, S., & Kaniamattam, M. (2024) · Technologies · doi:10.3390/technologies12070114 · Cited by 2
“For the next phase, engaging end users during these still early phases of development helps designers understand real-world challenges and preferences, through focus groups and usability testing, which has been shown to improve the accessibility and functionality for PWDs [41].”
BackgroundGeuenich, J., Joublin, F., Ceravola, A., Brandenburg, S., & Dodge, J. E. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3801978 · Cited by 2
“In contrast, the LLM-agents analyzed the structure of the HTML source code rather than assessing dynamic interactions.”
e-Accessibility: Making the Web Accessible to the Visually Impaired Persons
Ferreira, S. B. L., da Silveira, D. S., Chauvel, M. A., & Ferreira, M. G. D. A. L. (2008) · Americas Conference on Information Systems · Cited by 1
Mastrianni, A., & Sarcevic, A. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3573850 · Cited by 1
“Researchers can also more easily recruit participants from diferent geographic areas [7, 19].”
BackgroundAvsar, H., & Burkhardt, M. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-35992-7_32 · Cited by 1
Preserving the Spatial Information of Accessible UML Class Diagrams for the Visually Impaired
Matti, S. H. (2017) · RIT Scholar Works (Rochester Institute of Technology) · Cited by 1
The trade-off between usability and security in the context of eGovernment
Alshamsi, A. (2019) · Keele Research Repository (Keele University) · Cited by 1
Site Unseen: Website Accessibility Testing for Academic Libraries with Visually-Impaired Users
Waugh, D. (2019) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/gq0y-7650 · Cited by 3
Howard III, T. J. (2021) · Digital Commons - URI (University of Rhode Island) · doi:10.23860/thesis-howard-thomas-2021 · Cited by 1
AVALIAÇÃO DA METODOLOGIA DE CONFORMIDADE COM O WEB CONTENT ACCESSIBILITY GUIDELINES (WCAG-EM 1.0)
de Souza, E. R. (2015) · doi:10.17771/pucrio.acad.29516 · Cited by 4
Collaborative adaptive accessibility and human capabilities
Atkinson, M. T. (2012) · Loughborough University Institutional Repository (Loughborough University) · Cited by 2
Biswas, P. (2008) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1358628.1358736
“[4] take the approach of remote evaluation but still need to find disabled participants.”
BackgroundStudying information interaction in context: some lessons for traffic experiments
Colbert, M., & Oliver, A. (2012) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/2362724.2362763
Evaluation of remote usability techniques in a elderly. cohort
Yu, C.-H. (2010) · doi:10.6083/m4hh6h1k · Cited by 1
Horne, R. M. (2015) · UC Research Repository (University of Canterbury) · doi:10.26021/1514
Developer Driven and User Driven Usability Evaluations
Bruun, A. (2013) · VBN Forskningsportal (Aalborg Universitet)
Learning-Disabled Children: A Disregarded User Group
Bay, S., & Ziefle, M. (2008) · IGI Global eBooks · doi:10.4018/978-1-59904-871-0.ch009
Aplicando o Design Science Research no Desenvolvimento de um Sistema Colaborativo Assistivo
da Silva, D. M. A., Sartori, D. F., Hirata, C. M., & Berkenbrock, C. D. M. (2016) · Simpósio Brasileiro de Sistemas Colaborativos (SBSC) · doi:10.5753/sbsc.2016.9494
Using Social Media Websites to Support Scenario-Based Design of Assistive Technology
Yu, X. (2020) · IUScholarWorks (Indiana University) · doi:10.7912/c2/962
Corrêa, M. B. (2016) · RIUnB Institutional Repository (University of Brasília) · doi:10.26512/2016.07.d.21309
Integrating Accessibility Evaluation into Web Engineering Processes
Power, C., Freire, A. P., & Petrie, H. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-523-0.ch018
Self-citationDeveloping Accessible Websites for Differently Abled People Using Open Source Tools
Shirur, P. S., Raghuvanshi, S., & Bali, V. (2022) · International Journal of Software Innovation · doi:10.4018/ijsi.303576
Analysing usability and security issues in design and development of information systems
Arreymbi, J. (2012) · Greenwich Academic Literature Archive (University of Greenwich)
Usability of web-based MT post-editing environments for screen reader users
Vázquez, S. R., O'Brien, S., & Fitzpatrick, D. (2017) · Archive ouverte UNIGE (University of Geneva)
Saarinen, K. (2020) · Aaltodoc (Aalto University)
Sharif, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-07186-6_7
Metadata
- Authors
- Petrie, H., Hamilton, F., King, N., & Pavan, P.
- Affiliations
- University of York (GB)
- Designed for All, London, United Kingdom
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2006-04-22
- Pages
- 1133-1141
- OpenAlex
- W1973244543
- Semantic Scholar
- 779ceea17ddd7221f875c644ec05df9cb4812996
- Cited by
- 157
- References
- 22
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 8 · 2023: 7 · 2022: 8 · 2021: 13 · 2020: 8 · 2019: 5 · 2018: 2 · 2017: 7 · 2016: 6 · 2015: 8 · 2014: 3 · 2013: 9 · 2012: 7
- FWCI
- 9.99
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- remote usability evaluation, disabled users, participant recruitment, formative assessment, summative assessment, accessibility, user experience evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.