Remote usability evaluations with disabled people.
The DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index.
Abstract
Finding participants for evaluations with specific demographics can be a problem for usability and user experience specialists. In particular, finding participants with disabilities is especially problematic, yet testing with disabled people is becoming increasingly important. Two case studies are presented that explore using asynchronous remote evaluation techniques with disabled participants. These show that while quantitative data are comparable, the amount and richness of qualitative data are not likely to be comparable. The implications for formative and summative evaluations are discussed and a set of principles for local and remote evaluations with disabled users is presented.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875
Guidelines for Designing User Interface Software
Smith, S. L., & Mosier, J. N. (1986) · doi:10.21236/ada177198 · Cited by 756 · cited 1×
“The HCI community debated this issue in the late 1980s, when guidelines for interface design, such as those by Smith and Mosier [23] were evaluated and found to be difficult to use [4, 6, 14, 15 ].”
Method“User sensitive inclusive design”--- in search of a new paradigm
Newell, A. F., & Gregor, P. (2000) · Proceedings on the 2000 conference on Universal Usability - CUU '00 · doi:10.1145/355460.355470 · Cited by 344
User-Sensitive Inclusive Design
Newell, A. F., Gregor, P., Morgan, M., Pullin, G., & Macaulay, C. (2011) · Universal Access in the Information Society · doi:10.1007/s10209-010-0203-y · Cited by 252
Remote evaluation: the network as an extension of the usability laboratory
Hartson, H. R., Castillo, J. C., Kelso, J., & Neale, W. C. (1996) · Proceedings of the SIGCHI conference on Human factors in computing systems common ground - CHI '96 · doi:10.1145/238386.238511 · Cited by 195
Fundamental principles and priority setting for universal usability
Vanderheiden, G. (2000) · Proceedings on the 2000 conference on Universal Usability - CUU '00 · doi:10.1145/355460.355469 · Cited by 129 · cited 1×
“As Vanderheiden and colleagues have noted [ 25 , 26], there are probably as many definitions of these concepts, all closely related, as there are people discussing them, and considerable misunderstanding.”
BackgroundWebQuilt: A Proxy-based Approach to Remote Web Usability Testing
Hong, J., Heer, J., Waterson, S. J., & Landay, J. A. (2001) · Cited by 112
Application of guidelines for designing user interface software
MOSIER, J. N., & SMITH, S. L. (1986) · Behaviour & Information Technology · doi:10.1080/01449298608914497 · Cited by 100
Applying cognitive psychology to user-interface design
Wilson, J. G. (1988) · Applied Ergonomics · doi:10.1016/0003-6870(88)90050-6 · Cited by 88
WebQuilt: A proxy-based approach to remote web usability testing
(2001) · ACM Transactions on Information Systems · doi:10.1145/502115.502118 · Cited by 82 · cited 1×
“This technique is now widely used, but practitioners are aware of the problems of analyzing and interpreting the large amounts of data that are produced [8, 9 , 11].”
Method
Research works printed but not linked: 8
- Bradner , E. ( 2004 ). Keeping your distance. User experience, 3(6), 11 . Bradner, E. (2004). Keeping your distance. User experience, 3(6), 11.
- Disability Rights Commission ( 2004 ). The web: access and inclusion for disabled people . London : The Stationery Office . Disability Rights Commission (2004). The web: access and inclusion for disabled people. London: The Stationery Office.
- 10.5555/302163.302177 · doi:10.5555/302163.302177
- 10.1080/13614560412331326017 · doi:10.1080/13614560412331326017
- Law , C. , Barnicle , K. and Henry , S.L . ( 2000 ). Usability screening techniques: evaluating for a wider range of environments, circumstances and abilities . Proc. UPA 2000 . Available at: http://trace.wisc.edu/docs/usability_screen_techs_UPA 2000/usability_screening_techs.htm Law, C., Barnicle, K. and Henry, S.L. (2000). Usability screening techniques: evaluating for a wider range of environments, circumstances and abilities. Proc. UPA 2000. Available at: http://trace.wisc.edu/docs/usability_screen_techs_UPA2000/usability_screening_techs.htm
- Petrie , H. ( 2001 ). Acessibility and usability requirements for ICTs for disabled and elderly people: a functional classification approach . In J.G. Abascal and C. Nicolle (Eds.), Inclusive guidelines for human computer interaction. London : Taylor and Francis . 0-748409-48-3. Petrie, H. (2001). Acessibility and usability requirements for ICTs for disabled and elderly people: a functional classification approach. In J.G. Abascal and C. Nicolle (Eds.), Inclusive guidelines for human computer interaction. London: Taylor and Francis. 0-748409-48-3.
- Petrie , H. , and Hamilton , F . ( 2004 ). The Disability Rights Commission Formal Investigation into Web Site Accessibility. In Dearden, A. and Watts, L. (Eds.) , Proceedings HCI 2004: Design For Life ( Leeds, UK) , pp. 157 -- 158 . Petrie, H., and Hamilton, F. (2004). The Disability Rights Commission Formal Investigation into Web Site Accessibility. In Dearden, A. and Watts, L. (Eds.), Proceedings HCI 2004: Design For Life (Leeds, UK), pp. 157--158.
- Poulson , D. , Ashby , M. and Richardson , S . ( 1996 ). USERfit , A practical handbook on user-centred design for Assistive Technology , Brussels- Luxembourg : ESC-EC-EAEC. Poulson, D., Ashby, M. and Richardson, S. (1996). USERfit, A practical handbook on user-centred design for Assistive Technology, Brussels-Luxembourg: ESC-EC-EAEC.
Cited by
Internet Access by People with Intellectual Disabilities: Inequalities and Opportunities
Chadwick, D., Wesson, C., & Fullwood, C. (2013) · Future Internet · doi:10.3390/fi5030376 · Cited by 300
“Universal design incorporates “the design of products, environments, programmes and services to be usable by all people, to the greatest extent possible, without the need for adaptation or specialised design” [54] and is not apparent in the vast majority of websites that have been studied [56,57].”
BackgroundEvaluating existing audio CAPTCHAs and an interface optimized for non-visual use
Bigham, J. P., & Cavender, A. C. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518983 · Cited by 187
“observed, conducting studies with disabled people in a lab setting can be difficult, but remote studies can produce similar results [13].”
ResultWhat happened to remote usability testing?: an empirical study of three methods
Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., & Stage, J. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240838 · Cited by 177
From Mice to Men - 24 Years of Evaluation in CHI
Barkhuus, L., & Rode, J. A. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.2180963 · Cited by 180
“First, studies of technologies intended for specific target populations, such as the elderly or disabled (e.g. [ 33 ]) do not follow this trend.”
BackgroundThe Evaluation of Accessibility, Usability, and User Experience
(2009) · The Universal Access Handbook · doi:10.1201/9781420064995-31 · Cited by 171
WebinSitu: a comparative analysis of blind and sighted browsing behavior
Bigham, J. P., Cavender, A. C., Brudvik, J. T., Wobbrock, J. O., & Ladner, R. E. (2007) · Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1296843.1296854 · Cited by 153
“Previous work has advocated remote studies because they allow participants to use their existing assistive technology and software [24, 25, 16].”
BackgroundInvestigating touchscreen accessibility for people with visual impairments
McGookin, D., Brewster, S., & Jiang, W. (2008) · Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges · doi:10.1145/1463160.1463193 · Cited by 151
“Carrying out evaluations with visually impaired people is difficult, as the term “visual impairment” covers a range of impairments from poor sight to full blindness [18].”
BackgroundEnsuring Digital Accessibility Through Process and Policy
Lazar, J., Goldstein, D., & Taylor, A. (2015) · Elsevier eBooks · doi:10.1016/c2013-0-13367-3 · Cited by 359
Beyond Conformance: The Role of Accessibility Evaluation Methods
Brajnik, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85200-1_9 · Cited by 114
“[30], remote user testing for accessibility eases the logistic difficulties, but raises additional issues concerning the validity of results.”
BackgroundGravvitas: generic multi-touch presentation of accessible graphics
Goncu, C., & Marriott, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-23774-4_5 · Cited by 112
“Since there are relatively few blind people and it is often hard for them to travel, it is quite difficult to find blind participants (also pointed out in [32, 28]).”
WebAnywhere: a screen reader on-the-go
Bigham, J. P., Prince, C. M., & Ladner, R. E. (2008) · Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A) · doi:10.1145/1368044.1368060 · Cited by 109
“Such studies have been shown to yield similar quantitative results, although risk collecting less-informative qualitative feedback [25].”
ResultRepresenting users in accessibility research
Sears, A., & Hanson, V. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979268 · Cited by 96
“Testing of remote users [1, 9] and other innovative solutions may help future researchers recruit and test the needed representative users.”
BackgroundAlsaeedi, A. (2020) · Information · doi:10.3390/info11010040 · Cited by 101
“Despite the effectiveness of this approach, recruiting disabled people is a very difficult task [29].”
BackgroundHCI and Societal Issues: A Framework for Engagement
Hochheiser, H., & Lazar, J. (2007) · International Journal of Human-Computer Interaction · doi:10.1080/10447310701702717 · Cited by 93
The Expertise Effect on Web Accessibility Evaluation Methods
Brajnik, G., Yesilada, Y., & Harper, S. (2011) · Human–Computer Interaction · doi:10.1080/07370024.2011.601670 · Cited by 83
“A useful variant is remote user testing ; Petrie, Hamilton, King, and Pavan (2006) discussed pros and cons of remote usability evaluations with disabled people.”
Let your users do the testing: a comparison of three remote asynchronous usability testing methods
Bruun, A., Gull, P., Hofmeister, L., & Stage, J. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518948 · Cited by 85
Tools for web accessibility evaluation
Abascal, J., Arrue, M., & Valencia, X. (2019) · Human–Computer Interaction Series · doi:10.1007/978-1-4471-7440-0_26 · Cited by 88
Methodologies for Involving Older Adults in the Design Process
Newell, A. F., Arnott, J., Carmichael, A., & Morgan, M. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73279-2_110 · Cited by 88
“[28], who found that the observance of W3C guidelines did not necessarily lead to a site which was usable by disabled people.”
BackgroundKim, H. K., Han, S. H., Park, J., & Park, J. (2016) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2016.07.002 · Cited by 80
Representing users in accessibility research
Sears, A., & Hanson, V. L. (2012) · ACM Transactions on Accessible Computing · doi:10.1145/2141943.2141945 · Cited by 79
Evaluating e-learning accessibility by automated and student-centered methods
Kumar, K., & Owston, R. (2015) · Educational Technology Research and Development · doi:10.1007/s11423-015-9413-6 · Cited by 83
“…et al. 2007; Hartson et al. 1996; McFadden et al. 2002; Selvaraj 2004), though there are conflicting data as to whether the time on task or amount of qualitative data are affected by these methods of testing (Andreason et al. 2007; Hartson et al. 1996; McFadden et al. 2002; Petrie et al. 2006).”
MethodThe Effects of "Not Knowing What You Don't Know" on Web Accessibility for Blind Web Users
Bigham, J. P., Lin, I., & Savage, S. (2017) · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3132525.3132533 · Cited by 73
“Prior research has shown that remote studies can be effective for people with disabilities [21].”
BackgroundResultTigwell, G. W. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445620 · Cited by 73
“Prior work has looked at supporting remote evaluations and feedback with disabled people [11, 12, 53, 54].”
BackgroundAn evaluation of web accessibility metrics based on their attributes
Freire, A. P., Fortes, R. P. M., Turine, M. A. S., & Paiva, D. M. B. (2008) · Proceedings of the 26th annual ACM international conference on Design of communication · doi:10.1145/1456536.1456551 · Cited by 60
Gironimo, G. D., Matrone, G., Tarallo, A., Trotta, M., & Lanzotti, A. (2012) · Engineering With Computers · doi:10.1007/s00366-012-0274-x · Cited by 58
“However, the authors believe that an expert-based approach, such as the one described above, would make it easy to administer the tests also to disabled people [39].”
BackgroundThe potential of adaptive interfaces as an accessibility aid for older web users
Sloan, D., Atkinson, M. T., Machin, C. H. C., & Li, Y. (2010) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806033 · Cited by 53
“This can result in lower awareness of the design issues with developers [29].”
BackgroundCollaborative web accessibility improvement: challenges and possibilities
Takagi, H., Kawanaka, S., Kobayashi, M., Sato, D., & Asakawa, C. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639677 · Cited by 48
Web accessibility implementation in private sector organizations: motivations and business impact
Leitner, M.-L., Strauss, C., & Stummer, C. (2016) · Universal Access in the Information Society · doi:10.1007/s10209-014-0380-1 · Cited by 39
“, [32, 37]).”
User experience to improve the usability of a vision-based interface
Manresa-Yee, C., Ponsa, P., Varona, J., & Perales, F. J. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.06.004 · Cited by 46
“In order to evaluate them, we have to use procedures and techniques from usability engineering, such as interviews, satisfaction questionnaires, the identification of user profiles, and automated usability evaluation techniques (Dumas and Redish, 1999; Petrie et al., 2006; Stary and Eberle, 2008).”
MethodAutomatic evaluation of assistive interfaces
Biswas, P., & Robinson, P. (2008) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1378773.1378806 · Cited by 45
“[10] take the approach of remote evaluation but still need to find disabled participants.”
BackgroundOntology‐driven disability‐aware e‐learning personalisation with ONTODAPS
Nganji, J. T., Brayshaw, M., & Tompsett, B. (2012) · Campus-Wide Information Systems · doi:10.1108/10650741311288797 · Cited by 49
“As Petrie et al. (2006) observe, the challenge is to provide personalisation for those with multiple disabilities such as one who is both blind and deaf.”
BackgroundUsers’ design feedback in usability evaluation: a literature review
Følstad, A. (2017) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-017-0100-y · Cited by 52
Bigham, J. P., Brudvik, J. T., & Zhang, B. (2010) · Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1878803.1878812 · Cited by 41
“Commonly, when blind users perform an evaluation they focus on the most problematic accessibility problems and sometimes are not able to fully evaluate a web site because the accessibility problems discovered prevent full access to the site [18].”
BackgroundAkter, T., Ahmed, T., Kapadia, A. C., & Swaminathan, S. M. (2020) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3417003 · Cited by 45
“…methods have been widely adopted in the accessibility community [22, 26, 51] and are often encouraged, e.g., to avoid the difculties people with disabilities may face traveling to the facility [59], and are more suitable as participants use their preferred screen readers in their own settings [21].”
MethodValidity and reliability of web accessibility guidelines
Brajnik, G. (2009) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1639642.1639666 · Cited by 45
“They need to be well studied and understood for methods, and guidelines, that are expected to have a major impact.”
BackgroundDevelopment of a web-based survey system for evaluating affective satisfaction
Cho, Y., Park, J., Han, S. H., & Kang, S. (2011) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2011.01.009 · Cited by 38
“…methods (Hammontree et al., 1994; Hilbert and Redmiles, 1998, 1999), and usability studies have widely conducted to compare remote testing with laboratory testing (Andreasen et al., 2007; Brush et al., 2004; Hartson et al., 1996; McFadden et al., 2002; Petrie et al., 2006; Thompson et al., 2004).”
MethodConway, V., Brown, J., Hollier, S., & Nicholl, C. (2012) · The Australian Library Journal · doi:10.1080/00049670.2012.10736059 · Cited by 44
“In the work of Petrie et al. (2006) we see an assessment using people with disabilities to test the website accessibility.”
MethodResultInfluentialAndrzejczak, C., & Liu, D. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.01.052 · Cited by 38
“Petrie et al. (2006) found a specialized use for remote usability testing when testing those with disabilities.”
BackgroundHill, J. R., Brown, J. C., Campbell, N. L., & Holden, R. J. (2021) · JMIR Formative Research · doi:10.2196/26181 · Cited by 36
“Petrie et al [24] reported 2 case studies of remote usability testing—one with blind younger adults (n=8) and another with a more heterogeneous group of individuals with disabilities (n=51).”
BackgroundInfluentialAccessibility of Portuguese Public Universities' sites
Espadinha, C., Pereira, L. M., da Silva, F. M., & Lopes, J. B. (2011) · Disability and Rehabilitation · doi:10.3109/09638288.2010.498554 · Cited by 33
“Most authors choose Bobby as one of the automatic tools in their studies to assess the web accessibility [11,14,15,18–26], probably due to its 86% coverage of the WCAG 1.0 checkpoints.”
MethodInfluentialHill, J. R., Harrington, A. B., Adeoye, P., Campbell, N. L., & Holden, R. J. (2021) · JMIR mhealth and uhealth · doi:10.2196/26702 · Cited by 37
“These methods have, to our knowledge, not been adapted for fully remote projects with older adult users, although some work has been done with younger and older adults with various disabilities [18].”
BackgroundMethodAccessible on-line floor plans.
Goncu, C., Madugalla, A., Marinai, S., & Marriott, K. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741660 · Cited by 31
“It is worth pointing out that since there are relatively few blind people and it is often hard for them to travel, it is quite difficult to find blind participants (also pointed out in [26, 22]).”
BackgroundUnderstanding User Centred Design (UCD) for People with Special Needs
Thimbleby, H. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-70540-6_1 · Cited by 36
“bers of test subjects with specific special needs — studies show that special precautions are required [26] (a paper which is to be recommended for its principles).”
BackgroundTextSL: a command-based virtual world interface for the visually impaired
Folmer, E., Yuan, B., Carr, D., & Sapre, M. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639654 · Cited by 30
“A risk with remote evaluations is that user experience specialists often lack detailed understanding of how people with disabilities use their assistive technologies [26].”
Akter, T., Ahmed, T., Kapadia, A., & Swaminathan, M. (2022) · ACM Transactions on Accessible Computing · doi:10.1145/3506857 · Cited by 29
“Indeed, survey-based methods have been widely adopted in the accessibility community [37, 49, 94] and are often encouraged, for example, to avoid the difficulties that people with disabilities may face traveling to the facility [106] and are more suitable as participants use their preferred screen readers in their own…”
BackgroundThe interplay between web aesthetics and accessibility
Mbipom, G., & Harper, S. (2011) · The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/2049536.2049564 · Cited by 29
“As a result, such a person may be affected by multiple Web accessibility barriers which an evaluator may not be able to account for in one go [25, 34]; People with disabilities have different as sistive technologies and personal adaptations.”
BackgroundInfluentialSinabell, I., & Ammenwerth, E. (2022) · Universal Access in the Information Society · doi:10.1007/s10209-022-00949-w · Cited by 30
“…Think Aloud Protocol cannot beused as is, and will require modification to be useful, when evaluating websites with blind users” Petrie et al. [62] Software for blind people/ website Disabled (various impairment) Asynchronous/Synchronous Remote Usability Evaluation Challenges of conducting a…”
A Comparative Study of Synchronous and Asynchronous Remote Usability Testing Methods
Alghamdi, A. S., Al-Badi, A. H., & Mayhew, P. J. (2013) · Cited by 35
DIRETRIZES DE ACESSIBILIDADE: UMA ABORDAGEM COMPARATIVA ENTRE WCAG E E-MAG
Bach, C. F., Ferreira, S. B. L., da Silveira, D. S., & Nunes, R. R. (2009) · Revista Eletrônica de Sistemas de Informação · doi:10.5329/resi.2009.0801001 · Cited by 25
“Ou seja, não devemos esquecer que na Web encontramos todos os tipos de usuários e que a Web desempenha um papel importante no cotidiano de todas as pessoas, inclusive das que possuem necessidades especiais [Petrie et al. 2006].”
BackgroundHowie, S. R., & Gilardi, M. (2020) · Virtual Reality · doi:10.1007/s10055-020-00463-5 · Cited by 24
“Even with these forms of observation available, for remote studies conducted ‘in the wild’, setting up software or hardware for observing remote users could be a challenge (Petrie et al. 2006).”
BackgroundEvaluating a modified Google user interface via screen reader
Leporini, B., Andronico, P., Buzzi, M., & Castillo, C. (2008) · Universal Access in the Information Society · doi:10.1007/s10209-007-0111-y · Cited by 24
Exploratory Analysis of Collaborative Web Accessibility Improvement
Sato, D., Takagi, H., Kobayashi, M., Kawanaka, S., & Asakawa, C. (2010) · ACM Transactions on Accessible Computing · doi:10.1145/1857920.1857922 · Cited by 20
Contrasting usability evaluation methods with blind users
Miao, M., Pham, H. A., Friebe, J., & Weber, G. (2016) · Universal Access in the Information Society · doi:10.1007/s10209-014-0378-8 · Cited by 23
“[9] investigated local and asynchronous remote tests with blind participants.”
BackgroundEvaluating a search interface for visually impaired searchers
Sahib, N. G., Tombros, A., & Stockman, T. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23325 · Cited by 25
“Therefore, given the known difficulties in recruiting users with disabilities (Petrie, Hamilton, King, & Pavan, 2006), this setting allowed us to access a diverse set of users, and previous research (Bradner, 2004) has shown that remote user studies can be beneficial for formative evaluations because…”
BackgroundEvaluating accessibility-in-use
Vigo, M., & Harper, S. (2013) · International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461136 · Cited by 23
Online and ubiquitous HCI research
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00014-5 · Cited by 22
Working with research participants with disabilities
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00016-9 · Cited by 25
Including Heterogeneous Web Accessibility Guidelines in the Development Process
Arrue, M., Vigo, M., & Abascal, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-92698-6_37 · Cited by 29
“In our previous work [15], DOM and SAX technologies were used to navigate through the XML tree and the implementation required a big amount of source code compared with XQuery.”
BackgroundMethodPutting it into words : the impact of visual impairment on perception, experience and presence
Fryer, L. (2013) · Goldsmiths Research Online · Cited by 46
Sharif, A., Zhang, A. M., Reinecke, K., & Wobbrock, J. O. (2023) · 20th International Web for All Conference · doi:10.1145/3587281.3587284 · Cited by 21
“We were unsuccessful at finding SRUs who were actual teachers or news reporters, as is often the case with recruiting disabled participants, particularly those with specialties [65].”
BackgroundNour, R. (2022) · Computational and Mathematical Methods in Medicine · doi:10.1155/2022/8610844 · Cited by 24
“A tool-based audit to determine accessibility conformance and provide a report on violations can address some limitations of other methods, such as the cost of an expert assessment [13] or difficulties in recruiting disabled individuals for testing [14].”
MethodRutter, S., Zamani, E. D., McKenna-Aspell, J., & Wang, Y. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103278 · Cited by 22
Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility
Koutsabasis, P., Vlachogiannis, E., & Darzentas, J. (2010) · Journal of Usability Studies archive · doi:10.5555/2019116.2019120 · Cited by 21
Wood, R. E., Dixon, E., Elsayed-Ali, S., Shokeen, E., Lazar, A., & Lazar, J. K. (2021) · ACM Transactions on Accessible Computing · doi:10.1145/3460942 · Cited by 21
“Many researchers have examined the feasibility and benefits of remote usability testing methods [4, 26, 35, 70, 86].”
BackgroundMethodInfluentialNganji, J. T., & Brayshaw, M. (2015) · Science and Information Conference · doi:10.1109/sai.2015.7237128 · Cited by 20
“To successfully recommend suitable learning materials to a learner, the learning environment can make use of personalization.”
Improving deaf accessibility in remote usability testing
Schnepp, J. C., & Shiver, B. (2011) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049594 · Cited by 20
“Remote testing holds the potential to tap a large, geographically diverse Deaf population in a more cost-effective manner [6], particularly since many members of the Deaf community have embraced the Internet as a preferred means of communication [7].”
BackgroundA task‐based approach to assessing the accessibility of web sites
Craven, J., & Nietzio, A. (2007) · Performance Measurement and Metrics · doi:10.1108/14678040710760603 · Cited by 23
“Whilst there are arguments against it, there is evidence that remote testing can provide useful data even if it lacks the richness that can be extracted from face-to-face testing ( Petrie et al. , 2006”
BackgroundVolunteer-Based Online Studies With Older Adults and People with Disabilities
Li, Q., Gajos, K. Z., & Reinecke, K. (2018) · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3234695.3236360 · Cited by 20
“However, given that recruitment of specific populations is immensely difficult [12, 37], most studies with people with disabilities and older adults are forced to rely on small num bers.”
BackgroundCreating accessible online floor plans for visually impaired readers
Madugalla, A., Marriott, K., Marinai, S., Capobianco, S., & Goncu, C. (2020) · ACM Transactions on Accessible Computing · doi:10.1145/3410446 · Cited by 17
“As discussed by Petrie et al. [58], finding participants for studies is difficult, in part because of the difficulty that blind people face in traveling.”
BackgroundA large user pool for accessibility research with representative users
Dee, M., & Hanson, V. L. (2014) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2661334.2661361 · Cited by 17
“The accessibility literature often publishes on the challenges of recruiting participants, and has looked for new and creative ways of obtaining participants, such as online testing [3] [29].”
BackgroundA Pool of Representative Users for Accessibility Research: Seeing through the Eyes of the Users
Dee, M., & Hanson, V. L. (2016) · ACM Transactions on Accessible Computing · doi:10.1145/2845088 · Cited by 17
“The accessibility literature often publishes on the challenges of recruiting participants and has looked for new and creative ways of obtaining participants, such as online recruitment and testing [Bigham and Cavender 2009; Petrie et al. 2006].”
BackgroundTowards a tool for keystroke level modeling of skilled screen reading
Trewin, S., John, B. E., Richards, J., Swart, C., Brezin, J., Bellamy, R., & Thomas, J. (2010) · Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1878803.1878811 · Cited by 17
“However, quantitative data were sufficient for this purpose, and remote evaluation can be an effective way to gather such data from participants with disabilities over a long distance [19].”
BackgroundPower, C., & Petrie, H. (2019) · Human-computer interaction series · doi:10.1007/978-1-4471-7440-0_9 · Cited by 16
Self-citationEvaluating the design of inclusive interfaces by simulation
Biswas, P., & Robinson, P. (2010) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1719970.1720010 · Cited by 16
“Petrie and colleagues [36] take the approach of remote evaluation but can not avoid the need to find disabled participants.”
BackgroundRamôa, G., Moured, O., Schwarz, T., Müller, K., & Stiefelhagen, R. (2023) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3594806.3594818 · Cited by 15
“BVI people represent only 3.3% of the population in Europe [4], have mobility problems, and high unemployment rates [5, 12, 17, 19].”
BackgroundInfluentialEmpirical results from an evaluation of the accessibility of websites by dyslexic users
Freire, A. P., Petrie, H., & Power, C. (2011) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 15
Self-citationUnderstanding User Sensemaking in Machine Learning Fairness Assessment Systems
Gu, Z., Yan, J. N., & Rzeszotarski, J. M. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450092 · Cited by 20
“Though we were initially leery of conducting remote usability studies, in the past asynchronous [5] and synchronous [54] usability studies have proved effective,”
BackgroundPaternò, F., & Santoro, C. (2007) · Human-computer interaction series · doi:10.1007/978-1-84628-941-5_9 · Cited by 15
“Before analyzing the last dimension of the proposed framework in depth—the type of results an evaluation can deliver (i.e., qualitative vs. quantitative data)—we judge it useful to mention the work by Petrie et al. (2006) about remote evaluation.”
Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture
Oney, S., Lundgard, A., Krosnick, R., Nebeling, M., & Lasecki, W. S. (2018) · Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3242587.3242649 · Cited by 13
“• Tasks were run remotely, allowing participants to use their preferred screen reader and environment, which prior work has considered more ecologically valid [11, 55].”
BackgroundDiretrizes de Acessibilidade: Uma Abordagem Comparativa entre WCAG e e-MAG
Bach, C. F., Ferreira, S. B. L., da Silveira, D. S., & Nunes, R. R. (2009) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.5753/sbsi.2009.6167 · Cited by 14
Mastrianni, A., Kulp, L., & Sarcevic, A. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3443444 · Cited by 14
“Because researchers are not contained by budget and time to a specifc geographic area, remote testing can reach a wider range of participants, such as those with disabilities [28].”
BackgroundYao, T., Yoo, S., & Parker, C. (2021) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3520495.3520519 · Cited by 15
WebAnywhere: a screen reader on-the-go
Bigham, J. P., & Prince, C. M. (2007) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1296843.1296884 · Cited by 15
“Such studies have been shown to yield similar quantitative results, although risk collecting less-informative qualitative feedback [25].”
ResultEvaluation of software tools with deaf children
Mich, O. (2009) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1639642.1639692 · Cited by 14
How Do Low-Vision Individuals Experience Information Visualization?
Wang, Y., Zhao, Y., & Kim, Y. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642188 · Cited by 16
“While an online study may be the best solution for reaching a broader population [17, 52], diferent insights could be derived with in-person studies.”
Integrating Accessibility Evaluation into Web Engineering Processes
Power, C., Freire, A. P., & Petrie, H. (2009) · International Journal of Information Technology and Web Engineering · doi:10.4018/jitwe.2009100104 · Cited by 14
Self-citationSimulation to predict performance of assistive interfaces
Biswas, P., & Robinson, P. (2007) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1296843.1296885 · Cited by 13
“Practical user trials are also difficult and time consuming.”
Background‘Did You See That!?’ Enhancing the Experience of Sports Media Broadcast for Blind People
Goncu, C., & Finnegan, D. J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85623-6_24 · Cited by 16
“As we aware of disadvantages and problems around conducting remote evaluations with participants with disabilities [31], we gathered feedback on the online form from a member of the blind community working at Monash University before recruiting our participants.”
MethodSloan, D. R. (2006) · Discovery Research Portal (University of Dundee) · Cited by 14
Remote Inclusion of Vulnerable Users in mHealth Intervention Design: Retrospective Case Analysis
Straand, I. J., Baxter, K. A., & Følstad, A. (2024) · JMIR mhealth and uhealth · doi:10.2196/55548 · Cited by 10
“The benefits of a wider recruitment pool and efficiency gains, such as reduced travel time or inconvenience, were expected from past studies on remote research methods [15,20,57].”
BackgroundArfini, S., Bellani, P., Picardi, A., Yan, M., Fossa, F., & Caruso, G. (2023) · Studies in applied philosophy, epistemology and rational ethics · doi:10.1007/978-3-031-39991-6_4 · Cited by 10
Design and development of sign language questionnaires based on video and web interfaces
López, J. P., Bosch-Baliarda, M., Martín, C. A., Menéndez, J. M., Orero, P., Vilageliú, O. S., & Álvarez, F. (2019) · Artificial intelligence for engineering design analysis and manufacturing · doi:10.1017/s0890060419000374 · Cited by 10
“When online research is carried out, it becomes easier to increase the participation in the recruiting process and to reach users that would have been difficult to contact through face-to-face inter-views, especially in the case of people with disabilities (Petrie et al ., 2006).”
An Empirical Study to Evaluate the Accessibility of Arabic Websites by Low Vision Users
Akram, M. I., & Sulaiman, R. B. (2020) · International Conference on Information Technology and Multimedia (ICIMU) · doi:10.1109/icimu49871.2020.9243565 · Cited by 10
“But is it noted that quantitative data collected during remote assessment and laboratory session is somehow similar, but qualitative data collected during remote evaluation is not the same as evaluation performed in laboratory session [39].”
BackgroundAccessible Document Layout: An Interface for 2D Tactile Displays
Moured, O., Alzalabny, S., Schwarz, T., Rapp, B. E., & Stiefelhagen, R. (2023) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3594806.3594811 · Cited by 10
“BVI people represent only 3.3% of the population in Europe [10], have mobility problems, and high unemployment rates [12, 28, 32].”
BackgroundInfluentialLee, K., Hong, J., Jarjue, E., Mensah, E. E., & Kacorri, H. (2022) · International Web for All Conference (W4A) · doi:10.1145/3493612.3520448 · Cited by 10
“Approaches and tools for supporting remote user studies play an important role in human-computer interaction as they can help overcome some of the limitations of in-person studies, such as low recruitment of representative participants [38].”
BackgroundMethodInfluentialBorsci, S., Federici, S., Mele, M. L., & Conti, M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21006-3_4 · Cited by 9
Simulating HCI for special needs
Biswas, P. (2007) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/1328567.1328569 · Cited by 9
“[6] take the approach of remote evaluation but still need to find disabled participants.”
BackgroundBorwing, A. H., Bransford, J. D., & Deibel, K. (2011) · ResearchWorks at the University of Washington (University of Washington) · Cited by 14
The Trade-off between Usability and Security in the Context of eGovernment: A Mapping Study
Alshamsi, A., Williams, N., & András, P. (2016) · Electronic workshops in computing · doi:10.14236/ewic/hci2016.36 · Cited by 8
Remote Usability Testing Using Eyetracking
Chynał, P., & Szymański, J. M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23774-4_29 · Cited by 8
“So far remote usability tests are evaluated using standard methods such as remote surveys and video conferences [1], [7].”
Method
Metadata
- Authors
- Petrie, H., Hamilton, F., King, N., & Pavan, P.
- Affiliations
- University of York (GB)
- Designed for All, London, United Kingdom
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2006-04-22
- Pages
- 1133-1141
- OpenAlex
- W1973244543
- Semantic Scholar
- 779ceea17ddd7221f875c644ec05df9cb4812996
- Cited by
- 157
- References
- 22
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 8 · 2023: 7 · 2022: 8 · 2021: 13 · 2020: 8 · 2019: 5 · 2018: 2 · 2017: 7 · 2016: 6 · 2015: 8 · 2014: 3 · 2013: 9 · 2012: 7
- FWCI
- 9.99
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- remote usability evaluation, disabled users, participant recruitment, formative assessment, summative assessment, accessibility, user experience evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.