Usage of Subjective Scales in Accessibility Research
Abstract
Accessibility research studies often gather subjective responses to technology using Likert-type items, where participants respond to a prompt statement by selecting a position on a labeled response scale. We analyzed recent ASSETS papers, and found that participants in non-anonymous accessibility research studies gave more positive average ratings than those in typical usability studies, especially when responding to questions about a proposed innovation. We further explored potential positive response bias in an experimental study of two telephone information systems, one more usable than the other. We found that participants with visual impairment were less sensitive to usability problems than participants in a typical student sample, and that their subjective ratings didn't correlate as strongly with objective measures of performance. A deeper understanding of the mechanism behind this effect would help researchers to design better accessibility studies, and to interpret subjective ratings with more accuracy.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There exists the bias of comparing a new system to an old one when participants know which system was designed by the researchers.
“Second, there exists the bias of comparing a new system to an old one when participants know which system was designed by the researchers [5, 33].”
From Revamp: Enhancing Accessible Information Seeking Experience of Online Shopping for Blind or Low Vision Users (2021) · cites this paper as [33]
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · see citing works
Two-component models of socially desirable responding.
Paulhus, D. L. (1984) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.46.3.598 · Cited by 3,013 · see citing works
(2013) · Journal of Usability Studies archive · Cited by 1,357 · see citing works
Research Methods in Human-Computer Interaction
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2010) · doi:10.1016/b978-0-12-805390-4.09991-x · Cited by 1,012 · see citing works
Richman, W. L., Kiesler, S. B., Weisband, S. P., & Drasgow, F. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.5.754 · Cited by 821 · see citing works
Comparison of three one-question, post-task usability questionnaires
Sauro, J., & Dumas, J. S. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518946 · Cited by 511 · see citing works
Nielsen, J., & Levy, J. L. (1994) · Communications of the ACM · doi:10.1145/175276.175282 · Cited by 461 · see citing works
Determining what individual SUS scores mean
(2009) · Journal of Usability Studies archive · doi:10.5555/2835587.2835589 · Cited by 412 · see citing works
Development of the Quebec User Evaluation of Satisfaction with assistive Technology (QUEST)
Demers, L., Weiss‐Lambrou, R., & Ska, B. (1996) · Assistive Technology · doi:10.1080/10400435.1996.10132268 · Cited by 354 · see citing works
The impact of administration mode on response effects in survey measurement
Schwarz, N., Strack, F., Hippler, H., & Bishop, G. F. (1991) · Applied Cognitive Psychology · doi:10.1002/acp.2350050304 · Cited by 278 · see citing works
Dell, N. L., Vaidyanathan, V., Medhi, I., Cutrell, E. B., & Thies, W. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208589 · Cited by 267 · see citing works
Correlations among prototypical usability metrics
Sauro, J., & Lewis, J. R. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518947 · Cited by 221 · see citing works
Perceived disorientation: an examination of a new measure to assess web design effectiveness
Ahuja, J. S., & Webster, J. (2001) · Interacting with Computers · doi:10.1016/s0953-5438(01)00048-0 · Cited by 182 · see citing works
Powerful and consistent analysis of likert-type rating scales
Kaptein, M. C., Nass, C., & Markopoulos, P. (2010) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1753326.1753686 · Cited by 130 · see citing works
Assessing the deaf user perspective on sign language avatars
Kipp, M., Nguyen, Q., Héloir, A., & Matthes, S. (2011) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049557 · Cited by 103 · see citing works
Representing users in accessibility research
Sears, A., & Hanson, V. L. (2012) · ACM Transactions on Accessible Computing · doi:10.1145/2141943.2141945 · Cited by 79 · see citing works
DEEP: Design-Oriented Evaluation of Perceived Usability
Yang, T., Linder, J., & Bolchini, D. P. (2011) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2011.586320 · Cited by 63 · see citing works
Wheelchair-based game design for older adults
Gerling, K., Mandryk, R. L., & Kalyn, M. R. (2013) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2513383.2513436 · Cited by 43 · see citing works
Exploring Visual and Motor Accessibility in Navigating a Virtual World
Trewin, S. M., Laff, M. R., Hanson, V. L., & Cavender, A. C. (2009) · ACM Transactions on Accessible Computing · doi:10.1145/1530064.1530069 · Cited by 30 · see citing works
Self-citationHow voice augmentation supports elderly web users
Sato, D., Kobayashi, M., Takagi, H., Asakawa, C., & Tanaka, J. (2011) · The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/2049536.2049565 · Cited by 26 · see citing works
Evaluating quality and comprehension of real-time sign language video on mobile phones
Tran, J. J., Kim, J. O., Chon, J., Riskin, E. A., Ladner, R. E., & Wobbrock, J. O. (2011) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049558 · Cited by 22 · see citing works
A large user pool for accessibility research with representative users
Dee, M., & Hanson, V. L. (2014) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2661334.2661361 · Cited by 17 · see citing works
Flagged first, then by citations
2 of the 24 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
(2017) · doi:10.1145/2998181.2998364 · Cited by 258 · see citing works
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K. I., Yu, L., Shan, Q., Nichols, J. W., Wu, J., Fleizach, C., et al. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 145 · see citing works
The Placebo Effect of Artificial Intelligence in Human–Computer Interaction
Kosch, T., Welsch, R., Chuang, L., & Schmidt, A. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3529225 · Cited by 137 · see citing works
Review of Quantitative Empirical Evaluations of Technology for People with Visual Impairments
Brulé, E., Tomlinson, B. J., Metatla, O., Jouffrais, C., & Serrano, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376749 · Cited by 76 · see citing works
Abdolrahmani, A., Easley, W., Williams, M., Branham, S. M., & Hurst, A. K. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025528 · Cited by 58 · see citing works
Alonzo, O., Seita, M., Glasser, A., & Huenerfauth, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376563 · Cited by 46 · see citing works
What My Eyes Can't See, A Robot Can Show Me
Bonani, M., Oliveira, R., Correia, F., Rodrigues, A., Guerreiro, T., & Paiva, A. (2018) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3234695.3239330 · Cited by 41 · see citing works
Self-citationWang, R., Chen, Z., Zhang, M. R., Li, Z., Liu, Z., Dang, Z., Yu, C., & Chen, X. '. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445547 · Cited by 42 · see citing works
Kane, S. K., Guo, A., & Morris, M. R. (2020) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3416990 · Cited by 36 · see citing works
Lessons Learned in Designing AI for Autistic Adults
Begel, A., Tang, J., Andrist, S., Barnett, M., Carbary, T., Choudhury, P., Cutrell, E., Fung, A., Junuzovic, S., McDuff, D., et al. (2020) · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3373625.3418305 · Cited by 34 · see citing works
Hati, S. R. H., Fitriasih, R., & Safira, A. (2019) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-11-2018-0081 · Cited by 32 · see citing works
Accept or Address? Researchers’ Perspectives on Response Bias in Accessibility Research
Ming, J., Heung, S., Azenkot, S., & Vashistha, A. (2021) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3471216 · Cited by 27 · see citing works
Grussenmeyer, W., Garcia, J., & Jiang, F. (2016) · International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/2935334.2935367 · Cited by 23 · see citing works
Vashistha, A., Okeke, F., Anderson, R., & Dell, N. L. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174126 · Cited by 23 · see citing works
AlMeraj, Z., Alhuwail, D., Qadri, R., Shama, S., & Crabb, M. (2024) · Universal Access in the Information Society · doi:10.1007/s10209-023-00980-5 · Cited by 21 · see citing works
Working with research participants with disabilities
Lazar, J. K., Feng, J. H., & Hochheiser, H. (2017) · Elsevier eBooks · doi:10.1016/b978-0-12-805390-4.00016-9 · Cited by 21 · see citing works
“I knew that, I was just testing you”
Franz, R. L., Baecker, R., & Truong, K. N. (2018) · ACM Transactions on Accessible Computing · doi:10.1145/3226115 · Cited by 17 · see citing works
Usability Is Ageless: Conducting Usability Tests with Older Adults
Franz, R. L., & Neves, B. B. (2019) · Springer eBooks · doi:10.1007/978-981-13-3693-5_7 · Cited by 17 · see citing works
Alonzo, O., Trussell, J., Dingman, B., & Huenerfauth, M. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445038 · Cited by 16 · see citing works
Accessibility-Related Publication Distribution in HCI Based on a Meta-Analysis
Colley, M., Kränzle, T., & Rukzio, E. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519701 · Cited by 15 · see citing works
Verma, S., Gonthina, P. S., Hawks, Z., Nahar, D., Brooks, J. O., Walker, I. D., Wang, Y., de Aguiar, C., & Green, K. E. (2018) · International Conference on Pervasive Computing Technologies for Healthcare · doi:10.1145/3240925.3240978 · Cited by 13 · see citing works
Williams, K., Clarke, T., Gardiner, S., Zimmerman, J., & Tomasic, A. (2019) · ACM Transactions on Accessible Computing · doi:10.1145/3342282 · Cited by 12 · see citing works
DREEM: Moving from Empathy to Enculturation in Disability Related Human-Centered Design
Baltaxe-Admony, L. B., Duval, J., & Ringland, K. E. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675642 · Cited by 9 · see citing works
Bajcar, B., Borkowska, A., & Jach, K. (2020) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2020.1798084 · Cited by 8 · see citing works
Mack, K., McDonnell, E., Jain, D., Lu Wang, L., E. Froehlich, J., & Findlater, L. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445412 · Cited by 7 · see citing works
Remote Usage Data Collection and Analysis for Mobile Accessibility Applications
Ahmetovic, D., Bernareggi, C., Ducci, M., Gerino, A., & Mascetti, S. (2021) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops51409.2021.9430968 · Cited by 4 · see citing works
Large-Scale Anonymized Text-based Disability Discourse Dataset
Palonis, B., Dobesh, S. J., Bellscheidt, S., Mkaouer, M. W., Liu, Y., & Elglaly, Y. N. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3614476 · Cited by 4 · see citing works
de Oliveira, B., Braga, J. C., & Damaceno, R. J. P. (2018) · International Web for All Conference (W4A) · doi:10.1145/3192714.3192838 · Cited by 4 · see citing works
Investigating the Acceptance of Voice User Interfaces for Users With Disabilities
Jin, D. H., Kim, Y. M., & Lee, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3520149 · Cited by 2 · see citing works
Validating a Georeferenced Map Viewer Through Online and Manual Tests
Vaca, M. Á. D., Romero, F., & Builes, J. A. J. (2019) · International Conference on Inclusive Technologies and Education (CONTIE) · doi:10.1109/contie49246.2019.00026 · Cited by 1 · see citing works
Enhancing Quality of Life for People who are Blind or Low Vision Using Computing Technology
Rector, K. (2016) · ResearchWorks at the University of Washington (University of Washington)
Taxonomy for Tactile Interaction on Large Braille Displays
Adhami, H., Murr, P., Honcharov, A., & Mogilei, S. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-71801-4_17
Tabletop tangible maps and diagrams for visually impaired users
Ducasse, J. (2017) · doi:10.70675/3c9498ddzfbdbz4ab7z8bafz7dda0a7a88a5 · Cited by 1 · see citing works
- Authors
- Trewin, S., Marques, D., & Guerreiro, T.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- University of Lisbon (PT)
- Venue
- Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS '15
- Publisher
- ACM Press
- Published
- 2015
- Pages
- 59-67
- OpenAlex
- W2029870508
- Cited by
- 35
- References
- 24
- Citations by year
- 2024: 3 · 2023: 2 · 2022: 2 · 2021: 8 · 2020: 5 · 2019: 4 · 2018: 5 · 2017: 4 · 2016: 2
- FWCI
- 2.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion)
- Funders
- FCT (UID/CEC/ 00408/2013 SFRH/BD/98527/2013)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Keywords
- accessibility research, subjective ratings, response bias, Likert scale, visual impairment, usability evaluation, assets, objective performance measures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
