Volunteer-Based Online Studies With Older Adults and People with Disabilities
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
There are few large-scale empirical studies with people with disabilities or older adults, mainly because recruiting participants with specific characteristics is even harder than recruiting young and/or non-disabled populations. Analyzing four online experiments on LabintheWild with a total of 355,656 participants, we show that volunteer-based online experiments that provide personalized feedback attract large numbers of participants with diverse disabilities and ages and allow robust studies with these populations that replicate and extend the findings of prior laboratory studies. To find out what motivates people with disabilities to take part, we additionally analyzed participants' feedback and forum entries that discuss LabintheWild experiments. The results show that participants use the studies to diagnose themselves, compare their abilities to others, quantify potential impairments, self-experiment, and share their own stories -- findings that we use to inform design guidelines for online experiment platforms that adequately support and engage people with disabilities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Discovery of Grounded Theory; Strategies for Qualitative Research
Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968) · Nursing Research · doi:10.1097/00006199-196807000-00014 · Cited by 39,449
Perceiving an Object and Its Context in Different Cultures: A Cultural Look at New Look
Kitayama, S., Duffy, S., Kawamura, T., & Larsen, J. T. (2003) · Psychological Science · doi:10.1111/1467-9280.02432 · Cited by 820
Hartshorne, J. K., & Germine, L. T. (2015) · Psychological Science · doi:10.1177/0956797614567339 · Cited by 788
Germine, L., Nakayama, K., Duchaine, B. C., Chabris, C. F., Chatterjee, G., & Wilmer, J. B. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0296-9 · Cited by 669 · cited 2×
“Despite initial concerns about the quality of data collected from unsupervised online workers, robust and validated data quality methodologies have been de veloped for conducting a broad range of experiments, yielding results comparable to those obtained in conventional labora tory settings [18, 16, 20, 32, 39].”
BackgroundResultVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601
Memory load and the cognitive pupillary response in aging
Van Gerven, P. W. M., Paas, F., Van Merriënboer, J. J. G., & Schmidt, H. G. (2004) · Psychophysiology · doi:10.1111/j.1469-8986.2003.00148.x · Cited by 292 · cited 3×
“given that our large number of participants allowed us to an alyze the effect for all ages (rather than binning them into discrete “young” and “old” groups as in [47]).”
BackgroundResultInfluentialA convenient solution: Using MTurk to sample from hard-to-reach populations
Smith, N. A., Sabat, I. E., Martinez, L. R., Weaver, K., & Xu, S. (2015) · Industrial and Organizational Psychology · doi:10.1017/iop.2015.29 · Cited by 190 · cited 2×
“To study older adults and people with disabilities, some re searchers have therefore turned to the online labor markets, such as Amazon Mechanical Turk (MTurk) [44, 11, 41], which support the efficient recruitment of participants at low cost.”
Remote usability evaluations with disabled people.
Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124942 · Cited by 157 · cited 2×
“However, given that recruitment of specific populations is immensely difficult [12, 37], most studies with people with disabilities and older adults are forced to rely on small num bers.”
Background"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
Brewer, R., Morris, M. R., & Piper, A. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858198 · Cited by 111
Effect of age and Parkinson's disease on cursor positioning using a mouse
Keates, S., & Trewin, S. (2005) · Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1090785.1090800 · Cited by 100 · cited 4×
“Older adults have lower peak speeds than young adults [26].”
BackgroundMethodInfluentialRepresenting users in accessibility research
Sears, A., & Hanson, V. L. (2012) · ACM Transactions on Accessible Computing · doi:10.1145/2141943.2141945 · Cited by 79 · cited 1×
“They also risk generalizability and data quality if the same small number of participants repeatedly take part in similar studies [40].”
A Large Inclusive Study of Human Listening Rates
Bragg, D., Bennett, C., Reinecke, K., & Ladner, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174018 · Cited by 43 · cited 5×
“If the quote was taken from a comment in an LabintheWild experiment, the specific experiment and participant number is noted in brackets (with the exception of the Listening Rate Test, which only recorded session IDs).”
BackgroundInfluentialSelf-citation
Cited by
Li, Q., Joo, S. J., Yeatman, J. D., & Reinecke, K. (2020) · Scientific Reports · doi:10.1038/s41598-019-57204-1 · Cited by 183
Self-citationDo People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
“Both curiosity and opportunities for social comparison have been shown to increase engagement of online participants [32, 42] and multiple validation studies demonstrated that data collected on LabintheWild and other similar platforms are valid and lead to the same conclusions as data collected in traditional laborato…”
ResultSelf-citationMack, K., McDonnell, E., Jain, D., Lu Wang, L., E. Froehlich, J., & Findlater, L. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445412 · Cited by 7
“, [92])—particularly crowdsourced studies—and seven papers overall having only one or two participants.”
Background
Metadata
- Authors
- Li, Q., Gajos, K. Z., & Reinecke, K.
- Venue
- Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility
- Publisher
- ACM
- Published
- 2018-10-08
- Pages
- 229-241
- OpenAlex
- W2896439933
- Cited by
- 20
- References
- 12
- FWCI
- 2.85
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.