#Bias: The Opportunities and Challenges of Surveys That Recruit and Collect Data of Autistic Adults Online
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Internet-based online surveys are a crucial tool for researchers to learn about the understudied and often overlooked population of autistic adults. The recruitment and administration of online surveys can be cheaper, quicker, and have a wider reach compared with more traditional in-person methods. As online surveys become more prevalent, it is important to place strengths in the context of limitations and biases that can arise when recruiting and administering surveys online. In this perspective, we discuss two common issues that often appear in studies that use online tools to recruit and administer surveys to autistic adults and nonautistic volunteers: selection bias and sample identifiability. Selection bias is the distortion in effect estimates (e.g., relative risk, risk ratio, incidence rate) resulting from the factors that influence why a person chose to participate or how the researcher recruits and selects participants in a study. Sampling identifiability is the ability (or inability) to quantify and define the population of interest. We use a case example of an online survey study of suicidal ideation in autistic adults and describe how issues in selection bias and sample identifiability arise and may lead to challenges unique to studying autistic adults. We conclude with recommendations to improve the quality and utility of online survey research in autistic adults. Using online resources to recruit and collect data on autistic adults is an incredible tool with great potential; yet, authors need to consider the limitations, potential biases, and tools to overcome systematic error at each stage of the study. Lay summary What is the purpose of this article? Our purpose was to describe challenges in conducting and analyzing data from surveys of autistic adults, recruited and completed online. What is already known on the topic? Health outcomes for autistic adults are understudied by crucial areas of autism research. While researchers are interested in the outcomes of autistic adults, this type of research is difficult because many autistic adults are not formally diagnosed, so not available to recruit for studies through clinic registries. Furthermore, study participation can be a long, inconvenient, and stressful process. It is not surprising then that we are seeing internet surveys of autistic adults become a popular tool to reach this population. We wanted to offer an overview and recommendations of these issues to researchers and people who read research about topics pertaining to autistic adults. What are the perspectives of the authors? We are epidemiologists at Boston University and the University of Wisconsin-Madison. We both conduct research centered in improving health and well-being for autistic people across the life span. As people who study research methods, we have seen a lot of new research using survey methods. This research is intriguing, but all too often the articles need more information so we can be sure that the research is high quality. We want to share ways to improve this type of research and to help people in understanding the strengths and limitations of online survey research. What do the authors recommend? We offer a few considerations for researchers working in this area. (1) Make the steps you took to do the research as clear as possible. (2) Be specific about who you intend to study and who you ended up studying. (3) Present the demographics and characteristics of the participants. (4) If possible, consider using data analysis methods to account for selection bias and sample identifiability issues. (5) Do not make statements that are not supported for your study results. (6) Acknowledge that we are at the beginning of studying autistic adults. (7) Advocate for more funding for research in autistic adults. How will these recommendations help autistic adults now or in the future? Online surveys are an important tool for researchers to generate hypotheses and connect with a wider range of participants. However, online surveys have unique methodological challenges. We hope that this perspective raises the topic of bias and misinterpretation in online surveys and researchers continue to produce valid and meaningful work that is crucial to improving lives of autistic adults.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A structural approach to selection bias
Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004) · Epidemiology · doi:10.1097/01.ede.0000135174.63482.43 · Cited by 2,844 · cited 2×
“We used a directed acyclic graph 37 (Fig.”
Suicide, Suicide Attempts, and Suicidal Ideation
Klonsky, E. D., May, A. M., & Saffer, B. Y. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093204 · Cited by 1,675 · cited 1×
“We will use the association between autism and suicidal ideation (thinking about, considering, or planning suicide) 24 as the basis for understanding potential issues in online survey methods.”
Sex/Gender Differences and Autism: Setting the Scene for Future Research
Lai, M.-C., Lombardo, M. V., Auyeung, B., Chakrabarti, B., & Baron-Cohen, S. (2015) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2014.10.003 · Cited by 1,120
Delgado-Rodríguez, M., & Llorca, J. (2004) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2003.008466 · Cited by 1,064
Identifying the lost generation of adults with autism spectrum conditions
Lai, M.-C., & Baron-Cohen, S. (2015) · The Lancet Psychiatry · doi:10.1016/s2215-0366(15)00277-1 · Cited by 756 · cited 1×
“Administrative data may miss autistic adults who lack access to care because of known disparities, autistic adults without an autism diagnosis, or autistic adults who chose not to disclose their autism, 29–31 all of whom may be at the highest risk for suicide ideation.”
Risk markers for suicidality in autistic adults
Cassidy, S., Bradley, L., Shaw, R., & Baron-Cohen, S. (2018) · Molecular Autism · doi:10.1186/s13229-018-0226-4 · Cited by 621 · cited 2×
“A systematic review by Hedley and Uljarevic´ found that the prevalence of suicidal ideation ranged from 11% to 66% 25 and Cassidy et al. 11 reported that 72% of autistic adults scored above the cutoff for suicide risk in an online survey.”
Warrier, V., Greenberg, D. M., Weir, E., Buckingham, C., Smith, P., Lai, M.-C., Allison, C., & Baron-Cohen, S. (2020) · Nature Communications · doi:10.1038/s41467-020-17794-1 · Cited by 595
Cassidy, S., Bradley, P., Robinson, J., Allison, C., McHugh, M., & Baron-Cohen, S. (2014) · The Lancet Psychiatry · doi:10.1016/s2215-0366(14)70248-2 · Cited by 432 · cited 2×
“26 With the importance of acting quickly to change policy and practice, 32 using efficient online surveys in an unbiased and methodologically sound manner becomes more vital.”
Reporting Guidelines for Survey Research: An Analysis of Published Guidance and Reporting Practices
Bennett, C., Khangura, S., Brehaut, J. C., Graham, I. D., Moher, D., Potter, B. K., & M. Grimshaw, J. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1001069 · Cited by 397 · cited 2×
“…challenges are similar to issues in survey research in general, including relying on participant self-report and poor description of study methods 19 ; however, the paucity of research in autistic adults leaves us without the baseline knowledge needed to evaluate internal and external validity…”
Russell, G., Mandy, W., Elliott, D., White, R., Pittwood, T., & Ford, T. (2019) · Molecular Autism · doi:10.1186/s13229-019-0260-x · Cited by 366
Recruiting for health, medical or psychosocial research using Facebook: Systematic review
Thornton, L., Batterham, P. J., Fassnacht, D. B., Kay-Lambkin, F., Calear, A. L., & Hunt, S. (2016) · Internet Interventions · doi:10.1016/j.invent.2016.02.001 · Cited by 326 · cited 1×
“40,41 In Figure 2, we have created four different scenarios.”
Diagnosis of autism in adulthood: A scoping review
Huang, Y., Arnold, S. R., Foley, K.-R., & Trollor, J. N. (2020) · Autism · doi:10.1177/1362361320903128 · Cited by 289 · cited 1×
“17 While online surveys have strengths, the ease of implementation and nearly limitless opportunity for participation and recruitment can create bias and lead to misinterpretation in research studies.”
A 20‐year study of suicide death in a statewide autism population
Kirby, A. V., Bakian, A. V., Zhang, Y., Bilder, D. A., Keeshin, B. R., & Coon, H. (2019) · Autism Research · doi:10.1002/aur.2076 · Cited by 192
Gillespie-Lynch, K., Kapp, S. K., Shane-Simpson, C., Smith, D. S., & Hutman, T. (2014) · Intellectual and developmental disabilities · doi:10.1352/1934-9556-52.6.456 · Cited by 182 · cited 1×
“Specifically in regard to autism, the vibrant and diverse population of autistic adults who use internet platforms to communicate and find community, with a fairly broad range of verbal language abilities 15 can be leveraged to create more inclusive and representative study samples.”
Systematic Review of Suicide in Autism Spectrum Disorder: Current Trends and Implications
Hedley, D., & Uljarević, M. (2018) · Current Developmental Disorders Reports · doi:10.1007/s40474-018-0133-6 · Cited by 176 · cited 2×
“25 Demographic differences in samples may lead to differential associations in subgroups of autistic adults.”
Social media use among adults with autism spectrum disorders
Mazurek, M. O. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.02.004 · Cited by 159
Understanding and prevention of suicide in autism
Cassidy, S., & Rodgers, J. (2017) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30162-1 · Cited by 119 · cited 1×
“26 With the importance of acting quickly to change policy and practice, 32 using efficient online surveys in an unbiased and methodologically sound manner becomes more vital.”
Social anxiety in adult autism spectrum disorder
Bejerot, S., Eriksson, J. M., & Mörtberg, E. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.08.030 · Cited by 111 · cited 1×
“16 Online surveys can also reach participants without a formal diagnosis of autism, which can be difficult for many undiagnosed adults to attain.”
Cassidy, S. A., Bradley, L., Bowen, E., Wigham, S., & Rodgers, J. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2018.05.002 · Cited by 104 · cited 1×
“27 As a public health issue, it is important to accurately describe the extent of suicidal ideation in autistic people so that we have accurate estimates of occurrence of ideation and prevalence of factors associated with increased risk.”
Brief Report: A Survey of Autism Research Priorities Across a Diverse Community of Stakeholders
Frazier, T. W., Dawson, G., Murray, D., Shih, A., Sachs, J. S., & Geiger, A. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3642-6 · Cited by 97
Rai, D., Heuvelman, H., Dalman, C., Culpin, I., Lundberg, M., Carpenter, P., & Magnusson, C. (2018) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.1465 · Cited by 93 · cited 1×
“The solid lined arrows illustrate causal pathways that include intermediary variables, for example, through disparity and inequality autistic people develop depression, 38 which can lead to suicidal ideation.”
Roestorf, A., Bowler, D. M., Deserno, M. K., Howlin, P., Klinger, L., McConachie, H., Parr, J. R., Powell, P., Van Heijst, B. F. C., & Geurts, H. M. (2019) · Research in Autism Spectrum Disorders · doi:10.1016/j.rasd.2018.08.007 · Cited by 87
Cited by
Jadav, N., & Bal, V. H. (2022) · Autism Research · doi:10.1002/aur.2808 · Cited by 95
Moore, H. L., Brice, S., Powell, L., Ingham, B., Freeston, M., Parr, J. R., & Rodgers, J. (2022) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-021-05312-1 · Cited by 42
Are Robots Ready to Deliver Autism Inclusion?: A Critical Review
Rizvi, N., Wu, W., Bolds, M., Mondal, R., Begel, A., & Munyaka, I. N. S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642798 · Cited by 30
Metadata
- Authors
- Rubenstein, E., & Furnier, S.
- Venue
- Autism in Adulthood
- Publisher
- SAGE Publications
- Published
- 2021-06-01
- Volume
- 3
- Issue
- 2
- Pages
- 120-128
- ISSN
- 2573-9581 (ISSN-L)
- 2573-959X
- OpenAlex
- W3102111285
- Cited by
- 65
- References
- 22
- FWCI
- 5.95
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.