Cited by 1,072Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseEthics in Clinical Research

The DOI matches its Crossref record; 174 of the 177 references it cites were found in a registry or an index.

Conducting Clinical Research Using Crowdsourced Convenience Samples

Chandler, Jesse, Shapiro, Danielle·2016·Annual Review of Clinical Psychology

DOI 10.1146/annurev-clinpsy-021815-093623

Open this paper in CoChat — free account

Abstract

Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.

Integrity signals

Registry

Crossref · 10.1146/annurev-clinpsy-021815-093623

Cites

174 of 174 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citation
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Self-citation
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941

Show all 177 references

Research works printed but not linked: 12

Barnhoorn JS, 2014, Behav. Res. Methods, 530, 1
Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented at Asian Polit. Methodol. Conf., Taipai
Casey L, Chandler J, Levine AS, Proctor A, Strolovitch D. 2015. Demographic characteristics of a large sample of US workers. Mathematica Policy Res. Rep. In press
Downes-Le Guin T, 2006, ESOMAR World Res. Conf. Panel Res.
Higgins C, 2010, Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, 89
10.1007/s10683-011-9273-9 · DOI
10.1145/2676431.2676437 · DOI
Lang ASID, 2011, Code4Lib J., 15, 1
Lebreton P, 2015, Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco
10.1026//1618-3169.49.4.243 · DOI
Shao W, 2015, Digit. Cult. Educ., 7
Suri S, 2011, AAAI Workshops, 25th, San Francisco, 61

Sorted by citations

Items 401–500 of 1,001

Black Intragroup Empathic Responding to Police Interracial Violence: Effects of Victim Stereotypicality and Blacks’ Racial Identification
Johnson, J. D., Lecci, L. B., & Dovidio, J. F. (2019) · Social Psychological and Personality Science · doi:10.1177/1948550619859316 · Cited by 17
Evaluating the Effect of Daily Diary Instructional Phrases on Respondents’ Recall Time Frames: Survey Experiment
Stone, A. A., Wen, C. K. F., Schneider, S., & Junghaenel, D. U. (2019) · Journal of Medical Internet Research · doi:10.2196/16105 · Cited by 17
Treating Anxiety During Pregnancy: Patient Concerns About Pharmaceutical Treatment
Lemon, E. L., Vanderkruik, R. C., Arch, J. J., & Dimidjian, S. (2020) · Maternal and Child Health Journal · doi:10.1007/s10995-019-02873-7 · Cited by 18
Financially focused self-concept in disordered gambling: comparisons between non-disordered and disordered gambling subtypes
Tabri, N., Salmon, M. M., & Wohl, M. J. A. (2020) · Addiction Research & Theory · doi:10.1080/16066359.2020.1714039 · Cited by 17
Are you for real? Maximizing participant eligibility on Amazon's Mechanical Turk
Kim, H. S. A., & Hodgins, D. C. (2020) · Addiction · doi:10.1111/add.15065 · Cited by 17
The impact of childhood sexual abuse on sleep in adulthood
Higgs, E., Drolet, C. E., & Belicki, K. (2020) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2020.104567 · Cited by 17
Prevalence of social, cognitive, and emotional impairment among individuals with food addiction
Lacroix, E., & von Ranson, K. M. (2020) · Eating and Weight Disorders - Studies on Anorexia Bulimia and Obesity · doi:10.1007/s40519-020-01014-2 · Cited by 17
Behavioral economic interactions between cannabis and alcohol purchasing: Associations with disordered use.
Dolan, S. B., Spindle, T. R., Vandrey, R. G., & Johnson, M. W. (2020) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000397 · Cited by 17
Healthy or not? The impact of conflicting health-related information on attentional resources
Barnwell, P. V., Fedorenko, E. J., & Contrada, R. J. (2021) · Journal of Behavioral Medicine · doi:10.1007/s10865-021-00256-4 · Cited by 17
It’s The “Why”: Links between (Non)autonomous Sexual Motives, Sexual Assertiveness, and Women’s Orgasm in Casual Sex
Wongsomboon, V., Webster, G. D., & Burleson, M. H. (2021) · Archives of Sexual Behavior · doi:10.1007/s10508-021-02103-8 · Cited by 17
Mindfulness Profiles Among Gamblers: Exploring Differences in Gambling Behaviors, Motivations, Cognitions, and Mental Health
Stanmyre, J. F., Mills, D. J., Li, W., & Nower, L. (2021) · Mindfulness · doi:10.1007/s12671-021-01791-6 · Cited by 18
Factors Associated with Cell Phone Use While Driving: A Survey of Parents and Caregivers of Children Ages 4-10 Years
McDonald, C. C., Kennedy, E. E., Fleisher, L. G., & Zonfrillo, M. R. (2018) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2018.06.003 · Cited by 17
Reliability and validity of the Brief DSM-5 Alcohol Use Disorder Diagnostic Assessment: A systematic replication in a crowdsourced sample
Campbell, E. M., & Strickland, J. C. (2019) · Addictive Behaviors · doi:10.1016/j.addbeh.2019.01.007 · Cited by 17
How Informal Exchanges Impact Formal Sourcing Collaboration (and What Supply Managers Can Do about It)
Lu, J., Kaufmann, L., & Carter, C. R. (2020) · Journal of Supply Chain Management · doi:10.1111/jscm.12241 · Cited by 17
Anxiety and temporal self-appraisal: How people with anxiety evaluate themselves over time
Sokol, Y., Rosensweig, C., Levin, C., & Linzer, M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.09.081 · Cited by 16
Private decentralized crowdsensing with asynchronous blockchain access
Liang, Y., Li, Y., & Shin, B. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109088 · Cited by 16
Bullying Victimization and Adverse Childhood Experiences: Retrospective Reports of Relative Impact on Emotional Distress
Miller, H. H., Jenkins, L. N., Putzeys, S., Kaminski, S., & Woodall, M. (2023) · Journal of Child & Adolescent Trauma · doi:10.1007/s40653-023-00567-5 · Cited by 18
Attention! Do We Really Need Attention Checks?
Roth, Y., & Yakobi, O. (2024) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2377 · Cited by 18
Exploring privacy-personalization paradox: Facial recognition systems at business events
Ciftci, O., Berezina, K., & Soifer, I. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108335 · Cited by 16
The Promise and Pitfalls of Using Crowdsourcing in Research Prioritization for Back Pain: Cross-Sectional Surveys
Bartek, M. A., Truitt, A. R., Widmer-Rodriguez, S., Tuia, J. E., Bauer, Z., Comstock, B. A., Edwards, T. C., Lawrence, S. O., Monsell, S. E., Patrick, D. L., et al. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8821 · Cited by 16
OC-2-KB: integrating crowdsourcing into an obesity and cancer knowledge base curation system
Lossio-Ventura, J. A., Hogan, W. R., Modave, F. P., Guo, Y. M., He, Z., Yang, X., Zhang, H., & Bian, J. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0635-5 · Cited by 16
Methodological considerations for assessing trauma history via self-report.
Bardeen, J. R., & Benfer, N. (2018) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/tra0000398 · Cited by 16
Video-Based Communication Assessment: Development of an Innovative System for Assessing Clinician-Patient Communication
Mazor, K. M., King, A., Hoppe, R. B., Kochersberger, A. O., Jie, Y., & Reim, J. D. (2018) · JMIR Medical Education · doi:10.2196/10400 · Cited by 16
Outcomes of two randomized controlled trials, employing participants recruited through Mechanical Turk, of Internet interventions targeting unhealthy alcohol use
Cunningham, J. A., Godinho, A., & Bertholet, N. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0770-4 · Cited by 16
Associations Between Social Anxiety and Affective and Empathic Forecasts: A Replication and Extension in a Mechanical Turk Sample
Hall, K. A. A., Coleman, K., & Timpano, K. R. (2019) · Behavior Therapy · doi:10.1016/j.beth.2019.06.004 · Cited by 16
Predicting Gambling Situations: The Roles of Impulsivity, Substance Use, and Post-Traumatic Stress
Grubbs, J. B., & Chapman, H. A. (2019) · Substance Abuse Research and Treatment · doi:10.1177/1178221819852641 · Cited by 16
Enduring Cognitive and Linguistic Deficits in Individuals With a History of Concussion
Stockbridge, M. D., & Newman, R. S. (2019) · American Journal of Speech-Language Pathology · doi:10.1044/2019_ajslp-18-0196 · Cited by 19
Asymmetric Investor Materiality and the Effects of Disclosure
Doxey, M. M., Hatfield, R. C., Rippy, J., & Peel, R. K. (2019) · Auditing A Journal of Practice & Theory · doi:10.2308/ajpt-18-032 · Cited by 17
Attitudes and Beliefs Toward Computerized Cognitive Training in the General Population
Goghari, V. M., Krzyzanowski, D. J., Yoon, S. J., Dai, Y., & Toews, D. M. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.00503 · Cited by 18
Comparing the nature of grief and growth in bereaved, divorced, and unemployed individuals
Klurfeld, Z. B., Buqo, T., Sanderson, W. C., & Ward‐Ciesielski, E. F. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.06.008 · Cited by 16
A Meta-analysis of Studies on the Performance of the Crosswise Model
Schnell, R., & Thomas, K. (2021) · Sociological Methods & Research · doi:10.1177/0049124121995520 · Cited by 16
Public perceptions and disparities in access to telehealth orthopaedic services in the COVID-19 era
Puzzitiello, R. N., Moverman, M. A., Pagani, N. R., Ryan, S. P., Salzler, M. J., Jawa, A., & Menendez, M. E. (2021) · Journal of the National Medical Association · doi:10.1016/j.jnma.2021.02.007 · Cited by 16
Increased videoconferencing after COVID-19 stay-at-home orders increased depression and anxiety but did not impact appearance satisfaction or binge eating
Gullo, N. A., & Walker, D. C. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100080 · Cited by 16
Political partisanship and stigma against people who use drugs in opinions about allocating COVID-19 prevention resources to vulnerable populations
Schneider, K. E., Wilson, D. H., Dayton, L., Goodell, E. M. A., & Latkin, C. A. (2021) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2021.103301 · Cited by 16
Investigating the impact of ransomware splash screens
Yilmaz, Y., Çetin, O., Arief, B., & Hernandez-Castro, J. C. (2021) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2021.102934 · Cited by 16
Perfectionism predicts disordered eating and gambling via focused self-concept among those high in erroneous beliefs about their disordered behavior
Tabri, N., Werner, K. M., Milyavskaya, M., & Wohl, M. J. A. (2021) · Journal of Behavioral Addictions · doi:10.1556/2006.2021.00068 · Cited by 16
A longitudinal study of COVID-19 disclosure stigma and COVID-19 testing hesitancy in the United States
Dayton, L., Song, W., Kaloustian, I., Eschliman, E. L., Strickland, J. C., & Latkin, C. A. (2022) · Public Health · doi:10.1016/j.puhe.2022.08.003 · Cited by 16
Peace Data Standard: A Practical and Theoretical Framework for Using Technology to Examine Intergroup Interactions
Guadagno, R. E., Nelson, M., & Lee, L. L. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00734 · Cited by 16
Racial Climate and Inclusiveness in Academic Libraries: Perceptions of Welcomeness among Black College Students
Stewart, B., Ju, B., & Kendrick, K. D. (2019) · The Library Quarterly · doi:10.1086/700661 · Cited by 16
Masking wrongs through brilliance: the moderating effect of vision on the relationship between abusive supervision and employee outcomes
Fiset, J. E., Robinson, M. A., & Robertson, M. C. S. (2019) · European Journal of Work and Organizational Psychology · doi:10.1080/1359432x.2019.1637853 · Cited by 16
A Randomized Controlled Trial of a Self-Compassion Writing Intervention for Adults With Mental Illness
Urken, D., & LeCroy, C. W. (2020) · Research on Social Work Practice · doi:10.1177/1049731520972779 · Cited by 15
A quantitative analysis of resurgence following downshifts in alternative‐reinforcer magnitude
Ritchey, C. M., Kuroda, T., & Podlesnik, C. A. (2023) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.843 · Cited by 15
Do Mixed Emotions Matter in a Crisis? The Impact of Expressing Sadness and Sympathy on Organizational Reputation and Forgiveness
Ndone, J. (2022) · International Journal of Business Communication · doi:10.1177/23294884221106999 · Cited by 15
Social Interpretation Inflexibility and Functioning: Associations with Symptoms and Stress
Deng, W., Everaert, J., Bronstein, M. V., Joormann, J., & Cannon, T. D. (2023) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2023.42.1.29 · Cited by 15
Effects of pay rate and instructions on attrition in crowdsourcing research
Ritchey, C. M., Jimenez‐Gomez, C., & Podlesnik, C. A. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292372 · Cited by 15
Moderating the link between discrimination and adverse mental health outcomes: Examining the protective effects of cognitive flexibility and emotion regulation
Zhu, Y., & Deng, W. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282220 · Cited by 15
Perceptions of psychedelic personality change, determinants of use, setting and drug moderation: Toward a holistic model
Weiss, B., Sleep, C. E., Beller, N. M., Erritzøe, D., & Campbell, W. K. (2023) · Journal of Psychedelic Studies · doi:10.1556/2054.2023.00291 · Cited by 15
Estimating the smallest worthwhile difference of antidepressants: a cross-sectional survey
Sahker, E., Furukawa, T. A., Luo, Y., Ferreira, M. L., Okazaki, K., Chevance, A., Markham, S., Ede, R., Leucht, S., Cipriani, A., et al. (2024) · BMJ Mental Health · doi:10.1136/bmjment-2023-300919 · Cited by 16
Just the right push! Social Media as a Therapeutical catalyst: The impact of influencers’ motivational (vs. Neutral) communication on healthy consumption
Sestino, A., Nasta, L., Bernando, A., & Giannattasio, A. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102601 · Cited by 15
Low foreign language proficiency reduces optimism about the personal future
Oganian, Y., Heekeren, H. R., & Korn, C. W. (2018) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021818774789 · Cited by 15
Assessing Post-Traumatic Tonic Immobility Responses: The Scale for Tonic Immobility Occurring Post-Trauma
Lloyd, C., Lanius, R. A., Brown, M. F. D., Neufeld, R. J., Frewen, P., & McKinnon, M. C. (2019) · Chronic Stress · doi:10.1177/2470547018822492 · Cited by 18
Prevalence and clinical characteristics of skin picking among adults with skin disease symptoms
Dixon, L. J., & Snorrason, Í. (2019) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2019.100454 · Cited by 15
Does empathy undermine justice? Moderating the impact of empathic concern for a White policeman on responses to police interracial violence
Johnson, J. M., & Lecci, L. B. (2019) · British Journal of Social Psychology · doi:10.1111/bjso.12347 · Cited by 15
“Buzzwords”: Crowd-sourcing and quantifying U.S. young adult terminology for subjective effects of alcohol and marijuana use.
Linden‐Carmichael, A. N., Masters, L. D., & Lanza, S. T. (2020) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000344 · Cited by 15
Putting your money where your self is: Connecting dimensions of closeness and theories of personal identity
Woike, J. K., Collard, P., & Hood, B. M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0228271 · Cited by 16
A comparison of global and egocentric network approaches for assessing peer alcohol use among college students in the United States
Russell, A. M., Barry, A. E., & Patterson, M. S. (2020) · Drug and Alcohol Review · doi:10.1111/dar.13140 · Cited by 15
Development and Pretesting of Hookah Tobacco Public Education Messages for Young Adults
Phan, L., Villanti, A. C., Leshner, G., Wagener, T. L., Stevens, E. M., Johnson, A. C., & Mays, D. M. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17238752 · Cited by 15
Stigma and policy preference toward individuals who transition from prescription opioids to heroin
Goodyear, K., & Chavanne, D. (2020) · Addictive Behaviors · doi:10.1016/j.addbeh.2020.106784 · Cited by 15
Examining How Internet Users Trust and Access Electronic Health Record Patient Portals: Survey Study
Yin, R., Law, K. E., & Neyens, D. M. (2021) · JMIR Human Factors · doi:10.2196/28501 · Cited by 16
Just Love Me, Feed Me, Never Leave Me: Understanding Pet Food Anxiety, Feeding and Shopping Behavior of US Pet Owners in Covidian Times
Rombach, M., & Dean, D. (2021) · Animals · doi:10.3390/ani11113101 · Cited by 15
When people die by suicide: Introducing unacceptable loss thresholds as a potential missing link between suicide readiness states and actively suicidal clinical states
Capron, D. W., Bauer, B. W., & Bryan, C. J. (2021) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12820 · Cited by 15
Web-Based Single Session Intervention for Perceived Control Over Anxiety During COVID-19: Randomized Controlled Trial
Mullarkey, M. C., Dobias, M., Sung, J. Y., Ahuvia, I. L., Shumake, J., Beevers, C. G., & Schleider, J. L. (2022) · JMIR Mental Health · doi:10.2196/33473 · Cited by 17
Online crowdsourcing to study trauma and mental health symptoms in military populations: A case for Amazon’s Mechanical Turk (MTurk) platform.
Forkus, S. R., Contractor, A. A., Goncharenko, S., Goldstein, S. C., & Weiss, N. H. (2022) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/tra0001235 · Cited by 15
PFAS Contamination: Pathway from Communication to Behavioral Outcomes
Dong, X., & Yang, J. Z. (2023) · Journal of Health Communication · doi:10.1080/10810730.2023.2193144 · Cited by 17
Evaluating the co-use of opioids and cannabis for pain among current users using hypothetical purchase tasks
Bergeria, C. L., Dolan, S. B., Johnson, M. W., Campbell, C. M., & Dunn, K. E. (2020) · Journal of Psychopharmacology · doi:10.1177/0269881120914211 · Cited by 15
Ear Molding Therapy: Laypersons’ Perceptions, Preferences, and Satisfaction with Treatment Outcome
Vu, G. H., Azzolini, A., Humphries, L. S., Mazzaferro, D. M., Kalmar, C. L., Zimmerman, C. E., Swanson, J. W., Taylor, J. A., & Bartlett, S. P. P. (2020) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0000000000002902 · Cited by 15
Predicting personal protective equipment use, trauma symptoms, and physical symptoms in the USA during the early weeks of the COVID-19 lockdown (April 9–18, 2020)
O’Brien, W. H., Wang, S., Varga, A. V., Xu, H., Sims, T., Horan, K. A., & Lim, C. X. (2021) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2021.05.003 · Cited by 17
Reduced autobiographical memory specificity as a mediating factor between general anxiety symptoms and performance on problem‐solving tasks
Hallford, D. J., Noory, N., & Mellor, D. J. (2018) · Applied Cognitive Psychology · doi:10.1002/acp.3428 · Cited by 15
Examining the Factor Structure and Incremental Validity of the Barkley Deficits in Executive Functioning Scale – Short Form in a Community Sample
Clauss, K. E., Witte, T. K., & Bardeen, J. R. (2021) · Journal of Personality Assessment · doi:10.1080/00223891.2021.1887879 · Cited by 14
Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields
Salminen, J. O., Jung, S., Kamel, A. M., Froneman, W., & Jansen, B. J. J. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1136 · Cited by 14
In the Age of Social Media, How Does the Public Choose a Plastic Surgeon? A Crowdsourcing Analysis of Major Deciding Factors
Elmer, N. A., Veeramani, A., Hassell, N. E., Comer, C. D., Manstein, S. M., Shiah, E., Bustos, V. P., & Lin, S. J. (2023) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000010571 · Cited by 14
Comparing data quality from an online and in-person lab sample on dynamic theory of mind tasks
Krendl, A. C., Hugenberg, K., & Kennedy, D. P. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02152-y · Cited by 16
On the Selection of Item Scores or Composite Scores for Clinical Prediction
McClure, K. E., Ammerman, B. A., & Jacobucci, R. (2024) · Multivariate Behavioral Research · doi:10.1080/00273171.2023.2292598 · Cited by 19
It’s not just a game: social networks, isolation and mental health in online gamers
Prochnow, T. M., & Patterson, M. S. (2024) · Journal of Public Mental Health · doi:10.1108/jpmh-11-2023-0099 · Cited by 17
The threat of AI chatbot responses to crowdsourced open-ended survey questions
Traylor, F. (2024) · Energy Research & Social Science · doi:10.1016/j.erss.2024.103857 · Cited by 17
We Feel, We Understand: Examining the Moderating Effects of Publics' Empathy on Crisis Outcomes Across Crisis Types and Response Strategies
Ndone, J. (2025) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.70029 · Cited by 14
Online disinhibition and internet communication of non‐suicidal self‐injury
Corcoran, V. P., & Andover, M. S. (2020) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12659 · Cited by 16
E-Cigarette Demand: Impact of Commodity Definitions and Test–Retest Reliability
Strickland, J. C., Vsevolozhskaya, O. A., & Stoops, W. W. (2020) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaa139 · Cited by 14
Which Variations of a Brief Cognitive Bias Modification Session for Interpretations Lead to the Strongest Effects?
Steinman, S. A., Namaky, N., Toton, S. L., Meissel, E. E. E., John, A. T. S., Pham, N.-H., Werntz, A. J., Valladares, T. L., Gorlin, E. I., Arbus, S., et al. (2020) · Cognitive Therapy and Research · doi:10.1007/s10608-020-10168-3 · Cited by 14
Psychodynamic and sociopolitical predictors of COVID Distress and Gravity
Wagerman, S. A., Bedikian, A., & Ross, B. S. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110506 · Cited by 18
An evaluation of common risk factors for prolonged grief disorder using the international classification of diseases-11 criteria
Singer, J., McLean, E., Kahler, J., & Papa, A. (2021) · Aging & Mental Health · doi:10.1080/13607863.2021.1998359 · Cited by 14
How Do You Like Me Now? The Influence of “Likes” and Followers on Social Media in Plastic Surgery
Bhat, D., Kollu, T., Ricci, J. A., & Patel, A. (2022) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000008919 · Cited by 14
Which Groups of Plastic Surgery Patients Are Impacted by Social Media Use? An In-Depth Review of Social Media Engagement
Kinney, J. R., Kim, E., Friedman, R., Bustos, V. P., Hassell, N. E., Elmer, N. A., Lee, B. T., & Lin, S. J. (2023) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000010641 · Cited by 14
Modeling Treatment-Related Decision-Making Using Applied Behavioral Economics: Caregiver Perspectives in Temporally-Extended Behavioral Treatments
Gilroy, S. P., & Kaplan, B. A. (2020) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-020-00619-6 · Cited by 14
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention
Ophir, Y., Tikochinski, R., Klomek, A. B., & Reichart, R. (2021) · Clinical Psychological Science · doi:10.1177/21677026211022013 · Cited by 16
Computational Psychiatry in Borderline Personality Disorder
Fineberg, S. K., Stahl, D. S., & Corlett, P. R. (2017) · Current Behavioral Neuroscience Reports · doi:10.1007/s40473-017-0104-y · Cited by 14
The computerized scoring algorithm for the autobiographical memory test: updates and extensions for analyzing memories of English-speaking adults
Takano, K., Hallford, D. J., Vanderveren, E., Austin, D., & Raes, F. (2018) · Memory · doi:10.1080/09658211.2018.1507042 · Cited by 14
Learning the statistics of pronoun reference: By word or by category?
Ye, Y., & Arnold, J. E. (2023) · Cognition · doi:10.1016/j.cognition.2023.105546 · Cited by 13
Development and validation of a brief version of the emotion reactivity scale: The B-ERS
Veilleux, J. C., Schreiber, R. E., Warner, E. A., & Brott, K. H. (2023) · Current Psychology · doi:10.1007/s12144-023-05323-4 · Cited by 13
Although I know it: Social anxiety is associated with a deficit in positive updating even when the cost of avoidance is Obvious
Zabag, R., Rinck, M., Becker, E. S., Gilboa‐Schechtman, E., & Levy‐Gigi, E. (2023) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2023.11.041 · Cited by 13
Mental Model-Based Designs: The Study in Privacy Policy Landscape
Atashpanjeh, H., Paudel, R., & Al-Ameen, M. N. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2392064 · Cited by 13
Do thoughts about dieting matter? Testing the relationship between thoughts about dieting, body shape concerns, and state self-esteem
Sarfan, L. D., Clerkin, E. M., Teachman, B. A., & Smith, A. R. (2018) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2018.08.005 · Cited by 13
Sexual functioning of late adolescents and young adults in relationships: association with individual characteristics and relationship factors
Byers, E. S., O’Sullivan, L. F., & Hughes, K. (2019) · Sexual & Relationship Therapy · doi:10.1080/14681994.2019.1626982 · Cited by 13
Disruptive behaviour in the operating room is under-reported: an international survey
Fast, I., Villafranca, A., Henrichs, B., Magid, K., Christodoulou, C. C., & Jacobsohn, E. (2020) · Canadian Journal of Anesthesia/Journal canadien d anesthésie · doi:10.1007/s12630-019-01540-3 · Cited by 13
A Taxometric Analysis of Experiential Avoidance
Kirk, A., Broman‐Fulks, J. J., & Arch, J. J. (2020) · Behavior Therapy · doi:10.1016/j.beth.2020.04.008 · Cited by 13
The complex association of barriers and interest in internet-delivered cognitive behavior therapy for depression and anxiety: informing e-health policies through exploratory path analysis
Moskalenko, M. Y., Hadjistavropoulos, H. D., & Katapally, T. R. (2020) · Journal of Mental Health · doi:10.1080/09638237.2020.1793125 · Cited by 14
When convenience trumps quality of care: A population-based survey on direct to consumer orthodontics
Bous, R. M., Apostolopoulos, K., & Valiathan, M. (2021) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2020.10.025 · Cited by 13
Coercive Sexual Experiences that Include Orgasm Predict Negative Psychological, Relationship, and Sexual Outcomes
Chadwick, S. B., Grower, P. L., & van Anders, S. M. (2022) · Journal of Interpersonal Violence · doi:10.1177/08862605211073109 · Cited by 13
Reliability and validity of the PHQ-8 in first-time mothers who used assisted reproductive technology
Pavlov, C., Egan, K. N., & Limbers, C. A. (2022) · Human Reproduction Open · doi:10.1093/hropen/hoac019 · Cited by 13
Distress Among Parents of Individuals with Substance Use Disorders: Factors That Shape the Context of Care
Russell, B. S., D’Aniello, C., Tambling, R. R., & Horton, A. L. (2022) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-022-00884-0 · Cited by 13

Authors: Chandler, J., & Shapiro, D.
Affiliations: Mathematica Policy Research (US)
University of Michigan (US)
ORCID: Chandler, Jesse
Venue: Annual Review of Clinical Psychology
Publisher: Annual Reviews
Published: 2016-03-28
Volume: 12
Issue: 1
Pages: 53-81
ISSN: 1548-5943 (ISSN-L)
1548-5951
DOI: 10.1146/annurev-clinpsy-021815-093623
OpenAlex: W2126493218
Cited by: 1,072
References: 177
Citations by year: 2026: 27 · 2025: 70 · 2024: 105 · 2023: 103 · 2022: 150 · 2021: 168 · 2020: 122 · 2019: 115 · 2018: 111 · 2017: 77 · 2016: 24
FWCI: 17.58
Type: journal article
Language: en
Open access: closed
Locations: Annual Review of Clinical Psychology (publishedVersion)
PubMed (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
Keywords: crowdsourcing, Amazon Mechanical Turk, convenience samples, non-probability sampling, sample composition, data quality, participant recruitment, digital labour markets
MeSH terms: Humans, Biomedical Research, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Conducting Clinical Research Using Crowdsourced Convenience Samples

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks