Cited by 1,072Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseEthics in Clinical Research

The DOI matches its Crossref record; 174 of the 177 references it cites were found in a registry or an index.

Conducting Clinical Research Using Crowdsourced Convenience Samples

Chandler, Jesse, Shapiro, Danielle·2016·Annual Review of Clinical Psychology

DOI 10.1146/annurev-clinpsy-021815-093623

Open this paper in CoChat — free account

Abstract

Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.

Integrity signals

Registry

Crossref · 10.1146/annurev-clinpsy-021815-093623

Cites

174 of 174 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citation
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Self-citation
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941

Show all 177 references

Research works printed but not linked: 12

Barnhoorn JS, 2014, Behav. Res. Methods, 530, 1
Beramendi P, Duch RM, Matsuo A. 2014. When lab subjects meet real people: comparing different modes of experiments. Presented at Asian Polit. Methodol. Conf., Taipai
Casey L, Chandler J, Levine AS, Proctor A, Strolovitch D. 2015. Demographic characteristics of a large sample of US workers. Mathematica Policy Res. Rep. In press
Downes-Le Guin T, 2006, ESOMAR World Res. Conf. Panel Res.
Higgins C, 2010, Proc. NAACL HLT 2010 Workshop on Creat. Speech and Language Data with Amazon's Mechanical Turk, Los Angeles, 89
10.1007/s10683-011-9273-9 · DOI
10.1145/2676431.2676437 · DOI
Lang ASID, 2011, Code4Lib J., 15, 1
Lebreton P, 2015, Proc. SPIE Conf. Hum. Vis. Electron. Imaging, 20th, San Francisco
10.1026//1618-3169.49.4.243 · DOI
Shao W, 2015, Digit. Cult. Educ., 7
Suri S, 2011, AAAI Workshops, 25th, San Francisco, 61

Sorted by citations

Understanding and managing the threat of common method bias: Detection, prevention and control
Kock, F., Berbekova, A., & Assaf, A. G. (2021) · Tourism Management · doi:10.1016/j.tourman.2021.104330 · Cited by 1,692
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 815
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Self-citation
Customer service chatbots: Anthropomorphism and adoption
Sheehan, B., Jin, H. S., & Gottlieb, U. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.04.030 · Cited by 667
Mental health during the COVID-19 pandemic: Effects of stay-at-home policies, social distancing behavior, and social resources
Marroquín, B. M., Vine, V., & Morgan, R. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113419 · Cited by 667
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597
Demographics and Dynamics of Mechanical Turk Workers
Difallah, D. E., Filatova, E., & Ipeirotis, P. G. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159661 · Cited by 512
Comparing Amazon’s Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature
Mortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 441
Trust in a COVID-19 vaccine in the U.S.: A social-ecological perspective
Latkin, C. A., Dayton, L., Yi, G. T., Konstantopoulos, A. D., & Boodram, B. (2021) · Social Science & Medicine · doi:10.1016/j.socscimed.2021.113684 · Cited by 401
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 368
Modelling the contribution of the Big Five personality traits, health anxiety, and COVID-19 psychological distress to generalised anxiety and depressive symptoms during the COVID-19 pandemic
Nikčević, A. V., Marino, C., Kolubinski, D. C., Leach, D., & Spada, M. M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.10.053 · Cited by 344
The Politically Motivated Reasoning Paradigm, Part 1: What Politically Motivated Reasoning Is and How to Measure It
Kahan, D. M. (2016) · Emerging Trends in the Social and Behavioral Sciences · doi:10.1002/9781118900772.etrds0417 · Cited by 341
Mental distress among U.S. adults during the COVID-19 pandemic
Twenge, J. M., & Joiner, T. E. (2020) · Journal of Clinical Psychology · doi:10.1002/jclp.23064 · Cited by 330
“…than our panel (e.g., MTurk; Ackerman, 2019; Coppock & McClellan, 2019) do not consistently differ in the prevalence of clinical disorders, with some studies finding they show lower prevalence, some higher prevalence, and some no difference from probability samples (Chandler & Shapiro, 2016).”
Result
Exploring the structure of narcissism: Toward an integrated solution
Crowe, M. L., Lynam, D. R., Campbell, W. K., & Miller, J. D. (2019) · Journal of Personality · doi:10.1111/jopy.12464 · Cited by 317
The Influence of Climate Fiction
Schneider-Mayerson, M. (2018) · Environmental Humanities · doi:10.1215/22011919-7156848 · Cited by 281
Lie for a Dime
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 279
Self-citation
No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments
Kane, J., & Barabas, J. (2018) · American Journal of Political Science · doi:10.1111/ajps.12396 · Cited by 276
Misinformation about COVID-19: evidence for differential latent profiles and a strong association with trust in science
Agley, J. D., & Xiao, Y. (2021) · BMC Public Health · doi:10.1186/s12889-020-10103-x · Cited by 266
A self-report measure for the ICD-11 dimensional trait model proposal: The Personality Inventory for ICD-11.
Oltmanns, J. R., & Widiger, T. A. (2017) · Psychological Assessment · doi:10.1037/pas0000459 · Cited by 267
The use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2018) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 258
Anthropomorphic response: Understanding interactions between humans and artificial intelligence agents
Kim, J., & Im, I. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107512 · Cited by 233
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
McCredie, M. N., & Morey, L. C. (2018) · Assessment · doi:10.1177/1073191118760709 · Cited by 232
Comprehensive meta-analysis of resilience interventions
Liu, J. J. W., Ein, N., Gervasio, J., Battaion, M., Reed, M. J., & Vickers, K. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101919 · Cited by 259
Adaptive and Dark Personality in the COVID-19 Pandemic: Predicting Health-Behavior Endorsement and the Appeal of Public-Health Messages
Blagov, P. S. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550620936439 · Cited by 231
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 233
Self-citation
Opioid use and stigma: The role of gender, language and precipitating events
Goodyear, K., Haass‐Koffler, C. L., & Chavanne, D. (2018) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2017.12.037 · Cited by 218
Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203
The COVID-19 anxiety syndrome scale: development and psychometric properties.
Nikčević, A. V., & Spada, M. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113322 · Cited by 186
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 191
The Opportunities and Limitations of Using Mechanical Turk (MTURK) in Public Administration and Management Scholarship
Stritch, J. M., Pedersen, M. J., & Taggart, G. (2017) · International Public Management Journal · doi:10.1080/10967494.2016.1276493 · Cited by 181
Using online, crowdsourcing platforms for data collection in personality disorder research: the example of Amazon’s Mechanical Turk
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000191 · Cited by 179
COVID-19 vaccine intentions in the United States, a social-ecological framework
Latkin, C. A., Dayton, L., Yi, G. T., Konstantopoulos, A. D., Park, J. N., Maulsby, C. H., & Kong, X. (2021) · Vaccine · doi:10.1016/j.vaccine.2021.02.058 · Cited by 179
A Critical Appraisal of the Dark-Triad Literature and Suggestions for Moving Forward
Miller, J. D., Vize, C. E., Crowe, M. L., & Lynam, D. R. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419838233 · Cited by 178
Similarities and Differences in Tobacco Control Research Findings From Convenience and Probability Samples
Jeong, M., Zhang, D., Morgan, J. C., Ross, J. C., Osman, A., Boynton, M. H., Sheldon, J. M., & Brewer, N. T. (2018) · Annals of Behavioral Medicine · doi:10.1093/abm/kay059 · Cited by 177
The quality of data collected online: An investigation of careless responding in a crowdsourced sample
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168
The prevalence of aphantasia (imagery weakness) in the general population
Dance, C. J., Ipser, A., & Simner, J. (2022) · Consciousness and Cognition · doi:10.1016/j.concog.2021.103243 · Cited by 167
Cyberchondria: examining relations with problematic internet use and metacognitive beliefs
Fergus, T. A., & Spada, M. M. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2102 · Cited by 162
Common Concerns with MTurk as a Participant Pool: Evidence and SolutionsPreprint
Hauser, D., Paolacci, G., & Chandler, J. (2018) · doi:10.31234/osf.io/uq45c · Cited by 162
Self-citation
Identifying Predictors of Psychological Distress During COVID-19: A Machine Learning Approach
Prout, T. A., Zilcha‐Mano, S., Doorn, K. A., Békés, V., Christman-Cohen, I., Whistler, K., Kui, T., & Giuseppe, M. D. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.586202 · Cited by 164
Rethinking the link between cognitive sophistication and politically motivated reasoning.
Tappin, B. M., Pennycook, G., & Rand, D. G. (2021) · Journal of Experimental Psychology: General · doi:10.1037/xge0000974 · Cited by 154
Lifetime traumatic stressors and adverse childhood experiences uniquely predict concurrent PTSD, complex PTSD, and dissociative subtype of PTSD symptoms whereas recent adult non-traumatic stressors do not: results from an online survey study
Frewen, P., Zhu, J., & Lanius, R. (2019) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1606625 · Cited by 151
Uncovering the structure of agreeableness from self‐report measures
Crowe, M. L., Lynam, D. R., & Miller, J. D. (2017) · Journal of Personality · doi:10.1111/jopy.12358 · Cited by 147
Validation of the nine item ARFID screen ( NIAS ) subscales for distinguishing ARFID presentations and screening for ARFID
Murray, H. B., Dreier, M. J., Zickgraf, H. F., Becker, K. R., Breithaupt, L. E., Eddy, K. T., & Thomas, J. J. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23520 · Cited by 146
Are Relational Inferences from Crowdsourced and Opt-in Samples Generalizable? Comparing Criminal Justice Attitudes in the GSS and Five Online Samples
Thompson, A. J., & Pickett, J. T. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09436-7 · Cited by 132
Father Participation in Child Psychopathology Research
Parent, J. M., Forehand, R., Pomerantz, H., Peisch, V. D., & Seehuus, M. (2017) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-016-0254-5 · Cited by 127
Mapping the relationship between anxiety, anhedonia, and depression
Winer, E. S., Bryant, J., Bartoszek, G., Rojas, E., Nadorff, M. R., & Kilgore, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.06.006 · Cited by 124
The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers
Ophir, Y., Sisso, I., Asterhan, C. S. C., Tikochinski, R., & Reichart, R. (2019) · Clinical Psychological Science · doi:10.1177/2167702619865973 · Cited by 124
Participant carelessness and fraud: Consequences for clinical research and potential solutions.
Chandler, J., Sisso, I., & Shapiro, D. N. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000479 · Cited by 124
Self-citation
Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 132
Self-citation
An evaluation of DSM–5 Section III personality disorder Criterion A (impairment) in accounting for psychopathology.
Sleep, C. E., Lynam, D. R., Widiger, T. A., Crowe, M. L., & Miller, J. D. (2019) · Psychological Assessment · doi:10.1037/pas0000620 · Cited by 129
Emotion Regulation Difficulties Related to Depression and Anxiety: A Network Approach to Model Relations Among Symptoms, Positive Reappraisal, and Repetitive Negative Thinking
Everaert, J., & Joormann, J. (2019) · Clinical Psychological Science · doi:10.1177/2167702619859342 · Cited by 119
When negative interpretations persist, positive emotions don't! Inflexible negative interpretations encourage depression and social anxiety by dampening positive emotions
Everaert, J., Bronstein, M. V., Castro, A. A., Cannon, T. D., & Joormann, J. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103510 · Cited by 119
Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7
Agley, J. D., Xiao, Y., Nolan, R., & Golzarri‐Arroyo, L. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01665-8 · Cited by 118
Development and Initial Validation of the Duke Misophonia Questionnaire
Rosenthal, M. Z., Anand, D., Cassiello-Robbins, C., Williams, Z. J., Guetta, R. E., Trumbull, J., & Kelley, L. D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.709928 · Cited by 117
Trigger warning: Empirical evidence ahead
Bellet, B. W., Jones, P. J., & McNally, R. J. (2018) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2018.07.002 · Cited by 117
Behavioral and psychosocial factors associated with COVID-19 skepticism in the United States
Latkin, C. A., Dayton, L., Moran, M. B., Strickland, J. C., & Collins, K. (2021) · Current Psychology · doi:10.1007/s12144-020-01211-3 · Cited by 117
Factors that predict taking restaurant leftovers: Strategies for reducing food waste
Hamerman, E. J., Rudell, F., & Martins, C. M. (2017) · Journal of Consumer Behaviour · doi:10.1002/cb.1700 · Cited by 114
A preliminary cross-cultural study of Hikikomori and Internet Gaming Disorder: The moderating effects of game-playing time and living with parents
Stavropoulos, V., Anderson, E. L., Beard, C. L., Latifi, M. Q., Kuss, D. J., & Griffiths, M. D. (2018) · Addictive Behaviors Reports · doi:10.1016/j.abrep.2018.10.001 · Cited by 113
The uncanny valley: No need for any further judgments when an avatar looks eerie
Shin, M., Kim, S. J., & Biocca, F. A. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.016 · Cited by 113
A Technical Guide to Using Amazon's Mechanical Turk in Behavioral Accounting Research
Buchheit, S., Doxey, M. M., Pollard, T., & Stinson, S. R. (2017) · Behavioral Research in Accounting · doi:10.2308/bria-51977 · Cited by 115
Evaluating the assessment of the ICD-11 personality disorder diagnostic system.
Oltmanns, J. R., & Widiger, T. A. (2019) · Psychological Assessment · doi:10.1037/pas0000693 · Cited by 109
Rise in Use of Digital Mental Health Tools and Technologies in the United States During the COVID-19 Pandemic: Survey Study
Sorkin, D. H., Janio, E. A., Eikey, E. V., Schneider, M., Davis, K., Schueller, S. M., Stadnick, N. A., Zheng, K., Neary, M., Safani, D., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26994 · Cited by 108
Moving experimental psychology online: How to obtain high quality data when we can’t see our participants
Rodd, J. M. (2023) · Journal of Memory and Language · doi:10.1016/j.jml.2023.104472 · Cited by 117
Inattentive responding can induce spurious associations between task behaviour and symptom measures
Zorowitz, S., Solis, J., Niv, Y., & Bennett, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01640-7 · Cited by 107
Minding the gap: Subjective relative deprivation and depressive symptoms
Beshai, S., Mishra, S., Meadows, T. J. S., Parmar, P., & Huang, V. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.11.021 · Cited by 104
Measurement Invariance of the Internet Gaming Disorder Scale–Short-Form (IGDS9-SF) Between Australia, the USA, and the UK
Stavropoulos, V., Beard, C. L., Griffiths, M. D., Burleigh, T. L., Gomez, R., & Pontes, H. M. (2017) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-017-9786-3 · Cited by 98
How listening to music and engagement with other media provide a sense of belonging: An exploratory study of social surrogacy
Schäfer, K., & Eerola, T. (2018) · Psychology of Music · doi:10.1177/0305735618795036 · Cited by 116
The Politically Motivated Reasoning Paradigm, Part 2: Unanswered Questions
Kahan, D. M. (2016) · Wiley eBooks · doi:10.1002/9781118900772.etrds0418 · Cited by 99
Age of Initiation and Internet Gaming Disorder: The Role of Self-Esteem
Beard, C. L., Haas, A. L., Wickham, R. E., & Stavropoulos, V. (2017) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2017.0011 · Cited by 96
Social Media Contributions to Strong Black Woman Ideal Endorsement and Black Women’s Mental Health
Stanton, A. G., Jerald, M. C., Ward, L. M., & Avery, L. R. (2017) · Psychology of Women Quarterly · doi:10.1177/0361684317732330 · Cited by 96
An Assessment of the Rapid Decline of Trust in US Sources of Public Information about COVID-19
Latkin, C. A., Dayton, L., Strickland, J. C., Colon, B., Rimal, R. N., & Boodram, B. (2020) · Journal of Health Communication · doi:10.1080/10810730.2020.1865487 · Cited by 104
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94
Mechanisms of offline motor learning at a microscale of seconds in large-scale crowdsourced data
Bönstrup, M., Iturrate, I., Hebart, M. N., Censor, N., & Cohen, L. G. (2020) · npj Science of Learning · doi:10.1038/s41539-020-0066-9 · Cited by 103
Self-service technology kiosk design for restaurants: An QFD application
Park, S., Lehto, X. Y., & Lehto, M. R. (2020) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102757 · Cited by 94
A comprehensive comparison of the ICD-11 and DSM–5 section III personality disorder models.
McCabe, G. A., & Widiger, T. A. (2020) · Psychological Assessment · doi:10.1037/pas0000772 · Cited by 93
Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places
Gleibs, I. H. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0789-y · Cited by 93
Temporal self appraisal and continuous identity: Associations with depression and hopelessness
Sokol, Y., & Serper, M. R. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.10.033 · Cited by 99
Overlooked and Underestimated: Experiences of Ageism in Young, Middle-Aged, and Older Adults
Chasteen, A. L., Horhota, M., & Crumley-Branyon, J. J. (2020) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbaa043 · Cited by 92
Let’s face it: Are customers ready for facial recognition technology at quick-service restaurants?
Ciftci, O., Choi, E.-K., & Berezina, K. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.102941 · Cited by 92
What is the relationship between Aphantasia, Synaesthesia and Autism?
Dance, C. J., Jaquiery, M., Eagleman, D. M., Porteous, D., Zeman, A., & Simner, J. (2021) · Consciousness and Cognition · doi:10.1016/j.concog.2021.103087 · Cited by 89
Using Mechanical Turk Data in IS Research: Risks, Rewards, and Recommendations
Jia, R., Steelman, Z., & Reich, B. H. (2017) · Communications of the Association for Information Systems · doi:10.17705/1cais.04114 · Cited by 94
Building Blocks of Emotional Flexibility: Trait Mindfulness and Self-Compassion Are Associated with Positive and Negative Mood Shifts
Beshai, S., Prentice, J. L., & Huang, V. (2017) · Mindfulness · doi:10.1007/s12671-017-0833-8 · Cited by 108
Smartphones and the Neuroscience of Mental Health
Gillan, C. M., & Rutledge, R. B. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-101220-014053 · Cited by 95
Adverse Childhood Experiences and Disordered Gambling: Assessing the Mediating Role of Emotion Dysregulation
Poole, J. C., Kim, H. S. A., Dobson, K. S., & Hodgins, D. C. (2017) · Journal of Gambling Studies · doi:10.1007/s10899-017-9680-8 · Cited by 83
Assessment of fluctuation between grandiose and vulnerable narcissism: Development and initial validation of the FLUX scales.
Oltmanns, J. R., & Widiger, T. A. (2018) · Psychological Assessment · doi:10.1037/pas0000616 · Cited by 87
Why not eat alone? The effect of other consumers on solo dining intentions and the mechanism
Her, E., & Seo, S. (2017) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.10.019 · Cited by 82
A longitudinal study of vaccine hesitancy attitudes and social influence as predictors of COVID-19 vaccine uptake in the US
Latkin, C. A., Dayton, L., Miller, J., Yi, G. T., Balaban, A., Boodram, B., Uzzi, M., & Falade-Nwulia, O. O. (2022) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2022.2043102 · Cited by 81
Moving toward a metacognitive conceptualization of cyberchondria: examining the contribution of metacognitive beliefs, beliefs about rituals, and stop signals
Fergus, T. A., & Spada, M. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.09.003 · Cited by 80
Fear and anxiety in the face of COVID-19: Negative dispositions towards risk and uncertainty as vulnerability factors
Millroth, P., & Frey, R. (2021) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2021.102454 · Cited by 88
Moral polarization and out-party hostility in the US political context
Tappin, B. M., & McKay, R. T. (2019) · Journal of Social and Political Psychology · doi:10.5964/jspp.v7i1.1090 · Cited by 79
Applying the Gateway Belief Model to Genetically Modified Food Perceptions: New Insights and Additional Questions
Dixon, G. (2016) · Journal of Communication · doi:10.1111/jcom.12260 · Cited by 82
Lie prevalence, lie characteristics and strategies of self-reported good liars
Verigin, B. L., Meijer, E. H., Bogaard, G., & Vrij, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225566 · Cited by 88
Public Perceptions of Physician Attire and Professionalism in the US
Xun, H., Chen, J., Sun, A. H., Jenny, H. E., Liang, F., & Steinberg, J. P. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.17779 · Cited by 78
Framing fast and slow: a dual processing account of multimodal framing effects
Powell, T. E., Boomgaarden, H. G., Swert, K. D., & de Vreese, C. H. (2018) · Media Psychology · doi:10.1080/15213269.2018.1476891 · Cited by 81
A panel for lemons? Positivity bias, reputation systems and data quality on MTurk
Matherly, T. (2019) · European Journal of Marketing · doi:10.1108/ejm-07-2017-0491 · Cited by 79
Crowdsourcing Trauma: Psychopathology in a Trauma‐Exposed Sample Recruited via Mechanical Turk
van Stolk‐Cooke, K., Brown, A., Maheux, A. J., Parent, J. M., Forehand, R., & Price, M. (2018) · Journal of Traumatic Stress · doi:10.1002/jts.22303 · Cited by 74
Unpacking the construct of emotional attachment to objects and its association with hoarding symptoms
Yap, K., & Grisham, J. R. (2019) · Journal of Behavioral Addictions · doi:10.1556/2006.8.2019.15 · Cited by 71
Use of online crowdsourcing platforms for gambling research
Mishra, S., & Carleton, R. N. (2017) · International Gambling Studies · doi:10.1080/14459795.2017.1284250 · Cited by 71
The Impact of E-Cigarette Warnings, Warning Themes and Inclusion of Relative Harm Statements on Young Adults’ E-Cigarette Perceptions and Use Intentions
Wackowski, O. A., Sontag, J. M., Hammond, D. G., O’Connor, R. J., Ohman‐Strickland, P., Strasser, A. A., Villanti, A. C., & Delnevo, C. D. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16020184 · Cited by 70
Associations of negative cognitions, emotional regulation, and depression symptoms across four continents: International support for the cognitive model of depression
Chahar Mahali, S., Beshai, S., Feeney, J. R., & Mishra, S. (2020) · BMC Psychiatry · doi:10.1186/s12888-019-2423-x · Cited by 52

Authors: Chandler, J., & Shapiro, D.
Affiliations: Mathematica Policy Research (US)
University of Michigan (US)
ORCID: Chandler, Jesse
Venue: Annual Review of Clinical Psychology
Publisher: Annual Reviews
Published: 2016-03-28
Volume: 12
Issue: 1
Pages: 53-81
ISSN: 1548-5943 (ISSN-L)
1548-5951
DOI: 10.1146/annurev-clinpsy-021815-093623
OpenAlex: W2126493218
Cited by: 1,072
References: 177
Citations by year: 2026: 27 · 2025: 70 · 2024: 105 · 2023: 103 · 2022: 150 · 2021: 168 · 2020: 122 · 2019: 115 · 2018: 111 · 2017: 77 · 2016: 24
FWCI: 17.58
Type: journal article
Language: en
Open access: closed
Locations: Annual Review of Clinical Psychology (publishedVersion)
PubMed (publishedVersion)
RePEc: Research Papers in Economics (submittedVersion)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
Keywords: crowdsourcing, Amazon Mechanical Turk, convenience samples, non-probability sampling, sample composition, data quality, participant recruitment, digital labour markets
MeSH terms: Humans, Biomedical Research, Crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Conducting Clinical Research Using Crowdsourced Convenience Samples

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks