Conducting Clinical Research Using Crowdsourced Convenience Samples
Abstract
Crowdsourcing has had a dramatic impact on the speed and scale at which scientific research can be conducted. Clinical scientists have particularly benefited from readily available research study participants and streamlined recruiting and payment systems afforded by Amazon Mechanical Turk (MTurk), a popular labor market for crowdsourcing workers. MTurk has been used in this capacity for more than five years. The popularity and novelty of the platform have spurred numerous methodological investigations, making it the most studied nonprobability sample available to researchers. This article summarizes what is known about MTurk sample composition and data quality with an emphasis on findings relevant to clinical psychological research. It then addresses methodological issues with using MTurk--many of which are common to other nonprobability samples but unfamiliar to clinical science researchers--and suggests concrete steps to avoid these issues or minimize their impact.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Chandler and Shapiro (2016) demonstrate that MTurk-recruited research study participants are similarly honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is particularly well suited to reaching populations with social anxiety symptoms.
“Chandler and Shapiro (2016), including empirical work by Shapiro and colleagues (2013), demonstrate that MTurk-recruited research study participants are similarly honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is particularly well suited to reaching populations with social anxiety symptoms.”
From A Randomized Trial of Brief Online Interventions to Facilitate Treatment Seeking for Social Anxiety. (2022) · cites this paper as Chandler and Shapiro, 2016
MTurk workers are comparably honest, attentive, and reliable compared to non-MTurk samples, and MTurk is well suited to reaching populations with clinically relevant symptoms.
“However, as noted, research reviewed by Chandler and Shapiro (2016), including empirical work by Shapiro and colleagues (2013), illustrates that MTurk workers are comparably honest, attentive, and reliable compared to non-MTurk samples, and that MTurk is well suited to reaching populations with clinically relevant symptoms.”
From A Randomized Trial of Brief Online Interventions to Facilitate Treatment Seeking for Social Anxiety. (2022) · cites this paper as Chandler and Shapiro (2016)
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Self-citationInstructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Self-citationCrowdsourcing user studies with Mechanical Turk
Kittur, A., Ed, H., & Suh, B. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,941 · see citing works
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser
de Leeuw, J. R. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0458-y · Cited by 1,783 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1× · see citing works
“In direct comparison studies, MTurk samples are found to be more representative than college student samples, community samples recruited from college towns (Berinsky et al. 2012), or other online sources (Casler et al. 2013).”
BackgroundReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
Survey Errors and Survey Costs.
Faulkenberry, G. D., & Groves, R. M. (1990) · Journal of the American Statistical Association · doi:10.2307/2289634 · Cited by 1,441 · see citing works
Field Experiments: Design, Analysis, and Interpretation
GREEN, D. P. (2024) · doi:10.61700/sgeirzx0v0mjh1252 · Cited by 1,322 · see citing works
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L. W., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2011) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,315 · see citing works
The Body Appreciation Scale-2: Item refinement and psychometric evaluation
Tylka, T. L., & Wood-Barcalow, N. L. (2015) · Body Image · doi:10.1016/j.bodyim.2014.09.006 · Cited by 1,298 · see citing works
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Jeremy Freese, A. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,235 · see citing works
Investigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208 · see citing works
Self-citationUsing Mechanical Turk to Study Clinical Populations
Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013) · Clinical Psychological Science · doi:10.1177/2167702612469015 · Cited by 1,139 · see citing works
Self-citationThe viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,049 · see citing works
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1× · see citing works
“Further, performance on different attention checks in the same survey shows only a modest correlation, and associations between attentiveness items are equally strong regardless of their proximity, suggesting that attention to one question cannot be assumed to guarantee attention to measures administered in close prox…”
BackgroundChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Self-citationHuff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · see citing works
Crowdsourcing graphical perception
Heer, J., & Bostock, M. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 819 · see citing works
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798 · see citing works
Black, D. A., Gates, G. J., Sanders, S., & Taylor, L. J. (2000) · Demography · doi:10.2307/2648117 · Cited by 777 · see citing works
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772 · see citing works
Beginning the workday yet already depleted? Consequences of late-night smartphone use and sleep
Lanaj, K., Johnson, R. E., & Barnes, C. M. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2014.01.001 · Cited by 738 · cited 1× · see citing works
“Consequently, waves of data can be collected from workers over a period of days, allowing for diary studies and other forms of experience sampling (Boynton & Richman 2014, Lanaj et al. 2014, Usinger 2014).”
BackgroundIntuitive Cooperation and the Social Heuristics Hypothesis: Evidence from 15 Time Constraint Studies
Rand, D. G., Peysakhovich, A., Kraft‐Todd, G., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2013) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 730 · see citing works
Rand, D. G. (2011) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2011.03.004 · Cited by 686 · see citing works
Standards for Internet-Based Experimenting
Reips, U. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026/1618-3169.49.4.243 · Cited by 685 · see citing works
The role of impulsivity in the Dark Triad of personality
Jones, D. N., & Paulhus, D. L. (2011) · Personality and Individual Differences · doi:10.1016/j.paid.2011.04.011 · Cited by 618 · see citing works
Ruzich, E., Allison, C. L., Smith, P. L., Watson, P., Bonnie Auyeung, B., RING, H. A., & Baron‐Cohen, S. (2015) · Molecular Autism · doi:10.1186/2040-2392-6-2 · Cited by 612 · see citing works
The labor economics of paid crowdsourcing
Horton, J. J., & Chilton, L. B. (2010) · ACM Conference on Electronic Commerce (EC) · doi:10.1145/1807342.1807376 · Cited by 612 · see citing works
Internet-Delivered Psychological Treatments
Andersson, G. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093006 · Cited by 595 · see citing works
Weinberg, J. D., Jeremy Freese, A., & McElhattan, D. (2014) · Sociological Science · doi:10.15195/v1.a19 · Cited by 582 · see citing works
Standards for Internet-Based Experimenting
Reips, U. (2002) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026//1618-3169.49.4.243 · Cited by 557 · see citing works
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494 · see citing works
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463 · see citing works
Internet Research in Psychology
Gosling, S. D., & Mason, W. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015321 · Cited by 458 · see citing works
Austin, E. J. (2005) · Personality and Individual Differences · doi:10.1016/j.paid.2004.04.022 · Cited by 435 · see citing works
Collaborative learning in networks
Mason, W., & Watts, D. J. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1110069108 · Cited by 356 · see citing works
Cooperation and Contagion in Web-Based, Networked Public Goods Experiments
Suri, S., & Watts, D. J. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016836 · Cited by 355 · see citing works
Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review
Ranard, B. L., Ha, Y. P., Meisel, Z. F., Asch, D. A., Hill, S. S., Becker, L. B., Seymour, A. K., & Merchant, R. M. (2013) · Journal of General Internal Medicine · doi:10.1007/s11606-013-2536-8 · Cited by 324 · see citing works
Dimensions of Internet science
Reips, U., & Bošnjak, M. (2001) · Cited by 319 · see citing works
Cooperation and contagion in web-based, networked public goods experiments
Suri, S., & Watts, D. J. (2011) · ACM SIGecom Exchanges · doi:10.1145/1998549.1998550 · Cited by 318 · see citing works
The importance of assessing clinical phenomena in Mechanical Turk research.
Hall, K. A. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2015) · Psychological Assessment · doi:10.1037/pas0000217 · Cited by 311 · see citing works
Davenport, S. W., Bergman, S. M., Bergman, J. Z., & Fearrington, M. E. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.12.011 · Cited by 305 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 4× · see citing works
“The number of active users is unknown but is likely close to 15,000 individual US workers at any given time (Stewart et al. 2015; see also Fort et al. 2011), with perhaps a similar number of Indian users (see Indian Workers 56 Chandler · Shapiro A nn u.”
BackgroundResultInfluentialSelf-citationArbisi, P. A., & Ben‐Porath, Y. S. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.4.424 · Cited by 304 · see citing works
Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data
Benoit, K. R., Conway, D., Lauderdale, B. E., Laver, M., & Mikhaylov, S. (2016) · American Political Science Review · doi:10.1017/s0003055416000058 · Cited by 296 · see citing works
Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531663 · Cited by 279 · see citing works
Litman, L., Robinson, J., & Rosenzweig, C. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275 · see citing works
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2016) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 274 · see citing works
Using the Amazon Mechanical Turk for transcription of spoken language
Marge, M., Banerjee, S., & Rudnicky, A. I. (2010) · IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2010.5494979 · Cited by 272 · see citing works
Parent, J., McKee, L. G., Rough, J. N., & Forehand, R. (2015) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-015-9978-x · Cited by 270 · see citing works
Designing Incentives for Inexpert Human Raters
Shaw, A., Horton, J. J., & Chen, D. L. (2010) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1958824.1958865 · Cited by 253 · see citing works
Stigmatization of People with Pedophilia: Two Comparative Surveys
Jahnke, S., Imhoff, R., & Hoyer, J. (2014) · Archives of Sexual Behavior · doi:10.1007/s10508-014-0312-4 · Cited by 206 · see citing works
Clifford, S., & Jerit, J. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.5 · Cited by 206 · see citing works
Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults
Blumberg, S. J., & Luke, J. V. (2007) · Public Opinion Quarterly · doi:10.1093/poq/nfm047 · Cited by 192 · see citing works
Work experiences on MTurk: Job satisfaction, turnover, and information sharing
Brawley, A. M., & Pury, C. L. S. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2015.08.031 · Cited by 188 · see citing works
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Self-citationSimons, D. J., & Chabris, C. F. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051876 · Cited by 180 · see citing works
2010 Amendments to the 2002 "Ethical principles of psychologists and code of conduct".
(2010) · American Psychologist · doi:10.1037/a0020168 · Cited by 180 · see citing works
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D., & Schwarz, N. (2015) · SAGE Open · doi:10.1177/2158244015584617 · Cited by 173 · see citing works
Pearl, R. L., Puhl, R. M., & Dovidio, J. F. (2014) · Journal of Health Psychology · doi:10.1177/1359105313520338 · Cited by 166 · see citing works
QRTEngine: An easy solution for running online reaction time experiments using Qualtrics
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0530-7 · Cited by 166 · see citing works
Man enough? Masculine discrepancy stress and intimate partner violence
Reidy, D. E., Berke, D. S., Gentile, B., & Zeichner, A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.04.021 · Cited by 160 · see citing works
Daily Emotion in Non-Suicidal Self-Injury
Victor, S. E., & Klonsky, E. D. (2013) · Journal of Clinical Psychology · doi:10.1002/jclp.22037 · Cited by 160 · see citing works
Fergus, T. A. (2014) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.05.006 · Cited by 159 · see citing works
Crowdsourcing performance evaluations of user interfaces
Komarov, S., Reinecke, K., & Gajos, K. Z. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470684 · Cited by 157 · see citing works
Ethics and tactics of professional crowdwork
Silberman, M. ‘., Irani, L. C., & Ross, J. (2010) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1869086.1869100 · Cited by 144 · see citing works
Professional respondents in nonprobability online panels
Hillygus, D. S., Jackson, N., & Young, M. (2014) · Wiley eBooks · doi:10.1002/9781118763520.ch10 · Cited by 138 · see citing works
Negative affect intensity influences drinking to cope through facets of emotion dysregulation
Veilleux, J. C., Skinner, K. D., Reese, E. D., & Shaver, J. A. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.11.012 · Cited by 128 · see citing works
Enhancing social relationships through positive psychology activities: a randomised controlled trial
O’Connell, B. H., O’Shea, D., & Gallagher, S. (2016) · The Journal of Positive Psychology · doi:10.1080/17439760.2015.1037860 · Cited by 121 · see citing works
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 121 · see citing works
Participants at Your Fingertips
Johnson, D. R., & Borden, L. A. (2012) · Teaching of Psychology · doi:10.1177/0098628312456615 · Cited by 119 · see citing works
Swapping bricks for clicks: Crowdsourcing longitudinal data on Amazon Turk
Daly, T. M., & Nataraajan, R. (2015) · Journal of Business Research · doi:10.1016/j.jbusres.2015.05.001 · Cited by 119 · see citing works
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118 · see citing works
Self-citationThe face of quality in crowdsourcing relevance labels
Kazai, G., Kamps, J., & Milić-Frayling, N. (2012) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2396761.2398697 · Cited by 115 · see citing works
The public stigma of mental illness means a difference between you and me
Corrigan, P. W., Bink, A. B., Fokuo, J. K., & Schmidt, A. (2015) · Psychiatry Research · doi:10.1016/j.psychres.2014.12.047 · Cited by 114 · see citing works
Commonalities in grief responding across bereavement and non-bereavement losses
Papa, A., Lancaster, N. G., & Kahler, J. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.03.018 · Cited by 113 · see citing works
Opportunity costs of reward delays and the discounting of hypothetical money and cigarettes
Johnson, P. S., Herrmann, E. S., & Johnson, M. W. (2014) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.110 · Cited by 112 · see citing works
Non-suicidal self-injury disorder in a community sample of adults
Andover, M. S. (2014) · Psychiatry Research · doi:10.1016/j.psychres.2014.06.001 · Cited by 112 · see citing works
Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility
Schleider, J. L., & Weisz, J. R. (2015) · Journal of Child and Family Studies · doi:10.1007/s10826-015-0126-6 · Cited by 112 · see citing works
Attitudes Over Time: The Psychology of Panel Conditioning
Sturgis, P., Allum, N., & Brunton‐Smith, I. (2009) · Wiley eBooks · doi:10.1002/9780470743874.ch7 · Cited by 110 · see citing works
Jurs, S. G., & Glass, G. V. (1971) · The Journal of Experimental Education · doi:10.1080/00220973.1971.11011304 · Cited by 109 · see citing works
LGBT Demographics: Comparisons among population-based surveys
Gates, G. J. (2014) · eScholarship (California Digital Library) · Cited by 109 · see citing works
Evaluating and improving the usability of Mechanical Turk for low-income workers in India
Khanna, S., Ratan, A. L., Davis, J., & Thies, W. (2010) · ACM Symposium on Computing for Development (ACM DEV) · doi:10.1145/1926180.1926195 · Cited by 107 · see citing works
Short Forms of the Social Interaction Anxiety Scale and the Social Phobia Scale
Fergus, T. A., Valentiner, D. P., McGrath, P. B., Gier-Lonsway, S. L., & Kim, H.-S. (2012) · Journal of Personality Assessment · doi:10.1080/00223891.2012.660291 · Cited by 105 · see citing works
The Associations Among Dark Personalities and Sexual Tactics Across Different Scenarios
Jones, D. N., & Olderbak, S. G. (2013) · Journal of Interpersonal Violence · doi:10.1177/0886260513506053 · Cited by 98 · see citing works
Collecting response times using Amazon Mechanical Turk and Adobe Flash
Simcox, T., & Fiez, J. A. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0345-y · Cited by 97 · see citing works
Online crowdsourcing for efficient rating of speech: A validation study
Byun, T. M., Halpin, P. F., & Szeredi, D. (2014) · Journal of Communication Disorders · doi:10.1016/j.jcomdis.2014.11.003 · Cited by 95 · see citing works
Emotion regulation and obsessive–compulsive symptoms: A further examination of associations
Fergus, T. A., & Bardeen, J. R. (2014) · Journal of Obsessive-Compulsive and Related Disorders · doi:10.1016/j.jocrd.2014.06.001 · Cited by 93 · see citing works
Norr, A. M., Allan, N. P., Boffa, J. W., Raines, A. M., & Schmidt, N. B. (2015) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2015.02.001 · Cited by 92 · see citing works
The chronic disease concept of addiction: Helpful or harmful?
Wiens, T. K., & Walker, L. J. (2014) · Addiction Research & Theory · doi:10.3109/16066359.2014.987760 · Cited by 92 · see citing works
Miller, J. D., Few, L. R., Wilson, L. F., Gentile, B., Widiger, T. A., MacKillop, J., & Campbell, W. K. (2013) · Psychological Assessment · doi:10.1037/a0032536 · Cited by 88 · see citing works
‘Subtypes’ in the Presentation of Autistic Traits in the General Adult Population
Palmer, C. J., Paton, B. K., Enticott, P. G., & Hohwy, J. (2014) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-014-2289-1 · Cited by 87 · see citing works
Sexual Interest in Children Among an Online Sample of Men and Women
Wurtele, S. K., Simons, D. A., & Moreno, T. (2013) · Sexual Abuse · doi:10.1177/1079063213503688 · Cited by 84 · see citing works
Gupta, N., Martin, D., Hanrahan, B. V., & O’Neill, J. (2014) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/2660398.2660403 · Cited by 82 · see citing works
A positive effect of flowers rather than eye images in a large-scale, cross-cultural dictator game
Raihani, N., & Bshary, R. (2012) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2012.0758 · Cited by 82 · see citing works
Mechanical Turk is Not AnonymousPreprint
Lease, M., Hullman, J. R., Bigham, J. P., Bernstein, M. S., Kim, J., Lasecki, W. S., Bakhshi, S., Mitra, T., & Miller, R. C. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2228728 · Cited by 82 · see citing works
Development and validation of the Specific Loss of Interest and Pleasure Scale (SLIPS)
Winer, E. S., Veilleux, J. C., & Ginger, E. J. (2013) · Journal of Affective Disorders · doi:10.1016/j.jad.2013.09.010 · Cited by 79 · see citing works
Risks and Rewards of Crowdsourcing Marketplaces
Chandler, J. J., Paolacci, G., & Mueller, P. A. (2013) · Handbook of Human Computation · doi:10.1007/978-1-4614-8806-4_30 · Cited by 79 · see citing works
Self-citationPrivacy Attitudes of Mechanical Turk Workers and the U.S. Public
Kang, R., Brown, S. L., Dabbish, L., & Kiesler, S. B. (2014) · Symposium On Usable Privacy and Security · Cited by 78 · see citing works
Support matching and satisfaction in an online breast cancer support community
Vlahovic, T. A., Wang, Y., Kraut, R. E., & Levine, J. M. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557108 · Cited by 77 · see citing works
Arch, J. J. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.11.003 · Cited by 74 · see citing works
Lay beliefs about autism spectrum disorder among the general public and childcare providers
Mitchell, G. E., & Locke, K. D. (2015) · Autism · doi:10.1177/1362361314533839 · Cited by 74 · see citing works
Do Mechanical Turks dream of square pie charts?
Kosara, R., & Ziemkiewicz, C. (2010) · doi:10.1145/2110192.2110202 · Cited by 74 · see citing works
Honesty in an Online Labor Market
Suri, S., Goldstein, D. G., & Mason, W. (2013) · Cited by 70 · see citing works
Explaining the relationship between religiousness and substance use: Self-control matters.
DeWall, C. N., Pond, R. S., Carter, E. C., McCullough, M. E., Lambert, N. M., Fincham, F. D., & Nezlek, J. B. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0036853 · Cited by 68 · see citing works
Multidimensional Assessment of Beliefs About Emotion
Veilleux, J. C., Salomaa, A. C., Shaver, J. A., Zielinski, M. J., & Pollert, G. A. (2014) · Assessment · doi:10.1177/1073191114534883 · Cited by 68 · see citing works
Selectively Recruiting Participants from Amazon Mechanical Turk Using QualtricsPreprint
Péer, E., Paolacci, G., Chandler, J., & Mueller, P. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100631 · Cited by 67 · see citing works
Self-citationConvergent and Discriminant Validity of the Five Factor Form
Rojas, S. L., & Widiger, T. A. (2013) · Assessment · doi:10.1177/1073191113517260 · Cited by 66 · see citing works
An online daily diary study of alcohol use using A mazon's M echanical T urk
Boynton, M. H., & Richman, L. S. (2014) · Drug and Alcohol Review · doi:10.1111/dar.12163 · Cited by 63 · see citing works
Negative and positive urgency may both be risk factors for compulsive buying
Rose, P., & Segrist, D. J. (2014) · Journal of Behavioral Addictions · doi:10.1556/jba.3.2014.011 · Cited by 56 · see citing works
Lebowitz, M. S., Pyun, J. J., & Ahn, W. (2013) · Psychiatric Services · doi:10.1176/appi.ps.201300011 · Cited by 49 · see citing works
The credibility of exposure therapy: Does the theoretical rationale matter?
Arch, J. J., Twohig, M. P., Deacon, B. J., Landy, L. N., & Bluett, E. J. (2015) · Behaviour Research and Therapy · doi:10.1016/j.brat.2015.05.008 · Cited by 47 · cited 2× · see citing works
“Some researchers have used MTurk to recruit convenience samples of typically functioning adults to examine public attitudes about clinically relevant issues, such as mental health stigma (Corrigan et al. 2015) and attitudes toward therapy (Arch et al. 2015).”
BackgroundPsychopathic personality traits predict competitive wins and cooperative losses in negotiation
ten Brinke, L., Black, P., Porter, S., & Carney, D. R. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.02.001 · Cited by 46 · see citing works
Conducting real-time multiplayer experiments on the web
Hawkins, R. X. D. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0515-6 · Cited by 44 · see citing works
Cabrera, L. Y., Fitz, N. S., & Reiner, P. B. (2014) · Neuroethics · doi:10.1007/s12152-014-9223-2 · Cited by 42 · see citing works
Azzam, T., & Jacobson, M. R. (2013) · American Journal of Evaluation · doi:10.1177/1098214013490223 · Cited by 37 · see citing works
Miller, J. (2006) · Sage eBooks · doi:10.4135/9781412973380.n7 · Cited by 37 · see citing works
Ethical issues in psychological research on the Internet.
Buchanan, T., & Williams, J. E. (2010) · American Psychological Association eBooks · doi:10.1037/12076-016 · Cited by 36 · see citing works
Hoarding and eating pathology: the mediating role of emotion regulation
Raines, A. M., Boffa, J. W., Allan, N. P., Short, N. A., & Schmidt, N. B. (2015) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2014.11.005 · Cited by 35 · see citing works
Rose, P., & Segrist, D. J. (2012) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-012-9389-y · Cited by 35 · see citing works
Fergus, T. A. (2013) · Journal of Clinical Psychology · doi:10.1002/jclp.22034 · Cited by 33 · see citing works
Who is the Average Individual Investor? Numerical Skills and Implications for Accounting ResearchPreprint
Krische, S. D. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2426570 · Cited by 30 · see citing works
Abstract Analogical Reasoning in High‐Functioning Children with Autism Spectrum Disorders
Green, A. E., Kenworthy, L. E., Mosner, M. G., Gallagher, N. M., Fearon, E. W., Balhana, C. D., & Yerys, B. E. (2014) · Autism Research · doi:10.1002/aur.1411 · Cited by 28 · see citing works
Screening Amazon’s Mechanical Turk for Adults With ADHD
Wymbs, B. T., & Dawson, A. E. (2015) · Journal of Attention Disorders · doi:10.1177/1087054715597471 · Cited by 26 · see citing works
Generating Content for Scenario-Based Serious-Games Using CrowdSourcing
Sina, S., Rosenfeld, A., & Kraus, S. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8790 · Cited by 24 · see citing works
Investigating Several Factors Potentially Related to Endorsing Gambling as an Escape
Weatherly, J. N., & Cookman, M. L. (2014) · Current Psychology · doi:10.1007/s12144-014-9220-y · Cited by 23 · see citing works
Reese, E., & Veilleux, J. (2015) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntv054 · Cited by 23 · see citing works
Using Amazon Mechanical Turk to Transcribe Historical Handwritten Documents
Lang, A. S. I. D., & Rio-Ross, J. (2011) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 23 · see citing works
Crowd-powered positive psychological interventions
Morris, R. R., & Picard, R. W. (2014) · The Journal of Positive Psychology · doi:10.1080/17439760.2014.913671 · Cited by 23 · see citing works
Nichols, A. L., & Webster, G. D. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.01.043 · Cited by 22 · see citing works
Cooper, E. A., & Farid, H. (2014) · Public Understanding of Science · doi:10.1177/0963662514554354 · Cited by 21 · see citing works
Fergus, T. A., & Rowatt, W. C. (2014) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2014.09.003 · Cited by 21 · see citing works
Autism-spectrum traits predict humor styles in the general population
Eriksson, K. (2013) · Humor - International Journal of Humor Research · doi:10.1515/humor-2013-0030 · Cited by 21 · see citing works
Human Computation in Citizen Science
Lintott, C. J., & Reed, J. T. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_14 · Cited by 20 · see citing works
Interactive Effects of Anxiety Sensitivity and Emotion Regulation on Anxiety Symptoms
Allan, N. P., Norr, A. M., Macatee, R. J., Gajewska, A., & Schmidt, N. B. (2015) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-015-9484-z · Cited by 19 · see citing works
Wickham, R. E., Reed, D. E., & Williamson, R. E. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2014.12.049 · Cited by 19 · see citing works
Volunteering and Reemployment Status in Unemployed Emerging Adults
Konstam, V., Tomek, S. E., Celen‐Demirtas, S., & Sweeney, K. (2014) · Journal of Career Assessment · doi:10.1177/1069072714523248 · Cited by 19 · see citing works
Age-Related Slowing in Online Samples
Bui, D. C., Myerson, J., & Hale, S. (2015) · The Psychological Record · doi:10.1007/s40732-015-0135-2 · Cited by 18 · see citing works
A Comparison of Six Sample Providers Regarding Online Privacy Benchmarks
Schnorf, S., Sedley, A., Ortlieb, M., & Woodruff, A. (2014) · Cited by 17 · see citing works
Coping with causal uncertainty through alcohol use
Tobin, S. J., Loxton, N. J., & Neighbors, C. (2013) · Addictive Behaviors · doi:10.1016/j.addbeh.2013.11.009 · Cited by 16 · see citing works
Bridging the gap between eye tracking and crowdsourcing
Lebreton, P., Mäki, T., Skodras, E., Hupont, I., & Hirth, M. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2076745 · Cited by 16 · see citing works
Lebowitz, M. S., Ahn, W., & Oltman, K. A. (2015) · International Journal of Social Psychiatry · doi:10.1177/0020764015573086 · Cited by 14 · see citing works
Long-run learning in games of cooperation
Mason, W., Suri, S., & Watts, D. J. (2014) · ACM Conference on Economics and Computation (EC) · doi:10.1145/2600057.2602892 · Cited by 14 · see citing works
Cougle, J. R., Hawkins, K. A., Macatee, R. J., Zvolensky, M. J., & Sarawgi, S. (2014) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntu011 · Cited by 13 · see citing works
MTurk Crowdsourcing: A Viable Method for Rapid Discovery of Arabic Nicknames?
Higgins, C., McGrath, E. J., & Moretto, L. (2010) · Cited by 13 · see citing works
Nichols, A., & Webster, G. D. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.03.013 · Cited by 12 · see citing works
Yang, K., Friedman‐Wheeler, D. G., & Pronin, E. (2014) · Cognitive Therapy and Research · doi:10.1007/s10608-014-9597-9 · Cited by 12 · see citing works
Devlin, H. C., Johnson, S. L., & Gruber, J. (2015) · Cognitive Therapy and Research · doi:10.1007/s10608-015-9679-3 · Cited by 10 · see citing works
Characteristics of Borderline Personality Disorder and Disgust Sensitivity
Standish, A. J., Benfield, J. A., Bernstein, M. J., & Tragesser, S. L. (2014) · The Psychological Record · doi:10.1007/s40732-014-0039-6 · Cited by 8 · see citing works
Burke, S. E., Wang, K., & Dovidio, J. F. (2014) · Journal of Social and Clinical Psychology · doi:10.1521/jscp.2014.33.6.536 · Cited by 8 · see citing works
Gootzeit, J. H. (2014) · doi:10.17077/etd.grjd6zw4 · Cited by 7 · see citing works
Kim, J. (2014) · UWM Digital Commons (University of Wisconsin–Milwaukee) · Cited by 6 · see citing works
Shao, W., Guan, W., Clark, M. A., Liu, T., Santelices, C., Cortés, D. E., & Merchant, R. C. (2015) · PubMed · Cited by 6 · see citing works
Lawrence Berkeley National Lab. (LBNL), B., Greenblatt, J., USDOE Office of Science (SC), Yang, H.-C., Desroches, L., Young, S., Beraki, B., Price, S., Pratt, S., Willem, H., et al. (2013) · doi:10.2172/1171750 · Cited by 6 · see citing works
An Exploration of Mechanical Turk as a Feasible Recruitment Platform for Cancer Survivors
Carr, A. L. (2014) · CU Scholar (University of Colorado Boulder) · Cited by 5 · see citing works
Pilot study to evaluate the effectiveness of a mobile-based therapy and educational app for children
Howard, A. M., & MacCalla, J. (2014) · doi:10.1145/2668332.2676437 · Cited by 5 · see citing works
Cultural Differences in Playing Repeated Ultimatum Game Online with Virtual Humans
Nouri, E., & Traum, D. R. (2014) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2014.157 · Cited by 4 · see citing works
Cheating on online assessment tests: prevalence and impact on validity
Cavanagh, T. M., Kraiger, K., Gibbons, A., Henry, K., & Maynard, T. (2007) · Dominican Scholar (Dominican University of California) · doi:10.25675/3.024341 · Cited by 4 · see citing works
Shared Sense of Purpose and Well-Being among Veterans and Non-Veterans
Lynn, B. M.-D. (2014) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 4 · see citing works
U.S. Residential Miscellaneous Refrigeration Products: Results from Amazon Mechanical Turk Surveys
Lawrence Berkeley National Lab. (LBNL), B., Greenblatt, J., USDOE Office of Science (SC), Young, S., Yang, H.-C., Long, T., Beraki, B., Price, S., Pratt, S., Willem, H., et al. (2013) · doi:10.2172/1171618 · Cited by 4 · see citing works
Jiang, L., & Wagner, C. (2014) · Cited by 4 · see citing works
U.S. Residential Miscellaneous Refrigeration Products: Results from Amazon Mechanical Turk Surveys
Greenblatt, J. B., Young, S., Yang, H.-C., Long, T., Beraki, B., Price, S., Pratt, S., Willem, H., & Desroches, L. (2013) · doi:10.2172/1164375 · Cited by 4 · see citing works
Bernal, D. R. (2014) · Cited by 3 · see citing works
Using the Crowd to Generate Content for Scenario-Based Serious-GamesPreprint
Sina, S., Kraus, S., & Rosenfeld, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.5034 · Cited by 3 · see citing works
Effect of Internet Administered Mindfulness Training on Anxiety and Sleep Quality
Usinger, T. (2014) · CU Scholar (University of Colorado Boulder) · Cited by 1 · see citing works
Flagged first, then by citations
2 of the 177 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Understanding and managing the threat of common method bias: Detection, prevention and control
Kock, F., Berbekova, A., & Assaf, A. G. (2021) · Tourism Management · doi:10.1016/j.tourman.2021.104330 · Cited by 1,692 · see citing works
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 764 · see citing works
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668 · see citing works
Self-citationCustomer service chatbots: Anthropomorphism and adoption
Sheehan, B., Jin, H. S., & Gottlieb, U. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2020.04.030 · Cited by 667 · see citing works
Marroquín, B. M., Vine, V., & Morgan, R. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113419 · Cited by 637 · see citing works
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597 · see citing works
Demographics and Dynamics of Mechanical Turk Workers
Difallah, D. E., Filatova, E., & Ipeirotis, P. G. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159661 · Cited by 512 · see citing works
Mortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 441 · see citing works
Trust in a COVID-19 vaccine in the U.S.: A social-ecological perspective
Latkin, C. A., Dayton, L., Yi, G. T., Konstantopoulos, A. D., & Boodram, B. (2021) · Social Science & Medicine · doi:10.1016/j.socscimed.2021.113684 · Cited by 383 · see citing works
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 346 · see citing works
Nikčević, A. V., Marino, C., Kolubinski, D. C., Leach, D., & Spada, M. M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.10.053 · Cited by 344 · see citing works
Kahan, D. M. (2016) · Emerging Trends in the Social and Behavioral Sciences · doi:10.1002/9781118900772.etrds0417 · Cited by 341 · see citing works
Mental distress among U.S. adults during the COVID-19 pandemic
Twenge, J. M., & Joiner, T. E. (2020) · Journal of Clinical Psychology · doi:10.1002/jclp.23064 · Cited by 330 · see citing works
Exploring the structure of narcissism: Toward an integrated solution
Crowe, M. L., Lynam, D. R., Campbell, W. K., & Miller, J. D. (2019) · Journal of Personality · doi:10.1111/jopy.12464 · Cited by 317 · see citing works
The Influence of Climate Fiction
Schneider-Mayerson, M. (2018) · Environmental Humanities · doi:10.1215/22011919-7156848 · Cited by 281 · see citing works
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 279 · see citing works
Self-citationNo Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments
Kane, J., & Barabas, J. (2018) · American Journal of Political Science · doi:10.1111/ajps.12396 · Cited by 276 · see citing works
Agley, J. D., & Xiao, Y. (2021) · BMC Public Health · doi:10.1186/s12889-020-10103-x · Cited by 266 · see citing works
Oltmanns, J. R., & Widiger, T. A. (2017) · Psychological Assessment · doi:10.1037/pas0000459 · Cited by 249 · see citing works
The use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2018) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 249 · see citing works
Kim, J., & Im, I. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107512 · Cited by 233 · see citing works
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
McCredie, M. N., & Morey, L. C. (2018) · Assessment · doi:10.1177/1073191118760709 · Cited by 232 · see citing works
Comprehensive meta-analysis of resilience interventions
Liu, J. J. W., Ein, N., Gervasio, J., Battaion, M., Reed, M. J., & Vickers, K. (2020) · Clinical Psychology Review · doi:10.1016/j.cpr.2020.101919 · Cited by 231 · see citing works
Blagov, P. S. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550620936439 · Cited by 219 · see citing works
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 216 · see citing works
Self-citationOpioid use and stigma: The role of gender, language and precipitating events
Goodyear, K., Haass‐Koffler, C. L., & Chavanne, D. (2018) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2017.12.037 · Cited by 208 · see citing works
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203 · see citing works
The COVID-19 anxiety syndrome scale: development and psychometric properties.
Nikčević, A. V., & Spada, M. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113322 · Cited by 186 · see citing works
Scientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 182 · see citing works
Stritch, J. M., Pedersen, M. J., & Taggart, G. (2017) · International Public Management Journal · doi:10.1080/10967494.2016.1276493 · Cited by 181 · see citing works
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000191 · Cited by 179 · see citing works
COVID-19 vaccine intentions in the United States, a social-ecological framework
Latkin, C. A., Dayton, L., Yi, G. T., Konstantopoulos, A. D., Park, J. N., Maulsby, C. H., & Kong, X. (2021) · Vaccine · doi:10.1016/j.vaccine.2021.02.058 · Cited by 179 · see citing works
A Critical Appraisal of the Dark-Triad Literature and Suggestions for Moving Forward
Miller, J. D., Vize, C. E., Crowe, M. L., & Lynam, D. R. (2019) · Current Directions in Psychological Science · doi:10.1177/0963721419838233 · Cited by 178 · see citing works
Jeong, M., Zhang, D., Morgan, J. C., Ross, J. C., Osman, A., Boynton, M. H., Sheldon, J. M., & Brewer, N. T. (2018) · Annals of Behavioral Medicine · doi:10.1093/abm/kay059 · Cited by 177 · see citing works
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168 · see citing works
The prevalence of aphantasia (imagery weakness) in the general population
Dance, C. J., Ipser, A., & Simner, J. (2022) · Consciousness and Cognition · doi:10.1016/j.concog.2021.103243 · Cited by 167 · see citing works
Cyberchondria: examining relations with problematic internet use and metacognitive beliefs
Fergus, T. A., & Spada, M. M. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2102 · Cited by 162 · see citing works
Common Concerns with MTurk as a Participant Pool: Evidence and SolutionsPreprint
Hauser, D., Paolacci, G., & Chandler, J. (2018) · doi:10.31234/osf.io/uq45c · Cited by 156 · see citing works
Self-citationIdentifying Predictors of Psychological Distress During COVID-19: A Machine Learning Approach
Prout, T. A., Zilcha‐Mano, S., Doorn, K. A., Békés, V., Christman-Cohen, I., Whistler, K., Kui, T., & Giuseppe, M. D. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.586202 · Cited by 154 · see citing works
Rethinking the link between cognitive sophistication and politically motivated reasoning.
Tappin, B. M., Pennycook, G., & Rand, D. G. (2021) · Journal of Experimental Psychology: General · doi:10.1037/xge0000974 · Cited by 154 · see citing works
Frewen, P., Zhu, J., & Lanius, R. (2019) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1606625 · Cited by 151 · see citing works
Uncovering the structure of agreeableness from self‐report measures
Crowe, M. L., Lynam, D. R., & Miller, J. D. (2017) · Journal of Personality · doi:10.1111/jopy.12358 · Cited by 147 · see citing works
Murray, H. B., Dreier, M. J., Zickgraf, H. F., Becker, K. R., Breithaupt, L. E., Eddy, K. T., & Thomas, J. J. (2021) · International Journal of Eating Disorders · doi:10.1002/eat.23520 · Cited by 146 · see citing works
Thompson, A. J., & Pickett, J. T. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09436-7 · Cited by 130 · see citing works
Father Participation in Child Psychopathology Research
Parent, J. M., Forehand, R., Pomerantz, H., Peisch, V. D., & Seehuus, M. (2017) · Research on Child and Adolescent Psychopathology · doi:10.1007/s10802-016-0254-5 · Cited by 127 · see citing works
Mapping the relationship between anxiety, anhedonia, and depression
Winer, E. S., Bryant, J., Bartoszek, G., Rojas, E., Nadorff, M. R., & Kilgore, J. (2017) · Journal of Affective Disorders · doi:10.1016/j.jad.2017.06.006 · Cited by 124 · see citing works
The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers
Ophir, Y., Sisso, I., Asterhan, C. S. C., Tikochinski, R., & Reichart, R. (2019) · Clinical Psychological Science · doi:10.1177/2167702619865973 · Cited by 124 · see citing works
Participant carelessness and fraud: Consequences for clinical research and potential solutions.
Chandler, J., Sisso, I., & Shapiro, D. N. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000479 · Cited by 124 · see citing works
Self-citationCasey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 122 · see citing works
Self-citationSleep, C. E., Lynam, D. R., Widiger, T. A., Crowe, M. L., & Miller, J. D. (2019) · Psychological Assessment · doi:10.1037/pas0000620 · Cited by 122 · see citing works
Everaert, J., & Joormann, J. (2019) · Clinical Psychological Science · doi:10.1177/2167702619859342 · Cited by 119 · see citing works
Everaert, J., Bronstein, M. V., Castro, A. A., Cannon, T. D., & Joormann, J. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103510 · Cited by 119 · see citing works
Agley, J. D., Xiao, Y., Nolan, R., & Golzarri‐Arroyo, L. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01665-8 · Cited by 118 · see citing works
Development and Initial Validation of the Duke Misophonia Questionnaire
Rosenthal, M. Z., Anand, D., Cassiello-Robbins, C., Williams, Z. J., Guetta, R. E., Trumbull, J., & Kelley, L. D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.709928 · Cited by 117 · see citing works
Trigger warning: Empirical evidence ahead
Bellet, B. W., Jones, P. J., & McNally, R. J. (2018) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2018.07.002 · Cited by 117 · see citing works
Behavioral and psychosocial factors associated with COVID-19 skepticism in the United States
Latkin, C. A., Dayton, L., Moran, M. B., Strickland, J. C., & Collins, K. (2021) · Current Psychology · doi:10.1007/s12144-020-01211-3 · Cited by 117 · see citing works
Factors that predict taking restaurant leftovers: Strategies for reducing food waste
Hamerman, E. J., Rudell, F., & Martins, C. M. (2017) · Journal of Consumer Behaviour · doi:10.1002/cb.1700 · Cited by 114 · see citing works
Stavropoulos, V., Anderson, E. L., Beard, C. L., Latifi, M. Q., Kuss, D. J., & Griffiths, M. D. (2018) · Addictive Behaviors Reports · doi:10.1016/j.abrep.2018.10.001 · Cited by 113 · see citing works
The uncanny valley: No need for any further judgments when an avatar looks eerie
Shin, M., Kim, S. J., & Biocca, F. A. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.016 · Cited by 113 · see citing works
A Technical Guide to Using Amazon's Mechanical Turk in Behavioral Accounting Research
Buchheit, S., Doxey, M. M., Pollard, T., & Stinson, S. R. (2017) · Behavioral Research in Accounting · doi:10.2308/bria-51977 · Cited by 111 · see citing works
Evaluating the assessment of the ICD-11 personality disorder diagnostic system.
Oltmanns, J. R., & Widiger, T. A. (2019) · Psychological Assessment · doi:10.1037/pas0000693 · Cited by 109 · see citing works
Sorkin, D. H., Janio, E. A., Eikey, E. V., Schneider, M., Davis, K., Schueller, S. M., Stadnick, N. A., Zheng, K., Neary, M., Safani, D., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26994 · Cited by 108 · see citing works
Rodd, J. M. (2023) · Journal of Memory and Language · doi:10.1016/j.jml.2023.104472 · Cited by 101 · see citing works
Inattentive responding can induce spurious associations between task behaviour and symptom measures
Zorowitz, S., Solis, J., Niv, Y., & Bennett, D. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01640-7 · Cited by 100 · see citing works
Minding the gap: Subjective relative deprivation and depressive symptoms
Beshai, S., Mishra, S., Meadows, T. J. S., Parmar, P., & Huang, V. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.11.021 · Cited by 98 · see citing works
Stavropoulos, V., Beard, C. L., Griffiths, M. D., Burleigh, T. L., Gomez, R., & Pontes, H. M. (2017) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-017-9786-3 · Cited by 98 · see citing works
Schäfer, K., & Eerola, T. (2018) · Psychology of Music · doi:10.1177/0305735618795036 · Cited by 97 · see citing works
The Politically Motivated Reasoning Paradigm, Part 2: Unanswered Questions
Kahan, D. M. (2016) · Wiley eBooks · doi:10.1002/9781118900772.etrds0418 · Cited by 97 · see citing works
Age of Initiation and Internet Gaming Disorder: The Role of Self-Esteem
Beard, C. L., Haas, A. L., Wickham, R. E., & Stavropoulos, V. (2017) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2017.0011 · Cited by 96 · see citing works
Social Media Contributions to Strong Black Woman Ideal Endorsement and Black Women’s Mental Health
Stanton, A. G., Jerald, M. C., Ward, L. M., & Avery, L. R. (2017) · Psychology of Women Quarterly · doi:10.1177/0361684317732330 · Cited by 96 · see citing works
An Assessment of the Rapid Decline of Trust in US Sources of Public Information about COVID-19
Latkin, C. A., Dayton, L., Strickland, J. C., Colon, B., Rimal, R. N., & Boodram, B. (2020) · Journal of Health Communication · doi:10.1080/10810730.2020.1865487 · Cited by 95 · see citing works
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94 · see citing works
Mechanisms of offline motor learning at a microscale of seconds in large-scale crowdsourced data
Bönstrup, M., Iturrate, I., Hebart, M. N., Censor, N., & Cohen, L. G. (2020) · npj Science of Learning · doi:10.1038/s41539-020-0066-9 · Cited by 94 · see citing works
Self-service technology kiosk design for restaurants: An QFD application
Park, S., Lehto, X. Y., & Lehto, M. R. (2020) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2020.102757 · Cited by 94 · see citing works
A comprehensive comparison of the ICD-11 and DSM–5 section III personality disorder models.
McCabe, G. A., & Widiger, T. A. (2020) · Psychological Assessment · doi:10.1037/pas0000772 · Cited by 93 · see citing works
Gleibs, I. H. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0789-y · Cited by 93 · see citing works
Temporal self appraisal and continuous identity: Associations with depression and hopelessness
Sokol, Y., & Serper, M. R. (2016) · Journal of Affective Disorders · doi:10.1016/j.jad.2016.10.033 · Cited by 92 · see citing works
Overlooked and Underestimated: Experiences of Ageism in Young, Middle-Aged, and Older Adults
Chasteen, A. L., Horhota, M., & Crumley-Branyon, J. J. (2020) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbaa043 · Cited by 92 · see citing works
Let’s face it: Are customers ready for facial recognition technology at quick-service restaurants?
Ciftci, O., Choi, E.-K., & Berezina, K. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.102941 · Cited by 92 · see citing works
What is the relationship between Aphantasia, Synaesthesia and Autism?
Dance, C. J., Jaquiery, M., Eagleman, D. M., Porteous, D., Zeman, A., & Simner, J. (2021) · Consciousness and Cognition · doi:10.1016/j.concog.2021.103087 · Cited by 89 · see citing works
Using Mechanical Turk Data in IS Research: Risks, Rewards, and Recommendations
Jia, R., Steelman, Z., & Reich, B. H. (2017) · Communications of the Association for Information Systems · doi:10.17705/1cais.04114 · Cited by 88 · see citing works
Beshai, S., Prentice, J. L., & Huang, V. (2017) · Mindfulness · doi:10.1007/s12671-017-0833-8 · Cited by 86 · see citing works
Smartphones and the Neuroscience of Mental Health
Gillan, C. M., & Rutledge, R. B. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-101220-014053 · Cited by 84 · see citing works
Poole, J. C., Kim, H. S. A., Dobson, K. S., & Hodgins, D. C. (2017) · Journal of Gambling Studies · doi:10.1007/s10899-017-9680-8 · Cited by 83 · see citing works
Oltmanns, J. R., & Widiger, T. A. (2018) · Psychological Assessment · doi:10.1037/pas0000616 · Cited by 82 · see citing works
Why not eat alone? The effect of other consumers on solo dining intentions and the mechanism
Her, E., & Seo, S. (2017) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.10.019 · Cited by 82 · see citing works
Latkin, C. A., Dayton, L., Miller, J., Yi, G. T., Balaban, A., Boodram, B., Uzzi, M., & Falade-Nwulia, O. O. (2022) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2022.2043102 · Cited by 81 · see citing works
Fergus, T. A., & Spada, M. M. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.09.003 · Cited by 80 · see citing works
Millroth, P., & Frey, R. (2021) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2021.102454 · Cited by 80 · see citing works
Moral polarization and out-party hostility in the US political context
Tappin, B. M., & McKay, R. T. (2019) · Journal of Social and Political Psychology · doi:10.5964/jspp.v7i1.1090 · Cited by 79 · see citing works
Dixon, G. (2016) · Journal of Communication · doi:10.1111/jcom.12260 · Cited by 79 · see citing works
Lie prevalence, lie characteristics and strategies of self-reported good liars
Verigin, B. L., Meijer, E. H., Bogaard, G., & Vrij, A. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225566 · Cited by 78 · see citing works
Public Perceptions of Physician Attire and Professionalism in the US
Xun, H., Chen, J., Sun, A. H., Jenny, H. E., Liang, F., & Steinberg, J. P. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.17779 · Cited by 78 · see citing works
Framing fast and slow: a dual processing account of multimodal framing effects
Powell, T. E., Boomgaarden, H. G., Swert, K. D., & de Vreese, C. H. (2018) · Media Psychology · doi:10.1080/15213269.2018.1476891 · Cited by 75 · see citing works
A panel for lemons? Positivity bias, reputation systems and data quality on MTurk
Matherly, T. (2019) · European Journal of Marketing · doi:10.1108/ejm-07-2017-0491 · Cited by 75 · see citing works
Crowdsourcing Trauma: Psychopathology in a Trauma‐Exposed Sample Recruited via Mechanical Turk
van Stolk‐Cooke, K., Brown, A., Maheux, A. J., Parent, J. M., Forehand, R., & Price, M. (2018) · Journal of Traumatic Stress · doi:10.1002/jts.22303 · Cited by 72 · see citing works
Yap, K., & Grisham, J. R. (2019) · Journal of Behavioral Addictions · doi:10.1556/2006.8.2019.15 · Cited by 71 · see citing works
Use of online crowdsourcing platforms for gambling research
Mishra, S., & Carleton, R. N. (2017) · International Gambling Studies · doi:10.1080/14459795.2017.1284250 · Cited by 71 · see citing works
Wackowski, O. A., Sontag, J. M., Hammond, D. G., O’Connor, R. J., Ohman‐Strickland, P., Strasser, A. A., Villanti, A. C., & Delnevo, C. D. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16020184 · Cited by 70 · see citing works
Mahali, S. C., Beshai, S., Feeney, J. R., & Mishra, S. (2020) · BMC Psychiatry · doi:10.1186/s12888-019-2423-x · Cited by 69 · see citing works
- Authors
- Chandler, J., & Shapiro, D.
- Affiliations
- Mathematica Policy Research (US)
- University of Michigan (US)
- ORCID
- Chandler, Jesse
- Venue
- Annual Review of Clinical Psychology
- Publisher
- Annual Reviews
- Published
- 2016-03-28
- Volume
- 12
- Issue
- 1
- Pages
- 53-81
- ISSN
- 1548-5943 (ISSN-L)
- 1548-5951
- OpenAlex
- W2126493218
- Cited by
- 1,072
- References
- 177
- Citations by year
- 2026: 27 · 2025: 70 · 2024: 105 · 2023: 103 · 2022: 150 · 2021: 168 · 2020: 122 · 2019: 115 · 2018: 111 · 2017: 77 · 2016: 24
- FWCI
- 17.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Annual Review of Clinical Psychology (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Ethics in Clinical Research
- Keywords
- crowdsourcing, Amazon Mechanical Turk, convenience samples, non-probability sampling, sample composition, data quality, participant recruitment, digital labour markets
- MeSH terms
- Humans, Biomedical Research, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review