Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

Sheehan, Kim Bartel

doi:10.1080/03637751.2017.1342043

Cited by 413Mobile Crowdsensing and CrowdsourcingSurvey Methodology and NonresponseComputational and Text Analysis Methods

Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

Sheehan, Kim Bartel·2018·Communication Monographs

DOI 10.1080/03637751.2017.1342043

Open this paper in CoChat — free account

Abstract

Researchers in a variety of disciplines use Amazon’s crowdsourcing platform called Mechanical Turk as a way to collect data from a respondent pool that is much more diverse than a typical student sample. The platform also provides cost efficiencies over other online panel services and data can be collected very quickly. However, some researchers have been slower to try the platform, perhaps because of a lack of awareness of its functions or concerns with validity. This article provides an overview of Mechanical Turk as an academic research platform and a critical examination of its strengths and weaknesses for research. Guidelines for collecting data that address issues of validity, reliability, and ethics are presented.

Integrity signals

Registry

Crossref · 10.1080/03637751.2017.1342043

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Respondents were recruited via Amazon Mechanical Turk (MTurk), which has been utilized for many social science studies for sample recruitment, and data quality confirmed in previous studies.
“Respondents were recruited via Amazon Mechanical Turk (MTurk), which has been utilized for many social science studies for sample recruitment, and data quality confirmed in previous studies (e.g., Buhrmester et al., 2011; Sheehan, 2018).”
From Fostering Employee Trust via Effective Supervisory Communication during the COVID-19 Pandemic: Through the Lens of Motivating Language Theory (2022) · cites this paper as Sheehan, 2018
The MTurk population is generally more diverse than the typical sample of college students used in social science research.
“Another benefit of using MTurk is that the population is generally more diverse than the typical sample of college students used in social science research [60].”
From Internalized Racism and Mental Health: The Moderating Role of Collective Racial Self-Esteem (2024) · cites this paper as [60]

The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
WEIRD languages have misled us, too
Majid, A., & Levinson, S. C. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1000018x · Cited by 4,169 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Introducing the Short Dark Triad (SD3): A Brief Measure of Dark Personality Traits
Jones, D. N., & Paulhus, D. L. (2014) · Assessment · doi:10.1177/1073191113514105 · Cited by 2,162 · see citing works
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1× · see citing works
“Casler, Bickel, and Hackett (2013) found consistency in responses from snowball and MTurk samples, concluding that, “ for some behavioral tests, online recruitment and testing can be a valid – and sometimes even superior – method to in-person data collection ” (p. 2159).”
Background
The Online Laboratory: Conducting Experiments in a Real Labor Market∗Preprint
Zeckhauser, R. J. (2016) · Cited by 1,454 · see citing works
“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents
Huff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · see citing works
An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902 · see citing works
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 890 · see citing works
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798 · see citing works
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772 · see citing works
MOVING THE CROWD AT THREADLESS
Brabham, D. C. (2010) · Information Communication & Society · doi:10.1080/13691181003624090 · Cited by 561 · see citing works
A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples
Smith, S. M., Roster, C. A., Golden, L. L., & Albaum, G. (2016) · Journal of Business Research · doi:10.1016/j.jbusres.2015.12.002 · Cited by 415 · see citing works
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330 · see citing works
The importance of assessing clinical phenomena in Mechanical Turk research.
Hall, K. A. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2015) · Psychological Assessment · doi:10.1037/pas0000217 · Cited by 311 · see citing works
The Global Opportunity in Online Outsourcing
Kuek, S. C., Paradi-Guilford, C., Fayomi, T., Imaizumi, S., Ipeirotis, P. G., Pina, P., & Singh, M. (2015) · World Bank, Washington, DC eBooks · doi:10.1596/22284 · Cited by 306 · see citing works
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1× · see citing works
“Stewart et al. (2015) used a capture – recapture methodology from wildlife ecology to estimate that the average lab is sampling from about 7300 individuals on MTurk.”
Method
The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk
Litman, L., Robinson, J., & Rosenzweig, C. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275 · see citing works
Participant retention practices in longitudinal clinical research studies with high retention rates
Abshire, M., Dinglas, V. D., Cajita, M. I. A., Eakin, M. N., Needham, D. M., & Himmelfarb, C. D. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0310-z · Cited by 274 · see citing works
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185 · see citing works
Common (Mis)Beliefs about Memory: A Replication and Comparison of Telephone and Mechanical Turk Survey Methods
Simons, D. J., & Chabris, C. F. (2012) · PLoS ONE · doi:10.1371/journal.pone.0051876 · Cited by 180 · see citing works
Assessing the reliability of the M5-120 on Amazon’s mechanical Turk
Holden, C. J., Dennie, T., & Hicks, A. D. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.02.020 · Cited by 169 · see citing works
Walking the thin line between efficiency and accuracy: Validity and structural properties of the Dirty Dozen
Jonason, P. K., & Luévano, V. X. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.02.010 · Cited by 145 · see citing works
Amazon's Mechanical Turk for Academics: The HIT Handbook for Social Science Research
Sheehan, K. B., & Pittman, M. (2016) · Cited by 126 · see citing works
The benign online disinhibition effect: Could situational factors induce self-disclosure and prosocial behaviors?
Lapidot‐Lefler, N., & Barak, A. (2015) · Cyberpsychology Journal of Psychosocial Research on Cyberspace · doi:10.5817/cp2015-2-3 · Cited by 120 · see citing works
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118 · see citing works
Using Amazon's Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction
Gardner, R. M., Brown, D. L., & Boice, R. (2012) · Body Image · doi:10.1016/j.bodyim.2012.06.006 · Cited by 113 · see citing works
Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility
Schleider, J. L., & Weisz, J. R. (2015) · Journal of Child and Family Studies · doi:10.1007/s10826-015-0126-6 · Cited by 112 · see citing works
Amazon's Mechanical Turk: A Comment
Ford, J. B. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1277380 · Cited by 112 · see citing works
A Comparison of Three Online Recruitment Strategies for Engaging Parents
Dworkin, J., Hessel, H. M., Gliske, K., & Rudi, J. H. (2016) · Family Relations · doi:10.1111/fare.12206 · Cited by 99 · see citing works
The affective profiles in the USA: happiness, depression, life satisfaction, and happiness-increasing strategies
Schütz, E., Sailer, U., Nima, A. A., Rosenberg, P., Arntén, A.-C. A., Archer, T., & Garcia, D. (2013) · PeerJ · doi:10.7717/peerj.156 · Cited by 96 · see citing works
Mechanical Turk is Not AnonymousPreprint
Lease, M., Hullman, J. R., Bigham, J. P., Bernstein, M. S., Kim, J., Lasecki, W. S., Bakhshi, S., Mitra, T., & Miller, R. C. (2013) · SSRN Electronic Journal · doi:10.2139/ssrn.2228728 · Cited by 82 · see citing works
AI Gets a Brain
Barr, J., & Cabrera, L. F. (2006) · Queue · doi:10.1145/1142055.1142067 · Cited by 70 · see citing works
The impact of overtly listing eligibility requirements on MTurk: An investigation involving organ donation, recruitment scripts, and feelings of elevation
Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.08.020 · Cited by 46 · see citing works
A Primer for Conducting Survey Research using MTurk
Chambers, S., Nimon, K., & Anthony-McMann, P. E. (2016) · International Journal of Adult Vocational Education and Technology · doi:10.4018/ijavet.2016040105 · Cited by 41 · see citing works
The Good, The Bad and the Ugly: Why Crowdsourcing Needs Ethics
Schmidt, F. A. (2013) · International Conference on Cloud and Green Computing · doi:10.1109/cgc.2013.89 · Cited by 40 · see citing works
Practice-based considerations for using multi-stage survey design to reach special populations on Amazon’s Mechanical Turk
Springer, V. A., Martini, P. J., Lindsey, S. C., & Vezich, I. S. (2016) · Survey Practice · doi:10.29115/sp-2016-0029 · Cited by 34 · see citing works
Crowdsourcing for information retrieval
Alonso, O., & Lease, M. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010170 · Cited by 30 · see citing works
Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers’ Perceptions of Crowdsourcing Career
Deng, X. N., & Joshi, K. D. (2013) · International Conference on Information Systems · Cited by 22 · see citing works
Does the Sun revolve around the Earth? A comparison between the general public and online survey respondents in basic scientific knowledge
Cooper, E. A., & Farid, H. (2014) · Public Understanding of Science · doi:10.1177/0963662514554354 · Cited by 21 · see citing works

Flagged first, then by citations

Sorted by citations

Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions
Araujo, T. (2018) · Computers in Human Behavior · doi:10.1016/j.chb.2018.03.051 · Cited by 1,156 · see citing works
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · see citing works
I, Chatbot: Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents
Ashfaq, M., Yun, J., Yu, S., & Loureiro, S. M. C. (2020) · Telematics and Informatics · doi:10.1016/j.tele.2020.101473 · Cited by 1,100 · see citing works
Americans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728 · see citing works
Initial Challenges of Caregiving During COVID-19: Caregiver Burden, Mental Health, and the Parent–Child Relationship
Russell, B. S., Hutchison, M., Tambling, R. R., Tomkunas, A. J., & Horton, A. L. (2020) · Child Psychiatry & Human Development · doi:10.1007/s10578-020-01037-x · Cited by 715 · see citing works
Comparing Amazon’s Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature
Mortensen, K., & Hughes, T. L. (2018) · Journal of General Internal Medicine · doi:10.1007/s11606-017-4246-0 · Cited by 442 · see citing works
Knowledge hiding as a barrier to thriving: The mediating role of psychological safety and moderating role of organizational cynicism
Jiang, Z., Hu, X., Wang, Z., & Jiang, X. (2019) · Journal of Organizational Behavior · doi:10.1002/job.2358 · Cited by 289 · see citing works
The Relations between Hope and Subjective Well-Being: a Literature Overview and Empirical Analysis
Pleeging, E., Burger, M., & van Exel, J. N. A. (2019) · Applied Research in Quality of Life · doi:10.1007/s11482-019-09802-4 · Cited by 233 · see citing works
Internet Use, Risk Awareness, and Demographic Characteristics Associated With Engagement in Preventive Behaviors and Testing: Cross-Sectional Survey on COVID-19 in the United States
Li, S., Feng, B., Liao, W., & Pan, W. (2020) · Journal of Medical Internet Research · doi:10.2196/19782 · Cited by 201 · see citing works
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
ElSherief, M., Ziems, C., Muchlinski, D., Anupindi, V., Seybolt, J., Choudhury, M. D., & Yang, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.29 · Cited by 201 · see citing works
Keeping up with influencers: exploring the impact of social presence and parasocial interactions on Instagram
Kim, H. (2021) · International Journal of Advertising · doi:10.1080/02650487.2021.1886477 · Cited by 172 · see citing works
My Smart Speaker is Cool! Perceived Coolness, Perceived Values, and Users’ Attitude toward Smart Speakers
Ashfaq, M., Yun, J., & Yu, S. (2020) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2020.1841404 · Cited by 130 · see citing works
Gender Differences in Parents’ Communication With Their Adolescent Children about Sexual Risk and Sex-Positive Topics
Evans-Paulson, R., Widman, L. M., Kamke, K., & Stewart, J. (2019) · The Journal of Sex Research · doi:10.1080/00224499.2019.1661345 · Cited by 129 · see citing works
Exploring the Impact of Internal Communication on Employee Psychological Well-Being During the COVID-19 Pandemic: The Mediating Role of Employee Organizational Trust
Qin, Y. S., & Men, L. R. (2022) · International Journal of Business Communication · doi:10.1177/23294884221081838 · Cited by 125 · see citing works
Fostering Employee Trust via Effective Supervisory Communication during the COVID-19 Pandemic: Through the Lens of Motivating Language Theory
Men, L. R., Qin, Y. S., & Jin, J. (2022) · International Journal of Business Communication · doi:10.1177/23294884211020491 · Cited by 121 · see citing works
“Respondents were recruited via Amazon Mechanical Turk (MTurk), which has been utilized for many social science studies for sample recruitment, and data quality confirmed in previous studies (e.g., Buhrmester et al., 2011; Sheehan, 2018).”
Method
Racial differences in perceived food swamp and food desert exposure and disparities in self-reported dietary habits
Cooksey Stowers, K., Jiang, Q., Atoloye, A., Lucan, S., & Gans, K. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17197143 · Cited by 117 · see citing works
Does meaningful work reduce cyberloafing? Important roles of affective commitment and leader-member exchange
Usman, M., Javed, U., Shoukat, A., & Bashir, N. A. (2019) · Behaviour and Information Technology · doi:10.1080/0144929x.2019.1683607 · Cited by 116 · see citing works
Developing and Validating the Communication Resilience Processes Scale
Wilson, S. R., Kuang, K., Hintz, E. A., & Buzzanell, P. M. (2021) · Journal of Communication · doi:10.1093/joc/jqab013 · Cited by 119 · see citing works
A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk)
Çobanoğlu, C., Çavuşoğlu, M., & Türktarhan, G. (2021) · Journal of Global Business Insights · doi:10.5038/2640-6489.6.1.1177 · Cited by 90 · see citing works
Temporal Certainty and Skippable In-Stream Commercials: Effects of Ad Length, Timer, and Skip-ad Button on Irritation and Skipping Behavior
Jeon, Y. A., Son, H., Chung, A., & Drumwright, M. E. (2019) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2019.02.005 · Cited by 88 · see citing works
General framework, opportunities and challenges for crowdsourcing techniques: A Comprehensive survey
Bhatti, S. S., Gao, X., & Chen, G. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110611 · Cited by 85 · see citing works
Have an omnichannel seamless interaction experience! Dimensions and effect on consumer satisfaction
Rodríguez–Torrico, P., Trabold, L., Gutiérrez, S. S. M., & Cabezudo, R. S. J. (2020) · Journal of Marketing Management · doi:10.1080/0267257x.2020.1801798 · Cited by 86 · see citing works
Still Creepy After All These Years:The Normalization of Affective Discomfort in App Use
Seberger, J. S., Shklovski, I., Swiatek, E., & Patil, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502112 · Cited by 75 · see citing works
Collecting Online Survey Data: A Comparison of Data Quality among a Commercial Panel & MTurk
Zhang, B., & Gearhart, S. (2020) · Survey Practice · doi:10.29115/sp-2020-0015 · Cited by 73 · see citing works
From Lab-Testing to Web-Testing in Cognitive Research: Who You Test is More Important than how You Test
Uittenhove, K. L., Jeanneret, S., & Vergauwe, E. (2023) · Journal of Cognition · doi:10.5334/joc.259 · Cited by 73 · see citing works
Hospitality and Tourism Student Engagement and Hope During the COVID-19 Pandemic
Zhong, Y.-Y., Busser, J., Shapoval, V., & Murphy, K. (2021) · Journal of Hospitality and Tourism Education · doi:10.1080/10963758.2021.1907197 · Cited by 73 · see citing works
Communication outside of the home through social media during COVID-19
Pennington, N. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100118 · Cited by 72 · see citing works
The effects of emotions, individual attitudes towards vaccination, and social endorsements on perceived fake news credibility and sharing motivations
Ali, K. I., Li, C., Zain-ul-abdin, K., & Muqtadir, S. A. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107307 · Cited by 76 · see citing works
Americans’ distress early in the COVID-19 pandemic: Protective resources and coping strategies.
Park, C. L., Finkelstein‐Fox, L., Russell, B. S., Fendrich, M., Hutchison, M., & Becker, J. (2021) · Psychological Trauma Theory Research Practice and Policy · doi:10.1037/tra0000931 · Cited by 70 · see citing works
Sampling and validity
McEwan, B. L. (2020) · Annals of the International Communication Association · doi:10.1080/23808985.2020.1792793 · Cited by 69 · see citing works
Designing to Debias: Measuring and Reducing Public Managers’ Anchoring Bias
Nagtegaal, R., Tummers, L. G., Noordegraaf, M., & Bekkers, V. (2020) · Public Administration Review · doi:10.1111/puar.13211 · Cited by 75 · see citing works
Company Characteristics, Disability Inclusion Practices, and Employment of People with Disabilities in the Post COVID-19 Job Economy: A Cross Sectional Survey Study
Chan, F. H., Tansey, T. N., Iwanaga, K., Bezyak, J., Wehman, P., Phillips, B. N., Strauser, D. R., & Anderson, C. A. (2020) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-020-09941-8 · Cited by 68 · see citing works
Knowledge hide and seek: Role of ethical leadership, self-enhancement and job-involvement
Agarwal, U. A., Gupta, M., & Cooke, F. L. (2021) · Journal of Business Research · doi:10.1016/j.jbusres.2021.11.074 · Cited by 72 · see citing works
Fake news on Facebook: examining the impact of heuristic cues on perceived credibility and sharing intention
Ali, K. I., Li, C., Zain-ul-abdin, K., & Zaffar, M. A. (2021) · Internet Research · doi:10.1108/intr-10-2019-0442 · Cited by 66 · see citing works
Trust, distrust, and crowdfunding: A study on perceptions of institutional mechanisms
Strohmaier, D., Zeng, J., & Hafeez, M. (2019) · Telematics and Informatics · doi:10.1016/j.tele.2019.101252 · Cited by 65 · see citing works
Kratom use as more than a “self-treatment”
Smith, K. E., Dunn, K. E., Rogers, J. M., Grundmann, O., McCurdy, C. R., Garcia‐Romeu, A., Schriefer, D., Swogger, M. T., & Epstein, D. H. (2022) · The American Journal of Drug and Alcohol Abuse · doi:10.1080/00952990.2022.2083967 · Cited by 65 · see citing works
Reddit in communication research: current status, future directions and best practices
Hintz, E. A., & Betts, T. (2022) · Annals of the International Communication Association · doi:10.1080/23808985.2022.2064325 · Cited by 67 · see citing works
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 68 · see citing works
Consumer-brand identity and online advertising message elaboration
Graham, K. W., & Wilder, K. M. (2020) · Journal of Research in Interactive Marketing · doi:10.1108/jrim-01-2019-0011 · Cited by 61 · see citing works
The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems
Ziems, C., Yu, J., Wang, Y., Halevy, A. Y., & Yang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.261 · Cited by 73 · see citing works
Measuring Cumulative Stressfulness: Psychometric Properties of the COVID-19 Stressors Scale
Tambling, R. R., Russell, B. S., Park, C. L., Fendrich, M., Hutchinson, M., Horton, A. L., & Tomkunas, A. J. (2020) · Health Education & Behavior · doi:10.1177/1090198120979912 · Cited by 58 · see citing works
Investigating consumers’ adoption of AI chatbots for apparel shopping
Myin, M. T. A., & Watchravesringkan, K. (2024) · Journal of Consumer Marketing · doi:10.1108/jcm-03-2022-5234 · Cited by 43 · see citing works
Go WILD, Not WEIRD
Newson, M., Buhrmester, M. D., Xygalatas, D., & Whitehouse, H. (2020) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.38413 · Cited by 48 · see citing works
Searching for a Signal: Self-Reported Kratom Dose-Effect Relationships Among a Sample of US Adults With Regular Kratom Use Histories
Smith, K. E., Rogers, J. M., Dunn, K. E., Grundmann, O., McCurdy, C. R., Schriefer, D., & Epstein, D. H. (2022) · Frontiers in Pharmacology · doi:10.3389/fphar.2022.765917 · Cited by 54 · see citing works
Persuasive impact of loss and gain frames on intentions to exercise: A test of six moderators
Jensen, J. D., Ratcliff, C. L., Yale, R. N., Krakow, M., Scherr, C. L., & Yeo, S. K. (2017) · Communication Monographs · doi:10.1080/03637751.2017.1353699 · Cited by 54 · see citing works
Memorable Messages as Anticipatory Resilience: Examining Associations among Memorable Messages, Communication Resilience Processes, and Mental Health
Kuang, K., Tian, Z., Wilson, S. R., & Buzzanell, P. M. (2021) · Health Communication · doi:10.1080/10410236.2021.1993585 · Cited by 53 · see citing works
Conducting organizational survey and experimental research online: From convenient to ambitious in study designs, recruiting, and data quality
Quaquebeke, N. V., Salem, M., van Dijke, M., & Wenzel, R. (2022) · Organizational Psychology Review · doi:10.1177/20413866221097571 · Cited by 63 · see citing works
Awareness and perception of artificial intelligence operationalized integration in news media industry and society
Owsley, C. S., & Greenwood, K. (2022) · AI & Society · doi:10.1007/s00146-022-01386-2 · Cited by 52 · see citing works
Let it flow: the role of seamlessness and the optimal experience on consumer word of mouth in omnichannel marketing
Rodríguez–Torrico, P., Cabezudo, R. S. J., Gutiérrez, S. S. M., & Trabold, L. (2021) · Journal of Research in Interactive Marketing · doi:10.1108/jrim-06-2021-0154 · Cited by 54 · see citing works
Are you a cyberbully on social media? Exploring the personality traits using a fuzzy-set configurational approach
Hossain, M. A., Quaddus, M., Warren, M. J., Akter, S., & Pappas, I. O. (2022) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2022.102537 · Cited by 49 · see citing works
Growing evidence for separate neural mechanisms for attention and consciousness
Maier, A., & Tsuchiya, N. (2020) · Attention Perception & Psychophysics · doi:10.3758/s13414-020-02146-4 · Cited by 49 · see citing works
Resisting Covert Persuasion in Digital News: Comparing Inoculation and Reactance in the Processing of Native Advertising Disclosures and in Article Engagement Intentions
Amazeen, M. A. (2020) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699020952131 · Cited by 30 · see citing works
App Designs and Interactive Features to Increase mHealth Adoption: User Expectation Survey and Experiment
Lazard, A. J., Brennen, J. S., & Belina, S. P. (2021) · JMIR mhealth and uhealth · doi:10.2196/29815 · Cited by 48 · see citing works
Cues for Increasing Social Presence for Mobile Health App Adoption
Lazard, A. J., Brennen, J. S., Adams, E. T., & Love, B. (2020) · Journal of Health Communication · doi:10.1080/10810730.2020.1719241 · Cited by 45 · see citing works
Modelling determinants of walking and cycling adoption: A stage-of-change perspective
Biehl, A., Ermagun, A., & Stathopoulos, A. (2018) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2018.06.016 · Cited by 43 · see citing works
Linking high-performance work systems to affective commitment, job satisfaction, and career satisfaction: Thriving as a mediator and wasta as a moderator
Alothmany, R. D., Jiang, Z., & Manoharan, A. (2022) · The International Journal of Human Resource Management · doi:10.1080/09585192.2022.2157681 · Cited by 46 · see citing works
An Investigation of Misinformation Harms Related to Social Media During Humanitarian Crises
Tran, T., Valecha, R., Rad, P., & Rao, H. R. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-3817-9_10 · Cited by 42 · see citing works
Why does listening matter inside the organization? The impact of internal listening on employee-organization relationships
Qin, Y. S., & Men, L. R. (2021) · Journal of Public Relations Research · doi:10.1080/1062726x.2022.2034631 · Cited by 55 · see citing works
Dueling emergencies: Flood evacuation ridesharing during the COVID-19 pandemic
Borowski, E. C., Cedillo, V. L., & Stathopoulos, A. (2021) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2021.100352 · Cited by 40 · see citing works
Measurement of Adverse Childhood Experiences: It Matters
Reidy, D. E., Niolon, P. H., Estefan, L. F., Kearns, M. C., D’Inverno, A. S., Marker, C. D., & Merrick, M. T. (2021) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2021.05.043 · Cited by 40 · see citing works
Understanding the Influence of Authentic Leadership and Employee-Organization Relationships on Employee Voice Behaviors in Response to Dissatisfying Events at Work
Kim, Y., Lee, E., Kang, M., & Yang, S. (2022) · Management Communication Quarterly · doi:10.1177/08933189221085562 · Cited by 44 · see citing works
Effects of Fact-Checking Political Misinformation on Perceptual Accuracy and Epistemic Political Efficacy
York, C., Ponder, J. D., Humphries, Z., Goodall, C. E., Beam, M. A., & Winters, C. (2019) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699019890119 · Cited by 39 · see citing works
Inducing Positive Perspectives with Text Reframing
Ziems, C., Li, M., Zhang, A. L., & Yang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.257 · Cited by 46 · see citing works
The effects of leadership in corporate social advocacy on positive employee outcomes
Yang, S., Kang, M., Kim, Y., & Lee, E. (2022) · Journal of Public Relations Research · doi:10.1080/1062726x.2022.2123331 · Cited by 39 · see citing works
Exploring the characteristics of crowdsourcing: An online observational study
Bassi, H., Lee, C. J., Misener, L., & Johnson, A. M. (2019) · Journal of Information Science · doi:10.1177/0165551519828626 · Cited by 39 · see citing works
Conducting Unsupervised Virtual Reality User Studies Online
Mottelson, A., Petersen, G. B., Lilija, K., & Makransky, G. (2021) · Frontiers in Virtual Reality · doi:10.3389/frvir.2021.681482 · Cited by 39 · see citing works
The State of Assessment in Human-Animal Interaction Research
Rodriguez, K. E., Guérin, N. A., Gabriels, R. L., Serpell, J., Schreiner, P., & O’Haire, M. (2018) · Human-animal interaction bulletin · doi:10.1079/hai.2018.0022 · Cited by 39 · see citing works
An Investigation of Misinformation Harms Related to Social Media during Two Humanitarian Crises
Tran, T., Valecha, R., Rad, P., & Rao, H. (2019) · Information Systems Frontiers · doi:10.1007/s10796-020-10088-3 · Cited by 66 · see citing works
Automatically Detecting Image–Text Mismatch on Instagram with Deep Learning
Ha, Y.-I., Park, K., Kim, S. J., Joo, J., & Cha, M. (2021) · Journal of Advertising · doi:10.1080/00913367.2020.1843091 · Cited by 40 · see citing works
Frustration and ennui among Amazon MTurk workers
Fowler, C., Jiao, J., & Pitts, M. J. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01955-9 · Cited by 36 · see citing works
Situational Judgment Tests as a method for measuring personality: Development and validity evidence for a test of Dependability
Olaru, G., Burrus, J. T., MacCann, C., Zaromb, F. M., Wilhelm, O., & Roberts, R. D. (2019) · PLoS ONE · doi:10.1371/journal.pone.0211884 · Cited by 36 · see citing works
News in an Era of Content Confusion: Effects of News Use Motivations and Context on Native Advertising and Digital News Perceptions
Amazeen, M. A. (2019) · Journalism & Mass Communication Quarterly · doi:10.1177/1077699019886589 · Cited by 36 · see citing works
Policy Alienation and Street-level Bureaucrats’ Psychological Wellbeing: The Mediating Role of Alienative Commitment
Usman, M., Ali, M., Mughal, F., & Agyemang‐Mintah, P. (2020) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muaa043 · Cited by 36 · see citing works
Short‐term impacts of COVID‐19 on family caregivers: Emotion regulation, coping, and mental health
Russell, B. S., Hutchison, M., Park, C. L., Fendrich, M., & Finkelstein‐Fox, L. (2021) · Journal of Clinical Psychology · doi:10.1002/jclp.23228 · Cited by 34 · see citing works
Understanding crowdsourcing in science
Lenart-Gansiniec, R. A., Czakon, W., Sułkowski, Ł., & Poček, J. (2022) · Review of Managerial Science · doi:10.1007/s11846-022-00602-z · Cited by 36 · see citing works
Affection substitution: The effect of pornography consumption on close relationships
Hesse, C., & Floyd, K. (2019) · Journal of Social and Personal Relationships · doi:10.1177/0265407519841719 · Cited by 35 · see citing works
Motivational Differences Among Viewers of Traditional Sports, Esports, and NBA 2K League
Rogers, R. P., Farquhar, L., & Mummert, J. (2020) · Communication & Sport · doi:10.1177/2167479520942738 · Cited by 37 · see citing works
Hope Mediates the Relation between Income and Subjective Well-Being
Pleeging, E., Burger, M., & van Exel, J. N. A. (2020) · Journal of Happiness Studies · doi:10.1007/s10902-020-00309-6 · Cited by 35 · see citing works
Us and Them (and It): Social Orientation, Privacy Concerns, and Expected Use of Pandemic-Tracking Apps in the United States
Seberger, J. S., & Patil, S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445485 · Cited by 33 · see citing works
Online Data Collection in Auditory Perception and Cognition Research: Recruitment, Testing, Data Quality and Ethical Considerations
Eerola, T., Armitage, J. O., Lavan, N., & Knight, S. (2021) · Auditory Perception & Cognition · doi:10.1080/25742442.2021.2007718 · Cited by 34 · see citing works
Audience response to endemic and non-endemic sponsors of esports events
Rogers, R. P., Farquhar, L., & Mummert, J. (2020) · International Journal of Sports Marketing and Sponsorship · doi:10.1108/ijsms-09-2019-0107 · Cited by 33 · see citing works
Privacy Management and Health Information Sharing via Contact Tracing during the COVID-19 Pandemic: A Hypothetical Study on AI-Based Technologies
Hong, S. J., & Cho, H. (2021) · Health Communication · doi:10.1080/10410236.2021.1981565 · Cited by 32 · see citing works
Perceptions of and stigma toward BDSM practitioners
Brown, A. A., & Jefferson, S. E. (2022) · Current Psychology · doi:10.1007/s12144-022-03112-z · Cited by 31 · see citing works
Debunking misinformation in times of crisis: Exploring misinformation correction strategies for effective internal crisis communication
Kim, Y., & Lim, H. D. (2022) · Journal of Contingencies and Crisis Management · doi:10.1111/1468-5973.12447 · Cited by 30 · see citing works
Temporal patterns of owner-pet relationship, stress, and loneliness during the COVID-19 pandemic, and the effect of pet ownership on mental health: A longitudinal survey
Ogata, N., Weng, H., & Messam, L. L. M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0284101 · Cited by 30 · see citing works
The Use of Online Support Groups to Seek Information about Chronic Illness: Applying the Theory of Motivated Information Management
Kanter, E., Bevan, J. L., & Dorros, S. M. (2018) · Communication Quarterly · doi:10.1080/01463373.2018.1539406 · Cited by 29 · see citing works
Framing Syrian refugees: Examining news framing effects on attitudes toward refugee admissions and anti-immigrant sentiment
Liu, J. (2022) · Newspaper Research Journal · doi:10.1177/07395329221077251 · Cited by 29 · see citing works
The Structuration of Identification on Organizational Members’ Social Media
Piercy, C. W., & Carr, C. T. (2020) · International Journal of Business Communication · doi:10.1177/2329488420955215 · Cited by 28 · see citing works
Patterns of Communicating About Family Health History: Exploring Differences in Family Types, Age, and Sex
Campbell‐Salome, G., Rauscher, E. A., & Freytag, J. (2019) · Health Education & Behavior · doi:10.1177/1090198119853002 · Cited by 28 · see citing works
Online Privacy Fatigue: A Scoping Review and Research Agenda
van der Schyff, K., Foster, G. G., Renaud, K., & Flowerday, S. (2023) · Future Internet · doi:10.3390/fi15050164 · Cited by 28 · see citing works
Why do you use Yelp? Analysis of factors influencing customers’ website adoption and dining behavior
Salehi-Esfahani, S., & Kang, J. (2019) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2018.12.002 · Cited by 27 · see citing works
Internal Consent, Affirmative External Consent, and Sexual Satisfaction Among Young Adults
Javidi, H., Widman, L. M., Evans-Paulson, R., & Lipsey, N. P. (2022) · The Journal of Sex Research · doi:10.1080/00224499.2022.2048628 · Cited by 27 · see citing works
Parents’ attitudes towards the content of sex education in the USA: Associations with religiosity and political orientation
Hurst, J. L., Widman, L. M., Brasileiro, J., Maheux, A. J., Evans-Paulson, R., & Choukas‐Bradley, S. (2023) · Sex Education · doi:10.1080/14681811.2022.2162871 · Cited by 29 · see citing works
Executive Dysfunction and Emotion Dysregulation Explain the Effects of Insomnia Symptoms on Repetitive Negative Thinking
Cox, R. C., Jessup, S. C., & Olatunji, B. O. (2019) · Cognitive Therapy and Research · doi:10.1007/s10608-019-10033-y · Cited by 25 · see citing works
OurPlaces: Cross-Cultural Crowdsourcing Platform for Location Recommendation Services
Nguyen, L. V., Jung, J. J., & Hwang, M. (2020) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi9120711 · Cited by 24 · see citing works
Misinformation Harms: A Tale of Two Humanitarian Crises
Tran, T., Valecha, R., Rad, P., & Rao, H. (2020) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2020.3029685 · Cited by 24 · see citing works
Predictors of Mental Health Help-Seeking During COVID-19: Social Support, Emotion Regulation, and Mental Health Symptoms
Tambling, R. R., Russell, B. S., Fendrich, M., & Park, C. L. (2022) · The Journal of Behavioral Health Services & Research · doi:10.1007/s11414-022-09796-2 · Cited by 24 · see citing works
Why go the extra mile? Re-engaging quiet quitters through dialogic communication and community building in the workplace
Uysal, N., & Kim, Y. (2025) · Public Relations Review · doi:10.1016/j.pubrev.2024.102537 · Cited by 23 · see citing works
Associations of Alcohol, Marijuana, and Polysubstance use with Non-Adherence to Covid-19 Public Health Guidelines in a us Sample
Fendrich, M., Becker, J., Park, C. L., Russell, B. S., Finkelstein‐Fox, L., & Hutchison, M. (2021) · Substance Abuse · doi:10.1080/08897077.2021.1891603 · Cited by 23 · see citing works
Using the HEXACO-100 to measure Individual Entrepreneurial Orientation: Introducing the HEXACO-IEO
Howard, M. C. (2020) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2020.e00163 · Cited by 23 · see citing works

An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 768 · see citing works
Crowdsourcing in observational and experimental research
Zallot, C., Paolacci, G., Chandler, J., & Sisso, I. (2021) · Routledge eBooks · doi:10.4324/9781003025245-12 · Cited by 8 · see citing works
A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk)
Çobanoğlu, C., Çavuşoğlu, M., & Türktarhan, G. (2021) · Journal of Global Business Insights · doi:10.5038/2640-6489.6.1.1177 · Cited by 90 · see citing works
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M. P., & Shao, B. (2020) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 307 · see citing works
Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk
Shank, D. B. (2015) · The American Sociologist · doi:10.1007/s12108-015-9266-9 · Cited by 176 · see citing works

Authors: Sheehan, K. B.
Affiliations: University of Oregon (US)
Corresponding author: Sheehan, Kim Bartel
Venue: Communication Monographs
Publisher: Informa UK Limited
Published: 2017-07-04
Volume: 85
Issue: 1
Pages: 140-156
ISSN: 0363-7751 (ISSN-L)
1479-5787
DOI: 10.1080/03637751.2017.1342043
OpenAlex: W2724582867
Cited by: 408
References: 46
Citations by year: 2026: 13 · 2025: 33 · 2024: 39 · 2023: 43 · 2022: 78 · 2021: 73 · 2020: 64 · 2019: 49 · 2018: 12 · 2017: 2
FWCI: 41.62
Type: journal article
Language: en
Open access: closed
Locations: Communication Monographs (publishedVersion)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords: Amazon Mechanical Turk, crowdsourcing, online data collection, validity, reliability statistics, research ethics, cost efficiency, online panels

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks