The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Open accessAbstract
Data quality is one of the major concerns of using crowdsourcing websites such as Amazon Mechanical Turk (MTurk) to recruit participants for online behavioral studies. We compared two methods for ensuring data quality on MTurk: attention check questions (ACQs) and restricting participation to MTurk workers with high reputation (above 95% approval ratings). In Experiment 1, we found that high-reputation workers rarely failed ACQs and provided higher-quality data than did low-reputation workers; ACQs improved data quality only for low-reputation workers, and only in some cases. Experiment 2 corroborated these findings and also showed that more productive high-reputation workers produce the highest-quality data. We concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Data obtained from MTurk has been demonstrated to be valid and reliable when appropriate selection criteria and attention checks are used (Casler, Bickel, & Hackett, 2013; Peer, Vosgerau, & Acquisti, 2014), as was the case in this study.”
From The development and evaluation of a brief form of the Normative Male Alexithymia Scale (NMAS-BF). (2019) · cites this paper as Peer, Vosgerau, & Acquisti, 2014
The cited source says what the claim attributes to it.
In the citing paper
“Most of the sites we reviewed have supervision and reputation mechanisms designed to measure the quality of submitted work, and to allow researchers to invite workers with high reputation (similar to MTurk's”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
In the citing paper
“Amazon Mechanical Turk is an online sample recruiting platform that, although not nationally representative, yields high data quality through the use of attention checks 44 and worker reputation scores 45 .”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [45]
In the citing paper
“Following prior research, we selected participants with an approval rating above 95% on Amazon Mechanical Turk 45 .”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [45]
In the citing paper
“which showed that ACQs have low diagnostic ability when data quality is already high.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., (2014)
In the citing paper
“they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
In the citing paper
“they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
In the citing paper
“Using two attention-check questions (Peer et al., 2014), we checked whether participants read and paid attention to our instructions.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201272002 · Cited by 3,041
The Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Rosenberg, M. (2011) · PsycTESTS Dataset · doi:10.1037/t01038-000 · Cited by 1,226
Cited by
Sorted by citations
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Self-citationInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 2,122
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2020) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 1,098
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 815
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Marroquín, B. M., Vine, V., & Morgan, R. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113419 · Cited by 667
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597
Boas, T. C., Christenson, D. P., & Glick, D. M. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2018.28 · Cited by 570
Solution aversion: On the relation between ideology and motivated disbelief.
Campbell, T. H., & Kay, A. C. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037963 · Cited by 560
Instagram photos reveal predictive markers of depression
Reece, A. G., & Danforth, C. M. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0110-z · Cited by 519
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Are Manipulation Checks Necessary?
Hauser, D., Ellsworth, P. C., & Gonzalez, R. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00998 · Cited by 562
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474
Headphone screening to facilitate web-based auditory experiments
Woods, K. J. P., Siegel, M., Traer, J. A., & McDermott, J. H. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-017-1361-2 · Cited by 471
Forecasting the onset and course of mental illness with Twitter data
Reece, A. G., Reagan, A. J., Lix, K. L. M., Dodds, P. S., Danforth, C. M., & Langer, E. J. (2017) · Scientific Reports · doi:10.1038/s41598-017-12961-9 · Cited by 401
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2017) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 370
Shin, Y. H., Im, J., Jung, S. E., & Severt, K. (2018) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.10.011 · Cited by 358
Barbaro, N., & Pickett, S. M. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2015.05.026 · Cited by 350
Nikčević, A. V., Marino, C., Kolubinski, D. C., Leach, D., & Spada, M. M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.10.053 · Cited by 344
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 368
Understanding, explaining, and utilizing medical artificial intelligence
Cadario, R., Longoni, C., & Morewedge, C. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01146-0 · Cited by 342
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Survey-software implicit association tests: A methodological and empirical analysis
Carpenter, T. P., Pogacar, R., Pullig, C., Kouril, M., Aguilar, S. J., LaBouff, J. P., Isenberg, N., & Chakroff, A. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01293-3 · Cited by 342
Attitudes towards AI: measurement and associations with personality
Stein, J.-P., Messingschlager, T., Gnambs, T., Hutmacher, F., & Appel, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-53335-2 · Cited by 309
“With our data, we observed a very good Cronbach’s α of 0.86 for the CMQ.”
MethodInfluentialMTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2016) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 279
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies
Gaddis, S. (2017) · Sociological Science · doi:10.15195/v4.a19 · Cited by 285
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 279
A Note on Dropping Experimental Subjects who Fail a Manipulation Check
Aronow, P. M., Baron, J., & Pinson, L. E. (2019) · Political Analysis · doi:10.1017/pan.2019.5 · Cited by 280
Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 299
I Just Can’t Control Myself: A Self-Regulation Perspective on the Abuse of Deviant Employees
Mawritz, M. B., Greenbaum, R. L., Butts, M. M., & Graham, K. A. (2016) · Academy of Management Journal · doi:10.5465/amj.2014.0409 · Cited by 281
Ferrer, R. A., Klein, W. M. P., Persoskie, A., Avishai, A., & Sheeran, P. (2016) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9790-z · Cited by 262
Egelman, S., & Péer, E. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702249 · Cited by 303
The use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2018) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 258
Shin, Y. H., Moon, H. G., Jung, S. E., & Severt, K. (2017) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2017.10.010 · Cited by 259
Understanding Chinese consumer adoption of apparel mobile commerce: An extended TAM approach
Chi, T. (2018) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2018.07.019 · Cited by 271
Reconceptualizing Integration Quality Dynamics for Omnichannel Marketing
Hossain, T. M. T., Akter, S., Kattiyapornpong, U., & Dwivedi, Y. K. (2020) · Industrial Marketing Management · doi:10.1016/j.indmarman.2019.12.006 · Cited by 282
An innovation resistance theory perspective on purchase of eco-friendly cosmetics
Sadiq, M., Adil, M., & Paul, J. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2020.102369 · Cited by 262
Molyneux, L. (2017) · Digital Journalism · doi:10.1080/21670811.2017.1334567 · Cited by 256
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 233
Watch Your Tone: How a Brand's Tone of Voice on Social Media Influences Consumer Responses
Barcelos, R. H., Dantas, D. C., & Sénécal, S. (2017) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2017.10.001 · Cited by 257
Bardeen, J. R., & Fergus, T. A. (2016) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2016.02.002 · Cited by 213
Better together? Examining profiles of employee recovery experiences
Bennett, A. A., Gabriel, A. S., Calderwood, C., Dahling, J. J., & Trougakos, J. P. (2016) · Journal of Applied Psychology · doi:10.1037/apl0000157 · Cited by 219
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203
The effect of credit card versus mobile payment on convenience and consumers’ willingness to pay
Boden, J., Maier, E., & Wilken, R. (2019) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2019.101910 · Cited by 204
Kim, M., & Beehr, T. A. (2017) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051817727702 · Cited by 227
Validating Machine Learning Algorithms for Twitter Data Against Established Measures of Suicidality
Braithwaite, S., Giraud-Carrier, C. G., West, J. H., Barnes, M. D., & Hanson, C. L. (2016) · JMIR Mental Health · doi:10.2196/mental.4822 · Cited by 200
Redmiles, E. M., Kross, S., & Mazurek, M. L. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00014 · Cited by 193
The role of brand reputation in organic food consumption: A behavioral reasoning perspective
Ryan, J., & Casidy, R. (2018) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2018.01.002 · Cited by 205
The COVID-19 anxiety syndrome scale: development and psychometric properties.
Nikčević, A. V., & Spada, M. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113322 · Cited by 186
Moravec, P. L., Kim, A., & Dennis, A. R. (2018) · Information Systems Research · doi:10.1287/isre.2020.0927 · Cited by 185
The Role of MTurk in Education Research: Advantages, Issues, and Future Directions
Follmer, D. J., Sperling, R. A., & Suen, H. K. (2017) · Educational Researcher · doi:10.3102/0013189x17725519 · Cited by 184
Stubb, C., & Colliander, J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.04.024 · Cited by 216
Eco-friendly hotel stay and environmental attitude: A value-attitude-behaviour perspective
Sadiq, M., Adil, M., & Paul, J. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.103094 · Cited by 184
Why are conservatives more punitive than liberals? A moral foundations approach.
Silver, J. R., & Silver, E. (2017) · Law and Human Behavior · doi:10.1037/lhb0000232 · Cited by 193
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000191 · Cited by 179
Stubb, C., Nyström, A., & Colliander, J. (2019) · Journal of Communication Management · doi:10.1108/jcom-11-2018-0119 · Cited by 196
Lukasik, K. M., Waris, O., Soveri, A., Lehtonen, M., & Laine, M. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00004 · Cited by 177
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D., & Schwarz, N. (2015) · SAGE Open · doi:10.1177/2158244015584617 · Cited by 173
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168
Measuring patient-perceived quality of care in US hospitals using Twitter
Hawkins, J. B., Brownstein, J. S., Tuli, G., Runels, T., Broecker, K., Nsoesie, E. O., McIver, D. J., Rozenblum, R., Wright, A. T., Bourgeois, F. T., et al. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004309 · Cited by 168
Kim, M., & Beehr, T. A. (2018) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051818762337 · Cited by 184
Mechanical Turk data collection in addiction research: utility, concerns and best practices
Mellis, A. M., & Bickel, W. K. (2020) · Addiction · doi:10.1111/add.15032 · Cited by 175
Zickar, M. J., & Keith, M. G. (2022) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-120920-052946 · Cited by 231
Fergus, T. A. (2014) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.05.006 · Cited by 159
Cyberchondria: examining relations with problematic internet use and metacognitive beliefs
Fergus, T. A., & Spada, M. M. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2102 · Cited by 162
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157732 · Cited by 183
Su, W., Lyu, B., Chen, H., & Zhang, Y. (2020) · Baltic Journal of Management · doi:10.1108/bjm-09-2019-0335 · Cited by 181
The role of a “common is moral” heuristic in the stability and change of moral norms.
Lindström, B., Jangard, S., Selbing, I., & Olsson, A. (2017) · Journal of Experimental Psychology General · doi:10.1037/xge0000365 · Cited by 162
Beyond The Privacy Paradox: Objective Versus Relative Risk in Privacy Decision Making1
Adjerid, I., Péer, E., & Acquisti, A. (2016) · MIS Quarterly · doi:10.25300/misq/2018/14316 · Cited by 182
Self-citationSewall, C. J. R., Bear, T. M., Merranko, J., & Rosen, D. (2020) · Mobile Media & Communication · doi:10.1177/2050157920902830 · Cited by 139
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166
Common Concerns with MTurk as a Participant Pool: Evidence and SolutionsPreprint
Hauser, D., Paolacci, G., & Chandler, J. (2018) · doi:10.31234/osf.io/uq45c · Cited by 162
Reconsidering the voice effect when learning from a virtual human
Craig, S. D., & Schroeder, N. L. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.003 · Cited by 151
Nudge Me Right: Personalizing Online Security Nudges to People's Decision-Making Styles
Péer, E., Egelman, S., Harbach, M., Malkin, N., Mathur, A., & Frik, A. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106347 · Cited by 148
Extended Reality (XR) Remote Research: a Survey of Drawbacks and Opportunities
Ratcliffe, J., Soave, F., Bryan–Kinns, N., Tokarchuk, L. N., & Farkhatdinov, I. (2021) · University of the Arts London Research Online (University of the Arts London) · Cited by 145
Model-based exploration of the frontier of behaviours for deep learning system testing
Riccio, V., & Tonella, P. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409730 · Cited by 152
Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Robertson, R. E., Lazer, D. M. J., & Wilson, C. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186143 · Cited by 151
Coelho, F., Bairrada, C. M., & Coelho, A. (2019) · Psychology and Marketing · doi:10.1002/mar.21279 · Cited by 139
When Victims Help Their Abusive Supervisors: The Role of LMX, Self-Blame, and Guilt
Tröster, C., & Quaquebeke, N. V. (2020) · Academy of Management Journal · doi:10.5465/amj.2019.0559 · Cited by 143
Lee, K. Y., Sheehan, L., Lee, K. K., & Chang, Y. (2021) · Internet Research · doi:10.1108/intr-06-2020-0327 · Cited by 140
Using MTurk to Distribute a Survey or Experiment: Methodological Considerations
Hunt, N. C., & Scheetz, A. M. (2018) · Journal of Information Systems · doi:10.2308/isys-52021 · Cited by 141
Distinct Effects of Pride and Gratitude Appeals on Sustainable Luxury Brands
Septianto, F., Seo, Y., & Errmann, A. (2020) · Journal of Business Ethics · doi:10.1007/s10551-020-04484-7 · Cited by 144
Thompson, A. J., & Pickett, J. T. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09436-7 · Cited by 132
Scaling Up: Representing Gender Diversity in Survey Research
Magliozzi, D., Saperstein, A., & Westbrook, L. (2016) · Socius: Sociological Research for a Dynamic World · doi:10.1177/2378023116664352 · Cited by 146
Desai, S. C., & Reimers, S. J. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1066-z · Cited by 166
Running Behavioral Operations Experiments Using Amazon's Mechanical Turk
Lee, Y. S., Seo, Y. W., & Siemsen, E. (2018) · Production and Operations Management · doi:10.1111/poms.12841 · Cited by 128
Hu, Y., Xu, A., Hong, Y., Gal, D., Sinha, V. S., & Akkiraju, R. (2018) · Journal of Management Information Systems · doi:10.1080/07421222.2019.1628908 · Cited by 126
Self-perceived food addiction: Prevalence, predictors, and prognosis
Meadows, A., Nolan, L. J., & Higgs, S. (2017) · Appetite · doi:10.1016/j.appet.2017.03.051 · Cited by 119
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 132
Gottlieb, S., Keltner, D. J., & Lombrozo, T. (2018) · Cognitive Science · doi:10.1111/cogs.12648 · Cited by 138
Ghosting and destiny: Implicit theories of relationships predict beliefs about ghosting
Freedman, G., Powell, D. N., Le, B., & Williams, K. D. (2019) · Journal of Social and Personal Relationships · doi:10.1177/0265407517748791 · Cited by 120
Participant carelessness and fraud: Consequences for clinical research and potential solutions.
Chandler, J., Sisso, I., & Shapiro, D. N. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000479 · Cited by 124
Metadata
- Authors
- Peer, E., Vosgerau, J., & Acquisti, A.
- Affiliations
- Tilburg University (NL)
- Carnegie Mellon University (US)
- Venue
- Behavior Research Methods
- Published
- 2013-12-20
- Volume
- 46
- Issue
- 4
- Pages
- 1023-1031
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2105753174
- Cited by
- 1,618
- References
- 18
- Citations by year
- 2026: 68 · 2025: 96 · 2024: 128 · 2023: 167 · 2022: 175 · 2021: 212 · 2020: 210 · 2019: 147 · 2018: 119 · 2017: 132 · 2016: 91 · 2015: 49 · 2014: 17 · 2013: 1
- FWCI
- 79.80
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Behavior Research Methods (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- SSRN Electronic Journal (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- KiltHub Repository
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- data quality, Amazon Mechanical Turk, worker reputation, crowdsourcing, selection bias, labor productivity, participant recruitment
- MeSH terms
- Data Collection, Humans, Research Design, Patient Selection, Internet, Behavioral Research, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
