Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Open accessAbstract
Data quality is one of the major concerns of using crowdsourcing websites such as Amazon Mechanical Turk (MTurk) to recruit participants for online behavioral studies. We compared two methods for ensuring data quality on MTurk: attention check questions (ACQs) and restricting participation to MTurk workers with high reputation (above 95% approval ratings). In Experiment 1, we found that high-reputation workers rarely failed ACQs and provided higher-quality data than did low-reputation workers; ACQs improved data quality only for low-reputation workers, and only in some cases. Experiment 2 corroborated these findings and also showed that more productive high-reputation workers produce the highest-quality data. We concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MTurk data are valid and reliable when appropriate selection criteria and attention checks are used.
“Data obtained from MTurk has been demonstrated to be valid and reliable when appropriate selection criteria and attention checks are used (Casler, Bickel, & Hackett, 2013; Peer, Vosgerau, & Acquisti, 2014), as was the case in this study.”
From The development and evaluation of a brief form of the Normative Male Alexithymia Scale (NMAS-BF). (2019) · cites this paper as Peer, Vosgerau, & Acquisti, 2014
Amazon Mechanical Turk is not nationally representative but yields high data quality through attention checks and worker reputation scores.
“Amazon Mechanical Turk is an online sample recruiting platform that, although not nationally representative, yields high data quality through the use of attention checks 44 and worker reputation scores 45 .”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [45]
In studies 1-3B, participants were recruited from Lucid and Amazon Mechanical Turk, and MTurk participants were selected with an approval rating above 95%.
“Following prior research, we selected participants with an approval rating above 95% on Amazon Mechanical Turk 45 .”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [45]
Most of the reviewed sites had supervision and reputation mechanisms designed to measure the quality of submitted work and to allow researchers to invite workers with high reputation.
“Most of the sites we reviewed have supervision and reputation mechanisms designed to measure the quality of submitted work, and to allow researchers to invite workers with high reputation (similar to MTurk's”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
ACQs have low diagnostic ability when data quality is already high.
“which showed that ACQs have low diagnostic ability when data quality is already high.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., (2014)
The scales have been previously used with success to measure data quality on MTurk.
“they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
The scales used (IUIPC, NFC, RSES) are reliable and validated.
“they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
Attention-check questions from Peer et al. (2014) can be used to check whether participants read and paid attention to instructions.
“Using two attention-check questions (Peer et al., 2014), we checked whether participants read and paid attention to our instructions.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · see citing works
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996 · see citing works
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493 · see citing works
Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201272002 · Cited by 3,041 · see citing works
The Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Rosenberg, M. (2011) · PsycTESTS Dataset · doi:10.1037/t01038-000 · Cited by 1,226 · see citing works
Consequences of prejudice against the null hypothesis.
Greenwald, A. G. (1975) · Psychological Bulletin · doi:10.1037/h0076157 · Cited by 1,022 · cited 3× · see citing works
“Note that—as in Experiment 1—this conclusion relies on interpreting a null effect as meaningful, which is possible when samples are adequately powered (Greenwald, 1975).”
BackgroundInfluentialChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · see citing works
Measuring Social Desirability: Short Forms of the Marlowe-Crowne Social Desirability Scale
Fischer, D. G., & Fick, C. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053002011 · Cited by 580 · see citing works
Seriousness checks are useful to improve data validity in online research
Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0265-2 · Cited by 528 · see citing works
Are your participants gaming the system?
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 422 · see citing works
A K-Sample Significance Test for Independent Alpha Coefficients
Hakstian, A. R., & Whalen, T. E. (1976) · Psychometrika · doi:10.1007/bf02291840 · Cited by 256 · see citing works
Jordan, C. H. (2020) · Springer eBooks · doi:10.1007/978-3-319-24612-3_1155 · Cited by 76 · see citing works
Flagged first, then by citations
1 of the 18 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · see citing works
Self-citationInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136 · see citing works
TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2016) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 1,978 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718 · see citing works
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072 · see citing works
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · see citing works
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2020) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 942 · see citing works
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902 · see citing works
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 764 · see citing works
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668 · see citing works
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625 · see citing works
Marroquín, B. M., Vine, V., & Morgan, R. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113419 · Cited by 637 · see citing works
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 597 · see citing works
Boas, T. C., Christenson, D. P., & Glick, D. M. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2018.28 · Cited by 570 · see citing works
Instagram photos reveal predictive markers of depression
Reece, A. G., & Danforth, C. M. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0110-z · Cited by 519 · see citing works
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506 · see citing works
Are Manipulation Checks Necessary?
Hauser, D., Ellsworth, P. C., & Gonzalez, R. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00998 · Cited by 501 · see citing works
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474 · see citing works
Solution aversion: On the relation between ideology and motivated disbelief.
Campbell, T. H., & Kay, A. C. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0037963 · Cited by 560 · see citing works
Headphone screening to facilitate web-based auditory experiments
Woods, K. J. P., Siegel, M., Traer, J. A., & McDermott, J. H. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-017-1361-2 · Cited by 434 · see citing works
Forecasting the onset and course of mental illness with Twitter data
Reece, A. G., Reagan, A. J., Lix, K. L. M., Dodds, P. S., Danforth, C. M., & Langer, E. J. (2017) · Scientific Reports · doi:10.1038/s41598-017-12961-9 · Cited by 401 · see citing works
Conducting interactive experiments online
Arechar, A. A., Gächter, S., & Molleman, L. (2017) · Experimental Economics · doi:10.1007/s10683-017-9527-2 · Cited by 362 · see citing works
Shin, Y. H., Im, J., Jung, S. E., & Severt, K. (2018) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2017.10.011 · Cited by 358 · see citing works
Barbaro, N., & Pickett, S. M. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2015.05.026 · Cited by 350 · see citing works
Nikčević, A. V., Marino, C., Kolubinski, D. C., Leach, D., & Spada, M. M. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.10.053 · Cited by 344 · see citing works
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J., & Cho, T. S. (2018) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 346 · see citing works
Understanding, explaining, and utilizing medical artificial intelligence
Cadario, R., Longoni, C., & Morewedge, C. K. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01146-0 · Cited by 342 · see citing works
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319 · see citing works
Survey-software implicit association tests: A methodological and empirical analysis
Carpenter, T., Pogacar, R., Pullig, C., Kouril, M., Aguilar, S. J., LaBouff, J. P., Isenberg, N., & Chakroff, A. (2018) · Behavior Research Methods · doi:10.3758/s13428-019-01293-3 · Cited by 270 · see citing works
Attitudes towards AI: measurement and associations with personality
Stein, J.-P., Messingschlager, T., Gnambs, T., Hutmacher, F., & Appel, M. (2024) · Scientific Reports · doi:10.1038/s41598-024-53335-2 · Cited by 309 · see citing works
“With our data, we observed a very good Cronbach’s α of 0.86 for the CMQ.”
MethodInfluentialMTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330 · see citing works
Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2016) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 274 · see citing works
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies
Gaddis, S. M. (2017) · Sociological Science · doi:10.15195/v4.a19 · Cited by 307 · see citing works
Chandler, J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 279 · see citing works
A Note on Dropping Experimental Subjects who Fail a Manipulation Check
Aronow, P. M., Baron, J., & Pinson, L. E. (2019) · Political Analysis · doi:10.1017/pan.2019.5 · Cited by 280 · see citing works
Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 276 · see citing works
I Just Can’t Control Myself: A Self-Regulation Perspective on the Abuse of Deviant Employees
Mawritz, M. B., Greenbaum, R. L., Butts, M. M., & Graham, K. A. (2016) · Academy of Management Journal · doi:10.5465/amj.2014.0409 · Cited by 271 · see citing works
Ferrer, R. A., Klein, W. M. P., Persoskie, A., Avishai, A., & Sheeran, P. (2016) · Annals of Behavioral Medicine · doi:10.1007/s12160-016-9790-z · Cited by 262 · see citing works
Egelman, S., & Péer, E. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702249 · Cited by 259 · see citing works
The use of crowdsourcing in addiction science research: Amazon Mechanical Turk.
Strickland, J. C., & Stoops, W. W. (2018) · Experimental and Clinical Psychopharmacology · doi:10.1037/pha0000235 · Cited by 249 · see citing works
Shin, Y. H., Moon, H. G., Jung, S. E., & Severt, K. (2017) · Journal of Hospitality and Tourism Management · doi:10.1016/j.jhtm.2017.10.010 · Cited by 251 · see citing works
Understanding Chinese consumer adoption of apparel mobile commerce: An extended TAM approach
Chi, T. (2018) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2018.07.019 · Cited by 237 · see citing works
Reconceptualizing Integration Quality Dynamics for Omnichannel Marketing
Hossain, T. M. T., Akter, S., Kattiyapornpong, U., & Dwivedi, Y. K. (2020) · Industrial Marketing Management · doi:10.1016/j.indmarman.2019.12.006 · Cited by 259 · see citing works
An innovation resistance theory perspective on purchase of eco-friendly cosmetics
Sadiq, M., Adil, M., & Paul, J. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2020.102369 · Cited by 262 · see citing works
Molyneux, L. (2017) · Digital Journalism · doi:10.1080/21670811.2017.1334567 · Cited by 227 · see citing works
Crowdsourcing Samples in Cognitive Science
Stewart, N., Chandler, J., & Paolacci, G. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.06.007 · Cited by 216 · see citing works
Watch Your Tone: How a Brand's Tone of Voice on Social Media Influences Consumer Responses
Barcelos, R. H., Dantas, D. C., & Sénécal, S. (2017) · Journal of Interactive Marketing · doi:10.1016/j.intmar.2017.10.001 · Cited by 223 · see citing works
Bardeen, J. R., & Fergus, T. A. (2016) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2016.02.002 · Cited by 213 · see citing works
Better together? Examining profiles of employee recovery experiences
Bennett, A. A., Gabriel, A. S., Calderwood, C., Dahling, J. J., & Trougakos, J. P. (2016) · Journal of Applied Psychology · doi:10.1037/apl0000157 · Cited by 219 · see citing works
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019) · PLoS ONE · doi:10.1371/journal.pone.0226394 · Cited by 203 · see citing works
The effect of credit card versus mobile payment on convenience and consumers’ willingness to pay
Boden, J., Maier, E., & Wilken, R. (2019) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2019.101910 · Cited by 204 · see citing works
Kim, M., & Beehr, T. A. (2017) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051817727702 · Cited by 199 · see citing works
Validating Machine Learning Algorithms for Twitter Data Against Established Measures of Suicidality
Braithwaite, S., Giraud-Carrier, C. G., West, J. H., Barnes, M. D., & Hanson, C. L. (2016) · JMIR Mental Health · doi:10.2196/mental.4822 · Cited by 193 · see citing works
Redmiles, E. M., Kross, S., & Mazurek, M. L. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00014 · Cited by 193 · see citing works
The role of brand reputation in organic food consumption: A behavioral reasoning perspective
Ryan, J., & Casidy, R. (2018) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2018.01.002 · Cited by 191 · see citing works
The COVID-19 anxiety syndrome scale: development and psychometric properties.
Nikčević, A. V., & Spada, M. M. (2020) · Psychiatry Research · doi:10.1016/j.psychres.2020.113322 · Cited by 186 · see citing works
Moravec, P. L., Kim, A., & Dennis, A. R. (2018) · Information Systems Research · doi:10.1287/isre.2020.0927 · Cited by 185 · see citing works
The Role of MTurk in Education Research: Advantages, Issues, and Future Directions
Follmer, D. J., Sperling, R. A., & Suen, H. K. (2017) · Educational Researcher · doi:10.3102/0013189x17725519 · Cited by 184 · see citing works
Stubb, C., & Colliander, J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.04.024 · Cited by 188 · see citing works
Eco-friendly hotel stay and environmental attitude: A value-attitude-behaviour perspective
Sadiq, M., Adil, M., & Paul, J. (2021) · International Journal of Hospitality Management · doi:10.1016/j.ijhm.2021.103094 · Cited by 184 · see citing works
Why Are Conservatives More Punitive Than Liberals? A Moral Foundations Approach
Silver, J., & Silver, E. (2017) · Law and Human Behavior · doi:10.1037/lhb0000232 · Cited by 146 · see citing works
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017) · Personality Disorders Theory Research and Treatment · doi:10.1037/per0000191 · Cited by 179 · see citing works
Stubb, C., Nyström, A., & Colliander, J. (2019) · Journal of Communication Management · doi:10.1108/jcom-11-2018-0119 · Cited by 196 · see citing works
Lukasik, K. M., Waris, O., Soveri, A., Lehtonen, M., & Laine, M. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00004 · Cited by 177 · see citing works
It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Hauser, D., & Schwarz, N. (2015) · SAGE Open · doi:10.1177/2158244015584617 · Cited by 173 · see citing works
Brühlmann, F., Petralito, S., Aeschbach, L. F., & Opwis, K. (2020) · Methods in Psychology · doi:10.1016/j.metip.2020.100022 · Cited by 168 · see citing works
Measuring patient-perceived quality of care in US hospitals using Twitter
Hawkins, J. B., Brownstein, J. S., Tuli, G., Runels, T., Broecker, K., Nsoesie, E. O., McIver, D. J., Rozenblum, R., Wright, A. T., Bourgeois, F. T., et al. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004309 · Cited by 168 · see citing works
Kim, M., & Beehr, T. A. (2018) · Journal of Leadership & Organizational Studies · doi:10.1177/1548051818762337 · Cited by 166 · see citing works
Mechanical Turk data collection in addiction research: utility, concerns and best practices
Mellis, A. M., & Bickel, W. K. (2020) · Addiction · doi:10.1111/add.15032 · Cited by 164 · see citing works
Zickar, M. J., & Keith, M. G. (2022) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-120920-052946 · Cited by 163 · see citing works
Fergus, T. A. (2014) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2014.05.006 · Cited by 159 · see citing works
Cyberchondria: examining relations with problematic internet use and metacognitive beliefs
Fergus, T. A., & Spada, M. M. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2102 · Cited by 162 · see citing works
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161 · see citing works
Su, W., Lyu, B., Chen, H., & Zhang, Y. (2020) · Baltic Journal of Management · doi:10.1108/bjm-09-2019-0335 · Cited by 181 · see citing works
The role of a “common is moral” heuristic in the stability and change of moral norms.
Lindström, B., Jangard, S., Selbing, I., & Olsson, A. (2017) · Journal of Experimental Psychology General · doi:10.1037/xge0000365 · Cited by 150 · see citing works
Beyond The Privacy Paradox: Objective Versus Relative Risk in Privacy Decision Making1
Adjerid, I., Péer, E., & Acquisti, A. (2016) · MIS Quarterly · doi:10.25300/misq/2018/14316 · Cited by 165 · see citing works
Self-citationSewall, C. J. R., Bear, T. M., Merranko, J., & Rosen, D. (2020) · Mobile Media & Communication · doi:10.1177/2050157920902830 · Cited by 139 · see citing works
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166 · see citing works
Common Concerns with MTurk as a Participant Pool: Evidence and SolutionsPreprint
Hauser, D., Paolacci, G., & Chandler, J. (2018) · doi:10.31234/osf.io/uq45c · Cited by 156 · see citing works
Reconsidering the voice effect when learning from a virtual human
Craig, S. D., & Schroeder, N. L. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.07.003 · Cited by 151 · see citing works
Nudge Me Right: Personalizing Online Security Nudges to People's Decision-Making Styles
Péer, E., Egelman, S., Harbach, M., Malkin, N., Mathur, A., & Frik, A. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106347 · Cited by 148 · see citing works
Extended Reality (XR) Remote Research: a Survey of Drawbacks and Opportunities
Ratcliffe, J., Soave, F., Bryan–Kinns, N., Tokarchuk, L. N., & Farkhatdinov, I. (2021) · University of the Arts London Research Online (University of the Arts London) · Cited by 145 · see citing works
Model-based exploration of the frontier of behaviours for deep learning system testing
Riccio, V., & Tonella, P. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409730 · Cited by 141 · see citing works
Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Robertson, R. E., Lazer, D. M. J., & Wilson, C. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186143 · Cited by 136 · see citing works
Coelho, F., Bairrada, C. M., & Coelho, A. (2019) · Psychology and Marketing · doi:10.1002/mar.21279 · Cited by 139 · see citing works
When Victims Help Their Abusive Supervisors: The Role of LMX, Self-Blame, and Guilt
Tröster, C., & Quaquebeke, N. V. (2020) · Academy of Management Journal · doi:10.5465/amj.2019.0559 · Cited by 137 · see citing works
Lee, K. Y., Sheehan, L., Lee, K. K., & Chang, Y. (2021) · Internet Research · doi:10.1108/intr-06-2020-0327 · Cited by 140 · see citing works
Using MTurk to Distribute a Survey or Experiment: Methodological Considerations
Hunt, N. C., & Scheetz, A. M. (2018) · Journal of Information Systems · doi:10.2308/isys-52021 · Cited by 136 · see citing works
Distinct Effects of Pride and Gratitude Appeals on Sustainable Luxury Brands
Septianto, F., Seo, Y., & Errmann, A. (2020) · Journal of Business Ethics · doi:10.1007/s10551-020-04484-7 · Cited by 134 · see citing works
Thompson, A. J., & Pickett, J. T. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09436-7 · Cited by 130 · see citing works
Scaling Up: Representing Gender Diversity in Survey Research
Magliozzi, D., Saperstein, A., & Westbrook, L. (2016) · Socius: Sociological Research for a Dynamic World · doi:10.1177/2378023116664352 · Cited by 146 · see citing works
Desai, S. C., & Reimers, S. J. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1066-z · Cited by 127 · see citing works
Running Behavioral Operations Experiments Using Amazon's Mechanical Turk
Lee, Y. S., Seo, Y. W., & Siemsen, E. (2018) · Production and Operations Management · doi:10.1111/poms.12841 · Cited by 128 · see citing works
Hu, Y., Xu, A., Hong, Y., Gal, D., Sinha, V. S., & Akkiraju, R. (2018) · Journal of Management Information Systems · doi:10.1080/07421222.2019.1628908 · Cited by 126 · see citing works
Self-perceived food addiction: Prevalence, predictors, and prognosis
Meadows, A., Nolan, L. J., & Higgs, S. (2017) · Appetite · doi:10.1016/j.appet.2017.03.051 · Cited by 119 · see citing works
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017) · SAGE Open · doi:10.1177/2158244017712774 · Cited by 122 · see citing works
Gottlieb, S., Keltner, D. J., & Lombrozo, T. (2018) · Cognitive Science · doi:10.1111/cogs.12648 · Cited by 122 · see citing works
Ghosting and destiny: Implicit theories of relationships predict beliefs about ghosting
Freedman, G., Powell, D. N., Le, B., & Williams, K. D. (2019) · Journal of Social and Personal Relationships · doi:10.1177/0265407517748791 · Cited by 120 · see citing works
Participant carelessness and fraud: Consequences for clinical research and potential solutions.
Chandler, J., Sisso, I., & Shapiro, D. N. (2019) · Journal of Abnormal Psychology · doi:10.1037/abn0000479 · Cited by 124 · see citing works
- Authors
- Peer, E., Vosgerau, J., & Acquisti, A.
- Affiliations
- Tilburg University (NL)
- Carnegie Mellon University (US)
- Venue
- Behavior Research Methods
- Published
- 2013-12-20
- Volume
- 46
- Issue
- 4
- Pages
- 1023-1031
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2105753174
- Cited by
- 1,618
- References
- 18
- Citations by year
- 2026: 68 · 2025: 96 · 2024: 128 · 2023: 167 · 2022: 175 · 2021: 212 · 2020: 210 · 2019: 147 · 2018: 119 · 2017: 132 · 2016: 91 · 2015: 49 · 2014: 17 · 2013: 1
- FWCI
- 79.80
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Behavior Research Methods (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- SSRN Electronic Journal (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- KiltHub Repository
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- data quality, Amazon Mechanical Turk, worker reputation, crowdsourcing, selection bias, labor productivity, participant recruitment
- MeSH terms
- Data Collection, Humans, Research Design, Patient Selection, Internet, Behavioral Research, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
