The DOI matches its Crossref record; 17 of the 18 references it cites were found in a registry or an index.
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Open accessAbstract
Data quality is one of the major concerns of using crowdsourcing websites such as Amazon Mechanical Turk (MTurk) to recruit participants for online behavioral studies. We compared two methods for ensuring data quality on MTurk: attention check questions (ACQs) and restricting participation to MTurk workers with high reputation (above 95% approval ratings). In Experiment 1, we found that high-reputation workers rarely failed ACQs and provided higher-quality data than did low-reputation workers; ACQs improved data quality only for low-reputation workers, and only in some cases. Experiment 2 corroborated these findings and also showed that more productive high-reputation workers produce the highest-quality data. We concluded that sampling high-reputation workers can ensure high-quality data without having to resort to using ACQs, which may lead to selection bias if participants who fail ACQs are excluded post-hoc.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Data obtained from MTurk has been demonstrated to be valid and reliable when appropriate selection criteria and attention checks are used (Casler, Bickel, & Hackett, 2013; Peer, Vosgerau, & Acquisti, 2014), as was the case in this study.”
From The development and evaluation of a brief form of the Normative Male Alexithymia Scale (NMAS-BF). (2019) · cites this paper as Peer, Vosgerau, & Acquisti, 2014
The cited source says what the claim attributes to it.
In the citing paper
“Most of the sites we reviewed have supervision and reputation mechanisms designed to measure the quality of submitted work, and to allow researchers to invite workers with high reputation (similar to MTurk's”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
In the citing paper
“Amazon Mechanical Turk is an online sample recruiting platform that, although not nationally representative, yields high data quality through the use of attention checks 44 and worker reputation scores 45 .”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [45]
In the citing paper
“Following prior research, we selected participants with an approval rating above 95% on Amazon Mechanical Turk 45 .”
From Understanding, explaining, and utilizing medical artificial intelligence (2021) · cites this paper as [45]
In the citing paper
“which showed that ACQs have low diagnostic ability when data quality is already high.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., (2014)
In the citing paper
“they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
In the citing paper
“they are reliable and validated scales, and b) they have been previously used with success to measure data quality on MTurk (Peer et al., 2014).”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
In the citing paper
“Using two attention-check questions (Peer et al., 2014), we checked whether participants read and paid attention to our instructions.”
From Beyond the Turk: Alternative platforms for crowdsourcing behavioral research (2017) · cites this paper as Peer et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001) · Personality and Social Psychology Bulletin · doi:10.1177/0146167201272002 · Cited by 3,041
The Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Rosenberg, M. (2011) · PsycTESTS Dataset · doi:10.1037/t01038-000 · Cited by 1,226
Cited by
Sorted by citations
Items 301–400 of 1,000
Woznicki, N. W., Arriaga, A. S., Caporale-Berkowitz, N. A., & Parent, M. C. (2020) · Psychology of Sexual Orientation and Gender Diversity · doi:10.1037/sgd0000458 · Cited by 50
(2022) · doi:10.1145/3491102.3501957 · Cited by 47
Slovák, P., Rowan, K., Frauenberger, C., Gilad-Bachrach, R., Doces, M., Smith, B. M., Kamb, R., & Fitzpatrick, G. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2820007 · Cited by 46
Support for a general factor of well-being
Longo, Y., Coyne, I. J., Joseph, S., & Gustavsson, P. J. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.03.082 · Cited by 48
Lansford, K. L., Borrie, S. A., & Bystricky, L. (2016) · American Journal of Speech-Language Pathology · doi:10.1044/2015_ajslp-15-0059 · Cited by 46
Computer criminal behavior is related to psychopathy and other antisocial behavior
Seigfried‐Spellar, K. C., Villacís-Vukadinović, N., & Lynam, D. R. (2017) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2017.06.003 · Cited by 46
Is It My Luck or Loyalty? The Role of Culture on Customer Preferences for Loyalty Reward Types
Hwang, Y., & Mattila, A. S. (2017) · Journal of Travel Research · doi:10.1177/0047287517718353 · Cited by 46
Examining broad intellectual abilities obtained within an mTurk internet sample
Merz, Z. C., Lace, J. W., & Eisenstein, A. M. (2020) · Current Psychology · doi:10.1007/s12144-020-00741-0 · Cited by 52
Ying, S., Huang, Y., Qian, L., & Song, J. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122412 · Cited by 49
Komissarouk, S., Harpaz, G., & Nadler, A. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.12.019 · Cited by 45
Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk
Wu, M., & Quinn, A. J. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13317 · Cited by 45
Mindfulness, Money Attitudes, and Credit
Pereira, M. C., & Coelho, F. (2019) · Journal of Consumer Affairs · doi:10.1111/joca.12197 · Cited by 45
Morality, Deterrability, and Offender Decision Making
Herman, S., & Pogarsky, G. (2020) · Justice quarterly · doi:10.1080/07418825.2019.1709884 · Cited by 41
Hopefully that’s not wasted! The role of hope for reducing food waste
Khalil, M., Northey, G., Septianto, F., & Lang, B. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.03.080 · Cited by 48
El book or the libro ? Insights from acceptability judgments into determiner/noun code-switches
Couto, M. C. P., & Stadthagen-González, H. (2017) · International Journal of Bilingualism · doi:10.1177/1367006917728392 · Cited by 44
Black, J., Belicki, K., Emberley-Ralph, J., & McCann, A. (2020) · Death Studies · doi:10.1080/07481187.2020.1737274 · Cited by 53
Silber, H., Roßmann, J., & Gummer, T. (2022) · Field Methods · doi:10.1177/1525822x221115830 · Cited by 44
Taylor, S., & DiPietro, R. B. (2017) · International Journal of Hospitality & Tourism Administration · doi:10.1080/15256480.2017.1397585 · Cited by 44
“I’ll work out tomorrow”: The Procrastination in Exercise Scale
Kelly, S. M., & Walton, H. R. (2020) · Journal of Health Psychology · doi:10.1177/1359105320916541 · Cited by 48
Sanction risk perceptions, coherence, and deterrence *
Barnum, T. C., Nagin, D. S., & Pogarsky, G. (2021) · Criminology · doi:10.1111/1745-9125.12266 · Cited by 45
Gülserliler, E. G., Blackburn, J. D., & Wassenhove, L. N. V. (2021) · Journal of Industrial Ecology · doi:10.1111/jiec.13202 · Cited by 43
Bardeen, J. R., & Fergus, T. A. (2015) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-015-9522-x · Cited by 48
Dispositional self-forgiveness, guilt- and shame-proneness, and the roles of motivational tendencies
Carpenter, T. P., Tignor, S. M., Tsang, J.-A. C., & Willett, A. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.04.017 · Cited by 45
A relationship between processing speech in noise and dysarthric speech
Borrie, S. A., Baese‐Berk, M. M., Engen, K. J. V., & Bent, T. (2017) · The Journal of the Acoustical Society of America · doi:10.1121/1.4986746 · Cited by 43
Samore, T., Fessler, D. M. T., Sparks, A. M., & Holbrook, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253326 · Cited by 43
How Online Incivility Affects Consumer Engagement Behavior on Brands’ Social Media
Wolter, J. S., Bacile, T. J., & Xu, P. (2022) · Journal of Service Research · doi:10.1177/10946705221096192 · Cited by 48
Subtirelu, N. C., & Gopavaram, S. R. (2016) · Cited by 42
Body-related self-conscious emotions and reasons for exercise: A latent class analysis
Huellemann, K. L., Pila, E., Gilchrist, J. D., Nesbitt, A. E., & Sabiston, C. M. (2021) · Body Image · doi:10.1016/j.bodyim.2021.03.016 · Cited by 47
The Validity of Crowdsourcing Data in Studying Anger and Aggressive Behavior
Lütz, J. (2015) · Social Psychology · doi:10.1027/1864-9335/a000256 · Cited by 43
Meder, B., Fleischhut, N., Krumnau, N., & Waldmann, M. R. (2018) · Risk Analysis · doi:10.1111/risa.13178 · Cited by 44
Attitudes Toward Cognitive Enhancement: The Role of Metaphor and Context
Conrad, E. C., Humphries, S. A., & Chatterjee, A. K. (2019) · AJOB Neuroscience · doi:10.1080/21507740.2019.1595771 · Cited by 42
Emotional design of multimedia learning using background images with motivational cues
Chung, S., & Cheon, J. (2020) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12450 · Cited by 49
Big data prioritization in SCM decision-making: Its role and performance implications
Wilkin, C. L., Ferreira, A., Rotaru, K., & Gaerlan, L. R. (2020) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2020.100470 · Cited by 50
Andel, S. A., Arvan, M. L., & Shen, W. (2021) · Journal of Applied Psychology · doi:10.1037/apl0000934 · Cited by 44
Salerno, J. P., Doan, L., Sayer, L. C., Drotning, K. J., Rinderknecht, R. G., & Fish, J. N. (2021) · Psychology of Sexual Orientation and Gender Diversity · doi:10.1037/sgd0000520 · Cited by 47
Albayram, Y., Khan, M. M. H., & Fagan, M. E. (2017) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1306765 · Cited by 41
Cunningham, J. A., Godinho, A., & Kushnir, V. (2017) · Internet Interventions · doi:10.1016/j.invent.2017.08.005 · Cited by 41
Redmiles, E. M., Zhu, Z., Kross, S., Kuchhal, D., Dumitraş, T., & Mazurek, M. L. (2018) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3243734.3243740 · Cited by 41
Gender/sex diversity beliefs: Scale construction, validation, and links to prejudice
Schudson, Z. C., & van Anders, S. M. (2021) · Group Processes & Intergroup Relations · doi:10.1177/1368430220987595 · Cited by 41
Mixed emotional appeal enhances positive word-of-mouth: The moderating role of narrative person
Quach, S., Septianto, F., Thaichon, P., & Chiew, T. M. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2021.102618 · Cited by 43
"I need a better description": An Investigation Into User Expectations For Differential Privacy
Cummings, R., Kaptchuk, G., & Redmiles, E. M. (2023) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 41
Efficiency-Morality Trade-Offs in Repugnant Transactions: A Choice Experiment
Elias, J. J., Lacetera, N., & Macis, M. (2016) · National Bureau of Economic Research · doi:10.3386/w22632 · Cited by 41
Criminal Records, Positive Employment Credentials, and Race
DeWitt, S. E., & Denver, M. (2019) · Journal of Research in Crime and Delinquency · doi:10.1177/0022427819886111 · Cited by 48
Supplier dependence asymmetry and investment in innovation: The role of psychological uncertainty
Ma, S., Hofer, A. R., & Aloysius, J. A. (2021) · Journal of Purchasing and Supply Management · doi:10.1016/j.pursup.2021.100674 · Cited by 44
Lim, J. R., Liu, B. F., & Seate, A. A. (2022) · Risk Analysis · doi:10.1111/risa.13957 · Cited by 40
Lajante, M., Tojib, D., & Ho, T. H. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107792 · Cited by 45
Semantic prosody and judgment.
Hauser, D., & Schwarz, N. (2016) · Journal of Experimental Psychology General · doi:10.1037/xge0000178 · Cited by 40
Differentiating Cyberbullies and Internet Trolls by Personality Characteristics and Self-Esteem
Zezulka, L., & Seigfried‐Spellar, K. C. (2016) · The journal of digital forensics, security and law · doi:10.15394/jdfsl.2016.1415 · Cited by 50
Implications of metastereotypes for attitudes toward intergenerational contact
Fowler, C., & Gasiorek, J. (2018) · Group Processes & Intergroup Relations · doi:10.1177/1368430217744032 · Cited by 49
Public knowledge of late-life cognitive decline and dementia in an international sample
Van Patten, R., & Tremont, G. (2020) · Dementia · doi:10.1177/1471301218805923 · Cited by 40
Using Attention Testing to Select Crowdsourced Workers and Research Participants
Göritz, A. S., Borchert, K., & Hirth, M. (2019) · Social Science Computer Review · doi:10.1177/0894439319848726 · Cited by 40
Greenwood, B. N., Adjerid, I., Angst, C. M., & Meikle, N. L. (2020) · Journal of Business Ethics · doi:10.1007/s10551-020-04675-2 · Cited by 40
Bieleke, M., & Keller, L. (2020) · Personality and Individual Differences · doi:10.1016/j.paid.2020.110500 · Cited by 45
When normative framing saves Mr. Nature: Role of consumer efficacy in proenvironmental adoption
Khoa, D. T., Wang, C., & Guchait, P. (2021) · Psychology and Marketing · doi:10.1002/mar.21486 · Cited by 42
Ferrer, R. A., & Ellis, E. M. (2020) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2209 · Cited by 39
Septianto, F., Lee, M. S. W., & Putra, P. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2020.102317 · Cited by 35
Sticky Stigma: The Impact of Incarceration on Perceptions of Personality Traits and Deservingness
Brew, B., Alani, F., Li, A., & Wildeman, C. (2021) · Social Forces · doi:10.1093/sf/soab091 · Cited by 29
Wang, W., Kang, S., & Choi, S. B. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182010638 · Cited by 52
Pleasure Through Pain: An Empirical Examination of Benign Masochism in Tourism
Nørfelt, A., Köck, F., Karpen, I. O., & Josiassen, A. (2022) · Journal of Travel Research · doi:10.1177/00472875211067550 · Cited by 39
Olson, D. J. (2023) · Language Testing · doi:10.1177/02655322221139162 · Cited by 41
To Earmark or to Nonearmark? The Role of Control, Transparency, and Warm-Glow
Özer, Ö., Urrea, G., & Villa, S. (2024) · Manufacturing & Service Operations Management · doi:10.1287/msom.2022.0096 · Cited by 39
Wilson, A. B., McNellis, C. J., & Latham, C. K. (2018) · International Journal of Auditing · doi:10.1111/ijau.12108 · Cited by 39
Go Fix It: Comedy as an Agent of Political Activation*
Bode, L., & Becker, A. B. (2018) · Social Science Quarterly · doi:10.1111/ssqu.12521 · Cited by 39
Explaining the relationship between job insecurity and creativity
Probst, T. M., Chizh, A., Hu, S., Jiang, L., & Austin, C. (2019) · Career Development International · doi:10.1108/cdi-04-2018-0118 · Cited by 39
Samur, D., Tops, M., Slapšinskaitė, R., & Koole, S. L. (2020) · Cognition & Emotion · doi:10.1080/02699931.2020.1732876 · Cited by 40
Public Opinion on Cognitive Enhancement Varies across Different Situations
Dinh, C. T., Humphries, S. A., & Chatterjee, A. K. (2020) · AJOB Neuroscience · doi:10.1080/21507740.2020.1811797 · Cited by 39
Workplace cyberbullying: A criminological and routine activity perspective
Zhang, S., Leidner, D. E., Cao, X., & Liu, N. (2021) · Journal of Information Technology · doi:10.1177/02683962211027888 · Cited by 45
Public attitudes toward different management scenarios for “surplus” dairy calves
Ritter, C., Hötzel, M. J., & von Keyserlingk, M. A. G. (2022) · Journal of Dairy Science · doi:10.3168/jds.2021-21425 · Cited by 39
Designing Graphs that Promote Both Risk Understanding and Behavior Change
Okan, Y., Stone, E. R., & de Bruin, W. B. (2017) · Risk Analysis · doi:10.1111/risa.12895 · Cited by 38
“Oops, I Did It” or “It Wasn’t Me:” An Examination of Psychological Contract Breach Repair Tactics
Henderson, K. E., Welsh, E. T., & O’Leary-Kelly, A. M. (2019) · Journal of Business and Psychology · doi:10.1007/s10869-019-09624-z · Cited by 46
Lifestyles and routine activities: Do they enable different types of cyber abuse?
Vakhitova, Z., Alston‐Knox, C., Reynald, D. M., Townsley, M., & Webster, J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.07.012 · Cited by 38
Exploring potentials of digital nudging for business processes
Bammert, S., König, U., Roeglinger, M., & Wruck, T. (2020) · Business Process Management Journal · doi:10.1108/bpmj-07-2019-0281 · Cited by 40
Characterizing the choice to disclose nonsuicidal self‐injury
Ammerman, B. A., Wilcox, K. T., O’Loughlin, C. M., & McCloskey, M. S. (2020) · Journal of Clinical Psychology · doi:10.1002/jclp.23045 · Cited by 40
Hettiachchi, D., van Berkel, N., Kostakos, V., & Gonçalves, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415181 · Cited by 38
Construing online consumers’ information privacy decisions: The impact of psychological distance
Bandara, R. J., Fernando, M., & Akter, S. (2021) · Information & Management · doi:10.1016/j.im.2021.103497 · Cited by 37
Yang, X., & Thøgersen, J. B. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.086 · Cited by 48
Visual body size norms and the under‐detection of overweight and obesity
Oldham, M., & Robinson, E. (2017) · Obesity Science & Practice · doi:10.1002/osp4.143 · Cited by 37
Georgeac, O., & Rattan, A. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000561 · Cited by 36
Creativity on Paid Crowdsourcing Platforms
Oppenlaender, J., Milland, K., Visuri, A., Ipeirotis, P. G., & Hosio, S. J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376677 · Cited by 42
Xie, J., Ifie, K., & Gruber, T. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.03.030 · Cited by 39
Dancing Pigs or Externalities?
Redmiles, E. M., Mazurek, M. L., & Dickerson, J. P. (2018) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3219166.3219185 · Cited by 42
Fergus, T. A., & Bardeen, J. R. (2017) · Assessment · doi:10.1177/1073191116685807 · Cited by 39
Frustration and ennui among Amazon MTurk workers
Fowler, C., Jiao, J., & Pitts, M. J. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01955-9 · Cited by 41
Blockchain for sustainable consumption: an affordance and consumer value-based view
Hina, M., Islam, A. K. M. N., & Dhir, A. (2024) · Internet Research · doi:10.1108/intr-07-2023-0523 · Cited by 48
The Importance of Place in Communicating Climate Change to Different Facets of the American Public
Halperin, A., & Walton, P. (2017) · Weather Climate and Society · doi:10.1175/wcas-d-16-0119.1 · Cited by 36
Loaiza-Ramírez, J. P., Reimer, T. O., & Moreno-Mantilla, C. E. (2022) · Energy Research & Social Science · doi:10.1016/j.erss.2022.102753 · Cited by 36
Dogra, N., & Adil, M. (2022) · Journal Of Vacation Marketing · doi:10.1177/13567667221122103 · Cited by 36
Using Animal Portraiture to Activate Emotional Affect
Whitley, C. T., Kalof, L., & Flach, T. (2020) · Environment and Behavior · doi:10.1177/0013916520928429 · Cited by 35
Khoa, D. T., Wang, C.-Y., & Guchait, P. (2020) · Service Industries Journal · doi:10.1080/02642069.2020.1831477 · Cited by 36
The effects of age cues on preferences for organic food: The moderating role of message claim
Septianto, F., & Kemper, J. A. (2021) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2021.102641 · Cited by 35
Son, S.-C., Bae, J., & Kim, K. H. (2021) · Journal of Global Fashion Marketing · doi:10.1080/20932685.2021.1947151 · Cited by 36
Rogers, J. M., Smith, K. E., Strickland, J. C., & Epstein, D. H. (2021) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.789075 · Cited by 37
A Non-Factoid Question-Answering Taxonomy
Baranova-Bolotova, V., Blinov, V., Scholer, F., Croft, W. B., & Sanderson, M. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531926 · Cited by 50
Pham, M. N., Barbaro, N., Mogilski, J. K., & Shackelford, T. K. (2015) · Personality and Individual Differences · doi:10.1016/j.paid.2015.01.034 · Cited by 34
DOING L2 SPEECH RESEARCH ONLINE: WHY AND HOW TO COLLECT ONLINE RATINGS DATA
Nagle, C., & Rehman, I. (2021) · Studies in Second Language Acquisition · doi:10.1017/s0272263121000292 · Cited by 34
Pay-for-Performance and Interpersonal Deviance
Gläser, D., van Gils, S., & Quaquebeke, N. V. (2017) · Journal of Personnel Psychology · doi:10.1027/1866-5888/a000181 · Cited by 33
Psychological Distress Among the U.S. General Population During the COVID-19 Pandemic
Guerrini, C. J., Schneider, S. C., Guzick, A. G., Nwankwo, G. N. A., Canfield, I., Fedson, S., Gutierrez, A. M., Sheu, J. C., Song, A. Y., Villagran, A. M., et al. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.642918 · Cited by 34
Tse, J. S. Y., & Haslam, N. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.699750 · Cited by 41
Sherf, E. N., & Venkataramani, V. (2015) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2015.02.002 · Cited by 37
Development and Initial Validation of the Coalitional Mate Retention Inventory
Pham, M. N., Barbaro, N., & Shackelford, T. K. (2014) · Evolutionary Psychological Science · doi:10.1007/s40806-014-0001-5 · Cited by 34
Metadata
- Authors
- Peer, E., Vosgerau, J., & Acquisti, A.
- Affiliations
- Tilburg University (NL)
- Carnegie Mellon University (US)
- Venue
- Behavior Research Methods
- Published
- 2013-12-20
- Volume
- 46
- Issue
- 4
- Pages
- 1023-1031
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2105753174
- Cited by
- 1,618
- References
- 18
- Citations by year
- 2026: 68 · 2025: 96 · 2024: 128 · 2023: 167 · 2022: 175 · 2021: 212 · 2020: 210 · 2019: 147 · 2018: 119 · 2017: 132 · 2016: 91 · 2015: 49 · 2014: 17 · 2013: 1
- FWCI
- 79.80
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Behavior Research Methods (publishedVersion)
- Research portal (Tilburg University) (publishedVersion)
- PubMed (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- SSRN Electronic Journal (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- KiltHub Repository
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- data quality, Amazon Mechanical Turk, worker reputation, crowdsourcing, selection bias, labor productivity, participant recruitment
- MeSH terms
- Data Collection, Humans, Research Design, Patient Selection, Internet, Behavioral Research, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
