Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher’s question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our primary concern was the potential compromise in data quality and representativeness often associated with these platforms, as highlighted in recent studies [23,35,68].”
From Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Combined with superior prescreening tools, transparent demographic targeting, and cost-efficiency (Douglas et al., 2023), these strengths make prolific particularly well-suited for research in educational psychology.”
From Testing the testing effect on prolific: when retrieval practice fails to boost learning (2026) · cites this paper as Douglas et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prolific participants show higher attentiveness, longer and more appropriate completion times, fewer signs of disengagement, and greater response consistency (Albert and Smilek, 2023; Douglas et al., 2023).”
From Testing the testing effect on prolific: when retrieval practice fails to boost learning (2026) · cites this paper as Douglas et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Douglas et al. found that the data quality of samples recruited on Prolific is higher than on other labour markets and student samples, with participants paying more attention to the tasks and providing more meaningful answers [22].”
From Cognitive Forcing for Better Decision-Making: Reducing Overreliance on AI Systems Through Partial Explanations (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prolific and CloudResearch users consistently outperform Amazon Mechanical Turk (MTurk), Qualtrics, and SONA users on key data quality indicators (Douglas et al., 2023).”
From Testing the testing effect on prolific: when retrieval practice fails to boost learning (2026) · cites this paper as Douglas et al., 2023
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“, to account for the fact that we already “used up” all available degrees of freedom for the test of our a priori hypotheses) [43].”
BackgroundAmazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
The international personality item pool and the future of public-domain personality measures
Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006) · Journal of Research in Personality · doi:10.1016/j.jrp.2005.08.007 · Cited by 3,185
(1999) · Medical Entomology and Zoology · Cited by 3,151
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 3×
“[19] observed that MTurk (the platform with the most experienced workers) had worse data quality than Prolific (one of two platforms with less experienced workers) but better data quality than CrowdFlower (the other platform with less experienced workers).”
BackgroundInfluentialData quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,295
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · cited 3×
“, Cronbach’s alphas and internal validity between scales) [2].”
BackgroundCoppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Research works printed but not linked: 1
- Ternovski J, Orr L. A Note on Increases in Inattentive Online Survey-Takers Since 2020. Journal of Quantitative Description: Digital Media. 2022; 2: 1– 35.
Cited by
On the conversational persuasiveness of GPT-4
Salvi, F., Horta Ribeiro, M., Gallotti, R., & West, R. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02194-6 · Cited by 264
On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial
Salvi, F., Ribeiro, M. H., Gallotti, R., & West, R. (2024) · doi:10.21203/rs.3.rs-4429707/v1 · Cited by 285
Milli, S., Carroll, M., Wang, Y., Pandey, S., & Dragan, A. D. (2023) · arXiv.org · doi:10.48550/arxiv.2305.16941 · Cited by 200
Comparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166
“These findings corroborate observations from survey studies showing significantly fewer correct responses to attention-check questions in MTurk samples 38,41,64 .”
ResultOr they could just not use it? The dilemma of AI disclosure for audience trust in news
Toff, B., & Simon, F. M. (2025) · The International Journal of Press/Politics · doi:10.1177/19401612241308697 · Cited by 97
University students’ self-reported reliance on ChatGPT for learning: A latent profile analysis
Stojanov, A., Liu, Q., & Koh, J. H. L. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100243 · Cited by 87
Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
Dhillon, P. S., Molaei, S., Li, J., Golub, M., Zheng, S., & Robert, L. P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642134 · Cited by 111
Evaluating SoJump.com as a tool for online behavioral research in China
Ponte, A. D., Li, L., Ang, L., Lim, N., & Seow, W. J. (2024) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2024.100905 · Cited by 115
Formosa, P., Bankins, S., Matulionyte, R., & Ghasemi, O. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02081-0 · Cited by 63
Representativeness and response validity across nine opt-in online samples
Stagnaro, M. N., Druckman, J. N., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02438-z · Cited by 117
Galvan, M. J., Payne, B. K., Hannay, J. W., Georgeson, A. R., & Muscatell, K. A. (2023) · Annals of Behavioral Medicine · doi:10.1093/abm/kaad054 · Cited by 107
Alkhayyal, S., & Bajaba, S. M. (2024) · Acta Psychologica · doi:10.1016/j.actpsy.2024.104377 · Cited by 98
How do citizens perceive the use of Artificial Intelligence in public sector decisions?
Haesevoets, T., Verschuere, B., Van Severen, R., & Roets, A. (2024) · Government Information Quarterly · doi:10.1016/j.giq.2023.101906 · Cited by 74
ChatGPT usage in everyday life: A motivation-theoretic mixed-methods study
Wolf, V., & Maier, C. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102821 · Cited by 108
Representativeness and Response Validity Across Nine Opt-In Online Samples
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2024) · doi:10.31234/osf.io/h9j2d · Cited by 68
Community notes increase trust in fact-checking on social media
Drolsbach, C., Solovev, K., & Pröllochs, N. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae217 · Cited by 121
Large language models show amplified cognitive biases in moral decision-making
Cheung, V., Maier, M., & Lieder, F. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2412015122 · Cited by 64
Koning, B., & Voorveld, H. A. M. (2025) · Journal of Interactive Advertising · doi:10.1080/15252019.2025.2554149 · Cited by 52
Huynh, M.-T., & Aichner, T. (2025) · AI & Society · doi:10.1007/s00146-025-02378-8 · Cited by 91
Attention checks and how to use them: Review and practical recommendations
Muszyński, M. A. (2023) · Ask Research and Methods · doi:10.18061/ask.v32i1.0001 · Cited by 99
Hong, S. J. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108460 · Cited by 66
Ask ChatGPT first! Transforming learning experiences in the age of artificial intelligence
Gao, L. (., López-Pérez, M. E., Melero-Polo, I., & Trifu, A. (2024) · Studies in Higher Education · doi:10.1080/03075079.2024.2323571 · Cited by 46
Human vs. AI: can ChatGPT improve tourism product descriptions?
Guttentag, D. A., Litvin, S. W., & Teixeira, R. (2024) · Current Issues in Tourism · doi:10.1080/13683500.2024.2402563 · Cited by 65
On-Demand Delivery Platforms and Restaurant Sales
Li, Z., & Wang, G. (2024) · Management Science · doi:10.1287/mnsc.2021.01010 · Cited by 73
Large Language Models Show Amplified Cognitive Biases in Moral Decision-Making
Cheung, V., Maier, M., & Lieder, F. (2025) · doi:10.31234/osf.io/aj46b_v3 · Cited by 53
News bylines and perceived AI authorship: Effects on source and message credibility
Jia, H., Appelman, A., Wu, M., & Bien-Aimé, S. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100093 · Cited by 77
Metaphors of AI indicate that people increasingly perceive AI as warm and human-like
Cheng, M., Lee, A. Y., Rapuano, K. M., Niederhoffer, K. G., Liebscher, A., & Hancock, J. T. (2026) · Communications Psychology · doi:10.1038/s44271-025-00376-6 · Cited by 75
Faraon, M., Rönkkö, K., Milrad, M., & Tsui, E. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13492-x · Cited by 50
“Research has shown that Prolific participants outperform those from MTurk and Qualtrics in passing attention checks, providing meaningful responses, and following instructions (Douglas et al., 2023; Peer et al., 2022).”
Writing with AI Lowers Psychological Ownership, but Longer Prompts Can Help
Joshi, N., & Vogel, D. (2025) · Proceedings of the 7th ACM Conference on Conversational User Interfaces · doi:10.1145/3719160.3736608 · Cited by 50
The moral preferences of investors: Experimental evidence
Bonnefon, J., Landier, A., Sastry, P., & Thesmar, D. (2024) · Journal of Financial Economics · doi:10.1016/j.jfineco.2024.103955 · Cited by 82
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 102
Interplay of rationality and morality in using ChatGPT for academic misconduct
Zhang, L., Amos, C., & Pentina, I. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2325023 · Cited by 48
“Compared to other online respondent platforms maintained by commercial survey firms and in-person studies with undergraduate students, Prolific respondents yield higher data quality (Douglas, Ewell, and Brauer 2023; Peer et al. 2022).”
Luo, X., Wang, Z., Tilley, J. L., Balarajan, S., Bassey, U.-A., & Cheang, C. I. (2025) · JMIR Mental Health · doi:10.2196/77951 · Cited by 48
“Prolific provides opportunities to recruit diverse samples of individuals for online studies, while reducing common sampling biases related to quick responses and inattention [11].”
The Burden for High-Quality Online Data Collection Lies With Researchers, Not Recruitment Platforms
Cuskley, C. F., & Sulik, J. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916241242734 · Cited by 59
Overbye-Thompson, H. E., & Hamilton, K. A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0334616 · Cited by 62
Automated Annotation with Generative AI Requires Validation
Pangakis, N., Wolken, S., & Fasching, N. (2023) · arXiv · doi:10.48550/arxiv.2306.00176 · Cited by 47
“…accuracy or intercoder agreement) and analyze 1 Researchers have raised concerns about the quality of data collected from workers on popular crowdsourcing platforms like Amazon’s Mechanical Turk (Chmielewski and Kucker, 2020; Douglas et al., 2023) a few insular tasks on a small number of datasets.”
Huang, K., & Ball, C. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1146 · Cited by 46
The Inventory of Nonordinary Experiences (INOE): Evidence of validity in the United States and India
Taves, A., Ihm, E. D., Wolf, M., Barlev, M., Kinsella, M., & Vyas, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0287780 · Cited by 43
Jedicke, E. M., Arnold, C., & Lindenmeier, J. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2024.115127 · Cited by 55
“As an AI language model, I cannot”: Investigating LLM Denials of User Requests
Wester, J., Schrills, T., Pohl, H., & van Berkel, N. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642135 · Cited by 36
Leveraging ChatGPT for Automated Human-centered Explanations in Recommender Systems
Silva, Í. D. S., Marinho, L. B., Said, A., & Willemsen, M. C. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645171 · Cited by 65
Detecting the corruption of online questionnaires by artificial intelligence
Lebrun, B., Temtsin, S., Vonasch, A. J., & Bartneck, C. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1277635 · Cited by 54
Heiss, R., Bode, L., Adisuryo, Z. M., Brito, L. D. Q., Cuadra, A. R., Gao, P., Han, Y., Hearst, M., Huang, K., Kinyua, A., et al. (2024) · Health Communication · doi:10.1080/10410236.2023.2301201 · Cited by 46
AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
Schoenegger, P., Park, P. S., Karger, E., Trott, S., & Tetlock, P. E. (2025) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3707649 · Cited by 69
AI-Generated News Content: The Impact of AI Writer Identity and Perceived AI Human-Likeness
Lee, D. C., Jhang, J., & Baek, T. H. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2477739 · Cited by 43
Ruby, M. B., Graça, J., & Olli, E. (2024) · Appetite · doi:10.1016/j.appet.2024.107288 · Cited by 41
Wullenkord, M. C., Johansson, M. V., Loy, L. S., Menzel, C., & Reese, G. (2024) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2024.102278 · Cited by 46
Firm AI Adoption Intensity and Marketing Performance
Chen, J., Zhou, W., & Frankwick, G. L. (2023) · Journal of Computer Information Systems · doi:10.1080/08874417.2023.2277751 · Cited by 61
Chen, C., & Cui, Z. (2025) · Journal of Medical Internet Research · doi:10.2196/66083 · Cited by 44
Sustainable luxury brands: the moderating effects of salient identity-based goals
Li, J., & Kang, J. (2024) · Journal of Product & Brand Management · doi:10.1108/jpbm-04-2023-4452 · Cited by 40
Banker, S. M., Harrington, M., Schafer, M., Na, S., Heflin, M., Barkley, S., Trayvick, J., Peters, A. W., Thinakaran, A. A., Schiller, D., et al. (2025) · Nature Mental Health · doi:10.1038/s44220-025-00385-8 · Cited by 43
de Jong, S., Paananen, V., Tag, B., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710946 · Cited by 32
Tang, M., Hofreiter, S., Werner, C. H., Zielińska, A., & Karwowski, M. (2025) · The Journal of Creative Behavior · doi:10.1002/jocb.1519 · Cited by 32
Lim, J. S., Shin, D., Lee, C., Kim, J., & Zhang, J. (2025) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2025.2487679 · Cited by 32
Dupuis, H. E., & Girme, Y. U. (2023) · Personality and Social Psychology Bulletin · doi:10.1177/01461672231203123 · Cited by 37
Drone delivery and the value of customer privacy: A discrete choice experiment with U.S. consumers
Berke, A., Ding, G., Chin, C., Gopalakrishnan, K., Larson, K., Balakrishnan, H., & Li, M. Z. (2023) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2023.104391 · Cited by 33
Inoculation and accuracy prompting increase accuracy discernment in combination but not alone
Pennycook, G., Berinsky, A. J., Bhargava, P., Lin, H., Cole, R., Goldberg, B., Lewandowsky, S., & Rand, D. G. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02023-2 · Cited by 47
Mental health and academic experiences among U.S. college students during the COVID-19 pandemic
Roberts, M. E., Bell, E. A., & Meyer, J. L. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1166960 · Cited by 29
Revealing complexities when adult readers engage in the credibility evaluation of social media posts
Kuutila, M., Kiili, C., Kupiainen, R., Huusko, E., Li, J., Hosio, S. J., Mäntylä, M., Coiro, J., & Kiili, K. J. M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.108017 · Cited by 35
A Time-lagged Examination of Voluntary and Task-related Green Behavior in the Travel Industry
Dhir, A., Talwar, S., Raj, S., Sampat, B. H., & Nicolau, J. L. (2024) · Journal of Travel Research · doi:10.1177/00472875241230008 · Cited by 29
A network analysis of factors influencing the purchase intentions for refurbished electronics
Wallner, T. S., Haslbeck, J. M. B., Magnier, L., & Mugge, R. (2024) · Sustainable Production and Consumption · doi:10.1016/j.spc.2024.03.009 · Cited by 31
Li, W., Xu, S., Zheng, X., & Sun, R. (2024) · Science Communication · doi:10.1177/10755470241232352 · Cited by 31
Uncovering the reasons behind consumers’ shift from online to offline shopping
Kim, H., & Han, S. M. (2023) · Journal of Services Marketing · doi:10.1108/jsm-02-2023-0060 · Cited by 38
Smooth Talking: Generative Versus Scripted Coaching Chatbot Adoption and Efficacy Comparison
Terblanche, N. H. D. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2423125 · Cited by 35
Li, J., & Kang, J. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104268 · Cited by 36
The continued influence of AI-generated deepfake videos despite transparency warnings
Clark, S. P., & Lewandowsky, S. (2026) · Communications Psychology · doi:10.1038/s44271-025-00381-9 · Cited by 42
Vargiu, C., Nai, A., & Valli, C. (2024) · Political Psychology · doi:10.1111/pops.12969 · Cited by 33
AI contextual information shapes moral and aesthetic judgments of AI-generated visual art
Bara, I., Ramsey, R., & Cross, E. S. (2025) · Cognition · doi:10.1016/j.cognition.2025.106063 · Cited by 37
Beyond WEIRD: Can synthetic survey participants substitute for humans in global policy research?
Shrestha, P., Krpan, D., Koaik, F., Schnider, R., Sayess, D., & Binbaz, M. S. (2024) · Behavioral Science & Policy · doi:10.1177/23794607241311793 · Cited by 45
How funny is ChatGPT? A comparison of human- and A.I.-produced jokes
Gorenz, D., & Schwarz, N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305364 · Cited by 47
Zhu, Q., Duan, Y., & Sarkis, J. (2024) · Journal of Business Logistics · doi:10.1111/jbl.12405 · Cited by 31
How do people react to political bias in generative artificial intelligence (AI)?
Messer, U. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100108 · Cited by 43
Sociodemographic disparities in health literacy among American adults: A national survey study
Yu, Y., Altares, A., Leib, A. R., Bellows, L. L., Berry, C., Graham, D. J., & Mueller, M. P. (2025) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2025.103179 · Cited by 31
Landry, A., Druckman, J. N., & Willer, R. (2024) · Political Behavior · doi:10.1007/s11109-024-09934-w · Cited by 32
Measuring norm pluralism and perceived polarization in US politics
Panizza, F., Dimant, E., Kimbrough, E. O., & Vostroknutov, A. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae413 · Cited by 29
Role of “Likes” and “Dislikes” in Influencing User Behaviors on Social Media
Turel, O., & Qahri‐Saremi, H. (2024) · Journal of Management Information Systems · doi:10.1080/07421222.2024.2340829 · Cited by 31
Cybersecurity Risks and Incidents Disclosure: A Literature Review *
Amani, F., MAGNAN, M. L., & Moldovan, R. (2025) · Accounting Perspectives · doi:10.1111/1911-3838.12411 · Cited by 29
How people choose between criminal opportunities
Krajewski, A. T., Pickett, J. T., & Jacobs, B. A. (2025) · Criminology · doi:10.1111/1745-9125.70006 · Cited by 25
Why you shouldn’t trust data collected on MTurk
Kay, C. S. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02852-7 · Cited by 35
What over 1,000,000 participants tell us about online research protocols
Tomczak, J., Gordon, A., Adams, J., Pickering, J. S., Hodges, N., & Evershed, J. K. (2023) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2023.1228365 · Cited by 27
Wright, M., Galante, J., Corneille, J. S., Grabovac, A. D., Ingram, D. M., & Sacchet, M. D. (2024) · Mindfulness · doi:10.1007/s12671-024-02356-z · Cited by 29
Cummings, J. J., & Ingber, A. S. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108427 · Cited by 27
Zhan, E. S., Zheng, Q., Dong, C., & Thorson, E. (2025) · International Journal of Strategic Communication · doi:10.1080/1553118x.2025.2471527 · Cited by 22
Wardatzky, K., Inel, O., Rossetto, L., & Bernstein, A. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3716394 · Cited by 35
Fissel, E. R., & Lee, J. R. (2023) · Journal of Criminology · doi:10.1177/26338076231174639 · Cited by 29
Gender, Issue Stereotypes, and the Electoral Returns to Distributive Politics in the United States
Hamel, B. T., & Bauer, N. M. (2024) · Politics & Gender · doi:10.1017/s1743923x24000321 · Cited by 20
Chong, L., Lo, I.-P., Rayan, J. A., Dow, S. P., Ahmed, F., & Lykourentzou, I. (2025) · Design Science · doi:10.1017/dsj.2024.51 · Cited by 34
Singh, N., Awan, U., Basahel, S. B., & Alghafes, R. A. (2025) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2025.124219 · Cited by 22
Bui, H. T., Filimonau, V., & Sezerel, H. (2025) · Tourism Management Perspectives · doi:10.1016/j.tmp.2025.101392 · Cited by 31
Articulating scholarship in human resource management: Guidance for researchers
Budhwar, P. S., Wood, G., Chowdhury, S., Aguinis, H., Breslin, D., Collings, D. G., Cooke, F. L., Darabi, F., Eby, L. T. D. T., Martin, U. M., et al. (2024) · Human Resource Management Journal · doi:10.1111/1748-8583.12567 · Cited by 32
How Meaningless and Substantive Green Claims Jointly Determine Product Environmental Perceptions
Polonsky, M. J., Rotman, J., Weber, V., & Kumar, P. (2025) · International Journal of Advertising · doi:10.1080/02650487.2024.2377503 · Cited by 19
Reminders of Japanese redress increase Asian American support for Black reparations
Kraus, M. W., & Vinluan, A. C. (2023) · Communications Psychology · doi:10.1038/s44271-023-00033-w · Cited by 20
den Kamp, E. M. O., Bakker, A. B., Tims, M., Demerouti, E., & de Wijs, J. J. (2024) · Journal of Vocational Behavior · doi:10.1016/j.jvb.2024.103987 · Cited by 22
Investigating the Role of Cultural Values in Adopting Large Language Models for Software Engineering
Lambiase, S., Catolino, G., Palomba, F., Ferrucci, F., & Russo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3725529 · Cited by 41
Augmenting diversity in hiring decisions with artificial intelligence tools
Wilkens, U., Lutzeyer, I., Zheng, C. S.-M., Beşer, A., & Prilla, M. (2025) · The International Journal of Human Resource Management · doi:10.1080/09585192.2025.2492867 · Cited by 24
To Trust or Distrust AI: A Questionnaire Validation Study
Scharowski, N., Perrig, S. A. C., von Felten, N., Aeschbach, L. F., Opwis, K., Wintersberger, P., & Brühlmann, F. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732025 · Cited by 37
Krockow, E. M., Cheng, K. O., Maltby, J., & McElroy, E. (2023) · Communications Medicine · doi:10.1038/s43856-023-00379-6 · Cited by 27
Newlands, G., & Lutz, C. (2024) · Research in Social Stratification and Mobility · doi:10.1016/j.rssm.2024.100935 · Cited by 29
Motivated information acquisition and social norm formation
Dimant, E., Galeotti, F., & Villeval, M. C. (2024) · European Economic Review · doi:10.1016/j.euroecorev.2024.104778 · Cited by 32
Validating a brief measure of four facets of social evaluation
Koch, A. S., Smith, A., Fiske, S. T., Abele, A. E., Ellemers, N., & Yzerbyt, V. Y. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02489-y · Cited by 31
Metadata
- Authors
- Douglas, B. D., Ewell, P. J., & Brauer, M.
- Affiliations
- University of Wisconsin–Madison (US)
- Kenyon College (US)
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-03-14
- Volume
- 18
- Issue
- 3
- Pages
- e0279720
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC10013894
- OpenAlex
- W4324129606
- Cited by
- 1,532
- References
- 44
- Citations by year
- 2026: 558 · 2025: 571 · 2024: 329 · 2023: 69
- FWCI
- 347.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- BRITE Lab Grant: Behavioral Research Insights Through Experiments Lab, <https://brite.wisc.edu/>
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Keywords
- online data collection, data quality, crowdsourcing platforms, Amazon Mechanical Turk, Prolific, Qualtrics, Sona, attention checks, bot detection, online surveys
- MeSH terms
- Data Accuracy, Humans, Surveys and Questionnaires, Research Subjects
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.