Data quality of platforms and panels for online behavioral research
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
We examine key aspects of data quality for online behavioral research between selected platforms (Amazon Mechanical Turk, CloudResearch, and Prolific) and panels (Qualtrics and Dynata). To identify the key aspects of data quality, we first engaged with the behavioral research community to discover which aspects are most critical to researchers and found that these include attention, comprehension, honesty, and reliability. We then explored differences in these data quality aspects in two studies (N ~ 4000), with or without data quality filters (approval ratings). We found considerable differences between the sites, especially in comprehension, attention, and dishonesty. In Study 1 (without filters), we found that only Prolific provided high data quality on all measures. In Study 2 (with filters), we found high data quality among CloudResearch and Prolific. MTurk showed alarmingly low data quality even with data quality filters. We also found that while reputation (approval rating) did not predict data quality, frequency and purpose of usage did, especially on MTurk: the lowest data quality came from MTurk participants who report using the site as their main source of income but spend few hours on it per week. We provide a framework for future investigation into the ever-changing nature of data quality in online research, and how the evolving set of platforms and panels performs on these key aspects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“a research panel service identified as providing high data quality in online behavioral research (Peer et al., 2022).”
From “ChatGPT seems too good to be true”: College students’ use and perceptions of generative AI (2024) · cites this paper as Peer et al., 2022
In the citing paper
“There is empirical evidence proving the high data quality of Prolific for behavioral research (e.g., Eyal et al., 2021).”
From Development and Validation of a Theory-Based Questionnaire to Measure Different Types of Cognitive Load (2023) · cites this paper as Eyal et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 2×
“These mostly included testing participants ’ ability to pass attention-check questions (e.g., Paolacci et al., 2010, see also Göritz et al., 2021), reliability of validated scales (e.g., Buhrmester et al., 2016), replicability of known effects such as the Asian disease problem (e.g., Chandler et…”
Instructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957 · cited 4×
“Indeed, previous research has suggested that sites where participants ’ frequency of usage is lower (e.g., Prolific) show higher data quality (Peer et al., 2017).”
InfluentialThe Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913
The Dishonesty of Honest People: A Theory of Self-Concept Maintenance
Mazar, N., Amir, O., & Ariely, D. (2008) · Journal of Marketing Research · doi:10.1509/jmkr.45.6.633 · Cited by 2,875 · cited 1×
“Honesty was measured using an online version of Mazar et al. ’ s (2008) matrix problem task, which appeared at the end of the survey.”
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 3×
“…reputation (approval ratings — share of participants ’ submissions that were accepted by the researcher) can positively predict higher data quality, as high reputation was found to produce better data quality, to the point that even attention-check questions were unnecessary (Peer et al., 2014).”
InfluentialSpontaneous giving and calculated greed
Rand, D. G., Greene, J. D., & Nowak, M. A. (2012) · Nature · doi:10.1038/nature11467 · Cited by 1,334 · cited 2×
“Comprehension refers to the degree to which participants seem to understand a task ’ s instructions (Berinsky et al., 2014; Rand et al., 2012).”
A Domain-Specific Risk-Taking (DOSPERT) scale for adult populations
Blais, A.-R., & Weber, E. U. (2006) · Judgment and Decision Making · doi:10.1017/s1930297500000334 · Cited by 1,286 · cited 1×
“Reliability was measured using the NFC scale (Cacioppo et al., 1984), which measures the extent to which respondents like to engage in and enjoy thinking, and the Domain-Specific Risk-Taking Scale (DOSPERT; Blais & Weber, 2006), which evaluates self-reported risk-taking behavior.”
The viability of crowdsourcing for survey research
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0081-0 · Cited by 1,018
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 3×
“Comprehension refers to the degree to which participants seem to understand a task ’ s instructions (Berinsky et al., 2014; Rand et al., 2012).”
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levendusky, M. S., Simpser, A., & Wang, R. (2011) · Public Opinion Quarterly · doi:10.1093/poq/nfr020 · Cited by 758
Best practice recommendations for data screening
DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015) · Journal of Organizational Behavior · doi:10.1002/job.1962 · Cited by 679
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
Thomas, K. A., & Clifford, S. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.08.038 · Cited by 625
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494
Amazon mechanical Turk in organizational psychology: an evaluation and practical recommendations
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017) · Journal of Business and Psychology · doi:10.1007/s10869-016-9458-5 · Cited by 474
The truth about lies: A meta-analysis on dishonest behavior.
Gerlach, P., Teodorescu, K., & Hertwig, R. (2019) · Psychological Bulletin · doi:10.1037/bul0000174 · Cited by 454
Forecasting elections with non-representative polls
Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.06.001 · Cited by 371
Self-citationSchmidt, F. L., Le, H., & Ilies, R. (2003) · Psychological Methods · doi:10.1037/1082-989x.8.2.206 · Cited by 362 · cited 1×
“For example, personality researchers would be concerned about the reliability and validity of scales (e.g., Schmidt et al., 2003); experimentalists would require high attention to stimuli and would require that participants carefully read and comprehend instructions (e.g., Rand et al., 2012), and…”
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2019) · Journal of Management · doi:10.1177/0149206318811569 · Cited by 338
Keith, M. G., Tay, L., & Harms, P. D. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01359 · Cited by 319
Chandler, J. J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 277
Kim, Y., Dykema, J., Stevenson, J., Black, P., & Moberg, D. P. (2019) · Social Science Computer Review · doi:10.1177/0894439317752406 · Cited by 258
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185
MacInnis, B., Krosnick, J. A., Ho, A. S., & Cho, M.-J. (2018) · Public Opinion Quarterly · doi:10.1093/poq/nfy038 · Cited by 160
Feitosa, J., Joseph, D. L., & Newman, D. A. (2014) · Personality and Individual Differences · doi:10.1016/j.paid.2014.11.017 · Cited by 145
Thompson, A. J., & Pickett, J. T. (2019) · Journal of Quantitative Criminology · doi:10.1007/s10940-019-09436-7 · Cited by 139
Professional respondents in nonprobability online panels
Hillygus, D. S., Jackson, N., & Young, M. (2014) · Online Panel Research · doi:10.1002/9781118763520.ch10 · Cited by 138
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Domain-Specific Risk-Taking Scale
Blais, A.-R., & Weber, E. U. (2006) · PsycTESTS Dataset · doi:10.1037/t13084-000 · Cited by 102
The Use of Online Panel Data in Management Research: A Review and Recommendations:
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2019) · Sage Journals Data · doi:10.25384/sage.c.4310936.v1 · Cited by 98
Reply to MTurk, Prolific or panels? Choosing the right audience for online research
Litman, L., Moss, A., Rosenzweig, C., & Robinson, J. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3775075 · Cited by 57
The Use of Online Panel Data in Management Research: A Review and Recommendations
Porter, C. O. L. H., Outlaw, R., Gale, J. P., & Cho, T. S. (2018) · Sage Journals Data · doi:10.25384/sage.c.4310936 · Cited by 50
Using Attention Testing to Select Crowdsourced Workers and Research Participants
Göritz, A. S., Borchert, K., & Hirth, M. (2019) · Social Science Computer Review · doi:10.1177/0894439319848726 · Cited by 48
Cited by
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
Kirk, C. P., & Givi, J. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2024.114984 · Cited by 354
“ChatGPT seems too good to be true”: College students’ use and perceptions of generative AI
Baek, C., Tate, T., & Warschauer, M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100294 · Cited by 306
Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models
Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y., Ross, B. L., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, E., & Loaiza-Ganem, G. (2023) · neural information processing systems · doi:10.52202/075280-0165 · Cited by 290
Metadata
- Authors
- Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E.
- Affiliations
- Microsoft Research New York City (United States) (US)
- ORCID
- Rothschild, David
- Venue
- Behavior Research Methods
- Published
- 2021-09-29
- Volume
- 54
- Issue
- 4
- Pages
- 1643-1662
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC8480459
- OpenAlex
- W3202749896
- Cited by
- 1,718
- References
- 39
- Citations by year
- 2026: 383 · 2025: 447 · 2024: 419 · 2023: 305 · 2022: 143 · 2021: 19
- FWCI
- 187.63
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Behavior Research Methods (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- data quality, crowdsourcing platforms, Amazon Mechanical Turk, Prolific, Qualtrics, approval ratings, comprehension, dishonesty, reliability statistics, researcher reputation
- MeSH terms
- Data Accuracy, Attention, Humans, Names, Reproducibility of Results, Behavioral Research, Crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.