An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk
Abstract
Data collection using Internet-based samples has become increasingly popular in many social science disciplines, including advertising. This research examines whether one popular Internet data source, Amazon's Mechanical Turk (MTurk), is an appropriate substitute for other popular samples utilized in advertising research. Specifically, a five-sample between-subjects experiment was conducted to help researchers who utilize MTurk in advertising experiments understand the strengths and weaknesses of MTurk relative to student samples and professional panels. In comparisons across five samples, results show that the MTurk data outperformed panel data procured from two separate professional marketing research companies across various measures of data quality. The MTurk data were also compared to two different student samples, and results show the data were at least comparable in quality. While researchers may consider MTurk samples as a viable alternative to student samples when testing theory-driven outcomes, precautions should be taken to ensure the quality of data regardless of the source. Best practices for ensuring data quality are offered for advertising researchers who utilize MTurk for data collection.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Student samples are commonly used in marketing along with online panels and can provide a representative sample of the general consumer population.
“Student samples are commonly used in marketing along with online panels and can provide a representative sample of the general consumer population [73].”
From Consumer Responses to Generative AI Chatbots Versus Search Engines for Product Evaluation (2025) · cites this paper as [73]
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
The consideration of future consequences: Weighing immediate and distant outcomes of behavior.
Strathman, A., Gleicher, F., Boninger, D. S., & Edwards, C. S. (1994) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.66.4.742 · Cited by 1,782 · see citing works
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739 · see citing works
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 1× · see citing works
“…student and professional panel samples in domains outside of advertising is increasing (e.g., Bartneck et al. 2015; Berinsky, Huber, and Lenz 2012; Casler, Bickel, and Hackett 2013; Clifford, Jewell, and Waggoner 2015; Goodman, Cryder, and Cheema 2013; Heen, Lieberman, and Miethe 2014; Hamby and…”
BackgroundReputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617 · see citing works
Structure of Urban Movements: Polycentric Activity and Entangled Hierarchical Flows
Roth, C., Kang, S. M., Batty, M., & Barthélemy, M. (2011) · PLoS ONE · doi:10.1371/journal.pone.0015923 · Cited by 461 · see citing works
Economic games on the internet: the effect of $1 stakes
Amir, O., Rand, D. G., & Gal, Y. K. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031461 · Cited by 364 · see citing works
Bartneck, C., Duenser, A., Moltchanova, E., & Zawieska, K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121595 · Cited by 317 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Lou, C., & Yuan, S. (2019) · Journal of Interactive Advertising · doi:10.1080/15252019.2018.1533501 · Cited by 2,795 · see citing works
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532 · see citing works
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · see citing works
Americans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728 · see citing works
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668 · see citing works
Cheng, Y., & Jiang, H. (2020) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2020.1834296 · Cited by 469 · see citing works
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk
Sheehan, K. B. (2018) · Communication Monographs · doi:10.1080/03637751.2017.1342043 · Cited by 408 · see citing works
Consumer Responses to AI-Generated Charitable Giving Ads
Arango, L., Singaraju, S. P., & Niininen, O. (2023) · Journal of Advertising · doi:10.1080/00913367.2023.2183285 · Cited by 243 · see citing works
Boyraz, G., Legros, D. N., & Tigershtrom, A. (2020) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2020.102307 · Cited by 142 · see citing works
Development and Initial Validation of the Duke Misophonia Questionnaire
Rosenthal, M. Z., Anand, D., Cassiello-Robbins, C., Williams, Z. J., Guetta, R. E., Trumbull, J., & Kelley, L. D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.709928 · Cited by 117 · see citing works
Jang, W. (., Kim, J. (., Kim, S., & Chun, J. W. (2021) · Current Issues in Tourism · doi:10.1080/13683500.2020.1845126 · Cited by 116 · see citing works
Consumers’ willingness to pay for upcycled foods
Bhatt, S., Ye, H., Deutsch, J., Ayaz, H., & Suri, R. (2020) · Food Quality and Preference · doi:10.1016/j.foodqual.2020.104035 · Cited by 112 · see citing works
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94 · see citing works
The blame shift: Robot service failures hold service firms more accountable
Ryoo, Y., Jeon, Y. A., & Kim, W. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114360 · Cited by 89 · see citing works
Representativeness and Response Validity Across Nine Opt-In Online Samples
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2024) · doi:10.31234/osf.io/h9j2d · Cited by 68 · see citing works
Development and Validation of the Four Facet Mindful Eating Scale (FFaMES)
Carrière, K., Shireen, S. H., Siemers, N., Preißner, C. E., Starr, J., Falk, C., & Knäuper, B. (2022) · Appetite · doi:10.1016/j.appet.2021.105689 · Cited by 47 · see citing works
Mortality salience effects fail to replicate in traditional and novel measures
Sætrevik, B., & Sjåstad, H. (2022) · Meta-Psychology · doi:10.15626/mp.2020.2628 · Cited by 35 · see citing works
Demographic and Sociocultural Predictors of Face Image Satisfaction: The U.S. Body Project I
Frederick, D. A., Reynolds, T. A., Barrera, C. A., & Murray, S. B. (2022) · Body Image · doi:10.1016/j.bodyim.2022.01.016 · Cited by 37 · see citing works
Consumer Responses to Generative AI Chatbots Versus Search Engines for Product Evaluation
Kim, S., & Priluck, R. (2025) · Journal of Theoretical and Applied Electronic Commerce Research · doi:10.3390/jtaer20020093 · Cited by 20 · see citing works
“Student samples are commonly used in marketing along with online panels and can provide a representative sample of the general consumer population [73].”
- Authors
- Kees, J., Berry, C., Burton, S., & Sheehan, K.
- Affiliations
- Villanova University (US)
- University of Arkansas at Fayetteville (US)
- University of Oregon (US)
- Corresponding author
- Kees, Jeremy
- ORCID
- Berry, Christopher
- Venue
- Journal of Advertising
- Publisher
- Informa UK Limited
- Published
- 2017-01-02
- Volume
- 46
- Issue
- 1
- Pages
- 141-155
- ISSN
- 0091-3367 (ISSN-L)
- 1557-7805
- OpenAlex
- W2581057531
- Cited by
- 902
- References
- 10
- Citations by year
- 2026: 34 · 2025: 74 · 2024: 84 · 2023: 116 · 2022: 141 · 2021: 149 · 2020: 153 · 2019: 99 · 2018: 35 · 2017: 16
- FWCI
- 99.31
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Advertising (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Business Strategies and Management Research
- Keywords
- Amazon Mechanical Turk, data quality, advertising research, data collection methods, sample comparability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
