The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Lie for a Dime: When Most Prescreening Responses Are Honest but Most Study Participants Are Impostors
Open accessAbstract
The Internet has enabled recruitment of large samples with specific characteristics. However, when researchers rely on participant self-report to determine eligibility, data quality depends on participant honesty. Across four studies on Amazon Mechanical Turk, we show that a substantial number of participants misrepresent theoretically relevant characteristics (e.g., demographics, product ownership) to meet eligibility criteria explicit in the studies, inferred by a previous exclusion from the study or inferred in previous experiences with similar studies. When recruiting rare populations, a large proportion of responses can be impostors. We provide recommendations about how to ensure that ineligible participants are excluded that are applicable to a wide variety of data collection efforts, which rely on self-report.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Inside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Self-citationTurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences
Litman, L., Robinson, J., & Abberbock, T. (2017) · Behavior Research Methods · doi:10.3758/s13428-016-0727-z · Cited by 1,982
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 6×
“Agreeing to participate generated an observation that included the participant’s MTurk WorkerID (for technical details, see Peer, Pao-lacci, Chandler, & Mueller, 2012).”
BackgroundInfluentialConducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
Self-citationChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Self-citationAttributions, beliefs about control, and adjustment to breast cancer.
Taylor, S. E., Lichtman, R. R., & Wood, J. V. (1984) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.46.3.489 · Cited by 936
Interpretation by Physicians of Clinical Laboratory Results
Casscells, W., Schoenberger, A., & Graboys, T. B. (1978) · New England Journal of Medicine · doi:10.1056/nejm197811022991808 · Cited by 755
Social heuristics shape intuitive cooperation
Rand, D. G., Peysakhovich, A., Kraft-Todd, G. T., Newman, G. E., Wurzbacher, O., Nowak, M. A., & Greene, J. D. (2014) · Nature Communications · doi:10.1038/ncomms4677 · Cited by 720
Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of CrowdworkersPreprint
Berg, J. (2016) · RePEc: Research Papers in Economics · Cited by 669
(1984) · Serials Review · doi:10.1080/00987913.1984.10763527 · Cited by 657
Instagram photos reveal predictive markers of depression
Reece, A. G., & Danforth, C. M. (2017) · EPJ Data Science · doi:10.1140/epjds/s13688-017-0110-z · Cited by 519
Financial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463
Zhou, H., & Fishbach, A. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000056 · Cited by 459
Internet Research in Psychology
Gosling, S. D., & Mason, W. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015321 · Cited by 458
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1×
“Worker experience was estimated by summing the total number of Human Intelligence Tasks (HITs) that each worker completed within a large sample of researcher-posted HITs collected in prior years (data taken from Stewart et al., 2015).”
MethodSelf-citationScoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets
Farrell, A. M., Grenier, J. H., & Leiby, J. (2017) · The Accounting Review · doi:10.2308/accr-51447 · Cited by 288
Parent, J., McKee, L. G., N. Rough, J., & Forehand, R. (2016) · Journal of Abnormal Child Psychology · doi:10.1007/s10802-015-9978-x · Cited by 259
Using Nonnaive Participants Can Reduce Effect Sizes
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015) · Psychological Science · doi:10.1177/0956797615585115 · Cited by 185
How Childhood Advertising Exposure Can Create Biased Product Evaluations That Persist into Adulthood
Connell, P. M., Brucks, M. L., & Nielsen, J. H. (2014) · Journal of Consumer Research · doi:10.1086/675218 · Cited by 179
Pearl, R. L., Puhl, R. M., & Dovidio, J. F. (2015) · Journal of Health Psychology · doi:10.1177/1359105313520338 · Cited by 165
Banerjee, A., & Duflo, E. (2017) · North-Holland eBooks · Cited by 156
Jones, M. S., House, L., & Gao, Z. (2015) · Public Opinion Quarterly · doi:10.1093/poq/nfv015 · Cited by 135
Commonalities in grief responding across bereavement and non-bereavement losses
Papa, A., Lancaster, N. G., & Kahler, J. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.03.018 · Cited by 128
Using Mechanical Turk to Study Family Processes and Youth Mental Health: A Test of Feasibility
Schleider, J. L., & Weisz, J. R. (2015) · Journal of Child and Family Studies · doi:10.1007/s10826-015-0126-6 · Cited by 124
Using Nonnaive Participants Can Reduce Effect SizesPreprint
Chandler, J., Paolacci, G., Péer, E., Mueller, P. A., & Ratliff, K. A. (2015) · RePEc: Research Papers in Economics · Cited by 118
Self-citationOpportunity costs of reward delays and the discounting of hypothetical money and cigarettes
Johnson, P. S., Herrmann, E. S., & Johnson, M. W. (2014) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.110 · Cited by 117
Conditioned Superstition: Desire for Control and Consumer Brand Preferences
Hamerman, E. J., & Johar, G. V. (2013) · Journal of Consumer Research · doi:10.1086/670762 · Cited by 108
Concealment and fabrication by experienced research subjects
Devine, E. G., Waters, M., Putnam, M. A., Surprise, C., O’Malley, K., Richambault, C., Fishman, R. L., Knapp, C. M., Patterson, E. H., Sarid‐Segal, O., et al. (2013) · Clinical Trials · doi:10.1177/1740774513492917 · Cited by 106
Arch, J. J. (2013) · Behaviour Research and Therapy · doi:10.1016/j.brat.2013.11.003 · Cited by 87
Honesty in an Online Labor Market
Suri, S., Goldstein, D. G., & Mason, W. (2013) · Cited by 69
Selectively Recruiting Participants from Amazon Mechanical Turk Using QualtricsPreprint
Péer, E., Paolacci, G., Chandler, J., & Mueller, P. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100631 · Cited by 67
Self-citationMobile-only web survey respondents
Lugtig, P. J., Toepoel, V., & Amin, A. (2016) · Survey Practice · doi:10.29115/sp-2016-0020 · Cited by 53
Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.08.020 · Cited by 48
African American Career Aspirations: Examining the Relative Influence of Internalized Racism
Brown, D. L., & Segrist, D. (2016) · Journal of Career Development · doi:10.1177/0894845315586256 · Cited by 47
Using Mechanical Turk for research on cancer survivors
Arch, J. J., & Carr, A. L. (2016) · Psycho-Oncology · doi:10.1002/pon.4173 · Cited by 46
Springer, V. A., Martini, P. J., Lindsey, S. C., & Vezich, I. S. (2016) · Survey Practice · doi:10.29115/sp-2016-0029 · Cited by 34
The Association Between Bullying and Physical Health Among Gay, Lesbian, and Bisexual Individuals
Zou, C., Andersen, J. P., & Blosnich, J. R. (2013) · Journal of the American Psychiatric Nurses Association · doi:10.1177/1078390313510739 · Cited by 27
Fergus, T. A., & Rowatt, W. C. (2014) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2014.09.003 · Cited by 26
Volunteering and Reemployment Status in Unemployed Emerging Adults: A Time-worthy Investment?
Konstam, V., Tomek, S. E., Celen‐Demirtas, S., & Sweeney, K. (2014) · Journal of Career Assessment · doi:10.1177/1069072714523248 · Cited by 22
Yang, K., Friedman‐Wheeler, D. G., & Pronin, E. (2014) · Cognitive Therapy and Research · doi:10.1007/s10608-014-9597-9 · Cited by 14
Cougle, J. R., Hawkins, K. A., Macatee, R. J., Zvolensky, M. J., & Sarawgi, S. (2014) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntu011 · Cited by 14
Cited by
Data quality of platforms and panels for online behavioral research
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01694-3 · Cited by 1,718
MTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018) · Perspectives on Psychological Science · doi:10.1177/1745691617706516 · Cited by 767
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
Self-citationThe shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
“24 Bias may be particularly likely in studies of rare populations, attitudes, or behaviors, as fraudulent respondents may make up a disproportionate share of these rare categories (Chandler and Paolacci, 2017; Lopez and Hillygus, 2018).”
Background
Metadata
- Authors
- Chandler, J. J., & Paolacci, G.
- Venue
- Social Psychological and Personality Science
- Publisher
- SAGE Publications
- Published
- 2017-04-27
- Volume
- 8
- Issue
- 5
- Pages
- 500-508
- ISSN
- 1948-5506 (ISSN-L)
- 1948-5514
- OpenAlex
- W2609226177
- Cited by
- 277
- References
- 43
- FWCI
- 30.25
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.