The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures
Abstract
We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data—and discuss how this issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are likely insufficient to identify and mitigate emergent risks around data integrity. We discuss MTurk samples for two studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods—and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers now help target this abuse, the underlying limitations of IP screening procedures remain. Our findings emphasize the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MTurk Character Misrepresentation: Assessment and Solutions
Sharpe Wessling, K., Huber, J., & Netzer, O. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx053 · Cited by 330
Research works printed but not linked: 24
- Asay, H. S. , ElliottW. B., and RennekampK. 2017. Disclosure readability and the sensitivity of investors' valuation judgments to outside information. The Accounting Review92 ( 4): 1– 25. https://doi.org/10.2308/accr-51570 · DOI · Paper page
- Brandon, D. M. , LongJ. H., LoraasT. M., Mueller-PhillipsJ., and VansantB. 2014. Online instrument delivery and participant recruitment services: Emerging opportunities for behavioral accounting research. Behavioral Research in Accounting26 ( 1): 1– 23. https://doi.org/10.2308/bria-50651 · DOI · Paper page
- Brink, W. D. , LeeL. S., and PyzohaJ. S. 2019. Values of participants in behavioral accounting research: A comparison of the M-Turk population to a nationally representative sample. Behavioral Research in Accounting31 ( 1): 97– 117. https://doi.org/10.2308/bria-52103 · DOI
- Brink, W. D. , EatonT. V., GrenierJ. H., and ReffettA. 2017. Deterring unethical behavior in online labor markets. Journal of Business Ethics ( May): 1– 18. · DOI
- Buchheit, S. , DaltonD. W., PollardT., and StinsonS. 2019. Crowdsourcing intelligent research participants: A student versus MTurk comparison. Behavioral Research in Accounting31(2): 93– 106. · DOI
- Buchheit, S. , DoxeyM., PollardT., and StinsonS. 2018. A technical guide to using Amazon's Mechanical Turk in behavioral accounting research. Behavioral Research in Accounting30 ( 1): 111– 122. https://doi.org/10.2308/bria-51977 · DOI
- Buhrmester, M. D. , TalaifarS., and GoslingS. D. 2018. An evaluation of Amazon's Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science13 ( 2): 149– 154. https://doi.org/10.1177/1745691617706516 · DOI · Paper page
- Chandler, J. J. , and PaolacciG. 2017. Lie for a dime: When most prescreening responses are honest but most study participants are imposters. Social Psychological & Personality Science8 ( 5): 500– 508. https://doi.org/10.1177/1948550617698203 · DOI · Paper page
- Cooper, D. J. , DacinT., and PalmerD. 2013. Fraud in accounting, organizations, and society: Extending the boundaries of research. Accounting, Organizations and Society38 ( 6-7): 440– 457. https://doi.org/10.1016/j.aos.2013.11.001 · DOI · Paper page
- Farrell, A. M. , GrenierJ. H., and LeibyJ. 2017. Scoundrels or stars? Theory and evidence on the quality of workers in online labor markets. The Accounting Review92 ( 1): 93– 114. https://doi.org/10.2308/accr-51447 · DOI · Paper page
- Grenier, J. H. , ReffettA., SimonC. A., and WarneR. C. 2018. Researching juror judgment and decision making in cases of alleged auditor negligence: A toolkit for new scholars. Behavioral Research in Accounting30 ( 1): 99– 110. https://doi.org/10.2308/bria-51878 · DOI
- Griffith, E. , HammersleyJ. S., KadousK., and YoungD. 2015. Audits of complex estimates as verification of management numbers: How institutional pressures shape practice. Contemporary Accounting Research32 ( 3): 833– 863. https://doi.org/10.1111/1911-3846.12104 · DOI · Paper page
- Haug, C. J. 2015. Peer-review fraud—Hacking the scientific publication process. The New England Journal of Medicine373 ( 25): 2393– 2395. https://doi.org/10.1056/NEJMp1512330 · DOI · Paper page
- Hunt, N. C. , and ScheetzA. M. 2018. Using MTurk to distribute a survey or experiment: Methodological considerations. Journal of Information Systems33(1): 43– 65. · DOI
- Kan, I. P. , and DrummeyA. P. 2018. Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce. Computers in Human Behavior83: 243– 253. https://doi.org/10.1016/j.chb.2018.02.005 · DOI
- Kang, R. , BrownS., DabbishL., and KieslerS. 2014. Privacy attitudes of Mechanical Turk workers and the U.S. public. In 10th Symposium on Usable Privacy and Security. Menlo Park, CA: USENIX.
- Kennedy, R. , CliffordS., BurleighT., JewellR., and WaggonerP. 2018. The shape of and solutions to the MTurk quality crisis. Working paper, University of Houston and College of William & Mary. · DOI
- Matherly, T. 2019. A panel for lemons? Positivity bias, reputation systems and data quality on MTurk. European Journal of Marketing53 ( 2): 195– 223. https://doi.org/10.1108/EJM-07-2017-0491 · DOI
- McCallister, E. , GranceT., and ScarfoneK. A. 2010. Guide to Protecting the Confidentiality of Personally Identifiable Information (PII). Gaithersburg, MD.: National Institute of Standards and Technology. · DOI · Paper page
- Murphy, P. R. , and FreeC. 2016. Broadening the fraud triangle: Instrumental climate and fraud. Behavioral Research in Accounting28 ( 1): 41– 56. https://doi.org/10.2308/bria-51083 · DOI
- Raven, J. , and RavenJ. C. 2003. Raven progressive matrices. InHandbook of Nonverbal Assessment, edited byS. McCallumR. , 223– 237. Boston, MA: Springer US. · DOI · Paper page
- Raven, J. , RavenJ. C., and CourtJ. H. 2000. Standard Progressive Matrices. Oxford, U.K.: Psychology Press.
- Rennekamp, K. 2012. Processing fluency and investors' reactions to disclosure readability. Journal of Accounting Research50 ( 5): 1319– 1354. https://doi.org/10.1111/j.1475-679X.2012.00460.x · DOI · Paper page
- Thibodeau, J. C. , WilliamsL. T., and WitteA. L. 2019. Point and click data: An assessment of editorial perceptions and recommendations for the peer-review process in the new data frontier. Journal of Information Systems33(1): 129– 144. · DOI
Cited by
Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks
Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A. (2020) · The Quantitative Methods for Psychology · doi:10.20982/tqmp.16.5.p472 · Cited by 324
Ensuring survey research data integrity in the era of internet bots
Griffin, M., Martino, R. J., LoSchiavo, C., Comer-Carruthers, C., Krause, K. D., Stults, C. B., & Halkitis, P. N. (2022) · Quality & Quantity · doi:10.1007/s11135-021-01252-1 · Cited by 314
Pozzar, R., Hammer, M. J., Underhill-Blazey, M., Wright, A. A., Tulsky, J. A., Hong, F., Gundersen, D. A., & Berry, D. L. (2020) · Journal of Medical Internet Research · doi:10.2196/23021 · Cited by 314
Data Collection via Online Platforms: Challenges and Recommendations for Future Research
Newman, A., Bavik, Y. L., Mount, M., & Shao, B. (2021) · Applied Psychology · doi:10.1111/apps.12302 · Cited by 306
Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 269
Metadata
- Authors
- Dennis, S. A., Goodson, B. M., & Pearson, C. A.
- Venue
- Behavioral Research in Accounting
- Publisher
- American Accounting Association
- Published
- 2019-12-01
- Volume
- 32
- Issue
- 1
- Pages
- 119-134
- ISSN
- 1050-4753 (ISSN-L)
- 1558-8009
- OpenAlex
- W2900420874
- Cited by
- 186
- References
- 1
- FWCI
- 21.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures. https://app.cochat.ai/papers/10.2308/bria-18-044
- The paper
- Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020). Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures. Behavioral Research in Accounting. https://doi.org/10.2308/bria-18-044
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.