Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 186

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures

Dennis, Sean A., Goodson, Brian M., Pearson, Christopher A.·2020·Behavioral Research in Accounting

Abstract

We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data—and discuss how this issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are likely insufficient to identify and mitigate emergent risks around data integrity. We discuss MTurk samples for two studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods—and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers now help target this abuse, the underlying limitations of IP screening procedures remain. Our findings emphasize the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2308/bria-18-044

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 24

  1. Asay, H. S. , ElliottW. B., and RennekampK. 2017. Disclosure readability and the sensitivity of investors' valuation judgments to outside information. The Accounting Review92 ( 4): 1– 25. https://doi.org/10.2308/accr-51570 · DOI · Paper page
  2. Brandon, D. M. , LongJ. H., LoraasT. M., Mueller-PhillipsJ., and VansantB. 2014. Online instrument delivery and participant recruitment services: Emerging opportunities for behavioral accounting research. Behavioral Research in Accounting26 ( 1): 1– 23. https://doi.org/10.2308/bria-50651 · DOI · Paper page
  3. Brink, W. D. , LeeL. S., and PyzohaJ. S. 2019. Values of participants in behavioral accounting research: A comparison of the M-Turk population to a nationally representative sample. Behavioral Research in Accounting31 ( 1): 97– 117. https://doi.org/10.2308/bria-52103 · DOI
  4. Brink, W. D. , EatonT. V., GrenierJ. H., and ReffettA. 2017. Deterring unethical behavior in online labor markets. Journal of Business Ethics ( May): 1– 18. · DOI
  5. Buchheit, S. , DaltonD. W., PollardT., and StinsonS. 2019. Crowdsourcing intelligent research participants: A student versus MTurk comparison. Behavioral Research in Accounting31(2): 93– 106. · DOI
  6. Buchheit, S. , DoxeyM., PollardT., and StinsonS. 2018. A technical guide to using Amazon's Mechanical Turk in behavioral accounting research. Behavioral Research in Accounting30 ( 1): 111– 122. https://doi.org/10.2308/bria-51977 · DOI
  7. Buhrmester, M. D. , TalaifarS., and GoslingS. D. 2018. An evaluation of Amazon's Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science13 ( 2): 149– 154. https://doi.org/10.1177/1745691617706516 · DOI · Paper page
  8. Chandler, J. J. , and PaolacciG. 2017. Lie for a dime: When most prescreening responses are honest but most study participants are imposters. Social Psychological & Personality Science8 ( 5): 500– 508. https://doi.org/10.1177/1948550617698203 · DOI · Paper page
  9. Cooper, D. J. , DacinT., and PalmerD. 2013. Fraud in accounting, organizations, and society: Extending the boundaries of research. Accounting, Organizations and Society38 ( 6-7): 440– 457. https://doi.org/10.1016/j.aos.2013.11.001 · DOI · Paper page
  10. Farrell, A. M. , GrenierJ. H., and LeibyJ. 2017. Scoundrels or stars? Theory and evidence on the quality of workers in online labor markets. The Accounting Review92 ( 1): 93– 114. https://doi.org/10.2308/accr-51447 · DOI · Paper page
  11. Grenier, J. H. , ReffettA., SimonC. A., and WarneR. C. 2018. Researching juror judgment and decision making in cases of alleged auditor negligence: A toolkit for new scholars. Behavioral Research in Accounting30 ( 1): 99– 110. https://doi.org/10.2308/bria-51878 · DOI
  12. Griffith, E. , HammersleyJ. S., KadousK., and YoungD. 2015. Audits of complex estimates as verification of management numbers: How institutional pressures shape practice. Contemporary Accounting Research32 ( 3): 833– 863. https://doi.org/10.1111/1911-3846.12104 · DOI · Paper page
  13. Haug, C. J. 2015. Peer-review fraud—Hacking the scientific publication process. The New England Journal of Medicine373 ( 25): 2393– 2395. https://doi.org/10.1056/NEJMp1512330 · DOI · Paper page
  14. Hunt, N. C. , and ScheetzA. M. 2018. Using MTurk to distribute a survey or experiment: Methodological considerations. Journal of Information Systems33(1): 43– 65. · DOI
  15. Kan, I. P. , and DrummeyA. P. 2018. Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce. Computers in Human Behavior83: 243– 253. https://doi.org/10.1016/j.chb.2018.02.005 · DOI
  16. Kang, R. , BrownS., DabbishL., and KieslerS. 2014. Privacy attitudes of Mechanical Turk workers and the U.S. public. In 10th Symposium on Usable Privacy and Security. Menlo Park, CA: USENIX.
  17. Kennedy, R. , CliffordS., BurleighT., JewellR., and WaggonerP. 2018. The shape of and solutions to the MTurk quality crisis. Working paper, University of Houston and College of William & Mary. · DOI
  18. Matherly, T. 2019. A panel for lemons? Positivity bias, reputation systems and data quality on MTurk. European Journal of Marketing53 ( 2): 195– 223. https://doi.org/10.1108/EJM-07-2017-0491 · DOI
  19. McCallister, E. , GranceT., and ScarfoneK. A. 2010. Guide to Protecting the Confidentiality of Personally Identifiable Information (PII). Gaithersburg, MD.: National Institute of Standards and Technology. · DOI · Paper page
  20. Murphy, P. R. , and FreeC. 2016. Broadening the fraud triangle: Instrumental climate and fraud. Behavioral Research in Accounting28 ( 1): 41– 56. https://doi.org/10.2308/bria-51083 · DOI
  21. Raven, J. , and RavenJ. C. 2003. Raven progressive matrices. InHandbook of Nonverbal Assessment, edited byS. McCallumR. , 223– 237. Boston, MA: Springer US. · DOI · Paper page
  22. Raven, J. , RavenJ. C., and CourtJ. H. 2000. Standard Progressive Matrices. Oxford, U.K.: Psychology Press.
  23. Rennekamp, K. 2012. Processing fluency and investors' reactions to disclosure readability. Journal of Accounting Research50 ( 5): 1319– 1354. https://doi.org/10.1111/j.1475-679X.2012.00460.x · DOI · Paper page
  24. Thibodeau, J. C. , WilliamsL. T., and WitteA. L. 2019. Point and click data: An assessment of editorial perceptions and recommendations for the peer-review process in the new data frontier. Journal of Information Systems33(1): 129– 144. · DOI

Cited by

See all 186

Metadata

Authors
Dennis, S. A., Goodson, B. M., & Pearson, C. A.
Venue
Behavioral Research in Accounting
Published
2019-12-01
Volume
32
Issue
1
Pages
119-134
ISSN
  • 1050-4753 (ISSN-L)
  • 1558-8009
OpenAlex
W2900420874
Cited by
186
References
1
FWCI
21.24
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures. https://app.cochat.ai/papers/10.2308/bria-18-044
The paper
Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020). Online Worker Fraud and Evolving Threats to the Integrity of MTurk Data: A Discussion of Virtual Private Servers and the Limitations of IP-Based Screening Procedures. Behavioral Research in Accounting. https://doi.org/10.2308/bria-18-044

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.