The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks
Open accessAbstract
The Internet has been a popular source of data amongst academic researchers for many years, and for good reason.Online data collection is fast, provides access to hard-to-reach populations, and is often less expensive than in-lab recruitment.With these benefits also come risks, such as duplicate responses or participant inattention, which can significantly reduce data quality.Very recently, researchers have become aware of another concern associated with online data collection.Bots, also known as automatic survey-takers or fraudsters, have begun infiltrating scientific surveys, largely threatening the integrity of academic research conducted online.The aim of this paper is to warn researchers of the threat posed by bots and to highlight practical strategies that can be used to detect and prevent these bots.We first discuss strategies recommended in the literature that we implemented to identify bot responses from online survey data we collected in the past six months.We then share which strategies proved to be most and least effective in detecting bots.Finally, we discuss the implications of bot-generated data for the integrity of online research and the imminent future of bots in online data collection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Identifying careless responses in survey data.
Meade, A. W., & Craig, S. B. (2012) · Psychological Methods · doi:10.1037/a0028085 · Cited by 3,865 · cited 4×
“…structured in two ways: (a) as directed questions, requiring participants to provide a specific response (Huang et al., 2012; Maniaci & Rogge, 2014; Meade & Craig, 2012); and (b) as bogus questions, requiring participants to respond to items that everyone should agree to (i.e., universal truths…”
BackgroundInfluentialWright, K. B. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00259.x · Cited by 2,550
Detecting and Deterring Insufficient Effort Responding to Surveys
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012) · Journal of Business and Psychology · doi:10.1007/s10869-011-9231-8 · Cited by 1,295
An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027
Chandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950
Caring about carelessness: Participant inattention and its effects on research
Maniaci, M. R., & Rogge, R. D. (2014) · Journal of Research in Personality · doi:10.1016/j.jrp.2013.09.008 · Cited by 852
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 761
Ascertaining the validity of individual protocols from Web-based personality inventories
Johnson, J. A. (2005) · Journal of Research in Personality · doi:10.1016/j.jrp.2004.09.009 · Cited by 617
Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and Tradeoffs
Teitcher, J. E. F., Bockting, W. O., Bauermeister, J. A., Hoefer, C. J., Miner, M. H., & Klitzman, R. L. (2015) · Journal of Law, Medicine & Ethics · doi:10.1111/jlme.12200 · Cited by 542
AN EMPIRICAL ASSESSMENT OF DATA COLLECTION USING THE INTERNET
STANTON, J. M. (1998) · Personnel Psychology · doi:10.1111/j.1744-6570.1998.tb00259.x · Cited by 531
Insufficient effort responding: Examining an insidious confound in survey data.
Huang, J. L., Liu, M., & Bowling, N. A. (2015) · Journal of Applied Psychology · doi:10.1037/a0038510 · Cited by 416
A reliability analysis of Mechanical Turk data
Rouse, S. V. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2014.11.004 · Cited by 414
Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis
Woods, C. M. (2006) · Journal of Psychopathology and Behavioral Assessment · doi:10.1007/s10862-005-9004-7 · Cited by 400
Effects of Questionnaire Length on Response Quality
Herzog, A. R., & Bachman, J. G. (1981) · Public Opinion Quarterly · doi:10.1086/268687 · Cited by 392
Pozzar, R., Hammer, M. J., Underhill-Blazey, M., Wright, A. A., Tulsky, J. A., Hong, F., Gundersen, D. A., & Berry, D. L. (2020) · Journal of Medical Internet Research · doi:10.2196/23021 · Cited by 314
The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015) · Judgment and Decision Making · doi:10.1017/s1930297500005611 · Cited by 304 · cited 1×
“Additionally, researchers have highlighted the concern of professional participants, or “super workers,” and bots in these sample pools (Chandler, Mueller, & Paolacci, 2014; Chmielewski & Kucker, 2019; Dennis et al., 2019; Fort, Adda, & Cohen, 2011; Stewart et al., 2015).”
BackgroundEvidence for response bias as a source of error variance in applied assessment.
McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010) · Psychological Bulletin · doi:10.1037/a0019216 · Cited by 282
Chandler, J. J., & Paolacci, G. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550617698203 · Cited by 277
Methods to detect low quality data and its implication for psychological research
Buchanan, E. M., & Scofield, J. E. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1035-6 · Cited by 254
Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2018) · Journal of Business and Psychology · doi:10.1007/s10869-016-9479-0 · Cited by 230
Examining Completion Rates in Web Surveys via Over 25,000 Real-World Surveys
Liu, M., & Wronski, L. (2018) · Social Science Computer Review · doi:10.1177/0894439317695581 · Cited by 223
Whitehead, L. C. (2007) · Social Science & Medicine · doi:10.1016/j.socscimed.2007.03.005 · Cited by 209
Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000085 · Cited by 206
Methodological Challenges in Online Trials
Murray, E., Khadjesari, Z., White, I. R., Kalaitzaki, E., Godfrey, C., McCambridge, J., Thompson, S. G., & Wallace, P. (2009) · Journal of Medical Internet Research · doi:10.2196/jmir.1052 · Cited by 195
Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020) · Behavioral Research in Accounting · doi:10.2308/bria-18-044 · Cited by 186
The Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research
Konstan, J. A., Simon Rosser, B. R., Ross, M. W., Stanton, J., & Edwards, W. M. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2005.tb00248.x · Cited by 170
Assessing the reliability of the M5-120 on Amazon’s mechanical Turk
Holden, C. J., Dennie, T., & Hicks, A. D. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.02.020 · Cited by 169
Prepaid and Promised Incentives in Web Surveys: An Experiment
Bosnjak, M., & Tuten, T. L. (2003) · Social Science Computer Review · doi:10.1177/0894439303021002006 · Cited by 168
Identifying Multiple Submissions in Internet Research: Preserving Data Integrity
Bowen, A. M., Daniel, C. M., Williams, M. L., et al. (2008) · AIDS and Behavior · doi:10.1007/s10461-007-9352-2 · Cited by 165
Random Responding as a Threat to the Validity of Effect Size Estimates in Correlational Research
Credé, M. (2010) · Educational and Psychological Measurement · doi:10.1177/0013164410366686 · Cited by 160
Conducting an Internet-based Survey: Benefits, Pitfalls, and Lessons Learned
Alessi, E. J., & Martin, J. I. (2010) · Social Work Research · doi:10.1093/swr/34.2.122 · Cited by 159
Strategies to address participant misrepresentation for eligibility in Web‐based research
Kramer, J. M., Rubin, A., Coster, W. J., Helmuth, E., Hermos, J. A., Rosenbloom, D., Moed, R., Dooley, M., Kao, Y., Liljenquist, K. S., et al. (2014) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1415 · Cited by 138
Internet-Based Psychological Experimenting: Five Dos and Five Don’ts
Reips, U. (2002) · Social Science Computer Review · doi:10.1177/08939302020003002 · Cited by 135
Participants at Your Fingertips
Johnson, D. R., & Borden, L. A. (2012) · Teaching of Psychology · doi:10.1177/0098628312456615 · Cited by 119
What’s going on? Age, distraction, and multitasking during online survey taking
Zwarun, L., & Hall, A. E. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.09.041 · Cited by 112
Dupuis, M., Meier, E., & Cuneo, F. (2018) · Behavior Research Methods · doi:10.3758/s13428-018-1103-y · Cited by 112
How serious is the ‘carelessness’ problem on Mechanical Turk?
Aruguete, M. S., Huynh, H. P., Browne, B. L., Jurs, B. S., Flint, E., & McCutcheon, L. E. (2019) · International Journal of Social Research Methodology · doi:10.1080/13645579.2018.1563966 · Cited by 89
Out Damn Bot, Out: Recruiting Real People into Substance Use Studies on the Internet
Godinho, A., Schell, C., & Cunningham, J. A. (2019) · Substance Abuse · doi:10.1080/08897077.2019.1691131 · Cited by 86
McGonagle, A. K., Huang, J. L. W., & Walsh, B. M. (2015) · Applied Psychology · doi:10.1111/apps.12058 · Cited by 75
Cited by
Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 269
Who Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
Marshall, C. C., Goguladinne, P. S. R., Maheshwari, M., Sathe, A., & Shipman, F. M. (2023) · Proceedings of the 15th ACM Web Science Conference 2023 · doi:10.1145/3578503.3583622 · Cited by 20
Nakonechny, L., Cisneros, A., Moody, C. M., & Stellato, A. C. (2025) · Frontiers in Veterinary Science · doi:10.3389/fvets.2025.1634970 · Cited by 3
“Responses suspected of being from automated bots were identified and removed based on speed of survey completion and open-text anomalies, such as non-sensical text answers (46, 47).”
Metadata
- Authors
- Storozuk, A., Ashley, M., Delage, V., & Maloney, E. A.
- Venue
- The Quantitative Methods for Psychology
- Published
- 2020-05-01
- Volume
- 16
- Issue
- 5
- Pages
- 472-481
- ISSN
- 2292-1354 (ISSN-L)
- OpenAlex
- W3115100653
- Cited by
- 324
- References
- 40
- FWCI
- 26.77
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.