The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
The False positive problem of automatic bot detection in social science research
Open accessAbstract
The identification of bots is an important and complicated task. The bot classifier "Botometer" was successfully introduced as a way to estimate the number of bots in a given list of accounts and, as a consequence, has been frequently used in academic publications. Given its relevance for academic research and our understanding of the presence of automated accounts in any given Twitter discourse, we are interested in Botometer's diagnostic ability over time. To do so, we collected the Botometer scores for five datasets (three verified as bots, two verified as human; n = 4,134) in two languages (English/German) over three months. We show that the Botometer scores are imprecise when it comes to estimating bots; especially in a different language. We further show in an analysis of Botometer scores over time that Botometer's thresholds, even when used very conservatively, are prone to variance, which, in turn, will lead to false negatives (i.e., bots being classified as humans) and false positives (i.e., humans being classified as bots). This has immediate consequences for academic research as most studies in social science using the tool will unknowingly count a high number of human users as bots and vice versa. We conclude our study with a discussion about how computational social scientists should evaluate machine learning systems that are developed for identifying bots.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“They also demonstrate that Botometer's limits, even when applied very conservatively, are subject to fluctuation, which can result in both false positives and false negatives (i.e., the classification of humans as bots) and false negatives (i.e., the classification of bots as humans).”
From Deepfakes: current and future trends (2024) · cites this paper as Rauchfleisch and Kaiser 2020
The cited source says what the claim attributes to it.
In the citing paper
“Their findings demonstrate that Botometer ratings are inaccurate for estimating bots, particularly when used in an alternate language.”
From Deepfakes: current and future trends (2024) · cites this paper as Rauchfleisch and Kaiser 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 4×
“It is thus fair to say that the tool is arguably the most popular bot detection method in the social sciences and applications of the tools have been published both in field specific journals such as Political Communication [7] as well as high-ranked general-interest journals like Science [8] or Nature Communications…”
BackgroundMethodResultInfluentialThe relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 1×
“While the so-called Matthews correlation coefficient is better than the F1 score if researchers use only one specific threshold, PR curves are recommended [34, 35] if different threshold levels exist as in the case with Botometer.”
BackgroundSaito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1×
“While the percentage agreement is already an interesting indicator, we would highly recommend the use of a measurement like Krippendorff’s alpha [45] that takes into account chance agreement.”
MethodAUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
The spread of low-credibility content by social bots
Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2018) · Nature Communications · doi:10.1038/s41467-018-06930-7 · Cited by 1,125
Social bots distort the 2016 U.S. Presidential election online discussion
Bessi, A., & Ferrara, E. (2016) · First Monday · doi:10.5210/fm.v21i11.7090 · Cited by 993
Online Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981 · cited 5×
“We thus assume for our simulation that the Twitter population has 15% bots [4] and 85% non-bots.”
BackgroundMethodInfluentialTen quick tips for machine learning in computational biology
Chicco, D. (2017) · BioData Mining · doi:10.1186/s13040-017-0155-3 · Cited by 927
BotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909
Sun, X., & Xu, W. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2337313 · Cited by 638
The spread of low-credibility content by social bots
Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2018) · RePEc: Research Papers in Economics · Cited by 574
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419 · cited 4×
“Even though many new bot detection methods are developed every year—outperforming Botometer in some cases—[15, 16, 18], we focus on Botometer as it is by far the most used tool in studies analyzing bots on Twitter [12], especially in social science [5, 7, 10].”
BackgroundMethodInfluentialBots, #Strongerin, and #Brexit: Computational Propaganda During the UK-EU Referendum
Howard, P. N., & Kollanyi, B. (2016) · doi:10.2139/ssrn.2798311 · Cited by 392
The Brexit Botnet and User-Generated Hyperpartisan News
Bastos, M., & Mercea, D. (2019) · Social Science Computer Review · doi:10.1177/0894439317734157 · Cited by 385
The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3055135 · Cited by 381 · cited 9×
“At least researchers should be aware of the data used as training data for the classifier that might only cover certain types of bots [15, 16, 18].”
BackgroundMethodResultInfluentialPolitical Astroturfing on Twitter: How to Coordinate a Disinformation Campaign
Keller, F. B., Schoch, D., Stier, S., & Yang, J. (2020) · Political Communication · doi:10.1080/10584609.2019.1661888 · Cited by 378 · cited 1×
“Overall it is also important to broaden the perspective and focus more on political digital astroturfing [46] for which bots are just one potential dimension [47].”
Background(2020) · doi:10.1609/aaai.v34i01.5460 · Cited by 361
A decade of social bot detection
Cresci, S. (2020) · Communications of the ACM · doi:10.1145/3409116 · Cited by 323 · cited 7×
“Finally, there are other ways to identify bots on Twitter as Cresci [22] and Orabi et al.”
BackgroundInfluentialUnpacking the Social Media Bot: A Typology to Guide Research and Policy
Gorwa, R., & Guilbeault, D. (2020) · Policy & Internet · doi:10.1002/poi3.184 · Cited by 270
DeBot: Twitter Bot Detection via Warped Correlation
Chavoshi, N., Hamooni, H., & Mueen, A. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0096 · Cited by 260
Halligan, S., Altman, D. G., & Mallett, S. (2015) · European Radiology · doi:10.1007/s00330-014-3487-0 · Cited by 255
The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race
Cresci, S., Pietro, R. D., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · doi:10.48550/arxiv.1701.03017 · Cited by 237
Detection of Bots in Social Media: A Systematic Review
Orabi, M., Mouheb, D., Aghbari, Z. A., & Kamel, I. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102250 · Cited by 232
Social Bots in Election Campaigns: Theoretical, Empirical, and Methodological Implications
Keller, T. R., & Klinger, U. (2018) · Political Communication · doi:10.1080/10584609.2018.1526238 · Cited by 216
(2020) · Sabanci University · Cited by 197
Transparent and Reproducible Social Science Research
Christensen, G., Jeremy Freese, A., & Miguel, E. A. (2019) · University of California Press eBooks · doi:10.1525/9780520969230 · Cited by 171
Social fingerprinting: detection of spambot groups through DNA-inspired behavioral modeling
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2681672 · Cited by 168
RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter
Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., & Tesconi, M. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326015 · Cited by 163
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3412698 · Cited by 159 · cited 2×
“The Botometer creators [43] argue in their newest paper that some of the German politicians’ accounts used in our study might have a certain degree of automation.”
BackgroundDigital astroturfing in politics: Definition, typology, and countermeasures
Kovic, M., Rauchfleisch, A., Sele, M., & Caspar, C. (2018) · Studies in Communication Sciences · doi:10.24434/j.scoms.2018.01.005 · Cited by 149
Self-citationThe global organization of social media disinformation campaigns
Howard, P., & Bradshaw, S. (2018) · Journal of international affairs · Cited by 146
Transparent and Reproducible Social Science Research: How to Do Open Science
Christensen, G., Freese, J., & Miguel, E. A. (2019) · Cited by 129
Bradshaw, S., Howard, P. N., Kollányi, B., & Neudert, L. (2019) · Political Communication · doi:10.1080/10584609.2019.1663322 · Cited by 118
Are Social Bots on Twitter Political Actors? Empirical Evidence from a Ukrainian Social Botnet
Hegelich, S., & Janetzko, D. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i1.14764 · Cited by 96
Changing Perspectives: Is It Sufficient to Detect Social Bots?
Grimme, C., Assenmacher, D., & Adam, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-91521-0_32 · Cited by 93
Whose Lives Matter? Mass Shootings and Social Media Discourses of Sympathy and Policy, 2012–2014
Zhang, Y., Shah, D. V., Foley, J. M., Abhishek, A., Lukito, J., Suk, J., Kim, S. J., Sun, Z., Pevehouse, J. C. W., & Garlough, C. (2019) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmz009 · Cited by 92
BotChase: Graph-Based Bot Detection Using Machine Learning
Daya, A. A., Salahuddin, M. A., Limam, N., & Boutaba, R. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.2972405 · Cited by 74
On the Detection of Disinformation Campaign Activity with Network Analysis
Vargas, L., Emami, P., & Traynor, P. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop · doi:10.1145/3411495.3421363 · Cited by 74
Political Astroturfing on Twitter: How to Coordinate a Disinformation CampaignPreprint
Keller, F., Schoch, D., Stier, S., & Yang, J. (2019) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/a5gk6 · Cited by 66
LOBO: Evaluation of Generalization Deficiencies in Twitter Bot Classifiers
Echeverra, J., De Cristofaro, E., Kourtellis, N., Leontiadis, I., Stringhini, G., & Zhou, S. (2018) · Proceedings of the 34th Annual Computer Security Applications Conference · doi:10.1145/3274694.3274738 · Cited by 61
Political Bots and the Swedish General Election
Fernquist, J., Kaati, L., & Schroeder, R. (2018) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi.2018.8587347 · Cited by 57
Evolution of bot and human behavior during elections
Luceri, L., Deb, A., Giordano, S., & Ferrara, E. (2019) · First Monday · doi:10.5210/fm.v24i9.10213 · Cited by 56
“How Polling Becomes News: Communicating the Counting of Public Opinion”
Frankovic, K. A. (1998) · Political Communication · doi:10.1080/10584609.1998.11672656 · Cited by 7
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Fernández DM, Graziotin D, Wagner S, Seibold H. Open Science in Software Engineering; 2019. https://arxiv.org/abs/1904.06499. · arXiv
Cited by
Sorted by citations
Botometer 101: social bot practicum for computational social scientists
Yang, K.-C., Ferrara, E., & Menczer, F. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00177-5 · Cited by 159
Deepfakes: current and future trends
Gambín, Á. F., Yazidi, A., Vasilakos, A., Haugerud, H., & Djenouri, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10679-x · Cited by 145
He, L., He, C., Reynolds, T. L., Bai, Q., Huang, Y., Li, C., Zheng, K., & Chen, Y. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab047 · Cited by 110
Why Botter: How Pro-Government Bots Fight Opposition in Russia
Стукал, Д., Sanovich, S., Bonneau, R., & Tucker, J. A. (2022) · American Political Science Review · doi:10.1017/s0003055421001507 · Cited by 98
DeeProBot: a hybrid deep neural network model for social bot detection based on user profile data
Hayawi, K., Mathew, S. S., Venugopal, N., Masud, M. M., & Ho, P. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00869-w · Cited by 98
Metadata
- Authors
- Rauchfleisch, A., & Kaiser, J.
- Affiliations
- National Taiwan University (TW)
- Harvard University (US)
- Corresponding author
- Rauchfleisch, Adrian
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-10-22
- Volume
- 15
- Issue
- 10
- Pages
- e0241045
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7580919
- OpenAlex
- W3014935876
- Cited by
- 184
- References
- 49
- Citations by year
- 2026: 10 · 2025: 23 · 2024: 39 · 2023: 31 · 2022: 47 · 2021: 27 · 2020: 7
- FWCI
- 26.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Ministry of Science and Technology, Taiwan (108-2410-H-002 -007 -MY2)
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- bot detection, Botometer, false positives, false-negative results, Twitter, language influence, computational social science
- MeSH terms
- Machine Learning, Humans, Politics, Research, Social Sciences, Social Networking, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
