The DOI matches its Crossref record; 47 of the 49 references it cites were found in a registry or an index.
The False positive problem of automatic bot detection in social science research
Open accessAbstract
The identification of bots is an important and complicated task. The bot classifier "Botometer" was successfully introduced as a way to estimate the number of bots in a given list of accounts and, as a consequence, has been frequently used in academic publications. Given its relevance for academic research and our understanding of the presence of automated accounts in any given Twitter discourse, we are interested in Botometer's diagnostic ability over time. To do so, we collected the Botometer scores for five datasets (three verified as bots, two verified as human; n = 4,134) in two languages (English/German) over three months. We show that the Botometer scores are imprecise when it comes to estimating bots; especially in a different language. We further show in an analysis of Botometer scores over time that Botometer's thresholds, even when used very conservatively, are prone to variance, which, in turn, will lead to false negatives (i.e., bots being classified as humans) and false positives (i.e., humans being classified as bots). This has immediate consequences for academic research as most studies in social science using the tool will unknowingly count a high number of human users as bots and vice versa. We conclude our study with a discussion about how computational social scientists should evaluate machine learning systems that are developed for identifying bots.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“They also demonstrate that Botometer's limits, even when applied very conservatively, are subject to fluctuation, which can result in both false positives and false negatives (i.e., the classification of humans as bots) and false negatives (i.e., the classification of bots as humans).”
From Deepfakes: current and future trends (2024) · cites this paper as Rauchfleisch and Kaiser 2020
The cited source says what the claim attributes to it.
In the citing paper
“Their findings demonstrate that Botometer ratings are inaccurate for estimating bots, particularly when used in an alternate language.”
From Deepfakes: current and future trends (2024) · cites this paper as Rauchfleisch and Kaiser 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 4×
“It is thus fair to say that the tool is arguably the most popular bot detection method in the social sciences and applications of the tools have been published both in field specific journals such as Political Communication [7] as well as high-ranked general-interest journals like Science [8] or Nature Communications…”
BackgroundMethodResultInfluentialThe relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 1×
“While the so-called Matthews correlation coefficient is better than the F1 score if researchers use only one specific threshold, PR curves are recommended [34, 35] if different threshold levels exist as in the case with Botometer.”
BackgroundSaito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1×
“While the percentage agreement is already an interesting indicator, we would highly recommend the use of a measurement like Krippendorff’s alpha [45] that takes into account chance agreement.”
MethodAUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
The spread of low-credibility content by social bots
Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2018) · Nature Communications · doi:10.1038/s41467-018-06930-7 · Cited by 1,125
Social bots distort the 2016 U.S. Presidential election online discussion
Bessi, A., & Ferrara, E. (2016) · First Monday · doi:10.5210/fm.v21i11.7090 · Cited by 993
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Fernández DM, Graziotin D, Wagner S, Seibold H. Open Science in Software Engineering; 2019. https://arxiv.org/abs/1904.06499. · arXiv
Cited by
Sorted by citations
Items 101–183 of 183
Unveiling the Veiled Threat: The Impact of Bots on COVID-19 Health Communication
Ünlü, A., Truong, S., Sawhney, N., & Tammi, T. (2024) · Social Science Computer Review · doi:10.1177/08944393241275641 · Cited by 8
The Scamdemic Conspiracy Theory and Twitter’s Failure to Moderate COVID-19 Misinformation
Rossi, S. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.015 · Cited by 6
Jones, K., Nurse, J. R. C., & Li, S. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19303 · Cited by 6
Chomel, V., Panahi, M., & Chavalarias, D. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-21127-0_11 · Cited by 7
From Online Behaviours to Images: A Novel Approach to Social Bot Detection
Paolo, E. D., Petrocchi, M., & Spognardi, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35995-8_42 · Cited by 6
Cola, G., Mazza, M., & Tesconi, M. (2023) · IEEE Access · doi:10.1109/access.2023.3282580 · Cited by 7
Daume, S., Bjersér, P., & Galaz, V. (2023) · Current Opinion in Environmental Sustainability · doi:10.1016/j.cosust.2023.101384 · Cited by 10
Post-hoc Evaluation of Nodes Influence in Information Cascades: The Case of Coordinated Accounts
Marco, N. D., Brunetti, S., Cinelli, M., & Quattrociocchi, W. (2024) · ACM Transactions on the Web · doi:10.1145/3700644 · Cited by 8
Attributing coordinated social media manipulation: A theoretical model and typology
Thiele, D., Milzner, M., Heft, A., Gong, B., & Pfetsch, B. (2025) · New Media & Society · doi:10.1177/14614448251350100 · Cited by 7
Detecting and Measuring the Polarization Effects of Adversarial Botnets on Twitter
Lee, Y., Özer, M., Corman, S. R., & Davulcu, H. (2023) · ACM Symposium on Applied Computing · doi:10.1145/3555776.3577730 · Cited by 5
You Are a Bot! – Studying the Development of Bot Accusations on Twitter
Assenmacher, D., Fröhling, L., & Wagner, C. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31301 · Cited by 5
Security of Social Networks: Lessons Learned on Twitter Bot Analysis in the Literature
Alipour, S. A., Orji, R., & Zincir-Heywood, N. (2022) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3538969.3544450 · Cited by 5
Populismo de extrema derecha y redes sociales: ¿el futuro de la democracia en juego?
Capdevila Gómez, A., Casero Ripollés, A., Cozzo, E., Gayo Avello, D., Gerbaudo, P., Guerrero-Solé, F., Manchón, L. M., Moragas, F., Perales-García, C., Pérez-Altable, L., et al. (2023) · doi:10.31009/cr.2023.01 · Cited by 6
Ridge count thresholding to uncover coordinated networks during onset of the Covid-19 pandemic
Kirn, S. L., & Hinders, M. K. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00873-0 · Cited by 5
Don’t Shoot the Message: Regulating Disinformation Beyond Content
Keller, C. I. (2022) · Law, governance and technology series · doi:10.1007/978-3-031-07377-9_16 · Cited by 4
Identifying Behavioral Factors Leading to Differential Polarization Effects of Adversarial Botnets
Lee, Y., Özer, M., Corman, S. R., & Davulcu, H. (2023) · ACM SIGAPP Applied Computing Review · doi:10.1145/3610409.3610412 · Cited by 4
Forschungsgemeinschaft, D., & Gredel, E. (2022) · De Gruyter eBooks · doi:10.1515/9783110721447 · Cited by 5
Tourism content on Twitter (X) during a crisis
Garay, L., & Roelofsen, M. (2024) · Annals of Tourism Research Empirical Insights · doi:10.1016/j.annale.2024.100132 · Cited by 4
Social media and electoral prediction: ten years after
Gayo-Avello, D., Jungherr, A. M., & Rivero, G. (2025) · Edward Elgar Publishing eBooks · doi:10.4337/9781802207309.00040 · Cited by 3
Aguiar, J. C. D. (2025) · Aracê. · doi:10.56238/arev7n7-282 · Cited by 3
Bots don’t Vote, but They Surely Bother!: A Study of Anomalous Accounts in a National Referendum
Graells-Garrido, E., & Baeza-Yates, R. (2022) · Web Science Conference · doi:10.1145/3501247.3531576 · Cited by 8
The Platformisation of Consumer Culture
Caliandro, A., Gandini, A., Bainotti, L., & Anselmi, G. (2024) · Routledge eBooks · doi:10.5117/9789463729567 · Cited by 5
An Analysis of Social Bot Activity on X in Modern Japan
Ippa, S., Okubo, T., & Hashimoto, M. (2024) · IEEE Access · doi:10.1109/access.2024.3454536 · Cited by 3
Zeng, J., & Fang, H. (2025) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565251335412 · Cited by 5
Properties of Malicious Social Bots
Kolomeets, M., & Chechulin, A. A. (2023) · Proceedings of Telecommunication Universities · doi:10.31854/1813-324x-2023-9-1-94-104 · Cited by 3
Casas, A. (2024) · Political Communication · doi:10.1080/10584609.2024.2306503 · Cited by 6
Integrating higher-order relations for enhanced twitter bot detection
Reiche, S., Cohen, S., Simonov, K., & Friedrich, T. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01372-0 · Cited by 3
García-Rodríguez, A., Govezensky, T., Gershenson, C., Naumis, G. G., & Barrio, R. A. (2021) · Advances in Complex Systems · doi:10.1142/s021952592150017x · Cited by 6
Editorial: Five challenges in detection and mitigation of disinformation on social media
Bastos, M. (2022) · Online Information Review · doi:10.1108/oir-08-2021-563 · Cited by 3
Borchers, C., Klein, L., Morina, F., & Fischer, C. (2023) · doi:10.31219/osf.io/xu8gb · Cited by 3
Die KODAQS-Tools – Eine neue Ressource zur Beurteilung der Qualität von Forschungsdaten
Peters, Y., Krämer, F., Stroppe, A.-K., Sun, J., Daikeler, J., Yıldız, Ç., Dehne, J., Silber, H., Jünger, S., Weller, K., et al. (2026) · Publizistik · doi:10.1007/s11616-026-00941-x · Cited by 2
Miyazaki, K., Murayama, T., Matsui, A., Nishikawa, M., Uchiba, T., Kwak, H., & An, J. (2023) · ACM Web Science Conference (WebSci) · doi:10.1145/3578503.3583594 · Cited by 2
Nerino, V. (2023) · Frontiers in Sociology · doi:10.3389/fsoc.2023.1170447 · Cited by 3
Beyond Following: Augmenting Bot Detection with the Integration of Behavioral Patterns
Reiche, S., Cohen, S., Simonov, K., & Friedrich, T. (2024) · Studies in computational intelligence · doi:10.1007/978-3-031-53468-3_21 · Cited by 2
role of social bots in the Brazilian environmental debate:
Salles, D., de Medeiros, P. M., Martins, B., Regattieri, L. L., & Santini, R. M. (2024) · The International Review of Information Ethics · doi:10.29173/irie510 · Cited by 2
Username Squatting on Online Social Networks: A Study on X
Λεπίπας, Α., Borovykh, A., & Demetriou, S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3637637 · Cited by 2
Social Bots Spreading Disinformation About Finance: Research Trends, and Ethical Challenges
Pohl, J., Griesbach, M., samiei, A., & de Araujo, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-52082-2_6 · Cited by 2
Social Bots als Stimmen im Diskurs
Dreesen, P., & Krasselt, J. (2022) · De Gruyter eBooks · doi:10.1515/9783110721447-014 · Cited by 3
Don’t Shoot the Message: Regulating Disinformation Beyond Content
Keller, C. I. (2021) · Direito Público · doi:10.11117/rdp.v18i99.6057 · Cited by 2
Agency in Computer-Mediated Communication: Bots and US Political Elections
Piercy, C. W., Bisel, R. S., & Treem, J. W. (2022) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80071-597-420221032 · Cited by 1
Polarization and Shared Attention among Influential Amplifiers of 2018 U.S. Primary Candidates
Montez, D., & Shmargad, Y. (2025) · Journal of Quantitative Description Digital Media · doi:10.51685/jqd.2025.001 · Cited by 1
Linde, M., Chan, C., & Balluff, P. (2025) · Journalism & Mass Communication Quarterly · doi:10.1177/10776990251378122 · Cited by 1
FATe of Bots: Ethical Considerations of Social Bot Detection
Ng, L., Pan, E., Yoder, M. M., & Carley, K. M. (2026) · ACM Journal on Responsible Computing · doi:10.1145/3799706 · Cited by 5
Dissecting chirping patterns of invasive Tweeter flocks in the German Twitter forest
Reubold, J. L., Escher, S., Pflugmacher, J., & Strufe, T. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100228 · Cited by 1
Noisy digitalia – Kommunikationslärm in Born Digital-Quellen am Beispiel von Spam im Usenet
Kiechle, O. (2022) · De Gruyter eBooks · doi:10.1515/9783110757101-018 · Cited by 1
Entropy-Based Automation Detection on Twitter Using DNA Profiling
Gilmary, R., & Venkatesan, A. (2023) · SN Computer Science · doi:10.1007/s42979-023-02324-9 · Cited by 1
Cybersecurity in social networks: An ensemble model for Twitter bot detection
Darem, A. A., Alhashmi, A. A., Alanazi, M. H., Alanezi, A. F., Said, Y., Darem, L. A., & Hussain, M. M. (2024) · International Journal of ADVANCED AND APPLIED SCIENCES · doi:10.21833/ijaas.2024.11.014 · Cited by 1
Vitkova, L., Chechulin, A. A., & Kotenko, I. V. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-87178-9_42 · Cited by 1
A Toolkit for Analyzing and Visualizing Online Users via Reshare Cascade ModelingPreprint
Kong, Q., Ram, R., & Rizoiu, M.-A. (2020) · arXiv (Cornell University) · Cited by 1
Detecting Propaganda on the Sentence Level during the COVID-19 PandemicPreprint
Chang, R.-C., & Lin, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12269 · Cited by 1
Contrastive Clustering: Toward Unsupervised Bias Reduction for Emotion and Sentiment ClassificationPreprint
Mowery, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07448 · Cited by 1
Censorship on social media: The gatekeeping functions of shadowbans in the American TwitterversePreprint
Jaidka, K., Mukerjee, S., & Yphtach Lelkes, B. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4087843 · Cited by 1
Bots on Social Media: The Past, Present and Future
Rossi, S. (2024) · doi:10.22439/phd.19.2024 · Cited by 2
Ugwumba, N. K. (2025) · Research Square · doi:10.21203/rs.3.rs-8085397/v1 · Cited by 1
Antenore, M., Camacho-Rodriguez, J. M., & Panizzi, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01148
Social bots versus real humans : the framing of 'Trump's Wall' on Twitter
Parra-Novosad, N. (2020) · doi:10.32469/10355/78599
Corti, L., Zanetti, M., Tricella, G., & Bonati, M. R. (2021) · doi:10.2196/preprints.35043
Behavioral changes associated with interacting with bots on Twitter
Babutsitdze, Z., & Vincileoni, D. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.371
DON’T SHOOT THE MESSAGE: REGULATING DISINFORMATION BEYOND CONTENT IN BRAZIL
Keller, C. I. (2021) · International Journal of Social Sciences · doi:10.55367/pxgb6863
Identifying Behavioral Factors Leading to Differential Polarization Effects of Adversarial Botnets
(2023) · ACM SIGAPP Applied Computing Review · doi:10.1145/3610019.3610022
Muhle, F. (2023) · Springer eBooks · doi:10.1007/978-3-658-41172-5_7
Analyzing the Efficacy of Bot Detection Methods on Twitter/XPreprint
Javed, D., Zaman, N., Khan, N. A., Ray, S. K., Mazroa, A. A., Ayaz, R., & Das, S. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4665612
The Platformisation of Consumer Culture
Anselmi, G., Gandini, A., Gandini, A., & Bainotti, L. (2024) · Amsterdam University Press eBooks · doi:10.1515/9789048555109 · Cited by 16
Gonçalves, J. L., Torres, L. B. D. M., Freire, P. M. S., & Goldschmidt, R. R. (2025) · Brazilian Symposium on Information Systems · doi:10.5753/sbsi.2025.246620
Political Disinformation: “Fake News,” Bots, and Deep Fakes
Jungherr, A. M. (2026) · Oxford University Press eBooks · doi:10.1093/9780197851395.003.1516 · Cited by 2
Control of Information by a Few: Incoordinated behavior of social bots in information disseminationPreprint
Jia, Q., Ren, W., Luo, N., Xu, X., Zhang, L., & Zhang, H. (2025) · Research Square · doi:10.21203/rs.3.rs-6279146/v1
How the internet and its bots are sabotaging scientific research
Forshaw, M. J., & Schneider, J. (2025) · doi:10.64628/ab.qee6g4t9n
Como a internet e seus robôs estão sabotando a pesquisa científica
Forshaw, M. J., & Schneider, J. (2025) · doi:10.64628/ade.4xguf5pmm
A Compression-Based Approach to Detecting Automated and Coordinated Behavior on Social Media
Loru, E., Marco, N. D., Cinelli, M., & Quattrociocchi, W. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3778356 · Cited by 4
Borchers, C., Morina, F., Klein, L., & Fischer, C. (2025) · Journal of New Approaches in Educational Research · doi:10.1007/s44322-025-00047-8 · Cited by 5
Borchers, C., Morina, F., Klein, L., & Fischer, C. (2025) · doi:10.31219/osf.io/xu8gb_v2
On the Internet no-one knows you’re not a bot: ‘Botting’ on Reddit as participatory culture
Graham, T., & Carlon, D. (2026) · New Media & Society · doi:10.1177/14614448251409210
Risk Analysis in Automated Misinformation Detection
Yee, A. K. (2026) · Minds and Machines · doi:10.1007/s11023-026-09775-y
Zhao, X., Wang, X. J., Dai, Y., & Wang, L. (2026) · Online Information Review · doi:10.1108/oir-07-2025-0524
Masoudnia, H., Esfahani, A. S., Forghani, M., Moghimi, M., & Esfahany, H. S. (2026) · Social Science Computer Review · doi:10.1177/08944393261460815
Micro-Meso-Macro Dynamics of Aggregate Preference Formation
Ackland, R., & Novak, M. (2025) · Advanced sciences and technologies for security applications · doi:10.1007/978-3-032-17478-9_13
Pogorzelski, O. (2026) · Argumentation · doi:10.1007/s10503-026-09726-y
Wahlberg, L. (2026) · Scandinavian Political Studies · doi:10.1111/1467-9477.70060
Metadata
- Authors
- Rauchfleisch, A., & Kaiser, J.
- Affiliations
- National Taiwan University (TW)
- Harvard University (US)
- Corresponding author
- Rauchfleisch, Adrian
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-10-22
- Volume
- 15
- Issue
- 10
- Pages
- e0241045
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7580919
- OpenAlex
- W3014935876
- Cited by
- 184
- References
- 49
- Citations by year
- 2026: 10 · 2025: 23 · 2024: 39 · 2023: 31 · 2022: 47 · 2021: 27 · 2020: 7
- FWCI
- 26.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Ministry of Science and Technology, Taiwan (108-2410-H-002 -007 -MY2)
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- bot detection, Botometer, false positives, false-negative results, Twitter, language influence, computational social science
- MeSH terms
- Machine Learning, Humans, Politics, Research, Social Sciences, Social Networking, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
