The False positive problem of automatic bot detection in social science research
Open accessAbstract
The identification of bots is an important and complicated task. The bot classifier "Botometer" was successfully introduced as a way to estimate the number of bots in a given list of accounts and, as a consequence, has been frequently used in academic publications. Given its relevance for academic research and our understanding of the presence of automated accounts in any given Twitter discourse, we are interested in Botometer's diagnostic ability over time. To do so, we collected the Botometer scores for five datasets (three verified as bots, two verified as human; n = 4,134) in two languages (English/German) over three months. We show that the Botometer scores are imprecise when it comes to estimating bots; especially in a different language. We further show in an analysis of Botometer scores over time that Botometer's thresholds, even when used very conservatively, are prone to variance, which, in turn, will lead to false negatives (i.e., bots being classified as humans) and false positives (i.e., humans being classified as bots). This has immediate consequences for academic research as most studies in social science using the tool will unknowingly count a high number of human users as bots and vice versa. We conclude our study with a discussion about how computational social scientists should evaluate machine learning systems that are developed for identifying bots.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Botometer's limits, even when applied very conservatively, are subject to fluctuation, which can result in both false positives and false negatives.
“They also demonstrate that Botometer's limits, even when applied very conservatively, are subject to fluctuation, which can result in both false positives and false negatives (i.e., the classification of humans as bots) and false negatives (i.e., the classification of bots as humans).”
From Deepfakes: current and future trends (2024) · cites this paper as Rauchfleisch and Kaiser 2020
Botometer ratings are inaccurate for estimating bots, particularly when used in an alternate language.
“Their findings demonstrate that Botometer ratings are inaccurate for estimating bots, particularly when used in an alternate language.”
From Deepfakes: current and future trends (2024) · cites this paper as Rauchfleisch and Kaiser 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · see citing works
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 4× · see citing works
“It is thus fair to say that the tool is arguably the most popular bot detection method in the social sciences and applications of the tools have been published both in field specific journals such as Political Communication [7] as well as high-ranked general-interest journals like Science [8] or Nature Communications…”
BackgroundMethodResultInfluentialThe relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · cited 1× · see citing works
“While the so-called Matthews correlation coefficient is better than the F1 score if researchers use only one specific threshold, PR curves are recommended [34, 35] if different threshold levels exist as in the case with Botometer.”
BackgroundSaito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · see citing works
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1× · see citing works
“While the percentage agreement is already an interesting indicator, we would highly recommend the use of a measurement like Krippendorff’s alpha [45] that takes into account chance agreement.”
MethodAUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499 · see citing works
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973 · see citing works
The spread of low-credibility content by social bots
Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2018) · Nature Communications · doi:10.1038/s41467-018-06930-7 · Cited by 1,125 · see citing works
Social bots distort the 2016 U.S. Presidential election online discussion
Bessi, A., & Ferrara, E. (2016) · First Monday · doi:10.5210/fm.v21i11.7090 · Cited by 993 · see citing works
Online Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 928 · cited 5× · see citing works
“We thus assume for our simulation that the Twitter population has 15% bots [4] and 85% non-bots.”
BackgroundMethodInfluentialBotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909 · see citing works
Ten quick tips for machine learning in computational biology
Chicco, D. (2017) · BioData Mining · doi:10.1186/s13040-017-0155-3 · Cited by 875 · see citing works
Sun, X., & Xu, W. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2337313 · Cited by 638 · see citing works
The spread of low-credibility content by social bots
Shao, C., Ciampaglia, G. L., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2018) · RePEc: Research Papers in Economics · Cited by 574 · see citing works
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419 · cited 4× · see citing works
“Even though many new bot detection methods are developed every year—outperforming Botometer in some cases—[15, 16, 18], we focus on Botometer as it is by far the most used tool in studies analyzing bots on Twitter [12], especially in social science [5, 7, 10].”
BackgroundMethodInfluentialBots, #Strongerin, and #Brexit: Computational Propaganda During the UK-EU Referendum
Howard, P. N., & Kollanyi, B. (2016) · doi:10.2139/ssrn.2798311 · Cited by 392 · see citing works
The Brexit Botnet and User-Generated Hyperpartisan News
Bastos, M., & Mercea, D. (2019) · Social Science Computer Review · doi:10.1177/0894439317734157 · Cited by 385 · see citing works
The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3055135 · Cited by 381 · cited 9× · see citing works
“At least researchers should be aware of the data used as training data for the classifier that might only cover certain types of bots [15, 16, 18].”
BackgroundMethodResultInfluentialPolitical Astroturfing on Twitter: How to Coordinate a Disinformation Campaign
Keller, F. B., Schoch, D., Stier, S., & Yang, J. (2020) · Political Communication · doi:10.1080/10584609.2019.1661888 · Cited by 378 · cited 1× · see citing works
“Overall it is also important to broaden the perspective and focus more on political digital astroturfing [46] for which bots are just one potential dimension [47].”
Background(2020) · doi:10.1609/aaai.v34i01.5460 · Cited by 361 · see citing works
A decade of social bot detection
Cresci, S. (2020) · Communications of the ACM · doi:10.1145/3409116 · Cited by 323 · cited 7× · see citing works
“Finally, there are other ways to identify bots on Twitter as Cresci [22] and Orabi et al.”
BackgroundInfluentialUnpacking the Social Media Bot: A Typology to Guide Research and Policy
Gorwa, R., & Guilbeault, D. (2020) · Policy & Internet · doi:10.1002/poi3.184 · Cited by 270 · see citing works
DeBot: Twitter Bot Detection via Warped Correlation
Chavoshi, N., Hamooni, H., & Mueen, A. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0096 · Cited by 260 · see citing works
Halligan, S., Altman, D. G., & Mallett, S. (2015) · European Radiology · doi:10.1007/s00330-014-3487-0 · Cited by 255 · see citing works
The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race
Cresci, S., Pietro, R. D., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · doi:10.48550/arxiv.1701.03017 · Cited by 237 · see citing works
Detection of Bots in Social Media: A Systematic Review
Orabi, M., Mouheb, D., Aghbari, Z. A., & Kamel, I. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102250 · Cited by 232 · see citing works
Social Bots in Election Campaigns: Theoretical, Empirical, and Methodological Implications
Keller, T. R., & Klinger, U. (2018) · Political Communication · doi:10.1080/10584609.2018.1526238 · Cited by 216 · see citing works
(2020) · Sabanci University · Cited by 197 · see citing works
Social fingerprinting: detection of spambot groups through DNA-inspired behavioral modeling
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2681672 · Cited by 168 · see citing works
RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter
Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., & Tesconi, M. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326015 · Cited by 163 · see citing works
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3412698 · Cited by 159 · cited 2× · see citing works
“The Botometer creators [43] argue in their newest paper that some of the German politicians’ accounts used in our study might have a certain degree of automation.”
BackgroundDigital astroturfing in politics: Definition, typology, and countermeasures
Kovic, M., Rauchfleisch, A., Sele, M., & Caspar, C. (2018) · Studies in Communication Sciences · doi:10.24434/j.scoms.2018.01.005 · Cited by 149 · see citing works
Self-citationThe global organization of social media disinformation campaigns
Howard, P., & Bradshaw, S. (2018) · Journal of international affairs · Cited by 146 · see citing works
Transparent and Reproducible Social Science Research: How to Do Open Science
Christensen, G., Freese, J., & Miguel, E. A. (2019) · Cited by 129 · see citing works
Bradshaw, S., Howard, P. N., Kollányi, B., & Neudert, L. (2019) · Political Communication · doi:10.1080/10584609.2019.1663322 · Cited by 103 · see citing works
Are Social Bots on Twitter Political Actors? Empirical Evidence from a Ukrainian Social Botnet
Hegelich, S., & Janetzko, D. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v10i1.14764 · Cited by 96 · see citing works
Changing Perspectives: Is It Sufficient to Detect Social Bots?
Grimme, C., Assenmacher, D., & Adam, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-91521-0_32 · Cited by 93 · see citing works
Whose Lives Matter? Mass Shootings and Social Media Discourses of Sympathy and Policy, 2012–2014
Zhang, Y., Shah, D. V., Foley, J. M., Abhishek, A., Lukito, J., Suk, J., Kim, S. J., Sun, Z., Pevehouse, J. C. W., & Garlough, C. (2019) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmz009 · Cited by 92 · see citing works
BotChase: Graph-Based Bot Detection Using Machine Learning
Daya, A. A., Salahuddin, M. A., Limam, N., & Boutaba, R. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.2972405 · Cited by 74 · see citing works
On the Detection of Disinformation Campaign Activity with Network Analysis
Vargas, L., Emami, P., & Traynor, P. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop · doi:10.1145/3411495.3421363 · Cited by 74 · see citing works
LOBO: Evaluation of Generalization Deficiencies in Twitter Bot Classifiers
Echeverra, J., De Cristofaro, E., Kourtellis, N., Leontiadis, I., Stringhini, G., & Zhou, S. (2018) · Proceedings of the 34th Annual Computer Security Applications Conference · doi:10.1145/3274694.3274738 · Cited by 61 · see citing works
Political Astroturfing on Twitter: How to Coordinate a Disinformation CampaignPreprint
Keller, F., Schoch, D., Stier, S., & Yang, J. (2019) · doi:10.31235/osf.io/a5gk6 · Cited by 60 · see citing works
Political Bots and the Swedish General Election
Fernquist, J., Kaati, L., & Schroeder, R. (2018) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi.2018.8587347 · Cited by 57 · see citing works
Evolution of bot and human behavior during elections
Luceri, L., Deb, A., Giordano, S., & Ferrara, E. (2019) · First Monday · doi:10.5210/fm.v24i9.10213 · Cited by 56 · see citing works
Transparent and Reproducible Social Science Research
Christensen, G., Jeremy Freese, A., & Miguel, E. A. (2019) · University of California Press eBooks · doi:10.1525/9780520969230 · Cited by 36 · see citing works
“How Polling Becomes News: Communicating the Counting of Public Opinion”
Frankovic, K. A. (1998) · Political Communication · doi:10.1080/10584609.1998.11672656 · Cited by 7 · see citing works
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- [15]Echeverría J, Cristofaro E de, Kourtellis N, Leontiadis I, Stringhini G, Zhou S. LOBO. Proceedings of the 34th Annual Computer Security Applications Conference. New York, NY, USA: ACM; 2018. pp. 137–46.
- [16]Cresci S, Di Pietro R, Petrocchi M, Spognardi A, Tesconi M. The Paradigm-Shift of Social Spambots. In: Barrett R, Cummings R, Agichtein E, Gabrilovich E, editors. Proceedings of the 26th International Conference on World Wide Web Companion—WWW '17 Companion. New York, New York, USA: ACM Press; 2017. pp. 963–72.
- [19]Mazza M, Cresci S, Avvenuti M, Quattrociocchi W, Tesconi M. RTbust. In: Boldi P, Welles BF, Kinder-Kurlanda K, Wilson C, Peters I, et al., editors. Proceedings of the 10th ACM Conference on Web Science—WebSci '19. New York, New York, USA: ACM Press; 2019. pp. 183–92.
- [42]Fernández DM, Graziotin D, Wagner S, Seibold H. Open Science in Software Engineering; 2019. https://arxiv.org/abs/1904.06499. · arXiv
Cited by
Sorted by citations
Botometer 101: social bot practicum for computational social scientists
Yang, K.-C., Ferrara, E., & Menczer, F. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00177-5 · Cited by 159 · see citing works
Deepfakes: current and future trends
Gambín, Á. F., Yazidi, A., Vasilakos, A., Haugerud, H., & Djenouri, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10679-x · Cited by 145 · see citing works
He, L., He, C., Reynolds, T. L., Bai, Q., Huang, Y., Li, C., Zheng, K., & Chen, Y. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab047 · Cited by 110 · see citing works
Why Botter: How Pro-Government Bots Fight Opposition in Russia
Стукал, Д., Sanovich, S., Bonneau, R., & Tucker, J. A. (2022) · American Political Science Review · doi:10.1017/s0003055421001507 · Cited by 95 · see citing works
DeeProBot: a hybrid deep neural network model for social bot detection based on user profile data
Hayawi, K., Mathew, S. S., Venugopal, N., Masud, M. M., & Ho, P. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00869-w · Cited by 93 · see citing works
Ruffo, G., Semeraro, A., Giachanou, A., & Rosso, P. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100531 · Cited by 110 · see citing works
Bot, or not? Comparing three methods for detecting social bots in five political discourses
Martini, F., Samula, P., Keller, T. R., & Klinger, U. (2021) · Big Data & Society · doi:10.1177/20539517211033566 · Cited by 91 · see citing works
Computational Social Science and the Study of Political Communication
Theocharis, Y., & Jungherr, A. M. (2020) · Political Communication · doi:10.1080/10584609.2020.1833121 · Cited by 95 · see citing works
Artificial intelligence and the public arena
Jungherr, A. M., & Schroeder, R. (2023) · Communication Theory · doi:10.1093/ct/qtad006 · Cited by 98 · see citing works
Detecting computer-generated disinformation
Stiff, H., & Johansson, F. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00299-5 · Cited by 79 · see citing works
On the Detection of Disinformation Campaign Activity with Network Analysis
Vargas, L., Emami, P., & Traynor, P. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop · doi:10.1145/3411495.3421363 · Cited by 74 · see citing works
Coordinated inauthentic behavior and information spreading on twitter
Cinelli, M., Cresci, S., Quattrociocchi, W., Tesconi, M., & Zola, P. (2022) · Decision Support Systems · doi:10.1016/j.dss.2022.113819 · Cited by 74 · see citing works
Malicious URL Detection using Logistic Regression
Rupa, C., Srivastava, G., Bhattacharya, S., Maddikunta, P. K. R., & Gadekallu, T. R. (2021) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins51742.2021.9524269 · Cited by 69 · see citing works
Automatic detection of influential actors in disinformation networks
Smith, S., Kao, E., Mackin, E., Shah, D. C., Simek, O., & Rubin, D. (2020) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2011216118 · Cited by 55 · see citing works
Coordination patterns reveal online political astroturfing across the world
Schoch, D., Keller, F. B., Stier, S., & Yang, J. (2022) · Scientific Reports · doi:10.1038/s41598-022-08404-9 · Cited by 64 · see citing works
Public perception of generative AI on Twitter: an empirical study based on occupation and usage
Miyazaki, K., Murayama, T., Uchiba, T., An, J., & Kwak, H. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-023-00445-y · Cited by 62 · see citing works
AI-driven disinformation: a framework for organizational preparation and response
Karinshak, E., & Jin, Y. (2023) · Journal of Communication Management · doi:10.1108/jcom-09-2022-0113 · Cited by 65 · see citing works
Methodological Challenge: Addressing Bots in Online Research
Lawrence, P. R., Osborne, M. C., Sharma, D., Spratling, R. G., & Calamaro, C. (2023) · Journal of Pediatric Health Care · doi:10.1016/j.pedhc.2022.12.006 · Cited by 50 · see citing works
Digital Transformations of the Public Arena
Jungherr, A. M., & Schroeder, R. (2021) · Cambridge University Press eBooks · doi:10.1017/9781009064484 · Cited by 46 · see citing works
Weng, Z., & Lin, A. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192416376 · Cited by 44 · see citing works
Silenced on social media: the gatekeeping functions of shadowbans in the American Twitterverse
Jaidka, K., Mukerjee, S., & Lelkes, Y. (2023) · Journal of Communications · doi:10.1093/joc/jqac050 · Cited by 41 · see citing works
Bruno, M., Lambiotte, R., & Saracco, F. (2022) · EPJ Data Science · doi:10.1140/epjds/s13688-022-00330-0 · Cited by 40 · see citing works
Corti, L., Zanetti, M., Tricella, G., & Bonati, M. R. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00666-4 · Cited by 40 · see citing works
MGTAB: A Multi-Relational Graph-Based Twitter Account Detection Benchmark
Shi, S., Qiao, K., Liu, Z., Yang, J., Chen, C., Chen, J., & Yan, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130490 · Cited by 48 · see citing works
Conspiracy theories on Twitter: emerging motifs and temporal dynamics during the COVID-19 pandemic
Batzdorfer, V., Steinmetz, H., Biella, M., & Alizadeh, M. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00298-6 · Cited by 38 · see citing works
Ng, L. H. X., Robertson, D. C., & Carley, K. M. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100198 · Cited by 34 · see citing works
Winds of Change: Impact of COVID-19 on Vaccine-Related Opinions of Twitter Users
Poddar, S., Mondal, M., Misra, J., Ganguly, N., & Ghosh, S. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19334 · Cited by 32 · see citing works
Birkenmaier, L., Lechner, C. M., & Wagner, C. (2023) · Communication Methods and Measures · doi:10.1080/19312458.2023.2285765 · Cited by 30 · see citing works
Hays, C., Schutzman, Z., Raghavan, M., Walk, E., & Zimmer, P. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583214 · Cited by 29 · see citing works
Accuracy of health‐related information regarding COVID‐19 on Twitter during a global pandemic
Swetland, S. B., Rothrock, A. N., Andris, H., Davis, B., My, L. N. T., Davis, P. J., & Rothrock, S. G. (2021) · World Medical & Health Policy · doi:10.1002/wmh3.468 · Cited by 25 · see citing works
Understanding anti-immigration sentiment spreading on Twitter
Nasuto, A., & Rowe, F. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307917 · Cited by 30 · see citing works
Profiling users and bots in Twitter through social media analysis
Pastor-Galindo, J., Mármol, F. G., & Pérez, G. M. (2022) · Information Sciences · doi:10.1016/j.ins.2022.09.046 · Cited by 23 · see citing works
Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI
Liu, H., Das, A., Boltz, A., Zhou, D., Pinaroc, D., Lease, M., & Lee, M. K. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686962 · Cited by 31 · see citing works
Computational reproducibility in computational social science
Schoch, D., Chan, C.-H., Wagner, C., & Bleier, A. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00514-w · Cited by 24 · see citing works
Automation on Twitter: Measuring the Effectiveness of Approaches to Bot Detection
Beatson, O., Gibson, R., Cantijoch, M., & Elliot, M. J. (2021) · Social Science Computer Review · doi:10.1177/08944393211034991 · Cited by 22 · see citing works
Ruíz-Núñez, C., Segado-Fernández, S., Jiménez-Gómez, B., Hidalgo, P. J. J., Magdalena, C. S. R., Pollo, M. D. C. Á., García, A. S., & Herrera‐Peco, I. (2022) · Vaccines · doi:10.3390/vaccines10081240 · Cited by 22 · see citing works
Daume, S., Galaz, V., & Bjersér, P. (2023) · Social Media + Society · doi:10.1177/20563051231168370 · Cited by 22 · see citing works
Measuring the diffusion of conspiracy theories in digital information ecologies
Heft, A., & Buehling, K. (2022) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565221091809 · Cited by 21 · see citing works
Chen, A., Komi, M. M., Bessler, S., Mikles, S. P., & Zhang, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104439 · Cited by 21 · see citing works
Social Botomics: A Systematic Ensemble ML Approach for Explainable and Multi-Class Bot Detection
Dimitriadis, I., Georgiou, K., & Vakali, A. I. (2021) · Applied Sciences · doi:10.3390/app11219857 · Cited by 20 · see citing works
Friendship Preference: Scalable and Robust Category of Features for Social Bot Detection
Moghaddam, S. H., & Abbaspour, M. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3159007 · Cited by 21 · see citing works
TeBlunthuis, N., Hase, V., & Chan, C. (2024) · Communication Methods and Measures · doi:10.1080/19312458.2023.2293713 · Cited by 20 · see citing works
Multifaceted online coordinated behavior in the 2020 US presidential election
Tardelli, S., Nizzoli, L., Avvenuti, M., Cresci, S., & Tesconi, M. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00467-0 · Cited by 25 · see citing works
MulBot: Unsupervised Bot Detection Based on Multivariate Time Series
Mannocci, L., Cresci, S., Monreale, A., Vakali, A. I., & Tesconi, M. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020363 · Cited by 19 · see citing works
Detection and impact estimation of social bots in the Chilean Twitter network
Mendoza, M., Providel, E., dos Santos, M. L. B., & Valenzuela, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-57227-3 · Cited by 19 · see citing works
New Automation for Social Bots: From Trivial Behavior to AI-Powered Communication
Grimme, C., Pohl, J., Cresci, S., Lüling, R., & Preuss, M. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-18253-2_6 · Cited by 19 · see citing works
Super-amplifiers! The role of Twitter extended party networks in political elections
Yoon, N., Hemsley, J., Smith, A., Simpson, E., & Eakins, J. (2022) · Policy & Internet · doi:10.1002/poi3.295 · Cited by 18 · see citing works
Coordinated Behavior in Information Operations on Twitter
Cima, L., Mannocci, L., Avvenuti, M., Tesconi, M., & Cresci, S. (2024) · IEEE Access · doi:10.1109/access.2024.3393482 · Cited by 18 · see citing works
A Comparative Study of Bot Detection Techniques With an Application in Twitter Covid-19 Discourse
Antenore, M., Camacho, J. M., & Panizzi, E. (2022) · Social Science Computer Review · doi:10.1177/08944393211073733 · Cited by 17 · see citing works
Investigating the Validity of Botometer-Based Social Bot Studies
Gallwitz, F., & Kreil, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18253-2_5 · Cited by 17 · see citing works
Wang, R., Zhang, Y., Suk, J., & Holland Levin, S. (2024) · Social Media + Society · doi:10.1177/20563051241277606 · Cited by 16 · see citing works
Feature extractions and selection of bot detection on Twitter A systematic literature review
Al-azawi, R., & Al‐Mamory, S. O. (2022) · INTELIGENCIA ARTIFICIAL · doi:10.4114/intartif.vol25iss69pp57-86 · Cited by 16 · see citing works
Early Detection and Prevention of Malicious User Behavior on Twitter Using Deep Learning Techniques
Sánchez, R., Zubiaga, A., & Almeida, A. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3419171 · Cited by 15 · see citing works
Characterizing Retweet Bots: The Case of Black Market Accounts
Elmas, T., Overdorf, R., & Aberer, K. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19282 · Cited by 15 · see citing works
Measuring user engagement with low credibility media sources in a controversial online debate
Vilella, S., Semeraro, A., Paolotti, D., & Ruffo, G. (2022) · EPJ Data Science · doi:10.1140/epjds/s13688-022-00342-w · Cited by 15 · see citing works
Silver, N. A., Kierstead, E. C., Kostygina, G., Tran, H., Briggs, J., Emery, S., & Schillo, B. (2022) · Journal of Medical Internet Research · doi:10.2196/40331 · Cited by 14 · see citing works
Detection of automated behavior on Twitter through approximate entropy and sample entropy
Gilmary, R., Akila, V., & Vaiyapuri, G. (2021) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01647-9 · Cited by 14 · see citing works
Assembling a multi-platform ensemble social bot detector with applications to US 2020 elections
Ng, L. H. X., & Carley, K. M. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01211-2 · Cited by 13 · see citing works
Dissecting a social bot powered by generative AI: anatomy, new trends and challenges
López-Joya, S., Diaz-Garcia, J. A., Ruiz, M. D., & Martín-Bautista, M. J. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01410-5 · Cited by 13 · see citing works
Ngo, T., Wischnewski, M., Bernemann, R., Jansen, M., & Krämer, N. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107819 · Cited by 14 · see citing works
Evolution of the plandemic communication network among serial participants on Twitter
Xu, Y., Sun, Y., Hagen, L., Patel, M., & Falling, M. (2021) · New Media & Society · doi:10.1177/14614448211050928 · Cited by 12 · see citing works
Detection of fickle trolls in large-scale online social networks
Shafiei, H., & Dadlani, A. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00572-9 · Cited by 12 · see citing works
Deflating the Chinese balloon: types of Twitter bots in US-China balloon incident
Ng, L., & Carley, K. M. (2023) · EPJ Data Science · doi:10.1140/epjds/s13688-023-00440-3 · Cited by 17 · see citing works
Demystifying Misconceptions in Social Bots Research
Cresci, S., Yang, K.-C., Spognardi, A., Pietro, R. D., Menczer, F., & Petrocchi, M. (2025) · Social Science Computer Review · doi:10.1177/08944393251376707 · Cited by 12 · see citing works
Not all who are bots are evil: A cross-platform analysis of automated agent governance
Makhortykh, M., Urman, A., Münch, F. V., Heldt, A., Dreyer, S., & Kettemann, M. C. (2022) · New Media & Society · doi:10.1177/14614448221079035 · Cited by 11 · see citing works
Socio-Linguistic Characteristics of Coordinated Inauthentic Accounts
Burghardt, K., Rao, A., Chochlakis, G., Baruah, S., Guo, S., He, Z., Rojecki, A., Narayanan, S. S., & Lerman, K. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31305 · Cited by 11 · see citing works
Social Media, Populism, and Migration
Datts, M. (2020) · Media and Communication · doi:10.17645/mac.v8i4.3212 · Cited by 11 · see citing works
Do Bots Have Moral Judgement? The Difference Between Bots and Humans in Moral Rhetoric
Mutlu, E. Ç., Oghaz, T. A., Tütüncüler, E., & Garibay, I. I. (2020) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam49781.2020.9381386 · Cited by 11 · see citing works
Tran, H. D. (2021) · Journalism and Media · doi:10.3390/journalmedia2040042 · Cited by 11 · see citing works
Social bots spoil activist sentiment without eroding engagement
Li, L., Vásárhelyi, O., & Vedres, B. (2024) · Scientific Reports · doi:10.1038/s41598-024-74032-0 · Cited by 16 · see citing works
Vulnerability disclosure and cybersecurity awareness campaigns on twitter during COVID ‐19
Bahl, A., Sharma, A., & Asghar, M. R. (2021) · Security and Privacy · doi:10.1002/spy2.180 · Cited by 10 · see citing works
Mapping state-sponsored information operations with multi-view modularity clustering
Uyheng, J., Cruickshank, I. J., & Carley, K. M. (2022) · EPJ Data Science · doi:10.1140/epjds/s13688-022-00338-6 · Cited by 10 · see citing works
Agents of Discord: Modeling the Impact of Political Bots on Opinion Polarization in Social Networks
Lu, H.-C., & Lee, H. (2024) · Social Science Computer Review · doi:10.1177/08944393241270382 · Cited by 9 · see citing works
Wang, R., Walter, D., & Ophir, Y. (2023) · Social Science Computer Review · doi:10.1177/08944393231188472 · Cited by 9 · see citing works
Twitter Bots Influence on the Russo-Ukrainian War During the 2022 Italian General Elections
Faveri, F. L. D., Cosuti, L., Tricomi, P. P., & Conti, M. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-5177-2_3 · Cited by 9 · see citing works
An exploratory design science research on troll factories
Marcondes, F. S., Almeida, J. J., & Nováis, P. (2023) · Integrated Computer-Aided Engineering · doi:10.3233/ica-230720 · Cited by 9 · see citing works
Lost in Transformation: Rediscovering LLM-Generated Campaigns in Social Media
Britta, G., Pohl, J., Winkelmann, H., Stampe, L., & Grimme, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47896-3_6 · Cited by 9 · see citing works
The diffusion of malicious content on Twitter and its impact on security
Roumani, Y. (2024) · Information & Management · doi:10.1016/j.im.2024.103971 · Cited by 9 · see citing works
Task-agnostic representation learning of multimodal twitter data for downstream applications
Rivas, R., Paul, S., Hristidis, V., Papalexakis, E. E., & Roy–Chowdhury, A. K. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00570-x · Cited by 9 · see citing works
DNA-influenced automated behavior detection on twitter through relative entropy
Gilmary, R., Akila, V., Vaiyapuri, G., & Balamurali, D. (2022) · Scientific Reports · doi:10.1038/s41598-022-11854-w · Cited by 9 · see citing works
Surveying the Research on Fake News in Social Media: a Tale of Networks and Language.Preprint
Ruffo, G., Semeraro, A., Giachanou, A., & Rosso, P. (2021) · arXiv (Cornell University) · Cited by 9 · see citing works
Kim, J., Wang, Z., Shi, H., Ling, H.-K., & Evans, J. A. (2025) · Nature Communications · doi:10.1038/s41467-025-55868-0 · Cited by 8 · see citing works
BotSSCL: Social Bot Detection with Self-Supervised Contrastive Learning
Akhtar, M. M., Bhuiyan, N. S., Masood, R., Ikram, M., & Kanhere, S. S. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100318 · Cited by 8 · see citing works
Challenges in machine learning-based social bot detection: a systematic review
Alkathiri, N., & Slhoub, K. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00448-w · Cited by 8 · see citing works
García-Méndez, S., Leal, F., Malheiro, B., Burguillo, J. C., Veloso, B., Chis, A. E., & González–Vélez, H. (2022) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2022.102616 · Cited by 8 · see citing works
Wood, N. K., & Bindler, R. J. (2023) · Journal of Advanced Nursing · doi:10.1111/jan.15981 · Cited by 8 · see citing works
Tracing the dynamics of misinformation and vaccine stance in Finland amid COVID-19
Ünlü, A., Truong, S., Sawhney, N., Sivelä, J., & Tammi, T. (2024) · Information Communication & Society · doi:10.1080/1369118x.2024.2331756 · Cited by 8 · see citing works
Arroyo‐Machado, W., Herrera‐Viedma, E., & Torres‐Salinas, D. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24998 · Cited by 8 · see citing works
Uyheng, J., & Carley, K. M. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18098 · Cited by 8 · see citing works
Understanding Transformers for Bot Detection in TwitterPreprint
García-Silva, A., Berrío, C., & Gómez-Pérez, J. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06182 · Cited by 13 · see citing works
Patterns of human and bots behaviour on Twitter conversations about sustainability
Mouronte-López, M. L., Sánchez-Seco, J. G., & Benito, R. M. (2024) · Scientific Reports · doi:10.1038/s41598-024-52471-z · Cited by 7 · see citing works
Cyborgs for strategic communication on social media
Ng, L. H. X., Robertson, D. C., & Carley, K. M. (2024) · Big Data & Society · doi:10.1177/20539517241231275 · Cited by 7 · see citing works
Quali-quanti visual methods and political bots
Omena, J. J., Lobo, T., Tucci, G., Bitencourt, E. C., de Keulenaar, E. V., Kerche, F. W., Chao, J., Liedtke, M., Li, M., Paschoal, M. L., et al. (2024) · Journal of Digital Social Research · doi:10.33621/jdsr.v6i1.215 · Cited by 7 · see citing works
Bayesian identification of bots using temporal analysis of tweet storms
Kirn, S. L., & Hinders, M. K. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00783-7 · Cited by 7 · see citing works
Muhle, F. (2022) · Medien & Kommunikationswissenschaft · doi:10.5771/1615-634x-2022-1-2-40 · Cited by 7 · see citing works
To Whom Do Politicians Talk and Listen?
Keller, T. R. (2020) · Computational Communication Research · doi:10.5117/ccr2020.2.003.kell · Cited by 7 · see citing works
Automating communication in the digital society Editorial to the special issue
Katzenbach, C., & Pentzold, C. (2024) · New Media & Society · doi:10.1177/14614448241265655 · Cited by 7 · see citing works
Urman, A., & Makhortykh, M. (2024) · East European Politics · doi:10.1080/21599165.2024.2415640 · Cited by 6 · see citing works
Meng, X., Wang, X.-H., & Peng, T.-Q. (2026) · Communication Methods and Measures · doi:10.1080/19312458.2026.2613829 · Cited by 5 · see citing works
Few voices, strong echo: Measuring follower homogeneity of politicians’ Twitter accounts
Rusche, F. (2024) · New Media & Society · doi:10.1177/14614448221099860 · Cited by 6 · see citing works
- Authors
- Rauchfleisch, A., & Kaiser, J.
- Affiliations
- National Taiwan University (TW)
- Harvard University (US)
- Corresponding author
- Rauchfleisch, Adrian
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-10-22
- Volume
- 15
- Issue
- 10
- Pages
- e0241045
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC7580919
- OpenAlex
- W3014935876
- Cited by
- 184
- References
- 49
- Citations by year
- 2026: 10 · 2025: 23 · 2024: 39 · 2023: 31 · 2022: 47 · 2021: 27 · 2020: 7
- FWCI
- 26.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Ministry of Science and Technology, Taiwan (108-2410-H-002 -007 -MY2)
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- bot detection, Botometer, false positives, false-negative results, Twitter, language influence, computational social science
- MeSH terms
- Machine Learning, Humans, Politics, Research, Social Sciences, Social Networking, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
