LOBO: Evaluation of Generalization Deficiencies in Twitter Bot Classifiers
Open accessAbstract
Botnets in online social networks are increasingly often affecting the regular flow of discussion, attacking regular users and their posts, spamming them with irrelevant or offensive content, and even manipulating the popularity of messages and accounts. Researchers and cybercriminals are involved in an arms race, and new and updated botnets designed to defeat current detection systems are constantly developed, rendering such detection systems obsolete.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper argues that Twitter bot classifiers trained on single bot classes can achieve high accuracy but fail to generalize to unseen bot classes. It proposes a Leave-One-Botnet-Out (LOBO) evaluation methodology and shows a classifier trained on over 200,000 data points reaches over 97% accuracy yet can drop to less than 1% detection on some unseen botnets.
Key findings
- A classifier trained on over 200,000 data points achieved over 97% accuracy, but when tested on unseen bot classes via LOBO, it did not generalize well.
- Some target botnets completely misled the classifier, resulting in less than 1% detection accuracy, meaning 99% of the bots in that class were classified as users.
- The study collected more than 20 different bot classes and tested on over 1.5 million bots; the data includes some of the largest and most varied collections of bots used in literature.
- Adding a small portion of certain botnets to training can be enough to fully identify them, allowing a classifier to quickly scale and incrementally consider different and newly revealed bots.
Methods
- Compiled a botnet dataset aggregating content from a variety of bot datasets, some previously used in research as ground truth, plus a real-user dataset; each account contains a maximum of 3,200 tweets authored or retweeted due to Twitter API limitations.
- Built a dataset of 20 different botnet types with different purposes and proposed a “Leave-One-Botnet-Out” (LOBO) methodology: train on many diverse bots and test accuracy on bot classes with behaviors never seen before by the classifiers.
- Trained a Twitter bot classifier using a small number of commonly used features and evaluated its expected generalization by selectively leaving bot classes and behaviors out of the training data.
Limitations
- The paper states that datasets are constrained by Twitter’s API limitations, with each account containing a maximum of 3,200 tweets authored or retweeted.
- The paper states that machine-learning detection systems require example bot datasets that often contain biases; training on a single botnet can cause the system to overfit to that botnet’s idiosyncrasies rather than learn general bot characteristics.
- Evident from the provided text: the excerpt ends in the datasets section, so the full experimental setup, feature list, and detailed results are not available here.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 1 cited sources checked · 0 supported · 0 flagged
- Findingnone given§ 3. Boston University
The data used to train and test the classifier includes some of the largest and most varied collections of bots used in literature.
“includes some of the largest and most varied collections of bots used in literature.”
- Also stated in: § INTRODUCTION · plain
“Collects and combines the biggest and broadest botnet library to date”
Quote in the paperSubstantial claim - Also stated in: § INTRODUCTION · plain
- Findingown data§ 3. Boston University
The theoretically sound classifier does not generalize well to unseen bot classes.
“highlighting that it does not generalize well to unseen bot classes.”
- Also stated in: § INTRODUCTION · plain
“even this bot-detection system that has been trained with a variety of bots is incapable of detecting new bot families that were never observed before.”
Quote in the paperSubstantial claim - Also stated in: § INTRODUCTION · plain
- Findingown data§ INTRODUCTION
The study tests on over 1.5 million bots.
“testing on over 1.5 million bots.”
Quote in the paperSubstantial claim - Findingown data§ 3. Boston University
A Twitter bot classifier trained on a single bot dataset (over 200,000 data points) achieves over 97% accuracy.
“over 200,000 data points and show that it achieves over 97% accuracy”
Quote in the paperSubstantial claim - Findingown data§ INTRODUCTION
When bot classes are mixed equitably in a single dataset, the prediction power of the same classifier drops significantly.
“when we mix bot classes equitably in a single dataset”
Quote in the paperSubstantial claim
- Findingnone given§ INTRODUCTION
A Twitter bot classification strategy reaches accuracy values over 97% with a small number of commonly used features.
“reaches accuracy values over 97% with a small number of commonly used features”
Quote in the paperSubstantial claim - Findingnone given§ INTRODUCTION
Even a small portion of certain botnets can be enough to fully identify them when added to a common learning algorithm, allowing a classifier to quickly scale and incrementally consider different and newly revealed bots.
“even a small portion of certain botnets can be enough to fully identify them when adding it to a common learning algorithm”
Quote in the paperSubstantial claim - Interpretationreasoning§ INTRODUCTION
If a system was trained on a dataset containing only bot accounts belonging to one botnet, it would learn the idiosyncrasies of that specific botnet and become very accurate in detecting it.
“if a system was trained on a dataset containing only bot accounts belonging to one botnet, it would learn the idiosyncrasies”
Quote in the paperNot a substantial claim - Interpretationown data§ 3. Boston University
Methods trained and tested on single bot classes or datasets might not be able to generalize to new bot classes.
“argue that methods trained and tested on single bot classes or datasets might not able to generalize to new bot classes.”
Quote in the paperSubstantial claim - Interpretationreasoning§ INTRODUCTION
When trying to identify bots belonging to other botnets, a system trained on one botnet would perform very poorly, because it would overfit on a single family of bots.
“when trying to identify bots belonging to other botnets it would perform very poorly”
Quote in the paperSubstantial claim - Interpretationnone given§ INTRODUCTION
The methodology aids in identifying how much each target class is related to the rest of the bot classes without extensive and costly inspection.
“It further aids in identifying how much each target class is related to the rest of the bot classes without the need of extensive and costly inspection.”
Quote in the paperSubstantial claim - Interpretationreasoning§ INTRODUCTION
Even having multiple families of bots represented in the training set, there is no guarantee that the system will be able to identify new bot types or new botnets as they appear.
“Even having multiple families of bots represented in the training set, there is no guarantee that the system will be able to identify new bot types or new botnets as they appear.”
Quote in the paperSubstantial claim - Interpretationnone given§ INTRODUCTION
The way bot datasets are collected prevents them from being representative of all bots in Twitter.
“the way these datasets are collected prevents them from being representative of all bots in Twitter.”
Quote in the paperSubstantial claim
- May combine two claims. 3
May combine two claims.
Findingown data§ INTRODUCTIONSome target botnets completely mislead the classifier, resulting in less than 1% detection accuracy, meaning 99% of the bots in that class were classified as users.
“less than 1% detection accuracy, meaning 99% of the bots in that class were”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 3. Boston UniversityThe proposed methodology is empirically robust, using bot classes of varying sizes and characteristics and reaching similar results.
“We show that this methodology is empirically robust, using bot classes of varying sizes and characteristics and reaching similar results”
Quote in the paperSubstantial claimMay combine two claims.
Premisecitation§ INTRODUCTIONBot detection is an inherently adversarial problem, and once a defense mechanism is known, adversaries can modify their modus operandi and avoid detection.
“bot detection is an inherently adversarial problem, and once a defense mechanism is known, adversaries can modify their modus operandi and avoid detection [41].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [41] source
Not checked in the free run.
The paper relies on this without support.
Botnets in online social networks increasingly affect the regular flow of discussion, attack regular users and their posts, spam them with irrelevant or offensive content, and manipulate the popularity of messages and accounts.
“even manipulating the popularity of messages and accounts.”
The paper relies on this without support.
Researchers and cybercriminals are involved in an arms race, and new and updated botnets designed to defeat current detection systems are constantly developed, rendering such detection systems obsolete.
“Researchers and cybercriminals are involved in an arms race, and new and updated botnets designed to defeat current detection systems are constantly developed, rendering such detection systems obsolete.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Supervised methods' reliance on labeled datasets limits their ability to generalize to unseen types of bots.
“The reliance of supervised methods on labeled datasets limits their ability to generalize to unseen types of bots [16].”
From Unsupervised detection of coordinated fake-follower campaigns on social media (2024) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Dataset described as having both more than 20 and exactly 20 bot types
MinorThe introduction says the collected dataset contains more than 20 bot classes, while Section 3.1 describes the dataset as having 20 botnet types.
“more than 20”
“dataset of 20”
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
August, W. U., Moore, D., Voelker, G. M., & Savage, S. (2004) · IEEE Security & Privacy · doi:10.1109/msp.2004.9 · Cited by 1,519 · see citing works
Social bots distort the 2016 U.S. Presidential election online discussion
Bessi, A., & Ferrara, E. (2016) · First Monday · doi:10.5210/fm.v21i11.7090 · Cited by 993 · cited 1× · see citing works
“There have been also analysis on botnets promoting topics or products, including diet pills [28] and even political candidates [4].”
BotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909 · see citing works
Detecting spammers on social networks
Stringhini, G., Kruegel, C., & Vigna, G. (2010) · Proceedings of the 26th Annual Computer Security Applications Conference · doi:10.1145/1920261.1920263 · Cited by 791 · cited 2× · see citing works
“Such approaches look at either profile characteristics of fake accounts that distinguish them from legitimate ones [2, 31], at differences in the social graph of fake and legitimate accounts [7, 13, 27], at the way in which they are controlled by”
Self-citationMagno, G., Rodrigues, T., & Almeida, V. A. F. (2010) · Cited by 649 · see citing works
A multifaceted approach to understanding the botnet phenomenon
Rajab, M. A., Zarfoss, J., Monrose, F., & Terzis, A. (2006) · ACM Internet Measurement Conference (IMC) · doi:10.1145/1177080.1177086 · Cited by 595 · cited 1× · see citing works
“Large quantities of malicious accounts are often created and controlled by single mis-creants, forming so-called botnets [1].”
Grier, C., Thomas, K., Paxson, V., & Zhang, M. (2010) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/1866307.1866311 · Cited by 558 · see citing works
The DARPA Twitter Bot Challenge
Subrahmanian, V. S., Azaria, A., Durst, S., Kagan, V., Galstyan, A., Lerman, K., Zhu, L., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Computer · doi:10.1109/mc.2016.183 · Cited by 480 · cited 2× · see citing works
“hms show clear signs of an easy separation task, with accuracies over 95% in most cases. This level of accuracy in bot classification is not unheard of: it has been claimed before several times (e.g., [11,34,25]). To further reiterate that this is not a fluke, we also check the area under the ROC curve and the confusion…”
Design and Evaluation of a Real-Time URL Spam Filtering Service
Thomas, K., Grier, C., Ma, J. H., Paxson, V., & Song, D. X. (2011) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2011.25 · Cited by 470 · see citing works
Fame for sale: Efficient detection of fake Twitter followers
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.09.003 · Cited by 461 · cited 4× · see citing works
“This level of accuracy in bot classification is not unheard of: it has been claimed before several times (e.g., [11, 34, 25]).”
InfluentialSeven Months with the Devils: A Long-Term Study of Content Polluters on Twitter
Lee, K., Eoff, B. D., & Caverlee, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14106 · Cited by 448 · cited 1× · see citing works
“hms show clear signs of an easy separation task, with accuracies over 95% in most cases. This level of accuracy in bot classification is not unheard of: it has been claimed before several times (e.g., [11,34,25]). To further reiterate that this is not a fluke, we also check the area under the ROC curve and the confusion…”
Boshmaf, Y., Muslukhov, I., Beznosov, K., & Ripeanu, M. (2011) · Annual Computer Security Applications Conference · doi:10.1145/2076732.2076746 · Cited by 439 · see citing works
Suspended accounts in retrospect: an analysis of twitter spam
Thomas, K., Grier, C., Song, D., & Paxson, V. (2011) · Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference · doi:10.1145/2068816.2068840 · Cited by 439 · see citing works
Detecting and characterizing social spam campaigns
Gao, H., Hu, J. S., Wilson, C., Li, Z., Chen, Y., & Zhao, B. Y. (2010) · ACM Internet Measurement Conference (IMC) · doi:10.1145/1879141.1879147 · Cited by 420 · cited 2× · see citing works
“[20] analyze spam campaigns on Facebook.”
SybilInfer: Detecting Sybil Nodes using Social Networks.
Danezis, G., & Mittal, P. (2009) · Cited by 407 · see citing works
The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3055135 · Cited by 381 · see citing works
FRAUDAR: Bounding Graph Fraud in the Face of Camouflage
Hooi, B., Song, H. A., Beutel, A., Shah, N., Shin, K., & Faloutsos, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939747 · Cited by 305 · cited 1× · see citing works
“Other approaches leverage bots’ similarity in their operation, such as synchronization in posting messages [8], accesses by a common set of IP addresses [23, 32], or similar uses of the accounts [39].”
Yang, C., Harkreader, R., Zhang, J., Shin, S., & Gu, G. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187847 · Cited by 299 · cited 2× · see citing works
“[40] look at social relationships between spam accounts on Twitter, while Dave et al.”
Uncovering Large Groups of Active Malicious Accounts in Online Social Networks
Cao, Q., Yang, X., Yu, J., & Palow, C. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660269 · Cited by 274 · cited 2× · see citing works
“their operators [8, 32, 39], or at the content that they post, looking for signs of maliciousness [26, 29, 35].”
DeBot: Twitter Bot Detection via Warped Correlation
Chavoshi, N., Hamooni, H., & Mueen, A. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0096 · Cited by 260 · cited 2× · see citing works
“cores Botometer [15] (previously botornot) is a public API that provides a score based on whether an account is likely to be a bot or a user. It has been used to verify bot accounts in other research [9]. For our different bot classes, botometer does not perform well enough. This can be seen in Table1, %RW&L 8VHU&…”
You are how you click: Clickstream analysis for Sybil detection
Wang, G., Konolige, T., Wilson, C., Wang, X., Zheng, H.-T., & Zhao, B. Y. (2013) · Cited by 256 · see citing works
COMPA: Detecting Compromised Accounts on Social Networks.
Egele, M., Stringhini, G., Krügel, C., & Vigna, G. (2013) · Cited by 250 · see citing works
Self-citationTrafficking fraudulent accounts: the role of the underground market in Twitter spam and abuse
Thomas, K., McCoy, D., Grier, C., Kolcz, A., & Paxson, V. (2013) · Cited by 239 · see citing works
The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race
Cresci, S., Pietro, R. D., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · doi:10.48550/arxiv.1701.03017 · Cited by 237 · see citing works
Die Free or Live Hard? Empirical Evaluation and New Design for Fighting Evolving Twitter Spammers
Yang, C., Harkreader, R. C., & Gu, G. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-23644-0_17 · Cited by 235 · see citing works
WarningBird: Detecting Suspicious URLs in Twitter Stream.
Lee, S.-H., & Kim, J. (2012) · Open Access System for Information Sharing (Pohang University of Science and Technology) · Cited by 179 · see citing works
Stringhini, G., Wang, G., Egele, M., Kruegel, C., Vigna, G., Zheng, H., & Zhao, B. Y. (2013) · Proceedings of the 2013 conference on Internet measurement conference · doi:10.1145/2504730.2504731 · Cited by 169 · see citing works
Self-citationIntegro: Leveraging Victim Prediction for Robust Fake Account Detection in OSNs
Boshmaf, Y., Logothetis, D., Siganos, G., Leria, J., Lorenzo, J., Ripeanu, M., & Beznosov, K. (2015) · Proceedings 2015 Network and Distributed System Security Symposium · doi:10.14722/ndss.2015.23260 · Cited by 134 · see citing works
Of Bots and Humans (on Twitter)
Gilani, Z., Farahbakhsh, R., Gareth Tyson, A., et al. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3110090 · Cited by 131 · cited 3× · see citing works
“Datasets L,M,N and O have been identified by humans as bots, and were used as ground truth in [21].”
Measuring and fingerprinting click-spam in ad networks
Dave, V., Guha, S., & Zhang⋆, Y. (2012) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2342356.2342394 · Cited by 113 · cited 1× · see citing works
“[14] measure click-spam in ad networks, and Gao et al.”
Identifying Correlated Bots in Twitter
Chavoshi, N., Hamooni, H., & Mueen, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-47874-6_2 · Cited by 90 · cited 1× · see citing works
“Debot is a bot detection service that generates daily reports of bot activity, and stems from the work of [10][9].”
De Cristofaro, E., Friedman, A., Jourjon, G., Kaafar, M. A., & Shafiq, M. Z. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663729 · Cited by 85 · see citing works
Self-citationStringhini, G., Egele, M., Kruegel, C., & Vigna, G. (2012) · doi:10.1145/2342549.2342551 · Cited by 74 · see citing works
Self-citationEVILCOHORT: detecting communities of malicious accounts on online services
Stringhini, G., Mourlanne, P., Jacob, G., Egele, M., Kruegel, C., & Vigna, G. (2015) · Cited by 66 · see citing works
Self-citationDiscovery, Retrieval, and Analysis of the 'Star Wars' Botnet in Twitter
Echeverría, J., & Zhou, S. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3110074 · Cited by 56 · see citing works
Self-citationNilizadeh, S., Labrèche, F., Sedighian, A., Zand, A., Fernandez, J. M., Kruegel, C., Stringhini, G., & Vigna, G. (2017) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3133956.3134055 · Cited by 50 · cited 2× · see citing works
“[29] presented , a system that detects malicious messages (e.”
Self-citationExploiting Temporal Dynamics in Sybil Defenses
Liu, C., Gao, P., Wright, M., & Mittal, P. (2015) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2810103.2813693 · Cited by 33 · cited 2× · see citing works
“However, Liu et al [27] show that these techniques can be gamed by adversaries by exploiting the temporal dynamics used for detection.”
The Latent Community Model for Detecting Sybils in Social Networks.
Cai, Z., & Jermaine, C. (2012) · Network and Distributed System Security Symposium · Cited by 32 · see citing works
Measuring, Characterizing, and Detecting Facebook Like Farms
Ikram, M., Onwuzurike, L., Farooqi, S., Cristofaro, E. D., Friedman, A., Jourjon, G., Kaafar, M. A., & Shafiq, M. Z. (2017) · ACM Transactions on Privacy and Security · doi:10.1145/3121134 · Cited by 27 · see citing works
Self-citationFull Cycle Analysis of a Large-Scale Botnet Attack on Twitter
Besel, C., Echeverría, J., & Zhou, S. (2018) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2018.8508708 · Cited by 25 · see citing works
Self-citationData Science for Cyber-Security
Heard, N. A., Adams, N. M., Rubin‐Delanchy, P., et al. (2018) · Security science and technology · doi:10.1142/q0167 · Cited by 17 · see citing works
Discovery of the Twitter Bursty Botnet
Echeverría, J., Besel, C., & Zhou, S. (2018) · Security science and technology · doi:10.1142/9781786345646_007 · Cited by 16 · cited 6× · see citing works
“Some of them have obtained large botnets based on geographic anomalies [18] and temporal anomalies [17].”
InfluentialSelf-citation
Flagged first, then by citations
3 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- S. Narang . Green Coffee and Spam: Elaborate spam operation on Twitter uses nearly 750,000 accounts. https://symc.ly/2Cp7CJ5 , 2017 . S. Narang. Green Coffee and Spam: Elaborate spam operation on Twitter uses nearly 750,000 accounts. https://symc.ly/2Cp7CJ5, 2017.
Cited by
Sorted by citations
(2020) · doi:10.1609/aaai.v34i01.5460 · Cited by 361 · see citing works
Realtime Robust Malicious Traffic Detection via Frequency Domain Analysis
Fu, C.-P., Li, Q., Shen, M., & Xu, K. (2021) · Conference on Computer and Communications Security · doi:10.1145/3460120.3484585 · Cited by 273 · see citing works
“Moreover, the specialized features of botnets have been used in botnet traffic detection [16, 20, 27, 30].”
Method(2020) · Sabanci University · Cited by 197 · see citing works
The False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184 · see citing works
What types of COVID-19 conspiracies are populated by Twitter bots?
Ferrara, E. (2020) · First Monday · doi:10.5210/fm.v25i6.10633 · Cited by 223 · see citing works
“(1) Deep learning [53], (2) Anomaly detection [54]–[57], (3) Time series analysis [38], [58], [59].”
BackgroundMeaningful measures of human society in the twenty-first century
Lazer, D. M. J., Hargittai, E., Freelon, D., González‐Bailón, S., Munger, K., Ognyanova, K., & Radford, J. P. (2021) · Nature · doi:10.1038/s41586-021-03660-7 · Cited by 135 · see citing works
Machine learning-based social media bot detection: a comprehensive literature review
Aljabri, M., Zagrouba, R., Shaahid, A., Alnasser, F., Saleh, A., & Alomari, D. M. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-022-01020-5 · Cited by 132 · see citing works
“Numerous studies were able to detect social bots on Twitter starting (Echeverra et al. 2018) tested 20 unseen bot classes of varying sizes and characteristics using bot classifiers.”
BackgroundBot, or not? Comparing three methods for detecting social bots in five political discourses
Martini, F., Samula, P., Keller, T. R., & Klinger, U. (2021) · Big Data & Society · doi:10.1177/20539517211033566 · Cited by 91 · see citing works
“A weakness of these classi fi ers lies in their inability to detect new classes of bots that have not been represented in the training data, as shown by Echeverría et al. (2018).”
BackgroundA Total Error Framework for Digital Traces of Human Behavior on Online Platforms
Sen, I., Flöck, F., Weller, K., Weiß, B., & Wagner, C. (2021) · Public Opinion Quarterly · doi:10.1093/poq/nfab018 · Cited by 113 · see citing works
Frequency Domain Feature Based Robust Malicious Traffic Detection
Fu, C.-P., Li, Q., Shen, M., & Xu, K. (2023) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2022.3195871 · Cited by 84 · see citing works
“Moreover, the specialized features of botnets have been used in botnet traffic detection [67], [68].”
MethodThe COVID‐19 scamdemic: A survey of phishing attacks and their countermeasures during COVID‐19
Al-Qahtani, A. F., & Cresci, S. (2022) · IET Information Security · doi:10.1049/ise2.12073 · Cited by 68 · see citing works
“The latter test in particular has proven valuable in other tasks for identifying detectors' generalization deficiencies and for estimating their capacity to thwart future and unforeseen attacks [111].”
BackgroundThrowing Darts in the Dark? Detecting Bots with Limited Data using Neural Data Augmentation
Jan, S. T. K., Hao, Q., Hu, T., Pu, J., Oswal, S., Wang, G., & Viswanath, B. (2020) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40000.2020.00079 · Cited by 70 · see citing works
“A common way is supervised training with labeled bot data and benign user data [28], [29], [21], [22].”
BackgroundOn the Detection of Disinformation Campaign Activity with Network Analysis
Vargas, L., Emami, P., & Traynor, P. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop · doi:10.1145/3411495.3421363 · Cited by 74 · see citing works
“to the rapid and immense scale at which information diffuses in real-time. As a result, there is an increased interest in automating the moderation of such spaces using carefully designed algorithms [6, 7, 9, 34]. Such content moderation can be done through bot detection systems like Botometer [37, 38] or disinformati…”
Background#COVID-19 on Twitter: Bots, Conspiracies, and Social Media Activism
Ferrara, E. (2020) · Europe PMC (PubMed Central) · Cited by 63 · see citing works
Hays, C., Schutzman, Z., Raghavan, M., Walk, E., & Zimmer, P. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583214 · Cited by 29 · see citing works
“Building on prior work [18, 60], we show that expressive machine learning models trained on one dataset do not perform well when tested on others and that models trained on all but one dataset perform poorly when evaluated on the held-out one.”
BackgroundMethodInfluentialUnderstanding anti-immigration sentiment spreading on Twitter
Nasuto, A., & Rowe, F. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307917 · Cited by 30 · see citing works
Bot Datasets on Twitter: Analysis and Challenges
Samper-Escalante, L. D., Loyola‐González, O., Monroy, R., et al. (2021) · Applied Sciences · doi:10.3390/app11094105 · Cited by 24 · see citing works
Robust Benchmark for Propagandist Text Detection and Mining High-Quality Data
Ahmad, P. N., Liu, Y., Ali, G., Wani, M. A., & ELAffendi, M. A. (2023) · Mathematics · doi:10.3390/math11122668 · Cited by 21 · see citing works
Cai, Z., Tan, Z., Lei, Z., Zhu, Z., Wang, H., Zheng, Q., & Luo, M. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635843 · Cited by 31 · see citing works
“• LOBO [7] extracts 19 features from each users’ metadata and tweets and adopts random forest for classification.”
BackgroundMethodInfluentialSocial Botomics: A Systematic Ensemble ML Approach for Explainable and Multi-Class Bot Detection
Dimitriadis, I., Georgiou, K., & Vakali, A. I. (2021) · Applied Sciences · doi:10.3390/app11219857 · Cited by 20 · see citing works
Unsupervised detection of coordinated fake-follower campaigns on social media
Zouzou, Y., & Varol, O. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00499-6 · Cited by 19 · see citing works
“Secondly, these models struggle to generalize well on unseen bot types in the training dataset[14], but novel approaches can address these limitations with novel machine learning systems[11].”
BackgroundDetection and impact estimation of social bots in the Chilean Twitter network
Mendoza, M., Providel, E., dos Santos, M. L. B., & Valenzuela, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-57227-3 · Cited by 19 · see citing works
New Automation for Social Bots: From Trivial Behavior to AI-Powered Communication
Grimme, C., Pohl, J., Cresci, S., Lüling, R., & Preuss, M. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-18253-2_6 · Cited by 19 · see citing works
CGNN: A Compatibility-Aware Graph Neural Network for Social Media Bot Detection
Huang, H., Tian, H., Zheng, X., Zhang, X., Zeng, D. D., & Wang, F. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3396413 · Cited by 20 · see citing works
“To address this problem, researchers have proposed to construct robust feature-based methods by mixing multiple datasets [50], [51] or integrating models from different fi elds [52].”
MethodMulBot: Unsupervised Bot Detection Based on Multivariate Time Series
Mannocci, L., Cresci, S., Monreale, A., Vakali, A. I., & Tesconi, M. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020363 · Cited by 19 · see citing works
Akhtar, M. M., Masood, R., Ikram, M., & Kanhere, S. S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3644998 · Cited by 16 · see citing works
Bot Detection on Social Networks Using Persistent Homology
Nguyen, M., Aktas, M. E., & Akbaş, E. (2020) · Mathematical and Computational Applications · doi:10.3390/mca25030058 · Cited by 12 · see citing works
Bots are Less Central than Verified Accounts during Contentious Political EventsPreprint
González‐Bailón, S., & Domenico, M. D. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3637121 · Cited by 11 · see citing works
CALEB: A Conditional Adversarial Learning Framework to enhance bot detection
Dimitriadis, I., Dialektakis, G., & Vakali, A. I. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102245 · Cited by 15 · see citing works
CB-MTE: Social Bot Detection via Multi-Source Heterogeneous Feature Fusion
Cheng, M., Xiao, Y., Huang, T., Lei, C., & Zhang, C. (2025) · Sensors · doi:10.3390/s25113549 · Cited by 5 · see citing works
Lira, D. B., Xavier, F. W., & Digiampietri, L. A. (2021) · doi:10.1145/3466933.3466970 · Cited by 5 · see citing works
Security of Social Networks: Lessons Learned on Twitter Bot Analysis in the Literature
Alipour, S. A., Orji, R., & Zincir-Heywood, N. (2022) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3538969.3544450 · Cited by 5 · see citing works
Pietro, R. D., Raponi, S., Caprolu, M., & Cresci, S. (2020) · Advances in information security · doi:10.1007/978-3-030-60618-3_2 · Cited by 4 · see citing works
Sangeethapriya, R., & Akilandeswari, J. (2022) · Smart innovation, systems and technologies · doi:10.1007/978-981-16-6616-2_19 · Cited by 4 · see citing works
A Privacy-Preserving Architecture for the Protection of Adolescents in Online Social Networks
Charalambous, M., Papagiannis, P., Papasavva, A., Leonidou, P., Constaninou, R., Terzidou, L., Christophides, T., Nicolaou, P., Theofanis, O., Kalatzantonakis, G., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.12038 · Cited by 4 · see citing works
Behaviour and Bot Analysis on Online Social Networks
Alipour, S. A., Orji, R., & Zincir-Heywood, N. (2023) · International Journal of Technology and Human Interaction · doi:10.4018/ijthi.327789 · Cited by 3 · see citing works
“…subsets: those on which Botometer has been trained, which are publicly available on the Botometer repository (https://botometer.osome.iu.edu/bot-repository/datasets.html), such as Cresci-rtbust-2019; and those on which Botometer has not been trained, such as the ones in Echeverra et al. (2018).”
MethodBeyond trial-and-error: Predicting user abandonment after a moderation intervention
Tessa, B., Cima, L., Trujillo, A., Avvenuti, M., & Cresci, S. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112375 · Cited by 4 · see citing works
“While the independence assumption is easily verified in some domains of application, such as that of bot detection where each bot belongs to only one botnet [52], it does not hold in our context where any user can participate in multiple subreddits.”
Integrating higher-order relations for enhanced twitter bot detection
Reiche, S., Cohen, S., Simonov, K., & Friedrich, T. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01372-0 · Cited by 3 · see citing works
“To ensure effective generalization of bot detection algorithms to unseen data, it is crucial to recognize that achieving high scores on common machine learning metrics does not automatically imply robust generalization capabilities (Echeverría et al. 2018).”
The Advantage of the Right in Social Media News SharingPreprint
González‐Bailón, S., d’Andrea, V., Freelon, D., & Domenico, M. D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3895410 · Cited by 3 · see citing works
Demystifying Misconceptions in Social Bots Research
Cresci, S., Yang, K.-C., Spognardi, A., Pietro, R. D., Menczer, F., & Petrocchi, M. (2025) · Social Science Computer Review · doi:10.1177/08944393251376707 · Cited by 12 · see citing works
Kats, D., & Sharif, M. (2022) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3527188.3561928 · Cited by 3 · see citing works
“Most bot-detection algorithms rely on machine-learning (ML) algorithms [4, 14, 19, 58].”
BackgroundMethodInfluentialPolychronis, C., & Kogan, M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610033 · Cited by 2 · see citing works
Networks and Influencers in Online Propaganda Events: A Comparative Study of Three Cases in India
Sen, A., De, S., & Pal, J. (2024) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3653709 · Cited by 1 · see citing works
“Researchers in this area thus eventually focused on detecting coordinated behavior [51] among a set of user nodes, rather than considering a node in isolation, using semi-supervised and unsupervised techniques [18, 34].”
BackgroundWirth, K., Menchen-Trevino, E., & Moore, R. T. (2019) · International Conference on Social Media and Society (SMSociety) · doi:10.1145/3328529.3328547 · Cited by 2 · see citing works
Instrumental Bot Identities and the Politics of Online Discourse
Letourneau, A. M. (2025) · Social Media + Society · doi:10.1177/20563051251378233 · Cited by 1 · see citing works
“This presents challenges for detecting bots (Echeverrïa et al., 2018; Gorwa & Guilbeault, 2020; Iliou et al., 2019; Martini et al., 2021).”
Dissecting chirping patterns of invasive Tweeter flocks in the German Twitter forest
Reubold, J. L., Escher, S., Pflugmacher, J., & Strufe, T. (2022) · Online Social Networks and Media · doi:10.1016/j.osnem.2022.100228 · Cited by 1 · see citing works
Early Malware Characterization based on Online Social Networks
Sadighian, A., Abbes, I., & Oligeri, G. (2023) · International Conference on Advanced Communication Technologies and Networking (CommNet) · doi:10.1109/commnet60167.2023.10365252 · Cited by 1 · see citing works
“Taking into account the increasing rate of botnets in OSN, various bot and botnet detection approaches have been proposed in the literature [6], [7], [14], [15], [16].”
BackgroundMethodBotScout: A Social Bot Detection Algorithm Based on Semantics, Attributes and Neighborhoods
Lin, H., Chen, N., Chen, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5581-3_28 · Cited by 1 · see citing works
Saeed, M. H., Ali, S., Paudel, P., Blackburn, J., & Stringhini, G. (2024) · International Symposium on Research in Attacks, Intrusions and Defenses (RAID) · doi:10.1145/3678890.3678911 · Cited by 3 · see citing works
Self-citationBSG4Bot:Efficient Bot Detection Based on Biased Heterogeneous Subgraphs
Miao, H., Liu, Z., & Gao, J. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00097 · Cited by 1 · see citing works
Machine Learning models for Customer Relationship Analysis to Improve Satisfaction Rate in Banking
Jia, N., Bagam, L., Fabijanczyk, P., & Ceesay, E. N. (2022) · IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS) · doi:10.1109/iemtronics55184.2022.9795855 · Cited by 1 · see citing works
TED-On: A Total Error Framework for Digital Traces of Human Behavior on Online PlatformsPreprint
Sen, I., Flöck, F., Weller, K., Weiß, B., & Wagner, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08228 · Cited by 2 · see citing works
Optimizing global processing time in the detection of patterns related to suicide in social networks
Diaz, D. M., Rosas, F. J. L., Romo, J. C. M., Hernández-Vargas, M. A., & ZUÑIGA, I. C. (2020) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci51800.2020.00049
A Hierarchical Attention-Based Neural Network Model for Socialbot Detection in OSN
Fazil, M., Sah, A. K., & Abulaish, M. (2020) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wiiat50758.2020.00146
“The reason behind the generation of SD 4 is that existing approaches detect well a particular category of socialbot but fail for other categories [32].”
Backgroundto Protect the Public Opinion Against New Types of Bots?
Reubold, J. L., Escher, S., Wressnegger, C., & Strufe, T. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020758
On the Influence and Political Leaning of Overlap between Propaganda Communities
Sen, A., Gun, S., De, S., & Pal, J. (2024) · ACM Journal on Computing and Sustainable Societies · doi:10.1145/3640790
“Researchers in this area thus eventually focused on detecting coordinated behavior [36] among a set of user nodes, rather than considering a node in isolation, using semi-supervised and unsupervised techniques [10, 27].”
Botnet Identification on Twitter: A Novel Clustering Approach Based on Similarity
Samper-Escalante, L. D., Loyola‐González, O., Monroy, R., & Medina‐Pérez, M. A. (2024) · IEEE Access · doi:10.1109/access.2024.3471630
“We followed this methodology to evaluate our solution with five approaches from the literature of bot detection and botnet detection on Twitter: Echever-ria et al.’s [17] [43] 2021 work.”
BackgroundMethodInfluentialRoleScan: Enhancing Social Bot Detection Using Social Role Vector
Wen, W., Gao, M., Gong, Q., et al. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-2373-0_5
Exploring Writing Style Consistency to Timely Identify Heterogeneous Social Bots
Laudanna, S., Cardaioli, M., Sorbo, A. D., Visaggio, C. A., & Conti, M. (2025) · Lecture notes in social networks · doi:10.1007/978-3-031-85386-9_21
Unified network ensemble and chunk-based feature selection for improved social bot detection
Sharma, J., & Borah, S. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02807-0
An Explainable Feature-Based Approach for Understanding Social Bots Behaviour
López-Joya, S., Diaz-Garcia, J. A., Ruiz, M. D., & Martín-Bautista, M. J. (2026) · Applied System Innovation · doi:10.3390/asi9070150
- Authors
- Echeverra, J., De Cristofaro, E., Kourtellis, N., Leontiadis, I., Stringhini, G., & Zhou, S.
- Affiliations
- University College London
- Telefonica Research and Development (ES)
- Telefonica Research
- Boston University (US)
- University College London (GB)
- Venue
- Proceedings of the 34th Annual Computer Security Applications Conference
- Publisher
- ACM
- Published
- 2018-12-03
- Pages
- 137-146
- ISSN
- 1063-9527 (ISSN-L)
- OpenAlex
- W2892724803
- Semantic Scholar
- 2736b2b6a0120ca2c72b2673f1e3028be547ae16
- Cited by
- 61
- References
- 46
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 12 · 2023: 7 · 2022: 11 · 2021: 8 · 2020: 12 · 2019: 2
- FWCI
- 4.32
- Type
- conference paper
- Language
- en
- Open access
- green
- zenodo.org
- Locations
- Annual Computer Security Applications Conference (publishedVersion)
- OpenBU (Boston University) (submittedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- European Commission (HORIZON2020, 691025)
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- botnets, social networking sites, bot detection, spam, offensive content, cybercrime, arms race
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
