TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Twitter has become a vital social media platform while an ample amount of malicious Twitter bots exist and induce undesirable social effects. Successful Twitter bot detection proposals are generally supervised, which rely heavily on large-scale datasets. However, existing benchmarks generally suffer from low levels of user diversity, limited user information and data scarcity. Therefore, these datasets are not sufficient to train and stably benchmark bot detection measures. To alleviate these problems, we present TwiBot-20, a massive Twitter bot detection benchmark, which contains 229,573 users, 33,488,192 tweets, 8,723,736 user property items and 455,958 follow relationships. TwiBot-20 covers diversified bots and genuine users to better represent the real-world Twittersphere. TwiBot-20 also includes three modals of user information to support both binary classification of single users and community-aware approaches. To the best of our knowledge, TwiBot-20 is the largest Twitter bot detection benchmark to date. We reproduce competitive bot detection methods and conduct a thorough evaluation on TwiBot-20 and two other public datasets. Experiment results demonstrate that existing bot detection measures fail to match their previously claimed performance on TwiBot-20, which suggests that Twitter bot detection remains a challenging task and requires further research efforts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 1×
“We adopt Accuracy, F1-score and MCC [18] as evaluation metrics.”
MethodOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981 · cited 3×
“Property Semantic Neighbor varol-icwsm [27] pronbots [31] ✓ celebrity [31] ✓ gilani-17 [13] ✓ cresci-rtbust [19] ✓ cresci-stock [7] ✓ midterm-18 [32] ✓ botwiki [32] ✓ verified [32] ✓ PAN-19 3 ✓ caverlee [15] ✓ ✓ cresci-17 [6] ✓ ✓ TwiBot-20 ✓ ✓ ✓ are also encouraged to download and use the full…”
BackgroundBotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909 · cited 2×
“• Botometer [9]: Botometer is a publicly available demo that leverages more than one thousand features to identify Twitter bots on demand.”
BackgroundDetecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg?
Chu, Z., Gianvecchio, S., Wang, H., & Jajodia, S. (2012) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2012.75 · Cited by 634 · cited 3×
“[3], which strengthens the claim that TwiBot-20 annotation is generally trustworthy.”
BackgroundInfluentialDisinformation and social bot operations in the run up to the 2017 French presidential election
Ferrara, E. (2017) · First Monday · doi:10.5210/fm.v22i8.8005 · Cited by 524 · cited 1×
“Some Twitter bots exploit Twitter features to pursue malicious goals such as election interference [10, 11] and extreme propaganda [2].”
BackgroundDeep neural networks for bot detection
Kudugunta, S., & Ferrara, E. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.019 · Cited by 506 · cited 3×
“Similar to previous research [14, 32], we treat Twitter bot detection as a binary classification problem, where each user could either be human ( 𝑦 = 0) or bot ( 𝑦 = 1).”
BackgroundMethodFame for sale: Efficient detection of fake Twitter followers
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.09.003 · Cited by 475 · cited 1×
“Another early dataset is cresci-2015 [4] which provides a dataset of genuine accounts and fake followers.”
BackgroundDesign and Evaluation of a Real-Time URL Spam Filtering Service
Thomas, K., Grier, C., Ma, J., Paxson, V., & Song, D. (2011) · 2011 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2011.25 · Cited by 468
Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter
Lee, K., Eoff, B., & Caverlee, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14106 · Cited by 452 · cited 4×
“Dataset cresci-17 [6] and caverlee [15] contain both semantic and property information, while all other existing baselines only include user semantic or property information.”
BackgroundInfluentialArming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419 · cited 4×
“Property Semantic Neighbor varol-icwsm [27] pronbots [31] ✓ celebrity [31] ✓ gilani-17 [13] ✓ cresci-rtbust [19] ✓ cresci-stock [7] ✓ midterm-18 [32] ✓ botwiki [32] ✓ verified [32] ✓ PAN-19 3 ✓ caverlee [15] ✓ ✓ cresci-17 [6] ✓ ✓ TwiBot-20 ✓ ✓ ✓ are also encouraged to download and use the full…”
BackgroundMethodInfluentialThe Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3055135 · Cited by 381 · cited 6×
“Regarding dataset size, the number of users included in each dataset also range from 693 users in cresci-rtbust [19] to 50,538 in midterm-18 [32], and one of the most widely used dataset cresci-17 [6] contains 2,764 human users and 7,049 bots.”
BackgroundMethodInfluential(2020) · doi:10.1609/aaai.v34i01.5460 · Cited by 361
Empirical evaluation and new design for fighting evolving Twitter spammers
Yang, C., Harkreader, R., & Gu, G. (2013) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2013.2267732 · Cited by 346 · cited 1×
“Yang et al. [30] designed novel features to counter the evolution of Twitter bots.”
BackgroundTwitter spammer detection using data stream clustering
Miller, Z., Dickinson, B., Deitrick, W., Hu, W., & Wang, A. H. (2014) · Information Sciences · doi:10.1016/j.ins.2013.11.016 · Cited by 332
Towards Online Spam Filtering in Social Networks
Gao, H., Chen, Y., Lee, K., Palsetia, D., & Choudhary, A. (2012) · Cited by 237
The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race
Cresci, S., Pietro, R. D., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · doi:10.48550/arxiv.1701.03017 · Cited by 224 · cited 6×
“Regarding dataset size, the number of users included in each dataset also range from 693 users in cresci-rtbust [19] to 50,538 in midterm-18 [32], and one of the most widely used dataset cresci-17 [6] contains 2,764 human users and 7,049 bots.”
BackgroundMethodInfluentialOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C. A., Menczer, F., & Flammini, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03107 · Cited by 219 · cited 3×
“Property Semantic Neighbor varol-icwsm [27] pronbots [31] ✓ celebrity [31] ✓ gilani-17 [13] ✓ cresci-rtbust [19] ✓ cresci-stock [7] ✓ midterm-18 [32] ✓ botwiki [32] ✓ verified [32] ✓ PAN-19 3 ✓ caverlee [15] ✓ ✓ cresci-17 [6] ✓ ✓ TwiBot-20 ✓ ✓ ✓ are also encouraged to download and use the full…”
BackgroundWarningBird: A Near Real-Time Detection System for Suspicious URLs in Twitter Stream
Lee, S., & Kim, J. (2013) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2013.3 · Cited by 201 · cited 1×
“Lee et al. [17] proposed to verify URL redirections in tweets.”
Background(2020) · Sabanci University · Cited by 197
DNA-Inspired Online Behavioral Modeling and Its Application to Spambot Detection
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.29 · Cited by 194 · cited 1×
“Other features are also adopted such as information on the user homepage [16], social networks [22], and timeline of accounts [5].”
BackgroundDetect Me If You Can: Spam Bot Detection Using Inductive Representation Learning
Alhosseini, S. A., Tareaf, R. B., Najafi, P., & Meinel, C. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3316504 · Cited by 174 · cited 4×
“To prove that achieving good performance on one type of bot doesn’t necessarily indicate the ability to identify diversified bots, we train the community-based bot detector Alhos-seini et al. [1] on only one of the four interest domains in TwiBot-20 and evaluate it on the full test set.”
BackgroundMethodInfluentialTwitter Bot Detection Using Bidirectional Long Short-Term Memory Neural Networks and Word Embeddings
Wei, F., & Nguyen, U. T. (2019) · 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa48467.2019.00021 · Cited by 168 · cited 2×
“[29] used recurrent neural networks to identify bots with tweet semantics.”
MethodRTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter
Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., & Tesconi, M. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326015 · Cited by 163 · cited 3×
“Regarding dataset size, the number of users included in each dataset also range from 693 users in crescirtbust [19] to 50,538 in midterm-18 [32], and one of the most widely used dataset cresci-17 [6] contains 2,764 human users and 7,049 bots.”
BackgroundDetection of Novel Social Bots by Ensembles of Specialized Classifiers
Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3412698 · Cited by 159 · cited 1×
“Apart from that, verified-2019 [32], botwiki-2019 [32], cresci-rtbust-2019 [20], Astroturf [24] are also more recent bot detection datasets of various size and information completeness.”
BackgroundOf Bots and Humans (on Twitter)
Gilani, Z., Farahbakhsh, R., Tyson, G., Wang, L., & Crowcroft, J. (2017) · Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017 · doi:10.1145/3110025.3110090 · Cited by 124 · cited 3×
“Regarding user information, while caverlee [15] provides semantic and property information of Twitter users, gilani-17 [13] and others only include property information, and they vary greatly in user information completeness.”
BackgroundPerils and Challenges of Social Media and Election Manipulation Analysis: The 2018 US Midterms
Deb, A., Luceri, L., Badaway, A., & Ferrara, E. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3316486 · Cited by 90 · cited 1×
“Some Twitter bots exploit Twitter features to pursue malicious goals such as election interference [10, 11] and extreme propaganda [2].”
Background$FAKE: Evidence of Spam and Bot Activity in Stock Microblogs on Twitter
Cresci, S., Lillo, F., Regoli, D., Tardelli, S., & Tesconi, M. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15073 · Cited by 73 · cited 2×
“varol-icwsm [27] 2,573 0 0 0 pronbots [31] 21,965 750,991 0 0 celebrity [31] 5,971 879,954 0 0 gilani-17 [13] 2,653 104,515 0 0 cresci-rtbust [19] 693 28,968 0 0 cresci-stock [7] 13,276 551,603 0 0 midterm-18 [32] 50,538 909,684 0 0 botwiki [32] 698 29,082 0 0 verified [32] 1,987 83,383 0 0 PAN-193 11,568 0 369,246 0…”
BotWalk: Efficient Adaptive Exploration of Twitter Bot Networks
Minnich, A., Chavoshi, N., Koutra, D., & Mueen, A. (2017) · Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017 · doi:10.1145/3110025.3110163 · Cited by 60 · cited 1×
“Other features are also adopted such as information on the user homepage [16], social networks [22], and timeline of accounts [5].”
BackgroundFrom Reaction to Proaction: Unexplored Ways to the Detection of Evolving Spambots
Cresci, S., Petrocchi, M., Spognardi, A., & Tognazzi, S. (2018) · Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18 · doi:10.1145/3184558.3191595 · Cited by 28 · cited 1×
“[8] points out, studying user communities is essential in future bot detection endeavors, where TwiBot-20 enables it by providing users’ neighborhood information and the two other datasets fall short.”
BackgroundBOT SPAMMER DETECTION IN TWITTER USING TWEET SIMILARITY AND TIME INTERVAL ENTROPY
Perdana, R. S., Muliawati, T. H., & Alexandro, R. (2015) · Jurnal Ilmu Komputer dan Informasi · doi:10.21609/jiki.v8i1.280 · Cited by 20 · cited 1×
“[23] observed that bot users would generate a lot of repeated tweets, leading to a higher total tweet count.”
BackgroundGANs for Semi-Supervised Opinion Spam Detection
Stanton, G., & Irissappane, A. A. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/723 · Cited by 18
Cited by
Few voices, strong echo: Measuring follower homogeneity of politicians’ Twitter accounts
Rusche, F. (2024) · New Media & Society · doi:10.1177/14614448221099860 · Cited by 6
Metadata
- Authors
- Feng, S., Wan, H., Wang, N., Li, J., & Luo, M.
- Venue
- Proceedings of the 30th ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2021-10-26
- Pages
- 4485-4494
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W3175498457
- Cited by
- 197
- References
- 31
- FWCI
- 23.40
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.