The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; none of the 23 sources its claims cite is checked for support yet.
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Open accessAbstract
Malicious actors create inauthentic social media accounts controlled in part by algorithms, known as social bots, to disseminate misinformation and agitate online discussion. While researchers have developed sophisticated methods to detect abuse, novel bots with diverse behaviors evade detection. We show that different types of bots are characterized by different behavioral features. As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data. Moreover, tuning these models to recognize novel bots requires retraining with a significant amount of new annotations, which are expensive to obtain. To address these issues, we propose a new supervised learning method that trains classifiers specialized for each class of bots and combines their decisions through the maximum rule. The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets. Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining. We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an Ensemble of Specialized Classifiers (ESC) that trains separate Random Forest classifiers for different bot classes and combines them via the maximum rule to improve detection of novel social bots. ESC improves cross-domain bot recall and F1 over Botometer-v3 and is deployed in Botometer v4 with a cross-validation AUC of 0.99.
Key findings
- ESC improves cross-domain performance on unseen accounts: across six datasets, recall rises from 42% to 84%, precision from 52% to 64%, and F1 from 47% to 73%, an average F1 improvement of 56%. On combined-test, recall goes from 77% to 86%, precision stays at 70%, and F1 rises from 73% to 77% (5% improvement).
- In-domain performance is maintained: ESC achieves 0.96 AUC under 5-fold cross-validation, similar to Botometer’s 0.97 AUC, and the deployed Botometer v4 model yields a cross-validation AUC of 0.99.
- Different bot classes rely on different informative features: in cresci-17, traditional spambots are best separated by adjective-frequency standard deviation, social spambots by tweet sentiment arousal entropy, and fake followers by maximum friend-follower ratio.
- ESC adapts faster to new domains: when adding labeled examples from a held-out domain, ESC learns through a newly added specialized classifier while preserving old classifiers, requiring fewer labeled examples than retraining Botometer-v3 from scratch.
Methods
- The study uses labeled bot/human datasets from the Bot Repository plus datasets from Rauchfleisch and Kaiser, extracting over 1,200 features in six categories: metadata, retweet/mention networks, temporal features, content, and sentiment.
- ESC combines a human-detection classifier (RF0, corresponding to Botometer) with specialized Random Forest classifiers trained on balanced bot classes and human examples; bot scores are aggregated with the maximum rule, with Platt scaling in experiments.
- Evaluation includes 5-fold cross-validation for in-domain performance and hold-out cross-domain tests on cresci-stock, gilani-17, cresci-rtbust, kaiser-1, kaiser-2, kaiser-3, and combined-test, using F1 and AUC with bootstrap confidence intervals; adaptation is tested on varol-icwsm.
Limitations
- The paper states that cross-domain performance is highly sensitive to the datasets used in training and testing, making it easy to cherry-pick examples that make any method look bad; kaiser-1 and kaiser-3 are not completely hold-out because they include botwiki bots from ESC training, kaiser-3 shares accounts with kaiser-2 and kaiser-1, and kaiser-2 has only 27 bot accounts, making F1 sensitive to class imbalance.
- The paper notes that tweetbotornot2 uses a “verified” profile feature and tends to assign low bot scores to verified accounts, biasing comparisons in its favor on datasets where many human-labeled accounts are verified.
- The text makes evident that ground-truth labels can become invalid over time as accounts change, become inactive, or are compromised; the manual inspection of suspected mislabeled accounts covered only a random sample of 50 out of 332 highlighted accounts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
21 Own result · 18 Not checked yet · 16 No basis given · 14 Own interpretation · 3 Own method
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that different types of bots are characterized by different behavioral features.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data.”
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
“Both precision and recall tend to be higher for in-domain cases, demonstrating the limited generalization of supervised models across domains.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining.”
- Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
“It can also learn more efficiently from examples in new domains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using bot and human accounts in different datasets, we show that bot accounts exhibit greater heterogeneity in their discriminative behavioral features compared to human accounts.”
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
“Human accounts, on the other hand, exhibit consistent characteristics across datasets.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fig. 7 shows that ESC outperforms the Botometer baseline in most cases.”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“ESC yields F1 better than or comparable with tweetbotornot2 on all datasets except those from Rauchfleisch and Kaiser [35].”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“kaiser-2 includes only 27 bot accounts”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Manual inspection of a random sample of 50 of the accounts highlighted in Fig. 5 reveals that the human labels are no longer accurate for most of them - they have been inactive for years, are currently devoted to spam diffusion, and/or are controlled by third-party applications.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The scatter plot in Fig. 5 shows a good agreement between the bot scores obtained with ESC and the Botometer-v3 baseline (Spearman's ρ = 0 . 87).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The quote only partly states this claim. 3
- It may state two separate claims as one. 29
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets.”
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
“On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%. As a result, F1 increases from 47% to 73% (an improvement of 56%).”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that bot accounts tend to have lower cross-domain scores, resulting in lower recall.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the in-domain case (left), the linear classifier is sufficient to separate human and bot accounts. But in the cross-domain case (right), the same linear model fails, explaining the drop in recall: different features and distinct decision rules are needed to detect different classes of bots.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without loss of in-domain accuracy, the proposed approach effectively increases the recall of cross-domain bots.”
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
“On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%.”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the combined-test dataset, ESC outperforms tweetbotornot2 on both F1 and AUC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ESC is comparable to tweetbotornot2 on this dataset when using the AUC metric, which is not sensitive to class imbalance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AUC metric is not sensitive to class imbalance.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“kaiser-3 contains human accounts from kaiser-2 and bot accounts from kaiser-1 , so they are not independent.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“98% and 72% of accounts labeled human in kaiser-1 and kaiser-3 respectively are verified.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We focus on 332 accounts in this region that are labeled as human (75% are from caverlee ), which represent approximately 1% of the examples labeled as human in the training data.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the combined-test dataset, recall goes from 77% to 86%, precision stays at 70%, and F1 goes from 73% to 77% (an improvement of 5%).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“it uses a feature for 'verified' profiles and tends to assign a low bot score to them, even automated ones such as @twittersupport .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using 5-fold cross-validation, ESC achieves an Area Under the ROC Curve (AUC) of 0.96, similar to the Botometer classifier (0.97 AUC).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if we exclude botwiki from the training data, ESC still outperforms Botometer-v3 . For example, it yields an F1 score of 0.84 on kaiser-1 and 0.80 on kaiser-3 .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite impressive results when training and test sets are from the same domain - even using cross-validation - supervised models will miss new classes of bots, leading to low recall.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We reach similar conclusions using other hold-out datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Botometer-v3 baseline adapts more slowly.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We showed that this approach allows the system to learn about new domains in an efficient way, in the sense that fewer annotated examples are necessary.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our experiments show that cross-domain performance is highly sensitive to the datasets used in training and testing supervised models; it is easy to cherry-pick examples that make any given method look bad.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrated empirically that our proposed approach generalizes better than a monolithic classifier and is more robust to mislabeled training examples.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we are deploying ESC in the newest version of Botometer (v4)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fig. 8 shows how the F1 score on the test data improves as a function of the number of training examples from the hold-out domain.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Since ESC provides a good correspondence between scores and binary labels of human and bot accounts, no calibration is applied.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Bots with unseen characteristics are easily missed, as demonstrated by a drastic drop in recall when classifiers are faced with cross-domain accounts [18].”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The first baseline is the current version of Botometer, often considered the state-of-the-art method for bot detection [48].”
What the sources say
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In an arms race between abusive behaviors and countermeasures, novel social bots emerge everyday and evade purge from the platforms [15, 21].”
What the sources say
- Not checked yetcites [15] [15] Stefano Cresci, Marinella Petrocchi, Angelo Spognardi, and Stefano Tognazzi. 2019. Better safe than sorry: an adversarial approach to improve social bot detection. In Proceedings of the 10th ACM Conference on Web Science . 47-56. - source
- Not checked yetcites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“One common approach to address the lack of generalization is to retrain models with new labeled datasets [48].”
What the sources say
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: High-quality datasets of annotated social media accounts are expensive to acquire.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We obtain confidence intervals using bootstrapping with five samples of 80% of the accounts in the test set.”
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The kaiser-1 and kaiser-3 datasets include bots from botwiki , which are part of the ESC training data. Therefore these two cannot be considered completely hold-out datasets, but are included nonetheless because they were used to highlight weaknesses of Botometer-v3 in a recent independent report [35], so they provide us with an opportunity to demonstrate the performance of the latest Botometer”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The language agnostics version of Botometer (excluding English-based linguistic features) is used on the kaiser-2 dataset because of its German tweet content.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use 5-fold cross-validation for in-domain classification; for consistency we split training and test samples in cross-domain cases as well, reporting average precision and recall.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Botometer currently does not use the 'verified' status as a feature because it could lead to false negatives [45].”
What the sources say
- Not checked yetcites [45] [45] Onur Varol and Ismail Uluturk. 2020. Journalists on Twitter: self-branding, audiences, and involvement of bots. Journal of Computational Social Science 3, 1 (01 Apr 2020), 83-101. https://doi.org/10.1007/s42001-019-00056-6 - source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the old bot classes with more examples dominate and the classifier struggles to learn about the new domain.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.
In the paper
“Rauchfleisch and Kaiser [35] may overestimate the false positive rate by assuming that no politician account uses automation.”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we share these concerns and acknowledge that like any supervised learning model, Botometer makes mistakes.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Even a classifier trained on 1.5 million accounts and 22 classes of bots is incapable of identifying new classes of bots.”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Another source of bias is that Rauchfleisch and Kaiser [35] overlook accounts that are no longer available due to suspension, possibly leading to an underestimation of both precision and recall.”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the new version introduced in the paper is motivated by this issue and partly addresses it by improving cross-domain recall (false negatives).”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Echeverría et al. [18] proposed the leaveone-botnet-out methodology, to highlight how detection methods do not generalize well to unseen bot classes due to bias in the training data.”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ESC architecture quickly learns about new bots through the new classifier, which starts from scratch, while what was learned about the old ones is preserved in the existing classifiers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“since bots change and evolve continuously to evade detection, supervised learning algorithms need to adapt to new classes of bots [21, 44, 48].”
What the sources say
- Not checked yetcites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source
- Not checked yetcites [44] [44] Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018). -
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also include additional specialized classifiers when new annotated datasets become available.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Echeverria and Zhou [19] suggest that detecting different types of bots requires different types of features.”
What the sources say
- Not checked yetcites [19] [19] Juan Echeverria and Shi Zhou. 2017. Discovery of the Twitter Bursty Botnet. arXiv preprint arXiv:1709.06740 (2017). -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this way, the tool offers greater transparency about the decision process by allowing inspection of the outputs of different ensemble components and by providing interpretable class labels in output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proposed architecture is highly modular as each specialized classifier works independently, so one can substitute any part with different models as needed.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“They also report that some bot classes, like social spambots, are hard to distinguish from human accounts.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
High-quality datasets of annotated social media accounts are expensive to acquire.
“Unfortunately, high-quality datasets of annotated social media accounts are expensive to acquire.”
The authors state this as an assumption.
The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
“These datasets made an assumption that all accounts belonging to American and German politicians are human accounts.”
The paper relies on this without support.
Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.
“Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.”
The paper relies on this without support.
Periodic retraining to include newly annotated datasets helps systems adapt to unseen examples.
“Periodic retraining to include newly annotated datasets helps systems adapt to these unseen examples.”
The paper relies on this without support.
A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.
“A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.”
The paper relies on this without support.
The F1 score is sensitive to class imbalance.
“the F1 score is sensitive to this class imbalance”
The paper relies on this without support.
The AUC metric is not sensitive to class imbalance.
“the AUC metric, which is not sensitive to class imbalance”
The paper relies on this without support.
The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.
“The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.”
The paper relies on this without support.
No systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.
“Despite the existence of different types of bots, no systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An important limitation of these methods is that they rely on features crafted specifically to target previously observed malicious behaviors [18].”
From A language framework for modeling social media account behavior (2023) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These findings are consistent with prior analysis based on ad-hoc features [18].”
From A language framework for modeling social media account behavior (2023) · cites this paper as [18]
Citations
Cites
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
Self-citationPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · cited 2×
“The bot score returned by ESC is obtained by calibrating the score si∗ using Platt’s scaling [31].”
MethodWeaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018) · American Journal of Public Health · doi:10.2105/ajph.2018.304567 · Cited by 1,149 · cited 1×
“spreading misinformation [36, 37], amplifying popularity [9, 34, 45], or polarizing online discussion [39]. Bot activity has been reported in different domains, including politics [5, 20, 39], health [2, 3, 6, 17], and business [12, 13]. Due to the wide adoption of social media, every aspect of people’s life from news…”
BackgroundOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981 · cited 3×
“posed in this paper, extract various features from an account’s profile, social network, and content [16, 23, 29, 41, 43, 48, 49].”
BackgroundMethodSelf-citationBotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909
Self-citationRatkiewicz (2011) · doi:10.1145/1963192.1963301 · Cited by 541
Self-citationDeep neural networks for bot detection
Kudugunta, S., & Ferrara, E. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.019 · Cited by 506 · cited 1×
“Depending on the type of bots, similarity can be detected through tweet content [8, 25], temporal features in the timelines [7, 10], or retweeting behavior [28, 46].”
BackgroundBots increase exposure to negative and inflammatory content in online social systems
Stella, M., Ferrara, E., & De Domenico, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1803470115 · Cited by 492
The DARPA Twitter Bot Challenge
Subrahmanian, V. S., Azaria, A., Durst, S., Kagan, V., Galstyan, A., Lerman, K., Zhu, L., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Computer · doi:10.1109/mc.2016.183 · Cited by 480 · cited 2×
“number of links to legitimate accounts. Supervised machine learning approaches, such as the one proposed in this paper, extract various features from an account’s profile, social network, and content [16, 23, 29, 41, 43, 48, 49]. These methods rely on annotated datasets for learning the difference between bot and huma…”
BackgroundSelf-citationFame for sale: Efficient detection of fake Twitter followers
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.09.003 · Cited by 475
Research works printed but not linked: 9
- Jon-Patrick Allem , Emilio Ferrara , Sree Priyanka Uppu , Tess Boley Cruz, and Jennifer B Unger. 2017 . E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98. Jon-Patrick Allem, Emilio Ferrara, Sree Priyanka Uppu, Tess Boley Cruz, and Jennifer B Unger. 2017. E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98.
- Alessandro Bessi and Emilio Ferrara . 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 ( 2016 ). Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 (2016).
- Emilio Ferrara . 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 ( 2017 ). Emilio Ferrara. 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 (2017).
- Kyumin Lee , Brian David Eoff , and James Caverlee . 2011 . Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter . In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM). Kyumin Lee, Brian David Eoff, and James Caverlee. 2011. Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter. In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM).
- Fabian Pedregosa , Gaël Varoquaux , Alexandre Gramfort , Vincent Michel , Bertrand Thirion , Olivier Grisel , Mathieu Blondel , Peter Prettenhofer , Ron Weiss , Vincent Dubourg , 2011 . Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830.
- Chengcheng Shao , Giovanni Luca Ciampaglia , Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018 . The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787. Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018. The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787.
- Galen Stoking and Nami Sumida . 2018. Social Media Bots Draw Public?s Attention and Concern . Pew Research Center ( 15 Oct 2018 ). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/ Galen Stoking and Nami Sumida. 2018. Social Media Bots Draw Public?s Attention and Concern. Pew Research Center (15 Oct 2018). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/
- Onur Varol , Clayton A Davis , Filippo Menczer , and Alessandro Flammini . 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics ( 2018 ), 311--334. Onur Varol, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics (2018), 311--334.
- Onur Varol and Ismail Uluturk . 2018. Deception strategies and threats for online discussions. First Monday 23, 5 ( 2018 ). Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018).
Cited by
Sorted by citations
TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
Feng, S., Wan, H., Wang, N., Li, J., & Luo, M. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482019 · Cited by 199
“Apart from that, verified-2019 [32], botwiki-2019 [32], cresci-rtbust-2019 [20], Astroturf [24] are also more recent bot detection datasets of various size and information completeness.”
BackgroundThe False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184
“The Botometer creators [43] argue in their newest paper that some of the German politicians’ accounts used in our study might have a certain degree of automation.”
BackgroundBotometer 101: social bot practicum for computational social scientists
Yang, K.-C., Ferrara, E., & Menczer, F. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00177-5 · Cited by 159
“This tutorial focuses on V4 [41].”
BackgroundSelf-citationUncovering Coordinated Networks on Social Media: Methods and Case Studies
Pacheco, D., Hui, P.-M., Torres-Lugo, C., Truong, B. T., Flammini, A., & Menczer, F. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18075 · Cited by 140
Self-citationMachine learning-based social media bot detection: a comprehensive literature review
Aljabri, M., Zagrouba, R., Shaahid, A., Alnasser, F., Saleh, A., & Alomari, D. M. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-022-01020-5 · Cited by 132
“In the work of (Sayyadiharikandeh et al. 2020) for each class of bots, they recommended training specialized classifiers and combining their conclusions using the maximum rule.”
BackgroundMethod#Election2020: the first public Twitter dataset on the 2020 US Presidential election
Chen, E., Deb, A., & Ferrara, E. (2021) · Journal of Computational Social Science · doi:10.1007/s42001-021-00117-9 · Cited by 116
“We used Indiana University’s Botometer, a tool that assigns a bot-score to a Twitter account based on an account’s activity [14, 15].”
MethodDeeProBot: a hybrid deep neural network model for social bot detection based on user profile data
Hayawi, K., Mathew, S. S., Venugopal, N., Masud, M. M., & Ho, P. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00869-w · Cited by 98
“Social spambots tend to attack or support political candidates, and fake followers tend to have aggressive following patterns (Sayyadiharikandeh et al. 2020).”
BackgroundInfluentialBot, or not? Comparing three methods for detecting social bots in five political discourses
Martini, F., Samula, P., Keller, T. R., & Klinger, U. (2021) · Big Data & Society · doi:10.1177/20539517211033566 · Cited by 91
“4 Sayyadiharikandeh et al. (2020) responded to that critique by adopting a new version of Botometer (v4) in September 2020.”
BackgroundAn analysis of AstraZeneca COVID-19 vaccine misinformation and fear mongering on Twitter
Jemielniak, D., & Krempovych, Y. (2021) · Public Health · doi:10.1016/j.puhe.2021.08.019 · Cited by 77
Are You a Cyborg, Bot or Human?—A Survey on Detecting Fake News Spreaders
Shahid, W., Li, Y., Staples, D., Gilkar, G. A., Hakak, S., & Ghorbani, A. A. (2022) · IEEE Access · doi:10.1109/access.2022.3157724 · Cited by 79
“Bot Detection PyTrawler 2021 [63] BadBots [64] Cresci [65], [66], [67] PAN 2020 [68], [69], [70] Social Honeypot [71], [66]”
BackgroundSocial Bots and Their Coordination During Online Campaigns: A Survey
Khaund, T., Kirdemir, B., Agarwal, N., Liu, H., & Morstatter, F. (2021) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2021.3103515 · Cited by 78
Anatomy of an AI-powered malicious social botnet
Yang, K., & Menczer, F. (2024) · Journal of Quantitative Description: Digital Media · doi:10.51685/jqd.2024.icwsm.7 · Cited by 71
Self-citationDeep Learning Based Social Bot Detection on Twitter
Arın, E., & Kutlu, M. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3254429 · Cited by 66
“…of researchers worked on various problems about social bots, such as detecting automatically-generated posts [20] and accounts controlled by bots [21], [22], re-evaluating social bot detection systems [23], [24], predicting victims who are vulnerable to attacks of social bots [25], and others.”
BackgroundInfluentialTruică, C., Apostol, E., Nicolescu, R.-C., & Karras, P. (2023) · IEEE Access · doi:10.1109/access.2023.3331220 · Cited by 64
“Sayyadiharikandeh et al. [36] address the problem of fake news mitigation assuming that non-human entities in social networks are set up to spread false information in an automated way.”
BackgroundBotMoE: Twitter Bot Detection with Community-Aware Mixtures of Modal-Specific Experts
Liu, Y., Tan, Z., Wang, H., Feng, S., Zheng, Q., & Luo, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591646 · Cited by 61
“The distribution of Twitter bots is heterogeneous across the whole Twitter network [50? ], while bot behavior and characteristics greatly vary from one user community to another [54].”
BackgroundThe Spread of Propaganda by Coordinated Communities on Social Media
Hristakieva, K., Cresci, S., Martino, G. D. S., Conti, M., & Nakov, P. (2022) · ACM Web Science Conference (WebSci) · doi:10.1145/3501247.3531543 · Cited by 64
Identifying and characterizing superspreaders of low-credibility content on Twitter
DeVerna, M., Aiyappa, R., Pacheco, D. F., Bryden, J., & Menczer, F. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302201 · Cited by 73
“This approach ensures our work’s comparability with other pertinent studies [47].”
ResultSelf-citationPolitical audience diversity and news reliability in algorithmic ranking
Bhadani, S., Yamaya, S., Flammini, A., Menczer, F., Ciampaglia, G. L., & Nyhan, B. J. (2022) · Nature Human Behaviour · doi:10.1038/s41562-021-01276-5 · Cited by 45
“Second, because diversity is a property of the audience and not of its level of engagement, it is less susceptible to manipulation if one can detect inauthentic partisan accounts [44, 49, 52, 53].”
BackgroundSelf-citationDetecting bots in social-networks using node and structural embeddings
Dehghan, A., Siuta, K., Skorupka, A., Dubey, A., Betlen, A., Miller, D., Xu, W., Kamiński, B., & Prałat, P. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00796-3 · Cited by 53
Comparative analysis of social bots and humans during the COVID-19 pandemic
Chang, H.-C. H., & Ferrara, E. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00173-9 · Cited by 41
“Identifying bots is a methodological arms race, and as such, we employ the latest version of Botometer (version 4) [14].”
BackgroundMethodBruno, M., Lambiotte, R., & Saracco, F. (2022) · EPJ Data Science · doi:10.1140/epjds/s13688-022-00330-0 · Cited by 40
“To assign a bot score to each account, we used the Botometer API [20] in its most recent v4 version [21].”
MethodSystematic Literature Review of Social Media Bots Detection Systems
Ellaky, Z., Benabbou, F., & Ouahabi, S. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.04.004 · Cited by 38
“This work (Sayyadiharikandeh et al., 2020) suggested ensembles of specialized classifiers (ESC) to identify political bots using user information, behavior, friends, networks, temporal, and content features.”
A global comparison of social media bot and human characteristics
Ng, L. H. X., & Carley, K. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-96372-1 · Cited by 42
“…while successfully keeping their artificial identity undisclosed One of the problems with existing definitions is that they often define bots as being malicious and they highlight the nefarious use of bots 5,34–37 : “malicious actors”, “public opinion manipulation”, “malicious tasks” 18,38,39 .”
Graham, T., Bruns, A., Angus, D. J., Hurcombe, E., & Hames, S. (2020) · Media International Australia · doi:10.1177/1329878x20981780 · Cited by 38
“We use the state-of-the-art Botometer bot detection tool (Sayyadiharikandeh et al, 2020), which uses a machine learning-based approach to score a given Twitter account based on how likely it is to be fully automated.”
BackgroundMethodCharacterizing the roles of bots on Twitter during the COVID-19 infodemic
Xu, W., & Sasahara, K. (2021) · Journal of Computational Social Science · doi:10.1007/s42001-021-00139-3 · Cited by 36
“Botom-eter is a well-known tool for automatically detecting bots based on supervised machine learning models [17, 18].”
MethodEvolution of Public Opinion on COVID-19 Vaccination in Japan: Large-Scale Twitter Data Analysis
Kobayashi, R., Takedomi, Y., Nakayama, Y., Suda, T., Uno, T., Hashimoto, T., Toyoda, M., Yoshinaga, N., Kitsuregawa, M., & Rocha, L. E. C. (2022) · Journal of Medical Internet Research · doi:10.2196/41928 · Cited by 31
Bots, disinformation, and the first impeachment of U.S. President Donald Trump
Rossetti, M., & Zaman, T. R. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283971 · Cited by 34
“…aggregate news content or distribute links [2], financial bots which advertise commercial products or attempt to influence financial markets [2], “astroturf” bots which promote political figures and their policies [2, 3], and fake follower accounts used for inflating a user’s follower network [2].”
BackgroundMethod“This is Fake! Shared it by Mistake”:Assessing the Intent of Fake News Spreaders
Zhou, X., Shu, K., Phoha, V. V., Liu, H., & Zafarani, R. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512264 · Cited by 38
“We identify bots and trolls by collecting data from two wellestablished and widely accepted platforms, Botometer [29]7 and Bot Sentinel.”
MethodSmart, B., Watt, J., Benedetti, S., Mitchell, L., & Roughan, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19097-1_3 · Cited by 42
Lindelöf, G., Aledavood, T., & Keller, B. (2023) · Journal of Medical Internet Research · doi:10.2196/41319 · Cited by 36
Hays, C., Schutzman, Z., Raghavan, M., Walk, E., & Zimmer, P. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583214 · Cited by 29
“Following Botometer [60], we define combined bot-type datasets as follows.”
BackgroundMethodResultInfluentialZhao, B., Ren, W., Zhu, Y., & Zhang, H. (2023) · Journal of Information Technology & Politics · doi:10.1080/19331681.2023.2189201 · Cited by 30
“This study uses Botometer v4 (Sayyadiharikandeh, Varol, Yang, Flammini, & Menczer, 2020) to detect social bots.”
MethodSuárez-Lledó, V., & Álvarez‐Gálvez, J. (2022) · Journal of Medical Internet Research · doi:10.2196/36085 · Cited by 34
“Self-declared bots are extracted from Botwiki [28,32].”
MethodToward Mitigating Misinformation and Social Media Manipulation in LLM Era
Zhang, Y., Sharma, K., Du, L., & Liu, Y. (2024) · Companion Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589335.3641256 · Cited by 25
Plikynas, D., Rizgelienė, I., & Korvel, G. (2025) · IEEE Access · doi:10.1109/access.2025.3530688 · Cited by 30
“It is worth noting that cross-domain performance analysis for traditional ML methods has only been applied to bot detection task, where a collection of labeled datasets, rather than a single dataset, was used to evaluate model performance [48], [49].”
InfluentialStrategies of Chinese State Media on Twitter
Fan, Y., Pan, J., & Sheng, J. (2023) · Political Communication · doi:10.1080/10584609.2023.2233911 · Cited by 24
“23 This method uses supervised machine learning models to estimate the probability an account is a bot by relying on account features and behaviors (Davis et al., 2016; Sayyadiharikandeh et al., 2020).”
MethodZhang, L., Gou, Z., Fang, Z., Sivertsen, G., & Huang, Y. (2023) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24830 · Cited by 23
“…of the interaction between academia and society in the tweeting of scientific publications, while more data beyond bios (e.g., tweeter engagement data) and advanced methods or tools are necessary to make fine-grained distinctions, for example, the Botometer tool (Sayyadiharikandeh et al., 2020).”
BackgroundSEGCN: a subgraph encoding based graph convolutional network model for social bot detection
Liu, F., Li, Z., Yang, C., Gong, D., Lu, H., & Liu, F. (2024) · Scientific Reports · doi:10.1038/s41598-024-54809-z · Cited by 25
“To address the challenge of generalization ability and design generic social bot detection models, some researchers 27 designed various classifiers for bots using different datasets and combined these classifiers into an ensemble; Botometer-v3 28 , a social bot detection system that incorporates…”
MethodRuíz-Núñez, C., Segado-Fernández, S., Jiménez-Gómez, B., Hidalgo, P. J. J., Magdalena, C. S. R., Pollo, M. D. C. Á., García, A. S., & Herrera‐Peco, I. (2022) · Vaccines · doi:10.3390/vaccines10081240 · Cited by 22
“Congruent with the sensitivity settings of previous studies [22,23], we set the threshold to 0.”
MethodDaume, S., Galaz, V., & Bjersér, P. (2023) · Social Media + Society · doi:10.1177/20563051231168370 · Cited by 32
“Botometer returns several specialized bot scores as numeric values in the range of 0 to 1, with higher values indicating a larger likelihood that an account is automated, that is, a Twitter bot (see also Supplementary Section A.1.4).”
MethodInfluentialA language framework for modeling social media account behavior
Nwala, A. C., Flammini, A., & Menczer, F. (2023) · EPJ Data Science · doi:10.1140/epjds/s13688-023-00410-9 · Cited by 31
“astroturf-20 [18] 505 0 botometer-feedback-19 [19] 123 364 botwiki-19 [26] 695 0”
BackgroundMethodResultInfluentialSelf-citationSocial Botomics: A Systematic Ensemble ML Approach for Explainable and Multi-Class Bot Detection
Dimitriadis, I., Georgiou, K., & Vakali, A. I. (2021) · Applied Sciences · doi:10.3390/app11219857 · Cited by 22
Unsupervised detection of coordinated fake-follower campaigns on social media
Zouzou, Y., & Varol, O. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00499-6 · Cited by 25
“Secondly, these models struggle to generalize well on unseen bot types in the training dataset[14], but novel approaches can address these limitations with novel machine learning systems[11].”
BackgroundMethodSelf-citationFriendship Preference: Scalable and Robust Category of Features for Social Bot Detection
Moghaddam, S. H., & Abbaspour, M. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3159007 · Cited by 23
“Additionally, Botometer and some other state-of-the-art approaches in social bot detection rely on features designed for tracking pieces of evidence of malicious activities, such as the number of URLs [13], [71], [72].”
BackgroundShould we agree to disagree about Twitter’s bot problem?
Varol, O. (2023) · Online Social Networks and Media · doi:10.1016/j.osnem.2023.100263 · Cited by 20
Self-citationDetection and impact estimation of social bots in the Chilean Twitter network
Mendoza, M., Providel, E., dos Santos, M. L. B., & Valenzuela, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-57227-3 · Cited by 22
“To deal with these difficulties 14 , created Botometer V.4, training an ensemble of specialized classifiers (ESC).”
BackgroundMethodInfluentialMultifaceted online coordinated behavior in the 2020 US presidential election
Tardelli, S., Nizzoli, L., Avvenuti, M., Cresci, S., & Tesconi, M. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00467-0 · Cited by 25
“Here we resort to Botometer v4 [48] to obtain an estimate of the degree of automation in Twitter accounts.”
MethodA Comparative Study of Bot Detection Techniques With an Application in Twitter Covid-19 Discourse
Antenore, M., Camacho, J. M., & Panizzi, E. (2022) · Social Science Computer Review · doi:10.1177/08944393211073733 · Cited by 21
“Eventually, (Sayyadiharikandeh et al., 2020) introduces Botometer v4.”
Evaluative altmetrics: is there evidence for its application to research evaluation?
Arroyo‐Machado, W., & Torres‐Salinas, D. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1188131 · Cited by 16
Wang, R., Zhang, Y., Suk, J., & Holland Levin, S. (2024) · Social Media + Society · doi:10.1177/20563051241277606 · Cited by 16
“The bot detection was executed using Botometer v4 (Sayyadiharikandeh et al., 2020), Note .”
Estimating the Bot Population on Twitter via Random Walk Based Sampling
Fukuda, M., Nakajima, K., & Shudo, K. (2022) · IEEE Access · doi:10.1109/access.2022.3149887 · Cited by 16
“Botometer [29], [30] enables accurate classification of specified Twitter users as automated bots or humans through APIs.”
BackgroundMethodMulBot: Unsupervised Bot Detection Based on Multivariate Time Series
Mannocci, L., Cresci, S., Monreale, A., Vakali, A. I., & Tesconi, M. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020363 · Cited by 21
“However, with the very similar [25], this is the only work where a multi-class classification is addressed.”
BackgroundMethodResultInfluentialLGB: Language Model and Graph Neural Network-Driven Social Bot Detection
Zhou, M. L., Zhang, D., Wang, Y., Geng, Y., Dong, Y., & Tang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3573748 · Cited by 29
“Index Terms —Graph neural network, large language model, multimodal, social bot detection, social networks.”
Detecting Social Bot on the Fly using Contrastive Learning
Zhou, M. L., Zhang, D., Wang, Y., Geng, Y., & Tang, J. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615468 · Cited by 15
“Some features, such as user properties [48], temporal behaviors [34], tweet features [15], etc., are extracted for bot detection.”
BackgroundMethodExposure to social bots amplifies perceptual biases and regulation propensity
Yan, H. Y., Yang, K.-C., Shanahan, J. E., & Menczer, F. (2023) · Scientific Reports · doi:10.1038/s41598-023-46630-x · Cited by 24
“Quantifying perceptual biases about social bots may also help improve the design of machine learning algorithms to detect them 33–35 .”
BackgroundSelf-citationZhang, M., Chen, Z., Liu, X., & Liu, J. (2023) · Asian Journal of Communication · doi:10.1080/01292986.2023.2261112 · Cited by 15
“Each type has distinct characteristics and behaviors that contribute to their classi fi cation (Sayyadiharikandeh et al., 2020).”
BackgroundNegative affect variability differs between anxiety and depression on social media
Rutter, L. A., ten Thij, M., Lorenzo‐Luaces, L., Valdez, D., & Bollen, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0272107 · Cited by 19
Akhtar, M. M., Masood, R., Ikram, M., & Kanhere, S. S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3644998 · Cited by 16
“[111] identified that bot datasets vary regarding class labels and characteristics.”
BackgroundMethodAre We Secure from Bots? Investigating Vulnerabilities of Botometer
Torusdağ, M. B., Kutlu, M., & Selçuk, A. A. (2020) · doi:10.1109/ubmk50275.2020.9219433 · Cited by 15
Accou2vec: A Social Bot Detection Model Based on Community Walk
Liu, F., Yang, C., Li, Z., Gong, D., & Liu, F. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2023.3347040 · Cited by 14
Measuring user engagement with low credibility media sources in a controversial online debate
Vilella, S., Semeraro, A., Paolotti, D., & Ruffo, G. (2022) · EPJ Data Science · doi:10.1140/epjds/s13688-022-00342-w · Cited by 15
“Fortunately, Botometer allows the option of using the overall raw score (hence called BotScore), i.e. a score ranging in the interval [0 , 1] that assesses the likelihood of an account to be controlled by a bot by means of a language-independent classifier: see Figure 2 for two illustrative examples of calculating the…”
MethodResultInfluentialNoorian, Z., Ghenai, A., Moradisani, H., Zarrinkalam, F., & Alavijeh, S. Z. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3359010 · Cited by 13
Early Detection and Prevention of Malicious User Behavior on Twitter Using Deep Learning Techniques
Sánchez, R., Zubiaga, A., & Almeida, A. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3419171 · Cited by 18
“Subsequently, more advanced models emerged, leveraging over a thousand features and employing various algorithms for classification [27].”
MethodLe, T., Tran-Thanh, L., & Lee, D. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512215 · Cited by 14
“the last decade [2, 10, 37, 39, 42, 52, 54] primarily focus on developing computer models to effectively detect bots on social networks [4, 8].”
BackgroundAssembling a multi-platform ensemble social bot detector with applications to US 2020 elections
Ng, L. H. X., & Carley, K. M. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01211-2 · Cited by 14
“This step creates the ensemble model, combining the different individual models together, and can therefore better generalize data features as a whole [23].”
InfluentialStatin Twitter: Human and Automated Bot Contributions, 2010 to 2022
Slavin, S. D., Berman, A. N., Beam, A. L., Návar, A. M., & Mittleman, M. A. (2024) · Journal of the American Heart Association · doi:10.1161/jaha.123.032678 · Cited by 12
Dissecting a social bot powered by generative AI: anatomy, new trends and challenges
López-Joya, S., Diaz-Garcia, J. A., Ruiz, M. D., & Martín-Bautista, M. J. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01410-5 · Cited by 13
“The studies Sayyadiharikandeh et al. (2020) and Yang et al. (2022) propose a public Twitter bot detection tool called Botometer, which has become one of the most important in the field.”
Political polarization on Twitter during the COVID-19 pandemic: a case study in Brazil
Brum, P. P. V., Teixeira, M. C., Vimieiro, R., Araújo, E. F. D. M., Meira, W., & Pappa, G. L. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00949-x · Cited by 15
“After some unsuccessful experiments in classifying bots using traditional methods due to the restrictions limiting how much of the total content published by a user could be collected (only related to COVID-19), we chose to use Botometer 3 (Shao et al. 2018; Xu and Sasahara 2021; Sayyadiharikandeh et al. 2020; Yang et…”
MethodLi, Q., Liu, Q., Liu, S., Di, X., Chen, S., & Zhang, H. (2023) · Communication and the Public · doi:10.1177/20570473231166157 · Cited by 12
“The Botometer, developed by the Observatory on Social Media (OSoMe) at Indiana University, is a social bot detection tool that is based on supervised learning (Sayyadiharikandeh et al., 2020).”
BackgroundMethodInfluentialMechanisms Driving Online Vaccine Debate During the COVID-19 Pandemic
Seckin, O. C., Atalay, A., Otenen, E., Duygu, U., & Varol, O. (2024) · Social Media + Society · doi:10.1177/20563051241229657 · Cited by 19
“We generated bot scores of the accounts in our dataset using Botometer, a supervised bot detection algorithm developed by the Observatory on Social Media at Indiana University (Sayyadiharikandeh et al., 2019).”
MethodSelf-citationA longitudinal study of the top 1% toxic Twitter profiles
Qayyum, H., Zhao, B. Z. H., Wood, I., Ikram, M., Kourtellis, N., & Kâafar, M. A. (2023) · ACM Web Science Conference (WebSci) · doi:10.1145/3578503.3583619 · Cited by 11
“Spammer, Fake Follower bots from Botometer API v4 [48].”
BackgroundMethodInfluentialSocial bots spoil activist sentiment without eroding engagement
Li, L., Vásárhelyi, O., & Vedres, B. (2024) · Scientific Reports · doi:10.1038/s41598-024-74032-0 · Cited by 16
“Figure 4 visualizes the predicted daily change in the average number of daily tweet counts (Panel A), and the sentiment of the tweets (Panel B) grouped by different bot probabilities (65,70,75) and subbot category (astroturf or other type of bots).”
Lingam, G., & Das, S. K. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113019 · Cited by 11
Evolution of Malicious Social Bot Detection: From Individual Profiling to Group Analysis and Beyond
Liu, Y., Xu, S., Li, Y., & Yu, S. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0017 · Cited by 9
Suárez-Lledó, V., Ortega‐Martin, E., Carretero-Bravo, J., Ramos-Fiol, B., & Álvarez‐Gálvez, J. (2024) · JMIR Infodemiology · doi:10.2196/50021 · Cited by 10
Twitter Campaigning Strategies in Australian Federal Elections 2013–2019
Bruns, A., Angus, D. J., & Graham, T. (2021) · Social Media + Society · doi:10.1177/20563051211063462 · Cited by 9
Wang, R., Walter, D., & Ophir, Y. (2023) · Social Science Computer Review · doi:10.1177/08944393231188472 · Cited by 9
“Spammer is spambots from several datasets that the developing team used to train the model (See details in Sayyadiharikandeh et al., 2020).”
BackgroundMethodInfluentialBot Detection in Twitter: An Overview
López-Joya, S., Diaz-Garcia, J. A., Ruiz, M. D., & Martín-Bautista, M. J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42935-4_11 · Cited by 10
The anatomy of conspiracy theorists: Unveiling traits using a comprehensive Twitter dataset
Gambini, M., Tardelli, S., & Tesconi, M. (2024) · Computer Communications · doi:10.1016/j.comcom.2024.01.027 · Cited by 9
“Finally, we verify the presence of automated accounts by employing Botometer v4 to compute the bot scores for both conspiracy and random users [64].”
MethodTask-agnostic representation learning of multimodal twitter data for downstream applications
Rivas, R., Paul, S., Hristidis, V., Papalexakis, E. E., & Roy–Chowdhury, A. K. (2022) · Journal Of Big Data · doi:10.1186/s40537-022-00570-x · Cited by 9
“Botometer’s most recent version, v4 [47], is an ensemble classifier that combines Bot-ometer v3 [48] with random forest classifiers that are each trained on a specific class of Twitter bot.”
BackgroundMethodResultInfluentialTracing the dynamics of misinformation and vaccine stance in Finland amid COVID-19
Ünlü, A., Truong, S., Sawhney, N., Sivelä, J., & Tammi, T. (2024) · Information Communication & Society · doi:10.1080/1369118x.2024.2331756 · Cited by 7
“Conversely, when dealing with Finnish speakers, we applied the universal score (Sayyadiharikandeh et al., 2020).”
Cresswell, L., Espín-Noboa, L., Murphy, M. S. Q., Ramlawi, S., Walker, M. C., Karsai, M., & Corsi, D. J. (2022) · JMIR Research Protocols · doi:10.2196/34421 · Cited by 8
“Additionally, we will remove bot accounts [21], and tweets without geotags will be further analyzed to infer a location from their authors’ profile [22,23] (Figure 1).”
MethodRooting for the Same Team: Shared social identities in a polarized contextPreprint
Ajzenman, N., Ferman, B., & Sant'Anna, P. C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4326148 · Cited by 9
Alothali, E., Hayawi, K., & Alashwal, H. (2023) · Applied Sciences · doi:10.3390/app13074443 · Cited by 7
First public dataset to study 2023 Turkish general election
Najafi, A., Muğurtay, N., Zouzou, Y., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-58006-w · Cited by 7
“In this work, we use the Botometer system and its light-weight version called BotometerLite to evaluate Twitter accounts 72–74 .”
MethodSelf-citationMental health effects of COVID-19 lockdowns: A Twitter-based analysis
Colella, S., Dufourt, F., Hildebrand, V. A., & Vivès, R. (2023) · Economics & Human Biology · doi:10.1016/j.ehb.2023.101307 · Cited by 7
Unveiling the Veiled Threat: The Impact of Bots on COVID-19 Health Communication
Ünlü, A., Truong, S., Sawhney, N., & Tammi, T. (2024) · Social Science Computer Review · doi:10.1177/08944393241275641 · Cited by 8
“If the user ’ s primary language is English, we relied the on the default score else we used “ universal score ” (Sayyadiharikandeh et al., 2020).”
Debating stereotypes: Online reactions to the vice-presidential debate of 2020
Felmlee, D., Julien, C., & Francisco, S. C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280828 · Cited by 4
“Finally, using a random sample of 6,510 accounts from our dataset, we found that less than 10% were identified as bot accounts, according to Botometer (version 4), a tool to detect social bots on Twitter [69].”
MethodOpinion Dynamics: Bots and the Spiral of Silence
Shukla, A., Sahasrabudhe, N., & Moharir, S. (2023) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets56262.2023.10041283 · Cited by 6
“…has motivated numerous studies examining various aspects including the number of bots on various social networks [6], the fraction of overall activity that includes bots [7], identifying bots on social networks [8], and identifying and classifying the various types of social media bots [9].”
BackgroundAnalyzing Social Media Activities at Bellingcat
Bär, D., Calderon, F., Lawlor, M., Licklederer, S., Totzauer, M., & Feuerriegel, S. (2023) · ACM Web Science Conference (WebSci) · doi:10.1145/3578503.3583604 · Cited by 6
“Botometer exhibits a high accuracy (0.96 AUC (Sayyadiharikandeh et al. 2020)) and has been frequently used in previous works (Geissler et al. 2022; Shao et al. 2018; Wojcik et al. 2018).”
MethodInfluentialÜnlü, A., & Hupli, A. (2023) · Nordic Studies on Alcohol and Drugs · doi:10.1177/14550725231171022 · Cited by 7
“The TweetBotOrNot (Kearney, 2018) package in the R Programme and Botometer (Sayyadiharikandeh et al., 2020) web page were used to distinguish between non-bots and social bots.”
MethodCredibility of vaccine-related content on Twitter during COVID-19 pandemic
Yousefinaghani, S., Dara, R. A., Wang, A., MacKay, M., Papadopoulos, A., & Sharif, S. (2023) · PLOS Global Public Health · doi:10.1371/journal.pgph.0001385 · Cited by 8
Robust Graph Clustering via Meta Weighting for Noisy Graphs
Jo, H., Bu, F., & Shin, K. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615038 · Cited by 6
“However, real-world graphs, including social networks [3, 15], auction networks [43], SMS networks [46], review networks [56], computer networks [51], are often contaminated by noise edges [12, 36] due to click errors [19], bots [7, 48], and spam [33], to name a few.”
BackgroundCola, G., Mazza, M., & Tesconi, M. (2023) · IEEE Access · doi:10.1109/access.2023.3282580 · Cited by 7
“More specifically, we used the ‘‘Botome-ter’’ classifier [25] to estimate the presence of bots among enabled accounts.”
MethodBotCF: Improving the Social Bot Detection Performance By Focusing on the Community Features
Liu, F., Li, Z., Yang, C., Gong, D., Liu, F., Ma, R., & Borş, A. G. (2025) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2025.3600474 · Cited by 6
Exploring the Distinctive Tweeting Patterns of Toxic Twitter Users
Qayyum, H., Ikram, M., Zhao, B. Z. H., Wood, I. D., Kourtellis, N., & Kâafar, M. A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386402 · Cited by 5
Arroyo‐Machado, W., Herrera‐Viedma, E., & Torres‐Salinas, D. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24998 · Cited by 8
“Unlike the full Botometer API V4 (Sayyadiharikandeh et al., 2020), BotometerLite does not offer access to more detailed metrics.”
Twitter Perspectives on Cochlear Implantation: Sentiment and Thematic Analysis
Feier, J. S., Nguyen, K. K., & Choi, J. S. (2023) · Otolaryngology · doi:10.1002/ohn.292 · Cited by 5
Ünlü, A., Truong, S., Sawhney, N., Sivelä, J., & Tammi, T. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00257-8 · Cited by 7
“The algorithm extracts over 1,200 features from each Twitter account, such as metadata, content information, and sentiment from the 200 most recent tweets [59].”
Security of Social Networks: Lessons Learned on Twitter Bot Analysis in the Literature
Alipour, S. A., Orji, R., & Zincir-Heywood, N. (2022) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3538969.3544450 · Cited by 5
“Sayyadiharikandeh et al. demonstrated that bots of different kinds exhibit different behaviors [21].”
Background
Metadata
- Authors
- Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F.
- Affiliations
- Indiana University Bloomington (US)
- Sabancı Üniversitesi (TR)
- Venue
- Proceedings of the 29th ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2020-10-19
- Pages
- 2725-2732
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W3035082952
- Cited by
- 159
- References
- 34
- Citations by year
- 2026: 9 · 2025: 26 · 2024: 36 · 2023: 45 · 2022: 30 · 2021: 10 · 2020: 3
- FWCI
- 16.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Craig Newmark Philanthropies
- Defense Advanced Research Projects Agency (W911NF-17-C0094)
- John S. and James L. Knight Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- social bot detection, generalization to unseen classes, Botometer, misinformation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
