The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; none of the 23 sources its claims cite is checked for support yet.
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Open accessAbstract
Malicious actors create inauthentic social media accounts controlled in part by algorithms, known as social bots, to disseminate misinformation and agitate online discussion. While researchers have developed sophisticated methods to detect abuse, novel bots with diverse behaviors evade detection. We show that different types of bots are characterized by different behavioral features. As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data. Moreover, tuning these models to recognize novel bots requires retraining with a significant amount of new annotations, which are expensive to obtain. To address these issues, we propose a new supervised learning method that trains classifiers specialized for each class of bots and combines their decisions through the maximum rule. The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets. Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining. We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an Ensemble of Specialized Classifiers (ESC) that trains separate Random Forest classifiers for different bot classes and combines them via the maximum rule to improve detection of novel social bots. ESC improves cross-domain bot recall and F1 over Botometer-v3 and is deployed in Botometer v4 with a cross-validation AUC of 0.99.
Key findings
- ESC improves cross-domain performance on unseen accounts: across six datasets, recall rises from 42% to 84%, precision from 52% to 64%, and F1 from 47% to 73%, an average F1 improvement of 56%. On combined-test, recall goes from 77% to 86%, precision stays at 70%, and F1 rises from 73% to 77% (5% improvement).
- In-domain performance is maintained: ESC achieves 0.96 AUC under 5-fold cross-validation, similar to Botometer’s 0.97 AUC, and the deployed Botometer v4 model yields a cross-validation AUC of 0.99.
- Different bot classes rely on different informative features: in cresci-17, traditional spambots are best separated by adjective-frequency standard deviation, social spambots by tweet sentiment arousal entropy, and fake followers by maximum friend-follower ratio.
- ESC adapts faster to new domains: when adding labeled examples from a held-out domain, ESC learns through a newly added specialized classifier while preserving old classifiers, requiring fewer labeled examples than retraining Botometer-v3 from scratch.
Methods
- The study uses labeled bot/human datasets from the Bot Repository plus datasets from Rauchfleisch and Kaiser, extracting over 1,200 features in six categories: metadata, retweet/mention networks, temporal features, content, and sentiment.
- ESC combines a human-detection classifier (RF0, corresponding to Botometer) with specialized Random Forest classifiers trained on balanced bot classes and human examples; bot scores are aggregated with the maximum rule, with Platt scaling in experiments.
- Evaluation includes 5-fold cross-validation for in-domain performance and hold-out cross-domain tests on cresci-stock, gilani-17, cresci-rtbust, kaiser-1, kaiser-2, kaiser-3, and combined-test, using F1 and AUC with bootstrap confidence intervals; adaptation is tested on varol-icwsm.
Limitations
- The paper states that cross-domain performance is highly sensitive to the datasets used in training and testing, making it easy to cherry-pick examples that make any method look bad; kaiser-1 and kaiser-3 are not completely hold-out because they include botwiki bots from ESC training, kaiser-3 shares accounts with kaiser-2 and kaiser-1, and kaiser-2 has only 27 bot accounts, making F1 sensitive to class imbalance.
- The paper notes that tweetbotornot2 uses a “verified” profile feature and tends to assign low bot scores to verified accounts, biasing comparisons in its favor on datasets where many human-labeled accounts are verified.
- The text makes evident that ground-truth labels can become invalid over time as accounts change, become inactive, or are compromised; the manual inspection of suspected mislabeled accounts covered only a random sample of 50 out of 332 highlighted accounts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
21 Own result · 18 Not checked yet · 16 No basis given · 14 Own interpretation · 3 Own method
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that different types of bots are characterized by different behavioral features.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data.”
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
“Both precision and recall tend to be higher for in-domain cases, demonstrating the limited generalization of supervised models across domains.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining.”
- Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
“It can also learn more efficiently from examples in new domains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using bot and human accounts in different datasets, we show that bot accounts exhibit greater heterogeneity in their discriminative behavioral features compared to human accounts.”
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
“Human accounts, on the other hand, exhibit consistent characteristics across datasets.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fig. 7 shows that ESC outperforms the Botometer baseline in most cases.”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“ESC yields F1 better than or comparable with tweetbotornot2 on all datasets except those from Rauchfleisch and Kaiser [35].”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“kaiser-2 includes only 27 bot accounts”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Manual inspection of a random sample of 50 of the accounts highlighted in Fig. 5 reveals that the human labels are no longer accurate for most of them - they have been inactive for years, are currently devoted to spam diffusion, and/or are controlled by third-party applications.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The scatter plot in Fig. 5 shows a good agreement between the bot scores obtained with ESC and the Botometer-v3 baseline (Spearman's ρ = 0 . 87).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The quote only partly states this claim. 3
- It may state two separate claims as one. 29
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets.”
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
“On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%. As a result, F1 increases from 47% to 73% (an improvement of 56%).”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that bot accounts tend to have lower cross-domain scores, resulting in lower recall.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the in-domain case (left), the linear classifier is sufficient to separate human and bot accounts. But in the cross-domain case (right), the same linear model fails, explaining the drop in recall: different features and distinct decision rules are needed to detect different classes of bots.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without loss of in-domain accuracy, the proposed approach effectively increases the recall of cross-domain bots.”
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
“On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%.”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the combined-test dataset, ESC outperforms tweetbotornot2 on both F1 and AUC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ESC is comparable to tweetbotornot2 on this dataset when using the AUC metric, which is not sensitive to class imbalance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AUC metric is not sensitive to class imbalance.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“kaiser-3 contains human accounts from kaiser-2 and bot accounts from kaiser-1 , so they are not independent.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“98% and 72% of accounts labeled human in kaiser-1 and kaiser-3 respectively are verified.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We focus on 332 accounts in this region that are labeled as human (75% are from caverlee ), which represent approximately 1% of the examples labeled as human in the training data.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the combined-test dataset, recall goes from 77% to 86%, precision stays at 70%, and F1 goes from 73% to 77% (an improvement of 5%).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“it uses a feature for 'verified' profiles and tends to assign a low bot score to them, even automated ones such as @twittersupport .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using 5-fold cross-validation, ESC achieves an Area Under the ROC Curve (AUC) of 0.96, similar to the Botometer classifier (0.97 AUC).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if we exclude botwiki from the training data, ESC still outperforms Botometer-v3 . For example, it yields an F1 score of 0.84 on kaiser-1 and 0.80 on kaiser-3 .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite impressive results when training and test sets are from the same domain - even using cross-validation - supervised models will miss new classes of bots, leading to low recall.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We reach similar conclusions using other hold-out datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Botometer-v3 baseline adapts more slowly.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We showed that this approach allows the system to learn about new domains in an efficient way, in the sense that fewer annotated examples are necessary.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our experiments show that cross-domain performance is highly sensitive to the datasets used in training and testing supervised models; it is easy to cherry-pick examples that make any given method look bad.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrated empirically that our proposed approach generalizes better than a monolithic classifier and is more robust to mislabeled training examples.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we are deploying ESC in the newest version of Botometer (v4)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fig. 8 shows how the F1 score on the test data improves as a function of the number of training examples from the hold-out domain.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Since ESC provides a good correspondence between scores and binary labels of human and bot accounts, no calibration is applied.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Bots with unseen characteristics are easily missed, as demonstrated by a drastic drop in recall when classifiers are faced with cross-domain accounts [18].”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The first baseline is the current version of Botometer, often considered the state-of-the-art method for bot detection [48].”
What the sources say
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In an arms race between abusive behaviors and countermeasures, novel social bots emerge everyday and evade purge from the platforms [15, 21].”
What the sources say
- Not checked yetcites [15] [15] Stefano Cresci, Marinella Petrocchi, Angelo Spognardi, and Stefano Tognazzi. 2019. Better safe than sorry: an adversarial approach to improve social bot detection. In Proceedings of the 10th ACM Conference on Web Science . 47-56. - source
- Not checked yetcites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“One common approach to address the lack of generalization is to retrain models with new labeled datasets [48].”
What the sources say
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: High-quality datasets of annotated social media accounts are expensive to acquire.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We obtain confidence intervals using bootstrapping with five samples of 80% of the accounts in the test set.”
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The kaiser-1 and kaiser-3 datasets include bots from botwiki , which are part of the ESC training data. Therefore these two cannot be considered completely hold-out datasets, but are included nonetheless because they were used to highlight weaknesses of Botometer-v3 in a recent independent report [35], so they provide us with an opportunity to demonstrate the performance of the latest Botometer”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The language agnostics version of Botometer (excluding English-based linguistic features) is used on the kaiser-2 dataset because of its German tweet content.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use 5-fold cross-validation for in-domain classification; for consistency we split training and test samples in cross-domain cases as well, reporting average precision and recall.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Botometer currently does not use the 'verified' status as a feature because it could lead to false negatives [45].”
What the sources say
- Not checked yetcites [45] [45] Onur Varol and Ismail Uluturk. 2020. Journalists on Twitter: self-branding, audiences, and involvement of bots. Journal of Computational Social Science 3, 1 (01 Apr 2020), 83-101. https://doi.org/10.1007/s42001-019-00056-6 - source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the old bot classes with more examples dominate and the classifier struggles to learn about the new domain.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.
In the paper
“Rauchfleisch and Kaiser [35] may overestimate the false positive rate by assuming that no politician account uses automation.”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we share these concerns and acknowledge that like any supervised learning model, Botometer makes mistakes.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Even a classifier trained on 1.5 million accounts and 22 classes of bots is incapable of identifying new classes of bots.”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Another source of bias is that Rauchfleisch and Kaiser [35] overlook accounts that are no longer available due to suspension, possibly leading to an underestimation of both precision and recall.”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the new version introduced in the paper is motivated by this issue and partly addresses it by improving cross-domain recall (false negatives).”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Echeverría et al. [18] proposed the leaveone-botnet-out methodology, to highlight how detection methods do not generalize well to unseen bot classes due to bias in the training data.”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ESC architecture quickly learns about new bots through the new classifier, which starts from scratch, while what was learned about the old ones is preserved in the existing classifiers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“since bots change and evolve continuously to evade detection, supervised learning algorithms need to adapt to new classes of bots [21, 44, 48].”
What the sources say
- Not checked yetcites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source
- Not checked yetcites [44] [44] Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018). -
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also include additional specialized classifiers when new annotated datasets become available.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Echeverria and Zhou [19] suggest that detecting different types of bots requires different types of features.”
What the sources say
- Not checked yetcites [19] [19] Juan Echeverria and Shi Zhou. 2017. Discovery of the Twitter Bursty Botnet. arXiv preprint arXiv:1709.06740 (2017). -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this way, the tool offers greater transparency about the decision process by allowing inspection of the outputs of different ensemble components and by providing interpretable class labels in output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proposed architecture is highly modular as each specialized classifier works independently, so one can substitute any part with different models as needed.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“They also report that some bot classes, like social spambots, are hard to distinguish from human accounts.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
High-quality datasets of annotated social media accounts are expensive to acquire.
“Unfortunately, high-quality datasets of annotated social media accounts are expensive to acquire.”
The authors state this as an assumption.
The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
“These datasets made an assumption that all accounts belonging to American and German politicians are human accounts.”
The paper relies on this without support.
Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.
“Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.”
The paper relies on this without support.
Periodic retraining to include newly annotated datasets helps systems adapt to unseen examples.
“Periodic retraining to include newly annotated datasets helps systems adapt to these unseen examples.”
The paper relies on this without support.
A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.
“A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.”
The paper relies on this without support.
The F1 score is sensitive to class imbalance.
“the F1 score is sensitive to this class imbalance”
The paper relies on this without support.
The AUC metric is not sensitive to class imbalance.
“the AUC metric, which is not sensitive to class imbalance”
The paper relies on this without support.
The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.
“The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.”
The paper relies on this without support.
No systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.
“Despite the existence of different types of bots, no systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An important limitation of these methods is that they rely on features crafted specifically to target previously observed malicious behaviors [18].”
From A language framework for modeling social media account behavior (2023) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These findings are consistent with prior analysis based on ad-hoc features [18].”
From A language framework for modeling social media account behavior (2023) · cites this paper as [18]
Citations
Cites
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
Self-citationPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · cited 2×
“The bot score returned by ESC is obtained by calibrating the score si∗ using Platt’s scaling [31].”
MethodWeaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018) · American Journal of Public Health · doi:10.2105/ajph.2018.304567 · Cited by 1,149 · cited 1×
“spreading misinformation [36, 37], amplifying popularity [9, 34, 45], or polarizing online discussion [39]. Bot activity has been reported in different domains, including politics [5, 20, 39], health [2, 3, 6, 17], and business [12, 13]. Due to the wide adoption of social media, every aspect of people’s life from news…”
BackgroundOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981 · cited 3×
“posed in this paper, extract various features from an account’s profile, social network, and content [16, 23, 29, 41, 43, 48, 49].”
BackgroundMethodSelf-citationBotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909
Self-citationRatkiewicz (2011) · doi:10.1145/1963192.1963301 · Cited by 541
Self-citationDeep neural networks for bot detection
Kudugunta, S., & Ferrara, E. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.019 · Cited by 506 · cited 1×
“Depending on the type of bots, similarity can be detected through tweet content [8, 25], temporal features in the timelines [7, 10], or retweeting behavior [28, 46].”
BackgroundBots increase exposure to negative and inflammatory content in online social systems
Stella, M., Ferrara, E., & De Domenico, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1803470115 · Cited by 492
The DARPA Twitter Bot Challenge
Subrahmanian, V. S., Azaria, A., Durst, S., Kagan, V., Galstyan, A., Lerman, K., Zhu, L., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Computer · doi:10.1109/mc.2016.183 · Cited by 480 · cited 2×
“number of links to legitimate accounts. Supervised machine learning approaches, such as the one proposed in this paper, extract various features from an account’s profile, social network, and content [16, 23, 29, 41, 43, 48, 49]. These methods rely on annotated datasets for learning the difference between bot and huma…”
BackgroundSelf-citationFame for sale: Efficient detection of fake Twitter followers
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.09.003 · Cited by 475
Research works printed but not linked: 9
- Jon-Patrick Allem , Emilio Ferrara , Sree Priyanka Uppu , Tess Boley Cruz, and Jennifer B Unger. 2017 . E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98. Jon-Patrick Allem, Emilio Ferrara, Sree Priyanka Uppu, Tess Boley Cruz, and Jennifer B Unger. 2017. E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98.
- Alessandro Bessi and Emilio Ferrara . 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 ( 2016 ). Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 (2016).
- Emilio Ferrara . 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 ( 2017 ). Emilio Ferrara. 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 (2017).
- Kyumin Lee , Brian David Eoff , and James Caverlee . 2011 . Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter . In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM). Kyumin Lee, Brian David Eoff, and James Caverlee. 2011. Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter. In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM).
- Fabian Pedregosa , Gaël Varoquaux , Alexandre Gramfort , Vincent Michel , Bertrand Thirion , Olivier Grisel , Mathieu Blondel , Peter Prettenhofer , Ron Weiss , Vincent Dubourg , 2011 . Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830.
- Chengcheng Shao , Giovanni Luca Ciampaglia , Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018 . The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787. Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018. The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787.
- Galen Stoking and Nami Sumida . 2018. Social Media Bots Draw Public?s Attention and Concern . Pew Research Center ( 15 Oct 2018 ). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/ Galen Stoking and Nami Sumida. 2018. Social Media Bots Draw Public?s Attention and Concern. Pew Research Center (15 Oct 2018). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/
- Onur Varol , Clayton A Davis , Filippo Menczer , and Alessandro Flammini . 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics ( 2018 ), 311--334. Onur Varol, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics (2018), 311--334.
- Onur Varol and Ismail Uluturk . 2018. Deception strategies and threats for online discussions. First Monday 23, 5 ( 2018 ). Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018).
Cited by
Sorted by citations
Items 101–160 of 160
Elmas, T. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587650 · Cited by 5
“Past researched focus on detecting bots using their profle features [12, 18, 20, 31], content features [30], graph features [1] and temporal activity [3, 21], or a combination of them [6, 17, 25].”
BackgroundCalabrese, C. R., Zhang, J., & Yu, X. (2022) · Health & New Media Research · doi:10.22720/hnmr.2022.6.1.065 · Cited by 5
From Online Behaviours to Images: A Novel Approach to Social Bot Detection
Paolo, E. D., Petrocchi, M., & Spognardi, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35995-8_42 · Cited by 6
Rutter, L. A., Edinger, A., Lorenzo‐Luaces, L., ten Thij, M., Valdez, D., & Bollen, J. (2025) · Cognitive Therapy and Research · doi:10.1007/s10608-025-10580-7 · Cited by 6
“Finally, we ran a bot-analysis using Botometer (Sayyadiharikandeh et al., 2020 ) on each of these profiles and removed the accounts that were deemed very likely to be automated.”
Meng, X., Wang, X., & Peng, T. (2026) · Communication Methods and Measures · doi:10.1080/19312458.2026.2613829 · Cited by 7
“Next, supervised learning methods require labeled data to train classifiers for bot detection (Sayyadiharikandeh et al., 2020; Varol et al., 2017), whereas unsupervised learning uses clustering techniques without labeled training data.”
Zeng, K., Zhao, L., & Wang, X. (2025) · Sensors · doi:10.3390/s25134179 · Cited by 4
“ESC [21] extracted over 100 feature indicators by combining three groups of features: user attributes, Twitter behavioral data, and textual features.”
TG-OUT: temporal outlier patterns detection in Twitter attribute induced graphs
Dimitriadis, I., Poiitis, M., Faloutsos, C., & Vakali, A. I. (2022) · World Wide Web · doi:10.1007/s11280-021-00986-0 · Cited by 4
“Despite the fact that Botometer has recently released a new and improved version [44], the use of the first version was mandatory in order to eliminate the time gap between the data collection and data annotation.”
MethodLgt: long-range graph transformer for early rumor detection
Xia, J., Li, Y., & Yu, K. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01263-4 · Cited by 4
“For comparison, we consider the following baseline models: • Botometer-v4 (Sayyadiharikandeh et al. 2020) is a supervised machine learning tool for detecting whether a social media account is a bot.”
An Analysis of Social Bot Activity on X in Modern Japan
Ippa, S., Okubo, T., & Hashimoto, M. (2024) · IEEE Access · doi:10.1109/access.2024.3454536 · Cited by 3
“After the release of v1 in May 2014, v2 was released in May 2016, v3 in May 2018, and the latest version, v4, was released in September 2020, receiving more than 500,000 socialbot decision requests per day[12].”
Xu, W., Sasahara, K., Chu, J., Wang, B., Fan, W., & Hu, Z. (2025) · EPJ Data Science · doi:10.1140/epjds/s13688-025-00528-y · Cited by 4
Are patents linked on Twitter? A case study of Google patents
Orduña‐Malea, E., & Font-Julián, C. I. (2022) · Scientometrics · doi:10.1007/s11192-022-04519-y · Cited by 3
“Botometer is an application that scores Twitter users from 0 to 5 considering six variables, 5 and using lan-guage-independence features (Sayyadiharikandeh, 2020).”
MethodEnhancing the Detection of Social bots on Twitter using Ensemble machine Learning Technique
Mohanty, S., Dwivedy, S. P., Acharya, A. A., Mohapatra, S., Sahoo, S. S., Samal, S., & Samal, S. (2022) · International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC) · doi:10.1109/assic55218.2022.10088372 · Cited by 5
“Sayyadiharikandeh et al. [9] showed in their work that loss in performance was due to heterogeneous bot behavior.”
BackgroundHùng, N. V., Loi, T., Hương, N. T. M., Hang, T. T. M., & Hương, T. T. (2023) · Informatics and Automation · doi:10.15622/ia.22.4.4 · Cited by 4
Li, Y., Sun, X., Yang, R., Sun, X., Chen, S., Wang, S., Bhuiyan, M. Z. A., Zomaya, A. Y., & Xu, J. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02392-0 · Cited by 4
“Other works [11, 61, 62] consider substantial meta users and features in the elaborate machine learning models particularly for the consideration of scalability and generalization; however, they usually come with non-negligible time and computational cost.”
Demystifying Misconceptions in Social Bots Research
Cresci, S., Yang, K.-C., Spognardi, A., Pietro, R. D., Menczer, F., & Petrocchi, M. (2025) · Social Science Computer Review · doi:10.1177/08944393251376707 · Cited by 30
“However, the analysis only evaluates supervised detectors, which have known generalization issues (Echeverría et al., 2018; Yang et al., 2020; Sayyadiharikandeh et al., 2020; Rauchfleisch and Kaiser, 2020; Dimitriadis et al., 2021).”
InfluentialSelf-citationPolychronis, C., & Kogan, M. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610033 · Cited by 3
“We believe that this work could move our understanding of SSIO activity from general awareness of the division of labor that they engage in towards specific insight into the common social roles of these operations and how they contribute to operation longevity, reach, and ability to avoid detection.”
Backgroundrole of social bots in the Brazilian environmental debate:
Salles, D., de Medeiros, P. M., Martins, B., Regattieri, L. L., & Santini, R. M. (2024) · The International Review of Information Ethics · doi:10.29173/irie510 · Cited by 2
Username Squatting on Online Social Networks: A Study on X
Λεπίπας, Α., Borovykh, A., & Demetriou, S. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3637637 · Cited by 2
“We first run Botometer [78] for every”
Not our kind of crowd! How partisan bias distorts perceptions of political bots on Twitter (now X)
Lueders, A., Reiß, S., Dinkelberg, A., MacCarron, P., & Quayle, M. (2024) · British Journal of Social Psychology · doi:10.1111/bjso.12794 · Cited by 3
Machine Learning Based Detection System for Identifying Deepfake Images and Social Bots
Liyanage, C., Dematgoda, H., Harindi, D., Perera, N., Fernando, H., & Rathnayake, S. C. (2024) · International Conference on Advancements in Computing (ICAC) · doi:10.1109/icac64487.2024.10850968 · Cited by 2
Meng, X., Wang, X., & Zhao, X. (2025) · Internet Research · doi:10.1108/intr-07-2024-1123 · Cited by 2
Taxonomy and Bot Threats in Social Networks
Vitkova, L., Kolomeets, M., & Chechulin, A. A. (2022) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon54946.2022.9896268 · Cited by 3
“In [20], a supervised learning method is presented.”
MethodPublic interest in science or bots? Selective amplification of scientific articles on Twitter
Rahman, A., Mohammadi, E., & Alhoori, H. (2024) · Aslib Journal of Information Management · doi:10.1108/ajim-01-2024-0050 · Cited by 1
“Bots frequently change their approaches to social media, making it very difficult to detect their behavior (Sayyadiharikandeh et al., 2020).”
BackgroundA Review on Ensemble Techniques and Its Application on Social Bot Detection
Sharma, J., & Borah, S. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-6465-5_12 · Cited by 1
An Analysis of Relationship Between Social Bots and Echo Chambers on X in Modern Japan
Ippa, S., Okubo, T., & Hashimoto, M. (2024) · International Symposium on Computing and Networking · doi:10.1109/candar64496.2024.00014 · Cited by 2
“It receives over 500,000 social bot determination requests per day [25].”
Unmasking social bots: how confident are we?
Giroux, J., Ariyarathne, G., Nwala, A. C., & Fanelli, C. (2025) · EPJ Data Science · doi:10.1140/epjds/s13688-025-00536-y · Cited by 6
“…botometer-feedback-19 [57] 139 379 cresci-17 [11] 7049 2760 celebrity-19 [57] 0 20549 gilani-17 [21] 914 1576 verified-19 [58] 0 1891 astroturf-20 [46] 502 0 vendor-purchased-19 [57] 1069 17 varol-17 [53] 732 1496 political-bots-19 [57] 62 0 cresci-rtbust-19 [35] 352 340 botwiki-19 [58] 691 0…”
InfluentialAdaptive Social Bot Detection through Bridging the Feature Bias Between Source and Target Users
Sun, H., Peng, H., Cao, Y., Dai, Q., & Bai, X. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733268 · Cited by 2
“Through feature engineering [29, 40], natural language processing (NLP) [14, 37], and graph neural networks (GNNs) [10, 23], researchers can extract users’ feature representations, and utilize them to determine whether a user is a bot.”
A Community-Aware Spatio-Temporal Hypergraph Contrastive Learning Method for Social Bot Detection
Liu, F., Li, Z., & Ma, R. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3637684 · Cited by 1
“Early SBD methods primarily relied on manual feature engineering, identifying bots by mining account-level attributes such as profile metadata and posting behaviors [12], [13].”
Domain-based user embedding for competing events on social media
Xu, W., & Sasahara, K. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-025-00439-y · Cited by 3
“Bot score is a probability to evaluate how likely a Twitter user is to be a social bot, which is calculated by Botometer v4 [53].”
Malicious bot detection in Twitter/X social media platform with interpretable machine intelligence
Gupta, S., Sinha, A., & Saxena, B. (2026) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-026-05039-w · Cited by 1
Towards Automatic Annotation and Detection of Fake News
Akhtar, M. M., Karunanayake, I., Sharma, B., Masood, R., Ikram, M., & Kanhere, S. S. (2023) · Conference on Local Computer Networks · doi:10.1109/lcn58197.2023.10223359 · Cited by 2
Semantic Information Mining and Fusion Method for Bot Detection
Liang, L., Wang, X., & Liu, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44201-8_23 · Cited by 1
Investigating responses to US drone strikes in Yemen using Twitter data
Weiss, E., Ross, V., & Lyford, A. (2024) · Media War & Conflict · doi:10.1177/17506352231219694 · Cited by 1
“To obtain a body of relevant tweets, we queried the Academic Twitter API.”
InfluentialOn mission Twitter Profiles: A Study of Selective Toxic Behavior
Qayyum, H., Ikram, M., Zhao, B. Z. H., Wood, I. D., Kourtellis, N., & Kâafar, M. A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386248 · Cited by 1
“Tools like Botometer [41] can help identify less sophisticated influence attempts or promotional operations relying on a limited number of bot-operated accounts.”
BackgroundMethodPredicting the impact of online news articles – is information necessary?
Preiss, J. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11621-5 · Cited by 1
“This is supported by a brief exploration of the dataset in this work: the probability of being a bot account, measured by Botometer [47], was evaluated for 2500 randomly selected Twitter accounts from the list of tweeters / re-tweeters / followers appearing in the dataset.”
MethodUgwumba, N. K. (2025) · Research Square · doi:10.21203/rs.3.rs-8085397/v1 · Cited by 1
Suárez-Lledó, V., & Álvarez‐Gálvez, J. (2021) · doi:10.2196/preprints.36085
A Geolocated Dataset of COVID-19 Pandemic: Tweets with Location Information
Mohammed, I., & Abbas, A. S. (2021) · International Conference on Engineering Technology and its Applications (IICETA) · doi:10.1109/iiceta51758.2021.9717676
Achieving Participatory Smart Cities by Making Social Networks Safer
Sánchez, R., Zubiaga, A., & Almeida, A. (2022) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech55088.2022.9854284
“Preliminary works that employ simple features and sets of rules to detect malicious users [25] became obsolete due to the rapid sophistication experienced by malicious users and gave way to models that use more than a thousand features and several algorithms to perform the classification [26].”
MethodShort- and Long-Term Impact of Psychological Distance on Human Responses to a Terror AttackPreprint
Kušen, E., & Strembeck, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4181167
VIKAS: A Multimodal Framework to Aid in Effective Disaster Management
Prabhu, G. M., Gupta, T., Srujan, M. V., Soumya, A., Palorkar, A., & Chowdhury, A. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-2264-2_22
How to Find Social Robots exactly?
Ding, J., & Chen, Z. (2023) · doi:10.1145/3584871.3584873
“Botwiki [27] Manual labeling 689 0 Astroturf Manual labeling 505 0 Midterm-2018 Manual labeling 0 7459 Kaiser Manual labeling 875 499 -”
What is the best time to tweet a journal article? Quasi-randomized controlled trial
Fontenelle, L. F., & Sarti, T. D. (2024) · AtoZ novas práticas em informação e conhecimento · doi:10.5380/atoz.v13i0.89296
“Throughout the study period we continued to use Botometer (Sayyadiharikandeh, Varol, Yang, Flammini, and Menczer, 2020; Yang et al., 2019) to block accounts suspected of being malicious bots, and by December 2017 the account had 1,185 followers 4 .”
MethodImpact of Bots and Suspicious Accounts on Online Brand Personality and Brand Sentiment
Anand, V., & Kumar, D. V. S. (2024) · ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS) · doi:10.1109/icetsis61505.2024.10459671
“Using the OSoMe Botometer API developed by Indiana University [21], approximately 1160 tweets were identified as those generated by bots and suspicious accounts.”
MethodSocial Media as an Agent of Influence: Twitter Bots in Russia - Ukraine War
Taban, M. H., & Gür, İ. (2024) · Güvenlik Stratejileri Dergisi · doi:10.17752/guvenlikstrtj.1396705
Akhila, V. H., & Chacko, A. M. (2024) · ACM Symposium on Applied Computing · doi:10.1145/3605098.3636038
“Among the different types of bots Chatbots, Shopbots, Likebots etc [27]., Socialbot or social media bot is an automated software application designed to interact with social media users[10].”
BackgroundThe Relationship Between Emotional and Other Factors in Information Diffusion
Ippa, S., Okubo, T., & Hashimoto, M. (2025) · IEEE Access · doi:10.1109/access.2025.3535547
“It receives over 500,000 social bot determination requests per day [26].”
Endorsement Networks in the 2022 Brazilian Presidential Elections: a Case Study on Twitter Data
Dinkelberg, A., Santana, B. S., & Bazzan, A. L. C. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.3747
Nisotaki, E. M., & Mourlas, C. (2025) · Advances in wireless technologies and telecommunication book series · doi:10.4018/979-8-3693-9999-6.ch012
Gonçalves, J. L., Torres, L. B. D. M., Freire, P. M. S., & Goldschmidt, R. R. (2025) · Brazilian Symposium on Information Systems · doi:10.5753/sbsi.2025.246620
CAGCL: A Community-Aware Graph Contrastive Learning Model for Social Bot Detection
Wei, K., Teng, M., Du, H., et al. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761411
“…stages of the development of social bot detection, research primarily focused on extracting account attributes [25, 38], behavioral patterns [26, 27, 37], and semantic content features [17, 18], which were subsequently utilized in traditional supervised learning methods for bot…”
Paradowski, R. (2025) · Przegląd Politologiczny · doi:10.14746/pp.2025.30.3.11
“Among the 5,329 unique users, a sample of 14 high-volume accounts (responsible for over 20% of tweets) was examined using Botometer (Sayyadiharikandeh et al., 2020), confirming they were not generated by bots.”
Sinha, A., Saxena, B., & Sabherwa, S. (2025) · doi:10.1109/c2i666499.2025.11366994
Liu, Q., Zhang, W.-S., & Lu, J.-Z. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3682474
“Prior studies [44] have confirmed its high reliability and widespread adoption [43].”
Behavior Change as a Signal for Identifying Social Media Manipulation
Ariyarathne, I., Ariyarathne, G., Flammini, A., Menczer, F., & Nwala, A. C. (2026) · ACM Web Science Conference (WebSci) · doi:10.1145/3795766.3799748 · Cited by 1
“# Bots # Humans astroturf-20 [32] 505 0 botometer-feedback-19 [41] 123 364 botwiki-19 [42] 695 0 celebrity-19 [41] 0 20,911 cresci-17 [11] 5,812 2,744 cresci-rtbust-19 [25] 352 340 cresci-stock-18 [12] 6,926 6,155 gilani-17 [21] 1,058 1,381 midterm-18[42] 0 7,409 political-bots-19 [41] 62 0…”
Self-citationBotEvo: LLM-Driven social bot detection via behavioral evolution modeling and cross-modal fusion
Zhang, Y., Qiao, K., Shi, S., Liu, J., Liu, Z., Chen, J., Li, L., & Yan, B. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00814-3
Explainable Analysis of Fake Instagram Account Detection Using Graph and Behavioral Features
Miranda-Caramé, I., Rodríguez-Dueñas, A., Jiménez-Navarro, M. J., et al. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-29254-4_26
Ouyang, Q., Zhang, B., Li, R., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2856-0_8
Authenticity Detection by Fine-Tuning Transformer-Based Models
Abdelhady, A. S. (2026) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet65726.2026.11633234
“One paper [20] designed the Ensemble of Specialized Classifiers (ESC) way to help with detecting social bots.”
Wahlberg, L. (2026) · Scandinavian Political Studies · doi:10.1111/1467-9477.70060
Metadata
- Authors
- Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F.
- Affiliations
- Indiana University Bloomington (US)
- Sabancı Üniversitesi (TR)
- Venue
- Proceedings of the 29th ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2020-10-19
- Pages
- 2725-2732
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W3035082952
- Cited by
- 159
- References
- 34
- Citations by year
- 2026: 9 · 2025: 26 · 2024: 36 · 2023: 45 · 2022: 30 · 2021: 10 · 2020: 3
- FWCI
- 16.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Craig Newmark Philanthropies
- Defense Advanced Research Projects Agency (W911NF-17-C0094)
- John S. and James L. Knight Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- social bot detection, generalization to unseen classes, Botometer, misinformation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
