The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; none of the 23 sources its claims cite is checked for support yet.
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Open accessAbstract
Malicious actors create inauthentic social media accounts controlled in part by algorithms, known as social bots, to disseminate misinformation and agitate online discussion. While researchers have developed sophisticated methods to detect abuse, novel bots with diverse behaviors evade detection. We show that different types of bots are characterized by different behavioral features. As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data. Moreover, tuning these models to recognize novel bots requires retraining with a significant amount of new annotations, which are expensive to obtain. To address these issues, we propose a new supervised learning method that trains classifiers specialized for each class of bots and combines their decisions through the maximum rule. The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets. Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining. We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an Ensemble of Specialized Classifiers (ESC) that trains separate Random Forest classifiers for different bot classes and combines them via the maximum rule to improve detection of novel social bots. ESC improves cross-domain bot recall and F1 over Botometer-v3 and is deployed in Botometer v4 with a cross-validation AUC of 0.99.
Key findings
- ESC improves cross-domain performance on unseen accounts: across six datasets, recall rises from 42% to 84%, precision from 52% to 64%, and F1 from 47% to 73%, an average F1 improvement of 56%. On combined-test, recall goes from 77% to 86%, precision stays at 70%, and F1 rises from 73% to 77% (5% improvement).
- In-domain performance is maintained: ESC achieves 0.96 AUC under 5-fold cross-validation, similar to Botometer’s 0.97 AUC, and the deployed Botometer v4 model yields a cross-validation AUC of 0.99.
- Different bot classes rely on different informative features: in cresci-17, traditional spambots are best separated by adjective-frequency standard deviation, social spambots by tweet sentiment arousal entropy, and fake followers by maximum friend-follower ratio.
- ESC adapts faster to new domains: when adding labeled examples from a held-out domain, ESC learns through a newly added specialized classifier while preserving old classifiers, requiring fewer labeled examples than retraining Botometer-v3 from scratch.
Methods
- The study uses labeled bot/human datasets from the Bot Repository plus datasets from Rauchfleisch and Kaiser, extracting over 1,200 features in six categories: metadata, retweet/mention networks, temporal features, content, and sentiment.
- ESC combines a human-detection classifier (RF0, corresponding to Botometer) with specialized Random Forest classifiers trained on balanced bot classes and human examples; bot scores are aggregated with the maximum rule, with Platt scaling in experiments.
- Evaluation includes 5-fold cross-validation for in-domain performance and hold-out cross-domain tests on cresci-stock, gilani-17, cresci-rtbust, kaiser-1, kaiser-2, kaiser-3, and combined-test, using F1 and AUC with bootstrap confidence intervals; adaptation is tested on varol-icwsm.
Limitations
- The paper states that cross-domain performance is highly sensitive to the datasets used in training and testing, making it easy to cherry-pick examples that make any method look bad; kaiser-1 and kaiser-3 are not completely hold-out because they include botwiki bots from ESC training, kaiser-3 shares accounts with kaiser-2 and kaiser-1, and kaiser-2 has only 27 bot accounts, making F1 sensitive to class imbalance.
- The paper notes that tweetbotornot2 uses a “verified” profile feature and tends to assign low bot scores to verified accounts, biasing comparisons in its favor on datasets where many human-labeled accounts are verified.
- The text makes evident that ground-truth labels can become invalid over time as accounts change, become inactive, or are compromised; the manual inspection of suspected mislabeled accounts covered only a random sample of 50 out of 332 highlighted accounts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
21 Own result · 18 Not checked yet · 16 No basis given · 14 Own interpretation · 3 Own method
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We show that different types of bots are characterized by different behavioral features.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data.”
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
“Both precision and recall tend to be higher for in-domain cases, demonstrating the limited generalization of supervised models across domains.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining.”
- Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
“It can also learn more efficiently from examples in new domains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using bot and human accounts in different datasets, we show that bot accounts exhibit greater heterogeneity in their discriminative behavioral features compared to human accounts.”
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
“Human accounts, on the other hand, exhibit consistent characteristics across datasets.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fig. 7 shows that ESC outperforms the Botometer baseline in most cases.”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“ESC yields F1 better than or comparable with tweetbotornot2 on all datasets except those from Rauchfleisch and Kaiser [35].”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“kaiser-2 includes only 27 bot accounts”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Manual inspection of a random sample of 50 of the accounts highlighted in Fig. 5 reveals that the human labels are no longer accurate for most of them - they have been inactive for years, are currently devoted to spam diffusion, and/or are controlled by third-party applications.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The scatter plot in Fig. 5 shows a good agreement between the bot scores obtained with ESC and the Botometer-v3 baseline (Spearman's ρ = 0 . 87).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we see in § 4, the maximum rule has the effect of shifting scores of likely bots toward one and the scores of likely humans toward zero.”
- Also stated in: § 4.2 Cross-domain performance · p. 4 · plain
“ESC produces significantly higher scores than Botometer-v3 on bot accounts. This is a result of the maximum rule and leads to higher cross-domain recall, or better generalization.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.2 Cross-domain performance · p. 4 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the size of the hold-out dataset is small compared to the training size (over 67,000 examples in Botometerv3 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We performed load tests submitting queries from 20 simultaneous jobs running on a supercomputer, yielding an average response time below 100ms per query.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The final model yields an AUC of 0.99.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Supervised bot detection methods achieve high accuracy based on in-domain cross-validation [4].”
What the sources say
- Not checked yetcites [4] [4] Eiman Alothali, Nazar Zaki, Elfadil A Mohamed, and Hany Alashwal. 2018. Detecting social bots on Twitter: a literature review. In International Conference on Innovations in Information Technology . IEEE, 175-180. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The one exception is the high precision when testing on the cresci-17 domain, irrespective of the training datasets. This is due to the fact that cresci-17 includes spambots, which are represented in all datasets.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“These features are shown to be effective in identifying social bots and are described in detail in the literature [42, 43, 48].”
What the sources say
- Not checked yetcites [42] [42] Onur Varol, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics (2018), 311-334. -
- Not checked yetcites [43] [43] Onur Varol, Emilio Ferrara, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2017. Online human-bot interactions: Detection, estimation, and characterization. In Proc. Intl. AAAI Conf. on Web and Social Media (ICWSM) . - source
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This biases the results in favor of tweetbotornot2 .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that ESC can identify impurities in the training data.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The performance decay of current detection systems in the wild was reported by Cresci et al. [11], who showed that Twitter, human annotators, and state-of-the-art bot detection systems failed at discriminating between some new social spambots and genuine accounts.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate in this paper that the performance deterioration observed for out-of-domain accounts is due to heterogeneous bot behaviors that require different informative subsets of features.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This should provide ample scalability, considering the popularity of the tool among researchers and practitioners - at the time of this writing, it fields over half a million queries daily (Fig. 10).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that fewer labeled examples are needed to train a new specialized classifier when novel types of bots are observed in the wild.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proposed ESC method alleviates this problem because we can add a new classifier RF n + 1 to the ensemble to be trained with the new data, without retraining the existing classifiers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.
- The quote only partly states this claim. 3
- It may state two separate claims as one. 29
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets.”
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
“On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%. As a result, F1 increases from 47% to 73% (an improvement of 56%).”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that bot accounts tend to have lower cross-domain scores, resulting in lower recall.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the in-domain case (left), the linear classifier is sufficient to separate human and bot accounts. But in the cross-domain case (right), the same linear model fails, explaining the drop in recall: different features and distinct decision rules are needed to detect different classes of bots.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Without loss of in-domain accuracy, the proposed approach effectively increases the recall of cross-domain bots.”
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
“On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%.”
plaindirectn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the combined-test dataset, ESC outperforms tweetbotornot2 on both F1 and AUC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ESC is comparable to tweetbotornot2 on this dataset when using the AUC metric, which is not sensitive to class imbalance.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AUC metric is not sensitive to class imbalance.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“kaiser-3 contains human accounts from kaiser-2 and bot accounts from kaiser-1 , so they are not independent.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“98% and 72% of accounts labeled human in kaiser-1 and kaiser-3 respectively are verified.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We focus on 332 accounts in this region that are labeled as human (75% are from caverlee ), which represent approximately 1% of the examples labeled as human in the training data.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the combined-test dataset, recall goes from 77% to 86%, precision stays at 70%, and F1 goes from 73% to 77% (an improvement of 5%).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“it uses a feature for 'verified' profiles and tends to assign a low bot score to them, even automated ones such as @twittersupport .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using 5-fold cross-validation, ESC achieves an Area Under the ROC Curve (AUC) of 0.96, similar to the Botometer classifier (0.97 AUC).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if we exclude botwiki from the training data, ESC still outperforms Botometer-v3 . For example, it yields an F1 score of 0.84 on kaiser-1 and 0.80 on kaiser-3 .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Despite impressive results when training and test sets are from the same domain - even using cross-validation - supervised models will miss new classes of bots, leading to low recall.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We reach similar conclusions using other hold-out datasets.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Botometer-v3 baseline adapts more slowly.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We showed that this approach allows the system to learn about new domains in an efficient way, in the sense that fewer annotated examples are necessary.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our experiments show that cross-domain performance is highly sensitive to the datasets used in training and testing supervised models; it is easy to cherry-pick examples that make any given method look bad.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrated empirically that our proposed approach generalizes better than a monolithic classifier and is more robust to mislabeled training examples.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we are deploying ESC in the newest version of Botometer (v4)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Fig. 8 shows how the F1 score on the test data improves as a function of the number of training examples from the hold-out domain.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Since ESC provides a good correspondence between scores and binary labels of human and bot accounts, no calibration is applied.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Bots with unseen characteristics are easily missed, as demonstrated by a drastic drop in recall when classifiers are faced with cross-domain accounts [18].”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The first baseline is the current version of Botometer, often considered the state-of-the-art method for bot detection [48].”
What the sources say
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In an arms race between abusive behaviors and countermeasures, novel social bots emerge everyday and evade purge from the platforms [15, 21].”
What the sources say
- Not checked yetcites [15] [15] Stefano Cresci, Marinella Petrocchi, Angelo Spognardi, and Stefano Tognazzi. 2019. Better safe than sorry: an adversarial approach to improve social bot detection. In Proceedings of the 10th ACM Conference on Web Science . 47-56. - source
- Not checked yetcites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“One common approach to address the lack of generalization is to retrain models with new labeled datasets [48].”
What the sources say
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: High-quality datasets of annotated social media accounts are expensive to acquire.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We obtain confidence intervals using bootstrapping with five samples of 80% of the accounts in the test set.”
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The kaiser-1 and kaiser-3 datasets include bots from botwiki , which are part of the ESC training data. Therefore these two cannot be considered completely hold-out datasets, but are included nonetheless because they were used to highlight weaknesses of Botometer-v3 in a recent independent report [35], so they provide us with an opportunity to demonstrate the performance of the latest Botometer”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The language agnostics version of Botometer (excluding English-based linguistic features) is used on the kaiser-2 dataset because of its German tweet content.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We use 5-fold cross-validation for in-domain classification; for consistency we split training and test samples in cross-domain cases as well, reporting average precision and recall.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Botometer currently does not use the 'verified' status as a feature because it could lead to false negatives [45].”
What the sources say
- Not checked yetcites [45] [45] Onur Varol and Ismail Uluturk. 2020. Journalists on Twitter: self-branding, audiences, and involvement of bots. Journal of Computational Social Science 3, 1 (01 Apr 2020), 83-101. https://doi.org/10.1007/s42001-019-00056-6 - source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the old bot classes with more examples dominate and the classifier struggles to learn about the new domain.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.
In the paper
“Rauchfleisch and Kaiser [35] may overestimate the false positive rate by assuming that no politician account uses automation.”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we share these concerns and acknowledge that like any supervised learning model, Botometer makes mistakes.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Even a classifier trained on 1.5 million accounts and 22 classes of bots is incapable of identifying new classes of bots.”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Another source of bias is that Rauchfleisch and Kaiser [35] overlook accounts that are no longer available due to suspension, possibly leading to an underestimation of both precision and recall.”
What the sources say
- Not checked yetcites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the new version introduced in the paper is motivated by this issue and partly addresses it by improving cross-domain recall (false negatives).”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Echeverría et al. [18] proposed the leaveone-botnet-out methodology, to highlight how detection methods do not generalize well to unseen bot classes due to bias in the training data.”
What the sources say
- Not checked yetcites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ESC architecture quickly learns about new bots through the new classifier, which starts from scratch, while what was learned about the old ones is preserved in the existing classifiers.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“since bots change and evolve continuously to evade detection, supervised learning algorithms need to adapt to new classes of bots [21, 44, 48].”
What the sources say
- Not checked yetcites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source
- Not checked yetcites [44] [44] Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018). -
- Not checked yetcites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also include additional specialized classifiers when new annotated datasets become available.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Echeverria and Zhou [19] suggest that detecting different types of bots requires different types of features.”
What the sources say
- Not checked yetcites [19] [19] Juan Echeverria and Shi Zhou. 2017. Discovery of the Twitter Bursty Botnet. arXiv preprint arXiv:1709.06740 (2017). -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this way, the tool offers greater transparency about the decision process by allowing inspection of the outputs of different ensemble components and by providing interpretable class labels in output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proposed architecture is highly modular as each specialized classifier works independently, so one can substitute any part with different models as needed.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“They also report that some bot classes, like social spambots, are hard to distinguish from human accounts.”
What the sources say
- Not checked yetcites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
High-quality datasets of annotated social media accounts are expensive to acquire.
“Unfortunately, high-quality datasets of annotated social media accounts are expensive to acquire.”
The authors state this as an assumption.
The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.
“These datasets made an assumption that all accounts belonging to American and German politicians are human accounts.”
The paper relies on this without support.
Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.
“Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.”
The paper relies on this without support.
Periodic retraining to include newly annotated datasets helps systems adapt to unseen examples.
“Periodic retraining to include newly annotated datasets helps systems adapt to these unseen examples.”
The paper relies on this without support.
A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.
“A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.”
The paper relies on this without support.
The F1 score is sensitive to class imbalance.
“the F1 score is sensitive to this class imbalance”
The paper relies on this without support.
The AUC metric is not sensitive to class imbalance.
“the AUC metric, which is not sensitive to class imbalance”
The paper relies on this without support.
The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.
“The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.”
The paper relies on this without support.
No systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.
“Despite the existence of different types of bots, no systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An important limitation of these methods is that they rely on features crafted specifically to target previously observed malicious behaviors [18].”
From A language framework for modeling social media account behavior (2023) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These findings are consistent with prior analysis based on ad-hoc features [18].”
From A language framework for modeling social media account behavior (2023) · cites this paper as [18]
Citations
Cites
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
Self-citationPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884 · cited 2×
“The bot score returned by ESC is obtained by calibrating the score si∗ using Platt’s scaling [31].”
MethodWeaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018) · American Journal of Public Health · doi:10.2105/ajph.2018.304567 · Cited by 1,149 · cited 1×
“spreading misinformation [36, 37], amplifying popularity [9, 34, 45], or polarizing online discussion [39]. Bot activity has been reported in different domains, including politics [5, 20, 39], health [2, 3, 6, 17], and business [12, 13]. Due to the wide adoption of social media, every aspect of people’s life from news…”
BackgroundOnline Human-Bot Interactions: Detection, Estimation, and Characterization
Varol, O., Ferrara, E., Davis, C., Menczer, F., & Flammini, A. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14871 · Cited by 981 · cited 3×
“posed in this paper, extract various features from an account’s profile, social network, and content [16, 23, 29, 41, 43, 48, 49].”
BackgroundMethodSelf-citationBotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909
Self-citationRatkiewicz (2011) · doi:10.1145/1963192.1963301 · Cited by 541
Self-citationDeep neural networks for bot detection
Kudugunta, S., & Ferrara, E. (2018) · Information Sciences · doi:10.1016/j.ins.2018.08.019 · Cited by 506 · cited 1×
“Depending on the type of bots, similarity can be detected through tweet content [8, 25], temporal features in the timelines [7, 10], or retweeting behavior [28, 46].”
BackgroundBots increase exposure to negative and inflammatory content in online social systems
Stella, M., Ferrara, E., & De Domenico, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1803470115 · Cited by 492
The DARPA Twitter Bot Challenge
Subrahmanian, V. S., Azaria, A., Durst, S., Kagan, V., Galstyan, A., Lerman, K., Zhu, L., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Computer · doi:10.1109/mc.2016.183 · Cited by 480 · cited 2×
“number of links to legitimate accounts. Supervised machine learning approaches, such as the one proposed in this paper, extract various features from an account’s profile, social network, and content [16, 23, 29, 41, 43, 48, 49]. These methods rely on annotated datasets for learning the difference between bot and huma…”
BackgroundSelf-citationFame for sale: Efficient detection of fake Twitter followers
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.09.003 · Cited by 475
Research works printed but not linked: 9
- Jon-Patrick Allem , Emilio Ferrara , Sree Priyanka Uppu , Tess Boley Cruz, and Jennifer B Unger. 2017 . E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98. Jon-Patrick Allem, Emilio Ferrara, Sree Priyanka Uppu, Tess Boley Cruz, and Jennifer B Unger. 2017. E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98.
- Alessandro Bessi and Emilio Ferrara . 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 ( 2016 ). Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 (2016).
- Emilio Ferrara . 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 ( 2017 ). Emilio Ferrara. 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 (2017).
- Kyumin Lee , Brian David Eoff , and James Caverlee . 2011 . Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter . In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM). Kyumin Lee, Brian David Eoff, and James Caverlee. 2011. Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter. In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM).
- Fabian Pedregosa , Gaël Varoquaux , Alexandre Gramfort , Vincent Michel , Bertrand Thirion , Olivier Grisel , Mathieu Blondel , Peter Prettenhofer , Ron Weiss , Vincent Dubourg , 2011 . Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830.
- Chengcheng Shao , Giovanni Luca Ciampaglia , Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018 . The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787. Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018. The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787.
- Galen Stoking and Nami Sumida . 2018. Social Media Bots Draw Public?s Attention and Concern . Pew Research Center ( 15 Oct 2018 ). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/ Galen Stoking and Nami Sumida. 2018. Social Media Bots Draw Public?s Attention and Concern. Pew Research Center (15 Oct 2018). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/
- Onur Varol , Clayton A Davis , Filippo Menczer , and Alessandro Flammini . 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics ( 2018 ), 311--334. Onur Varol, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics (2018), 311--334.
- Onur Varol and Ismail Uluturk . 2018. Deception strategies and threats for online discussions. First Monday 23, 5 ( 2018 ). Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018).
Cited by
Sorted by citations
TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
Feng, S., Wan, H., Wang, N., Li, J., & Luo, M. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482019 · Cited by 199
“Apart from that, verified-2019 [32], botwiki-2019 [32], cresci-rtbust-2019 [20], Astroturf [24] are also more recent bot detection datasets of various size and information completeness.”
BackgroundThe False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184
“The Botometer creators [43] argue in their newest paper that some of the German politicians’ accounts used in our study might have a certain degree of automation.”
BackgroundBotometer 101: social bot practicum for computational social scientists
Yang, K.-C., Ferrara, E., & Menczer, F. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00177-5 · Cited by 159
“This tutorial focuses on V4 [41].”
BackgroundSelf-citationUncovering Coordinated Networks on Social Media: Methods and Case Studies
Pacheco, D., Hui, P.-M., Torres-Lugo, C., Truong, B. T., Flammini, A., & Menczer, F. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18075 · Cited by 140
Self-citationMachine learning-based social media bot detection: a comprehensive literature review
Aljabri, M., Zagrouba, R., Shaahid, A., Alnasser, F., Saleh, A., & Alomari, D. M. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-022-01020-5 · Cited by 132
“In the work of (Sayyadiharikandeh et al. 2020) for each class of bots, they recommended training specialized classifiers and combining their conclusions using the maximum rule.”
BackgroundMethod
Metadata
- Authors
- Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F.
- Affiliations
- Indiana University Bloomington (US)
- Sabancı Üniversitesi (TR)
- Venue
- Proceedings of the 29th ACM International Conference on Information & Knowledge Management
- Publisher
- ACM
- Published
- 2020-10-19
- Pages
- 2725-2732
- ISSN
- 2155-0751 (ISSN-L)
- OpenAlex
- W3035082952
- Cited by
- 159
- References
- 34
- Citations by year
- 2026: 9 · 2025: 26 · 2024: 36 · 2023: 45 · 2022: 30 · 2021: 10 · 2020: 3
- FWCI
- 16.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Craig Newmark Philanthropies
- Defense Advanced Research Projects Agency (W911NF-17-C0094)
- John S. and James L. Knight Foundation
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- social bot detection, generalization to unseen classes, Botometer, misinformation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
