CoChat
Sign Up
Cited by 159Spam and Phishing DetectionMisinformation and Its ImpactsHate Speech and Cyberbullying Detection

Detection of Novel Social Bots by Ensembles of Specialized Classifiers

Open access
Sayyadiharikandeh, Mohsen, Varol, Onur, Yang, Kai-Cheng, Flammini, Alessandro, Menczer, Filippo·2020·Proceedings of the 29th ACM International Conference on Information & Knowledge Management
DOI 10.1145/3340531.3412698
Open this paper in CoChat — free accountOpen PDF

Abstract

Malicious actors create inauthentic social media accounts controlled in part by algorithms, known as social bots, to disseminate misinformation and agitate online discussion. While researchers have developed sophisticated methods to detect abuse, novel bots with diverse behaviors evade detection. We show that different types of bots are characterized by different behavioral features. As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data. Moreover, tuning these models to recognize novel bots requires retraining with a significant amount of new annotations, which are expensive to obtain. To address these issues, we propose a new supervised learning method that trains classifiers specialized for each class of bots and combines their decisions through the maximum rule. The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets. Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining. We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes an Ensemble of Specialized Classifiers (ESC) that trains separate Random Forest classifiers for different bot classes and combines them via the maximum rule to improve detection of novel social bots. ESC improves cross-domain bot recall and F1 over Botometer-v3 and is deployed in Botometer v4 with a cross-validation AUC of 0.99.

Key findings

  • ESC improves cross-domain performance on unseen accounts: across six datasets, recall rises from 42% to 84%, precision from 52% to 64%, and F1 from 47% to 73%, an average F1 improvement of 56%. On combined-test, recall goes from 77% to 86%, precision stays at 70%, and F1 rises from 73% to 77% (5% improvement).
  • In-domain performance is maintained: ESC achieves 0.96 AUC under 5-fold cross-validation, similar to Botometer’s 0.97 AUC, and the deployed Botometer v4 model yields a cross-validation AUC of 0.99.
  • Different bot classes rely on different informative features: in cresci-17, traditional spambots are best separated by adjective-frequency standard deviation, social spambots by tweet sentiment arousal entropy, and fake followers by maximum friend-follower ratio.
  • ESC adapts faster to new domains: when adding labeled examples from a held-out domain, ESC learns through a newly added specialized classifier while preserving old classifiers, requiring fewer labeled examples than retraining Botometer-v3 from scratch.

Methods

  • The study uses labeled bot/human datasets from the Bot Repository plus datasets from Rauchfleisch and Kaiser, extracting over 1,200 features in six categories: metadata, retweet/mention networks, temporal features, content, and sentiment.
  • ESC combines a human-detection classifier (RF0, corresponding to Botometer) with specialized Random Forest classifiers trained on balanced bot classes and human examples; bot scores are aggregated with the maximum rule, with Platt scaling in experiments.
  • Evaluation includes 5-fold cross-validation for in-domain performance and hold-out cross-domain tests on cresci-stock, gilani-17, cresci-rtbust, kaiser-1, kaiser-2, kaiser-3, and combined-test, using F1 and AUC with bootstrap confidence intervals; adaptation is tested on varol-icwsm.

Limitations

  • The paper states that cross-domain performance is highly sensitive to the datasets used in training and testing, making it easy to cherry-pick examples that make any method look bad; kaiser-1 and kaiser-3 are not completely hold-out because they include botwiki bots from ESC training, kaiser-3 shares accounts with kaiser-2 and kaiser-1, and kaiser-2 has only 27 bot accounts, making F1 sensitive to class imbalance.
  • The paper notes that tweetbotornot2 uses a “verified” profile feature and tends to assign low bot scores to verified accounts, biasing comparisons in its favor on datasets where many human-labeled accounts are verified.
  • The text makes evident that ground-truth labels can become invalid over time as accounts change, become inactive, or are compromised; the manual inspection of suspected mislabeled accounts covered only a random sample of 50 out of 332 highlighted accounts.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3340531.3412698

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 23 cited sources checked · 0 supported · 0 flagged

  • Findingnone given
    § ABSTRACT · p. 1

    Different types of bots are characterized by different behavioral features.

    “We show that different types of bots are characterized by different behavioral features.”
    Quote in the paperNot a substantial claim
  • Findingreasoning
    § ABSTRACT · p. 1

    Supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data.

    “As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data.”
    • Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · boosted
      “Both precision and recall tend to be higher for in-domain cases, demonstrating the limited generalization of supervised models across domains.”
    Quote in the paperSubstantial claim
  • Findingnone given
    § ABSTRACT · p. 1

    Novel bot behaviors are learned with fewer labeled examples during retraining.

    “Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining.”
    • Also stated in: § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1 · plain
      “It can also learn more efficiently from examples in new domains.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1

    Bot accounts exhibit greater heterogeneity in their discriminative behavioral features compared to human accounts.

    “Using bot and human accounts in different datasets, we show that bot accounts exhibit greater heterogeneity in their discriminative behavioral features compared to human accounts.”
    • Also stated in: § 2.2 Cross-domain performance comparison · p. 2 · plain
      “Human accounts, on the other hand, exhibit consistent characteristics across datasets.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4.2 Cross-domain performance · p. 5

    ESC outperforms the Botometer baseline in most cases.

    “Fig. 7 shows that ESC outperforms the Botometer baseline in most cases.”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 3
  • May combine two claims. 29
  • Classified as probably not a substantial claim. 24

  • The quote only partly states this claim.

    Findingown data
    § ABSTRACT · p. 1

    Botometer v4 has a cross-validation AUC of 0.99.

    “We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § ABSTRACT · p. 1

    ESC can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets.

    “The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56% in F1 score for unseen accounts across datasets.”
    • Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
      “On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%. As a result, F1 increases from 47% to 73% (an improvement of 56%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 2.2 Cross-domain performance comparison · p. 2

    Bot accounts tend to have lower cross-domain scores, resulting in lower recall.

    “This suggests that bot accounts tend to have lower cross-domain scores, resulting in lower recall.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 2.3 Predictability of different bot classes · p. 3

    In-domain, a linear classifier on two most informative features is sufficient to separate human and bot accounts in caverlee; cross-domain, the same linear model fails.

    “For the in-domain case (left), the linear classifier is sufficient to separate human and bot accounts. But in the cross-domain case (right), the same linear model fails, explaining the drop in recall: different features and distinct decision rules are needed to detect different classes of bots.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1

    The proposed approach effectively increases the recall of cross-domain bots without loss of in-domain accuracy.

    “Without loss of in-domain accuracy, the proposed approach effectively increases the recall of cross-domain bots.”
    • Also stated in: § 4.2 Cross-domain performance · p. 5 · plain
      “On average across the six datasets, recall goes from 42% to 84% (an improvement of 100%) while precision increases from 52% to 64%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Cross-domain performance · p. 5

    On combined-test dataset, ESC outperforms tweetbotornot2 on both F1 and AUC.

    “On the combined-test dataset, ESC outperforms tweetbotornot2 on both F1 and AUC.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.2 Cross-domain performance · p. 5

    ESC is comparable to tweetbotornot2 on this dataset when using the AUC metric, which is not sensitive to class imbalance.

    “ESC is comparable to tweetbotornot2 on this dataset when using the AUC metric, which is not sensitive to class imbalance.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The AUC metric is not sensitive to class imbalance.

  • May combine two claims.

    Findingown data
    § 4.2 Cross-domain performance · p. 5

    kaiser-3 contains human accounts from kaiser-2 and bot accounts from kaiser-1, so they are not independent.

    “kaiser-3 contains human accounts from kaiser-2 and bot accounts from kaiser-1 , so they are not independent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Cross-domain performance · p. 5

    tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets in AUC.

    “The AUC metric is comparable on those datasets, while tweetbotornot2 wins on gilani-17 and ESC wins on the other datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 4.2 Cross-domain performance · p. 5

    98% and 72% of accounts labeled human in kaiser-1 and kaiser-3 respectively are verified.

    “98% and 72% of accounts labeled human in kaiser-1 and kaiser-3 respectively are verified.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 In-domain performance · p. 4

    332 accounts in the disagreement region are labeled as human, 75% from caverlee, representing approximately 1% of human-labeled training examples.

    “We focus on 332 accounts in this region that are labeled as human (75% are from caverlee ), which represent approximately 1% of the examples labeled as human in the training data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Cross-domain performance · p. 5

    On combined-test dataset, recall goes from 77% to 86%, precision stays at 70%, and F1 goes from 73% to 77% (an improvement of 5%).

    “On the combined-test dataset, recall goes from 77% to 86%, precision stays at 70%, and F1 goes from 73% to 77% (an improvement of 5%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 4.2 Cross-domain performance · p. 5

    tweetbotornot2 uses a feature for 'verified' profiles and tends to assign a low bot score to them, even automated ones such as @twittersupport.

    “it uses a feature for 'verified' profiles and tends to assign a low bot score to them, even automated ones such as @twittersupport .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 In-domain performance · p. 4

    ESC achieves an in-domain AUC of 0.96, similar to the Botometer classifier (0.97 AUC).

    “Using 5-fold cross-validation, ESC achieves an Area Under the ROC Curve (AUC) of 0.96, similar to the Botometer classifier (0.97 AUC).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Cross-domain performance · p. 5

    Even if botwiki is excluded from training data, ESC still outperforms Botometer-v3, yielding F1 0.84 on kaiser-1 and 0.80 on kaiser-3.

    “Even if we exclude botwiki from the training data, ESC still outperforms Botometer-v3 . For example, it yields an F1 score of 0.84 on kaiser-1 and 0.80 on kaiser-3 .”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.

  • May combine two claims.

    Findingreasoning
    § 7 CONCLUSION · p. 7

    Despite impressive results when training and test sets are from the same domain, supervised models will miss new classes of bots, leading to low recall.

    “Despite impressive results when training and test sets are from the same domain - even using cross-validation - supervised models will miss new classes of bots, leading to low recall.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 4.3 Model adaptation · p. 5

    The authors reach similar conclusions using other hold-out datasets.

    “We reach similar conclusions using other hold-out datasets.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingreasoning
    § 4.3 Model adaptation · p. 5

    The Botometer-v3 baseline adapts more slowly than ESC.

    “The Botometer-v3 baseline adapts more slowly.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 7 CONCLUSION · p. 7

    The approach allows the system to learn about new domains efficiently, in the sense that fewer annotated examples are necessary.

    “We showed that this approach allows the system to learn about new domains in an efficient way, in the sense that fewer annotated examples are necessary.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7 CONCLUSION · p. 7

    Cross-domain performance is highly sensitive to the datasets used in training and testing supervised models; it is easy to cherry-pick examples that make any given method look bad.

    “our experiments show that cross-domain performance is highly sensitive to the datasets used in training and testing supervised models; it is easy to cherry-pick examples that make any given method look bad.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7 CONCLUSION · p. 7

    The proposed approach generalizes better than a monolithic classifier and is more robust to mislabeled training examples.

    “We demonstrated empirically that our proposed approach generalizes better than a monolithic classifier and is more robust to mislabeled training examples.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 6 BOTOMETER-V4 DEPLOYMENT · p. 6

    The authors are deploying ESC in the newest version of Botometer (v4).

    “we are deploying ESC in the newest version of Botometer (v4)”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning
    § 4.3 Model adaptation · p. 5

    Fig. 8 shows how the F1 score on the test data improves as a function of the number of training examples from the hold-out domain.

    “Fig. 8 shows how the F1 score on the test data improves as a function of the number of training examples from the hold-out domain.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § 6 BOTOMETER-V4 DEPLOYMENT · p. 6

    ESC provides a good correspondence between scores and binary labels of human and bot accounts, so no calibration is applied.

    “Since ESC provides a good correspondence between scores and binary labels of human and bot accounts, no calibration is applied.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1

    Bots with unseen characteristics are easily missed, as demonstrated by a drastic drop in recall when classifiers are faced with cross-domain accounts.

    “Bots with unseen characteristics are easily missed, as demonstrated by a drastic drop in recall when classifiers are faced with cross-domain accounts [18].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1 Baseline bot detection models · p. 3

    The current version of Botometer is often considered the state-of-the-art method for bot detection.

    “The first baseline is the current version of Botometer, often considered the state-of-the-art method for bot detection [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1

    Novel social bots emerge everyday and evade purge from platforms.

    “In an arms race between abusive behaviors and countermeasures, novel social bots emerge everyday and evade purge from the platforms [15, 21].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [15] [15] Stefano Cresci, Marinella Petrocchi, Angelo Spognardi, and Stefano Tognazzi. 2019. Better safe than sorry: an adversarial approach to improve social bot detection. In Proceedings of the 10th ACM Conference on Web Science . 47-56. - source

      Not checked in the free run.

    • cites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Detection of Novel Social Bots by Ensembles of Specialized Classifiers · p. 1

    Retraining supervised models to address lack of generalization requires new labeled datasets.

    “One common approach to address the lack of generalization is to retrain models with new labeled datasets [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source

      Not checked in the free run.

    • Rests on an assumption: High-quality datasets of annotated social media accounts are expensive to acquire.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 4.2 Cross-domain performance · p. 4

    Confidence intervals were obtained using bootstrapping with five samples of 80% of the accounts in the test set.

    “We obtain confidence intervals using bootstrapping with five samples of 80% of the accounts in the test set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § 4.2 Cross-domain performance · p. 5

    kaiser-1 and kaiser-3 datasets include bots from botwiki, which are part of ESC training data, so they cannot be considered completely hold-out.

    “The kaiser-1 and kaiser-3 datasets include bots from botwiki , which are part of the ESC training data. Therefore these two cannot be considered completely hold-out datasets, but are included nonetheless because they were used to highlight weaknesses of Botometer-v3 in a recent independent report [35], so they provide us with an opportunity to demonstrate the performance of the latest Botometer”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -

      Not checked in the free run.

    • The authors state this as an assumption: The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 4.2 Cross-domain performance · p. 4

    The language agnostic version of Botometer (excluding English-based linguistic features) was used on the kaiser-2 dataset because of its German tweet content.

    “The language agnostics version of Botometer (excluding English-based linguistic features) is used on the kaiser-2 dataset because of its German tweet content.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 2.2 Cross-domain performance comparison · p. 2

    The study used 5-fold cross-validation for in-domain classification and split training and test samples in cross-domain cases, reporting average precision and recall.

    “We use 5-fold cross-validation for in-domain classification; for consistency we split training and test samples in cross-domain cases as well, reporting average precision and recall.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5 RELATED WORK · p. 6

    Botometer currently does not use the 'verified' status as a feature because it could lead to false negatives.

    “Botometer currently does not use the 'verified' status as a feature because it could lead to false negatives [45].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] [45] Onur Varol and Ismail Uluturk. 2020. Journalists on Twitter: self-branding, audiences, and involvement of bots. Journal of Computational Social Science 3, 1 (01 Apr 2020), 83-101. https://doi.org/10.1007/s42001-019-00056-6 - source

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning
    § 4.3 Model adaptation · p. 5

    Because the hold-out dataset is small compared to the training size and decision trees use Gini gain, old bot classes with more examples dominate and the classifier struggles to learn about the new domain.

    “As a result, the old bot classes with more examples dominate and the classifier struggles to learn about the new domain.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 5 RELATED WORK · p. 6

    Rauchfleisch and Kaiser [35] may overestimate the false positive rate by assuming that no politician account uses automation.

    “Rauchfleisch and Kaiser [35] may overestimate the false positive rate by assuming that no politician account uses automation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 5 RELATED WORK · p. 6

    Like any supervised learning model, Botometer makes mistakes.

    “we share these concerns and acknowledge that like any supervised learning model, Botometer makes mistakes.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5 RELATED WORK · p. 6

    Even a classifier trained on 1.5 million accounts and 22 classes of bots is incapable of identifying new classes of bots.

    “Even a classifier trained on 1.5 million accounts and 22 classes of bots is incapable of identifying new classes of bots.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § 5 RELATED WORK · p. 6

    Rauchfleisch and Kaiser [35] overlook accounts that are no longer available due to suspension, possibly leading to an underestimation of both precision and recall.

    “Another source of bias is that Rauchfleisch and Kaiser [35] overlook accounts that are no longer available due to suspension, possibly leading to an underestimation of both precision and recall.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] [35] Adrian Rauchfleisch and Jonas Kaiser. 2020. The False Positive Problem of Automatic Bot Detection in Social Science Research. SSRN Electronic Journal (01 2020). https://doi.org/10.2139/ssrn.3565233 -

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation
    § 5 RELATED WORK · p. 6

    The new version introduced in the paper is motivated by the false positive/negative issue and partly addresses it by improving cross-domain recall (false negatives).

    “the new version introduced in the paper is motivated by this issue and partly addresses it by improving cross-domain recall (false negatives).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5 RELATED WORK · p. 6

    Echeverría et al. [18] proposed the leave-one-botnet-out methodology to highlight how detection methods do not generalize well to unseen bot classes due to bias in the training data.

    “Echeverría et al. [18] proposed the leaveone-botnet-out methodology, to highlight how detection methods do not generalize well to unseen bot classes due to bias in the training data.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] [18] Juan Echeverría, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, Shi Zhou, et al. 2018. LOBO-evaluation of generalization deficiencies in Twitter bot classifiers. In Proc. of the Annual Computer Security Applications Conf. ACM, 137-146. -

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § 4.3 Model adaptation · p. 5

    The ESC architecture quickly learns about new bots through the new classifier, which starts from scratch, while what was learned about the old ones is preserved in the existing classifiers.

    “the ESC architecture quickly learns about new bots through the new classifier, which starts from scratch, while what was learned about the old ones is preserved in the existing classifiers.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 5 RELATED WORK · p. 6

    Bots change and evolve continuously to evade detection, so supervised learning algorithms need to adapt to new classes of bots.

    “since bots change and evolve continuously to evade detection, supervised learning algorithms need to adapt to new classes of bots [21, 44, 48].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [21] [21] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96-104. - source

      Not checked in the free run.

    • cites [44] [44] Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018). -

      Not checked in the free run.

    • cites [48] [48] Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behav. and Emerging Technologies 1, 1 (2019), 48-61. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 7 CONCLUSION · p. 7

    Additional specialized classifiers can be included when new annotated datasets become available.

    “We can also include additional specialized classifiers when new annotated datasets become available.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5 RELATED WORK · p. 6

    Echeverria and Zhou [19] suggest that detecting different types of bots requires different types of features.

    “Echeverria and Zhou [19] suggest that detecting different types of bots requires different types of features.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19] [19] Juan Echeverria and Shi Zhou. 2017. Discovery of the Twitter Bursty Botnet. arXiv preprint arXiv:1709.06740 (2017). -

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 5 RELATED WORK · p. 6

    Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.

    “Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 6 BOTOMETER-V4 DEPLOYMENT · p. 7

    The tool offers greater transparency about the decision process by allowing inspection of the outputs of different ensemble components and by providing interpretable class labels in output.

    “In this way, the tool offers greater transparency about the decision process by allowing inspection of the outputs of different ensemble components and by providing interpretable class labels in output.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 7 CONCLUSION · p. 7

    The proposed architecture is highly modular as each specialized classifier works independently, so one can substitute any part with different models as needed.

    “The proposed architecture is highly modular as each specialized classifier works independently, so one can substitute any part with different models as needed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5 RELATED WORK · p. 6

    Cresci et al. [11] report that some bot classes, like social spambots, are hard to distinguish from human accounts.

    “They also report that some bot classes, like social spambots, are hard to distinguish from human accounts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In Proc. Intl. Conf. of the Web Companion . 963-972. - source

      Not checked in the free run.

48 of 48

  • The paper relies on this without support.

    High-quality datasets of annotated social media accounts are expensive to acquire.

    “Unfortunately, high-quality datasets of annotated social media accounts are expensive to acquire.”

  • The authors state this as an assumption.

    The datasets from Rauchfleisch and Kaiser assume that all accounts belonging to American and German politicians are human accounts.

    “These datasets made an assumption that all accounts belonging to American and German politicians are human accounts.”

  • The paper relies on this without support.

    Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.

    “Real-world applications of social bot detection always face the challenge of recognizing novel behaviors that are not represented in the training data.”

  • The paper relies on this without support.

    Periodic retraining to include newly annotated datasets helps systems adapt to unseen examples.

    “Periodic retraining to include newly annotated datasets helps systems adapt to these unseen examples.”

  • The paper relies on this without support.

    A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.

    “A common approach is to train a new classifier from scratch including both old and new training data, which may not be efficient.”

  • The paper relies on this without support.

    The F1 score is sensitive to class imbalance.

    “the F1 score is sensitive to this class imbalance”

  • The paper relies on this without support.

    The AUC metric is not sensitive to class imbalance.

    “the AUC metric, which is not sensitive to class imbalance”

  • The paper relies on this without support.

    The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.

    “The decision trees use Gini gain as a feature selection criterion, therefore the number of examples sharing the same informative features affects the selection of those features.”

  • The paper relies on this without support.

    No systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.

    “Despite the existence of different types of bots, no systems have been presented in the literature to automatically identify the type of a bot as the method proposed here.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing detection methods rely on features crafted specifically to target previously observed malicious behaviors.

    “An important limitation of these methods is that they rely on features crafted specifically to target previously observed malicious behaviors [18].”

    From A language framework for modeling social media account behavior (2023) · cites this paper as [18]

    The Related Work section (p. 6) says supervised bot detectors use account features and annotated datasets, and the paper’s introduction (p. 1) says bots with unseen characteristics are easily missed. This supports the limitation involving previously observed behavior, but the copy does not say the features were crafted specifically to target that behavior. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • These findings are consistent with prior analysis based on ad-hoc features.

    “These findings are consistent with prior analysis based on ad-hoc features [18].”

    From A language framework for modeling social media account behavior (2023) · cites this paper as [18]

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. Jon-Patrick Allem , Emilio Ferrara , Sree Priyanka Uppu , Tess Boley Cruz, and Jennifer B Unger. 2017 . E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98. Jon-Patrick Allem, Emilio Ferrara, Sree Priyanka Uppu, Tess Boley Cruz, and Jennifer B Unger. 2017. E-cigarette surveillance with social media data: social bots, emerging topics, and trends. JMIR public health and surveillance 3, 4 (2017), e98.
  2. Alessandro Bessi and Emilio Ferrara . 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 ( 2016 ). Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 US Presidential election online discussion. First Monday 21, 11 (2016).
  3. Emilio Ferrara . 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 ( 2017 ). Emilio Ferrara. 2017. Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election. First Monday 22, 8 (2017).
  4. Kyumin Lee , Brian David Eoff , and James Caverlee . 2011 . Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter . In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM). Kyumin Lee, Brian David Eoff, and James Caverlee. 2011. Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter. In Proc. AAAI Intl. Conf. on Web and Social Media (ICWSM).
  5. Fabian Pedregosa , Gaël Varoquaux , Alexandre Gramfort , Vincent Michel , Bertrand Thirion , Olivier Grisel , Mathieu Blondel , Peter Prettenhofer , Ron Weiss , Vincent Dubourg , 2011 . Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825--2830.
  6. Chengcheng Shao , Giovanni Luca Ciampaglia , Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018 . The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787. Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menczer. 2018. The spread of low-credibility content by social bots. Nature communications 9, 1 (2018), 4787.
  7. Galen Stoking and Nami Sumida . 2018. Social Media Bots Draw Public?s Attention and Concern . Pew Research Center ( 15 Oct 2018 ). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/ Galen Stoking and Nami Sumida. 2018. Social Media Bots Draw Public?s Attention and Concern. Pew Research Center (15 Oct 2018). https://www.journalism.org/2018/10/15/social-media-bots-draw-publics-attention-and-concern/
  8. Onur Varol , Clayton A Davis , Filippo Menczer , and Alessandro Flammini . 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics ( 2018 ), 311--334. Onur Varol, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2018. Feature Engineering for Social Bot Detection. Feature Engineering for Machine Learning and Data Analytics (2018), 311--334.
  9. Onur Varol and Ismail Uluturk . 2018. Deception strategies and threats for online discussions. First Monday 23, 5 ( 2018 ). Onur Varol and Ismail Uluturk. 2018. Deception strategies and threats for online discussions. First Monday 23, 5 (2018).

Cited by

Sorted by citations

Authors
Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F.
Affiliations
  • Indiana University Bloomington (US)
  • Sabancı Üniversitesi (TR)
ORCID
Venue
Proceedings of the 29th ACM International Conference on Information & Knowledge Management
Publisher
ACM
Published
2020-10-19
Pages
2725-2732
ISSN
2155-0751 (ISSN-L)
DOI
10.1145/3340531.3412698
OpenAlex
W3035082952
Cited by
159
References
34
Citations by year
2026: 9 · 2025: 26 · 2024: 36 · 2023: 45 · 2022: 30 · 2021: 10 · 2020: 3
FWCI
16.69
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM International Conference on Information and Knowledge Management (CIKM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Craig Newmark Philanthropies
  • Defense Advanced Research Projects Agency (W911NF-17-C0094)
  • John S. and James L. Knight Foundation
Topics
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
social bot detection, generalization to unseen classes, Botometer, misinformation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review