The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Open accessAbstract
Recent studies in social media spam and automation provide anecdotal argumentation of the rise of a new generation of spambots, so-called social spambots. Here, for the first time, we extensively study this novel phenomenon on Twitter and we provide quantitative evidence that a paradigm-shift exists in spambot design. First, we measure current Twitter's capabilities of detecting the new social spambots. Later, we assess the human performance in discriminating between genuine accounts, social spambots, and traditional spambots. Then, we benchmark several state-of-the-art techniques proposed by the academic literature. Results show that neither Twitter, nor humans, nor cutting-edge applications are currently capable of accurately detecting the new social spambots. Our results call for new approaches capable of turning the tide in the fight against this raising phenomenon. We conclude by reviewing the latest literature on spambots detection and we highlight an emerging common research trend based on the analysis of collective behaviors. Insights derived from both our extensive experimental campaign and survey shed light on the most promising directions of research and lay the foundations for the arms race against the novel social spambots. Finally, to foster research on this novel phenomenon, we make publicly available to the scientific community all the datasets used in this study.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper provides quantitative evidence that a new generation of Twitter “social spambots” mimics genuine users closely enough to evade Twitter, human annotators, and several state-of-the-art detection tools. It argues for a paradigm shift toward group-level/collective-behavior detection and releases annotated datasets.
Key findings
- Twitter’s own suspensions barely affect social spambots: survival rates were 95.2%, 96.1%, and 99.6% for social spambots #1–#3, close to genuine accounts at 96.5%; by contrast, traditional spambots #2 were 1.0% alive / 99.0% suspended, and fake followers were 25.4% alive / 73.5% suspended.
- Human annotators failed on social spambots: accuracy 0.2355 with 1,065 false negatives, versus 0.9136 on traditional spambots and 0.9201 on genuine accounts; Fleiss’ kappa was 0.186 for social spambots, 0.007 for traditional spambots, and 0.410 for genuine accounts.
- Established detection tools also failed: BotOrNot? achieved F-Measure 0.288 and MCC 0.174 on test set #1; Yang et al. achieved F-Measure 0.261 and MCC 0.043; Miller et al. performed worst.
- Group-level approaches performed much better: Ahmed et al. modified with fastgreedy community detection reached MCC 0.886 on test set #1 and 0.847 on test set #2; Cresci et al.’s digital DNA reached MCC 0.952 and 0.867, respectively.
Methods
- The study used Twitter datasets: 3,474 genuine accounts / 8,377,522 tweets; social spambots #1 (991 accounts / 1,610,176 tweets), #2 (3,457 / 428,542), #3 (464 / 1,418,626); traditional spambots #1–#4; fake followers (3,351 / 196,027); and two mixed test sets of 1,982 and 928 accounts.
- Twitter monitoring used API error codes 63 (suspended) and 50 (deleted) to measure account survival. Crowdsourcing used CrowdFlower with 4,428 accounts, at least 3 contributors per account, 247 trusted contributors from 42 countries, 13,284 answers, majority voting, and a 70% test-question accuracy threshold.
- The authors reproduced and benchmarked BotOrNot?, Yang et al.’s supervised classifier, Miller et al.’s stream clustering, Ahmed et al.’s graph clustering (modified with fastgreedy), and Cresci et al.’s digital DNA, evaluating precision, recall, specificity, accuracy, F-Measure, and MCC on the test sets.
Limitations
- Stated: BotOrNot?’s authors note its performance against evolved spambots may be worse than originally reported; Viswanath et al.’s technique cannot directly spot individual tampering accounts; and Table 7 is explicitly not complete but illustrative of an emerging trend.
- Stated/evident: 142 accounts (3.2%) were deleted, suspended, or protected during the crowdsourcing campaign, leaving 4,286 instead of 4,428; contributors were tech-savvy Twitter users and filtered by >70% test accuracy, so human-performance results may not generalize to all users.
- Evident: the datasets come from specific Twitter campaigns and years and rely on manual annotation; the social-spambot ground truth used group comparison, and the paper does not establish performance on other platforms or future spambot generations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 17 cited sources checked · 0 supported · 0 flagged
- Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 6
The authors measured a slightly higher agreement for the classification of social spambots than for traditional ones, with κ = 0.186.
“Surprisingly, we measured a slightly higher agreement for the classification of social spambots than for traditional ones, with κ = 0 . 186.”
Quote in the paperSubstantial claim- Rests on an assumption: The level of agreement measured by Fleiss' kappa can be interpreted as a proxy for the difficulty of a task.
- Findingown data§ 3.1 Twitter monitoring · p. 4
Social spambots feature very small differences with respect to genuine accounts (∼ ± 3%) in ratios of alive, deleted, and suspended accounts.
“As shown, social spambots feature very small differences with respect to genuine accounts ( ∼ ± 3%).”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Rests on an assumption: The genuine accounts that answered the crowdsensing question are human-operated.
- Findingown data§ 3.1 Twitter monitoring · p. 3
Genuine accounts feature a very high survival rate (96.5%).
“As shown in Table 2, genuine accounts feature a very high survival rate (96.5%).”
Quote in the paperNot a substantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Rests on an assumption: The genuine accounts that answered the crowdsensing question are human-operated.
- Findingown data§ 2. DATASETS · p. 2
The automated accounts in social spambots #1 were similar to genuine ones in every way.
“Surprisingly, we found such automated accounts to be similar to genuine ones in every way.”
Quote in the paperSubstantial claim - Findingown data§ 3.1 Twitter monitoring · p. 4
Some of these differences between social spambots and genuine accounts are not even statistically significant, according to a chi-square test.
“Some of these differences are not even statistically significant, according to a chi-square test.”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Rests on an assumption: The genuine accounts that answered the crowdsensing question are human-operated.
- Findingown data§ 2. DATASETS · p. 2
The deceitful activity of social spambots #3 was carried out by spamming URLs pointing to the advertised products.
“The deceitful activity was carried out by spamming URLs pointing to the advertised products.”
Quote in the paperNot a substantial claim - Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 6
Human contributors showed very little agreement while classifying traditional spambots, as represented by κ = 0.007.
“Instead, they showed very little agreement while classifying traditional spambots, as represented by κ = 0 . 007.”
Quote in the paperSubstantial claim- Rests on an assumption: The level of agreement measured by Fleiss' kappa can be interpreted as a proxy for the difficulty of a task.
- Findingown data§ 3.1 Twitter monitoring · p. 4
Traditional spambots #1 have largely managed to evade suspension, despite dating back to 2009.
“So far, traditional spambots #1 have largely managed to evade suspension, despite dating back to 2009.”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Findingown data§ 2. DATASETS · p. 2
Social spambots #3 advertise products on sale on Amazon.com.
“Further, we uncovered a third group of social bots, social spambots #3 , which advertise products on sale on Amazon.com .”
Quote in the paperNot a substantial claim - Findingreasoning§ 3.1 Twitter monitoring · p. 3
Among the no longer available genuine accounts, the vast majority have been deleted by the original owner, rather than suspended by Twitter.
“In addition, among the no longer available accounts, the vast majority have been deleted by the original owner, rather than suspended by Twitter.”
Quote in the paperNot a substantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Rests on an assumption: The genuine accounts that answered the crowdsensing question are human-operated.
- Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 6
Human contributors showed a decent agreement for the classification of genuine accounts, with κ = 0.410.
“In our experiment, human contributors showed a decent agreement for the classification of genuine accounts, with κ = 0 . 410.”
Quote in the paperNot a substantial claim- Rests on an assumption: The level of agreement measured by Fleiss' kappa can be interpreted as a proxy for the difficulty of a task.
- Findingown data§ 2. DATASETS · p. 3
89.29% (464 accounts) of the accounts that tweeted suspicious Amazon.com URLs were certified as spambots.
“Finally, 89.29% (464 accounts) of the accounts that tweeted suspicious Amazon.com URLs were also certified as spambots.”
Quote in the paperSubstantial claim- Rests on an assumption: The internal manual verification correctly certifies the automated nature of the accounts.
- Findingown data§ 2. DATASETS · p. 3
Among all distinct retweeters of the Italian political candidate, 50.05% (991 accounts) were certified as spambots.
“Among all the distinct retweeters of the Italian political candidate, 50.05% (991 accounts) were certified as spambots.”
Quote in the paperSubstantial claim- Rests on an assumption: The internal manual verification correctly certifies the automated nature of the accounts.
- Findingown data§ 2. DATASETS · p. 2
Every profile of social spambots #1 was accurately filled in with detailed yet fake personal information such as a (stolen) photo, (fake) shortbio, (fake) location, etc.
“Every profile was accurately filled in with detailed - yet fake personal information such as a (stolen) photo, (fake) shortbio, (fake) location, etc.”
Quote in the paperSubstantial claim - Findingown data§ 3.1 Twitter monitoring · p. 4
Traditional spambots #1 have differences ∼ ± 8% that are highly significant (p < 0.01) for alive and suspended accounts.
“Traditional spambots #1 have differences ∼ ± 8% that are highly significant ( p < 0 . 01) for alive and suspended accounts.”
Quote in the paperNot a substantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 6
The traditional account-by-account annotation process used by human workers to evaluate social media accounts is no longer viable when applied to the detection of the novel wave of social spambots.
“However, the results of our crowdsourcing experiment confirmed that the traditional 'account-byaccount'annotation process used by human workers to evaluate social media accounts is no longer viable when applied to the detection of the novel wave of social spambots.”
Quote in the paperSubstantial claim - Findingown data§ 2. DATASETS · p. 2
Social spambots #2 spent several months promoting the #TALNTS hashtag.
“We also discovered a second group of social bots, which we labeled social spambots #2 , who spent several months promoting the #TALNTS hashtag.”
Quote in the paperNot a substantial claim - Findingreasoning§ 2. DATASETS · p. 2
Social spambots #3 interleaved spam tweets with harmless and genuine ones.
“Similarly to the retweeters of the Italian political candidate, also this family of spambots interleaved spam tweets with harmless and genuine ones.”
Quote in the paperNot a substantial claim - Findingown data§ 2. DATASETS · p. 2
Every time the political candidate posted a new tweet from his official account, all the automated accounts retweeted it in a time span of just a few minutes.
“However, every time the political candidate posted a new tweet from his official account, all the automated accounts retweeted it in a time span of just a few minutes.”
Quote in the paperSubstantial claim - Findingcitation§ Keywords · p. 2
The paper provides empirical evidence of the existence of a novel wave of Twitter spambots, which up to now has been just theorized.
“We provide empirical evidence of the existence of a novel wave of Twitter spambots, which, up to now, has been just theorized [16].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [16] E. Ferrara, O. Varol, C. Davis, F. Menczer, and A. Flammini. The rise of social bots. Commun. ACM, 59(7):96--104, 2016. source
Not checked in the free run.
- Findingown data§ 2. DATASETS · p. 2
The most common form of human-to-spambot interaction was a human reply to one of the spambot tweets quotes.
“The most common form of such human-to-spambot interaction was represented by a human reply to one of the spambot tweets quotes.”
Quote in the paperNot a substantial claim - Findingreasoning§ 3.2 Crowdsourcing: tasks and results · p. 6
By comparing the behaviors of different accounts rather than analyzing them one by one, the authors were able to spot the social spambots among all collected accounts.
“By comparing the behaviors of different accounts, rather than by analyzing them one by one, we were able to spot the social spambots among all the collected accounts, as thoroughly described in Section 2.”
Quote in the paperSubstantial claim - Findingown data§ 2. DATASETS · p. 2
Social spambots #1 showed a tweeting behavior that was apparently similar to genuine accounts, with a few tweets posted every day, mainly quotes from popular people, songs, and YouTube videos.
“Furthermore, the accounts showed a tweeting behavior that was apparently similar to those of genuine accounts, with a few tweets posted every day, mainly quotes from popular people, songs, and YouTube videos.”
Quote in the paperSubstantial claim - Findingreasoning§ 3.1 Twitter monitoring · p. 3
Fake followers have mostly been detected and suspended by Twitter.
“Conversely, the simplest kind of malicious accounts, fake followers , have mostly been detected and suspended by Twitter.”
Quote in the paperNot a substantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 5
Human detection performances for traditional spambots and genuine accounts are quite satisfactory, with an accuracy of 0.91 and 0.92 respectively.
“Human detection performances for the two other groups of accounts, namely traditional spambots and genuine accounts, are instead quite satisfactory, with an accuracy of 0.91 and 0.92 respectively.”
Quote in the paperSubstantial claim - Findingown data§ 2. DATASETS · p. 3
94.50% (3,457 accounts) of the accounts who tweeted the #TALNTS hashtag resulted as spambots.
“Similarly, 94.50% (3,457 accounts) of the accounts who tweeted the #TALNTS hashtag resulted as spambots.”
Quote in the paperSubstantial claim- Rests on an assumption: The internal manual verification correctly certifies the automated nature of the accounts.
- Findingown data§ 6. CONCLUDING REMARKS · p. 9
The authors' experiments highlighted that the majority of existing automated systems, as well as crowdsourcing, erroneously label social spambots as genuine (human-operated) accounts.
“Indeed, our experiments highlighted that the majority of existing automated systems, as well as crowdsourcing, erroneously label social spambots as genuine (human-operated) accounts.”
Quote in the paperSubstantial claim - Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 7
The approach of Ahmed et al. [2] failed to identify 2 distinct clusters, since accounts of both test sets were assigned to a single cluster.
“However, the approach failed to identify 2 distinct clusters, since accounts of both our test-sets were assigned to a single cluster.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [2] F. Ahmed and M. Abulaish. A generic statistical approach for spam detection in online social networks. Computer Communications, 36(10):1120--1129, 2013. source
Not checked in the free run.
- Findingnone given§ 4. ESTABLISHED TECHNIQUES · p. 7
Novel social spambots tweet contents similar to that of genuine accounts, such as retweets of genuine tweets and famous quotes.
“However, novel social spambots tweet contents similar to that of genuine accounts (e.g., retweets of genuine tweets and famous quotes).”
Quote in the paperSubstantial claim - Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 6
BotOrNot?'s detection performances are particularly bad for the accounts of test set #1, where the spambots are from the social spambots #1 group.
“Its detection performances are particularly bad for the accounts of test set #1 - where the spambots are from the social spambots #1 group.”
Quote in the paperNot a substantial claim - Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 6
BotOrNot? achieves rather unsatisfactory results for the accounts of both test set #1 and test set #2.
“As shown in Table 7, BotOrNot? achieves rather unsatisfactory results for the accounts of both test set #1 and test set #2 (such datasets are described in Table 1).”
Quote in the paperSubstantial claim - Findingreasoning§ 4. ESTABLISHED TECHNIQUES · p. 6
For BotOrNot? on test set #1, the low F-Measure and MCC values (0.288 and 0.174) are mainly due to low Recall.
“The low values of F-Measure and Mathews Correlation Coefficient (MCC), respectively 0.288 and 0.174, are mainly due to the low Recall.”
Quote in the paperNot a substantial claim - Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 6
The supervised system of C. Yang et al. [44] fails to correctly classify the novel social spambots.
“Results in Table 7 show that the system fails to correctly classify the novel social spambots.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [44] C. Yang, R. Harkreader, and G. Gu. Empirical evaluation and new design for fighting evolving Twitter spammers. IEEE Trans. Inform. Forens. Sec., 8(8):1280--1293, 2013. source
Not checked in the free run.
- Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 6
For the C. Yang et al. system, the worst results in both test set #1 and test set #2 are related to the Recall metric.
“Similarly to the results of the BotOrNot? service, the worst results of this system in both test set #1 and test set #2 are related to the Recall metric.”
Quote in the paperNot a substantial claim - Findingown data§ 5. EMERGING TRENDS · p. 8
As measured by the LCS metric, social spambots #1 and #3 feature a level of similarity much higher than that of genuine accounts.
“More specifically, as measured by the LCS metric, social spambots #1 and #3 feature a level of similarity much higher than that of genuine accounts.”
Quote in the paperSubstantial claim - Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 7
A grid search simulation to test the best parameter configuration for MCL did not yield a better result.
“We also performed a grid search simulation in order to test the best parameter configuration for MCL 8 , but to no avail.”
Quote in the paperSubstantial claim - Findingown data§ 6. CONCLUDING REMARKS · p. 9
In the authors' experiments, neither humans nor state-of-the-art spambot detection applications managed to accurately detect the accounts belonging to the new wave of spambots.
“In our experiments, neither humans nor state-of-the-art spambot detection applications managed to accurately detect the accounts belonging to this new wave of spambots.”
Quote in the paperSubstantial claim - Findingown data§ 5. EMERGING TRENDS · p. 8
The digital DNA-based technique of Cresci et al. [14] achieves excellent detection performances.
“Results reported in Table 7 demonstrate that the digital DNA-based technique [14] achieves excellent detection performances.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [14] S. Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, and M. Tesconi. DNA-inspired online behavioral modeling and its application to spambot detection. IEEE Intelligent Systems, 5(31):58--64, 2016. source
Not checked in the free run.
- Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 7
The unsupervised approach of Miller et al. [31] achieved the worst performances among all techniques benchmarked in this study.
“As shown in Table 7, this achieved the worst performances among all those that we have benchmarked in this study.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [31] Z. Miller, B. Dickinson, W. Deitrick, W. Hu, and A. H. Wang. Twitter spammer detection using data stream clustering. Information Sciences, 260:64--73, 2014. source
Not checked in the free run.
- Findingown data§ 6. CONCLUDING REMARKS · p. 9
The authors' long-lasting experiment on malicious accounts survival rate in Twitter demonstrated that spambot detection is still an open issue.
“Our long-lasting experiment on malicious accounts survival rate in Twitter demonstrated that spambot detection is still an open issue.”
Quote in the paperSubstantial claim - Findingown data§ 5. EMERGING TRENDS · p. 8
Applying the digital DNA technique to the authors' datasets, the similarity curve of genuine accounts is significantly different from that of social spambots.
“Applying the technique to our datasets, the similarity curve of genuine accounts is significantly different from that of social spambots, as shown in Figure 4(c).”
Quote in the paperSubstantial claim - Interpretationown data§ 3.1 Twitter monitoring · p. 4
Social spambots appear to be more similar to genuine accounts than to traditional spambots, with regards to Twitter suspensions.
“Notably, social spambots appear to be more similar to genuine accounts than to traditional spambots, with regards to Twitter suspensions.”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Rests on an assumption: The genuine accounts that answered the crowdsensing question are human-operated.
- Interpretationown data§ ABSTRACT · p. 1
The paper provides quantitative evidence that a paradigm-shift exists in spambot design.
“Here, for the first time, we extensively study this novel phenomenon on Twitter and we provide quantitative evidence that a paradigm-shift exists in spambot design.”
Quote in the paperSubstantial claim - Study design and conductreasoning§ 3.2 Crowdsourcing: tasks and results · p. 5
By designing test questions with traditional spambots and genuine accounts, and by enforcing the policy of at least 70% correct answers, the authors can guarantee that all trusted contributors are typically able to detect traditional spambots and to distinguish them from genuine accounts.
“Notably, by designing test questions with traditional spambots and genuine accounts, and by enforcing the policy of at least 70% correct answers, we can guarantee that all our trusted contributors are typically able to detect traditional spambots and to distinguish them from genuine accounts.”
Quote in the paperNot a substantial claim - Study design and conductreasoning§ 3.2 Crowdsourcing: tasks and results · p. 5
Only trusted contributors who answered correctly to more than 70% of the test questions have been considered in the study.
“Only the trusted contributors who answered correctly to more than the 70% of the test questions have been considered in our study.”
Quote in the paperSubstantial claim - Interpretationown data§ 1. INTRODUCTION · p. 1
The characteristics of the new wave of social spambots are such that standard classification approaches, where a single account is evaluated according to a set of established features tested over known datasets, are no longer successful.
“Unfortunately, the characteristics of the new wave of social spambots are such that standard classification approaches, where a single account is evaluated according to a set of established features tested over known datasets, are no longer successful.”
Quote in the paperSubstantial claim - Interpretationown data§ 3.2 Crowdsourcing: tasks and results · p. 5
These results highlight the existence of a striking difference between traditional and social spambots.
“These important results further highlight the existence of a striking difference between traditional and social spambots.”
Quote in the paperSubstantial claim - Interpretationown data§ 3.1 Twitter monitoring · p. 4
These results suggest an increased difficulty for the detection of social spambots.
“Even if the difference between the survival rate of social spambots and that of traditional spambots #1 is marginal, these results nonetheless suggest an increased difficulty for the detection of social spambots.”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
- Premisecitation§ 5. EMERGING TRENDS · p. 8
Since 2013, a number of research teams independently started to formalize new approaches, including algorithmic ones, for detecting coordinated and synchronized behavior that characterizes groups of automated malicious accounts.
“However, since 2013, a number of research teams independently started to formalize new approaches, also from the algorithmic point of view, for detecting the coordinated and synchronized behavior that characterizes groups of automated malicious accounts [6].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [6] A. Beutel, W. Xu, V. Guruswami, C. Palow, and C. Faloutsos. CopyCatch: stopping group attacks by spotting lockstep behavior in social networks. In WWW. ACM, 2013. source
Not checked in the free run.
- Interpretationreasoning§ 5. EMERGING TRENDS · p. 9
A large enough group of spambots will still leave traces of automation, no matter how sophisticated a single spambot can be.
“Focusing on groups has the advantage that, no matter how sophisticated a single spambot can be, a large enough group of spambots will still leave traces of automation, since they do have a common goal (e.g., increasing someone's reputation score).”
Quote in the paperSubstantial claim- Rests on an assumption: Social spambots have a common goal, such as increasing someone's reputation score.
- Interpretationreasoning§ 4. ESTABLISHED TECHNIQUES · p. 6
The C. Yang et al. classifier labeled social spambots as genuine accounts.
“This means that also this classifier labeled social spambots as genuine accounts.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4. ESTABLISHED TECHNIQUES · p. 6
BotOrNot? has a tendency to label social spambots #1 as genuine accounts.
“In turn, this represents a tendency of labeling social spambots #1 as genuine accounts.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 4. ESTABLISHED TECHNIQUES · p. 7
An approach almost solely based on tweet content will not be able to achieve satisfactory results for detecting novel social spambots.
“For this reason, an approach almost solely based on tweet content will not be able to achieve satisfactory results.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6. CONCLUDING REMARKS · p. 9
By accurately mimicking the characteristics of genuine users, social spambots are intrinsically harder to detect than those studied by Academia in the past years.
“By accurately mimicking the characteristics of genuine users, these spambots are intrinsically harder to detect than those studied by Academia in the past years.”
Quote in the paperSubstantial claim - Interpretationown data§ 5. EMERGING TRENDS · p. 8
The technique proposed by Viswanath et al. [40] is capable of spotting the differences between groups of genuine accounts and the new wave of social spambots.
“Thus, the technique proposed in [40] is capable of spotting the differences between groups of genuine accounts and the new wave of social spambots.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [40] B. Viswanath, M. A. Bashir, M. B. Zafar, S. Bouget, S. Guha, K. P. Gummadi, A. Kate, and A. Mislove. Strength in numbers: Robust tamper detection in crowd computations. In COSN. ACM, 2015. source
Not checked in the free run.
- Interpretationreasoning§ 6. CONCLUDING REMARKS · p. 9
The already difficult problem of detecting spambots in social media is bound to worsen because of the emergence of a new wave of social spambots.
“Moreover, the already difficult problem to detect spambots in social media is bound to worsen, since the emergence of a new wave of so-called social spambots.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5. EMERGING TRENDS · p. 9
By performing analyses at group level, the emerging trend might be able to significantly raise the bar for social spambots to evade detection.
“By performing analyses at group level, this emerging trend might be able to significantly raise the bar for social spambots to evade detection.”
Quote in the paperNot a substantial claim - Premisecitation§ 5. EMERGING TRENDS · p. 8
Although working at group level, the digital DNA technique of Cresci et al. [14] is capable of spotting single spambot accounts.
“Notably, although working at group level, [14] is capable of spotting single spambot accounts.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [14] S. Cresci, R. Di Pietro, M. Petrocchi, A. Spognardi, and M. Tesconi. DNA-inspired online behavioral modeling and its application to spambot detection. IEEE Intelligent Systems, 5(31):58--64, 2016. source
Not checked in the free run.
- Premisecitation§ 4. ESTABLISHED TECHNIQUES · p. 7
Among the 126 features used by Miller et al. [31], 95 are based on the textual content of tweets.
“Among the 126 features, 95 are based on the textual content of tweets.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [31] Z. Miller, B. Dickinson, W. Deitrick, W. Hu, and A. H. Wang. Twitter spammer detection using data stream clustering. Information Sciences, 260:64--73, 2014. source
Not checked in the free run.
- Premisecitation§ 4. ESTABLISHED TECHNIQUES · p. 7
Social spambots might be so sophisticatedly designed that it is very difficult to distinguish them from genuine accounts if observed one by one.
“Our initial claim, supported by preliminary work [16, 48], is that social spambots might be so sophisticatedly designed to make it very difficult to distinguish them from genuine accounts, if observed one by one.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [16] E. Ferrara, O. Varol, C. Davis, F. Menczer, and A. Flammini. The rise of social bots. Commun. ACM, 59(7):96--104, 2016. source
Not checked in the free run.
- cites [48] J. Zhang, R. Zhang, Y. Zhang, and G. Yan. The rise of social botnets: Attacks and countermeasures. arXiv preprint arXiv:1603.02714, 2016.
Not checked in the free run.
- Classified as probably not a substantial claim. 13
- May combine two claims. 21
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 5Overall, the authors collected 13,284 answers given by 247 trusted contributors from 42 different countries.
“Overall, we collected 13,284 answers given by 247 trusted contributors from 42 different countries.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 5Out of the 247 participating contributors, 60 of them (∼24%) evaluated the campaign, leading to an aggregated score of 3.7/5.
“Out of the 247 participating contributors, 60 of them ( ∼ 24%) evaluated our campaign, leading to a convincing aggregated score of 3.7/5, as shown in detail in Table 5.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone given§ 3.1 Twitter monitoring · p. 4Traditional spambots #2 and fake followers show massive differences: ∼ ± 96% and ∼ ± 72%, respectively.
“Instead, traditional spambots #2 and fake followers show massive differences: ∼ ± 96% and ∼ ± 72%, respectively.”
Quote in the paperNot a substantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
May combine two claims.
Findingnone given§ 4. ESTABLISHED TECHNIQUES · p. 6Neither Twitter nor human operators are currently capable of identifying novel social spambots.
“So far, we have demonstrated that neither Twitter nor human operators are currently capable of identifying novel social spambots.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 2. DATASETS · p. 2The authors found tens of human accounts who tried to engage in conversation with some of the spambots.
“Amazingly, we also found tens of human accounts who tried to engage in conversation with some of the spambots.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingcitation§ 2. DATASETS · p. 2Social spambots #1 accounts represented credible sources of information since they all had thousands of followers and friends, the majority of which were genuine users.
“Those accounts also represented credible sources of information since they all had thousands of followers and friends, the majority of which were genuine users 2 .”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 3.1 Twitter monitoring · p. 4All 3 groups of social spambots feature very high survival rates, respectively 95.2%, 96.1%, and 99.6%.
“All the 3 groups of social spambots feature very high survival rates, respectively 95.2%, 96.1%, and 99.6%.”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
May combine two claims.
Findingown data§ 3.2 Crowdsourcing: tasks and results · p. 5Overall, the human annotators obtained an accuracy of less than 0.24 on the social spambots, with more than 1,000 False Negatives (FN), meaning that contributors classified more than 1,000 accounts as genuine, when they actually belonged to the dataset of the last generation of spambots.
“As reported in Table 6, overall, the human annotators obtained an accuracy of less than 0.24 on the social spambots, with more than 1,000 False Negatives (FN), meaning that contributors classified more than 1,000 accounts as genuine, when they actually belonged to the dataset of the last generation of spambots.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 2. DATASETS · p. 2The vast majority of social spambots #2 tweets were harmless messages, occasionally interspersed by tweets mentioning a specific genuine (human) account and suggesting him to buy the VIP version of the app from a Web store.
“The vast majority of tweets were harmless messages, occasionally interspersed by tweets mentioning a specific genuine (human) account and suggesting him to buy the VIP version of the app from a Web store.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 3.1 Twitter monitoring · p. 4Only 8.6% of traditional spambots #1 have been suspended, while 88.9% of them are still alive.
“Indeed, only 8.6% of the bots have been suspended, while 88.9% of them are still alive.”
Quote in the paperSubstantial claim- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
May combine two claims.
Findingown data§ 5. EMERGING TRENDS · p. 8Genuine accounts feature distributions of join date and number of followers that almost uniformly span across the possible range of values, whereas social spambots have anomalous distributions.
“Whereas the genuine accounts feature distributions that almost uniformly span across the possible range of values, social spambots have anomalous distributions.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5. EMERGING TRENDS · p. 7The established works benchmarked in Section 4 largely failed to detect the new wave of social spambots, with the notable exception of the unsupervised approach proposed in [2].
“As shown in Table 7, the established works benchmarked in Section 4 largely failed to detect the new wave of social spambots, one notable exception being the unsupervised approach proposed in [2].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [2] F. Ahmed and M. Abulaish. A generic statistical approach for spam detection in online social networks. Computer Communications, 36(10):1120--1129, 2013. source
Not checked in the free run.
May combine two claims.
Findingown data§ 4. ESTABLISHED TECHNIQUES · p. 7The modified implementation using fastgreedy community detection proved effective in detecting social spambots, with MCC = 0.886 for test set #1 and MCC = 0.847 for test set #2.
“As reported in Table 7, our modified implementation proved effective in detecting social spambots, with an MCC = 0.886 for test set #1 and MCC = 0.847 for test set #2 .”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 3.2 Crowdsourcing: tasks and results · p. 6Human contributors were able to correctly detect traditional spambots, as shown by the 0.91 accuracy, but contributors rarely agreed on the class.
“This interesting result shows that, overall, the human contributors were able to correctly detect traditional spambots, as shown by the 0.91 accuracy, but also that contributors rarely agreed on the class.”
Quote in the paperSubstantial claim- Rests on an assumption: The level of agreement measured by Fleiss' kappa can be interpreted as a proxy for the difficulty of a task.
Classified as probably not a substantial claim.
Study design and conductreasoning§ 3.2 Crowdsourcing: tasks and results · p. 5The authors recruited contributors who were tech-savvy and Twitter users themselves.
“We only recruited contributors who were tech-savvy and Twitter users themselves, in order to be reasonably sure about their knowledge of Twitter and its dynamics.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationcitation§ 3.1 Twitter monitoring · p. 4Twitter's spambot detection mechanisms are still unable to accurately identify traditional spambots #1, while recent solutions proposed by Academia have succeeded in this task.
“This seems to suggest that Twitter's spambot detection mechanisms are still unable to accurately identify such accounts, while recent solutions proposed by Academia have succeeded in this task [44].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [44] C. Yang, R. Harkreader, and G. Gu. Empirical evaluation and new design for fighting evolving Twitter spammers. IEEE Trans. Inform. Forens. Sec., 8(8):1280--1293, 2013. source
Not checked in the free run.
- Rests on an assumption: Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
May combine two claims.
Interpretationown data§ ABSTRACT · p. 1Neither Twitter, nor humans, nor cutting-edge applications are currently capable of accurately detecting the new social spambots.
“Results show that neither Twitter, nor humans, nor cutting-edge applications are currently capable of accurately detecting the new social spambots.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 3.2 Crowdsourcing: tasks and results · p. 6Humans generally failed in classifying social spambots (accuracy = 0.2355) and were more in agreement on this mistake than they were when correctly classifying traditional spambots.
“These results imply that humans generally failed in classifying social spambots (accuracy = 0 . 2355) and, furthermore, that they also were more in agreement on this mistake than they were when (correctly) classifying traditional spambots.”
Quote in the paperSubstantial claim- Rests on an assumption: The level of agreement measured by Fleiss' kappa can be interpreted as a proxy for the difficulty of a task.
Classified as probably not a substantial claim.
Premisecitation§ 4. ESTABLISHED TECHNIQUES · p. 6BotOrNot? is a publicly-available service to evaluate the similarity of a Twitter account with the known characteristics of social spambots.
“BotOrNot? is a publicly-available service 7 to evaluate the similarity of a Twitter account with the known characteristics of social spambots [15].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [15] C. A. Davis, O. Varol, E. Ferrara, A. Flammini, and F. Menczer. BotOrNot: A system to evaluate social bots. In WWW companion. ACM, 2016. source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 2. DATASETS · p. 3The internal manual annotation was carried out by comparing every account to all the others, in order to highlight possible similarities and common behaviors.
“Our internal manual annotation has been carried out by comparing every account to all the others, in order to highlight possible similarities and common behaviors.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 2. DATASETS · p. 2The political candidate was able to reach many more accounts in addition to his direct followers and managed to alter Twitter engagement metrics during the electoral campaign.
“Thus, the political candidate was able to reach many more accounts in addition to his direct followers and managed to alter Twitter engagement metrics during the electoral campaign.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 3.2 Crowdsourcing: tasks and results · p. 5Humans might not be able to detect social spambots in the wild, and to distinguish them from genuine accounts.
“More worryingly, they also suggest that humans might not be able to detect social spambots in the wild, and to distinguish them from genuine accounts.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Premisecitation§ ABSTRACT · p. 1Recent studies provide anecdotal argumentation of the rise of a new generation of spambots, so-called social spambots.
“Recent studies in social media spam and automation provide anecdotal argumentation of the rise of a new generation of spambots, so-called social spambots .”
- Also stated in: § 1. INTRODUCTION · p. 1 · hedged
“Recent investigations anecdotally highlight how new waves of social spambots are rising [16, 48].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [16] E. Ferrara, O. Varol, C. Davis, F. Menczer, and A. Flammini. The rise of social bots. Commun. ACM, 59(7):96--104, 2016. source
Not checked in the free run.
- cites [48] J. Zhang, R. Zhang, Y. Zhang, and G. Yan. The rise of social botnets: Attacks and countermeasures. arXiv preprint arXiv:1603.02714, 2016.
Not checked in the free run.
- Also stated in: § 1. INTRODUCTION · p. 1 · hedged
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning§ 3.2 Crowdsourcing: tasks and results · p. 5Each account was required to be classified by at least 3 different contributors, with the final class decided by majority voting.
“Furthermore, we required each account to be classified by at least 3 different contributors, with the final class decided by majority voting.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Interpretationown data§ 4. ESTABLISHED TECHNIQUES · p. 7Low values of both Precision and Recall mean incomplete and unreliable spambot detection.
“Low values of both Precision and Recall mean incomplete and unreliable spambot detection.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationcitation§ 5. EMERGING TRENDS · p. 8The technique proposed by Viswanath et al. [40] cannot directly spot the tampering accounts, so detection and removal of single accounts must be performed using a separate methodology.
“However, the technique cannot directly spot the tampering accounts, and, thus, detection and removal of the single accounts must be performed using a separate methodology.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [40] B. Viswanath, M. A. Bashir, M. B. Zafar, S. Bouget, S. Guha, K. P. Gummadi, A. Kate, and A. Mislove. Strength in numbers: Robust tamper detection in crowd computations. In COSN. ACM, 2015. source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone given§ 5. EMERGING TRENDS · p. 9A further property and advantage of the recent research wave is that it proposes ad-hoc detection algorithms rather than adopting generic and off-the-shelf machine learning algorithms.
“In addition, a further property - and advantage - of this recent research wave is that it proposes ad-hoc detection algorithms, rather than adopting generic and off-the-shelf machine learning algorithms.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
Interpretationreasoning§ 5. EMERGING TRENDS · p. 9A paradigm-shift can be observed for research and development of spambot detection systems, which may exploit the new concepts to achieve better resilience and robustness and to withstand the next evolution of social media spambots.
“We can observe a paradigm-shift for research and development of spambot detection systems, which may exploit the new concepts to achieve better resilience and robustness and to withstand the next evolution of social media spambots.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 6. CONCLUDING REMARKS · p. 9The promising outcome of the novel group-level approaches indicates that this is a favorable research avenue.
“The promising outcome of these novel approaches clearly indicates that this is a favorable research avenue.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisereasoning§ 5. EMERGING TRENDS · p. 8All emerging trend systems propose novel algorithmic solutions and investigate groups of accounts as a whole, marking a significant difference with previous literature.
“Despite being based on different key concepts, all these systems propose novel algorithmic solutions and investigate groups of accounts as a whole, marking a significant difference with the previous literature.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationown data§ 6. CONCLUDING REMARKS · p. 9The authors demonstrated the need for novel analytic tools capable of turning the tide in the arms race against sophisticated spambots.
“We demonstrated the need for novel analytic tools capable of turning the tide in the arms race against such sophisticated spambots.”
Quote in the paperSubstantial claim
The paper relies on this without support.
Twitter's API error codes 63 and 50 accurately indicate that an account has been suspended by Twitter or deleted by its original owner, respectively.
“API queries to a suspended account result in Twitter responding with the error code 63. API queries to accounts that have been deleted by their original owner result in Twitter responding with the error code 50.”
The paper relies on this without support.
The internal manual verification correctly certifies the automated nature of the accounts.
“All the accounts collected in this process have then undergone an internal manual verification phase to certify their automated nature.”
The paper relies on this without support.
The genuine accounts that answered the crowdsensing question are human-operated.
“All the replies to our questions were manually verified and all the 3,474 accounts that answered were certified as humans.”
The paper relies on this without support.
The level of agreement measured by Fleiss' kappa can be interpreted as a proxy for the difficulty of a task.
“The level of agreement can be also interpreted as a proxy for the difficulty of a task.”
The paper relies on this without support.
Social spambots have a common goal, such as increasing someone's reputation score.
“since they do have a common goal (e.g., increasing someone's reputation score).”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Supervised machine learning models for fraud detection can be evaded by malicious actors if they change their behaviors over time.
“these models can be evaded by malicious actors if they change their behaviors over time [40, 92]”
From Six Million (Suspected) Fake Stars on GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware (2026) · cites this paper as [40]
The Introduction (p. 1) says that standard classification approaches no longer succeed against a new wave of evolving social spambots. In Established Techniques (p. 6), a supervised classifier labels those spambots as genuine accounts. The evidence supports evasion in social-spambot detection, but does not establish the claim for fraud-detection models generally. (LLM, conf=0.86)
Experimental — accuracy not yet measuredAnomalous accounts show significant similarities in content-sharing behavior.
“we examined the behavioral similarities between accounts during annotation as described in the literature [12] and noticed significant similarities in the content-sharing behavior of anomalous accounts”
From Unsupervised detection of coordinated fake-follower campaigns on social media (2024) · cites this paper as [12]
Malicious actors continuously evolve their behaviors to evade platform detection.
“malicious actors continuously evolve their behaviors to evade platform detection [40, 58]”
From Six Million (Suspected) Fake Stars on GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware (2026) · cites this paper as [40]
Fake followers can be bought for as low as 12$ per 1,000 followers in the surface Web.
“Fake followers can be bought for as low as 12$ per 1,000 followers in the surface Web.”
From A decade of social bot detection (2020) · cites this paper as [5]
A third wave of bots spread through online social networks from 2016 onwards.
“Time went by and new studies acknowledged the rise of a third wave of bots that spread through online social networks from 2016 onwards [5, 13], as shown in panel C of Figure 4.”
From A decade of social bot detection (2020) · cites this paper as [5]
Botnets leave behind more traces of their automation and coordination than those left behind by sophisticated single bots.
“As such, botnets leave behind more traces of their automation and coordination than those left behind by sophisticated single bots [5].”
From A decade of social bot detection (2020) · cites this paper as [5]
Hundreds of tech-savvy social media users were able to tell apart newer bots from legitimate users only 24% of the times, but could spot older bots 91% of the times.
“Moreover, hundreds of tech-savvy social media users that participated in a crowdsourcing experiment were able to tell apart newer bots from legitimate users only 24% of the times. The same users were instead able of spotting older bots 91% of the times [5].”
From A decade of social bot detection (2020) · cites this paper as [5]
Botometer's generality and ease of deployment are counterbalanced by reduced bot detection accuracy.
“The generality and ease of deployment of this detector are however counterbalanced by a reduced bot detection accuracy [5, 17].”
From A decade of social bot detection (2020) · cites this paper as [5]
Only 5% of newer bots are removed from social platforms, whereas older ones are removed 60% of the times.
“Results showed that only 5% of newer bots are removed from social platforms, whereas older ones are removed 60% of the times [5].”
From A decade of social bot detection (2020) · cites this paper as [5]
Cresci et al. [11] report that some bot classes, like social spambots, are hard to distinguish from human accounts.
“They also report that some bot classes, like social spambots, are hard to distinguish from human accounts.”
From Detection of Novel Social Bots by Ensembles of Specialized Classifiers (2020) · cites this paper as [11]
Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.
“Cresci et al. [11] highlight that it is hard for human annotators to assign one label to one account to describe its bot class.”
From Detection of Novel Social Bots by Ensembles of Specialized Classifiers (2020) · cites this paper as [11]
Cresci et al. [11] showed that Twitter, human annotators, and state-of-the-art bot detection systems failed at discriminating between some new social spambots and genuine accounts.
“The performance decay of current detection systems in the wild was reported by Cresci et al. [11], who showed that Twitter, human annotators, and state-of-the-art bot detection systems failed at discriminating between some new social spambots and genuine accounts.”
From Detection of Novel Social Bots by Ensembles of Specialized Classifiers (2020) · cites this paper as [11]
By promoting online activity, bots can interact with humans and influence their standing against specific topics such as political issues.
“By promoting online activity, bots can interact with humans and influence their standing against specific topics such as political issues [10, 7, 12, 13].”
From Influence of augmented humans in online interactions during voting events (2019) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Social spambots #2 deletion percentage is overstated
MinorOne deleted account out of 3,457 is 0.029%, which rounds to 0.0%, not 0.1%.
Reported 0.1 · recomputed 0.028927
“| social spambots #2 | 3,457 | 3,322 (96.1%) | 1 (0.1%) | 134 (3.8%) |”
Social spambots #2 suspension percentage is understated
MinorThe 134 suspended accounts out of 3,457 constitute 3.876%, which rounds to 3.9%, not 3.8%.
Reported 3.8 · recomputed 3.876193
“| social spambots #2 | 3,457 | 3,322 (96.1%) | 1 (0.1%) | 134 (3.8%) |”
Social-spambot account totals differ between tables
MinorTable 1 lists 991 accounts for social spambots #1 and 464 for #3, while Table 2 lists 994 and 467, respectively. The paper does not explain the additional three accounts in each group.
“| social spambots #1 | retweeters of an Italian political candidate | 991 | 1,610,176 | 2012 | 3.1, 3.2 |”
“| social spambots #1 | 994 | 946 (95.2%) | 2 (0.2%) | 46 (4.6%) |”
Traditional spambots #1 suspension difference does not match Table 2
MinorTable 2 reports suspension rates of 8.6% for traditional spambots #1 and 0.1% for genuine accounts. Their difference is 8.5 percentage points, rather than the 8.7 reported in Table 3.
Reported 8.7 · recomputed 8.5
“| traditional spambots #1 | - 7.6% *** | - 0.8% | +8.7% *** |”
Genuine-account suspension percentage is understated
MinorSix suspended accounts out of 3,474 are 0.173%, which rounds to 0.2%, not the reported 0.1%.
Reported 0.1 · recomputed 0.172712
“| genuine accounts | 3,474 | 3,353 (96.5%) | 115 (3.3%) | 6 (0.1%) |”
Cites
Finding community structure in very large networks
Clauset, A., Newman, M. E. J., & Moore, C. (2004) · Physical Review E · doi:10.1103/physreve.70.066111 · Cited by 7,390 · see citing works
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973 · see citing works
Gwet, K. L. (2014) · Medical Entomology and Zoology · Cited by 1,095 · see citing works
Density-Based Clustering over an Evolving Data Stream with Noise
Cao, F., Estert, M., Qian, W., & Zhou, A. (2006) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972764.29 · Cited by 1,033 · see citing works
BotOrNot: A System to Evaluate Social Bots
Davis, C. A., Varol, O., Ferrara, E., Flammini, A., & Menczer, F. (2016) · Proceedings of the 25th International Conference Companion on World Wide Web - WWW 16 Companion · doi:10.1145/2872518.2889302 · Cited by 909 · see citing works
Detecting spammers on social networks
Stringhini, G., Kruegel, C., & Vigna, G. (2010) · Proceedings of the 26th Annual Computer Security Applications Conference · doi:10.1145/1920261.1920263 · Cited by 791 · see citing works
Graph Clustering Via a Discrete Uncoupling Process
van Dongen, S. (2008) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/040608635 · Cited by 726 · see citing works
Magno, G., Rodrigues, T., & Almeida, V. A. F. (2010) · Cited by 649 · see citing works
Uncovering social spammers: social honeypots + machine learning
Lee, K., Caverlee, J., & Webb, S. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835522 · Cited by 619 · see citing works
Design and Evaluation of a Real-Time URL Spam Filtering Service
Thomas, K., Grier, C., Ma, J. H., Paxson, V., & Song, D. X. (2011) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2011.25 · Cited by 470 · see citing works
Fame for sale: Efficient detection of fake Twitter followers
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2015) · Decision Support Systems · doi:10.1016/j.dss.2015.09.003 · Cited by 461 · cited 1× · see citing works
“Fake followers are another kind of malicious accounts that recently gained interest both from platform administrators and from the scientific world [13].”
BackgroundSelf-citationAiding the detection of fake accounts in large scale social online services
Cao, Q., Sirivianos, M., Yang, X., & Pregueiro, T. (2012) · Cited by 449 · see citing works
Seven Months with the Devils: A Long-Term Study of Content Polluters on Twitter
Lee, K., Eoff, B. D., & Caverlee, J. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14106 · Cited by 448 · see citing works
Detecting spam in a Twitter network
Schoenebeck, S. Y., Romero, D. M., Schoenebeck, G. R., & boyd, D. (2009) · First Monday · doi:10.5210/fm.v15i1.2793 · Cited by 347 · see citing works
Empirical evaluation and new design for fighting evolving Twitter spammers
Yang, C., Harkreader, R., & Gu, G. (2013) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2013.2267732 · Cited by 346 · see citing works
Understanding and combating link farming in the twitter social network
Ghosh, S., Viswanath, B., Kooti, F., Sharma, N. K., Korlam, G., Benevenuto, F., Ganguly, N., & Gummadi, K. P. (2012) · International World Wide Web Conference (WWW) · doi:10.1145/2187836.2187846 · Cited by 341 · see citing works
CopyCatch: stopping group attacks by spotting lockstep behavior in social networks
Beutel, A., Xu, W., Guruswami, V., Palow, C., & Faloutsos, C. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488400 · Cited by 335 · see citing works
Twitter spammer detection using data stream clustering
Miller, Z., Dickinson, B., Deitrick, W., Hu, W., & Wang, A. H. (2014) · Information Sciences · doi:10.1016/j.ins.2013.11.016 · Cited by 332 · see citing works
Uncovering social network Sybils in the wild
Yang, Z., Wilson, C., Wang, X., Gao, T., Zhao, B. Y., & Dai, Y. (2014) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2556609 · Cited by 290 · see citing works
Uncovering Large Groups of Active Malicious Accounts in Online Social Networks
Cao, Q., Yang, X., Yu, J., & Palow, C. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660269 · Cited by 274 · see citing works
Trafficking fraudulent accounts: the role of the underground market in Twitter spam and abuse
Thomas, K., McCoy, D., Grier, C., Kolcz, A., & Paxson, V. (2013) · Cited by 239 · see citing works
Detecting spammers and content promoters in online video social networks
Benevenuto, F., Rodrigues, T., Almeida, V., Almeida, J., & Gonçalves, M. (2009) · Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1571941.1572047 · Cited by 220 · see citing works
Stein, T., Chen, E., & Mangla, K. (2011) · Proceedings of the 4th Workshop on Social Network Systems · doi:10.1145/1989656.1989664 · Cited by 219 · see citing works
WarningBird: A Near Real-Time Detection System for Suspicious URLs in Twitter Stream
Lee, S., & Kim, J. K. (2013) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2013.3 · Cited by 200 · see citing works
A generic statistical approach for spam detection in Online Social Networks
Ahmed, F., & Abulaish, M. (2013) · Computer Communications · doi:10.1016/j.comcom.2013.04.004 · Cited by 199 · see citing works
Towards Online Spam Filtering in Social Networks
Gao, H., Chen, Y., Lee, K., Palsetia, D., & Choudhary, A. (2012) · Cited by 199 · see citing works
DNA-Inspired Online Behavioral Modeling and Its Application to Spambot Detection
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.29 · Cited by 194 · see citing works
Self-citationSocial Turing Tests: Crowdsourcing Sybil DetectionPreprint
Wang, G., Mohanlal, M., Wilson, C., Wang, X., Metzger, M. J., Zheng, H.-T., & Zhao, B. Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.3856 · Cited by 173 · see citing works
Stringhini, G., Wang, G., Egele, M., Kruegel, C., Vigna, G., Zheng, H., & Zhao, B. Y. (2013) · Proceedings of the 2013 conference on Internet measurement conference · doi:10.1145/2504730.2504731 · Cited by 169 · cited 1× · see citing works
“Recent investigations anecdotally highlight how new waves of social spambots are rising [16, 48].”
BackgroundAn analysis of social network-based Sybil defenses
Viswanath, B., Post, A., Gummadi, K. P., & Mislove, A. E. (2010) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1851275.1851226 · Cited by 161 · see citing works
Online Social Spammer Detection
Hu, X., Tang, J., & Liu, H. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8728 · Cited by 133 · see citing works
Suspicious Behavior Detection: Current Trends and Future Directions
Jiang, M., Cui, P., & Faloutsos, C. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.5 · Cited by 131 · see citing works
Yu, R., He, X., & Liu, Y. (2015) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2811268 · Cited by 99 · see citing works
The Rise of Social Botnets: Attacks and Countermeasures
Zhang, J., Zhang, R., Zhang, Y., & Yan, G. (2016) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2016.2641441 · Cited by 96 · see citing works
StreamKM++: A Clustering Algorithm for Data Streams
Ackermann, M. R., Raupach, C., Lammersen, C., et al. (2010) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611972900.16 · Cited by 90 · see citing works
Early filtering of ephemeral malicious accounts on Twitter
Lee, S., & Kim, J. K. (2014) · Computer Communications · doi:10.1016/j.comcom.2014.08.006 · Cited by 75 · see citing works
Stringhini, G., Egele, M., Kruegel, C., & Vigna, G. (2012) · doi:10.1145/2342549.2342551 · Cited by 74 · see citing works
Detecting Spammers and Content Promoters in Online Video Social Networks
Benevenuto, F., Rodrigues, T., Almeida, J. M., et al. (2009) · IEEE INFOCOM Workshops · doi:10.1109/infcomw.2009.5072127 · Cited by 63 · see citing works
Strength in Numbers: Robust Tamper Detection in Crowd Computations
Viswanath, B., Bashir, M. A., Zafar, M. B., Bouget, S., Guha, S., Gummadi, K. P., Kate, A., & Mislove, A. (2015) · Proceedings of the 2015 ACM on Conference on Online Social Networks · doi:10.1145/2817946.2817964 · Cited by 43 · see citing works
Inferring lockstep behavior from connectivity pattern in large graphs
Jiang, M., Cui, P., Beutel, A., Faloutsos, C., & Yang, S. (2016) · Knowledge and Information Systems · doi:10.1007/s10115-015-0883-y · Cited by 43 · see citing works
Avvenuti, M., Bellomo, S., Cresci, S., Polla, M. N. L., & Tesconi, M. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3051155 · Cited by 42 · see citing works
Self-citationAnti-Reconnaissance Tools: Detecting Targeted Socialbots
Paradise, A., Puzis, R., & Shabtai, A. (2014) · IEEE Internet Computing · doi:10.1109/mic.2014.81 · Cited by 40 · see citing works
Xie, Y., Yu, F., Ke, Q., Abadi, M., Gillum, E., Vitaldevaria, K., Walter, J., Huang, J., & Mao, Z. M. (2012) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2382196.2382235 · Cited by 38 · see citing works
ND-Sync: Detecting Synchronized Fraud Activities
Γιάτσογλου, Μ., Chatzakou, D., Shah, N., Beutel, A., Faloutsos, C., & Vakali, A. I. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-18032-8_16 · Cited by 36 · see citing works
Uncovering deception in social media
Liu, H., Han, J., & Motoda, H. (2014) · Social Network Analysis and Mining · doi:10.1007/s13278-014-0162-z · Cited by 21 · see citing works
A Criticism to Society (As Seen by Twitter Analytics)
Cresci, S., Petrocchi, M., Spognardi, A., Tesconi, M., & Pietro, R. D. (2014) · 2014 IEEE 34th International Conference on Distributed Computing Systems Workshops · doi:10.1109/icdcsw.2014.31 · Cited by 20 · see citing works
Self-citationSDHM: A hybrid model for spammer detection in Weibo
Liu, Y., Wu, B., Wang, B., & Li, G. (2014) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2014.6921699 · Cited by 20 · see citing works
The Rise of Social Botnets: Attacks and CountermeasuresPreprint
Zhang, J., Zhang, R., Zhang, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.02714 · Cited by 4 · see citing works
Flagged first, then by citations
2 of the 50 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- M. Jiang, P. Cui, A. Beutel, C. Faloutsos, and S. Yang. Catching synchronized behaviors in large networks: A graph mining approach. ACM Trans. Knowl. Discov. Data, 10(4):35:1--35:27, 2016.
Cited by
Sorted by citations
Tucker, J. A., Guess, A., Barbera, P., Vaccari, C., Siegel, A., Sanovich, S., Stukal, D., & Nyhan, B. (2018) · doi:10.2139/ssrn.3144139 · Cited by 1,222 · see citing works
Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., Quinn, S. C., & Dredze, M. (2018) · American Journal of Public Health · doi:10.2105/ajph.2018.304567 · Cited by 1,149 · see citing works
Arming the public with artificial intelligence to counter social bots
Yang, K., Varol, O., Davis, C. A., Ferrara, E., Flammini, A., & Menczer, F. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.115 · Cited by 419 · see citing works
“While supervised methods have proven to be effective in many cases, they do not perform well at detecting coordinated social bots that post human-generated content (Cresci et al., 2017; Grimme et al., 2018; Chen and Subramanian, 2018).”
BackgroundSocial Media, Echo Chambers, and Political Polarization
Barberá, P. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.004 · Cited by 373 · see citing works
Measuring objective and subjective well-being: dimensions and data sources
Voukelatou, V., Gabrielli, L., Miliou, I., Cresci, S., Sharma, R. R., Tesconi, M., & Pappalardo, L. (2020) · International Journal of Data Science and Analytics · doi:10.1007/s41060-020-00224-2 · Cited by 392 · see citing works
Self-citationHarnessing Artificial Intelligence Capabilities to Improve Cybersecurity
Zeadally, S., Adi, E., Baig, Z. A., & Khan, I. A. (2020) · IEEE Access · doi:10.1109/access.2020.2968045 · Cited by 334 · see citing works
“In contrast, malicious bots also emerged to cheat in online games [49], post spams [50], [51] and spread malware [52].”
Facts and Where to Find Them: Empirical Research on Internet Platforms and Content Moderation
Keller, D., & Leerssen, P. (2020) · Data Archiving and Networked Services (DANS) · doi:10.1017/9781108890960
A decade of social bot detection
Cresci, S. (2020) · Communications of the ACM · doi:10.1145/3409116 · Cited by 323 · see citing works
“th the recent rise of the adversarial ones. they share common goals. As such, botnets leave behind more traces of their automation and coordination than those left behind by sophisticated single bots [5]. Devising techniques for spotting suspiciously coordinated and synchronized behaviors is thus likely to yield bette…”
BackgroundInfluentialSelf-citationMisinformation, Disinformation, and Online Propaganda
Guess, A. M., & Lyons, B. A. (2020) · Social Media and Democracy · doi:10.1017/9781108890960.003 · Cited by 284 · see citing works
Bots and Misinformation Spread on Social Media: Implications for COVID-19
Himelein-Wachowiak, M., Giorgi, S., Devoto, A., Rahman, M. M., Ungar, L., Schwartz, H. A., Epstein, D. H., Leggio, L., & Curtis, B. L. (2021) · Journal of Medical Internet Research · doi:10.2196/26933 · Cited by 238 · see citing works
“In light of findings that humans are largely unable to distinguish social bots from genuine (human) accounts [33], it is likely that humans unknowingly contribute to the spread of misinformation as well.”
InfluentialTwiBot-20: A Comprehensive Twitter Bot Detection Benchmark
Feng, S., Wan, H., Wang, N., Li, J., & Luo, M. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482019 · Cited by 199 · see citing works
“Regarding dataset size, the number of users included in each dataset also range from 693 users in cresci-rtbust [19] to 50,538 in midterm-18 [32], and one of the most widely used dataset cresci-17 [6] contains 2,764 human users and 7,049 bots.”
BackgroundMethodInfluential(2020) · Sabanci University · Cited by 197 · see citing works
TweepFake: about detecting deepfake tweets
Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251415 · Cited by 185 · see citing works
“[19] proved that a human is not reliable on this labeling task.”
BackgroundSelf-citationThe State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
Chatzimparmpas, A., Martins, R. M., Jusufi, I., Kucher, K., Fabrice Rossi, A., & Kerren, A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14034 · Cited by 184 · see citing works
“Squares [RAL ∗ 17] is a performance visualization for multi-class classification problems.”
BackgroundMethodInfluentialThe False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184 · see citing works
“At least researchers should be aware of the data used as training data for the classifier that might only cover certain types of bots [15, 16, 18].”
BackgroundMethodResultInfluentialA survey on deep learning for cybersecurity: Progress, challenges, and opportunities
Macas, M., Wu, C., & Fuertes, W. (2022) · Computer Networks · doi:10.1016/j.comnet.2022.109032 · Cited by 192 · see citing works
RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter
Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., & Tesconi, M. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326015 · Cited by 163 · see citing works
Self-citationTwitter Bot Detection Using Bidirectional Long Short-Term Memory Neural Networks and Word Embeddings
Wei, F., & Nguyen, U. T. (2019) · 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa48467.2019.00021 · Cited by 168 · see citing works
“We prepared two test sets following [56].”
Detection of Novel Social Bots by Ensembles of Specialized Classifiers
Sayyadiharikandeh, M., Varol, O., Yang, K.-C., Flammini, A., & Menczer, F. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3412698 · Cited by 159 · see citing works
Botometer 101: social bot practicum for computational social scientists
Yang, K.-C., Ferrara, E., & Menczer, F. (2022) · Journal of Computational Social Science · doi:10.1007/s42001-022-00177-5 · Cited by 159 · see citing works
“These accounts might come from a dif‐ ferent context, use different languages other than English [52, 53], or show novel behavioral patterns [34, 45, 54].”
BackgroundDetecting Bots on Russian Political Twitter
Stukal, D., Sanovich, S., Bonneau, R., & Tucker, J. A. (2017) · Big Data · doi:10.1089/big.2017.0038 · Cited by 140 · see citing works
“26 However, it was only in recent years that both the scope 27 and sophistication 28 of bot activity on Twitter increased to such an extent that it began to draw serious attention from political actors, including state actors.”
A Hybrid Approach for Detecting Automated Spammers in Twitter
Fazil, M., & Abulaish, M. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2825958 · Cited by 137 · see citing works
“The experimental proof of the existence of social spambots and the challenges arising due to their presence are discussed in [24].”
BackgroundCharting the Landscape of Online Cryptocurrency Manipulation
Nizzoli, L., Tardelli, S., Avvenuti, M., Cresci, S., Tesconi, M., & Ferrara, E. (2020) · IEEE Access · doi:10.1109/access.2020.3003370 · Cited by 132 · see citing works
Self-citationYuan, X., Schuchard, R., & Crooks, A. T. (2019) · Social Media + Society · doi:10.1177/2056305119865465 · Cited by 143 · see citing works
“This is not surprising as the emerging field of social bot research is still working toward establishing norms (Cresci, Di Pietro, Petrocchi, Spognardi, & Tesconi, 2017; Ferrara et al., 2016) and a multitude of different methodologies must be employed to account for the vast array of different…”
BackgroundDetect Me If You Can: Spam Bot Detection Using Inductive Representation Learning
Alhosseini, S. A., Tareaf, R. B., Najafi, P., & Meinel, C. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3316504 · Cited by 133 · see citing works
Of Bots and Humans (on Twitter)
Gilani, Z., Farahbakhsh, R., Gareth Tyson, A., et al. (2017) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1145/3110025.3110090 · Cited by 131 · see citing works
The Future of False Information Detection on Social Media: New Perspectives and Trends
Guo, B., Ding, Y., Yao, L., Liang, Y., & Yu, Z. (2021) · ACM Computing Surveys · doi:10.1145/3393880 · Cited by 122 · see citing works
“Cresci et al. [26] also leverage characteristics of group accounts to detect malicious bots.”
BackgroundMachine learning-based social media bot detection: a comprehensive literature review
Aljabri, M., Zagrouba, R., Shaahid, A., Alnasser, F., Saleh, A., & Alomari, D. M. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-022-01020-5 · Cited by 132 · see citing works
“The authors recommend considering the three types of bot class as in (Cresci et al. 2017) dataset.”
InfluentialCashtag Piggybacking: Uncovering Spam and Bot Activity in Stock Microblogs on Twitter
Cresci, S., Lillo, F., Regoli, D., Tardelli, S., & Tesconi, M. (2019) · ACM Transactions on the Web · doi:10.1145/3313184 · Cited by 105 · see citing works
Self-citationWho Falls for Online Political Manipulation?
Badawy, A., Lerman, K., & Ferrara, E. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3316494 · Cited by 105 · see citing works
“…Kollanyi et al. 2016; Marwick and Lewis 2017; Shorey and Howard 2016; Wang et al. 2016; Woolley 2016; Woolley and Howard 2016] and all over the world, including in South America [Forelle et al. 2015; Suárez-Serrato et al. 2016], the U.K. [Howard and Kollanyi 2016], and Italy [Cresci et al. 2017].”
BackgroundThe role of bot squads in the political propaganda on Twitter
Caldarelli, G., Nicola, R. D., Vigna, F. D., Petrocchi, M., & Saracco, F. (2020) · Communications Physics · doi:10.1038/s42005-020-0340-4 · Cited by 135 · see citing works
“, [12]: exactly what is revealed by the entropy-based filtering.”
BackgroundSelf-citationSiegel, A. A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.005 · Cited by 117 · see citing works
Demystifying Social Bots: On the Intelligence of Automated Social Media Actors
Assenmacher, D., Clever, L., Frischlich, L., Quandt, T., Trautmann, H., & Grimme, C. (2020) · Social Media + Society · doi:10.1177/2056305120939264 · Cited by 100 · see citing works
Heidari, M., Jones, J. H., & Uzuner, Ö. (2020) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw51313.2020.00071 · Cited by 99 · see citing works
“The data set for this research is the Cresci 2017 data set [1], which is a labeled data set of different types of bots and genuine (human) users.”
BackgroundMethodInfluentialDisinformation and social bot operations in the run up to the 2017 French presidential election
Ferrara, E. (2017) · First Monday · doi:10.5210/fm.v22i8.8005 · Cited by 524 · see citing works
Twitter spam account detection based on clustering and classification methods
Adewole, K. S., Han, T., Wu, W., Song, H. H., & Sangaiah, A. K. (2018) · The Journal of Supercomputing · doi:10.1007/s11227-018-2641-x · Cited by 95 · see citing works
DeeProBot: a hybrid deep neural network model for social bot detection based on user profile data
Hayawi, K., Mathew, S. S., Venugopal, N., Masud, M. M., & Ho, P. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00869-w · Cited by 93 · see citing works
Ruffo, G., Semeraro, A., Giachanou, A., & Rosso, P. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100531 · Cited by 110 · see citing works
Bot, or not? Comparing three methods for detecting social bots in five political discourses
Martini, F., Samula, P., Keller, T. R., & Klinger, U. (2021) · Big Data & Society · doi:10.1177/20539517211033566 · Cited by 91 · see citing works
“Studies have already shown that human coders are by no means better able to identify automated accounts (Cresci et al., 2017a, 2017b), at least in manual content analysis, while sophisticated digital forensics is a different story.”
BackgroundFeng, S., Wan, H., Wang, N., Li, J., & Luo, M. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481949 · Cited by 96 · see citing works
“• cresci-17 [5] is a public dataset with 4 components: genuine accounts, social spambots, traditional spambots and fake followers.”
Leveraging Analysis of User Behavior to Identify Malicious Activities in Large-Scale Social Networks
Al‐Qurishi, M., Hossain, M. S., Alrubaian, M., Rahman, S. M. M., & Alamri, A. M. (2017) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2017.2753202 · Cited by 83 · see citing works
Contrast Pattern-Based Classification for Bot Detection on Twitter
Loyola‐González, O., Monroy, R., Rodríguez‐Ruiz, J., López‐Cuevas, A., & Mata-Sánchez, J. I. (2019) · IEEE Access · doi:10.1109/access.2019.2904220 · Cited by 80 · see citing works
“Account Information: an account information feature is extracted from the Twitter account that has been allegedly used to post a message [14].”
BackgroundMethodInfluentialDetecting malicious activity in Twitter using deep learning techniques
Ilias, L., & Roussaki, I. G. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107360 · Cited by 81 · see citing works
DeepSBD: A Deep Neural Network Model With Attention Mechanism for SocialBot Detection
Fazil, M., Sah, A. K., & Abulaish, M. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3102498 · Cited by 86 · see citing works
A Deep Learning Approach for Robust Detection of Bots in Twitter Using Transformers
Martin-Gutierrez, D., Hernández-Peñaloza, G., Belmonte-Hernández, A., Lozano-Díez, A., & Álvarez, F. (2021) · IEEE Access · doi:10.1109/access.2021.3068659 · Cited by 79 · see citing works
“In [11], authors annotated more than 8000 accounts and proposed a classifier which achieved a considerable level of accuracy for such set of samples.”
BackgroundDetecting computer-generated disinformation
Stiff, H., & Johansson, F. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00299-5 · Cited by 79 · see citing works
“Early machine learning-based approaches to detecting automation of social media accounts were often based on relatively simple measurements related to posting behavior, posted content, and account properties of individual accounts [6], but such approaches are not working as well today, due to newer generations of bots…”
BackgroundLingam, G., Rout, R. R., & Somayajulu, D. V. L. N. (2019) · Applied Intelligence · doi:10.1007/s10489-019-01488-3 · Cited by 73 · see citing works
Unsupervised Social Bot Detection via Structural Information Theory
Peng, H. M., Zhang, J., Huang, X., Hao, Z., Li, A., Yu, Z., & Yu, P. S. (2024) · ACM Transactions on Information Systems · doi:10.1145/3660522 · Cited by 82 · see citing works
An Empirical Study of Machine learning Algorithms for Social Media Bot Detection
Heidari, M., Jones, J. H., & Uzuner, Ö. (2021) · IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS) · doi:10.1109/iemtronics52119.2021.9422605 · Cited by 76 · see citing works
“In the study conducted by Cresci et al. [13], the authors identified multiple types of spam bots, including promoter bots, URL spam bots, and fake followers.”
BackgroundMethodInfluentialAre You a Cyborg, Bot or Human?—A Survey on Detecting Fake News Spreaders
Shahid, W., Li, Y., Staples, D., Gilkar, G. A., Hakak, S., & Ghorbani, A. A. (2022) · IEEE Access · doi:10.1109/access.2022.3157724 · Cited by 77 · see citing works
“The automated nature of bots makes it easy to achieve a large scale impact when spreading misinformation [15].”
BackgroundAmplifying influence through coordinated behaviour in social networks
Weber, D., & Neumann, F. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00815-2 · Cited by 72 · see citing works
“This is especially difficult given inau-thentic behaviour is hard for humans to judge by eye (Cresci et al. 2017; Benkler et al. 2018; Jamieson 2020).”
BackgroundDetection of Malicious Social Bots Using Learning Automata With URL Features in Twitter Network
Rout, R. R., Lingam, G., & Somayajulu, D. V. L. N. (2020) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2020.2992223 · Cited by 64 · see citing works
“In this section, the performance of our proposed LA-MSBD algorithm is evaluated by considering two Twitter data sets, namely, Social Honeypot 1 data set [22] and The Fake Project 2 data set [21].”
BackgroundMethodInfluentialThe limited reach of fake news on Twitter during 2019 European elections
Cinelli, M., Cresci, S., Galeazzi, A., Quattrociocchi, W., & Tesconi, M. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234689 · Cited by 64 · see citing works
Self-citationA Large-scale Behavioural Analysis of Bots and Humans on Twitter
Gilani, Z., Farahbakhsh, R., Gareth Tyson, A., & Crowcroft, J. (2019) · ACM Transactions on the Web · doi:10.1145/3298789 · Cited by 62 · see citing works
“tested known bots on Twitter to see if they are perceived differently than humans [14].”
BackgroundLOBO: Evaluation of Generalization Deficiencies in Twitter Bot Classifiers
Echeverra, J., De Cristofaro, E., Kourtellis, N., Leontiadis, I., Stringhini, G., & Zhou, S. (2018) · Proceedings of the 34th Annual Computer Security Applications Conference · doi:10.1145/3274694.3274738 · Cited by 61 · see citing works
Deep Learning Based Social Bot Detection on Twitter
Arın, E., & Kutlu, M. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3254429 · Cited by 65 · see citing works
Misinformation and Its Correction
Wittenberg, C. E., & Berinsky, A. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.009 · Cited by 63 · see citing works
BERT Model for Fake News Detection Based on Social Bot Activities in the COVID-19 Pandemic
Heidari, M., Zad, S., Hajibabaee, P., Malekzadeh, M., HekmatiAthar, S., Uzuner, Ö., & Jones, J. H. (2021) · IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) · doi:10.1109/uemcon53757.2021.9666618 · Cited by 57 · see citing works
“Cresci authors bought Fake followers of Twitter accounts from different websites [46].”
BackgroundMethodCognitive Network Science for Understanding Online Social Cognitions: A Brief Review
Stella, M. (2021) · Topics in Cognitive Science · doi:10.1111/tops.12551 · Cited by 53 · see citing works
“…of thinking about specific news, with relevance for understanding about how social bots (Ferrara et al., 2016, Stella et al., 2018), fake accounts (Cresci et al., 2017) and trolls (Monakhov, 2020) influenced other users; (v) profile online users in terms of their digital footprint based on the…”
BackgroundBotMoE: Twitter Bot Detection with Community-Aware Mixtures of Modal-Specific Experts
Liu, Y., Tan, Z., Wang, H., Feng, S., Zheng, Q., & Luo, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591646 · Cited by 61 · see citing works
“However, in part due to the awareness of these feature-based detectors, bot operators began to tamper with user metadata [6, 8], leaving featurebased approaches less effective.”
BackgroundBetter Safe Than Sorry: An Adversarial Approach to Improve Social Bot Detection
Cresci, S., Petrocchi, M., Spognardi, A., & Tognazzi, S. (2019) · Proceedings of the 10th ACM Conference on Web Science · doi:10.1145/3292522.3326030 · Cited by 52 · see citing works
“g- [5, 8, 11, 13, 22, 48], and networkcharacteristics [5, 31, 47, 48, 51–53] of the accounts. However, until now, new detection systems have been developed only as a consequence of spambot evolutions [12]. In fact, no work has ever been done towards studying, and possibly anticipating, such evolutions. In other words,…”
BackgroundSelf-citationBots, Elections, and Social Media: A Brief Overview
Ferrara, E. (2020) · Lecture notes in social networks · doi:10.1007/978-3-030-42699-6_6 · Cited by 60 · see citing works
“Other recent surveys propose complementary or alternative taxonomies that are worth considering as well [71,20,20,86].”
BackgroundСтукал, Д., Sanovich, S., Tucker, J. A., & Bonneau, R. (2019) · SAGE Open · doi:10.1177/2158244019827715 · Cited by 55 · see citing works
Influence of augmented humans in online interactions during voting events
Stella, M., Cristoforetti, M., & De Domenico, M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0214210 · Cited by 51 · see citing works
“By promoting online activity, bots can interact with humans and influence their standing against specific topics such as political issues [7, 10, 12, 13].”
BackgroundGANBOT: a GAN-based framework for social bot detection
Najari, S., Salehi, M., & Farahbakhsh, R. (2021) · Social Network Analysis and Mining · doi:10.1007/s13278-021-00800-9 · Cited by 50 · see citing works
“They can influence and bias users’ opinion via spreading malicious manipulated contents (Ferrara et al. 2016; Halawa et al. 2019) such as spam or fake review contents (Cresci et al. 2017; Aghakhani et al. 2018) on social media.”
BackgroundMethodBot detection using a single post on social media
Mohammad, S., Khan, M. U. S., Ali, M., Liu, L., Shardlow, M., & Nawaz, R. (2019) · World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4) · doi:10.1109/worlds4.2019.8903989 · Cited by 48 · see citing works
“We have used a publicly available dataset [11] for our experiments.”
MethodMeasuring Bot and Human Behavioral Dynamics
Pozzana, I., & Ferrara, E. (2020) · Frontiers in Physics · doi:10.3389/fphy.2020.00125 · Cited by 49 · see citing works
Ephemeral Astroturfing Attacks: The Case of Fake Twitter Trends
Elmas, T., Overdorf, R., Özkalay, A. F., & Aberer, K. (2021) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp51992.2021.00035 · Cited by 50 · see citing works
Facts and Where to Find Them: Empirical Research on Internet Platforms and Content Moderation
Keller, D., & Leerssen, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.011 · Cited by 37 · see citing works
Systematic Literature Review of Social Media Bots Detection Systems
Ellaky, Z., Benabbou, F., & Ouahabi, S. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.04.004 · Cited by 38 · see citing works
“They used four known datasets ICWSM17 (Varol et al., 2017), RAID11 (Liu et al., 2017), WWW17 (Yang et al., 2013), and Cresci-17 (Cresci et al., 2017).”
InfluentialDemocratic Transparency in the Platform Society
Gorwa, R., & Ash, T. G. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.013 · Cited by 67 · see citing works
Bots and Computational Propaganda: Automation for Communication and Control
Woolley, S. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.006 · Cited by 39 · see citing works
“Spambots, conversely, are built to generate noise (Cresci et al. 2017 ).”
BackgroundInfluentialA Survey of Attacks Against Twitter Spam Detectors in an Adversarial Environment
Imam, N. H., & Vassilakis, V. G. (2019) · Robotics · doi:10.3390/robotics8030050 · Cited by 36 · see citing works
“A recent study noted that the increase in OSN spammers, who distribute unsolicited spam and advertise untrustworthy products, affects the public’s perception of companies, which can eventually lead to people developing biased opinions [16].”
BackgroundBots in Social and Interaction Networks
Mendoza, M., Tesconi, M., & Cresci, S. (2020) · ACM Transactions on Information Systems · doi:10.1145/3419369 · Cited by 36 · see citing works
“By taking into account patterns of coordination, this annotation process offers higher guarantees with regards to the correct labeling of sophisticated bots that, if analyzed individually, are likely to be mistaken for legitimate accounts [13].”
BackgroundMethodResultInfluentialSelf-citationAn unsupervised method for social network spammer detection based on user information interests
Koggalahewa, D. N., Xu, Y., & Foo, E. (2022) · Journal Of Big Data · doi:10.1186/s40537-021-00552-5 · Cited by 36 · see citing works
“The Social Honeypot [18], HSpam14 [19] and The Fake Project [20] datasets are used to evaluate our approach.”
BackgroundMethodAn Attention-Based Graph Neural Network for Spam Bot Detection in Social Networks
Zhao, C., Xin, Y., Li, X., Zhu, H., Yang, Y., & Chen, Y. (2020) · Applied Sciences · doi:10.3390/app10228160 · Cited by 36 · see citing works
“The behavioral features of a single user may seem legitimate, but when multiple such users appear as a group, they will show anomaly features [8].”
BackgroundBots, disinformation, and the first impeachment of U.S. President Donald Trump
Rossetti, M., & Zaman, T. R. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283971 · Cited by 34 · see citing works
“For this reason, other methods have been developed which focus on analyzing collective behaviors and similarities among groups of accounts [23, 29, 30, 33, 34], or including such analysis in a hybrid “tiered” approach [35].”
BackgroundMethodBot stamina: examining the influence and staying power of bots in online social networks
Schuchard, R., Crooks, A. T., Stefanidis, A., & Croitoru, A. (2019) · Applied Network Science · doi:10.1007/s41109-019-0164-x · Cited by 36 · see citing works
“Relying upon a single bot detection platform helped validate this study’s applied network analysis methods, but a sole source detection algorithm is not sufficient for overcoming known specific limitations that currently challenge all open-source bot detection results (Subrahmanian et al. 2016; Cresci et al. 2017).”
BackgroundMethodInfluentialLingam, G., Rout, R. R., Somayajulu, D. V. L. N., & Ghosh, S. K. (2020) · IEEE Systems Journal · doi:10.1109/jsyst.2020.3034416 · Cited by 33 · see citing works
“…Q -value function; 3) to develop a SIU-ICD algorithm to minimize the disseminating of spam-content through influential communities in Twitter network; 4) experimentation using Social Honeypot dataset [16] and The Fake Project dataset [17] demonstrates the efficacy of our proposed algorithms.”
MethodGrossi, V., Rapisarda, B., Giannotti, F., et al. (2018) · International Journal of Data Science and Analytics · doi:10.1007/s41060-018-0126-x · Cited by 32 · see citing works
“The ability of combiningdata sensedboth byopportunistic and participatory methods is expected to open new avenues for learning systems, capable of recognizing and mapping human behaviour at global and local scale [15,16].”
BackgroundDemocratic Creative Destruction? The Effect of a Changing Media Landscape on Democracy
Nielsen, R. K., & Fletcher, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.008 · Cited by 35 · see citing works
Lingam, G., Rout, R. R., Somayajulu, D. V. L. N., & Das, S. K. (2020) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3320269.3384770 · Cited by 32 · see citing works
Smart, B., Watt, J., Benedetti, S., Mitchell, L., & Roughan, M. (2022) · Social Informatics · doi:10.1007/978-3-031-19097-1_3 · Cited by 24 · see citing works
“[9] measured Twitter’s current capabilities of detecting social spambots.”
Real-time Detection of Content Polluters in Partially Observable Twitter Networks
Nasim, M., Nguyen, A., Lothian, N., Cope, R. C., & Mitchell, L. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3184558.3191574 · Cited by 32 · see citing works
“The importance of detecting bots in online social media has produced an active research area on this topic [9, 21].”
BackgroundDon’t @ Me: Experimentally Reducing Partisan Incivility on Twitter
Munger, K. (2021) · Journal of Experimental Political Science · doi:10.1017/xps.2020.14 · Cited by 31 · see citing works
“Increasingly sophisticated “social spam bots” have been developed that are difficult for human coders to detect (Cresci et al., 2017).”
BackgroundShehnepoor, S., Togneri, R. B., Liu, W., & Bennamoun, M. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3123876 · Cited by 34 · see citing works
Screening Out Social Bots Interference: Are There Any Silver Bullets?
Zago, M., Nespoli, P., Papamartzivanos, D., Perez, M. G., Marmol, F. G., Kambourakis, G., & Perez, G. M. (2019) · IEEE Communications Magazine · doi:10.1109/mcom.2019.1800520 · Cited by 30 · see citing works
Emotional Bots: Content-based Spammer Detection on Social Media
Andriotis, P., & Takasu, A. (2018) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs.2018.8630760 · Cited by 33 · see citing works
“The final dataset ( WWW17 ) [7] consists of various accounts (14,398 in total).”
BackgroundHays, C., Schutzman, Z., Raghavan, M., Walk, E., & Zimmer, P. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583214 · Cited by 29 · see citing works
“…during the 2018 US elections. stock-2018 [13] are accounts with high volumes of tweets that tagged stock microblogs with cashtags. genuine-accounts-cresci [11] is purportedly a random sample of human Twitter users, confirmed to be genuine by their response to a nat-ural language question.”
BackgroundMethodInfluentialKouvela, M., Dimitriadis, I., & Vakali, A. I. (2020) · International Conference on Management of Emergent Digital EcoSystems (MEDES) · doi:10.1145/3415958.3433075 · Cited by 30 · see citing works
“The same observation is reached in [50] and in [16], which proceed to a more fine-grained classification of bots, providing separate datasets for each type of bots (celebrities, porn bots,etc.).”
BackgroundCresci, S., Petrocchi, M., Spognardi, A., & Tognazzi, S. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3184558.3191595 · Cited by 28 · see citing works
“Most notably, spambot evolution still goes on: recent investigations highlight that new waves of social spambots are rising [1, 3, 6].”
BackgroundSelf-citationBots Acting Like Humans: Understanding and Preventing Harm
Daniel, F., Cappiello, C., & Benatallah, B. (2019) · IEEE Internet Computing · doi:10.1109/mic.2019.2893137 · Cited by 28 · see citing works
Characterizing Social Bots Spreading Financial Disinformation
Tardelli, S., Avvenuti, M., Tesconi, M., & Cresci, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49570-1_26 · Cited by 29 · see citing works
Self-citationRostami, R. R. (2020) · Webology · doi:10.14704/web/v17i1/a204 · Cited by 26 · see citing works
T-Bot: AI-based social media bot detection model for trend-centric twitter network
Gera, S., & Sinha, A. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00897-6 · Cited by 25 · see citing works
“The rapid rise in the number of users participating in OSN conversations has inclined the researchers towards detecting social media bots (Cresci et al. (2017), Perna and Tagarelli (2018), Varol et al. (2017), Morstatter et al. (2016), Zhang and Gupta (2018b)).”
BackgroundFine-Tuned Understanding: Enhancing Social Bot Detection With Transformer-Based Classification
Sallah, A., Alaoui, E. A. A., Agoujil, S., Wani, M. A., Hammad, M. A., Maleh, Y., & El‐Latif, A. A. A. (2024) · IEEE Access · doi:10.1109/access.2024.3440657 · Cited by 27 · see citing works
Comparative Media Regulation in the United States and Europe
Fukuyama, F., & Grotto, A. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108890960.010 · Cited by 26 · see citing works
Application of the Benford’s law to Social bots and Information Operations activities
Madahali, L., & Hall, M. (2020) · International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA) · doi:10.1109/cybersa49311.2020.9139709 · Cited by 26 · see citing works
“These accounts are from Crecsi 2017 dataset [37].”
BackgroundMethodRandom Forest Twitter Bot Classifier
Schnebly, J., & Sengupta, S. (2019) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc.2019.8666593 · Cited by 24 · see citing works
Analyzing Twitter Users' Behavior Before and After Contact by the Russia's Internet Research Agency
Dutta, U., Hanscom, R., Zhang, J. S., Han, R. Y., Lehman, T. S., Lv, Q., & Mishra, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449164 · Cited by 24 · see citing works
- Authors
- Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M.
- Affiliations
- University of Pisa (IT)
- Nokia Bell Labs & University of Padua, Paris, France
- National Research Council (IT)
- Technical University of Denmark (DK)
- Venue
- Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion
- Publisher
- ACM Press
- Published
- 2017
- Pages
- 963-972
- OpenAlex
- W4289253852
- Cited by
- 381
- References
- 50
- Citations by year
- 2026: 31 · 2025: 42 · 2024: 48 · 2023: 48 · 2022: 48 · 2021: 46 · 2020: 62 · 2019: 37 · 2018: 15 · 2017: 4
- FWCI
- 24.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International World Wide Web Conference (WWW) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion)
- Funders
- EU (654024, 675320)
- Regione Toscana (SmartNews: Social sensing for Breaking News)
- Topics
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- Twitter, spam social media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
