Empirical evaluation and new design for fighting evolving Twitter spammers

The feature sets from [8, 2] are used to train and test machine learning classifiers in order to distinguish between humans and spammers.

“Such feature sets are then used to train and test machine learning classifiers in order to distinguish between humans and spammers.”

Works [8, 2] propose a list of features to be extracted from manually classified datasets of accounts.

“Both of them propose a list of features to be extracted from manually classified datasets of accounts.”

The features proposed by Yang et al. [2] are robust against evasion techniques adopted by spammers.

“Instead, here we evaluate how those features, which proved to be quite robust against evasion techniques adopted by spammers, perform in detecting fake Twitter followers.”

Even though the proposed features were originally designed for spam detection, this paper is the first to consider them to spot fake followers on Twitter.

“Even if the proposed features have been originally designed for spam detection, here, for the first time, we consider them to spot another category of Twitter accounts, i.e., the fake followers.”

The authors of [2] suggested considering some new features, making evasion more difficult for spammers.

“Thus, they suggested to consider some new features, making evasion more difficult for spammers.”

API ratio (tweets sent from API / total number of tweets) has higher values for suspicious accounts.

“6. API ratio ( tweets sent from API total number of tweets ): higher values for suspicious accounts;”

API URL ratio (tweets sent from API and containing URLs / total number of tweets sent from API) is higher for suspicious accounts.

“7. API URL ratio ( tweets sent from API and containing URLs total number of tweets sent from API ): such ratio is higher for suspicious accounts;”

Average neighbors' followers is commonly higher for legitimate accounts than for spammers.

“3. average neighbors' followers : the average number of followers of the account's friends. This feature aims at reflecting the quality of the choice of friends of an account. The feature has been found to be commonly higher for legitimate accounts than for spammers;”

The authors of [2] observed that Twitter spammers often modify their behavior in order to evade existing spam detection techniques.

“The authors of [2] observed that Twitter spammers often modify their behavior in order to evade existing spam detection techniques.”

Age of the account: the more an account is aged, the more it could be considered a good one.

“1. age of the account (this feature also appears in [28]): the more an account is aged, the more it could be considered a good one;”

For classifiers based on the feature set of [2], RF, J48, and D have MCC greater than 0.98.

“In particular, the RF, J48 and D classifiers have MCC greater than 0.98.”

The classifiers built over the feature set of Yang et al. [2] achieve slightly better results than those built over the feature set of Stringhini et al. [8].

“The ones built over the feature set of [2] achieve slightly better results.”

The other features in [2] are less robust with regards to evasion techniques.

“However, as claimed by the same authors, they are less robust with regards to evasion techniques and thus we decided not to include them in our evaluation.”

All classifiers based on the feature set of [2] have a higher AUC than the corresponding classifiers built with the feature set of [8].

“In addition, all the classifiers based on the feature set by [2] have a higher AUC , when compared with the ones built with the feature set by [8].”

The lower precision and recall of the [8] feature set compared to the [2] feature set show that the features of Stringhini et al. [8] tend to consider some human accounts as fake followers.

“The lower precision and recall with respect to the ones obtained using the set of Yang et al. [2] show that the features of Stringhini et al. [8] exhibit the tendency to consider as fake followers some human accounts.”

Beyond the features directly available from the account profile lookup, the authors of [2] propose some graph-, automation-, and timing-based features.

“Beyond the features directly available from the account profile lookup, the authors propose some graph-, automation-, and timing-based features.”

Bidirectional link ratio is lower for spammer accounts than for legitimate accounts.

“2. bidirectional link ratio ( bidirectional links friends ), where a bidirectional link occurs when two accounts follow each other: this feature has been tested to be lower for spammer accounts than for legitimate accounts;”

Following rate reflects the speed at which an account follows other accounts; spammers usually feature high values of this rate.

“9. following rate : this metric reflects the speed at which an accounts follows other accounts. Spammers usually feature high values of this rate.”

For classifiers based on the feature set of [2], RF, J48, and D have precision and recall around 0.99.

“Similarly, precision and recall are around 0.99 for all of them.”

API tweet similarity is higher for suspicious accounts.

“8. API tweet similarity : this metric considers only the number of similar tweets sent from API. The notion of tweet similarity is as in Section 5.1. This metric is higher for suspicious accounts;”

With both the [2] and [8] feature sets, BN, kNN, and LR classifiers achieve overall worse performances than the other classifiers.

“With both the [2] and [8] feature sets, BN, kNN and LR classifiers achieve, overall, worse performances.”

Average neighbors' tweets is lower for spammers than for legitimate accounts.

“4. average neighbors' tweets : the average number of tweets of the account's followers. This feature is lower for spammers than for legitimate accounts;”

Followings to median neighbor's followers (ratio between number of friends and median of followers of its friends) has been found higher for spammers than for legitimate accounts.

“5. followings to median neighbor's followers : defined as the ratio between the number of friends and the median of the followers of its friends. This feature has been found higher for spammers than for legitimate accounts;”

The decision tree with the feature set of Yang et al. [2] has 17 leaves, 33 nodes, and a height of 8, with the bi-directional link ratio as the root.

“The decision tree with the feature set of Yang et al. [2] has 17 leaves, 33 nodes and a height of 8, with the bi-directional link ratio as the root.”

Even reducing the features, it is possible to have a detection rate higher than 0.95, as shown for Stringhini et al. [8] and Yang et al. [2] in the last lines of Table 13.

“Overall, the results of this experiment show that, even reducing the features, it is possible to have a detection rate higher than 0.95 (as in the last lines of Table 13, for [8] and [2], respectively).”

The pruned version of Stringhini et al. [8] reduces its recall by 0.017, while the pruned version of Yang et al. [2] drops recall by only 0.004.

“Furthermore, we clearly observe that the pruned version of Stringhini et al. [8] reduces its recall of 0.017, while the one of Yang et al. [2] only drops of 0.004”

There are differences between the fake followers in the BAS dataset and the spam accounts characterized in [8] and [2].

“Looking at the tree structure, we observe some interesting differences between the fake followers in our BAS dataset and the spam accounts characterized in [8] and [2].”

The feature set proposed by Yang et al. [2] seems to be slightly more effective than the one proposed by Stringhini et al. [8] when used in feature-based classifiers aiming at fake followers detection.

“Similarly, we have seen that the feature set proposed by Yang et al. [2] seems to be slightly more effective than the one proposed by Stringhini et al. [8], when used in feature-based classifiers aiming at fake followers detection.”

Countermeasures could be taken to evade some of the simplest features the classifiers are built upon.

“We have to point out, however, that countermeasures could be taken to evade some of the simplest features our classifiers are built upon [2].”

The SVM classifier achieves remarkable results, especially with the feature set of [2].

“The SVM classifier, instead, achieves remarkable results, especially with the feature set of [2].”

In [2], the average neighbor's tweets feature was found to be lower for spammers.

“A similar behavior has been observed for the average neighbor's tweets feature, that has been found to be lower for spammers in [2], but higher for our fake followers.”

In the BAS dataset, the average neighbor's tweets feature is higher for fake followers.

“A similar behavior has been observed for the average neighbor's tweets feature, that has been found to be lower for spammers in [2], but higher for our fake followers.”

After pruning, the Yang et al. [2] classifier is able to miss fewer fakes than the Stringhini et al. [8] classifier.

“meaning that the latter is able to miss fewer fakes than the former one after pruning.”

All the classifiers of the previous sections are classifiers of Class B, with the exception of the classifier with the feature set of Yang et al. [2], which belongs to Class C.

“Then, all the classifiers of the previous sections are classifiers of Class B , with the exception of the classifier with the feature set of Yang et al. [2], that belongs to Class C .”

For the feature set of Yang et al. [2], reducing the number of nodes from 33 to 11 decreases accuracy by 0.007 and MCC by 0.014 only.

“for example, for the feature set of Yang et al. [2], reducing the number of nodes from 33 to 11 decreases the accuracy of 0.007 and the MCC of 0.014, only.”

Leaving out the bi-link ratio from the Yang et al. [2] feature set causes an evident decrease in accuracy, precision, recall, F-Measure, AUC, and an even noticeable decrease in MCC for the less pruned trees (subtree raising 0.25 and reduced error 3 folds).

“The results in Table 13 show an evident decrease in accuracy , precision and recall for the less pruned trees (subtree raising 0.25 and reduced error 3 folds), as well as for both F-Measure and AUC , and an even noticeable decrease of the MCC measure.”

The Class C classifiers, built over all the features considered in the study, perform better than all the other ones, including the classifiers using the feature sets of Yang et al. [2] and Stringhini et al. [8], as reported in Table 12.

“We start observing that the Class C classifiers, built over all the features considered in our study, perform better than all the other ones, including the classifiers using the feature sets of Yang et al. [2] and Stringhini et al. [8], as reported in Table 12.”

The social bots studied by Chao Yang et al. appeared as more popular and credible, given the relatively large number of their social connections, and were no longer spamming the same messages over and over again.

“Conversely, the social bots studied by Chao Yang et al. appeared as more popular and credible, given the relatively large number of their social connections. In addition, they were no longer spamming the same messages over and over again.”

Noteworthy works published by Chao Yang et al. between 2011 and 2013 provided the first evidence and theoretical foundations to study social bot evolution.

“Noteworthy works published by Chao Yang et al. between 2011 and 2013 provided the first evidence and the theoretical foundations to study social bot evolution [34].”

The first wave of social bots that populated OSNs until around 2011 was made of rather simplistic bots with few social connections and posted messages and clear signs of automation.

“The first wave of social bots that populated OSNs until around 2011 was made of rather simplistic bots. Accounts with very low reputation due to few social connections and posted messages, and featuring clear signs of automation as shown in panel A of Figure 4.”

Twitter's spambot detection mechanisms are still unable to accurately identify traditional spambots #1, while recent solutions proposed by Academia have succeeded in this task.

“This seems to suggest that Twitter's spambot detection mechanisms are still unable to accurately identify such accounts, while recent solutions proposed by Academia have succeeded in this task [44].”

The supervised system of C. Yang et al. [44] fails to correctly classify the novel social spambots.

“Results in Table 7 show that the system fails to correctly classify the novel social spambots.”

New waves of malicious accounts present different and advanced features, making their detection with existing systems extremely challenging.

“New waves of malicious accounts present different and advanced features, making their detection with existing systems extremely challenging [4], [6].”