Quantifying controversy on social media

The paper's new random-walk-based controversy measure outperforms existing controversy measures in capturing the intuitive notion of controversy.

“We find that our new random-walk-based measure outperforms existing ones in capturing the intuitive notion of controversy”

• Also stated in: § Paper · plain

  “Section 8 shows that our method outperforms existing ones [27,40].”

• cites [27] sourceCould not verify

  The Performance Optimization > Comparison with other Algorithms section compares graph-layout algorithms and measures layout quality, not controversy. Searches for the random-walk-based controversy measure and its claimed advantage found no passage addressing either point; page numbers were unavailable in our copy. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

For all measures the median score of controversial topics is higher than for non-controversial ones.

“For all measures the median score of controversial topics is higher than for non-controversial ones.”

• AssumptionRests on an assumption: The topics in Table 2 are correctly labeled as controversial or non-controversial.

BCC and EC show varying degrees of separation and overlap.

“BCC and EC show varying degrees of separation and overlap”

• AssumptionRests on an assumption: The topics in Table 2 are correctly labeled as controversial or non-controversial.

All the measures agree that the Political blogs dataset presents only a moderate level of controversy.

“all the measures agree that it presents only a moderate level of controversy.”

By manual inspection, most trending hashtags are not related to controversial discussions.

“By manual inspection we find that most trending hashtags are not related to controversial discussions [19].”

• cites [19] sourceNot checked — no full text of the source

The 5-category Fleiss' κ value between our discretized controversy scores and human annotations is 0.35.

“The κ value is 0.35, which represents a 'fair' level of agreement, according to Landis and Koch [36].”

Building the retweet graph with a threshold τ = 1 did not produce reliable results.

“We remark that, in preliminary experimentation with this approach, building the retweet graph with a threshold τ = 1 did not produce reliable results.”

All the measures are able to distinguish controversial graphs to some extent, in the sense that they return higher values for the controversial cases.

“All the measures are able to distinguish controversial graphs to some extent, in the sense that they return higher values for the controversial cases.”

The random-walk-based measure generalizes well, as it agrees with previously defined measures when tested on datasets from existing work.

“In addition, it also generalizes well as it agrees with previously-defined measures when tested on datasets from existing work.”

The two baselines GMCK and MBLB fail to separate the two groups.

“The two baselines GMCK and MBLB instead fail to separate the two groups.”

• AssumptionRests on an assumption: The topics in Table 2 are correctly labeled as controversial or non-controversial.

The variance of the sentiment expressed on a topic is a reliable indication of controversy.

“We also find that the variance of the sentiment expressed on a topic is a reliable indication of controversy.”

All features except content-based ones are useful in detecting controversial topics, to different extents.

“Our study shows that all features, except from content-based ones, are useful in detecting controversial topics, to different extents.”

One of the paper's newly proposed measures, based on random walks, discriminates controversial topics with great accuracy.

“We find that one of our newly-proposed measures, based on random walks, is able to discriminate controversial topics with great accuracy.”

The Twitter politics dataset is clearly one of the most controversial ones across all measures.

“the Twitter politics dataset is clearly one of the most controversial one across all measures.”

BCC has a very wide distribution.

“while BCC has a very wide distribution”

• AssumptionRests on an assumption: The topics in Table 2 are correctly labeled as controversial or non-controversial.

Content features are vastly less helpful than other features in detecting controversial topics.

“and show that content features are vastly less helpful in this task.”

The difference among the tested graph partitioning algorithms (METIS, spectral clustering, label propagation, and affiliation-graph-based models) is not significant.

“The difference among these methods is not significant, however from visual inspection METIS generates the cleanest partitions.”

RWC is the best measure on our datasets.

“Figures 7 and 8 clearly show that RWC is the best measure on our datasets.”

• AssumptionRests on an assumption: The topics in Table 2 are correctly labeled as controversial or non-controversial.

The hybrid content-and-retweet graph approach shows the same pattern as content graphs, with sparse graphs and overlapping partitions.

“The same pattern applies for the hybrid approach, as shown in Figure 4.”

For Twitter, endorsement features (i.e., retweets) are the most useful features for detecting controversial topics.

“Particularly for Twitter, we find endorsement features (i.e., retweets) to be the most useful.”

The Karate club graph is an exception to the pattern that measures return higher values for controversial cases.

“The only exception is Karate club.”

The manually selected non-controversial topics contain no clear instance of controversy in a sample of tweets.

“In addition to our intuition that these topics are non-controversial, we manually check a sample of tweets, and we are unable to identify any clear instance of controversy.2”

Our approach works well on a number of datasets extracted in-the-wild from Twitter.

“We have shown that our approach works well on a number of datasets extracted in-the-wild from Twitter.”

Our system was able to identify controversial hashtags in the wild.

“our system was able to identify the following controversial hashtags:”

All the measures proposed in this work show the same trend as MBLB.

“All the measures proposed in this work show the same trend as MBLB.”

Using correlation clustering for graph partitioning with negative edges for mentions yields qualitatively worse results than METIS.

“Alas, the results are qualitatively worse than those obtained by METIS.”

Among the measures studied, the random-walk-based RWC most neatly separates controversial topics from non-controversial ones.

“Among the measures we studied, the random-walk-based RWC most neatly separates controversial topics from non-controversial ones.”

Our approach also generalizes well to datasets from different domains.

“We have shown that our approach also generalizes well to datasets from different domains.”

Our pipeline is able to find the two communities with opposing views (grouping together gun control supporters and moderates, as suggested in the original study) without any external help.

“Our pipeline is able to find the two communities with opposing views (grouping together gun control supporters and moderates, as suggested in the original study) without any external help.”

Using the difference in conductance with cuts obtained by METIS and by random partitions yields equally poor results.

“Unfortunately, we obtain equally poor results by using the difference in conductance with cuts obtained by METIS and by random partitions.”

Simple content-based representations do not work in general.

“We have also shown that simple content-based representations do not work in general, while sentiment analysis offers promising results.”

Conductance in all its variants yields poor results.

“Conductance (in all its variants) also yields poor results.”

RWC and EC follow very closely the original measure (Pearson correlation coefficients r of 0.944 and 0.949, respectively).

“Both RWC and EC follow very closely the original measure (Pearson correlation coefficients r of 0.944 and 0.949, respectively)”

RWC is the best-performing score among the ones in this study.

“Given that RWC is the best-performing score among the ones in this study, we focus our attention solely on it henceforth.”

In our dataset, the mention graphs are extremely sparse because we focus on short-lived events.

“However, in our dataset the mention graphs are extremely sparse given that we focus on short-lived events.”

A majority of the 924 trending hashtags collected have an RWC score around zero.

“A majority of these hashtags have an RWC score around zero.”

Controversial topics have a higher variance in sentiment than non-controversial topics.

“Indeed, controversial topics have a higher variance than non-controversial ones, as shown in Figure 12.”

For a given value of p1, a smaller value of p2 generates a larger RWC score, as the communities are more well separated.

“for a given value of p 1 , a smaller value of p 2 generates a larger RWC score, as the communities are more well separated.”

Merging mentions into the retweet graph does not provide any noticeable improvement.

“Merging the mentions into the retweet graph does not provide any noticeable improvement.”

Most hashtags that are reported as trending on Twitter concern topics that are not controversial.

“most hashtags that are reported as trending on Twitter concern topics that are not controversial.”

Controversial and non-controversial topics do not differ in whether their sentiment is more positive or negative.

“While it is not possible to say that controversial topics are more positive or negative than noncontroversial ones, we can detect a difference in their variance.”

Sentiment analysis offers promising results.

“We have also shown that simple content-based representations do not work in general, while sentiment analysis offers promising results.”

All measures agree with the conclusions drawn in the original paper that the Gun control topic is highly controversial.

“All measures agree with the conclusions drawn in the original paper that this topic is highly controversial.”

BCC shows a more jagged behavior in the first half of the plot (r = 0.743), due to the discrete nature of shortest paths.

“while BCC shows a more jagged behavior in the first half of the plot (r = 0.743), due to the discrete nature of shortest paths.”

Previously-used measures are not reliable.

“We have shown that previously-used measures are not reliable and demonstrated that controversy can be identified both in the retweet and topic-induced follow graph.”

The measures agree with previous independent results on external datasets.

“Additionally, our measures agree with previous independent results on external datasets, which further decreases the likelihood of overfitting.”

Simple measures such as size of the cut of the partitions do not generalize across different graphs.

“Simple measures such as size of the cut of the partitions do not generalize across different graphs.”

Using the all-pairs expected hitting time computed with SimRank and computing the SPID of this distribution yields mixed results.

“We compute the SPID (ratio of variance to mean) of this distribution, however results are mixed.”

For a given value of p2, a larger value of p1 generates a larger RWC score, as the communities are denser.

“Conversely, for a given value of p 2 , a larger value of p 1 generates a larger RWC scores, as the communities are denser.”

GMCK reports an almost constant moderate value of controversy during the whole period (r = 0.542), with barely noticeable peaks and dips.

“Conversely, GMCK reports an almost constant moderate value of controversy during the whole period (r = 0.542), with barely noticeable peaks and dips.”

In the content representation, an unpaired Wilcoxon rank-sum test at p=0.05 did not reject the null hypothesis of no difference between controversial and non-controversial topics.

“We use an unpaired Wilcoxon rank-sum test at the p = 0.05 significance level, but we are unable to reject the null hypothesis that there is no difference in these measures between the controversial and non-controversial topics.”

Using a single hashtag to define a topic may miss part of the relevant posts, because opposing sides may use different hashtags.

“Using a single hashtag may thus miss part of the relevant posts.”

Our approach provides results that are similar to the state-of-the-art approach (BiasWatch).

“Our approach thus provides results that are similar to the state-of-the-art approach.”

If the two partitions are not well separated, the paths that connect vertices across the two partitions will pass through one of the many edges in the cut, leading to betweenness values for C similar to the rest of the graph.

“In this case, the paths that connect vertices across the two partitions will pass through one of the many edges in the cut, leading to betweenness values for C similar to the rest of the graph.”

If the two partitions are well-separated, the shortest paths that connect vertices of the two partitions will pass through the edges in the cut, leading to high betweenness values for edges in C.

“In this case, the shortest paths that connect vertices of the two partitions will pass through the edges in the cut, leading to high betweenness values for edges in C.”

Using a threshold of τ = 2 retweets per edge is a good trade-off between high selectivity and noise reduction for building retweet graphs.

“Using τ = 2 retweets as threshold proves to be a good trade-off between high selectivity (which hinders analysis) and noise reduction.”

• AssumptionThe authors state this as an assumption: A single retweet on a topic is not enough of a signal to infer endorsement.

Using the entire social graph would not necessarily produce well-separated partitions that correspond to single topics of discussion, because additional edges blur those partitions.

“Note that using the entire social graph would not necessarily produce well-separated partitions that correspond to single topics of discussion, as those partitions would be “blurred” by the existence of additional edges that are due to other reasons (e.g., o ine social connections).”

The probabilities P_AB in RWC are not skewed by the size of each partition, because the random walk starts with equal probability from each partition.

“The aforementioned probabilities have the following desirable properties: (i) they are not skewed by the size of each partition, as the random walk starts with equal probability from each partition”

If the two partitions X and Y are well separated, then label propagation will assign different extreme (±1) R-values to the two partitions, leading to higher values of the MBLB measure.

“The rationale for this measure is that, if the two partitions X and Y are well separated, then label propagation will assign different extreme (±1) R-values to the two partitions, leading to higher values of the MBLB measure.”

• AssumptionRests on an assumption: Extreme values of R (close to -1 or 1) correspond to users who belong most clearly to either side of the controversy.

If the two partitions are not well separated, then the cut will consist of strong ties.

“On the other hand, if the two partitions are not well separated, then the cut will consist of strong ties.”

Higher values of the GMCK boundary connectivity measure correspond to higher degrees of controversy.

“Higher values of the measure correspond to higher degrees of controversy.”

Larger differences in the size of the two partitions (reflected in the value of ∆A) lead to decreased values for the MBLB measure.

“Note also that larger differences in the size of the two partitions (reflected in the value of ∆A) lead to decreased values for the measure”

The two-dimensional layouts produced by ForceAtlas2 indicate a layout with maximum modularity.

“Based on this observation, the twodimensional layouts produced by this algorithm indicate a layout with maximum modularity.”

The authors of [37] provide datasets containing human annotations for controversy score (range [-2,2]) for 500 randomly selected users.

“The authors also provide datasets which contain human annotations for controversy score (in the range [-2,2]) for 500 randomly selected users.”

• cites [37] sourceNot checked — no full text of the source

The paper's work is one of the few that quantifies the degree of controversy using language- and domain-independent methods.

“Our work is one of the few that quantifies the degree of controversy using language and domain independent methods.”

If the two partitions represent two sides of a controversy, then boundary vertices will be more strongly connected to internal vertices than to other boundary vertices of either partition.

“The reasoning for this measure is that, if the two partitions represent two sides of a controversy, then boundary vertices will be more strongly connected to internal vertices than to other boundary vertices of either partition.”

• cites Guerra et al. [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A force-directed embedding also maximizes modularity.

“According to Noack [43], a force-directed embedding also maximizes modularity.”

• cites Noack [43]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The proposed variant of RWC can handle cases where the random walker gets stuck (dangling vertices) by using restarts.

“This variant can handle cases where the random walker gets stuck (i.e., dangling vertices), by using restarts.”

The probabilities P_AB in RWC are not skewed by the total degree of vertices in each partition, because they are conditional on ending in either partition.

“(ii) they are not skewed by the total degree of vertices in each partition, as the probabilities are conditional on ending in either partition”

If the two partitions are well-separated, then the cut will consist of edges that bridge structural holes.

“The intuition here is that, if the two partitions are well-separated, then the cut will consist of edges that bridge structural holes [7].”

• cites [7] sourceNot checked — no full text of the source

Directly applying the original hashtag clustering approach results in extremely noisy clusters because very popular hashtags co-occur with a large number of hashtags.

“However, this simple approach presents one drawback, in that very popular hashtags such as #ff or #follow co-occur with a large number of hashtags.Hence, directly applying the original approach results in extremely noisy clusters.”

• cites Feng et al. [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The bag-of-words representation of content is not a good basis for the task of quantifying controversy.

“This result suggests that the bag-of-words representation of content is not a good basis for our task.”

The findings are mostly based on datasets from Twitter.

“We present our findings mostly on datasets coming from Twitter.”

• AssumptionRests on an assumption: Twitter is one of the main venues for online public discussion and one of the few for which data is available.

The proposed measures generalize well to datasets from other social media and the Web.

“In addition, our measures generalize well to datasets from other social media and the Web.”

• Also stated in: § 10.3 Conclusions · boosted

  “Besides, our measures gracefully generalize to datasets from other domains and previous studies.”

The tones with which controversial topics are debated are stronger.

“In practice, the”

Sentiment analysis is able to detect the stronger tones of controversial topics.

“In practice, the”

The main experiment uses only 20 topics.

“we use only 20 topics for our main experiment.”

There is not enough signal in the content representation to discern between controversial and non-controversial topics with confidence.

“Therefore, there is not enough signal in the content representation to discern between controversial and non-controversial topics with confidence.”

The difference between Political blogs and Twitter politics suggests that the measures are more geared towards capturing the dynamics of controversy as it unfolds on social media, which might differ from more traditional blogs.

“This difference suggests that the measures are more geared towards capturing the dynamics of controversy as it unfolds on social media, which might differ from more traditional blogs.”

The paper's methods are useful for measuring the degree of polarization of a topic in a domain-agnostic fashion.

“Our paper contributes methods that are useful in this setting, and enable measuring the degree of polarization of a topic in a domain-agnostic fashion.”

Using content for the task of quantifying controversy might not be straightforward.

“It also agrees with our earlier attempts to use content to build the graph used in the pipeline (see Section 4) -which suggests that using content for the task of quantifying controversy might not be straightforward.”

Given the small number of examples, there is a real risk of overfitting the measures to the dataset.

“Given the small number of examples, the risk of overfitting our measures to the dataset is real.”

Only 40% of the topics were used during the development of the measures to reduce overfitting risk.

“We reduce this risk by using only 40% of the topics during the development of the measures.”