CoChat
Sign Up
Cited by 348Sentiment Analysis and Opinion MiningMisinformation and Its ImpactsSocial Media and Politics

The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index; 5 of the 22 sources its claims cite were checked for support.

Quantifying controversy on social media

Open access
Garimella, Kiran, Morales, Gianmarco De Francisci, Gionis, Aristides, Mathioudakis, Michael·2018·ACM Transactions on Social Computing
DOI 10.1145/3140565
Open this paper in CoChat — free accountOpen PDF

Abstract

Which topics spark the most heated debates on social media? Identifying those topics is not only interesting from a societal point of view but also allows the filtering and aggregation of social media content for disseminating news stories. In this article, we perform a systematic methodological study of controversy detection by using the content and the network structure of social media. Unlike previous work, rather than studying controversy in a single hand-picked topic and using domain-specific knowledge, we take a general approach to study topics in any domain . Our approach to quantifying controversy is based on a graph-based three-stage pipeline, which involves (i) building a conversation graph about a topic, (ii) partitioning the conversation graph to identify potential sides of the controversy, and (iii) measuring the amount of controversy from characteristics of the graph. We perform an extensive comparison of controversy measures, different graph-building approaches, and data sources. We use both controversial and non-controversial topics on Twitter, as well as other external datasets. We find that our new random-walk-based measure outperforms existing ones in capturing the intuitive notion of controversy and show that content features are vastly less helpful in this task.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This paper presents a domain-agnostic, graph-based pipeline for quantifying controversy on social media by building conversation graphs, partitioning them into two sides, and scoring how separated those sides are. Across Twitter topics and external datasets, a new random-walk-based measure (RWC) best captures controversy, while content-only features are largely unhelpful.

Key findings

  • The proposed Random Walk Controversy (RWC) measure most cleanly separates controversial from non-controversial topics; baselines GMCK and MBLB “fail to separate the two groups,” especially on retweet graphs.
  • An efficient RWC variant using random walk with restart is almost identical to Monte Carlo RWC (Pearson’s r = 0.96) and up to 200 times faster.
  • Content-based graph building and bag-of-words features do not reliably distinguish controversial topics; an unpaired Wilcoxon rank-sum test at p = 0.05 could not reject the null of no difference. Sentiment variance is more useful: controversial topics have variance at least 2, non-controversial at most 1.5.
  • On external datasets, all measures distinguish controversial graphs to some extent except Karate club; Political blogs is judged only moderately controversial, while Twitter politics is among the most controversial. In evolving Venezuela data, RWC and EC correlate with MBLB at r = 0.944 and 0.949, BCC at r = 0.743, and GMCK at r = 0.542.

Methods

  • Design: a three-stage pipeline — (i) build a conversation graph for a topic, (ii) partition it into two sides using METIS, and (iii) compute a controversy measure from graph characteristics.
  • Data: Twitter hashtags trending Feb 27–Jun 15, 2015, with 10 manually chosen controversial and 10 non-controversial topics; topics expanded from seed hashtags via a co-occurrence similarity measure. Graphs include retweet and follow graphs, plus content and hybrid variants. External datasets from prior work, a 56-day Venezuela dataset, synthetic Erdős-Rényi graphs with planted communities, and a three-month deployment over 924 trending hashtags were also used.
  • Analysis: compares RWC, BCC, EC, and baselines GMCK and MBLB using beanplots, correlations, simulations, and external-dataset validation; also tests user-level controversy scores against BiasWatch.

Limitations

  • Stated: findings are mostly on Twitter; controversial topics were manually picked, which may introduce bias; only 20 topics are used in the main experiment, risking overfitting; the pipeline relies on graph partitioning; multisided controversies are deferred; evaluation depends on subjective labeling of controversy.
  • Stated: sentiment analysis relies on tools that do not work reliably for non-English languages, so it cannot be applied to topics such as #russia_march.
  • Evident from the text: content-based approaches produced sparse graphs and overlapping partitions, and several negative results (mentions graph, cuts, modularity, SPID) are reported, indicating the method’s scope is limited to the graph-structural pipeline that worked.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3140565

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 of 22 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

Show all 83 claims

  • Probably not a substantial claim. 28
  • It may state two separate claims as one. 29
  • The quote only partly states this claim. 4
53 of 53

  • The authors state this as an assumption.

    Controversial and polarized topics induce conversation graphs with clustered structure, representing different opinions and points of view.

    “In particular, we assume that controversial and polarized topics induce graphs with clustered structure, representing different opinions and points of view.”

  • The authors state this as an assumption.

    Users can be split into two sides according to their point of view on a topic.

    “assuming that users are split into two sides according to their point of view on the topic, which are these two sides?”

  • The authors state this as an assumption.

    There is homophily in the social network, so on a given topic people agree more often than not with people they follow.

    “The rationale for using this graph is based on an assumption of the presence of homophily in the social network, which is a common trait in this setting.”

  • The paper relies on this without support.

    Retweets typically indicate endorsement.

    “Retweets typically indicate endorsement.5 Users who retweet signal endorsement of the opinion expressed in the original tweet by propagating it further.”

  • The authors state this as an assumption.

    A single retweet on a topic is not enough of a signal to infer endorsement.

    “We presume that a single retweet on a topic is not enough of a signal to infer endorsement.”

  • The paper relies on this without support.

    The quality of the topic affects critically the entire pipeline.

    “Since the quality of the topic affects critically the entire pipeline, we want to avert this issue and ensure minimal noise is introduced in the expanded set of hashtags.”

  • The paper relies on this without support.

    The 10 manually selected hashtags represent controversial topics of discussion.

    “We first manually pick a set of 10 hashtags that we know represent controversial topics of discussion.”

  • The paper relies on this without support.

    In a controversial discussion, there are authoritative users on both sides, as evidenced by a large degree in the graph.

    “It is based on the rationale that, in a controversial discussion, there are authoritative users on both sides, as evidenced by a large degree in the graph.”

  • The paper relies on this without support.

    High degree is a proxy for authoritativeness, as it means that a user has received a large number of endorsements on the specific topic.

    “High degree is a proxy for authoritativeness, as it means that a user has received a large number of endorsements on the specific topic.”

  • The paper relies on this without support.

    Retweet graphs are inherently directed.

    “retweet graphs (one of our main considerations in this paper) are inherently directed”

  • The paper relies on this without support.

    Directed retweet graphs are very often star-like.

    “since these directed graphs are very often star-like”

  • The paper relies on this without support.

    There are a few authoritative users who generate information that spreads through the graph.

    “there are a few authoritative users who generate information that spreads through the graph.”

  • The paper relies on this without support.

    Extreme values of R (close to -1 or 1) correspond to users who belong most clearly to either side of the controversy.

    “Intuitively, extreme values of R (close to -1 or 1) correspond to users who belong most clearly to either side of the controversy.”

  • The paper relies on this without support.

    Collecting a large number of samples is computationally intensive, and leads to slow evaluation of RWC.

    “Nevertheless, collecting a large number of samples is computationally intensive, and leads to slow evaluation of RWC.”

  • The authors state this as an assumption.

    When computing RWC_user(u,X), user u belongs to side X of the controversy.

    “Assuming that user u belongs to side X of the controversy (i.e., u ∈ X )”

  • The paper relies on this without support.

    The human annotations for controversy score provided by the authors of [37] are a valid ground truth for controversy.

    “The authors also provide datasets which contain human annotations for controversy score (in the range [-2,2]) for 500 randomly selected users.”

  • The paper relies on this without support.

    The external datasets used in previous studies [1,10,27] have a known ground truth for controversy.

    “we test all the measures on a set of external datasets used in previous studies [1,10,27] to validate the measures against a known ground truth.”

  • The paper relies on this without support.

    The topics in Table 2 are correctly labeled as controversial or non-controversial.

    “The beanplot is divided into two groups, one for controversial topics (left/dark) and one for non-controversial ones (right/light).”

  • The authors state this as an assumption.

    The two sides of a controversial topic use different vocabularies in reference to the issue at hand, despite sharing the hashtag.

    “The underlying assumption is that the two sides, while sharing the use of the hashtag for the topic, use different vocabularies in reference to the issue at hand.”

  • The authors state this as an assumption.

    The authors' intuitive labelling of topics as controversial/polarized is reasonable for developing methods that can be adapted to large-scale systems.

    “This might not always be true, but given that the alternative is to hand-label/survey the thousands of users, we presume that this assumption is reasonable for developing methods that can be adapted to large scale systems.”

  • The paper relies on this without support.

    Ground truths for controversial topics are hard to find, especially for ephemeral issues.

    “Unfortunately, ground truths for controversial topics are hard to find, especially for ephemeral issues.”

  • The paper relies on this without support.

    Twitter is one of the main venues for online public discussion and one of the few for which data is available.

    “Twitter is one of the main venues for online public discussion, and one of the few for which data is available.”

  • The paper relies on this without support.

    Graph partitioning is a hard but well studied problem.

    “Given that graph partitioning is a hard but well studied problem, we rely on off-the-shelf techniques for this step.”

  • The paper relies on this without support.

    Not all controversies involve only two sides with opposing views; some discussions are multifaceted or have three or more competing views.

    “Not all controversies involve only two sides with opposing views.Some times discussions are multifaceted, or there are three or more competing views on the field.”

  • The paper relies on this without support.

    Sentiment analysis technologies do not work reliably for languages other than English.

    “Moreover, this feature relies on technologies that do not work reliably for languages other than English and hence cannot be applied for topics such as #russia_march.”

  • The paper relies on this without support.

    The hashtags represent the intuitive notion of controversy that the authors strive to capture.

    “Moreover, the hashtags represent the intuitive notion of controversy that we strive to capture, so human judgement is an important ingredient we want to use.”
26 of 26

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Hitting-time percentile interpretation is reversed

    Major

    The defined ρ X (u) counts vertices that reach X + in fewer steps than u does. A vertex very close to X + relative to most others would therefore have ρ X (u) near 0, not near 1; the analogous claim for a vertex far from Y + is reversed as well.

  • User controversy score cannot take the stated negative values

    Major

    The stated formula is a ratio of nonnegative probabilities, so it can only produce values from 0 to 1. It cannot assign negative scores to one side or use 0 as the neutral midpoint as claimed.

2 of 2

Citations

Cites

Flagged first, then by citations

Show all 39 references

Research works printed but not linked: 6

  1. Michael Conover , Jacob Ratkiewicz , Matthew Francisco , Bruno Gonçalves , Filippo Menczer , and Alessandro Flammini . 2011 . Political polarization on twitter . In Proceedings of the International Conference on Weblogs and Social Media (ICWSM’11) . Michael Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Gonçalves, Filippo Menczer, and Alessandro Flammini. 2011. Political polarization on twitter. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM’11).
  2. Michael Dimock , Carroll Doherty , Jocelyn Kiley , and Russ Oates . 2014. Political polarization in the American public: How increasing ideological uniformity and partisan antipathy affect politics, compromise and everyday life . Pew Research Center , Washington, DC ( 2014 ). Michael Dimock, Carroll Doherty, Jocelyn Kiley, and Russ Oates. 2014. Political polarization in the American public: How increasing ideological uniformity and partisan antipathy affect politics, compromise and everyday life. Pew Research Center, Washington, DC (2014).
  3. Pedro Henrique Calais Guerra , Wagner Meira Jr , Claire Cardie , and Robert Kleinberg . 2013 . A measure of polarization on social media networks based on community boundaries . In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM’13) . Pedro Henrique Calais Guerra, Wagner Meira Jr, Claire Cardie, and Robert Kleinberg. 2013. A measure of polarization on social media networks based on community boundaries. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM’13).
  4. George Karypis and Vipin Kumar . 1998. A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices . University of Minnesota, Department of Computer Science and Engineering, Army HPC Research Center , Minneapolis, MN ( 1998 ). George Karypis and Vipin Kumar. 1998. A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. University of Minnesota, Department of Computer Science and Engineering, Army HPC Research Center, Minneapolis, MN (1998).
  5. Manfred Klenner , Michael Amsler , Nora Hollenstein , and Gertrud Faaß . 2014 . Verb polarity frames: A new resource and its application in target-specific polarity classification . In Proceedings of the Conference on Natural Language Processing (KONVENS’14) . 106--115. Manfred Klenner, Michael Amsler, Nora Hollenstein, and Gertrud Faaß. 2014. Verb polarity frames: A new resource and its application in target-specific polarity classification. In Proceedings of the Conference on Natural Language Processing (KONVENS’14). 106--115.
  6. Mikalai Tsytsarau , Themis Palpanas , and Kerstin Denecke . 2011 . Scalable detection of sentiment-based contradictions. DiversiWeb , WWW 11 (2011), 105 -- 112 . Mikalai Tsytsarau, Themis Palpanas, and Kerstin Denecke. 2011. Scalable detection of sentiment-based contradictions. DiversiWeb, WWW 11 (2011), 105--112.

Cited by

Sorted by citations

See all 348

Related

Metadata

Authors
Garimella, K., Morales, G. D. F., Gionis, A., & Mathioudakis, M.
Affiliations
  • Aalto University (FI)
  • Qatar Computing Research Institute, Doha, Qatar
  • Centre National de la Recherche Scientifique (FR)
ORCID
Venue
ACM Transactions on Social Computing
Publisher
Association for Computing Machinery (ACM)
Published
2018-01-18
Volume
1
Issue
1
Pages
1-27
ISSN
  • 2469-7818 (ISSN-L)
  • 2469-7826
DOI
10.1145/3140565
OpenAlex
W2792053130
Semantic Scholar
5afbbda067ec11447bc3c330d36598137a4acc89
Cited by
348
References
39
Citations by year
2026: 22 · 2025: 39 · 2024: 44 · 2023: 52 · 2022: 51 · 2021: 52 · 2020: 48 · 2019: 29 · 2018: 10
FWCI
19.65
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Social Computing (publishedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
Funders
  • European Commission (654024)
  • Academy of Finland (286211)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Communication › Social Media and Politics
Keywords
controversy detection, social media, Twitter, graph partitioning, content features, news dissemination

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-13, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-13 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review