CoChat
Sign Up
Cited by 20Social Media and PoliticsComplex Network Analysis TechniquesOpinion Dynamics and Social Influence

Separating Polarization from Noise: Comparison and Normalization of Structural Polarization Measures

Open access
Salloum, Ali, Chen, Ted Hsuan Yun, Kivelä, Mikko·2022·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3512962
Open this paper in CoChat — free accountOpen PDF

Abstract

Quantifying the amount of polarization is crucial for understanding and studying political polarization in political and social systems. Several methods are used commonly to measure polarization in social networks by purely inspecting their structure. We analyse eight of such methods and show that all of them yield high polarization scores even for random networks with similar density and degree distributions to typical real-world networks. Further, some of the methods are sensitive to degree distributions and relative sizes of the polarized groups. We propose normalization to the existing scores and a minimal set of tests that a score should pass in order for it to be suitable for separating polarized networks from random noise. The performance of the scores increased by 38%-220% after normalization in a classification task of 203 networks. Further, we find that the choice of method is not as important as normalization, after which most of the methods have better performance than the best-performing method before normalization. This work opens up the possibility to critically assess and compare the features and performance of different methods for measuring structural polarization.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3512962

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Post-hoc normalization via randomization tests remains fundamentally unprincipled, as it applies arbitrary null models retroactively without rigorous justification.

    “Attempts to correct these biases, such as post-hoc normalization via randomization tests [38], remain fundamentally unprincipled, as they apply arbitrary null models retroactively without rigorous justification.”

    From DSP: A Statistically-Principled Structural Polarization Measure (2026) · cites this paper as [38]

    Section 3 (Methods; page unavailable) specifies what each null model preserves and explains why randomization is used to separate structural bias from polarization, rather than presenting the models as arbitrary. Section 5.3 (Normalization of the scores; page unavailable) reports that every normalized score outperformed the best unnormalized score. The paper uses and supports this approach; it does not characterize it as fundamentally unprincipled. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Overall, denoising [38] often moves the polarization scores away from their ideal values rather than correcting them.

    “Overall, denoising [38] often moves the polarization scores away from their ideal values rather than correcting them.”

    From DSP: A Statistically-Principled Structural Polarization Measure (2026) · cites this paper as [38]

    Section 5.3 says denoising corrects for the expected effects of network size and degree distribution, and reports improved performance for all scores after normalization. The Discussion (Section 6) recommends the procedure because it removes substantial noise. The paper does not describe denoising as often moving scores away from ideal values; it reports the opposite effect on its evaluation measures. Page numbers were not available in our copy. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The proposed solution of standardizing the RWC score by its values in a random graph null model requires considerable computational resources.

    “In addition, it requires considerable computational resources.”

    From DSP: A Statistically-Principled Structural Polarization Measure (2026) · cites this paper as [38]

    Section 5.3 (page unavailable) says normalization requires randomizing the network multiple times and recomputing the score; the authors sampled each network 500 times. This supports the claim that the procedure involves repeated computation, but the paper does not characterize its resource requirements as considerable. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Recent structural polarization measures such as Betweenness Centrality Controversy, Boundary Polarization, Dipole Polarization, Krackhardt E/I Ratio, and Random Walk Controversy can produce high polarization scores even for random networks, indicating sensitivity to elementary network features such as average degree and network size.

    “Other more recent measures, such as Betweenness Centrality Controversy [15], Boundary Polarization [16], Dipole Polarization [29], Krackhardt E/I Ratio [22], and Random Walk Controversy [15], can produce high polarization scores even for random networks, indicating an undesirable sensitivity to elementary network features such as average degree and network size [38].”

    From DSP: A Statistically-Principled Structural Polarization Measure (2026) · cites this paper as [38]

  • RWC has systematic bias.

    “A key critique of RWC (as well as of other measures) is its systematic bias [38].”

    From DSP: A Statistically-Principled Structural Polarization Measure (2026) · cites this paper as [38]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 69

  1. Robert Adcock and David Collier. 2001. Measurement validity: A shared standard for qualitative and quantitative research. American political science review 95, 3 (2001), 529-546.
  2. Leman Akoglu. 2014. Quantifying Political Polarity Based on Bipartite Opinion Networks. Proceedings of the Interna- tional AAAI Conference on Web and Social Media 8, 1 (May 2014). https://ojs.aaai.org/index.php/ICWSM/article/view/ 14524
  3. Alessia Amelio and Clara Pizzuti. 2012. Analyzing Voting Behavior in Italian Parliament: Group Cohesion and Evolution. In 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. IEEE, 140-146. https://doi.org/10.1109/ASONAM.2012.33 · DOI
  4. James P Bagrow. 2012. Communities and bottlenecks: Trees and treelike networks have high modularity. Physical Review E 85, 6 (2012), 066118.
  5. Delia Baldassarri and Peter Bearman. 2007. Dynamics of political polarization. American sociological review 72, 5 (2007), 784-811.
  6. Delia Baldassarri and Andrew Gelman. 2008. Partisans without constraint: Political polarization and trends in American public opinion. Amer. J. Sociology 114, 2 (2008), 408-446.
  7. Albert-László Barabási and Réka Albert. 1999. Emergence of Scaling in Random Networks. Science 286, 5439 (1999), 509-512. https://doi.org/10.1126/science.286.5439.509 · DOI · Paper page
  8. Fabian Baumann, Philipp Lorenz-Spreen, Igor M Sokolov, and Michele Starnini. 2020. Modeling echo chambers and polarization dynamics in social networks. Physical Review Letters 124, 4 (2020), 048301.
  9. Loris Belcastro, Riccardo Cantini, Fabrizio Marozzo, Domenico Talia, and Paolo Trunfio. 2020. Learning political polarization on social media using neural networks. IEEE Access 8 (2020), 47177-47187.
  10. Jonathan Bright. 2018. Explaining the emergence of political fragmentation on social media: The role of ideology and extremism. Journal of Computer-Mediated Communication 23, 1 (2018), 17-33.
  11. Ted Hsuan Yun Chen, Ali Salloum, Antti Gronow, Tuomas Ylä-Anttila, and Mikko Kivelä. 2020. Polarization of Climate Politics Results from Partisan Sorting: Evidence from Finnish Twittersphere. arXiv:2007.02706 (2020). · arXiv
  12. Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. 2009. Power-law distributions in empirical data. SIAM review 51, 4 (2009), 661-703.
  13. Alessandro Cossard, Gianmarco De Francisci Morales, Kyriaki Kalimeri, Yelena Mejova, Daniela Paolotti, and Michele Starnini. 2020. Falling into the echo chamber: the italian vaccination debate on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 130-140.
  14. Wesley Cota, Silvio C Ferreira, Romualdo Pastor-Satorras, and Michele Starnini. 2019. Quantifying echo chamber effects in information spreading over political communication networks. EPJ Data Science 8, 1 (2019), 35.
  15. Carlo Dal Maso, Gabriele Pompa, Michelangelo Puliga, Gianni Riotta, and Alessandro Chessa. 2014. Voting behavior, coalitions and government strength through a complex network analysis. PloS one 9, 12 (2014), e116046.
  16. Kareem Darwish. 2019. Quantifying Polarization on Twitter: The Kavanaugh Nomination. In Social Informatics. Springer International Publishing, Cham, 188-201.
  17. Juan Manuel Ortiz de Zarate, Marco Di Giovanni, Esteban Zindel Feuerstein, and Marco Brambilla. 2020. Measuring Controversy in Social Networks Through NLP. In String Processing and Information Retrieval. Springer International Publishing, Cham, 194-209.
  18. Paul DiMaggio, John Evans, and Bethany Bryson. 1996. Have American's social attitudes become more polarized? American journal of Sociology 102, 3 (1996), 690-755.
  19. Fahim Djatmiko, Ridi Ferdiana, and Muhammad Faris. 2019. A Review of Sentiment Analysis for Non-English Language. In 2019 International Conference of Artificial Intelligence and Information Technology (ICAIIT). IEEE, 448-451.
  20. Hanif Emamgholizadeh, Milad Nourizade, Mir Saman Tajbakhsh, Mahdieh Hashminezhad, and Farzaneh Nasr Esfahani. 2020. A framework for quantifying controversy of social network debates using attributed networks: biased random walk (BRW). Social Network Analysis and Mining 10, 1 (2020), 1-20.
  21. P Erdős and A Rényi. 1959. On random graphs I. Publicationes Mathematicae 6 (1959), 290-297.
  22. Dario Fasino, Arianna Tonetto, and Francesco Tudisco. [n.d.]. Generating large scale-free networks with the Chung-Lu random graph model. Networks ([n. d.]). https://doi.org/10.1002/net.22012 · DOI
  23. Morris P Fiorina and Samuel J Abrams. 2008. Political polarization in the American public. Annu. Rev. Polit. Sci. 11 (2008), 563-588.
  24. Santo Fortunato and Darko Hric. 2016. Community detection in networks: A user guide. Physics reports 659 (2016), 1-44.
  25. Kiran Garimella. 2018. Polarization on Social Media. Ph.D. Dissertation. Aalto University. http://urn.fi/URN:ISBN:978- 952-60-7833-5
  26. Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. 2018. Political Discourse on Social Media: Echo Chambers, Gatekeepers, and the Price of Bipartisanship. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW '18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 913-922. https://doi.org/10.1145/3178876.3186139 · DOI · Paper page
  27. Noé Gaumont, Maziyar Panahi, and David Chavalarias. 2018. Reconstruction of the socio-semantic dynamics of political activist Twitter networks-Method and application to the 2017 French presidential election. PloS one 13, 9 (2018), e0201879.
  28. Laetitia Gauvin, Mathieu Génois, Márton Karsai, Mikko Kivelä, Taro Takaguchi, Eugenio Valdano, and Christian L Vestergaard. 2018. Randomized reference models for temporal networks. arXiv:1806.04032 (2018). · arXiv
  29. P.H. Guerra, Wagner Meira Jr, Claire Cardie, and R. Kleinberg. 2013. A measure of polarization on social media NetworksBased on community boundaries. Proceedings of the 7th International Conference on Weblogs and Social Media, ICWSM 2013 (01 2013), 215-224.
  30. Roger Guimera, Marta Sales-Pardo, and Luís A Nunes Amaral. 2004. Modularity from fluctuations in random graphs and complex networks. Physical Review E 70, 2 (2004), 025101.
  31. Eszter Hargittai, Jason Gallo, and Matthew Kane. 2008. Cross-ideological discussions among conservative and liberal bloggers. Public Choice 134, 1-2 (2008), 67-86.
  32. Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. 1983. Stochastic blockmodels: First steps. Social networks 5, 2 (1983), 109-137.
  33. Michael Hout and Christopher Maggio. 2020. Immigration, Race, and Political Polarization. https://doi.org/10.31235/ osf.io/p7q2w · DOI
  34. Luca Iandoli, Simonetta Primario, and Giuseppe Zollo. 2021. The impact of group polarization on the quality of online debate in social media: A systematic literature review. Technological Forecasting and Social Change 170 (2021), 120924.
  35. Lorien Jasny, Joseph Waggle, and Dana R Fisher. 2015. An empirical examination of echo chambers in US climate policy networks. Nature Climate Change 5, 8 (2015), 782-786.
  36. David R Jones. 2001. Party polarization and legislative gridlock. Political Research Quarterly 54, 1 (2001), 125-141.
  37. George Karypis and Vipin Kumar. 1998. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on scientific Computing 20, 1 (1998), 359-392.
  38. B. W. Kernighan and S. Lin. 1970. An efficient heuristic procedure for partitioning graphs. The Bell System Technical Journal 49, 2 (1970), 291-307. https://doi.org/10.1002/j.1538-7305.1970.tb01770.x · DOI
  39. Justin H Kirkland and Justin H Gross. 2014. Measurement and theory in legislative networks: The evolving topology of Congressional collaboration. Social Networks 36 (2014), 97-109.
  40. David Krackhardt and Robert N Stern. 1988. Informal networks and organizational crises: An experimental simulation. Social psychology quarterly (1988), 123-140.
  41. Andrea Lancichinetti, Filippo Radicchi, and José J Ramasco. 2010. Statistical significance of communities in networks. Physical Review E 81, 4 (2010), 046110.
  42. Priya Mahadevan, Dmitri Krioukov, Kevin Fall, and Amin Vahdat. 2006. Systematic topology analysis and generation using degree correlations. ACM SIGCOMM Computer Communication Review 36, 4 (2006), 135-146.
  43. Christos Makridis and Jonathan T Rothwell. 2020. The real cost of political polarization: evidence from the COVID-19 pandemic. Available at SSRN 3638373 (2020).
  44. Moritz Markgraf and Manfred Schoch. 2019. Quantification of Echo Chambers: A Methodological Framework Considering Multi-party Systems. In Proceedings of the 27th European Conference on Information Systems (ECIS). https://aisel.aisnet.org/ecis2019_rp/91/
  45. Lilliana Mason. 2015. "I disrespectfully agree": The differential effects of partisan sorting on social and issue polarization. American Journal of Political Science 59, 1 (2015), 128-145.
  46. Jennifer McCoy, Tahmina Rahman, and Murat Somer. 2018. Polarization and the global crisis of democracy: Common patterns, dynamics, and pernicious consequences for democratic polities. American Behavioral Scientist 62, 1 (2018), 16-42.
  47. Colin McDiarmid and Fiona Skerman. 2017. Modularity of regular and treelike graphs. Journal of Complex Net- works 6, 4 (10 2017), 596-619. https://doi.org/10.1093/comnet/cnx046 arXiv:https://academic.oup.com/comnet/article- pdf/6/4/596/25451962/cnx046.pdf · DOI
  48. Colin McDiarmid and Fiona Skerman. 2020. Modularity of Erdős-Rényi random graphs. Random Structures & Algorithms 57, 1 (2020), 211-243.
  49. Atsushi Miyauchi and Yasushi Kawase. 2016. Z-score-based modularity for community detection in networks. PloS one 11, 1 (2016), e0147805.
  50. Michael Molloy and Bruce Reed. 1995. A critical point for random graphs with a given degree sequence. Random structures & algorithms 6, 2-3 (1995), 161-180.
  51. James Moody and Peter J Mucha. 2013. Portrait of political party polarization. Network Science 1, 1 (2013), 119-121.
  52. Alfredo Jose Morales, Javier Borondo, Juan Carlos Losada, and Rosa M Benito. 2015. Measuring political polarization: Twitter shows the two sides of Venezuela. Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 3 (2015), 033114.
  53. Zachary P Neal. 2020. A sign of the times? Weak and strong polarization in the US Congress, 1973-2016. Social Networks 60 (2020), 103-112.
  54. Matti Nelimarkka, Salla-Maaria Laaksonen, and Bryan Semaan. 2018. Social media is polarized, social media is polarized: towards a new design agenda for mitigating polarization. In Proceedings of the 2018 Designing Interactive Systems Conference. 957-970.
  55. Matti Nelimarkka, Jean Philippe Rancy, Jennifer Grygiel, and Bryan Semaan. 2019. (Re) Design to Mitigate Political Polarization: Reflecting Habermas' ideal communication space in the United States of America and Finland. Proceedings of the ACM on Human-computer Interaction 3, CSCW (2019), 1-25.
  56. Mark EJ Newman. 2006. Modularity and community structure in networks. Proceedings of the national academy of sciences 103, 23 (2006), 8577-8582.
  57. Chiara Orsini, Marija M Dankulov, Pol Colomer-de Simón, Almerima Jamakovic, Priya Mahadevan, Amin Vahdat, Kevin E Bassler, Zoltán Toroczkai, Marián Boguná, Guido Caldarelli, et al. 2015. Quantifying randomness in real networks. Nature communications 6, 1 (2015), 1-10.
  58. Mert Ozer, Mehmet Yigit Yildirim, and Hasan Davulcu. 2017. Measuring the Polarization Effects of Bot Accounts in the US Gun Control Debate on Social Media. Proceedings of ACM Conference (Conference'17) (2017).
  59. Tiago P Peixoto. 2013. Parsimonious module inference in large networks. Physical review letters 110, 14 (2013), 148701.
  60. Abdullateef Rabab'ah, Mahmoud Al-Ayyoub, Yaser Jararweh, and Mohammed Al-Kabi. 2016. Measuring the Controversy Level of Arabic Trending Topics on Twitter. In 2016 7th International Conference on Information and Communication Systems (ICICS). https://doi.org/10.1109/IACS.2016.7476097 · DOI
  61. Ammar Rashed, Mucahid Kutlu, Kareem Darwish, Tamer Elsayed, and Cansın Bayrak. 2020. Embeddings-Based Clustering for Target Specific Stances: The Case of a Polarized Turkey. arXiv:2005.09649 [cs.SI] · arXiv
  62. Anna Rumshisky, Mikhail Gronas, Peter Potash, Mikhail Dubov, Alexey Romanov, Saurabh Kulshreshtha, and Alex Gribov. 2017. Combining Network and Language Indicators for Tracking Conflict Intensity. In Social Informatics, Giovanni Luca Ciampaglia, Afra Mashhadi, and Taha Yasseri (Eds.). Springer International Publishing, Cham, 391-404.
  63. Matteo Serafino, Giulio Cimini, Amos Maritan, Andrea Rinaldo, Samir Suweis, Jayanth R Banavar, and Guido Caldarelli. 2021. True scale-free networks hidden by finite size effects. Proceedings of the National Academy of Sciences 118, 2 (2021).
  64. Shiliang Sun, Chen Luo, and Junyu Chen. 2017. A review of natural language processing techniques for opinion mining systems. Information fusion 36 (2017), 10-25.
  65. William MK Trochim and James P Donnelly. 2001. Research methods knowledge base. Vol. 2. Atomic Dog Pub.
  66. Andrew Scott Waugh, Liuyi Pei, James H. Fowler, Peter J. Mucha, and Mason A. Porter. 2011. Party Polarization in Congress: A Network Science Approach. arXiv:0907.3509 [physics.soc-ph] · arXiv
  67. Ingmar Weber, Venkata R Kiran Garimella, and Alaa Batayneh. 2013. Secular vs. islamist polarization in egypt on twitter. In Proceedings of the 2013 IEEE/ACM international conference on advances in social networks analysis and mining. 290-297.
  68. Pan Zhang and Cristopher Moore. 2014. Scalable detection of statistically significant communities and hierarchies, using message passing for modularity. Proceedings of the National Academy of Sciences 111, 51 (2014), 18144-18149.
  69. Yilin Zhang and Karl Rohe. 2018. Understanding Regularized Spectral Clustering via Graph Conductance. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS'18).

Cited by

Sorted by citations

Authors
Salloum, A., Chen, T. H. Y., & Kivelä, M.
Affiliations
  • Aalto University (FI)
  • University of Helsinki (FI)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2022-03-30
Volume
6
Issue
CSCW1
Pages
1-33
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3512962
OpenAlex
W4226515364
Cited by
20
References
9
Citations by year
2026: 3 · 2025: 9 · 2024: 5 · 2023: 1 · 2022: 2
FWCI
6.48
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, other-oa)
  • Työväentutkimus Vuosikirja (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Academy of Finland (320781)
Topics
  • Social Sciences › Communication › Social Media and Politics
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Complex Network Analysis Techniques
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Opinion Dynamics and Social Influence
Keywords
structural polarization, polarization measures, social networks, normalization, degree distribution, method comparison, network classification, political polarization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review