CoChat
Sign Up
Cited by 135Misinformation and Its ImpactsEthics and Social Impacts of AISocial Media and Politics

Recommender systems and the amplification of extremist content

Open access
Whittaker, Joe, Looney, Seán, Reed, Alastair, Votta, Fabio·2021·Internet Policy Review
DOI 10.14763/2021.2.1565
Open this paper in CoChat — free account

Abstract

Recommendation algorithms potentially amplifying extremist content has become a policy concern in recent years. We conduct a novel empirical experiment on three platforms (YouTube, Reddit, and Gab) to test this phenomenon. We find that YouTube’s “Recommended for you” system does promote extreme content. We synthesise the findings into the policy debate and argue that co-regulation may provide some solutions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.14763/2021.2.1565

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. Agresti, A. (2013). Categorical Data Analysis . John Wiley & Sons.
  2. Ayres, I., & Braithwaite, J. (1992). Responsive regulation: Transcending the deregulation debate . Oxford University Press.
  3. Bakshy, E., Messing, S., & Adamic, L. (2015). Exposure to Ideologically Diverse News and Opinion on Facebook. Science Express , 1-5. https://doi.org/10.1111/j.1460-2466.2008.00410.x Baugut, P., & Neumann, K. (2020). Online propaganda use during Islamist radicalization. Information Communication and Society , 23 (11), 1570-1592. https://doi.org/10.1080/1369118x.2019.1594333 · DOI · Paper page
  4. Berger, J. M. (2018a). Extremism . MIT Press.
  5. Bozdag, E. (2013). Bias in algorithmic filtering and personalization. Ethics and Information Technology , 15 (3), 209-227. https://doi.org/10.1007/s10676-013-9321-6 · DOI · Paper page
  6. Bucher, T. (2017). The algorithmic imaginary: Exploring the ordinary affects of Facebook algorithms. Information, Communication & Society , 20 (1), 30-44. https://doi.org/10.1080/1369118X.2016.11540 86 · DOI
  7. Conway, M. (2016). Violent Extremism and Terrorism Online in 2016. The Year in Review', Vox Pol .
  8. Conway, M. (2020). Routing the Extreme Right: Challenges for Social Media Platforms'. RUSI Journal . https://doi.org/10.1080/03071847.2020.1727157 · DOI
  9. Copland, S. (2020). Reddit quarantined: Can changing platform affordances reduce hateful material online?'. Internet Policy Review , 9 (4), 1-26. https://doi.org/10.14763/2020.4.1516 · DOI
  10. Courtois, C., Slechten, L., & Coenen, L. (2018). Challenging Google Search filter bubbles in social and political information: Disconforming evidence from a digital methods case study. Telematics and Informatics , 35 (7), 2006-2015. https://doi.org/10.1016/j.tele.2018.07.004 · DOI
  11. DeVito, M. A. (2016). From Editors to Algorithms. Digital Journalism , 1-21. https://doi.org/10.1080/2 1670811.2016.1178592 · DOI
  12. Dylko, I. (2018). Impact of Customizability Technology on Political Polarization. Journal of Information Technology and Politics , 15 (1), 19-33. https://doi.org/10.1080/19331681.2017.1354243 · DOI
  13. Eslami, M. (2015). 'I always assumed that I wasn't really that close to [her]'. CHI'15 Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems , 153-162. https://doi.org/1 0.1145/2702123.2702556
  14. Gaudette, T. (2020). Upvoting Extremism: Collective identity formation and the extreme right on Reddit. New Media and Society . https://doi.org/10.1177/1461444820958123 · DOI
  15. Haim, M., Graefe, A., & Brosius, H. B. (2018). Burst of the Filter Bubble?: Effects of personalization on the diversity of Google News'. Digital Journalism , 6 (3), 330-343. https://doi.org/10.1080/216708 11.2017.1338145 · DOI
  16. Helberger, N., Pierson, J., & Poell, T. (2018). Governing online platforms: From contested to cooperative responsibility. The Information Society , 34 (1), 1-14. https://doi.org/10.1080/01972243.2 017.1391913 · DOI
  17. Heldt, A. (2019). Let's Meet Halfway: Sharing New Responsibilities in a Digital Age'. Journal of Information Policy , 9 , 336-369. https://doi.org/10.5325/jinfopoli.9.2019.0336 · DOI
  18. Holbrook, D. (2015). Designing and Applying an 'Extremist Media Index'. Perspectives on Terrorism , 9 (5), 57-68. https://doi.org/10.1088/0031-9155/49/9/004 · DOI
  19. Holbrook, D. (2017a). Terrorism as process narratives: A study of pre-arrest media usage and the emergence of pathways to engagement. In Terrorism and Political Violence, In Press .
  20. Kraska-Miller, M. (2013). Nonparametric Statistics for Social and Behavioral Sciences . CRC Press.
  21. Ledwich, M., & Zaitsev, A. (2019). Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization . http://arxiv.org/abs/1912.11211. · arXiv
  22. Marsden, C. T. (2010). Net Neutrality: Towards a Co-regulatory Solution . Bloomsbury Academic.
  23. Mittelstadt, B. (2016). Automation, algorithms, and politics| auditing for transparency in content personalization systems. International Journal of Communication , 10 , 12.
  24. Möller, J., Trilling, D., Helberger, N., & van Es, B. (2018). Do not blame it on the algorithm: An empirical assessment of multiple recommender systems and their impact on content diversity. Information, Communication & Society , 21 (7), 959-977. https://doi.org/10.1080/1369118X.2018.144 4076 · DOI
  25. Munger, K., & Phillips, J. (2020). Right-Wing YouTube: A Supply and Demand Perspective. International Journal of Press/Politics . https://doi.org/10.1177/1940161220964767 · DOI · Paper page
  26. Napoli, P. M. (2014). Automated media: An institutional theory perspective on algorithmic media production and consumption. Communication Theory , 24 (3), 340-360. https://doi.org/10.1111/com t.12039 · DOI
  27. Napoli, P. M. (2015). Social media and the public interest: Governance of news platforms in the realm of individual and algorithmic gatekeepers. Telecommunications Policy , 39 (9), 751-760. http s://doi.org/10.1016/j.telpol.2014.12.003 · DOI
  28. O'Callaghan, D. (2015). Down the (White) Rabbit Hole: The Extreme Right and Online Recommender Systems. Social Science Computer Review , 33 (4), 459-478. https://doi.org/10.1177/08944393145553 29 · DOI
  29. Ottoni, R. (2018). Analyzing Right-wing YouTube Channels: Hate, Violence and Discrimination'. Proceedings Ofthe 10th ACM Conference on Web Science . https://doi.org/10.1145/3201064.3201081 · DOI
  30. Palzer, C. (2003). Self-monitoring v. Self-regulation v. Co-regulation. In W. Closs, S. Nikoltchev, & European Audiovisual Observatory (Eds.), Co-regulation of the media in Europe (pp. 29-31).
  31. Pariser, E. (2011). The filter bubble: What the Internet is hiding from you . Penguin Press.
  32. Ribeiro, M. H. (2019). Auditing Radicalization Pathways on YouTube . ACM Symposium on Neural Gaze Detection. http://arxiv.org/abs/1908.08313. · arXiv
  33. Ricci, F., Rokach, L., & Shapira, B. (2011). Recommender Systems Handbook . Springer. https://doi.org/ 10.1007/978-0-387-85820-3 · DOI
  34. Schmid, A. P. (2013). Radicalisation, De-Radicalisation, Counter-Radicalisation: A Conceptual Discussion and Literature Review [Research Paper]. International Centre for Counter-Terrorism. http://www.icct.n l/download/file/ICCT-Schmid-Radicalisation-De-Radicalisation-Counter-Radicalisation-March-201 3.pdf.
  35. Schmitt, J. B. (2018). Counter-messages as prevention or promotion of extremism?! The potential role of YouTube. Journal of Communication , 68 (4), 758-779. https://doi.org/10.1093/joc/jqy029 · DOI
  36. Seaver, N. (2018). Captivating algorithms: Recommender systems as traps. Journal of Material Culture . https://doi.org/10.1177/1359183518820366 · DOI
  37. Stefanija, A. P ., & Pierson, J. (2020). Practical AI Transparency: Revealing Datafication and Algorithmic Identities. Journal of Digital Social Research , 2 (3), 84-125.
  38. Sunstein, C. R. (2002). The law of group polarization. The Journal of Political Philosophy , 10 (2), 175-195. https://doi.org/10.1002/9780470690734.ch4 · DOI
  39. Suzor, N. (2018). Digital constitutionalism: Using the rule of law to evaluate the legitimacy of governance by platforms. Social Media + Society , 4 (3). https://doi.org/10.1177/2056305118787812 · DOI
  40. van Der Vegt, I. (2020). Online influence, offline violence: Language Use on YouTube surrounding the 'Unite the Right' rally. Journal of Computational Social Science , 4 , 333-354. https://doi.org/10.10 07/s42001-020-00080-x
  41. Whittaker, J. (2020). Online Echo Chambers and Violent Extremism. In S. M. Khasru, R. Noor, & Y. Li (Eds.), The Digital Age, Cyber Space, and Social Media: The Challenges of Security & Radicalization (pp. 129-150). Dhaka Institute for Policy, Advocacy, and Governance.

Cited by

Sorted by citations

Authors
Whittaker, J., Looney, S., Reed, A., & Votta, F.
Affiliations
  • Swansea University (GB)
  • University of Amsterdam (NL)
Corresponding author
Whittaker, Joe
ORCID
Venue
Internet Policy Review
Publisher
Internet Policy Review, Alexander von Humboldt Institute for Internet and Society
Published
2021-06-30
Volume
10
Issue
2
ISSN
2197-6775 (ISSN-L)
DOI
10.14763/2021.2.1565
OpenAlex
W3175993430
Cited by
135
References
4
Citations by year
2026: 42 · 2025: 24 · 2024: 41 · 2023: 17 · 2022: 10 · 2021: 1
FWCI
21.93
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Internet Policy Review (publishedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • PEARL (University of Plymouth) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Econstor (Econstor) (submittedVersion, cc-by)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Communication › Social Media and Politics
Keywords
recommender systems, algorithmic amplification, extremist content, YouTube, Reddit, Gab, content moderation, co-regulation, platform policies, online extremism

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review