Do not blame it on the algorithm: an empirical assessment of multiple recommender systems and their impact on content diversity
Open accessAbstract
In the debate about filter bubbles caused by algorithmic news recommendation, the conceptualization of the two core concepts in this debate, diversity and algorithms, has received little attention in social scientific research. This paper examines the effect of multiple recommender systems on different diversity dimensions. To this end, it maps different values that diversity can serve, and a respective set of criteria that characterizes a diverse information offer in this particular conception of diversity. We make use of a data set of simulated article recommendations based on actual content of one of the major Dutch broadsheet newspapers and its users (N=21,973 articles, N=500 users). We find that all of the recommendation logics under study proved to lead to a rather diverse set of recommendations that are on par with human editors and that basing recommendations on user histories can substantially increase topic diversity within a recommendation set.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A number of interesting ideas are emerging, from voting on different algorithmic policymakers to using LLMs to summarize diverse human opinions and generate consensus statements.
“A number of interesting ideas are emerging, from voting on different algorithmic policymakers 184 to using LLMs to summarize diverse human opinions 185 and generate consensus statements 153 .”
From Machine culture (2023) · cites this paper as [153]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
(2011) · Cited by 4,430 · see citing works
Software Framework for Topic Modelling with Large Corpora
Rek, R. R. U., & Sojka, P. (2010) · doi:10.13140/2.1.2393.1847 · Cited by 3,766 · see citing works
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245 · see citing works
(2000) · Applied Physics Letters · doi:10.1162/jmlr.2003.3.4-5.993 · Cited by 3,030 · see citing works
Introduction to Recommender Systems Handbook
Ricci, F., Rokach, L., & Shapira, B. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_1 · Cited by 2,378 · see citing works
Filter Bubbles, Echo Chambers, and Online News Consumption
Flaxman, S., Goel, S., & Rao, J. M. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfw006 · Cited by 1,992 · see citing works
Improving recommendation lists through topic diversification
Ziegler, C.-N., McNee, S. M., Konstan, J. A., & Lausen, G. (2005) · International World Wide Web Conference (WWW) · doi:10.1145/1060745.1060754 · Cited by 1,920 · see citing works
LDAvis: A method for visualizing and interpreting topics
Sievert, C., & Shirley, K. E. (2014) · doi:10.3115/v1/w14-3110 · Cited by 1,447 · see citing works
Thinking critically about and researching algorithms
Kitchin, R. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1154087 · Cited by 1,189 · see citing works
Stroud, N. J. (2011) · doi:10.1093/acprof:oso/9780199755509.001.0001 · Cited by 950 · see citing works
Explaining the user experience of recommender systems
Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9118-4 · Cited by 799 · see citing works
Niche news: the politics of news choice
(2011) · Choice Reviews Online · doi:10.5860/choice.49-1894 · Cited by 695 · see citing works
Bias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649 · see citing works
In Search of a Standard: four models of democracy and their normative implications for journalism
Strömbäck, J. (2005) · Journalism Studies · doi:10.1080/14616700500131950 · Cited by 572 · see citing works
Should we worry about filter bubbles?
Zuiderveen Borgesius, F. J., Trilling, D., Möller, J., Bodó, B., de Vreese, C. H., & Helberger, N. (2016) · Internet Policy Review · doi:10.14763/2016.1.401 · Cited by 541 · see citing works
Self-citationDiversity in recommender systems – A survey
Kunaver, M., & Požrl, T. (2017) · Knowledge-Based Systems · doi:10.1016/j.knosys.2017.02.009 · Cited by 506 · see citing works
Four models of the public sphere in modern democracies
Ferree, M. M., Gamson, W. A., Gerhards, J., & Rucht, D. (2002) · Theory and Society · doi:10.1023/a:1016284431021 · Cited by 469 · see citing works
Survey Article: Deliberation, Democracy, and the Systemic Turn
Owen, D. A., & Smith, G. (2015) · Journal of Political Philosophy · doi:10.1111/jopp.12054 · Cited by 453 · see citing works
Smyth, B., & McClave, P. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44593-5_25 · Cited by 428 · see citing works
Burst of the Filter Bubble?: Effects of personalization on the diversity of <i>Google News</i>
Haim, M., Graefe, A., & Brosius, H.-B. (2018) · Digital Journalism · doi:10.1080/21670811.2017.1338145 · Cited by 399 · see citing works
Fletcher, R., & Nielsen, R. K. (2017) · Journal of Communication · doi:10.1111/jcom.12315 · Cited by 399 · see citing works
Individual and Routine Forces in Gatekeeping
Shoemaker, P. J., Eichholz, M., Kim, E., & Wrigley, B. (2001) · Journalism & Mass Communication Quarterly · doi:10.1177/107769900107800202 · Cited by 394 · see citing works
Exposure diversity as a design principle for recommender systems
Helberger, N., Karppinen, K., & D’Acunto, L. (2016) · Information Communication & Society · doi:10.1080/1369118x.2016.1271900 · Cited by 368 · see citing works
Self-citationA survey of serendipity in recommender systems
Kotkov, D., Wang, S., & Veijalainen, J. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.08.014 · Cited by 334 · see citing works
Smedt, T. D., & Daelemans, W. M. P. (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2343710 · Cited by 240 · see citing works
Building Large Corpora from the Web Using a New Efficient Tool Chain
Schäfer, R., & Bildhauer, F. (2012) · Language Resources and Evaluation · Cited by 217 · see citing works
Down the (White) Rabbit Hole: The Extreme Right and Online Recommender Systems
O'Callaghan, D., Greene, D., Conway, M., Carthy, J., & Cunningham, P. (2015) · Social Science Computer Review · doi:10.1177/0894439314555329 · Cited by 196 · see citing works
Ricotta, C., & Szeidl, L. (2006) · Theoretical Population Biology · doi:10.1016/j.tpb.2006.06.003 · Cited by 196 · see citing works
Automating the News: How Personalized News Recommender System Design Choices Impact News Reception
Beam, M. A. (2014) · Communication Research · doi:10.1177/0093650213497979 · Cited by 159 · see citing works
Dylko, I. B., Dolgov, I., Hoffman, W. B., Eckhart, N. R., Molina, M. D., & Aaziz, O. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2017.03.031 · Cited by 158 · see citing works
Multiobjective Pareto-Efficient Approaches for Recommender Systems
Ribeiro, M. T., Ziviani, N., de Moura, E. S., et al. (2014) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2629350 · Cited by 135 · see citing works
Karppinen, K. (2013) · Fordham University Press eBooks · doi:10.5422/fordham/9780823245123.001.0001 · Cited by 119 · see citing works
What makes news more multiperspectival? A field analysis
Benson, R. (2009) · Poetics · doi:10.1016/j.poetic.2009.09.002 · Cited by 107 · see citing works
On Competition, Access and Diversity in Media, Old and New
van Cuilenburg, J. J. (1999) · New Media & Society · doi:10.1177/14614449922225555 · Cited by 98 · see citing works
Measuring and Explaining the Diversity of Voices and Viewpoints in the News
Masini, A., Aelst, P. V., Zerback, T., Reinemann, C., Mancini, P., Mazzoni, M., Damiani, M., & Coen, S. (2017) · Journalism Studies · doi:10.1080/1461670x.2017.1343650 · Cited by 82 · see citing works
Understanding the role of latent feature diversification on choice difficulty and satisfaction
Willemsen, M. C., Graus, M. P., & Knijnenburg, B. P. (2016) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-016-9178-6 · Cited by 79 · see citing works
How Technology Encourages Political Selective Exposure
Dylko, I. B. (2015) · Communication Theory · doi:10.1111/comt.12089 · Cited by 72 · see citing works
Media coverage in times of political crisis: A text mining approach
de Fortuny, E. J., Smedt, T. D., Martens, D., et al. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.04.013 · Cited by 53 · see citing works
Ways of Computing Diverse Collaborative Recommendations
Bridge, D., & Kelly, J. P. (2006) · Lecture notes in computer science · doi:10.1007/11768012_6 · Cited by 36 · see citing works
Duncan, J., & Reid, J. (2013) · Communicatio · doi:10.1080/02500167.2013.864448 · Cited by 22 · see citing works
Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource
Tulkens, S., Emmery, C., & Daelemans, W. M. P. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3nghzpdnmazi · Cited by 17 · see citing works
The Limits of Empirical Indicators: Media Pluralism as an Essentially Contested Concept
Karppinen, K. (2015) · Palgrave Macmillan UK eBooks · doi:10.1057/9781137304308_17 · Cited by 11 · see citing works
Napoli, P. M. (2015) · Palgrave Macmillan UK eBooks · doi:10.1057/9781137304308_8 · Cited by 8 · see citing works
Hierarchical Re-estimation of Topic Models for Measuring Topical Diversity
Azarbonyad, H., Dehghani, M., Kenter, T., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-56608-5_6 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Sunstein, C. R. (2007). Republic.com 2.0. Princeton, NJ: Princeton University Press.
Cited by
Sorted by citations
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548 · see citing works
Role of fairness, accountability, and transparency in algorithmic affordance
Shin, D., & Park, Y. J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.04.019 · Cited by 643 · see citing works
Swart, J. (2021) · Social Media + Society · doi:10.1177/20563051211008828 · Cited by 381 · see citing works
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384 · see citing works
Hermann, E. (2022) · New Media & Society · doi:10.1177/14614448211022702 · Cited by 214 · see citing works
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193 · see citing works
A review and provocation: On polarization and platforms
Kreiss, D., & McGregor, S. C. (2024) · New Media & Society · doi:10.1177/14614448231161880 · Cited by 151 · see citing works
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
Bountouridis, D., Harambam, J., Makhortykh, M., Marrero, M., Tintarev, N., & Hauff, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287583 · Cited by 92 · see citing works
Bureaucracy as a Lens for Analyzing and Designing Algorithmic Systems
Pääkkönen, J., Nelimarkka, M., Haapoja, J., & Lampinen, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376780 · Cited by 35 · see citing works
- Authors
- Möller, J., Trilling, D., Helberger, N., & van Es, B.
- Affiliations
- University of Amsterdam (NL)
- eScience Center, Amsterdam, Netherlands
- Corresponding author
- Möller, Judith
- Venue
- Information, Communication & Society
- Publisher
- Informa UK Limited
- Published
- 2018-03
- Volume
- 21
- Issue
- 7
- Pages
- 959-977
- ISSN
- 1369-118X (ISSN-L)
- 1468-4462
- OpenAlex
- W2790980374
- Cited by
- 387
- References
- 45
- Citations by year
- 2026: 26 · 2025: 31 · 2024: 40 · 2023: 65 · 2022: 62 · 2021: 69 · 2020: 58 · 2019: 29 · 2018: 7
- FWCI
- 65.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Information Communication & Society (publishedVersion, cc-by-nc-nd)
- UvA-DARE (University of Amsterdam) (submittedVersion, cc-by-nc-nd)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Funders
- European Research Council (ERC-grant 638514)
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Sociology and Political Science › Media Influence and Politics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- recommender systems, content diversity, filter bubbles, topic diversity, user history, newspaper articles
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
