Should I Follow the Crowd?: A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
The use of IR methodology in the evaluation of recommender systems has become common practice in recent years. IR metrics have been found however to be strongly biased towards rewarding algorithms that recommend popular items "the same bias that state of the art recommendation algorithms display. Recent research has confirmed and measured such biases, and proposed methods to avoid them. The fundamental question remains open though whether popularity is really a bias we should avoid or not; whether it could be a useful and reliable signal in recommendation, or it may be unfairly rewarded by the experimental biases. We address this question at a formal level by identifying and modeling the conditions that can determine the answer, in terms of dependencies between key random variables, involving item rating, discovery and relevance. We find conditions that guarantee popularity to be effective or quite the opposite, and for the measured metric values to reflect a true effectiveness, or qualitatively deviate from it. We exemplify and confirm the theoretical findings with empirical results. We build a crowdsourced dataset devoid of the usual biases displayed by common publicly available data, in which we illustrate contradictions between the accuracy that would be measured in a common biased offline experimental setting, and the actual accuracy that can be measured with unbiased observations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · cited 1×
“The effectiveness of majority taste makes indeed statistical sense: the items that many people like (according to the records of observed user activity) are liked by many people (in test data for evaluation) [19].”
BackgroundThe MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 6,627
Social Influence: Compliance and Conformity
Cialdini, R. B., & Goldstein, N. J. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142015 · Cited by 6,077 · cited 1×
“Crowd dynamics are moreover known to be exposed to external and internal influence and bias factors [26,27,29], such as mass media [7], marketing, opinion management [6], algorithmic bias [28], or social conformity [13].”
BackgroundAmazon.com recommendations: item-to-item collaborative filtering
Linden, G., Smith, B., & York, J. (2003) · IEEE Internet Computing · doi:10.1109/mic.2003.1167344 · Cited by 5,403
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 1×
“Recent studies show that majority formation involves a degree of chance, by which different outcomes are possible as to what choices make it to the top of popularity [31].”
BackgroundFILTER BUBBLE: Wie wir im Internet entmündigt werden
Pariser, E. (2012) · doi:10.3139/9783446431164 · Cited by 1,849
The filter bubble: how the new personalized web is changing what we read and how we think
(2012) · Choice Reviews Online · doi:10.5860/choice.50-0926 · Cited by 1,319
THE PROBABILITY RANKING PRINCIPLE IN IR
ROBERTSON, S. E. (1977) · Journal of Documentation · doi:10.1108/eb026647 · Cited by 1,103
Social Influence and the Collective Dynamics of Opinion Formation
Moussaïd, M., Kämmer, J. E., Analytis, P. P., & Neth, H. (2013) · PLoS ONE · doi:10.1371/journal.pone.0078433 · Cited by 407
Cited by
Bias and Debias in Recommender System: A Survey and Future Directions
Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., & He, X. (2023) · ACM Transactions on Information Systems · doi:10.1145/3564284 · Cited by 741
“SIGIR even presents the Best Paper award to the paper on this topic in 2018 [1] and 2020 [2], respectively.”
BackgroundCausal Intervention for Leveraging Popularity Bias in Recommendation
Zhang, Y., Feng, F., He, X., Wei, T., Song, C., Ling, G., & Zhang, Y. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462875 · Cited by 443
Disentangling User Interest and Conformity for Recommendation with Causal Embedding
Zheng, Y., Gao, C., Li, X., He, X., Li, Y., & Jin, D. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3449788 · Cited by 356
“Existing causal solutions for recommender systems formulate the problem as eliminating popularity bias, from the perspective of items [1, 3, 5, 11, 12, 24, 35, 43].”
BackgroundModel-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
Wei, T., Feng, F., Chen, J., Wu, Z., Yi, J., & He, X. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467289 · Cited by 274
Popularity Bias is not Always Evil: Disentangling Benign and Harmful Bias for Recommendation
Zhao, Z., Chen, J., Zhou, S., He, X., Cao, X., Zhang, F., & Wu, W. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3218994 · Cited by 117
“For example, some work [7], [18] argues that such skewed distribution may be caused by user conformity, deviating from reflecting”
BackgroundHuang, J., Oosterhuis, H., de Rijke, M., & van Hoof, H. (2020) · Fourteenth ACM Conference on Recommender Systems · doi:10.1145/3383313.3412252 · Cited by 64
“These biased policies may result in detrimental performance if exposed to actual users [3, 19, 24, 28].”
Background
Metadata
- Authors
- Cañamares, R., & Castells, P.
- Venue
- The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
- Publisher
- ACM
- Published
- 2018-06-27
- Pages
- 415-424
- OpenAlex
- W2799048248
- Cited by
- 127
- References
- 9
- FWCI
- 12.16
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.