The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Calibrated recommendations
Open accessAbstract
When a user has watched, say, 70 romance movies and 30 action movies, then it is reasonable to expect the personalized list of recommended movies to be comprised of about 70% romance and 30% action movies as well. This important property is known as calibration, and recently received renewed attention in the context of fairness in machine learning. In the recommended list of items, calibration ensures that the various (past) areas of interest of a user are reflected with their corresponding proportions. Calibration is especially important in light of the fact that recommender systems optimized toward accuracy (e.g., ranking metrics) in the usual offline-setting can easily lead to recommendations where the lesser interests of a user get crowded out by the user's main interests-which we show empirically as well as in thought-experiments. This can be prevented by calibrated recommendations. To this end, we outline metrics for quantifying the degree of calibration, as well as a simple yet effective re-ranking algorithm for post-processing the output of recommender systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
An analysis of approximations for maximizing submodular set functions—I
Nemhauser, G. L., Wolsey, L. A., & Fisher, M. L. (1978) · Mathematical Programming · doi:10.1007/bf01588971 · Cited by 4,412
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 3,907
Collaborative Filtering for Implicit Feedback Datasets
Hu, Y., Koren, Y., & Volinsky, C. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.22 · Cited by 3,170
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“Various fairness criteria have been proposed in the literature, including calibration , equal(ized) odds , equal opportunity , and statistical parity [12, 16, 33].”
BackgroundThe use of MMR, diversity-based reranking for reordering documents and producing summaries
Carbonell, J., & Goldstein, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291025 · Cited by 2,329
The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
Carbonell, J., & Goldstein, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130369 · Cited by 2,156
Improving recommendation lists through topic diversification
Ziegler, C.-N., McNee, S. M., Konstan, J. A., & Lausen, G. (2005) · Proceedings of the 14th international conference on World Wide Web - WWW '05 · doi:10.1145/1060745.1060754 · Cited by 1,892
Evaluating Recommendation Systems
Shani, G., & Gunawardana, A. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_8 · Cited by 1,325
Bennett, J. R., & Lanning, S. (2007) · Cited by 1,216
One-Class Collaborative Filtering
Pan, R., Zhou, Y., Cao, B., Liu, N. N., Lukose, R., Scholz, M., & Yang, Q. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.16 · Cited by 1,015
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
An analysis of approximations for maximizing submodular set functions - 1Preprint
Fisher, M. L., Nemhauser, G. L., & Wolsey, L. A. (1978) · RePEc: Research Papers in Economics · Cited by 837
Expected reciprocal rank for graded relevance
Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009) · Proceedings of the 18th ACM conference on Information and knowledge management · doi:10.1145/1645953.1646033 · Cited by 832
(2001) · Cited by 683
Avoiding monotony: improving the diversity of recommendation lists
Zhang, M., & Hurley, N. (2008) · Proceedings of the 2008 ACM conference on Recommender systems · doi:10.1145/1454008.1454030 · Cited by 456
Exploiting query reformulations for web search result diversification
Santos, R. L. T., Macdonald, C., & Ounis, I. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772780 · Cited by 441
Training and testing of recommender systems on data missing not at random
Steck, H. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835895 · Cited by 365
Self-citationNovelty and Diversity in Top-N Recommendation -- Analysis and Evaluation
Hurley, N., & Zhang, M. (2011) · ACM Transactions on Internet Technology · doi:10.1145/1944339.1944341 · Cited by 312
Inherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Foster, D. (1998) · Biometrika · doi:10.1093/biomet/85.2.379 · Cited by 204
Diversity by proportionality: an election-based approach to search result diversification
Dang, V., & Croft, W. B. (2012) · Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2348283.2348296 · Cited by 189
Learning Non-Discriminatory Predictors
Woodworth, B., Gunasekar, S., Ohannessian, M. I., & Srebro, N. (2017) · arXiv · doi:10.48550/arxiv.1702.06081 · Cited by 182
Coverage, redundancy and size-awareness in genre diversity for recommender systems
Vargas, S., Baltrunas, L., Karatzoglou, A., & Castells, P. (2014) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2645710.2645743 · Cited by 174
Beyond Parity: Fairness Objectives for Collaborative Filtering
Yao, S., & Huang, B. (2017) · arXiv · doi:10.48550/arxiv.1705.08804 · Cited by 169
Explicit relevance models in intent-oriented information retrieval diversification
Vargas, S., Castells, P., & Vallet, D. (2012) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2348283.2348297 · Cited by 79
Optimal greedy diversity for recommendation
Ashkan, A., Kveton, B., Berkovsky, S., & Wen, Z. (2015) · Cited by 78
Promoting diversity in recommendation by entropy regularizer
Qin, L., & Zhu, X. (2013) · International Joint Conference on Artificial Intelligence · Cited by 74
Adaptive, Personalized Diversity for Visual Discovery
Teo, C. H., Nassif, H., Hill, D., Srinivasan, S., Goodman, M., Mohan, V., & Vishwanathan, S. V. N. (2016) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2959100.2959171 · Cited by 62
Bayesian Low-Rank Determinantal Point Processes
Gartrell, M., Paquet, U., & Koenigstein, N. (2016) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2959100.2959178 · Cited by 58
A survey on measuring indirect discrimination in machine learning
Zliobaite, I. (2015) · arXiv · doi:10.48550/arxiv.1511.00148
Cited by
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
Causal Intervention for Leveraging Popularity Bias in Recommendation
Zhang, Y., Feng, F., He, X., Wei, T., Song, C., Ling, G., & Zhang, Y. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462875 · Cited by 443
A Survey on the Fairness of Recommender Systems
Wang, Y., Ma, W., Zhang, M., Liu, Y., & Ma, S. (2023) · ACM Transactions on Information Systems · doi:10.1145/3547333 · Cited by 356
The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation
Abdollahpouri, H., Mansoury, M., Burke, R., & Mobasher, B. (2020) · Fourteenth ACM Conference on Recommender Systems · doi:10.1145/3383313.3418487 · Cited by 183
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646
Metadata
- Authors
- Steck, H.
- Venue
- Proceedings of the 12th ACM Conference on Recommender Systems
- Publisher
- ACM
- Published
- 2018-09-27
- Pages
- 154-162
- OpenAlex
- W2911936302
- Cited by
- 301
- References
- 33
- FWCI
- 15.55
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.