The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
On Sampled Metrics for Item Recommendation
Open accessAbstract
The task of item recommendation requires ranking a large catalogue of items given a context. Item recommendation algorithms are evaluated using ranking metrics that depend on the positions of relevant items. To speed up the computation of metrics, recent work often uses sampled metrics where only a smaller set of random items and the relevant items are ranked. This paper investigates sampled metrics in more detail and shows that they are inconsistent with their exact version, in the sense that they do not persist relative statements, e.g., recommender A is better than B, not even in expectation. Moreover, the smaller the sampling size, the less difference there is between metrics, and for very small sampling size, all metrics collapse to the AUC metric. We show that it is possible to improve the quality of the sampled metrics by applying a correction, obtained by minimizing different criteria such as bias or mean squared error. We conclude with an empirical evaluation of the naive sampled metrics and their corrected variants. To summarize, our work suggests that sampling should be avoided for metric calculation, however if an experimental study needs to sample, the proposed corrections can improve the quality of the estimate.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135
Neural Collaborative Filtering
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., & Chua, T.-S. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052569 · Cited by 6,652
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 3,907
Collaborative Filtering for Implicit Feedback Datasets
Hu, Y., Koren, Y., & Volinsky, C. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.22 · Cited by 3,170
Statistical Inference under Order Restrictions (The Theory and Application of Isotonic Regression)
Vincze, I., Barlow, R. E., Bartholomew, D. J., Bremner, J. M., & Brunk, H. D. (1973) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1402630 · Cited by 842
Explainable Reasoning over Knowledge Graphs for Recommendation
Wang, X., Wang, D., Xu, C., He, X., Cao, Y., & Chua, T.-S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015329 · Cited by 785
Collaborative Memory Network for Recommendation Systems
Ebesu, T., Shen, B., & Fang, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3209991 · Cited by 275
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model
Bengio, Y., & Senecal, J.-S. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2007.912312 · Cited by 223
A Generic Coordinate Descent Framework for Learning from Implicit Feedback
Bayer, I., He, X., Kanagal, B., & Rendle, S. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052694 · Cited by 223
Self-citationUnbiased offline recommender evaluation for missing-not-at-random implicit feedback
Yang, L., Cui, Y., Xuan, Y., Wang, C., Belongie, S., & Estrin, D. (2018) · Proceedings of the 12th ACM Conference on Recommender Systems · doi:10.1145/3240323.3240355 · Cited by 207
Efficient top-n recommendation for very large scale binary rated datasets
Aiolli, F. (2013) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2507157.2507189 · Cited by 117
Selection of Negative Samples for One-class Matrix Factorization
Yu, H., Bilenko, M., & Lin, C. (2017) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611974973.41 · Cited by 72
OpenRec: A Modular Framework for Extensible and Adaptable Recommendation Algorithms
Yang, L., Bagdasaryan, E., Gruenstein, J., Hsieh, C.-K., & Estrin, D. L. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159681 · Cited by 64
Cited by
Contrastive Learning for Sequential Recommendation
Xie, X., Sun, F., Liu, Z., Wu, S., Gao, J., Zhang, J., Ding, B., & Cui, B. (2022) · 2022 IEEE 38th International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00099 · Cited by 663
A Survey of Recommendation Systems: Recommendation Models, Techniques, and Application Fields
Ko, H., Lee, S., Park, Y., & Choi, A. (2022) · Electronics · doi:10.3390/electronics11010141 · Cited by 658
Learning Intents behind Interactions with Knowledge Graph for Recommendation
Wang, X., Huang, T., Wang, D., Yuan, Y., Liu, Z., He, X., & Chua, T.-S. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450133 · Cited by 578
Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
Qiu, R., Huang, Z., Yin, H., & Wang, Z. (2022) · Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3488560.3498433 · Cited by 460
Wu, L., He, X., Wang, X., Zhang, K., & Wang, M. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3145690 · Cited by 413
Metadata
- Authors
- Krichene, W., & Rendle, S.
- Venue
- Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2020-08-20
- Pages
- 1748-1757
- OpenAlex
- W3081170586
- Cited by
- 495
- References
- 14
- FWCI
- 45.84
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.