Evaluating Collaborative Filtering Recommender Algorithms: A Survey
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Due to the explosion of available information on the Internet, the need for effective means of accessing and processing them has become vital for everyone. Recommender systems have been developed to help users to find what they may be interested in and business owners to sell their products more efficiently. They have found much attention in both academia and industry. A recommender algorithm takes into account user–item interactions, i.e., rating (or purchase) history of users on items, and their contextual information, if available. It then provides a list of potential items for each target user, such that the user is likely to positively rate (or purchase) them. In this paper, we review evaluation metrics used to assess performance of recommendation algorithms. We also survey a number of classical and modern recommendation algorithms and compare their performance in terms of different evaluation metrics on five benchmark datasets. Our experiments show that there is no golden recommendation algorithm showing the best performance in all evaluation metrics. We also find large variability across the datasets. This indicates that one should carefully consider the evaluation criteria in choosing a recommendation algorithm for a particular application.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135 · cited 31×
“COLLABORATIVE FILTERING RECOMMENDATION ALGORITHMS Collaborative Filtering (CF) RSs consider item- or user-based similarities and extract the list of recommendations based 74004 VOLUME 6, 2018 on them.”
BackgroundMethodInfluentialThe proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039
Evaluating collaborative filtering recommender systems
Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004) · ACM Transactions on Information Systems · doi:10.1145/963770.963772 · Cited by 5,798 · cited 10×
“These metrics are mostly useful where the predicted ratings are shown to the users of system [4].”
BackgroundMethodInfluentialGroupLens: an open architecture for collaborative filtering of netnews
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., & Riedl, J. (1994) · Proceedings of the 1994 ACM conference on Computer supported cooperative work - CSCW '94 · doi:10.1145/192844.192905 · Cited by 5,058
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
Fab: content-based, collaborative recommendation
Balabanović, M., & Shoham, Y. (1997) · Communications of the ACM · doi:10.1145/245108.245124 · Cited by 2,934
Context-Aware Recommender Systems
Adomavicius, G., & Tuzhilin, A. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_7 · Cited by 1,102
Critiquing-based recommenders: survey and emerging trends
Chen, L., & Pu, P. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9108-6 · Cited by 281 · cited 1×
“The quality of RSs can be enhanced by various approaches; for example, by better knowing users and items [43], [49]–[51] through incorporating features of their profiles such as demographic information or keywords [31], [33], [52], [53] or using data mining techniques [31], [33], [52], [54], utilizing additional conte…”
Method
Metadata
- Authors
- Jalili, M., Ahmadian, S., Izadi, M., Moradi, P., & Salehi, M.
- Venue
- IEEE Access
- Published
- 2018
- Volume
- 6
- Pages
- 74003-74024
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2907935579
- Cited by
- 172
- References
- 8
- FWCI
- 28.52
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.