The DOI matches its Crossref record; 181 of the 181 references it cites were found in a registry or an index.
Evaluating Recommender Systems: Survey and Framework
Open accessAbstract
The comprehensive evaluation of the performance of a recommender system is a complex endeavor: many facets need to be considered in configuring an adequate and effective evaluation setting. Such facets include, for instance, defining the specific goals of the evaluation, choosing an evaluation method, underlying data, and suitable evaluation metrics. In this article, we consolidate and systematically organize this dispersed knowledge on recommender systems evaluation. We introduce the Framework for Evaluating Recommender systems (FEVR), which we derive from the discourse on recommender systems evaluation. In FEVR, we categorize the evaluation space of recommender systems evaluation. We postulate that the comprehensive evaluation of a recommender system frequently requires considering multiple facets and perspectives in the evaluation. The FEVR framework provides a structured foundation to adopt adequate evaluation configurations that encompass this required multi-facetedness and provides the basis to advance in the field. We outline and discuss the challenges of a comprehensive evaluation of recommender systems and provide an outlook on what we need to embrace and do to move forward as a research community.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Matrix Factorization Techniques for Recommender Systems
Koren, Y., Bell, R., & Volinsky, C. (2009) · Computer · doi:10.1109/mc.2009.263 · Cited by 11,673
Adomavicius, G., & Tuzhilin, A. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.99 · Cited by 10,310
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135
A Design Science Research Methodology for Information Systems Research
Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007) · Journal of Management Information Systems · doi:10.2753/mis0742-1222240302 · Cited by 7,832 · cited 1×
“At the heart of any evaluation activity is the comparison of the objectives (target performance) to the observed results (actual performance) [170].”
ResultAttitude-behavior relations: A theoretical analysis and review of empirical research
Ajzen, I., & Fishbein, M. (1977) · Psychological Bulletin · doi:10.1037/0033-2909.84.5.888 · Cited by 7,611 · cited 1×
“However, self-reports may not be consistent with user behavior [43, 141, 211] because the link between an individual’s attitude and behavior is generally not very strong [7].”
Neural Collaborative Filtering
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., & Chua, T.-S. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052569 · Cited by 6,652
Evaluating collaborative filtering recommender systems
Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004) · ACM Transactions on Information Systems · doi:10.1145/963770.963772 · Cited by 5,798
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841
LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., & Wang, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401063 · Cited by 4,430
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 3,907
Hybrid Recommender Systems: Survey and Experiments
Burke, R. (2002) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1021240730564 · Cited by 3,871
Resnick, P., & Varian, H. R. (1997) · Communications of the ACM · doi:10.1145/245108.245121 · Cited by 3,772
Graph Convolutional Neural Networks for Web-Scale Recommender Systems
Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W. L., & Leskovec, J. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219890 · Cited by 3,733
A Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,602
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Collaborative Filtering for Implicit Feedback Datasets
Hu, Y., Koren, Y., & Volinsky, C. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.22 · Cited by 3,170
Information bias in health research: definition, pitfalls, and adjustment methods
Althubaiti, A. (2016) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s104807 · Cited by 3,018
Social information filtering: algorithms for automating “word of mouth”
Shardanand, U., & Maes, P. (1995) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '95 · doi:10.1145/223904.223931 · Cited by 2,823
Content-Based Recommendation Systems
Pazzani, M. J., & Billsus, D. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-72079-9_10 · Cited by 2,600
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562
Collaborative Filtering Recommender Systems
Schafer, J. B., Frankowski, D., Herlocker, J., & Sen, S. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-72079-9_9 · Cited by 2,120
Improving recommendation lists through topic diversification
Ziegler, C.-N., McNee, S. M., Konstan, J. A., & Lausen, G. (2005) · Proceedings of the 14th international conference on World Wide Web - WWW '05 · doi:10.1145/1060745.1060754 · Cited by 1,892
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 1,511
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
The Netflix Recommender System: Algorithms, Business Value, and Innovation
Gomez-Uribe, C. A., & Hunt, N. (2016) · ACM Transactions on Management Information Systems · doi:10.1145/2843948 · Cited by 1,418
SoRec: social recommendation using probabilistic matrix factorization
Ma, H., Yang, H., Lyu, M. R., & King, I. (2008) · Proceedings of the 17th ACM conference on Information and knowledge management · doi:10.1145/1458082.1458205 · Cited by 1,401
A method for assessing the quality of a randomized control trial
Chalmers, T. C., Smith, H., Blackburn, B., Silverman, B., Schroeder, B., Reitman, D., & Ambroz, A. (1981) · Controlled Clinical Trials · doi:10.1016/0197-2456(81)90056-8 · Cited by 1,349
Variational Autoencoders for Collaborative Filtering
Liang, D., Krishnan, R. G., Hoffman, M. D., & Jebara, T. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186150 · Cited by 1,338
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 2×
“Reproducibility is a minimum necessary condition for a inding to be believable and informative” [91].”
BackgroundResultParticipant Observation as a Data Collection Method
Kawulich, B. (2008) · Forum qualitative Sozialforschung · doi:10.17169/fqs-6.2.466 · Cited by 1,215
The YouTube video recommendation system
Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., Gupta, S., He, Y., Lambert, M., Livingston, B., et al. (2010) · Proceedings of the fourth ACM conference on Recommender systems · doi:10.1145/1864708.1864770 · Cited by 1,191
Factors relevant to the validity of experiments in social settings.
Campbell, D. T. (1957) · Psychological Bulletin · doi:10.1037/h0040950 · Cited by 1,140
E-Commerce Product Recommendation Agents: Use, Characteristics, and Impact1
Xiao, B., & Benbasat, I. (2007) · MIS Quarterly · doi:10.2307/25148784 · Cited by 1,081
Joint Deep Modeling of Users and Items Using Reviews for Recommendation
Zheng, L., Noroozi, V., & Yu, P. S. (2017) · Proceedings of the Tenth ACM International Conference on Web Search and Data Mining · doi:10.1145/3018661.3018665 · Cited by 1,075
An Agenda for Purely Confirmatory Research
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463078 · Cited by 1,042 · cited 2×
“, observations or theory) [217] and where the factors measured (and the instruments used for measuring these factors) are carefully selected may help determine whether an intervention was effective [42]; explaining presumed causal links in real-world interventions is often too complex for experimental methods.”
BackgroundxDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems
Lian, J., Zhou, X., Zhang, F., Chen, Z., Xie, X., & Sun, G. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220023 · Cited by 1,040
Collaborative Filtering Recommender Systems
Ekstrand, M. D. (2011) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000009 · Cited by 1,037
Solving the apparent diversity-accuracy dilemma of recommender systems
Zhou, T., Kuscsik, Z., Liu, J.-G., Medo, M., Wakeling, J. R., & Zhang, Y.-C. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1000488107 · Cited by 972
Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI 2010)
Adams, R. P., Dahl, G. E., & Murray, I. (2010) · Cited by 966
Recommender Systems: An Introduction
Jannach, D., Zanker, M., Felfernig, A., & Friedrich, G. (2010) · doi:10.1017/cbo9780511763113 · Cited by 886
A user-centric evaluation framework for recommender systems
Pu, P., Chen, L., & Hu, R. (2011) · Proceedings of the fifth ACM conference on Recommender systems · doi:10.1145/2043932.2043962 · Cited by 850
On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach
Salzberg, S. L. (1997) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009752403260 · Cited by 827
Research-paper recommender systems: a literature survey
Beel, J., Gipp, B., Langer, S., & Breitinger, C. (2016) · International Journal on Digital Libraries · doi:10.1007/s00799-015-0156-0 · Cited by 823
The validity of self-report measures of proenvironmental behavior: A meta-analytic review
Kormos, C., & Gifford, R. (2014) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2014.09.003 · Cited by 806
Explaining the user experience of recommender systems
Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9118-4 · Cited by 799
Recommender systems: from algorithms to user experience
Konstan, J. A., & Riedl, J. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9112-x · Cited by 762
Rank and relevance in novelty and diversity metrics for recommender systems
Vargas, S., & Castells, P. (2011) · Proceedings of the fifth ACM conference on Recommender systems · doi:10.1145/2043932.2043955 · Cited by 709
Controlled experiments on the web: survey and practical guide
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-008-0114-1 · Cited by 690
Beyond accuracy: evaluating recommender systems by coverage and serendipity
Ge, M., Delgado-Battenfeld, C., & Jannach, D. (2010) · Proceedings of the fourth ACM conference on Recommender systems · doi:10.1145/1864708.1864761 · Cited by 663
DRN: A Deep Reinforcement Learning Framework for News Recommendation
Zheng, G., Zhang, F., Zheng, Z., Xiang, Y., Yuan, N. J., Xie, X., & Li, Z. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3185994 · Cited by 630
Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks
Quadrana, M., Karatzoglou, A., Hidasi, B., & Cremonesi, P. (2017) · Proceedings of the Eleventh ACM Conference on Recommender Systems · doi:10.1145/3109859.3109896 · Cited by 591
Fairness of Exposure in Rankings
Singh, A., & Joachims, T. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220088 · Cited by 591
Under what conditions does theory obstruct research progress?
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.216 · Cited by 574
The long tail : how endless choice is creating unlimited demand
Anderson, C. (2006) · Cited by 554
A Survey on Conversational Recommender Systems
Jannach, D., Manzoor, A., Cai, W., & Chen, L. (2022) · ACM Computing Surveys · doi:10.1145/3453154 · Cited by 521
Diversity in recommender systems – A survey
Kunaver, M., & Požrl, T. (2017) · Knowledge-Based Systems · doi:10.1016/j.knosys.2017.02.009 · Cited by 504
Sequence-Aware Recommender Systems
Quadrana, M., Cremonesi, P., & Jannach, D. (2019) · ACM Computing Surveys · doi:10.1145/3190616 · Cited by 504
Is seeing believing?: how recommender system interfaces affect users' opinions
Cosley, D., Lam, S. K., Albert, I., Konstan, J. A., & Riedl, J. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642713 · Cited by 467
Kaminskas, M., & Bridge, D. (2017) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2926720 · Cited by 465
Proceedings of the 13th International Society for Music Information Retrieval Conference, ISMIR 2012
Gouyon, F., Herrera, P., Martins, L. G., & Müller, M. A. (2012) · Cited by 446
The long tail of recommender systems and how to leverage it
Park, Y.-J., & Tuzhilin, A. (2008) · Proceedings of the 2008 ACM conference on Recommender systems · doi:10.1145/1454008.1454012 · Cited by 436
Online controlled experiments at large scale
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2488217 · Cited by 433
Are we really making much progress? A worrying analysis of recent neural recommendation approaches
Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019) · Proceedings of the 13th ACM Conference on Recommender Systems · doi:10.1145/3298689.3347058 · Cited by 431
Smyth, B., & McClave, P. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44593-5_25 · Cited by 422
Towards Conversational Recommender Systems
Christakopoulou, K., Radlinski, F., & Hofmann, K. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939746 · Cited by 412
The Philosophy of Information Retrieval Evaluation
Voorhees, E. M. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45691-0_34 · Cited by 389
Evaluating recommender systems from the user’s perspective: survey of the state of the art
Pu, P., Chen, L., & Hu, R. (2012) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9115-7 · Cited by 377
MyMediaLite: a free recommender system library
Gantner, Z., Rendle, S., Freudenthaler, C., & Schmidt-Thieme, L. (2011) · Proceedings of the fifth ACM conference on Recommender systems · doi:10.1145/2043932.2043989 · Cited by 368
Celma, Ò. (2010) · Springer eBooks · doi:10.1007/978-3-642-13287-2 · Cited by 368
Fairness in Recommendation Ranking through Pairwise Comparisons
Beutel, A., Chen, J., Doshi, T., Qian, H., Wei, L., Wu, Y., Heldt, L., Zhao, Z., Hong, L., Chi, E. H., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330745 · Cited by 362
Equity of Attention: Amortizing Individual Fairness in Rankings
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210063 · Cited by 337
Measurement, Design, and Analysis
Pedhazur, E. J., & Schmelkin, L. P. (2013) · Psychology Press eBooks · doi:10.4324/9780203726389 · Cited by 335
Item popularity and recommendation accuracy
Steck, H. (2011) · Proceedings of the fifth ACM conference on Recommender systems · doi:10.1145/2043932.2043957 · Cited by 330
Miller, S. M., & Landsberger, H. A. (1959) · American Sociological Review · doi:10.2307/2089456 · Cited by 329
Advances in Collaborative Filtering
Koren, Y., Rendle, S., & Bell, R. (2022) · Recommender Systems Handbook · doi:10.1007/978-1-0716-2197-4_3 · Cited by 324
Designing and Evaluating Explanations for Recommender Systems
Tintarev, N., & Masthoff, J. (2011) · Recommender Systems Handbook · doi:10.1007/978-0-387-85820-3_15 · Cited by 322
Multistakeholder recommendation: Survey and research directions
Abdollahpouri, H., Adomavicius, G., Burke, R., Guy, I., Jannach, D., Kamishima, T., Krasnodebski, J., & Pizzato, L. (2020) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-019-09256-1 · Cited by 319
A Survey of Accuracy Evaluation Metrics of Recommendation Tasks
GunawardanaAsela, & ShaniGuy (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1755883 · Cited by 314
Novelty and Diversity in Top-N Recommendation -- Analysis and Evaluation
Hurley, N., & Zhang, M. (2011) · ACM Transactions on Internet Technology · doi:10.1145/1944339.1944341 · Cited by 312
50 Years of Test (Un)fairness: Lessons for Machine Learning
Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287600 · Cited by 297 · cited 1×
“One way to address group fairness from the user perspective is to disaggregate the user-oriented metrics to measure and compare to which extent user groups are provided with lower-quality recommendations (e.g., References [69, 73, 74, 108, 142, 158, 196]).”
BackgroundTasteWeights: a visual interactive hybrid recommender system
Bostandjiev, S., O'Donovan, J., & Höllerer, T. (2012) · Proceedings of the sixth ACM conference on Recommender systems · doi:10.1145/2365952.2365964 · Cited by 274
Surprise: A Python library for recommender systems
Hug, N. (2020) · Journal of Open Source Software · doi:10.21105/joss.02174 · Cited by 264
Semantics-Aware Content-Based Recommender Systems
de Gemmis, M., Lops, P., Musto, C., Narducci, F., & Semeraro, G. (2015) · Recommender Systems Handbook · doi:10.1007/978-1-4899-7637-6_4 · Cited by 258
Evaluation of recommendations: rating-prediction and ranking
Steck, H. (2013) · Proceedings of the 7th ACM conference on Recommender systems · doi:10.1145/2507157.2507160 · Cited by 256
Scalable Coordinate Descent Approaches to Parallel Matrix Factorization for Recommender Systems
Yu, H.-F., Hsieh, C.-J., Si, S., & Dhillon, I. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.168 · Cited by 253
On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected
Adamopoulos, P., & Tuzhilin, A. (2015) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2559952 · Cited by 233
Content-Based Video Recommendation System Based on Stylistic Visual Features
Deldjoo, Y., Elahi, M., Cremonesi, P., Garzotto, F., Piazzolla, P., & Quadrana, M. (2016) · Journal on Data Semantics · doi:10.1007/s13740-016-0060-9 · Cited by 233
Offline A/B Testing for Recommender Systems
Gilotte, A., Calauzènes, C., Nédelec, T., Abraham, A., & Dollé, S. (2018) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3159652.3159687 · Cited by 233
What do usability evaluators do in practice?: an explorative study of think-aloud testing
Nørgaard, M., & Hornbæk, K. (2006) · Proceedings of the 6th conference on Designing Interactive systems · doi:10.1145/1142405.1142439 · Cited by 230
Measuring the Business Value of Recommender Systems
Jannach, D., & Jugovac, M. (2019) · ACM Transactions on Management Information Systems · doi:10.1145/3370082 · Cited by 230
Person-Situation: Segmentation's Missing Link
Dickson, P. R. (1982) · Journal of Marketing · doi:10.1177/002224298204600407 · Cited by 226
Swets, J. A. (1963) · Science · doi:10.1126/science.141.3577.245 · Cited by 219
A new approach to evaluating novel recommendations
Celma, Ò., & Herrera, P. (2008) · Proceedings of the 2008 ACM conference on Recommender systems · doi:10.1145/1454008.1454038 · Cited by 215
Recommender Systems: Techniques, Applications, and Challenges
Ricci, F., Rokach, L., & Shapira, B. (2022) · Recommender Systems Handbook · doi:10.1007/978-1-0716-2197-4_1 · Cited by 214
User perception of differences in recommender algorithms
Ekstrand, M. D., Harper, F. M., Willemsen, M. C., & Konstan, J. A. (2014) · Proceedings of the 8th ACM Conference on Recommender systems · doi:10.1145/2645710.2645737 · Cited by 204
A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
Ferrari Dacrema, M., Boglio, S., Cremonesi, P., & Jannach, D. (2021) · ACM Transactions on Information Systems · doi:10.1145/3434185 · Cited by 199
Comparative recommender system evaluation: benchmarking recommendation frameworks
Said, A., & Bellogín, A. (2014) · Proceedings of the 8th ACM Conference on Recommender systems · doi:10.1145/2645710.2645746 · Cited by 194
Batch learning from logged bandit feedback through counterfactual risk minimization
Swaminathan, A., & Joachims, T. (2015) · Journal of Machine Learning Research · Cited by 193
Recommendations with a Purpose
Jannach, D., & Adomavicius, G. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959186 · Cited by 187
Measuring the impact of online personalisation: Past, present and future
Zanker, M., Rook, L., & Jannach, D. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.06.006 · Cited by 183
Recommender Systems: Issues, Challenges, and Research Opportunities
Khusro, S., Ali, Z., & Ullah, I. (2016) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-10-0557-2_112 · Cited by 182
Digital nudging with recommender systems: Survey and future directions
Jesse, M., & Jannach, D. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2020.100052 · Cited by 180
Offline and online evaluation of news recommender systems at swissinfo.ch
Garcin, F., Faltings, B., Donatsch, O., Alazzawi, A., Bruttin, C., & Huber, A. (2014) · Proceedings of the 8th ACM Conference on Recommender systems · doi:10.1145/2645710.2645745 · Cited by 177
Acquiring and Revising Preferences in a Critique-Based Mobile Recommender System
Ricci, F., & Nguyen, Q. N. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.43 · Cited by 175
Metrics for Evaluating the Serendipity of Recommendation Lists
Murakami, T., Mori, K., & Orihara, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-78197-4_5 · Cited by 174
Evaluating Stochastic Rankings with Expected Exposure
Diaz, F., Mitra, B., Ekstrand, M. D., Biega, A. J., & Carterette, B. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411962 · Cited by 171
Evaluating Recommender Systems with User Experiments
Knijnenburg, B. P., & Willemsen, M. C. (2015) · Springer eBooks · doi:10.1007/978-1-4899-7637-6_9 · Cited by 170
Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit
Ekstrand, M. D., Ludwig, M., Konstan, J. A., & Riedl, J. T. (2011) · Proceedings of the fifth ACM conference on Recommender systems · doi:10.1145/2043932.2043958 · Cited by 169
McInerney, J., Lacker, B., Hansen, S., Higley, K., Bouchard, H., Gruson, A., & Mehrotra, R. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240354 · Cited by 166
Impact of data characteristics on recommender systems performance
Adomavicius, G., & Zhang, J. (2012) · ACM Transactions on Management Information Systems · doi:10.1145/2151163.2151166 · Cited by 164
Rate it again: increasing recommendation accuracy by user re-rating
Amatriain, X., Pujol, J. M., Tintarev, N., & Oliver, N. (2009) · Proceedings of the third ACM conference on Recommender systems · doi:10.1145/1639714.1639744 · Cited by 163
Beel, J., Genzmehr, M., Langer, S., Nürnberger, A., & Gipp, B. (2013) · Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation · doi:10.1145/2532508.2532511 · Cited by 161
Chao, Y.-L., & Lam, S.-P. (2011) · Environment and Behavior · doi:10.1177/0013916509350849 · Cited by 160
Research paper recommender system evaluation: a quantitative literature survey
Beel, J., Langer, S., Genzmehr, M., Gipp, B., Breitinger, C., & Nürnberger, A. (2013) · Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation · doi:10.1145/2532508.2532512 · Cited by 159
Seven pitfalls to avoid when running controlled experiments on the web
Crook, T., Frasca, B., Kohavi, R., & Longbotham, R. (2009) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1557019.1557139 · Cited by 157
McNee, S. M., Riedl, J., & Konstan, J. A. (2006) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1125451.1125660 · Cited by 148
Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
Anelli, V. W., Bellogín, A., Ferrara, A., Malitesta, D., Merra, F. A., Pomo, C., Donini, F. M., & Noia, T. D. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463245 · Cited by 148
Using Recommendation Agents to Cope with Information Overload
Aljukhadar, M., Sénécal, S., & Daoust, C.-E. (2012) · International Journal of Electronic Commerce · doi:10.2753/jec1086-4415170202 · Cited by 137
Knijnenburg, B. P., Reijmer, N. J. M., & Willemsen, M. C. (2011) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2043932.2043960 · Cited by 132
User-centric evaluation of a K-furthest neighbor collaborative filtering recommender algorithm
Said, A., Fields, B., Jain, B. J., & Albayrak, Ş. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441933 · Cited by 130
Evaluating Recommender Systems
Gunawardana, A., Shani, G., & Yogev, S. (2012) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_15 · Cited by 124
Why Psychology Cannot be an Empirical Science
Smedslund, J. (2015) · Integrative Psychological and Behavioral Science · doi:10.1007/s12124-015-9339-x · Cited by 116
A General Framework for Counterfactual Learning-to-Rank
Agarwal, A., Takatsu, K., Zaitsev, I., & Joachims, T. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331202 · Cited by 115
Dimensions and Metrics for Evaluating Recommendation Systems
Avazpour, I., Pitakrat, T., Grunske, L., & Grundy, J. (2013) · Springer eBooks · doi:10.1007/978-3-642-45135-5_10 · Cited by 114
Recommendations in Signed Social Networks
Tang, J., Aggarwal, C. C., & Liu, H. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2882971 · Cited by 113
Learning Distributed Representations from Reviews for Collaborative Filtering
Almahairi, A., Kastner, K., Cho, K., & Courville, A. C. (2015) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2792838.2800192 · Cited by 110
Beel, J., & Langer, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24592-8_12 · Cited by 109
Parra, D., & Amatriain, X. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22362-4_22 · Cited by 107
Contrasting Offline and Online Results when Evaluating Recommendation Algorithms
Rossetti, M., Stella, F. A., & Zanker, M. (2016) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2959100.2959176 · Cited by 106
An Evaluation Methodology for Collaborative Recommender Systems
Cremonesi, P., Turrin, R., Lentini, E., & Matteucci, M. (2008) · International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution · doi:10.1109/axmedis.2008.13 · Cited by 100
Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems
Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A. V., & Turrin, R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23765-2_11 · Cited by 99
LensKit for Python: Next-Generation Software for Recommender Systems Experiments
Ekstrand, M. D. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412778 · Cited by 99
Escaping the McNamara Fallacy: Toward More Impactful Recommender Systems Research
Jannach, D., & Bauer, C. (2020) · AI Magazine · doi:10.1609/aimag.v41i4.5312 · Cited by 97
Self-citationUnifying explicit and implicit feedback for collaborative filtering
Liu, N. N., Xiang, E. W., Zhao, M., et al. (2010) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1871437.1871643 · Cited by 95
Break the Loop: Gender Imbalance in Music Recommenders
Ferraro, A., Serra, X., & Bauer, C. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446033 · Cited by 92
Self-citationQuantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317595 · Cited by 92
Returning is Believing: Optimizing Long-term User Engagement in Recommender Systems
Wu, Q., Wang, H., Hong, L., & Shi, Y. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133025 · Cited by 92
Recommender Systems in Computer Science and Information Systems – A Landscape of Research
Jannach, D., Zanker, M., Ge, M., & Gröning, M. (2012) · Lecture notes in business information processing · doi:10.1007/978-3-642-32273-0_7 · Cited by 90
Offline evaluation options for recommender systems
Cañamares, R., Castells, P., & Moffat, A. (2020) · Information Retrieval · doi:10.1007/s10791-020-09371-3 · Cited by 90
Matrix co-factorization for recommendation with rich side information and implicit feedback
Fang, Y., & Si, L. (2011) · doi:10.1145/2039320.2039330 · Cited by 88
Updatable, Accurate, Diverse, and Scalable Recommendations for Interactive Applications
Paudel, B., Christoffel, F., Newell, C., & Bernstein, A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2955101 · Cited by 87
Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
Tamm, Y.-M., Damdinov, R., & Vasilev, A. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3478848 · Cited by 85
Performance Evaluation of Recommender Systems
Chen, M. (2017) · International Journal of Performability Engineering · doi:10.23940/ijpe.17.08.p7.12461256 · Cited by 80
Auditing Search Engines for Differential Satisfaction Across Demographics
Mehrotra, R., Anderson, A., Díaz, F., Sharma, A. K., Wallach, H., & Yılmaz, E. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054197 · Cited by 77
Improving accountability in recommender systems research through reproducibility
Bellogín, A., & Said, A. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-021-09302-x · Cited by 77
Recommender Systems: Sources of Knowledge and Evaluation Metrics
Parra, D., & Sahebi, S. (2012) · Studies in computational intelligence · doi:10.1007/978-3-642-33326-2_7 · Cited by 74
Exploring author gender in book rating and recommendation
Ekstrand, M. D., & Kluver, D. (2021) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-020-09284-2 · Cited by 72
Context-Aware Recommender Systems: From Foundations to Recent Developments
Adomavičius, G., Bauman, K., Tuzhilin, A., & Unger, M. (2021) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_6 · Cited by 72
A pragmatic procedure to support the user-centric evaluation of recommender systems
Knijnenburg, B. P., Willemsen, M. C., & Kobsa, A. (2011) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2043932.2043993 · Cited by 71
A meta‐analytic synthesis of the question–behavior effect
Spangenberg, E. R., Kareklas, I., Devezer, B., & Sprott, D. E. (2015) · Journal of Consumer Psychology · doi:10.1016/j.jcps.2015.12.004 · Cited by 71
Fairness in Music Recommender Systems: A Stakeholder-Centered Mini Review
Dinnissen, K., & Bauer, C. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.913608 · Cited by 69
Self-citationAccounting for taste: using profile similarity to improve recommender systems
Bonhard, P., Harries, C., McCarthy, J. D., & Sasse, M. A. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124930 · Cited by 62
LensKit for Python: Next-Generation Software for Recommender Systems Experiments
Ekstrand, M. D. (2020) · ScholarWorks (Boise State University) · Cited by 62
User Perception of Next-Track Music Recommendations
Kamehkhosh, I., & Jannach, D. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3079628.3079668 · Cited by 59
Schedl, M., Brandl, S., Lesota, O., Parada‐Cabaleiro, E., Penz, D., & Rekabsaz, N. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505791 · Cited by 54
Semantics and Content-Based Recommendations
Musto, C., de Gemmis, M., Lops, P., Narducci, F., & Semeraro, G. M. (2012) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_7 · Cited by 52
Support the underground: characteristics of beyond-mainstream music listeners
Kowald, D., Müllner, P., Zangerle, E., Bauer, C., Schedl, M., & Lex, E. (2021) · EPJ Data Science · doi:10.1140/epjds/s13688-021-00268-9 · Cited by 51
Self-citationTrust Your Neighbors: A Comprehensive Survey of Neighborhood-Based Methods for Recommender Systems
Νικολακόπουλος, Α. Ν., Ning, X., Desrosiers, C., & Karypis, G. (2021) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_2 · Cited by 51
Fairness and discrimination in recommendation and retrieval
Ekstrand, M. D., Burke, R., & Díaz, F. (2019) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3298689.3346964 · Cited by 50
Chen, C. C., Shih, S.-Y., & Lee, M. (2016) · Decision Support Systems · doi:10.1016/j.dss.2016.06.017 · Cited by 48
Enabling cross-continent provider fairness in educational recommender systems
Gómez, E., Zhang, C. S., Boratto, L., Salamó, M., & Ramos, G. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.08.025 · Cited by 47
Online field experiments: a selective survey of methods
Chen, Y., & Konstan, J. A. (2015) · Journal of the Economic Science Association · doi:10.1007/s40881-015-0005-3 · Cited by 46
Evaluating Group Recommender Systems
Felfernig, A., Boratto, L., Stettinger, M., & Tkalčič, M. (2018) · Springer briefs in electrical and computer engineering · doi:10.1007/978-3-319-75067-5_3 · Cited by 46
Knijnenburg, B. P., & Willemsen, M. C. (2009) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/1639714.1639793 · Cited by 44
Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems
Champiri, Z. D., Asemi, A., & Salim, S. S. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-018-1324-5 · Cited by 39
A Meta-Model for Information Systems Quality
Russo, D., Ciancarini, P., Falasconi, T., & Tomasi, M. (2018) · ACM Transactions on Management Information Systems · doi:10.1145/3230713 · Cited by 25
Evaluating Recommender Systems for Supportive Technologies
Freyne, J., & Berkovsky, S. (2013) · Human-computer interaction series · doi:10.1007/978-1-4471-4778-7_8 · Cited by 24
Zangerle, E., Gassler, W., Pichl, M., Steinhauser, S., & Specht, G. (2016) · International Symposium on Open Collaboration · doi:10.1145/2957792.2957804 · Cited by 24
Self-citationToward Better Interactions in Recommender Systems
Zhao, Q., Adomavičius, G., Harper, F. M., Willemsen, M. C., & Konstan, J. A. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998211 · Cited by 21
Value and Impact of Recommender Systems
Jannach, D., & Zanker, M. (2012) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_14 · Cited by 21
Liang, Y., & Willemsen, M. C. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3474253 · Cited by 19
Community-Based Recommender Systems: Analyzing Business Models from a Systems Operator's Perspective
Chen, P.-Y. S., Chou, Y., & Kauffman, R. J. (2009) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2009.117 · Cited by 18
Evaluating recommendation and search in the labor market
Reusens, M., Lemahieu, W., Baesens, B., & Sels, L. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.04.007 · Cited by 18
Mixed methods for evaluating user satisfaction
Garcia-Gathright, J. I., Hosey, C., Thomas, B. S., Carterette, B., & Díaz, F. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3241622 · Cited by 16
A factual and perceptional framework for assessing diversity effects of online recommender systems
Matt, C., Heß, T., & Weiß, C. H. (2019) · Internet Research · doi:10.1108/intr-06-2018-0274 · Cited by 15
User-Centric Evaluation Framework for Multimedia Recommender Systems
Knijnenburg, B. P., Meesters, L., Marrow, P. B., & Bouwhuis, D. G. (2010) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-642-12630-7_47 · Cited by 14
Situation awareness for recommender systems
Richthammer, C., & Pernul, G. (2018) · Electronic Commerce Research · doi:10.1007/s10660-018-9321-z · Cited by 14
Experiences in Applying Mixed-Methods Approach in Information Systems Research
Peng, G. C. A., & Annansingh, F. (2013) · IGI Global eBooks · doi:10.4018/978-1-4666-2491-7.ch014 · Cited by 5
Mixed Method Development of Evaluation Metrics
Thomas, B. S., Chandar, P., Hosey, C., & Díaz, F. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3470802 · Cited by 4
Cited by
Recommender Systems in the Era of Large Language Models (LLMs)
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., et al. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3392335 · Cited by 481
Metadata
- Authors
- Zangerle, E., & Bauer, C.
- Venue
- ACM Computing Surveys
- Published
- 2022-12-23
- Volume
- 55
- Issue
- 8
- Pages
- 1-38
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4292092796
- Cited by
- 283
- References
- 181
- FWCI
- 51.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.