The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Evaluation of session-based recommendation algorithms
Open accessAbstract
Recommender systems help users find relevant items of interest, for example on e-commerce or media streaming sites. Most academic research is concerned with approaches that personalize the recommendations according to long-term user profiles. In many real-world applications, however, such long-term profiles often do not exist and recommendations therefore have to be made solely based on the observed behavior of a user during an ongoing session. Given the high practical relevance of the problem, an increased interest in this problem can be observed in recent years, leading to a number of proposals for session-based recommendation algorithms that typically aim to predict the user’s immediate next actions. In this work, we present the results of an in-depth performance comparison of a number of such algorithms, using a variety of datasets and evaluation measures. Our comparison includes the most recent approaches based on recurrent neural networks like gru4rec, factorized Markov model approaches such as fism or fossil, as well as simpler methods based, e.g., on nearest neighbor schemes. Our experiments reveal that algorithms of this latter class, despite their sometimes almost trivial nature, often perform equally well or significantly better than today’s more complex approaches based on deep neural networks. Our results therefore suggest that there is substantial room for improvement regarding the development of more sophisticated session-based recommendation algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 4×
“gru4rec models user sessions with the help of an RNN with Gated Recurrent Units [7] in order to predict the probability of the subsequent events (e.”
MethodInfluentialMining association rules between sets of items in large databases
Agrawal, R., Imieliński, T., & Swami, A. (1993) · Proceedings of the 1993 ACM SIGMOD international conference on Management of data · doi:10.1145/170035.170072 · Cited by 14,907
(2010) · Cited by 8,261
Amazon.com recommendations: item-to-item collaborative filtering
Linden, G., Smith, B., & York, J. (2003) · IEEE Internet Computing · doi:10.1109/mic.2003.1167344 · Cited by 5,403
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Mining association rules between sets of items in large databases
Agrawal, R., Imieliński, T., & Swami, A. (1993) · ACM SIGMOD Record · doi:10.1145/170036.170072 · Cited by 4,551
BPR: Bayesian Personalized Ranking from Implicit Feedback
Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.2618 · Cited by 4,242
Factorizing personalized Markov chains for next-basket recommendation
Rendle, S., Freudenthaler, C., & Schmidt-Thieme, L. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772773 · Cited by 2,692
Google news personalization: scalable online collaborative filtering
Das, A. S., Datar, M., Garg, A., & Rajaram, S. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242610 · Cited by 1,511
Data Preparation for Mining World Wide Web Browsing Patterns
Cooley, R., Mobasher, B., & Srivastava, J. (1999) · Knowledge and Information Systems · doi:10.1007/bf03325089 · Cited by 1,455
Norris, J. R. (1997) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810633 · Cited by 1,239
The YouTube video recommendation system
Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., Gupta, S., He, Y., Lambert, M., Livingston, B., et al. (2010) · Proceedings of the fourth ACM conference on Recommender systems · doi:10.1145/1864708.1864770 · Cited by 1,191
STAMP: Short-Term Attention/Memory Priority Model for Session-based Recommendation
Liu, Q., Zeng, Y., Mokhosi, R., & Zhang, H. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219950 · Cited by 960
Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
Hidasi, B., & Karatzoglou, A. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271761 · Cited by 925
Improved Recurrent Neural Networks for Session-based Recommendations
Tan, Y. K., Xu, X., & Liu, Y. (2016) · Proceedings of the 1st Workshop on Deep Learning for Recommender Systems · doi:10.1145/2988450.2988452 · Cited by 779
Fusing Similarity Models with Markov Chains for Sparse Sequential Recommendation
He, R., & McAuley, J. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0030 · Cited by 769
Personalized news recommendation based on click behavior
Liu, J., Dolan, P., & Pedersen, E. R. (2010) · Proceedings of the 15th international conference on Intelligent user interfaces · doi:10.1145/1719970.1719976 · Cited by 762
Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques
Adomavicius, G., & Kwon, Y. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.15 · Cited by 675
FISM: factored item similarity models for top-N recommender systems
Kabbur, S., Ning, X., & Karypis, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487589 · Cited by 651
Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks
Quadrana, M., Karatzoglou, A., Hidasi, B., & Cremonesi, P. (2017) · Proceedings of the Eleventh ACM Conference on Recommender Systems · doi:10.1145/3109859.3109896 · Cited by 591
Recurrent Marked Temporal Point Processes: Embedding Event History to Vector
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., & Song, L. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939875 · Cited by 536
Where you like to go next: successive point-of-interest recommendation
Chen, C., Yang, H., Lyu, M. R.-T., & King, I. (2013) · International Joint Conference on Artificial Intelligence · Cited by 529
Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations
Hidasi, B., Quadrana, M., Karatzoglou, A., & Tikk, D. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959167 · Cited by 523
Sequence-Aware Recommender Systems
Quadrana, M., Cremonesi, P., & Jannach, D. (2019) · ACM Computing Surveys · doi:10.1145/3190616 · Cited by 504
Self-citationWhen Recurrent Neural Networks meet the Neighborhood for Session-Based Recommendation
Jannach, D., & Ludewig, M. (2017) · Proceedings of the Eleventh ACM Conference on Recommender Systems · doi:10.1145/3109859.3109872 · Cited by 490
Self-citationA Dynamic Recurrent Model for Next Basket Recommendation
Yu, F., Liu, Q., Wu, S., Wang, L., & Tan, T. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2914683 · Cited by 486
A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion
Sordoni, A., Bengio, Y., Vahabi, H., Lioma, C., Grue Simonsen, J., & Nie, J.-Y. (2015) · Proceedings of the 24th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2806416.2806493 · Cited by 472
Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks
Zhang, Y., Dai, H., Xu, C., Feng, J., Wang, T., Bian, J., Wang, B., & Liu, T.-Y. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8917 · Cited by 337
News recommender systems – Survey and roads ahead
Karimi, M., Jannach, D., & Jugovac, M. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.04.008 · Cited by 336
Self-citationContext-aware music recommendation based on latenttopic sequential patterns
Hariri, N., Mobasher, B., & Burke, R. (2012) · Proceedings of the sixth ACM conference on Recommender systems · doi:10.1145/2365952.2365979 · Cited by 283
Playlist prediction via metric embedding
Chen, S., Moore, J. L., Turnbull, D., & Joachims, T. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339643 · Cited by 282
What recommenders recommend: an analysis of recommendation biases and possible countermeasures
Jannach, D., Lerche, L., Kamehkhosh, I., & Jugovac, M. (2015) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-015-9165-3 · Cited by 249
Self-citationAutomated Generation of Music Playlists: Survey and Experiments
Bonnin, G., & Jannach, D. (2015) · ACM Computing Surveys · doi:10.1145/2652481 · Cited by 225
Self-citationLi, Z., Zhao, H., Liu, Q., Huang, Z., Mei, T., & Chen, E. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220014 · Cited by 209
E-commerce in Your Inbox: Product Recommendations at Scale
Grbovic, M., Radosavljevic, V., Djuric, N., Bhamidipati, N., Savla, J., Bhagwan, V., & Sharp, D. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788627 · Cited by 209
Recommendations with a Purpose
Jannach, D., & Adomavicius, G. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959186 · Cited by 187
Self-citationMeta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation
Vasile, F., Smirnova, E., & Conneau, A. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959160 · Cited by 177
Jannach, D., Ludewig, M., & Lerche, L. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9194-1 · Cited by 174
Self-citationInferring a Personalized Next Point-of-Interest Recommendation Model with Latent Behavior Patterns
He, J., Li, X., Liao, L., Song, D., & Cheung, W. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9994 · Cited by 167
Multi-Rate Deep Learning for Temporal Recommendation
Song, Y., Elkahky, A. M., & He, X. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2914726 · Cited by 167
Impact of data characteristics on recommender systems performance
Adomavicius, G., & Zhang, J. (2012) · ACM Transactions on Management Information Systems · doi:10.1145/2151163.2151166 · Cited by 164
Predicting The Next App That You Are Going To Use
Baeza-Yates, R., Jiang, D., Silvestri, F., & Harrison, B. (2015) · Proceedings of the Eighth ACM International Conference on Web Search and Data Mining · doi:10.1145/2684822.2685302 · Cited by 161
Long and Short-Term Recommendations with Recurrent Neural Networks
Devooght, R., & Bersini, H. (2017) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3079628.3079670 · Cited by 142
Adaptation and Evaluation of Recommendations for Short-term Shopping Goals
Jannach, D., Lerche, L., & Jugovac, M. (2015) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2792838.2800176 · Cited by 133
Self-citationModelling Contextual Information in Session-Aware Recommender Systems with Neural Networks
Twardowski, B. (2016) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2959100.2959162 · Cited by 126
Unified Point-of-Interest Recommendation with Temporal Interval Assessment
Liu, Y., Liu, C., Liu, B., Qu, M., & Xiong, H. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939773 · Cited by 125
A learning agent for wireless news access
Billsus, D., Pazzani, M. J., & Chen, J. (2000) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/325737.325768 · Cited by 121
A case study on the effectiveness of recommendations in the mobile internet
Jannach, D., & Hegelich, K. (2009) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/1639714.1639749 · Cited by 108
Self-citationSequence-aware Recommender Systems
Quadrana, M., Cremonesi, P., & Jannach, D. (2018) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3209219.3209270 · Cited by 106
Self-citationUnifying nearest neighbors collaborative filtering
Verstrepen, K., & Goethals, B. (2014) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2645710.2645731 · Cited by 100
#nowplaying Music Dataset: Extracting Listening Behavior from Twitter
Zangerle, E., Pichl, M., Gässler, W., & Specht, G. (2014) · doi:10.1145/2661714.2661719 · Cited by 99
Which app will you use next? collaborative filtering with interactional context
Natarajan, N., Shin, D., & Dhillon, I. S. (2013) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2507157.2507186 · Cited by 98
The Natural Language Of Playlists.
McFee, B., & Lanckriet, G. R. G. (2011) · doi:10.5281/zenodo.1418118 · Cited by 89
Personalized next-song recommendation in online karaokes
Wu, X., Liu, Q., Chen, E., He, L., Lv, J., Cao, C., & Hu, G. (2013) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2507157.2507215 · Cited by 88
Statistical models of music-listening sessions in social media
Zheleva, E., Guiver, J., Rodrigues, E. M., & Milić-Frayling, N. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772794 · Cited by 77
Factored MDPs for detecting topics of user sessions
Tavakol, M., & Brefeld, U. (2014) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2645710.2645739 · Cited by 76
Hypergraph Models Of Playlist Dialects.
McFee, B., & Lanckriet, G. R. G. (2012) · doi:10.5281/zenodo.1415617 · Cited by 73
Adapting Recommendations to Contextual Changes Using Hierarchical Hidden Markov Models
Aghdam, M. H., Hariri, N., Mobasher, B., & Burke, R. (2015) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2792838.2799684 · Cited by 68
Deep Sequential Recommendation for Personalized Adaptive User Interfaces
Soh, H., Sanner, S., White, M., & Jamieson, G. A. (2017) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3025171.3025207 · Cited by 59
On the Value of Reminders within E-Commerce Recommendations
Lerche, L., Jannach, D., & Ludewig, M. (2016) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/2930238.2930244 · Cited by 55
Self-citationImpact of Recommender Systems on Sales Volume and Diversity
Lee, D., & Hosanagar, K. (2014) · Cited by 50
30Music Listening and Playlists Dataset
Turrin, R., Quadrana, M., Condorelli, A., Pagano, R., & Cremonesi, P. (2015) · Virtual Community of Pathological Anatomy (University of Castilla La Mancha) · Cited by 45
Learning Student and Content Embeddings for Personalized Lesson Sequence Recommendation
Reddy, S., Labutov, I., & Joachims, T. (2016) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2876034.2893375 · Cited by 35
Hidden Conditional Random Fields with Deep User Embeddings for Ad Targeting
Djuric, N., Radosavljević, V., Grbovic, M., & Bhamidipati, N. L. (2014) · Proceedings · doi:10.1109/icdm.2014.150 · Cited by 16
Personalized Recommendation Meets Your Next Favorite
Song, Q., Cheng, J., Yuan, T., & Lu, H. (2015) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2806416.2806598 · Cited by 9
Cited by
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 3,485
Deep Learning based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07435 · Cited by 635
A Survey on Session-based Recommender Systems
Wang, S., Cao, L., Wang, Y., Sheng, Q. Z., Orgun, M. A., & Lian, D. (2022) · ACM Computing Surveys · doi:10.1145/3465401 · Cited by 570
Are we really making much progress? A worrying analysis of recent neural recommendation approaches
Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019) · Proceedings of the 13th ACM Conference on Recommender Systems · doi:10.1145/3298689.3347058 · Cited by 431
Self-citationRecommendation system based on deep learning methods: a systematic review and new directions
Da’u, A., & Salim, N. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09744-1 · Cited by 402
Metadata
- Authors
- Ludewig, M., & Jannach, D.
- Venue
- User Modeling and User-Adapted Interaction
- Published
- 2018-10-01
- Volume
- 28
- Issue
- 4-5
- Pages
- 331-390
- ISSN
- 0924-1868 (ISSN-L)
- 1573-1391
- OpenAlex
- W2795199972
- Cited by
- 336
- References
- 66
- FWCI
- 57.05
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.