The Netflix Recommender System: Algorithms, Business Value, and Innovation
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
This article discusses the various algorithms that make up the Netflix recommender system, and describes its business purpose. We also describe the role of search and related algorithms, which for us turns into a recommendations problem as well. We explain the motivations behind and review the approach that we use to improve the recommendation algorithms, combining A/B testing focused on improving member retention and medium term engagement, as well as offline experimentation using historical member engagement data. We discuss some of the issues in designing and interpreting A/B tests. Finally, we describe some current areas of focused innovation, which include making our recommender system global and language aware.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Matrix Factorization Techniques for Recommender Systems
Koren, Y., Bell, R., & Volinsky, C. (2009) · Computer · doi:10.1109/mc.2009.263 · Cited by 11,673
(2012) · Cited by 9,062
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012
Analysis of Incomplete Multivariate Data
Booth, D. E. (2000) · Technometrics · doi:10.1080/00401706.2000.10486013 · Cited by 5,627
Probabilistic Matrix Factorization
Mnih, A., & Salakhutdinov, R. (2007) · UCL Discovery (University College London) · Cited by 4,517
Factorization meets the neighborhood: a multifaceted collaborative filtering model
Koren, Y. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401944 · Cited by 3,852
Rendle, S. (2010) · 2010 IEEE International Conference on Data Mining · doi:10.1109/icdm.2010.127 · Cited by 3,189
The Paradox of Choice: Why More Is Less
Schwartz, B. N. (2004) · Works - Scholarship, Research, & Creative Expression (Swarthmore College) · Cited by 2,512
Hierarchical Dirichlet Processes
Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000302 · Cited by 2,120
Bennett, J. R., & Lanning, S. (2007) · Cited by 1,216
Large-scale validation and analysis of interleaved search evaluation
Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012) · ACM Transactions on Information Systems · doi:10.1145/2094072.2094078 · Cited by 227
Improving the sensitivity of online controlled experiments by utilizing pre-experiment data
Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013) · Proceedings of the sixth ACM international conference on Web search and data mining · doi:10.1145/2433396.2433413 · Cited by 223
A/B Testing: The Most Powerful Way to Turn Clicks Into Customers
Siroker, D., & Koomen, P. (2013) · Cited by 149
Research works printed but not linked: 2
- Amatriain Xavier, 2012, Netflix Recommendations: Beyond the 5 stars (Part 2). Retrieved
- Paterek Arkadiusz, 2007, Proceedings of KDD Cup and Workshop. 5--8.
Cited by
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137
Deep Learning Based Recommender System: A Survey and New Perspectives
Zhang, S., Yao, L., Sun, A., & Tay, Y. (2020) · ACM Computing Surveys · doi:10.1145/3285029 · Cited by 3,485
If...Then: Algorithmic Power and Politics
Bucher, T. (2018) · doi:10.1093/oso/9780190493028.001.0001 · Cited by 1,223
Recommender Systems with Generative Retrieval
Rajput, S., Mehta, N., Singh, A., Keshavan, R. H., Vu, T., Heldt, L., Hong, L., Tay, Y., Tran, V., Samost, J., et al. (2023) · neural information processing systems · doi:10.52202/075280-0452 · Cited by 1,085
Machine learning in mental health: a scoping review of methods and applications
Shatte, A. B. R., Hutchinson, D. M., & Teague, S. J. (2019) · Psychological Medicine · doi:10.1017/s0033291719000151 · Cited by 1,047
Metadata
- Authors
- Gomez-Uribe, C. A., & Hunt, N.
- Affiliations
- Netflix (United States) (US)
- Venue
- ACM Transactions on Management Information Systems
- Published
- 2015-12-28
- Volume
- 6
- Issue
- 4
- Pages
- 1-19
- ISSN
- 2158-656X (ISSN-L)
- 2158-6578
- OpenAlex
- W2210543184
- Cited by
- 1,418
- References
- 15
- Citations by year
- 2026: 130 · 2025: 185 · 2024: 186 · 2023: 142 · 2022: 133 · 2021: 178 · 2020: 156 · 2019: 132 · 2018: 92 · 2017: 51 · 2016: 31
- FWCI
- 155.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Management Information Systems (publishedVersion)
- Funders
- National Geographic Society
- Topics
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Management Science and Operations Research › Advanced Bandit Algorithms Research
- Keywords
- Netflix, recommender systems, recommendation algorithms, A/B testing, member retention, user engagement, search algorithms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.