Statistical Significance of the Netflix Challenge
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Inspired by the legacy of the Netflix contest, we provide an overview of what has been learned—from our own efforts, and those of others—concerning the problems of collaborative filtering and recommender systems. The data set consists of about 100 million movie ratings (from 1 to 5 stars) involving some 480 thousand users and some 18 thousand movies; the associated ratings matrix is about 99% sparse. The goal is to predict ratings that users will give to movies; systems which can do this accurately have significant commercial applications, particularly on the world wide web. We discuss, in some detail, approaches to “baseline” modeling, singular value decomposition (SVD), as well as kNN (nearest neighbor) and neural network models; temporal effects, cross-validation issues, ensemble methods and other considerations are discussed as well. We compare existing models in a search for new models, and also discuss the mission-critical issues of penalization and parameter shrinkage which arise when the dimensions of a parameter space reaches into the millions. Although much work on such problems has been carried out by the computer science and machine learning communities, our goal here is to address a statistical audience, and to provide a primarily statistical treatment of the lessons that have been learned from this remarkable set of data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357
Evaluating collaborative filtering recommender systems
Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004) · ACM Transactions on Information Systems · doi:10.1145/963770.963772 · Cited by 5,798
GroupLens: an open architecture for collaborative filtering of netnews
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., & Riedl, J. (1994) · Proceedings of the 1994 ACM conference on Computer supported cooperative work - CSCW '94 · doi:10.1145/192844.192905 · Cited by 5,058
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 4×
“Salakhutdinov, Mnih and Hinton (2007) report that this”
BackgroundInfluentialSparse Principal Component Analysis
Zou, H., Hastie, T., & Tibshirani, R. (2006) · Journal of Computational and Graphical Statistics · doi:10.1198/106186006x113430 · Cited by 3,152
Robust De-anonymization of Large Sparse Datasets
Narayanan, A., & Shmatikov, V. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.33 · Cited by 2,372
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300 · cited 3×
“Our discussion draws from Casella (1985), Copas (1983), Efron (1975, 1983, 1986, 1996, 2004), Efron et al. (2004), Efron and Morris (1971, 1972a, 1972b, 1973a, 1973b, 1975, 1977), Houwelingen (2001), Morris (1983), Stein (1981), Ye (1998) and Zou et al. (2007).”
BackgroundInfluentialA View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187 · cited 1×
“Although derived independently, we note that Paterek (2007) proposed a related model, except that in Paterek’s model, the movie functions which determine the user weights [corresponding to our Wi,j (k) here] do not depend on k, that is, on the user’s ratings.”
MethodRestricted Boltzmann machines for collaborative filtering
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273596 · Cited by 1,885
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · cited 1×
“See also Herlocker et al. (2000), Tintarev and Masthoff (2007) and Ungar and Foster (1998).”
BackgroundParametric empirical Bayes inference: theory and applications
Morris, C. N. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477920 · Cited by 1,448 · cited 1×
“Our discussion draws from Casella (1985), Copas (1983), Efron (1975, 1983, 1986, 1996, 2004), Efron et al. (2004), Efron and Morris (1971, 1972a, 1972b, 1973a, 1973b, 1975, 1977), Houwelingen (2001), Morris (1983), Stein (1981), Ye (1998) and Zou et al. (2007).”
Stein's Estimation Rule and its Competitors—An Empirical Bayes Approach
Efron, B., & Morris, C. (1973) · Journal of the American Statistical Association · doi:10.1080/01621459.1973.10481350 · Cited by 981
Lessons from the Netflix prize challenge
Bell, R. M., & Koren, Y. (2007) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1345448.1345465 · Cited by 727
Regression, Prediction and Shrinkage
Copas, J. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01258.x · Cited by 612 · cited 1×
“Our discussion draws from Casella (1985), Copas (1983), Efron (1975, 1983, 1986, 1996, 2004), Efron et al. (2004), Efron and Morris (1971, 1972a, 1972b, 1973a, 1973b, 1975, 1977), Houwelingen (2001), Morris (1983), Stein (1981), Ye (1998) and Zou et al. (2007).”
Cited by
Big Data: New Tricks for Econometrics
Varian, H. R. (2014) · Journal of Economic Perspectives · doi:10.1257/jep.28.2.3 · Cited by 1,557
The use of machine learning algorithms in recommender systems: a systematic review
Portugal, I., Alencar, P., & Cowan, D. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.12.020 · Cited by 743
Discrimination in the Age of Algorithms
Kleinberg, J., Ludwig, J., Mullainathan, S., & Sunstein, C. R. (2018) · Journal of Legal Analysis · doi:10.1093/jla/laz001 · Cited by 401
Item Response Theory: A Statistical Framework for Educational and Psychological Measurement
Chen, Y., Li, X., Liu, J., & Ying, Z. (2025) · Statistical Science · doi:10.1214/23-sts896 · Cited by 30
Metadata
- Authors
- Feuerverger, A., He, Y., & Khatri, S.
- Venue
- Statistical Science
- Publisher
- Institute of Mathematical Statistics
- Published
- 2012-05-01
- Volume
- 27
- Issue
- 2
- Pages
- 202-231
- ISSN
- 0883-4237 (ISSN-L)
- OpenAlex
- W2073214884
- Cited by
- 57
- References
- 14
- FWCI
- 7.86
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.