The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
Abstract
Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“training models with differentially-private mechanisms often reduces accuracy [34] because it causes models to fail to capture the long tails of the data distribution [19,20,67]. Moreover, it increases training time, which can further reduce accuracy because current LMs are limited by the cost of training [35, 38, 55].”
From Extracting Training Data from Large Language Models (2020) · cites this paper as [20]
In the citing paper
“The tradeoffs between privacy and utility of models have been studied extensively: differentially-private training typically prevents models from capturing the long tails of the data distribution and thus hurts utility [19, 20, 67].”
From Extracting Training Data from Large Language Models (2020) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Research works printed but not linked: 28
- D. Arpit, S. Jastrzkebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. "A closer look at memorization in deep networks". In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org. 2017, pp. 233-242.
- B. Barz and J. Denzler. "Do We Train on Test Data? Purging CIFAR of Near-Duplicates". In: Journal of Imaging 6.6 (2020), p. 41.
- M. Belkin, D. J. Hsu, and P. Mitra. "Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate". In: Advances in Neural Information Processing Systems. 2018, pp. 2300-2311.
- M. Belkin, D. Hsu, and J. Xu. "Two models of double descent for weak features". In: arXiv preprint arXiv:1903.07571 (2019). · arXiv
- P. L. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler. "Benign Overfitting in Linear Regression". In: arXiv preprint arXiv:1906.11300 (2019). · arXiv
- M. Belkin, S. Ma, and S. Mandal. "To Understand Deep Learning We Need to Understand Kernel Learning". In: ICML. Vol. 80. Proceedings of Machine Learning Research. PMLR, 2018, pp. 541-549. URL: http://proceedings.mlr.press/v80/belkin18a.html.
- E. Bagdasaryan, O. Poursaeed, and V. Shmatikov. "Differential privacy has disparate impact on model accuracy". In: Advances in Neural Information Processing Systems. 2019, pp. 15453-15462.
- M. Belkin, A. Rakhlin, and A. B. Tsybakov. "Does data interpolation contradict statistical optimality?" In: arXiv preprint arXiv:1806.09471 (2018). · arXiv
- N. Carlini, Ú. Erlingsson, and N. Papernot. "Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications". In: arXiv preprint arXiv:1910.13427 (2019). · arXiv
- V. Feldman. "Does Learning Require Memorization? A Short Tale about a Long Tail". In: CoRR abs/1906.05271 (2019). Extended abstract in STOC 2020. arXiv: 1906.05271. URL: http://arxiv. org/abs/1906.05271. · arXiv
- G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. "Densely connected convolutional networks". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 4700-4708.
- T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani. "Surprises in High-Dimensional Ridgeless Least Squares Interpolation". In: arXiv preprint arXiv:1903.08560 (2019). · arXiv
- K. He, X. Zhang, S. Ren, and J. Sun. "Deep Residual Learning for Image Recognition". In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016, pp. 770-778.
- Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. "Fantastic Generalization Mea- sures and Where to Find Them". In: ICLR. 2020. URL: https://openreview.net/forum?id= SJgIPJBFvH.
- Z. Jiang, C. Zhang, K. Talwar, and M. C. Mozer. "Characterizing Structural Regularities of Labeled Data in Overparameterized Models". In: CoRR abs/2002.03206 (2020). arXiv: 2002.03206. URL: https://arxiv.org/abs/2002.03206. · arXiv
- T. Liang and A. Rakhlin. "Just interpolate: Kernel" ridgeless" regression can generalize". In: arXiv preprint arXiv:1808.00387 (2018). · arXiv
- V. Muthukumar, K. Vodrahalli, and A. Sahai. "Harmless interpolation of noisy data in regression". In: arXiv preprint arXiv:1903.09139 (2019). · arXiv
- B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. "Exploring generalization in deep learning". In: Advances in Neural Information Processing Systems. 2017, pp. 5947-5956.
- V. Nagarajan and J. Z. Kolter. "Uniform convergence may be unable to explain generalization in deep learning". In: Advances in Neural Information Processing Systems. 2019, pp. 11611-11622.
- G. Pruthi, F. Liu, M. Sundararajan, and S. Kale. Estimating Training Data Influence by Tracking Gradient Descent. 2020. arXiv: 2002.08484 [cs.LG]. · arXiv
- A. Rakhlin and X. Zhai. "Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenon". In: COLT. Vol. 99. PMLR, 2019, pp. 2595-2623. URL: http://proceedings.mlr. press/v99/rakhlin19a.html.
- C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. "Going deeper with convolutions". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, pp. 1-9.
- M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon. "An Empirical Study of Example Forgetting during Deep Neural Network Learning". In: ICLR. OpenReview.net, 2019. URL: https://openreview.net/forum?id=BJlxm30cKm.
- G. Van Horn and P. Perona. "The devil is in the tails: Fine-grained classification in the wild". In: arXiv preprint arXiv:1709.01450 (2017). · arXiv
- A. J. Wyner, M. Olson, J. Bleich, and D. Mease. "Explaining the success of adaboost and random forests as interpolating classifiers". In: The Journal of Machine Learning Research 18.1 (2017), pp. 1558-1590.
- C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar. "Representer point selection for explaining deep neural networks". In: Advances in Neural Information Processing Systems. 2018, pp. 9291-9301.
- X. Zhu, D. Anguelov, and D. Ramanan. "Capturing long-tail distributions of object subcategories". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, pp. 915- 922.
- C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. "Understanding deep learning requires rethinking generalization". In: ICLR. 2017. URL: https://openreview.net/forum?id=Sy8gdB9xx.
Cited by
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
Deduplicating Training Data Makes Language Models Better
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.577 · Cited by 304
Self-citationDoes learning require memorization? a short tale about a long tail
Feldman, V. (2020) · Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing · doi:10.1145/3357713.3384290 · Cited by 287
Self-citationData-efficient Fine-tuning for LLM-based Recommendation
Lin, X., Wang, W., Li, Y., Yang, S., Feng, F., Wei, Y., & Chua, T.-S. (2024) · arXiv · doi:10.48550/arxiv.2401.17197 · Cited by 165
The Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142
Metadata
- Authors
- Feldman, V., & Zhang, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- arXiv
- 2008.03703
- References
- 2
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. https://app.cochat.ai/papers/10.48550/arxiv.2008.03703
- The paper
- Feldman, V., & Zhang, C. (2020). What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. arXiv. https://doi.org/10.48550/arxiv.2008.03703
