CoChat
Sign Up
PreprintCited by 28Machine Learning and Data ClassificationStochastic Gradient Optimization TechniquesDomain Adaptation and Few-Shot Learning

Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt

Open access
Mindermann, Sören, Brauner, Jan, Razzak, Muhammed, Sharma, Mrinank, Kirsch, Andreas, Xu, Winnie, Höltgen, Benedikt, Gomez, Aidan N., Morisot, Adrien, Farquhar, Sebastian, Gal, Yarin·2022·arXiv
DOI 10.48550/arxiv.2206.07137arXiv 2206.07137
Open this paper in CoChat — free accountOpen PDF

Abstract

Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection (RHO-LOSS), a simple but principled technique which selects approximately those points for training that most reduce the model's generalization loss. As a result, RHO-LOSS mitigates the weaknesses of existing data selection methods: techniques from the optimization literature typically select 'hard' (e.g. high loss) points, but such points are often noisy (not learnable) or less task-relevant. Conversely, curriculum learning prioritizes 'easy' points, but such points need not be trained on once learned. In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT). On the large web-scraped image dataset Clothing-1M, RHO-LOSS trains in 18x fewer steps and reaches 2% higher final accuracy than uniform data shuffling.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.07137

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Gathering large sets of data is inefficient, and training on them is prohibitively expensive.

    “In practice, gathering large sets of data is inefficient, and training on them is prohibitively expensive; worse, there is no guarantee that the resulting model will handle complex scenarios or generalize better.”

    From TT-SI: Self-Improving LLM Agents with Test-Time Training (2026) · cites this paper as Mindermann et al., 2022

    The Introduction (p. 1) says training large models can take weeks or months, supporting the concern about training burden, but it does not establish that training is prohibitively expensive. It does not say gathering large datasets is inefficient; the Experiments section (p. 4) instead describes web-scraped data as abundant. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Reliance on extremely large training datasets leads to substantial annotation and computational costs, rendering agent development prohibitively expensive.

    “Computation Cost: The reliance on extremely large training datasets D train ( N ≫ 10 4 ) leads to substantial annotation and computational costs, both scaling with N , rendering agent development prohibitively expensive (Mirzasoleiman et al., 2020; Mindermann et al., 2022).”

    From TT-SI: Self-Improving LLM Agents with Test-Time Training (2026) · cites this paper as Mindermann et al., 2022

    The Introduction (p. 1) says models train on vast amounts of web-scraped data, that training can take weeks or months, and that training time constrains development. It does not establish annotation costs, costs scaling with dataset size, a threshold of N ≫ 10⁴, or that agent development is prohibitively expensive. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Alain, G., Lamb, A., Sankar, C., Courville, A., and Bengio, Y. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481 , 2015. · arXiv
  2. Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G. E., and Hinton, G. E. Large scale distributed neural network training through online distillation. arXiv preprint arXiv:1804.03235 , 2018. · arXiv
  3. Baayen, R. H. Word frequency distributions , volume 18. Springer Science & Business Media, 2001.
  4. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pp. 41-48, 2009.
  5. Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pp. 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/ blundell15.html .
  6. Borsos, Z., Mutn` y, M., and Krause, A. Coresets via bilevel optimization for continual learning and streaming. arXiv preprint arXiv:2006.03875 , 2020. · arXiv
  7. Bottou, L. and LeCun, Y. Large scale online learning. Advances in neural information processing systems , 16:217224, 2004.
  8. Chen, P., Liao, B. B., Chen, G., and Zhang, S. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning , pp. 1062-1070. PMLR, 2019.
  9. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. · arXiv
  10. Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selection via proxy: Efficient data selection for deep learning. International Conference on Learning Representations , 2020.
  11. Darlow, L. N., Crowley, E. J., Antoniou, A., and Storkey, A. J. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018. · arXiv
  12. Farquhar, S., Gal, Y., and Rainforth, T. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665 , 2021. · arXiv
  13. Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pp. 1050-1059. PMLR, 2016.
  14. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 2016.
  15. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems , 32:103-112, 2019.
  16. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pp. 448456. PMLR, 2015.
  17. Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.00762 , 2019. · arXiv
  18. Johnson, T. B. and Guestrin, C. Training deep models faster with robust, approximate importance sampling. Advances in Neural Information Processing Systems , 31: 7265-7275, 2018.
  19. Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture , pp. 1-12, 2017.
  20. Katharopoulos, A. and Fleuret, F. Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043 , 2017. · arXiv
  21. Katharopoulos, A. and Fleuret, F. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning , pp. 25252534. PMLR, 2018.
  22. Kawaguchi, K. and Lu, H. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics , pp. 669-679. PMLR, 2020.
  23. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016. · arXiv
  24. Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning. arXiv preprint arXiv:2012.10630 , 2020. · arXiv
  25. Kirsch, A., Van Amersfoort, J., and Gal, Y. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. arXiv preprint arXiv:1906.08158 , 2019. · arXiv
  26. Kirsch, A., Rainforth, T., and Gal, Y. Active learning under pool set distribution shift and noisy data. CoRR , abs/2106.11719, 2021. URL https://arxiv.org/ abs/2106.11719 . · arXiv
  27. Komatsuzaki, A. One epoch is all you need. arXiv preprint arXiv:1906.06669 , 2019. · arXiv
  28. Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto , 2009.
  29. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
  30. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551, 1989.
  31. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, November 1998.
  32. LeCun, Y. A., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade , pp. 9-48. Springer, 2012.
  33. Li, M. and Sethi, I. K. Confidence-based active learning. IEEE transactions on pattern analysis and machine intelligence , 28(8):1251-1261, 2006.
  34. Loshchilov, I. and Hutter, F. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343 , 2015. · arXiv
  35. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. · arXiv
  36. McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162 , 2018. · arXiv
  37. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) , 54(6):1-35, 2021.
  38. Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning , pp. 6950-6960. PMLR, 2020.
  39. Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H., and Gal, Y. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.11582 , 2021. · arXiv
  40. Osawa, K., Swaroop, S., Jain, A., Eschenhagen, R., Turner, R. E., Yokota, R., and Khan, M. E. Practical deep learning with bayesian principles. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , pp. 4287-4299, 2019.
  41. Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. arXiv preprint arXiv:2107.07075 , 2021. · arXiv
  42. Pleiss, G., Zhang, T., Elenberg, E., and Weinberger, K. Q. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems , 33:17044-17056, 2020.
  43. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pp. 35053506, 2020.
  44. Ren, M., Zeng, W., Yang, B., and Urtasun, R. Learning to reweight examples for robust deep learning. In International conference on machine learning , pp. 4334-4343. PMLR, 2018.
  45. Rolnick, D., Veit, A., Belongie, S., and Shavit, N. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694 , 2017. · arXiv
  46. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015. · arXiv
  47. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multibillion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. · arXiv
  48. Tian, Y., Henaff, O. J., and Oord, A. v. d. Divide and contrast: Self-supervised learning from uncurated data. arXiv preprint arXiv:2105.08054 , 2021. · arXiv
  49. Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 , 2018. · arXiv
  50. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018. · arXiv
  51. Wilson, A. G. and Izmailov, P. Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint arXiv:2002.08791 , 2020. · arXiv
  52. Xiao, T., Xia, T., Yang, Y ., Huang, C., and Wang, X. Learning from massive noisy labeled data for image classification. In CVPR , 2015.
  53. Yadav, C. and Bottou, L. Cold case: The lost mnist digits. In Advances in Neural Information Processing Systems 32 , 2019.
  54. Yi, K. and Wu, J. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.
  55. Zhang, J., Yu, H.-F., and Dhillon, I. S. Autoassist: A framework to accelerate training of deep neural networks. arXiv preprint arXiv:1905.03381 , 2019. · arXiv

Cited by

Sorted by citations

Authors
Mindermann, S., Brauner, J., Razzak, M., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., & Gal, Y.
Affiliations
  • University of Oxford (GB)
  • University of Toronto (CA)
  • Cohere (Canada) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.07137
arXiv
2206.07137
OpenAlex
W4283073619
Cited by
28
References
2
Citations by year
2025: 7 · 2024: 8 · 2023: 12 · 2022: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
training data selection, training efficiency, curriculum learning, web-scale data, image classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review