CoChat
Sign Up
PreprintCited by 1Machine Learning and Data ClassificationStochastic Gradient Optimization TechniquesAdversarial Robustness in Machine Learning

Optimizing ML Training with Metagradient Descent

Open access
Engstrom, Logan, Ilyas, Andrew, Chen, Benjamin, Feldmann, Axel, Moses, William, Madry, Aleksander·2025·arXiv
DOI 10.48550/arxiv.2503.13751arXiv 2503.13751
Open this paper in CoChat — free accountOpen PDF

Abstract

A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.13751

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modern frameworks for automatic differentiation can efficiently compute gradients of gradients with minimal overhead.

    “Fortunately, modern frameworks for automatic differentiation can efficiently compute gradients of gradients with minimal overhead [13, 25].”

    From End-to-End Test-Time Training for Long Context (2025) · cites this paper as [25]

    Section 2.2 (p. 4) describes automatic differentiation as a way to calculate metagradients and says an ordinary gradient takes only a small constant factor more time than evaluating the function. But it explicitly includes a storage cost and says direct AD for metagradients can require terabytes of storage even for MNIST training. The paper therefore does not support the unqualified claim that gradients of gradients have minimal overhead. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 91

  1. [ATS+23] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. 'SemDeDup: Data-efficient learning at web-scale through semantic deduplication'. In: arXiv preprint arXiv:2303.09540 (2023). · arXiv
  2. [BAC+21] Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. 'The iWildCam 2021 competition dataset'. In: arXiv preprint arXiv:2105.03494 . 2021. · arXiv
  3. [BBC+22] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe LlinaresLópez, Fabian Pedregosa, and Jean-Philippe Vert. 'Efficient and modular implicit differentiation'. In: Advances in neural information processing systems 35 (2022), pp. 5230-5242.
  4. [BBY+22] Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. 'WinoGAViL: Gamified association benchmark to challenge vision-andlanguage models'. In: Advances in Neural Information Processing Systems . 2022.
  5. [BCT92] Preston Briggs, Keith D Cooper, and Linda Torczon. 'Rematerialization'. In: Proceedings of the ACM SIGPLAN 1992 conference on Programming language design and implementation . 1992, pp. 311-321.
  6. [Ben00] Yoshua Bengio. 'Gradient-based optimization of hyperparameters'. In: Neural computation 12.8 (2000), pp. 1889-1900.
  7. [BGM+18] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. 'From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge'. In: IEEE Transactions on Medical Imaging (2018).
  8. [BGV14] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 'Food-101-mining discriminative components with random forests'. In: European conference on computer vision . 2014.
  9. [BKB+20] Quentin Bertrand, Quentin Klopfenstein, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, and Joseph Salmon. 'Implicit differentiation of lasso-type models for hyperparameter optimization'. In: International Conference on Machine Learning . PMLR. 2020, pp. 810-821.
  10. [BKM+22] Quentin Bertrand, Quentin Klopfenstein, Mathurin Massias, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, and Joseph Salmon. 'Implicit differentiation for fast hyperparameter selection in non-smooth convex learning'. In: Journal of Machine Learning Research 23.149 (2022), pp. 1-43.
  11. [BMA+19] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. 'ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models'. In: Neural Information Processing Systems (NeurIPS) . 2019.
  12. [BNL+22] Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Grosse. 'If Influence Functions are the Answer, Then What is the Question?' In: ArXiv preprint arXiv:2209.05364 . 2022. · arXiv
  13. [BSF94] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 'Learning long-term dependencies with gradient descent is difficult'. In: IEEE Transactions on Neural Networks . 1994.
  14. [CAC+81] Gregory J Chaitin, Marc A Auslander, Ashok K Chandra, John Cocke, Martin E Hopkins, and Peter W Markstein. 'Register allocation via coloring'. In: Computer languages 6.1 (1981), pp. 4757.
  15. [CFW+18] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. 'Functional Map of the World'. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . June 2018.
  16. [CH20] Xiangning Chen and Cho-Jui Hsieh. 'Stabilizing differentiable architecture search via perturbationbased regularization'. In: International conference on machine learning . PMLR. 2020, pp. 15541565.
  17. [CHL17] Gong Cheng, Junwei Han, and Xiaoqiang Lu. 'Remote sensing image scene classification: Benchmark and state of the art'. In: Proceedings of the IEEE . 2017.
  18. [CKL+22] Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability . 2022. arXiv: 2103.00065 [cs.LG] . URL: https://arxiv.org/abs/2103.00065 . · arXiv
  19. [CMK+14] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 'Describing textures in the wild'. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2014.
  20. [CNL11] Adam Coates, Andrew Ng, and Honglak Lee. 'An analysis of single-layer networks in unsupervised feature learning'. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics . 2011.
  21. [CXR+22] Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelley, and Erik Meijer. 'Gradient descent: The ultimate optimizer'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 82148225.
  22. [CXZ+16] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 'Training Deep Nets with Sublinear Memory Cost'. In: CoRR abs/1604.06174 (2016). arXiv: 1604.06174 . URL: http://arxiv. org/abs/1604.06174 . · arXiv
  23. [DDS+09] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 'Imagenet: A large-scale hierarchical image database'. In: Computer Vision and Pattern Recognition (CVPR) . 2009.
  24. [DFE+22] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness . 2022. arXiv: 2205.14135 [cs.LG] . URL: https://arxiv.org/abs/2205.14135 . · arXiv
  25. [DJP+24] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 'The llama 3 herd of models'. In: arXiv preprint arXiv:2407.21783 (2024). · arXiv
  26. [EVW+10] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 'The Pascal Visual Object Classes (VOC) Challenge'. In: International Journal of Computer Vision . 2010.
  27. [FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. 'Model-agnostic meta-learning for fast adaptation of deep networks'. In: International conference on machine learning . PMLR. 2017, pp. 11261135.
  28. [FDF+17] Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 'Forward and reverse gradient-based hyperparameter optimization'. In: International Conference on Machine Learning (ICML) . 2017.
  29. [FFP04] Li Fei-Fei, Rob Fergus, and Pietro Perona. 'Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories'. In: 2004 conference on computer vision and pattern recognition workshop . IEEE. 2004, pp. 178-178.
  30. [FIW+22] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. 'Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)'. In: ICML . 2022.
  31. [GBA+23] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. 'Studying large language model generalization with influence functions'. In: arXiv preprint arXiv:2308.03296 (2023). · arXiv
  32. [GDG17] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 'Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain'. In: arXiv preprint arXiv:1708.06733 (2017). · arXiv
  33. [GIF+24] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 'DataComp: In search of the next generation of multimodal datasets'. In: Advances in Neural Information Processing Systems . 2024.
  34. [GLU12] Andreas Geiger, Philip Lenz, and Raquel Urtasun. 'Are we ready for autonomous driving? The KITTI vision benchmark suite'. In: 2012 IEEE conference on computer vision and pattern recognition . 2012.
  35. [HAM+21] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 'Meta-learning in neural networks: A survey'. In: IEEE transactions on pattern analysis and machine intelligence 44.9 (2021), pp. 5149-5169.
  36. [HBB+20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 'Measuring massive multitask language understanding'. In: arXiv preprint arXiv:2009.03300 (2020). · arXiv
  37. [HBD+19] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 'EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification'. In: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing . 2019.
  38. [HBK+21] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 'Measuring Mathematical Problem Solving With the MATH Dataset'. In: NeurIPS (2021).
  39. [HBM+20] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization . 2020. arXiv: 2006.16241 [cs.CV] . · arXiv
  40. [HBM+22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 'Training compute-optimal large language models'. In: arXiv preprint arXiv:2203.15556 . 2022. · arXiv
  41. [HNM19] Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. 'Data cleansing for models trained with SGD'. In: Advances in Neural Information Processing Systems 32 (2019).
  42. [Hub64] Peter J. Huber. 'Robust estimation of a location parameter'. In: The Annals of Mathematical Statistics . 1964.
  43. [HY20] Jiaoyang Huang and Horng-Tzer Yau. 'Dynamics of Deep Neural Networks and Neural Tangent Hierarchy'. In: Proceedings of the 37th International Conference on Machine Learning . 2020.
  44. [HZB+19] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. 'Natural adversarial examples'. In: arXiv preprint arXiv:1907.07174 (2019). · arXiv
  45. [JHV+17] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 'CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2017.
  46. [JS08] Yaochu Jin and Bernhard Sendhoff. 'Pareto-based multiobjective machine learning: An overview and case studies'. In: IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.3 (2008), pp. 397-415.
  47. [KB15] Diederik P. Kingma and Jimmy Ba. 'Adam: A Method for Stochastic Optimization'. In: International Conference on Learning Representations (ICLR) . 2015.
  48. [KDJ20] MJ Zico Kolter, David Duvenaud, and Matt Johnson. 'Deep implicit layers-neural odes, deep equilibirum models, and beyond, 2020'. In: NeurIPS Tutorial (2020).
  49. [KKR+24] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 'Openassistant conversations-democratizing large language model alignment'. In: Advances in Neural Information Processing Systems 36 (2024).
  50. [KL17] Pang Wei Koh and Percy Liang. 'Understanding Black-box Predictions via Influence Functions'. In: International Conference on Machine Learning . 2017.
  51. [KSD+13] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. '3d object representations for finegrained categorization'. In: Proceedings of the IEEE international conference on computer vision workshops . 2013.
  52. [KSM+20] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, et al. 'WILDS: A Benchmark of in-the-Wild Distribution Shifts'. In: arXiv preprint arXiv:2012.07421 (2020). · arXiv
  53. [LFX+24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 'Deepseek-v3 technical report'. In: arXiv preprint arXiv:2412.19437 . 2024. · arXiv
  54. [LHV+23] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 'The flan collection: Designing data and methods for effective instruction tuning'. In: International Conference on Machine Learning . PMLR. 2023, pp. 22631-22648.
  55. [LKY22] Yiwei Lu, Gautam Kamath, and Yaoliang Yu. 'Indiscriminate Data Poisoning Attacks on Neural Networks'. In: arXiv preprint arXiv:2204.09092 (2022). · arXiv
  56. [LKY23] Yiwei Lu, Gautam Kamath, and Yaoliang Yu. 'Exploring the limits of model-targeted indiscriminate data poisoning attacks'. In: International Conference on Machine Learning . PMLR. 2023, pp. 22856-22879.
  57. [LM20] Guillaume Leclerc and Aleksander Madry. 'The two regimes of deep network training'. In: arXiv preprint arXiv:2002.10376 . 2020. · arXiv
  58. [LMB+14] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 'Microsoft coco: Common objects in context'. In: European conference on computer vision (ECCV) . 2014.
  59. [LSY18] Hanxiao Liu, Karen Simonyan, and Yiming Yang. 'Darts: Differentiable architecture search'. In: arXiv preprint arXiv:1806.09055 (2018). · arXiv
  60. [LVD20] Jonathan Lorraine, Paul Vicol, and David Duvenaud. 'Optimizing millions of hyperparameters by implicit differentiation'. In: International conference on artificial intelligence and statistics . PMLR. 2020, pp. 1540-1552.
  61. [MDA15] Dougal Maclaurin, David Duvenaud, and Ryan Adams. 'Gradient-based hyperparameter optimization through reversible learning'. In: International conference on machine learning (ICML) . 2015.
  62. [MRK+13] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 'Finegrained visual classification of aircraft'. In: arXiv preprint arXiv:1306.5151 (2013). · arXiv
  63. [MS21] Paul Micaelli and Amos J Storkey. 'Gradient-based hyperparameter optimization over long horizons'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 10798-10809.
  64. [Nau08] Uwe Naumann. 'Optimal Jacobian accumulation is NP-complete'. In: Math. Program. 112.2 (Apr. 2008), pp. 427-441. ISSN: 0025-5610.
  65. [NWC+11] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. 'Reading digits in natural images with unsupervised feature learning'. In: NIPS workshop on deep learning and unsupervised feature learning . 2011.
  66. [NZ08] Maria-Elena Nilsback and Andrew Zisserman. 'Automated flower classification over a large number of classes'. In: 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . 2008.
  67. [Pea96] Barak A Pearlmutter. 'An investigation of the gradient descent process in neural networks'. In: PhD thesis, Carnegie Mellon University . 1996.
  68. [PGI+23] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 'TRAK: Attributing Model Behavior at Scale'. In: Arxiv preprint arXiv:2303.14186 . 2023. · arXiv
  69. [PVZ+12] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 'Cats and dogs'. In: 2012 IEEE conference on computer vision and pattern recognition . IEEE. 2012, pp. 3498-3505.
  70. [RDK+22] William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 'The dollar street dataset: Images representing the geographic and socioeconomic diversity of the world'. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track . 2022.
  71. [RFK+19] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. 'Meta-learning with implicit gradients'. In: Advances in neural information processing systems 32 (2019).
  72. [RKH+21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 'Learning transferable visual models from natural language supervision'. In: arXiv preprint arXiv:2103.00020 . 2021. · arXiv
  73. [RLZ+24] Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. 'GeoDE: a geographically diverse evaluation dataset for object recognition'. In: Advances in Neural Information Processing Systems . 2024.
  74. [RRS+19] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 'Do ImageNet Classifiers Generalize to ImageNet?' In: International Conference on Machine Learning (ICML) . 2019.
  75. [SGB+22] Damien Scieur, Gauthier Gidel, Quentin Bertrand, and Fabian Pedregosa. 'The curse of unrolling: Rate of differentiating through optimization'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 17133-17145.
  76. [SN17] Leslie N. Smith Smith and Topin Nicholay. 'Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates'. In: ArXiv preprint arXiv:1708.07120 . 2017. · arXiv
  77. [SRR+22] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 'Beyond the imitation game: Quantifying and extrapolating the capabilities of language models'. In: arXiv preprint arXiv:2206.04615 (2022). · arXiv
  78. [SSS+11] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. 'The German traffic sign recognition benchmark: a multi-class classification competition'. In: The 2011 international joint conference on neural networks . 2011.
  79. [SSS+22] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 'Challenging bigbench tasks and whether chain-of-thought can solve them'. In: arXiv preprint arXiv:2210.09261 (2022). · arXiv
  80. [TMH+24] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 'Gemma: Open models based on gemini research and technology'. In: arXiv preprint arXiv:2403.08295 (2024). · arXiv
  81. [TSF+16] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 'YFCC100M: The New Data in Multimedia Research'. In: Communications of the ACM (2016).
  82. [VLW+18] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. 'Rotation equivariant CNNs for digital pathology'. In: Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11 . 2018.
  83. [Wer90] Paul J Werbos. 'Backpropagation through time: what it does and how to do it'. In: Proceedings of the IEEE 78.10 (1990), pp. 1550-1560.
  84. [WGX+19] Haohan Wang, Songwei Ge, Eric P Xing, and Zachary C Lipton. 'Learning robust global representations by penalizing local predictive power'. In: Neural Information Processing Systems (NeurIPS) (2019).
  85. [WWS+22] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 'Chain-of-thought prompting elicits reasoning in large language models'. In: Advances in neural information processing systems 35 (2022), pp. 24824-24837.
  86. [XHE+10] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 'Sun database: Large-scale scene recognition from abbey to zoo'. In: Computer Vision and Pattern Recognition (CVPR) . 2010.
  87. [XMG+24] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 'Less: Selecting influential data for targeted instruction tuning'. In: arXiv preprint arXiv:2402.04333 (2024). · arXiv
  88. [YLH+14] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 'From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions'. In: Transactions of the Association for Computational Linguistics . 2014.
  89. [ZP00] Geoffrey Zweig and Mukund Padmanabhan. 'Exact alpha-beta computation in logarithmic space with application to MAP word graph construction'. In: Sixth International Conference on Spoken Language Processing, ICSLP 2000 / INTERSPEECH 2000, Beijing, China, October 16-20, 2000 . ISCA, 2000, pp. 855-858. DOI: 10.21437/ICSLP.2000-404 . URL: https://doi.org/10. 21437/ICSLP.2000-404 . · DOI
  90. [ZPK+19] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. 'A large-scale study of representation learning with the visual task adaptation benchmark'. In: arXiv preprint arXiv:1910.04867 . 2019. · arXiv
  91. [ZSP+21] Miao Zhang, Steven W Su, Shirui Pan, Xiaojun Chang, Ehsan M Abbasnejad, and Reza Haffari. 'idarts: Differentiable architecture search with stochastic implicit gradients'. In: International Conference on Machine Learning . PMLR. 2021, pp. 12557-12566.

Cited by

Sorted by citations

  • End-to-End Test-Time Training for Long Context

    Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., et al. (2025) · arXiv

    “Fortunately, modern frameworks for automatic di ff erentiation can e ffi ciently compute gradients of gradients with minimal overhead [13, 25].”

Authors
Engstrom, L., Ilyas, A., Chen, B., Feldmann, A., Moses, W., & Madry, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.13751
arXiv
2503.13751
OpenAlex
W6910513755
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
data selection, data poisoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review