CoChat
Sign Up
PreprintCited by 1Domain Adaptation and Few-Shot LearningMachine Learning and AlgorithmsReinforcement Learning in Robotics

More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms

Open access
Zakerinia, Hossein, Behjati, Amin, Lampert, Christoph H.·2024·arXiv
DOI 10.48550/arxiv.2402.04054arXiv 2402.04054
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a new framework for studying meta-learning methods using PAC-Bayesian theory. Its main advantage over previous work is that it allows for more flexibility in how the transfer of knowledge between tasks is realized. For previous approaches, this could only happen indirectly, by means of learning prior distributions over models. In contrast, the new generalization bounds that we prove express the process of meta-learning much more directly as learning the learning algorithm that should be used for future tasks. The flexibility of our framework makes it suitable to analyze a wide range of meta-learning mechanisms and even design new mechanisms. Other than our theoretical contributions we also show empirically that our framework improves the prediction quality in practical meta-learning mechanisms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.04054

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning g treats the optimizer as a learned algorithm, using execution feedback for program revision.

    “Learning g treats the optimizer as a learned algorithm (Zakerinia et al., 2024), using execution feedback for program revision (Agrawal et al., 2026).”

    From VERSE: Verified Self-Evolving Optimizer for Agent Harnesses (2026) · cites this paper as Zakerinia et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 36

  1. Alquier, P. User-friendly introduction to PAC-Bayes bounds. Foundations and Trends in Machine Learning, 17(2):174- 303, 2024.
  2. Amit, R. and Meir, R. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In International Conference on Machine Learning (ICML), 2018.
  3. Baxter, J. A model of inductive bias learning. Journal of Artificial Intelligence Research (JAIR), 12:149-198, 2000.
  4. Chen, Q., Shui, C., and Marchand, M. Generalization bounds for meta-learning: An information-theoretic anal- ysis. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  5. Denevi, G., Ciliberto, C., Grazzi, R., and Pontil, M. Learning-to-learn stochastic gradient descent with biased regularization. In International Conference on Machine Learning (ICML), 2019.
  6. Ding, N., Chen, X., Levinboim, T., Goodman, S., and Sori- cut, R. Bridging the gap between practice and PAC-Bayes theory in few-shot meta-learning. In Conference on Neu- ral Information Processing Systems (NeurIPS), 2021.
  7. Farid, A. and Majumdar, A. Generalization bounds for meta- learning via PAC-Bayes and uniform stability. Conference on Neural Information Processing Systems (NeurIPS), 2021.
  8. Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional Conference on Machine Learning (ICML), 2017.
  9. Friedman, L. and Meir, R. Adaptive meta-learning via data- dependent PAC-Bayes bounds. In Conference on Lifelong Learning Agents (CoLLAs), 2023.
  10. Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
  11. Guan, J. and Lu, Z. Fast-rate PAC-Bayesian generalization bounds for meta-learning. In International Conference on Machine Learning (ICML), 2022.
  12. Guedj, B. A primer on PAC-Bayesian learning. arXiv preprint arXiv:1901.05353, 2019. · arXiv
  13. Hellström, F. and Durisi, G. Evaluated CMI bounds for meta learning: Tightness and expressiveness. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
  14. Hellström, F., Durisi, G., Guedj, B., and Raginsky, M. Gen- eralization bounds: Perspectives from information theory and PAC-Bayes. arXiv preprint arXiv:2309.04381, 2023. · arXiv
  15. Kingma, D. P., Salimans, T., and Welling, M. Varia- tional dropout and the local reparameterization trick. In Conference on Neural Information Processing Systems (NeurIPS), 2015.
  16. Lee, K., Maji, S., Ravichandran, A., and Soatto, S. Meta- learning with differentiable convex optimization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  17. Li, Z., Zhou, F., Chen, F., and Li, H. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835, 2017. · arXiv
  18. Liu, T., Lu, J., Yan, Z., and Zhang, G. PAC-Bayes bounds for meta-learning with data-dependent prior. arXiv preprint arXiv:2102.03748, 2021. · arXiv
  19. Maurer, A. A note on the PAC Bayesian theorem. arXiv preprint arXiv:cs.LG/0411099, 2004. · arXiv
  20. Maurer, A. Transfer bounds for linear feature learning. Machine Learning, 75(3):327-350, 2009.
  21. Maurer, A., Pontil, M., and Romera-Paredes, B. The benefit of multitask representation learning. Journal of Machine Learning Research (JMLR), 17(81):1-32, 2016.
  22. McAllester, D. A. Some PAC-Bayesian theorems. In Confer- ence on Computational Learning Theory (COLT), 1998.
  23. Nguyen, C., Do, T.-T., and Carneiro, G. PAC-Bayes meta- learning with implicit task-specific posteriors. IEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI), 45(1):841-851, 2022.
  24. Nichol, A., Achiam, J., and Schulman, J. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018. · arXiv
  25. Pentina, A. and Lampert, C. H. A PAC-Bayesian bound for lifelong learning. In International Conference on Machine Learning (ICML), 2014.
  26. Pentina, A. and Lampert, C. H. Lifelong learning with non-iid tasks. In Conference on Neural Information Pro- cessing Systems (NeurIPS), 2015.
  27. Pérez-Ortiz, M., Rivasplata, O., Shawe-Taylor, J., and Szepesvári, C. Tighter risk certificates for neural net- works. The Journal of Machine Learning Research, 22 (1):10326-10365, 2021.
  28. Ravi, S. and Larochelle, H. Optimization as a model for few- shot learning. In International Conference on Learning Representations (ICLR), 2017.
  29. Rezazadeh, A. A unified view on PAC-Bayes bounds for meta-learning. In International Conference on Machine Learning (ICML), 2022.
  30. Riou, C., Alquier, P., and Chérief-Abdellatif, B.-E. Bayes meets Bernstein at the meta level: an analysis of fast rates in meta-learning with PAC-Bayes. arXiv preprint arXiv:2302.11709, 2023. · arXiv
  31. Rothfuss, J., Fortuin, V., Josifoski, M., and Krause, A. PACOH: Bayes-optimal meta-learning with PAC- guarantees. In International Conference on Machine Learning (ICML), 2021.
  32. Rothfuss, J., Josifoski, M., Fortuin, V., and Krause, A. Scal- able PAC-Bayesian Meta-Learning via the PAC-Optimal Hyper-Posterior: From Theory to Practice. Journal of Machine Learning Research (JMLR), 2023.
  33. Scott, J., Zakerinia, H., and Lampert, C. H. PeFLL: Per- sonalized Federated Learning by Learning to Learn. In International Conference on Learning Representations (ICLR), 2024.
  34. Seldin, Y., Laviolette, F., Cesa-Bianchi, N., Shawe-Taylor, J., and Auer, P. PAC-Bayesian inequalities for martin- gales. IEEE Transactions on Information Theory, 58(12): 7086-7093, 2012.
  35. Thrun, S. and Pratt, L. (eds.). Learning to Learn. Kluwer Academic Press, 1998.
  36. Zhao, D., Kobayashi, S., Sacramento, J., and von Oswald, J. Meta-learning via hypernetworks. In NeurIPS Workshop on Meta-Learning (MetaLearn), 2020.

Cited by

Sorted by citations

Authors
Zakerinia, H., Behjati, A., & Lampert, C. H.
ORCID
Lampert, Christoph H.
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.04054
arXiv
2402.04054
OpenAlex
W4391667454
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
meta-learning, generalization bounds, knowledge transfer between tasks, meta-learner

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review