CoChat
Sign Up
PreprintCited by 4Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization

Open access
Zhang, Yufeng, Zhang, Fengzhuo, Yang, Zhuoran, Wang, Zhaoran·2023·arXiv
DOI 10.48550/arxiv.2305.19420arXiv 2305.19420
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we conduct a comprehensive study of In-Context Learning (ICL) by addressing several open questions: (a) What type of ICL estimator is learned by large language models? (b) What is a proper performance metric for ICL and what is the error rate? (c) How does the transformer architecture enable ICL? To answer these questions, we adopt a Bayesian view and formulate ICL as a problem of predicting the response corresponding to the current covariate, given a number of examples drawn from a latent variable model. To answer (a), we show that, without updating the neural network parameters, ICL implicitly implements the Bayesian model averaging algorithm, which is proven to be approximately parameterized by the attention mechanism. For (b), we analyze the ICL performance from an online learning perspective and establish a $\mathcal{O}(1/T)$ regret bound for perfectly pretrained ICL, where $T$ is the number of examples in the prompt. To answer (c), we show that, in addition to encoding Bayesian model averaging via attention, the transformer architecture also enables a fine-grained statistical analysis of pretraining under realistic assumptions. In particular, we prove that the error of pretrained model is bounded by a sum of an approximation error and a generalization error, where the former decays to zero exponentially as the depth grows, and the latter decays to zero sublinearly with the number of tokens in the pretraining dataset. Our results provide a unified understanding of the transformer and its ICL ability with bounds on ICL regret, approximation, and generalization, which deepens our knowledge of these essential aspects of modern language models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.19420

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • One major reason why debate strategies can be effective is the ability of LLMs to adapt to additional information given in-context.

    “One major reason why debate strategies can be an effective tool is the ability of LLMs to adapt to additional information given in-context (Zhang et al., 2023).”

    From Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (2023) · cites this paper as Zhang et al., 2023

    The Introduction and §3 (Preliminary) describe LLMs learning from examples supplied in a prompt, supporting the in-context adaptation part of the claim. The paper does not address debate strategies or establish that this ability makes them effective. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. Agarwal, A., Kakade, S., Krishnamurthy, A. and Sun, W. (2020). Flambe: Structural com- plexity and representation learning of low rank MDPs. Advances in Neural Information Processing Systems, 33 20095-20107.
  2. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T. and Zhou, D. (2022). What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661. · arXiv
  3. Anthony, M., Bartlett, P. L., Bartlett, P. L. et al. (1999). Neural network learning: Theoreti- cal foundations, vol. 9. cambridge university press Cambridge.
  4. Bai, Y., Chen, F., Wang, H., Xiong, C. and Mei, S. (2023). Transformers as statisti- cians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637. · arXiv
  5. Bartlett, P. L., Foster, D. J. and Telgarsky, M. J. (2017). Spectrally-normalized margin bounds for neural networks. Neural Information Processing Systems.
  6. Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A. and Hjelm, D. (2018). Mutual information neural estimation. In International Conference on Machine Learning. PMLR.
  7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020). Language models are few-shot learners. Neural Information Processing Systems.
  8. Chan, S. C., Santoro, A., Lampinen, A. K., Wang, J. X., Singh, A., Richemond, P. H., McClelland, J. and Hill, F. (2022). Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05055. · arXiv
  9. Dai, D., Sun, Y., Dong, L., Hao, Y., Sui, Z. and Wei, F. (2022). Why can GPT learn In- Context? Language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559. · arXiv
  10. Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. · arXiv
  11. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M. and Hon, H.-W. (2019). Unified language model pre-training for natural language understanding and gener- ation. Advances in neural information processing systems, 32.
  12. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J. and Sui, Z. (2022). A survey for in-context learning. arXiv preprint arXiv:2301.00234. · arXiv
  13. Edelman, B. L., Goel, S., Kakade, S. and Zhang, C. (2021). Inductive biases and variable creation in self-attention mechanisms. arXiv preprint arXiv:2110.10090. · arXiv
  14. Elbrächter, D., Perekrestenko, D., Grohs, P. and Bölcskei, H. (2021). Deep neural network approximation theory. IEEE Transactions on Information Theory, 67 2581-2623.
  15. Feng, G., Gu, Y., Zhang, B., Ye, H., He, D. and Wang, L. (2023). Towards revealing the mys- tery behind chain of thought: a theoretical perspective. arXiv preprint arXiv:2305.15408. · arXiv
  16. Fukumizu, K. (2015). Nonparametric bayesian inference with kernel mean embedding. In Modern Methodology and Applications in Spatial-Temporal Modeling. Springer, 1-24.
  17. Garg, S., Tsipras, D., Liang, P. and Valiant, G. (2022). What can transformers learn in- context? A case study of simple function classes. arXiv preprint arXiv:2208.01066. · arXiv
  18. Garnelo, M. and Czarnecki, W. M. (2023). Exploring the space of key-value-query models with intention. arXiv preprint arXiv:2305.10203. · arXiv
  19. Hahn, M. and Goyal, N. (2023). A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971. · arXiv
  20. Honovich, O., Shaham, U., Bowman, S. R. and Levy, O. (2022). Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782. · arXiv
  21. Hron, J., Bahri, Y., Sohl-Dickstein, J. and Novak, R. (2020). Infinite attention: NNGP and NTK for deep attention networks. In International Conference on Machine Learning.
  22. Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S. et al. (2022). OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017. · arXiv
  23. Jiang, H. (2023). A latent space theory for emergent abilities in large language models. arXiv preprint arXiv:2304.09960. · arXiv
  24. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F. and Liu, Q. (2019). Tiny- bert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351. · arXiv
  25. Ke, G., He, D. and Liu, T.-Y. (2020). Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595. · arXiv
  26. Kim, H. J., Cho, H., Kim, J., Kim, T., Yoo, K. M. and Lee, S.-g. (2022). Self-generated in- context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082. · arXiv
  27. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. · arXiv
  28. Ledent, A., Mustafa, W., Lei, Y. and Kloft, M. (2021). Norm-based generalisation bounds for deep multi-class convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35.
  29. Li, Y., Ildiz, M. E., Papailiopoulos, D. and Oymak, S. (2023). Transformers as algorithms: Generalization and stability in in-context learning. arXiv preprint arXiv:2301.07067. · arXiv
  30. Liao, R., Urtasun, R. and Zemel, R. (2020). A pac-bayesian approach to generalization bounds for graph neural networks. arXiv preprint arXiv:2012.07690. · arXiv
  31. Lin, S. and Zhang, J. (2019). Generalization bounds for convolutional neural networks. arXiv preprint arXiv:1910.01487. · arXiv
  32. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L. and Chen, W. (2021). What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804. · arXiv
  33. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H. and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55 1-35.
  34. Malladi, S., Wettig, A., Yu, D., Chen, D. and Arora, S. (2022). A kernel-based view of lan- guage model fine-tuning. arXiv preprint arXiv:2210.05643. · arXiv
  35. Michel, P., Levy, O. and Neubig, G. (2019). Are sixteen heads really better than one? Ad- vances in neural information processing systems, 32.
  36. Min, S., Lewis, M., Zettlemoyer, L. and Hajishirzi, H. (2021). Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943. · arXiv
  37. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H. and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837. · arXiv
  38. Neyshabur, B., Bhojanapalli, S. and Srebro, N. (2017). A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564. · arXiv
  39. Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S. P. and Lucchi, A. (2022). Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. arXiv preprint arXiv:2206.03126. · arXiv
  40. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A. et al. (2022). Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35 27730- 27744.
  41. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J. et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR.
  42. Rubin, O., Herzig, J. and Berant, J. (2021). Learning to retrieve prompts for in-context learn- ing. arXiv preprint arXiv:2112.08633. · arXiv
  43. Song, L., Huang, J., Smola, A. and Fukumizu, K. (2009). Hilbert space embeddings of con- ditional distributions with applications to dynamical systems. In International Conference on Machine Learning.
  44. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. (2017). Attention is all you need. In Neural Information Processing Systems.
  45. von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A. and Vladymyrov, M. (2022). Transformers learn in-context by gradi- ent descent. arXiv preprint arXiv:2212.07677. · arXiv
  46. Vuckovic, J., Baratin, A. and Combes, R. T. d. (2020). A mathematical theory of attention. arXiv preprint arXiv:2007.02876. · arXiv
  47. Wang, X., Zhu, W. and Wang, W. Y. (2023). Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916. · arXiv
  48. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D. and Hajishirzi, H. (2022). Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. · arXiv
  49. Wasserman, L. (2000). Bayesian model selection and model averaging. Journal of Mathemat- ical Psychology, 44 92-107.
  50. Wei, C., Chen, Y. and Ma, T. (2022a). Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems, 35 12071-12083.
  51. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M. and Le, Q. V. (2021). Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. · arXiv
  52. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D. et al. (2022b). Emergent abilities of large language mod- els. arXiv preprint arXiv:2206.07682. · arXiv
  53. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q. and Zhou, D. (2022c). Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv
  54. Wies, N., Levine, Y. and Shashua, A. (2023). The learnability of in-context learning. arXiv preprint arXiv:2303.07895. · arXiv
  55. Xie, S. M., Raghunathan, A., Liang, P. and Ma, T. (2021). An explanation of in-context learning as implicit Bayesian inference. arXiv preprint arXiv:2111.02080. · arXiv
  56. Yang, G. (2020). Tensor programs II: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548. · arXiv
  57. Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, S. J. and Kumar, S. (2019). Are trans- formers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv:1912.10077. · arXiv
  58. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R. and Smola, A. J. (2017). Deep sets. Neural Information Processing Systems.
  59. Zhang, F., Liu, B., Wang, K., Tan, V. Y., Yang, Z. and Wang, Z. (2022a). Relational reason- ing via set transformers: Provable efficiency and applications to MARL. arXiv preprint arXiv:2209.09845. · arXiv
  60. Zhang, Z., Zhang, A., Li, M. and Smola, A. (2022b). Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493. · arXiv
  61. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q. and Chi, E. (2022a). Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. · arXiv
  62. Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H. and Ba, J. (2022b). Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. · arXiv

Cited by

Sorted by citations

Authors
Zhang, Y., Zhang, F., Yang, Z., & Wang, Z.
Affiliations
  • Northwestern University (US)
  • National University of Singapore (SG)
  • Yale University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2305.19420
arXiv
2305.19420
OpenAlex
W4379087156
Cited by
4
References
1
Citations by year
2025: 3 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
in-context learning, Bayesian model averaging, large language models, transformer models, attention mechanism, generalization error, approximation error, pre-training, latent variable models, error rate analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review