CoChat
Sign Up
Cited by 563Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

What Makes Good In-Context Examples for GPT-3?

Open access
Liu, Jiachang, Shen, Dinghan, Zhang, Yizhe, Dolan, Bill, Carin, Lawrence, Chen, Weizhu·2022·Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
DOI 10.18653/v1/2022.deelio-1.10
Open this paper in CoChat — free accountOpen PDF

Abstract

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen. Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. 2022.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.deelio-1.10

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing works on ICL for coding tasks treat code examples as monotonic units and do not explore the contributing features within a code snippet.

    “Although existing works advance in optimizing ICL for various tasks including coding, they consider code examples as monotonic units and do not explore the contributing features in a code snippet.”

    From What Makes In-Context Examples Effective for Code Generation? (2026) · cites this paper as Liu et al. [47]

    The Trial section describes selecting whole in-context examples by similarity to a test sample, which supports the general point that this work selects examples as units. But Section 3, Experimental Setup, lists sentiment analysis, table-to-text generation, and question answering—not coding. The paper does not establish that existing coding-task work treats code snippets as indivisible units or neglects their contributing features. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Similarity-based methods select the top-k most similar exemplars, which are simple and fast but often yield homogeneous exemplars and sub-optimal results.

    “Similarity-based methods select the topk most similar exemplars [17], [25]-[28], which are simple and fast but often yield homogeneous exemplars and sub-optimal results.”

    From Post-Incorporating Code Structural Knowledge Into Pretrained Models via ICL for Code Translation (2025) · cites this paper as [17]

    Section 2.3 describes selecting the k nearest examples by embedding similarity. The paper does not say those examples are often homogeneous or that the method is fast; its introduction instead reports that retrieval-based examples outperform random sampling. It therefore supports the selection mechanism but not the claimed drawbacks or speed advantage. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • LLMs can perform better for Text-to-SQL through in-context learning.

    “However, LLMs can perform better for Text-to-SQL through in-context learning, in which only a few examples are provided in the input prompts.”

    From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [28]

    The Trial section reports that selected in-context examples improved GPT-3’s performance on several tasks, including table-to-text generation and question answering. The Experimental Setup section lists the tasks studied; Text-to-SQL is not among them, so the source does not establish the claim for that task. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The performance of in-context learning (ICL) strongly relies on the quality of demonstrations.

    “However, the performance of ICL heavily relies on the quality of demonstrations, e.g., the selected examples.”

    From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [20]

  • Verbal credit assignment from critiques lets models reach high-quality refinement trajectories through in-context learning, which standard scalar-reward exploration cannot reach.

    “This suggests that verbal credit assignment from critiques enables models to access high-quality refinement trajectories via in-context learning (Liu et al., 2022)-trajectories unreachable via standard exploration with scalar rewards alone.”

    From Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback (2025) · cites this paper as Liu et al., 2022

    The Trial and Fine-tuning sections discuss selecting retrieved examples to improve GPT-3’s in-context learning. Our copy does not address verbal credit assignment from critiques, refinement trajectories, or a comparison with scalar-reward exploration; searches for those terms found no supporting passage. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533-1544.
  2. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. · arXiv
  3. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. · arXiv
  4. Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. 2018. Skeleton- to-response: Dialogue generation guided by retrieval memory. arXiv preprint arXiv:1809.05296. · arXiv
  5. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017. Faithful to the original: Fact aware neural abstractive summarization. arXiv preprint arXiv:1711.04434. · arXiv
  6. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. · arXiv
  7. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine trans- lation. In AAAI, pages 5133-5140.
  8. Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit frame- work for predicting structured outputs. In Advances in Neural Information Processing Systems, pages 10052-10062.
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. · arXiv
  10. Frank Jäkel, Bernhard Schölkopf, and Felix A Wich- mann. 2008. Generalization and similarity in exem- plar models of categorization: Insights from machine learning. Psychonomic Bulletin & Review, 15(2):256- 271.
  11. Nora Kassner and Hinrich Schütze. 2020. Bert- knn: Adding a knn search component to pretrained language models for better qa. arXiv preprint arXiv:2005.00766. · arXiv
  12. Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710. · arXiv
  13. Guillaume Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining. arXiv preprint arXiv:1901.07291. · arXiv
  14. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401. · arXiv
  15. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: A simple ap- proach to sentiment and style transfer. arXiv preprint arXiv:1804.06437. · arXiv
  16. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
  17. Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1329-1338.
  18. Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to- text generation dataset. In Proceedings of EMNLP.
  19. Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text genera- tion with exemplar-based adaptive decoding. arXiv preprint arXiv:1904.04428. · arXiv
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  21. Nazneen Fatema Rajani, Ben Krause, Wengpeng Yin, Tong Niu, Richard Socher, and Caiming Xiong. 2020. Explaining and improving model behavior with k nearest neighbor representations. arXiv preprint arXiv:2010.09030. · arXiv
  22. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. · arXiv
  23. Nils Reimers and Iryna Gurevych. 2020. Mak- ing monolingual sentence embeddings multilin- gual using knowledge distillation. arXiv preprint arXiv:2004.09813. · arXiv
  24. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980. · arXiv
  25. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
  26. Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:1610.07149. · arXiv
  27. Eiichiro Sumita and HDA Hitoshi. 1991. Experiments and prospects of example-based machine translation. In 29th Annual Meeting of the Association for Com- putational Linguistics, pages 185-192.
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998-6008.
  29. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. · arXiv
  30. Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. · arXiv
  31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771. · arXiv
  32. Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhou- jun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7281-7288.
  33. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. · arXiv
  34. Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In Proceed- ings of the 39th International ACM SIGIR conference on Research and Development in Information Re- trieval, pages 55-64.
  35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In Advances in neural informa- tion processing systems, pages 5753-5763.
  36. Morteza Ziyadi, Yuting Sun, Abhishek Goswami, Jade Huang, and Weizhu Chen. 2020. Example- based named entity recognition. arXiv preprint arXiv:2008.10570. · arXiv

Cited by

Sorted by citations

Authors
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W.
Affiliations
  • Duke University (US)
  • Microsoft (United States) (US)
  • Meta (United States) (US)
ORCID
Venue
Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
Publisher
Association for Computational Linguistics
Published
2022
Pages
100-114
DOI
10.18653/v1/2022.deelio-1.10
OpenAlex
W3122241445
Cited by
563
References
16
Citations by year
2026: 101 · 2025: 172 · 2024: 131 · 2023: 140 · 2022: 11 · 2021: 8
FWCI
62.74
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
in-context learning, in-context examples, GPT-3, large language models, prompt engineering, knowledge extraction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review