CoChat
Sign Up
Cited by 77Topic ModelingDomain Adaptation and Few-Shot LearningNatural Language Processing Techniques

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

Open access
Voronov, Anton, Wolf, Lena, Ryabinin, Max·2024·Findings of the Association for Computational Linguistics ACL 2024
DOI 10.18653/v1/2024.findings-acl.375
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples.The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning.In this work, we conduct a comprehensive study of the template format's influence on the in-context learning performance.We evaluate the impact of the prompt template across 21 models (from 770M to 70B parameters) and 4 standard classification datasets.We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level.More importantly, the best templates do not transfer between different setups and even between models of the same family.Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works.As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates.This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.Demonstrations (x, y) ("Worst film ever", 1) ("Awesome, I like it", 0) v I : text: {x} v O : target: {C[y]} C = (positive, negative) Intra-sep: " "; inter-sep: "\n" Templates v I : input: {x} v O : It was {C[y]}.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-acl.375

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Voronov et al. (2024) showed that no prompt format consistently performs best across models.

    “Voronov et al. (2024) further showed that no prompt format consistently performs best across models.”

    From Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs (2025) · cites this paper as Voronov et al., 2024

  • Existing studies have reported that LLMs are highly sensitive to prompt phrasing.

    “Although existing studies have reported that LLMs are highly sensitive to prompt phrasing (Voronov et al., 2024;Mizrahi et al., 2024)”

    From Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs (2025) · cites this paper as Voronov et al., 2024

  • Previous studies typically rely on heuristic evaluation, such as regular-expression-based answer extraction or log-likelihood scoring over candidates.

    “To investigate this, we find that previous studies (Voronov et al., 2024;Chatterjee et al., 2024) typically rely on heuristic evaluation, such as regular-expression-based answer extraction or loglikelihood scoring over candidates.”

    From Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs (2025) · cites this paper as Voronov et al., 2024

    Section 4.2 discusses DIRECT, CHANNEL, and CALIBRATION prediction methods, while Section 5.5 discusses sensitivity to prompt templates. Neither passage establishes that previous studies typically use heuristic evaluation or mentions regular-expression answer extraction or log-likelihood scoring over candidates. Searches for those terms and related descriptions found no passage addressing the claim. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. What learn- ing algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations.
  2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. Preprint, arXiv:2304.01373. · arXiv
  3. Sid Black, Stella Biderman, Eric Hallahan, Quentin An- thony, Leo Gao, Laurence Golding, Horace He, Con- nor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open- source autoregressive language model. Preprint, arXiv:2204.06745. · arXiv
  4. Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14014-14031, Toronto, Canada. Association for Computational Linguistics. · DOI
  5. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  6. Shivam Garg, Dimitris Tsipras, Percy Liang, and Gre- gory Valiant. 2022. What can transformers learn in- context? a case study of simple function classes. In Advances in Neural Information Processing Systems.
  7. Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. 2023. Prototypical calibration for few- shot learning of language models. In The Eleventh International Conference on Learning Representa- tions.
  8. Tin Kam Ho. 1995. Random decision forests. In Pro- ceedings of 3rd international conference on docu- ment analysis and recognition, volume 1, pages 278- 282. IEEE.
  9. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. Preprint, arXiv:2203.15556. · arXiv
  10. Bairu Hou, Joe O'Connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023. PromptBoosting: Black- box text classification with ten forward passes. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 13309-13324.
  11. PMLR.
  12. Jaccard. 1912. The distribution of the flora of the alpine zone. In New Phytologist, volume 11, pages 37-50.
  13. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438. · DOI · Paper page
  14. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hin- ton. 2012. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems, volume 25.
  15. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. Preprint, arXiv:1612.01474. · arXiv
  16. Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
  17. Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. 2023. The language of prompting: What linguistic properties make a prompt successful? In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 9210-9232, Singapore. Association for Computational Linguistics. · DOI
  18. Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
  19. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022a. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI · Paper page
  20. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022b. Fantastically or- dered prompts and where to find them: Overcom- ing few-shot prompt order sensitivity. Preprint, arXiv:2104.08786. · arXiv
  21. Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettle- moyer, and Hannaneh Hajishirzi. 2023. Z-icl: Zero- shot in-context learning with pseudo-demonstrations. Preprint, arXiv:2212.09865. · arXiv
  22. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5316-5330, Dublin, Ireland. As- sociation for Computational Linguistics. · DOI
  23. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022b. Noisy channel language model prompting for few-shot text classification. Preprint, arXiv:2108.04106. · arXiv
  24. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022c. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791-2809, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
  25. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022d. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI · Paper page
  26. Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. 2020. A metric learning reality check. In Computer Vision -ECCV 2020, pages 681-699, Cham. Springer International Publishing.
  27. Tai Nguyen and Eric Wong. 2023. In-context example selection with influences. Preprint, arXiv:2302.11042. · arXiv
  28. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics. · DOI · Paper page
  29. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. Preprint, arXiv:1409.1556. · arXiv
  30. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
  31. C. Spearman. 1904. The proof and measurement of association between two things. American Journal of Psychology, 15:88-103.
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. · arXiv
  33. Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023. Large lan- guage models are implicitly topic models: Explain- ing and finding good demonstrations for in-context learning. Preprint, arXiv:2301.11916. · arXiv
  34. Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023. The icl consistency test. Preprint, arXiv:2312.04945. · arXiv
  35. Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
  36. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. Larger language models do in-context learning dif- ferently. Preprint, arXiv:2303.03846. · arXiv
  37. Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong. 2023. Self-adaptive in-context learn- ing: An information compression perspective for in- context example selection and ordering. Preprint, arXiv:2212.10375. · arXiv
  38. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. In Interna- tional Conference on Learning Representations.
  39. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. Preprint, arXiv:2205.01068. · arXiv
  40. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28.
  41. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Im- proving few-shot performance of language models. CoRR, abs/2102.09690. · arXiv
  42. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. 2024. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In The Twelfth International Conference on Learning Repre- sentations.

Cited by

Sorted by citations

Metadata

Authors
Voronov, A., Wolf, L., & Ryabinin, M.
Affiliations
National Research University Higher School of Economics (RU)
Venue
Findings of the Association for Computational Linguistics ACL 2024
Publisher
Association for Computational Linguistics
Published
2024
Pages
6287-6310
DOI
10.18653/v1/2024.findings-acl.375
OpenAlex
W4402670305
Cited by
77
References
4
Citations by year
2026: 27 · 2025: 27 · 2024: 23
FWCI
12.16
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
in-context learning, prompt templates, template selection, test-time augmentation, large language models, few-shot learning, assessment consistency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review