The DOI matches its DataCite record; 58 of the 58 references it cites were found in a registry or an index.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Open accessAbstract
Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Min et al. [44] has shown that rather than learning new tasks, in-context learning [42] works by activating patterns from the training data.”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [44]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Self-citationChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602
Automatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,101
Research works printed but not linked: 4
- Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fer- nandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP.
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop.
- Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. ArXiv.
- Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019b. A survey of zero-shot learning: Set- tings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST).
Cited by
Sorted by citations
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
Towards Mitigating LLM Hallucination via Self Reflection
Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., & Fung, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.123 · Cited by 362
Query Rewriting in Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.322 · Cited by 602
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253
Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2024) · JMIR Medical Informatics · doi:10.2196/55318 · Cited by 293
GraphGPT: Graph Instruction Tuning for Large Language Models
Tang, J., Yang, Y., Wei, W., Shi, L., Su, L., Cheng, S., Yin, D., & Huang, C. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657775 · Cited by 249
Lu, G., Ju, X., Chen, X., Pei, W., & Cai, Z. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112031 · Cited by 224
STaR: Bootstrapping Reasoning With Reasoning
Zelikman, E., Wu, Y., Mu, J., & Goodman, N. D. (2022) · arXiv · doi:10.48550/arxiv.2203.14465 · Cited by 182
GPT-RE: In-context Learning for Relation Extraction using Large Language Models
Wan, Z., Cheng, F., Mao, Z., Liu, Q., Song, H., Li, J., & Kurohashi, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.214 · Cited by 241
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H. W., Shu, T., Song, M., Xing, E. P., & Hu, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.222 · Cited by 406
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 571
Huang, X., Ruan, W., Huang, W., Jin, G., Dong, Y., Wu, C., Bensalem, S., Mu, R., Yi, Q., Zhao, X., et al. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10824-0 · Cited by 228
Dynamic prompt-based virtual assistant framework for BIM information search
Zheng, J., & Fischer, M. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105067 · Cited by 154
Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3764579 · Cited by 385
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Prompt engineering as a new 21st century skill
Federiakin, D. A., Molerov, D., Zlatkin‐Troitschanskaia, O., & Maur, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1366434 · Cited by 223
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 346
Can language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142
Personality Traits in Large Language ModelsPreprint
Serapio‐García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Abdulhai, M., Faust, A., & Matarić, M. J. (2023) · Research Square · doi:10.21203/rs.3.rs-3296728/v1 · Cited by 137
Biomedical knowledge graph-optimized prompt generation for large language models
Soman, K., Rose, P. W., Morris, J. H., Akbas, R. E., Smith, B., Peetoom, B., Villouta-Reyes, C., Cerono, G., Shi, Y., Rizk-Jackson, A., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae560 · Cited by 135
Explaining machine learning models with interactive natural language conversations using TalkToModel
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00692-8 · Cited by 125
GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills
Abdelghani, R., Wang, Y.-H., Yuan, X., Wang, T., Lucas, P., Sauzéon, H., & Oudeyer, P.-Y. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00340-7 · Cited by 129
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.153 · Cited by 120
Self-citationA survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 282
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Dai, S.-C., Xiong, A., & Ku, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.669 · Cited by 215
Schneider, J., Meske, C., & Kuss, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00851-0 · Cited by 145
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
Compositional Chain-of-Thought Prompting for Large Multimodal Models
Mitra, C., Huang, B., Darrell, T., & Herzig, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01367 · Cited by 150
Materials science in the era of large language models: a perspective
Lei, G., Docherty, R., & Cooper, S. J. (2024) · Digital Discovery · doi:10.1039/d4dd00074a · Cited by 130
Active Example Selection for In-Context Learning
Zhang, Y., Shi, F., & Tan, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.622 · Cited by 180
Artificial intelligence co-piloted auditing
Gu, H., Schreyer, M., Moffitt, K. C., & Vasarhelyi, M. A. (2024) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2024.100698 · Cited by 151
Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models
Louis, A., DIJCK, G. V., & Spanakis, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30232 · Cited by 132
LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
Qu, L., Wu, S., Fei, H., Nie, L., & Chua, T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612012 · Cited by 118
Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text
Mihindukulasooriya, N., Tiwari, S. M., Enguix, C. F., & Lata, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47243-5_14 · Cited by 126
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89
Prompting PaLM for Translation: Assessing Strategies and Performance
Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., & Foster, G. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.859 · Cited by 185
Continual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2025) · ACM Computing Surveys · doi:10.1145/3735633 · Cited by 291
Is GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings
Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1272229 · Cited by 115
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.779 · Cited by 171
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Yang, C., Zhu, Y. T., Wang, L., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3699518 · Cited by 128
In-context Examples Selection for Machine Translation
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., & Ghazvininejad, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.564 · Cited by 161
Self-citationOn Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76
Few-shot In-context Learning on Knowledge Base Question Answering
Li, T., Ma, X., Zhuang, A., Gu, Y., Su, Y., & Chen, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.385 · Cited by 126
Wu, Z., Wang, Y., Ye, J., & Kong, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.79 · Cited by 158
Evaluating Generative AI in Mental Health: Systematic Review of Capabilities and Limitations
Wang, L., Bhanushali, T., Huang, Z., Yang, J., Badami, S., & Hightow-Weidman, L. (2025) · JMIR Mental Health · doi:10.2196/70014 · Cited by 73
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Nadas, M. D., Dioşan, L. S., & Tomescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3589503 · Cited by 161
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 91
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 145
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Honovich, O., Shaham, U., Bowman, S. R., & Levy, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.108 · Cited by 160
Code4Struct: Code Generation for Few-Shot Event Structure Prediction
Wang, X., Li, S., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.202 · Cited by 128
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 84
Extracting Financial Data from Unstructured Sources: Leveraging Large Language Models
Li, H., Gao, H., Wu, C., & Vasarhelyi, M. A. (2024) · Journal of Information Systems · doi:10.2308/isys-2023-047 · Cited by 76
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
Toward large reasoning models: A survey of reinforced reasoning with large language models
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101370 · Cited by 533
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors
Li, P., Sun, T., Tang, Q., Yan, H. R., Wu, Y., Huang, X., & Qiu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.855 · Cited by 100
Yang, C.-H. H., Gu, Y., Liu, Y.-C., Ghosh, S., Bulyko, I., & Stolcke, A. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389673 · Cited by 68
LLaFS: When Large Language Models Meet Few-Shot Segmentation
Zhu, L., Chen, T., Ji, D., Ye, J., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00296 · Cited by 65
DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
Du, M., Yue, X., Chow, S. S. M., Wang, T., Huang, C., & Sun, H. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3616592 · Cited by 82
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2024) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 133
Automated generation of discharge summaries: leveraging large language models with clinical data
Ganzinger, M., Kunz, N., Fuchs, P., Lyu, C. K., Loos, M., Dugas, M., & Pausch, T. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-01618-7 · Cited by 59
PLeak: Prompt Leaking Attacks against Large Language Model Applications
Hui, B., Yuan, H., Gong, N., Burlina, P., & Cao, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670370 · Cited by 55
One-Shot Labeling for Automatic Relevance Estimation
MacAvaney, S., & Soldaini, L. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3592032 · Cited by 65
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3660147 · Cited by 51
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Li, X. L., Kuncoro, A., Hoffmann, J., d’Autume, C. D. M., Blunsom, P., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.812 · Cited by 84
Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning
Lee, D., Ahrabian, K., Jin, W., Morstatter, F., & Pujara, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.36 · Cited by 80
HiGPT: Heterogeneous Graph Language Model
Tang, J., Yang, Y., Wei, W., Shi, L., Xia, L., Yin, D., & Huang, C. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671987 · Cited by 80
LLMs to the Moon? Reddit Market Sentiment Analysis with Large Language Models
Deng, X., Bashlovkina, V., Han, F., Baumgartner, S., & Bendersky, M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587605 · Cited by 51
What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis
Deng, X., Bashlovkina, V., Han, F., Baumgartner, S., & Bendersky, M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587324 · Cited by 55
Complementary Explanations for Effective In-Context Learning
Ye, X., Iyer, S. V., Çelikyılmaz, A., Stoyanov, V., Durrett, G., & Pasunuru, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.273 · Cited by 79
Prompting is not a substitute for probability measurements in large language models
Hu, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.306 · Cited by 60
Xpert: empowering incident management with query recommendations via large language models
Jiang, Y., Zhang, C., He, S., Yang, Z., Ma, M., Qin, S., Kang, Y., Dang, Y., Rajmohan, S., Lin, Q., et al. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639081 · Cited by 45
Cost-Effective In-Context Learning for Entity Resolution: A Design Space Exploration
Fan, M., Han, X., Fan, J., Chai, C., Tang, N., Li, G., & Du, X. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00284 · Cited by 58
KoSEL: Knowledge subgraph enhanced large language model for medical question answering
Zeng, Z., Cheng, Q., Hu, X., Zhuang, Y., Liu, X., He, K., & Liu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112837 · Cited by 46
Jin, W., Gao, Y., Tao, T., Wang, X., Wang, N., Wu, B., & Zhao, B. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/5920142 · Cited by 50
Large language models meet NLP: a survey
Qin, L., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y., Li, M., Che, W., & Yu, P. S. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50472-3 · Cited by 203
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S., Lee, S., & Kim, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.155 · Cited by 91
A comparative study of cross-lingual sentiment analysis
Přibáň, P., Šmíd, J., Steinberger, J., & Mištera, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123247 · Cited by 49
Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models
Zhang, T., Irsan, I. C., Thung, F., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697009 · Cited by 63
Zhang, H., Sediq, A. B., Afana, A., & Erol‐Kantarci, M. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901312 · Cited by 40
Zero-Shot Cross-Lingual Summarization via Large Language Models
Wang, J., Liang, Y., Meng, F., Zou, B., Li, Z., Qu, J., & Zhou, J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.2 · Cited by 81
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41
Learning In-context Learning for Named Entity Recognition
Chen, J., Lu, Y., Lin, H., Lou, J., Jia, W., Dai, D., Wu, H. R., Cao, B., Han, X., & Sun, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.764 · Cited by 55
Towards Making the Most of LLM for Translation Quality Estimation
Huang, H., Wu, S., Liang, X., Wang, B., Shi, Y., Wu, P., Yang, M., & Zhao, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_30 · Cited by 36
Generative Type Inference for Python
Peng, Y. F., Wang, C., Wang, W., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00031 · Cited by 47
Remadi, A., Hage, K. E., Hobeika, Y., & Bugiotti, F. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102313 · Cited by 36
Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
Ye, F., Fang, M., Li, S., & Yılmaz, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.398 · Cited by 72
Large Language Model for Multiobjective Evolutionary Optimization
Liu, F., Lin, X., Yao, S., Wang, Z., Tong, X., Yuan, M., & Zhang, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-3538-2_13 · Cited by 42
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.380 · Cited by 43
Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., & Sun, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.609 · Cited by 172
Metadata
- Authors
- Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 11048-11064
- arXiv
- 2202.12837
- OpenAlex
- W4385567149
- Cited by
- 752
- References
- 57
- Citations by year
- 2026: 135 · 2025: 227 · 2024: 179 · 2023: 191 · 2022: 20
- FWCI
- 85.30
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2044660)
- Office of Naval Research (N00014-18, N00014-18-1-2826, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- in-context learning, demonstrations, large language models, label noise, input distribution, label space, GPT-3, classification tasks, multiple-choice tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
