Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Open accessAbstract
Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Min et al. has shown that in-context learning works by activating patterns from the training data.
“Min et al. [44] has shown that rather than learning new tasks, in-context learning [42] works by activating patterns from the training data.”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [44]
In §6 (page unavailable), the authors suggest that, under a strict definition of learning, models do not learn new tasks at test time and may instead use priors from pretraining. But they qualify this as a suggestion and add that, under a broader definition, models do learn from demonstrations. The citing sentence presents the proposed explanation as a categorical account of how in-context learning works. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Self-citationExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Self-citationChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,991 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
Automatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,101 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1× · see citing works
“This is in line with Reynolds and McDonell (2021) who claim that the demonstrations are for task location and the intrinsic ability to perform the task is obtained at pretraining time.”
ResultLevesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
The Third PASCAL Recognizing Textual Entailment Challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · ACL-PASCAL@ACL · doi:10.3115/1654536.1654538 · Cited by 801 · see citing works
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
Barbieri, F., Camacho-Collados, J., Anke, L. E., & Neves, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.148 · Cited by 756 · see citing works
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747 · see citing works
A SICK cure for the evaluation of compositional distributional semantic models
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Raffaella Bernardi, B., & Zamparelli, R. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39qdhuevzbqa · Cited by 680 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · see citing works
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
Self-citationGood debt or bad debt: Detecting semantic orientations in economic texts
Malo, P., Sinha, A., Korhonen, P., Wallenius, J., & Takala, P. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23062 · Cited by 513 · see citing works
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
The CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 465 · see citing works
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
Hate Speech Dataset from a White Supremacy Forum
de Gibert, O., Pérez, N., Pablos, A. G., & Cuadros, M. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5102 · Cited by 428 · see citing works
Unsupervised Domain Adaptation for Zero-Shot Learning
Kodirov, E., Xiang, T., Fu, Z., & Gong, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.282 · Cited by 415 · see citing works
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · see citing works
Datasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · see citing works
CommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 296 · see citing works
Cross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · see citing works
Self-citationDREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
Sun, K., Yu, D., Chen, J., Yu, D., Choi, Y., & Cardie, C. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00264 · Cited by 245 · see citing works
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A., & Pavlick, E. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.167 · Cited by 203 · see citing works
Muppet: Massive Multi-task Representations with Pre-Finetuning
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.468 · Cited by 183 · see citing works
Self-citationETHOS: a multi-label hate speech detection dataset
Mollas, I., Chrysopoulou, Z., Karlos, S., & Tsoumakas, G. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00608-2 · Cited by 181 · see citing works
What Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148 · see citing works
Noisy Channel Language Model Prompting for Few-Shot Text Classification
Min, S., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.365 · Cited by 142 · see citing works
Self-citationReframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140 · see citing works
Self-citationAn Explanation of In-context Learning as Implicit Bayesian Inference
Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2111.02080 · Cited by 138 · see citing works
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Logan IV, R., Balazevic, I., Wallace, E., Petroni, F., Singh, S., & Riedel, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.222 · Cited by 133 · see citing works
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
Self-citationEfficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S. V., Pasunuru, R., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.804 · Cited by 122 · see citing works
Self-citationCrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 120 · see citing works
CLIMATE-FEVER: A Dataset for Verification of Real-World Climate ClaimsPreprint
Diggelmann, T., Boyd-Graber, J. L., Bulian, J., Ciaramita, M., & Leippold, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00614 · Cited by 91 · see citing works
Meta-learning via Language Model In-context Tuning
Chen, Y., Zhong, R., Zha, S., Karypis, G., & He, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.53 · Cited by 84 · see citing works
SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Gordon, A. S., Kozareva, Z., & Roemmele, M. (2012) · Cited by 84 · see citing works
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions
Tafjord, O., Gardner, M., Lin, K., & Clark, P. E. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1608 · Cited by 64 · see citing works
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2021) · arXiv · doi:10.48550/arxiv.2110.15943 · Cited by 61 · see citing works
Self-citationGround-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S., Lee, S., & Kim, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.155 · Cited by 55 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Reasoning
Razeghi, Y., Logan, R. L., Gardner, M., & Singh, S. (2022) · arXiv · doi:10.48550/arxiv.2202.07206 · Cited by 53 · see citing works
The Turking Test: Can Language Models Understand Instructions?Preprint
Efrat, A., & Levy, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11982 · Cited by 43 · see citing works
Investigating Societal Biases in a Poetry Composition SystemPreprint
Sheng, E., & Uthus, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.02686 · Cited by 24 · see citing works
Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango
Madaan, A., & Yazdanbakhsh, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07686 · Cited by 24 · see citing works
QASC: A Dataset for Question Answering via Sentence Composition
Khot, T., Clark, P., Guerquin, M., Jansen, P., & Sabharwal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6319 · Cited by 20 · see citing works
McCreery, C. H., Katariya, N., Kannan, A., Chablani, M., & Amatriain, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.13546 · Cited by 12 · see citing works
QUAREL: A Dataset and Models for Answering Questions about Qualitative Relationships
Tafjord, O., Clark, P. E., Gardner, M., Yih, W.-T., & Sabharwal, A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017063 · Cited by 5 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 4
- Michael Chen, Mike D'Arcy, Alisa Liu, Jared Fer- nandez, and Doug Downey. 2019. CODAH: An adversarially-authored question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP.
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges Workshop.
- Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020. Ethos: an online hate speech detection dataset. ArXiv.
- Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019b. A survey of zero-shot learning: Set- tings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST).
Cited by
Sorted by citations
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · see citing works
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369 · see citing works
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367 · see citing works
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · see citing works
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
Towards Mitigating LLM Hallucination via Self Reflection
Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., & Fung, P. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.findings-emnlp.123 · Cited by 563 · see citing works
Query Rewriting in Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.322 · Cited by 391 · see citing works
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441 · see citing works
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., & Wang, Y. (2024) · JMIR Medical Informatics · doi:10.2196/55318 · Cited by 253 · see citing works
GraphGPT: Graph Instruction Tuning for Large Language Models
Tang, J., Yang, Y., Wei, W., Shi, L., Su, L., Cheng, S., Yin, D., & Huang, C. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657775 · Cited by 249 · see citing works
Lu, G., Ju, X., Chen, X., Pei, W., & Cai, Z. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112031 · Cited by 197 · see citing works
GPT-RE: In-context Learning for Relation Extraction using Large Language Models
Wan, Z., Cheng, F., Mao, Z., Liu, Q., Song, H., Li, J., & Kurohashi, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.214 · Cited by 199 · see citing works
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H. W., Shu, T., Song, M., Xing, E. P., & Hu, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.222 · Cited by 233 · see citing works
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 253 · see citing works
Huang, X., Ruan, W., Huang, W., Jin, G., Dong, Y., Wu, C., Bensalem, S., Mu, R., Yi, Q., Zhao, X., et al. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10824-0 · Cited by 181 · see citing works
Dynamic prompt-based virtual assistant framework for BIM information search
Zheng, J., & Fischer, M. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.105067 · Cited by 154 · see citing works
Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3764579 · Cited by 263 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
Prompt engineering as a new 21st century skill
Federiakin, D. A., Molerov, D., Zlatkin‐Troitschanskaia, O., & Maur, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1366434 · Cited by 145 · see citing works
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
Can language models learn from explanations in context?
Lampinen, A., Dasgupta, I., Chan, S., Mathewson, K., Tessler, M., Creswell, A., McClelland, J., Wang, J., & Hill, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.38 · Cited by 142 · see citing works
Personality Traits in Large Language ModelsPreprint
Serapio‐García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Abdulhai, M., Faust, A., & Matarić, M. J. (2023) · Research Square · doi:10.21203/rs.3.rs-3296728/v1 · Cited by 137 · see citing works
Biomedical knowledge graph-optimized prompt generation for large language models
Soman, K., Rose, P. W., Morris, J. H., Akbas, R. E., Smith, B., Peetoom, B., Villouta-Reyes, C., Cerono, G., Shi, Y., Rizk-Jackson, A., et al. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae560 · Cited by 135 · see citing works
Explaining machine learning models with interactive natural language conversations using TalkToModel
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00692-8 · Cited by 125 · see citing works
GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills
Abdelghani, R., Wang, Y.-H., Yuan, X., Wang, T., Lucas, P., Sauzéon, H., & Oudeyer, P.-Y. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00340-7 · Cited by 129 · see citing works
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.153 · Cited by 120 · see citing works
Self-citationA survey of multilingual large language models
Qin, L., Chen, Q., Zhou, Y., Zhi, C., Li, Y., Liao, L., Li, M., Che, W., & Yu, P. S. (2025) · Patterns · doi:10.1016/j.patter.2024.101118 · Cited by 197 · see citing works
LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
Dai, S.-C., Xiong, A., & Ku, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.669 · Cited by 152 · see citing works
Schneider, J., Meske, C., & Kuss, P. M. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-024-00851-0 · Cited by 203 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Compositional Chain-of-Thought Prompting for Large Multimodal Models
Mitra, C., Huang, B., Darrell, T., & Herzig, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01367 · Cited by 115 · see citing works
Materials science in the era of large language models: a perspective
Lei, G., Docherty, R., & Cooper, S. J. (2024) · Digital Discovery · doi:10.1039/d4dd00074a · Cited by 109 · see citing works
Active Example Selection for In-Context Learning
Zhang, Y., Shi, F., & Tan, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.622 · Cited by 100 · see citing works
Artificial intelligence co-piloted auditing
Gu, H., Schreyer, M., Moffitt, K. C., & Vasarhelyi, M. A. (2024) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2024.100698 · Cited by 120 · see citing works
Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models
Louis, A., DIJCK, G. V., & Spanakis, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30232 · Cited by 108 · see citing works
LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
Qu, L., Wu, S., Fei, H., Nie, L., & Chua, T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612012 · Cited by 104 · see citing works
Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text
Mihindukulasooriya, N., Tiwari, S. M., Enguix, C. F., & Lata, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47243-5_14 · Cited by 105 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
Prompting PaLM for Translation: Assessing Strategies and Performance
Vilar, D., Freitag, M., Cherry, C. A., Luo, J., Ratnakar, V., & Foster, G. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.859 · Cited by 125 · see citing works
Continual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2025) · ACM Computing Surveys · doi:10.1145/3735633 · Cited by 136 · see citing works
Is GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings
Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1272229 · Cited by 94 · see citing works
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.779 · Cited by 103 · see citing works
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Yang, C., Zhu, Y. T., Wang, L., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3699518 · Cited by 93 · see citing works
In-context Examples Selection for Machine Translation
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., & Ghazvininejad, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.564 · Cited by 106 · see citing works
Self-citationOn Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
Few-shot In-context Learning on Knowledge Base Question Answering
Li, T., Ma, X., Zhuang, A., Gu, Y., Su, Y., & Chen, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.385 · Cited by 75 · see citing works
Wu, Z., Wang, Y., Ye, J., & Kong, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.79 · Cited by 99 · see citing works
Evaluating Generative AI in Mental Health: Systematic Review of Capabilities and Limitations
Wang, L., Bhanushali, T., Huang, Z., Yang, J., Badami, S., & Hightow-Weidman, L. (2025) · JMIR Mental Health · doi:10.2196/70014 · Cited by 73 · see citing works
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71 · see citing works
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72 · see citing works
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Nadas, M. D., Dioşan, L. S., & Tomescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3589503 · Cited by 99 · see citing works
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 67 · see citing works
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Honovich, O., Shaham, U., Bowman, S. R., & Levy, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.108 · Cited by 66 · see citing works
Code4Struct: Code Generation for Few-Shot Event Structure Prediction
Wang, X., Li, S., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.202 · Cited by 63 · see citing works
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 71 · see citing works
Extracting Financial Data from Unstructured Sources: Leveraging Large Language Models
Li, H., Gao, H., Wu, C., & Vasarhelyi, M. A. (2024) · Journal of Information Systems · doi:10.2308/isys-2023-047 · Cited by 71 · see citing works
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 72 · see citing works
Toward large reasoning models: A survey of reinforced reasoning with large language models
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101370 · Cited by 186 · see citing works
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors
Li, P., Sun, T., Tang, Q., Yan, H. R., Wu, Y., Huang, X., & Qiu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.855 · Cited by 80 · see citing works
Yang, C.-H. H., Gu, Y., Liu, Y.-C., Ghosh, S., Bulyko, I., & Stolcke, A. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389673 · Cited by 58 · see citing works
LLaFS: When Large Language Models Meet Few-Shot Segmentation
Zhu, L., Chen, T., Ji, D., Ye, J., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00296 · Cited by 59 · see citing works
DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass
Du, M., Yue, X., Chow, S. S. M., Wang, T., Huang, C., & Sun, H. (2023) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3576915.3616592 · Cited by 63 · see citing works
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2024) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 92 · see citing works
Automated generation of discharge summaries: leveraging large language models with clinical data
Ganzinger, M., Kunz, N., Fuchs, P., Lyu, C. K., Loos, M., Dugas, M., & Pausch, T. M. (2025) · Scientific Reports · doi:10.1038/s41598-025-01618-7 · Cited by 56 · see citing works
<scp>PLeak:</scp> Prompt Leaking Attacks against Large Language Model Applications
Hui, B., Yuan, H., Gong, N., Burlina, P., & Cao, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670370 · Cited by 55 · see citing works
One-Shot Labeling for Automatic Relevance Estimation
MacAvaney, S., & Soldaini, L. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3592032 · Cited by 65 · see citing works
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52 · see citing works
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3660147 · Cited by 51 · see citing works
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Li, X. L., Kuncoro, A., Hoffmann, J., d’Autume, C. D. M., Blunsom, P., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.812 · Cited by 59 · see citing works
Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning
Lee, D., Ahrabian, K., Jin, W., Morstatter, F., & Pujara, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.36 · Cited by 50 · see citing works
HiGPT: Heterogeneous Graph Language Model
Tang, J., Yang, Y., Wei, W., Shi, L., Xia, L., Yin, D., & Huang, C. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671987 · Cited by 50 · see citing works
LLMs to the Moon? Reddit Market Sentiment Analysis with Large Language Models
Deng, X., Bashlovkina, V., Han, F., Baumgartner, S., & Bendersky, M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587605 · Cited by 46 · see citing works
What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis
Deng, X., Bashlovkina, V., Han, F., Baumgartner, S., & Bendersky, M. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587324 · Cited by 46 · see citing works
Complementary Explanations for Effective In-Context Learning
Ye, X., Iyer, S. V., Çelikyılmaz, A., Stoyanov, V., Durrett, G., & Pasunuru, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.273 · Cited by 46 · see citing works
Prompting is not a substitute for probability measurements in large language models
Hu, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.306 · Cited by 60 · see citing works
Xpert: empowering incident management with query recommendations via large language models
Jiang, Y., Zhang, C., He, S., Yang, Z., Ma, M., Qin, S., Kang, Y., Dang, Y., Rajmohan, S., Lin, Q., et al. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639081 · Cited by 45 · see citing works
Cost-Effective In-Context Learning for Entity Resolution: A Design Space Exploration
Fan, M., Han, X., Fan, J., Chai, C., Tang, N., Li, G., & Du, X. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00284 · Cited by 44 · see citing works
KoSEL: Knowledge subgraph enhanced large language model for medical question answering
Zeng, Z., Cheng, Q., Hu, X., Zhuang, Y., Liu, X., He, K., & Liu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112837 · Cited by 45 · see citing works
Jin, W., Gao, Y., Tao, T., Wang, X., Wang, N., Wu, B., & Zhao, B. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/5920142 · Cited by 47 · see citing works
Large language models meet NLP: a survey
Qin, L., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y., Li, M., Che, W., & Yu, P. S. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50472-3 · Cited by 127 · see citing works
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Yoo, K. M., Kim, J., Kim, H. J., Cho, H., Jo, H., Lee, S., Lee, S., & Kim, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.155 · Cited by 55 · see citing works
A comparative study of cross-lingual sentiment analysis
Přibáň, P., Šmíd, J., Steinberger, J., & Mištera, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123247 · Cited by 41 · see citing works
Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models
Zhang, T., Irsan, I. C., Thung, F., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697009 · Cited by 41 · see citing works
Zhang, H., Sediq, A. B., Afana, A., & Erol‐Kantarci, M. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901312 · Cited by 40 · see citing works
Zero-Shot Cross-Lingual Summarization via Large Language Models
Wang, J., Liang, Y., Meng, F., Zou, B., Li, Z., Qu, J., & Zhou, J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.2 · Cited by 62 · see citing works
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
Learning In-context Learning for Named Entity Recognition
Chen, J., Lu, Y., Lin, H., Lou, J., Jia, W., Dai, D., Wu, H. R., Cao, B., Han, X., & Sun, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.764 · Cited by 44 · see citing works
Towards Making the Most of LLM for Translation Quality Estimation
Huang, H., Wu, S., Liang, X., Wang, B., Shi, Y., Wu, P., Yang, M., & Zhao, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_30 · Cited by 36 · see citing works
Generative Type Inference for Python
Peng, Y. F., Wang, C., Wang, W., Gao, C., & Lyu, M. R.-T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00031 · Cited by 36 · see citing works
Remadi, A., Hage, K. E., Hobeika, Y., & Bugiotti, F. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102313 · Cited by 36 · see citing works
Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
Ye, F., Fang, M., Li, S., & Yılmaz, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.398 · Cited by 51 · see citing works
Large Language Model for Multiobjective Evolutionary Optimization
Liu, F., Lin, X., Yao, S., Wang, Z., Tong, X., Yuan, M., & Zhang, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-3538-2_13 · Cited by 35 · see citing works
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.380 · Cited by 35 · see citing works
Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., & Sun, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.609 · Cited by 78 · see citing works
Meshkin, H., Zirkle, J., Arabidarrehdor, G., Chaturbedi, A., Chakravartula, S., Mann, J., Thrasher, B., & Li, Z. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae354 · Cited by 34 · see citing works
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning
Gupta, P., Jiao, C., Yeh, Y.-T., Mehri, S., Eskénazi, M., & Bigham, J. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.33 · Cited by 46 · see citing works
- Authors
- Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 11048-11064
- arXiv
- 2202.12837
- OpenAlex
- W4385567149
- Cited by
- 752
- References
- 57
- Citations by year
- 2026: 135 · 2025: 227 · 2024: 179 · 2023: 191 · 2022: 20
- FWCI
- 85.30
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2044660)
- Office of Naval Research (N00014-18, N00014-18-1-2826, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- in-context learning, demonstrations, large language models, label noise, input distribution, label space, GPT-3, classification tasks, multiple-choice tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
