The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Open accessAbstract
GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "N/A". We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“using the demonstrations with gold labels significantly improves the performance over no demonstrations”
From Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022) · cites this paper as Zhao et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Despite incontext learning consistently outperforming zeroshot inference on a wide range of tasks (Zhao et al., 2021; Liu et al., 2021)”
From Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022) · cites this paper as Zhao et al., 2021
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We use contextual calibration (Zhao et al., 2021) and class-balanced sampling when these techniques improve performance on a validation set.”
From Adapting Language Models to Compress Contexts (2023) · cites this paper as Zhao et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“This finding is consistent with the recency bias proposed by (Zhao et al., 2021): LMs are biased towards outputting answers that are towards the end of the prompt.”
From Understanding the Dark Side of LLMs' Intrinsic Self-Correction (2025) · cites this paper as Zhao et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“LLMs are shown to be sensitive to the prompt format (Zhao et al., 2021;Lu et al., 2021;Wei et al., 2023;Madaan & Yazdanbakhsh, 2022)”
From Large Language Models as Optimizers (2023) · cites this paper as Zhao et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“For instance, simple modifications such as calibrating output probabilities (Zhao et al., 2021; Holtzman et al., 2021) or using a noisy channel (Min et al., 2022a) have improved performance on a range of tasks.”
From Emergent Abilities of Large Language Models (2022) · cites this paper as Zhao et al., 2021
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Zhong et al. [152] first define a template using a discrete search method such as AutoPrompt [125]'s, initialize virtual tokens based on this discovered prompt, and then fine-tune the embeddings to increase task accuracy.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [152]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 4,834
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · cited 1×
“Few-shot learning—the ability to learn tasks with limited examples—is an important aspect of intelligence (Lake et al., 2015; Yogatama et al., 2019).”
BackgroundA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS , 2020.
- Schwartz, R., Sap, M., Konstas, I., Zilles, L., Choi, Y ., and Smith, N. A. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In ACL , 2017.
Cited by
Sorted by citations
Items 301–400 of 543
Ilagan, J. B., & Ilagan, J. R. (2023) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/azhf9 · Cited by 10
Democratizing Reasoning Ability: Tailored Learning from Large Language Model
Wang, Z., Huang, S., Liu, Y., Wang, J., Song, M., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.120 · Cited by 22
Delmas, M., Wysocka, M., & Freitas, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00520 · Cited by 13
Detect-Then-Resolve: Enhancing Knowledge Graph Conflict Resolution with Large Language Model
Peng, H., Zhang, P., Tang, J., Xu, H., & Zeng, W. (2024) · Mathematics · doi:10.3390/math12152318 · Cited by 9
Efficient and explainable sequential recommendation with language model
Li, Z., Zou, L., Li, C., & Li, C. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104122 · Cited by 11
Language model collaboration for relation extraction from classical Chinese historical documents
Tang, X., Wang, L., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104286 · Cited by 11
Few-shot Subgoal Planning with Language Models
Logeswaran, L., Fu, Y., Lee, M., & Lee, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.402 · Cited by 19
Few-shot Reranking for Multi-hop QA via Language Model Prompting
Khalifa, M., Logeswaran, L., Lee, M., Lee, H., & Wang, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.885 · Cited by 18
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations
Saha, S., Hase, P., Rajani, N. F., & Bansal, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.137 · Cited by 10
POE: Process of Elimination for Multiple Choice Reasoning
Ma, C., & Du, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.273 · Cited by 11
ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models
Mekala, D., Wolfe, J., & Roy, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.354 · Cited by 21
DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning
Venkateswaran, P., Duesterwald, E., & Isahagian, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.310 · Cited by 10
Classifying Conspiratorial Narratives at Scale: False Alarms and Erroneous Connections
Diab, A., Nefriana, R., & Lin, Y. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31318 · Cited by 9
FLamE: Few-shot Learning from Natural Language Explanations
Zhou, Y., Zhang, Y., & Tan, C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.372 · Cited by 12
Towards Few-Shot Identification of Morality Frames using In-Context Learning
Roy, S., Nakshatri, N. S., & Goldwasser, D. (2022) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/2022.nlpcss-1.20 · Cited by 13
Generating Data for Symbolic Language with Large Language Models
Ye, J., Li, C., Kong, L., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.523 · Cited by 18
The Benefits of Label-Description Training for Zero-Shot Text Classification
Gao, L., Ghosh, D., & Gimpel, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.853 · Cited by 14
MEAL: Stable and Active Learning for Few-Shot Prompting
Köksal, A., Schick, T., & Schuetze, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.36 · Cited by 19
Tomanek, K., Tobin, J., Venugopalan, S., Cave, R., Seaver, K., Green, J. R., & Heywood, R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447177 · Cited by 8
Meta-tuning Language Models to Answer Prompts Better.Preprint
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · arXiv (Cornell University) · Cited by 8
Self-citationA Prompt Example Construction Method Based on Clustering and Semantic Similarity
Chen, D., & Wang, J. (2024) · Systems · doi:10.3390/systems12100410 · Cited by 9
Soliman, G., Zaki, H., & Kilany, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100131 · Cited by 13
From local to global: Leveraging document graph for named entity recognition
Shang, Y.-M., Mao, H., Tian, T., Huang, H., & Mao, X.-L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113017 · Cited by 7
Leveraging Large Language Models for Wireless Symbol Detection via In-Context Learning
Abbas, M., Kar, K., & Chen, T. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901317 · Cited by 13
Prompt choice for large language models: Business communications
Komarova, E. V. (2025) · Philology and Culture · doi:10.26907/2782-4756-2025-79-1-66-74 · Cited by 11
van Velzen, M., van der Willigen, R. F., de Beer, V. J., de Graaf-Waar, H. I., Janssen, E., van Leeuwen, S., van der Willigen, M. F., van der Willigen, M. J., Renardus, G., Maaroufi, R. E., et al. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1644084 · Cited by 8
Revisiting Automated Prompting: Are We Actually Doing Better?
Zhou, Y., Zhao, Y., Shumailov, I., Mullins, R., & Gal, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.155 · Cited by 14
Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts
Le, T. N., Bai, F., & Ritter, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.197 · Cited by 13
Reliable Gradient-free and Likelihood-free Prompt Tuning
Shen, M., Ghosh, S. K., Sattigeri, P., Das, S., Bu, Y., & Wornell, G. W. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.183 · Cited by 8
Metric-Based In-context Learning: A Case Study in Text Simplification
Vadlamannati, S., & Şahin, G. G. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.18 · Cited by 8
PILLOW: Enhancing Efficient Instruction Fine-tuning via Prompt Matching
Qi, Z., Tan, X., Shi, S., Qu, C. Y., Xu, Y., & Qi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.45 · Cited by 9
Valuing Training Data via Causal Inference for In-Context Learning
Zhou, X., Ye, W., Lee, Z., Zou, L., & Zhang, S. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3546761 · Cited by 7
Advancing Few-Shot Named Entity Recognition with Large Language Model
Xiao, Y., Zou, J., & Yang, Q. (2025) · Applied Sciences · doi:10.3390/app15073838 · Cited by 10
Dual Context-Guided Continuous Prompt Tuning for Few-Shot Learning
Zhou, J., Tian, L., Yu, H., Xiao, Z., Su, H., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.8 · Cited by 7
Unified Prompt Learning Makes Pre-Trained Language Models Better Few-Shot Learners
Jin, F., Lu, J., & Zhang, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095738 · Cited by 7
Can Transformers Learn Optimal Filtering for Unknown Systems?
Du, Z., Balim, H., Oymak, S., & Özay, N. (2023) · IEEE Control Systems Letters · doi:10.1109/lcsys.2023.3335318 · Cited by 13
Revolutionizing High School Physics Education: A Novel Dataset
Anand, A., Addala, K., Baghel, K. K. S., Goel, A., Hira, M., Gupta, R., & Shah, R. R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49601-1_5 · Cited by 8
Universal Self-Adaptive Prompting
Wan, X., Sun, R., Nakhost, H., Dai, H., Eisenschlos, J. M., Arık, S. Ö., & Pfister, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.461 · Cited by 21
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
Lu, X., Brahman, F., West, P., Jung, J., Chandu, K. R., Ravichander, A., Ammanabrolu, P., Jiang, L., Ramnath, S., Dziri, N., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.424 · Cited by 30
Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning
Liu, H., & Wang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1060 · Cited by 10
SINC: Self-Supervised In-Context Learning for Vision-Language Tasks
Chen, Y.-S., Song, Y.-Z., Yeo, C. Y., Liu, B., Fu, J., & Shuai, H.-H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01415 · Cited by 7
Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
Yang, S., Kim, J., Jang, J., Ye, S., Lee, H., & Seo, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00666 · Cited by 12
Inspecting the concept knowledge graph encoded by modern language models
Aspillaga, C., Mendoza, M., & Soto, Á. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.263 · Cited by 9
Contextualized Sentiment Analysis using Large Language ModelsPreprint
Breitung, C., Kruthof, G., & Müller, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4615038 · Cited by 8
Data Similarity is Not Enough to Explain Language Model Performance
Yauney, G., Reif, E., & Mimno, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.695 · Cited by 13
Prompting with Pseudo-Code Instructions
Mishra, M., Kumar, P., Bhat, R. A., Murthy, R., Contractor, D., & Tamilselvam, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.939 · Cited by 18
Injecting Bias through Prompts: Analyzing the Influence of Language on LLMs
Torres, N., Ulloa, C., Araya, I., Ayala, M., & Jara, S. (2024) · International Conference of the Chilean Computer Science Society (SCCC) · doi:10.1109/sccc63879.2024.10767653 · Cited by 7
Shih, Y.-C., Wu, C.-Y., Huang, S., & Tsai, C.-Y. (2026) · Journal of Medical Internet Research · doi:10.2196/87193 · Cited by 6
On the Multilingual Capabilities of Very Large-Scale English Language Models
Armengol-Estapé, J., Bonet, O. D. G., & Melero, M. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4swrjz4r87aw · Cited by 6
Large Language Models as SocioTechnical Systems
Dhole, K. (2023) · doi:10.18653/v1/2023.bigpicture-1.6 · Cited by 7
Zhao, Y., Zhang, Y., Sui, D., Cao, L., Zhao, J., & Liu, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.331 · Cited by 13
Knowledge Mining using Generative AI for Causal Discovery in Electronics Production
Meier, S., Töper, F., Gebele, J., Rachinger, B., Klarmann, S., & Franke, J. (2024) · International Spring Seminar on Electronics Technology · doi:10.1109/isse61612.2024.10604102 · Cited by 6
Large Language Model-Based Optimization for System-Level Test Program Generation
Schwachhofer, D., Domanski, P., Becker, S., Wagner, S., Sauer, M., Pflüger, D., & Polian, I. (2024) · Proceedings · doi:10.1109/dft63277.2024.10753556 · Cited by 6
A Simple, Yet Effective Approach to Finding Biases in Code Generation
Mouselinos, S., Malinowski, M., & Michalewski, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.718 · Cited by 13
Investigating the Learning Behaviour of In-Context Learning: A Comparison with Supervised Learning
Wang, X., Wang, Y., Xu, C., Geng, X., Zhang, B., Tao, C., Rudzicz, F., Mercer, R. E., & Jiang, D. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230559 · Cited by 15
EdgeAL: An Edge Estimation Based Active Learning Approach for OCT Segmentation
Kadir, M. A., Alam, H., & Sonntag, D. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_8 · Cited by 6
LLM-driven Instruction Following: Progresses and Concerns
Yin, W., Ye, Q., Liu, P., Ren, X., & Schütze, H. (2023) · doi:10.18653/v1/2023.emnlp-tutorial.4 · Cited by 8
Large language models for biomolecular analysis: From methods to applications
Feng, R., Zhang, C., & Zhang, Y. (2024) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2024.117540 · Cited by 7
SE Factual Knowledge in Frozen Giant Code Model: A Study on FQN and Its Retrieval
Huang, Q. Y., Liao, D., Xing, Z., Yuan, Z., Lu, Q., Xu, X., & Lu, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3436883 · Cited by 13
An Empirical Study on Few-shot Knowledge Probing for Pretrained Language ModelsPreprint
He, T., Cho, K., & Glass, J. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02772 · Cited by 12
Grimoire is All You Need for Enhancing Large Language ModelsPreprint
Chen, D., Song, S., Yu, Q., Li, Z., Wang, W., Xiong, F., & Tang, B. (2024) · Research Square · doi:10.21203/rs.3.rs-3845612/v1 · Cited by 13
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning
Chen, W., Huang, Z., Xie, L., Lin, B., Li, H., Lu, L., Tian, X., Cai, D., Zhang, Y., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.01658 · Cited by 5
Karasenko, A., & Baier, D. (2025) · Journal of Business Economics · doi:10.1007/s11573-025-01232-z · Cited by 9
An Empirical Study on Commit Message Generation Using LLMs via In-Context Learning
Wu, Y., Wang, Y., Li, Y., Tao, W., Yu, S., Yang, H.-W., Jiang, W., & Li, J. (2025) · International Conference on Software Engineering · doi:10.1109/icse55347.2025.00091 · Cited by 22
Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service
Ha, H., Lee, J., Han, W., & Chun, B.-G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.159 · Cited by 5
A novel prompt-tuning method: Incorporating scenario-specific concepts into a verbalizer
Ma, Y., Luo, S., Shang, Y.-M., Li, Z., & Liu, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123204 · Cited by 6
Confidence Estimation For LLM-Based Dialogue State Tracking
Sun, Y.-J., Dey, S., Hakkani-Tür, D. Z., & Tür, G. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832237 · Cited by 9
On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook
Fan, M., Wang, C., Chen, C., Liu, Y., & Huang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02375-w · Cited by 40
The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
Chochlakis, G., Potamianos, A., Lerman, K., & Narayanan, S. S. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00041 · Cited by 11
Data-Efficient Playlist Captioning With Musical and Linguistic Knowledge
Gabbolini, G., Hennequin, R., & Epure, E. V. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.784 · Cited by 6
Prompt Engineers or Librarians? An Exploration
Zhang, B. (2023) · Medical Reference Services Quarterly · doi:10.1080/02763869.2023.2250680 · Cited by 15
Domain Adaptation for Conversational Query Production with the RAG Model Feedback
Wang, A., Song, L., Ge, X., & Su, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.612 · Cited by 6
Weber, L., Bruni, E., & Hupkes, D. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.20 · Cited by 25
SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA
Tonglet, J., Reusens, M., Borchert, P., & Baesens, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.837 · Cited by 8
Can ChatGPT be Your Personal Medical Assistant?
Biswas, M. R., Islam, A., Shah, Z., Zaghouani, W., & Belhouari, S. B. (2023) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams60348.2023.10375477 · Cited by 5
Hint-Enhanced In-Context Learning Wakes Large Language Models Up For Knowledge-Intensive Tasks
Wang, Y., Guo, Q., Ni, X., Shi, C., Liu, L., Jiang, H., & Yang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447527 · Cited by 9
Who Knows the Answer? Finding the Best Model and Prompt for Each Query Using Confidence-Based Search
Gerych, W., Rizk, Y., Isahagian, V., Muthusamy, V., Duesterwald, E., & Venkateswaran, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29763 · Cited by 5
Leveraging Generative AI for Rapid Design and Verification of a Vector Processor SoC
Salcedo, W., Achour, S., & McBeth, C. (2024) · IEEE Design and Test · doi:10.1109/mdat.2024.3404117 · Cited by 5
Ilagan, J. R., Ilagan, J. B., Basallo, C. L., & Alabastro, Z. M. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.05.123 · Cited by 7
TI-Prompt: Towards a Prompt Tuning Method for Few-shot Threat Intelligence Twitter Classification
You, Y., Jiang, Z., Zhang, K., Jiang, J., Wang, X., Zhang, Z., Wang, S., & Feng, H. (2022) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac54236.2022.00046 · Cited by 6
Alfred: A System for Prompted Weak Supervision
Yu, P., & Bach, S. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.46 · Cited by 7
Are Experts Needed? On Human Evaluation of Counselling Reflection Generation
Wu, Z., Balloccu, S., Reiter, E., Helaoui, R., Recupero, D. R., & Riboni, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.382 · Cited by 6
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models
Kang, J., Xu, W. H., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.622 · Cited by 11
Robustness of Learning from Task Instructions
Gu, J., Zhao, H., Xu, H., Nie, L., Mei, H., & Yin, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.875 · Cited by 21
Wang, Z., Wang, T., Mekala, D., & Shang, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.244 · Cited by 7
“John is 50 years old, can his son be 65?” Evaluating NLP Models’ Understanding of Feasibility
Gupta, H., Varshney, N., Mishra, S. R., Pal, K. K., Sawant, S. A., Scaria, K. J., Goyal, S., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.30 · Cited by 13
ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories
Xia, H., Dong, Q., Li, L., Xu, J., Liu, T., Qin, Z., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.133 · Cited by 9
Exploring Large Language Models for Low-Resource IT Information Extraction
Bhavya, B., Isaza, P. T., Deng, Y., Nidd, M., Azad, A. P., Shwartz, L., & Zhai, C. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00157 · Cited by 7
Understanding the Dark Side of LLMs' Intrinsic Self-Correction
Zhang, Q., Wang, D., Qian, H., Li, Y., Zhang, T., Huang, M., Xu, K., Li, H., Yan, L., & Qiu, H. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1314 · Cited by 36
Thompson, K. N., Chandler, K. L., Morgan, C., Khashabi, D., Delinski, E. A., & Durme, B. V. (2025) · Journal of Advanced Academics · doi:10.1177/1932202x251356323 · Cited by 5
Helmy, H., Ibrahim, A., Arabi, M., Sattar, A. H. M. S., & Serag, A. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100182 · Cited by 4
Zhang, Z., Tu, Z., Chen, Y., Xiao, X., Feng, Y., & Zhang, W. (2025) · Journal of Research in Personality · doi:10.1016/j.jrp.2025.104680 · Cited by 4
Lee, S., Uhm, M., Jeong, H. D., & Lee, G. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.104199 · Cited by 6
Large Language Models for Sexual, Reproductive, and Maternal Health Rights*
Sewunetie, W. T., Beza, A., Abebe, H., Abuhay, T. M., Admass, W. S., Hassen, H., Haile, T. T., Hailemariam, H. A., Debebe, L., Moges, N. B., et al. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00091 · Cited by 4
Agentic Large Language Models for Generating Large-Scale Urban Daily Activity Patterns
Zhang, Y., Zhang, K., Pang, Y., & Sekimoto, Y. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825138 · Cited by 5
Evaluating Large Language Models: Frameworks and Methodologies for AI/ML System Testing
Pandhare, H. V. (2024) · International Journal of Scientific Research and Management (IJSRM) · doi:10.18535/ijsrm/v12i09.ec08 · Cited by 5
Commonsense Knowledge Salience Evaluation with a Benchmark Dataset in E-commerce
Qu, Y., Zhang, N., Chen, H., Dai, Z., Wang, C., Wang, X., Chen, Q., & Chen, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.2 · Cited by 5
Distinguishability Calibration to In-Context Learning
Li, H., Yan, H., Li, Y., Qian, L., He, Y., & Gui, L. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.102 · Cited by 8
Weissweiler, L., Hofmann, V., Köksal, A., & Schütze, H. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1225791 · Cited by 4
Metadata
- Authors
- Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S.
- Affiliations
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.09690
- OpenAlex
- W3132736064
- Cited by
- 513
- References
- 33
- Citations by year
- 2026: 5 · 2025: 69 · 2024: 114 · 2023: 206 · 2022: 86 · 2021: 33
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- few-shot learning, GPT-3, GPT-2, prediction variance, natural language prompts
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
