Learning To Retrieve Prompts for In-Context Learning
Open accessAbstract
In-context learning is a recent paradigm in natural language understanding, where a large pretrained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters.However, performance has been shown to strongly depend on the selected training examples (termed prompts).In this work, we propose an efficient method for retrieving prompts for in-context learning using annotated data and an LM.Given an inputoutput pair, we estimate the probability of the output given the input and a candidate training example as the prompt, and label training examples as positive or negative based on this probability.We then train an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time.We evaluate our approach on three sequence-tosequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes Efficient Prompt Retrieval (EPR), which uses a language model to label which training examples are good in-context prompts and trains a dense retriever from that signal. EPR substantially outperforms prior prompt-retrieval baselines on BREAK, MTOP, and SMCALFLOW.
Key findings
- With GPT-NEO as both scoring and inference LM, EPR improves over the best baseline from 26.0 to 31.9 on BREAK, from 57.0 to 64.2 on MTOP, and from 51.4 to 54.3 on SMCALFLOW.
- In the one-shot BREAK setting, EPR reaches 23.0 LF-EM, outperforming the best baseline by 8.5 points and BM25-ORACLE by 5 points.
- When GPT-NEO is used as a proxy scoring LM for larger inference LMs, EPR outperforms BM25 and CBR by 2–8 points across all datasets and pretrained models.
- Prompt-copying analysis shows high abstract copying in MTOP and SMCALFLOW (>80%); on MTOP, 84.5% of examples were abstractly copied, and EPR achieved 71.6% EM on that copied subset versus 64.2% overall.
Methods
- EPR generates training data by retrieving candidate prompts with an unsupervised retriever (BM25 or SBERT; BM25 chosen), scoring each candidate with a scoring LM as Prob(y | candidate, x), and labeling top-k as positive and bottom-k as negative (k=5, L=50).
- It trains a dense retriever with DPR-style contrastive learning using BERT-base input and prompt encoders, then uses FAISS maximum inner-product search at inference to retrieve prompts within the inference LM’s context limit.
- Evaluation covers three structured sequence-to-sequence tasks—BREAK, MTOP, and SMCALFLOW—using GPT-NEO as scoring LM and GPT-NEO, GPT-J, GPT-3, or CODEX as inference LMs, with LF-EM/NEM/EM metrics and a 2,048-token context size.
Limitations
- Paper-stated: the authors note that large language models exhibit bias, and since EPR is trained on signal from such LMs, it might also exhibit these biases.
- Paper-stated: the method scores each training example independently at training time, while at test time the LM observes a sequence of examples; modeling dependence between examples is left to future work.
- Evident from the text: evaluation is limited to three English structured tasks mapping utterances to meaning representations, and the approach relies on a scoring LM as a proxy for the inference LM.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 3 cited sources checked · 0 supported · 0 flagged
- Findingown data
EPR substantially improves performance compared to BM25 and CBR on the BREAK and MTOP test sets.
“Table 3 shows test results on BREAK and MTOP corroborating that EPR substantially improves performance compared to BM25 and CBR.”
Quote in the paperSubstantial claim - Findingnone given
BM25 outperforms SBERT for prompt retrieval.
“BM25 outperforms SBERT for prompt retrieval, and consequently we use BM25 in all of our supervised approaches to retrieve the set of candidates, Ē.”
Quote in the paperSubstantial claim - Findingown data
EPR substantially outperforms all other baselines in the LM-as-a-service setting.
“EPR substantially outperforms all other baselines.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The candidate set Ē includes good prompt candidates and hard negatives.
- Findingreasoning
CODEX outperforms GPT-3.
“CODEX outperforms GPT-3, which can be explained by the fact that it was trained on code, and our datasets involve mapping to programs or meaning representations.”
Quote in the paperNot a substantial claim - Findingown data
Accuracy on examples where copying occurred is much higher compared to accuracy where no copying happened.
“Moreover, accuracy on examples where copying occurred is much higher compared to accuracy where no copying happened.”
Quote in the paperNot a substantial claim
- Findingown data
In the LM-as-a-service setup using GPT-Neo, performance is generally robust with respect to the hyperparameters k (number of prompts labeled as positive or negative) and L (number of prompts retrieved by the unsupervised retriever).
“We test the effect of k, the number of prompts labeled as positive or negative, and L, the number of prompts retrieved by the unsupervised retriever.Table 8 shows that performance is is generally robust w.r.t these hyperparameters.”
- Also stated in: § Additional information · plain
“In the LM-as-a-service setup, using GPT-Neo, we search for other values for L and k, and note that the choice of our hyperparameters is robust.”
Quote in the paperNot a substantial claim - Also stated in: § Additional information · plain
- Findingown data
Random sampling of training examples leads to poor performance.
“For the unsupervised methods, the RANDOM baseline shows that random sampling of training examples leads to poor performance.”
Quote in the paperSubstantial claim - Findingown data
When the scoring LM and inference LM are identical using GPT-NEO, EPR improves performance compared to the best baseline from 57% to 64.2% on MTOP.
“When the scoring LM and inference LM are identical (using GPT-NEO (Black et al., 2021)), performance compared to the best baseline improves from 26% to 31.9% on BREAK, from 57% to 64.2% on MTOP, and from 51.4% to 54.3% on SMCALFLOW.”
- Also stated in: § Model · plain
“Specifically, when comparing to the best baseline, it improves performance from 26.0 to 31.9 on BREAK, from 57.0 to 64.2 on MTOP, and from 51.4 to 54.3 on SMCALFLOW.”
Quote in the paperSubstantial claim - Also stated in: § Model · plain
- Findingnone given
When using CODEX as the scoring LM and inference LM, performance remains roughly the same: 29.5 to 29.3.
“Analogously, when using CODEX as the scoring LM and inference LM performance remains roughly the same: 29.5→29.3.”
Quote in the paperNot a substantial claim - Findingown data
EPR substantially outperforms prior work and multiple baselines across the board.
“We evaluate our approach on three sequence-tosequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.”
- Also stated in: § Similar examples · plain
“We observe that EPR substantially improves performance compared to prior work on prompt retrieval.”
Quote in the paperSubstantial claim - Also stated in: § Similar examples · plain
- Findingown data
When the scoring LM and inference LM are identical using GPT-NEO, EPR improves performance compared to the best baseline from 26% to 31.9% on BREAK.
“When the scoring LM and inference LM are identical (using GPT-NEO (Black et al., 2021)), performance compared to the best baseline improves from 26% to 31.9% on BREAK, from 57% to 64.2% on MTOP, and from 51.4% to 54.3% on SMCALFLOW.”
- Also stated in: § Model · plain
“Specifically, when comparing to the best baseline, it improves performance from 26.0 to 31.9 on BREAK, from 57.0 to 64.2 on MTOP, and from 51.4 to 54.3 on SMCALFLOW.”
Quote in the paperSubstantial claim - Also stated in: § Model · plain
- Findingown data
When the scoring LM is GPT-NEO and the inference LM is a larger LM, EPR substantially outperforms prior baselines, including BM25 and CBR, by 2-8 points on all datasets and all pretrained models.
“First, the trends are similar to the LM-as-a-service setup, i.e., EPR substantially outperforms prior baselines, including our best unsupervised baseline, BM25, and the best supervised baseline, CBR, by 2-8 points on all datasets and all pretrained models.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The candidate set Ē includes good prompt candidates and hard negatives.
- Findingown data
Copying happens mostly from highly-ranked prompts.
“We observe that copying happens mostly from highly-ranked prompts.”
Quote in the paperNot a substantial claim - Findingown data
When the scoring LM and inference LM are identical using GPT-NEO, EPR improves performance compared to the best baseline from 51.4% to 54.3% on SMCALFLOW.
“When the scoring LM and inference LM are identical (using GPT-NEO (Black et al., 2021)), performance compared to the best baseline improves from 26% to 31.9% on BREAK, from 57% to 64.2% on MTOP, and from 51.4% to 54.3% on SMCALFLOW.”
- Also stated in: § Model · plain
“Specifically, when comparing to the best baseline, it improves performance from 26.0 to 31.9 on BREAK, from 57.0 to 64.2 on MTOP, and from 51.4 to 54.3 on SMCALFLOW.”
Quote in the paperSubstantial claim - Also stated in: § Model · plain
- Findingreasoning
BRUTEFORCE performs worse than BM25.
“Last, BRUTEFORCE performs worse than BM25.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The signal provided by the scoring LM for training goes beyond surface text similarity.
“This is surprising since BM25-ORACLE has access to the output sequence y test at test time, illustrating that the signal provided by the scoring LM for training goes beyond surface text similarity.”
Quote in the paperSubstantial claim - Interpretationown data
Using the LM itself to label examples is an effective approach for obtaining a strong prompt retriever.
“This shows that using the LM itself to label examples is an effective approach for obtaining a strong prompt retriever.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The candidate set Ē includes good prompt candidates and hard negatives.
- Interpretationreasoning
Some form of generalization to new structures is taking place.
“Nevertheless, even though accuracy is much lower in cases where no copying occurred, accuracy is not negligible, which shows that some form of generalization to new structures is taking place.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Using an LM for labeling examples is a better proxy for training a retriever compared to previously-proposed surface similarity heuristics.
“We argue that using an LM for labeling examples is a better proxy for training a retriever compared to previously-proposed surface similarity heuristics.”
Quote in the paperSubstantial claim - Interpretationown data
The high quality of candidates provided by BM25 applied on y is shown by reaching more than 50% LF-EM with just a single prompt.
“This shows the high quality of candidates provided by BM25 (applied on the y), as one can reach more than 50% LF-EM with just a single prompt.”
Quote in the paperSubstantial claim - Premisecitation
In-context learning performance strongly depends on the selected training examples (prompts).
“However, performance has been shown to strongly depend on the selected training examples (termed prompts).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Liu et al. (2021a)
Not checked in the free run.
- Interpretationnone given
The score s(ē_l) is a better proxy for the utility of a training example at inference time compared to prior approaches.
“We argue this score is a better proxy for the utility of a training example at inference time compared to prior approaches.”
Quote in the paperNot a substantial claim - Interpretationown data
Using a smaller LM (GPT-NEO) is an effective strategy for training a retriever that will be applied on other LMs.
“Thus, using a smaller LM (GPT-NEO) is an effective strategy for training a retriever that will be applied on other LMs.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The candidate set Ē includes good prompt candidates and hard negatives.
- Interpretationown data
GPT-NEO serves as a good proxy for choosing training examples.
“Thus, GPT-NEO serves as a good proxy for choosing training examples.”
Quote in the paperSubstantial claim
- May combine two claims. 10
- Classified as probably not a substantial claim. 5
- The quote only partly states this claim. 1
May combine two claims.
Findingown dataANYCORRECT-ORACLE reaches 53.6%, 20 points above LM-ORACLE.
“ANYCORRECT-ORACLE reaches 53.6%, 20 points above LM-ORACLE.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataUsing the target sequence y only for the unsupervised retriever leads to slightly higher performance than using the training pair (x, y).
“For both, we experimented with passing the retriever the training pair (x, y) or the target sequence y only, and found that using y leads to slightly higher performance.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingreasoningEPR outperforms BM25-ORACLE on MTOP and SMCALFLOW and is comparable on BREAK.
“Interestingly, EPR outperforms BM25-ORACLE on MTOP and SMCALFLOW and is comparable on BREAK.”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataIn the BREAK example where EPR is correct and CBR is incorrect, all training examples retrieved by EPR perform an argmax (argmin in the original utterance) and return in the final step a code, while the third example retrieved by CBR does not perform an argmax or argmin and does not involve a code.
“All training examples retrieved by EPR perform an argmax (argmin in the original utterance), and return in the final step”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingreasoningGPT-J performs slightly better than GPT-NEO across the board.
“Zooming in on different inference LMs, GPT-J performs slightly better than GPT-NEO across the board, since it was trained on the same data and using the same procedure as GPT-NEO.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the one-shot setup, EPR outperforms the best baseline by 8.5% and BM25-ORACLE by 5%.
“Indeed, EPR outperforms the best baseline by 8.5%, and BM25-ORACLE by 5%.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataOn MTOP, 84.5% of examples were abstractly copied, and on that subset EPR achieves 71.6% EM, compared to 64.2% on the entire validation set.
“For example, on MTOP, 84.5% of the examples were abstractly copied, and on that subset of examples, EPR achieves 71.6% EM, compared to 64.2% on the entire validation set.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-J outperforms CODEX (except on MTOP) and GPT-3 despite being 30x smaller.
“Surprisingly, GPT-J outperforms CODEX (except on MTOP) and GPT-3 despite being 30x smaller.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn examples from the development sets, EPR is correct while CBR is incorrect.
“Tables 9, 10, and 11 provide more examples for cases where EPR is correct while CBR is incorrect along with the top-3 prompts for each method.”
- Also stated in: § 6 Conclusions · plain
“An example from MTOP development set where EPR is correct and CBR is incorrect along with the top-3 training examples retrieved from each retriever.”
- Also stated in: § 6 Conclusions · plain
“An example from SMCALFLOW development set where EPR is correct and CBR is incorrect along with the top-3 training examples retrieved from each retriever.”
Quote in the paperNot a substantial claim- Also stated in: § 6 Conclusions · plain
May combine two claims.
Findingown dataThe rate of copying is much higher in MTOP and SM-CALFLOW compared to BREAK, where in MTOP and SMCALFLOW abstract copying reaches more than 80%.
“We observe that the rate of copying is much higher in MTOP and SM-CALFLOW compared to BREAK, where in MTOP and SMCALFLOW abstract copying reaches more than 80%.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationLarge language models have been shown to exhibit various kinds of bias.
“Large language models have been shown to exhibit various kinds of bias (Bender et al., 2021)”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Bender et al., 2021
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningEPR might also exhibit the biases that large language models exhibit, because EPR is trained on the signal obtained from such large LMs.
“since EPR is trained on the signal obtained from such large LMs, it might also exhibit these biases.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationLF-EM is a better metric compared to Normalized Exact Match (NEM) for evaluating BREAK because it measures whether two meaning representations are semantically equivalent.
“On BREAK, we evaluate performance on the development set with LF-EM (Hasson and Berant, 2021), which is a better metric compared to Normalized Exact Match (NEM), the official metric, as it measures whether two meaning representations are semantically equivalent.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Hasson and Berant, 2021
Not checked in the free run.
May combine two claims.
Interpretationown dataThe supervision provided by the scoring LM is strong, and training a better retriever from this signal can substantially enhance performance.
“The performance of LM-ORACLE is substantially higher than EPR, showing that the supervision provided by the scoring LM is strong, and training a better retriever from this signal can substantially enhance performance.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataEPR captures both lexical and structure similarity.
“shows that EPR captures both lexical and structure similarity.”
Quote in the paperSubstantial claim
The authors state this as an assumption.
The candidate set Ē includes good prompt candidates and hard negatives.
“This should lead to relevant positive examples, assuming that the set of candidates, Ē includes good prompt candidates and hard negatives, since all candidates have high similarity with (x, y) according to R u (y, D).”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Instance-level demonstrations generally perform better than task-level demonstrations.
“Although instance-level demonstrations generally perform better than task-level demonstrations, it requires a labeled training set in advance for retrieval.”
From What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs? (2023) · cites this paper as [32]
Section 4.2 compares demonstrations retrieved for each test utterance with randomly sampled training examples, and the Experimental Results report better performance for retrieval on the three tasks studied. The paper does not explicitly compare instance-level with task-level demonstrations as categories, so it does not establish the broader claim that the former generally perform better. The Background section also assumes a training set of input-output pairs for retrieval. (LLM, conf=0.79)
Experimental — accuracy not yet measuredRetrieval allows LLMs to access databases, mitigating hallucinations.
“Retrieval allows LLMs to access databases, mitigating hallucinations (Li et al. 2022;Wang, Yang, and Wei 2023;Rubin, Herzig, and Berant 2022;Liu et al. 2022).”
From Expel: LLM agents are experiential learners (2024) · cites this paper as Rubin, Herzig, and Berant 2022
The Background: Prompt Retrieval Problem setup section describes retrieving examples from a training set and supplying them to a language model as a prompt. This supports retrieval of stored examples, but the paper does not establish that retrieval gives LLMs access to databases or mitigates hallucinations. (LLM, conf=0.81)
Experimental — accuracy not yet measuredRetrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge.
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021;Poesia et al., 2021;Rubin et al., 2022;Madaan et al., 2022;Shrivastava et al., 2023).”
From Learning Performance-Improving Code Edits (2023) · cites this paper as Rubin et al., 2022
Earlier research has shown several positive pedagogical impacts of using NLP-based social conversational agents on children's perception of their own competencies and their attitudes towards learning.
“earlier research has showed several positive pedagogical impacts of using NLP-based social conversational agents on children's perception of their own competencies and their attitudes towards learning ([51]).”
From GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills (2024) · cites this paper as [51]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
Ankerst, M. (1999) · doi:10.1145/304181.304187 · Cited by 3,982 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
The probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
(2002) · Cited by 1,635 · see citing works
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsPreprint
Du, N., Huang, Y.-P., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y.-Z., Krikun, M., Zhou, Y.-Q., Yu, A. W., Firat, O., et al. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2112.06905 · Cited by 1,379 · see citing works
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305 · see citing works
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279 · see citing works
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 1× · see citing works
“Recently, Min et al. (2022) showed that the model does not rely on the ground truth input-label mapping provided in the demonstrations as much as previously thought.”
BackgroundRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, W. X., Dong, D., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.466 · Cited by 479 · see citing works
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382 · see citing works
(2020) · International Conference on Machine Learning · Cited by 358 · see citing works
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
Efficient Natural Language Response Suggestion for Smart ReplyPreprint
Henderson, M., Al‐Rfou, R., Strope, B. P., Sung, Y.-H., Lukacs, L., Guo, R., Kumar, S., Miklos, B., & Kurzweil, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.00652 · Cited by 315 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274 · see citing works
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTPreprint
Khattab, O., & Zaharia, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12832 · Cited by 270 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Nearest Neighbor Machine Translation
Khandelwal, U., Fan, A., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2020) · arXiv · doi:10.48550/arxiv.2010.00710 · Cited by 165 · see citing works
What Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148 · see citing works
Noisy Channel Language Model Prompting for Few-Shot Text Classification
Min, S., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.365 · Cited by 142 · see citing works
Constrained Language Models Yield Few-Shot Semantic Parsers
Shin, R., Lin, C., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., & Van Durme, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.608 · Cited by 139 · see citing works
An Explanation of In-context Learning as Implicit Bayesian Inference
Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2111.02080 · Cited by 138 · see citing works
B reak It Down: A Question Understanding Benchmark
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00309 · Cited by 136 · see citing works
Self-citationMTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
Li, H., Arora, A., Chen, S., Gupta, A., Gupta, S., & Mehdad, Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.257 · Cited by 133 · see citing works
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
Case-based Reasoning for Natural Language Queries over Knowledge Bases
Das, R., Zaheer, M., Thai, D., Godbole, A., Perez, E., Lee, J. Y., Tan, L., Polymenakos, L., & McCallum, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.755 · Cited by 121 · see citing works
Task-Oriented Dialogue as Dataflow Synthesis
Andreas, J., Bufe, J., Burkett, D., Chen, C., Clausman, J., Crawford, J., Crim, K., DeLoach, J., Dorner, L., Eisner, J., et al. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00333 · Cited by 98 · see citing works
Wang, S., Xu, Y., Fang, Y., Liu, Y., Sun, S., Xu, R. P., Zhu, C., & Zeng, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.226 · Cited by 92 · see citing works
BERTese: Learning to Speak to BERT
Haviv, A., Berant, J., & Globerson, A. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.316 · Cited by 84 · see citing works
Self-citationHuman Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Yi‐chong, X., Zhu, C., Wang, S., Sun, S., Cheng, H., Liu, X., Gao, J., He, P., Zeng, M., & Huang, X. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/383 · Cited by 43 · see citing works
Improving the Cluster Structure Extracted from OPTICS Plots.
Schubert, E., & Gertz, M. (2018) · LWDA · Cited by 33 · see citing works
Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval
Alon, U., Xu, F. F., He, J., Sengupta, S., Roth, D., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2201.12431 · Cited by 32 · see citing works
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Saunshi, N., Malladi, S., & Arora, S. (2020) · arXiv · doi:10.48550/arxiv.2010.03648 · Cited by 32 · see citing works
Controllable Semantic Parsing via Retrieval Augmentation
Pasupat, P., Zhang, Y., & Guu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.607 · Cited by 30 · see citing works
Improving Evidence Retrieval for Automated Explainable Fact-Checking
Samarinas, C., Hsu, W., & Lee, M. L. (2021) · doi:10.18653/v1/2021.naacl-demos.10 · Cited by 26 · see citing works
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example DesignPreprint
Levine, Y., Wies, N., Jannai, D., Navon, D., Hoshen, Y., & Shashua, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04541 · Cited by 9 · see citing works
Question Decomposition with Dependency GraphsPreprint
Hasson, M., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08647 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Geoffrey E. Hinton and Sam T. Roweis. 2002. Stochas- tic neighbor embedding. In NIPS.
- Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest Neigh- bor Machine Translation. In Proceedings of ICLR.
Cited by
Sorted by citations
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · see citing works
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671470 · Cited by 827 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
“…2021; Holtzman et al., 2021; Min et al., 2021a), better ways of choosing labeled examples for the demonstrations (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), meta-training with an explicit in-context learning objective (Chen et al., 2021; Min et al., 2021b), and learning to follow…”
BackgroundA Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649 · see citing works
Recommender Systems in the Era of Large Language Models (LLMs)
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., et al. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3392335 · Cited by 481 · see citing works
Automatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · see citing works
What does a platypus look like? Generating customized prompts for zero-shot image classification
Pratt, S., Covert, I., Liu, R., & Farhadi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01438 · Cited by 240 · see citing works
Text Classification via Large Language Models
Sun, X., Li, X., Li, J., Wu, F. Q., Guo, S., Zhang, T., & Wang, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.603 · Cited by 273 · see citing works
Lu, G., Ju, X., Chen, X., Pei, W., & Cai, Z. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112031 · Cited by 197 · see citing works
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., & Chau, D. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.51 · Cited by 244 · see citing works
Self-Planning Code Generation with Large Language Models
Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., & Jiao, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672456 · Cited by 158 · see citing works
Expel: LLM agents are experiential learners
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29936 · Cited by 153 · see citing works
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Mari Ostendorf, E., Yih, W.-T., Smith, N. A., Zettlemoyer, L., & Yu, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.71 · Cited by 136 · see citing works
Ma, Y., Cao, Y., Hong, Y. S., & Sun, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.710 · Cited by 154 · see citing works
GPT-3-Driven Pedagogical Agents to Train Children’s Curious Question-Asking Skills
Abdelghani, R., Wang, Y.-H., Yuan, X., Wang, T., Lucas, P., Sauzéon, H., & Oudeyer, P.-Y. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00340-7 · Cited by 129 · see citing works
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
Baek, J., Aji, A. F., & Saffari, A. (2023) · NLRSE · doi:10.18653/v1/2023.nlrse-1.7 · Cited by 122 · see citing works
Language Models of Code are Few-Shot Commonsense Learners
Madaan, A., Zhou, S., Alon, U., Yang, Y., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.90 · Cited by 131 · see citing works
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623343 · Cited by 120 · see citing works
LLM for SoC Security: A Paradigm Shift
Saha, D., Tarek, S., Yahyaei, K., Saha, S. K., Zhou, J., Tehranipoor, M. M., & Farahmandi, F. (2024) · IEEE Access · doi:10.1109/access.2024.3427369 · Cited by 113 · see citing works
Active Example Selection for In-Context Learning
Zhang, Y., Shi, F., & Tan, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.622 · Cited by 117 · see citing works
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106 · see citing works
Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations
Huang, X., Lian, J., Lei, Y., Yao, J., Lian, D., & Xie, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3731446 · Cited by 89 · see citing works
Li, J., Liu, Y., Fan, W., Wei, X.-Y., Liu, H., Tang, J., & Li, Q. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3393356 · Cited by 85 · see citing works
DB-GPT: Large Language Model Meets Database
Zhou, X., Sun, Z., & Li, G. (2024) · Data Science and Engineering · doi:10.1007/s41019-023-00235-6 · Cited by 83 · see citing works
ChatGPT as Your Vehicle Co-Pilot: An Initial Attempt
Wang, S., Zhu, Y., Li, Z., Wang, Y., Li, L., & He, Z. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3325300 · Cited by 82 · see citing works
In-context Examples Selection for Machine Translation
Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., & Ghazvininejad, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.564 · Cited by 106 · see citing works
Wu, Z., Wang, Y., Ye, J., & Kong, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.79 · Cited by 99 · see citing works
PromptCap: Prompt-Guided Image Captioning for VQA with GPT-3
Hu, Y., Hua, H., Yang, Z., Shi, W., Smith, N. A., & Luo, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00277 · Cited by 136 · see citing works
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · see citing works
A Practical Survey on Zero-shot Prompt Design for In-context Learning
Columbia University, N. Y. C., & Li, Y. (2023) · Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings · doi:10.26615/978-954-452-092-2_069 · Cited by 68 · see citing works
Open-Vocabulary SAM: Segment and Recognize Twenty-Thousand Classes Interactively
Yuan, H., Li, X., Zhou, C., Li, Y., Chen, K., & Loy, C. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72775-7_24 · Cited by 70 · see citing works
Han, B., Susnjak, T., & Mathrani, A. (2024) · Applied Sciences · doi:10.3390/app14199103 · Cited by 68 · see citing works
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“This can consist of training input-output pairs (Rubin et al., 2021), history event flows (Weston et al.”
BackgroundA Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Honovich, O., Shaham, U., Bowman, S. R., & Levy, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.108 · Cited by 66 · see citing works
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 88 · see citing works
Diverse Demonstrations Improve In-context Compositional Generalization
Levy, I., Bogin, B., & Berant, J. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.78 · Cited by 66 · see citing works
Self-citationKnowledge extraction for additive manufacturing process via named entity recognition with LLMs
Liu, X. E., Erkoyuncu, J. A., Fuh, J. Y. H., Lu, W. F., & Li, B. (2024) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2024.102900 · Cited by 58 · see citing works
Dolphins: Multimodal Language Model for Driving
Ma, Y., Cao, Y., Sun, J., Pavone, M., & Xiao, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72995-9_23 · Cited by 55 · see citing works
Can We Edit Factual Knowledge by In-Context Learning?
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., & Chang, B. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.296 · Cited by 54 · see citing works
Prompt-Based Learning for Unpaired Image Captioning
Zhu, P., Wang, X., Zhu, L., Sun, Z., Zheng, W., Wang, Y., & Chen, C. W. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3265842 · Cited by 52 · see citing works
Li, J., Li, H., Pan, Z., Sun, D., Wang, J., Zhang, W., & Pan, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.184 · Cited by 51 · see citing works
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
Baek, J., Aji, A., & Saffari, A. (2023) · Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023) · doi:10.18653/v1/2023.matching-1.7 · Cited by 51 · see citing works
“To further improve their performances, some work (Rubin et al., 2022; Liu et al., 2022a) proposes retrieving relevant samples to the input question from the training dataset and prepending them in the prompt under few-show learning.”
BackgroundUnified Demonstration Retriever for In-Context Learning
Li, X., Lv, K., Yan, H., Lin, T., Zhu, W., Ni, Y., Xie, G., Wang, X., & Qiu, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.256 · Cited by 49 · see citing works
WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences
Liu, X., Lai, H., Yu, H., Xu, Y., Zeng, A., Du, Z., Zhang, P., Dong, Y., & Tang, J. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599931 · Cited by 64 · see citing works
Complementary Explanations for Effective In-Context Learning
Ye, X., Iyer, S. V., Çelikyılmaz, A., Stoyanov, V., Durrett, G., & Pasunuru, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.273 · Cited by 46 · see citing works
Successive Prompting for Decomposing Complex Questions
Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.81 · Cited by 46 · see citing works
Batch Prompting: Efficient Inference with Large Language Model APIs
Cheng, Z., Kasai, J., & Yu, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.74 · Cited by 54 · see citing works
Yang, L., Wang, Z., Li, Z., Na, J., & Yu, J. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103724 · Cited by 45 · see citing works
Cost-Effective In-Context Learning for Entity Resolution: A Design Space Exploration
Fan, M., Han, X., Fan, J., Chai, C., Tang, N., Li, G., & Du, X. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00284 · Cited by 44 · see citing works
Learning In-context Learning for Named Entity Recognition
Chen, J., Lu, Y., Lin, H., Lou, J., Jia, W., Dai, D., Wu, H. R., Cao, B., Han, X., & Sun, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.764 · Cited by 44 · see citing works
Ma, J., Zhao, J., & Hou, Y. (2024) · Sensors · doi:10.3390/s24175502 · Cited by 45 · see citing works
Wu, J., Wu, X., Qiu, Z., Li, M., Lin, S., Zhang, Y., Zheng, Y., Yuan, C., & Yang, J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae079 · Cited by 41 · see citing works
Chan, C.-F., Wong, P. K.-Y., Guo, X., Cheng, J. C. P., Chan, J. C. F., Leung, P.-H., & Tao, X. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106305 · Cited by 41 · see citing works
A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction
Shui, R., Cao, Y., Wang, X., & Chua, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.490 · Cited by 43 · see citing works
Explainable Knowledge reasoning via thought chains for knowledge-based visual question answering
Qiu, C., Xie, Z., Liu, M., & Hu, H. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103726 · Cited by 33 · see citing works
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32 · see citing works
Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study
Li, Y., Huo, Y., Jiang, Z., Zhong, R., He, P., Su, Y., Briand, L., & Lyu, M. R.-T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3475375 · Cited by 32 · see citing works
Active Learning Principles for In-Context Learning with Large Language Models
Margatina, K., Schick, T., Αλέτρας, Ν., & Dwivedi-Yu, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.334 · Cited by 42 · see citing works
Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning
Tang, R., Kong, D., Huang, L., & Xue, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.284 · Cited by 31 · see citing works
Large Language Models as Evaluators for Recommendation Explanations
Zhang, X., Li, Y., Wang, J., Sun, B., Ma, W., Sun, P., & Zhang, M. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688075 · Cited by 36 · see citing works
Liang, Y., Wang, J., Zhu, H., Wang, L., Qian, W., & Lan, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.263 · Cited by 37 · see citing works
FLTRNN: Faithful Long-Horizon Task Planning for Robotics with Large Language Models
Zhang, J., Tang, L., Song, Y., Meng, Q., Qian, H., Shao, J., Song, W., Zhu, S., & Gu, J. J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611663 · Cited by 30 · see citing works
Finding Support Examples for In-Context Learning
Li, X., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.411 · Cited by 48 · see citing works
Zhao, Z., Zhou, G., Xie, Z., Wu, L., & Huang, J. X. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3658673 · Cited by 29 · see citing works
Patchview: LLM-powered Worldbuilding with Generative Dust and Magnet Visualization
Chung, J. J. Y., & Kreminski, M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676352 · Cited by 30 · see citing works
Shi, P., Zhang, R., Bai, H., & Lin, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.384 · Cited by 35 · see citing works
Fagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 43 · see citing works
ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback
Ye, J., Gao, J., Wu, Z., Feng, J., Yu, C., & Kong, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.269 · Cited by 37 · see citing works
Better Zero-Shot Reasoning with Self-Adaptive Prompting
Wan, X., Sun, R., Dai, H., Arık, S. Ö., & Pfister, T. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.216 · Cited by 35 · see citing works
A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Qiu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2025) · ACM Computing Surveys · doi:10.1145/3729218 · Cited by 34 · see citing works
Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge
Chen, J., Shi, W., Fu, Z., Cheng, S., Li, L., & Xiao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.550 · Cited by 30 · see citing works
GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models
Rajpoot, P. K., & Parikh, A. P. (2023) · Workshop on Financial Technology and Natural Language Processing (FinNLP) · doi:10.18653/v1/2023.finnlp-2.5 · Cited by 31 · see citing works
What Makes Multimodal In-Context Learning Work?
Baldassini, F. B., Shukor, M., Cord, M., Soulier, L., & Piwowarski, B. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00161 · Cited by 25 · see citing works
Exploring Effective Factors for Improving Visual In-Context Learning
Sun, Y., Chen, Q., Wang, J., Wang, J., & Li, Z. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3554410 · Cited by 32 · see citing works
In-Context Learning for Text Classification with Many Labels
Milios, A., Reddy, S., & Bahdanau, D. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.14 · Cited by 29 · see citing works
Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
Ye, S., Hwang, H., Yang, S., Yun, H., Kim, Y., & Seo, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29909 · Cited by 37 · see citing works
ScatterShot: Interactive In-context Example Curation for Text Transformation
Wu, T., Shen, H., Weld, D. S., Heer, J., & Ribeiro, M. T. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584059 · Cited by 26 · see citing works
Mishra, B. D., Tafjord, O., & Clark, P. E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.644 · Cited by 24 · see citing works
MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning
Zhang, C., Lin, K., Yang, Z., Wang, J., Li, L., Lin, C.-C., Liu, Z., & Wang, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01295 · Cited by 24 · see citing works
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 23 · see citing works
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
BackgroundHow Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?
Lo, L. Y., & Qu, H. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456333 · Cited by 23 · see citing works
How Effective are Large Language Models in Generating Software Specifications?
Xie, D., Yoo, B., Jiang, N., Kim, M., Tan, L., Zhang, X., & Lee, J. S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00014 · Cited by 47 · see citing works
Shan, T., Zhang, F., Chan, A. P. C., Zhu, S., & Li, K. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103655 · Cited by 23 · see citing works
MoT: Memory-of-Thought Enables ChatGPT to Self-Improve
Li, X., & Qiu, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.392 · Cited by 36 · see citing works
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
Qiu, L., Shaw, P. M., Pasupat, P., Shi, T., Herzig, J., Pitler, E., Sha, F., & Toutanova, K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.624 · Cited by 27 · see citing works
Self-citationPre-Training to Learn in Context
Gu, Y., Dong, L., Wei, F., & Huang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.267 · Cited by 33 · see citing works
ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval
Yu, Y., Zhuang, Y., Zhang, R., Yu, M., Shen, J., & Zhang, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.748 · Cited by 26 · see citing works
Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation
Yuan, X., Wang, T., Wang, Y.-H., Fine, E., Abdelghani, R., Sauzéon, H., & Oudeyer, P.-Y. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.820 · Cited by 21 · see citing works
Boosting Your Context by Dual Similarity Checkup for In-Context Learning Medical Image Segmentation
Gao, J. Y., Lao, Q., Kang, Q., Liu, P., Du, C., Li, K., & Zhang, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3440311 · Cited by 21 · see citing works
Retrieval-style In-context Learning for Few-shot Hierarchical Text Classification
Chen, H., Zhao, Y. H., Chen, Z., Wang, M., Li, L., Zhang, M., & Zhang, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00697 · Cited by 23 · see citing works
Multi-stage guided code generation for Large Language Models
Han, Y., & Lyu, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109491 · Cited by 25 · see citing works
In-context Learning for Few-shot Multimodal Named Entity Recognition
Cai, C., Wang, Q., Liang, B., Qin, B., Yang, M., Wong, K., & Xu, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.196 · Cited by 21 · see citing works
Dr.ICL: Demonstration-Retrieved In-context Learning
Luo, M., Xu, X., Dai, Z., Pasupat, P., Kazemi, M., Baral, C. R., Imbrasaitė, V., & Zhao, V. Y. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0012 · Cited by 51 · see citing works
LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery
Chen, K., Du, Y., You, T., Islam, M., Guo, Z., Jin, Y., Chen, G., & Heng, P. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610603 · Cited by 21 · see citing works
Managing Linux servers with LLM-based AI agents: An empirical evaluation with GPT4
Cao, Q., Wang, F., Lindley, L. C., & Wang, Z. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100570 · Cited by 22 · see citing works
Enhancing named entity recognition with external knowledge from large language model
Li, Q., Xie, T., Zhang, J., Ma, K., Su, J., Yang, K., & Wang, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113471 · Cited by 22 · see citing works
How Do In-Context Examples Affect Compositional Generalization?
An, S., Lin, Z., Fu, Q., Chen, B., Zheng, N., Lou, J., & Zhang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.618 · Cited by 19 · see citing works
- Authors
- Rubin, O., Herzig, J., & Berant, J.
- Affiliations
- Tel Aviv University (IL)
- Venue
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2022
- Pages
- 2655-2671
- OpenAlex
- W4287891464
- Semantic Scholar
- f9838a3be5c94bb2674a0e224de349b50e18f3c4
- Cited by
- 374
- References
- 54
- Citations by year
- 2026: 57 · 2025: 97 · 2024: 91 · 2023: 119 · 2022: 10
- FWCI
- 43.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- European Commission
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- in-context learning, dense retrievers, training data selection, pre-trained language models, annotated dataset, meaning representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
