OpenPrompt: An Open-source Framework for Prompt-learning
Open accessAbstract
Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to $cloze$-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks. However, no standard implementation framework of prompt-learning is proposed yet, and most existing prompt-learning codebases, often unregulated, only provide limited implementations for specific scenarios. Since there are many details such as templating strategy, initializing strategy, and verbalizing strategy, etc. need to be considered in prompt-learning, practitioners face impediments to quickly adapting the desired prompt learning methods to their applications. In this paper, we present {OpenPrompt}, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility, and its combinability allows the freedom to combine different PLMs, task formats, and prompting modules in a unified paradigm. Users could expediently deploy prompt-learning frameworks and evaluate the generalization of them on different NLP tasks without constraints. OpenPrompt is publicly released at {\url{ https://github.com/thunlp/OpenPrompt}}.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
People without programming skills or NLP knowledge can make NLP models generate desired interactions by crafting effective prompt strategies.
“In this paradigm, people without programming skills or NLP knowledge can nonetheless make NLP models generate desired interactions by crafting efective prompt strategies (e.g., natural language instructions, examples, and templates) [3,15].”
From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [15]
The Introduction (p. 1) says textual prompts or demonstrations can effectively stimulate pretrained language models, supporting the role of prompt strategies. But the Background (p. 3) describes implementing prompt-learning as complex and does not establish that people without programming skills or NLP knowledge can obtain desired interactions this way. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 2× · see citing works
“Once the PLM is determined, researchers could deploy a known valid prompt-learning pipeline (e.g., RoBERTa for few-shot sentiment classification) or explore other uses of PLM that could exploit its potential.”
BackgroundPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1× · see citing works
“As illustrated in Figure 1, OpenPrompt provides the full life-cycle of prompt-learning based on PyTorch (Paszke et al., 2019).”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 4× · see citing works
“GPT-3 (Brown et al., 2020) is one of the representative works adopting this objec-tive.”
BackgroundInfluentialBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1× · see citing works
“Different evaluation metrics are then used for different types of task, For example, exact match for classification tasks and BLEU score (Papineni et al., 2002) for generation tasks.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 2× · see citing works
“Once the PLM is determined, researchers could deploy a known valid prompt-learning pipeline (e.g., RoBERTa for few-shot sentiment classification) or explore other uses of PLM that could exploit its potential.”
BackgroundExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1× · see citing works
“The PromptTrainer is a controller that controls the data flow and the training process with some unique attributes, users can also implement the training process in a conventional fashion. include T5 (Raffel et al., 2020), MASS (Song et al., 2019) and BART (Lewis et al., 2020), etc.”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1× · see citing works
“The PromptTrainer is a controller that controls the data flow and the training process with some unique attributes, users can also implement the training process in a conventional fashion. include T5 (Raffel et al., 2020), MASS (Song et al., 2019) and BART (Lewis et al., 2020), etc.”
BackgroundCharacter-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · cited 3× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1× · see citing works
“A series of studies of prompt-learning (Liu et al., 2021a) have been proposed to investigate the strategies of constructing templates (Schick and Sch¨utze, 2021; Gao et al., 2021; Liu et al., 2021b), verbalizers (Hu et al., 2021), optimization (Lester et al., 2021), and application (Li and Liang,…”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 2× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
MethodDBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., et al. (2015) · Semantic Web · doi:10.3233/sw-140134 · Cited by 3,737 · cited 2× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
BackgroundA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 2× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
MethodThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 6× · see citing works
“Previous works design a wide variety of templates, including manually written template (Schick and Schütze, 2021) and pure soft template (Lester et al., 2021).”
BackgroundMethodInfluentialLoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 1× · see citing works
“In addition to prompt-based methods, there are also other techniques exploring the parameter-efficient stimulation of large-scale PLMs (Houlsby et al., 2019; Hu et al., 2022; He et al., 2022; Ding et al., 2022).”
MethodPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 3× · see citing works
“, 2021b) Prefix-tuning LM, Seq2Seq Soft tokens Text Generation (Li and Liang, 2021) LM-BFF MLM A.”
BackgroundMethodParameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 1× · see citing works
“In addition to prompt-based methods, there are also other techniques exploring the parameter-efficient stimulation of large-scale PLMs (Houlsby et al., 2019; Hu et al., 2022; He et al., 2022; Ding et al., 2022).”
MethodExploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2001.07676 · Cited by 1,932 · cited 2×
“Previous works design a wide variety of templates, including manually written template (Schick and Schütze, 2021) and pure soft template (Lester et al.”
BackgroundLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 2× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
BackgroundPre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · cited 1× · see citing works
“Pre-trained language models (PLMs) (Han et al., 2021a; Qiu et al., 2020) have been widely proven to be effective in natural language understanding and generation, ushering in a new era of modern natural language processing (NLP).”
BackgroundMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 4× · see citing works
“…of prompt-learning (Liu et al., 2021a) have been proposed to investigate the strategies of constructing templates (Schick and Sch¨utze, 2021; Gao et al., 2021; Liu et al., 2021b), verbalizers (Hu et al., 2021), optimization (Lester et al., 2021), and application (Li and Liang, 2021; Han et…”
BackgroundMethodInfluentialPre-Trained Models: Past, Present and FuturePreprint
Han, X., Zhang, Z.-Y., Ding, N., Gu, Y.-X., Liu, X., Huo, Y., Qiu, J.-Z., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.48550/arxiv.2106.07139 · Cited by 1,145 · see citing works
Self-citationSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 3× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
BackgroundMethodLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908 · cited 3×
“A series of studies of prompt-learning (Liu et al., 2021a) have been proposed to investigate the strategies of constructing templates (Schick and Sch¨utze, 2021; Gao et al., 2021; Liu et al., 2021b), verbalizers (Hu et al., 2021), optimization (Lester et al., 2021), and application (Li and Liang,…”
BackgroundMASS: Masked Sequence to Sequence Pre-training for Language GenerationPreprint
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02450 · Cited by 834 · cited 1× · see citing works
“The PromptTrainer is a controller that controls the data flow and the training process with some unique attributes, users can also implement the training process in a conventional fashion. include T5 (Raffel et al., 2020), MASS (Song et al., 2019) and BART (Lewis et al., 2020), etc.”
BackgroundDunn, I. (2009) · doi:10.1002/9780470551455.ch10 · Cited by 571 · cited 2×
“Originated in T5 (Raffel et al., 2019) and GPT-3 (Brown et al., 2020), researchers find that PLMs can be effectively stimulated by textual prompts or demonstrations, especially in low-data scenarios.”
BackgroundTowards a Unified View of Parameter-Efficient Transfer LearningPreprint
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04366 · Cited by 514 · see citing works
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 2× · see citing works
“Moreover, the performance of a prompt-learning pipeline varies greatly with the choice of templates and verbalizers (Zhao et al., 2021), creating more barriers for implementations.”
BackgroundPPT: Pre-trained Prompt Tuning for Few-shot LearningPreprint
Gu, Y.-X., Han, X., Liu, Z.-Y., & Huang, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2109.04332 · Cited by 507 · cited 1× · see citing works
“Gu et al. (2021) report a mix of manual template tokens and soft (trainable) to-kens sometimes yields better results than separate manual template and soft template.”
ResultSelf-citationPTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11259 · Cited by 478 · cited 5× · see citing works
“Pre-trained language models (PLMs) (Han et al., 2021a; Qiu et al., 2020) have been widely proven to be effective in natural language understanding and generation, ushering in a new era of modern natural language processing (NLP).”
BackgroundMethodInfluentialSelf-citationThe WebNLG Challenge: Generating Text from RDF Data
Gardent, C., Shimorina, A., Narayan, S., & Perez-Beltrachini, L. (2017) · International Conference on Natural Language Generation · doi:10.18653/v1/w17-3518 · Cited by 470 · cited 2× · see citing works
“And the evaluation tasks include WebNLG (Gardent et al., 2017) for conditional generation, GLUE (Wang et al., 2018) and Super-GLUE (Wang et al., 2019) for natural language understanding; SemEval (Hendrickx et al., 2010) for relation extraction; Few-NERD (Ding et al., 2021b) for fine-grained entity typing; MNLI (William…”
MethodKnowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationPreprint
Hu, S.-D., Ding, N., Wang, H., Liu, Z.-Y., Wang, J.-G., Li, J.-Z., Wu, W., & Sun, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2108.02035 · Cited by 457 · cited 5× · see citing works
“Additional to manually-defined verbalizers, we implement automatic verbalizers like AutomaticVerbalizer and KnowledgeableVerbalizer (Hu et al., 2021).”
BackgroundMethodInfluentialSelf-citationSemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of NominalsPreprint
Hendrickx, I., Kim, S., Kozareva, Z., Nakov, P., Séaghdha, D. Ó., Padó, S., Pennacchiotti, M., Romano, L., & Szpakowicz, S. (2010) · International Workshop on Semantic Evaluation · doi:10.48550/arxiv.1911.10422 · Cited by 367 · see citing works
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Vu, T., Lester, B., Constant, N., Al-Rfou, R., & Cer, D. M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2110.07904 · Cited by 357 · cited 1×
“These practical studies imply that we may use prompts to more effectively and efficiently dig the knowledge kept in PLMs, leading to a deeper understanding of the underlying principles of their mechanisms (Wei et al., 2021; Qin et al., 2021; Vu et al., 2021).”
BackgroundHow many data points is a prompt worth?Preprint
Scao, T. L., & Rush, A. M. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.08493 · Cited by 329 · cited 1× · see citing works
“Practically, this paradigm is surprisingly effective in low-data regime (Le Scao and Rush, 2021; Gao et al., 2021).”
BackgroundCutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsPreprint
Logan, R. L., IV, Balavzevi'c, I., Wallace, E., Petroni, F., Singh, S., & Riedel, S. (2021) · Findings · doi:10.48550/arxiv.2106.13353 · Cited by 253 · cited 1× · see citing works
“Logan IV et al. (2021) design null template with simple concatenation of the inputs and an appended <mask> token.”
MethodFew-NERD: A Few-shot Named Entity Recognition Dataset
Ding, N., Xu, G., Chen, Y., Wang, X., Han, X., Xie, P., Zheng, H.-T., & Liu, Z. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.248 · Cited by 245 · cited 5× · see citing works
“, 2010), Few-NERD (Ding et al., 2021b) for information extraction; MNLI (Williams et al.”
BackgroundInfluentialSelf-citationPrompt-Learning for Fine-Grained Entity TypingPreprint
Ding, N., Chen, Y., Han, X., Xu, G., Wang, X.-B., Xie, P.-J., Zheng, H.-T., Liu, Z.-Y., Li, J.-Z., & Kim, H.-G. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2108.10604 · Cited by 179 · cited 6× · see citing works
“For example, with appropriate template, zero-shot prompt-learning could even outperform 32-shot fine-tuning (Ding et al., 2021a).”
BackgroundInfluentialSelf-citationCrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 120 · cited 1× · see citing works
“Then the final prediction will be based on a mapping from the target texts to the labels (Ye et al., 2021; Du et al., 2021).”
BackgroundAll NLP Tasks Are Generation Tasks: A General Pretraining FrameworkPreprint
Du, Z.-X., Qian, Y., Liu, X., Ding, M., Qiu, J.-Z., Yang, Z.-L., & Tang, J. (2021) · arXiv.org · doi:10.48550/arxiv.2103.10360 · Cited by 69 · cited 1× · see citing works
“Then the final prediction will be based on a mapping from the target texts to the labels (Ye et al., 2021; Du et al., 2021).”
BackgroundExploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning
Qin, Y., Wang, X., Su, Y., Lin, Y.-K., Ding, N., Liu, Z.-Y., Li, J.-Z., Hou, L., Li, P., Sun, M., et al. (2021) · arXiv.org · doi:10.48550/arxiv.2110.07867 · Cited by 40 · cited 2×
“These practical studies imply that we may use prompts to more effectively and efficiently dig the knowledge kept in PLMs, leading to a deeper understanding of the underlying principles of their mechanisms (Wei et al., 2021; Qin et al., 2021; Vu et al., 2021).”
BackgroundSelf-citationWhy Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Wei, C., Xie, S. M., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2106.09226 · Cited by 32 · cited 2× · see citing works
“These practical studies imply that we may use prompts to more effectively and efficiently dig the knowledge kept in PLMs, leading to a deeper understanding of the underlying principles of their mechanisms (Wei et al., 2021; Qin et al., 2021; Vu et al., 2021).”
BackgroundDelta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.06904 · Cited by 20 · cited 1× · see citing works
“In addition to prompt-based methods, there are also other techniques exploring the parameter-efficient stimulation of large-scale PLMs (Houlsby et al., 2019; Hu et al., 2022; He et al., 2022; Ding et al., 2022).”
MethodSelf-citation
Flagged first, then by citations
3 of the 45 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL .
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of EMNLP , pages 38-45, Online.
Cited by
Sorted by citations
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 893 · see citing works
“In this paradigm, people without programming skills or NLP knowledge can nonetheless make NLP models generate desired interactions by crafting efective prompt strategies (e.g., natural language instructions, examples, and templates) [3, 15].”
BackgroundKnowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., Wu, W., & Sun, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.158 · Cited by 298 · see citing works
Self-citationNo more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y. F., Zhang, H., & Lyu, M. R.-T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549113 · Cited by 153 · see citing works
“Our implementation of prompt is based on OpenPrompt [7].”
MethodIs GPT-3 a Good Data Annotator?
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., & Bing, L. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.626 · Cited by 149 · see citing works
Prompt Engineering Paradigms for Medical Applications: Scoping Review
Zaghir, J., Naguib, M., Bjelogrlic, M., Névéol, A., Tannier, X., & Lovis, C. (2024) · Journal of Medical Internet Research · doi:10.2196/60501 · Cited by 143 · see citing works
“Figure 4 [16,20,26,41,47, 88-123 88-123 ] illustrates Figure 5 [ 10-87 10-87 ]”
Prompt-learning for Fine-grained Entity Typing
Ding, N., Chen, Y., Han, X., Xu, G., Wang, X., Xie, P., Zheng, H., Liu, Z., Li, J., & Kim, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.512 · Cited by 152 · see citing works
Self-citationAdvances of machine learning in materials science: Ideas and techniques
Chong, S. S., Ng, Y. S., Wang, H., & Zheng, J. (2023) · Frontiers of Physics · doi:10.1007/s11467-023-1325-z · Cited by 109 · see citing works
Exploring prompt pattern for generative artificial intelligence in automatic question generation
Wang, L., Song, R., Guo, W., & Yang, H. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2412082 · Cited by 53 · see citing works
“The applications of LLMs, such as ChatGPT, in education have recently made significant progress (Ding et al., 2021; Liu, Yuan, et al., 2023b).”
PPT: Backdoor Attacks on Pre-trained Models via Poisoned Prompt Tuning
Du, W., Zhao, Y., Li, B., Liu, G., & Wang, S. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/96 · Cited by 36 · see citing works
Generative Multi-Modal Knowledge Retrieval with Large Language Models
Long, X., Zeng, J., Meng, F., Ma, Z., Zhang, K., Zhou, B., & Zhou, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29837 · Cited by 34 · see citing works
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Zhong, W., Gao, Y., Ding, N., Qin, Y., Liu, Z., Zhou, M., Wang, J., Yin, J. P., & Duan, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.313 · Cited by 29 · see citing works
Self-citationXPrompt: Exploring the Extreme of Prompt Tuning
Ma, F., Zhang, C., Ren, L., Wang, J., Wang, Q., Wu, W., Quan, X., & Song, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.758 · Cited by 31 · see citing works
Pre-trained Language Models Can be Fully Zero-Shot Learners
Zhao, X., Ouyang, S., Yu, Z., Wu, M., & Li, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.869 · Cited by 29 · see citing works
MANNER: A Variational Memory-Augmented Model for Cross Domain Few-Shot Named Entity Recognition
Fang, J., Wang, X., Meng, Z., Xie, P., Huang, F., & Jiang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.234 · Cited by 17 · see citing works
“Moreover, prompt-based techniques (Cui et al., 2021; Ding et al., 2022; Chen et al., 2022) have also been proposed to address few-shot NER tasks.”
BackgroundDiscoPrompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition
Chan, C., Liu, X., Cheng, J., Li, Z., Song, Y., Wong, G. Y., & See, S. C. W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.4 · Cited by 20 · see citing works
Fei, Y., Zhao, M., Nie, P., Wattenhofer, R. P., & Sachan, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.587 · Cited by 20 · see citing works
Toward Human Readable Prompt Tuning: Kubrick’s The Shining is a good movie, and a good prompt too?
Shi, W., Han, X., Gonen, H., Holtzman, A., Tsvetkov, Y., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.733 · Cited by 18 · see citing works
OpenICL: An Open-Source Framework for In-context Learning
Wu, Z., Wang, Y., Ye, J., Wu, Z., Feng, J., Xu, J., & Qiao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.47 · Cited by 13 · see citing works
On the Compositional Generalization Gap of In-Context Learning
Hosseini, A., Vani, A., Bahdanau, D., Sordoni, A., & Courville, A. C. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.22 · Cited by 12 · see citing works
Zhuang, Y., Zhang, J., Li, X., Liu, C., Yu, Y., Dong, W., & He, K. (2024) · JMIR Medical Informatics · doi:10.2196/63020 · Cited by 9 · see citing works
Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
Colombo, P., Pellegrain, V., Boudiaf, M., Tami, M., Storchan, V., Ayed, I. B., & Piantanida, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.257 · Cited by 7 · see citing works
Jurgens, D., Seth, A., Sargent, J., Aghighi, A., & Geraci, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.616 · Cited by 7 · see citing works
Prompt-based Distribution Alignment for Domain Generalization in Text Classification
Chen, J., & Zhang, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.690 · Cited by 6 · see citing works
“, 2019) as the default PLM, building the model on the OpenPrompt framework (Ding et al., 2022).”
MethodScalable Prompt Generation for Semi-supervised Learning with Language Models
Zhou, Y., Maharjan, S., & Liu, B. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.58 · Cited by 10 · see citing works
BMInf: An Efficient Toolkit for Big Model Inference and Tuning
Han, X., Zeng, G., Zhao, W., Liu, Z., Zhang, Z., Zhou, J., Zhang, J., Chao, J., & Sun, M. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.22 · Cited by 5 · see citing works
“…level, we will continue to improve our model quantization methods to achieve better performance, and work with other toolkits such as OpenPrompt (Ding et al., 2021) to explore more effective ways to tune big models; (3) At the implementation level, we will provide long-term maintenance for this…”
MethodSelf-citationMedTem2.0: Prompt-based Temporal Classification of Treatment Events from Discharge Summaries
Cui, Y., Han, L., & Nenadić, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.27 · Cited by 5 · see citing works
“Consequently, this probability is used to predict y , thereby bypassing the need for extensive labelled datasets (Liu et al., 2023; Ding et al., 2021 We used OpenPrompt , a toolkit for implementing prompt learning in downstream tasks (Ding et al., 2021).”
InfluentialAbdel‐Salam, R. (2022) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/2022.wanlp-1.48 · Cited by 5 · see citing works
“For MARBERT with prompt-tuning, openprompt library was used (Ding et al., 2021), which used P-tuning.”
MethodPrompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognition
Wang, C., Jian, P., & Huang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.45 · Cited by 5 · see citing works
Lin, X., Fu, Y., Yang, Z., Lin, N., & Jiang, S. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.27 · Cited by 5 · see citing works
“Prompt-learning is a new paradigm in modern natural language processing to adapt pre-trained language models (PLMs) to down-stream NLP tasks, which modifies the input text with a textual template and directly uses PLMs to conduct pre-trained tasks.”
BackgroundMethodExploring Lottery Prompts for Pre-trained Language Models
Chen, Y., Ding, N., Wang, X., Hu, S., Zheng, H.-T., Liu, Z., & Xie, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.860 · Cited by 6 · see citing works
Self-citationManifold-Based Verbalizer Space Re-embedding for Tuning-Free Prompt-Based Classification
Wang, H., Zhao, S., Liu, C., Xi, N., Cai, M., Qin, B., & Liu, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29880 · Cited by 4 · see citing works
CPL-NoViD: Context-Aware Prompt-Based Learning for Norm Violation Detection in Online Communities
He, Z., May, J., & Lerman, K. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31335 · Cited by 5 · see citing works
Tripathi, S., Sukumaran, R., Dheer, S., & Cook, T. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4785683 · Cited by 4 · see citing works
A Prompt Based Approach for Euphemism Detection
Maimaitituoheti, A., Yang, Y., & Fan, X.-C. (2022) · FLP · doi:10.18653/v1/2022.flp-1.2 · Cited by 4 · see citing works
“OpenPrompt (Ding et al., 2022) is an open-source prompt learning framework based on PyTorch (Paszke et al., 2019), which provided the implementation of many templates and verbalizers, including manually created templates, prefix-tuning templates, p-tuning templates, ptr templates, manually created…”
MethodData-efficient end-to-end Information Extraction for Statistical Legal Analysis
Hwang, W., Eom, S., Lee, H., Park, H. J., & Seo, M. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.12 · Cited by 3 · see citing works
Memory-Based Invariance Learning for Out-of-Domain Text Classification
Chen, J., & Zhang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.101 · Cited by 2 · see citing works
“We utilize RoBERTa BASE (Liu et al., 2019) as the primary Pretrained Language Models (PLMs) in our study, following the OpenPrompt framework (Ding et al., 2022), for two text classification tasks.”
MethodAbdel‐Salam, R. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.52 · Cited by 3 · see citing works
“However, in subtask-2B the goal was to predict the disinformation class of a given text. subtask-2A In this subtask, the same experiments conducted in subtask-1A were used in this subtask. subtask-2B In this subtask, prompt-tuning was utilized using openprompt library (Ding et al., 2021).”
MethodPrompterator: Iterate Efficiently towards More Effective Prompts
Sučík, S., Skala, D., Švec, A., Hraška, P., & Šuppa, M. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.43 · Cited by 3 · see citing works
Decoder Tuning: Efficient Language Understanding as Decoding
Cui, G., Li, W., Ding, N., Huang, L., Liu, Z., & Sun, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.840 · Cited by 2 · see citing works
Self-citationCEIL: A General Classification-Enhanced Iterative Learning Framework for Text Clustering
Zhao, M. M., Wang, M., Ma, Y., Niu, D., & Wu, H. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583457 · Cited by 2 · see citing works
“We implement our CEIL framework based on PyTorch [14] and OpenPrompt [6], and incorporate distilbert-base-nli-stsb-mean-tokens from Sentence Transformer [19] as the initial checkpoint of the language model.”
MethodMulti-Source Probing for Open-Domain Conversational Understanding
Li, Y., Zhou, H., Zhou, J., & Huang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.769 · Cited by 1 · see citing works
“We implemented the above models with Py-Torch (Paszke et al., 2017), OpenPrompt (Ding et al., 2021) and Huggingface Library (Wolf et al., 2019a).”
MethodNo Evidence, No Problem: When Less is More for Out of Context Multimodal Misinformation Detection
Chen, S., Naseem, U., & Nasim, M. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3795462 · Cited by 1 · see citing works
“In this work, we leverage the OpenPrompt framework [5] to design effective prompt templates for multimodal misinformation detection.”
Multi-Mask Label Mapping for Prompt-Based Learning
Qi, J., Zhang, R., Kim, J., Chen, J., Qin, W., & Mao, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26579 · Cited by 1 · see citing works
CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels
Cho, H., Kim, Y., & Lee, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.239 · Cited by 2 · see citing works
Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models
Choi, J. (2024) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2024.13201 · Cited by 1 · see citing works
“Liang et al. [16] demonstrated a few-shot technique for generating relevance, while studies like those by Sun et al. [30] and Dai et al. [6] have successfully applied few-shot learning in various scenarios.”
BackgroundMethodMultihop Factual Claim Verification Using Natural Language Prompts
Rahman, M. M., Islam, M. S., Laskar, M. T. R., Hossain, M. A., & Kamal, A. R. M. (2023) · doi:10.21428/594757db.63abb0f0
Ameer, I., Bölücü, N., Xu, H., & Bataineh, A. A. (2023) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2023.wassa-1.56
Keep Skills in Mind: Understanding and Implementing Skills in Commonsense Question Answering
Bao, M., Liu, Q., Zhang, K., Liu, Y., Yue, L., Li, L., & Zhou, J. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/557
Transductive Learning for Textual Few-Shot Classification in API-based Embedding ModelsPreprint
Colombo, P., Pellegrain, V., Boudiaf, M., Storchan, V., Tami, M., Ayed, I. B., Hudelot, C., & Piantanida, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13998
“Prompt-based FSL : Prompt-based FSL involves the use of natural language prompts or templates to guide the model to perform a specific task (Ding et al., 2021; Liu et al., 2023).”
MethodLearning to Predict Task Transferability via Soft Prompt
Feng, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.546
“…and the knowledge in a large pretrained language model can be elicited with prompts (Liu et al., 2021a; Li and Liang, 2021; Lester et al., 2021; Ding et al., 2021), we conduct prompt tuning, a lightweight alternative to fine-tuning, on each task and store each task prompt in a prompt pool.”
BackgroundBeyond Human Review: Levereging ChatGPT for Label Noise DetectionPreprint
Cichecki, I., Kocoń, J., Kazienko, P., Kaszyca, O., Szydło, D., & Kochanek, M. (2023) · doi:10.36227/techrxiv.170326715.56351742/v1
PromptER: Prompt Contrastive Learning for Generalized Entity Resolution
Dai, C., Tang, Q., Ma, W., Wu, Y., Zhou, H., & Ding, H. (2024) · doi:10.5121/csit.2024.140102
“…pre-training and fine-tuning: Although methods based on PLMs have achieved impressive results in entity resolution tasks, recent research [15, 16] shows that the gap between the objectives of pre-training and fine-tuning, which restricts the full utilization of knowledge in pre-trained…”
MethodInfluence of Prompt Terms on Relevance Evaluation with GPTs
Choi, J. (2024) · doi:10.5121/csit.2024.140603
“In the broader context of prompt-learning, or “prompting,” the approach is increasingly recognized as a frontier in natural language processing, seamlessly bridging the gap between the pre-training and fine-tuning phases of model development [25, 26].”
BackgroundAlleviating the medical strain: a triage method via cross-domain text classification
Xiao, X., Wang, S., Jiang, F., Qi, T., & Wang, W. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1468519
Li, X., Wu, X., Xiao, L., Liu, T., Huang, P. S., & Shan, J. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00355
“The framework combines LLM-based utterance generation, parameterized prompts encoding structured schemas, and function-guided sampling to ensure consistency with executable operations, inspired by prompt-based and instruction-driven synthesis methods [5]–[8].”
Knowledge-Aware Prototypical Prompt Verbalizer: A New Approach for Few-Shot Text Classification
Yuan, P. (2025) · International Conference on Applied Mathematics, Modelling and Intelligent Computing (CAMMIC) · doi:10.1145/3745533.3745602
“The model is implemented using the PyTorch framework [27] and the OpenPrompt framework [28].”
- Authors
- Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H.-T., & Sun, M.
- Affiliations
- Tsinghua University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2111.01998
- OpenAlex
- W4309444617
- Cited by
- 60
- References
- 45
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 14 · 2023: 28 · 2022: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- prompt learning, pre-trained language models, open source software framework, verbalization, sequence-to-sequence generation, natural language processing toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
