PTR: Prompt Tuning with Rules for Text Classification
Open accessAbstract
Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the rich knowledge distributed in PLMs to better serve downstream tasks. Prompt tuning has achieved promising results on some few-class classification tasks such as sentiment classification and natural language inference. However, manually designing lots of language prompts is cumbersome and fallible. For those auto-generated prompts, it is also expensive and time-consuming to verify their effectiveness in non-few-shot scenarios. Hence, it is still challenging for prompt tuning to address many-class classification tasks. To this end, we propose prompt tuning with rules (PTR) for many-class text classification and apply logic rules to construct prompts with several sub-prompts. In this way, PTR is able to encode prior knowledge of each class into prompt tuning. We conduct experiments on relation classification, a typical and complicated many-class classification task, and the results show that PTR can significantly and consistently outperform existing state-of-the-art baselines. This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Han et al. propose prompt tuning with rules (PTR), which uses manually crafted sub-templates to compose a complete template using logic rules.
“Han et al. [41] propose prompt tuning with rules (PTR), which uses manually crafted sub-templates to compose a complete template using logic rules.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [41]
PTR inserts several virtual tokens whose embeddings can be tuned together with the pre-trained LMs parameters using training samples.
“To enhance the representation ability of the resulting template, they also insert several virtual tokens whose embeddings can be tuned together with the pre-trained LMs parameters using training samples.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [41]
Experiment results demonstrate the effectiveness of PTR prompt design method in relation classification tasks.
“Experiment results demonstrate the effectiveness of this prompt design method in relation classification tasks.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [41]
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“…al., 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al., 2020), to text classification (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021), fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks.”
BackgroundPan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020; Schick et al., 2020).”
BackgroundRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 4× · see citing works
“, 2019), RoBERTa (Liu et al., 2019) and T5 (Raffel et al.”
BackgroundInfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1× · see citing works
“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”
BackgroundPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1× · see citing works
“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”
BackgroundExploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2020) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2001.07676 · Cited by 1,932 · cited 2×
“tasks such as natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”
BackgroundLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · cited 3× · see citing works
“ries of research work on knowledge probing (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019), language prompts have been proven to effectively stimulate knowledge from PLMs.”
BackgroundSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · cited 1× · see citing works
“For those knowledge-enhanced PLMs, we select SPANBERT (Joshi et al., 2020), KNOWBERT (Peters et al.”
MethodText Summarization with Pretrained Encoders
Liu, Y., & Lapata, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1387 · Cited by 1,779 · cited 1× · see citing works
“, 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al.”
BackgroundPre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · cited 1× · see citing works
“It is now a consensus of the NLP community to fine-tune PLMs for specific tasks instead of learning models from scratch (Qiu et al., 2020).”
BackgroundMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 2× · see citing works
“Gao et al. (2020) take seq-to-seq models to generate prompt candidates, and then use each of them for prompt tuning and verify their effectiveness on development sets.”
BackgroundMethod(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 2× · see citing works
“By applying self-supervised learning, PLMs can capture a wealth of linguistic (Jawahar et al., 2019), semantic (Yenicelik et al.”
BackgroundMethodPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206 · cited 1× · see citing works
“, 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al.”
BackgroundPosition-aware Attention and Supervised Data Improve Slot Filling
Zhang, Y., Zhong, V., Chen, D., Angeli, G., & Manning, C. D. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1004 · Cited by 1,009 · cited 6× · see citing works
“Four popular benchmarks of relation classification are used for our experiments, including TA-CRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Stoica et al., 2021), and Se-mEval 2010 Task 8 (Hendrickx et al., 2010).”
MethodInfluentialSemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
Hendrickx, I., Kim, S., Kozareva, Z., Nakov, P., Séaghdha, D. Ó., Padó, S., Pennacchiotti, M., Romano, L., & Szpakowicz, S. (2009) · SEW@NAACL-HLT · doi:10.3115/1621969.1621986 · Cited by 929 · cited 2× · see citing works
“Although these PLMs can capture rich knowledge (Jawahar et al., 2019; Hewitt and Manning, 2019; Petroni et al., 2019; Yenicelik et al., 2020) from massive corpora, a fine-tuning process with extra task-specific data is still required to transfer their knowledge for downstream tasks.”
BackgroundLiu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908 · cited 4×
“Moreover, human-picked prompts have achieved promising results on few-class classification tasks such as sentiment classification and natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”
BackgroundInfluentialGraph Convolution over Pruned Dependency Trees Improves Relation Extraction
Zhang, Y., Qi, P., & Manning, C. D. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1244 · Cited by 809 · cited 3× · see citing works
“We carry out our experiments on four relaction clas-sification datasets: TACRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Sto-ica et al., 2021), and SemEval 2010 Task 8 (Se-mEval) (Hendrickx et al., 2010).”
MethodLUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentionPreprint
Yamada, I., Asai, A., Shindo, H., Takeda, H., & Matsumoto, Y. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.01057 · Cited by 802 · see citing works
Efficient Intent Detection with Dual Sentence Encoders
Casanueva, I., Temcinas, T., Gerz, D., Henderson, M., & Vulic, I. (2020) · NLP4CONVAI · doi:10.48550/arxiv.2003.04807 · Cited by 750
Matching the Blanks: Distributional Similarity for Relation Learning
Baldini Soares, L., FitzGerald, N., Ling, J., & Kwiatkowski, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1279 · Cited by 731 · see citing works
Knowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661 · cited 1× · see citing works
“, 2020), KNOWBERT (Peters et al., 2019), LUKE (Yamada et al.”
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · cited 2× · see citing works
“, 2019; Liu and Lapata, 2019), question answering (Adiwardana et al., 2020), to”
Background(2019) · doi:10.18653/v1/n19-1419 · Cited by 601 · cited 1× · see citing works
“Although these PLMs have captured rich knowledge (Jawahar et al., 2019; Hewitt and Manning, 2019; Petroni et al., 2019; Yenicelik et al., 2020) from massive corpora, a fine-tuning process with extra task-specific data is still required to transfer the knowledge for downstream tasks.”
BackgroundEliciting Knowledge from Language Models Using Automatically Generated PromptsPreprint
Shin, T., Razeghi, Y., Logan, R. L., IV, Wallace, E., & Singh, S. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.15980 · Cited by 584 · cited 2× · see citing works
“Shin et al. (2020) further explore gradient-guided search to automatically generate both templates and label words.”
BackgroundMultilingual Universal Sentence Encoder for Semantic RetrievalPreprint
Yang, Y., Cer, D. M., Ahmad, A., Guo, M., Law, J., Constant, N., Ábrego, G. H., Yuan, S., Tar, C., Sung, Y.-H., et al. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1907.04307 · Cited by 572 · see citing works
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1131 · Cited by 454 · see citing works
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., & Smith, N. (2020) · arXiv · doi:10.48550/arxiv.2002.06305 · Cited by 412 · cited 1× · see citing works
“Besides fine-tuning language models for specific tasks, recent studies have explored better optimization and regularization techniques to improve fine-tuning (Lee et al., 2019; Dodge et al., 2020).”
MethodKnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511998 · Cited by 389 · see citing works
CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases
Ren, X., Wu, Z., He, W.-Q., Qu, M., Voss, C. R., Ji, H., Abdelzaher, T., & Han, J. (2016) · The Web Conference · doi:10.1145/3038912.3052708 · Cited by 321 · see citing works
A Simple Method for Commonsense Reasoning
Trinh, T. H., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1806.02847 · Cited by 293 · see citing works
Commonsense Knowledge Mining from Pretrained Models
Davison, J., Feldman, J., & Rush, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1109 · Cited by 264 · cited 1× · see citing works
“ries of research work on knowledge probing (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019), language prompts have been proven to effectively stimulate knowledge from PLMs.”
BackgroundDialoGPT: Large-Scale Generative Pre-training for Conversational Response GenerationPreprint
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00536 · Cited by 258 · cited 1× · see citing works
“From dialogue (Zhang et al., 2020), summarization (Zhang et al.”
Few-NERD: A Few-shot Named Entity Recognition Dataset
Ding, N., Xu, G., Chen, Y., Wang, X., Han, X., Xie, P., Zheng, H.-T., & Liu, Z. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.248 · Cited by 245 · see citing works
Self-citationAutomatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification
Schick, T., Schmid, H., & Schütze, H. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2010.13641 · Cited by 233 · cited 1×
“To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020; Schick et al., 2020).”
BackgroundLearning from Context or Names? An Empirical Study on Neural Relation Extraction
Peng, H., Gao, T., Han, X., Lin, Y., Li, P., Liu, Z., Sun, M., & Zhou, J. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.01923 · Cited by 229 · cited 1×
“, 2020), to text classification (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021), fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks.”
BackgroundSelf-citationTACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task
Alt, C., Gabryszak, A., & Hennig, L. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.14855 · Cited by 192 · cited 9×
“TACREV: one dataset built based on the original TACRED dataset.”
InfluentialRe-TACRED: Addressing Shortcomings of the TACRED Dataset
Stoica, G., Platanios, E. A., & Póczos, B. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2104.08398 · Cited by 158 · cited 7×
“We introduce the reversed setting of TACRED, TACREV, Re-TACRED in Section 5.3 and report the corresponding results in Table 2.”
BackgroundMethodInfluentialOpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction
Han, X., Gao, T., Yao, Y., Ye, D., Liu, Z., & Sun, M. (2019) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d19-3029 · Cited by 152 · cited 1× · see citing works
“Our model is implemented based on the toolkit Transformers (Wolf et al., 2020) and Open-NRE (Han et al., 2019).”
MethodSelf-citationMixout: Effective Regularization to Finetune Large-scale Pretrained Language ModelsPreprint
Lee, C., Cho, K., & Kang, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11299 · Cited by 150 · cited 1× · see citing works
“Besides fine-tuning language models for specific tasks, recent studies have explored better optimization and regularization techniques to improve fine-tuning (Lee et al., 2019; Dodge et al., 2020).”
BackgroundNeural Architectures for Fine-grained Entity Type ClassificationPreprint
Shimaoka, S., Stenetorp, P., Inui, K., & Riedel, S. (2016) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.1606.01341 · Cited by 122 · see citing works
Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language InferencePreprint
Zhang, J., Hashimoto, K., Liu, W.-H., Wu, C.-S., Wan, Y., Yu, P. S., Socher, R., & Xiong, C. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2010.13009 · Cited by 100 · see citing works
Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning
Zhang, J., Bui, T., Yoon, S., Chen, X., Liu, Z., Xia, C., Tran, Q. H., Chang, W. H., & Yu, P. S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.144 · Cited by 77 · see citing works
Self-citationAn Improved Baseline for Sentence-level Relation ExtractionPreprint
Zhou, W., & Chen, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01373 · Cited by 64 · see citing works
Prototypical Representation Learning for Relation ExtractionPreprint
Ding, N., Wang, X., Fu, Y., Xu, G., Wang, R., Xie, P., Shen, Y., Huang, F., Zheng, H.-T., & Zhang, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11647 · Cited by 60 · cited 1× · see citing works
“, 2020), to text classification (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021), fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks.”
BackgroundSelf-citationHow does BERT capture semantics? A closer look at polysemous words
Yenicelik, D., Schmidt, F., & Kilcher, Y. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.15 · Cited by 51 · cited 2× · see citing works
“, 2019), semantic (Yenicelik et al., 2020), syntactic (Hewitt and Manning, 2019) and world”
BackgroundPrompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning
Zhang, R., Yu, Y., Shetty, P., Song, L., & Zhang, C. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.09735 · Cited by 29
Flagged first, then by citations
5 of the 54 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · see citing works
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and TasksPreprint
Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07602 · Cited by 386 · see citing works
“Prompt tuning and P-tuning have been proved quite effective in many NLU tasks, including knowledge probing, sentimental analysis (Li et al., 2021), relation extraction (Han et al., 2021), entity typing (Ding et al., 2021) and so on.”
BackgroundPre-Trained Models: Past, Present and FuturePreprint
Han, X., Zhang, Z.-Y., Ding, N., Gu, Y.-X., Liu, X., Huo, Y., Qiu, J.-Z., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.48550/arxiv.2106.07139 · Cited by 1,145 · see citing works
“By designing, generating and searching discrete (Petroni et al., 2019; Gao et al., 2021; Hu et al., 2021) or continuous (Liu et al., 2021b; Han et al., 2021; Lester et al., 2021) prompts and using MLM for specific downstream tasks, these models could (1) bridge the gap between pre-training and…”
BackgroundMethodSelf-citationLarge Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029 · see citing works
PPT: Pre-trained Prompt Tuning for Few-shot LearningPreprint
Gu, Y.-X., Han, X., Liu, Z.-Y., & Huang, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2109.04332 · Cited by 507 · see citing works
“Hybrid Prompt Tuning In hybrid prompt tuning, both soft and hard prompts are used (Liu et al., 2021; Han et al., 2021b).”
BackgroundMethodInfluentialSelf-citationKnowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationPreprint
Hu, S.-D., Ding, N., Wang, H., Liu, Z.-Y., Wang, J.-G., Li, J.-Z., Wu, W., & Sun, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2108.02035 · Cited by 457 · see citing works
“, 2021) , Relation Extraction (Han et al., 2021; Chen et al., 2021), and Knowledge Probing (Petroni et al.”
BackgroundMethodSelf-citationKnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511998 · Cited by 389 · see citing works
“We conducted 8-, 16-, and 32-shot experiments following LM-BFF [15, 22] to measure the average performance across five different randomly sampled data based on every experiment using a fixed set of seeds Sseed.”
BackgroundMethodLu, Y., Liu, J., Zhang, Y., Liu, Y., & Tian, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00514 · Cited by 262 · see citing works
“Recently, some works [12,22,24,48], also known as prompt tuning , attempt to learn soft (continuous) prompts directly instead of searching for discrete prompts.”
BackgroundAugGPT: Leveraging ChatGPT for Text Data Augmentation
Dai, H.-X., Liu, Z., Liao, W., Huang, X.-K., Cao, Y.-H., Wu, Z., Zhao, L., Xu, S.-C., Zeng, F., Liu, W., et al. (2023) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536934 · Cited by 322 · see citing works
Self-citationText Classification via Large Language Models
Sun, X., Li, X., Li, J., Wu, F. Q., Guo, S., Zhang, T., & Wang, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.603 · Cited by 273 · see citing works
Reframing Instructional Prompts to GPTk’s LanguagePreprint
Mishra, S., Khashabi, D., Baral, C., Choi, Y., & Hajishirzi, H. (2021) · Findings · doi:10.48550/arxiv.2109.07830 · Cited by 287 · see citing works
“For example, they are studied in the context of classification (Petroni et al., 2019; Schick and Schütze, 2021) or applied as prompt composition and prompt decomposition for relation extraction (Han et al., 2021) and named entity recognition (Cui et al., 2021) tasks, respectively.”
BackgroundA Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems
Yi, Z., Ouyang, J., Xu, Z., Liu, Y., Liao, T., Luo, H., & Shen, Y. (2025) · ACM Computing Surveys · doi:10.1145/3771090 · Cited by 136 · see citing works
“The mainstream tuning-free prompting methods include In-context Learning ( ICL ), Chain-of-thought ( CoT ) and Retrieval Augmented Generation ( RAG ).”
A Survey of Knowledge Enhanced Pre-Trained Language Models
Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., & Li, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3310002 · Cited by 224 · see citing works
“Besides, PTR [27] incorporates logic rules to construct task-specific prompts composed with sub-prompts designed manually, so that the model can encode task-related prior knowledge in the prompt-tuning and generate prompts that are more interpretable.”
BackgroundMethodInfluentialSelf-citationActive Prompting with Chain-of-Thought for Large Language ModelsPreprint
Diao, S., Wang, P., Lin, Y., Liu, X., & Zhang, T. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2302.12246 · Cited by 254 · see citing works
Template-free Prompt Tuning for Few-shot NERPreprint
Ma, R., Zhou, X., Gui, T., Tan, Y., Li, L., Zhang, Q., & Huang, X. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2109.13532 · Cited by 213 · see citing works
“Meanwhile, several approaches have explore continuous prompts for both text classification and generation tasks Li and Liang (2021); Liu et al. (2021); Han et al. (2021).”
BackgroundDifferentiable Prompt Makes Pre-trained Language Models Better Few-shot LearnersPreprint
Zhang, N., Li, L., Chen, X., Deng, S., Bi, Z., Tan, C., Huang, F., & Chen, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13161 · Cited by 124 · see citing works
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y. F., Zhang, H., & Lyu, M. R.-T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549113 · Cited by 153 · see citing works
“For the example shown in Figure 1, prompt tuning outputs the final predicted class by a verbalizer [16, 49].”
BackgroundMethodInfluentialA Comprehensive Study of Knowledge Editing for Large Language ModelsPreprint
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S.-M., Wang, M.-R., Xi, Z.-K., Mao, S., Zhang, J., Ni, Y., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.01286 · Cited by 194 · see citing works
“Knowledge Augmentation for LLMs LLMs still face unknown questions, and many knowledge-augmented methods are proposed to help the model deal with this task [105–107].”
MethodInformation Retrieval: Recent Advances and Beyond
Hambarde, K., & Proença, H. P. M. C. (2023) · IEEE Access · doi:10.1109/access.2023.3295776 · Cited by 176 · see citing works
“Though prompt tuning has gained outstanding outcomes in domains such as data extraction [247], [248], text classification [249], [250], and fact probing [243], [251], there has existed no extended work on prompt tuning.”
BackgroundZhou, B., Li, X., Liu, T., Xu, K., Liu, W., & Bao, J. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102333 · Cited by 181 · see citing works
Prompt-Learning for Fine-Grained Entity TypingPreprint
Ding, N., Chen, Y., Han, X., Xu, G., Wang, X.-B., Xie, P.-J., Zheng, H.-T., Liu, Z.-Y., Li, J.-Z., & Kim, H.-G. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2108.10604 · Cited by 179 · see citing works
“Despite the success of fine-tuning PLMs, the huge objective form gap between pre-training and fine-tuning still hinders the full use of per-trained knowledge for downstream tasks (Liu et al., 2021b; Han et al., 2021b; Hu et al., 2021).”
BackgroundSelf-citationA comprehensive survey on relation extraction: recent advances and new frontiers
Zhao, X., Deng, Y., Yang, M., Wang, L., Zhang, R., Cheng, H., Lam, W., Shen, Y., & Xu, R. (2024) · ACM Computing Surveys · doi:10.1145/3674501 · Cited by 168 · see citing works
“Recent works [60, 133] show that prompt learning can effectively leverage the knowledge encoded in the PLMs, especially for few-shot RE tasks in Section 4.1.2.”
InfluentialAn Improved Baseline for Sentence-level Relation ExtractionPreprint
Zhou, W., & Chen, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01373 · Cited by 64 · see citing works
“Since then, the proposed techniques have been incorporated into several followup works (Chen et al., 2022; Wang et al., 2022b,a; Lu et al., 2022; Han et al., 2021; Kulkarni et al., 2022) that are published before this version of the paper.”
MethodFedPrompt: Communication-Efficient and Privacy-Preserving Prompt Tuning in Federated Learning
Zhao, H., Du, W., Li, F., Li, P., & Liu, G. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095356 · Cited by 86 · see citing works
“Moreover, PTR (Han et al. 2021) applies logic rules to build templates that are more suitable for text classification tasks.”
Zhang, S., Khan, S. A., Shen, Z., Naseer, M., Chen, G., & Khan, F. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00339 · Cited by 86 · see citing works
ChatAug: Leveraging ChatGPT for Text Data AugmentationPreprint
Dai, H.-X., Liu, Z., Liao, W., Huang, X.-K., Wu, Z., Zhao, L., Liu, W., Liu, N.-H., Li, S., Zhu, D.-J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2302.13007 · Cited by 147 · see citing works
Self-citationMultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction TuningPreprint
Xu, Z., Shen, Y., & Huang, L.-F. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10773 · Cited by 146 · see citing works
“Prompt tuning (Liu et al., 2021; Li and Liang, 2021; Han et al., 2022; Wang et al., 2022b; Sanh et al., 2022) aims to learn a task-specific prompt by reformulating the downstream tasks to the format that the model was initially trained on and has shown competitive performance across various natural…”
BackgroundPacked Levitated Marker for Entity and Relation Extraction
Ye, D., Lin, Y., Li, P., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.337 · Cited by 144 · see citing works
“More-over, to build the connection between entity type and relation type, we add an auxiliary loss for predicting the type of object entity (Zhou and Chen, 2021; Han et al., 2021).”
Self-citationLFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5Preprint
Qin, C., & Joty, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07298 · Cited by 50 · see citing works
Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors
Zhang, K., Gutierrez, B. J., & Su, Y. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.11159 · Cited by 141
“Following previous work (Sainz et al., 2021; Lu et al., 2022; Han et al., 2022; Chen et al., 2022), we report the micro averaged F1 with the none-of-the-above relation excluded.”
BackgroundMethodInfluentialWang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 176 · see citing works
“Numerous studies have explored the impact of prompt engineering on AI generative models across domains, including image generation tasks [85] and classification tasks [86], highlighting the critical role of excellent prompt design for LLMs like ChatGPT [87], [88].”
BackgroundLv, X., Lin, Y.-K., Cao, Y., Hou, L., Li, J.-Z., Liu, Z., Li, P., & Zhou, J. (2022) · Findings · doi:10.18653/v1/2022.findings-acl.282 · Cited by 131 · see citing works
“To make triple prompts more expressive, inspired by Han et al. (2021), we also add some soft prompts to P Tp to form the final triple prompts P T .”
BackgroundSelf-citationExploring Visual Prompts for Adapting Large-Scale Models
Bahng, H., Jahanian, A., Sankaranarayanan, S., & Isola, P. (2022) · arXiv · doi:10.48550/arxiv.2203.17274 · Cited by 129 · see citing works
“Currently, prompting methods are dominantly NLP-specific [4, 40–42, 13, 30, 27, 36, 14, 48, 15, 26], despite the fact that the framework serves a general purpose: adapt a frozen pre-trained model by modifying the data space .”
MethodLog Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129 · see citing works
“2) Prompt Tuning: Recently, prompt tuning [37], [38], [39], [40] has been proposed to close the gap between pre-training and downstream tasks.”
BackgroundLISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Pan, R., Liu, X., Diao, S., Pi, R.-J., Zhang, J., Han, C., & Zhang, T. (2024) · Neural Information Processing Systems · doi:10.48550/arxiv.2403.17919 · Cited by 127
“In general, PEFT methods can be grouped into three classes: 1) Prompt Learning methods [8, 28, 29, 30, 31, 32, 33], 2) Adapter methods [7, 9, 34, 35, 36, 37, 38], and 3) Selective methods [39, 39, 40, 41].”
Dynamic Prefix-Tuning for Generative Template-based Event Extraction
Liu, X., Huang, H., Shi, G., & Wang, B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.358 · Cited by 105 · see citing works
“Tsimpoukelli et al., 2021), initializing with discrete prompts(Zhong et al., 2021; Qin and Eisner, 2021; Hambardzumyan et al., 2021) and hybrid prompt tuning(Liu et al., 2021b,a; Han et al., 2021).”
BackgroundPrototypical Verbalizer for Prompt-based Few-shot Tuning
Cui, G., Hu, S.-D., Ding, N., Huang, L., & Liu, Z.-Y. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.09770 · Cited by 123
Self-citationAugmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting
Wen, Z., & Fang, Y. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591641 · Cited by 73 · see citing works
Learning from models beyond fine-tuning
Zheng, H., Shen, L., Tang, A., Luo, Y., Hu, H., Du, B., Wen, Y., & Tao, D. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00961-0 · Cited by 114 · see citing works
“Building on the concept of minimal parameter tuning, PTR (prompt tuning method with rules) 23 creates prompts using manual templates and virtual tokens, and then fine-tunes them with rules.”
Towards Large-Scale 3D Representation Learning with Multi-Dataset Point Prompt Training
Wu, X., Tian, Z., Wen, X., Peng, B., Liu, X., Yu, K., & Zhao, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01849 · Cited by 59 · see citing works
“The prompt templates could be heuristic designed [7, 59, 26], automatically generated [61, 20], or learned as task-specific parameters [43, 47, 24, 31, 15].”
MethodA Unified Evaluation of Textual Backdoor Learning: Frameworks and BenchmarksPreprint
Cui, G., Yuan, L., He, B.-X., Chen, Y., Liu, Z.-Y., & Sun, M. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.08514 · Cited by 114 · see citing works
“…security of PLMs. Specifically, we recognize that the novel adaptation methods, including prompt-based learning (data-efficient tuning) [33, 19, 21, 10] and delta tuning (parameter-efficient tuning) [14, 20, 29], are important in PLM deployment and they bring unique backdoor security…”
Self-citationTELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
Karmaker, S. (., & Feng, D. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.11430 · Cited by 111
“Unsurprisingly, the quality and effectiveness of the prompt can greatly influence the performance of the LLMs for a particular task, and therefore, designing appropriate prompts with the right amount of detail has become more important than ever (Liu et al., 2023; Han et al., 2022).”
BackgroundAdversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis
Wu, H., & Shi, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.174 · Cited by 73 · see citing works
Prompting Large Language Models With the Socratic Method
Chang, E. Y. (2023) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc57344.2023.10099179 · Cited by 77 · see citing works
CPM-2: Large-scale Cost-effective Pre-trained Language ModelsPreprint
Zhang, Z.-Y., Gu, Y.-X., Han, X., Chen, S.-Q., Xiao, C.-J., Sun, Z., Yao, Y., Qi, F., Guan, J., Ke, P., et al. (2021) · AI Open · doi:10.48550/arxiv.2106.10715 · Cited by 104 · see citing works
“More details of this part are included in our paper (Han et al., 2021).”
BackgroundSelf-citationOntology-enhanced Prompt-tuning for Few-shot Learning
Ye, H., Zhang, N., Deng, S., Chen, X., Chen, H., Xiong, F., Chen, X., & Chen, H. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511921 · Cited by 76 · see citing works
“Originally from GPT-3, prompt-tuning has been applied to various of tasks including relation extraction [20], event extraction [21, 59], named entity recognition [5, 7], entity typing [13], and so on.”
BackgroundMethodParadigm Shift in Natural Language Processing
Sun, T., Liu, X.-Y., Qiu, X., & Huang, X. (2022) · Machine Intelligence Research · doi:10.1007/s11633-022-1331-6 · Cited by 94 · see citing works
Black-box Prompt Learning for Pre-trained Language ModelsPreprint
Diao, S., Huang, Z., Xu, R., Li, X., Lin, Y., Zhou, X., & Zhang, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.08531 · Cited by 53 · see citing works
“…(Wallace et al., 2019; Shin et al., 2020; Jiang et al., 2020; Yuan et al., 2021; Haviv et al., 2021; Gao et al., 2021; Ben-David et al., 2021) and continuous prompt (Zhong et al., 2021; Qin and Eisner, 2021; Hambardzumyan et al., 2021; Liu et al., 2021c; Han et al., 2021; Li and Liang, 2021).”
How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?
Xu, X., Zhu, Y.-Q., Wang, X.-H., & Zhang, N. (2023) · SUSTAINLP · doi:10.48550/arxiv.2305.01555 · Cited by 86
“Most previous works focus on fine-tuning (Soares et al., 2019; Ye et al., 2022) or prompt-tuning (Chen et al., 2022; Han et al., 2021b) with relatively small language models (e.g., RoBERTa (Liu et al., 2019)).”
BackgroundMethodInfluentialPushing The Limit of LLM Capacity for Text Classification
Zhang, Y.-Z., Wang, M.-Y., Ren, C.-Y., Li, Q.-C., Tiwari, P., Wang, B.-Y., & Qin, J. (2024) · The Web Conference · doi:10.1145/3701716.3715528 · Cited by 84 · see citing works
Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models
Yu, S.-X., Muñoz, J. P., & Jannesari, A. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2305.11414 · Cited by 83
“Various methods for prompt engineering have been proposed, including prompt templates [24], prompt tuning and instruction tuning [24] [9, 4], automated prompt generating [31, 22], and in-context learning [14, 15, 20, 11].”
MethodLightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting
Chen, X., Li, L., Deng, S.-M., Tan, C.-Q., Xu, C.-L., Huang, F., Si, L., Chen, H.-J., & Zhang, N. (2021) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2109.00720 · Cited by 82
“A series of research work (Schick and Schütze, 2021; Schick et al., 2020; Shin et al., 2020; Han et al., 2021; Ben-David et al., 2022; Poth et al., 2021; Ben-David et al., 2022) have emerged, which implies that prompt-tuning can effectively stimulate knowledge from PLMs compared with standard fine-tuning, thus, induci…”
BackgroundKnowledge Mechanisms in Large Language Models: A Survey and Perspective
Wang, M.-R., Yao, Y., Xu, Z., Qiao, S.-F., Deng, S.-M., Wang, P., Chen, X., Gu, J.-C., Jiang, Y., Xie, P.-J., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2407.15017 · Cited by 81
“• Knowledge Augmentation is proposed to assist the model in handling unknown knowledge for LLMs (Zhang et al., 2019; Han et al., 2022).”
UniRel: Unified Representation and Interaction for Joint Relational Triple Extraction
Tang, W., Xu, B., Zhao, Y., Mao, Z.-D., Liu, Y., Liao, Y., & Xie, H.-Y. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2211.09039 · Cited by 81
“In terms of technical designs, SSAN (Xu et al., 2021) also delves into the self-attention layer within the Transformer to model structural interaction, but it makes extra adaptions for standard self-attention mechanism and focuses on document-level RE tasks.”
BackgroundMDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction
Zhang, L., Chen, Q., Wang, W., Deng, C., Zhang, S.-L., Li, B., Wang, W., & Cao, X. (2021) · Findings · doi:10.48550/arxiv.2110.06651 · Cited by 81
“Inspired by (Lewis et al., 2019; Zhang104 et al., 2020; Han et al., 2021), where pre-trained lan-105 guage models (PLMs) trained on objectives close106 to final downstream tasks achieve enhanced repre-107 sentations and improve fine-tune performance, we108 further propose a novel self-supervised…”
BackgroundMethodPrompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability
Rodriguez, A. D., Dearstyne, K. R., & Cleland‐Huang, J. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00087 · Cited by 71 · see citing works
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot GeneralizationPreprint
Xu, H., Chen, Y., Du, Y., Shao, N., Wang, Y.-G., Li, H., & Yang, Z.-L. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2201.06910 · Cited by 77 · see citing works
“, 2020; Le Scao and Rush, 2021), while the latter inserts learnable parameters in the template and directly optimize them during training (Han et al., 2021; Liu et al., 2021; Lester et al., 2021).”
BackgroundPre-trained Language Model with Prompts for Temporal Knowledge Graph Completion
Xu, W.-J., Liu, B., Peng, M., Jia, X., & Peng, M. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.07912 · Cited by 75
“We add a soft prompt [EVE] before the beginning of each fact tuple due to introducing soft prompts in the input sentences can improve the expressiveness of the sentences(Han et al., 2021).”
BackgroundChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse RelationsPreprint
Chan, C., Jiayang, C., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y.-Q. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14827 · Cited by 75 · see citing works
“…for many downstream tasks in the Natural Language Processing (NLP) field, the sensitivity and performance variance of the prompting method has been reported in a lot of works (Han et al., 2021; Chan et al., 2023a; Zhong et al., 2021; Liu et al., 2021a; Li et al., 2023b; Chan and Chan, 2023).”
BackgroundSentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment AnalysisPreprint
Li, C., Gao, F., Bu, J., Xu, L., Chen, X., Gu, Y., Shao, Z., Zheng, Q., Zhang, N., Wang, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08306 · Cited by 76 · see citing works
“Prompt-tuning (Jiang et al. 2020; Shin et al. 2020; Schick and Schutze 2021; Han et al. 2021), as a new fine-tuning paradigm, has emerged which is proposed to bridge the gap of objective forms between pre-training and fine-tuning.”
Chan, C., Jiayang, C., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y.-Q. (2024) · Findings · doi:10.18653/v1/2024.findings-eacl.47 · Cited by 73
“…for many downstream tasks in the Natural Language Processing (NLP) field, the sensitivity and performance variance of the prompting method has been reported in a lot of works (Han et al., 2021; Chan et al., 2023a; Zhong et al., 2021; Liu et al., 2021a; Li et al., 2023b; Chan and Chan, 2023).”
BackgroundContinual Prompt Tuning for Dialog State Tracking
Zhu, Q., Li, B., Mi, F., Zhu, X.-Y., & Huang, M. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.06654 · Cited by 72
“some works (Qin and Eisner, 2021; Liu et al., 2021; Han et al., 2021) combine hard text prompts and soft prompts whose embeddings are learned through back-propagation.”
BackgroundMask-guided BERT for few-shot text classification
Liao, W., Liu, Z., Dai, H., Wu, Z., Zhang, Y., Huang, X., Chen, Y., Jiang, X., Liu, D., Zhu, D., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128576 · Cited by 58 · see citing works
Self-citationDecoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning
Chen, X., Li, L., Zhang, N.-Y., Liang, X.-Z., Deng, S.-M., Tan, C.-Q., Huang, F., Si, L., & Chen, H.-J. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.14704 · Cited by 69
Summarization as Indirect Supervision for Relation Extraction
Lu, K., Hsu, I.-H., Zhou, W., Ma, M. D., & Chen, M. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.09837 · Cited by 69
“Specifically, Han et al. (2021) propose prompt tuning methods for RE by applying logic rules to construct hierarchical prompts.”
BackgroundMethodPrompt-Learning for Short Text Classification
Zhu, Y. Q., Wang, Y., Qiang, J., & Wu, X. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3332787 · Cited by 83 · see citing works
“Althoughautomaticprompt generationmethodscanavoidintensivelabor,mostofthese methodscannotachievecomparableperformancetomanually selectedpromptmethodsinthereal-worldscenario[42].”
MethodInfluentialDeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base PopulationPreprint
Zhang, N., Xu, X., Tao, L., Yu, H., Ye, H., Qiao, S., Xie, X., Chen, X., Li, Z., Li, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.03335 · Cited by 15 · see citing works
“We utilize a SOTA few-shot RE method, KnowPrompt (Chen et al., 2022b) which incorporates knowledge into prompt-tuning with synergistic optimization, to conduct 8, 16-, and 32-shot experiments compared with other baselines, such as GDPNet (Xue et al., 2021) and PTR (Han et al., 2021).”
MethodHarnessing domain insights: A prompt knowledge tuning method for aspect-based sentiment analysis
Sun, X., Zhang, K., Liu, Q., Bao, M., & Chen, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111975 · Cited by 63 · see citing works
Contrastive Conditional Latent Diffusion for Audio-Visual Segmentation
Mao, Y., Zhang, J., Xiang, M., Lv, Y., Li, D., Zhong, Y., & Dai, Y. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3580269 · Cited by 39 · see citing works
“In this case, e ff ective transfer learning with suitable model tuning strategies can be a promising research direction to improve the e ff ectiveness of our solution further, e.g. prompt tuning [71], [72], [73].”
A Survey of Knowledge Enhanced Pre-trained ModelsPreprint
Yang, J., Xiao, G., Shen, Y.-L., Jiang, W., Hu, X., Zhang, Y., & Peng, J. (2021) · arXiv.org · doi:10.48550/arxiv.2110.00269 · Cited by 64 · see citing works
“By integrating symbol reasoning system into the learning pipeline, knowledge form unchanged KEPLMs reconcile the advantages of effective perception of PLMs and reasoning and interpretability of symbolic representation [29].”
BackgroundMethodInfluentialPromptBoosting: Black-Box Text Classification with Ten Forward Passes
Hou, B.-R., O’Connor, J., Andreas, J., Chang, S.-Y., & Zhang, Y. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.09257 · Cited by 63
“Since verifying the automatically-generated prompts is time-consuming, the PTR (Han et al., 2021) method incorporates logic rules to construct prompts and to encode prior knowledge into prompt-based learning.”
MethodA Review on Language Models as Knowledge Bases
AlKhamissi, B., Li, M., Celikyilmaz, A., Diab, M., & Ghazvininejad, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06031 · Cited by 62 · see citing works
“A number of papers show that soft prompting, when optimized for specific tasks such as relation extraction and natural language understanding, can achieve better performance with minimal tuning (Lester et al., 2021; Han et al., 2021).”
BackgroundBlack Box Few-Shot Adaptation for Vision-Language models
Ouali, Y., Bulat, A., Matinez, B., & Tzimiropoulos, G. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01424 · Cited by 33 · see citing works
OpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H.-T., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2111.01998 · Cited by 60 · see citing works
“Pre-trained language models (PLMs) (Han et al., 2021a; Qiu et al., 2020) have been widely proven to be effective in natural language understanding and generation, ushering in a new era of modern natural language processing (NLP).”
BackgroundMethodInfluentialSelf-citationEfficient utilization of pre-trained models: A review of sentiment analysis via prompt learning
Bu, K., Liu, Y., & Ju, X. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111148 · Cited by 57 · see citing works
Huang, Q. Y., Yuan, Z., Xing, Z., Xu, X., Zhu, L., & Lu, Q. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556912 · Cited by 42 · see citing works
“Different from manually defined, uniform prompts in NLP tasks [16, 23, 27, 49, 72], our approach automatically extract code lines surrounding a focused code line with FQNs as its contextual FQN prompt.”
MethodPanDa: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation
Zhong, Q., Ding, L., Liu, J., Du, B., & Tao, D. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3376453 · Cited by 37 · see citing works
Tailor: A Soft-Prompt-Based Approach to Attribute-Based Controlled Text Generation
Yang, K., Liu, D., Lei, W., Yang, B., Xue, M., Chen, B., & Xie, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.25 · Cited by 53 · see citing works
“Inspired by the composition of discrete prompts (Han et al., 2021) to accomplish a complex task, our intuitive idea is to combine single-attribute prompts as a multi-attribute prompt to hint a PLM for multi-attribute CTG. Fixed GPT-2 Fixed GPT-2 in Figure 2 (bottom), Tailor introduces a…”
MethodGPS: Genetic Prompt Search for Efficient Few-Shot Learning
Xu, H., Chen, Y., Du, Y., Shao, N., Wang, Y.-G., Li, H., & Yang, Z.-L. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.17041 · Cited by 56
“Han et al. (2021) applied rules in prompt tuning to deal with the hard many-class text classification tasks.”
BackgroundMethodPADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains
Ben-David, E., Oved, N., & Reichart, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00468 · Cited by 54 · see citing works
Hierarchical Prompt Learning for Multi-Task Learning
Liu, Y., Lu, Y., Liu, H., An, Y., Xu, Z., Yao, Z., Zhang, B., Xiong, Z., & Gui, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01048 · Cited by 42 · see citing works
“Recently, automatic prompt generation [18, 26, 30, 33, 57, 78] has emerged as a promising way to adapt language models effectively.”
BackgroundOne for All: A Unified Generative Framework for Image Emotion Classification
Shi, G., Deng, S., Wang, B., Feng, C., Zhuang, Y., & Wang, X. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3341840 · Cited by 73 · see citing works
“Existing works have explored prefix tuning [62], [63], [64], initialized with discrete prompts [59], [65], [66], [67], and hybrid tuning using both hard and soft prompts [68], [69].”
BackgroundChatGPT-powered Conversational Drug Editing Using Retrieval and Domain FeedbackPreprint
Liu, S.-C., Wang, J.-X., Yang, Y., Wang, C., Liu, L., Guo, H., & Xiao, C. (2023) · arXiv.org · doi:10.48550/arxiv.2305.18090 · Cited by 51 · see citing works
“In natural language processing (NLP), prompt design or prompt engineering [39] has proven to be an effective paradigm for generalizing well-trained LLMs to various NLP downstream tasks, including but not limited to sentiment classification [40, 41], textual entailment [42, 43], text summarization [44, 45, 46].”
MethodRelation Extraction as Open-book Examination
Chen, X., Li, L., Zhang, N., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531746 · Cited by 33 · see citing works
“Typically, PTR [13] applies logic rules in prompt-tuning with several sub-prompts and encode prior knowledge.”
BackgroundMethodInfluentialSimilarity-Aware Multimodal Prompt Learning for fake news detection
Jiang, Y., Yu, X., Wang, Y., Xu, X., Song, X., & Maynard, D. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119446 · Cited by 58 · see citing works
Exploring Prompt-based Few-shot Learning for Grounded Dialog GenerationPreprint
Zheng, C., & Huang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06513 · Cited by 34 · see citing works
Sun, Y., Zheng, Y., Hao, C., & Qiu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03564 · Cited by 48 · see citing works
“PTR (Han et al., 2021) incorporates logic rules to compose task-specific prompts with several simple subprompts.”
BackgroundMethodA Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks
Wahba, Y., Madhavji, N., & Steinbacher, J. (2022) · International Conference on Machine Learning, Optimization, and Data Science · doi:10.48550/arxiv.2211.02563 · Cited by 48
“, classification) limits the ability to fully utilize the knowledge encoded in PLMs [7], (ii) the high computational cost and the large set of trainable parameters make these models impractical for training from scratch, (iii) dealing with rare words is a challenge for PLMs [8], and (iv) the performance of PLMs may no…”
BackgroundPrompt-Based Meta-Learning For Few-shot Text Classification
Zhang, H., Zhang, X., Huang, H., & Yu, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.87 · Cited by 40 · see citing works
“The gap between pre-train and downstream tasks often causes difficulty in fine-tuning of PLMs like GPT (Brown et al., 2020), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), etc. Inspired by GPT-3, which presents the strong potential of FSL by prompting, numerous work has applied prompt-tuning for Relation Clas…”
MethodFan, Z., & Chen, C. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103646 · Cited by 50 · see citing works
GAP: A novel Generative context-Aware Prompt-tuning method for relation extraction
Chen, Z., Li, Z., Zeng, Y., Zhang, C., & Ma, H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123478 · Cited by 43 · see citing works
Generative Multi-Modal Knowledge Retrieval with Large Language Models
Long, X.-W., Zeng, J.-L., Meng, F.-D., Ma, Z.-Y., Zhang, K., Zhou, B.-W., & Zhou, J. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2401.08206 · Cited by 47
“To mitigate these challenges, parameter-efficient fine-tuning methods (Han et al. 2022; Ding et al. 2023; Zhang et al. 2023a) have been proposed which reduce costs by updating only a subset of parameters.”
MethodConsistent Prototype Learning for Few-Shot Continual Relation Extraction
Chen, X., Wu, H., & Shi, X. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.409 · Cited by 47 · see citing works
“Specifically, we firstly enrich the sentence x into a specific input sequence { x, [SEP] } based on the prompt learning (Han et al., 2021b; Chen et al., 2022), and then encode the input sequence by the encoder to obtain the contextualized representation.”
BackgroundMethodLearning Visual Prompt for Gait Recognition
Ma, K., Fu, Y., Cao, C., Hou, S., Huang, Y., & Zheng, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00063 · Cited by 34 · see citing works
“It has gained considerable popularity [16, 19, 41] in the field of natu-ral language processing (NLP) due to its ability to integrate task-specific cues for each individual task.”
Prompt Tuning in Code Intelligence: An Experimental Evaluation
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R.-T. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3313881 · Cited by 44 · see citing works
“There is also a verbalizer [9], [13] that maps the tokens predicted by the model to a class.”
BackgroundInfluentialDebiased Fine-Tuning for Vision-language Models by Prompt Regularization
Zhu, B.-E., Niu, Y., Lee, S., Hur, M., & Zhang, H.-W. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2301.12429 · Cited by 45
“To this end, “prompt tuning” is pro-a posed to fine-tune the token embeddings in a prompt using the task data (Liu et al. 2021a,b; Han et al. 2021; Zhou et al. 2021).”
Pre-trained Language Models Can be Fully Zero-Shot Learners
Zhao, X., Ouyang, S., Yu, Z.-G., Wu, M.-L., & Li, L. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.06950 · Cited by 45
“Prompting has been applied to a large variety of tasks such as Text Classification (Schick and Schütze, 2021a), Natural Language Understanding (Xu et al., 2022), Knowledge Probing (Petroni et al., 2019), and Relation Extraction (Han et al., 2021).”
BackgroundMethodInfluentialPushing the Limits of ChatGPT on NLP Tasks
Sun, X., Dong, L.-F., Li, X., Wan, Z., Wang, S.-H., Zhang, T.-W., Li, J., Cheng, F., Lyu, L., Wu, F., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2306.09719 · Cited by 44
- Authors
- Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M.
- Affiliations
- Beijing National Research Center for Information Science and Technology (CN)
- Beijing Academy of Artificial Intelligence (CN)
- Corresponding author
- Liu, Zhiyuan
- Sun, Maosong
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Volume
- 3
- Pages
- 182-192
- ISSN
- 2666-6510 (ISSN-L)
- arXiv
- 2105.11259
- OpenAlex
- W4309811444
- Cited by
- 478
- References
- 54
- Citations by year
- 2026: 27 · 2025: 92 · 2024: 131 · 2023: 137 · 2022: 77 · 2021: 14
- FWCI
- 35.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- arxiv.org/licenses/nonexclusive-distrib/1.0/
- Locations
- AI Open (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62236004)
- Tsinghua University
- National Key Research and Development Program of China (2020AAA0106500)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- prompt tuning, text classification, pre-trained language models, relation classification, entity typing, intent classification, prior knowledge encoding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
