PTR: Prompt Tuning with Rules for Text Classification

“…al., 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al., 2020), to text classification (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021), fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks.”

“To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020; Schick et al., 2020).”

“, 2019), RoBERTa (Liu et al., 2019) and T5 (Raffel et al.”

“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”

“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”

“tasks such as natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”

“, 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al.”

“The hard prompt tuning applies a template with [MASK] token to the original input and maps the predicted label word to the corresponding class (Han et al., 2022; Shin et al., 2020).”

“Han et al. design the basic sub-prompts manually for text classification and apply logical rules to combine these sub-prompts into the final task-specific prompts [20].”

“We illustrate how we leverage the ESCO taxonomy to construct and populate sub-prompts, as proposed by Han et al. [10].”

“Several studies [16], [21], [22] have demonstrated that a prompt template with special learnable tokens can make prompt learning more effective.”

“The third works by prompt composition or decomposition [15, 6], which constructs multiple sub-prompts for better predictions.”

“Multi-prompt learning is a collection of these two engineering prompts aimed at improving model generality, and there are typically several methods of multi-template learning, such as prompt augmentation [12], prompt composition [13], and prompt decomposition [14].”

“[11] used heuristic rules to define several sub-prompt and formed the final pattern by combining several sub-templates.”

“A series of research works based on manual prompts have achieved promising performance on few-shot sentiment classification [56], fake news detection [27], and natural language inference [18].”

“Numerous tasks have been addressed through prompt-tuning, such as Text Classification [19], Natural Language Understanding [20] [25], Relation Extraction [26] [27], and Knowledge Probing [25] [28], etc.”

“To address this issue of verbalizer ambiguity, Han et al. (2021) proposed manually creating logic rules to combine multiple sub-prompts into a final prompt for each class.”

“[13] proposed prompt tuning with rules for many-class classification tasks.”

“This method has shown great results on few-shot and zero-shot learning tasks such as classification (Gao et al., 2021; Schick and Schütze, 2021; Han et al., 2021) and text generation (Schick and Schütze, 2020; Li and Liang, 2021).”

“Cross-linguistic adaptation is the method for adapting or tuning existing NLP models [45, 46] for good performance over a different portion of language usually those for low-resource languages.”

“…by finetuning a masked language model (Wu and He, 2019, Joshi et al., 2020, Yamada et al., 2020, Wang et al., 2021b, Lyu and Chen, 2021, Paolini et al., 2021, Wang et al., 2022, Li et al., 2023b) or prompting a masked language model (Han et al., 2022b, Han et al., 2022a, Zhang et al., 2023b).”

“Han et al. [23] proposed a rule-based approach to designing more efficient templates, which split the prompts in the template into several sub-prompts and combined them into a whole after constructing the sub-prompts.”

“These soft prompts are learnable vectors rather than pre-existing vocabulary items (Qin and Eisner, 2021; Zhong et al., 2021; Han et al., 2022).”

“Utilizing the powerful natural language understanding ability of MLM, cloze style prompt-tuning is commonly adopted in many tasks and has achieved convincing performance [5–8].”

“For low-resource IE, instruction prompting can be effective as it allows the model to execute tasks using its pre-existing knowledge and language understanding.”

“Prompt-based methods prepend random soft tokens to the input (typically as a prefix) and train their embeddings while keeping the backbone LLM frozen [31, 18, 73, 44], also introducing inference overhead.”

“The soft prompt (Liu et al. 2023; Tsimpoukelli et al. 2021; Han et al. 2022) is more flexible, with the generation of prompts being learned as a task itself.”

“To address these challenges, prompt tuning Han et al. [2022], Lester et al. [2021], Li and Liang [2021] has emerged as an efficient alternative, serving as a case study in achieving targeted model improvements with significantly reduced costs.”

“However, most of these methods cannot achieve comparable performance to manually selected prompt methods in the real-world scenario [56].”

“A prompt template is employed to encapsulate a original text into a new format featuring prompts and masks, as exemplified in the work of [27], [28], [29].”

“Previous works have shown that a trainable defined template can amplify the role of prompt guidance [23], [24].”

“Prompting methods are categorized into discrete prompts [36,11,8] and continuous prompts [29,21,14].”

“For instance, Han et al. [30] proposed a "PTR" prompt tuning method for handling complex multi-class tasks.”

“PTR and KnowPrompt have moderate computational complexity, while GRASP has high complexity due to the addition of a trigger prediction process during inference.”

“Later, prompt learning has been applied in many information extraction tasks, e.g., entity extraction (Cui et al., 2021; Liu et al., 2022a; Ding et al., 2021) and relation extraction (Han et al., 2022).”

“[16] defined various subprompts and proposed multiprompts for combining them.”

“One of the recommended [18] approaches is to create several simple sub-prompts and then combine them to make task-specific prompts according to the logic rules.”

“Recently, Prompt tuning (Gu et al. 2021; Liu et al. 2021; Wei et al. 2021; Han et al. 2022) aims to insert a set of prompt tokens into the input sequence of PLMs, thereby alleviating the difference between the data distributions of pre-training and downstream tasks (Liu et al.”

“Prompt-based fine-tuning (P ROMPT ) (Lester et al., 2021; Han et al., 2021) has recently been the most popular way to fine-tune a PLM.”

“Recently, (Gao et al., 2021; Han et al., 2021; Wei et al., 2021) found that the prompt-based models achieve remarkable few-shot performance, and reformulate the traditional paradigm of finetuning to prompt-tuning, which could better utilize the knowledge of the pre-trained models.”

“Prompt learning can show the potential to further improve performance by narrowing the disparity between language model pre-training and target task training [29-30].”

“Thanks to these advances, prompt-tuning has been widely applied to various tasks such as text classification [38], relation extraction [39], and question answer [40], demonstrating its versatility and effectiveness across different application scenarios.”

“The implementation of Prompt is becoming more diverse, with methods that freeze the entire parameters of PLMs [22], [23] and meth-ods combining fine-tuning with Prompt [11], [24].”

“The prompt technique [18, 20, 15] was originally employed in NLP tasks.”

“Han et al. (2022) propose prompt-tuning methods for RE by applying logic rules to construct hierarchical prompts.”

“As another option, classification tasks can benefit from the PTR (prompt tuning with rules) framework, which suggests encoding prior task knowledge into rules, decomposing into subtasks, and combining them to achieve the desired result (Han et al., 2022).”

“Following [16], we consider three different styles of prompting: hard , soft , and mixed prompting .”

“While it can achieve performance close to that of the pre-training and fine-tuning paradigm, it fails to surpass it [15,16] primarily due to the data-driven nature of deep learning (i.e., lack of training samples poses a significant challenge).”

“However, while auto-generated prompts can avoid intensive labor, they often do not perform as well as manually selected templates in the real-world scenario, and the computational expenses and time required to validate the efficacy of these approaches are huge [18].”

“Then, to reduce heuristic efforts, advanced approaches automatically search for prompts arranged from discrete tokens(Jiang et al., 2020; Shin et al., 2020), or learn vector-based continuous prompts(Han et al., 2022; Lester et al., 2021; Li & Liang, 2021; Zhong et al., 2021).”

“These prototypes are refined iteratively using clustering methods, such as K-means [13], to accurately capture the evolving distribution of relational semantics.”

“41 The verbalizer, which maps label words to categories, has been shown to play a vital role in prompt-tuning.”

“Additionally, to solve text classification, Han et al. [49] leveraged pre-defined logic rules to merge manually designed sub-prompts together as the final task-specific prompts.”

“It is used in many tasks of natural language processing, including classification [25,26], natural language understanding [27], and relation extraction [28].”

“@5 @10 TagDC [22] 0.429 0.311 0.189 0.483 0.570 0.678 0.434 0.387 0.288 PORFIT [31] 0.511 0.368 0.217 0.571 0.668 0.774 0.516 0.456 0.330 Post2Vec [42] 0.550 0.384 0.223 0.624 0.707 0.799 0.558 0.478 0.339 PTM4Tag [16] 0.488 0.339 0.199 0.557 0.628 0.720 0.496 0.423 0.303 CDR4Tag [23] evaluation…”

“Recently, prompt learning [26] [28] [29] was proposed, which aims to enhance the generalization of pre-trained language models to downstream tasks.”

“However, there exists certain gaps between the training and fine-tuning process of these pre-trained models, which cannot maximize the ability of the models [14–17].”

“…in order to evaluate the stability of the log parsing method in a small number of data sample environments, this paper sets up five samples of 4, 8, 16, 32, and 64 from the same labeled data, and the samples selected by completely random sampling are used as the training sets for the five sizes.”

“With the continuous development, some work reveals that prompt-based methods also exploit superficial cues and lead to the bias of language models toward predicting certain answers (Kavumba et al., 2022; Zhao et al., 2021) using the same prompt formats.”

“Following this, PTR [19] designed a set of sub-prompt templates which would be later combined by logical rules into specific task prompts, thus incorporating prior knowledge into the prompts.”

“Meanwhile, generating continuous prompts through neural networks for both text classification and generation tasks (Han et al., 2021; Li and Liang, 2021) have been proposed.”

“Shin et al. [16] adopted gradient search to automatically search for prompt templates and demonstrated through experiments that this approach has superior performance.”

“Han et al. [39] generated a set of rules called PTR, which decomposes a difficult task into simpler subtasks.”

“This approach ensures that pre-trained knowledge is fully lever-aged in the fine-tuning process (Gao et al., 2021a; Han et al., 2022; Liu et al., 2023), while also incorporating task-specific information through prompts (Schick and Schütze, 2021).”

“…et al., 2021; Lester et al., 2021; Zhong et al., 2021b; Wallace et al., 2021; Haviv et al., 2021; Jiang et al., 2020; Chen et al., 2022; Qin and Eisner, 2021; Liu et al., 2021; Han et al., 2021; Zhong et al., 2021a; Lu et al., 2022; Ben-David et al., 2022; Wang et al., 2022a; Zhou et al., 2023b).”

“…exploration into various aspects of prompt-based learning, including template engineering (such as Prefix-Tuning [8], and AutoPrompt [9]), answer engineering (such as PET and LMBFF), and multi-prompt learning (such as PTR [10] and TemplateNER [11]), leading to impressive advancements in NLP tasks.”

“[20] We propose a method that encodes prior knowledge in classification tasks into rules, then designs sub-cues according to the rules, and finally combines the sub-cues to handle the classification task.”

“However, even though fine-tuning PLMs have achieved promising performance, several studies [14], [15], [16] suggest that the major obstacle in PLM-based methods is the gap between objective forms used in pre-training and fine-tuning, preventing PLMs from taking full advantage of given information.”

“More recently, some works[31], [32], [33] have ventured into learning continuous prompts as an alternative to seeking discrete prompts.”

“For relation extraction, in order to solve the problem of manyto-one mapping when prompt learning is applied to relation extraction tasks, [45] proposes a model of combining several sub-prompts based on logical rules.”

“For example, [10] and [11] combines both discrete prompts and soft prompts, however, they fine-tune the soft prompts jointly with the entire pre-trained model; P-tuning v2 [12] further enhanced the universality of [10] on a broader selection of models and tasks at the cost of introducing soft prompts (more trainable p…”

“Recently, thanks to the success of pre-trained language models (PLMs), fine-tuning PLMs has become a mainstream approach for various text classification tasks (Han et al., 2021a).”

“Template length Different from manual templates (used by PTR) or manually initialized continuous templates (used by KnowPrompt), whose lengths are predefined and limited, EPPAC can work with randomly initialized templates of any length.”

“Existing studies have explored prompt construction from multiple perspectives, including manually designed discrete prompts [6], [26], continuous soft prompts represented by learnable vectors [6], [8], [11], [27], and automatically generated prompt templates based on search strategies [28].”

“These methods relied on expert-defined frameworks, where the relationships between influencing factors such as historical trends, economic indicators, and visitor behaviors were manually established [5].”

“Han et al. [18] defined a set of logical rules and used Prompts embedded with these rules as input to generate text related to specified categories as the output.”