PTR: Prompt Tuning with Rules for Text Classification

“…al., 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al., 2020), to text classification (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021), fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks.”

“To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020; Schick et al., 2020).”

“, 2019), RoBERTa (Liu et al., 2019) and T5 (Raffel et al.”

“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”

“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”

“tasks such as natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”

“, 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al.”

“The disparity between pre-training tasks and downstream applications is believed by researchers to hinder the optimal utilization of PLMs’ knowledge (Han et al., 2021).”

“Later, task-specific prompts are used to guide PLM fine-tuning and perform well in a low-resource setting for several tasks, such as text classification (Han et al., 2021; Hu et al., 2022), relation extraction (Chen et al., 2022), and entity typing (Ding et al., 2021; Huang et al., 2022).”

“…automatically searching for a prompt or prompt-embedding via optimization) has emerged, with methods such as prefix-tuning (Li & Liang, 2021), P-tuning (Liu et al., 2021b), prompt-tuning with rules (Han et al., 2021), knowledgeable prompt tuning (Hu et al., 2021) and many more (Liu et al., 2021a).”

“Han et al. [4] applied rule-guided quick modification to increase decision boundaries but Wen and Fang [5] applied graph-grounded pre-training to increase the performance of low-resource classification.”

“PTR [88] encodes prior knowledge through logical rules for prompt tuning in text classification.”

“Han et al. (2022) demonstrated that hybrid approaches combining prompting with explicit rules can improve classification performance by incorporating domain knowledge.”

“In RE, both discrete and continuous prompts have been used to incorporate task instructions and relation semantics into the input representation [ 24 , 25 ].”

“PTR [39] uses sub-prompts to form prompts for specific tasks according to logical rules, and encodes prior knowledge into prompt templates.”

“Specifically, hard templates introduce prior knowledge through manually designed discrete natural language [22], while soft templates use learnable continuous vectors to adaptively capture task-related features [28], [29], [30], [31].”

“Recent approaches [5,23,24,39,41,47] have aimed to address this issue by learning a “soft prompt,” i.e., a continuous vector, through backpropagation while freezing the backbone model’s parameters.”

“Additionally, Han et al. propose the PTR model [12] developed a set of sub-prompts and further formulated logical rules to integrate these sub-prompts into task-specific prompts, with prior knowledge embedded into the template design.”

“…via episodic training [6], [7], yet struggle with multimodal alignment complexity; (2) Prompt-based tuning leveraging PLMs for few-shot adaptation [19], [20], but underutilize visual modalities; (3) Contrastive learning learning discriminative representations through hard negative mining [14],…”

“There also exist works [13, 22] that use a hybrid prompt tuning method that inserts trainable parameters into hard templates instead of using a purely learnable soft template.”

“Prompt-based approaches [21,27] have demonstrated strong performance in supervised settings but are rarely adapted to zero-shot scenarios, and most do not leverage relation descriptions to initialize or guide prompt representations.”

“Representative PEFT approaches include Adapter [24] , Prefix-Tuning [25] , and Prompt-based Tuning [26] .”

“To better highlight entities, several methods have introduced auxiliary entity-aware features, including entity markers [22, 23], external descriptions [24], and entity masking strategies [25].”

“Han et al. (2022) applied logic rules to construct prompts with several sub-prompts for relation classification, consistently outperforming existing state-of-the-art baselines without introducing additional model layers, manual annotations, or augmented data.”

“PTR [19]: is a prompt learning method suitable for small sample relation extraction.”

“Han et al. [22] proposed PTR, which leverages logical rules to construct structured prompts that better represent relational semantics, while Chen et al. [21] developed KnowPrompt, which jointly optimizes virtual type and answer words under knowledge constraints to enhance entity-type awareness.”

“Han et al. [7] introduced the Pattern-based Text Representation (PTR) framework, which uses rule-based sub-prompts to activate PLMs’ internal knowledge for complex intent classification.”

“The importance of prompt engineering has been highlighted in numerous prior studies [15]–[17].”

“For the relation extraction task, Han et al. [35] introduce PTR, which employs a rule-based prompt-tuning method for relation classification.”

“For example, Liu et al. studied how optimizing prompt keywords affects the output quality of image generation models [15], while Han et al. explored strategies for designing efficient prompts for classification tasks [16].”

“Moreover, prompt tuning uses the verbalizer V (an injective function) [17, 18] to obtain the final expected class.”

“The RE community has recently witnessed promising efforts to incorporate prompt-tuning [10, 11, 12], a new widespread paradigm for leveraging PLMs’ knowledge [13].”

“For example, Han et al. [22] use logic rules to generate prompts with several sub-prompts on text clas-sification tasks, which consistently outperforms existing state-of-the-art baselines without introducing any additional model parameters.”

“Among these approaches, Han et al. (2021) pro-pose a PTR model to creatively apply logic rules and decompose the prompt into several sub-prompts.”

“This paradigm mainly adds the template and verbal-izes to achieve the effect of stimulating PLM capabilities as much as [20] used heuristic rules to define several sub-prompt and formed the final pattern by combining several sub-templates.”

“However because fine-tuning state-of-the-art LLMs became more cost-intensive and access to LLMs shifted to APIs, researchers started to explore prompt-based approaches of synthetic data creation (Han et al., 2022), which rely on few-shot learning capabilities of LLMs (Brown et al., 2020).”

“…experience of constructing background data sets in papers, projects, patents, and standards, and determined to use nearly 200,000 current national standard bibliographic data of China, the United States, Britain, Germany, France, and Japan as the background data set of this analysis report [15].”

“To reduce the dependency on labeled data, many studies have focused on how to activate knowledge in PLMs. Inspired † denotes the first author (co-first authors) * denotes the corresponding author by GPT-3 [8], a series of prompt-based methods [9] [10] [11] have been proposed.”

“For RE, we use R-BERT [16] in rich-resource scenario and PTR [17] in few-shot scenario.”

“A relation extraction model called PTR( [25]) was proposed, which employs logic rules to create prompts composed of multiple sub-prompts [26], [27].”

“Recently, prompts methods have been proposed to address this problem by designing specific input templates for the specific downstream tasks [4,25,45,52, 54,79,91].”

“Other works [38], [39] mine prompts from the training corpus based on heuristic rules/semantic relations.”

“In the domain of information extraction, the exploration of fine-tuning pre-trained language models for supervised relation extraction has yielded promising outcomes in recognizing a diverse range of relations[11, 12].”

“Inspired by recent progress in PLMs and prompt tuning [4], [5] in natural language processing, we propose a new paradigm named Span-based Prompt tuning (SPTNet) for video grounding.”

“There is also a class of approaches that obtain continuous prompts in the embedding space of the model, and some work on vectorial adjustment [30], [31] with discrete prompts [32], [33] and mixed prompts [34], [35].”

“To alleviate the differences in the structured and unstructured knowledge injection process, we unify all forms of knowledge into the same semantic space by using soft prompts (Lester et al., 2021; Han et al., 2021).”

“Prompt-based tuning (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021b; Huang et al., 2022; Gu et al., 2021; Liu et al., 2021a; Han et al., 2021; Hou et al., 2022; Vu et al., 2021; Liu et al., 2022) is an innovative method to use the power of PLMs efficiently.”

“Specifically, we do Prompt+LM finetuning (Liu et al., 2021a) in which both the prompt and model parameters are trainable (Ben-David et al., 2021; Liu et al., 2021b; Han et al., 2021).”

“Although deep learning techniques have led to substantialbreakthroughsinRE,thefieldfacesthechallengethat by incorporating manually constructed templates [38], [39] or explanatory definitions [36], [37].”

“Discriminative methods like SpanBERT (Joshi et al., 2020), LUKE (Yamada et al., 2020), and IRE (Zhou and Chen, 2022) learn relation representations for classification, while prompt-based approaches like PTR (Han et al., 2022b), KnowPrompt (Chen et al., 2022a), and GenPT (Han et al., 2022a) design ver-balizers for each…”

“It has diverse applications, including knowledge graph construction [5,6], information retrieval [7,8], text summarization generation [3], sentiment classification [9,10], question answering [11], and more.”

“For example, a study by Gao et al. [29], Wang et al. [30], and Zhu et al. [31] successfully applied prompt tuning in text classification, while Han et al. [32] and Chen et al. [15] focused on relation extraction.”

“The other baseline models, which use human mobility data, are typically faster in embedding learning 00, as human mobility data provides critical insights into regional interactions and the functional relationships between regions, enabling the model to converge quickly.”

“…a shift from objective engineering to prompt engineering: this includes both the manual design of templates (Petroni et al., 2019) and automatic prompt learning (Jiang et al., 2020), as well as various options to ensemble (Schick and Schütze, 2021b) and compose (Han et al., 2022) multiple prompts.”

“Han et al. (2022) proposes a prompt tuning method with rules "PTR", which encodes the prior knowledge of a classification task into a rule.”

“In prompt-tuning, downstream tasks are formalized as language modeling problems by inserting language prompts, and the results of language modeling are heuristically mapped to the solutions of downstream tasks (Schick et al., 2020; Han et al., 2022).”

“In order to construct a task-specific prompt, [5] apply some logical rules in the sub-prompts construction.”