PTR: Prompt Tuning with Rules for Text Classification

“…al., 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al., 2020), to text classification (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021), fine-tuned PLMs have been demonstrated their effectiveness on almost all important NLP tasks.”

“To alleviate this issue, prompt tuning has been proposed to bridge the gap of objective forms in pre-training and fine-tuning (Brown et al., 2020; Schick et al., 2020).”

“, 2019), RoBERTa (Liu et al., 2019) and T5 (Raffel et al.”

“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”

“Recently, some continuous prompts have also been proposed (Li and Liang, 2021; Lester et al., 2021), which directly use a series of learnable continuous embeddings as prompt templates and get rid of the trouble of designing prompts.”

“tasks such as natural language inference (Schick and Schütze, 2021; Liu et al., 2021).”

“, 2020), summarization (Zhang et al., 2019; Liu and Lapata, 2019), question answering (Adiwardana et al.”

“The evolution of prompts ranges from mere textual prompt, CoT prompt with more logic descriptions to more structured prompts such as Program of Thoughts [21], Tree of Thoughts [34], and Graph of Thoughts [35], and even directly using logical rules as prompts [36].”

“Ptr (Han et al., 2021) adopts manually crafted sub-templates and generates complete templates by logic rules.”

“Several approaches (Li and Liang, 2021; Liu et al., 2021; Han et al., 2021) have been proposed to search continuous prompts.”

“TANL, NLI, PTR, and Know-Prompt exploit label semantics.”

“Other works (Shin et al., 2020; Han et al., 2021b; Scao and Rush, 2021; Utama et al., 2021) mine prompts from the training corpus based on heuristic rules/semantic relations.”

“…tasks and adding some prompt tokens into the input, including human-designed explainable prompts (Brown et al., 2020; Schick and Schütze, 2021a,b; Han et al., 2021b; Ding et al., 2021; Hu et al., 2021) and automatically searched prompts (Jiang et al., 2020; Shin et al., 2020; Gao et al., 2021).”

“Research has shown that prompt tuning can perform well in various NLP tasks [8–11] and their few-shot scenarios [12, 13].”

“Following this idea, semantic typing tasks have been reformulated as prompt-based learning (Ding et al., 2021; Han et al., 2021), natural language inference (Yin et al., 2019; Sainz et al., 2021), question answering (Levy et al., 2017; Li et al., 2019; Du and Cardie, 2020), and translation (Paolini…”

“Prompt-tuning [8],[9],[10]isarepresentativeandpowerful method that has remarkable superiority over the classical fine-tuning paradigm, especially in zero-shot and few-shot scenarios.”

“Specifically, HCRP and MapRE also use textual label in their model, but their models lead to sub-optimal performance, proving the effectiveness of dropping out textual labels during training.”

“The scores from dodrans and entropy indicate that only if all classes are considered equally important the PTR model should be preferred.”

“This approach has been shown to be effective in various NLP tasks, including text classification (Han et al., 2021; Wang et al., 2022) and question answering (Yao et al., 2022).”

“To achieve it, PTR (Han et al., 2021) introduces manual sub-prompts for entity recognition and relation classification, respectively.”

“With the rise of LLMs, prompting-based methods [71, 245, 260, 313] achieved notable results by leveraging LLMs’ internal knowledge.”

“…NLP tasks, such as text classification (Schick and Schütze, 2021a; Hu et al., 2021), text matching (Jiang et al., 2022), named entity recognition (Cui et al., 2021; Chen et al., 2021), and relation extraction (Han et al., 2021; Chen et al., 2022a), particularly on few-shot and zero-shot settings.”

“Han et al. [31] researched strategies for designing prompts for classification tasks.”

“There are also recent works that improve prompt-tuning based on heuristic rules (Han et al., 2021), automatic prompt construction (Gao et al., 2021a; Shin et al., 2020) and continuous prompt learning (Liu et al., 2021b,a; Hu et al., 2021; Gu et al., 2021).”

“Moreover, PTR (Han et al. 2021) applies logic rules to build templates that are more suitable for text classification tasks.”

“Most prompt-based methods first construct semantic templates as prompts to obtain masked word predictions from a pretrained language model (PLM), then map these predictions into task-specific labels [12], [13].”

“It makes the model difficult to fully utilize the knowledge of pre-trained models, resulting in suboptimal performance on downstream tasks (Lester et al., 2021; Gu et al., 2022; Han et al., 2022).”

“To address this data scarcity, few-and zero-shot RE works try to leverage knowledge from LM pre-training or auxiliary sources (Petroni et al., 2019; Chia et al., 2022; Han et al., 2022; Chen et al., 2022a; Zhao et al., 2023; Zhou et al., 2023a; Wan et al., 2023; Li et al., 2023b; Sun et al., 2024).”

“For low-resource IE, instruction prompting can be effective as it allows the model to execute tasks using its pre-existing knowledge and language understanding.”

“Lastly, as prompting methods have been widely adopted to other tasks such as natural language inference (Gao et al., 2021a) and relation extraction (Han et al., 2021), it is possible to extend our method to these tasks.”

“(2022), which leverage prompt tuning with rules (Han et al., 2022) to learn optimal discrete tokens to fill in [MASK] within the template such as “@CHEMICAL$ [MASK] [MASK] [MASK] @GENE$.”

“Prompt-ing has been explored in various tasks, including text classification (Hu et al., 2021; Han et al., 2021; Schick and Schütze, 2021a,b), information extraction (Lester et al., 2021; Chen et al., 2021) and text generation (Dou et al., 2021; Li and Liang, 2021).”

“Prompt-tuning-based approaches construct well-curated prompts to distill the hidden relational knowledge from large language models (LLMs) [7, 8, 20].”

“In NLP, prompting on text classification tasks has also achieved remarkable results [13, 20, 22, 30].”

“, 2021), which prepends a sequence of task-specific vectors to the input, while keeping LLM parameters frozen; and hard-soft prompt hybrid tuning (Liu et al., 2021; Han et al., 2022), which inserts tunable embeddings into a hard prompt template.”

“Meanwhile, due to the continuity of parameters in neural networks, continuous prompts for both text classification and generation tasks (Li and Liang, 2021; Liu et al., 2021b; Han et al., 2021) have been proposed.”

“This training imparts a certain level of code knowledge to CodeBERT, and further fine-tuning with downstream task (Schick and Schütze 2021; Li and Liang 2021; Lester et al. 2021; Gu et al. 2022; Han et al. 2022) corpora enhances its understanding of specific tasks.”

“For relation extraction, PTR employs logic rules to enhance prompts with several sub-prompts [12].”

“While most of current works about prompt-tuning is in text classification tasks, some works have applied it to information extraction tasks [Cui et al., 2021; Han et al., 2021; Si et al., 2021] and text generation tasks [Li and Liang, 2021].”

“Prompt Tuning (Schick and Schütze, 2021; Liu et al., 2023a) (Shin et al., 2020; Gao et al., 2021a; Han et al., 2022) directly concatenate explicit discrete tokens with the original text and maintain them unchanged throughout entire training process, which do not introduce any parameters.”

“Recently, a series of studies propose to use prompt tuning (Han et al., 2021; Schick and Schütze, 2021a; Chen et al., 2022b) to bridge the gap between pre-training and fine-tuning.”

“[11] introduces a more efficient method to construct prompts with several sub-prompts that employs prompt tuning with rules without searching.”

“The strength of prompting has been validated in a wide range of NLP applications, including text classification[37],[38],[39],[40],[41],machinetranslation[42] andrelationextraction[43],[44].”

“…makes better use of prior knowledge contained in pretrained language models, this new paradigm is beginning to become popular in NLP tasks and has achieved promising performance (Seoh et al., 2021; Han et al., 2021; Cui et al., 2021; Hou et al., 2022; Hu et al., 2022; Chen et al., 2022).”

“The other baseline models, which use human mobility data, are typically faster in embedding learning 00, as human mobility data provides critical insights into regional interactions and the functional relationships between regions, enabling the model to converge quickly.”

“Drawing inspiration from the notable success of prompt learning within the field of multi-task fine-tuning in natural language processing [11, 24, 51], we design tailored prompts for each modality-distortion conditions instead of training individual model with a myriad of parameters.”

“Inspired by these observations, prompt-based tuning has been widely developed for many text applications [4, 5, 19] and has generally surpassed vanilla fine-tuning in low-data regimes.”

“initializing with discrete prompts [36], [37], [38] and hybrid prompt tuning [39], [40], [41].”

“Liu, Han, et al. [29], [30] use hybrid templates to achieve promising performance on multiple tasks.”

“People have been using leverage prompts and giant models to solve IE tasks like named entity recognition (Cui et al., 2021), semantic parsing (Shin et al., 2021), and relations extraction (Chen et al., 2022; Han et al., 2021) in a zero-shot or few-shot way.”

“(Label Words) is inhibitor of Following the simple template proposed by (Han et al., 2021), we reformulate each example by appending to it a sentence containing its two entities, with masked tokens between them.”

“However, most of the studies focus on supervised or few-shot learning on text classification [12, 16, 18, 20], event detection [30], relation classification [6, 18] and name entity recognition [21, 29, 34].”

“…emerged (Li and Liang, 2021; Liu et al., 2021b) to improve the process of prompting, with methods such as prefix-tuning (Li and Liang, 2021), P-tuning (Liu et al., 2021b), prompt-tuning with rules (Han et al., 2021), knowledgeable prompt tuning (Hu et al., 2021) and many more (Liu et al., 2021a).”

“, 2021), text classification (Gao et al., 2021; Han et al., 2021; Chen et al., 2021; Chai et al., 2020), natural language inference (Shin et al.”

“Prompt tunings (Han et al. 2022; Jin et al. 2021; Zang et al. 2022) is an efficient, low-cost way of adapting a pre-trained foundation model to new downstream tasks.”

“Later, task-specific prompts are used to guide PLM fine-tuning and perform well in a low-resource setting for several tasks, such as text classification (Han et al., 2021; Hu et al., 2022), relation extraction (Chen et al., 2022), and entity typing (Ding et al., 2021; Huang et al., 2022).”

“PTR (Han et al. 2021) and KnowPrompt (Chen et al. 2022) use different prompt and answer word engineering for RE.”

“Depending on resource scenarios, it can be mainly divided into two kinds of studies: full prompt tuning (FPT) and parameter-efficient ones (PEPT).”

“Other methods include prompt tuning [22-24], where prompts are fine-tuned on a specific task, and prompt programming [25], where prompts are generated using a programming language.”

“Moreover, PTR (Han et al., 2021) and KnowPrompt (Chen et al., 2021) are the prompt-based fine-tuning baseline models.”

“, 2021a), PTR (Han et al., 2021), and many other related works (Li and Liang, 2021b; Qin and Eisner, 2021).”

“More-over, to build the connection between entity type and relation type, we add an auxiliary loss for predicting the type of object entity (Zhou and Chen, 2021; Han et al., 2021).”

“, 2020b; Qin and Eisner, 2021), compositing (Han et al., 2021), or decoupling (Ozturkler et al.”

“…prompt learning uses soft prompts , which are embedded in a continuous space and fine-tuned using training samples [21], and (3) hybrid prompt learning combines both hard and soft prompts to enable pre-trained LM to perform tasks by inserting tunable embeddings into hard prompts [20, 25, 27].”

“Composition is more suitable for composable tasks such as relation extraction [38].”

“…divided into two classes: discrete prompts (Gao et al., 2020; Shin et al., 2020; Ben-David et al., 2022) which is a sequence of discrete tokens and continuous prompts (Zhong et al., 2021; Han et al., 2022; Qin & Eisner, 2021) which is a sequence of continuous vectors followed by input text vectors.”

“We follow Chen et al. (2022) and Han et al. (2021), and evaluate on 8, 16 and 32 shots.”

“More recently, a new methodology named prompt learning has made waves in the community by demonstrating astounding few-shot capabilities on LRE (Han et al., 2021; Chen et al., 2022d).”

“…Gao et al., 2021; Ben-David et al., 2022; Su et al., 2022a; Diao et al., 2022; Datta et al., 2023; Pan et al., 2024) and continuous prompts (Hambardzumyan et al., 2021; Zhong et al., 2021; Han et al., 2021; Li and Liang, 2021; Qin and Eisner, 2021; Liu et al., 2021a) based on the format of prompts.”

“3) hybrid prompt (Han et al., 2022) combines untunable hard prompts and tuneable soft prompts, allowing the models to easily learn suitable templates with small manual intervention.”

“However, LLMs are quite sensitive to prompts that are fed to them (Loya et al., 2023; Sclar et al., 2023), and therefore, designing appropriate prompts with the right amount of detail has become more important than ever (Liu et al., 2023a; Han et al., 2022).”

“In a word, PT can encode task-specific knowledge [19] 1 3 Lifelong iris presentation attack detection without… and make better use of frozen pre-trained models [20].”

“Additionally, common methods [7,14,26] in multimodal FND, such as cascading, addition, and specially designed neural networks, primarily facilitate the fusion of surface features but may lose deeper multimodal information.”

“Prompt tuning has been advanced for different goals in language models [19, 20, 38], while visual prompting has been explored for zero-shot recognition tasks [66, 67].”

“To further equip the models with similar compositional ability, recent works also combine different prompts to accomplish multi-task or multi-attribute controls (Liu et al., 2021; Han et al., 2021; Yang et al., 2022).”

“The construction of templates is a key factor that affects the performance of prompt learning methods [34], [35].”

“PTR (Han et al., 2021) manually designs some essential sub-prompts and composes them into final prompts by applying logic rules.”

“While there are extensive works devoted to explaining language models in the context of text classification tasks (Shi et al., 2022; Han et al., 2021; Shi et al., 2023), relatively few attempts (Zhao and Shan, 2024; Vafa et al., 2021) are proposed to investigate the importance of prompt texts on…”

“The soft prompt [31], [33], [34] is different from the hard prompt in that the tokens of the prompt are represented as continuous vectors instead of fixed discrete words.”

“proposed a text classification framework named “PTR” [11],”