Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“We begin by experimenting with EMMET for model editing with varied batch sizes on GPT2-XL, GPT-J and Llama-2-7b on the CounterFact and zsRE (Levy et al., 2017) datasets.”

“These models have been shown to perform well in a zero-shot setting, i.e., for a previously unseen relation type (Levy et al., 2017), and for extracting entities that belong to non-standard types (Roth et al., 2018).”

“Surfacing knowledge from large corpora is a crucial step when dealing with knowledge intensive language tasks (Levy et al., 2017; Dinan et al., 2019; Elsahar et al., 2018; Petroni et al., 2021), such as open-domain question answering (Voorhees et al., 1999; Joshi et al., 2017; Yang et al., 2018;…”

“Specifically, these datasets can be divided into two groups: textual-output datasets (i.e., generation tasks) and categorical-output datasets (i.e., classification tasks).”

“Levy et al. (2017) leverage the MRC framework to settle the zero-shot relation extraction.”

“Most probabilities lie on two and three hops rather than zero and one hop, so key-value extraction based text generative models (Eric et al., 2017; Levy et al., 2017; Qian et al., 2018) are not suitable for this task.”

“5) We first test MEMIT on zsRE (Levy et al., 2017), a question-answering task from which we extract 10,000 real-world facts; zsRE tests MEMIT’s ability to add correct information.”

“87 RelationExtraction (Levy et al., 2017) 79.”

“Motivated by previous works handling information extraction via machine reading comprehension (MRC) (Levy et al., 2017; Li et al., 2020; Du and Cardie, 2020b; Liu et al., 2020), we note implicit EAE may be more akin to MRC, as both of them are document-level tasks.”

“Further efforts pay attention to learning relations from the text in various specific scenarios, such as zero-few-shot scenarios (Levy et al., 2017; Han et al., 2018b; Gao et al., 2019; Soares et al., 2019; Ye & Ling, 2019) and open-domain scenarios (Banko et al., 2007; Fader et al., 2011; Mausam et…”

“Inspired by the current trend of formalizing NLP problems as question answering tasks (Levy et al., 2017; McCann et al., 2018; Li et al., 2019a,b; Gardner et al., 2019; Raffel et al., 2019), we pro-pose a new framework for text classification by formalizing it as a SQuAD-style machine reading…”

“, 2020) improved story generation by training a Language Model with knowledge triples converted into sentences using predefined templates (Levy et al., 2017).”

“, 2017), RelationExtraction (RE) (Levy et al., 2017), and TextbookQA (TQ) (Kembhavi et al.”

“We note that the one exception is the Zero-shot RE task (Levy et al., 2017), a trivial task in which the query will always contain the title of the page to be retrieved.”

“(2) zsRE includes 151,631 questions based on relational knowledge from Wikipedia, which we randomly shuffle into train/dev/test splits with 80/10/10% of the data (Levy et al., 2017).”

“However, due to the absence of out-of-scope instances in datasets like ZsRE and MQUAKE, we employ GPT-4 to generate corresponding queries and answers based on the edit descriptors, further details of which are provided in Appendix A.1.”

“In (Levy et al. 2017), sets of crowdsourced paraphrastic questions are written for each relation type in the ontology.”

“Levy et al. (2017) show that RE can be addressed as answering reading comprehension questions and improved by the training process of a machine reading comprehensive task.”

“We then perform large scale editing of GPT2-XL (Radford (Levy et al., 2017).”

“Interestingly, we find that we can fit the reader just relation type along with its first argument in KB completion (Levy et al., 2017; Li et al., 2019), an entity category in (nested) NER (Li et al., 2020), an auxiliary verb or a wh -expression in ellipsis resolution (Aralikatte et al., 2021), and…”

“[107] used the BiDAF model [109] with an additional NIL node in the output layer for this task on the WikiReading [42] dataset with additional negative samples.”

“Low resource Relation Extraction methods gained a lot of attention recently (Levy et al., 2017; Tarvainen and Valpola, 2017; Lin et al., 2019; Li and Qian, 2020; Hu et al., 2021, 2020), since these methods require fewer labeled data and deep neural networks could expand limited labeled information…”

“CounterFact and ZsRE Meng et al. (2022) processed P ARA R EL relations differently from Dai et al. (2022).”

“To that end, several benchmarks have been proposed to measure consistency of LMs (Ravichander et al., 2020; Elazar et al., 2021), while other datasets have been adapted for that purpose (Levy et al., 2017; Wang et al., 2021c).”

“We use the Natural Questions (NQ) [19], TriviaQA [15], HotpotQA [50], FEVER [45], T-REx [7], zsRE [20], and Wizard of Wikipedia (WoW) [6] datasets from the KILT [29] benchmark.”

“…et al., 2021), HOVER (Jiang et al., 2020), FEVEROUS (Aly et al., 2021) of the Fact Checking task, the T-REx (Elsahar et al., 2018), zsRE (Levy et al., 2017) and LAMA (Petroni et al., 2019a) to examine factual knowledge contained in pretrained language models, the WikiBio (Lebret et al.,…”

“…of Dialog (Wei et al., 2018; Cho and May, 2020; Aliannejadi et al., 2021), IE (Mausam et al., 2012; Zlabinger et al., 2020; Radev et al., 2020), QA (Levy et al., 2017; Zhang et al., 2018b; Min et al., 2020), Sum (Henderson et al., 2014; Syed et al., 2020; Hasan et al., 2021) and SA (Sheng and…”

“This model transfer was successfully demonstrated by Levy et al. (2017), who took a relation extraction task and used templates to pose it as QA in order to transfer a SQuAD-trained model to the task.”

“For the problem of zero-shot extraction of relations, Rockt¨aschel et al. (2015); De-meester et al. (2016) proposed the use of logic rules, Levy et al. (2017) proposed to address the problem by formulating it as a reading comprehension challenge, while Obamuyide and Vlachos (2018) proposed to…”

“26 2,284,168 5,000 5,000 zsRE [27] Slot Filling 1 147,909 3,724 4,966 NQ [24] Open Domain QA 1.”

“The current methods for editing knowledge mainly aim at incorporating triple-fact knowledge, which concentrate on question-answer (QA) tasks,i.e., ZsRE (Levy et al., 2017).”

“…to evaluate the compression models for various abilities: ArxivQA and SQuAD (Rajpurkar et al. 2016) for information extraction, RelationExtraction (Levy et al. 2017) for relation extraction, HotpotQA (Yang et al. 2018) for multi-hop reasoning, and RACE (Lai et al. 2017) for reading comprehension.”

“" T5 could predict new categories of data practices due to its language understanding capabilities that enable few and zero-shot inference [22, 50, 53].”

“However, the cumulative addition of extra parameters compromises the original inference process, evidenced by the pronounced deterioration in locality observed in ZSRE.”

“As [17], they experiment their methodology only with NLP tasks using FEVER [24] and zsRE [22] datasets.”

“For BERT-style models, fact-checking dataset FEVER (Thorne et al., 2018) and question-answer dataset zsRE (Levy et al., 2017) are usally adopted.”

“As LLMs become increasingly integrated into various applications, ranging from information retrieval to content generation, the accuracy of their outputs becomes paramount.”

“Machine Reading Comprehension (MRC) The aim of MRC models (Levy et al., 2017; Li et al., 2020, 2019; McCann et al., 2018; Shen, Huang, Gao, & Chen, 2017) is to extract answer spans from a context through a given query. The task can be formulated as two classification problems: predicting the start and end positions of…”

“Other semantic tasks such as Named Entity Recognition, Relation Extraction and Semantic Role Labelling have also been reformulated as entailment by automatically converting data into the entailment format (White et al., 2017; Poliak et al., 2018a; Levy et al., 2017; Sainz et al., 2021).”

“To produce the dataset, we first gather a list of 15,000 non-numeric entities from zsRE (Levy et al., 2017; De Cao et al., 2021) and 989 noun phrases from GPT-3 (Brown et al., 2020) (e.g., GHOST HUNTING ) for a total of 15,989 topics.”

“…experiments on 7 KILT tasks: HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), Natural Questions (NQ) (Kwiatkowski et al., 2019), T-REx (Elsahar et al., 2018), Zero Shot RE (zsRE) (Levy et al., 2017), FEVER (Thorne et al., 2018), and Wizard of Wikipedia (WoW) (Dinan et al., 2018).”

“Experimental Result We first focus on the performance on zero-shot Relation Extraction (zsRE), which is measured on the dev set of 12 relations that are ensured to have no overlap with the 84 relations of the train set (Levy et al., 2017).”

“In detail, they typically derive structured facts as knowledge updates from meticulously crafted datasets (Vrandeˇci´c and Krötzsch, 2014; Levy et al., 2017).”

“In sentence-level synthetic data, there exists merely one relation triplet within a sentence.”

“…relation-specific patterns or generic hyper-patterns (i.e., templates for patterns) for cloze questions (or, equivalently, basic Who/Where/When questions), relation extraction can be cast into masked-word predictions over an LM, leveraging neural learning for machine reading comprehension [15].”

“One approach is to directly extend previous MRC models by introducing a no-answer score to the score vector of the answer span (Levy et al. 2017; Clark and Gardner 2017).”

“We construct the benchmark of modified facts from two datasets, T-REx (Elsahar et al., 2018) and Zero-shot Relation Extraction (zsRE) (Levy et al., 2017).”

“We evaluate knowledge items from two widely used datasets: Counterfact (Meng et al., 2022a) and ZsRE (Levy et al., 2017).”

“…problems, where the query itself has high overlap with the relevant passage and there is a unique answer, for example factoid Open-domain QA such as Natural Questions (Kwiatkowski et al., 2019), HotPotQA (Yang et al., 2018), fact-checking (Thorne et al., 2018), and slot-filling (Levy et al., 2017).”

“Despite the greater naturalness and diversity than past RE solutions [62], [284], their question generation module was based on templates obtained via context-independent rule-based strategies.”

“Zero-shot relation classification (ZSRC) was first proposed by (Levy et al., 2017), which is able to extract new relations by reducing relation classification to answering simple reading comprehension questions.”

“Other zero-shot problems studied in NLP involve text classification (Yin et al., 2019), entity typing (Zhou et al., 2018), word sense disambiguation (Kumar et al., 2019) and relation extraction (Levy et al., 2017).”

“…weights in a range of layers; MeLLo , which manages multi-hop knowledge editing by decomposing subproblems and detecting conflicts; PMET , which optimizes FFN hidden states for precise weight updates, achieving SOTA performance in COUNTERFACT (Meng et al., 2022a) and ZsRE (Levy et al., 2017).”

“The ZsRE (Levy et al., 2017) and CounterFact (Meng et al., 2022) datasets are primarily utilized to assess whether edited models can recall the newly injected facts, as depicted in Figure 1(a).”

“Map Relations to Natural Language (NL) To convert ConceptNet triples into NL, we follow a common practice and map every relation r in the triple to its NL template, and fill in s and o in the template (Levy et al., 2017).”

“Tests were conducted on 3 LLMs using ZsRE dataset under 20 × 100 configuration while keeping all other variables constant (such as the number of LLM editing layers) to ensure fair comparison.”

“We use two knowledge-intensive datasets including Zero Shot RE (Levy et al., 2017) (ZS) and T-REx (Elsahar et al., 2018).”

“Two popular model editing datasets Zero-Shot Relation Extraction (Z S RE) [33] and C OUNTER -”

“In particular, the KILT benchmark (Petroni et al., 2020b), standardizes two zero-shot slot filling tasks, zsRE (Levy et al., 2017) and T-REx (Elsahar et al., 2018), providing a competitive evaluation framework to drive advancements in slot filling.”

“LLM Editing Benchmarks The widely used datasets for LLM editing are ZsRE [16] and COUN-TERFACT [3].”

“There are two mainstream learning paradigms widely explored in low-shot relation extraction, namely zeroshot RE (Levy et al., 2017) and few-shot RE (Han et al.”

“These cross-lingual word embeddings have shown to help the task of cross-lingual information extraction (Levy et al., 2017), False Friends and Cognate detection (Merlo and Ro-driguez, 2019), and Unsupervised Neural Machine Translation (Artetxe et al., 2018b).”

“ESIM (Levy et al., 2017).”

“Levy et al. [93] provide a pre-defined set that maps each relational type R ( x, y ) in knowledge data to at least one readable natural language sentence.”

“As a result, previous work has found ways to align existing relation ontologies with questions, either through human curation (Levy et al., 2017) — which limits the approach to very small ontologies — or with a small set of fixed question templates (Du and Cardie, 2020) — which relies heavily on…”

“, 2018), RelationExtraction (Levy et al., 2017), RACE (Lai et al.”

“To be capable of extracting relations that were not specified in advance, zero-shot relation extraction [ Levy et al. , 2017; Chen and Li, 2021; Chia et al. , 2022 ] is proposed to invent new models to predict new relations.”

“The MEMIT (Mass Editing Memory in a Trans-former) method proposed by Meng et al. (2023) demonstrates its effectiveness in editing up to 10,000 factual associations in transformer models on zsRE (Levy et al., 2017) and their proposed COUNTERFACT dataset, designed to test factual recall.”

“…idea, semantic typing tasks have been reformulated as prompt-based learning (Ding et al., 2021; Han et al., 2021), natural language inference (Yin et al., 2019; Sainz et al., 2021), question answering (Levy et al., 2017; Li et al., 2019; Du and Cardie, 2020), and translation (Paolini et al., 2021).”

“Several works study cross-task transfer by unifying the inputoutput format, e.g., relation extraction as machine reading comprehension (Levy et al., 2017), named entity recognition as machine reading comprehension (Li et al., 2020).”

“…data in KILT benchmark (Petroni et al., 2021), including TrivaQA (Joshi et al., 2017), kwiatkowski2019naturaluestion (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), Fever (Thorne et al., 2018), zsRE (Levy et al., 2017), AY2, T-REx (Elsahar et al., 2018b) and WoW (Dinan et al., 2018).”

“One dataset is used for fact verification, called FEVER [46], and finally, two slot-filling datasets for relation extraction are used in our experiments; zsRE [23] and T-REx [9].”

“We have evaluated PQuAD using two pre-trained transformer-based language models, namely ParsBERT (Farahani et al., 2021) and XLM-RoBERTa (Conneau et al., 2020), as well as BiDAF (Levy et al., 2017) which is an attention-based model proposed for MRC. Model EM F1 HasAns EM HasAns F1 NoAns EM/F1 BNA 54.4 71.4 43.9 66.4 8…”

“There are also some ZSL methods for dealing with the training sample shortage problem in other domains, such as Natural Language Processing (NLP) including text classification [19–21], entity linking [22, 23], relation extraction [24] and machine translation [25].”

“We extract potential edits from the validation set of zsRE [24], following the editing literature [16, 30].”

“We conduct experiments on two Wikipedia-sourced datasets used in Model Editing (ME) meth-ods: the Zero-Shot Relation Extraction (zsRE) dataset (Levy et al., 2017) and the T-REx dataset (Elsahar et al., 2018).”

“[14] Entity Linking - 3,396 WNED-CWEB [14] Entity Linking - 5,599 T-REx [12] Slot Filling 2,284,168 5,000 Zero Shot RE [25] Slot Filling 147,909 3,724 Natural Questions [24] Open Domain QA 87,372 2,837 HotpotQA [58] Open Domain QA 88,869 5,600 TriviaQA [21] Open Domain QA 61,844 5,359 ELI5 [13]…”

“The T5 model was pretrained on the Natural Questions dataset (NQ) [19], and we simulate deployment on the zsRE dataset [20] extracting potential edits from [15]’s validation split.”

“Levy et al. [46] showed that relation extraction can be approached by asking MRC models to answer simple questions instead of classifications.”

“…Aguilar et al., 2014) which are limited by their pre-defined schemas, to directly using natural language to encode semantic arguments ( QA-SRL ; He et al. 2015; Stanovsky et al. 2016; FitzGerald et al. 2018; Roit et al. 2020) and events (Levy et al., 2017; Liu et al., 2020; Du and Cardie, 2020).”

“Based on BiDAF, Levy et al. (2017) propose the BiDAF-No-Answer model to predict the distribution of answer candidates and the probability of a question being unan-swerable at the same time.”

“ZSRE Dataset.”

“Evaluating the effectiveness of knowledge editing and constructing corresponding datasets is also a research hotspot (Thorne, Vlachos, Christodoulopoulos and Mittal, 2018; Levy et al., 2017; Meng et al., 2022; Onoe et al., 2023; Zhong et al., 2023; Wang et al., 2024).”

“Taking inspiration from recent NLP works on RE [27, 69], we employ zero-shot learning (ZSL) [26] to infer relations from the texts.”

“[27] transformed the relation extraction into a MRC problem in zero-shot setting and achieved improvement.”

“uestion representation h0, following Seo et al. (2019). Note the difference, however, that we concatenate the negative example instead of considering it as an independent example with noanswer option Levy et al. (2017). During training, we find that adding tf-idf matching scores on the word-level logits of the negative…”

“Knowledge-intensive language tasks (KILTs), such as fact checking [41] and slot filling [27], have gained much attention in recent years.”

“, 2017), RelationExtraction (Levy et al., 2017), TextbookQA (Kembhavi et al.”

“The out-of-domain set we evaluate on has examples from BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017).”