Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“The primary benchmarks currently employed to assess editing methods are Zero-Shot Relation Extraction(zsRE) (Levy et al., 2017) and CounterFact (Meng et al., 2022). zsRE is a question-answering dataset designed for relation-specific queries.”

“We follow the evaluation metrics setup of Counter-Fact and ZsRE as outlined in Meng et al. (2023a,b); Li et al. (2024a).”

“[730] As of October 2022 , Messi ranks 9th all time in goals scored from direct free kicks with 60 , the most among active players.”

“These tasks are assessed after 10 to 40 edits on the Zero-Shot Relation Extraction (ZsRE) dataset(Levy et al., 2017), comparing the results against pre-editing benchmarks.”

“ZS RE (Levy et al., 2017; Mitchell et al., 2022a) is a Question Answering dataset, where each instance contains a factual statement and a paraphrased prompt.”

“Such methods intuitively transform RE into other targets like question answering (Levy et al., 2017; Li et al., 2019), natural language inference (Sainz et al., 2021) and translation (Paolini et al., 2021; Wang et al., 2021a).”

“…(ICL) is a new paradigm using the language model (LM) to perform many NLP tasks [58, 59], including entity recognition [60], relation extraction [61], document extraction [62] without Given a segment of a legal case document as enclosed within angular brackets, enlist the likely label that…”

“In Figure 2, we utilized the zsRE[Levy et al., 2017], ELKEN[Peng et al., 2024], 20Q[Porada et al., 2021, Gupta et al., 2023], and CKD[Huang et al., 2024] datasets as editing corpora.”

“We use ZsRE and LLaMA2-7B for the analysis.”

“We introduce recent datasets and knowledge editing methods in this section.”

“CounterFact (Meng et al., 2022) and zsRE (Levy et al., 2017) are prominent datasets to evaluate update algorithms.”

“We find that for the ZSRE task, LMCORE achieves a significantly higher F-1 score (74.80) compared to the second-best RAG model (49.95).”

“…filling, and dialogue categories: HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), Natural Questions (NQ) (Kwiatkowski et al., 2019), T-REx (Elsahar et al., 2018), Zero Shot RE (zsRE) (Levy et al., 2017)), FEVER (Thorne et al., 2018), and Wizard of Wikipedia (WoW) (Dinan et al., 2018).”

“Conversely, word-level linguistic knowledge is centered around individual words, capturing specific attributes such as part-of-speech tags and sentiment polarity.”

“[21] introduced KILT, a new benchmark of knowledge intensive tasks, which contains among others two slot filling datasets, zero-shot RE which was first presented in [14] and T-REx introduced in [3].”

“Zero-shot relation extraction (ZSRE, Levy et al. 2017) is a slot filling task that aims to predict the object when given the subject and the relation.”

“Recently, [Levy et al., 2017] propose to reduce RE slot-filling to a question answering problem.”

“We utilize GPT-4o and GPT-J (6B) models to generate demonstrations using both few-shot and zero-shot generation (Levy et al., 2017).”

“There exist many specific IE tasks: Named Entity Recognition (Lample et al., 2016; Lin et al., 2019), Relational Triple Extraction (Zheng et al., 2017; Levy et al., 2017), Event Extraction (Lin et al., 2018; Wadden et al., 2019; Du and Cardie, 2020; Li et al., 2022c,b,a), etc.”

“Previously, zero-shot relation extraction was approached as a slot-filling task [59] and solved through the use of reading comprehension techniques.”

“Some work tackles low-resource RE by indirect supervision, which solves RE by other tasks such as machine reading comprehension (Levy et al., 2017), textual entailment (Sainz et al., 2021), and abstractive summarization (Lu et al., 2022).”

“Zero-shot RE methods apply indirect supervision and convert RE tasks to other NLP tasks (Levy et al. 2017; Obamuyide and Vlachos 2018; Sainz et al. 2021; Lu et al. 2022).”

“We evaluate retrieval on eight datasets: FEVER (Thorne et al., 2018), T-REx (Elsahar et al., 2018), Zero Shot RE (Levy et al., 2017), Natural Questions (Kwiatkowski et al., 2019), HotPotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), and Wizard of Wikipedia (Dinan et…”

“The ability to automatically frame wet lab experiments in the form of text game quests and leverage the latest text-game agent advances to interactively train agents to perform them has implications for significantly improving procedural text understanding [Levy et al., 2017] and in the reproducibility of scientific e…”

“Following Levy et al. (2017), we treat relation extraction as a special case of reading comprehension, that is, we “ask” for the unknown tail entity.”

“More recently the work by (Levy et al., 2017) builds on Question-Answering techniques to build new datasets and models for RE: The relations are represented as questions and the answers are the connected entities.”

“For negative examples, we generate synthetic noises, similarly to translation/generation errors, by adopting a naive but effective method to change words or sentences in a positive example (Levy et al., 2017; Zhang et al., 2019; Yang et al., 2019).”

“…(LSTM) networks [30] and gated recurrent units (GRUs) [23] were shown to be highly effective for representing sequences, and have been widely used for a variety of tasks, such as machine translation [17, 38, 56], reading comprehension [36, 53], speech recognition [18, 22, 24] and more [16, 40].”

“Procedural text understanding is a highly challenging task for today’s learning algorithms (Lucy and Gauthier, 2017; Levy et al., 2017).”

“Recently, Levy et al. (2017) presented an approach that bridges question answering and query-driven answer extraction.”

“In this paper, we employ the most widely adopted editing datasets ZsRE (Levy et al., 2017) and Counterfact (Meng et al., 2022a) as editing datasets across all experiments.”

“For instance, Levy et al. [23] proposed a method to cast the relation extraction task as a QA task by parameterizing each relation type R(x,y) as a question Q(x), with y being the answer.”

“Datasets & Evaluation Metrics We use the ZsRE (Levy et al., 2017) and COUNTERFACT (Meng et al., 2023) in EasyEdit 2 for evaluation.”

“Closer to our work, Textual Entailment (Dagan et al., 2006) has been used as a source of external supervision to solve several text classification tasks (Yin et al., 2019, 2020; Wang et al., 2021; Sainz and Rigau, 2021; McCann et al., 2018; White et al., 2017), Named Entity Recognition (Li et al., 2022a; Poliak et al.,…”

“…answer generator (GoldQ-Base) achieves an average F1-score of 43.6, and after fine-tuning the answer module, the baseline with gold questions achieves the highest average F1-score of 58.9, outperforming the previous BiDAF method (Levy et al., 2017), which had an average F1-score of 39.6.”

“Out toolkit leans on recent work which has successfully recast several IE tasks as Textual Entailment (TE) tasks (White et al., 2017; Poliak et al., 2018; Levy et al., 2017; Sainz et al., 2021).”

“Levy et al. [34] used a zero-shot approach to train MRC model on templatized questions and inferenced it on unseen relations.”

“Formulating NLP tasks as MRC tasks has been a rising trend in recent years, such as dependency parsing (Gan et al., 2021), relation extraction (Levy et al., 2017), named entity recognition (Li et al., 2020), sentiment analysis (Chen et al., 2021; Mao et al., 2021).”

“Levy et al. (2017) consider relation extraction as a machine reading comprehension.”

“…benefit of applicability to free-flowing texts, and is therefore complementary to these other measures which focus on adding facts of the canonical form ( subject, relation, object , related works (Cohen et al., 2023a; Elazar et al., 2021; Hase et al., 2023; Levy et al., 2017; Meng et al., 2022a)).”

“Our experiments are conducted on two datasets: CounterFact [4] and zsRE [26].”

“We choose the entity-centric ZSRE (Levy et al., 2017) and the relation-centric dataset RIPE (Cohen et al., 2023) as our testing sets.”

“(ii) zsRE (De Cao et al., 2021; Levy et al., 2017), zero-shot relation extraction, derives from a factual question-answering task.”

“MRC datasets are important sources of transfer learning for zero-shot settings in many other NLP tasks (Wu et al., 2020; Levy et al., 2017; Lyu et al., 2021; Du and Cardie, 2020; Li et al., 2019).”

“(Levy et al., 2017) models RE as a simple QA problem, i.e., giving the head entity and the relation and predicting the head entity.”

“Miwa and Bansal (2016) use a sequential BiL-STM for NER and a Tree-LSTM over the shortest dependency path between candidate arguments given by an external parser and Li et al. (2017) apply this model to biomedical data.”

“, 2018), QA-RE (Levy et al., 2017), NewsQA (Trischler et al.”

“…2018; Kwiatkowski et al., 2019) to train and vali-184 date the model as in-domain settings, and unveiled six out of the twelve test datasets 1 (Dua et al., 2019; Lai et al., 2017; Kembhavi et al., 2017; Levy et al., 2017; Saha et al., 2018) to validate the trained model as out-of-domain settings.”

“Slot-filling data To extract the contexts for each triple in our dataset we use the distant super-vision method described by Levy et al. (2017).”

“For example, Levy et al. [86] first formulated the RE task as a question answering (QA) task, where the relations are defined by natural-language question templates.”

“A recent successful trend involves treating various other natural language processing tasks as an MRC task, such as relation extraction (Levy et al., 2017), named entity recognition (Li et al., 2019), event extraction (Liu et al., 2020) and coreference resolution (Wu et al., 2020).”

“This includes indirect supervision sources such as question answering and reading comprehension (Wu et al., 2020; Lyu et al., 2021; Levy et al., 2017; Li et al., 2019b; Du and Cardie, 2020), natural language inference (Li et al.”

“The comparison covers both sequential editing performance and the general editing capability of large language models (LLMs).”

“To quantitatively evaluate this aspect, we use the Zero-Shot Relation Extraction (zsRE) dataset (Levy et al., 2017) to measure knowledge capacity, following the methodology of Mitchell et al. (2022); Cao et al. (2021).”

“In reality, an IR corpus may expand as new documents continuously enter the system.”

“For the fine-tuning results, we report out-of-domain results, which include 6 datasets: BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), Relation Extraction (RE) (Levy et al., 2017), and TextbookQA (TQA) (Kembhavi et al., 2017).”

“This result validates the capability of SKR to rewrite retrieved knowledge, confirming the effectiveness of this approach.”

“In this study, we train two meta-networks using the respective training splits from the ZsRE [26] and C OUNTER F ACT datasets for GPT-2 XL, adhering to the default hyperparameter settings.”

“Another interestingthreadis tocast informationextraction as an MRC task [17, 34–36].”

“• Slot Filling : T-REx (ElSahar et al., 2018) and zero-shot RE (Levy et al., 2017).”

“We take inspiration from similar works [10, 12] over purely textual document collections like Wikipedia.”

“Six separate datasets are kept for testing— BioASQ(Tsatsaronis et al., 2015), DROP(Dua et al., 2019), DuoRC(Saha et al., 2018), RACE(Lai et al., 2017), RelationExtraction (Levy et al., 2017), TextbookQA (Kembhavi et al., 2017).”

“Levy et al. (2017) have shown that a small crowdsourced dataset of questions about relations can be exploited to perform zero-shot learning for relation extraction.”

“Zero-Shot Relation Extraction (ZSRE) was first introduced by Levy et al. (2017), and existing meth-ods can be roughly categorized into three types.”

“…retrieval methods; they are inherent to the level of personal detail required to apply equitable justice. and Grishman, 2015; Adel et al., 2016; Levy et al., 2017; Karita et al., 2019; Luo et al., 2019), but techniques have emerged for cross-sentence or even document-level relation extraction…”

“ZsRE (Levy et al., 2017) is a question-answering (QA) dataset that generates an equivalence neighbor through back-translation.”

“We use 9 datasets with various characteristics: FEVER (Thorne et al., 2018), AY2 (Hoffart et al., 2011), TREX (ElSahar et al., 2018), zsRE (Levy et al., 2017), NQ (Kwiatkowski et al., 2019), TQA (Joshi et al., 2017), WOW (Dinan et al., 2019), NQ-320k (Tay et al., 2022), and Hot-potQA (Yang et al.,…”

“The results on the MCF and ZsRE datasets are in Table 4.”

“For the knowledge editing performance, we utilize two standard benchmarks: ZsRE [34] for question answering, and CounterFact [13] for factual corrections.”

“We increase the number of edits from 3K to 100K using the ZsRE dataset.”

“Question Answering (Levy et al., 2017) or Natural Language Inference (Obamuyide and Vlachos, 2018; Sainz et al., 2021) paradigms, leveraging supervised data from these auxiliary tasks.”

“Following prior works [37, 38], each Counterfact metric is defined for a large language model 𝑓 𝜃 , with a knowledge prompt ( 𝑠 𝑖 𝑖 ) , an edited target output 𝑜 𝑖 , and the model’s original output 𝑜 𝑐𝑖 : • Efficacy (edit success) : The ratio of cases where 𝑜 𝑖 has a higher probability…”

“In each database, we omit a randomly selected table, making a sub-set of the queries associated with that organization unanswerable , simulating a scenario in which required information is simply not present (Levy et al., 2017; Rajpurkar et al., 2018) in the organization.”

“Bi-zsRE (Wang et al., 2023) translates the zsRE dataset (De Cao et al., 2021; Levy et al., 2017) into Chinese with the API interfaces of gpt-3.”

“…this includes question answering datasets Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and Hot-PotQA (Yang et al., 2018), fact verification dataset FEVER (Thorne et al., 2018), and slot-filling datasets Zero Shot RE (Levy et al., 2017) and T-REx (ElSahar et al., 2018).”

“…2019), HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), a dialogue generation dataset, Wizard of Wikipedia (WoW) (Di-nan et al., 2019), and two slot filling dataset, Zero Shot RE (zsRE) (Levy et al., 2017 require the model to exploit document knowledge fully.”

“Similar to QA-SRL, QA-ZRE (Levy et al., 2017) formulates this problem as a question-answer dataset.”

“, for relation extraction [Levy et al., 2017], slot filling [Du et al.”

“, 2017) 674 RelEx (Levy et al., 2017) 2,948 TextbookQA (Kembhavi et al.”

“Previous NLP benchmarks are usually for evaluate general language understanding, such as slot filling (El-sahar et al., 2019; Levy et al., 2017), QA (Yang et al., 2018; Rajpurkar et al., 2016; Joshi et al., 2017; Fan et al., 2019; Kwiatkowski et al., 2019; Liu et al., 2020; Ding et al., 2019; Clark…”

“Beyond that, with the successful application of machine reading comprehension (MRC) in many NLP problems [7, 9], MRC is also used to NER task with the advantage of significant prior information of the entity category.”

“This approach also extends more recent work on text-based slot filling as QA (Levy et al. 2017; Du et al. 2021; Fuisz et al. 2022) by developing a much broader, multimodal computer vision-based approach.”

“To handle unanswerable queries in Web QA, we use the updated BIDAF model [26] that can handle null answers 6.”

“Levy et al [30] turned it into a reading comprehension problem by associating multiple textual questions with each relation type and learning answers for them.”

“In the literature, zero-shot RC has been reformulated as other tasks such as reading comprehension (Levy et al., 2017), textual entailment (Obamuyide and Vlachos, 2018, Sainz et al., 2021, Wang et al., 2022b), summarization (Lu et al., 2022), span-prediction (Cohen et al., 2020), question answering…”

“RelationExtraction [82] is a multiturn QA task created from datasets, such as ACE04, ACE05, and CoNLL04, where entities and relations are converted into QA pairs using templates.”

“A widely adopted strategy for evaluating the effectiveness of KE methods is to test whether the model can reproduce newly injected knowledge, as seen in benchmarks like ZsRE (Levy et al., 2017) and COUNTERFACT (Meng et al., 2022a).”

“Datasets and Benchmarks CounterFact (Meng et al., 2022b) and zsRE (Levy et al., 2017) have been used widely to evaluate KE methods.”

“In this case, an ideal output would be: “ 黄 真真 - 导 演 - 《 闺 蜜 2 》 ” Each model—Gemini, LLaMA, and ChatGPT—processes the input using its pre-trained knowledge, generating structured outputs in the form of triples.”

“ZsRE is a comprehensive question-answering (QA) dataset(Levy et al., 2017).”