Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“Most existing knowledge editing datasets assess triplet-level editing, including ZsRE (Levy et al., 2017), CounterFact (Meng et al., 2022a), Fact Verification (Mitchell et al., 2022), Calibration (Dong et al., 2022), MQuAKE (Zhong et al., 2023), RaKE (Wei et al., 2023), RIPPLEEDITS (Cohen et al., 2023 (Mitchell et al.…”

“Methods like the re-formulation of structured prediction tasks into a question-answering (QA) format (He et al., 2015; Levy et al., 2017; Du and Cardie, 2020) intend to make such tasks more amenable for annotation.”

“2020), enables the model to learn information apart from feature representation when performing relation extraction, or to learn novel relations with the help of external knowledge (Levy et al. 2017), pre-trained”

“…EE approaches, one approach that has been studied in supervised EE (Du and Cardie, 2020; Liu et al., 2020), zero-shot relation extraction (RE) (Levy et al., 2017), and zero-shot text classification (TC) (Yin et al., 2019) queries over text spans to detect events and extract arguments (Lyu et…”

“Levy et al [20] treat zero-shot relation extraction as a question answering task, they define 10 question templates and train a reading comprehension model to match questions and relations.”

“The first [32] is a preliminary study, whose experiments include a single toy example based on a beta distribution with just one feature and two side information features.”

“Inspired by recent success in formulating non-MRC NLP tasks as MRC tasks (Levy et al., 2017; Li et al., 2020c), we propose an MRC framework for SRL, which can effectively utilize the semantic information provided by frame files.”

“The closest work to ours in the information extraction space is [14] where they apply machine comprehension techniques to extract relations.”

“For EQA, we use the MRQA benchmark [15], including SQuAD [44], TriviaQA [20], NaturalQues-tion [24], NewQA [54], SearchQA [14], HotpotQA [66], BioASQ [55], DROP [13], DuoRC [47], RACE [25], RelationExtraction [27], TextbookQA [21].”

“However, existing benchmarks are largely static or centered on single-fact updates, and seldom require handling ordered event sequences or enforcing temporal consistency (Thede et al., 2025; Levy et al., 2017; Jang et al., 2022).”

“• ZsRE (Levy et al., 2017) is a question-answering (QA) dataset.”

“This also inspired the mention of ZsRE, a popular model editing evaluation dataset, as a model of the task (Levy et al., 2017).”

“These include CounterFactual [107] (used by [59, 118, 167]), PKU-SafeRLHF [74] (used by [75, 76, 188]), SQuAD [125] (used by [119, 130]), and ZsRE [181] (used by [160, 167]).”

“We show the statistics of ZeroGR-Train Dataset in the Table 5 and Figure 7.”

“We employ four widely used datasets, including Multi-Counterfact (MCF) (Meng et al., 2022), ZsRE (Levy et al., 2017), MQuAKE (Zhong et al., 2023), and Wiki-Counterfact (WCF) (Zhang et al., 2024b).”

“…the longest matching sequence of words between a generated text and a reference text, emphasizing both precision and recall in terms of sequence [25] Zero Shot RE SF 147,909 3,724 NQ [23] Natural Questions ODQA 87,372 2,837 HoPo [66] HotpotQA ODQA 88,869 5,600 TQA [20] TriviaQA ODQA 61,844…”

“…TriviaQA [9], SearchQA [5], HotpotQA [39], NaturalQuestions [12] Out-of-domain BioASQ [28], DROP [4], DuoRC [32], RACE [13], RelationExtraction [15], Text-bookQA [10] Mathematics (Problem Solving) In-domain GSM8K [3] Out-of-domain SVAMP: Subtraction, Addition, Common-Division, Multiplication…”

“The dataset comprises over 700 relation types, with our study focusing on the 31 most common ones, consistent with prior research [Levy et al. , 2017; Elazar et al. , 2021; Meng et al. , 2022a; Zhong et al. , 2023; Wei et al. , 2024; Yin et al. , 2024; Ma et al. , 2024].”

“…2021), HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019); slot filling with T-REx (Elsahar et al., 2018), Zero Shot RE (Levy et al., 2017); entity disambiguation on AIDA CoNLL-YAGO, WNED-WIKI and WNED-CEWB; dialog with Wizard of Wikipedia (Dinan et al., 2018).”

“We use the zsRE (Levy et al., 2017) dataset, which is a popular question-answering benchmark for knowledge editing.”

“• zsRE (Levy et al., 2017) is one of the most popular question answering (QA) datasets which use question rephrasing as the equivalence neighborhood.”

“Levy et al. (2017) converted the tasks of relation extraction into question answering (QA) tasks.”

“Low-shot RE, including few-shot RE (Han et al., 2018) and zero-shot RE (Levy et al., 2017), trains a model to identify unseen relation labels in a test set with a few support examples or even none of the examples.”

“ZeroshotRE is a KBQA corpus consisting of several questions generated based on 1,192 crowd-sourced question templates (e.g., “Where did x graduate from?””

“[Levy et al. 2017] aim to extract relation in natural language sentences that are only specified at test-time.”

“Inspired by current academic research on machine reading comprehension (MRC) [30], [31], some researchers have also started to turn the entity extraction problem into a MRC task about entity types.”

“We have particularly in mind the question answering format [11] which has already shown success for zero-shot relation extraction [19].”

“…questions from textbooks TriviaQA (Joshi et al., 2017) 7,785 Trivia questions with contexts collected using a Bing search RelationExtraction (Levy et al., 2017) 2,948 Generated samples using a knowledge base BioASQ (Tsatsaronis et al., 2015) 1,504 Medical articles and Liang, 2021), and LoRA…”

“The two prominent datasets are ZsRE (Levy et al., 2017) and COUNTERFACT (Meng et al., 2022).”

“7) for ZsRE, while CrispEdit - Seq uses ( n, γ ) = 4, CrispEdit uses ( n, γ ) = (1 , 000 , 0 .”

“Initial efforts concentrating on foundational knowledge include LAMA [143], which utilizes cloze-style probes for factual retrieval, and datasets like ZsRE [103] and COUNTERFACT [125], which introduce counterfactuals to facilitate causal tracing to specific layers or heads.”

“To evaluate KE methods, we utilize two widely recognized benchmarks: the Counterfact dataset [9] and the ZsRE dataset [19].”

“Current evaluations of KE methods primarily focused on general-domain benchmarks, such as WikiData (Meng et al., 2022) and counterfactual datasets (Levy et al., 2017), which aimed to up-date LLMs with new knowledge that contradicted common facts.”

“Tasks in existing datasets include QA, sentence completion, choose, and cloze tests (Zeng et al., 2024; Zhong et al., 2023; Levy et al., 2017a; Meng et al., 2022; Zhang et al., 2024c; Wang et al., 2024c).”

“Most widely used knowledge editing datasets (Levy et al., 2017; Meng et al., 2022a; Zhang et al., 2024; Thede et al., 2025) represent knowledge as simple subject-relation-object triples.”

“We evaluate KEDiT on the following tasks included in KILT: FEVER (Thorne et al., 2018), AIDA CoNLL-YAGO (AY2; Hoffart et al., 2011), Zero Shot RE (Levy et al., 2017), Natural Questions (NQ; Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), and ELI5 (Fan et al.,…”

“The benchmark spans three data domains: two widely-used machine unlearning datasets, TOFU [18] and the Harry Potter books [19, 20], as well as a popular model editing dataset, ZsRE [21].”

“We use two well-known datasets: ZsRE (Levy et al., 2017) and C OUNTER F ACT (Meng et al., 2022).”

“Additionally, we present a comparison between MKEB and other datasets, including CounterFact (Meng et al., 2022), zsRE (Levy et al., 2017), MQuAKE (Zhong et al., 2023), and MLaKE (Wei et al., 2024), in Table 2.”

“Levy et al. [4] transfer ZSRE as a reading comprehensive task, which relies on human effort to design question and answer template for relation classes.”

“We first conduct experiments on zsRE Levy et al. (2017), which is a question-answering task containing real-world facts and test the ability of adding correct information to LLMs. zsRE is a prediction task and only prediction-based metrics are evaluated.”

“[9] reframe ZeroRC as a question-answering problem.”

“Entity and relation extraction (ERE) contains two sub-tasks called named entity recognition (NER) [8, 10, 21, 23] and relation classification (RC) [13, 14], which is the fundamental step of automatic knowledge graphs (KGs) [15] construction, knowledge discovery and intelligent question answering…”

“, 2018), kilt_zsre (Levy et al., 2017), lama-conceptnet (Petroni et al.”

“Recently, there is a trend to cast non-QA NLP tasks, such as information extraction (Levy et al., 2017; Li et al., 2019; Li et al., 2020), text classification and more (McCann et al., 2018; Keskar et al., 2019) into MRC, which can achieve comparable or improved results on the original task, thanks…”

“Following (Levy et al., 2017) and (Yatskar, 2018), we implement several effective models.”

“In recent years, several methods that tackle the relation extraction (RE) task with question answering (QA) have shown impressive results (Levy et al., 2017; Li et al., 2019; Cohen et al., 2021).”

“At step 𝑡 , the model reads the input instruction 𝑋 𝑡 and generates the assistant response 𝑌 𝑡 = ( 𝑦 𝑡, 1 , 𝑦 𝑡, | 𝑌 𝑡 | ) .”

“ZsRE (Levy et al., 2017) evaluates text-based question-answering edits and robustness to rephrased queries.”

“…15 datasets widely adopted in mechanistic interpretability and post-training research: OpenBookQA [40], Gender [41], RTE [42], IOI [43], Docstring [44], SST2 [45], Winogrande [46], Reverse [47], Greater Than [48], FEVER [49], zsRE [50], Induction [15], Bool [51], Arithmetic [52], and SA [53].”

“Following prior work, we use COUNTERFACT (single-hop counterfactual edits introduced with MEMIT) (Meng et al., 2023) and ZsRE (zero-shot relation extraction) (Levy et al., 2017).”

“We select KnowEdit (Zhang et al., 2024c) for knowledge editing tasks, mainly on ZsRE dataset (Levy et al., 2017).”

“Zero-shot relation extraction (ZSRE) aims to identify semantic relations between entities without relying on labeled training instances (Levy et al., 2017).”

“Additionally, on the ZsRE dataset, although the overall trend was similar to that on the CounterFact dataset, there were some fluctuations in the related metrics, which may be attributed to the characteristics of the dataset itself or the model’s adaptability to different datasets.”

“For the benchmark, we evaluated MindBridge on two commonly used knowledge editing datasets: the ZsRE dataset (Levy et al., 2017) and the Counterfact dataset (Meng et al., 2022a).”

“In our experiment, two popular model editing datasets Z S RE (Levy et al., 2017) and C OUNTER -”

“Levy et al. [14] first introduced the zero-shot relation extraction task and proposed a paradigm based on machine reading comprehension (MRC).”

“For dataset, we use two prominent knowledge datasets: ZsRE (Levy et al., 2017) and C OUNTER -”

“And QA forms [21] can include different types of question-and-answer tasks.”

“Recent works (Levy et al., 2017; Obamuyide and Vlachos, 2018; Zhao et al., 2023) formulated the zero-shot RE to other NLP tasks such as reading comprehension (Levy et al., 2017; Zhao et al., 2023) and textual entailment (Oba-muyide and Vlachos, 2018).”

“Zero-shot Relation Extraction: Zero-shot relation extraction methods typically assume that test data only contains novel classes and that descriptions for those novel classes are readily available (Levy et al., 2017; Obamuyide and Vlachos, 2018; Lockard et al., 2020; Chen and Li, 2021).”

“[29] first converted the relation extraction task into the form of reading comprehension, which could adapt to zero-shot scenario problems.”

“Levy et al. [17] first reduce the relation extraction to answering crowd-source questions and effectively generalize to zero-shot scenarios.”

“They also provide several baselines, such as BiDAF-No-Answer (BNA), which is originated from the BiDAF model [26] with an unanswerable classification module [19].”

“This work reflects an increasing convergence between language models, knowledge bases, and question answering, such that problems in one have been shown as solvable in another [22, 29, 37].”

“With the experimental results shown in the Table 15 and Table 16, we can see that our model outperforms the baseline provided by [11] in both unseen entities setting and unseen relations setting with the help of our Answer Triggering model.”

“Datasets & LLMs : Following previous works (Mitchell et al., 2022a; Meng et al., 2022, 2023; Li et al., 2024b), we also use the EDIT sets of Z S RE (Levy et al., 2017) and C OUNTER F ACT (Meng et al., 2022) to evaluate our method.”

“In each database, we omit a randomly selected table, making a sub-set of the queries associated with that organization unanswerable , simulating a scenario in which required information is simply not present (Levy et al., 2017; Rajpurkar et al., 2018) in the organization.”

“We consider five text classification datasets: i) FEVER-F ACT C HECKING (Thorne et al., E (Levy et al., 2017) and W IKI T EXT (Merity, 2016) datasets.”

“Currently, in most knowledge editing papers, researchers often evaluate the performance of knowledge editing techniques using idealized datasets like COUNTERFACT (Meng et al., 2022a) and ZsRE (Levy et al., 2017).”

“We conduct experiments on the knowledge-editing task of Zero-Shot Relation Extraction (zsRE) (Levy et al., 2017).”

“For instance, Levy et al. (2017) reduced RE to the problem of answering simple reading comprehension questions.”

“To name a few, authors in [9], [33] cast relation”

“In order to apply DrQA to the LAMA probe, we take inspiration from [Levy et al., 2017] and map each cloze-style template to a natural question template ( e.g. , “ X was born in [Mask] ” to “Where was X born?”).”

“Following [10], we evaluate WISE on the closed-book question-answering (QA) dataset ZsRE [46], and also evaluate its ability to correct Hallucination in SelfCheckGPT [48].”

“We train our model on the processed version of the C4 dataset (Raffel et al., 2020) from Red-Pajama (Computer, 2023).”

“The first datasets for this task, such as zsRE (Levy et al., 2017) or ParaRel (Elazar et al., 2021), were initially created to test encoder-based models such as BERT (Devlin et al., 2018), but then started being used for decoder models evaluation (for example Yao et al., 2023; Meng et al., 2022).”

“Existing benchmarks (Meng et al., 2022a; Levy et al., 2017) for knowledge intervention evaluation suffer from two major limitations.”

“We evaluate X-KDE and Baselines using two widely used benchmarks: Bi-ZsRE dataset (Wang et al., 2023a) and MzSRE dataset (Wang et al., 2023d).”

“Moreover, these datasets are all composed of data in a single format with a single task like QA (Levy et al., 2017) or sentence completion (Meng et al., 2022), which leads to a disparity between experiments and practical applications.”

“ZsRE (Levy et al., 2017) is a question-answering dataset consisting of 10,000 real-world facts, used to test model editing methods for adding correct information.”

“For evaluation, we conduct experiments on 10 , 000 ( 2 , 000 ) samples of COUNTERFACT (Meng et al. 2022a) dataset for GPT2-XL (GPT-J), 1 , 000 samples of COUNTERFACT for Llama-2 and 10 , 000 samples of Zero-shot Relation Extraction (zsRE) (Levy et al. 2017; Meng et al. 2022a) for GPT2-XL.”

“Dataset The zsRE dataset (Levy et al., 2017) and the Counterfact dataset (Meng et al., 2022) are the most commonly used model editing datasets. zsRE consists of an edit prompt along with several paraphrased versions of that prompt.”

“), generalization (Gen), locality (Loc), and portability (Port) scores for multiple language models evaluated using the ZsRE dataset and the MEMIT editing method.”

“Levy et al. [25] simplified the relation extraction task to a single QA task by associating one or more natural language questions with each relation slot.”

“Low-shot RE, including few-shot RE [3] and zero-shot RE [54], trains a model to recognize unseen relation labels in a test set with a few support instances or even none of the instances.”