Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“The original SERAC is finetuned on supervised data, but we are interested in the zero-shot performance, so we train SERAC on ZsRE using the same hyperparameters as Mitchell et al. (2022b) and evaluate it on unseen datasets without additional finetuning.”

“To integrate external knowledge for NLP tasks such as question answering (Kwiatkowski et al., 2019; Joshi et al., 2017; Yang et al., 2018), slot filling (Levy et al., 2017), and fact verification (Thorne et al., 2018), systems employ a retrieve-then-read pipeline of two models (Chen et al., 2017).”

“Many downstream tasks rely on relational triple extraction, such as building knowledge graph (Wang et al., 2018; Nayak et al., 2021), reading comprehension (Levy et al., 2017; Zhao et al., 2021) and biomedical text mining (Wei et al., 2016; Zeng et al., 2022).”

“Some studies [3, 4] cast zero-shot relation extraction task into other tasks (i.e., question answering and textual entailment) to fully leverage the corresponding labeled data and models.”

“Prompts have been widely adopted in many NLP tasks (Levy et al., 2017; McCann et al., 2018, inter alia), paving the way for formulating TE tasks as Question Answering (QA) (Du and Cardie, 2020b; Chen et al., 2020b, inter alia).”

“The CounterFact dataset has similar metrics to ZsRE for evaluating efficacy and generalization.. Additionally, CounterFact includes multiple generation prompts with the same meaning as the original prompt to test the quality of generated text, specifically focusing on fluency and consistency.”

“We initially employ two popular model editing datasets, ZsRE (Levy et al., 2017) and COUNTER-FACT (Meng et al., 2022).”

“Researches on zero-shot RE tasks refer to a lot of cutting-edge techniques, e.g., incorporating the text description of relation types [18], [19], logic rules of knowledge graph [20], prototypical networks [21], prompt-based learning [22], and formulating RE tasks into question answering,…”

“[14] Entity Linking - 3,396 WNED-CWEB [14] Entity Linking - 5,599 T-REx [12] Slot Filling 2,284,168 5,000 Zero Shot RE [25] Slot Filling 147,909 3,724 Natural Questions [24] Open Domain QA 87,372 2,837 HotpotQA [58] Open Domain QA 88,869 5,600 TriviaQA [21] Open Domain QA 61,844 5,359 ELI5 [13]…”

“We evaluated Larimar on the ZsRE benchmark (Levy et al., 2017 comprehension.”

“Weusethree representativedatasetsfocusingonknowledgeinsertionand modification:WikiData counterfact [70] for counterfactual edits, WikiData recent [71] for recent factual updates, and ZsRE [72] for assessing generalization to related contexts.”

“…limitation by constructing explicit factual updates: ZsRE and CounterFact test whether models can revise individual facts while preserving locality [Levy et al., 2017, De Cao et al., 2021, Meng et al., 2022], while MQuAKE and RippleEdits further evaluate whether such updates propagate to logically…”

“ParaphraseRC [44], TextbookQA [24], RelationExtraction [28] and RACE [27].”

“We follow the recent work Alphaedit in using two benchmark datasets: Multi-CounterFact (MCF) (Meng et al., 2022) and ZsRE (Levy et al., 2017), editing 2,000 knowledge items for each dataset.”

“2) ZsRE (Zero-Shot Relation Extraction) Evaluates the model’s ability to extract and geralize factual relations.”

“To further demonstrate scalability and broader feasibility, we extend our evaluation to the ZsRE dataset (Levy et al., 2017) with 3,000 edits to benchmark performance in a structured editing setting.”

“Standard benchmarks include CounterFact and zsRE (Meng et al., 2022c; Levy et al., 2017).”

“Two widely used benchmarks were adopted: Counter-Fact (Meng et al., 2022) and ZsRE (Levy et al., 2017).”

“The ZsRE and CounterFact datasets used in [Meng et al. 2022], as well as KnowEdit [Zhang et al. 2024b], are examples of such approaches, where models must learn to connect entities through predefined relations.”

“Existing work mainly focuses on plain text language models and benchmark datasets, such as ZsRE (Levy et al. 2017) and CounterFact (Meng et al. 2022).”

“We evaluate MobiEdit on two standard datasets widely used in factual knowledge editing: ZsRE [10], a zero-shot relation extraction dataset derived from WikiRE, and CounterFact [14], a curated benchmark of factual edits targeting named entities (e.g., people, locations), with truth and…”

“…of multi-hop reasoning questions, constructed by collecting single-hop questions from Wikipedia-based datasets such as SQuAD (Rajpurkar et al., 2016), Natural Questions (Kwiatkowski et al., 2019), MLQA (Lewis et al., 2019), T-Rex (Elsahar et al., 2018), and Zero Shot RE (Levy et al., 2017).”

“The datasets used in the experiments are ZSRE [10], a question-answering task, and COUNTERFACT [6], a collection of factual statements.”

“Based on these datasets, several innovative methods have been proposed to address the unanswerability issue ranging from probability prediction and extraction [4, 12, 15, 18, 32] to leveraging large language models [1, 13, 14, 37, 38].”

“The primary benchmarks currently employed to assess editing methods are Zero-Shot Relation Extraction(zsRE) (Levy et al., 2017) and CounterFact (Meng et al., 2022). zsRE is a question-answering dataset designed for relation-specific queries.”

“ZORE aims to extract novel relational facts where the target relation types are not observed in the training set (Levy et al., 2017).”

“Questions are linked to relevant scientific literature found on PubMed, with the abstracts of these articles serving as the context for MRC. RelationExtraction (RelEx) comprises question-answer pairs that express Wikidata relations between entities mentioned in Wikipedia articles (Levy et al., 2017).”

“…in a hidden layer representation for cross-domain (Liu et al., 2018; Li et al., 2019b; Du et al., 2020) or cross-task decision making (Johnson et al., 2017; Levy et al., 2017; Eriguchi et al., 2018; Lee et al., 2019; Wang et al.; Keung et al., 2019; Vernikos et al., 2020; Wang et al., 2022).”

“Treating NLP tasks other than MRC as MRC problems strengthens neural networks’ reasoning processes, including event extraction (Du and Cardie, 2020; Liu et al., 2020), relation extraction (Li et al., 2019; Levy et al., 2017), and named-entity recognition (Li et al., 2020).”

“For EQA, we use the MRQA benchmark [15], including SQuAD [44], TriviaQA [20], NaturalQues-tion [24], NewQA [54], SearchQA [14], HotpotQA [66], BioASQ [55], DROP [13], DuoRC [47], RACE [25], RelationExtraction [27], TextbookQA [21].”

“The parameters of the training used were set to the following values: batch size equals to 32 , learning rate equals to 5 e-5 , the optimizer chosen is Adam (Kingma and Ba, 2015) with default parameters.”

“Possible exceptions include Lin et al. (2016), who present a rule-based, feature-rich system for handling ellipsis and coreference in Chinese medical dialogues, but the synergy between the two subsystems is limited; and Banjade et al.”

“Levy et al. extended BiDAF (Levy et al., 2017; Seo et al., 2016) to deal with unanswerable questions by effectively setting a threshold p, whereby the model will output no answer if the model’s highest confidence in any answers is less than p.”

“[Levy et al. , 2017] reduced relation extraction to answering simple reading comprehension questions.”

“In the literature, zero-shot RE has been reformulated as other tasks such as reading comprehension (Levy et al., 2017), textual entailment (Sainz et al., 2021, Rahimi and Surdeanu, 2023), summarization (Lu et al., 2022), span-prediction (Cohen et al., 2020), question answering (Cetoli, 2020),…”

“Table 13: Non-compositional results on classic ME dataset ZsRE [50] and Counterfact [8] following [11] on Llama3-8B.”

“We first evaluate COMEM on Zero-Shot Relation Extraction (zsRE, Levy et al. (2017)) dataset with 10,000 knowledge edits following (Cao et al., 2021; Mitchell et al., 2022a; Meng et al., 2022b).”

“By linking one or more natural language questions to each relation, Levy et al. [9] have reframed the relation extraction job as a reading comprehension issue.”

“We evaluate AlphaEdit using two widely adopted benchmarks: the Counterfact dataset (Meng et al., 2022) and the ZsRE dataset (Levy et al., 2017).”

“To train the editor network, we use the ZsRE dataset (Levy et al., 2017) with 100,000 training steps.”

“ZsRE [Levy et al., 2017] contains tens of thousands of examples and predates modern LLMs, limiting its suitability for large-scale evaluation.”

“We conduct our evaluation on two standard benchmarks for structured knowledge editing, CounterFact (Meng et al., 2022) and ZsRE (Levy et al., 2017).”

“• MQUAKE (Zhong et al., 2024): Although originally designed for assessing multi-hop knowledge editing, we adapt this dataset to a single-hop setting analogous to ZsRE .”

“We assess editing performance on two standard factual editing benchmarks: C OUNTER F ACT [Meng et al. , 2022] and Z S RE [Levy et al. , 2017].”

“Evaluation is performed on the MRQA test set consisting of BioASQ [11], DROP [12], DuoRC [13], RACE [14], RelationExtraction [15], and TextbookQA [16].”

“First, zero-shot prompting was used; for each generated subquery derived from both the segmented text and the original records, the top 5 most semantically relevant CCC coding rules were retrieved and incorporated into the prompt [30].”

“Datasets and Metrics We adopt three widely-used evaluation datasets from knowledge editing: ZSRE (Levy et al., 2017), Counterfact (Meng et al., 2023a), and RIPE (Cohen et al., 2024 AlphaEdit (Fang et al., 2025) resolves disturbances to existing knowledge caused by the perturbations in…”

“We conduct ablation studies on both the unstructured CounterFact-uns dataset and the structured ZsRE dataset, to evaluate the effectiveness and robustness of our method across different knowledge formats.”

“To examine this hypothesis, we conduct experiments on the several benchmark datasets [23, 44].”

“Experiments are conducted on two standard benchmarks, CounterFact (Meng et al., 2022), and zsRE (Levy et al., 2017), where we construct specific candidate libraries for subjects and prompts to simulate realistic attack scenarios.”

“ZsRE [Levy et al., 2017] is a widely used factual probing dataset derived from relation extraction in a question-answering format.”

“We could not find the license for zsRE (Levy et al., 2017).”

“We evaluate HORSE on two mainstream datasets: zsRE (question-answer pairs) [17], CounterFact (counterfactual statements) [7].”

“Early works framed ZSRE as reading comprehension (Levy et al., 2017) or textual entailment tasks (Obamuyide and Vlachos, 2018), with later improvements like entailment templates (Rahimi and Surdeanu, 2023).”

“The editing performance was evaluated using two types of datasets: factual knowledge-based datasets, including ZsRE (Levy et al. 2017) and Coun-terFact (Meng et al. 2022), and a conceptual knowledge-based dataset, ConceptEdit (Wang et al. 2024b).”

“Following Meng et al. [23]’s work, we employ Coun-terFact ( CF ) [23] and Zero-Shot Relation Extraction ( zsRE ) [12, 26].”

“We use two widely used knowledge editing benchmarks: C OUNTER F ACT (Meng et al., 2023a) and ZS RE (Levy et al., 2017).”

“Previous work has established four datasets for the evaluation of knowledge editors: zsre (Levy et al., 2017), CounterFact (Meng et al., 2023a), MQuAKE (Zhong et al., 2024), and RippleEdits (Cohen et al., 2023).”

“…evaluation, we experiment over 11 information-seeking datasets, including NQ (Kwiatkowski et al., 2019), Trivi-aQA (TQA) (Joshi et al., 2017), zsRE (Levy et al., 2017), T-rex (Elsahar et al., 2018), and HotpotQA (HQA) (Yang et al., 2018), using the versions provided in KILT (Petroni et al., 2021).”

“In the setting, meta-training is conducted on the zsRE [21] dataset, which comes with paraphrasing.”

“The evaluation results are presented in Table 2 for C OUNTER F ACT and Table 10 for ZsRE (provided in Appendix due to space constraints).”

“The following experiments are uniformly set as follows: base model is GPTJ (Achiam et al., 2023) model, dataset is factual dataset zsRE (Levy et al., 2017).”

“Following the literature (Meng et al., 2022; 2023), we use Multi-CounterFact (MCF) (Meng et al., 2022) and zsRE (Levy et al., 2017) as datasets and evaluate the editing performance on GPT2-XL (Radford et al., 2019) and GPT-J (6B) (Wang & Komatsuzaki, 2021).”

“Levy et al. [20] transformed relation extraction into zero-shot MRC problems.”

“To evaluate LLMs’ preference for context knowledge, we adopt the test set of MRQA (Fisch et al., 2019), covering BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtrac-tion (Levy et al., 2017), and TextbookQA (Kemb-havi et al., 2017) across variou…”

“To account for these drawbacks, recent works have proposed clever adaptation methods to recast RE into adjacent NLP tasks, such as a question-answering (Levy et al., 2017) and natural language inference (Obamuyide and Vlachos, 2018; Sainz et al., 2021, 2022; Xu et al., 2023).”

“QA as annotation is proposed to transform the relation extraction task into a reading comprehension task (Levy et al., 2017; He et al., 2015; Michael et al., 2018).”

“The Reading Comprehension Modeling paradigm is widely used with answering systems in the field of NLP [5], Question Answering [6], Information Retrieval [7] and other domains.”

“Zero-shot relation extraction (ZeroRE) is a former task in RE (Levy et al., 2017).”

“BiDAF-No-Answer(BNA) BNA[22] is a classic model for machine reading comprehension-based question answering tasks.”

“Levy et al. (2017) reformulated the zero-shot slot filling task to a reading comprehension task.”

“Levy et al [7] reduced RC to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot.”

“It also includes 3 small out-of-domain datasets (RelationExtraction [18], DuoRC [19], RACE [20]) with size of 127 for fine-tuning.”

“In this paper, we show that it is possible to reduce IE from documents to a problem of answering simple reading comprehension questions (Levy et al., 2017) (Figure 1) by leveraging the notion that language models are good few-shot learners (Brown et al., 2020) and the fact that a class-aware QA…”

“A I] 1 7 A pr A recently introduced suite of benchmarks, KILT (Knowledge Intensive Language Tasks) [Petroni et al., 2020b], standardizes two zero-shot slot filling tasks: zsRE [Levy et al., 2017] and T-REx [Elsahar et al., 2018].”

“…(Roth et al., 2018), knowledge graph building (Das et al., 2018), NER (Li et al., 2019), etc. Compare with some studies directly leveraging MRC (Levy et al., 2017; Das et al., 2018), we have a totally different main contribution, i.e., a new document paradigm which can provide multi-level…”

“Six other evaluation datasets are included: BioASQ (Tsatsaronis et al., 2012), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationsExtraction (Levy et al., 2017) , and TextbookQA (Kembhavi et al., 2017).”

“Each sample in ZsRE contains a subject string and a set of answers as editing targets for assessing editing success, a rephrased question for evaluating generalization, and a locality question for measuring specificity.”

“Specifically, AlphaSet1 contains 200 K 0 facts from Wikipedia, for which high-similarity paraphrases are created and their answers adjusted by a large language model, together with 800 randomly sampled ZsRE examples.”

“We train our editor using trajectories constructed from CounterFact (Meng et al., 2022), and evaluate continual-editing performance on two standard benchmarks, ZsRE (Levy et al., 2017) and FEVER (Thorne et al., 2018).”

“On the ZsRE dataset with T = 1000 as an example, orange points denote the similarity between the generated key and the last subject token of the unseen paraphrased prompt.”

“We use ZsRE (Levy et al., 2017) and WikiData counterfact (Cohen et al., 2024).”

“Below, we provide an example of a ZsRE record.”

“To further demonstrate scalability and broader feasibility, we extend our evaluation to the ZsRE dataset (Levy et al., 2017) with 3,000 edits to benchmark performance in a structured editing setting.”

“The benchmark spans three data domains: two widely-used machine unlearning datasets, TOFU [18] and the Harry Potter books [19, 20], as well as a popular model editing dataset, ZsRE [21].”