Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“IKE (Zheng et al., 2023), as the most competitive method, achieves good performance on real-world datasets ZsRE by providing additional editing examples to prompt the model to learn the current task format.”

“We use ZsRE and LLaMA2-7B for the analysis.”

“Specifically, question answering [6] and text entailment [7] are two prominent examples.”

“, 2017) 674 RelEx (Levy et al., 2017) 2,948 TextbookQA (Kembhavi et al.”

“ZSRE (Levy et al., 2017) includes claims like “Los Angeles is known for its food” (labeled as false), and CounterFact (Meng et al., 2022) includes prompts like “Tom Hank’s profession is a” with the label “actor” – a Google search lists 13 professions, suggesting it would be inappropriate to treat…”

“For ZsRE and C OUNTER F ACT , we use their established test sets, each with 10,000 records.”

“Other commonly used knowledge editing datasets include ZsRE (Levy et al., 2017; Yao et al., 2023), WikiBio (Hartvigsen et al., 2023), WikiData (Cohen et al., 2023), and ConvSent (Mitchell et al., 2022).”

“• ZsRE question answering task (Levy et al., 2017) was first used for factual knowledge evaluation by (Cao et al., 2021), later being extended and adopted by (Mitchell et al., 2022a).”

“By updating only conditional memory, EngramEdit achieves near-perfect editing success on both CounterFact [9] and ZsRE [10].”

“In this section, we present evaluations on MO-IKE and our baselines across three standard knowledge editing benchmarks: C OUNTER F ACT (Meng et al., 2022), Z S RE (Levy et al., 2017), and W IKI -”

“We build on the Zero-Shot Relation Extraction (ZsRE) benchmark [29], which pairs each factual (subject, relation, object) triple with crowd-sourced paraphrases of the underlying relation.”

“ZsRE (Levy et al. 2017) is a few-shot factual editing benchmark.”

“Evaluation is commonly conducted on benchmarks such as zsRE (Levy et al. 2017), CounterFact (Meng et al. 2022a), and UnKE (Deng et al. 2024), measuring edit success, para-phrase generalization, and preservation of non-target knowledge.”

“Static editing benchmarks such as C OUNTER F ACT (Meng et al., 2022) and zsRE (Levy et al., 2017) cast an edit as a single ( s, r, o → o ∗ ) rewrite, with no temporal dimension.”

“We evaluate with the R-ROME [23] method on ZSRE [34, 57], WikiCounterFact [64], WikiBio [64, 26], and WikiRecent [64, 10], following the proxy-set protocol of recent study [12].”

“Despite their empirical success on constrained benchmarks like CounterFact [27] and zsRE [22], these editing techniques suffer from severe limitations, including brittle generalization under paraphrasing and sequential interference [18].”

“We evaluate performance on two common knowledge editing benchmarks: COUNTERFACT (single-hop counterfactual edit dataset) presented with MEMIT [Meng et al., 2022b] and ZsRE (zero-shot relation extraction) [Levy et al., 2017].”

“Levy et al. [2017] reduced relation extraction to answering reading comprehension questions by associating natural-language questions with each relation slot.”

“UnKEBench (Deng et al., 2025) tests editing over noisy, unstructured data, and ZsRE (Levy et al., 2017) focuses on QA-based factual samples.”

“Incorporating attention mechanisms into the proposed semantic clone detection model significantly enhances its overall performance [79,96].”

“Question-answering has been widely used to elicit structured information from text (Levy et al., 2017; Li et al., 2019; Du and Cardie, 2020), by leveraging the cognitive process of interrogative inquiry, which facilitates the construction of situation models (Chi et al., 1989; Graesser and Person,…”

“On the ZSRE dataset, where generalization primarily involves paraphrasing of edit samples, DE-based methods also perform strongly.”

“We conduct a thorough ablation study on the COUNTERFACT dataset using Llama-3-8B-Instruct to isolate the individual contributions of each key component in our EtCon framework, with re-sults presented in Table 4.”

“To mitigate the degradation of harmlessness caused by fingerprint injection for SFT, we follow prior works (Wang et al., 2025; Xu et al., 2024) by selecting 50 regularization samples from ZsRE (Levy et al., 2017) to mix with the fingerprint dataset.”

“…knowledge editing datasets are created based on temporal changes or counter-fact modifications of world knowledge either from existing QA datasets (Levy et al., 2017; Yao et al., 2023) or Wikipedia (Cohen et al., 2024a; Zhong et al., 2023), in which Zhong et al. (2023) first propose multi-hop…”

“Datasets For evaluating the reliability, generality, and related-locality of factual editing, we generate two evaluation datasets using GPT-4o mini based on the zsRE question-answering dataset [18] and the Counterfact dataset [23].”

“ZsRE (Levy et al., 2017) was originally introduced for zero-shot relation extraction and later adapted for KE by (De Cao et al., 2021; Mitchell et al., 2022).”

“These experiments are presented as averages from 5 random runs with different random seeds and the same hyperparameters.”

“The foundational datasets primarily focus on knowledge representation types, including: 1) commonsense knowledge (Levy et al., 2017; Porada et al., 2021; Meng et al., 2022; Gupta et al., 2023), 2) biased knowledge (Chen et al., 2024), 3) counterfactual knowledge (Meng et al., 2022), 4) conceptual…”

“For instance, MEMIT performs well on GPT-J-6B with Counterfact but poorly on ZsRE (0.92 vs. 0.20), and FT-L shows exceptionally high locality (0.99) on Mistral-7B-v0.”

“To investigate this, we train a model using the KV cache from all layers and evaluate it on 1000 instances from ZsRE (Levy et al., 2017).”

“RelationExtraction (Levy et al., 2017) Given labeled slot-filling examples, relations between entities are transformed into QA pairs using templates.”

“Reference [9] proposes a relation extraction model based on question-answering through prompt learning.”

“Beyond reforming RE as foundational linguistic tasks used for pre-training, some works convert RE into summarization [28], reading comprehension [17], and machine translation [33] with elaborate templates to conveniently induce the relevant knowledge contained in the upstream models that benefit RE.”

“We edit 10,000 samples from the Multi-COUNTERFACT dataset (Meng et al. 2022b), where the editing updates original correct answers to target incorrect answers.”

“Traditionally, ZSL has demonstrated versatility across tasks [39], including image classification [40], relation extraction [23], and node classification [37, 43].”

“Unlike other datasets such as ZsRE (Levy et al., 2017) or CounterFact (Meng et al., 2022a), which draw from known Wiki knowledge or focus on false facts respectively, we opt for RealTimeQA due to its alignment with real-world scenarios, offering a more fitting context for our knowledge updating needs.”

“For example, Levy et al.[19] regard ZeroRE as a slot-filling problem resolved by reading comprehension.”

“(3) In order to convert ConceptNet triples into natural language that the model can understand, this paper follows a common practice of mapping each relation in a triple to its natural language template, and filling the template with verbs and Prepositions [5].”

“The Zero-Shot dataset (Levy et al., 2017) contains 1,192 question templates spanning 120 Wikidata properties.”

“For example, Levy et al. [28] treat the parametrized query R(x,?) as a templated question in task of ralation extraction.”

“, 2020) on the zsRE dataset (Levy et al., 2017) and a DistilGPT-2 model (Sanh et al.”

“In addition to KGC models, relation extraction (RE) models (Levy et al., 2017) can also derive knowledge from text.”

“…in-domain reading comprehension datasets (Natural Questions [3], NewsQA [4] and SQuAD [5]) and three small out-of-domain reading comprehension datasets (RelationExtraction [6], DuoRC [7], RACE [8]) for training a QA system which will be evaluated on test examples from out-of-domain datasets.”

“[20] treat the relation extraction task as MRC task.”

“, 2017), and zero-shot relation extraction (RE) via reading comprehension (Levy et al., 2017).”

“For example, Levy et al. (2017) transformed the task of relation extraction to a QA task: each relation type R(x,y) can be parameterized as a question q(x) whose answer is y.”

“[34] achieve zero shot relation extraction by regarding the relation extraction task as a simple QA task.”

“We selected 300 random samples from the ZsRE dataset (Levy et al., 2017) as targets for the ROME (Meng et al., 2022a) editing method.”

“…we consider three categories of knowledge-intensive tasks: 1) multi-hop QA , including 2 https://www.wikipedia.org/ HotPotQA (HotPot) [46], MuSiQue [15], 2WikiMultiHopQA (WQA) [5], and StrategyQA [47]; 2) slot filling , using zsRE [48] and T-REx [49]; and 3) long-form QA , using ELI5 [50].”

“The MRQA dataset comprises 6 datasets for training and evaluation, namely HotpotQA (Yang et al., 2018), Natural Questions (Kwiatkowski et al., 2019), NewsQA (Trischler et al., 2017), SQuAD (Rajpurkar et al., 2016), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017), as well as 6 additional datasets for ou…”

“Some recent studies have applied MRC to event extraction, where certain research [3], [13], [14], [15] adopts manually designed, context-independent questions that often fail to provide sufficient contextual evidence for QA, leading to inadequate information interaction.”

“…of Dialog (Wei et al., 2018; Cho and May, 2020; Aliannejadi et al., 2021), IE (Mausam et al., 2012; Zlabinger et al., 2020; Radev et al., 2020), QA (Levy et al., 2017; Zhang et al., 2018b; Min et al., 2020), Sum (Henderson et al., 2014; Syed et al., 2020; Hasan et al., 2021) and SA (Sheng and…”

“Other similar works using indirect supervision can be found in [9] and [15], who study zero-shot relation extraction by transforming it into a machine comprehension and textual entailment problem respectively.”

“In nature, MRC models are dedicated for extracting NE mention spans regardless of flat or nested nature [21].”

“For the SelfCheckGPT, we follow the evaluation settings from prior research (Wang et al., 2024c), using perplexity (PPL) as the metric for evaluating reliability, with locality assessed in the same manner as for the ZsRE and excluding generalization.”

“…addition to this new benchmark, we evaluate the effectiveness and scalability of UltraEdit on four widely used model editing datasets, namely zsRE (Levy et al., 2017), FEVER (Thorne et al., 2018), WikiBigEdit (Thede et al., 2025), and UnKE (Deng et al., 2025), across six diverse models including…”

“ZsRE (Levy et al., 2017) and COUNTERFACT (Meng et al., 2023) focus on synthetic updates, while AToKE (Yin et al., 2024) and Chro-noEdit (Ge et al., 2024) evaluate temporal consistency but remain limited in scale and scope.”

“We evaluate this transferability on two non-QA tasks using the first 300 instances of each: (i) Fact Verification using the FEVER dataset [39], measured by accuracy; and (ii) Slot Filling using the Zero-Shot-RE dataset [23], measured by F1 score.”

“5000 on zsRE (Levy et al., 2017; Zhang et al., 2024)”

“We use ZsRE [21] and CounterFact [9] as controlled streams of factual updates.”

“The out-of-domain benchmarks include DROP, DuoRC, and RelationExtraction.”

“We conduct experiments on three benchmarks: ZsRE (Levy et al., 2017), Counter-Fact (CF) (Meng et al., 2022, 2023), and Wik-iBigEdit (Thede et al., 2025).”

“We evaluate knowledge editing performance us-ing the MQuAKE-Remastered (Zhong et al., 2025) and ZSRE (Levy et al., 2017) datasets, and use OpenHermesPreferences (Huang et al., 2024) as anchor data for continual preference optimization to suppress behavior-level distribution drift.”

“[26] G QA ✓ - - - WikiBigEdit[39] G QA/Wikifacts ✓ ✓ ✓ - MQuAKE[54] G QA(multi-hop) - ✓ - - DocTER[47] G Documents - - - - ThinkEval[4] G Graph - - - - UniEdit[8] G Graph - - - - MedEditBench[9] M QA - - ✓ - MedMKEB[50] M Multi-modal - - - - MedCF/MedFE[49] M QA/Explanation ✓ - - - TiEBe[3] G…”

“5-2B on ZsRE (Levy et al., 2017) with LoRA (Hu et al., 2021) of rank 8 and alpha 16.”

“We use 500 factual updates each from zsRE [14] and CounterFact [4], with three paraphrase probes per item, and Qwen3-0.”

“We evaluate end-to-end editing on CounterFact [5], ZSRE [20], and KnowEdit [21], covering factual replacement, question-answer rephrasing, and diverse knowledge domains.”

“Compared with the Re-edit baseline, our method provides better and more stable preservation of the remaining edits, especially in the Reverse-50 setting on the ZsRE dataset.”

“We evaluate KLOD on ZsRE (Levy et al., 2017) and CounterFact (Meng et al., 2022) under the Breadth-First optimization protocol of LocFT-BF (Yang et al., 2026).”

“Benchmarks commonly used to evaluate knowledge editing, such as ZsRE (Levy et al., 2017), CounterFact (Meng et al., 2022), MQuAKE (Zhong et al., 2023), and RippleEd-its (Cohen et al., 2024) typically evaluate localized updates to known facts, including counterfactual or outdated associations.”

“A large follow-up literature measures locality, generalization, and persistence trade-offs on behavioral benchmarks (CounterFact, zsRE, and derivatives: Meng et al., 2022; Levy et al., 2017).”

“Datasets & Models : Following the previous work LKS [22], we utilize the zsRE [38] question-answering dataset and the Counterfact dataset [18] for knowledge editing evaluation.”

“ZsRE (Levy et al., 2017) is a question-answering benchmark derived via back-translation, providing each example with a rephrased paraphrase (used to measure generalization) and an out-of-scope neighboring question (used to measure locality).”

“We evaluated the performance of ACKE on the ZsRE dataset (Levy et al., 2017) for ques - tion answering (QA) and the SelfCheckGPT dataset (Manakul et al., 2023) for hallucination correction.”

“We evaluate ORE on three widely used knowledge editing benchmarks: ZsRE (Levy et al., 2017), CounterFact (Meng et al., 2022), and Bi-ZsRE (Wang et al., 2024a).”

“Zero-shot Relation Extraction (ZsRE) [15] is a context-free question answering benchmark designed to evaluate factual knowledge storage and retrieval in language models.”

“5, IWCKEdit delivers improvements of 14.81%, 11.40%, and 11.23% in Efficacy, Generalization, and Specificity on multi-counterfact, and 27.18%, 27.98%, and 7.25% on ZsRE.”

“ZsRE tests relation extraction style edits [10], and MQuAKE-CF stresses multi-hop consequences after edits [28].”

“Table 2 shows a representative what -type instance from the forget split, drawn from the ZSRE source.”

“Architecture-agnostic techniques include quantization and low-rank decomposition (Levy et al., 2017).”

“Knowledge Editing : we apply Rank-One Model Editing (ROME) (Meng et al., 2022) to a Llama3-8B base model ( M 1 ) (Grattafiori et al., 2024), performing 10 sequential zsRE-derived (Levy et al., 2017) edits to create M 2 .”

“We evaluate HoReN on three benchmarks that progressively stress its core separation guarantee: ZsRE [13], the standard lifelong editing benchmark with semantically well-separated edit and locality queries; WikiBigEdit [25], a harder regime in which locality queries share subjects and relations with…”

“For fact-checking, slot filling, and dialogue, we use the knowledge sources provided by KILT (Petroni et al., 2021), and select FEVER (Thorne et al., 2018) for fact-checking, Zero-Shot RE (Levy et al., 2017) for slot filling, and Wizard of Wikipedia (Dinan et al., 2018) for dialogue.”