Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“To address this issue, zero-shot relation extraction (ZSRE) has been proposed [2], [3], where the model learns a limited set of known relations during training but is able to generalize to unseen relations during testing.”

“In this section, we present some cases to demonstrate how HiEdit and other competitive baseline methods (such as FT, AlphaEdit, and RLEdit) generate knowledge from earlier (Figure 10) and recent (Figure 9) edits after 2,000, 8,000, 10,000, and 20,000 sequential model edits on the ZsRE dataset and Llama-3-8B.”

“…evaluate the performance of our proposed method, we selected six representative datasets covering four types of KIQA tasks: Multi-hop QA (Hot-potQA (HoPo) [15], WikiMultiHopQA (WQA) [16], Strate-gyQA (SQA) [17]), Slot Filling (zsRE [18]), Fact Checking (FEVER [19]), and Long-form QA (ELI5 [20]).”

“ZsRE is derived from the original question–answer pairs in the Natural Questions corpus, where each instance is rewritten into a relational query paired with its corresponding factual answer.”

“EAC demonstrates strong performance in single-sequential editing; however, when extended to batch-sequential editing, the original approach exhibits clear limitations.”

“We evaluate on six benchmarks from the FlashRAG collection (Jin et al., 2025): Natural Questions (NQ) (Kwiatkowski et al., 2019), BoolQ (Clark et al., 2019), FEVER (Thorne et al., 2018), zsRE (Levy et al., 2017), HotpotQA (Yang et al., 2018), and SQuAD (Rajpurkar et al., 2016).”

“To evaluate the performance of knowledge editing, we employed three standard benchmarks: CounterFact (Meng et al., 2022a), ZsRE (Levy et al., 2017), and Wik-iFactDiff (Ammar Khodja et al., 2024).”

“We employ three datasets encompassing both knowledge insertion and knowledge modification tasks in our experiments: WikiData Counterfact (Cohen et al. 2024), WikiData Recent (Cohen et al. 2024), and ZsRE (Levy et al. 2017).”

“ZsRE [38] is a widely used benchmark for factual editing in a question-answering format.”

“We use three standard benchmarks for factual knowledge updating, including CounterFact (Meng et al., 2022), a dataset designed to distinguish between memo-rization and deep knowledge updates; ZsRE (Levy et al., 2017), a QA dataset where relations are defined by natural language; and MQuAKE-Remastered (Zhong et al., 20…”

“We conduct experiments on two widely used knowledge editing benchmarks, ZsRE (Levy et al., 2017) and Counterfact (Meng et al., 2022), together with our proposed EVK-Bench.”

“Bi-ZsRE (Wang et al., 2024b) extends ZsRE to Chinese–English pairs, enabling bilingual consistency evaluation.”

“As a pioneer work, Levy et al. reformulate RE as a reading comprehension task [14].”

“We select PROMPT, IKE, ROME, MEMIT, SERAC, and MELLO as baselines and evaluate them on our BidirectionalCounterfact benchmark, as well as the standard CounterFact[9], zsRE[6], and MQUAKE-CF-3K[22] datasets.”

“Levy et al. [6] first proposed the joint extraction method based on Machine Reading Comprehension (MRC) in 2017.”

“We evaluate our method using two widely adopted benchmarks: the ZsRE dataset Levy et al. (2017) and the WikiData counterfact dataset Meng et al. (2022a).”

“The most popular approach is to convert it into reading comprehension [3, 4] or textual entailment [5, 6] tasks, utilizing the external knowledge of these tasks to pre-train the matching ability of label semantics.”

“B Detailed information about datasets B.0.1 Generation-Based Datasets zsRE (Levy et al., 2017) The Zero-Shot Relation Extraction (zsRE) dataset is widely used in KE evaluations, particularly in QA tasks.”

“The empirical assessment of ReFEdit leverages two standard benchmarks: the Counterfact dataset [8] and the ZsRE dataset [22].”

“In the current study, Li et al. [1] and Levy et al. [7] proposed a new entity-relation extraction paradigm to handle the above task, and they transformed the task of entity-relation extraction into machine reading comprehension (MRC) involving understanding context information and predicting the…”

“Levy et al. [21] treat zero-shot relation extraction as a question answering task.”

“Other methods, such as Zero-Shot Relation Slot-Filling, rely on external entity detection modules [7].”

“Besides QA tasks we also conducted evaluations on multiple choice dataset TruthfulQA(Lin et al., 2021) and slot-filling dataset Zero-shot RE(Levy et al., 2017), results can be seen in Appendix A.3 Evaluation .”

“…Appendix A. Knowledge Editing Datasets We use two widely adopted context-free question answering (QA) datasets in knowledge editing research: WikiData counterfact [24] and Zero-Shot Relation Extraction ( ZsRE )[64], both of which serve as counterfactual benchmarks for modifying knowledge in LLMs.”

“We demonstrate their performance on CounterFact (Meng et al., 2022), ZsRE (Levy et al., 2017), and Wiki-recent (Zhang et al., 2024).”

“We conducted experiments using GPT2-XL [16], GPT-J [17] and LLaMA-2 [21] models on the zsRE [18] and CounterFact [19] datasets, respectively.”

“To meet the challenge of knowledge transfer, some previous research handles ZSRE by formulating it into a proxy task formulation, such as reading comprehension (Levy et al., 2017), textual entailment (Obamuyide and Vlachos, 2018), and attribute representation matching (Chen and Li, 2021).”

“Levy et al. [31] extract the unseen relationships by training a reading comprehension model to answer specific questions, which is framed as a slot filling task.”

“Machine Reading Comprehension (MRC) [1] is a task involving NLP and AI, aimed at enabling computers to understand and answer questions from given texts.”

“We use the QA-ZRE relation extraction benchmark [27].”

“In our experiments, we mainly conduct experiments on zsRE (Levy et al., 2017), which is one of the most prevalent Question Answering datasets extended and adopted for knowledge editing (De Cao et al., 2021; Mitchell et al., 2021).”

“More precisely, Levy et al. [15] state that each relation type 𝑅 ( 𝑥,𝑦 ) is translated to a natural-language question 𝑞𝑥 whose answer is 𝑦 .”

“As a case in point, a multi-level representation method introduced by Levy et al. [24] is rooted in an attention mechanism in the MRC model, while Yu et al. [25] Introduced a QANET method that used convolutional neural networks to capture sentence-level local dependencies and capture global…”

“Natural language processing [17, 23, 31, 37, 58], videos [11, 15, 16, 40, 69, 70], mobile and wireless security [54], emoji predictions [4, 73], human activity recognition [65], or neuroimaging data [2] are other applications.”

“In the early research on unanswerable questions, Levy et al. (2017) re-defined the BiDAF model (Seo et al., 2017) to allow it to output whether the given question is unanswerable.”

“In the direction of unanswerable question in MRC research, early efforts by Levy et al. (2017) involved redefining the BiDAF model (Seo et al., 2016), enabling the model to determine whether a provided question is unanswerable.”

“Levy et al.[2] (2017) treated the relational extraction task as a reading comprehension task.”

“Inspired by the current trend of casting NLP problems as question answering [5] [6], we propose a new framework to deal with REST API extraction problem.”

“As for our system, we transform the problem into the machine reading comprehension task (Levy et al. 2017; Zhao et al. 2020), and handle the issue of the open growth of non-predefined relations through the question answering mechanism.”

“As the baseline for relation extraction in (Levy et al., 2017), Random NE randomly chooses a name entity, which does not exist in a question, from a set of statements for each sample.”

“Zero-shot learning has been used to overcome this limit: for example, one can understand relation extraction as a question answering problem [7].”

“These results demonstrate that DipEdit consistently outperforms baseline methods across all metrics on both the CounterFact and ZsRE datasets.”

“All experiments use the ZsRE dataset under the standard knowledge-editing evaluation protocol.”

“We sample 200 data points from the ZsRE validation set as the remaining knowledge dataset for both tasks.”

“We conduct an ablation study to disentangle the respective contributions of spectral decoupling Table 2: Our method is integrated into existing knowledge-editing methods and compared against their original implementations, evaluated on CounterFact and ZsRE across LLaMA3 , GPT-J , and GPT2-XL .”

“Levy et al. [31] formulate RE as a machine reading comprehension task by transforming it into question-answering problems.”

“We present a quantitative evaluation of knowledge editing on the WikiData Counterfact , WikiData Recent , and ZsRE benchmarks, reporting the experimental results in Tables 1, 2, and 3.”

“Other benchmarks include ZsRE [19], which focuses on relation extraction; WikiData [6]; and ConvSent [26], each covering different domains and use cases, ranging from biomedical to general-purpose conversational contexts.”

“Following Meng et al. [23]’s work, we employ Coun-terFact ( CF ) [23] and Zero-Shot Relation Extraction ( zsRE ) [12, 26].”

“[122] only used shared-entity negatives for training relation extraction systems.”

“For QA, we apply a BART base model (Lewis et al., 2020) and the Zero-Shot Relation Extraction (zsRE) dataset (Levy et al., 2017).”

“For GPT-2, T5-XL, and GPT-J, we use the zero-shot Relation Extraction (zsRE) question answering dataset (Levy et al., 2017) with the identical train/val splits with De Cao et al. (2021); Mitchell et al. (2022), where the unrelated tuples ( x ui , y ui ) mi =1 are sampled from Natural Questions (NQ;…”

“It has learned that just before naming the album is a good point at which to initiate a memory read.”

“The popular model editing dataset Zero-Shot Relation Extraction (Z S RE) (Levy et al., 2017) used in previous work (Cao et al., 2021; Meng et al., 2022; Yao et al., 2023) was adopted in our experiments.”