Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“Zero-shot relation extraction approaches (Rockt ¨ aschel et al., 2015; Demeester et al., 2016; Levy et al., 2017; Oba-muyide and Vlachos, 2018) can extract unseen relations, but at lower performance levels, and are unable to continually exploit newly available supervision to improve performance…”

“Levy et al. (2017) connect nature language question with relation query to tackle zero shot relation extraction problem.”

“0 are posed by humans, and exhibit much more diversity and fidelity than those in other automatic constructed datasets [20, 27].”

“Omer et al. and Christo-pher et al. predicted an additional no-answer probability to detect unanswerable cases [22], [23].”

“ZSL has recently been adapted to many NLP tasks, including entity recognition [35], relation extraction [36], and text classification [37].”

“Note that the zsRE (Levy et al 2017) dataset and those subsequently listed were also employed for forgetting purposes by the same authors (see Table 2).”

“For slot-filling tasks, we evaluate our method on the Zero-Shot RE [21] and T-REx [12] datasets.”

“[12] reduces the zero-shot relation to the problem of answering simple reading comprehension questions to potentially extract facts of new types that were neither specified nor observed a priori.”

“…includes research investigating reformulating a task into a different format, such as reducing tasks to NLI (White et al., 2017; Wang et al., 2018) or reading comprehension (Levy et al., 2017; Wu et al., 2020), or even reducing all tasks to a single format (Kumar et al., 2016; McCann et al., 2018).”

“can be posed as question answering (Levy et al., 2017; McCann et al., 2018).”

“UW-RE [116] 32M 120 Precision, Recall, F1 1.”

“We extensively compare 6 different choices of A : TriviaQA (Joshi et al., 2017), NewsQA (Trischler et al., 2017), SQuAD (Ra-jpurkar et al., 2016), relation extraction (RE) dataset in QA format (QA-RE for short) (Levy et al., 2017), Large QA-SRL (FitzGerald et al., 2018), and QAMR (Michael et al., 2017).”

“Zero-Shot Relation Extraction Our work is re-lated to zero-shot RE (Levy et al., 2017).”

“Our dataset, Coupled Entity Prompts , is based on zSRE (Levy et al., 2017) and Counterfact (Meng et al., 2023).”

“The authors converted commonsense triplets from ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019) into readable sentences using a template-18 based method (Levy et al., 2017).”

“Levy et al. (2017) demonstrate that relation extraction can be solved using machine reading comprehension formulation.”

“Inspired by Levy et al. (2017) we reduce RC instances into QA instances.”

“According to the developers of the library, they used machine-translated SimpleQuestions and a dataset for zero-shot relation extraction [23] to train the model.”

“This is inspired by previous studies demonstrating the effectiveness of framing various information extraction tasks such as named entity recognition [11], relation extraction [12], and event extraction [13] as question answering (QA) by harnessing the well-developed machine reading comprehension models.”

“…representations that are more amenable for cross-domain (Liu et al., 2018; Li et al., 2019b; Du et al., 2020) or cross-task decision making (Johnson et al., 2017; Levy et al., 2017; Eriguchi et al., 2018; Lee et al., 2019; Wang et al.; Keung et al., 2019; Vernikos et al., 2020; Wang et al., 2022).”

“We evaluate RASE on fact-checking dataset FEVER (Thorne et al., 2018a) and Zero-Shot Relation Extraction (ZsRE) dataset (Levy et al., 2017).”

“There is a fundamental connection between machine reading comprehension and natural language inference [15], both tasks rely heavily on reasoning skills, and both are general because many NLP tasks can be cast into MRC [19], [20] or NLI [21].”

“, 2011), Zero-Shot RE (Levy et al., 2017), T-REx (Elsahar et al.”

“In this work, we employ the most widely adopted editing datasets ZsRE [38] and COUNTERFACT [16] as editing datasets across all experiments in the work.”

“Following the previous work (Mitchell et al., 2022a; Meng et al., 2022; 2023), this section defines each ZsRE metric given a LLM f θ , a knowledge fact prompt ( s i , r i ) , an edited target output o i , and the model’s original output o c i : • Efficacy : Efficacy is calculated as the average top-1 accuracy on the e…”

“, 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al.”

“…al. 2016; Rajpurkar, Jia, and Liang 2018), CNN/Daily Mail (Hermann et al. 2015), RACE (Lai et al. 2017), NarrativeQA (Kocisk´y et al. 2018), etc. Based on these large scale datasets, a lot of deep learning based models have been built, such as BiDAF (Seo et al. 2017), QANet (Yu et al. 2018), etc.”

“This includes, for example, Named Entity Recognition (NER, Li et al., 2020; Arora and Park, 2023), entity relation extraction (Levy et al., 2017; Li et al., 2019; Zhang et al., 2022) and slot filling (Gao et al., 2019).”

“Levy et al. (2017) (Vrandeˇci´c and Krötzsch, 2014), Free-base (Bollacker et al., 2008) and DBPedia (Auer et al., 2007) to obtain high-quality relations be-tween entities, and then mining data sources such as Wikipedia for texts mentioning both head and tail entities to construct training datasets…”

“RelationExtraction [82] is a multiturn QA task created from datasets, such as ACE04, ACE05, and CoNLL04, where entities and relations are converted into QA pairs using templates.”

“Rajpurkar et al. (2018) experimented on SQuAD 2.0 using the BiDAF-No-Answer (BNA) model proposed by Levy et al. (2017) and the DocumentQA No-Answer (DocQA) model from Clark and Gardner (2018).”

“Datasets: We experiment with SQuAD 1.1 (Ra-jpurkar et al., 2016) as the source dataset and the following 10 datasets as out-of-domain datasets: NewsQA (Trischler et al., 2017), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), Hot-potQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019),…”

“Most zero-shot studies for NLP (Wang et al., 2019) deal with machine translation, cross-lingual retrieval and entity/relation extraction (Levy et al., 2017; Pasupat and Liang, 2014), which are not suitable for our task, because they identify values that are explicitly mentioned rather than…”

“The query here can be directly formed by the word sequence of { e i , r ij } , or built from templates as in [21].”

“Unlike other knowledge-intensive NLP tasks such as fact checking (Thorne et al., 2018) and slot filling (Levy et al., 2017; Elsahar et al., 2018), knowledge-intensive generation tasks further require models to generate natural language response, which are more challenging.”

“In a disparate research, a zero-shot methodology has also been used to investigate relation linking [29].”

“…be considered to tackle KI tasks, such as large collections of question-answer pairs (Lewis et al., 2021; Huber et al., 2021), structured knowledge sources (Berant et al., 2013; Levy et al., 2017; Elsahar et al., 2018) or domain specific collections (Tsatsaronis et al., 2015; Saikh et al., 2021).”

“These models are then tested on four new datasets including RACE, TextbookQA, BioASQ, and RelationExtraction.”

“To this end, the task of zeroshot relation extraction (ZS-RE), which is to predict unseen relations with given entity mentions and their sentences, was proposed and has been investigated by some studies [44, 45, 46, 47, 48, 49].”

“Typical works extract the entity-relation triplet by multi-turn question answering (QA) structure (Levy et al. 2017; Li et al. 2019; Zhao et al. 2020).”

“The most related work to zero-shot learning for relation classification uses zero-shot learning to extract unseen relation types by listing questions that define the relation’s slot values [17].”

“Levy et al. (2017) have formulated the relation extraction task as a reading comprehension problem by associating one or more natural language questions with each relation.”

“Recently, there is a trend to cast non-QA NLP tasks, such as information extraction (Levy et al., 2017; Li et al., 2019, 2020), coreference resolution (Hou, 2020; Wu et al., 2020) and more (McCann et al., 2018) into MRC, which can achieve comparable or improved results.”

“In particular, by listing questions that define the relation’s slot values [2, 27].”

“…the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016), QA Zero-shot Relationship Extraction (ZRE; we use the 0 th split) (Levy et al., 2017), QA Semantic Role Labeling (SRL) (He et al., 2015), Commonsense Question Answering (CQA) (Tal-mor et al., 2018) and the two…”

“…et al., 2017; Yu et al., 2020a), span-based classification (Jiang et al., 2020; Yu et al., 2020b; Wang et al., 2021; Ye et al., 2022), MRC-based methods (Levy et al., 2017; Li et al., 2020; Liu et al., 2020) and generation-based methods (Zeng et al., 2018; Yan et al., 2021a; Hsu et al., 2022).”

“Levy et al. (2017) showed that relation extraction can be reduced to answer simple reading comprehension questions.”

“…format dataset, commonsense knowledge dataset includes PEP3k and 20Q (Porada et al., 2021; Gupta et al., 2023), factual knowledge includes ZsRE (Levy et al., 2017), CounterFact (Meng et al., 2022a), Fact Verification (Mitchell et al., 2022) , Calibration (Dong et al., 2022), MQuAKE (Zhong et…”

“The datasets used in our experiments are: (1) zsRE [Levy et al., 2017]: A question-answering dataset with 19,086 facts.”

“ZsRE (Levy et al., 2017) is a question answering dataset whose queries require models to answer the questions based on the information within the queries.”

“In subsequent research, zero-shot tasks were occasionally tackled from another problem’s perspective (Levy et al., 2017; Obamuyide and Vlachos, 2018), which implies training a model on a task for which annotated data is available and performing inference on a different one.”

“We used two lifelong model editing datasets: ZsRE (Levy et al., 2017) and SelfCheckGPT (Manakul et al., 2023).”

“(Levy et al. 2017; Cetoli 2020; Bragg et al. 2021) have learned this task by extending the machine reading-comprehension techniques.”

“In this work, we use ZsRE (Levy et al., 2017) and C OUNTER F ACT (Meng et al., 2022a), two widely used datasets, for our experiments.”

“Casting a natural language problem to a question-answering setting has previously been explored in a variety of tasks such as relation classification (Levy et al., 2017; Cohen et al., 2022), and semantic role labeling (He et al., 2015; Klein et al., 2022).”

“A variety of benchmark tasks on EQA have been released and at-tracted great interest (Rajpurkar et al., 2016; Fisch et al., 2019; Rajpurkar et al., 2018; Lai et al., 2017; Trischler et al., 2017; Levy et al., 2017; Joshi et al., 2017).”

“We improve upon the reduction of (Levy et al., 2017) by making it two-directional and demonstrate that by using span-based classification rather than objective one is more effective in a fully supervised setup: by converting RC datasets to the QA-inspired spanprediction form, we can train fully-supervised models for b…”

“Zero-shot entity and relation extraction (Levy et al., 2017; Chen and Li, 2021) aim to extract novel entities and their relations by transferring semantic knowledge from seen classes to unseen ones.”

“Two commonly used datasets for this purpose are ZsRE (Levy et al., 2017), a Question Answering (QA) dataset that employs question rephrasings generated through back-translation, and CounterFact (Meng et al., 2023a), a more challenging dataset that includes counterfacts starting with low scores in comparison to correct…”

“…[231] MultiHop-RAG [227] Long-form QA ELI5 [62] WebCPM [192] NarrativeQA (NQA) [118] Information Extraction Entity Linking ZESHEL [150] CoNLL [90] Relation Extraction T-REx [59] ZsRE [125] Text Understanding Text Classification TREC [136] SST-2 [220] Text Summarization WikiASP [85] XSum [172]”

“…performance using ROUGE-1 F1 and exact match (EM) scores on the following datasets: BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), Relation Extraction (RE) (Levy et al., 2017), and TextbookQA (TQA) (Kembhavi et al., 2017).”

“For the example of factual knowledge, this could be achieved by collecting reading times of sentences from datasets that contain information about real-world entities [e.g. 94], coupled with a questionnaire that gauges familiarity with the relevant piece of knowledge.”

“For RE, tasks are rephrased as question-answering (Levy et al., 2017), often injecting latent knowledge contained in relation labels into prompt construction (Chen et al., 2022) and iteratively fine-tuning prompts to enhance the model’s ability to focus on semantic cues (Son et al., 2022).”

“In the ield of Natural Language Processing (NLP), zero-shot learning has been utilized for tasks such as translation [14, 27], spoken language understanding [50], named entity recognition [28], and relation extraction [21, 43].”

“• Multilingual editing dataset : We build a machine translated multilingual knowledge editing dataset ( MzsRE ) in 12 languages: English, Czech, German, Dutch, Spanish, French, Por-tugues, Russian, Thai, Turkish, Vietnamese and Chinese using the zsRE testset (Levy et al., 2017).”

“Knowledge-Enhanced Dialogue Generation (WoW) Zero-Shot Slot Filling (ZSRE) Input Query: Do you ever enjoy listening to gospel music?”

“, 2021) and relation extraction (Levy et al., 2017; Obamuyide and Vlachos, 2018) have been”

“…outperforms methods that rely on weakly supervised QA models (Dhingra et al., 2018; Lewis et al., 2019) by up to 27.8 F1 points compared to the next-best method, while it can even outperform methods that rely on supervised QA models (Levy et al., 2017) by up to 12.4 F1 points in certain settings.”

“Since we allow querying the label of any training example, generalization to our test data is achievable, while out-of-domain generalization (Levy et al., 2017; Yogatama et al., 2019; Talmor and Berant, 2019) may be statistically impossible.”

“This includes BioASQ (Wiese et al., 2017), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017).”

“No-answer scoring [26], [27] and answer verification modules [5], [28] were used in some models to determine whether a question is unanswerable.”

“Datasets The vanilla knowledge editing is evaluated on two datasets: Zero-Shot Relation Extraction (zsRE) (Levy et al., 2017) and C OUNTER F ACT (Meng et al., 2022a).”

“This guarantees that the information to be updated does not appear in the original training sets of the LLMs. Specifically, we utilize two widely used context-free question-answering (QA) datasets, namely WikiData counterfact [8] and Zero-Shot Relation Extraction ( ZsRE ) [33].”

“In order to facilitate models’ evaluation based on both specificity and implication awareness, we introduce DepEdit , a new dataset for evaluating edits in language models using the zsRE dataset (Levy et al., 2017).”

“Levy et al. (2017) tackled this task by using reading comprehension models with predefined question templates.”

“Question & Answering Table 1 presents the results on the ZsRE [17] dataset for the question answering task, evaluated across a range of total edits from 1 to 1000 .”

“This approach has been widely applied to traditional machine learning models for classification tasks such as entity recognition [38], relation extraction [34], document classification [41], and text classification [45].”

“Recent works emphasize the importance of combining retrieval with robust re-ranking techniques to ensure that the retrieved context is both relevant and precise [45].”

“We conduct experiments on the knowledge-editing task of Zero-Shot Relation Extraction (zsRE) (Levy et al., 2017).”

“…perspective of knowledge editing, ensuring the benchmark comprehensively extraction and later adapted by De Cao et al. (2021) and Mitchell et al. (2021) for knowledge editing tasks. zsRE focuses on regular, well-defined knowledge items, making it an ideal baseline for evaluating the reliability…”

“Levy et al. (2017) addresses relation extraction by associating natural language questions with each relation slot.”

“We randomly select 100 samples from the ZsRE (Levy et al., 2017) as the editing dataset.”

“…al., 2016; Jia and Liang, 2017; Zhao et al., 2017; Ebrahimi et al., 2017; Jia et al., 2019b); The model lacks for domain adaptation abilities (Mou et al., 2016; Daumé III, 2009): a model trained on one domain can hardly generalize to new test distributions (Fisch et al., 2019; Levy et al., 2017).”

“[39] add an extra trainable bias to the confidence score of start and end position and apply SoftMax to the new score to obtain the probability distributions of no answer.”