Zero-Shot Relation Extraction via Reading Comprehension

“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”

“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”

“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”

“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”

“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”

“For the task of identifying predicate-argument structures, QA-SRL (He et al., 2015) was proposed as an open schema for semantic roles, in which the relation between an argument and a predicate is expressed as a natural-language question containing the predicate (“Where was someone educated?”) whose…”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“Rockt¨aschel et al. (2015) and Demeester et al. (2016) use inference rules to predict hidden knowledge-base relations from observed natural-language relations.”

“Slot-Filling Data We use the WikiReading dataset (Hewlett et al., 2016) to collect labeled”

“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”

“When the type of facts (relations) are predefined, one can use crowdsourcing (Liu et al., 2016) or distant supervision (Hoffmann et al., 2011) to collect examples and train an extraction model for each relation type.”

“Verga et al. (2017) predict facts for entity pairs that were not observed in the original matrix; this is equivalent to extracting seen relation types with unseen entities (see Section 6.1).”

“Bronstein et al. (2015) explore a similar zero-shot setting for event-trigger identification, in which R N +1 is specified by a set of trigger words at test time.”

“Following the methodology of InfoboxQA (Morales et al., 2016), we generate negative examples by matching (for the same entity e ) a question q that pertains to one relation with a sentence s that expresses another relation.”

“For this task, several different question answering datasets (Dua et al., 2019; Saha et al., 2018; Lai et al., 2017; Levy et al., 2017; Kembhavi et al., 2017; Rajpurkar et al., 2016) are converted into a singular, extractive format.”

“For this task, several different question answering datasets (Dua et al., 2019; Saha et al., 2018; Lai et al., 2017; Levy et al., 2017; Kembhavi et al., 2017; Rajpurkar et al., 2016) are converted into a singular, extractive format.”

“The Zero-shot Relation Extraction dataset (Levy et al., 2017) contains negative examples generated with distant supervision.”

“…recently, mainly for neural machine translation (NMT) [Bahdanau et al. 2014; Luong et al. 2015; Vaswani et al. 2017], reading comprehension [Levy et al. 2017; Seo et al. 2016], image captioning [Xu et al. 2015], and more [Ba et al. 2014; Bahdanau et al. 2016; Chorowski et al. 2015; Mnih et…”

“[95] irst formulated relation extraction as a QA task.”

“, 2015), relation extraction (Levy et al., 2017), coreference resolution (Wu et al.”

“Second, in our main model, we map relation names to natural language (e.g., IsA → “is a”; HasSubevent→ “has subevent”) so the model can learn to represent these concepts with language, as opposed to learning a special embedding from scratch for each relation (Levy et al., 2017).”

“…and many classical neural architectures have been proposed, such as sequence tagging (Lample et al., 2016; Zheng et al., 2017; Lin et al., 2019), span classification (Sohrab and Miwa, 2018; Lin et al., 2018; Wad-den et al., 2019), and MRC (Levy et al., 2017; Li et al., 2020; Du and Cardie, 2020).”

“…2019), HotpotQA (Yang et al., 2018c), TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019); slot filling with T-REx (Elsahar et al., 2018), Zero Shot RE (Levy et al., 2017); entity disambiguation on AIDA CoNLL-YAGO, WNED-WIKI and WNED-CWEB; dialogue with Wizard of Wikipedia (Dinan et al., 2019).”

“For example, Levy et al. (2017) transformed the task of relation extraction to a QA task: each relation type R(x, y) can be parameterized as a question q(x) whose answer is y. For example, the relation EDUCATED-AT can be mapped to “Where did x study?”. Given a question q(x), if a non-null answer y can be extracted fro…”

“same property as in q and keeping their answer expressions. As an example, for the WIKI6 We use the WIKIDATA triples present in WIKIREADING. 7 We thus use a disjoint subset of WIKIREADING compared to Levy et al. (2017) to construct WIKIHOP. DATA property country, this would be the set fFrance;Russia;:::g. Graph traver…”

“Levy et al. (2017) extract relational facts from documents by answering questions using reading comprehension methods, where the questions are converted from entity-relation pairs.”

“…al., 2020) Symmetric 50,000 ccnews_title_text (Hamborg et al., 2017) Asymmetric 25,000 triviaqa (Joshi et al., 2017) Asymmetric 50,000 zero_shot_re (Levy et al., 2017) Asymmetric 15,000 flickr30k_captions (Young et al., 2014) Symmetric 25,000 xsum (Narayan et al., 2018) Asymmetric 10,000…”

“Levy et al. (2017) and Obamuyide and Vlachos (2018) study zero-shot relation extraction by converting it into a machine comprehension and textual entailment problem respectively.”

“Finally, it has been used in other problems related to NLP, such as entity extraction from web pages [117], fine-grained named entity typing [99], cross-lingual document retrieval [47], and relation extraction [85].”

“In the information extraction literature, Levy et al. (2017) propose the zero-shot relation extraction task and reduce the task to answering crowd-sourced reading comprehension questions.”

“As with SRL, we use a dataset that maps relations to a set of questions so that relation extraction can be treated as question answering: QA-ZRE (Levy et al., 2017).”

“[211] turn zero-shot RC into a reading comprehension problem to comprehend unseen labels by a template converter.”

“Note that there are also some works attempting to handle RC in a zero-shot scenario (Levy et al. 2017), incorporating extra information to classify relations never appearing in training sets.”

“Our work is highly inspired by Levy et al. (2017).”

“2) We also use the zsRE question-answering dataset (Levy et al., 2017).”

“Zero Shot RE (Levy et al., 2017) is a dataset designed to translate relation extraction into a reading comprehension problem.”

“The prominent benchmarks for evaluating factual KE are the Zero-Shot Relation Extraction (zsRE) (Levy et al., 2017; De Cao et al., 2021) and CounterFact (Meng et al., 2022). zsRE is a question-answering dataset for relation-specific queries, which includes human generated paraphrases that are used…”

“RelationExtraction (Levy et al., 2017) Given a slot-filling dataset, 5 relations among entities are systematically transformed into question-answer pairs using templates.”

“But different from our work, most of the above methods (Levy et al., 2017; Li et al., 2019b; FitzGerald et al., 2018; Du and Cardie, 2020) adopt human-designed, context-independent questions, which may not provide enough contextual evidence for question-answering.”

“, 2018), Zero Shot RE (Levy et al., 2017)) and filter it down to 760K good single-hop questions using this pipeline.”

“For the datasets, we performed counterfactual update experiments on two datasets: Zero-Shot Relation Extraction (zsRE) (Levy et al. 2017) and C OUNTER F ACT (Meng et al. 2022a).”

“Lu et al. (2022a) introduced a unified text-to-structure generation that can model different IE tasks universally.”

“Generative recommendation systems do not require the computation of ranking scores for each item followed by database indexing, but instead accomplish item recommendations through the direct generation of IDs.”

“…is a tendency to apply MRC on many NLP tasks, including named entity recognition (Li et al. 2020), entity relation extraction (Li et al. 2019b; Levy et al. 2017), and summarization (McCann et al. 2018), etc. Due to the advantages of MRC framework, we naturally transform ASTE into a multi-turn…”

“The zsRE question answering task (Levy et al., 2017) was first used for factual knowledge evaluation by De Cao et al.”

“, 2016), relation extraction (Verga et al., 2016; Levy et al., 2017), semantic parsing (Bapna et al.”

“Further studies extended LAMA by the automated discovery of prompts [25], finetuning LLMs for better probing [3,32,68], or a purely unsupervised way of probing knowledge from LMs [50].”

“…answering: NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018); slot filling: Zero Shot RE (Levy et al., 2017) and T-REx (Elsahar et al., 2018); entity linking: AIDA CoNLL-YAGO (Hoffart et al., 2011); dialogue: Wizard of Wikipedia (Dinan et…”

“Levy et al. (2017) and McCann et al. (2018) tried to formulate relation extraction as single-turn QA tasks.”

“The previous work of (Levy et al., 2017; Clark and Gardner, 2018) outputted a no-answer score depending on the probability of all answer spans.”

“18QA models are even used directly for relation extraction [1, 161, 168].”

“We use two datasets for evaluation, CounterFact (Meng et al., 2022) and zsRE (Levy et al., 2017).”

“…qa Kwiatkowski et al. 2019 kilt_trex qa/closed-book qa Elsahar et al. 2018 kilt_wow cg/dialogue Dinan et al. 2019 kilt_zsre qa/closed-book qa Levy et al. 2017 lama-conceptnet qa/closed-book qa Petroni et al. 2019, 2020 lama-google_re qa/closed-book qa Petroni et al. 2019, 2020 lama-squad…”

“For knowledge modification, we have selected the following four datasets: ZsRE [173], Wik-iBio [155], Wikidata recent [149], and Convsent [145].”

“For that we fine-tune a BART base model (Lewis et al., 2020) with a standard seq2seq objective on the Zero-Shot Relation Extraction (zsRE) dataset by Levy et al. (2017).”

“This prompt for QA-ZRE (Levy et al., 2017), a dataset for zero-shot relation extraction, shows how to manipulate strings and do conditional statements with Jinja.”

“[87] add an extra trainable bias to the confidence score of start and end position and apply SoftMax to the new score to obtain the probability distributions of no answer.”

“For example, Levy et al. [86] first formulated the RE task as a question answering (QA) task, where the relations are defined by natural-language question templates.”

“Following Levy et al. (2017), we introduce a trainable no-answer bias when computing softmax.”

“To address this problem, previous works (Levy et al., 2017; Clark and Gardner, 2018) predict a no-answer probability in addition to the distribution over answer spans, so as to jointly learn no-answer detection as well as answer extraction.”

“, [351, 361, 119]).”

“Zero-Shot Relation Slot-Filling (Levy et al., 2017) aims to predict the tail entity based on the provided head entity and relation, but also relies on other methods for entity d t ction.”

“ment), which allows the use of readily available question answering (or entailment) training data (Yin et al., 2019; Levy et al., 2017).”

“Levy et al. (2017) reduced relation extraction to answering simple reading comprehension questions, yielding models that generalize better in the zero-shot setting.”

“Furthermore, to cope with the wrong labeling problem caused by distant supervision, Besides, a few models (Levy et al., 2017; Qiu et al., 2018) borrowed the reading comprehension techniques to document-level RE.”

“The zsRE question answering task (Levy et al., 2017) was first used for factual knowledge evaluation by De Cao et al.”

“For the unanswerable questions, the form emphasized that they all needed to be incorrect but plausible (Levy et al. 2017).”

“…Over the past few years, neural models have achieved remarkable success in RE (Nguyen and Grishman, 2015b,a; Adel et al., 2016; Yin et al., 2017; Levy et al., 2017; Su et al., 2018; Song et al., 2018; Luo et al., 2019), in which the input representation usually comes from shallow neural…”

“The popular model editing dataset Zero-Shot Relation Extraction (Z S RE) (Levy et al., 2017) used in previous work (Cao et al., 2021; Meng et al., 2022; Yao et al., 2023) was adopted in our experiments.”

“For a given rule text and question pair, a negative example is generated by sampling a random question from the set of all possible questions, excluding the question itself and questions sourced from the same document using a methodology similar to the work of Levy et al. (2017).”

“from other resources may be necessary to solve a significant percentage of questions. Besides these standard tasks, we are aware that there is a trend of formalizing tasks such as relation extraction (Levy et al., 2017), word prediction (Chu et al., 2017), and judgment prediction (Long et al., 2018) as extractive or no…”

“These categories in our taxonomy are inspired from early reading comprehension datasets which introduced notions of unanswerable queries that models should identify [60, 42, 20, 80].”

“First, while the typical study (Levy et al., 2017) cannot perform zero-shot relation classification without putting more human effort on it, as they solve this problem via pre-defining question templates.”

“…capability of reading comprehension (Gardner et al., 2019) and many recent works have been implementing this strategy to test machines’ understanding of linguistic formalisms: He et al. (2015); Michael et al. (2018); Levy et al. (2017); Jia et al. (2018); Ning et al. (2020); Du and Cardie (2020).”

“We focus on using two slot filling datasets from the KILT benchmark (Petroni et al., 2021): T-REx (Elsahar et al., 2018) and zero-shot relation extraction (Levy et al., 2017).”

“Inspired by the current trend of formalizing NLP tasks into machine reading comprehension (MRC) tasks [19]–[23], we use BioBERT in the MRC framework to perform BioNER (referred to BioBERT-MRC).”

“• zsRE is a question answering (QA) dataset built upon zero-shot relation extraction (Levy et al. 2017).”

“In zero-shot relation extraction (RE), they reduced relation slot filling to an RC problem so as to build very large distant training data and improve zero-shot learning performance (Levy et al., 2017).”

“These experiments will be expanded to other models in future iterations of the paper.”

“, relation extraction is transformed to question answering (Levy et al., 2017; Abdou et al., 2019)); or (ii) meta-learning (Andrychowicz et al.”

“As a result, we select some data instances from the ZsRE and the C OUNTER F ACT dataset.”

“Specifi-cally, as shown in Figure 3, USM first verbalizes all label schemas (Levy et al. 2017; Li et al. 2020; Lu et al. 2022) and learns the schema-text joint embedding to build a shared label text semantic space.”

“…et al., 2015) PubMed Documents DROP (Dua et al., 2019) Wikipedia DuoRC (Saha et al., 2018) Wikipedia+IMDB RACE (Lai et al., 2017) English Exam RE (Levy et al., 2017) Wikipedia Textbook (Kembhavi et al., 2017) K12 Textbook For evaluation metrics, we use Mean Reciprocal Rank (MRR) as the criteria.”

“However, relation extraction has traditionally focused on extracting relations from sentences relying on entity linking systems to identify the subject/object and building models to learn the predicates in a sentence [3, 22].”

“, 2020), with some extensions allowing for unanswerable questions (Levy et al., 2017; Rajpurkar et al., 2018) or multiple answer spans (Dua et al.”

“Levy et al. (2017) translated the information extraction task into the reading comprehension task with good results.”

“However, relation extraction has traditionally focused on extracting relations from sentences relying on entity linking systems to identify the subject/object and building models to learn the predicates in a sentence [5, 26].”

“Finally, we systematically validate the mitigation capabilities of the two proposed defense strategies against different types of manipulated knowledge spread.”

“2019) and have shown promising results in zero shot prediction by leveraging their language understanding capabilities (Levy et al., 2017; Zhou et al., 2018; Yin et al., 2019).”

“Except for the RelationExtraction dataset, all the out-of-domain datasets only have a strong relationship with one or two in-domain datasets but are positioned far from the others.”

“ZSL has shown great potential in various NLP tasks, such as entity recognition [66], relation extraction [67], document classification [68], and text classification [65].”

“subject/object and building models to learn the predicates in a sentence [8, 23].”

“Specifically, for seq2seq models, we use the zsRE question-answering dataset (Levy et al., 2017) using question rephrasings generated by backtranslation as the equivalence neighborhood and train/val splits generated by De Cao et al.”

“shuffle into train/dev/test splits with 80/10/10% of the data (Levy et al., 2017).”

“The Bahdanau [22] or Luong [23] Attention has been widely adopted into neural machine translation [25], reading comprehension [26] and computer vision [27].”

“Levy et al. (2017) showed that a related and complementary task, that of entity/attribute relation extraction, can be reduced to a question answering problem.”

“, 2015) 2016 Parsing Seq2ASeq (M)LM (Choe and Charniak, 2016) 2017 Relation Extraction Class MRC (Levy et al., 2017) 2018 Text Summarization SeqLab Seq2Seq (McCann et al.”

“Beyond boosting the performance in a wide range of applications, attention mechanism is also playing a key role in improving the explainability of deep learning models, benefiting machine translation [19], machine reading comprehension [20], image captioning [21], and document classification [22].”

“Existing slot-filling tasks (Levy et al., 2017; Surdeanu, 2013) focus on a single-hop , static corpus setting, whereas our task considers a dynamic setting which requires the system to traverse the corpus.”

“While most approaches rephrase tasks as a language modeling problem, some use prompts to reformulate them as different tasks for which large amounts of training data are available (Levy et al., 2017; McCann et al., 2018; Yin et al., 2019; Sun et al., 2021; Sainz et al., 2021).”

“We develop three setups that follow the same procedure as (Levy et al., 2017) for Zero-Shot relation extraction to evaluate how our model generalizes to: 1) unseen predicates, 2) unseen sub-types and 3) unseen obj-types.”

“tracting relations that were not specified in advance, zero-shot RE (Levy et al., 2017) is proposed to invent new models to predict new relations.”