The DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index.
Zero-Shot Relation Extraction via Reading Comprehension
Open accessAbstract
We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot.This reduction has several advantages: we can (1) learn relationextraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training examples.Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relation types with high accuracy, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels, setting the bar for future work on this task.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ZsRE (Levy et al., 2017) is a context-free question-answering (QA) dataset generated by BART (Lewis et al., 2020) with human quality control, and it has been widely studied in the field of knowledge editing.”
From Knowledge Decoupling via Orthogonal Projection for Lifelong Editing of Large Language Models (2025) · cites this paper as Levy et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To facilitate comprehensive assessment of model editing capabilities, the dataset is augmented with two types of auxiliary samples: (1) paraphrased variants of the original question, which test the model's ability to generalize the update, and (2) unrelated but structurally similar questions, which assess the specificity of the edit by ensuring unrelated knowledge remains unaffected.”
From UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models (2025) · cites this paper as Levy et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“ZsRE (Zero-shot Relation Extraction) dataset (Levy et al., 2017) is a widely adopted benchmark for evaluating factual consistency and knowledge editing in language models.”
From UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models (2025) · cites this paper as Levy et al., 2017
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“where each record contains an editing statement x e , a paraphrase prompt x ′ e , and a locality prompt x loc .”
From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The ZsRE question-answering task [46] is extensively studied within the model editing literature [18, 19, 31, 15, 11]”
From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“Hyperparameters In our experiments, we initialized word embeddings with GloVe (Pennington et al., 2014), and did not fine-tune them.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 4×
“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
BackgroundMethodInfluentialTeaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“To support relation extraction, our dataset deviates from recent reading comprehension formulations (Hermann et al., 2015; Rajpurkar et al., 2016), and introduces negative examples – question-sentence pairs that have no answers ( A = ∅ ).”
MethodOpen Information Extraction from the Web
Etzioni, O., Banko, M., Cafarella, M. J., Soderland, S., & Weld, D. S. (2007) · CACM · doi:10.1145/1409360.1409378 · Cited by 2,620
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698 · cited 5×
“We start from the BiDAF model (Seo et al., 2016), whose input is two sequences of words: a sentence s and a question q.”
BackgroundMethodInfluentialSelf-citationEnd-to-End Relation Extraction using LSTMs on Sequences and Tree Structures
Miwa, M., & Bansal, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1105 · Cited by 1,438 · cited 1×
“Lastly, we retrain an off-the-shelf relation extraction system (Miwa and Bansal, 2016), which has shown promising results on a number of benchmarks.”
MethodRepresenting Text for Joint Embedding of Text and Knowledge Bases
Toutanova, K., Chen, D., Pantel, P., Poon, H., Choudhury, P., & Gamon, M. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1174 · Cited by 827 · cited 1×
“Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”
BackgroundSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678 · cited 4×
“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
BackgroundMethodInfluentialLarge-scale Simple Question Answering with Memory Networks
Bordes, A., Usunier, N., Chopra, S., & Weston, J. (2015) · arXiv · doi:10.48550/arxiv.1506.02075 · Cited by 528 · cited 2×
“The Simple QA dataset (Bordes et al., 2015) was created by annotating questions about individual Freebase facts (e.”
MethodWikidata: a new platform for collaborative data collection
Vrandečić, D. (2012) · The Web Conference · doi:10.1145/2187980.2188242 · Cited by 528 · cited 3×
“WikiReading was collected by aligning each Wikidata (Vrandeˇci´c, 2012) relation R ( e, a ) with the corresponding Wikipedia article D for the entity e , under the reasonable assumption that the relation can be derived from the article’s text.”
MethodDynamic Coattention Networks For Question AnsweringPreprint
Xiong, C., Zhong, V., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01604 · Cited by 504 · cited 2×
“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
BackgroundInjecting Logical Background Knowledge into Embeddings for Relation Extraction
Rocktäschel, T., Singh, S., & Riedel, S. (2015) · North American Chapter of the Association for Computational Linguistics · doi:10.3115/v1/n15-1118 · Cited by 281
Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language
He, L., Lewis, M., & Zettlemoyer, L. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1076 · Cited by 197 · cited 2×
“For the task of identifying predicate-argument structures, QA-SRL (He et al., 2015) was proposed as an open schema for semantic roles, in which the relation between an argument and a predicate is expressed as a natural-language question containing the predicate (“Where was someone educated?”) whose…”
BackgroundSelf-citationLifted Rule Injection for Relation Embeddings
Demeester, T., Rocktäschel, T., & Riedel, S. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1146 · Cited by 142 · cited 1×
“Rockt¨aschel et al. (2015) and Demeester et al. (2016) use inference rules to predict hidden knowledge-base relations from observed natural-language relations.”
BackgroundWikiReading: A Novel Large-scale Language Understanding Task over Wikipedia
Hewlett, D., Lacoste, A., Jones, L., Polosukhin, I., Fandrianto, A., Han, J. J., Kelcey, M., & Berthelot, D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1145 · Cited by 130 · cited 7×
“Slot-Filling Data We use the WikiReading dataset (Hewlett et al., 2016) to collect labeled”
BackgroundMethodInfluentialLearning Recurrent Span Representations for Extractive Question AnsweringPreprint
Lee, K., Salant, S., Kwiatkowski, T., Parikh, A. P., Das, D., & Berant, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01436 · Cited by 118 · cited 2×
“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
BackgroundMulti-Perspective Context Matching for Machine ComprehensionPreprint
Wang, Z., Mi, H., Hamza, W., & Florian, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04211 · Cited by 109 · cited 2×
“The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
BackgroundEffective Crowd Annotation for Relation Extraction
Liu, A., Soderland, S., Bragg, J., Lin, C. H., Ling, X., & Weld, D. S. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1104 · Cited by 68 · cited 1×
“When the type of facts (relations) are predefined, one can use crowdsourcing (Liu et al., 2016) or distant supervision (Hoffmann et al., 2011) to collect examples and train an extraction model for each relation type.”
MethodSeed-Based Event Trigger Labeling: How far can event descriptions get us?
Bronstein, O., Dagan, I., Li, Q., Ji, H., & Frank, A. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-2061 · Cited by 47 · cited 1×
“Bronstein et al. (2015) explore a similar zero-shot setting for event-trigger identification, in which R N +1 is specified by a set of trigger words at test time.”
BackgroundGeneralizing to Unseen Entities and Entity Pairs with Row-less Universal Schema
Verga, P. W., Neelakantan, A., & McCallum, A. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1058 · Cited by 41 · cited 1×
“Verga et al. (2017) predict facts for entity pairs that were not observed in the original matrix; this is equivalent to extracting seen relation types with unseen entities (see Section 6.1).”
BackgroundInfluentialLearning to Answer Questions from Wikipedia Infoboxes
Morales, A., Premtoon, V., Avery, C., Felshin, S., & Katz, B. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1199 · Cited by 16 · cited 3×
“Following the methodology of InfoboxQA (Morales et al., 2016), we generate negative examples by matching (for the same entity e ) a question q that pertains to one relation with a sentence s that expresses another relation.”
MethodInfluential
Research works printed but not linked: 3
- Morgan Kaufmann Publishers Inc., San Fran- cisco, CA, USA, IJCAI'07, pages 2670-2676. http://dl.acm.org/citation.cfm?id=1625275.1625705. 7 http://nlp.cs.washington.edu/zeroshot
- Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies- Volume 1. Association for Computational Linguis- tics, pages 541-550.
- Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation ex- traction with matrix factorization and universal schemas. In Proceedings of the 2013 Confer- ence of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. Association for Computa- tional Linguistics, Atlanta, Georgia, pages 74-84. http://www.aclweb.org/anthology/N13-1008.
Cited by
Sorted by citations
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
“For this task, several different question answering datasets (Dua et al., 2019; Saha et al., 2018; Lai et al., 2017; Levy et al., 2017; Kembhavi et al., 2017; Rajpurkar et al., 2016) are converted into a singular, extractive format.”
BackgroundThe Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
“For this task, several different question answering datasets (Dua et al., 2019; Saha et al., 2018; Lai et al., 2017; Levy et al., 2017; Kembhavi et al., 2017; Rajpurkar et al., 2016) are converted into a singular, extractive format.”
BackgroundKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
“The Zero-shot Relation Extraction dataset (Levy et al., 2017) contains negative examples generated with distant supervision.”
BackgroundMethodcode2vec: Learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 962
“…recently, mainly for neural machine translation (NMT) [Bahdanau et al. 2014; Luong et al. 2015; Vaswani et al. 2017], reading comprehension [Levy et al. 2017; Seo et al. 2016], image captioning [Xu et al. 2015], and more [Ba et al. 2014; Bahdanau et al. 2016; Chorowski et al. 2015; Mnih et…”
BackgroundMethodInfluentialSelf-citationRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
“[95] irst formulated relation extraction as a QA task.”
Method
Metadata
- Authors
- Levy, O., Seo, M., Choi, E., & Zettlemoyer, L.
- Affiliations
- Bar-Ilan University (IL)
- University of Washington (US)
- Venue
- Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)
- Published
- 2017
- Pages
- 333-342
- OpenAlex
- W2624677889
- Cited by
- 482
- References
- 22
- Citations by year
- 2026: 20 · 2025: 42 · 2024: 52 · 2023: 84 · 2022: 62 · 2021: 82 · 2020: 63 · 2019: 56 · 2018: 21
- FWCI
- 17.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Computational Natural Language Learning (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1252835, 1562364)
- Nvidia
- Tencent
- Defense Advanced Research Projects Agency (FA8750-13-2-0019, FA8750)
- Army Research Office (W911NF-16-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- zero-shot relation extraction, reading comprehension, natural language queries, distant supervision, zero-shot learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
