CoChat
Sign Up
Cited by 261Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

REBEL: Relation Extraction By End-to-end Language generation

Open access
Huguet Cabot, Pere-Lluís, Navigli, Roberto·2021·Findings of the Association for Computational Linguistics: EMNLP 2021
DOI 10.18653/v1/2021.findings-emnlp.204
Open this paper in CoChat — free accountOpen PDF

Abstract

Extracting relation triplets from raw text is a crucial task in Information Extraction, enabling multiple applications such as populating or validating knowledge bases, factchecking, and other downstream tasks.However, it usually involves multiple-step pipelines that propagate errors or are limited to a small number of relation types.To overcome these issues, we propose the use of autoregressive seq2seq models.Such models have previously been shown to perform well not only in language generation, but also in NLU tasks such as Entity Linking, thanks to their framing as seq2seq tasks.In this paper, we show how Relation Extraction can be simplified by expressing triplets as a sequence of text and we present REBEL, a seq2seq model based on BART that performs end-to-end relation extraction for more than 200 different relation types.We show our model's flexibility by finetuning it on an array of Relation Extraction and Relation Classification benchmarks, with it attaining state-of-the-art performance in most of them.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.findings-emnlp.204

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is a massive difference between end-to-end information extraction systems like REBEL and LLMs.

    “we first note that there is a massive difference between end-to-end information extraction systems like REBEL and LLMs.”

    From HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) · cites this paper as [34]

    The Introduction identifies REBEL as an end-to-end relation-extraction system built by pre-training BART. That supports the example, but the paper does not compare REBEL with LLMs or establish a “massive difference” between them. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • REBEL generates around half the number of nodes and edges compared to both GPT-3.5 Turbo and both Llama models.

    “compared to both GPT-3.5 Turbo and both Llama models, REBEL generates around half the number of nodes and edges.”

    From HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) · cites this paper as [34]

    The Experimental Setup (§4) says REBEL generates relation triplets, and the Relation Extraction results (§5.1) report precision, recall, and F1. Our copy does not report numbers of generated nodes or edges or compare them with GPT-3.5 Turbo or Llama (searched for those models and node/edge counts). (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 43 · Cited by 261

Research works printed but not linked: 37

  1. Heike Adel and Hinrich Schütze. 2017. Global nor- malization of convolutional neural networks for joint entity and relation classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1723-1729, Copenhagen, Denmark. Association for Computa- tional Linguistics. · DOI
  2. Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: A thorough eval- uation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1558- 1569, Online. Association for Computational Lin- guistics. · DOI
  3. Michele Bevilacqua, Rexhina Blloshmi, and Roberto Navigli. 2021. One spring to rule them both: Sym- metric amr semantic parsing and generation with- out a complex pipeline. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12564- 12573.
  4. Michele Bevilacqua, Marco Maru, and Roberto Nav- igli. 2020. Generationary or "how we went beyond word sense inventories and learned to gloss". In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7207-7221, Online. Association for Computa- tional Linguistics. · DOI
  5. Rexhina Blloshmi, Simone Conia, Rocco Tripodi, and Roberto Navigli. 2021. Generating senses and roles: An end-to-end model for dependency-and span- based semantic role labeling. In Proceedings of the Thirtieth International Joint Conference on Artifi- cial Intelligence, IJCAI-21, pages 3786-3793. Inter- national Joint Conferences on Artificial Intelligence Organization. Main Track. · DOI
  6. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Represen- tations.
  7. Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Pro- ceedings of the 9th International Conference on Se- mantic Systems, I-SEMANTICS '13, page 121-124, New York, NY, USA. Association for Computing Machinery. · DOI
  8. Markus Eberts and Adrian Ulges. 2020. Span-based joint entity and relation extraction with transformer pre-training. In ECAI, pages 2006-2013.
  9. Markus Eberts and Adrian Ulges. 2021. An end-to-end model for entity-level relation extraction using multi- instance learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3650-3660, Online. Association for Computational Linguistics.
  10. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  11. Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural net- work for joint entity and relation extraction. In Pro- ceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Techni- cal Papers, pages 2537-2547, Osaka, Japan. The COLING 2016 Organizing Committee.
  12. Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug- related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885-892. Text Mining and Natural Language Processing in Pharmacogenomics. · DOI
  13. Arzoo Katiyar and Claire Cardie. 2017. Going out on a limb: Joint extraction of entity mentions and re- lations without dependency trees. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 917-928, Vancouver, Canada. Association for Computational Linguistics. · DOI
  14. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
  15. Makoto Miwa and Mohit Bansal. 2016. End-to-end re- lation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105-1116, Berlin, Germany. Association for Computational Linguis- tics. · DOI
  16. Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table repre- sentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1858-1869, Doha, Qatar. Associa- tion for Computational Linguistics. · DOI
  17. Tapas Nayak and Hwee Tou Ng. 2020. Effective mod- eling of encoder-decoder architecture for joint entity and relation extraction. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8528- 8535. · DOI
  18. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. In 9th Inter- national Conference on Learning Representations, ICLR 2021.
  19. Sachin Pawar, Pushpak Bhattacharyya, and Girish Pal- shikar. 2017. End-to-end relation extraction using neural networks and Markov Logic Networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics: Volume 1, Long Papers, pages 818-827, Valencia, Spain. Association for Computational Lin- guistics.
  20. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
  21. Sebastian Riedel, Limin Yao, and Andrew McCal- lum. 2010. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases, pages 148-163, Berlin, Heidelberg. Springer Berlin Heidelberg. · DOI
  22. Dan Roth and Wen-tau Yih. 2004. A linear program- ming formulation for global inference in natural lan- guage tasks. In Proceedings of the Eighth Confer- ence on Computational Natural Language Learn- ing (CoNLL-2004) at HLT-NAACL 2004, pages 1-8, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
  23. George Stoica, Emmanouil Antonios Platanios, and Barnabas Poczos. 2021. Re-tacred: Addressing shortcomings of the tacred dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13843-13850.
  24. Bruno Taillé, Vincent Guigue, Geoffrey Scoutheeten, and Patrick Gallinari. 2020. Let's Stop Incorrect Comparisons in End-to-end Relation Extraction! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3689-3701, Online. Association for Computa- tional Linguistics. · DOI
  25. Jue Wang and Wei Lu. 2020. Two are better than one: Joint entity and relation extraction with table- sequence encoders. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1706-1721, Online. As- sociation for Computational Linguistics. · DOI
  26. Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. TPLinker: Single-stage joint extraction of entities and relations through token pair linking. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 1572-1582, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics. · DOI
  27. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity- aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 6442-6454, On- line. Association for Computational Linguistics. · DOI
  28. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777, Florence, Italy. Association for Computational Lin- guistics. · DOI · Paper page
  29. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via con- volutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344, Dublin, Ireland. Dublin City Uni- versity and Association for Computational Linguis- tics.
  30. Daojian Zeng, Haoran Zhang, and Qianying Liu. 2020. Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning. In The Thirty-Fourth AAAI Conference on Artificial In- telligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Ed- ucational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9507-9514. AAAI Press.
  31. Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu, Shengping Liu, and Jun Zhao. 2019. Learning the extraction order of multiple relational facts in a sen- tence with reinforcement learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 367-377, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  32. Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 506-514, Melbourne, Aus- tralia. Association for Computational Linguistics. · DOI
  33. Ranran Haoran Zhang, Qianying Liu, Aysa Xuemo Fan, Heng Ji, Daojian Zeng, Fei Cheng, Daisuke Kawahara, and Sadao Kurohashi. 2020. Minimize exposure bias of Seq2Seq models in joint entity and relation extraction. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pages 236-246, Online. Association for Computa- tional Linguistics. · DOI
  34. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor An- geli, and Christopher D. Manning. 2017. Position- aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics. · DOI
  35. Suncong Zheng, Yuexing Hao, Dongyuan Lu, Hongyun Bao, Jiaming Xu, Hongwei Hao, and Bo Xu. 2017. Joint entity and relation extraction based on a hybrid neural network. Neurocomput- ing, 257:59-66. Machine Learning and Signal Processing for Big Multimedia Analysis. · DOI
  36. Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pages 207-212, Berlin, Germany. Association for Compu- tational Linguistics. · DOI
  37. Wenxuan Zhou and Muhao Chen. 2021. An im- proved baseline for sentence-level relation extrac- tion. CoRR, abs/2102.01373. · arXiv

Authors
Huguet Cabot, P.-L., & Navigli, R.
Venue
Findings of the Association for Computational Linguistics: EMNLP 2021
Publisher
Association for Computational Linguistics
Published
2021
Pages
2370-2381
DOI
10.18653/v1/2021.findings-emnlp.204
OpenAlex
W3214342214
Cited by
261
References
43
Citations by year
2026: 47 · 2025: 62 · 2024: 66 · 2023: 68 · 2022: 18
FWCI
17.79
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by-nc-nd)
Funders
  • European Commission (860801, H2020-EU.1.3.1, H2020)
  • Horizon 2020 Framework Programme (H2020-EU.1.3.1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
relation extraction, BART, information extraction, relation classification, knowledge base construction, fact-checking, rebel groups

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review