CoChat
Sign Up
PreprintCited by 1Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Retrieval-Augmented Generation-based Relation Extraction

Open access
Efeoglu, Sefika, Paschke, Adrian·2024·arXiv
DOI 10.48550/arxiv.2404.13397arXiv 2404.13397

Preprint — published as 10.1177/22104968251385519

Open this paper in CoChat — free accountOpen PDF

Abstract

Information Extraction (IE) is a transformative process that converts unstructured text data into a structured format by employing entity and relation extraction (RE) methodologies. The identification of the relation between a pair of entities plays a crucial role within this framework. Despite the existence of various techniques for relation extraction, their efficacy heavily relies on access to labeled data and substantial computational resources. In addressing these challenges, Large Language Models (LLMs) emerge as promising solutions; however, they might return hallucinating responses due to their own training data. To overcome these limitations, Retrieved-Augmented Generation-based Relation Extraction (RAG4RE) in this work is proposed, offering a pathway to enhance the performance of relation extraction tasks. This work evaluated the effectiveness of our RAG4RE approach utilizing different LLMs. Through the utilization of established benchmarks, such as TACRED, TACREV, Re-TACRED, and SemEval RE datasets, our aim is to comprehensively evaluate the efficacy of our RAG4RE approach. In particularly, we leverage prominent LLMs including Flan T5, Llama2, and Mistral in our investigation. The results of our study demonstrate that our RAG4RE approach surpasses performance of traditional RE approaches based solely on LLMs, particularly evident in the TACRED dataset and its variations. Furthermore, our approach exhibits remarkable performance compared to previous RE methodologies across both TACRED and TACREV datasets, underscoring its efficacy and potential for advancing RE tasks in natural language processing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.13397

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A critical paradigm of long-term memory optimisation is Retrieval-Augmented Generation (RAG), which incorporates relevant external memory into the reasoning process via retrieval.

    “A critical paradigm of long-term memory optimisation is Retrieval-Augmented Generation (RAG), which incorporates relevant external memory into the reasoning process via retrieval (Wang et al., 2023a; Efeoglu and Paschke, 2024; Gao et al., 2025c).”

    From A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (2025) · cites this paper as Efeoglu and Paschke, 2024

    The Introduction and §3.2 describe RAG retrieving relevant information from an embedding database and adding it to a prompt for the model. The paper does not characterize RAG as a paradigm of long-term memory optimisation; its focus is relation extraction. Page numbers are unavailable in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 26

  1. R. Grishman, Information Extraction, IEEE Expert 30(5) (2015), 8-15.
  2. C. Wang, X. Liu, Z. Chen, H. Hong, J. Tang and D. Song, DeepStruct: Pretraining of Language Models for Structure Prediction, in: Findings of the Association for Computational Linguistics: ACL 2022, S. Muresan, P. Nakov and A. Villavicencio, eds, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 803-823. doi:10.18653/v1/2022.findings-acl.67. https://aclanthology.org/2022.findings-acl.67 . · DOI
  3. Z. Chen, Z. Li, Y. Zeng, C. Zhang and H. Ma, GAP: A novel Generative context-Aware Prompt-tuning method for relation extraction, Expert Systems with Applications 248 (2024), 123478. doi:https://doi.org/10.1016/j.eswa.2024.123478. https://www.sciencedirect.com/science/article/pii/S0957417424003439. · DOI
  4. A.D. Cohen, S. Rosenman and Y. Goldberg, Relation classification as two-way span-prediction, arXiv preprint arXiv:2010.04829 (2020). · arXiv
  5. W. Zhou and M. Chen, An Improved Baseline for Sentence-level Relation Extraction, in: Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Pro- cessing (Volume 2: Short Papers), Y. He, H. Ji, S. Li, Y. Liu and C.-H. Chang, eds, Association for Computational Linguistics, Online only, 2022, pp. 161-168. https://aclanthology.org/2022.aacl-short.21.
  6. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel and D. Kiela, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Curran Associates Inc., Red Hook, NY, USA, 2020. ISBN 9781713829546.
  7. N. Mihindukulasooriya, S. Tiwari, C.F. Enguix and K. Lata, Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Gener- ation from Text, in: The Semantic Web -ISWC 2023, T.R. Payne, V. Presutti, G. Qi, M. Poveda-Villalón, G. Stoilos, L. Hollink, Z. Kaoudi, G. Cheng and J. Li, eds, Springer Nature Switzerland, Cham, 2023, pp. 247-265.
  8. R. Ahmad, M. Critelli, S. Efeoglu, E. Mancini, C. Ringwald, X. Zhang and A. Merono Penuela, Draw Me Like My Triples: Leveraging Generative AI for Wikidata Image Completion, 2023, The 4th Wikidata Workshop ; Conference date: 07-11-2023. https://wikidataworkshop.github.io/2023/.
  9. S. Hertling and H. Paulheim, OLaLa: Ontology matching with large language models, in: Proceedings of the 12th Knowledge Capture Conference 2023, 2023, pp. 131-139.
  10. Y. Zhang, V. Zhong, D. Chen, G. Angeli and C.D. Manning, Position-aware Attention and Supervised Data Improve Slot Filling, in: Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, M. Palmer, R. Hwa and S. Riedel, eds, Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp. 35-45. doi:10.18653/v1/D17-1004. https://aclanthology.org/D17-1004 . · DOI
  11. C. Alt, A. Gabryszak and L. Hennig, TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter and J. Tetreault, eds, Association for Computational Linguistics, Online, 2020, pp. 1558-1569. doi:10.18653/v1/2020.acl-main.142. https://aclanthology.org/2020.acl-main.142 . · DOI
  12. G. Stoica, E.A. Platanios and B. Poczos, Re-TACRED: Addressing Shortcomings of the TACRED Dataset, Proceed- ings of the AAAI Conference on Artificial Intelligence 35(15) (2021), 13843-13850. doi:10.1609/aaai.v35i15.17631. https://ojs.aaai.org/index.php/AAAI/article/view/17631. · DOI
  13. I. Hendrickx, S.N. Kim, Z. Kozareva, P. Nakov, D. Ó Séaghdha, S. Padó, M. Pennacchiotti, L. Romano and S. Szpakowicz, SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals, in: Proceedings of the 5th International Workshop on Semantic Evaluation, K. Erk and C. Strapparava, eds, Association for Computational Linguistics, Uppsala, Sweden, 2010, pp. 33-38. https://aclanthology.org/S10-1006.
  14. H.W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S.S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E.H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q.V. Le and J. Wei, Scaling Instruction-Finetuned Language Models, arXiv, 2022. doi:10.48550/ARXIV.2210.11416. https://arxiv.org/abs/2210.11416 . · DOI · Paper page
  15. H. Touvron, L. Martin, K.R. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D.M. Bikel, L. Blecher, C.C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A.S. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I.M. Kloumann, A.V. Korenev, P.S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E.M. Smith, R. Subramanian, X. Tan, B. Tang, R. Taylor, A. Williams, J.X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov and T. Scialom, Llama 2: Open Foundation and Fine- Tuned Chat Models, ArXiv abs/2307.09288 (2023). https://api.semanticscholar.org/CorpusID:259950998. · arXiv
  16. A.Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D.S. Chaplot, D.d.l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al., Mistral 7B, arXiv preprint arXiv:2310.06825 (2023). · arXiv
  17. S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang and X. Wu, Unifying large language models and knowledge graphs: A roadmap, IEEE Trans- actions on Knowledge and Data Engineering (2024).
  18. S. Pawar, G.K. Palshikar and P. Bhattacharyya, Relation Extraction : A Survey, arXiv, 2017.
  19. M. Aydar, O. Bozal and F. Ozbay, Neural relation extraction: a survey, arXiv, 2020.
  20. E. Agichtein and L. Gravano, Snowball: Extracting Relations from Large Plain-Text Collections, in: Proceedings of the Fifth ACM Confer- ence on Digital Libraries, DL '00, Association for Computing Machinery, New York, NY, USA, 2000, pp. 85-94-. ISBN 158113231X.
  21. Y.S. Kai Zhang Bernal Jiménez Gutiérrez, Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors, in: Findings of ACL, 2023.
  22. E. Melz, Enhancing LLM Intelligence with ARM-RAG: Auxiliary Rationale Memory for Retrieval Augmented Generation, arXiv e-prints (2023), arXiv-2311.
  23. Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun and H. Wang, Retrieval-augmented generation for large language models: A survey, arXiv preprint arXiv:2312.10997 (2023). · arXiv
  24. O. Ovadia, M. Brief, M. Mishaeli and O. Elisha, Fine-tuning or retrieval? comparing knowledge injection in llms, arXiv preprint arXiv:2312.05934 (2023). · arXiv
  25. N. Reimers and I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, in: Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3982-3992.
  26. W. Xiong, Y. Song, P. Wang and S. Li, Rationale-Enhanced Language Models are Better Continual Relation Learners, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino and K. Bali, eds, Association for Computational Linguistics, Singapore, 2023, pp. 15489-15497. doi:10.18653/v1/2023.emnlp-main.958. https://aclanthology.org/2023.emnlp-main.958. · DOI

Cited by

Sorted by citations

Authors
Efeoglu, S., & Paschke, A.
ORCID
Paschke, Adrian
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2404.13397
arXiv
2404.13397
OpenAlex
W4395064975
Cited by
1
References
0
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
relation extraction, retrieval-augmented generation, large language models, hallucinations, information extraction, TACRED, named entity recognition, Flan-T5, Llama 2, Mistral

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review