Revisiting Relation Extraction in the era of Large Language Models
Open accessAbstract
Relation extraction (RE) is the core NLP task of inferring semantic relationships between entities from text.Standard supervised RE techniques entail training modules to tag tokens comprising entity spans and then predict the relationship between them.Recent work has instead treated the problem as a sequence-tosequence task, linearizing relations between entities as target strings to be generated conditioned on the input.Here we push the limits of this approach, using larger language models (GPT-3 and Flan-T5 large) than considered in prior work and evaluating their performance on standard RE tasks under varying levels of supervision.We address issues inherent to evaluating generative approaches to RE by doing human evaluations, in lieu of relying on exact matching.Under this refined evaluation, we find that: (1) Few-shot prompting with GPT-3 achieves near SOTA performance, i.e., roughly equivalent to existing fully supervised models; (2) Flan-T5 is not as capable in the fewshot setting, but supervising and fine-tuning it with Chain-of-Thought (CoT) style explanations (generated via GPT-3) yields SOTA results.We release this model as a new baseline for RE tasks 1 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This dependency is a shared limitation of agentic memory systems that rely on LLM-based structural inference, as they are susceptible to extraction errors and hallucinations.
“This dependency is a shared limitation of agentic memory systems that rely on LLM-based structural inference, as they are susceptible to extraction errors and hallucinations (Pan et al., 2024; Xi et al., 2025; Wadhwa et al., 2023).”
From MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents (2026) · cites this paper as Wadhwa et al., 2023
The Few-Shot RE with Flan-T5 section reports non-conforming outputs and out-of-domain relations, supporting the point that LLM-based relation extraction can make errors. The paper does not examine agentic memory systems or establish that hallucinations are a shared limitation of those systems. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,953 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Support vector machine learning for interdependent and structured output spaces
Tsochantaridis, I., Hofmann, T., Joachims, T., & Altun, Y. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015341 · Cited by 1,493 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,342 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · see citing works
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
DocRED: A Large-Scale Document-Level Relation Extraction Dataset
Yao, Y., Ye, D., Li, P., Han, X., Lin, Y., Liu, Z., Liu, Z., Huang, L., Zhou, J., & Sun, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1074 · Cited by 550 · see citing works
Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism
Zeng, X., Zeng, D., He, S., Liu, K., & Zhao, J. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1047 · Cited by 549 · see citing works
Unified Structure Generation for Universal Information Extraction
Lu, Y., Liu, Q., Dai, D., Xiao, X., Lin, H., Han, X., Sun, L., & Wu, H. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.395 · Cited by 481 · see citing works
Gurulingappa, H., Rajput, A. M., Roberts, A., Fluck, J., Hofmann‐Apitius, M., & Toldo, L. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.04.008 · Cited by 447 · see citing works
A Linear Programming Formulation for Global Inference in Natural Language Tasks
Roth, D., & Yih, W.-T. (2004) · Cited by 441 · see citing works
Span-Based Joint Entity and Relation Extraction with Transformer Pre-Training
Eberts, M., & Ulges, A. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200321 · Cited by 413 · see citing works
REBEL: Relation Extraction By End-to-end Language generation
Huguet Cabot, P.-L., & Navigli, R. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.204 · Cited by 261 · see citing works
Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction
Nayak, T. K., & Ng, H. T. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i05.6374 · Cited by 245 · see citing works
Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders
Wang, J., & Lu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.133 · Cited by 242 · see citing works
Want To Reduce Labeling Cost? GPT-3 Can Help
Wang, S., Liu, Y., Xu, Y., Zhu, C., & Zeng, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.354 · Cited by 208 · see citing works
CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning
Zeng, D., Zhang, R. H., & Liu, Q. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6495 · Cited by 201 · see citing works
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 120 · see citing works
ChemProt: a disease chemical biology database
Taboureau, O., Nielsen, S. K., Audouze, K., Weinhold, N., Edsgärd, D., Roque, F. S., Kouskoumvekaki, I. A., Bora, A., Curpan, R. F., Jensen, T. S., et al. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq906 · Cited by 105 · see citing works
DeepStruct: Pretraining of Language Models for Structure Prediction
Wang, C., Liu, X., Chen, Z., Hong, H., Tang, J., & Song, D. X. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.67 · Cited by 92 · see citing works
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot GeneralizationPreprint
Xu, H., Chen, Y., Du, Y., Shao, N., Wang, Y.-G., Li, H., & Yang, Z.-L. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2201.06910 · Cited by 77 · see citing works
An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning
Eberts, M., & Ulges, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.319 · Cited by 67 · see citing works
ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization
Xu, H., Chen, Y., Du, Y., Shao, N., Yanggang, W., Li, H., & Yang, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.312 · Cited by 39 · see citing works
Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction!
Taillé, B., Guigue, V., Scoutheeten, G., & Gallinari, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.301 · Cited by 38 · see citing works
Improving In-Context Few-Shot Learning via Self-Supervised Training
Chen, M., Du, J., Pasunuru, R., Mihaylov, T., Iyer, S. V., Stoyanov, V., & Kozareva, Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.260 · Cited by 28 · see citing works
CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild
Yao, Y., Du, J., Lin, Y., Li, P., Liu, Z., Zhou, J., & Sun, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.366 · Cited by 19 · see citing works
CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task LearningPreprint
Zeng, D., Zhang, R. H., & Liu, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.10438 · Cited by 11 · see citing works
Span-based Joint Entity and Relation Extraction with Transformer Pre-training
Eberts, M., & Ulges, A. (2019) · arXiv · doi:10.48550/arxiv.1909.07755
Flagged first, then by citations
Research works printed but not linked: 2
- Brown Tom, Mann Benjamin, Ryder Nick, Subbiah Melanie, Jared D Kaplan Prafulla Dhariwal, Neelakantan Arvind, Shyam Pranav, Sastry Girish, Askell Amanda, Agarwal Sandhini, Herbert-Voss Ariel, Krueger Gretchen, Henighan Tom, Child Rewon, Ramesh Aditya, Ziegler Daniel, Wu Jeffrey, Winter Clemens, Hesse Chris, Chen Mark, Sigler Eric, Litwin Mateusz, Gray Scott, Chess Benjamin, Clark Jack, Berner Christopher, Sam McCandlish Alec Radford, Sutskever Ilya, and Amodei Dario. 2020a. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877– 1901. Curran Associates, Inc.
- Paolini Giovanni, Athiwaratkun Ben, Krone Jason, Ma Jie, Achille Alessandro, ANUBHAI RISHITA, Santos Cicero Nogueira dos, Xiang Bing, and Soatto Stefano. 2021. Structured prediction as translation between augmented natural languages. In International Conference on Learning Representations.
Cited by
Sorted by citations
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Miao, J., Thongprayoon, C., Suppadungsuk, S., Valencia, O. A. G., & Cheungpasitporn, W. (2024) · Medicina · doi:10.3390/medicina60030445 · Cited by 176 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
Dennstädt, F., Zink, J., Putora, P. M., Hastings, J., & Čihorić, N. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02575-4 · Cited by 165 · see citing works
A survey of large language models for cyber threat detection
Chen, Y., Cui, M., Wang, D., Cao, Y., Yang, P., Jiang, B., Lü, Z., & Liu, B. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104016 · Cited by 156 · see citing works
Ma, Y., Cao, Y., Hong, Y. S., & Sun, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.710 · Cited by 154 · see citing works
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106 · see citing works
Chain of Thought Utilization in Large Language Models and Application in Nephrology
Miao, J., Thongprayoon, C., Suppadungsuk, S., Krisanapan, P., Radhakrishnan, Y., & Cheungpasitporn, W. (2024) · Medicina · doi:10.3390/medicina60010148 · Cited by 108 · see citing works
A review of AI edge devices and lightweight CNN and LLM deployment
Sun, K., Wang, X., Miao, X., & Zhao, Q. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128791 · Cited by 76 · see citing works
Actionable Cyber Threat Intelligence Using Knowledge Graphs and Large Language Models
Fieblinger, R., Alam, T., & Rastogi, N. (2024) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw61312.2024.00018 · Cited by 66 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
Fan, Z., & Chen, C. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103646 · Cited by 50 · see citing works
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 49 · see citing works
Enhancing supply chain visibility with knowledge graphs and large language models
AlMahri, S., Xu, L., & Brintrup, A. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2575841 · Cited by 58 · see citing works
Jain, R., Sojitra, D., Acharya, A., Saha, S., Jatowt, A., & Dandapat, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.418 · Cited by 34 · see citing works
LEAP: LLM instruction-example adaptive prompting framework for biomedical relation extraction
Zhou, H., Li, M., Xiao, Y., Yang, H., & Zhang, R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae147 · Cited by 42 · see citing works
An LLM-assisted ETL pipeline to build a high-quality knowledge graph of the Italian legislation
Colombo, A., Bernasconi, A., & Ceri, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104082 · Cited by 34 · see citing works
Li, J., Jia, Z., & Zheng, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.334 · Cited by 32 · see citing works
Šuster, S., Baldwin, T., & Verspoor, K. M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1749 · Cited by 30 · see citing works
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27 · see citing works
Retrieval-Augmented Code Generation for Universal Information Extraction
Guo, Y., Li, Z., Jin, X., Liu, Y., Zeng, Y., Liu, W., Li, X., Pan, Y., Bai, L., Guo, J., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_3 · Cited by 46 · see citing works
Barron, R., Grantcharov, V., Wanna, S., Eren, M. E., Bhattarai, M., Solovyev, N., Tompkins, G., Nicholas, C., Rasmussen, K. Ø., Matuszek, C., et al. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00258 · Cited by 31 · see citing works
KLLMs4Rec: Knowledge graph-enhanced LLMs sentiment extraction for personalized recommendations
Cui, Y., Wang, K., Yu, H., Guo, X., & Cao, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127430 · Cited by 26 · see citing works
Testing prompt engineering methods for knowledge extraction from text
Polat, F., Tiddi, I., & Groth, P. (2024) · Semantic Web · doi:10.3233/sw-243719 · Cited by 26 · see citing works
Retrieval And Structuring Augmented Generation with Large Language Models
Jiang, P., Ouyang, S., Jiao, Y., Zhong, M., Tian, R., & Han, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736557 · Cited by 22 · see citing works
Information Extraction in Low-Resource Scenarios: Survey and Perspective
Deng, S., Ma, Y., Zhang, N., Cao, Y., & Hooi, B. (2024) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/ickg63256.2024.00013 · Cited by 21 · see citing works
Are Large Language Models Reliable Argument Quality Annotators?
Mirzakhmedova, N., Gohsen, M., Chang, C. H., & Stein, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63536-6_8 · Cited by 21 · see citing works
Core Concept Identification in Educational Resources via Knowledge Graphs and Large Language Models
Reales, D., Manrique, R., & Grévisse, C. (2024) · SN Computer Science · doi:10.1007/s42979-024-03341-y · Cited by 17 · see citing works
A survey on cutting-edge relation extraction techniques based on language models
Diaz-Garcia, J. A., & Amador, J. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11280-0 · Cited by 23 · see citing works
LLM-IE: a python package for biomedical generative information extraction with large language models
Hsu, E., & Roberts, K. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf012 · Cited by 16 · see citing works
InstructIE: A Bilingual Instruction-based Information Extraction Dataset
Gui, H., Qiao, S., Zhang, J., Ye, H., Sun, M., Liang, L., Pan, J. Z., Chen, H., & Zhang, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77847-6_4 · Cited by 15 · see citing works
LTACL: long-tail awareness contrastive learning for distantly supervised relation extraction
Yan, T., Zhang, X., & Luo, Z. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01226-w · Cited by 14 · see citing works
Dietrich, J., & Hollstein, A. (2024) · Drug Safety · doi:10.1007/s40264-024-01499-1 · Cited by 17 · see citing works
Wu, H., Zhang, Y., Han, Z., Hou, Y., Wang, L., Liu, S., Gong, Q., & Ge, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113057 · Cited by 17 · see citing works
GEOSatDB: global civil earth observation satellite semantic database
Lin, M., Jin, M., Li, J., & Bai, Y. (2024) · Big Earth Data · doi:10.1080/20964471.2024.2331992 · Cited by 13 · see citing works
A review on the reliability of knowledge graph: from a knowledge representation learning perspective
Yang, Y., Chen, J., & Xiang, Y. (2024) · World Wide Web · doi:10.1007/s11280-024-01316-w · Cited by 12 · see citing works
Text-to-structure interpretation of user requests in BIM interaction
Wei, Y., Li, X., & Petzold, F. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106119 · Cited by 12 · see citing works
PIVOINE: Instruction Tuning for Open-world Entity Profiling
Lu, K., Pan, X., Song, K., Zhang, H., Yu, D., & Chen, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1009 · Cited by 12 · see citing works
Enhancing Relation Extraction from Biomedical Texts by Large Language Models
Asada, M., & Fukuda, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60615-1_1 · Cited by 12 · see citing works
From document-centric to data-centric public service provision
Konstantinidis, I., Kapantai, E., Michailidis, A., Deligiannis, A. P., Berberidis, C., Magnisalis, I. D., & Peristeras, V. (2024) · Digital Government Research and Practice · doi:10.1145/3676279 · Cited by 17 · see citing works
A closer look at ChatGPT's role in concept map generation for education
Schicchi, D., Limongelli, C., Monteleone, V., & Taibi, D. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2025.2497110 · Cited by 11 · see citing works
Zhou, B., & Li, K. (2025) · Geosciences · doi:10.3390/geosciences15100382 · Cited by 11 · see citing works
Automatically learning linguistic structures for entity relation extraction
Yang, W., Chen, Y., Xu, J., Qin, Y., & Chen, P. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103904 · Cited by 11 · see citing works
LLM Instruction-Example Adaptive Prompting (LEAP) Framework for Clinical Relation ExtractionPreprint
Zhou, H., Li, M., Xiao, Y., Yang, H. Q., & Zhang, R. (2023) · medRxiv · doi:10.1101/2023.12.15.23300059 · Cited by 11 · see citing works
Li, Y., Viswaroopan, D., He, W., Li, J., Zuo, X., Xu, H., & Tao, C. (2025) · Research Square · doi:10.21203/rs.3.rs-6201919/v1 · Cited by 11 · see citing works
RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extraction
Meng, S., Hu, X., Liu, A., Li, S., Ma, F., Yang, Y., & Wen, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.316 · Cited by 10 · see citing works
Kwon, M., Bang, J., Hwang, S., Jang, J., & Lee, W. (2025) · Electronics · doi:10.3390/electronics14040659 · Cited by 10 · see citing works
Few-shot relational triple extraction with hierarchical prototype optimization
Gao, C., Zhang, X., Jin, Z. G., Shang, W., Ma, Y., Li, L., Ding, Z., & Liang, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110779 · Cited by 9 · see citing works
Towards Generating High-Quality Knowledge Graphs by Leveraging Large Language Models
Ezzabady, M. K., Ieng, F., Khorashadizadeh, H., Benamara, F., Groppe, S., & Sahri, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_31 · Cited by 9 · see citing works
Large language models can extract metadata for annotation of human neuroimaging publications
Turner, M. D., Appaji, A., Rakib, N. A., Golnari, P., Rajasekar, A., Anitha Rathnam, K., V, Sahoo, S. S., Wang, Y., Wang, L., & Turner, J. A. (2025) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2025.1609077 · Cited by 9 · see citing works
LLM With Relation Classifier for Document-Level Relation Extraction
Li, X., Chen, K., Long, Y., & Zhang, M. (2026) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2026.3668580 · Cited by 9 · see citing works
Enhancing the coverage of SemRep using a relation classification approach
Ming, S., Zhang, R., & Halil Kilicoglu, B. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104658 · Cited by 7 · see citing works
TCohPrompt: task-coherent prompt-oriented fine-tuning for relation extraction
Long, J., Yin, Z., Liu, C., & Huang, W. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01563-4 · Cited by 8 · see citing works
Liu, W., Xiao, Y., Cheng, S., Zeng, D., Zhou, L., Kong, W., Zhang, M., & Chen, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126099 · Cited by 9 · see citing works
DIRel: Joint relational triple extraction through dual implicit relation
Zhang, L. F., & Zheng, N. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129374 · Cited by 8 · see citing works
A Study of Biomedical Relation Extraction Using GPT Models.
Zhang, J., Wibert, M., Zhou, H., Peng, X., Chen, Q., Keloth, V. K., Hu, Y., Zhang, R., Xu, H., & Raja, K. (2024) · PubMed Central · Cited by 7 · see citing works
Han, P., Liang, G., & Wang, Y. (2025) · Electronics · doi:10.3390/electronics14030593 · Cited by 7 · see citing works
Efficient unified information extraction model based on large language models
Zhang, X., Cai, S., Shen, X.-R., Yang, H. Q., Hu, W., & Zhang, Y. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113302 · Cited by 7 · see citing works
Bhasuran, B., Manoharan, S., Oviya, I. R., Murugesan, G., Prabahar, A., & Raja, K. (2024) · Biomedicines · doi:10.3390/biomedicines12071535 · Cited by 7 · see citing works
Semantically Guided Representation Learning For Action Anticipation
Diko, A., Avola, D., Prenkaj, B., Fontana, F., & Cinque, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73390-1_26 · Cited by 6 · see citing works
Ruixiang, L., Qiu-Jun, D., Xianhui, L., Chenglin, Z., & Weidong, Z. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05969-y · Cited by 6 · see citing works
Detecting Linguistic Indicators for Stereotype Assessment with Large Language Models
Görge, R., Mock, M., & Allende‐Cid, H. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732181 · Cited by 6 · see citing works
Inkeraction: An Interaction Modality Powered by Ink Recognition and Synthesis
Shi, L., Campbell, R., Chi, P., Cirimele, M., Cleron, M., Climer, K., Fleming, C. Q., Ganti, A., Gervais, P., Gonnet, P., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642498 · Cited by 6 · see citing works
Sarol, M. J., Hong, G., Guerra, E., & Halil Kilicoglu, B. (2024) · Database · doi:10.1093/database/baae079 · Cited by 6 · see citing works
Abisheka, P., Deisy, C., & Sharmila, P. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102257 · Cited by 6 · see citing works
Deep mining the textual gold in relation extraction
Sharma, T., & Emmert‐Streib, F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11042-4 · Cited by 6 · see citing works
A discrete convolutional network for entity relation extraction
Yang, W., Qin, Y., Wang, K., Hu, Y., Huang, R., & Chen, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2024.107117 · Cited by 6 · see citing works
A Knowledge Extraction Framework on Cyber Threat Reports with Enhanced Security Profiles
Cai, Y., Qiu, J., Zhang, F., Li, Q., & Chen, L. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729880 · Cited by 6 · see citing works
Task-Oriented Dynamic Knowledge Distillation for Continuous Few-Shot Relation Extraction
Yang, H., Zhang, X., Gao, C., Shang, W., Du, K., Cai, W., Wang, D., & Li, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113914 · Cited by 5 · see citing works
AGLHunter: Automated Threat Hunting Using In-Context Learning-Enhanced LLM
Cui, M., Jiang, Z., Yao, Y., Ma, C., He, Q., Yang, P., & Feng, H. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033364 · Cited by 5 · see citing works
Automated Mining of Structured Knowledge from Text in the Era of Large Language Models
Zhang, Y., Zhong, M., Ouyang, S., Jiao, Y., Zhou, S., Ding, L., & Han, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671469 · Cited by 6 · see citing works
Document-level relation extraction with Double graph guidance for long-tailed distributions
Liu, B., & Qi, G. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110237 · Cited by 5 · see citing works
Integrating Large Language Models and Knowledge Graphs for Next-level AGI
Luo, L., Yang, C., Kharlamov, E., & Pan, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715866 · Cited by 7 · see citing works
Large Language Models for Few-Shot Automatic Term Extraction
Banerjee, S., Chakravarthi, B. R., & McCrae, J. P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_10 · Cited by 5 · see citing works
Çetinkaya, A. (2025) · Information · doi:10.3390/info16110960 · Cited by 4 · see citing works
Swarup, A., Bhandarkar, A., Dizon-Paradis, O. P., Wilson, R., & Woodard, D. L. (2024) · IEEE Access · doi:10.1109/access.2024.3494737 · Cited by 5 · see citing works
Semantics-enabled biomedical literature analytics
Halil Kilicoglu, B., Ensan, F., McInnes, B. T., & Wang, L. L. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104588 · Cited by 4 · see citing works
EE-LCE: An Event Extraction Framework Based on LLM-Generated CoT Explanation
Yu, Y., Wang, Y., Ma, Y., Li, J., Lu, K., Huang, Z., & Chua, T. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5492-2_3 · Cited by 5 · see citing works
Nair, P. C., Gupta, D., & Devi, B. I. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.502 · Cited by 4 · see citing works
Roll, D. S., Kurt, Z., Li, Y.-L., & Woo, W.-L. (2025) · Italian National Conference on Sensors · doi:10.3390/s25113352 · Cited by 5 · see citing works
A survey of Few-Shot Relation Extraction combining meta-learning with prompt learning
Hang, T., Wu, W., Feng, J., Djigal, H., & Huang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130534 · Cited by 4 · see citing works
Nachtegael, C., Stefani, J. D., & Lenaerts, T. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292356 · Cited by 4 · see citing works
Reliable Data Generation and Selection for Low-Resource Relation Extraction
Yu, J., Wang, X., & Chen, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29915 · Cited by 4 · see citing works
Pan, D., Sun, Y., Xu, B., Li, J., Yang, Z., Luo, L., Lin, H., & Wang, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3578781 · Cited by 3 · see citing works
Towards scalable and cross-lingual specialist language models for oncology
Rohanian, M., Mehra, T., Miglino, N., Nooralahzadeh, F., Krauthammer, M. O., & WICKI, A. N. (2025) · Scientific Reports · doi:10.1038/s41598-025-19282-2 · Cited by 3 · see citing works
XLORE 3: A Large-Scale Multilingual Knowledge Graph from Heterogeneous Wiki Knowledge Resources
Zeng, K., Jin, H., Lv, X., Zhu, F., Hou, L., Zhang, Y., Pang, F., Qi, Y., Liu, D., Li, J., et al. (2024) · ACM Transactions on Information Systems · doi:10.1145/3660521 · Cited by 3 · see citing works
A homogeneous multimodality sentence representation for relation extraction
Wang, K., Chen, Y., Yang, W., & Qin, Y. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102968 · Cited by 3 · see citing works
Plant-Disease Relation Model through BERT-BiLSTM-CRF Approach
Riyanto, S., Sitanggang, I. S., Djatna, T., & Atikah, T. D. (2024) · Indonesian Journal of Electrical Engineering and Informatics (IJEEI) · doi:10.52549/ijeei.v12i1.5154 · Cited by 3 · see citing works
Effective Sentence-Level Relation Extraction Model Using Entity-Centric Dependency Tree
Park, S., & Kim, H. (2022) · PeerJ Computer Science · doi:10.2139/ssrn.4272160 · Cited by 4 · see citing works
Zhao, D., Han, D., Yuan, Y., Ning, B., Li, M., He, Z., & Song, S. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681012 · Cited by 3 · see citing works
How Does Memorization Impact LLMs' Social Reasoning? An Assessment using Seen and Unseen Queries
Amirizaniani, M., Sivachenko, M., Lavergne, A., Shah, C. S., & Mashhadi, A. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703576 · Cited by 4 · see citing works
MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents
Jiang, D., Li, Y., Li, G., & Li, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1709 · Cited by 2 · see citing works
Privacy Requirements Acquisition and Analysis
Hosseini, M. B., & Breaux, T. D. (2025) · Springer eBooks · doi:10.1007/978-3-031-73143-3_9 · Cited by 2 · see citing works
Enhancing Zero-Shot Relation Extraction through Staged Interaction with Large Language Models
Zhang, Y., Duan, P., Yang, Y., & Xiong, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887575 · Cited by 2 · see citing works
Knowledge Enhancement with LLMs for Few-Shot Medical Relation Extraction
Zhang, K., Li, Y., Wu, P., Li, Y., Zhang, C., & Zan, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-3752-2_10 · Cited by 2 · see citing works
RelCheck: Improving Relation Extraction with Ontology-Guided and LLM-Based Validation
Ourekouch, M., Koulali, M.-A., & Erradi, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94575-5_24 · Cited by 2 · see citing works
Multilingual Relation Extraction: A Survey
Ali, M., Speck, R., Zahera, H. M., Saleem, M., Moussallem, D., & Ngomo, A.-C. N. (2025) · IEEE Access · doi:10.1109/access.2025.3604258 · Cited by 2 · see citing works
Can an LLM reproduce an expert knowledge base in neuroeducation?
Bukina, T. V., Khramova, M. V., Smirnov, N. M., Kurkin, S., & Hramov, A. E. (2026) · The European Physical Journal Special Topics · doi:10.1140/epjs/s11734-026-02534-1 · Cited by 2 · see citing works
PotatoG-DKB: a potato gene-disease knowledge base mined from biological literature
Xie, C., Gao, J., Chen, J., & Zhao, X. (2024) · PeerJ · doi:10.7717/peerj.18202 · Cited by 2 · see citing works
- Authors
- Wadhwa, S., Amir, S., & Wallace, B.
- Affiliations
- Northeastern University (US)
- ORCID
- Wallace, Byron
- Venue
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Volume
- 2023
- Pages
- 15566-15589
- ISSN
- 0736-587X (ISSN-L)
- PMC
- PMC10482322
- OpenAlex
- W4385571451
- Cited by
- 207
- References
- 32
- Citations by year
- 2026: 53 · 2025: 74 · 2024: 65 · 2023: 14 · 2022: 1
- FWCI
- 36.50
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1750978)
- National Institutes of Health (5r01lm012086-08)
- U.S. National Library of Medicine (R01-LM012086)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- relation extraction, large language models, GPT-3, Flan-T5, few-shot prompting, chain-of-thought prompting, human evaluation, sequence-to-sequence generation, entity relationships, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
