The DOI matches its Crossref record; 177 of the 177 references it cites were found in a registry or an index.
Large language models for generative information extraction: a survey
Open accessAbstract
Information Extraction (IE) aims to extract structural knowledge from plain natural language texts. Recently, generative Large Language Models (LLMs) have demonstrated remarkable capabilities in text understanding and generation. As a result, numerous works have been proposed to integrate LLMs for IE tasks based on a generative paradigm. To conduct a comprehensive systematic review and exploration of LLM efforts for IE tasks, in this study, we survey the most recent advancements in this field. We first present an extensive overview by categorizing these works in terms of various IE subtasks and techniques, and then we empirically analyze the most advanced methods and discover the emerging trend of IE tasks with LLMs. Based on a thorough review conducted, we identify several insights in technique and promising research directions that deserve further exploration in future studies. We maintain a public repository and consistently update related works and resources on GitHub (LLM4IE repository).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Within structured tool invocation approaches, the language model implicitly performs additional tasks such as information extraction and named entity recognition (NER). This process involves mapping fragments of unstructured user input onto the fields of a structured tool input schema [8].”
From A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue (2026) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 2×
“It entails utilizing task-specific instructions, known as prompts, to guide the behavior of model [13, 213, 214].”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2×
“It entails utilizing task-specific instructions, known as prompts, to guide the behavior of model [13, 213, 214].”
MethodLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 1×
“In IE, the extraction targets exhibit intricate structures where entities are presented as span structures (string structures) and relationships are represented as triple structures [4].”
Proceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Research works printed but not linked: 130
- Srihari R K, Li W. Information extraction supported question answering. In: Proceedings of the 8th Text REtrieval Conference. 1999
- Jia P, Liu Y, Zhao X, Li X, Hao C, Wang S, Yin D. MILL: mutual verification with large language models for zero-shot query expansion. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 2498–2518
- Huang J, She Q, Jiang W, Wu H, Hao Y, Xu T, Wu F. QDMR-based planning-and-solving prompting for complex reasoning tasks. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 13395–13406
- Lou J, Lu Y, Dai D, Jia W, Lin H, Han X, Sun L, Wu H. Universal information extraction as unified semantic matching. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 13318–13326
- Sainz O, García-Ferrero I, Agerri R, de Lacalle O L, Rigau G, Agirre E. GoLLIE: annotation guidelines improve zero-shot information-extraction. In: Proceedings of the ICLR 2024. 2024
- Paolini G, Athiwaratkun B, Krone J, Ma J, Achille A, Anubhai R, dos Santos C N, Xiang B, Soatto S. Structured prediction as translation between augmented natural languages. In: Proceedings of the 9th International Conference on Learning Representations. 2021
- Li P, Sun T, Tang Q, Yan H, Wu Y, Huang X, Qiu X. CodeIE: large code generation models are better few-shot information extractors. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 15339–15353
- Wang X, Li S, Ji H. Code4Struct: code generation for few-shot event structure prediction. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 3640–3663
- Xu X, Zhu Y, Wang X, Zhang N. How to unleash the power of large language models for few-shot relation extraction? In: Proceedings of the 4th Workshop on Simple and Efficient Natural Language Processing. 2023, 190–200
- Foppiano L, Lambard G, Amagasa T, Ishii M. Mining experimental data from materials science literature with large language models: an evaluation study. Science and Technology of Advanced Materials: Methods, 2024, 4(1): 2356506
- Feng Y, Pratapa A, Mortensen D. Calibrated seq2seq models for efficient and generalizable ultra-fine entity typing. In: Proceedings of the Findings of the Association for Computational Linguistics. 2023, 15550–15560
- Zhang S, Shen Y, Tan Z, Wu Y, Lu W. De-bias for generative extraction in unified NER task. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 808–818
- Wang L, Li R, Yan Y, Yan Y, Wang S, Wu W, Xu W. InstructionNER: a multi-task instruction-based generative framework for few-shot NER. 2022, arXiv preprint arXiv: 2203.03903 · arXiv
- Xia Y, Zhao Y, Wu W, Li S. Debiasing generative named entity recognition by calibrating sequence likelihood. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 1137–1148
- Chen X, Li L, Qiao S, Zhang N, Tan C, Jiang Y, Huang F, Chen H. One model for all domains: collaborative domain-prefix tuning for cross-domain NER. In: Proceedings of the 32nd International Joint Conference on Artificial Intelligence. 2023, 559
- Xie T, Li Q, Zhang Y, Liu Z, Wang H. Self-improving for zero-shot named entity recognition with large language models. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 583–593
- Zhang X, Tan M, Zhang J, Zhu W. NAG-NER: a unified nonautoregressive generation framework for various NER tasks. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 676–686
- Chen J, Lu Y, Lin H, Lou J, Jia W, Dai D, Wu H, Cao B, Han X, Sun L. Learning in-context learning for named entity recognition. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 13661–13675
- Tang X, Wang J, Su Q. Small language model is a good guide for large language model in Chinese entity relation extraction. 2024, arXiv preprint arXiv: 2402.14373 · arXiv
- Kim S, Seo K, Chae H, Yeo J, Lee D. VerifiNER: verification-augmented NER via knowledge-grounded reasoning with large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 2441–2461
- Lu J, Yang Z, Wang Y, Liu X, Namee B M, Huang C. PaDeLLMNER: parallel decoding in large language models for named entity recognition. 2024, arXiv preprint arXiv: 2402.04838 · arXiv
- Liu J, Wang J, Huang H, Zhang R, Yang M, Zhao T. Improving LLM-based health information extraction with in-context learning. In: Proceedings of the 9th China Health Information Processing Conference. 2024, 49–59
- Wu C, Ke W, Wang P, Luo Z, Li G, Chen W. ConsistNER: towards instructive NER demonstrations for LLMs with the consistency of ontology and context. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 19234–19242
- Zaratiana U, Tomeh N, Holat P, Charnois T. GliNER: generalist model for named entity recognition using bidirectional transformer. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 5364–5376
- Munnangi M, Feldman S, Wallace B, Amir S, Hope T, Naik A. On-the-fly definition augmentation of LLMs for biomedical NER. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 3833–3854
- Yan F, Yu P, Chen X. LTNER: Large language model tagging for named entity recognition with contextualized entity marking. 2024, arXiv preprint arXiv: 2404.05624 · DOI
- Jiang G, Luo Z, Shi Y, Wang D, Liang J, Yang D. ToNER: type-oriented named entity recognition with generative language model. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 16251–16262
- Jiang G, Ding Z, Shi Y, Yang D. P-ICL: point in-context learning for named entity recognition with large language models. 2024, arXiv preprint arXiv: 2405.04960 · arXiv
- Xie T, Zhang J, Zhang Y, Liang Y, Li Q, Wang H. Retrieval augmented instruction tuning for open ner with large language models. 2024, arXiv preprint arXiv:2406.17305 · arXiv
- Ye J, Xu N, Wang Y, Zhou J, Zhang Q, Gui T, Huang X. LLM-DA: data augmentation via large language models for few-shot named entity recognition. 2024, arXiv preprint arXiv: 2402.14568 · arXiv
- Ma M D, Wang X, Kung P N, Brantingham P J, Peng N, Wang W. STAR: boosting low-resource information extraction by structure-to-text data generation with large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 18751–18759
- Li B, Yu D, Ye W, Zhang J, Zhang S. Sequence generation with label augmentation for relation extraction. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 13043–13050
- Qi J, Ji K, Wang X, Yu J, Zeng K, Hou L, Li J, Xu B. Mastering the task of open information extraction with large language models and consistent reasoning environment. 2023, arXiv preprint arXiv: 2310.10590 · arXiv
- Li G, Wang P, Liu J, Guo Y, Ji K, Shang Z, Xu Z. Meta in-context learning makes large language models better zero and few-shot relation extractors. In: Proceedings of the 33rd International Joint Conference on Artificial Intelligence. 2024
- Shi Z, Luo H. CRE-LLM: a domain-specific Chinese relation extraction framework with fine-tuned large language model. 2024, arXiv preprint arXiv: 2404.18085 · arXiv
- Li G, Wang P, Ke W, Guo Y, Ji K, Shang Z, Liu J, Xu Z. Recall, retrieve and reason: towards better in-context relation extraction. In: Proceedings of the 33rd International Joint Conference on Artificial Intelligence. 2024
- Li G, Xu Z, Shang Z, Liu J, Ji K, Guo Y. Empirical analysis of dialogue relation extraction with large language models. In: Proceedings of the 33rd International Joint Conference on Artificial Intelligence. 2024
- Efeoglu S, Paschke A. Retrieval-augmented generation-based relation extraction. 2024, arXiv preprint arXiv: 2404.13397 · arXiv
- Ozyurt Y, Feuerriegel S, Zhang C. In-context few-shot relation extraction via pre-trained language models. 2023, arXiv preprint arXiv: 2310.11085 · arXiv
- Xue L, Zhang D, Dong Y, Tang J. AutoRE: document-level relation extraction with large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 211–220
- Liu Y, Peng X, Du T, Yin J, Liu W, Zhang X. ERA-CoT: improving chain-of-thought through entity relationship analysis. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 8780–8794
- Li G, Ke W, Wang P, Xu Z, Ji K, Liu J, Shang Z, Luo Q. Unlocking instructive in-context learning with tabular prompting for relational triple extraction. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024
- Ding Z, Huang W, Liang J, Xiao Y, Yang D. Improving recall of large language models: a model collaboration approach for relational triple extraction. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 8890–8901
- Ni X, Li P, Li H. Unified text structuralization with instruction-tuned language models. 2023, arXiv preprint arXiv: 2303.14956 · arXiv
- Zaratiana U, Tomeh N, Holat P, Charnois T. An autoregressive text-to-graph framework for joint entity and relation extraction. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 19477–19487
- Peng L, Wang Z, Yao F, Wang Z, Shang J. MetaIE: distilling a meta model from LLM for all kinds of information extraction tasks. 2024, arXiv preprint arXiv: 2404.00457 · arXiv
- Atuhurra J, Dujohn S C, Kamigaito H, Shindo H, Watanabe T. Distilling named entity recognition models for endangered species from large language models. 2024, arXiv preprint arXiv: 2403.15430 · arXiv
- Tang X, Su Q, Wang J, Deng Z. CHisIEC: an information extraction corpus for ancient Chinese history. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 3192–3202
- Cai Z, Kung P N, Suvarna A, Ma M, Bansal H, Chang B, Brantingham P J, Wang W, Peng N. Improving event definition following for zero-shot event detection. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024
- Zhou Y, Shen T, Geng X, Long G, Jiang D. ClarET: pre-training a correlation-aware context-to-event transformer for event-centric generation and classification. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 2559–2575
- Huang K H, Hsu I, Natarajan P, Chang K W, Peng N. Multilingual generative language models for zero-shot cross-lingual event argument extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 4633–4646
- Ma Y, Wang Z, Cao Y, Li M, Chen M, Wang K, Shao J. Prompt for extraction? PAIE: prompting argument interaction for event argument extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 6759–6774
- Liu X, Huang H, Shi G, Wang B. Dynamic prefix-tuning for generative template-based event extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 5216–5228
- Cai E, O’Connor B. A Monte Carlo language model pipeline for zero-shot sociopolitical event extraction. In: Proceedings of the NeurIPS 2023. 2023
- Lu D, Ran S, Tetreault J, Jaimes A. Event extraction as question generation and answering. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 1666–1688
- Hsu I H, Xie Z, Huang K, Natarajan P, Peng N. AMPERE: AMR-aware prefix for generation-based event argument extraction model. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 10976–10993
- Lin Z, Zhang H, Song Y. Global constraints with prompting for zero-shot event argument classification. In: Proceedings of the Findings of the Association for Computational Linguistics: EACL 2023. 2023, 2482–2493
- Zhang X F, Blum C, Choji T, Shah S, Vempala A. ULTRA: unleash LLMs’ potential for event argument extraction through hierarchical modeling and pair-wise self-refinement. In: Proceedings of the Findings of the Association for Computational Linguistics ACL 2024. 2024
- Zhou H, Qian J, Feng Z, Hui L, Zhu Z, Mao K. LLMs learn task heuristics from demonstrations: a heuristic-driven prompting strategy for document-level event argument extraction. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 11972–11990
- Hsu I H, Huang K H, Boschee E, Miller S, Natarajan P, Chang K W, Peng N. DEGREE: a data-efficient generation-based event extraction model. In: Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022, 1890–1908
- Gao J, Zhao H, Wang W, Yu C, Xu R. EventRL: enhancing event extraction with outcome supervision for large language models. 2024, arXiv preprint arXiv: 2402.11430 · arXiv
- Wei X, Chen Y, Cheng N, Cui X, Xu J, Han W. CollabKG: a learnable human-machine-cooperative information extraction toolkit for (event) knowledge graph construction. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024
- Wang J, Chang Y, Li Z, An N, Ma Q, Hei L, Luo H, Lu Y, Ren F. TechGPT-2.0: a large language model project to solve the task of knowledge graph construction. 2024, arXiv preprint arXiv: 2401.04507 · arXiv
- Xiao X, Wang Y, Xu N, Wang Y, Yang H, Wang M, Luo Y, Wang L, Mao W, Zeng D. YAYI-UIE: a chat-enhanced instruction tuning framework for universal information extraction. 2023, arXiv preprint arXiv: 2312.15548 · arXiv
- Xu J, Sun M, Zhang Z, Zhou J. ChatUIE: exploring chat-based unified information extraction using large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 3146–3152
- Gui H, Yuan L, Ye H, Zhang N, Sun M, Liang L, Chen H. IEPile: unearthing large scale schema-conditioned information extraction corpus. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 127–146
- Li Z, Zeng Y, Zuo Y, Ren W, Liu W, Su M, Guo Y, Liu Y, Lixiang L, Hu Z, Bai L, Li W, Liu Y, Yang P, Jin X, Guo J, Cheng X. KnowCoder: coding structured knowledge into LLMs for universal information extraction. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 8758–8779
- Tang R, Han X, Jiang X, Hu X. Does synthetic data generation of LLMs help clinical text mining? 2023, arXiv preprint arXiv: 2303.04360 · arXiv
- Evans J, Sadruddin S, D’Souza J. Astro-NER–astronomy named entity recognition: is GPT a good domain expert annotator? 2024, arXiv preprint arXiv: 2405.02602 · arXiv
- Naraki Y, Yamaki R, Ikeda Y, Horie T, Naganuma H. Augmenting NER datasets with LLMs: towards automated and refined annotation. 2024, arXiv preprint arXiv: 2404.01334 · arXiv
- Chen F, Feng Y. Chain-of-thought prompt distillation for multimodal named entity recognition and multimodal relation extraction. 2023, arXiv preprint arXiv: 2306.14122 · arXiv
- Wadhwa S, Amir S, Wallace B. Revisiting relation extraction in the era of large language models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 15566–15589
- Bian J, Zheng J, Zhang Y, Zhu S. Inspire the large language model by external knowledge on biomedical named entity recognition. 2023, arXiv preprint arXiv: 2309.12278 · arXiv
- Geng S, Josifosky M, Peyrard M, West R. Flexible grammar-based constrained decoding for language models. 2023, arXiv preprint arXiv: 2305.13971 · arXiv
- Nie B, Shao Y, Wang Y. Know-adapter: towards knowledge-aware parameter-efficient transfer learning for few-shot named entity recognition. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 9777–9786
- Monajatipoor M, Yang J, Stremmel J, Emami M, Mohaghegh F, Rouhsedaghat M, Chang K W. LLMs in biomedicine: a study on clinical named entity recognition. 2024, arXiv preprint arXiv: 2404.07376 · arXiv
- Dunn A, Dagdelen J, Walker N, Lee S, Rosen A S, Ceder G, Persson K, Jain A. Structured information extraction from complex scientific text with fine-tuned large language models. 2022, arXiv preprint arXiv: 2212.05238 · arXiv
- Cheung J, Zhuang Y, Li Y, Shetty P, Zhao W, Grampurohit S, Ramprasad R, Zhang C. POLYIE: a dataset of information extraction from polymer material scientific literature. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024
- Ma M D, Taylor A, Wang W, Peng N. DICE: data-efficient clinical event extraction with generative models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 15898–15917
- Hu Y, Ameer I, Zuo X, Peng X, Zhou Y, Li Z, Li Y, Li J, Jiang X, Xu H. Zero-shot clinical entity recognition using ChatGPT. 2023, arXiv preprint arXiv: 2303.16416 · arXiv
- Labrak Y, Rouvier M, Dufour R. A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024
- Gao J, Zhao H, Yu C, Xu R. Exploring the feasibility of ChatGPT for event extraction. 2023, arXiv preprint arXiv: 2303.03836 · arXiv
- Gui H, Zhang J, Ye H, Zhang N. InstructIE: a Chinese instruction-based information extraction dataset. 2023, arXiv preprint arXiv: 2305.11527 · arXiv
- Han R, Peng T, Yang C, Wang B, Liu L, Wan X. Is information extraction solved by ChatGPT? an analysis of performance, evaluation criteria, robustness and errors. 2023, arXiv preprint arXiv: 2305.14450 · arXiv
- Li B, Fang G, Yang Y, Wang Q, Ye W, Zhao W, Zhang S. Evaluating ChatGPT’s information extraction capabilities: an assessment of performance, explainability, calibration, and faithfulness. 2023, arXiv preprint arXiv: 2304.11633 · arXiv
- Fei H, Zhang M, Zhang M, Chua T S. XNLP: an interactive demonstration system for universal structured NLP. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024
- Bhagavatula C, Le Bras R, Malaviya C, Sakaguchi K, Holtzman A, Rashkin H, Downey D, Yih S W T, Choi Y. Abductive commonsense reasoning. In: Proceedings of the 8th International Conference on Learning Representations. 2020
- Luo P, Xu T, Liu C, Zhang S, Xu L, Li M, Chen E. Bridging gaps in content and knowledge for multimodal entity linking. In: Proceedings of the ACM Multimedia 2024. 2024
- Yang H, Zhao X, Huang S, Li Q, Xu G. LATEX-GCL: large language models (LLMs)-based data augmentation for text-attributed graph contrastive learning. 2024, arXiv preprint arXiv: 2409.01145 · arXiv
- Gao L, Biderman S, Black S, Golding L, Hoppe T, Foster C, Phang J, He H, Thite A, Nabeshima N, Presser S, Leahy C. The pile: an 800GB dataset of diverse text for language modeling. 2020, arXiv preprint arXiv: 2101.00027 · arXiv
- Zhao H, Zheng S, Wu L, Yu B, Wang J. LANE: logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. 2024, arXiv preprint arXiv: 2407.02833 · arXiv
- Zheng Z, Qiu Z, Hu X, Wu L, Zhu H, Xiong H. Generative job recommendations with large language model. 2023, arXiv preprint arXiv: 2307.02157 · arXiv
- Wu L, Qiu Z, Zheng Z, Zhu H, Chen E. Exploring large language model for graph data understanding in online job recommendations. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 9178–9186
- Chen B, Zhang Z, Langrené N, Zhu S. Unleashing the potential of prompt engineering in large language models: a comprehensive review. 2023, arXiv preprint arXiv: 2310.14735 · arXiv
- Zhao Z, Lin F, Zhu X, Zheng Z, Xu T, Shen S, Li X, Yin Z, Chen E. DynLLM: when large language models meet dynamic graph recommendation. 2024, arXiv preprint arXiv: 2405.07580 · DOI
- Xu D, Zhang Z, Lin Z, Wu X, Zhu Z, Xu T, Zhao X, Zheng Y, Chen E. Multi-perspective improvement of knowledge graph completion with large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 11956–11968
- Li X, Zhou J, Chen W, Xu D, Xu T, Chen E. Visualization recommendation with prompt-based reprogramming of large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024, 13250–13262
- Peng W, Yi J, Wu F, Wu S, Zhu B B, Lyu L, Jiao B, Xu T, Sun G, Xie X. Are you copying my model? Protecting the copyright of large language models for EaaS via backdoor watermark. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 7653–7668
- Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E H, Le Q V, Zhou D. Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1800
- Chu Z, Chen J, Chen Q, Yu W, He T, Wang H, Peng W, Liu M, Qin B, Liu T. A survey of chain of thought reasoning: advances, frontiers and future. 2023, arXiv preprint arXiv: 2309.15402 · arXiv
- Beurer-Kellner L, Müller M N, Fischer M, Vechev M. Prompt sketching for large language models. 2023, arXiv preprint arXiv: 2311.04954 · arXiv
- Zheng L, Yin L, Xie Z, Huang J, Sun C, Yu C H, Cao S, Kozyrakis C, Stoica I, Gonzalez J E, Barrett C, Sheng Y. Efficiently programming large language models using SGLang. 2023, arXiv preprint arXiv: 2312.07104 · arXiv
- Liu Z, Wu L, He M, Guan Z, Zhao H, Feng N. Dr.E bridges graphs with large language models through words. 2024, arXiv preprint arXiv: 2406.15504 · arXiv
- Guan Z, Zhao H, Wu L, He M, Fan J. LangTopo: aligning language descriptions of graphs with tokenized topological modeling. 2024, arXiv preprint arXiv: 2406.13250 · arXiv
- Zha R, Zhang L, Li S, Zhou J, Xu T, Xiong H, Chen E. Scaling up multivariate time series pre-training with decoupled spatial-temporal representations. In: Proceedings of the 40th IEEE International Conference on Data Engineering. 2024, 667–678
- Liu Q, Wu X, Zhao X, Zhu Y, Zhang Z, Tian F, Zheng Y. Large language model distilling medication recommendation model. 2024, arXiv preprint arXiv: 2402.02803 · arXiv
- Qiao S, Ou Y, Zhang N, Chen X, Yao Y, Deng S, Tan C, Huang F, Chen H. Reasoning with language model prompting: a survey. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 5368–5393
- Sang E F T K, De Meulder F. Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: Proceedings of the 7th Conference on Natural Language Learning. 2003, 142–147
- Roth D, Yih W T. A linear programming formulation for global inference in natural language tasks. In: Proceedings of the 8th Conference on Computational Natural Language Learning. 2004, 1–8
- Walker C, Strassel S, Medero J, Maeda K. Ace 2005 multilingual training corpus-linguistic data consortium. See catalog.ldc.upenn.edu/LDC2006T06 website, 2005
- Liu Z, Xu Y, Yu T, Dai W, Ji Z, Cahyawijaya S, Madotto A, Fung P. CrossNER: evaluating cross-domain named entity recognition. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 13452–13460
- Chen P, Xu H, Zhang C, Huang R. Crossroads, buildings and neighborhoods: a dataset for fine-grained location recognition. In: Proceedings of 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022, 3329–3339
- Pradhan S, Elhadad N, South B R, Martínez D, Christensen L, Vogel A, Suominen H, Chapman W W, Savova G. Task 1: ShARe/CLEF eHealth evaluation lab 2013. In: Proceedings of the Working Notes for CLEF 2013 Conference. 2013
- Mowery D L, Velupillai S, South B R, Christensen L, Martínez D, Kelly L, Goeuriot L, Elhadad N, Pradhan S, Savova G, Chapman W W. Task 2: ShARe/CLEF eHealth evaluation lab 2014. In: Proceedings of the Working Notes for CLEF 2014 Conference. 2014, 31–42
- Jiang H, Hua Y, Beeferman D, Roy D. Annotating the tweebank corpus on named entity recognition and building NLP models for social media analysis. In: Proceedings of the 13th Language Resources and Evaluation Conference. 2022, 7199–7208
- Zhang Q, Fu J, Liu X, Huang X. Adaptive co-attention network for named entity recognition in tweets. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 2018, 5674–5681
- Ushio A, Barbieri F, Silva V, Neves L, Camacho-Collados J. Named entity recognition in twitter: a dataset and analysis on short-term temporal shifts. In: Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 2022, 309–319
- Wang X, Tian J, Gui M, Li Z, Wang R, Yan M, Chen L, Xiao Y. WikiDiverse: a multimodal entity linking dataset with diversified contextual topics and entity types. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 4785–4797
- Stoica G, Platanios E A, Poczos B. Re-TACRED: addressing shortcomings of the TACRED dataset. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 13843–13850
- Satyapanich T, Ferraro F, Finin T. CASIE: extracting cybersecurity event information from text. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 8749–8757
- Kim J D, Wang Y, Yamamoto Y. The genia event extraction shared task, 2013 edition -overview. In: Proceedings of BioNLP Shared Task 2013 Workshop. 2013, 8–15
- Zamai A, Zugarini A, Rigutini L, Ernandes M, Maggini M. Show less, instruct more: Enriching prompts with definitions and guidelines for zero-shot ner. 2024, arXiv preprint arXiv:2407.01272 · DOI
- Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu P J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020, 21(1): 140
- Du Z, Qian Y, Liu X, Ding M, Qiu J, Yang Z, Tang J. GLM: general language model pretraining with autoregressive blank infilling. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022, 320–335
- Taori R, Gulrajani I, Zhang T, Dubois Y, Li X. Stanford alpaca: An instruction-following llama model. See github.com/tatsulab/stanford_alpaca website, 2023
- Chiang W L, Li Z, Lin Z, Sheng Y, Wu Z, Zhang H, Zheng L, Zhuang S, Zhuang Y, Gonzalez J E, Stoica I, Xing E P. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. See vicuna.lmsys.org websit, 2023
- Rozière B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan X E, Adi Y, Liu J, Sauvestre R, Remez T, Rapin J, Kozhevnikov A, Evtimov I, Bitton J, Bhatt M, FerrerC C, Grattafiori A, Xiong W, Défossez A, Copet J, Azhar F, Touvron H, Martin L, Usunier N, Scialom T, Synnaeve G. Code llama: open foundation models for code. 2023, arXiv preprint arXiv: 2308.12950 · arXiv
- Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog, 2019, 1(8): 9
- Wang B. Mesh-Transformer-JAX: model-parallel implementation of Transformer language model with JAX. See github.com/kingoflolz/mesh-transformer-jax website, 2021
- Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, Schulman J, Hilton J, Kelton F, Miller L, Simens M, Askell A, Welinder P, Christiano P, Leike J, Lowe R. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 27730–27744
Cited by
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“This section will discuss some of the key research challenges identified from our analysis that the academic community will need to address in future work.”
Next-Generation Database Interfaces: A Survey of LLM-Based Text-to-SQL
Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., & Huang, X. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3609486 · Cited by 388
Collaborative large language models for automated data extraction in living systematic reviews
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Riaz, Z. B., Raina, A., Zhou, S., He, H., Saeidi, A., Hasan, B., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae325 · Cited by 75
Wen, S., Fei, L., Zhuang, W., Pan, X., Yu, W., Bao, J., & Li, X. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103235 · Cited by 71
Metadata
- Authors
- Xu, D., Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Wang, Y., & Chen, E.
- Affiliations
- University of Science and Technology of China (CN)
- City University of Hong Kong (HK)
- Tencent (China) (CN)
- Corresponding author
- Xu, Tong
- Zhao, Xiangyu
- Wu, Xian
- Chen, Enhong
- Venue
- Frontiers of Computer Science
- Published
- 2024-11-11
- Volume
- 18
- Issue
- 6
- ISSN
- 2095-2228 (ISSN-L)
- 2095-2236
- OpenAlex
- W4404239715
- Cited by
- 326
- References
- 151
- Citations by year
- 2026: 149 · 2025: 144 · 2024: 33
- FWCI
- 48.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Frontiers of Computer Science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- CityU Scholars (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Impact Fund
- City University of Hong Kong (9360163)
- Tencent
- Ant Group
- Aromatic Plant Research Center
- National Natural Science Foundation of China (62222213, 62072423)
- Innovation and Technology Fund
- Hong Kong Institute for Data Science (9360163)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- large language models, information extraction, knowledge extraction from text
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large language models for generative information extraction: a survey. https://app.cochat.ai/papers/10.1007/s11704-024-40555-y
- The paper
- Xu, D., Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Wang, Y., & Chen, E. (2024). Large language models for generative information extraction: a survey. Frontiers of Computer Science. https://doi.org/10.1007/s11704-024-40555-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.