CoChat
Sign Up
Cited by 11Topic ModelingAdvanced Text Analysis TechniquesBiomedical Text Mining and Ontologies

Large Language Models for Scientific Information Extraction: An Empirical Study for Virology

Open access
Shamsabadi, Mahsa, D’Souza, Jennifer, Auer, Sören·2024·Findings of the Association for Computational Linguistics: EACL 2024
DOI 10.18653/v1/2024.findings-eacl.26
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we champion the use of structured and semantic content representation of discourse-based scholarly communication, inspired by tools like Wikipedia infoboxes or structured Amazon product descriptions.These representations provide users with a concise overview, aiding scientists in navigating the dense academic landscape.Our novel automated approach leverages the robust text generation capabilities of LLMs to produce structured scholarly contribution summaries, offering both a practical solution and insights into LLMs' emergent abilities.For LLMs, the prime focus is on improving their general intelligence as conversational agents.We argue that these models can also be applied effectively in information extraction (IE), specifically in complex IE tasks within terse domains like Science.This paradigm shift replaces the traditional modular, pipelined machine learning approach with a simpler objective expressed through instructions.Our results show that finetuned FLAN-T5 with 1000x fewer parameters than the state-of-the-art GPTdavinci is competitive for the task.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-eacl.26

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent systems have shown strong general-purpose language capabilities and encouraging results in scientific text mining, information extraction, and structured generation.

    “Recent systems have shown strong general-purpose language capabilities and encouraging results in scientific text mining, information extraction, and structured generation [7][8][9][10].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [7]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics. · DOI · Paper page
  2. Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. 2021. Ext5: Towards extreme multi- task scaling for transfer learning. arXiv preprint arXiv:2111.10952. · arXiv
  3. Sören Auer, Allard Oelen, Muhammad Haris, Markus Stocker, Jennifer D'Souza, Kheir Eddine Farfar, Lars Vogt, Manuel Prinz, Vitalis Wiens, and Mo- hamad Yaser Jaradeh. 2020. Improving access to scientific literature with knowledge graphs. Biblio- thek Forschung und Praxis, 44(3):516-529.
  4. Jeroen Baas, Michiel Schotten, Andrew Plume, Gré- goire Côté, and Reza Karimi. 2020. Scopus as a curated, high-quality bibliometric data source for aca- demic research in quantitative science studies. Quan- titative Science Studies, 1(1):377-386.
  5. Tim Berners-Lee, James Hendler, and Ora Lassila. 2001. The semantic web. Scientific american, 284(5):34- 43.
  6. Caroline Birkle, David A Pendlebury, Joshua Schnell, and Jonathan Adams. 2020. Web of science as a data source for research on scientific and scholarly activity. Quantitative Science Studies, 1(1):363-376.
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. · arXiv
  9. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
  10. Lisa Ehrlinger and Wolfram Wöß. 2016. Towards a def- inition of knowledge graphs. SEMANTiCS (Posters, Demos, SuCCESS), 48(1-4):2.
  11. Dieter Fensel, Umutcan Şimşek, Kevin Angele, Elwin Huaman, Elias Kärle, Oleksandra Panasiuk, Ioan Toma, Jürgen Umbrich, Alexander Wahler, Dieter Fensel, et al. 2020. Introduction: what is a knowl- edge graph? Knowledge graphs: Methodology, tools and selected use cases, pages 1-10.
  12. Ivo M. Foppa. 2017. 7 -o. diekmann, j. heesterbeek, and j.a. metz (1991) and p. van den driessche and j. wat- mough (2002): The spread of infectious diseases in heterogeneous populations. In Ivo M. Foppa, editor, A Historical Introduction to Mathematical Modeling of Infectious Diseases, pages 157-194. Academic Press, Boston. · DOI
  13. Suzanne Fricke. 2018. Semantic scholar. Journal of the Medical Library Association: JMLA, 106(1):145.
  14. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, pages 124-133, San- tiago de Compostela, Spain. Association for Compu- tational Linguistics. · DOI
  15. R Brian Haynes, Cynthia D Mulrow, Edward J Huth, Douglas G Altman, and Martin J Gardner. 1990. More informative abstracts revisited. Annals of inter- nal medicine, 113(1):69-76.
  16. Ginny Hendricks, Dominika Tkaczyk, Jennifer Lin, and Patricia Feeney. 2020. Crossref: The sustain- able source of community-owned scholarly metadata. Quantitative Science Studies, 1(1):414-427.
  17. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. · arXiv
  18. Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556. · arXiv
  19. Sally Hopewell, Mike Clarke, David Moher, Elizabeth Wager, Philippa Middleton, Douglas G Altman, Ken- neth F Schulz, and Consort Group. 2008. Consort for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS medicine, 5(1):e20.
  20. Mohamad Yaser Jaradeh, Allard Oelen, Kheir Ed- dine Farfar, Manuel Prinz, Jennifer D'Souza, Gábor Kismihók, Markus Stocker, and Sören Auer. 2019. Open research knowledge graph: next generation in- frastructure for semantic scholarly knowledge. In Proceedings of the 10th International Conference on Knowledge Capture, pages 243-246.
  21. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language under- standing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 4163-4174.
  22. Rob Johnson, Anthony Watkinson, and Michael Mabe. 2018. The stm report. An overview of scientific and scholarly publishing. 5th edition October.
  23. Salomon Kabongo, Jennifer D'Souza, and Sören Auer. 2023a. Orkg-leaderboards: A systematic workflow for mining leaderboards as a knowledge graph. arXiv preprint arXiv:2305.11068. · arXiv
  24. Salomon Kabongo, Jennifer D'Souza, and Sören Auer. 2023b. Zero-shot entailment of leaderboards for em- pirical ai research. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2023.
  25. Salomon Kabongo, Jennifer D'Souza, and Sören Auer. 2023c. Zero-shot entailment of leaderboards for em- pirical ai research. In 2023 ACM/IEEE Joint Confer- ence on Digital Libraries (JCDL), pages 237-241. · DOI
  26. Salomon Kabongo, Jennifer D'Souza, and Sören Auer. 2021a. Automated mining of leaderboards for empir- ical ai research. In Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, December 1-3, 2021, Proceedings 23, pages 453-470. Springer.
  27. Salomon Kabongo, Jennifer D'Souza, and Sören Auer. 2021b. Automated mining of leaderboards for empir- ical ai research. In Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, December 1-3, 2021, Proceedings 23, pages 453-470. Springer.
  28. Salomon Kabongo, Jennifer D'Souza, and Sören Auer. 2023d. Orkg-leaderboards: a systematic workflow for mining leaderboards as a knowledge graph. Inter- national Journal on Digital Libraries, pages 1-14.
  29. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question an- swering, text classification, and regression via span extraction. arXiv preprint arXiv:1904.09286. · arXiv
  30. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with a single QA system. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics. · DOI · Paper page
  31. Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Ro- dríguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio López, Umesh Nandal, et al. 2017. Overview of the biocreative vi chemical-protein inter- action track. In Proceedings of the sixth BioCreative challenge evaluation workshop, volume 1, pages 141- 146.
  32. Hemant Kulkarni. 1996. Structured abstracts: still more. Annals of Internal Medicine, 124(7):695-696.
  33. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
  34. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian- feng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487-4496.
  35. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688. · arXiv
  36. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. · arXiv
  37. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791-2809, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
  38. Antonio Miranda, Farrokh Mehryary, Jouni Luoma, Sampo Pyysalo, Alfonso Valencia, and Martin Krallinger. 2021. Overview of drugprot biocreative vii track: quality evaluation and large scale text min- ing of drug-gene/protein relations. In Proceedings of the seventh BioCreative challenge evaluation work- shop, pages 11-21.
  39. Takeo Nakayama, Nobuko Hirai, Shigeaki Yamazaki, and Mariko Naito. 2005. Adoption of structured abstracts by general medical journals and format for a structured abstract. Journal of the Medical Library Association, 93(2):237.
  40. Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xi- angru Tang, Aadit Vyas, Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open- domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 432-447, Online. Association for Computa- tional Linguistics. · DOI
  41. Allard Oelen, Mohamad Yaser Jaradeh, Kheir Eddine Farfar, Markus Stocker, and Sören Auer. 2019. Com- paring research contributions in a scholarly knowl- edge graph. In CEUR Workshop Proceedings 2526 (2019), volume 2526, pages 21-26. Aachen: RWTH Aachen.
  42. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
  43. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5551.
  44. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789.
  45. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
  46. Allen H Renear and Carole L Palmer. 2009. Strate- gic reading, ontologies, and the future of scientific publishing. Science, 325(5942):828-832.
  47. Steven Sanche, Yen Ting Lin, Chonggang Xu, Ethan Romero-Severson, Nicolas W Hengartner, and Ruian Ke. 2020. The novel coronavirus, 2019-ncov, is highly contagious and more infectious than initially estimated. arXiv preprint arXiv:2002.03268. · arXiv
  48. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. · arXiv
  49. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR.
  50. David Shotton. 2009. Semantic publishing: the coming revolution in scientific journal publishing. Learned Publishing, 22(2):85-94.
  51. Luciana B Sollaci and Mauricio G Pereira. 2004. The introduction, methods, results, and discussion (imrad) structure: a fifty-year survey. Journal of the medical library association, 92(3):364.
  52. Markus Stocker, Allard Oelen, Mohamad Yaser Jaradeh, Muhammad Haris, Omar Arab Oghli, Golsa Hei- dari, Hassan Hussein, Anna-Lena Lorenz, Salomon Kabenamualu, Kheir Eddine Farfar, et al. 2023. Fair scientific information with the open research knowl- edge graph. FAIR Connect, 1(1):19-21.
  53. Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chieh- Han Wu, Yuxiao Dong, and Anshul Kanakia. 2020a. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1):396-413.
  54. Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, et al. Cord-19: The covid-19 open research dataset. ArXiv.
  55. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 5776-5788.
  56. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652. · arXiv
  57. Vitalis Wiens, Markus Stocker, and Sören Auer. 2020. Towards customizable chart visualizations of tabular data using knowledge graphs. In Digital Libraries at Times of Massive Societal Transition: 22nd Interna- tional Conference on Asia-Pacific Digital Libraries, ICADL 2020, Kyoto, Japan, November 30-December 1, 2020, Proceedings 22, pages 71-80. Springer.

Cited by

Sorted by citations

Authors
Shamsabadi, M., D’Souza, J., & Auer, S.
Affiliations
Technische Informationsbibliothek (TIB) (DE)
ORCID
Venue
Findings of the Association for Computational Linguistics: EACL 2024
Publisher
Association for Computational Linguistics
Published
2024
Pages
374-392
DOI
10.18653/v1/2024.findings-eacl.26
OpenAlex
W7126444615
Cited by
11
References
1
Citations by year
2026: 9 · 2025: 1 · 2024: 1
FWCI
2.65
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Findings of the Association for Computational Linguistics: EACL (publishedVersion, cc-by)
Funders
  • Deutsche Forschungsgemeinschaft (460234259)
  • Bundesministerium für Bildung und Forschung (01lS22070)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
Keywords
large language models, scientific information extraction, virology, scholarly communication, Wikipedia infoboxes, Flan-T5, parameter-efficient fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review