CoChat
Sign Up
Cited by 21Topic ModelingBiomedical Text Mining and OntologiesAdvanced Text Analysis Techniques

SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents

Open access
Zhang, Qi, Chen, Zhijia, Pan, Huitong, Caragea, Cornelia, Latecki, Longin Jan, Dragut, Eduard·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2024.emnlp-main.726
Open this paper in CoChat — free accountOpen PDF

Abstract

Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations).Several datasets have been proposed for training and validating SciIE models.However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context.In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles.Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations.To capture the intricate use and interactions among entities in full texts, our dataset contains a finegrained tag set for relations.Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation.We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-main.726

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • More recent datasets and evaluations show that extracting fine-grained full-text relations remains challenging and often requires more than local span prediction.

    “More recent datasets and evaluations likewise show that extracting fine-grained full-text relations remains challenging and often requires more than local span prediction [29,30].”

    From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [29]

    The Introduction says SciER annotates entire papers with finer-grained relations and calls this harder than annotating abstracts; §5.1 reports significant room for improvement in end-to-end relation extraction. Section 4.2 describes baselines that use global context or model interactions among entities and relations, but the paper does not establish that approaches limited to local span prediction are generally insufficient. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: ScienceIE -extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546-555, Vancouver, Canada. Association for Computational Linguistics. · DOI
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
  3. Satadisha Saha Bhowmick, Eduard C Dragut, and Weiyi Meng. 2022. Boosting entity mention detection for targetted twitter streams with global contextual em- beddings. In 2022 IEEE 38th International Confer- ence on Data Engineering (ICDE), pages 1085-1097. IEEE.
  4. Satadisha Saha Bhowmick, Eduard C Dragut, and Weiyi Meng. 2023. Globally aware contextual embeddings for named entity recognition in social media streams. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 1544-1557. IEEE.
  5. Zhijia Chen, Lihong He, Arjun Mukherjee, and Eduard C Dragut. 2024. Comquest: Large scale user comment crawling and integration. In SIGMOD Conference Companion, pages 432-435.
  6. Zhijia Chen, Weiyi Meng, and Eduard Dragut. 2022. Web record extraction with invariants. Proceedings of the VLDB Endowment, 16(4):959-972.
  7. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers an- chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610, Online. Association for Computational Linguistics. · DOI · Paper page
  8. Mark Davies and Joseph L Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, pages 1047-1051.
  9. Kata Gábor, Davide Buscaldi, Anne-Kathrin Schu- mann, Behrang QasemiZadeh, Haïfa Zargayouna, and Thierry Charnois. 2018. SemEval-2018 task 7: Semantic relation extraction and classification in scientific papers. In Proceedings of the 12th Inter- national Workshop on Semantic Evaluation, pages 679-688, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  10. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  11. Nikita Gautam, David Shumway, Megan Kowalcyk, Sarthak Khanal, Doina Caragea, Cornelia Caragea, Hande Mcginty, and Samuel Dorevitch. 2023. Lever- aging existing literature on the web and deep neural models to build a knowledge graph focused on water quality and health risks. In Proceedings of the ACM Web Conference 2023, pages 4161-4171.
  12. Paul Groth, Mike Lauruhn, Antony Scerri, and Ron Daniel Jr. 2018. Open information extraction on scientific text: An evaluation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3414-3423, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  13. Jenny Heddes, Pim Meerdink, Miguel Pieters, and Maarten Marx. 2021. The automatic detection of dataset names in scientific articles. Data, 6(8):84.
  14. Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2021. TDMSci: A specialized corpus for scientific literature entity tag- ging of tasks datasets and metrics. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 707-714, Online. Association for Com- putational Linguistics. · DOI
  15. Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, et al. 2024. Im- proving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Association, page ocad259.
  16. Sarthak Jain, Madeleine van Zuylen, Hannaneh Ha- jishirzi, and Iz Beltagy. 2020. SciREX: A challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7506-7516, Online. Association for Computational Linguistics. · DOI · Paper page
  17. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learn- ing for biomedical IE? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497-4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  18. Yuhan Li, Jian Wu, Zhiwei Yu, Börje F Karlsso, Wei Shen, Manabu Okumura, and Chin-Yew Lin. 2023. Unlocking science: Novel dataset and benchmark for cross-modality scientific information extraction. arXiv preprint arXiv:2311.08189. · arXiv
  19. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Pro- ceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and In- tegration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online. Association for Computational Linguistics. · DOI · Paper page
  20. Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. 2021. Noisy-labeled NER with confidence estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3437-3445, Online. Association for Computa- tional Linguistics. · DOI
  21. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The seman- tic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics. · DOI · Paper page
  22. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics. · DOI
  23. Yi Luan, Mari Ostendorf, and Hannaneh Hajishirzi. 2017. Scientific information extraction with semi- supervised neural tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2641-2651, Copenhagen, Denmark. Association for Computational Linguistics. · DOI
  24. Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. Large language model is not a good few-shot informa- tion extractor, but a good reranker for hard samples! In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10572-10601, Sin- gapore. Association for Computational Linguistics. · DOI
  25. Wolfgang Otto, Matthäus Zloch, Lu Gan, Saurav Kar- makar, and Stefan Dietze. 2023. GSAP-NER: A novel task, corpus, and baseline for scholarly entity extrac- tion focused on machine learning models and datasets. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 8166-8176, Singapore. Association for Computational Linguistics. · DOI
  26. Huitong Pan, Qi Zhang, Cornelia Caragea, Eduard Dragut, and Longin Jan Latecki. 2024a. Flowlearn: Evaluating large vision-language models on flowchart understanding. arXiv preprint arXiv:2407.05183. · arXiv
  27. Huitong Pan, Qi Zhang, Cornelia Caragea, Eduard Dragut, and Longin Jan Latecki. 2024b. SciDMT: A large-scale corpus for detecting scientific men- tions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14407-14417, Torino, Italia. ELRA and ICCL.
  28. Huitong Pan, Qi Zhang, Eduard Dragut, Cornelia Caragea, and Longin Jan Latecki. 2023. DMDD: A large-scale dataset for dataset mentions detection. Transactions of the Association for Computational Linguistics, 11:1132-1146. · DOI
  29. Seo Park and Cornelia Caragea. 2023. Multi-task knowl- edge distillation with embedding constraints for schol- arly keyphrase boundary classification. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13026-13042, Singapore. Association for Computational Linguis- tics. · DOI
  30. Dong Pham, Xanh Ho, Quang Thuy Ha, and Akiko Aizawa. 2023. Solving label variation in scientific information extraction via multi-task learning. In Pro- ceedings of the 37th Pacific Asia Conference on Lan- guage, Information and Computation, pages 243-256, Hong Kong, China. Association for Computational Linguistics.
  31. Behrang QasemiZadeh and Anne-Kathrin Schumann. 2016. The ACL RD-TEC 2.0: A language resource for evaluating term extraction and entity recognition methods. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1862-1868, Portorož, Slovenia. European Language Resources Association (ELRA).
  32. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpose natural language process- ing task solver? In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 1339-1384, Singapore. Associa- tion for Computational Linguistics. · DOI
  33. Mobashir Sadat and Cornelia Caragea. 2022. SciNLI: A corpus for natural language inference on scientific text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7399-7409, Dublin, Ireland. Association for Computational Linguistics. · DOI
  34. Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023. Gollie: Annotation guidelines improve zero-shot information-extraction. In The Twelfth In- ternational Conference on Learning Representations.
  35. Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje Karlsson, Tingting Ma, Yuzhong Qu, and Chin-Yew Lin. 2022. TIARA: Multi-grained retrieval for robust question answering over large knowledge base. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8108-8121, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  36. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085. · arXiv
  37. Vijay Viswanathan, Luyu Gao, Tongshuang Wu, Pengfei Liu, and Graham Neubig. 2023. DataFinder: Scien- tific dataset recommendation from natural language descriptions. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 10288-10303, Toronto, Canada. Association for Computational Lin- guistics. · DOI
  38. Vijay Viswanathan, Graham Neubig, and Pengfei Liu. 2021. CitationIE: Leveraging the citation graph for scientific information extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 719-731, Online. Association for Computational Linguistics. · DOI
  39. Somin Wadhwa, Silvio Amir, and Byron Wallace. 2023. Revisiting relation extraction in the era of large lan- guage models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 15566-15589, Toronto, Canada. Association for Computational Lin- guistics. · DOI · Paper page
  40. Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. GPT-RE: In-context learning for relation extraction using large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3534-3547, Singapore. Association for Computational Linguistics. · DOI
  41. Qingyun Wang, Manling Li, Xuan Wang, Nikolaus Parulian, Guangxing Han, Jiawei Ma, Jingxuan Tu, Ying Lin, Ranran Haoran Zhang, Weili Liu, Aabhas Chauhan, Yingjun Guan, Bangzheng Li, Ruisong Li, Xiangchen Song, Yi Fung, Heng Ji, Jiawei Han, Shih-Fu Chang, James Pustejovsky, Jasmine Rah, David Liem, Ahmed ELsayed, Martha Palmer, Clare Voss, Cynthia Schneider, and Boyan Onyshkevych. 2021. COVID-19 literature knowledge graph con- struction and drug repurposing report generation. In · DOI

Cited by

Sorted by citations

Authors
Zhang, Q., Chen, Z., Pan, H., Caragea, C., Latecki, L. J., & Dragut, E.
Affiliations
  • Temple University (US)
  • University of Illinois Chicago (US)
ORCID
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2024
Pages
13083-13100
DOI
10.18653/v1/2024.emnlp-main.726
OpenAlex
W4404782499
Cited by
21
References
0
Citations by year
2026: 15 · 2025: 6
FWCI
5.08
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2107213, 2333789, 2107518)
  • Temple University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
scientific information extraction, named entity recognition, relation extraction, scientific datasets, scientific method, out-of-distribution evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review