CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Solving Label Variation in Scientific Information Extraction via Multi-Task Learning

Pham, Dong, Ho, Xanh, Ha, Quang-Thuy, Aizawa, Akiko·2023·arXiv
DOI 10.48550/arxiv.2312.15751arXiv 2312.15751
Open this paper in CoChat — free account

Abstract

Scientific Information Extraction (ScientificIE) is a critical task that involves the identification of scientific entities and their relationships. The complexity of this task is compounded by the necessity for domain-specific knowledge and the limited availability of annotated data. Two of the most popular datasets for ScientificIE are SemEval-2018 Task-7 and SciERC. They have overlapping samples and differ in their annotation schemes, which leads to conflicts. In this study, we first introduced a novel approach based on multi-task learning to address label variations. We then proposed a soft labeling technique that converts inconsistent labels into probabilistic distributions. The experimental results demonstrated that the proposed method can enhance the model robustness to label noise and improve the end-to-end performance in both ScientificIE tasks. The analysis revealed that label variations can be particularly effective in handling ambiguous instances. Furthermore, the richness of the information captured by label variations can potentially reduce data size requirements. The findings highlight the importance of releasing variation labels and promote future research on other tasks in other domains. Overall, this study demonstrates the effectiveness of multi-task learning and the potential of label variations to enhance the performance of ScientificIE.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.15751

Cites

Statistics

Figures

Table 2: The noise samples occur in label variation. For the relation part, we use 'Sem' [*blue] to denote relation in the SemEval dataset, while 'Sci' [*green] denotes the SciERC dataset.
Table 2: The noise samples occur in label variation. For the relation part, we use 'Sem' [*blue] to denote relation in the SemEval dataset, while 'Sci' [*green] denotes the SciERC dataset.CC-BY · From the PDF
Figure 1 Illustration of Multi-Task Learning based on SpERT architecture to handle label variation.
Figure 1 Illustration of Multi-Task Learning based on SpERT architecture to handle label variation.CC-BY · From the PDF
Figure 2 The impact of data quantity on the performance (SciERC testing). The gold label was trained on SciERC annotations with the SpERT baseline model, and the variation label was trained using the proposed model.
Figure 2 The impact of data quantity on the performance (SciERC testing). The gold label was trained on SciERC annotations with the SpERT baseline model, and the variation label was trained using the proposed model.CC-BY · From the PDF
Table 7: The error samples are from the predictions from the proposed method: (a) the common sources of error, (b) predictions are redundant the relations, and (c) correct predictions are missing in the gold label. [*red] is predictions, [*green] is the gold label of SciERC, [*blue] is the gold label of SemEval-2018, OST is 'OtherScientificTerm'.
Table 7: The error samples are from the predictions from the proposed method: (a) the common sources of error, (b) predictions are redundant the relations, and (c) correct predictions are missing in the gold label. [*red] is predictions, [*green] is the gold label of SciERC, [*blue] is the gold label of SemEval-2018, OST is 'OtherScientificTerm'.CC-BY · From the PDF
Figure 3 The distribution of common relations in the overlapped abstracts of the two datasets.
Figure 3 The distribution of common relations in the overlapped abstracts of the two datasets.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 5 The co-occurrence distribution of 'Used-for/Usage' in two overlapped corpora.
Figure 5 The co-occurrence distribution of 'Used-for/Usage' in two overlapped corpora.CC-BY · From the PDF
Figure 6 The co-occurrence distribution of 'Evaluate-for/Result' in two overlapped corpora.
Figure 6 The co-occurrence distribution of 'Evaluate-for/Result' in two overlapped corpora.CC-BY · From the PDF
Figure 7 The co-occurrence distribution of 'Feature-of/Model' in two overlapped corpora.
Figure 7 The co-occurrence distribution of 'Feature-of/Model' in two overlapped corpora.CC-BY · From the PDF
Figure 8 The co-occurrence distribution of 'Part-of/Part-whole' in two overlapped corpora.
Figure 8 The co-occurrence distribution of 'Part-of/Part-whole' in two overlapped corpora.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 26

  1. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pages 15-21. Association for Computational Linguistics.
  2. Elisa Bassignana and Barbara Plank. 2022a. CrossRE: A cross-domain dataset for relation extraction. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 3592-3604, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
  3. Elisa Bassignana and Barbara Plank. 2022b. What do you mean by relation extraction? a survey on datasets and study on scientific relation classification. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics: Student Research Workshop, pages 67-83, Dublin, Ireland. Association for Computational Linguistics. · DOI
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
  6. Markus Eberts and Adrian Ulges. 2019. Span-based joint entity and relation extraction with transformer pre-training. The 24th European Conference on Arti- ficial Intelligence (ECAI 2020).
  7. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Bar- bara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator dis- agreement via soft-label multi-task learning. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2591-2597, Online. Association for Computational Linguistics. · DOI
  8. Kata Gábor, Davide Buscaldi, Anne-Kathrin Schu- mann, Behrang QasemiZadeh, Haïfa Zargayouna, and Thierry Charnois. 2018. SemEval-2018 task 7: Semantic relation extraction and classification in scientific papers. In Proceedings of the 12th Inter- national Workshop on Semantic Evaluation, pages 679-688, New Orleans, Louisiana. Association for Computational Linguistics. · DOI
  9. Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tat- sunori Hashimoto, and Michael S Bernstein. 2021. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-14.
  10. Sarthak Jain, Madeleine van Zuylen, Hannaneh Ha- jishirzi, and Iz Beltagy. 2020. SciREX: A challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7506- 7516, Online. Association for Computational Lin- guistics. · DOI · Paper page
  11. Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsource an- notation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 291-297, Lisbon, Portugal. Association for Compu- tational Linguistics. · DOI
  12. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86.
  13. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators' dis- agreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 10528-10539, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  14. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics. · DOI · Paper page
  15. Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  16. Rebecca J. Passonneau, Ansaf Salleb-Aoussi, Vikas Bhardwaj, and Nancy Ide. 2010. Word sense annota- tion of polysemous words by multiple annotators. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Asso- ciation (ELRA).
  17. Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671-10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  18. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of the 14th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 742-751.
  19. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  20. Victor S Sheng, Foster Provost, and Panagiotis G Ipeiro- tis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614-622.
  21. Alexandra Uma, Tommaso Fornaciari, Anca Dumi- trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021a. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338- 347, Online. Association for Computational Linguis- tics. · DOI
  22. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021b. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385-1470.
  23. Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020. Denoising relation extraction from document- level distant supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3683-3688, On- line. Association for Computational Linguistics. · DOI
  24. Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4904-4917. Association for Computational Linguistics.
  25. Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of annotation: From zero to many labels. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7620-7632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  26. Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 50-61, Online. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Metadata

Authors
Pham, D., Ho, X., Ha, Q.-T., & Aizawa, A.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2312.15751
arXiv
2312.15751
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review