Biomedical Named Entity Recognition with Multilingual BERT
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
We present the approach of the Turku NLP group to the PharmaCoNER task on Spanish biomedical named entity recognition.We apply a CRF-based baseline approach and multilingual BERT to the task, achieving an Fscore of 88% on the development data and 87% on the test set with BERT.Our approach reflects a straightforward application of a state-of-the-art multilingual model that is not specifically tailored to either the language nor the application domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite their success in simple sentiment classification, directly applying BERT to aspect-based sentiment analysis (ABSA), which is a fine-grained SA task, shows less significant improvement [202].”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [202]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“In our second experiment we utilize BERT (Devlin et al., 2018), a transformer (Vaswani et al., 2017) based attentive neural architecture.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“The model is optimized with Adam (Kingma and Ba, 2014) with a batch size of 16 and a learning rate of 2e-5 warmed up from 2e-7 over the first training epoch.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“In our second experiment we utilize BERT (Devlin et al., 2018), a transformer (Vaswani et al., 2017) based attentive neural architecture.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“In our second experiment we utilize BERT (Devlin et al., 2018), a transformer (Vaswani et al., 2017) based attentive neural architecture.”
Method(2014) · Cited by 33,419
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 2×
“(Ma and Hovy, 2016; Lample et al., 2016).”
BackgroundEnd-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567
Fixing Weight Decay Regularization in Adam
Loshchilov, I., & Hutter, F. (2018) · Cited by 1,986
How Multilingual is Multilingual BERT?
Pires, T., Schlinger, E., & Garrette, D. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1493 · Cited by 1,351 · cited 2×
“Thus we conduct our experiments with the multilingual BERT model (Pires et al., 2019) trained on a Wikipedia corpus, covering 104 languages.”
BackgroundMethodPeng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899 · cited 1×
“Whereas pre-trained BERT models have shown strong performance for English NER tasks (Peng et al., 2019), to our knowledge no pretrained Spanish BERT models are readily available 4 .”
MethodOverview of BioNLP'09 shared task on event extraction
Kim, J.-D., Ohta, T., Pyysalo, S., Kano, Y., & Tsujii, J. (2009) · Proceedings of the Workshop on BioNLP Shared Task - BioNLP '09 · doi:10.3115/1572340.1572342 · Cited by 588 · cited 2×
“The task data is distributed in the simple stand-off format first introduced for the BioNLP Shared Task 2009 (Kim et al., 2009).”
MethodSelf-citationIntroduction to the bio-entity recognition task at JNLPBA
Kim, J.-D., Ohta, T., Tsuruoka, Y., Tateisi, Y., & Collier, N. (2004) · Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications - JNLPBA '04 · doi:10.3115/1567594.1567610 · Cited by 573 · cited 2×
“We note that compared to English language biomedical NER resources, the number of annotations is somewhat limited; for example, the JNLPBA shared task (Kim et al., 2004) data contains over 50,000 training examples of similar”
BackgroundOverview of BioCreative II gene mention recognition
Smith, L., Tanabe, L. K., Ando, R. J. N., Kuo, C.-J., Chung, I.-F., Hsu, C.-N., Lin, Y.-S., Klinger, R., Friedrich, C. M., Ganchev, K., et al. (2008) · Genome Biology · doi:10.1186/gb-2008-9-s2-s2 · Cited by 486 · cited 1×
“…limited; for example, the JNLPBA shared task (Kim et al., 2004) data contains over 50,000 training examples of similar types, the BioCreative II GM data (Smith et al., 2008) over 18,000 gene mentions, and the BioCre-ative CHEMDNER (Krallinger et al., 2015) data over 80,000 chemical mentions.”
BackgroundThe CHEMDNER corpus of chemicals and drugs and its annotation principles
Krallinger, M., Rabal, O., Leitner, F., Vazquez, M., Salgado, D., Lu, Z., Leaman, R., Lu, Y., Ji, D., Lowe, D. M., et al. (2015) · Journal of Cheminformatics · doi:10.1186/1758-2946-7-s1-s2 · Cited by 404 · cited 2×
“We note that compared to English language biomedical NER resources, the number of annotations is somewhat limited; for example, the JNLPBA shared task (Kim et al., 2004) data contains over 50,000 training examples of similar types, the BioCreative II GM data (Smith et al., 2008) over 18,000 gene mentions, and the BioC…”
BackgroundTaggerOne: joint named entity recognition and normalization with semi-Markov Models
Leaman, R., & Lu, Z. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw343 · Cited by 305 · cited 1×
“…of manually annotated corpora and dedicated NER meth-ods have been introduced for the analysis of English biomedical and clinical texts (e.g. (Leaman and Lu, 2016; Crichton et al., 2017; Weber et al., 2019)), there has been comparatively little work on these basic resources for other…”
MethodA neural network multi-task learning approach to biomedical named entity recognition
Crichton, G., Pyysalo, S., Chiu, B., & Korhonen, A. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1776-8 · Cited by 229 · cited 1×
“…corpora and dedicated NER meth-ods have been introduced for the analysis of English biomedical and clinical texts (e.g. (Leaman and Lu, 2016; Crichton et al., 2017; Weber et al., 2019)), there has been comparatively little work on these basic resources for other languages, including Spanish.”
BackgroundSelf-citationHow multilingual is Multilingual BERT?Preprint
Pires, T., Schlinger, E., & Garrette, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01502 · Cited by 135 · cited 2×
“Thus we conduct our experiments with the multilingual BERT model (Pires et al., 2019) trained on a Wikipedia corpus, covering 104 languages.”
BackgroundMethodPharmaCoNER: Pharmacological Substances, Compounds and proteins Named Entity Recognition track
Agirre, A. G., Marimon, M., Intxaurrondo, A., Rabal, O., Villegas, M., & Krallinger, M. (2019) · Proceedings of The 5th Workshop on BioNLP Open Shared Tasks · doi:10.18653/v1/d19-5701 · Cited by 86 · cited 1×
“The PharmaCoNER task focuses on pharmacological compound mentions in Spanish clinical texts, promoting the development of biomedical text mining tools for non-English data (Gonzalez-Agirre et al., 2019).”
BackgroundPeng, Y., Yan, S., & Lu, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05474 · Cited by 72 · cited 1×
“Whereas pre-trained BERT models have shown strong performance for English NER tasks (Peng et al., 2019), to our knowledge no pretrained Spanish BERT models are readily available 4 .”
MethodSoares, F., Villegas, M., González-Agirre, A., Krallinger, M., & Armengol-Estapé, J. (2019) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/w19-1916 · Cited by 62 · cited 1×
“by Soares et al. (2019) https://github.”
HUNER: improving biomedical NER with pretraining
Weber, L., Münchmeyer, J., Rocktäschel, T., Habibi, M., & Leser, U. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz528 · Cited by 58 · cited 1×
“…corpora and dedicated NER meth-ods have been introduced for the analysis of English biomedical and clinical texts (e.g. (Leaman and Lu, 2016; Crichton et al., 2017; Weber et al., 2019)), there has been comparatively little work on these basic resources for other languages, including Spanish.”
Background(2012) · Research Explorer (The University of Manchester)
Self-citation
Research works printed but not linked: 1
- Yoshimasa Tsuruoka, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Junichi Tsujii. 2005. Developing a robust part- of-speech tagger for biomedical text. In Panhel- lenic Conference on Informatics, pages 382-392.
Cited by
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“speed up ELMo’s inference on NER. Tsai et al. [168] leveraged knowledge distillation to run a small BERT for NER on a single CPU. Besides, BERT is also used on domain-specific NER, such as biomedicine [52, 91], etc. 8 Future Directions Though PTMs have proven their power for various NLP tasks, challenges still exist du…”
MethodUnifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
“These models are most effective for tasks that require understanding the entire sentence, such as text classification [26] and named entity recognition [53].”
BackgroundChinese clinical named entity recognition with variant neural structures based on BERT methods
Li, X., Zhang, H., & Zhou, X.-H. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103422 · Cited by 186
Large Language Models for UAVs: Current State and Pathways to the Future
Javaid, S., Fahim, H., He, B., & Saeed, N. (2024) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2024.3446799 · Cited by 208
“It showed remarkable performance improvements on leaderboards for tasks such as named entity recognition [72], [73], sentiment analysis [74], [75], and especially question answering and natural language inference, where the full-sentence context from both directions can be crucial for understanding…”
BackgroundCan BERT Dig It? Named Entity Recognition for Information Retrieval in the Archaeology Domain
Brandsen, A., Verberne, S., Lambers, K., & Wansleeben, M. (2022) · Journal on Computing and Cultural Heritage · doi:10.1145/3497842 · Cited by 99
“This model is often used when no single-language model is available [25, 28, 35].”
Metadata
- Authors
- Hakala, K., & Pyysalo, S.
- Affiliations
- University of Turku (FI)
- Venue
- Proceedings of The 5th Workshop on BioNLP Open Shared Tasks
- Published
- 2019
- Pages
- 56-61
- OpenAlex
- W2987972786
- Cited by
- 96
- References
- 22
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 17 · 2023: 25 · 2022: 22 · 2021: 14 · 2020: 11
- FWCI
- 35.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on BioNLP Open Shared Tasks (BioNLP-OST) (publishedVersion, cc-by)
- Funders
- China Scholarship Council
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical named entity recognition, multilingual BERT, Spanish language, conditional random fields, clinical text annotation, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
