Biomedical Named Entity Recognition with Multilingual BERT
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
We present the approach of the Turku NLP group to the PharmaCoNER task on Spanish biomedical named entity recognition.We apply a CRF-based baseline approach and multilingual BERT to the task, achieving an Fscore of 88% on the development data and 87% on the test set with BERT.Our approach reflects a straightforward application of a state-of-the-art multilingual model that is not specifically tailored to either the language nor the application domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite their success in simple sentiment classification, directly applying BERT to aspect-based sentiment analysis (ABSA), which is a fine-grained SA task, shows less significant improvement [202].”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [202]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“In our second experiment we utilize BERT (Devlin et al., 2018), a transformer (Vaswani et al., 2017) based attentive neural architecture.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“The model is optimized with Adam (Kingma and Ba, 2014) with a batch size of 16 and a learning rate of 2e-5 warmed up from 2e-7 over the first training epoch.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“In our second experiment we utilize BERT (Devlin et al., 2018), a transformer (Vaswani et al., 2017) based attentive neural architecture.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“In our second experiment we utilize BERT (Devlin et al., 2018), a transformer (Vaswani et al., 2017) based attentive neural architecture.”
Method(2014) · Cited by 33,419
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 2×
“(Ma and Hovy, 2016; Lample et al., 2016).”
BackgroundEnd-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567
Fixing Weight Decay Regularization in Adam
Loshchilov, I., & Hutter, F. (2018) · Cited by 1,993
Research works printed but not linked: 1
- Yoshimasa Tsuruoka, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Junichi Tsujii. 2005. Developing a robust part- of-speech tagger for biomedical text. In Panhel- lenic Conference on Informatics, pages 382-392.
Cited by
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“speed up ELMo’s inference on NER. Tsai et al. [168] leveraged knowledge distillation to run a small BERT for NER on a single CPU. Besides, BERT is also used on domain-specific NER, such as biomedicine [52, 91], etc. 8 Future Directions Though PTMs have proven their power for various NLP tasks, challenges still exist du…”
MethodUnifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
“These models are most effective for tasks that require understanding the entire sentence, such as text classification [26] and named entity recognition [53].”
BackgroundChinese clinical named entity recognition with variant neural structures based on BERT methods
Li, X., Zhang, H., & Zhou, X.-H. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103422 · Cited by 186
Large Language Models for UAVs: Current State and Pathways to the Future
Javaid, S., Fahim, H., He, B., & Saeed, N. (2024) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2024.3446799 · Cited by 211
“It showed remarkable performance improvements on leaderboards for tasks such as named entity recognition [72], [73], sentiment analysis [74], [75], and especially question answering and natural language inference, where the full-sentence context from both directions can be crucial for understanding…”
BackgroundCan BERT Dig It? Named Entity Recognition for Information Retrieval in the Archaeology Domain
Brandsen, A., Verberne, S., Lambers, K., & Wansleeben, M. (2022) · Journal on Computing and Cultural Heritage · doi:10.1145/3497842 · Cited by 99
“This model is often used when no single-language model is available [25, 28, 35].”
Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis
Yu, H., Fan, L., Li, L., Zhou, J., Ma, Z., Xian, L., Hua, W., He, S., Jin, M., Zhang, Y., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00171-8 · Cited by 101
Hadi, M. A., & Fard, F. H. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10314-x · Cited by 71
“Previous studies (Liu et al. 2018; Hakala and Pyysalo 2019; Edunov et al. 2019) developed Named Entity Recognition systems with the help of PLMs.”
BackgroundA pre-training and self-training approach for biomedical named entity recognition
Gao, S., Kotevska, O., Sorokine, A., & Christian, J. B. (2021) · PLOS ONE · doi:10.1371/journal.pone.0246310 · Cited by 46
“Following previous work on applying BERT for NER [59], during training and inference, we only use the label from the first subword token associated with each word.”
MethodNamed entity recognition using neural language model and CRF for Hindi language
Sharma, R., Morwal, S., & Agarwal, B. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101356 · Cited by 50
E-NER — An Annotated Named Entity Recognition Corpus of Legal Text
Au, T. W. T., Lampos, V., & Cox, I. J. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.22 · Cited by 59
Multilingual Protest News Detection - Shared Task 1, CASE 2021
Hürriyetoğlu, A., Mutlu, O., Yörük, E., Liza, F. F., Kumar, R., & Ratan, S. (2021) · Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) · doi:10.18653/v1/2021.case-1.11 · Cited by 41
“New benchmark data sets such as XTREME (Hu et al., 2020) and system proposals such as mBERT (Devlin et al., 2019a), XLM (Lam-ple and Conneau, 2019), mBART (Liu et al., 2020), and XLM-R (Conneau et al., 2020) have demonstrated promising results on various tasks (Hakala and Pyysalo, 2019).”
BackgroundDepression Classification From Tweets Using Small Deep Transfer Learning Language Models
Rizwan, M., Mushtaq, M. F., Akram, U., Mehmood, A., Ashraf, I., & Sahelices, B. (2022) · IEEE Access · doi:10.1109/access.2022.3223049 · Cited by 56
“Several studies used BERT for biomedical NER which has been done by combining the conditional random field (CRF) layer with multi-lingual BERT [15].”
MethodDeb, S., & Chanda, A. K. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2022.100253 · Cited by 46
Song, Y., Miret, S., & Liu, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.201 · Cited by 83
Zou, J., & Kanoulas, E. (2020) · ACM Transactions on Information Systems · doi:10.1145/3388640 · Cited by 48
“Recognizing entities in biomedical texts is a research direction of its own, with significant recent work on neural methods, which further progresses the state-of-the-art [Hakala and Pyysalo 2019; Wang et al. 2019].”
An ERNIE-Based Joint Model for Chinese Named Entity Recognition
Wang, Y., Sun, Y., Ma, Z., Gao, L., & Xu, Y. (2020) · Applied Sciences · doi:10.3390/app10165711 · Cited by 35
“[22] applied the multilingual BERT to Spanish biomedical NER.”
MethodPre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05006 · Cited by 34
“Besides English, there is much work exploring the pre-trained language models on the BioNER of other languages, including Chinese [43, 91, 129, 130, 252, 272], Spanish [6, 75, 159], French [41], Korean [109], Russian [154], Arabic [23], Italian [28].”
BackgroundFraile Navarro, D., Kocaballi, A. B., Dras, M., & Berkovsky, S. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3569893 · Cited by 28
“, trained on clinical text corpora [2], have allowed their application for complex NLP tasks including named entity recognition [49], summarization [110], or question answering [51], also in the medical domain and the health industry.”
BackgroundNi, P., Li, G.-M., Hung, P., & Chang, V. I. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107975 · Cited by 33
“And other SOTA methods adopt pre-trained language models to obtain words or even characters representations [58, 59, 60].”
MethodJaber, A., & Martínez, P. (2022) · Methods of Information in Medicine · doi:10.1055/s-0042-1742388 · Cited by 29
T-FREX: A Transformer-based Feature Extraction Method from Mobile App Reviews
Motger, Q., Miaschi, A., Dell’Orletta⋄, F., Franch, X., & Marco, J. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00030 · Cited by 28
“…publicly available for generic types of NER (e.g., dates, locations, persons, e-mail addresses...) and for some specific domains (e.g. medical and legal domains) [29]–[31], to the best of our knowledge there are no proposals for the identification of app feature entities exploiting fine-tuned LLMs.”
BackgroundSegura-Bédmar, I., Camino-Perdones, D., & Guerrero‐Aspizua, S. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04810-y · Cited by 30
“Recently, transformers [20] and Bidirectional Encoder Representations from Transformers [21] have outperformed traditional and deep learning models for most of NLP applications [22–25], and in particular, for NER in the biomedical domain [17, 26].”
Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text
Lange, L., Iurshina, A., Adel, H., & Strötgen, J. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.14 · Cited by 27
MedBERT: A Pre-trained Language Model for Biomedical Named Entity Recognition
Vasantharajan, C., Tun, K. Z., Ho, T.-N., Jain, S., Rong, T., & Siong, C. E. (2022) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.23919/apsipaasc55919.2022.9980157 · Cited by 44
“1: Overview of the pre-training of MedBERT. and [30].”
BackgroundAnnotating the Pandemic: Named Entity Recognition and Normalisation in COVID-19 Literature
Colic, N., Furrer, L., & Rinaldi, F. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.27 · Cited by 22
“While it is trained to predict whether a sentence follows another and randomly blacked out words, the resulting language model can be finetuned for different tasks, such as NER (Hakala and Pyysalo, 2019) and NEN, or adapted for different domains through further training.”
Review of Natural Language Processing in Pharmacology
Trajanov, D., Trajkovski, V., Dimitrieva, M., Dobreva, J., Jovanovik, M., Klemen, M., Žagar, A., & Robnik‐Šikonja, M. (2023) · Pharmacological Reviews · doi:10.1124/pharmrev.122.000715 · Cited by 25
“CRFsuite library[162] NER, Drug Discovery, ADE [178] [36] [243] [72] [13] [73] [203] [159] [136] URL: http://www.”
PharmKE: Knowledge Extraction Platform for Pharmaceutical Texts Using Transfer Learning
Jofche, N., Mishev, K., Stojanov, R., Jovanovik, M., Zdravevski, E., & Trajanov, D. (2023) · Computers · doi:10.3390/computers12010017 · Cited by 17
“For the recognition of biomedical named entities for content in Spanish, Hakala and Pyysalo [23] proposed a method based on conditional random fields (CRFs) and multilingual BERT.”
MethodRevisiting Open Domain Query Facet Extraction and Generation
Samarinas, C., Dharawat, A., & Zamani, H. (2022) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3539813.3545138 · Cited by 24
“We can cast the facet extraction problem as a sequence labeling task, which has been successfully used for entity recognition [9], keyphrase extraction [21], and question answering [26].”
MethodEmbedding Electronic Health Records to Learn BERT-based Models for Diagnostic Decision Support
Tang, R., Yao, H., Zhu, Z., Sun, X., Hu, G., Yichong, L., & Xie, G. (2021) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi52183.2021.00055 · Cited by 18
ADMit: Improving NER in automotive domain with domain adversarial training and multi-task learning
Park, C., Jeong, S., & Kim, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120007 · Cited by 14
Uronen, L., Salanterä, S., Hakala, K., Hartiala, J., & Moen, H. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104695 · Cited by 14
Self-citationGeospatRE: extraction and geocoding of spatial relation entities in textual documents
Syed, M. A., Arsevska, E., Mathieu Roche, A., & Teisseire, M. (2023) · Cartography and Geographic Information Science · doi:10.1080/15230406.2023.2264753 · Cited by 15
“More precisely, the measure for evaluating NER is the F-Score which is the harmonic mean of Precision and Recall (Hakala & Pyysalo, 2019; Resnik & Lin, 2010).”
Xiong, Y., Chen, S., Tang, B., Chen, Q., Wang, X., Yan, J., & Zhou, Y. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04236-y · Cited by 16
BERT-based tourism named entity recognition: making use of social media for travel recommendations
Fudholi, D. H., Zahra, A., Rani, S., Huda, S. N., Paputungan, I. V., & Zukhri, Z. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1731 · Cited by 14
“Hakala & Pyysalo (2019) performed NER on the Spanish dataset.”
MethodLee, H., & Jeong, O. (2023) · Sensors · doi:10.3390/s23020685 · Cited by 15
“In other words, to extract where the entity is located from the history of the conversation with the user, token classification [32,33] can be performed using POS tagging together to predict the tag for the token of the input s quence as one of the types of named entity, B-(begin), I-(inside), and O-(outside), hich is…”
MethodInfluentialBonifacio, L., Vilela, P. A., Lobato, G. R., & Fernandes, E. R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61377-8_46 · Cited by 14
Effects of data and entity ablation on multitask learning models for biomedical entity recognition
Rodriguez, N. E., Nguyen, M., & McInnes, B. T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104062 · Cited by 13
Named Entity Recognition of Ancient Poems Based on Albert‐BiLSTM‐MHA‐CRF Model
Zhou, F., Wang, C., & Wang, J. (2022) · Wireless Communications and Mobile Computing · doi:10.1155/2022/6507719 · Cited by 11
Panagoulias, D. P., Palamidas, A. F., Virvou, M. K., & Tsihrintzis, G. A. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113975 · Cited by 16
Improving biomedical named entity recognition through transfer learning and asymmetric tri-training
Bhattacharya, M., Bhat, S., Tripathy, S., Bansal, A., & Choudhary, M. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.244 · Cited by 14
Knowledge Graph Construction and Representation Method for Potato Diseases and Pests
Yang, W., Yang, S., Wang, G., Liu, Y., Lü, J., & Yuan, W. (2023) · Agronomy · doi:10.3390/agronomy14010090 · Cited by 10
Lancheros, B. S., Pastor, G. C., & Mitkov, R. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09738-8 · Cited by 13
“The CRAFT EN + EN Augmented specifies the concatenation of the original dataset with the augmented version obtained by entity replacement.”
InfluentialSimAC: simulating agile collaboration to generate acceptance criteria in user story elaboration
Li, Y., Keung, J., Yang, Z., Ma, X., Zhang, J., & Liu, S. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00448-7 · Cited by 18
Sony, P., Shanmugam, G. S., & Nagarajan, S. (2023) · SN Computer Science · doi:10.1007/s42979-023-02392-x · Cited by 12
“The main research areas in NLP that uses BERT effectively are named entity recognition [105], machine translation [106], question answering [107], sentiment analysis [108], anaphora resolution [109], to name a few.”
Efficacy of BERT embeddings on predicting disaster from Twitter dataPreprint
Chanda, A. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10698 · Cited by 28
“In different types of NLP tasks such as text classification [28], text summarization [15], entity recognition [8], BERT model outperformed traditional embedding learning models.”
BackgroundChandrakala, S., & Raj, S. A. A. (2022) · International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES) · doi:10.1109/icses55317.2022.9914194 · Cited by 8
“Especially, BERT, a language representation transformer model that generates embedding of a word depending on its context outperforms traditional embedding models and it is widely used in variety of NLP task such as text classification[14], entity recognition[15], text summarization[16],etc.”
MethodEASAL: Entity-Aware Subsequence-Based Active Learning for Named Entity Recognition
Liu, Y., Hu, J., Chen, Z., Wan, X., & Chang, T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26069 · Cited by 7
Multilingual BERT Cross-Lingual Transferability with Pre-trained Representations on Tangut: A Survey
Lu, X.-M., Liu, W., Jiang, S., & Liu, C. (2023) · doi:10.1109/icnlp58431.2023.00048 · Cited by 6
Studying and recommending information highlighting in Stack Overflow answers
Ahmed, S. S., Wang, S., Tian, Y., Chen, T.-H. P., & Zhang, H. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107478 · Cited by 7
“BERT has been proven effective in embedding the context for NER task [39, 6, 13].”
MethodToward Recognizing More Entity Types in NER: An Efficient Implementation using Only Entity Lexicons
Peng, M., Ma, R., Zhang, Q., Zhao, L., Wei, M., Sun, C., & Huang, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.60 · Cited by 5
“…like the bi-directional long short-term memory network (LSTM) plus a CRF layer (Huang et al., 2015), the convolutional neural network (CNN) plus a CRF layer, the combination of LSTM and CNN (Chiu and Nichols, 2016), and the BERT based LSTM+CRF model (Jiang et al., 2019; Hakala and Pyysalo, 2019).”
BackgroundLee, S.-M., Tan, Y.-H., & Yu, H. (2023) · International Conference on Artificial Intelligence and Law · doi:10.1145/3594536.3595144 · Cited by 5
“Nonetheless, recent studies have shown that advanced machine learning and deep learning techniques, such as pretraining language models and fine-tuning them to learn downstream tasks, including NER, have been successful in addressing this challenge [16][19][22][23].”
BackgroundNICE: Neural Integrated Custom Engines.
Buj, D. M., Garcia, D., Parcheta, Z., & Casacuberta, F. (2020) · Cited by 4
NLP for Student and Teacher: Concept for an AI based Information Literacy Tutoring System.
Libbrecht, P., Declerck, T., Schlippe, T., Mandl, T., & Schiffner, D. (2020) · Cited by 5
SymptomID: A Framework for Rapid Symptom Identification in Pandemics Using News Reports
Gu, K., Vosoughi, S., & Prioleau, T. (2021) · ACM Transactions on Management Information Systems · doi:10.1145/3462441 · Cited by 5
Rhouma, R., McMahon, C., Mcgillivray, D., Massood, H., Kanwal, S., Khan, M., Lo, T., Lam, J., & Smith, C. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101519 · Cited by 9
Yan, J., Zong, C., & Xu, J. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3588314 · Cited by 4
“To further validate the generalizability of LASL, we separately test three different target NER models: (1) LSTMCRF, a basic sequence labeling method with Bi-LSTM as word encoder; (2) BERTCRF, the same model as described in Section 3.1; and (3) Globalpointer [42], a span-based NER model that leveraging the relative po…”
MethodInfluentialuOttawa at SemEval-2023 Task 6: Deep Learning for Legal Text Understanding
Almuslim, I., Stilwell, S., Suresh, S. K., & Inkpen, D. Z. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.79 · Cited by 6
“In the biomedical field, Schneider et al. (2020) and Hakala and Pyysalo (2019) both describe NER models for the biomedical domain.”
BackgroundGeoXTag: Relative Spatial Information Extraction and Tagging of Unstructured Text
Syed, M. A., Arsevska, E., Mathieu Roche, A., & Teisseire, M. (2022) · AGILE GIScience Series · doi:10.5194/agile-giss-3-16-2022 · Cited by 4
“More precisely, the measure for evaluating NER is the F-Score which is the harmonic mean of Precision and Recall (Hakala and Pyysalo, 2019; Resnik and Lin, 2010).”
BackgroundMethodA refinement strategy for identification of scientific software from bioinformatics publications
Jiang, L., Kang, X., Huang, S., & Yang, B. (2022) · Scientometrics · doi:10.1007/s11192-022-04381-y · Cited by 3
“A number of deep learning algorithms, such as RNN (Mikolov et al., 2010), LSTM (Hochreiter, 1998), BERT (Devi et al., 2019), LSTM-CRF (Zeng et al., 2017), have been developed for some promising sequential text processing applications, especially for NER (Cho et al., 2020; Hakala & Pyysalo, 2019).”
MethodLiang, S., Hartmann, M., & Sonntag, D. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.31 · Cited by 9
“…typologically close to English through cross-lingual transfer from large-scale pre-trained multilingual BERT models (Lauscher et al., 2020; Souza et al., 2019; Jørgensen et al., 2021; Hakala and Pyysalo, 2019; Souza et al., 2019) or English language models (Artetxe et al., 2020; Plank, 2019).”
BackgroundEvaluating Data Augmentation for Medication Identification in Clinical Notes
Ixa, U., Koontz, J. C., Oronoz, M., HiTZ - Ixa, U., Pérez, A., & HiTZ - Ixa, U. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_063 · Cited by 3
“Hakala and Pyysalo (2019) combine BERT with a final CRF layer for PharmaCoNER (Gonzalez-Agirre et al., 2019), the first shared task on detecting drug and chemical entities in Spanish medical documents.”
Wang, B., Tang, D., & Wu, Q. (2022) · International Conference on Applied Machine Learning (ICAML) · doi:10.1109/icaml57167.2022.00034 · Cited by 3
“[12] used multilingual BERT for NER in biomedical field.”
MethodSkoltechNLP at SemEval-2021 Task 5: Leveraging Sentence-level Pre-training for Toxic Span Detection
Dale, D., Markov, I. L., Logacheva, V., Kozlova, O., Semenov, N., & Panchenko, A. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.126 · Cited by 8
“…to successfully use it for sequence labeling tasks of different levels: part-of-speech tagging and syntactic parsing (Koto et al., 2020), named entity recognition (Hakala and Pyysalo, 2019), semantic role labeling (He et al., 2019), detection of Machine Translation errors (Moura et al., 2020).”
MethodTowards Improved Distantly Supervised Multilingual Named-Entity Recognition for Tweets
Eskander, R., Mishra, S., Mehta, S., Samaniego, S., & Haghighi, A. (2022) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2022.mrl-1.12 · Cited by 2
“While mostly utilized for sentence-level NLP tasks such as hate-speech identification (Mandl et al., 2021) and claim matching (Kazemi et al., 2021), LaBSE has also proven efficient for NER (Hakala and Pyysalo, 2019).”
BackgroundLayer, M., Neubert, S., Boda, B., & Stelzer, R. (2023) · Symposium Design for X (DFX) · doi:10.35199/dfx2023.24 · Cited by 2
The Impact of Pause and Filler Word Encoding on Dementia Detection with Contrastive Learning
Soleimani, R., Guo, S., Haley, K. L., Jacks, A., & Lobaton, E. J. (2024) · Applied Sciences · doi:10.3390/app14198879 · Cited by 7
Li, L., Wang, H., Li, J., Xu, X., Wang, Y., & Ren, T. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679698 · Cited by 3
“Among them, encoder-only PLMs like BERT are most effective for tasks that require understanding the entire sentence, such as text classification [38] and named entity recognition [17].”
Decoding Domain-Specific NER: A Performance Evaluation of ChatGPT, Bi-LSTM, and BERT
Rafique, K. A., Pansuriya, M., Wawrzik, F., & Grimm, C. (2024) · International Conference on Machine Intelligence Theory and Applications (MiTA) · doi:10.1109/mita60795.2024.10751683 · Cited by 2
“While Hakala et al. (2019) [10] used Spanish biomedical NER to apply a CRF-based baseline approach and multilingual BERT to achieve excellent results.”
Structured Representation of Pre-Defined Information Backflow in Standards and Directives
Layer, M., Luttmer, J., Nagarajah, A., & Stelzer, R. H. (2024) · Standards · doi:10.3390/standards4040013 · Cited by 2
AMU-EURANOVA at CASE 2021 Task 1: Assessing the stability of multilingual BERT
Bouscarrat, L., Bonnefoy, A., Capponi, C., & Ramisch, C. (2021) · Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) · doi:10.18653/v1/2021.case-1.21 · Cited by 4
Toward A Robust Method for Understanding the Replicability of Research.
Gelman, B. U., Clark, C., Friedman, S., Kuter, U., & Gentile, J. E. (2021) · National Conference on Artificial Intelligence · Cited by 2
Transformer-based approach to variable typing
Rey, C. A., Danguilan, J. L., Mendoza, K. P. R., & Remolona, M. F. M. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e20505 · Cited by 2
“In the ER task, a span of string generally referred to as a document is fed into the system where mentions of entities are located and classified according to a predefined set of labels [20,21].”
BackgroundFirdaus, F., Rajagede, R. A., Sari, A., Hanifah, S., & Perwitasari, D. A. (2024) · International Journal of Engineering · doi:10.5829/ije.2024.37.09c.04 · Cited by 3
Biomedical-named entity recognition using CUDA accelerated KNN algorithm
Bali, M., Anandaraj, S. P., & Duraisamy, J. H. (2023) · TELKOMNIKA (Telecommunication Computing Electronics and Control) · doi:10.12928/telkomnika.v21i4.24065 · Cited by 1
A Lightweight Global Taxonomy Induction System for E-Commerce Concept Labels
Kejriwal, M., & Shen, K. (2023) · Lecture notes in social networks · doi:10.1007/978-3-031-33065-0_5 · Cited by 1
Discovering Named Entities at Scale of a Data Mining Perspective on Large Text Corpora
Su, J. (2024) · IEEE International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA) · doi:10.1109/eebda60612.2024.10485772 · Cited by 1
“Notwithstanding, despite the fact that most of NER strategies utilize extra contingent irregular fields (RCF) to find significant relationships between's adjoining marks, they typically miss the mark concerning integrating all logical data from text into the profound learning layers[21].”
BackgroundSun, Y., Yang, Y., & Fu, W. (2024) · Asia Conference on Computer Vision, Image Processing and Pattern Recognition (CVIPPR) · doi:10.1145/3663976.3664023 · Cited by 2
“This particular type of model excels in tasks that necessitate a thorough understanding of entire sentences, such as text classification [29] and named entity recognition [9].”
BackgroundZeng, W., & Huson, D. H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.06.01.543218 · Cited by 1
“Traditional statistical models 32–35 are gradually being superseded by transformer-based language models or other neural networks 31,36,37 .”
Colic, N., Beeler, P. E., Csajka, C., Foufi, V., Gaspar, F., Pogam, M. L., Lisibach, A., Lovis, C., Lutters, M., & Rinaldi, F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68763-2_15
AMU-EURANOVA at CASE 2021 Task 1: Assessing the stability of multilingual BERTPreprint
Bouscarrat, L., Bonnefoy, A., Capponi, C., & Ramisch, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14625 · Cited by 1
“Multilingual BERT has already been used for multilingual named entity recognition and showed great results compared to state-of-the-art models (Hakala and Pyysalo, 2019).”
ResultAttention-based Persian Language Modeling
Yekkehkhani, A., Pouromid, M., & Bahrani, M. (2022) · International Computer Conference, Computer Society of Iran (CSICC) · doi:10.1109/csicc55295.2022.9780494
Segura-Bédmar, I., Camino-Perdones, D., & Guerrero‐Aspizua, S. (2022) · Research Square · doi:10.21203/rs.3.rs-1569369/v1 · Cited by 1
Automated Web Annotator of Biomedical Entities in Spanish Language
Iza, J., Morejon, S., & Uyaguari, A. (2022) · International Conference on Information Systems and Software Technologies (ICI2ST) · doi:10.1109/ici2st57350.2022.00018
“However, this tool relies heavily on lexical resources and morphological analyses of English text [3].”
MethodInfluentialChat agents respond more empathetically by using hearsay experience
Narimatsu, H., Sugiyama, H., Mizukami, M., & Arimoto, T. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.960087
“, 2018)1, which can be applied to various tasks by fine-tuning with a small amount of data and has actually been used in recent years for domain application of NE extraction (Hakala and Pyysalo, 2019; Li et al., 2020).”
BackgroundSimple Knowledge Graph Completion Model Based on Differential Negative Sampling and Prompt Learning
Duan, L., Wang, J., Luo, B., & Sun, Q. (2023) · Information · doi:10.3390/info14080450 · Cited by 1
Named Entities Based on the BERT-BILSTM-ACRF Model Recognition Research
Wang, J., & Guo, Y. (2023) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3639233.3639347
“In the work of [7], the authors and others proposed a biomedical named entity recognition method based on multi-language Bert.”
Zhixiang, W., & Yolwas, N. (2024) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1117/12.3034807
A Method of Network Attack Named Entity Recognition based on Deep Active Learning
Wang, L., Ma, Y., Li, M., Li, H., & Zhang, P. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00044
“In recent years, significant progress has been made in deep learning-based named entity recognition methods with large annotated datasets[5-6],In particular, using pre-training language models such as BERT[7] and RoBERTa[8]and so on.”
BackgroundINDOBERT embedding for flood detection in Indonesia
Purwaningsih, T., Winarko, E., & Mustofa, K. (2025) · AIP conference proceedings · doi:10.1063/5.0236876
Knowledge Graph-based Text Sentiment Analysis via Three-way Decision
He, Y., Qian, W., Cai, X., & Huang, J. (2025) · Data Intelligence · doi:10.3724/2096-7004.di.2025.0063
Data Quality Assessment Approaches for Event-based Surveillance Systems
Syed, M. A., Arsevska, E., Mathieu Roche, A., & Teisseire, M. (2025) · Data Intelligence · doi:10.3724/2096-7004.di.2025.0062
Research on Optimization and Application of Retrieval-LLM-NER Method
Shen, T., Yi, J., Ni, Z., Ji, X., Gao, J., & Guo, W. (2025) · Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS) · doi:10.1109/acctcs66275.2025.00014
“By utilizing bidirectional encoding [3], BERT could deeply understand textual context, significantly improving NER accuracy.”
Zafar, A., Nowaczyk, S., & Sarmadi, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-32643-0_17
An Empirical Study of MuRIL-based architectures for Hindi Health Domain NER
Mishra, S., Pandere, S., & Sinha, S. (2026) · IEEE Guwahati Subsection Conference (GCON) · doi:10.1109/gcon69192.2026.11648992
“Although mBERT has been successfully applied to NER tasks in multiple languages such as Portuguese [17], Spanish [18], Chinese [19], Russian [20], German [21], and Slavic languages [22], systematic exploration of these models for Hindi NER, particularly in domain-specific settings, remains limited.”
Metadata
- Authors
- Hakala, K., & Pyysalo, S.
- Affiliations
- University of Turku (FI)
- Venue
- Proceedings of The 5th Workshop on BioNLP Open Shared Tasks
- Published
- 2019
- Pages
- 56-61
- OpenAlex
- W2987972786
- Cited by
- 96
- References
- 22
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 17 · 2023: 25 · 2022: 22 · 2021: 14 · 2020: 11
- FWCI
- 35.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on BioNLP Open Shared Tasks (BioNLP-OST) (publishedVersion, cc-by)
- Funders
- China Scholarship Council
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical named entity recognition, multilingual BERT, Spanish language, conditional random fields, clinical text annotation, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
