SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,301–2,400 of 3,596
Automated Orthodontic Diagnosis from a Summary of Medical Findings
Ohtsuka, T., Kajiwara, T., Tanikawa, C., Shimizu, Y., Nagahara, H., & Ninomiya, T. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.21 · Cited by 2
CoAug: Combining Augmentation of Labels and Labelling Rules
Menon, R. R., Wang, B., Araki, J., Zhou, Z., Feng, Z., & Ren, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.577 · Cited by 2
MetaFill: Text Infilling for Meta-Path Generation on Heterogeneous Information Networks
Liu, Z., Duan, K., Yang, J., Xu, H., Zhang, M., & Wang, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.341 · Cited by 2
Extracting Operator Trees from Model Embeddings
Reusch, A., & Lehner, W. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.6 · Cited by 2
Yang, X., Kang, P.-B., Xu, F.-J., Jin, Y., Tang, Y., Su, Y.-Y., Qiu, J.-P., & Cheng, J. J. (2023) · Scientia Sinica Chimica · doi:10.1360/ssc-2023-0142 · Cited by 2
Knowledge-Infused Dynamic Embedding for Predicting the Severity of Suicidal Ideation in Social Media
Lee, D. M., & Moradi, H. (2022) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci58124.2022.00139 · Cited by 3
“While there are many to utilize, not all may provide an accurate representation of every concept [4, 15, 23].”
BackgroundValiyev, G., Eles, P., Kok, A., & D’Ercole, R. (2023) · International Conference on Military Communications and Information Systems (ICMCIS) · doi:10.1109/icmcis59922.2023.10253491 · Cited by 2
Bondarenko, D. A., Ferrod, R., & Di, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47546-7_28 · Cited by 2
Enhancing aspect-based financial sentiment analysis through contrastive learning
Ivanenko, V. (2023) · Innovative technologies and scientific solutions for industries · doi:10.30837/itssi.2023.25.138 · Cited by 2
Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need
Wu, J., Chao, W., Zhou, X., & Luo, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.828 · Cited by 5
Biomedical Named Entity Recognition Through Deep Reinforcement Learning
Zhao, Z., Zou, Y., Wang, M., Xu, B., Yang, Z., & Lin, H. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10386014 · Cited by 2
Automatic Generation of Discharge Summary of EMRs Based on Multi-granularity Information Fusion
Zhao, B., Zan, H., Niu, C., Chang, H., & Zhang, K. (2024) · Communications in computer and information science · doi:10.1007/978-981-99-9864-7_17 · Cited by 2
A Keyphrase-Centric Search Engine for Scientific Papers
Lahiri, A., Sanyal, D. K., & Mukherjee, I. (2023) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/3632754.3632772 · Cited by 2
CoLAL: Co-learning Active Learning for Text Classification
Le, L., Zhao, G., Zhang, X., Zuccon, G., & Demartini, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29235 · Cited by 6
Ayanouz, S., Abdelhakim, B. A., & Ahmed, M. B. (2024) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v13.i2.pp1732-1740 · Cited by 2
“This is where BioBert [6], BlueBert [7], and SciBert [8] ISSN: 2252-8938 Using natural language processing to evaluate the impact of specialized … (Soufyane Ayanouz) 1733 come into play.”
BackgroundEfficient Token Augmented Fine-Tuning
Weber, L. J., Ramalingam, K. J., Beyer, M., Kirchheim, A., & Zimmermann, A. P. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487678 · Cited by 2
“To confirm whether [24] AND [5].”
Document-level Relation Extraction with Progressive Self-distillation
Wang, Q., Mao, Z., Gao, J., & Zhang, Y. (2024) · ACM Transactions on Information Systems · doi:10.1145/3656168 · Cited by 4
“For all the three +PSD variants, we use cased BERT-base [ 12 ] and RoBERTa-large [ 33 ] as encoder on DocRED and Re-DocRED, and SciBERT-base [ 4 ] as encoder on CDR and GDA.”
Consolidating Separate Degradations Model via Weights Fusion and Distillation
Daultani, D., & Larochelle, H. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00053 · Cited by 2
“Pretraining methods have been widely popular nowadays in computer vision [3, 7] and natural language processing [2, 3, 7, 19, 30] to retain/transfer information from commonly used datasets, for example vision datasets such as CIFAR-100 [17] and ImageNet [23].”
MethodBBPM: A Study of Information Pre-retrieval Models Based on Medical BERT Model
Guo, S., Duan, Y., Huang, J., Zheng, D., Pang, P. C., & Tong, H. H. Y. (2023) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc59590.2023.10507322 · Cited by 2
“We considered three models pre-processed for medical data: BioBERT [9] and SciBERT [10], along with Longformer [11], a model specifically tailored for processing long texts.”
MethodDynamic Heterogeneous Graph Learning: An Adaptive Research Academic Network
Yang, M., & Yang, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3401460 · Cited by 2
Wang, P., Li, S., Li, D., Long, K., Tang, J., & Wang, T. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657705 · Cited by 4
“One-Shot+SciBERT: We use SciBERT [2] to encode the examples and retrieve the most similar demonstration from the training set.”
MethodEnhancing Generalizability in Biomedical Entity Recognition: Self-Attention PCA-CLS Model
Mundotiya, R. K., Priya, J. L., Kuwarbi, D., & Singh, T. (2024) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2024.3429234 · Cited by 2
“Leveraging language models as contextual information providers, competitive results were achieved using extensive corpora, as evidenced by BioBERT [3], which utilizes biomedical literature abstracts from PubMed, and SciBERT [4], drawing on scientific papers, including PubMed.”
Gupta, A., Mittal, D., Goel, O., & Jha, S. K. (2024) · Journal of Materials Science · doi:10.1007/s10853-024-09980-z · Cited by 5
“An additional layer of the corpus can also be used before this domain-specific corpora, with scientific words-based corpora without specificity to any domain like in the popular NLP tool SciBERT [ 20].”
Software Mention Recognition with a Three-Stage Framework Based on BERTology Models at SOMD 2024
Thị, T. N., Viet, A. N., Văn, T. Đ., & Nguyen, N. L.-T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_18 · Cited by 2
CeER: A Nested Name Entity Recognition Model Incorporating Gaze Feature
Yu, J., Kong, W., & Liu, F. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7232-2_3 · Cited by 6
Jeon, D., & Lee, C. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125242 · Cited by 2
Data Augmentation Method Utilizing Template Sentences for Variable Definition Extraction
Nagayama, K., Kato, S., & Kano, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_11 · Cited by 3
“Kang et al. [8] developed a method of extracting technical terms and their definitions simultaneously from texts using SciB-ERT [9] and attained an F1-score of 70.8% in extracting definitions from 50 papers included in the ACL Anthology.”
MethodInfluentialInsights into the Development Trends of Industrial Large Language Models
Lai, Y., Bai, J., You, Y., & Dejun, N. (2024) · Trends in Computer Science and Information Technology · doi:10.17352/tcsit.000084 · Cited by 3
BioBridge: Unified Bio-Embedding With Bridging Modality in Code-Switched EMR
Jeon, J., Cho, S., Lee, D., Lee, C., & Kim, J. (2024) · IEEE Access · doi:10.1109/access.2024.3467251 · Cited by 2
“Using pre-trained encoder base models in general domains in NLP and applying them to clinical notes is challenging, and many attempts have been made to solve it [20], [21], [55].”
Wang, T., Ye, R., & Long, T. (2024) · International Symposium on Educational Technology · doi:10.1109/iset61814.2024.00088 · Cited by 2
A Personalized Federated Framework for Document-Level Biomedical Relation Extraction
Xiao, Y., Jin, Y., Zhang, H., Huo, X., Liu, Q., & Zheng, Z. (2024) · International Conference on Data-driven Optimization of Complex Systems (DOCS) · doi:10.1109/docs63458.2024.10704513 · Cited by 2
“In our experiments, we chose Huggingface’s transformers [21], initialized with pretrained cased SciBERT [22] as our base model.”
MethodTechnology roadmap toward the completion of whole-brain architecture with BRA-driven development
Yamakawa, H., Tawatsuji, Y., Ashihara, Y., Fukawa, A., Arakawa, N., Takahashi, K., & Matsuo, Y. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101300 · Cited by 2
“Although the use of natural language processing (NLP) in scientific research has progressed (Beltagy, Lo, & Cohan, 2019), its application has seen rapid expansion in recent years with the advent of LLMs.”
BackgroundMikhaylov, D. V., & Emelyanov, G. M. (2024) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661824700792 · Cited by 2
Boosting Entity Recognition by leveraging Cross-task Domain Models for Weak Supervision
Agrawal, S., Merugu, S., & Sembium, V. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3680009 · Cited by 2
“• Legacy Cross-task Encoder There are many domains where pre-trained in-domain BERT-based encoders are openly available, such as BioBERT [24] for biomedical corpora, FinBERT [2] for financial services corpora, SciBERT [3] for science literature etc.”
Entity and relationship extraction based on span contribution evaluation and focusing framework
Li, Q., Yao, N., Zhou, N., & Zhao, J. K. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101744 · Cited by 3
Context-Aware Search for Environmental Data Using Dense Retrieval
Wetzel, S., & Mäs, S. (2024) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi13110380 · Cited by 2
Semantic-guided attention and adaptive gating for document-level relation extraction
Ding, X., Duan, S., & Zhang, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-78051-9 · Cited by 2
“For CDR, we apply SciBERT 36 as the encoder.”
MethodDocument-level relation extraction via dual attention fusion and dynamic asymmetric loss
Ding, X., Ding, D., Zhou, G., Lu, J., & Zhu, T. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01632-8 · Cited by 2
“In the experiment of the CDR dataset, we introduce SciBERT [34] encoder, which is a pre-trained language model in the biomedical domain of scientific texts.”
Fine-Tuning Models for Biomedical Relation Extraction
Creanga, C., Dinu, L. P., & Gîfu, D. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.637 · Cited by 2
Chinese paper classification based on pre-trained language model and hybrid deep learning method
Luo, X., Mutalib, S., & Aris, S. R. S. (2024) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v14.i1.pp641-649 · Cited by 2
Estimation of Relevance Between Datasets for Enhancing Accessibility of Research Artifacts
Ito, K., & Matsubara, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0868-3_8 · Cited by 2
Galuščáková, P., Oard, D. W., & Nair, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674136 · Cited by 3
Gómez, E., Cuesta, H., Gkiokas, A., Gómez-Cañón, J. S., Porcaro, L., & Yesiler, F. (2024) · ACM eBooks · doi:10.1145/3674127.3702959 · Cited by 5
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
Developing a Medical Question-Answering Chatbot for Egyptian Arabic: A Comparative Study
Ismail, E., Yahya, M., Nabil, M., Ashraf, Y., Elshaer, Z., & Khoriba, G. A. (2024) · International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) · doi:10.1109/miucc62295.2024.10783570 · Cited by 3
“This research also mentions using pre-trained language models such as BERT, SciBERT [30], BioBERT [31], and PubMedBERT [32], all types of LLMs commonly used as baselines for a MedMCQA dataset.”
Wang, J., Dao, J., Zhang, Y., Xu, D., & Zhang, X. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121759 · Cited by 2
Benchmarking with a Language Model Initial Selection for Text Classification Tasks
Riyadi, A., Kovacs, M., Serdült, U., & Kryssanov, V. V. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7010003 · Cited by 3
Sriram, R., Sundaram, S. S., & Sophie, S. L. (2024) · Journal of Electronics Electromedical Engineering and Medical Informatics · doi:10.35882/jeeemi.v7i1.616 · Cited by 3
Reading comprehension powered semantic fusion network for identification of N-ary drug combinations
Zhang, H., Zhan, P., Yang, C., Yan, Y., Cai, Z., Shan, G., Jiang, B., Chen, B. Y., Gu, Q., & Zhou, Q. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110096 · Cited by 3
Leveraging LLMs for LLM-Generated Fake News Detection: Insights from COVID-19 Misinformation
Hong, D. N., Hashimoto, Y., Paik, I., & Thang, T. C. (2024) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn63059.2024.10847475 · Cited by 4
Qin, N., Liu, L., Wei, L., & Chen, H. (2025) · Journal of Alloys and Compounds Communications · doi:10.1016/j.jacomc.2025.100067 · Cited by 4
A Comprehensive Study on Advancements in Text Mining and Natural Language Processing
Mahire, S., Saravanan, U. K. B., Murali, M., & Prabakaran, N. (2024) · International Conference on Emerging Research in Computational Science (ICERCS) · doi:10.1109/icercs63125.2024.10895330 · Cited by 2
Konduru, K. K., Natalia, F., Sudirman, S., & Al‐Jumeily, D. (2024) · International Conference on Developments in eSystems Engineering (DeSE) · doi:10.1109/dese63988.2024.10911889 · Cited by 2
“They are the original, or cased, BERT [6], PubMedBERT [5], SciBERT [20], ClinicalBERT [7], and DistilBERT [21].”
Evidence Extraction for Automated Medical Coding: Preliminary Evaluation
Jiang, X., Khan, K., Vasantha, S. T., & Haider, S. (2024) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3711542.3711580 · Cited by 4
Ngo, N.-H., Nguyen, A.-D., Thi, Q.-T. P., & Dang, T. H. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4285-4_15 · Cited by 2
HsscBERT: pre-training domain model for the full text of Chinese humanity and social science
Shen, S., Zhao, Z., Yan, D., Jiang, C., & Wang, D. (2025) · npj Heritage Science · doi:10.1038/s40494-025-01747-2 · Cited by 2
“Beltagy et al. 31 released SciBERT, which out-performed BERT on scienti fi c text, addressing the scarcity of large-scale, high-quality scienti fi c data.”
Saleh, H., McCann, M., Breslin, J. G., & El–Sappagh, S. (2025) · Journal of Advances in Information Technology · doi:10.12720/jait.16.6.854-868 · Cited by 2
Mechanical Property Prediction of Materials via a Literature Mining Approach
Zhang, R., Zhang, Y., Chen, Q., & Han, Y. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035212 · Cited by 2
Generalized Knowledge-Enhanced Framework for Biomedical Entity and Relation Extraction
Nguyen, M., & Le, P. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6585-3_19 · Cited by 2
Measuring research difficulty of academic papers: A case study in natural language processing
Li, H., Li, J., Zhao, Y., Zhang, H., Yang, Y., Hu, Z., & Zhang, C. (2025) · Data Science and Informetrics · doi:10.1016/j.dsim.2025.06.001 · Cited by 2
Sakhovskiy, A., & Tutubalina, E. V. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729901 · Cited by 5
“In recent years, advancements in biomedical Natural Language Processing (NLP) have been largely driven by the development of domain-specific pre-trained Language Models (LMs) [1, 4, 17, 42, 44, 59, 84].”
Yu, F., An, W., Wang, H., & Yin, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05397-w · Cited by 4
Dao, H. N., Hashimoto, Y., Paik, I., & Thang, T. C. (2025) · IEEE Access · doi:10.1109/access.2025.3592943 · Cited by 2
Patel, D., Lain, A. D., Vijayaraghavan, A., Faghih-Mirzaei, N., Mweetwa, M. N., Wang, M., Beck, T., & Posma, J. M. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag418 · Cited by 2
“Three models were fine-tuned using their original weights: BioBERT (Lee et al. 2020), SciBERT (Beltagy et al. 2019), and SapBERT (Liu et al. 2021).”
InfluentialKabootari, M., Abdeahad, Y., Kheirkhah, Y., & Kheirkhah, E. (2025) · Computing · doi:10.1007/s00607-025-01546-4 · Cited by 2
Zhang, B., Chen, J., Liu, S., Schunkert, H., & Navab, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04978-0_58 · Cited by 2
Integrating insights into radicalization: A text‐mining systematic review
Knorr, A., Podolin-Danner, N., Cicco, G. D., & Kossowska, M. (2025) · Political Psychology · doi:10.1111/pops.70070 · Cited by 4
“Specifically, we used SciBERT embeddings (Beltagy et al., 2019) with agglomerative clustering to provide an additional assessment of corpus structure, which is less sensitive to document length (Zhang et al., 2022).”
Automated neural patent landscaping in the small data regime using citations and CPC codes
Erana, T. I., & Finlayson, M. A. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09483-5 · Cited by 2
Lu, J., Li, F., Feng, L., Yu, Z., Lu, J., Qin, M., She, S., Jiang, H., Guo, Z., Xia, J., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf534 · Cited by 3
“To strengthen our evaluation, we included two widely used domain-specific models in the biomedical field, like BioBERT [31] and SciBERT [32], as baselines.”
Kommineni, V. K., König‐Ries, B., & Samuel, S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3204 · Cited by 2
Improving Scientific Document Retrieval with Academic Concept Index
Lee, J., Lee, J., Kweon, W., Jin, B., Zhang, Y., Yoon, S., Lee, D., Yu, H., Han, J., & Kang, S. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3844616 · Cited by 2
Big Green at WNUT 2020 Shared Task-1: Relation Extraction as Contextualized Sequence Classification
Miller, C., & Vosoughi, S. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.36 · Cited by 3
Natural Language Processing Application to the Chemical Domain
Iwakura, T., & Yoshikawa, H. (2020) · Journal of Natural Language Processing · doi:10.5715/jnlp.27.969 · Cited by 2
“…セッションが開催され,それぞれ,9件,12件の発表が集まった.国外においても,CHEMDNER (Krallinger, Leitner, Rabal, Vazquez, Oyarzabal, and Valencia 2015)やCHEMPROT (Krallinger, Rabal, Akhondi, et al. 2017),ChEMU (He, Nguyen, Akhondi, Druckenbrodt, Thorne, Hoessel, Afzal, Zhai, Fang, Yoshikawa, Albahem, Cavedon, Cohn,…”
Analyzing the Domain Robustness of Pretrained Language Models, Layer by Layer
Kashyap, A. R., Mehnaz, L., Malik, B., Waheed, A., Hazarika, D., Kan, M., & Shah, R. R. (2021) · National University of Singapore · Cited by 6
Semantic Retrieval of Microbiome Information Based on Deep Learning
Alphonse, J., Binosh, A. N., Raj, S., Pal, S., & Melethadathil, N. (2021) · Lecture notes in electrical engineering · doi:10.1007/978-981-33-6987-0_4 · Cited by 2
Task Definition and Integration For Scientific-Document Writing Support
Narimatsu, H., Koyama, K., Dohsaka, K., Higashinaka, R., Minami, Y., & Taira, H. (2021) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2021.sdp-1.3 · Cited by 2
“We will also consider the use of domain-specific pretrained language models, such as SciBERT (Beltagy et al., 2019), in order to improve performance.”
MethodBiomedical Interpretable Entity Representations
García-Olano, D., Onoe, Y., Baldini, I., Ghosh, J., Wallace, B., & Varshney, K. R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.311 · Cited by 5
Learning Latent Structures for Cross Action Phrase Relations in Wet Lab Protocols
Kulkarni, C., Chan, J., Fosler‐Lussier, E., & Machiraju, R. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.525 · Cited by 3
Zhang, T., Wang, C., Qiu, M., Yang, B., Cai, Z., He, X., & Huang, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.197 · Cited by 2
ArgFuse: A Weakly-Supervised Framework for Document-Level Event Argument Aggregation
Kar, D., Sarkar, S., & Goyal, P. (2021) · Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) · doi:10.18653/v1/2021.case-1.5 · Cited by 5
An Investigation of Language Model Interpretability via Sentence Editing
Stevens, S., & Su, Y. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.34 · Cited by 4
“Pre-trained language models (PLMs) [1, 2, 3] are pervasively used in language-related tasks, but interpreting their predictions is notoriously difficult because of their parameters’ complex inter-dependencies.”
BackgroundExploring Language Patterns in a Medical Licensure Exam Item Bank
Padhee, S., Swygert, K. A., & Micir, I. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669681 · Cited by 2
Extracting Impacts of Non-pharmacological Interventions for COVID-19 From Modelling Study
Yang, Y., Cao, Z., Zhao, P., Zeng, D. D., Zhang, Q., & Luo, Y. (2021) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi53945.2021.9624840 · Cited by 3
Kaddari, Z., Berrich, J., Rahmoun, N., Belouali, S., & Bouchentouf, T. (2021) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds53782.2021.9626759 · Cited by 2
Soshnikov, D., Petrova, T. A., Soshnikova, V., & Grunin, A. A. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6010004 · Cited by 2
Weber, L. J., Kirchheim, A., & Zimmermann, A. P. (2022) · doi:10.5121/csit.2022.120304 · Cited by 2
Towards Automation of Topic Taxonomy Construction
Dauxais, Y., Zaratiana, U., Laneuville, M., Hernandez, S. D., Holat, P., & Grosman, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-01333-1_3 · Cited by 2
Ameri, K., Hempel, M., Sharif, H., Lopez, J., & Perumalla, K. S. (2022) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp2020022 · Cited by 3
Speaker Diarization and BERT-Based Model for Question Set Generation from Video Lectures
Nittala, S., Agarwal, P. R., Vishnu, R., & Shanbhag, S. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-0095-2_42 · Cited by 2
Ensemble-based Fine-Tuning Strategy for Temporal Relation Extraction from the Clinical Narrative
Wang, L., Miller, T., Bethard, S. J., & Savova, G. (2022) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2022.clinicalnlp-1.11 · Cited by 6
Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis
Kulick, S., Ryant, N., & Santorini, B. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.44 · Cited by 5
MinpakuBERT: A Language Model for Understanding Cultural Properties in Museum
Mibayashi, R., Ueta, M., Kawahara, T., Matsumoto, N., Yoshimura, T., Aihara, K., Kando, N., Shoji, Y., Nakajima, Y., Yamamoto, T., et al. (2022) · IIAI International Congress on Advanced Applied Informatics (IIAI-AAI) · doi:10.1109/iiaiaai55812.2022.00013 · Cited by 2
“For example, Beltagy et al. propose SciBERT [9], a domain-specific additional learning model for chemistry.”
BackgroundDrug Protein Interaction Extraction Using SciBERT Based Deep Learning Model
GabAllah, N. A., & Rafea, A. (2022) · Advances in intelligent systems and computing · doi:10.1007/978-3-031-14054-9_16 · Cited by 2
Lara-Clares, A., Lastra-Díaz, J. J., & García-Serrano, A. M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0276539 · Cited by 2
Domain Word Extension Using Curriculum Learning
Seong, S., & Cha, J.-W. (2023) · Sensors · doi:10.3390/s23063064 · Cited by 2
“Scibert [4] creates language models from the data of computer science and broad biomedical domain randomly sampled from Semantic Scholar [12].”
Topics in Contextualised Attention Embeddings
Talebpour, M., de Herrera, A. G. S., & Jameel, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_15 · Cited by 6
Guan, S., & Jiang, T. (2022) · International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI) · doi:10.1109/mlbdbi58171.2022.00078 · Cited by 4
“We employ SciBERT 24 to generate initial vectors for paper nodes.”
MethodInfluentialLessons from shortcomings in machine learning for medical imaging
Varoquaux, G. P., & Cheplygina, V. (2023) · OECD eBooks · doi:10.1787/b885eecd-en · Cited by 4
On the Impact of the Vocabulary for Domain-Adaptive Pretraining of Clinical Language Models
Lamproudis, A., & Henriksson, A. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-38854-5_16 · Cited by 2
Building a Corpus for Biomedical Relation Extraction of Species Mentions
Khettari, O. E., Quiniou, S., & Chaffron, S. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.21 · Cited by 4
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.