SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,501–2,600 of 3,596
Burmaoğlu, S., ARAVACIK, E. D., & Söyler, A. (2025) · Scientometrics · doi:10.1007/s11192-025-05457-1 · Cited by 2
“To directly compare publications and patents, we calculated cosine similarity between the SciBERT embeddings of each publication and all patents in our sample, identifying the most semantically similar patent for each publication.”
InfluentialHYDRA: Bridging Dynamic and Static User Modeling for Scientific Article Recommendation
Elayadi, K., & Yanes, N. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.537 · Cited by 1
Semi-Automatic Extraction and Analysis of Health Equity Covariates in Registered Research Projects
Nananukul, N., & Kejriwal, M. (2025) · Applied Sciences · doi:10.3390/app152211853 · Cited by 1
RanAT4BIE: Random Adversarial Training for Biomedical Information Extraction
Chen, J., Lv, S., & Su, L. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227694 · Cited by 1
Zhao, R., Li, W., Gonzalez-Aguirre, J., Chen, Y., Yuan, J. S., Pakrasi, H. B., Fei, C. J., Park, S., Roell, G. W., Chen, Y., et al. (2025) · Bioresource Technology · doi:10.1016/j.biortech.2025.133646 · Cited by 1
The Privacy Quagmire: Where Computer Scientists and Lawyers May Disagree
Zhao, Y., Chandrasekaran, V., Wies, T., & Subramanian, L. (2025) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/3772356.3772422 · Cited by 1
“We optionally filter out unlikely relationships by computing SciBERT [3] similarity scores between terms and removing pairs with similarity below a threshold.”
DiagDiff: A Diagnosis-Aware Diffusion Model for Radiology Report Generation
Niu, Y., Wang, L., Zhang, Y., Li, S., & Zhao, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_8 · Cited by 1
MediHerb: A multi-modal enhanced framework for disease inference via herbal knowledge
Liu, X., Chen, F., Pan, J., Ai, C., Guo, F., & Tang, J. (2025) · Computational Biomedicine · doi:10.70401/cbm.2025.0003 · Cited by 1
Shi, J., Wang, L., Liu, L., & Su, J. (2025) · BMC Genomics · doi:10.1186/s12864-025-12342-x · Cited by 1
“Building on this foundation, transformer-based models pretrained on domain-specific corpora, such as BioBERT [39], BioMegatron [40], and SciBERT [41], have significantly improved the performance of biomedical information extraction tasks.”
Chen, S., Hu, G., Ren, S., & Shu, T. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.104022 · Cited by 5
A Flexible and Configurable System to Author Name Disambiguation
Rodrigues, N. S., & Ralha, C. G. (2025) · IEEE Access · doi:10.1109/access.2025.3589957 · Cited by 2
Automated generation of research workflows from academic papers: a full-text mining framework
Zhang, H., & Zhang, C. (2025) · Journal of Informetrics · doi:10.1016/j.joi.2025.101732 · Cited by 1
“For research workflow paragraph classification, we used the “base” variants of BERT, RoBERTa, and SciBERT with a batch size of 64 and learning rates from 1e-5 to 5e-5.”
InfluentialAlmutairi, N., Coenen, F., & Elhanashi, A. (2025) · IEEE Access · doi:10.1109/access.2025.3640222 · Cited by 1
Challenges in Evaluating Clinical Remission in Severe Asthma: Insights from the Mayo Clinic
Lim, K. G., Deb, A., Corbridge, T. C., Lee, L., Kelloway, J. S., Simsek, S., Manoharan, M., Barman, H., & Wagner, T. (2025) · Advances in Therapy · doi:10.1007/s12325-025-03433-y · Cited by 1
“…from Transformers-based model was trained to classify the context of a mentioned disease or symptom in a patient’s clinical note as Yes (confirmed disease), No (ruled out disease), Maybe (possibility of disease), or Other (alternative context, such as family history) [25, 26, 29, 30].”
Large Language Models in Economics
Ash, E., Hansen, S. E., Muvdi, Y., & Marangon, C. (2025) · Springer eBooks · doi:10.1007/978-3-031-88240-1_8 · Cited by 1
SPECTER-BS: effective citation recommendation using SPECTER with bibliographic scoring
Nguyen, S. N., Long, N. H., Dinh, T. N., Pham, P., & Vo, B. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-025-02677-y · Cited by 2
“In this study, we presented a novel citation recommendation model SPECTER-BS that synergistically combines SPECTER [46] for textual embedding with an advanced bibliographic stage to calculate relevance scores for scientific articles.”
Modular Model Adaptation for Online Learning in Streaming Text Classification
Kim, M.-S., Liu, L., & Kwon, H.-Y. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3651427 · Cited by 2
“Thevocabularyindica-tordrawsinspirationfromfindingsthatincompletevocabulary coverageintheembeddinglayer,especiallyindomain-specific contexts,canhinderdownstreamperformance[61],[62].”
Food Hazard Text Classification and Entity Detection using BERT Model
Rachim, M. F. D., & Ruskanda, F. Z. (2025) · International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA) · doi:10.1109/icaicta67604.2025.11335102 · Cited by 1
“For example, models such as BioBERT [16] and SciBERT [17] are trained with biomedical literature and have shown strong transferability for food hazard detection tasks due to overlap in technical terminology.”
A Multi-Stage NLP Framework for Knowledge Discovery from Crop Disease Research Literature
Polpinij, J., Kaenampornpan, M., Khoo, C. S. G., Cheng, W., & Luaphol, B. (2026) · Mathematics · doi:10.3390/math14020299 · Cited by 2
Prediction of patent grant using interpretable citation-guided graph neural networks
Tong, X., Lu, Y., & Motohashi, K. (2026) · Scientometrics · doi:10.1007/s11192-025-05515-8 · Cited by 1
Utilizing large model for content-based identification of disruptive and developmental research
Huang, S., Lu, W., Huang, Y., Liu, Y., Cheng, Q., & Wang, Q. (2026) · Scientometrics · doi:10.1007/s11192-025-05513-w · Cited by 1
Evaluating Natural Language Processing and Named Entity Recognition for Bioarchaeological Data Reuse
Lien-Talks, A. G. W. (2026) · Heritage · doi:10.3390/heritage9010035 · Cited by 1
“In the current landscape, more powerful transformer-based Large Language Models (here onwards LLMs) such as BERT [14], BioBERT [15], SciBERT [16], and GPT-4 [17] have significantly advanced information extraction capabilities.”
Extracting Data on Patient Perspectives from Social Media
Luschi, A., Polisena, J., & Iadanza, E. (2026) · Health informatics · doi:10.1007/978-3-032-11284-2_7 · Cited by 1
Rahman, M. H. (2026) · Open Access Journal on Engineering Applications · doi:10.64886/oajea.0102.002 · Cited by 1
Miao, Q., Zhou, R. H. F. L., Meng, J., & Wei, Y. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356825 · Cited by 1
“Early models such as BioBERT [16] and SciBERT [17] achieved strong performance on NER, RE, and QA by learning biomedical semantics from corpora like PubMed.”
Where You Cite Matters: Section-Aware Embeddings for Citation Intent Classification
Nam, S., Phan, T. A., Kim, G., Jung, J. J., & Bui, K.-H. N. (2025) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf68649.2025.11365141 · Cited by 1
“Transformer-based models, including BERT [2] and SciBERT [3], have advanced the performance of citation intent classification.”
Zhang, X., Zhang, J., Pan, Y., Wang, C., Liu, J., Chong, X., & Wang, M. (2026) · BMC Geriatrics · doi:10.1186/s12877-026-07046-0 · Cited by 1
“Consequently, rural empty-nest elders confront high travel costs and delayed care [47], exacerbating health inequities [48].”
ProfFîlos: AI Approach for Automated Assessment of Student Translations from English to Arabic
Gawich, M., Abouelenine, S., & Alfonse, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-17557-1_4 · Cited by 1
Economic Profile Generation from Textual Data Using an Algorithm Framework
Babalola, O. T., Ojokoh, B. A., & Boyinbode, O. K. (2026) · Annals of Data Science · doi:10.1007/s40745-026-00687-3 · Cited by 1
Leveraging citation intent for publication relatedness
Phan, T. A., & Jung, J. J. (2026) · Discover Computing · doi:10.1007/s10791-026-09995-x · Cited by 2
“Regarding the citation intent classification model, we use the scibert-scivocab-uncased [48] with 12 layers, 768 hidden size, and 31,116 vocabulary size.”
AI-assisted multi-target classification for research-policy alignment in conservation science
McCarthy, C., Brooks, C. M., Sternberg, T., Shaney, K. J., & Hoshino, B. (2026) · Ecological Informatics · doi:10.1016/j.ecoinf.2026.103669 · Cited by 1
Farouk, G. M., Ismail, S., & Aref, M. M. (2026) · IEEE Access · doi:10.1109/access.2026.3667674 · Cited by 2
Kumar, S., Pal, A., & Chatterjee, S. (2026) · npj Systems Biology and Applications · doi:10.1038/s41540-026-00676-5 · Cited by 4
A comparative evaluation of transformer models for medical abstract classification
AnsariShiri, M., & Mansouri, N. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21140-w · Cited by 1
“Its effectiveness has been demonstrated on several scientific NLP tasks, including sequence tagging and text classification, making it a strong candidate for analyzing scientific and medical abstracts where precise technical language is prevalent [55].”
Bharti, P. K. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01059-z · Cited by 1
Zhao, H., Zhao, Y., & Zhang, C. (2026) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2026.124598 · Cited by 1
Moghadasi, M. N., & Ghaderi, F. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401965 · Cited by 1
“Architectural Scalability Correlation: The scalability distribution strongly correlates with architectural paradigms, with encoder-only and compression-optimized models demonstrating superior scalability compared to generative and specialized architectures.”
Patel, V. P., Jolkver, E., & Schwerk, A. M. K. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0018 · Cited by 2
“Early efforts centered on domain-pretrained encoders such as BioBERT [ 21 ], PubMedBERT [ 35 ], and SciBERT [ 36 ], each trained on large biomedical corpora.”
DustNET: Enabling machine learning and AI models of dusty plasmas
Wang, Z. (. A., Burton, J., Dormagen, N., Du, C.-R., Feng, Y., Foster, J. E., Glenn, S., Klein, M., Knapek, C. A., Matthews, L., et al. (2026) · Physics of Plasmas · doi:10.1063/5.0337599 · Cited by 1
Automatic Detection of Core Sections in Scientific Papers
Văruicu, M. E., Popescu, P., & Mihăescu, M. C. (2026) · Acta Universitatis Sapientiae Informatica · doi:10.1007/s44427-026-00028-5 · Cited by 1
Yamamoto, R., Takahashi, A., Terayama, K., Kumagai, Y., & Oba, F. (2026) · Advanced Intelligent Systems · doi:10.48550/arxiv.2603.26060 · Cited by 1
Fine-Tuning Large Language Models for the Efficient and Concurrent Extraction of Fuel Properties
Alshehri, A. S. (2026) · Applied Sciences · doi:10.3390/app16073320 · Cited by 1
Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching
Pan, Y., Ning, Z., Wang, L., & Du, Y. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729273 · Cited by 2
“We include scientific PLM retrievers SciBERT [17], SPECTER [18], and SciNCL [27], as well as the general-purpose dense retriever COCO-DR [38].”
Knowledge Evolution in the Mobile Industry via Embedding-Based Topic Growth and Typology Analysis
Jeon, S., Jung, W., & Cho, K. (2026) · Systems · doi:10.3390/systems14040415 · Cited by 1
Turdalyuly, M., Тұрсынхан, А., Yerimbetova, A., Turdalykyzy, T., Sakenov, B., Мукажанов, Н., & Baisholan, N. (2026) · Computers · doi:10.3390/computers15050268 · Cited by 1
Iqbal, A., Haq, M. I. U., Wahid, A., Muhammad, S., yahya, M., & Shahid, A. (2026) · IEEE Access · doi:10.1109/access.2026.3688794 · Cited by 1
Aspect-Aware Content-Based Recommendations for Mathematical Research Papers
Satpute, A., Greiner-Petter, A., Gießing, N., Teschke, O., Schubotz, M., Aizawa, A., & Gipp, B. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809531 · Cited by 1
“Let AS = { 𝑎𝑠 1 , . . . 𝑛 } denote the set of 𝑛 aspects defined in the dataset.”
Oh, J.-H., Nam, H.-J., Seo, S. H., & Park, H.-S. (2026) · Applied Sciences · doi:10.3390/app16104622 · Cited by 1
A Comparative Evaluation of AI Approaches to Large-Scale Scientific Subject Classification
Tanácsi, R., & Micsik, A. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10050151 · Cited by 1
“The authors compared traditional deep learning approaches and pre-trained language models (PLMs), and they found that SciBERT [5], which was pre-trained on a massive corpus of scientific publications, had the best performance.”
A Multi-Stage Hybrid Retrieval Framework for the Scientific Literature with Cross-Encoder Re-Ranking
Al-Joofi, W., Sagheer, A., & Hamdoun, H. (2026) · Applied Sciences · doi:10.3390/app16104813 · Cited by 1
Rostam, Z. R. K., & Kertész, G. (2026) · IEEE International Conference on Computational Cybernetics (ICCC) · doi:10.1109/iccc68994.2026.11511643 · Cited by 1
Zhu, E., & Hu, Y. (2026) · AI Magazine · doi:10.1002/aaai.70062 · Cited by 1
Liu, J., Pan, D., Kang, Y., Luo, L., Sun, Y., & Lin, H. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_6 · Cited by 1
Chawla, R., Dahiya, S., Singh, G., & Fortino, G. (2026) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2026.3691181 · Cited by 1
“…language models (LLMs) have demonstrated strong performance in document-level tasks in domains such as scientific question answering, legal analysis, and financial summarization, enabled by advances such as chain-of-thought (CoT) prompting and retrieval-augmented generation (RAG) [14], [15], [16].”
A Purpose-Aware Semantic Reasoning Model for Patent Infringement Detection in the DIKWP Network
Guo, Z., & Duan, Y. (2026) · Electronics · doi:10.3390/electronics15112449 · Cited by 1
An integrated LDA and SciBERT Model for Topic Analysis: A Case Study of Network Science
Wang, J., Lu, M., Wang, W., & Huang, X. (2026) · SAGE Open · doi:10.1177/21582440261426946 · Cited by 1
“Secondly, the SciBERT model (Beltagy et al., 2019) is used to identify similar words for each term in each topic within their context-specific semantic spaces, thus enriching the semantic context and reducing semantic information loss, thereby obtaining fine-grained topic features and enhancing…”
Nabil, H. A. M., Lutfi, S. L., & Mulki, H. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3878 · Cited by 1
Zhang, C., Hao, J., & Mao, Y. (2026) · Library & Information Science Research · doi:10.1016/j.lisr.2026.101414 · Cited by 1
The evolution of AI-integrated genome editing and its challenges
Wong, M. T. J., Zulkifli, N. D., Ravichandran, T., Vasodavan, K., Bin-Alshaibah, O., Himel, G. M. S., Achuthan, A., Anthonysamy, M. A., & Das, K. T. (2026) · Mammalian Genome · doi:10.1007/s00335-026-10248-x · Cited by 1
“…processing such as SciBERT can mine the biomedical literature to extract gene-disease associations, genetic variants, and biomolecular pathway relationships, enabling the synthesis of disparate data sources to contex-tualize genome editing findings (Beltagy et al. 2019 ; Lee and Lee 2020 ).”
Zhang, X., Guan, X., & Zheng, J. (2026) · Global and Planetary Change · doi:10.1016/j.gloplacha.2026.105592 · Cited by 1
Evaluation and Explainability of Unsupervised Scholarly Collaboration Recommendations
Noor, M. A., Sheppard, J. W., & Clark, J. A. (2026) · International Conference on Machine Learning and Applications · doi:10.1109/icmla69890.2026.00122 · Cited by 1
“Despite these advances, evaluation in collaboration recommendation still relies heavily on historical co-authorship.”
Liu, J., Luo, L., Pan, D., Kang, Y., Sun, Y., Lin, H., Yu, L., & Zhuang, H. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00473-8 · Cited by 1
Buenstorf, G., Hindiyeh, A., Koenig, J., Neumann, L. P., & Özgun, B. (2026) · Scientometrics · doi:10.1007/s11192-026-05736-5 · Cited by 1
V., N. S. P., & Rao, M. V. P. C. S. (2026) · International journal of intelligent engineering and systems · doi:10.22266/ijies2026.0831.22 · Cited by 1
Bailo, D., Lauciani, V., Stefano, R. D., & Bernardi, F. (2026) · Earth Science Informatics · doi:10.1007/s12145-026-02197-5 · Cited by 1
“However, models’ training and fine-tuning present important limitations for scientific knowledge management, as shown in Table 1 .”
HukukBERT: Domain-Specific Language Model for Turkish Law
Öztürk, M. U., Türkoğlu, T., & Buz-Yalug, B. (2026) · Journal of Computational Law and Legal Technology · doi:10.47852/bonviewjcllt620210346 · Cited by 2
“The limitations of general-domain models have driven a significant paradigm shift toward DAPT across highly specialized fields, including biomedicine (e.g., BioBERT [9]) and science (e.g., SciBERT [10]) [11].”
Natural language embeddings of synthesis and testing conditions enhance glass dissolution prediction
Mannan, S., Nambudiripad, K. S., Mandal, I., Gosvami, N. N., & Krishnan, N. M. A. (2026) · Digital Discovery · doi:10.1039/d6dd00191b · Cited by 1
Methods for Extracting Information from Messages from Primary Care Providers to Specialists
Ding, X., Barnett, M. L., Mehrotra, A., & Miller, T. A. (2020) · doi:10.18653/v1/2020.nlpmc-1.1 · Cited by 1
An Improved Mask Approach Based on Pointer Network for Domain Adaptation of BERT
Lu, P., Jiang, D., & Li, Y. (2020) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1646/1/012072 · Cited by 1
Learning Informative Representations of Biomedical Relations with Latent Variable Models
Shah, H., & Fauqueur, J. (2020) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2020.sustainlp-1.3 · Cited by 1
SU-NLP at WNUT-2020 Task 2: The Ensemble Models
Fayoumi, K., & Yeniterzi, R. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.61 · Cited by 1
Saha, S., Das, S., Khurana, P., & Srihari, R. K. (2020) · Cited by 1
Gim, J. (2020) · JOURNAL OF ADVANCED INFORMATION TECHNOLOGY AND CONVERGENCE · doi:10.14801/jaitc.2020.10.2.139 · Cited by 1
“Research has been conducted to improve LM's performance through domain-specific pre-training, and PubMedBERT [35], SciBERT [36], ClinicalBERT [37], etc., have been proposed and are widely used to perform natural languages processing tasks such as NER, Relation Extraction, and Document…”
MethodCOPER: a Query-Adaptable Semantics-based Search Engine for Persian COVID-19 Articles
Khanmohammadi, R., Mirshafiee, M. S., & Allahyari, M. (2021) · International Conference on Web Research (ICWR) · doi:10.1109/icwr51868.2021.9443151 · Cited by 1
PSED: A Dataset for Selecting Emphasis in Presentation Slides
Shirani, A., Tran, G., Trinh, H., Dernoncourt, F., Lipka, N., Echevarria, J., Solorio, T., & Asente, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.377 · Cited by 1
Towards an Automated Classification Approach for Software Engineering Research
Kaplan, A., & Keim, J. (2021) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3463274.3463358 · Cited by 1
“There is also a BERT model that was trained specifically on scientific text that promises better results on such text [2].”
BackgroundCarrillo-Larco, R. M., Castillo‐Cara, M., & Lovón-Melgarejo, J. (2021) · Wellcome Open Research · doi:10.12688/wellcomeopenres.16867.1 · Cited by 1
BioTMPy: A Deep Learning-Based Tool to Classify Biomedical Literature
Alves, N., Rodrigues, R., & Rocha, M. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-86258-9_12 · Cited by 1
An Empirical Study of Span Modeling in Science NER
Jiang, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86324-1_4 · Cited by 1
A Relation-aware Attention Neural Network for Modeling the Usage of Scientific Online Resources
Xu, Y., Huang, H., Feng, C., Zhou, C. D., Zhang, J., & Hu, Y. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533951 · Cited by 1
“Ours-scibert To illustrate the importance of in-domain pre-training, we replaced general BERT base with SciBERT based [26] which is a pre-trained language model based on BERT but trained on a large corpus of scientific publication.”
MethodGlobal Research on Coronaviruses: Metadata-Based Analysis for Public Health Policies
Warin, T. (2021) · JMIR Medical Informatics · doi:10.2196/31510 · Cited by 3
Will Sorafenib Help? Treatment-aware Reranking in Precision Medicine Search
Rybinski, M., & Karimi, S. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482220 · Cited by 1
“Reranking with SciBERT [7], using a monoBERT is successfully applied by SCUT-CCNL team in TREC PM 2019 [20], resulting in close to stateof-the-art effectiveness for both tasks of scientific literature and clinical trials retrieval.”
MethodEasy Semantification of Bioassays
Anteghini, M., D’Souza, J., dos Santos, V. A. P. M., & Auer, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-08421-8_14 · Cited by 1
Zhang, Y., & Ma, Q. (2021) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3486622.3494002 · Cited by 1
“The recent language modelling models such as BERT [5], SciBERT [2] are effective on multiple NLP tasks.”
MethodInfluentialChange Summarization of Diachronic Scholarly Paper Collections by Semantic Evolution Analysis
Paharia, N., Pozi, M. S. M., & Jatowt, A. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00067 · Cited by 1
Beyond the Polarities: Sentiment Analysis of French Restaurant Reviews Using BERT-based Models
Kang, H. J., & Eshkol-Taravella, I. (2021) · International Conference on Behavioral, Economic, and Socio-Cultural Computing · doi:10.1109/besc53957.2021.9635309 · Cited by 1
How to pretrain an efficient cross-disciplinary language model: The ScilitBERT use case
de la Broise, J.-B., Bernard, N., Dubuc, J.-P., Perlato, A., & Latard, B. (2021) · International Conference on Information Technology Research (ICITR) · doi:10.1109/icitr54349.2021.9657164 · Cited by 1
Multi-Round Parsing-based Multiword Rules for Scientific Knowledge Extraction
Kuebler, J., Tong, L., & Jiang, M. (2021) · IEEE International Conference on Big Knowledge (ICBK) · doi:10.1109/ickg52313.2021.00051 · Cited by 2
Jiang, X., He, K., & Yang, B. (2022) · IEEE Access · doi:10.1109/access.2022.3159338 · Cited by 1
RES: An Interpretable Replicability Estimation System for Research Publications
Wang, Z., Feng, Q., Chatterjee, M., Zhao, X., Liu, Y., Li, Y., Singh, A. K., Shipman, F. M., III, Hu, X., & Caverlee, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21737 · Cited by 1
Separate then Constrain: A Hierarchical Network for End-to-End Triples Extraction
Wang, H., Fu, Y., Hu, L., Jiang, W., & Pu, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05933-9_42 · Cited by 1
BioCite: A Deep Learning-based Citation Linkage Framework for Biomedical Research Articles
Roy, S. S., & Mercer, R. E. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.23 · Cited by 1
Multi-Task Learning for Chemical Named Entity Recognition with Chemical Compound Paraphrasing
Watanabe, T., Tamura, A., Ninomiya, T., Makino, T., & Iwakura, T. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.294 · Cited by 1
Ma, X., Lu, Y., Lu, Y., & Jiang, M. (2022) · Current Bioinformatics · doi:10.2174/1574893617666220718104518 · Cited by 1
Utilizing Out-Domain Datasets to Enhance Multi-task Citation Analysis
Mercier, D., Rizvi, S. T. R., Rajashekar, V., Ahmed, S., & Dengel, A. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-10161-8_6 · Cited by 1
DrNote: An open medical annotation service
Frei, J., Soto‐Rey, I., & Krämer, F. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000086 · Cited by 3
Learning Ontology Classes from Text by Clustering Lexical Substitutes Derived from Language Models1
Revenko, A., Mireles, V., Breit, A., Bourgonje, P., Schneider, J. M., Khvalchik, M., & Rehm, G. (2022) · Studies on the semantic web · doi:10.3233/ssw220018 · Cited by 1
How do Patent Claims Reflect Scientific Knowledge : Insights from Human Genomics
Lai, M. Y. (2022) · Portland International Conference on Management of Engineering and Technology (PICMET) · doi:10.23919/picmet53225.2022.9882736 · Cited by 1
“Therefore, in this work, I choose to follow a BERT model trained on scientific corpus provided by Beltagy et al. [17], which have successful use cases in biomedical field [18, 19] relevant to our needs and the aforementioned limitation.”
MethodSemantic Data Science in the COVID‐19 Pandemic
DeBellis, M., & Dutta, B. (2022) · Wiley eBooks · doi:10.1002/9781119865339.ch15 · Cited by 1
Thesis Review and Analysis Automated System
Lostaunau, J. P., Soto, A., & Barrientos, A. (2022) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct56874.2022.9953855 · Cited by 1
“Prioritization over the models chosen after the filter reduced the possible candidate models to SciBERT [22] and SpERT [23] The models meet the established requirements, were trained with scientific articles and are BERT distributions, which means that they are derived from a general purpose model,…”
Method
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.