SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,401–2,500 of 3,596
DISTANT: Distantly Supervised Entity Span Detection and Classification
Yano, K., Miwa, M., & Ananiadou, S. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.14 · Cited by 2
Yepes, A. J., & Verspoor, K. M. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.4 · Cited by 3
Deep Learning Models for Automatic De-identification of Clinical Text
Sriram, R., Sundaram, S. S., & Sophie, S. L. (2023) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-39811-7_10 · Cited by 3
CLARK: Building Conversational Intelligence for Knowledge Management in the Space Domain
Henriquez, M., Herd, A., Danėnas, P., Dilijonas, D., & Ontiveros, J. A. M. (2023) · European Conference on Knowledge Management · doi:10.34190/eckm.24.1.1792 · Cited by 2
Named Entity Recognition in a Very Homogenous Domain
Agarwal, O., & Nenkova, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.138 · Cited by 3
Arora, H., Shinde, K., & Ghosal, T. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00015 · Cited by 2
“(3) Shared Feature Extraction (SFE): Introduced in [26], the model uses SciBERT [3] embeddings and feeds the encoded representations directly to a shared layer for feature extraction.”
MethodA Joint Entity and Relation Extraction Approach Using Dilated Convolution and Context Fusion
Kong, W., Xia, Y., Yao, W., & Lu, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_11 · Cited by 2
PGB: A PubMed Graph Benchmark for Heterogeneous Network Representation Learning
Lee, E. W., & Ho, J. C. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615128 · Cited by 4
“SPECTER [12] learns the document embedding using a SciBERT model [5] pre-trained on relatedness signals derived from the citation graph.”
MethodAlshouha, B., Serrano‐Guerrero, J., Elizondo, D. A., Romero, F. P., & Olivas, J. Á. (2024) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.104093 · Cited by 3
Machine learning at the edge to improve in-field safeguards inspections
Shoman, N., Williams, K., Balsara, B., Ramakrishnan, A., Kakish, Z., Coram, J. L., Honnold, P., Rivas, T., & Smartt, H. A. (2024) · Annals of Nuclear Energy · doi:10.1016/j.anucene.2024.110398 · Cited by 2
A Semantic-Driven Generative Information Extraction Framework Oriented to Patent Text
Zhao, Y., Wang, T., & Zhang, S. (2023) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa-bdcloud-socialcom-sustaincom59178.2023.00124 · Cited by 2
“We employ ChemProt dataset [19] as a relation extraction test data for evaluating the SGIE framework to generate performance metrics.”
MethodA Narrative Review on Human Anxiety Detection in Social Media using Natural Language Processing
Senanayake, A., & Liyanage, S. R. (2023) · The journal of desk research review and analysis. · doi:10.4038/jdrra.v1i2.20 · Cited by 2
“Vanilla BERT has become the foundation for many LLM builds intending to handle different tasks like Bio BERT for biological concept analysis (Lee et al., 2020), Clinical-BERT for biomedical text analysis (Alsentzer et al., 2019), SciBERT for scientific literature analysis (Beltagy et al., 2019).”
MethodCamemBERT-bio : Un modèle de langue français savoureux et meilleur pour la santé
Touchent, R., Romary, L., & de la Clergerie, É. V. (2023) · HAL · Cited by 30
Learning Section Weights for Multi-label Document Classification
Fard, M. M., Bayod, P. S., Motarjem, K., Nejadi, M. A., Akhondi, S. A., & Thorne, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_34 · Cited by 2
Abstract Hardware Grounding Towards the Automated Design of Automation Systems
Shi, Y.-Z., Qiao, X., Meng, F., Ruan, L., & Wang, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-0780-8_9 · Cited by 2
MeDa-BERT: A medical Danish pretrained transformer model
Pedersen, J. S., Laursen, M. S., Vinholt, P. J., & Savarimuthu, T. R. (2023) · University of Southern Denmark Research Portal (University of Southern Denmark) · Cited by 4
A Self-supervised Approach for Semantic Indexing in the Context of COVID-19 PandemicPreprint
Ebadi, N., & Najafirad, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03544 · Cited by 2
Domain-Transferable Method for Named Entity Recognition TaskPreprint
Mikhailov, V., & Shavrina, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12170 · Cited by 2
“SciBERT [5] and BioBERT [6] achieved significant improvement over original BERT [7] on a number of downstream tasks within the domains.”
BackgroundLearning to Emphasize: Dataset and Shared Task Models for Selecting Emphasis in Presentation SlidesPreprint
Shirani, A., Tran, G., Trinh, H., Dernoncourt, F., Lipka, N., Asente, P., Echevarria, J., & Solorio, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03237 · Cited by 2
Unsupervised Extractive Summarization by Human Memory SimulationPreprint
Cardenas, R., Gallé, M., & Cohen, S. B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08392 · Cited by 2
“In addition to the aforementioned unsupervised baselines, we compare our models against a supervised baseline based on SciBert (Beltagy et al., 2019) and using the pretrained models served by HuggingFace.3 We add a linear classifier layer on top of the transformer model and fine-tune it over the same subset used to fi…”
MethodRegularizing Models via Pointwise Mutual Information for Named Entity RecognitionPreprint
Jeong, M., & Kang, J. (2021) · arXiv (Cornell University) · Cited by 2
Zhang, J., Bolanos, L. D., Li, T., Tanwar, A., Freire, G., Yang, X., Ive, J., Gupta, V., & Guo, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01935 · Cited by 5
“0 (Lee et al., 2019) pre-trained on PubMed and PMC, (3) Clinical BERT (Alsentzer et al., 2019) pre-trained based on BioBERT and MIMIC-III discharge summaries, (4) SciBERT (Beltagy et al., 2019) pre-trained for scientific literature.”
MethodDoes constituency analysis enhance domain-specific pre-trained BERT models for relation extraction?Preprint
Tang, A., Deléger, L., Bossy, R., Zweigenbaum, P., & Nédellec, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.02955 · Cited by 3
BioAct: Biomedical Knowledge Base Construction using Active LearningPreprint
Wright, D., Gentile, A. L., Faux, N. G., & Beck, K. L. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.14.488416 · Cited by 4
“In all experiments we used SciBERT [4] as the machine learning model, with the entropy of the output classifier used for ranking in uncertainty sampling.”
MethodInfluentialSchoeberl, C., Toney, A., & Dunham, J. (2023) · doi:10.51593/20220030 · Cited by 4
Named entity extraction for food safety events monitoring
Lee, Z. H. N. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/172106 · Cited by 3
Large Language Models for Multi-Choice Question Classification of Medical SubjectsPreprint
Ponce-López, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.14582 · Cited by 2
Abstractive Summarization Model for Summarizing Scientific ArticlePreprint
Ülker, M., & Özer, A. B. (2024) · Preprints.org · doi:10.20944/preprints202405.1123.v1 · Cited by 2
Text Mining for Bioinformatics Using Biomedical Literature
Lamúrias, A., Sousa, D., & Couto, F. M. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00017-8 · Cited by 2
Trajectories of Change: Approaches for Tracking Knowledge EvolutionPreprint
Schlattmann, R., & Vogl, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.00391 · Cited by 2
“After experimenting with several models, including SciBERT [56], PhysBERT [57], and OpenAI’s text-embedding-3-large , we found text-embedding-3-large generated embeddings that were most accurate and meaningful in terms of representation tuning.”
Zucco, C. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00249-9 · Cited by 2
Horizon Scans can be accelerated using novel information retrieval and artificial intelligence toolsPreprint
Schmidt, L., Sharma, O., Marshall, C., & Gonzalez-Moral, S. G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.01627 · Cited by 4
Investigating the originality of scientific papers across time and domain: A quantitative analysisPreprint
Culbert, J. H., Kenett, Y. N., & Mayr, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.07892 · Cited by 2
“…sentences s i , and the embedding vector from the BERT model at layer k defined as BERT k ( s To investigate whether SciBERT is a better embedding model for calculating DSI of scientific papers, we computed the DSI using the cased AllenAI’s SciBERT model (Beltagy et al. 2019) as embedding model.”
InfluentialXiong, X., Wang, H., Wu, K., Yang, Z., Zhao, H., Wang, H., Liu, H., & Xiong, D. (2026) · doi:10.36227/techrxiv.177155935.57684125/v1 · Cited by 2
Boukkouri, H. E. (2021) · theses.fr (ABES) · doi:10.70675/4c63eb46zd8cfz4a5dz80e9z666563e756d8 · Cited by 10
SolutionTailor: Scientific Paper Recommendation Based on Fine-Grained Abstract Analysis
Takahashi, T., & Katsurai, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_40 · Cited by 1
The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through
Singh, S., & Singh, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.249 · Cited by 1
A review on method framework construction of Chinese Information Science
Li, B., Tian, L., Zhang, Y., Zhang, H., & Zhang, C. (2022) · Data and Information Management · doi:10.1016/j.dim.2022.100023 · Cited by 1
Query Resolution of Literature Knowledge Graphs Using Hybrid Document Embeddings
Muhammad, I., Coenen, F., Gamble, C. L., Kearney, A., & Williamson, P. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21441-7_7 · Cited by 1
AcX: system, techniques, and experiments for acronym expansion
Pereira, J. L. M., Casanova, J., Galhardas, H., & Shasha, D. E. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3551793.3551812 · Cited by 3
Investigation of Biomedical Named Entity Recognition Methods
Çelikten, A., Onan, A., & Bulut, H. (2023) · Engineering cyber-physical systems and critical infrastructures · doi:10.1007/978-3-031-31956-3_18 · Cited by 1
Sengupta, S., Ghosh, S., Nakov, P., & Mitra, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.27019 · Cited by 1
“For example, SciBERT (Beltagy, Lo, and Cohan 2019) when applied to COVID-QA (M ¨ oller et al. 2020) yields mediocre performance (Sengupta et al. 2022), about 0.45 F1 and 0.25 EM, and even when fine-tuned, it does not improve significantly: 0.54 F1 and 0.29 EM.”
MethodInfluentialMulti-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction
Li, Y., Martschat, S., & Ponzetto, S. P. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.1 · Cited by 1
“…intermediate pre-training: We use B ERT B ASE 3 (Devlin et al., 2019) as the baseline model representing the canonical text domain, and SciBERT 4 (Beltagy et al., 2019), which we expect to be more closely related to the domains of our measurement extraction corpus, because it was pre-trained…”
MethodMuLMS-AZ: An Argumentative Zoning Dataset for the Materials Science Domain
Schrader, T. P., Bürkle, T., Henning, S., Tan, S., Finco, M., Grünewald, S., Indrikova, M., Hildebrand, F., & Friedrich, A. (2023) · Workshop on Computational Approaches to Discourse (CODI) · doi:10.18653/v1/2023.codi-1.1 · Cited by 1
Tracking the Evolution of Covid-19 Symptoms through Clinical Conversations
da Silva, T. C., de Macêdo, J. F., & Magalhães, R. P. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.6 · Cited by 1
Automating Citation Intent Classification on SciCite Dataset
Lenin, A., Chauhan, P., Mehta, T., Baliyan, N., & Sharma, D. K. (2023) · International Conference on Computer, Electronics & Electrical Engineering & their Applications (IC2E3) · doi:10.1109/ic2e357697.2023.10262812 · Cited by 1
A Scientific Document Retrieval and Reordering Method by Incorporating HFS and LSD
Feng, Z., & Tian, X. (2023) · Applied Sciences · doi:10.3390/app132011207 · Cited by 1
Adding Linguistic Information to Transformer Models Improves Biomedical Event Detection?
Zanella-Calzada, L. A., & Toussaint, Y. (2023) · Annals of Computer Science and Information Systems · doi:10.15439/2023f2076 · Cited by 1
“The transformer model, BERT [6], and four BERT variants pretrained in the biomedical domain, BioBERT [7], SciBERT [8], PubMedBERT [20], and BioMedRoBERTa [21], are used and compared for the detection of event triggers.”
MethodFraunhofer IAIS, S. A., Gupta, V., Viciano, A., Medien-Doktor Gesundheit, I. F. J., Wormer, H., Lehrstuhl f¨ur Wissenschaftsjournalismus, I. F. J., Mousavinezhad, N., & Fraunhofer IAIS, S. A. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_049 · Cited by 1
Meng, Z. (2023) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml60161.2023.10424807 · Cited by 1
“Some other models based on BERT were presented, such as BioBERT model and SciBERT model, which use the specialized data from different fields to train better embedding vectors for the corresponding task [9, 10].”
MethodEnhancing Academic Title Generation Using SciBERT and Linguistic Rules
Callegari, E., Vajdečka, P., Xhura, D., & Ingason, A. K. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.10 · Cited by 1
SciCN: A Scientific Dataset for Chinese Named Entity Recognition
Yang, J., Ji, B., Li, S., Ma, J., & Yu, J. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.035594 · Cited by 1
RehabGPT: a MaaS-based solution to large model building for digital rehabilitation
Wang, H., Zhong, Z., Dai, L., Wu, F. H., Ma, X., Xue, J., Pavel, S., Lam, P. L. K. M., Zhang, Q., & Lin, F. (2024) · International Workshop on Advanced Imaging Technology (IWAIT) · doi:10.1117/12.3019532 · Cited by 1
“In the fields of science [23], medicine [24], and law [25], pre-trained models based on technical corpora have shown good application prospects.”
BackgroundTuan, K. N., & Văn, T. Đ. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_13 · Cited by 1
RepoFromPaper: An Approach to Extract Software Code Implementations from Scientific Publications
Stankovski, A., & Garijo, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_7 · Cited by 3
A Data-Driven Approach for Integrating Multi-Source Scientific Categorization Labels
Wu, H., Li, J., & Wu, D. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.08.190 · Cited by 1
Improving Related Work Generation through Knowledge Aggregation from Citation Networks
Niu, K., Wang, H., Wang, Z., & Han, M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650144 · Cited by 1
“Following SPECTER, we utilized a contrastive learning approach based on Scibert [19], specializing in academic text analysis.”
MethodWang, H., Li, H., Fu, L., Liu, X., & Xu, J. (2024) · doi:10.1109/icsip61881.2024.10671517 · Cited by 2
Wang, X., Lin, Z., Shi, J. J., & Sun, Y. (2024) · Journal of Health Communication · doi:10.1080/10810730.2024.2426810 · Cited by 3
“We fine-tuned it with our labeled input sequences, as recent studies demonstrate the model’s adaptability with domain-specific data (Beltagy et al., 2019; J. Lee et al., 2020; Müller et al., 2020).”
Dhawan, M., & Purohit, L. (2024) · SN Computer Science · doi:10.1007/s42979-024-03472-2 · Cited by 1
“Similarly, clinical notes were used as corpora in the training of BERT-based transfer learning models by Si et al. [18].”
eCommerce Search and Discovery
Kallumadi, S. T., & King, T. H. (2024) · ACM eBooks · doi:10.1145/3674127.3674140 · Cited by 1
ODSBAHIA-PTBR: A Natural Language Processing Model to Support Sustainable Development Goals
Santos, E. S., & Moraes, L. E. (2024) · Revista de Gestão Social e Ambiental · doi:10.24857/rgsa.v18n12-039 · Cited by 1
S2-HTC: Hierarchical Text Classification via Fusing the Structural and Semantic Information
Shen, Y., Yan, Y., Yin, D., & Shen, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5569-1_4 · Cited by 1
Pham, H.-L., Mibayashi, R., Yamamoto, T., Kato, M. P., Yamamoto, Y., Shoji, Y., & Ohshima, H. (2025) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2024dat0003 · Cited by 1
Integration of BERT Models in NAO Robot for Library Assistance
Bouari, Y. A., & Romero, R. A. F. (2024) · Brazilian Symposium on Robotics and Workshop on Robotics in Education (SBR/WRE) · doi:10.1109/sbr/wre63066.2024.10838042 · Cited by 1
“Beltagy et al. (2019) introduced SciBERT, demonstrating the adaptability of BERT to specific domains, relevant to adapting BERT to library-specific terminologies [12].”
Pre-Trained Language Models with Topic Attention for Supervised Document Structure Learning
Pham, D., & Le, T. M. V. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825111 · Cited by 1
Toward Automatic Dataset Discovery From Scientific Publications
Kumar, S., Ghosal, T., & Ekbal, A. (2025) · IEEE Access · doi:10.1109/access.2025.3532767 · Cited by 2
Becker, M., Han, K., Rezapour, R., Diesner, J., & Witt, A. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09928-6 · Cited by 1
“For instance, SciBERT (Beltagy et al., 2019), a language model tailored for scientific text through fine-tuning BERT on a large corpus of scientific publications, has become one of the most widely used models for scientific document categorization (Cai et al., 2022; Maheshwari et al., 2021).”
Бубнов, А. С., Gallini, N. I., Grishin, I. Y., Kobozeva, I. M., Loukachevitch, N., Панич, М. Б., Раевский, Е. Н., Садковский, Ф. А., & Тимиргалеева, Р. Р. (2024) · Doklady Mathematics · doi:10.1134/s106456242470234x · Cited by 1
Leveraging Knowledge Graphs for Paper Recommendation Systems
Mataoui, M., Bahloul, H., & Ziane, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-82150-9_18 · Cited by 2
Text-guided diverse-expression diffusion model for molecule generation
Weng, W., Jiang, H., 孔, X. 祥. K., & Pau, G. (2025) · Chinese Physics B · doi:10.1088/1674-1056/adbedd · Cited by 1
A Bottom-Up Approach to Knowledge Graph Construction for Health Innovation Ecosystems
Barbosa, A., & Filho, R. H. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.03.141 · Cited by 1
A Joint Entity and Relation Extraction Model Driven by Fine-Grained Feature Extraction
Yu, H., Cao, H., Dong, C., Wang, H., Liang, H., & Cui, Y. (2025) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001425540096 · Cited by 1
Transformer-Based Named Entity Recognition for Automated Server Provisioning
Damavandi, H., Jalali, H., & Pishgoo, B. (2025) · International Conference on Web Research (ICWR) · doi:10.1109/icwr65219.2025.11006240 · Cited by 1
“Domainspecific models such as BioBERT [20], SciBERT [21], ClinicalBERT [22], and FinBERT [23] further illustrate the advantage of domain-tailored transformers in capturing specialized vocabularies and context.”
The Probing Machine: Can Using GenAI Tools Help With Better Reflections During User Interviews
Tran, C., Parayno, R. L., Venkitachalam, B. K., Deja, J. A., & Deja, J. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93418-6_6 · Cited by 1
Advancing Sustainable Communities in Scientific Oss: a Replication Study With Astropy
Sun, J., Patil, A. A., Li, Y., Guo, J., & Zhou, S. (2025) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00027 · Cited by 1
“With advances in large language models (LLMs), more LLM-based tools have emerged to support SE tasks [97] and scientific research [98], [99].”
Dynamic task balancing for joint information extraction
Hao, A., Sun, S., Su, J., Hui, S. C., & Luu, A. T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130638 · Cited by 1
Identifying Important Citation Through Cascading Models and NLP Techniques
Hannachi, F., Hammache, A., & Saidani, F. R. (2024) · International Conference of the African Federation of Operational Research Societies (AFROS) · doi:10.1109/afros62115.2024.11037110 · Cited by 1
RiskBERT: A Pre-Trained Insurance-Based Language Model for Text Classification
Hussain, R. G. (2025) · International Journal of Innovative Technology and Exploring Engineering · doi:10.35940/ijitee.f1097.14070625 · Cited by 1
Audrito, D., Spada, I., Mignone, R., Sulis, E., & Di, L. (2025) · Computer law & security review · doi:10.1016/j.clsr.2025.106171 · Cited by 3
Silva, M. O., & Moro, M. M. (2025) · Linguamática · doi:10.21814/lm.17.1.443 · Cited by 1
Elliott, T. J., Levitt, S. P., Nixon, K. J., & Bekker, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96262-2_15 · Cited by 1
Efficient Domain-Adaptive Continual Pretraining for the Process Industry in the German Language
Zhukova, A., Matt, C. E., & Gipp, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02551-7_8 · Cited by 3
Reduction of supervision for biomedical knowledge discovery
Theodoropoulos, C., Coman, A. C., Henderson, J., & Moens, M. (2025) · BMC Bioinformatics · doi:10.1186/s12859-025-06187-0 · Cited by 1
“This performance gap is most profound on the GAD dataset, followed by BioInfer, highlighting the competitiveness and effectiveness of our proposed methods.”
Zhang, H., Yao, R., Jia, Q., Qi, S., He, Y., Zhao, J., & Chuan, L. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1636024 · Cited by 1
Fine-Tuning Pretrained Language Models for Automated Research Papers Classification
Mghari, M., Mhamedi, A., & Hibaoui, A. E. (2025) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc66714.2025.11135408 · Cited by 1
“…in natural language processing (NLP), particularly the emergence of transformer-based large language models (LLMs) such as GPT [1], BERT [2] , RoBERTa [3], and their domain-specific variants like SciBERT [4], have demonstrated remarkable performance in various text classification tasks.”
Enhanced entity resolution and iterative inference mechanisms for document-level relation extraction
Chang, G., Yang, H., Tan, X., Chen, K., & Zhang, X. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02814-7 · Cited by 2
“For CDR and GDA datasets in the biomedical domain, this study employs SciBERT [ 30 ], which is a BERT model pretrained exclusively on biomedical corpora and suitable for NLP tasks in the biomedical context.”
Scilinkbert: a BERT-style language model for understanding scientific texts with citations
Yu, J.-Y., Yang, D., & Lee, K.-H. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07838-y · Cited by 2
“The second approach builds models from scratch, as in SciBERT [3], PubMedBERT [9], and BioLinkBERT [39], which ensures deeper domain alignment but at the cost of significant computational overhead.”
VMHQA: A Vietnamese Multi-choice Dataset for Mental Health Domain Question Answering
Nguyen, T. A. H., Nguyen, Q.-D., Nguyen, H. M., Nguyen, A. H., & Nguyen, L. T. T. (2025) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.7678 · Cited by 3
“At the same time, SciBERT [26], pre-trained on scientific publications, enhances performance in various NLP tasks such as sequence tagging and dependency parsing, particularly within scientific domains.”
Knowledge-Enriched Cell-Type Annotation in Single-Cell Transcriptomics via LLM Embeddings
Fabbricatore, A., Buffa, F. M., & Tangherloni, A. (2025) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb66090.2025.11177124 · Cited by 1
“While specialised biomedical variants of these models (e.g., BioBERT [30], SciBERT [31]), GeneBERT [32], DNABERT [33], and BioGPT [34] exist, recent work indicates that even generic LLMs (e.g., DistilBERT [35]) can be fine-tuned efficiently for biomedical and non-biomedical tasks, achieving…”
SMURF: Federated Multimodal Retrieval for Scientific Data via Embedding Alignment
Oh, S. Y., Khan, A., Foster, I. T., & Chard, K. (2025) · IEEE International Conference on eScience · doi:10.1109/escience65000.2025.00091 · Cited by 3
“Each modality is encoded using a pre-trained model: SciBERT [39] for text, Vision Transformer (ViT) [40] for images, and ChemBERTa [41] for SMILES strings.”
Virk, V., Giri, S., & Giri, K. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-33633-1.00016-2 · Cited by 2
Li, Y., Liu, J., Song, N., & Pei, L. (2025) · Scientometrics · doi:10.1007/s11192-025-05429-5 · Cited by 1
“Specifically, we utilized the pre-trained SciBERT model (Beltagy et al., 2019) to encode each abstract into a contextual embedding.”
Survice-BERT: A BERT Model for Named Entity Recognition in Infectious Disease Surveillance Reports
Cheon, S., & Ahn, I. (2025) · Journal of Bacteriology and Virology · doi:10.4167/jbv.2025.55.3.235 · Cited by 1
Cross-Lingual Keyword Extraction for Pesticide Terminology in Brazilian Portuguese and English
de Souza, J. V., Amamou, H., Chen, R., Salari, E., Gubelmann, R., Niklaus, C., Serpa, T., Lima, M. F., Pinto, P. T., Kshirsagar, S., et al. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5815 · Cited by 1
“In particular, domain-specific BERT variants, such as BioBERT [Lee et al ., 2020] for biomedical text, SciBERT [Beltagy et al ., 2019] for scientific literature, and LEGAL-BERT [Chalkidis et al ., 2020] for the legal domain, have demonstrated superior performance in capturing domain-specific…”
Mikhaylov, D. V., & Emelyanov, G. M. (2025) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661825601789 · Cited by 1
“The latest version of this model showed an average quality in all tasks of the SciDocs benchmark [23] about 1.3 times higher than the well-known allenai/scibert-scivocab-uncased model [2].”
InfluentialLattesRex: Building ChatBots for Semi-Structured Documents
Darcio, L., Santos, K. S., Spellen, A., Soares, E., Real, L., & da Silva, A. S. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37819 · Cited by 1
“No dom ´ ınio cient ´ ıfico, modelos especializados como o SciBERT [Beltagy et al. 2019] e o SPECTER [Cohan et al. 2020] melhoraram significati-vamente tarefas como classificac¸ ˜ ao e similaridade sem ˆ antica de documentos acad ˆ emicos.”
InfluentialNPL v. NLP: analysing the links between science and technology with citations and semantics
Liu, J., Amdaoud, M., Mescheba, W., Quemener, J., Sapinho, D., Deltorn, J., & Guellec, D. (2025) · Scientometrics · doi:10.1007/s11192-025-05460-6 · Cited by 2
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.