SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,301–1,400 of 3,596
Jiang, Q., Sun, Y., Chuo, M., Dong, F., Ji, N., Ji, X., Li, J., Wang, Z., & Liu, Y. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.128597 · Cited by 12
Fully Attentional Network for Low-Resource Academic Machine Translation and Post Editing
İlhami, S. E. L., & Hanbay, D. (2022) · Applied Sciences · doi:10.3390/app122211456 · Cited by 11
Applications of data analysis on scholarly long documents
Banerjee, B., Ingram, W. A., Wu, J. F., & Fox, E. A. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020935 · Cited by 10
“Instead, we use BERT [17] and SciBERT [18] as base models for fine-tuning.”
MethodZhao, J., Song, R., Yue, C., Wang, Z., & Xu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103442 · Cited by 10
CLeBPI: Contrastive Learning for Bug Priority Inference
Wang, W., Wu, C., & He, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107302 · Cited by 12
Retrieval-Augmented Domain Adaptation of Language Models
Xu, B., Zhao, C., Jiang, W., Zhu, P., Dai, S., Pang, C., Sun, Z., Wang, S., & Sun, Y. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.5 · Cited by 22
Maroudas, S., Legkas, S., Malakasiotis, P., & Chalkidis, I. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.8 · Cited by 11
Zero-Shot Entailment of Leaderboards for Empirical AI Research
Kabongo, S., D’Souza, J., & Auer, S. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00042 · Cited by 15
“Posited as document-level IE, it uses a 2-level BERT+BiLSTM method to get token representations which are passed to a CRF layer to identify mentions.”
MethodInfluentialShellGPT: Generative Pre-trained Transformer Model for Shell Language Understanding
Shi, J., Jiang, S., Xu, B., Liang, J., Xiao, Y., & Wang, W. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00082 · Cited by 21
“…language models have been successful in general domain natural language understanding tasks, it has also been shown that their performance deteriorates on domain-specific tasks, which is because of the distribution difference between general domain texts and domain-specific texts [15], [16].”
BackgroundAssessing the evolution of research topics in a biological field using plant science as an example
Shiu, S., & Lehti‐Shiu, M. D. (2024) · PLoS Biology · doi:10.1371/journal.pbio.3002612 · Cited by 12
On the Trade-off between Redundancy and Cohesiveness in Extractive Summarization
Cardenas, R., Gallé, M., & Cohen, S. B. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15191 · Cited by 12
The gene function prediction challenge: Large language models and knowledge graphs to the rescue
Sunil, R. S., Lim, S. C., Itharajula, M., & Mutwil, M. (2024) · Current Opinion in Plant Biology · doi:10.1016/j.pbi.2024.102665 · Cited by 11
“LLMs can be fine-tuned for task-specific datasets, such as BioGPT, BioBERT and SciBERT in scientific domains [46–48], or incorporated into tools and pipelines [49,50].”
Chang, Y., Huang, M.-S., Huang, Y.-H., & Lin, Y.-H. (2025) · Scientific Reports · doi:10.1038/s41598-025-99290-4 · Cited by 10
Park, S., & Kim, Y. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.10.009 · Cited by 10
AUTH @ CLSciSumm 20, LaySumm 20, LongSumm 20
Gidiotis, A., Stefanidis, S. P., & Tsoumakas, G. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.28 · Cited by 12
A comparative analysis of system features used in the TREC-COVID information retrieval challenge
Chen, J., & Hersh, W. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103745 · Cited by 14
Noh, J., & Kavuluru, R. (2021) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3459930.3469533 · Cited by 14
“Here we use SciBERT which is trained on scientific literature both from biomedicine and computer science.”
MethodInfluentialZhang, J., Trujillo, L. B., Li, T., Tanwar, A., Freire, G., Yang, X., Ive, J., Gupta, V., & Guo, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.690 · Cited by 13
“0 (Lee et al., 2019) pre-trained on PubMed and PMC, (3) Clinical BERT (Alsentzer et al., 2019) pre-trained based on BioBERT and MIMIC-III discharge summaries, (4) SciBERT (Beltagy et al., 2019) pre-trained for scientific literature.”
MethodNamed Entity Recognition in Long Documents: An End-to-end Case Study in the Legal Domain
Keshavarz, H., Vagena, Z., Kouki, P., Fountalis, I., Mabrouki, M., Belaweid, A., & Vasiloglou, N. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020873 · Cited by 10
A Universal Activation Function for Deep Learning
Hwang, S.-Y., & Kim, J.-J. (2023) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2023.037028 · Cited by 10
LADER: Log-Augmented DEnse Retrieval for Biomedical Literature Search
Jin, Q., Shin, A., & Lu, Z. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592005 · Cited by 11
“Table 2: LADER results on the HEAD queries compared to BERT DOT [14]; LA: log augmentation; DR: dense retrieval.”
MethodInfluentialEmbedding Recycling for Language Models
Saad-Falcon, J., Singh, A., Soldaini, L., D’Arcy, M., Cohan, A., & Downey, D. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.145 · Cited by 13
Self-citationA self-supervised language model selection strategy for biomedical question answering
Arabzadeh, N., & Bagheri, E. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104486 · Cited by 10
UZH@CRAFT-ST: a Sequence-labeling Approach to Concept Recognition
Furrer, L., Cornelius, J., & Rinaldi, F. (2019) · Workshop on BioNLP Open Shared Tasks (BioNLP-OST) · doi:10.18653/v1/d19-5726 · Cited by 11
Recent Progress of Named Entity Recognition over the Most Popular Datasets
Albared, M., Ocaña, M. G., Ghareb, A. S., & Al-Moslmi, T. (2019) · International Conference of Intelligent Computing and Engineering (ICOICE) · doi:10.1109/icoice48418.2019.9035170 · Cited by 10
AT-BERT: Adversarial Training BERT for Acronym Identification Winning Solution for SDU@AAAI-21Preprint
Zhu, D., Lin, W., Zhang, Y., Zhong, Q., Zeng, G., Wu, W., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03700 · Cited by 10
“Specifically, considering the training data is relatively small, we adopt the pre-trained BERT model as sentence encoder, which is pre-trained on general domain corpora and shows a significant improvement on the performance of downstream tasks with supervised fine-tuning (Beltagy, Lo, and Cohan 2019).”
MethodLinguistically Informed Masking for Representation Learning in the Patent DomainPreprint
Althammer, S., Buckley, M., Hofstätter, S., & Hanbury, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05768 · Cited by 10
“We compare the language models BERT [6] (pre-trained on general purpose language) and SciBERT [4] (pre-trained on scientific language for representing patent language).”
InfluentialPlant phenotype relationship corpus for biomedical relationships between plants and phenotypes
Cho, H., Kim, B., Choi, W., Lee, D., & Lee, H. (2022) · Scientific Data · doi:10.1038/s41597-022-01350-1 · Cited by 10
Uni-Fold MuSSe: De Novo Protein Complex Prediction with Protein Language ModelsPreprint
Zhu, J., He, Z., Li, Z., Ke, G., & Zhang, L. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.02.14.528571 · Cited by 10
“Previous studies in Natural Language Processing (Lee et al., 2020; Beltagy et al., 2019; Sun et al., 2020) proved the effectiveness of further pre-training on domain transfer tasks, whereas the target domain generally has massive unlabeled data.”
BackgroundDrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsPreprint
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.03.535368 · Cited by 30
“An alternative method, when enough in-domain data is available, is to directly pre-train models from scratch (SciBERT (Beltagy et al., 2019), PubMedBERT (Gu et al.”
MethodBuilding a Knowledge Graph to Enrich Chatgpt Responses in Manufacturing Service DiscoveryPreprint
Li, Y., & Starly, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4517533 · Cited by 10
Construction of a Japanese Financial Benchmark for Large Language ModelsPreprint
Hirano, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4769124 · Cited by 10
Amiri, M., & Bocklitz, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.17277 · Cited by 11
“The evaluated domain-pretrained models include SciBERT [35], PubMedBERT [40], Biomed-BERT [60], ChemicalBERT [42], ChemBERTa [43], and MatSciBERT [39].”
Elazar, Y., Bhagia, A., Magnusson, I., Ravichander, A., Schwenk, D., Suhr, A., Walsh, P., Groeneveld, D., Soldaini, L., Singh, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.20707 · Cited by 9
Improving biomedical Named Entity Recognition with additional external contexts
Tho, B. D., Nguyen, M.-T., Le, D. T., Ying, L.-L., Inoue, S., & Nguyen, T.-T. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104674 · Cited by 12
Leveraging peer-review aspects for extractive and abstractive summarization of scientific articles
Majadly, M., & Last, M. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00665-z · Cited by 9
Ma, Y., Qing, L., Kang, Y., Liu, J., Zhang, Y., Cheng, Q., Lu, W., & Liu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127226 · Cited by 13
He, Y., Deng, K., & Han, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0322182 · Cited by 9
“Beltagy et al. [22] integrated the BERT architecture with graph neural networks, proposing an advanced automated patent classification model to support scholars’ and busi-nesses’ technological research.”
Krzyzanowski, A., Pickett, S. D., & Pogány, P. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00359 · Cited by 12
Asadollahi, H., Meouche, R. E., Zheng, Z., Eslahi, M., & Farazdaghi, E. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110680 · Cited by 11
Abbasiantaeb, Z., Verberne, S., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104264 · Cited by 9
Liu, H., Chen, Z., Li, P., Liu, Y.-Z., Liu, X., Xu, R. X., & Sun, M. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104896 · Cited by 12
KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models
Nguyen, L., Barcelos, E. I., French, R. H., & Wu, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09527-5_34 · Cited by 9
Large Language Models in Bio-Ontology Research: A Review
Manda, P. (2025) · Bioengineering · doi:10.3390/bioengineering12111260 · Cited by 15
Choudhary, A., Pamidimokkala, S., Krithiga, R., & M., B. R. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1706369 · Cited by 13
Sohrab, M. G., Nguyen, A.-K. D., Miwa, M., & Takamura, H. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.38 · Cited by 10
Visual Summary Identification From Scientific Publications via Self-Supervised Learning
Yamamoto, S., Lauscher, A., Ponzetto, S. P., Glavaš, G., & Morishima, S. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.719004 · Cited by 14
“In this paper, we propose an approach which falls into the second category and is based on pretrained Transformers (Beltagy et al., 2019; Devlin et al., 2019).”
MethodSelf-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer
Xu, W., Saxon, M., Sra, M., & Wang, W. Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21410 · Cited by 13
FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations
Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.660 · Cited by 9
Knowledge-Guided Paraphrase Identification
Wang, H., Ma, F., Wang, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.72 · Cited by 11
Chemical identification and indexing in PubMed full-text articles using deep learning and heuristics
de Almeida, T. D., Antunes, R., Silva, J., Almeida, J. R., & Matos, S. (2022) · Database · doi:10.1093/database/baac047 · Cited by 11
Automating Transfer Credit Assessment-A Natural Language Processing-Based Approach
Chandrasekaran, D., & Mago, V. (2022) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2022.027236 · Cited by 9
Mahbub, M., Begoli, E., Martins, S. B., Peluso, A., Tamang, S. R., & Peterson, G. D. (2023) · IEEE Access · doi:10.1109/access.2023.3235265 · Cited by 12
A Named Entity Recognition Model for Manufacturing Process Based on the BERT Language Model Scheme
Shrivastava, M., Seri, K., & Wagatsuma, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-24667-8_50 · Cited by 9
Towards InnoGraph: A Knowledge Graph for AI Innovation
Massri, M. B., Spahiu, B., Grobelnik, M., Alexiev, V., Palmonari, M., & Roman, D. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587614 · Cited by 14
“Some academic KGs ofer models that can be especially useful for InnoGraph e.g. SciBERT [3] and SciCo [5] of Semantic Scholar.”
MethodTADA: Efficient Task-Agnostic Domain Adaptation for Transformers
Hung, C.-C., Lange, L., & Strötgen, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.31 · Cited by 9
Barman, H., Sikirica, V., Carlson, K., Silvert, E., Carlson, K., Boyer, S. W., Glaser, R., Morava, É., Wagner, T., & Lanpher, B. C. (2023) · Molecular Genetics and Metabolism · doi:10.1016/j.ymgme.2023.107695 · Cited by 9
A study of BERT-based methods for formal citation identification of scientific data
Yang, N., Zhang, Z., & Huang, F. (2023) · Scientometrics · doi:10.1007/s11192-023-04833-z · Cited by 9
“…of models such as the bidirectional encoder representations from transformer (BERT) model (Devlin et al., 2019) and its derivatives, such as ALBERT (Lan et al., 2019) BioBERT (Lee et al., 2020) and SciBERT (Beltagy et al., 2019), have made machine replacement of human for text analysis possible.”
MethodHaviana, S. F. C., Mulyono, S., & Badieah, B. (2023) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.1109/eecsi59885.2023.10295797 · Cited by 9
“M ETHODOLOGY The experiment was conducted by using eight variant of language model which is : BERT base-uncased [20], Albert base-v2 [21], DeBERTa [22], DistillBERT [23], Electra [24], RoBERTa [25], SciBERT [26], dan XLNet [27].”
MethodFocused Contrastive Loss for Classification With Pre-Trained Language Models
He, J., Li, Y., Zhai, Z., Fang, B., Thorne, C., Druckenbrodt, C., Akhondi, S. A., & Verspoor, K. M. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3327777 · Cited by 11
“Furthermore, considering the common batch sizes used in NLP tasks, e.g., 8, 16, and 32, the benefits of using FoCL will be more significant when a task has smaller number of classes.”
MethodWord embeddings for retrieving tabular data from research publications
Berenguer, A., Mazón, J.-N., & Tomás, D. (2023) · Machine Learning · doi:10.1007/s10994-023-06472-0 · Cited by 10
“SciBERT (Beltagy et al., 2019): another variation of BERT specifically designed for scientific texts.”
Biomedical Named Entity Recognition via Dictionary-based Synonym Generalization
Fu, Z., Su, Y., Meng, Z., & Collier, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.903 · Cited by 16
Citance-Contextualized Summarization of Scientific Papers
Syed, S., Hakimi, A., Al‐Khatib, K., & Potthast, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.573 · Cited by 15
SOAP classifier for free-text clinical notes with domain-specific pre-trained language models
de Oliveira, J. M., Antunes, R. S., & da Costa, C. A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123046 · Cited by 11
Gupta, K., Ahmad, A., Ghosal, T., & Ekbal, A. (2024) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00393-y · Cited by 9
“However, we also assess the impact of imbalanced document distribution in two sub-tests by selecting 10 evenly distributed articles and another 10 unevenly distributed articles.”
Wang, Y., Yi, X., & Zhang, C. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101550 · Cited by 10
“After being trained on the more extensive training data, SciBERT is still the best-performing model, which again validates its superiority to academic text data processing.”
InfluentialYu, D., & Xiang, B. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101551 · Cited by 9
Duan, Z., Gu, H., Ke, Y., & Zhou, D. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112118 · Cited by 9
Academic expert finding using BERT pre-trained language model
Mannix, I. A., & Yulianti, E. (2024) · International Journal of Advances in Intelligent Informatics · doi:10.26555/ijain.v10i2.1497 · Cited by 9
“…BERT (mBERT) [15], which is trained with more than 100 languages, including Indonesian and Javanese; (2) Indonesian BERT (IndoBERT) [16], trained on approximately 250 million Indonesian language sentences; (3) Scientific BERT (SciBERT) [17], trained using 1.14 million English scientific articles.”
BackgroundInfluentialAdaptive feature extraction for entity relation extraction
Yang, W., Qin, Y., Huang, R., & Chen, Y. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101712 · Cited by 9
Blockchain Projects in Environmental Sector: Theoretical and Practical Analysis
Vaccargiu, M., & Tonelli, R. (2024) · Earth · doi:10.3390/earth5030020 · Cited by 9
“The first one, called SciBERT , is trained on a dataset of scientific articles [23], while the second one, called ClimateBERT , is trained on a dataset related to climate, sustainability, and the environment [24].”
Enriched BERT Embeddings for Scholarly Publication Classification
Wolff, B., Seidlmayer, E., & Förstner, K. U. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_16 · Cited by 11
“Selecting Pre-trained Models For selecting promising pre-trained models for our downstream task, we rely on the insights from the SciRepEval benchmark [12] and chose the top-performing models identified for scientific tasks as potential candidates: SciBERT [1], SciNCL [9] and SPECTER2 [12].”
MethodHe, G., Lin, C., Ren, J., & Duan, P. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101596 · Cited by 9
TDGI: Translation-Guided Double-Graph Inference for Document-Level Relation Extraction
Zhang, L., Zhong, Y., Zheng, Q., Liu, J., Wang, Q., Wang, J., & Chang, X. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3528246 · Cited by 9
“In addition, we pick up the pretrained cased BERT-base [20] or BERT-large [20] models to encode documents from DocRED dataset, meanwhile use the cased SciBERT [21] model on CDR and GDA.”
Augmenting LLMs to Securely Retrieve Information for Construction and Facility Management
Krütli, D., & Hanne, T. (2025) · Information · doi:10.3390/info16020076 · Cited by 12
“Similar re fi nement approaches were employed SciBERT [20] for the scienti fi c literature, ClimateBERT [21] for climate-related text, LEGAL-BERT [22] for the Similar refinement approaches were employed with SciBERT [20] for the scientific literature, ClimateBERT [21] for climate-related text, and…”
Li, B., Qiu, X., Tan, X., Yang, L., Tao, J., Fang, Z., & Huang, J. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06299-3 · Cited by 10
“Now, fine-tuning BERT for various downstream tasks, such as Bio-BERT [46] for biomedical information mining, Clinical-BERT [47, 48]for clinical tasks, and SCI-BERT [49] for scientific publications, has demonstrated significant potential.”
Wang, J., Jatowt, A., & Cai, Y. (2025) · ACM Transactions on the Web · doi:10.1145/3723352 · Cited by 12
“A notable example is the work of Dhingra et al. [21], which modifies the pre-training process by incorporating timestamp information into the masked language modeling (MLM) objective.”
How did we get there? AI applications to biological networks and sequences
Anteghini, M., Gualdi, F., & Oliva, B. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110064 · Cited by 9
Benchmarking transformer embedding models for biomedical terminology standardization
Lahiri, A., Shukla, S., Stear, B. J., Ahooyi, T. M., Beigel, K., Margolskee, E. M., & Taylor, D. M. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100683 · Cited by 12
CBAG: Conditional biomedical abstract generation
Sybrandt, J., & Safro, I. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253905 · Cited by 12
Effective distributed representations for academic expert search
Berger, M., Zavrel, J., & Groth, P. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.7 · Cited by 16
Vaidhya, T., & Kaushal, A. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.34 · Cited by 9
Case Study on well-known Topic Modeling Methods for Document Classification
Ozdemirci, S., & Turan, M. (2021) · International Conference on Inventive Computation Technologies (ICICT) · doi:10.1109/icict50816.2021.9358473 · Cited by 9
“In the future, models such as RoBERTa [24], SciBERT [25], ALBERT [26] etc. can be used instead of BERT.”
MethodMulti-source Neural Topic Modeling in Multi-view Embedding Spaces
Gupta, P., Chaudhary, Y., & Schütze, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.332 · Cited by 9
On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles
Gosangi, R., Arora, R., Gheisarieha, M., Mahata, D., & Zhang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.359 · Cited by 11
A Multilingual Approach to Identify and Classify Exceptional Measures against COVID-19
Tziafas, G., de Saint-Phalle, E., de Vries, W., Egger, C. M., & Caselli, T. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.5 · Cited by 9
A Semantic Annotation Pipeline towards the Generation of Knowledge Graphs in Tribology
Kügler, P., Marian, M., Dorsch, R., Schleich, B., & Wartzack, S. J. (2022) · Lubricants · doi:10.3390/lubricants10020018 · Cited by 9
“The SciBERT model [61] is pre-trained on scientific papers improving the performance of downstream tasks with scientific vocabulary.”
MethodKerasBERT: Modeling the Keras Language
Shorten, C., & Khoshgoftaar, T. M. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00041 · Cited by 9
Improving Biomedical Named Entity Recognition with a Unified Multi-Task MRC Framework
Tong, Y., Zhuang, F., Wang, D., Ying, H., & Wang, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746482 · Cited by 14
Lee, S., & Na, S. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.231 · Cited by 11
Multi-task Hierarchical Cross-Attention Network for Multi-label Text Classification
Lu, J., Zhang, H., Shen, Z., Shi, K., Yang, L., Xu, B., Zhang, S., & Lin, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17189-5_13 · Cited by 9
Basuki, S., & Tsuchiya, M. (2022) · IEEE Access · doi:10.1109/access.2022.3225871 · Cited by 15
Usha, M. S., Smrity, A. M., & Das, S. (2022) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit57492.2022.10055784 · Cited by 9
“The followings are a brief overview of the projected model’s tangible information: • To overcome the shortage of high quality, extensive labeled scientific data [9], we have employed SciBERT, a pre-trained language model based on BERT [10].”
MethodKim, J., Kim, J., Lee, A., & Kim, J. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283342 · Cited by 9
“SciBERT follows the same architecture as BERT but is pretrained on scientific text instead [18].”
MethodA novel self-attention enriching mechanism for biomedical question answering
Kaddari, Z., & Bouchentouf, T. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120210 · Cited by 12
Oh, J. H., Tannenbaum, A., & Deasy, J. O. (2023) · Frontiers in Genetics · doi:10.3389/fgene.2023.1161047 · Cited by 9
“, 2022), SciBERT (Beltagy et al., 2019), ClinicalBERT (Huang et al.”
MethodIaaSI: a device based interoperability as a service for IoMT devices
Kotha, A., Manohar, K., & Venkanna, U. (2023) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-023-04669-8 · Cited by 9
Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords
Golchin, S., Surdeanu, M., Tavabi, N., & Kiapour, A. M. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.2 · Cited by 13
Modeling Information Change in Science Communication with Semantically Matched Paraphrases
Wright, D., Pei, J., Jurgens, D., & Augenstein, I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.117 · Cited by 24
Evaluating the Knowledge Dependency of Questions
Moon, H., Yang, Y., Yu, H., Lee, S., Jeong, M., Park, J., Shin, J., Kim, M., & Choi, S. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.718 · Cited by 18
“A total of 8 types of facts and 32 questions were presented for each dataset, and a questionnaire was conducted on a total of 96 questions.”
Background
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.