SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,801–1,900 of 3,596
Shi, M., Janowicz, K., Liu, Z., Karimi, M., Majić, I., & Fortacz, A. (2025) · Transactions in GIS · doi:10.1111/tgis.70058 · Cited by 5
Jeon, W., Park, M., An, D., Nam, W., Shin, J., Lee, S., & Lee, S. (2025) · JMIR Medical Informatics · doi:10.2196/67513 · Cited by 6
Zhang, K., Dong, C., Guo, Y., Zhou, W., Yu, G., & Mi, J. (2025) · Systems · doi:10.3390/systems13060417 · Cited by 5
“In technical discourse analysis, specialized BERT variants such as SciBERT [37] and RoBERTa‑wwm [38] have demonstrated improved performance by incorporating domain‑specific training data and structural knowledge.”
Molefe, T., Jiang, Y., Magubane, A., Mguni, L. L., Sikeyi, L. L., Jiang, C., Liu, X., & Yao, Y. (2025) · Fuel Processing Technology · doi:10.1016/j.fuproc.2025.108285 · Cited by 5
Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
Yun, H. S., Zhang, K. Y. C., Kouzy, R., Marshall, I. J., Li, J. J., & Wallace, B. C. (2025) · PubMed · doi:10.48550/arxiv.2502.07963 · Cited by 6
“De-Spin adopted a combination of rule-based methods and machine learning models—BERT (Devlin, 2018), BioBERT (Lee et al., 2020), and SciBERT (Beltagy et al., 2019)—to extract trial design information and identify spin.”
Are patents with female inventors under-cited? Evidence from text estimation
Hochberg, Y. V., Kakhbod, A., Li, P., & Sachdeva, K. (2022) · Journal of Financial Economics · doi:10.1016/j.jfineco.2026.104307 · Cited by 8
Li, B., Xu, W., Xu, Z., Li, J., & Peng, P. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02269-7 · Cited by 5
“SciBERT [3] pre-trained BERT on a large multi-domain corpus of scientific publications that achieves good results in scientific-related text tasks in ACL-ARC [4] and SciCite [9].”
MethodDevice Fabrication Knowledge Extraction from Materials Science Literature
Wadhwa, N., Sarath, S., Shah, S., Reddy, S., Mitra, P., Jain, D., & Rai, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17811 · Cited by 5
“We support three types of embeddings: embeddings learned using word2vec objective: Mat-word2vec (Kim et al. 2017b) and mat2vec (Tshitoyan et al. 2019); embeddings with sub-word information: Matfasttext (Kim et al. 2020); and contextualized word embeddings: Mat-ELMo (Kim et al. 2020) and SciBERT (Beltagy, Lo, and Cohan…”
MethodKA-NER: Knowledge Augmented Named Entity Recognition
Nie, B., Li, C., & Wang, H. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-6471-7_5 · Cited by 6
Extracting Material Property Measurement Data from Scientific Articles
Panapitiya, G., Parks, F., Sepulveda, J., & Saldanha, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.438 · Cited by 11
ACTA 2.0: A Modular Architecture for Multi-Layer Argumentative Analysis of Clinical Trials
Molinet, B., Marro, S., Cabrio, E., Villata, S., & Mayer, T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/859 · Cited by 7
“The SciBERT [Beltagy et al., 2019] uncased base model with pretrained weights is used for the sentence representations, finetuned with a learning rate of 2e-5, batch size of 8, maximum”
MethodInterpreting Patient Descriptions using Distantly Supervised Similar Case Retrieval
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532003 · Cited by 5
“This can be explained by the observation that HeadQA primarily consists of factual questions, which clearly 7 shows the results obtained for these two sets of questions, for the SciBERT-FT and PubMedBERT-FT baselines, as well as our proposed model, where we used the RQE pre-training task and 𝑘 = 5.”
MethodInfluentialAuxiliary Learning for Named Entity Recognition with Multiple Auxiliary Biomedical Training Data
Watanabe, T., Ichikawa, T., Tamura, A., Iwakura, T., Ma, C., & Kato, T. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.13 · Cited by 5
Tang, A., Deléger, L., Bossy, R., Zweigenbaum, P., & Nédellec, C. (2022) · Database · doi:10.1093/database/baac070 · Cited by 5
“Conventionally, an RE task is treated as a special kind of text classification problem as in (4, 7).”
BackgroundMethodA Cross‐Domain Ontology Semantic Representation Based on NCBI‐BlueBERT Embedding
Zhao, L., Junjie, W. A. N. G., Wang, C., & Guo, M. (2022) · Chinese Journal of Electronics · doi:10.1049/cje.2020.00.326 · Cited by 5
Text Summarization towards Scientific Information Extraction
Keller, A., Furst, J. D., Raicu, D. S., Hastings, P., & Tchoua, R. (2022) · IEEE International Conference on eScience · doi:10.1109/escience55777.2022.00036 · Cited by 5
Karapetian, K., Jeon, S. M., Kwon, J., & Suh, Y. (2022) · Journal of Medical Internet Research · doi:10.2196/41100 · Cited by 5
“SciBERT [60] ClinicalBERT [59] PubMedBERT [58] BioBERT [57] Vanilla BERT [ Table 5 presents the performance results of the R-BERT models in terms of the different properties of the 8 data sets.”
Parwez, M. A., Fazil, M., Arif, M., Nafis, M. T., & Auwul, M. R. (2023) · Computational Intelligence and Neuroscience · doi:10.1155/2023/2989791 · Cited by 6
“…the feld of biomedical domain to develop models such as BioWordVec [33] and BioBERT [34], as well as other do-main-specifc models such as SciBERT [35] trained on various scientifc and biomedical corpuses, ClinicalBERT [36] trained on clinical notes for various NLP tasks, and MatSciBERT [37]…”
MethodNASA Science Mission Directorate Knowledge Graph Discovery
Timmer, R. C., Mark, M. D., Khoo, F. S., Martins, M. S. R., Berea, A., Renard, G., & Bugbee, K. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587585 · Cited by 5
“In the future, we aim to create NER algorithms similar to AstroBERT [13], and SciBERT [4].”
MethodInfluentialTian, Y., Li, W., Wang, S., & Gu, Z. (2023) · Transactions in GIS · doi:10.1111/tgis.13059 · Cited by 7
“Six models, LSA, GloVe, RoBERTa, SPECTER, SciBERT, and Doc2vec are used to conduct semantic analysis and identify similar topics.”
MethodInfluentialRethinking Dictionaries and Glyphs for Chinese Language Pre-training
Wang, Y., Wang, J., Zhao, D., & Zheng, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.70 · Cited by 10
Chen, C., Shen, Z., Klein, D., Stanovsky, G., Downey, D., & Lo, K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.844 · Cited by 6
Self-citationMIReAD: Simple Method for Learning High-quality Representations from Scientific Documents
Razdaibiedina, A., & Brechalov, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.46 · Cited by 7
An Empirical Study on Active Learning for Multi-label Text Classification
Wang, M., & Liu, M. (2023) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2023.insights-1.12 · Cited by 7
Pérez, N., Cuadros, M., & Rigau, G. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102682 · Cited by 5
Soft Prompt Transfer for Zero-Shot and Few-Shot Learning in EHR Understanding
Wang, Y., Peng, X., Shen, T., Clarke, A., Schlegel, C., Martin, P., & Long, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-46671-7_2 · Cited by 7
Software Entity Recognition with Noise-Robust Learning
Nguyen, T. D., Di, Y., Lee, J., Chen, M., & Zhang, T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00203 · Cited by 8
“Thus, developers have spent great effort to develop domain-specific NER models, such as BioBERT [57] for biomedical literature, ClinicalBERT [58] for clinical documents, or SciBERT [59] for scientific literature.”
PEDL+: protein-centered relation extraction from PubMed at your fingertip
Weber, L., Barth, F., Lorenz, L. J., Konrath, F., Huska, K., Wolf, J., & Leser, U. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad603 · Cited by 5
“It is a version of the PLM SciBERT (Beltagy et al. 2019) fine-tuned on a mixture of gold-standard data and distantly supervised data derived from multiple pathway databases.”
Ghosh, M., Mukherjee, S., Santra, P., Na, G., & Basuchowdhuri, P. (2024) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3632410.3632442 · Cited by 7
“Recently, the biomedical NLP literature has seen a notable surge in research focusing on PICO frame extraction, particularly following the advent of transformer-based frameworks such as BERT [9], SciB-ERT [6], and XLMNet [29] in the NLP literature.”
BackgroundMethodEvans, M.-T. C., Latifi, M., Ahsan, M., & Haider, J. (2024) · Information · doi:10.3390/info15020091 · Cited by 6
How to write a CHI paper (asking for a friend)
Robinson, R., Alvarez, A., & Mekler, E. D. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3644051 · Cited by 6
Deep Pre-Training Transformers for Scientific Paper Representation
Wang, J., Yang, Z., & Cheng, Z. (2024) · Electronics · doi:10.3390/electronics13112123 · Cited by 5
SSuieBERT: Domain Adaptation Model for Chinese Space Science Text Mining and Information Extraction
Liu, Y., Li, S., Deng, Y., Hao, S., & Wang, L. (2024) · Electronics · doi:10.3390/electronics13152949 · Cited by 5
A natural language processing system for the efficient extraction of cell markers
Cheng, P., Peng, Y., Zhang, X., Chen, S., Fang, B.-B., Li, Y.-Z., & Sun, Y. (2024) · Scientific Reports · doi:10.1038/s41598-024-72204-6 · Cited by 6
“Specifically, the sentences were initially input into the SciBERT model[23], and the tokenizer and parser components of the SciBERT model were used for part-of-speech tagging and syntactic dependency parsing of the sentences.”
Overview of the NLPCC2024 Shared Task 6: Scientific Literature Survey Generation
Tian, Y., Gu, X., Li, A., Zhang, H., Xu, R., Li, Y., & Liu, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9443-0_35 · Cited by 11
Enhancing keyphrase extraction from long scientific documents using graph embeddings
Martínez-Cruz, R., Mahata, D., López-López, Á. J., & Portela, J. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06579-y · Cited by 5
“With the increasing reliance on digitization in diverse industry sectors such as legal, medical, scientific, and financial, where documents can span multiple pages, the need to identify keyphrases from these lengthy documents has become paramount.”
Exploring lightweight language models for materials informatics with AlchemBERT
Wang, Y., Liu, X., Chen, H., Yang, T., Liu, X., & Wen, X. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102724 · Cited by 7
Investigating the Effect of Lexical Segmentation in Transformer-based Models on Medical Datasets
Nguyen, V. A., Karimi, S., & Xing, Z. (2019) · Cited by 8
PERLEX: A Bilingual Persian-English Gold Dataset for Relation Extraction
Asgari-Bidhendi, M., Nasser, M., Janfada, B., & Bidgoli, B. M. (2021) · Scientific Programming · doi:10.1155/2021/8893270 · Cited by 8
Ramponi, A., Giampiccolo, S., Tomasoni, D., Priami, C., & Lombardo, R. (2020) · IEEE Access · doi:10.1109/access.2020.3014862 · Cited by 5
Representing Semantified Biological Assays in the Open Research Knowledge Graph
Anteghini, M., D’Souza, J., dos Santos, V. A. P. M., & Auer, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64452-9_8 · Cited by 9
Medical Knowledge-enriched Textual Entailment Framework
Yadav, S., Pallagani, V., & Sheth, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.161 · Cited by 5
Wecker, H., Friedrich, A., & Adel, H. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.15 · Cited by 7
Synthetic Target Domain Supervision for Open Retrieval QA
Reddy, R. G., Iyer, B., Sultan, M. A., Zhang, R., Sil, A., Castelli, V., Florian, R., & Roukos, S. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463085 · Cited by 9
“, with scientific [7] and biomedical text [3, 25].”
BackgroundContext, Language Modeling, and Multimodal Data in Finance
Das, S. R., Goggins, C., He, J., Karypis, G., Krishnamurthy, S., Mahajan, M., Prabhala, N. R., Slack, D., van Dusen, R., Yue, S., et al. (2021) · The Journal of Financial Data Science · doi:10.3905/jfds.2021.1.063 · Cited by 5
Guo, Y., Ge, Y., Al-Garadi, M. A., & Sarker, A. (2021) · Social Media Mining for Health Research and Applications Workshop and Shared Tasks (SMM4H) · doi:10.18653/v1/2021.smm4h-1.8 · Cited by 5
“SciBERT is trained on more general domain data such as computer science text.”
MethodInfluentialA Simple yet Effective Method for Sentence Ordering
Shen, A., & Baldwin, T. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.16 · Cited by 5
The Effect of Transfer Learning on Turkish Text Classification
Şahin, G., & Diri, B. (2021) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu53274.2021.9477910 · Cited by 5
Yazi, F. S., Vong, W.-T., Raman, V., Then, P., & Lunia, M. J. (2021) · International Conference on Information Retrieval & Knowledge Management (CAMP) · doi:10.1109/camp51653.2021.9498061 · Cited by 5
Zhu, X., Gu, Y., & Xiao, Z. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.799349 · Cited by 8
“• SciBERT (Beltagy et al., 2019) is another BERT varient pretrained on a corpus consisting of 1.”
BackgroundFine-Grained Chemical Entity Typing with Multimodal Knowledge Representation
Sun, C., Li, W., Xiao, J., Parulian, N. N., Zhai, C., & Ji, H. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669360 · Cited by 5
“Finally the increase of the performance from SciBERT-CD and SciBERT-CG to our full model shows the benefit of learning a unified multimodal representation for chemical entities in chemistry literature.”
BackgroundMethodInfluentialTransformer-Encoder-Based Mathematical Information Retrieval
Reusch, A., Thiele, M., & Lehner, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13643-6_14 · Cited by 5
Zainab, K., Srivastava, G., & Mago, V. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04933-2 · Cited by 6
“To get the context of a word these three methods are used together [36].”
BackgroundMethodDid AI get more negative recently?
Beese, D., Altunbaş, B., Güzeler, G., & Eger, S. (2023) · Royal Society Open Science · doi:10.1098/rsos.221159 · Cited by 10
“We feed each paper as concatenation of title and abstract separated by special tokens to SciBERT: [CLS] <title> [SEP] <abstract> .”
MethodInfluentialDong, B., Wang, Z., Li, Z., Duan, Z., Xu, J., Pan, T., Zhang, R., Liu, N., Li, X., Wang, J., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-39543-2 · Cited by 6
“PLMs compensate for the negative effects of insufficient training data by transferring pre-training results to downstream tasks and have achieved impressive success in natural language processing (NLP) tasks 24–29 .”
BackgroundPavlova, V., & Makhlouf, M. (2023) · Workshop on Biomedical Natural Language Processing · doi:10.18653/v1/2023.bionlp-1.31 · Cited by 3
“To date, various meth-ods have been developed to pre-train biomedical language models: pre-training from scratch (Gu et al., 2021; Beltagy et al., 2019), continual pre-training (Lee et al., 2019; Huang et al., 2019), and hybrid approaches (Tai et al., 2020; Poerner et al., 2020; Sachidananda et…”
BackgroundMethodCan we Pretrain a SotA Legal Language Model on a Budget From Scratch?
Niklaus, J., & Giofré, D. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.11 · Cited by 8
Exploring Partial Knowledge Base Inference in Biomedical Entity Linking
Yuan, H., Lu, K., & Yuan, Z. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.3 · Cited by 5
Tweet recommendation using Clustered Bert and Word2vec Models
Kakar, S., Dhaka, D., & Mehrotra, M. (2023) · International Conference on Smart Applications, Communications and Networking (SmartNets) · doi:10.1109/smartnets58706.2023.10215867 · Cited by 5
“Bert, on the other hand stands for bidirectional encoder decoder transformer models and have been successfully used in several studies like [11], [37], [38], [39].”
MethodJoint Biomedical Entity and Relation Extraction Based on Feature Filter Table Labeling
Sun, Z. J., Xing, L., Zhang, L., Cai, H., & Guo, M. (2023) · IEEE Access · doi:10.1109/access.2023.3331504 · Cited by 5
CoVariance-based Causal Debiasing for Entity and Relation Extraction
Ren, L., Liu, Y., Cao, Y., & Ouyang, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.173 · Cited by 7
Grouin, C., Grabar, N., & Processing, S. E. F. T. I. Y. S. O. N. L. (2023) · Yearbook of Medical Informatics · doi:10.1055/s-0043-1768752 · Cited by 7
Zhang, B., Naderi, N., Mishra, R., & Teodoro, D. (2024) · JMIR AI · doi:10.2196/42630 · Cited by 7
“In this reranking step (Figure 1B), we created an ensemble of claim-checking models—robustly optimized BERT approach (RoBERTa)–Large [48], BioMedRoBERTa-base [49], and SciBERT-base [50]—which were fine-tuned on the FEVER [51] and SciFact [52] data sets.”
MethodCMBEE: A constraint-based multi-task learning framework for biomedical event extraction
Hu, J., Tang, B., Lyu, N., He, Y., & Xiong, Y. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104599 · Cited by 5
Rank Your Summaries: Enhancing Bengali Text Summarization Via Ranking-Based Approach
Shahariar, G. M., Talukder, T., Sotez, R. A. K., & Shawon, M. T. R. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-8937-9_11 · Cited by 5
Predicting the sample size of randomized controlled trials using natural language processing
Windisch, P. Y., Dennstädt, F., Koechli, C., Forster, R. E., Schröder, C., Aebersold, D. M., & Zwahlen, D. R. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae116 · Cited by 5
Kmetty, Z., Kollányi, B., & Boros, K. (2024) · SN Computer Science · doi:10.1007/s42979-024-03553-2 · Cited by 6
“…to learn the biomedical language [18], SciBERT was pre-trained on a random sample of 1.14 million scientific (mainly computer science and biomedical) articles [19], while FinBERT was pre-trained on a large financial corpus consisting of more than 46,000 news articles with 29 million words [20].”
InfluentialPersonalized Patent Claim Generation and MeasurementPreprint
Lee, J.-S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.03502 · Cited by 6
PeopleMap: Visualization Tool for Mapping Out Researchers using Natural Language ProcessingPreprint
Saad-Falcon, J., Shaikh, O., Wang, Z. J., Wright, A. P., Richardson, S., & Chau, D. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06105 · Cited by 5
Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context SizePreprint
Yoshida, D., Ettinger, A., & Gimpel, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07027 · Cited by 7
Extracting Qualitative Causal Structure with Transformer-Based NLPPreprint
Friedman, S., Magnusson, I., & Schmer-Galunder, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13304 · Cited by 5
“For our scientific claims dataset, we use the fine-tuned SciB-ERT transformer variant [Beltagy et al. , 2019] as the input layer of our architecture.”
MethodLocal Citation Recommendation with Hierarchical-Attention Text Encoder and SciBERT-based RerankingPreprint
Gu, N., Gao, Y.-Q., & Hahnloser, R. H. R. (2021) · European Conference on Information Retrieval · doi:10.48550/arxiv.2112.01206 · Cited by 50
“We design a reranker based on SciBERT [1],”
Automatic question answering for multiple stakeholders, the epidemic question answering dataset
Goodwin, T. R., Demner‐Fushman, D., Lo, K., Wang, L. L., Dang, H. T., & Soboroff, I. M. (2022) · Scientific Data · doi:10.1038/s41597-022-01533-w · Cited by 5
Self-citationPatent Search Using Triplet Networks Based Fine-Tuned SciBERTPreprint
Acikalin, U. U., & Kutlu, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.11497 · Cited by 5
“Therefore, we exploit SciBERT [1], which is pre-trained on large multi-domain corpus of scientific publications, instead of using the original BERT.”
MethodAmgpt: A Large Language Model for Contextual Querying in Additive ManufacturingPreprint
Chandrasekhar, A., Chan, J., Ogoke, F., Ajenifujah, O., & Farimani, A. B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4870552 · Cited by 6
Ding, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.04850 · Cited by 5
Automatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · arXiv · doi:10.48550/arxiv.2311.11967 · Cited by 5
AI4Research: a survey of artificial intelligence for scientific research
Chen, Q., Yang, M., Qin, L., Liu, J., Yan, Z., Guan, J., Peng, D., Ji, Y., Li, H., Hu, M., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-5000-y · Cited by 4
Building a large gene expression-cancer knowledge base with limited human annotations
Marchesin, S., Menotti, L., Giachelle, F., Silvello, G., & Alonso, O. (2023) · Database · doi:10.1093/database/baad061 · Cited by 4
Explanatory argumentation in natural language for correct and incorrect medical diagnoses
Molinet, B., Marro, S., Cabrio, E., & Villata, S. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00306-1 · Cited by 4
“For the entity detection task, we experiment with different transformer-based Language Models such as BERT [87], SciBERT [53], BioBERT [54], PubMed-BERT [55] and UmlsBERT [57] initialized with their respective pre-trained weights.”
MethodInfluentialAutomatic literature screening using the PAJO deep-learning model for clinical practice guidelines
Lin, Y., Li, J., Xiao, H., Zheng, L., Xiao, Y., Song, H., Fan, J., Xiao, D., Ai, D., Fu, T., et al. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02328-8 · Cited by 4
“Various bidirectional-encoder-representations-from-transform-ers (BERT) methods have been adapted for text processing problems, including “A Lite” BERT [12], “scientific” BERT [13], and “biomedical” BERT [14].”
MethodLeveraging shortest dependency paths in low-resource biomedical relation extraction
Enayati, S., & Vučetić, S. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02592-2 · Cited by 4
“In all the experiments, we applied SciBERT [30] as the encoder.”
MethodNie, P., Ning, J., Lin, M., Yang, Z., & Wang, L. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104719 · Cited by 4
Liu, J., Castillo‐Cara, M., & García‐Castro, R. (2025) · Expert Systems · doi:10.1111/exsy.70073 · Cited by 4
A soft-masking continual pre-training method based on domain knowledge relevance
Zhao, L., Feng, J., Qiu, H., Gu, C., & Qin, H. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104224 · Cited by 5
Wang, Q., Li, H., Ma, M., & Li, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05382-3 · Cited by 4
“As shown in Table 1 (adapted from Beltagy et al. (2019)), SciBERT achieves an average +2.11% F1 improvement with fine-tuning across tasks such as named entity recognition (NER), relation classification (REL), and citation intent classification (CLS).”
InfluentialKim, H., & Gim, G.-Y. (2025) · Applied Sciences · doi:10.3390/app15179254 · Cited by 4
TForMIX: A Method That Combines LLM and Multidimensional Modeling for Technological Foresight
Rosa, G. F., Avelino, J. O., Cavalcanti, M. C., & Duarte, J. C. (2025) · IEEE Access · doi:10.1109/access.2025.3605116 · Cited by 4
Foundation Model for Medical Imaging: A Comprehensive Review
Jiao, L., Yang, J., Li, R., Liu, F., Liu, X., Chen, P., Guo, Y., Li, L., Shang, R., & Ma, W. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3618796 · Cited by 10
“Representative encoder-only architectures include BioBERT [62], ClinicalBERT [63], SciBERT [64], COVID-twitter-BERT [65], PubMedBERT [66], BioFLAIR [67], BioMedBERT [68], BioLinkBERT [69], and others, which are pretrained on different large-scale medical text data using the BERT initialization…”
Multi-Step LLM Pipeline for Enhancing TTP Extraction in Cyber Threat Intelligence
Kim, H. R., Lee, D., Lee, I., Lee, S., & Lee, S. (2025) · IEEE Access · doi:10.1109/access.2025.3622350 · Cited by 4
Gusenbauer, M., Endermann, J., Huber, H., Strasser, S. I., Granitzer, A., & Ströhle, T. (2025) · Scientometrics · doi:10.1007/s11192-025-05490-0 · Cited by 4
Citation importance-aware document representation learning for large-scale science mapping
Liang, Z., van Eck, N. J., Wu, X., Mao, J., & Li, G. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104557 · Cited by 5
“More recently, representation learning techniques have provided a promising way to capture the semantic information of scientific documents (Beltagy et al., 2019; Lee et al., 2020; Shen et al., 2023).”
InfluentialFrom Statistical Models to LLMs: A Comprehensive Survey of Language Model Evolution
Hassanpour, H., & Majidi, M. (2024) · Journal of Artificial Intelligence Applications and Innovations · doi:10.61838/jaiai.1.4.5 · Cited by 6
Automated SNOMED CT Concept Annotation in Clinical Text Using Bi-GRU Neural Networks
Noori, A., Devkota, P., Mohanty, S. D., & Manda, P. (2025) · International Conference on Bioinformatics Research and Applications (ICBRA) · doi:10.1145/3774976.3774989 · Cited by 4
“Building on these foundations, transformer-based models—such as BERT, BioBERT [13], SciBERT [3], and ClinicalBERT—have achieved state-of-the-art re-sults by learning deep, contextualized representations from large biomedical corpora.”
Quantum Machine Learning for Drug Discovery: Taxonomy, Research Challenges, and the Road Ahead
Duong, Y. H. P., Rizvi, S. M. A., McNiven, B. D. E., Nguyen, T. T., Shin, H., Dobre, O. A., & Duong, T. Q. (2026) · ACM Computing Surveys · doi:10.1145/3785660 · Cited by 7
“SciBERT [16] addressed the challenge of acquiring labelled data in scientific domains by leveraging unsupervised pre-training on a large corpus of scientific data.”
Mathematical and Artificial Intelligence Techniques in Modern Drug Discovery: A Review
Agrwal, A., Kumar, R., Maheshwari, S., Sharma, M., Batra, C. M., Sharma, S., & Gaur, V. (2025) · Drug Development Research · doi:10.1002/ddr.70212 · Cited by 4
AI- and ML-Driven Strategies for Drug Repurposing: Techniques, Applications, and Challenges
Cong, Y., Nakagawa, S., Ogura, A., Mineta, K., Gojobori, T., & Endo, T. (2026) · Springer eBooks · doi:10.1007/978-3-031-98022-0_23 · Cited by 4
Automated comparison of scientific research based on academic genealogy
Marinosyan, A. K., Grigoriev, S. G., Михайлович, Л. И., & Anikieva, M. A. (2026) · Informatics and Education · doi:10.32517/0234-0453-2025-40-6-16-27 · Cited by 5
Campo, A. O. D., Lituiev, D., Varma, G., Manoharan, M., Ravi, S. K., Aman, A., Kansagra, A. J., Greshock, J., Venkatakrishnan, A., & Batavia, A. S. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03345-z · Cited by 5
“These proprietary models are fine-tuned versions of SciBERT cased [39] (basis for models 1–5 specified above) and ClinicalBERT [40] (basis for model 6).”
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.