SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 501–600 of 3,596
DisenCite: Graph-Based Disentangled Representation Learning for Context-Specific Citation Generation
Wang, Y., Song, Y., Li, S., Cheng, C., Ju, W., Zhang, M., & Wang, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21397 · Cited by 49
Moro, G., & Valgimigli, L. (2021) · Sensors · doi:10.3390/s21196430 · Cited by 35
“In particular, we selected SciBERT [3] because it already contains knowledge on the scientific articles’ domain, and we did not perform any structural change to the architecture of the network.”
BackgroundMethodInfluentialMapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 38
Stylianou, N., & Vlahavas, I. P. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103767 · Cited by 38
Explaining ambiguity in scientific language
Sterner, B. (2022) · Synthese · doi:10.1007/s11229-022-03792-x · Cited by 40
“The most popular deep learning algorithms in NLP over the past few years have large gaps in coverage for scientific terminology: SciBERT, for example, has training data from computer science and biomedicine but not ecology or evolution (Beltagy et al., 2019).”
BackgroundNaliyatthaliyazchayil, P., Muthyala, R., Gichoya, J. W., & Purkayastha, S. (2025) · Journal of Medical Internet Research · doi:10.2196/74142 · Cited by 36
“The allenai/scibert_scivocab_uncased variant of SciBERT [14,15], implemented through the SentenceTransformer framework, was used to generate embeddings for both the ground truth primary diagnosis (from MIMIC-IV clinical notes) and the LLM-predicted diagnosis.”
Few-shot Named Entity Recognition: Definition, Taxonomy and Research Directions
Moscato, V., Postiglione, M., & Sperlí, G. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3609483 · Cited by 41
“Experiments show that this simple approach has worse F1-scores than domain-specific pre-trained language models (e.g., BioBERT [69], SciBERT [8]), but it could be a good alternative in low-resource languages and domains where huge domain-specific language model variants are not available.”
BackgroundDeep learning for predicting patent application outcome: The fusion of text and network embeddings
Jiang, H., Fan, S., Zhang, N., & Zhu, B. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101402 · Cited by 38
The changing landscape of text mining: a review of approaches for ecology and evolution
Farrell, M. J., Guillarme, N. L., Brierley, L., Hunter, S. B., Scheepens, D. R., Willoughby, A. R., Yates, A., & Mideo, N. (2024) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2024.0423 · Cited by 42
“While there exist pre-trained language models designed to work with scientific literature, they are either explicitly biomedical focused, such as bioBERT [53] and PubMedBERT [54], or the majority of the training corpus is from the biomedical domain, e.g. scispaCy [14] and sciBERT [51].”
BackgroundMethodInfluentialHERGen: Elevating Radiology Report Generation with Longitudinal Data
Wang, F., Du, S., & Yu, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73001-6_11 · Cited by 69
Polignano, M., Musto, C., de Gemmis, M., Lops, P., & Semeraro, G. M. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3474272 · Cited by 41
“Even if WE techniques have been largely used to provide users with recommendations [32], the adoption of contextual word representations such as BERT [13], Elmo [4], Universal Sentence Encoder (USE) [9] in recommendation tasks is poorly investigated.”
MethodGERNERMED: An open German medical NER model
Frei, J., & Krämer, F. (2021) · Software Impacts · doi:10.1016/j.simpa.2021.100212 · Cited by 34
SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model
Andrade, S. R., & Walsh, H. S. (2023) · AIAA AVIATION Forum · doi:10.2514/6.2023-3437 · Cited by 32
Multi-Stage Pre-training for Low-Resource Domain Adaptation
Zhang, R., Reddy, R. G., Sultan, M. A., Castelli, V., Ferritto, A., Florian, R., Kayi, E. S., Roukos, S., Sil, A., & Ward, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.440 · Cited by 45
Word embeddings for biomedical natural language processing: A survey
Chiu, B., & Baker, S. (2020) · Language and Linguistics Compass · doi:10.1111/lnc3.12402 · Cited by 37
End-to-end Argument Mining with Cross-corpora Multi-task Learning
Morio, G., Ozaki, H., Morishita, T., & Yanai, K. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00481 · Cited by 68
“RoBERTa/SciBERT (Mayer et al., 2020), 8 a Transformer (Vaswani et al., 2017)-based AM parser that uses RoBERTa (Liu et al., 2019b) or SciBERT (Beltagy et al., 2019).”
MethodHarnessing GPT-3.5 for text parsing in solid-state synthesis – case study of ternary chalcogenides
Thway, M., Low, A. K. Y., Khetan, S., Dai, H., Recatala‐Gomez, J., Chen, A. P., & Hippalgaonkar, K. (2024) · Digital Discovery · doi:10.1039/d3dd00202k · Cited by 37
Wu, W., Zhang, C., & Zhao, Y. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70005 · Cited by 43
“We use the pre-trained language models (PLMs) such as BERT (Devlin et al., 2019), SciBERT (Beltagy et al., 2019), etc. as the text encoder to obtain the text representation.”
EventPlus: A Temporal Event Understanding Pipeline
Ma, M. D., Sun, J., Yang, M., Huang, K.-H., Wen, N., Singh, S., Han, R., & Peng, N. (2021) · doi:10.18653/v1/2021.naacl-demos.7 · Cited by 45
CataLM: empowering catalyst design through large language models
Wang, L., Chen, X., Du, Y., Zhou, Y., Gao, Y., & Cui, W. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02473-0 · Cited by 43
“The control method that should be used is anodized copper (AN-Cu) Cu(OH)(2) catalysts by a simple electrochemical synthesis method.”
A Domain-adaptive Pre-training Approach for Language Bias Detection in News
Krieger, J.-D., Spinde, T., Ruas, T., Kulshrestha, J., & Gipp, B. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530932 · Cited by 34
“Adapting a pre-trained language model to a specific domain becomes essential when the target domain differs strongly from the pre-training ground truth [4, 15, 20].”
BackgroundInfluentialA survey on cutting-edge relation extraction techniques based on language models
Diaz-Garcia, J. A., & Amador, J. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11280-0 · Cited by 47
hmBERT: Historical Multilingual Language Models for Named Entity RecognitionPreprint
Schweter, S., März, L., Schmid, K., & Çano, E. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.15575 · Cited by 32
“[6] showed that using a domain-specific vocabulary leads to performance improvements compared to using a general domain vocabulary.”
Huang, T.-H., Huang, C.-Y., Ding, C. C., Hsu, Y.-C., & Giles, C. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02367 · Cited by 32
El-allaly, E.-D., Sarrouti, M., En-Nahnahi, N., & Alaoui, S. O. E. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103968 · Cited by 34
Chintalapati, S. S., Bragg, J., & Wang, L. L. (2022) · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3517428.3544796 · Cited by 31
“We experiment with a Random Forest classifier using tf-idf word representations, as well as classifiers based on BERT [14] and SciBERT [3].”
MethodClinicalRadioBERT: Knowledge-Infused Few Shot Learning for Clinical Notes Named Entity Recognition
Rezayi, S., Dai, H., Liu, Z., Wu, Z., Hebbar, A., Burns, A. H., Zhao, L., Zhu, D., Li, Q., Liu, W., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21014-3_28 · Cited by 61
KEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 43
Sequential sentence classification in research papers using cross-domain multi-task learning
Brack, A., Entrup, E., Stamatakis, M., Buschermöhle, P., Hoppe, A., & Ewerth, R. (2024) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00392-z · Cited by 35
“To the best of our knowledge, a unified approach for different types of texts that differ noticeably by their structure and semantic context of sentences, as is the case for abstracts and full papers, has not been proposed yet.”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora
Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.-W. D., Wei, X., Arnold, A., & Ren, X. (2022) · doi:10.18653/v1/2022.bigscience-1.1 · Cited by 33
Torkamaan, H., Steinert, S., Pera, M. S., Kudina, O., Freire, S. K., Verma, H., Kelly, S., Sekwenz, M.-T., Yang, J., van Nunen, K., et al. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2431068 · Cited by 43
Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies
Pergola, G., Kochkina, E., Gui, L., Liakata, M., & He, Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.169 · Cited by 50
Jeong, Y., & Kim, E. (2022) · IEEE Access · doi:10.1109/access.2022.3180830 · Cited by 45
The accuracy of field classifications for journals in Scopus
Thelwall, M. A., & Pinfield, S. (2024) · Scientometrics · doi:10.1007/s11192-023-04901-4 · Cited by 31
“Alternatives Marginally more accurate cosines might have been achieved if the SciB-ERT (Beltagy et al., 2019) language model had been used to detect the senses of the terms in the abstracts rather than using the ngram approach.”
BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition
Li, Y., Shetty, P., Liu, L., Zhang, C., & Song, L. (2021) · doi:10.18653/v1/2021.acl-long.482 · Cited by 41
“The original BERT (Devlin et al., 2019) is used for CoNLL 2003 and LaptopReview datasets, bioBERT (Lee et al., 2019) for NCBI-Disease and SciBERT (Belt-agy et al., 2019) for BC5CDR.”
BackgroundBrack, A., Hoppe, A., Stocker, M., Auer, S., & Ewerth, R. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00306-x · Cited by 37
“Beltagy et al. [9] present benchmarks for several scientific datasets and Peng et al. [81] especially for the biomedical domain.”
BackgroundIvanisenko, T., Сергеевич, Д. П., & Иванисенко, В. А. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252111811 · Cited by 40
Keyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized EmbeddingsPreprint
Sahrawat, D., Mahata, D., Kulkarni, M., Zhang, H., Gosangi, R., Stent, A. J., Sharma, A., Kumar, Y., Shah, R. R., & Zimmermann, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08840 · Cited by 36
LLM-based feature generation from text for interpretable machine learning
Balek, V., Sýkora, L., Sklenák, V., & Kliegr, T. (2025) · Machine Learning · doi:10.1007/s10994-025-06867-1 · Cited by 40
“For experiments on BANKING-77, Hate Speech, and Food Hazard datasets, which were several times larger than M17+ and CORD-19, we used the Random Forest Classifier, which resulted in a very similar performance to the Gradient Boost Classifier but was faster.”
Liu, P., Tao, J., & Ren, Z. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00977-6 · Cited by 45
“BERT and its derivatives like RoBERTa 25 and domain-specific variants such as SciBERT 26 and Chem-BERTa 27 IUPAC name recognition .”
Zeng, Q., Bai, Y., Tan, Z., Feng, S., Liang, Z., Zhang, Z., & Jiang, M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679608 · Cited by 60
“The demonstration 𝑑 𝑖 employed for expanding the 𝑘 -th layer of the demo taxonomy We use the first five sub-taxonomies from WordNet’s training set as demonstrations”
MethodKing, D., Shen, Z., Subramani, N., Weld, D. S., Beltagy, I., & Downey, D. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.51 · Cited by 48
Self-citationMeadows, J., & Freitas, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00594 · Cited by 43
Language models for materials discovery and sustainability: Progress, challenges, and opportunities
Pei, Z., Yin, J., & Zhang, J. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101495 · Cited by 30
External features enriched model for biomedical question answering
Xu, G., Rong, W., Wang, Y., Ouyang, Y., & Xiong, Z. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04176-7 · Cited by 35
Salatino, A. A., Osborne, F., & Motta, E. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00305-y · Cited by 38
“Garcia-Silva et al. [23] focus on the task of classifying scientific publications against a taxonomy of scientific disciplines, taking advantage of BERT [20] and its different flavours specialised in the scientific domain: BioBERT [33] and SciBERT [5].”
Local Citation Recommendation with Hierarchical-Attention Text Encoder and SciBERT-Based Reranking
Gu, N., Gao, Y., & Hahnloser, R. H. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_19 · Cited by 58
DisorBERT: A Double Domain Adaptation Model for Detecting Signs of Mental Disorders in Social Media
Aragón, M. E., López-Monroy, A. P., González, L. C., Losada, D. E., & Montes, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.853 · Cited by 60
SemEval-2021 Task 8: MeasEval – Extracting Counts and Measurements and their Related Contexts
Harper, C. A., Cox, J., Kohler, C., Scerri, A., Daniel, R., & Groth, P. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.38 · Cited by 43
Multi-Task Attentive Residual Networks for Argument Mining
Galassi, A., Lippi, M., & Torroni, P. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3275040 · Cited by 37
Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications
Pei, J., & Jurgens, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.784 · Cited by 21
“We use the best-performing SciBERT model (r=0.70) as the regressor for sentence-level certainty in the following analyses.”
MethodInfluentialAWS CORD-19 Search: A Neural Search Engine for COVID-19 Literature
Bhatia, P., Liu, L., Arumae, K., Pourdamghani, N., Deshpande, S., Snively, B., Mona, M., Wise, C., Price, G., Ramaswamy, S., et al. (2022) · Studies in computational intelligence · doi:10.1007/978-3-030-93080-6_11 · Cited by 30
TermInformer: unsupervised term mining and analysis in biomedical literature
Tiwari, P., Uprety, S., Dehdashti, S., & Hossain, M. S. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05335-2 · Cited by 33
A Gradually Soft Multi-Task and Data-Augmented Approach to Medical Question Understanding
Mrini, K., Dernoncourt, F., Yoon, S., Bui, T., Chang, W. H., Farcas, E., & Nakashole, N. (2021) · doi:10.18653/v1/2021.acl-long.119 · Cited by 42
Text Classification Using Neural Network Language Model (NNLM) and BERT: An Empirical Comparison
Esmaeilzadeh, A., & Taghva, K. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-82199-9_12 · Cited by 29
LegalBert-pt: A Pretrained Language Model for the Brazilian Portuguese Legal Domain
Silveira, R., Ponte, C., Almeida, V. M. S. S. D., Pinheiro, V. C. M., & Furtado, V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45392-2_18 · Cited by 44
Pairwise Multi-Class Document Classification for Semantic Relations between Wikipedia Articles
Ostendorff, M., Ruas, T., Schubotz, M., Rehm, G., & Gipp, B. (2020) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3383583.3398525 · Cited by 32
Soni, S., & Roberts, K. E. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa271 · Cited by 29
BioReader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature
Frisoni, G., Mizutani, M., Moro, G., & Valgimigli, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.390 · Cited by 21
Truly standard‐essential patents? A semantics‐based analysis
Brachtendorf, L., Gaessler, F., & Harhoff, D. (2022) · Journal of Economics & Management Strategy · doi:10.1111/jems.12500 · Cited by 29
S-NER: A Concise and Efficient Span-Based Model for Named Entity Recognition
Yu, J., Ji, B., Li, S., Ma, J., Liu, H., & Xu, H. (2022) · Sensors · doi:10.3390/s22082852 · Cited by 37
“For a fair comparison with prior work, we also use SciBERT [35] when evaluating S-NER on SciERC.”
MethodInfluentialTCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing
Ren, M., Huang, H., Zhou, Y., Cao, Q., Bu, Y., & Gao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_16 · Cited by 46
A BERT-based One-Pass Multi-Task Model for Clinical Temporal Relation Extraction
Lin, C., Miller, T., Dligach, D., Sadeque, F., Bethard, S. J., & Savova, G. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.7 · Cited by 37
Portelli, B., Lenzi, E., Chersoni, E., Serra, G., & Santus, E. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.149 · Cited by 29
CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding
Wright, D., & Augenstein, I. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.157 · Cited by 41
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.896 · Cited by 74
Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End
Chen, Y., & Eger, S. (2023) · Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2023.eval4nlp-1.6 · Cited by 24
Structure-inducing pre-training
McDermott, M. B. A., Yap, B., Szolovits, P., & Žitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00647-z · Cited by 33
“Baseline We compare against the published SciBERT model [91] as our per-token comparison and lack an associated per-sample comparison as we don’t know of any published per-sample models in the academic papers modality.”
BackgroundMethodInfluentialMultilingual translation for zero-shot biomedical classification using BioTranslator
Xu, H., Woicik, A., Poon, H., Altman, R. B., & Wang, S. (2023) · Nature Communications · doi:10.1038/s41467-023-36476-2 · Cited by 48
Graph Neural Networks and Representation Embedding for Table Extraction in PDF Documents
Gemelli, A., Vivoli, E., & Marinai, S. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956590 · Cited by 35
Automatic detection of health misinformation: a systematic review
Schlicht, I. B., Fernandez, E., Chulvi, B., & Rosso, P. (2023) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-023-04619-4 · Cited by 42
“Also some researchers used domain specific transformers (Kotonya and Toni 2020) such as SciBERT (Beltagy et al. 2019) or applied a domain adaptation onto transformer embed-dings (Dharawat et al. 2020) and (Hossain et al. 2020).”
Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data
Lowell, D., Howard, B. E., Lipton, Z. C., & Wallace, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.408 · Cited by 36
SciNLI: A Corpus for Natural Language Inference on Scientific Text
Sadat, M., & Caragea, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.511 · Cited by 57
CORD19STS: COVID-19 Semantic Textual Similarity DatasetPreprint
Guo, X. B., Mirzaalian, H., Sabir, E., Jaiswal, A., & AbdAlmageed, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02461 · Cited by 34
AlzheimerRAG: Multimodal Retrieval-Augmented Generation for Clinical Use Cases
Lahiri, A. K., & Hu, Q. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7030089 · Cited by 31
“• Early Diagnosis and Monitoring : Assess the system’s ability to recommend diagnostic tools and interpret results for early detection.”
Pre-training Methods in Information RetrievalPreprint
Fan, Y., Xie, X., Cai, Y., Chen, J., Ma, X., Li, X., Zhang, R., & Guo, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13853 · Cited by 34
Towards employing native information in citation function classification
Zhang, Y., Zhao, R., Wang, Y., Chen, H., Mahmood, A., Zaib, M., Zhang, W. E., & Sheng, Q. Z. (2022) · Scientometrics · doi:10.1007/s11192-021-04242-0 · Cited by 28
“Recently, Beltagy et al. (2019) proposed SciBERT which shares a similar training scheme and architecture with BERT, but was trained on 1.14M scientific papers.”
BackgroundMethodInfluentialTong, Y., Chen, Y., & Shi, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.424 · Cited by 37
Pride, D., Cancellieri, M., & Knoth, P. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43849-3_13 · Cited by 28
Kungfupanda at SemEval-2020 Task 12: BERT-Based Multi-TaskLearning for Offensive Language Detection
Dai, W., Yu, T., Liu, Z., & Fung, P. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.272 · Cited by 44
Improving BERT-based Query-by-Document Retrieval with Multi-task Optimization
Abolghasemi, A., Verberne, S., & Azzopardi, L. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_1 · Cited by 48
A review on Natural Language Processing Models for COVID-19 research
Hall, K., Chang, V. I. C., & Jayne, C. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100078 · Cited by 33
An empirical assessment of different word embedding and deep learning models for bug assignment
Wang, R., Ji, X., Xu, S., Tian, Y., Jiang, S., & Huang, R. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.111961 · Cited by 34
A study of the generalizability of self-supervised representations
Tendle, A., & Hasan, M. R. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100124 · Cited by 44
“We will create domain-specific representations (Beltagy et al., 2019) to compare them with domain-agnostic representations.”
Methodpysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasksPreprint
Perez, J. M., Rajngewerc, M., Giudici, J. C., Furman, D., Luque, F. M., Alemany, L. A., & Martínez, M. V. (2023) · Research Square · doi:10.21203/rs.3.rs-3570648/v1 · Cited by 39
Automatic extraction of materials and properties from superconductors scientific literature
Foppiano, L., de Castro, P. B., Suárez, P. O., Terashima, K., Takano, Y., & Ishii, M. (2022) · Science and Technology of Advanced Materials Methods · doi:10.1080/27660400.2022.2153633 · Cited by 28
Rehman, T., Sanyal, D. K., Chattopadhyay, S., Bhowmick, P. K., & Das, P. P. (2023) · IEEE Access · doi:10.1109/access.2023.3292300 · Cited by 31
Miranda-Escalada, A., Mehryary, F., Luoma, J., Estrada, D., Gascó, L., Pyysalo, S., Valencia, A., & Krallinger, M. (2023) · Database · doi:10.1093/database/baad080 · Cited by 36
“Some of the most common transformer-based pre-trained LMs are BERT (34), BioBERT (20), SciBERT (35) and PubMedBERT (36).”
BackgroundLarge language model enhanced corpus of CO2 reduction electrocatalysts and synthesis procedures
Chen, X., Gao, Y., Wang, L., Cui, W., Huang, J., Du, Y., & Wang, B. (2024) · Scientific Data · doi:10.1038/s41597-024-03180-9 · Cited by 35
“In the first step, we used the SciBERT model 31 to convert each word token into an embedding vector.”
MethodBreaking Down Walls of Text: How Can NLP Benefit Consumer Privacy?
Ravichander, A., Black, A. W., Norton, T. B., Wilson, S., & Sadeh, N. (2021) · doi:10.18653/v1/2021.acl-long.319 · Cited by 41
Deep learning meets bibliometrics: A survey of citation function classification
Zhang, Y., Wang, Y., Sheng, Q. Z., Yao, L., Chen, H., Wang, K., Mahmood, A., Zhang, W. E., Zaib, M., Sagar, S., et al. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101608 · Cited by 42
A survey of knowledge organization systems of research fields: Resources and challenges
Salatino, A. A., Aggarwal, T., Mannocci, A., Osborne, F., & Motta, E. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00363 · Cited by 35
“For instance, Pisu et al. (2024) introduced an AI-driven pipeline that extracts research concepts from articles and then determines their semantic relationships (hierarchical or synonymous) using SciBERT (Beltagy et al., 2019).”
Apel, M., Betzer, A., & Scherer, B. (2021) · Finance research letters · doi:10.1016/j.frl.2022.103600 · Cited by 29
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs.
Ghosh, M., Mukherjee, S., Ganguly, A., Basuchowdhuri, P., Naskar, S. K., & Ganguly, D. (2024) · Methods · doi:10.1016/j.ymeth.2024.04.005 · Cited by 27
“• SciBERT [17]: Similar to BioBERT work, we finetune the SciBERT language model for the downstream the PICO frame extraction task by considering it as sequence labeling task.”
MethodInfluentialAutomating the removal of obsolete TODO comments
Gao, Z., Xia, X., Lo, D., Grundy, J., & Zimmermann, T. (2021) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 27
Few-Shot Class-Incremental Learning for Named Entity Recognition
Wang, R., Yu, T., Zhao, H., Kim, S., Mitra, S., Zhang, R., & Henao, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.43 · Cited by 39
Weakly Supervised Named Entity Tagging with Learnable Logical Rules
Li, J., Ding, H., Shang, J., McAuley, J., & Feng, Z. (2021) · doi:10.18653/v1/2021.acl-long.352 · Cited by 33
“In our experiments, we use SciBert for two biomedical datasets and Bert for CoNLL2003 dataset.”
BackgroundMethodPre-train or Annotate? Domain Adaptation with a Constrained Budget
Bai, F., Ritter, A., & Xu, W. H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.409 · Cited by 43
“The overall training of SciBERT took 7 days (5 days for the first stage and 2 days for the second stage) with an estimated cost of 1,340 USD.”
MethodInfluentialCOVID-HateBERT: a Pre-trained Language Model for COVID-19 related Hate Speech Detection
Li, M., Liao, S., Okpala, E., Tong, M., Costello, M., Cheng, L., Hu, H., & Luo, F. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00043 · Cited by 27
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.