SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,901–2,000 of 3,596
How Green are Neural Language Models? Analyzing Energy Consumption in Text Summarization Fine-tuning
Rehman, T., Sanyal, D. K., & Chattopadhyay, S. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-9724-3_47 · Cited by 6
Akande, O. A., Ijiga, O. M., Bamigwojo, O. V., & James, A. O. (2026) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/26jan1453 · Cited by 4
Chen, H., Zhou, Y., Li, R., Illa, A. M., Cleveland, A. D., & Ding, J. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105005 · Cited by 7
An overview of artificial intelligence approaches for automating evidence synthesis
Harasgama, S., Pearce, H., Loftus, L., Painter, H., & Ford, J. A. (2026) · Public Health · doi:10.1016/j.puhe.2026.106220 · Cited by 5
SciNUP: Natural Language User Interest Profiles for Scientific Literature Recommendation
Arustashvili, M., & Balog, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_51 · Cited by 5
Transfer Learning-based Model Training for Short-term Load Forecasting
Jiang, B., Hongyuan, Y., Wang, Y., Yang, G., & Hua, G. (2026) · Journal of Modern Power Systems and Clean Energy · doi:10.35833/mpce.2024.000940 · Cited by 4
Lv, J., Liu, G., Zhou, X., & Chen, Y. (2026) · Probiotics and Antimicrobial Proteins · doi:10.1007/s12602-026-11013-x · Cited by 4
“A pre-trained language (i.e., SciBERT [147]) was used to extract accurate text representations that guide peptide generation.”
Kertész, G., Czere, J. T., Zrubka, Z., Gulàcsi, L., & Péntek, M. (2026) · JMIR Formative Research · doi:10.2196/86647 · Cited by 4
“While BERT itself is a model pretrained on general data [36], there are existing pretrained alternatives with scientific or biomedical domains, such as Scientific BERT (SciBERT) [37], Biomedical BERT (BioBERT) [38], or Biomedical Language Understanding Evaluation BERT (BlueBERT) [39].”
Using Center Vector and Drug Molecular Information for Drug Drug Interaction Extraction
Duan, B., Qin, L., & Peng, J. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669610 · Cited by 4
RadTex: Learning Efficient Radiograph Representations from Text Reports
Quigley, K., Cha, M., Liao, R., Chauhan, G., Horng, S., Berkowitz, S. J., & Golland, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16876-5_3 · Cited by 5
“To address some of these differences, we use a vocab list specific to the scientific domain, scibert_scivocab_uncased [3], take text only from the Findings section of the report, and lengthen the maximum caption size that the model can accept to 170 tokens (95 th percentile of finding section length).”
MethodTransforming Drug-Drug Interaction Extraction from Biomedical Literature
Zaikis, D., Kokkas, S., & Vlahavas, I. P. (2022) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3549737.3549753 · Cited by 4
“• SciBERT [1] similarly, is a domain-specific language representationmodel pre-trained on scientific texts from Semantic Scholar and has its own WordPiece vocabulary to match the”
BackgroundComprehensive study of pre-trained language models: detecting humor in news headlines
Shatnawi, F., Abdullah, M., Hammad, M., & Al‐Ayyoub, M. (2022) · Soft Computing · doi:10.1007/s00500-022-07573-z · Cited by 8
“Natural Language Processing (NLP) is considered the primary pillar for obtaining the linguistic knowledge features from text (Beltagy et al 2019).”
BackgroundTab-Cleaner: Weakly Supervised Tabular Data Cleaning via Pre-training for E-commerce Catalog
Cheng, K., Li, X., Wang, Z., Zhang, C., Huang, B., Xu, Y., Dong, X. L., & Sun, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.18 · Cited by 8
W&G-BERT: A Pretrained Language Model for Automotive Warranty and Goodwill Text
Weber, L. J., Ramalingam, K. J., Beyer, M., Kirchheim, A., & Zimmermann, A. P. (2023) · International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd57115.2023.10206043 · Cited by 4
“Besides W&G-specific MLM pretraining, it is still beneficial for W&G-AER downstream tasks to use SciBERT as a pretrained model.”
MethodInfluentialMemorization of Named Entities in Fine-Tuned BERT Models
Diera, A., Lell, N., Garifullina, A., & Scherp, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40837-3_16 · Cited by 6
“The most notable variants include RoBERTa [32], DistilBERT [45], DeBERTa [23], and domain-specific models such as SciBERT [5] or ClinicalBERT [3].”
Menad, S., Abdeddaïm, S., & Soualmia, L. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42935-4_3 · Cited by 4
Steegh, E., & Sileno, G. (2023) · International Conference on Artificial Intelligence and Law · doi:10.1145/3594536.3595171 · Cited by 7
Information Extraction and Machine Learning for Archaeological Texts
Brandsen, A. (2023) · Quantitative archaeology and archaeological modelling · doi:10.1007/978-3-031-37156-1_11 · Cited by 4
CocoSciSum: A Scientific Summarization Toolkit with Compositional Controllability
Ding, Y., Qin, Y., Liu, Q., & Kan, M. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.47 · Cited by 8
Ai, X., & Kavuluru, R. (2023) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi57859.2023.00108 · Cited by 6
“PURE uses a contextualized language model (such as SciBERT [24] or PubMedBERT [4] in our case) to process the input sequence and obtain contextualized embeddings for each token.”
MethodHu, B., Lin, A., & Brinson, L. C. (2024) · Integrating materials and manufacturing innovation · doi:10.1007/s40192-024-00363-5 · Cited by 8
“Examples in the materials science domain include SciBERT, MatBERT, and MatSciBERT, all of which selected to pre-train BERT-based transformers with domain-specific corpora [28–30].”
Training a language model to learn the syntax of commands
Hussain, Z., Nurminen, J. K., & Ranta-aho, P. O. (2024) · Array · doi:10.1016/j.array.2024.100355 · Cited by 4
GTR: An explainable Graph Topic-aware Recommender for scholarly document
Ni, P., Wang, X., Lv, B., & Wu, L. (2024) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2024.101439 · Cited by 4
An Empirical Study for the GPT-Based LLM in Paper Similarity Measurement
Yang, J. (2024) · International Conference on Electronic Communication and Artificial Intelligence (ICECAI) · doi:10.1109/icecai62591.2024.10674909 · Cited by 6
“Given these advantages of the Transformer model in text processing, we utilized the SciBERT model, based on the Transformer architecture, in our subsequent experiments to test paper similarity [14].”
MethodLeveraging Large Language Models on the Traditional Scientific Writing Workflow
Procko, T., Newcomb, A., Elvira, T., & Ochoa, O. (2024) · Conference on AI, Science, Engineering, and Technology (AIxSET) · doi:10.1109/aixset62544.2024.00028 · Cited by 6
“SciBERT is pretrained on a random sample of approximately 1.4 million papers from Semantic Scholar, where 18% of its training corpus comes from papers in the computer science domain, and 82% comes from papers in the biomedical domain [41].”
InfluentialPath-Aware Reasoning Network for document-level relation extraction with co-regularization loss
Yin, Y., Ai, X., Yu, Y., Qu, X., & Wei, W. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112931 · Cited by 4
MedNote BERT: Modeling Medical Notes and Predicting Drug-Disease Interactions
Mishra, P., & Sharma, S. (2024) · 2024 International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa63652.2024.10836617 · Cited by 1
Improving inference via rich path information and logic rules for document-level relation extraction
Su, H., Xie, S., Yu, H., Yuan, C., Wang, X., & Luo, X. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-024-02336-8 · Cited by 4
“We utilize the cased BE RT BASE [34] as the encoder for DocRED dataset and the cased SciBERT [41] as the encoder for the CDR and GDA datasets.”
EHealth: A Chinese Biomedical Language Model Built via Multi-Level Text Discrimination
Wang, Q., Dai, S., Xu, B., Lyu, Y., Wu, H. J., & Wang, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3536177 · Cited by 4
Small Language Models for Emotion Recognition in Polish Stock Market Investor Opinions
Koptyra, B., Oleksy, M., Dzięcioł, E., & Kocoń, J. (2024) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw65004.2024.00065 · Cited by 4
“• Insights into overcoming class imbalance and the potential benefits of domain-specific pre-training [2].”
The influence of grant renewal on research content: evidence from NIH-funded PIs
Li, B., & Bai, A. (2025) · Scientometrics · doi:10.1007/s11192-025-05295-1 · Cited by 4
“…various level, including Word2vec for word-level semantic vector representations (Mikolov et al., 2013), SciBERT for sentence-level representations (Beltagy et al., 2019), and SPECTER for document-level representations (Cohan et al., 2020), among others, have been employed to measure research…”
Modeling and Analysis of Question Answering for Long Context and Compound Questions Using BERT Model
Sivakumar, S., & Meenakshi, S. P. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.529 · Cited by 4
Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., Xie, Q., Xie, G., & Gong, X. (2025) · The VLDB Journal · doi:10.1007/s00778-025-00919-7 · Cited by 8
Wu, X., She, Y., Wang, X., Lu, H., & Gao, Q. (2025) · Applied Sciences · doi:10.3390/app15105674 · Cited by 4
AI-Powered Knowledge Graphs for Neuromorphic and Energy-Efficient Computing
Gautam, A., Patton, R. M., Potok, T. E., Kannan, R., Aimone, J. B., & Severa, W. (2025) · ACM Great Lakes Symposium on VLSI (GLSVLSI) · doi:10.1145/3716368.3735295 · Cited by 5
EMK-KEN: A high-performance approach for assessing knowledge value in citation network
Liu, C., & Qu, Z. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113949 · Cited by 4
Wu, D., Wu, H., Meng, F., & Li, J. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70018 · Cited by 4
“Content-based classification techniques have gained prominence with advancements in natural language processing (NLP) (Beltagy et al., 2019; Singhal et al., 2023).”
Influential+VeriRel: Verification Feedback to Enhance Document Retrieval for Scientific Fact Checking
Deng, X., Wang, X., & Stevenson, M. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760822 · Cited by 4
“6, a pre-trained model SciBERT [1], effective in scientific domains [8, 20, 24]”
ReviewGuard: Enhancing Deficient Peer Review Detection via LLM-Driven Data Augmentation
Zhang, H., Li, R., Shrestha, S., Mamidala, S. H., Putta, R., Aggarwal, A., Xiao, T., Ding, J., & Chen, H. (2025) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl67857.2025.00032 · Cited by 6
“• SciBERT-scivocab-uncased [44]: Pre-trained on a corpus of 1.14 million scientific abstracts, it provides domain-adapted embeddings and vocabulary optimized for technical discourse in machine learning and related fields.”
InfluentialSCUT-CCNL at TREC 2019 Precision Medicine Track.
Liu, X., Li, L., Yang, Z., & Dong, S. (2019) · Text REtrieval Conference · Cited by 4
Iwatsuki, K., Boudin, F., & Aizawa, A. (2020) · HAL · Cited by 4
SciNER: A Novel Scientific Named Entity Recognizing Framework
Yan, T., Huang, H., & Mao, X.-L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_65 · Cited by 5
Nguyen, A. T. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.42 · Cited by 4
Pawate, A. G., Madhavan, V., & Chandak, D. (2020) · doi:10.18653/v1/2020.textgraphs-1.12 · Cited by 4
Improved Biomedical Entity Recognition via Longer Context Modeling
Stylianou, N., Kosmoliaptsis, P., & Vlahavas, I. P. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-79150-6_4 · Cited by 5
Experiments with Contextualized Word Embeddings for Keyphrase Extraction
Dascalu, C., & Trăușan-Matu, Ș. (2021) · International Conference on Control Systems and Computer Science · doi:10.1109/cscs52396.2021.00079 · Cited by 4
KU-DMIS at BioASQ 9: Data-centric and model-centric approaches for biomedical question answering.
Yoon, W., Yoo, J., Seo, S., Sung, M., Jeong, M., Kim, G., & Kang, J. (2021) · CLEF (Working Notes) · Cited by 5
Adaptor: Objective-Centric Adaptation Framework for Language Models
Štefánik, M., Novotný, V. S., Groverová, N., & Sojka, P. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.26 · Cited by 7
Dual Reasoning Based Pairwise Representation Network for Document Level Relation Extraction
Li, Y., Liu, Y., Gu, X., Yue, Y., Fan, H., & Li, B. (2022) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859653 · Cited by 4
“We use BERT-base [20] as the encoder on DocRED, and SciBERT [25] on CDR and GDA.”
MethodOverview of NLPCC2022 Shared Task 5 Track 1: Multi-label Classification for Scientific Literature
Liu, M., Zhang, H., Tian, Y., Zong, T., Cai, B., Xu, R., & Li, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17189-5_28 · Cited by 4
Kalla, A., Mukhopadhyay, S., Ralte, Z., & Kar, I. (2023) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs57279.2023.10112876 · Cited by 4
“For the task of Adverse signal detection, employing pretrained models [2] has been effective in improving many natural language processing tasks such as name entity recognition, event extraction, etc.”
MethodHey, T. (2023) · OECD eBooks · doi:10.1787/7b7b1bce-en · Cited by 5
Detecting science-based health disinformation: a stylometric machine learning approach
Williams, J. A., AlEroud, A. F., & Zimmerman, D. (2023) · Journal of Computational Social Science · doi:10.1007/s42001-023-00213-y · Cited by 5
“We used two more variants of BERT, SciBERT [50] and CoronaBERT [51] models which process the research literature on SARS-CoV-2 together with SARS-CoV and MERS-CoV.”
MethodOnline Research Topic Modeling and Recommendation Utilizing Multiview Autoencoder-Based Approach
Paul, D., Chakdar, D., Saha, S., & Mathew, J. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3253502 · Cited by 4
“To perform embedding of this article, we utilized a scientific version of BERT [24] named SciBERT [6].”
MethodA Full-fledged Commit Message Quality Checker Based on Machine Learning
Faragó, D., Färber, M., & Petrov, C. (2023) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00108 · Cited by 4
Urteaga, I., Draidia, M. Z., Lancewicki, T., & Khadivi, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.675 · Cited by 4
“Many have replicated the pre-train-then-fine-tune strategy in different domains, e.g., pre-training BERT with scientific (Beltagy et al., 2019) and biomedical corpora (Lee et al., 2020; Alsentzer et al., 2019; Gu et al., 2021); or in-house, industry-specific TLMs (Kalyan et al., 2021).”
BackgroundSemantics-enabled biomedical literature analytics
Kilicoglu, H., Ensan, F., McInnes, B. T., & Wang, L. L. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104588 · Cited by 4
Toward the novel AI tasks in infection biology
Yakimovich, A. (2024) · mSphere · doi:10.1128/msphere.00591-23 · Cited by 4
“The arrival of models like AlphaFold (10), large lan guage model architectures (19, 20), and concepts like foundational models (FM) (21) is steering the field away from the “sorting machine” paradigm toward the “Swiss army knife” paradigm.”
BackgroundFSN: Joint Entity and Relation Extraction Based on Filter Separator Network
Dai, Q., Yang, W., Wei, F., He, L. J., & Liao, Y. (2024) · Entropy · doi:10.3390/e26020162 · Cited by 4
Identifying Influential Citations in Scientific Papers
James, M., Palakkat, V., & Jones, G. J. F. (2023) · Irish Conference on Artificial Intelligence and Cognitive Science (AICS) · doi:10.1109/aics60730.2023.10470500 · Cited by 5
“We employ four trans-former models, namely BERT [10], SciBERT [6], BART [5], and XLNet [4], to determine citation polarity.”
MethodWang, M., Vijayaraghavan, A., Beck, T., & Posma, J. M. (2024) · Journal of Proteome Research · doi:10.1021/acs.jproteome.3c00367 · Cited by 4
Gendrin, A., Harrison, E., Noveras, J., Souliotis, L., Vince, H., Smit, I. A., Costa, F., Milward, D., Dimitrievska, S., Metcalfe, P., et al. (2024) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2024.100152 · Cited by 4
Yang, S., Liu, J., Jin, F., & Lu, Y. (2024) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/tb-2024-0404 · Cited by 5
Win-Win Cooperation: Bundling Sequence and Span Models for Named Entity Recognition
Ji, B., Liu, H., Li, S., Ma, J., & Yu, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3423838 · Cited by 4
“Specifically, we use the bert-large-cased [65] as the BERTlarge, the biobert-pubmed-pmc [66] as the BioBERT, and the scibert_scivocab_cased [31] as the SciBERT.”
Hao, A., Yuan, H., Hui, S. C., & Su, J. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05851-1 · Cited by 4
“The pre-trained BERT models include the cased and uncased versions of SciBERT [67], BioBERT [12], and PubMedBERT [21]: • SciBERT—”
MethodCite-worthiness Detection on Social Media: A Preliminary Study
Hafid, S., Ammar, W., Bringay, S., & Todorov, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65794-8_2 · Cited by 4
Stampe, L., Pohl, J., Britta, G., & Grimme, C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650595 · Cited by 5
“…various families of underlying models, such as: • LASER (Facebook, [21]) • WordEmbeddings (e.g. GloVe by Stanford, [22]) • GPT (OpenAI, [23]) • MiniLM (Microsoft, [24]) • MPNET ([25], also based on BERT) • T5 (Google, [26]) • BERT (Google, [20]), also RoBERTa (Facebook AI, [27]) or SciBERT, [28])”
Zhu, Z., Cheng, X., Zhang, Y., Chen, Z., Long, Q., Li, H., Huang, Z., Wu, X., & Zheng, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72384-1_29 · Cited by 4
Predicting Scientific Impact Through Diffusion, Conformity, and Contribution Disentanglement
Xue, Z., He, G., Jiang, Z., Gu, S., Kang, Y., Zhao, S. X., & Lu, W. P. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679546 · Cited by 7
“SciBERT [3] is fine-tuned to encode the title and abstract of the paper.”
A deep neural network model for classifying pharmacy practice publications into research domains
Adeosun, S. O., Faibille, A. B., Qadir, A. N., Mutwol, J. T., & McMannen, T. (2024) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2024.10.009 · Cited by 4
TC-BERT: large-scale language model for Korean technology commercialization documents
Kim, T., Oh, C., Hwang, H., Lee, E., Kim, Y., Choi, Y., Kim, S.-J., Choi, H., & Song, K. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06597-6 · Cited by 4
“It promotes research on the development of large-scale LMs specialized in specific domains (e.g., medical [9, 25], biomedical [7, 8], education [5, 10, 11], scientific [26, 27], and financial [6, 12]) has been actively advanced.”
Zhang, H., Lin, G., Chen, K., Lin, N., Cheng, L., & Yang, A. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112570 · Cited by 4
Unlocking wisdom: enhancing biomedical question answering with domain knowledge
Azad, B., Alavi, M. A. A., Jafarzadeh, P., Ensan, F., & Androutsos, D. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02338-0 · Cited by 4
“These models are either fine-tuned on biomedical data [8–10], or are integrated into a transfer learning framework.”
Bibliographic network enhanced local citation recommendation
Jiang, T., Xu, Z., Wu, C., & Zhao, D. (2025) · The Electronic Library · doi:10.1108/el-08-2024-0251 · Cited by 4
Sarwar, T., Yepes, A. J., & Cavedon, L. (2025) · Journal of Medical Internet Research · doi:10.2196/73325 · Cited by 4
“Recognizing this limitation led to the development of domain-specific BERT (bidirectional encoder representations from transformers) models, such as BioBERT [19], ClinicalBERT [20], PubMedBERT [21], BioMed-RoBERTa [22], SciBERT [23], MedCPT [24], and GatorTron [25].”
Exploiting pivot words to classify and summarize discourse facets of scientific papers
Quatra, M. L., Cagliero, L., & Baralis, E. (2020) · Scientometrics · doi:10.1007/s11192-020-03532-3 · Cited by 4
Towards Aspect Extraction and Classification for Opinion Mining with Deep Sequence Networks
Kersting, J., & Geierhos, M. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-63787-3_6 · Cited by 5
Attending to Long-Distance Document Context for Sequence Labeling
Jörke, M., Gillick, J., Sims, M., & Bamman, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.330 · Cited by 4
Umapathy, A., Radhakrishnan, K., Jain, K., & Singh, R. K. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.34 · Cited by 4
IDSOU at WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets
Ohashi, S., Kajiwara, T., Chu, C., Takemura, N., Nakashima, Y., & Nagahara, H. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.62 · Cited by 7
Frugal neural reranking: evaluation on the Covid-19 literature
de Almeida, T. D., & Matos, S. (2020) · Workshop on NLP for COVID-19 (NLP-COVID19) · doi:10.18653/v1/2020.nlpcovid19-2.3 · Cited by 4
Nicholson, J. M. (2021) · Matter · doi:10.1016/j.matt.2021.02.007 · Cited by 4
Protocol for a reproducible experimental survey on biomedical sentence similarity
Lara-Clares, A., Lastra-Díaz, J. J., & García-Serrano, A. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248663 · Cited by 4
BreakingBERT@IITK at SemEval-2021 Task 9: Statement Verification and Evidence Finding with Tables
Jindal, A., Gupta, A., Srivastava, J., Menghwani, P., Malik, V., Kaushik, V., & Modi, A. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.40 · Cited by 4
Huang, C.-Y., & Huang, T.-H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.215 · Cited by 5
CLaC-BP at SemEval-2021 Task 8: SciBERT Plus Rules for MeasEval
Therien, B., Bagherzadeh, P., & Bergler, S. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.49 · Cited by 4
Learning Bill Similarity with Annotated and Augmented Corpora of Bills
Kim, J., Griggs, E., Kim, I. S. L., & Oh, A. H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.787 · Cited by 4
“10050 different BERT models pretrained for various domains and tasks (Chalkidis et al., 2020; Beltagy et al., 2019; Lee et al., 2019; Feng et al., 2020).”
Taju, S. W., Shah, S. M. A., & Ou, Y. (2021) · Analytical Biochemistry · doi:10.1016/j.ab.2021.114416 · Cited by 4
Navigating the Kaleidoscope of COVID-19 Misinformation Using Deep Learning
Chen, Y., & Hasan, M. R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.485 · Cited by 4
Lentschat, M., Buche, P., Dibie-Barthélemy, J., & Mathieu Roche, A. (2021) · International Journal of Intelligent Information and Database Systems · doi:10.1504/ijiids.2021.10042229 · Cited by 4
Interpretable Aspect-Aware Capsule Network for Peer Review Based Citation Count Prediction
Li, S., Li, Y., Zhao, W. X., Ding, B., & Wen, J.-R. (2021) · ACM Transactions on Information Systems · doi:10.1145/3466640 · Cited by 7
“Comparing the two BERT-based variants, it can be observed that SciBERT performs better than BERT, since it has been specially pretrained on scholarly papers.”
MethodA Multi-task based Bilateral-Branch Network for Imbalanced Citation Intent Classification
Hu, T., Li, J., Fukumoto, F., & Zhou, R. (2022) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom53663.2022.9721746 · Cited by 4
Why Did You Not Compare with That? Identifying Papers for Use as Baselines
Bedi, M., Pandey, T., Bhatia, S., & Chakraborty, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_4 · Cited by 9
Nuntachit, N., & Sugunnasil, P. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4030030 · Cited by 4
“SciBERT, which uses SCIVOCAB, was trained from scratch and used a single TPU v3 with eight cores for one week [18].”
MethodInfluentialNamed Entity Recognition for Cancer Immunology Research Using Distant Supervision
Trieu, H.-L., Miwa, M., & Ananiadou, S. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.17 · Cited by 6
Enriching Contextualized Representations with Biomedical Ontologies: Extending KnowBert to UMLS
Piat, G., Semmar, N., Allauzen, A., Essafi, H., & Tourille, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-10464-0_52 · Cited by 4
Ambavi, H., Vaishnaw, K., Vyas, U., Tiwari, A., & Singh, M. P. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3417428 · Cited by 4
Improving Accessibility to Arabic ETDs Using Automatic Classification
Abdelrahman, E., & Fox, E. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16802-4_18 · Cited by 4
Tanwar, A., Zhang, J., Ive, J., Gupta, V., & Guo, Y. (2022) · Studies in computational intelligence · doi:10.1007/978-3-031-14771-5_2 · Cited by 4
Language Models for Deep Learning Programming: A Case Study with Keras
Shorten, C., & Khoshgoftaar, T. M. (2022) · Advances in intelligent systems and computing · doi:10.1007/978-981-19-6153-3_6 · Cited by 7
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.