SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 2,701–2,800 of 3,596
Publication recommendation in incomplete networks based on graph learning
Liu, J., & Zhang, J. (2025) · Scientometrics · doi:10.1007/s11192-024-05219-5 · Cited by 1
“Research also indicates that due to its fine-tuning in scientific and technical realms, SciBERT demonstrates a superior understanding of domain-specific contexts and concepts, leading to improved performance in related tasks.”
InfluentialLingBERT, Linguistic Knowledge Injection into Attention Mechanism Based on a Hybrid Masking Strategy
Mechouma, T., Biskri, I., & Robert, S. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00253 · Cited by 1
“Although, TinyBERT, DistilBERT and SciBERT retain the same static masking strategy used in BERT [4] [7] [17] .”
Isaac, L. D., Arunkumar, S., Sundaramurthi, V., & Bommannan, G. (2025) · AIP conference proceedings · doi:10.1063/5.0262896 · Cited by 1
A century of knowledge growth in sedimentology
Xu, Y., Hu, X., Garzanti, E., Qi, J., Kang, H., Nong, H., Wu, L., Ding, J., Wang, X., & Wang, C. (2025) · Gondwana Research · doi:10.1016/j.gr.2025.04.017 · Cited by 1
Chatrinan, K., Noraset, T., & Tuarob, S. (2025) · IEEE Access · doi:10.1109/access.2025.3586729 · Cited by 1
Wei, Y., & Fan, J. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01236-0 · Cited by 1
“Most of the methods used in existing studies can be categorized into two types: one is secondary training of LLMs [90, 91], and the other is designing prompt words to achieve data extraction.”
Mallia, B. (2025) · Artificial intelligence · doi:10.5772/intechopen.1010889 · Cited by 1
A two-stage model for unified sentence- and document-level biomedical event extraction
Su, F., Zhang, Y., Jiao, P., Zhao, Z., Li, B., Li, F., & Ji, D. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112001 · Cited by 1
Text Mining for Process–Structure–Properties Relationships in Metals
Verma, A. K., Zhang, Z., Seo, J., Kuo, R., Jiang, R., Strubell, E., & Rollett, A. D. (2025) · Integrating materials and manufacturing innovation · doi:10.1007/s40192-025-00420-7 · Cited by 2
“As a result, domain-specific variants of pre-trained language models—such as SciBERT [17], BioBERT [18], and MatBERT [19]—have been developed to capture context-specific concepts and entities.”
Enhancing Scientific Knowledge Graph Generation Pipelines with LLMs and Human-in-the-Loop
Tsaneva, S., Dessì, D., Osborne, F., & Sabou, M. (2024) · Open Research Online (The Open University) · Cited by 6
Evaluating neural multi-field document representations for patent classification
Pujari, S. C., Mantiuk, F., Giereth, M., Strötgen, J., & Friedrich, A. (2023) · OPUS (Augsburg University) · Cited by 3
Predicting the presence of inline citations in academic text using binary classification
Vajdečka, P., Callegari, E., Xhura, D., & Ásmundsson, A. S. (2023) · DSpace repository (University of Tartu) · Cited by 4
Kai, H., & Yaoting, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116229 · Cited by 1
Extracting T Cell Function and Differentiation Characteristics from the Biomedical LiteraturePreprint
Czech, E., & Hammerbacher, J. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/643767 · Cited by 1
Zhang, T., Wang, C., Qiu, M., Yang, B., He, X., & Huang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10327 · Cited by 1
Gender prediction using limited Twitter DataPreprint
Burghoorn, M., de Boer, M., & Raaijmakers, S. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02005 · Cited by 1
Akama‐Garren, E. H., Anand, P., Cristea‐Platon, T., Lenehan, P. J., Lindemer, E., Bade, S., Liukasemsarn, S., O’Horo, J. C., Hurt, R. T., Williams, A. W., et al. (2021) · medRxiv · doi:10.1101/2021.01.03.20248997 · Cited by 2
Hierarchical Transformer Networks for Longitudinal Clinical Document Classification.Preprint
Si, Y., & Roberts, K. E. (2021) · arXiv (Cornell University) · Cited by 1
Interpretable Self-supervised Multi-task Learning for COVID-19 Information Retrieval and ExtractionPreprint
Ebadi, N., & Najafirad, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08252 · Cited by 1
“SPECTER is initialized with SciBERT [18] and trained on a bipartite graph of citations to capture document-level relatedness and minimize a triplet loss between related articles and maximize over unrelated ones.”
MOOCRep: A Unified Pre-trained Embedding of MOOC EntitiesPreprint
Pandey, S., & Srivastava, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05154 · Cited by 1
Multi-Round Parsing-based Multiword Rules for Scientific OpenIEPreprint
Kuebler, J., Tong, L., & Jiang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02074 · Cited by 1
Ranking Scientific Papers Using Preference Learning.Preprint
Dycke, N., Simpson, E., Kuznetsov, I., & Gurevych, I. (2021) · arXiv (Cornell University) · Cited by 1
Attention-based Approaches for Text Analytics in Social Media and Automatic Summarization
Barba, J. Á. G. (2021) · RiuNet (Universitat Politècnica de València) · doi:10.4995/thesis/10251/172245 · Cited by 4
(2021) · PeerJ Computer Science · doi:10.7287/peerj-cs.443v0.1/reviews/2 · Cited by 1
Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems ResearchPreprint
Gruetzemacher, R., & Paradice, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3944942 · Cited by 2
Contextualization and Generalization in Entity and Relation ExtractionPreprint
Taillé, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.07558 · Cited by 3
An Efficient ‘Paper Mining’ System to Search Academic Papers Using SPECTER ModelPreprint
Yamazaki, Y., Suzuki, T., Kumar, A., Ayada, E. S., Reserva, R. L., Imai, M., Miyashiro, D., & Umemura, K. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4191461 · Cited by 1
Similarity between Units of Natural Language: The Transition from Coarse to Fine EstimationPreprint
Mu, W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.14275 · Cited by 1
Exploring the opportunities and limitations of large language models in biomedical sectorPreprint
Albarqawi, A. (2022) · Research Square · doi:10.21203/rs.3.rs-2401418/v1 · Cited by 1
Tohalino, J. A. V. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.55.2023.tde-11042023-090221 · Cited by 1
Galiani, S., Gálvez, R. H., & Nachman, I. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4469037 · Cited by 1
Understanding Telecom Language Through Large Language ModelsPreprint
Bariah, L., Zou, H., Zhao, Q., Mouhouche, B., Bader, F., & Debbah, M. A. (2023) · doi:10.36227/techrxiv.23501271.v1 · Cited by 1
Aspect-based Document Similarity for Literature Recommender Systems
Ostendorff, M. (2023) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-9937 · Cited by 1
Custom Named Construct Recognition in the Business and Management Literature
Bhattacharya, A. (2023) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2023-15467 · Cited by 1
Ghosh, M., Ganguly, D., Basuchowdhuri, P., & Naskar, S. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.03401 · Cited by 1
“The baselines we have chosen are well-established and widely used in this field, providing a solid point of reference for our experiments (Beltagy et al., 2019; Schweter and Akbik, 2020; Jain et al., 2020).”
BackgroundMethodInfluentialAbstractive Summarization of Scientific Documents: Models and Evaluation Techniques
Rehman, T., Chattopadhyay, S., & Sanyal, D. K. (2023) · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE) · doi:10.1145/3632754.3632771 · Cited by 1
“In the third work [13], we passed the source document through SciBERT [2] and input the resulting token embeddings into the coverage-augmented pointer-generator network.”
MethodExtraction and normalization of simple and structured entities in medical documents
Wajsbürt, P. (2021) · theses.fr (ABES) · doi:10.70675/2f777bcaz72aaz400ezb9f4za52428477079 · Cited by 18
Tahri, C. (2023) · theses.fr (ABES) · doi:10.70675/de7f6f4cz266dz4745zab2fz833e8155f8b3 · Cited by 2
On the data scarcity problem of neural-based named entity recognition
Zhou, R. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/173481 · Cited by 1
Bansal, N., C, S. D. R., Pathak, A., & Narayanan, M. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.17.613424 · Cited by 2
“Other models include SciBERT, [17], BlueBERT [18] and PubMedBERT [19], which were trained on a large corpus from PubMed, and Knowledge-based BERT (K-BERT) [20], limited to NER and QA.”
MethodYang, L., Zhou, X., Fan, J., Xie, X., & Zhu, S. (2024) · Qeios · doi:10.32388/ev7ddp · Cited by 1
NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical TextPreprint
Kailas, P., Homilius, M., Deo, R. C., & MacRae, C. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.11477 · Cited by 1
“No-table MLM models in the biomedical domain include SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2019), ClinicalBERT (Alsentzer et al., 2019), and Bi-oLM (Lewis et al., 2020), which are trained on the semantic scholar , scientific abstracts from PubMed and PMC or the MIMIC-III dataset…”
MethodA Comprehensive Survey on the Integration of Reinforcement Learning and NLP for Stock Market TradingPreprint
Ferrell, B., & McInnes, B. T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5135573 · Cited by 1
The BERT Model Family and Encoder-Only Transformers
Ciapparelli, M., & de Varda, A. G. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00507-x · Cited by 1
Hechtner, F., Schmidt, L., Seebeck, A., & Weiß, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5146523 · Cited by 4
Klein, A. Z., Weissenbacher, D., O’Connor, K., Elyaderani, A., Amaro, I. F., Onishi, T., Golder, S., Spiegel, K., Scotch, M., & Gonzalez‐Hernandez, G. (2025) · medRxiv · doi:10.1101/2025.04.25.25326298 · Cited by 7
“…in general domains (DistilBERT-Base [18], BERT-Base [15], and RoBERTa-Large [19]), biomedical domains (BioBERT [20], BiomedBERT-Large-Abstract [21], and BiomedBERT-Base-Abstract-Fulltext [22]), and, more specifically, COVID-19 domains (COVID-SciBERT [23] and COVID-Twitter-BERT [24]) (Figure 1).”
BioMedTools: a language model-powered community for biomedical computational toolsPreprint
Liu, S., Xing, H., Han, M., Zhang, D., Gong, L., Liu, D., Chen, J., Cai, P., & Hu, Q. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.02.651919 · Cited by 2
EXTRACTING KEYPHRASES FROM SCIENTIFIC CONTENTPreprint
Thota, V. N. T. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5259431 · Cited by 1
Partisan Bias and the Resilience of High-Quality SciencePreprint
Choi, D. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ujyec_v5 · Cited by 1
Yasrab, R., Agrawal, R., Saber-Ayad, M. M., & El-Hadidi, M. S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.26.661832 · Cited by 1
An Artificial Intelligence Driven Semantic Similarity-Based Pipeline for Rapid LiteraturePreprint
Dhakal, A., Paudel, K., & Sigdel, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.15292 · Cited by 2
“Prior studies have shown that such models often underperform in specialized domains, as they may not capture domain-specific terminology, citation context, or methodological nuances with the same fidelity as models trained on in-domain corpora [20, 21, 22].”
Consumer Medical Question Answering: Challenges and Approaches
Nguyen, V. (2024) · ANU Open Research (Australian National University) · doi:10.25911/zj6y-ab57 · Cited by 1
Improving Access to ETD Elements Through Chapter Categorization and Summarization
Banerjee, B. (2024) · VTechWorks (Virginia Tech) · Cited by 2
Lin, G., van Eck, N. J., Haiyan, H., & Hu, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.04190 · Cited by 1
“To compute this, we used SciBERT (Beltagy et al., 2019), a BERT-based language model trained on scientific text (Devlin et al., 2019), to transform the titles and abstracts of both the HCPs and citing papers into embedding vectors.”
Durán, T. G., Far, A. J., Diaz, A., Barroso, M., Roldán, A., Colinas, N., & Cancellario, T. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.24.696373 · Cited by 2
“Pre-trained Pre-trained transformers acquire linguistic representations from extensive corpora that typically include full-text documents, enabling generalisation beyond the specific text types used during fine-tuning (Beltagy et al.”
PERELMAN: Pipeline for scientific literature meta-analysis. Technical reportPreprint
Sherki, D., Merkulov, D., Savina, A., & Muravleva, E. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.21727 · Cited by 1
“…de Schoot et al. , 2021 ] , scientific document infrastructure [ Lo et al. , 2020; Lopez, 2009; Zhong et al. , 2019 ] , text-focused scientific IE [ Beltagy et al. , 2019; Luan et al. , 2018; Cheung et al. , 2024 ] , and chart benchmarks that expose persistent numerical-reasoning gaps in VLMs…”
Corpas, M., Freidin, M. B., Valdivia-Silva, J., Baker, S., Fatumo, S., & Guio, H. (2026) · medRxiv · doi:10.64898/2026.02.10.26346004 · Cited by 1
Predicting New Concept-Object Associations in Astronomy by Mining the LiteraturePreprint
Li, J., Ting, Y.-S., Accomazzi, A., Ghosal, T., & Ramachandra, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.14335 · Cited by 2
“…representation for scientific corpora, enabling scalable pipelines for entity recognition, normalization, and relation mining (Devlin et al., 2019; Beltagy et al., 2019; Brown et al., 2020), and have been used directly for knowledge graph completion and inductive link prediction (Yao et al.,…”
UniFAR: A Unified Facet-Aware Retrieval Framework for Scientific DocumentsPreprint
Dou, Z., Zhang, Z., Wang, D., Ban, Y., & Zhuang, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.23766 · Cited by 1
Thyroid Cancer Risk Prediction from Multimodal Datasets Using Large Language ModelPreprint
Ray, P. (2026) · medRxiv · doi:10.64898/2026.03.05.26347766 · Cited by 1
Mercadier, Y. (2020) · theses.fr (ABES) · doi:10.70675/ac846a22za984z4dcezb563z95ea4094b2a8 · Cited by 11
Argument Mining on Clinical Trials
Mayer, T. (2020) · theses.fr (ABES) · doi:10.70675/8c2cf7b5z08cbz40e4za421z0115aa4a8eaa · Cited by 3
Espejel, J. L. (2021) · theses.fr (ABES) · doi:10.70675/31684827z4a48z48dfz8dd7zc76a5467c3e8 · Cited by 3
Epidemic Event Extraction in Multilingual and Low-resource Settings
Mutuvi, S. (2022) · theses.fr (ABES) · doi:10.70675/82e91446zf499z4438zaaa7z66548ffb98b0 · Cited by 3
Leveraging linguistic and semantic information for relation extraction from domain-specific texts
Tang, A. (2023) · theses.fr (ABES) · doi:10.70675/5e2de3fdz3bb1z4e21z877ez380f3d91a22a · Cited by 3
RWGBench: Evaluating Scholarly Positioning in Related Work GenerationPreprint
Xie, A., Su, W., Mao, J., Liu, Y., Zhang, M., Ma, S., & Ai, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.24894 · Cited by 1
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Purkayastha, S., Li, Z., Lauscher, A., Qu, L., & Gurevych, I. (2025) · arXiv · doi:10.48550/arxiv.2504.11042 · Cited by 1
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
“However, we employed SPECTER2 (Singh et al., 2022) and SciB-ERT (Beltagy et al., 2019) embeddings, designed for scientific texts and have shown superior performance in domain-specific applications.”
Claim Extraction in Biomedical Publications using Deep Discourse Model and Transfer LearningPreprint
Achakulvisut, T., Bhagavatula, C., Acuna, D., & Kording, K. P. (2019) · arXiv (Cornell University) · doi:10.4230/tgdk.3.3.4 · Cited by 40
“…use of language in a great number of research areas and consumer-oriented applications, as for example in the analysis of biomedical literature (Beltagy et al., 2019), the generation of EEG reports (Biswal et al., 2019), the development of more advanced chatbots (Budzianowski and Vuli´c, 2019)…”
BackgroundMed-KGMA: A novel AI-driven medical support system leveraging knowledge graphs and medical advisors
Varshney, S., Jain, B., Singh, P., Rani, D., & Mehra, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110929
From citation intent to knowledge contribution: Classifying what cited papers actually contribute
Quan, Z., Liang, Z., Ma, M., Wei, J., Li, G., & Mao, J. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70127
“Our Dual-Path Fusion model uses SciBERT as the encoding backbone and models the citing context and document-level metadata in parallel.”
InfluentialOnco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology
Gupta, S. K., Basu, A., Taylor, B., Kothari, A., & Singh, H. (2024) · arXiv · doi:10.48550/arxiv.2404.06680
“Even within the clinical data domain, though there has been progress in developing embedding models for tasks like named entity recognition and information extraction, the work on retrieval is limited (Gu et al. (2020); Beltagy et al. (2019); Goldstein and Shahar (2016); Kanwal and Rizzo (2022)).”
MethodConflict and Cooperation: AI Research and Development in terms of the Economy of ConventionsPreprint
Solans, D., Tauchmann, C., Farrell, A., Kappler, K. E., Huber, H.-H., Castillo, C., & Kersting, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12591
Cascade Neural Ensemble for Identifying Scientifically Sound ArticlesPreprint
Ambalavanan, A. K., & Devarakonda, M. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.06222
Verb Focused Answering from CORD-19
George, E. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58323-1_22
Predictions For Pre-training Language ModelsPreprint
Guo, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09031
A Brief Review of Relation Extraction Based on Pre-Trained Language Models
Xu, T., & Zhang, F. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200755 · Cited by 2
AutoNLU: An On-demand Cloud-based Natural Language Understanding System for Enterprises
Le, N., Lai, T., Bui, T., & Kim, D. S. (2020) · doi:10.18653/v1/2020.aacl-demo.2
Discovering Key Topics From Short, Real-World Medical Inquiries via Natural Language Processing
Ziletti, A., Berns, C., Treichel, O., Weber, T., Liang, J. J., Kammerath, S., Schwaerzler, M., Virayah, J., Ruau, D., Ma, X., et al. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.672867
AutoNLU: An On-demand Cloud-based Natural Language Understanding System for EnterprisesPreprint
Le, N., Lai, T., Bui, T., & Kim, D. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.13470
“, 2018) and SciBERT (Beltagy et al., 2019).”
MethodParmar, M., Ambalavanan, A. K., Guan, H., Banerjee, R., Pabla, J., & Devarakonda, M. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02017
“In this scorer, we used a SciBERT model that was previously fine-tuned on the Clinical Hedges dataset to identify articles describing treatments.”
MethodInfluentialConstructing a Relevance-oriented Dataset for Training Transformer Rankers for Medical Search
Zheng, Z., & He, B. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313521
“Existing works mainly focus on pre-training transformer-based language models like BERT in specific domains, such as BioBERT [6] and SciBERT [7].”
BackgroundMetric-Type Identification for Multilevel Header Numerical Tables in Scientific Papers
Suadaa, L. H., Kamigaito, H., Okumura, M., & Takamura, H. (2021) · Journal of Natural Language Processing · doi:10.5715/jnlp.28.1247
Multilingual Sentiment Analysis
Přibáň, P. (2020) · Digital Library (University of West Bohemia)
Topological Analysis of Averaged Sentence Embeddings
Holmes, W. J. (2020) · Journal of Bioresource Management
An Explanatory Query-Based Framework for Exploring Academic ExpertisePreprint
Cocarascu, O., Mclean, A., French, P. M. W., & Toni, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13728
“Concretely, we experiment with: BERT [ Devlin et al. , 2019 ] , RoBERTa [ Liu et al. , 2019], ALBERT [Lan et al. , 2020], DistilBERT [Sanh et al. , 2019], and SciBERT [Beltagy et al. , 2019].”
Incorporating Visual Layout Structures for Scientific Text Classification.Preprint
Shen, Z., Lo, K., Wang, L. L., Kuehl, B., Weld, D. S., & Downey, D. (2021) · arXiv (Cornell University) · Cited by 3
Self-citationAIM in Genomic Basis of Medicine: Applications
Kamada, M., & Okuno, Y. (2021) · Springer eBooks · doi:10.1007/978-3-030-58080-3_264-1
Finding Pragmatic Differences Between Disciplines
Kezar, L., & Pujara, J. (2021) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2021.sdp-1.10
University of Glasgow Terrier Team and UFMG at the TREC 2020 Precision Medicine Track.
Ueda, A., Santos, R. L. T., Macdonald, C., & Ounis, I. (2020) · Text REtrieval Conference
Ionita, R., Corlatescu, D.-G., Dascălu, M., & McNamara, D. S. (2021) · International Conference on Control Systems and Computer Science · doi:10.1109/cscs52396.2021.00078
Improving Korean Emotion Classification via Colloquial-Adaptive Pretraining
Lee, J., Kim, D., Ro, Y., & Kang, P. (2021) · Journal of Korean Institute of Industrial Engineers · doi:10.7232/jkiie.2021.47.4.342
MDS_UNCC Question Answering System for Biomedical Data with Preliminary Error Analysis.
Gopalakrishnan, S., Padithala, S., Demirhan, H., & Zadrozny, W. (2021) · CLEF (Working Notes)
Generating Diverse Extended Summaries of Scientific Articles
Reddy, S. M., Agarwal, S., & Saha, S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533999
“IIITBH-IITP [35] used CNN based model with SciBERT [36] em-3 beddings.”
MethodSemantic Influence Score: Tracing Beautiful Minds Through Knowledge Diffusion and Derivative Works
Sridhar, P., Karanji, D., Sampatrao, G. S., Danda, S., & Saha, S. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-87101-7_11
Dirkson, A. R., Verberne, S., & Kraaij, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11308 · Cited by 3
“For the biomedical data sets, we additionally fine-tune two domain-specific BERT models BioBERT (base-cased) (Lee et al., 2019) and SciB-ERT (scivocab-cased) (Beltagy et al., 2019).”
MethodInfluentialVarghese, N., & Punithavalli, M. (2021) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca51532.2021.9544727
Alvarez, M. C. Z., Barreneche, J. G., & Aristizabal, J. K. N. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-86702-7_17
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.