SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 3,301–3,400 of 3,596
AI Application in Semiconductor Manufacturing: A Patent-driven Approach
Kim, D., & Zo, H. (2026) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2026.138
Mijatović, A., Ursić, L., Bralić, N., Bandić, R., Ćaćić, B., Buljan, I., & Marušić, A. (2026) · JMIR Medical Informatics · doi:10.2196/72657
“…of the Bidirectional Encoder Representations from Transformers (BERT) model specifically designed for scientific and biomedical text processing that is pretrained on a vast corpus of scientific literature consisting of 18% of papers from computer science and 82% from the biomedical field [30].”
SAFE: Probabilistic Framework to Characterize Honest Reviewers
Kwatra, D., & Bhatnagar, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-23241-0_22
A Hybrid Framework for Personalized Scientific Articles Recommendation with LLM-Driven Explanations
Elayadi, K., & Yanes, N. (2025) · IEEE Afro-Mediterranean Conference on Artificial Intelligence (AMCAI) · doi:10.1109/amcai66110.2025.11474352
Generative AI in Life Sciences: Concepts and Applications from Molecules to Medicine
Wang, Z., Sannyasi, A., & Jiang, J. (2026) · Springer eBooks · doi:10.1007/978-3-032-14222-1_4
Chen, X., Meng, X., Lv, S., Zhang, Y., & Chen, Y. J. (2026) · Scientometrics · doi:10.1007/s11192-026-05594-1
Bandaru, V. N. R., Jyothika, C., Gupta, J. C., Guthala, S., & Naidu, J. C. V. (2026) · International Journal of Engineering Research and Sustainable Technologies (IJERST) · doi:10.63458/ijerst.v4i1.152
A Hybrid Method for Low-Resource Named Entity Recognition
Duc, D. M. (2026) · Journal of Applied Data Sciences · doi:10.47738/jads.v7i2.1161
“Moreover, multilingual and domain-specific adaptations of these models have further extended the applicability of NER in diverse settings such as biomedical, legal, and e-commerce domains [5].”
Optimizing SciBERT for Efficient Question Answering Through Quantization
Bulla, T., Gani, P., Kinagi, S. M., Muttagi, V., Kulkarni, U., & Meena, S. M. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18282-1_13
Sonar, P. S., Majumder, S., Sriram, S., & Nivethitha, V. (2026) · International Conference on Signal Analysis for Smart Systems (SIGNASS) · doi:10.1109/signass67826.2026.11479902
“Performance: B-series better than regul Nice performance on dirty clinical text [12].”
MTMinePy: An Interactive Platform for Multilingual Text Mining and Academic Visualization
Ren, Y., Liu, X., Zhao, X., & Liang, Y. (2026) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait69154.2026.11477324
“The benefits of tailoring these models to specific contexts are evident; for instance, SciBERT [5], pretrained on scientific literature, significantly outperforms general-domain BERT on downstream scientific tasks, demonstrating the value of domain-adaptive pre-training.”
Chowdhury, R. R., Jain, N., Shinde, R., Jain, A. K., & Bisane, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-18480-1_44
ONCORAG: A Knowledge Graph-Augmented Rag Framework for Mechanism-Aware Oncology Recommendations
Su, L., Qi, J., Li, M., Ma, S., Qian, W., Li, X., Shao, M., & Zhang, L. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460327
Tao, D. M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.16329
“Prominent scientific paper embedding models include: • SPECTER Cohan et al. [2020]: Built on SciBERT Beltagy et al. [2019], SPECTER fine-tunes the encoder using a triplet loss on citation graphs.”
InfluentialKudlak, Z., & Sherman, J. (2026) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/15485129261440549
Zhou, S. (2026) · ResearchWorks at the University of Washington (University of Washington)
Full-text Russian-language search in a distributed repository of patent information
Коробкин, Д. М., Fomenkov, S. A., & Bobunov, A. (2026) · VESTNIK OF ASTRAKHAN STATE TECHNICAL UNIVERSITY SERIES MANAGEMENT COMPUTER SCIENCE AND INFORMATICS · doi:10.24143/2072-9502-2026-2-41-52
Sentiment-Guided Financial News Summarization Using Transformer Models
Janadri, S., Deepika, E. J., Amrutha., A., Vincent, R., R, N. S., & Kannadaguli, P. (2026) · Contemporary Computing Innovations Conference (CCIC) · doi:10.1109/ccic68129.2026.11486152
Financial Sentiment Analysis using Natural Language Processing and Machine Learning
Hossain, M. M., Alim, M. A., Akif, S. M. C., Hussain, M. S., & Hoq, E. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491339
“These models serve as classical baselines commonly used in sentiment analysis tasks due to their robustness and interpretability [4, 9]. ii) Embedding-Based Models We extracted dense vector representations using Word2Vec [5], Doc2Vec [6], SBERT [7], and FinBERT embeddings [8].”
Named entity recognition for climate change research
Poleksić, A., & Martinčić-Ipšić, S. (2026) · Climatic Change · doi:10.1007/s10584-026-04324-z
“This observation is largely consistent with the comparison between SciBERT and CliSciBERT (see Appendix H, Figure H8), except for the Person type.”
InfluentialGhojogh, B., & Ghodsi, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-10738-1_11
Fei, Z., & Chen, H. (2026) · International Conference on Electronics Technology and Artificial Intelligence (ETAI) · doi:10.1109/etai68332.2026.11485044
“Other notable models include SciBERT [11], pre-trained on broad scientific literature, and methods enhancing relationship understanding through improved pre-training objectives, such as the contrastive learning approach by Su et al. [12].”
Multi-Granularity Embedding Meets LLM: Improving Table Matching in Scientific Documents
Li, C., Ma, Z., Li, H., Liao, C., Chen, J., & Wang, J. (2026) · International Conference on Electronics Technology and Artificial Intelligence (ETAI) · doi:10.1109/etai68332.2026.11485336
“Models such as SciBERT demonstrate that incorporating scientific text during pretraining can substantially improve performance on downstream scientific NLP tasks [2].”
SPARK: Self-Play with Asymmetric Reward from Knowledge GraphsPreprint
Park, H., Kim, T., & Choi, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.05546
ExperMatch: A Unified Benchmark for Bidirectional and Cross-Domain Expertise Matching
Chen, W., Chen, K., Qiu, Y.-X., Zhang, W., Lu, M., Chen, Q., Liu, J., Chan, W. K., & Mao, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0378-9_26
Yea, S.-J., Jang, H., & Kim, J. U. (2026) · Journal of Biomedical Semantics · doi:10.1186/s13326-026-00355-8
“To address this limitation, domain-adapted models such as SciBERT [53], BioBERT [16], and PubMedBERT [54] have been developed.”
Ensembles of Large Language Models for Identifying EQ-5D Studies in Pubmed Based on their Abstracts
Rostam, Z. R. K., Péntek, M., Czere, J. T., Zrubka, Z., Gulàcsi, L., & Kertész, G. (2026) · IEEE International Conference on Computational Cybernetics (ICCC) · doi:10.1109/iccc68994.2026.11511629
“• Assess the performance of nine Gemini and Gemma models using few-shot”
Yang, E., Han, J.-T., & Yoo, B.-S. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104796
Semantic Similarity Analysis of Theses Using Sentence Embeddings and Knowledge Graphs
Cusi, I. J. D. P., Mamani, L. C., Zuñiga-Rojas, G., Alvarez-Mamani, E., & Olivera, H. V. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-20322-9_7
Chen, N. (2026) · Research Square · doi:10.21203/rs.3.rs-9625547/v1
Lin, X., Sun, Y., Chen, L., Zhang, X., Dai, L., Chen, H., Xiang, H., & Yu, Z. (2026) · BMC Biology · doi:10.1186/s12915-026-02627-2
“Textual Encoder: The textual encoder utilizes SciBERT [23], a variant of BERT pre-trained on a large corpus of scientific publications.”
An Annotated Dataset for Sentence Classification from Educational Problem Sets
Benghachoua, W., Afifi, N., & Hilal, I. (2026) · Data · doi:10.3390/data11100280
PI-Embedding: Scientific Idea Representation Learning via Citation Intent and Paper Provenance
Wang, C., & Sun, Z. X. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519295
“Learning effective representations of scientific knowledge is vital for literature retrieval [8] [9].”
Optimized Feature Extraction and Alignment for Cross-Modal Molecule Retrieval
Qian, L., Li, M., Peng, N., & Long, T. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7141-3_30
Entity Knowledge Graph Enhanced Vision Language Model for Multi-Organ Diagnosis
Yu, J., Jiang, M., Chu, C. W., & Dou, Q. (2026) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi61048.2026.11515502
“To address the domain gap between medical terminology and general language, we employ the pre-trained SciBERT [23] as the text encoder to acquire entity embeddings.”
Local citation relationship classification for recommendation: a SciBERT-based DualFocusNet approach
Lu, Y., Yuan, M., & Chen, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05627-9
“It outputs a non-linear transformation of the input, effectively capturing complex patterns in data (Beltagy et al., 2019).”
InfluentialAcronym Disambiguation with Augmented and Optimized DistilBERT
Kamal, A., Siddiqui, A., & Bano, Z. (2026) · International Conference on AI Innovations and Industry (ICAIII) · doi:10.1109/icaiii69475.2026.11521654
“Domain-specific variants like SciBERT [6] further boosted performance, achieving 91 – 93% accuracy in biomedical datasets.”
Ma, Z., Deng, Y., Qiao, Z., Fu, Y., Yue, S., & Ren, J. (2026) · IEEE Transactions on Networking · doi:10.1109/ton.2026.3695944
Rostam, Z. R. K., Péntek, M., Czere, J. T., Zrubka, Z., Gulàcsi, L., & Kertész, G. (2026) · IEEE Access · doi:10.1109/access.2026.3697415
Retrieval-Augmented AI for Interpreting ICH Quality Guidelines and Product Control Strategies
Petrova, E., Georgiev, I., Stoyanov, N., & Kolev, P. (2026) · Pharmacophore · doi:10.51847/fgaycls0yd
The role of preprints in neuroscience scholarly communication: A citation analysis
Rasuli, B., Seyfzadehdarabad, F., Thibaut, A., & Gosseries, O. (2026) · Quantitative Science Studies · doi:10.1162/qss.a.490
“old wine in a new bottle”: Are different terminologies on dual processing conceptually the same?
Güss, C. D., & Powell, J. (2026) · Europe’s Journal of Psychology · doi:10.5964/ejop.18353
“…definitions is the sci-BERT-trained model known as “SCEPTER”, which has been pretrained on over 1 million scientific texts, primarily from the biomedical domain, which is no longer possi ble due to recent anti-scraping measures that most online journals have implemented (Beltagy et al., 2019).”
Compass: Navigating Global Marine Lead Data Integration through Expert-Guided LLM AgentPreprint
Liu, Y., Lu, B., Jin, M., Sang, Z., Jiang, S., Zhou, L., Wang, X., Zhou, C., & Zhang, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.29966
“ChemAgent [35] and SciAgents [14] leverage tool-use and ontological knowledge graphs for multi-agent reasoning, while other approaches explore self-updating mechanisms and automated environment generation [19, 30].”
Yang, L., Shi, Y., Han, F., & Cai, Y. (2026) · Biomedicines · doi:10.3390/biomedicines14061231
Morales, D., Gutierrez, A., Navarro, L., & Ríos, P. (2026) · Pharmacophore · doi:10.51847/rgdsedvjds
Tong, Q., Zhou, L., Liu, X., Liu, X., Wu, N., Wang, Y., Luo, H., Bin, Y., Tian, J., Han, D., et al. (2026) · aBIOTECH · doi:10.1016/j.abiote.2026.100059
“From these hits, three models, BioBERT [26], SciBERT [27], and XGBoost [28] were constructed to obtain a refined corpus of 11,068 publications, which were then converted into machine-readable text using GROBID (GeneRation Of BIbliographic Data) and CERMINE (Content ExtRactor and MINEr) (Fig.”
DLT-Corpus: A Large-Scale Text Collection for the Distributed Ledger Technology Domain
Cruz, W. H., Devine, P., Vadgama, N., Tasca, P., & Xu, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817553
“• Sentiment analysis dataset 6 : 23,301 cryptocurrency news headlines and brief descriptions with crowdsourced annotations from active community members, addressing the need for domain-specific labeled data.”
Chen, X., Ma, Y., Wu, K., Pang, X., Li, G., Ma, R., Yang, L., Peng, C., Zhi, J., & Yuan, J. (2026) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi15060243
Automatic generation of titles for research papers using language models
Rehman, T., Sanyal, D. K., & Chattopadhyay, S. (2026) · International Journal on Digital Libraries · doi:10.1007/s00799-026-00443-1
“We use the commonly used automatic text summarization evaluation metrics, including ROUGE [10], METEOR [11], MoverScore [12], BERTScore [13] and SciBERTScore [41], to assess the quality of the generated titles with reference to the author-crafted titles.”
AutoTail-BSFGM: Class-Balance-Aware Fine-Tuning for Chinese Scholarly Text ClassificationPreprint
Xiang, A., Yang, Y., & Shen, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.03576
“More recent systems increasingly combine metadata, abstracts, full text, and language models to characterize research aims, topics, and disciplinary relations (Wu et al., 2025; Beltagy et al., 2019; Lo et al., 2020; Singh et al., 2023; Zhang et al., 2024).”
Khan, M. S. R., & Park, H. W. (2026) · Annals of the International Communication Association · doi:10.1093/anncom/wlag018
Darold, D. L. (2026) · Research Square · doi:10.21203/rs.3.rs-9589637/v1
AI-powered pipeline for generating European project proposals
Guridi, P. H. (2026) · UPCommons institutional repository (Universitat Politècnica de Catalunya)
Álvarez, Y. D., Pinto-Elías, R., Magadán-Salazar, A., Castro-Sánchez, N. A., & Fuentes-Pacheco, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28393-1_25
Hierarchical Graph-Language Models for Sequential Sentence Classification
Nguyễn, P., Le, T. M. V., & McAuley, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1465-5_12
Types of Embeddings and Their Application in Intellectual Academic Genealogy
Marinosyan, A. K. (2026) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105526700184
“Transformer models—such as BERT [10], Sentence-BERT [11], SciBERT [12], and E5 [13]—generate dense vectors (ranging from hundreds to thousands of dimensions) that account for the context of each word in a sentence.”
PathoNER : State-of-the-Art Disease Extraction with a Domain-Task-Specific SLM
Ifrah, H., Mellah, Y., Chadli, S., & Saber, M. (2026) · International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · doi:10.1109/iraset68627.2026.11538539
“Other studies, including those conducted by SciBERT [7] and BlueBERT [8], indicate that combining clinical and scientific texts highlights the significance of domain-specific linguistic modeling for achieving high-precision entity extraction.”
Ding, Y., & Moreira, F. (2026) · Research Policy · doi:10.1016/j.respol.2026.105546
Feasibility of Zero-Shot Language Models on Answer Selection Task
Chuy, C. N., Ding, C., & Hu, Q. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3780028
Ivanitskii, A. D., & Lebedeva, N. A. (2026) · International Conference on Information Technologies in Engineering Education (Inforino) · doi:10.1109/inforino68543.2026.11556637
“• Proposal of relevant TRIZ principles, examples from other industries, patent analogies [10].”
An Efficient Model for Academic Paper Title Generation using Summarization Approach
Ali, A. A. (2026) · Iraqi Journal for Computers and Informatics · doi:10.25195/ijci.v52i1.701
“SciBERTScore is grounded on the scientific and technical scientific papers-trained SciBERT, and thus it is more suitable in assessing scholarly text in scientific fields that are related to computer science [33].”
Learning Selective Deferral Policies for Reliable Medical Text Classification
Albalawi, T., & Alzahrani, A. (2026) · Technologies · doi:10.3390/technologies14060359
“For comparative backbone experiments, the same sequence classification pipeline is also instantiated with BioBERT [5] and SciBERT [28].”
Arman, M., Kabir, M. F., Alamgir, F. M., Aklima, K., Kasedullah, M., Kabir, E., Akand, A. R., Imtihan, A. A., & Sánchez, M. Á. R. (2026) · Machine Learning · doi:10.1007/s10994-026-07095-x · Cited by 1
“Large pre-trained language models (Brown et al., 2020; Devlin et al., 2019), especially those fine-tuned on scientific text such as SciBERT (Beltagy et al., 2019) and BioBERT, demonstrate superior performance on scientific NER and RE (Krenn et al., 2022; Luan et al., 2018).”
InfluentialCross-lingual Text-to-Thangka Generator based on Diffusion Transformer
Wang, Y., Tsering, D., Wang, X. R., Yu, Y., Li, D., & Tashi, N. (2026) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit70033.2026.11557735
“Sub-sequent studies [29] [30] demonstrated that continued pre-training with MLM on domain-specific corpora significantly enhances model performance in that domain.”
Медианные значения метрик соответствия для классов SUPORT и CONTRADICT
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.168.94.4
Кузнецов, И. (2026) · Международный научно-исследовательский журнал · doi:10.60797/irj.2026.168.94
Liang, J., Fan, J., Zhang, G., & van Driel, W. D. (2026) · IEEE Electronic Components and Technology Conference (ECTC) · doi:10.1109/ectc51846.2026.00169
“…including GROBID for bibliographic and structural extraction from PDF documents [4], large-scale machine-readable corpora such as S2ORC containing over 80 million academic papers with structured metadata [5], and domain-adapted language models such as SciBERT trained on scientific publications [6].”
Основные этапы комплексной оценки соответствия научных утверждений информации из первоисточника
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.168.94.3
Современные методы повышения качества цитатно-осведомленной суммаризации
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.168.94.2
Privacy-Aware Retrieval-Augmented Generation for Intelligent IoMT Healthcare Systems
Yogi, M. K., & Donga, D. (2026) · Journal of Ubiquitous Computing and Communication Technologies · doi:10.36548/jucct.2026.3.002
“Recent studies have shown that domain-specific language models trained on large-scale scientific and biomedical literature achieve considerable performance gains for downstream healthcare natural language processing tasks [6], [12].”
Основные этапы цитатно-осведомленной суммаризации
Кузнецов, И. (2026) · Cifra LLC Journals · doi:10.60797/irj.2026.168.94.1
Fahim, A. M., & Rahman, M. R. (2026) · Discover Computing · doi:10.1007/s10791-026-10228-4
“Beltagy et al. [17] presented SciBERT, a version of BERT trained on scientific literature, demonstrating that domain adaptation can yield better outcomes for specialized NLP tasks, a concept that can be extended to clickbait detection.”
Liu, J.-H., Chen, P.-J., & Chen, J.-R. (2026) · Scientific Reports · doi:10.1038/s41598-026-59616-2
“Domain adaptation of embedding models typically follows one of three paradigms: (1) continued pre-training on domain-specific corpora 13 , (2) supervised fine-tuning on labeled query-document pairs 14 , or (3) hybrid approaches combining both strategies 15 .”
Constructing and Exploiting a Parallel Corpus for Transformer-Based Biomedical Text Simplification
Chaabéne, M., Camara, M., Ellouze, S., & Jaoua, M. (2026) · International Symposium of Systems, Advanced Technologies and Knowledge (ISSATK) · doi:10.1109/issatk69667.2026.11566879
“Empirical studies demonstrate that, when appropriately adapted, transformer models consistently outperform earlier neural architectures in terms of semantic preservation and readability [9], [10].”
Zhou, J., & Xue, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1947-6_5
Iyengar, A. R., Downs, E. P., Mehta, S., Rangarajan, N., Woo, M. S., Sredni, S. T., Donato, R. D., Simsek, S., Wilson, D. M., Krieser, K., et al. (2026) · The Breast Journal · doi:10.1155/tbj/7212120
“To identify aBC patients, a second NLP algorithm (“staging model”) [6–8] was used on clinical notes to assign cancer stages.”
A Survey on Multilingual Clinical Entity Linking in the LLM Age
Vassileva, S., Koychev, I. K., & Boytcheva, S. (2026) · Proceedings of the international neural networks society · doi:10.1007/978-3-032-30153-6_3
Qasim, H. F. (2026) · Informatica · doi:10.31449/inf.v50i13.14270
“SciBERT contains domain-specific vocabulary and is therefore well suited for processing technical documents [12].”
InfluentialLegal Domain Adaptation of Modern BERT ModelsPreprint
Stammbach, D., & Henderson, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.28538
“Moreover, BERT checkpoints have been adapted to many different domains, for example scientific text [3] or environmental text [30].”
Analyzing ESG Disclosure Patterns in Vietnamese Commercial Banks
Bui, M. P., & Ngoc, M. N. T. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-8392-7.ch009
Named Entity Recognition for Adverse Drug Event Detection in Electronic Medical Records Using XLNet
Dhiman, A., Singh, P., & Kanwar, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-22289-3_1
Sun, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-58880-6
Intelligent Analysis Framework for Academic Papers with Local LLM Integration
Xiaoye, H., & Wu, Y. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-30836-8_13
Context aware knowledge discovery for clinical support using transfer learning
Paliwal, G., & Bunglowala, A. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21765-x
“In a similar vein, Si et al.‘s [45] training of BERT-based TL models made use of clinical notes as corpora.”
CLEAR-X: Contrastive learning with explainable re-ranking for academic reviewer recommendation
Ran, Y., Li, X., Huang, J., Ai, M., Sun, F., Krebs, B., & Jia, T. (2026) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2026.118738
LLM-Enhanced Rapid Review (LLMEn-RR): a methodology proposal
Irace, M. B., Simões, J. L., Argôlo, M., Barbosa, C. E., Nóbrega, L., Martinez, L. F., & de Souza, J. M. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582469
“Utilizing transformer-based mechanisms, LLMs capture deep contextual, syntactic, and semantic patterns, enabling advanced capabilities such as text summarization, question answering, semantic search, classification, and content generation [17]–[19].”
MSKGCF: A Framework for Multimodal Scientific Knowledge Graph Construction and Fusion
Maoliniyazi, A., Ma, C., & Meng, X. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581529
“While pre-trained language models (PLMs) such as SciBERT [5] have advanced text-based extraction, they often struggle to align visual semantics with textual descriptions.”
Lee, Y., Hur, M., & Min, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05729-4
“This is particularly salient for scientific literature, which often contains domain-specific terminology, complex argumentative structures, and long-range dependencies across paragraphs (Beltagy et al., 2019).”
An Empirical Comparison of LoRA and DoRA for Parameter-Efficient Fine-Tuning for Climate Change
Ribarić, M., Poleksić, A., & Martinčić-Ipšić, S. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591981
“Fine-tuning uses a pretrained language neural model, such as SciBERT [10], as a foundation.”
InfluentialBhattacharjee, N., Chutia, A., & Nath, S. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.377
Kunal Achintya Reddy, S., Raaghavan, M. S., Bollena, N., Thara, S., & Veena, G. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.002
Siras, A. S., Nagamalle, N., & Aravinda Kumar, P. (2025) · IEEE India Geoscience and Remote Sensing Symposium (InGARSS) · doi:10.1109/ingarss67683.2025.11584034
A Comparative Study on Domain-Specific LLMs: Architecture and Performance
Haidura, N., Assaf, H., & Kobti, Z. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_47 · Cited by 1
INSPIRE: A System that Models Authorial Intent for Scholarly Paper Recommendation
Yano, M., Gohourou, D., Hananto, V. R., & Kryssanov, V. V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_50
MANILA26: SIGIR 2026 Tutorial on Information Retrieval for Climate Change Impact
de Rijke, M., & Salim, F. D. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808642
“While SPECTER [4] and SciBERT [1] serve as primary baselines, their performance in climate impact retrieval is hindered by a citation-proximity objective that fails to bridge the gap between disparate disciplines like hydrology and economics.”
Materials research analytics platform: Interactive visualization and analysis of MRS conference data
Agarwal, S., & Misra, S. (2026) · MRS Communications · doi:10.1557/s43579-026-01008-8
Low-Resource Topic Modeling for Domain Experts: A Pilot Study in CO2 Mineralization Literature
Sabatino, G., Rosa, G. D., Tripodo, A., & Bella, M. D. (2026) · Analytics · doi:10.3390/analytics5040045
Remelgado, H., Gomes, M., Silva, E. C. E., & Bicho, E. G. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30801-6_24
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.