SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 3,501–3,596 of 3,596
A Domain-Tuned Multimodal Agent for NWP Forecast IntelligencePreprint
Chalumuri, A., Singh, P., Gera, A., Routray, A., Devarajan, P. K., & Prasad, V. S. (2026) · Research Square · doi:10.21203/rs.3.rs-10248854/v1
GUENNICH, O., & Yousfi, A. (2026) · ECONOMIC COMPUTATION AND ECONOMIC CYBERNETICS STUDIES AND RESEARCH · doi:10.24818/18423264/60.3.26.15
Chanaa, H., Nfaoui, E. H., & Loqman, C. (2026) · Future Internet · doi:10.3390/fi18090497
Xu, A., Chang, B., Shu, K., Jian, L., & Song, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05807-7
“SciBERT is a pre-trained model specifically designed for scientific texts, trained on a large multi-domain corpus of scholarly publications (Beltagy et al., 2019).”
Hammad, R., Ottom, M. A., & Baker, A. B. (2026) · Journal of Computer Science · doi:10.3844/jcssp.2026.2692.2699
Samarek, R., & Martinek, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.26218
“Contextual language models of the BERT type [Devlin et al., 2019] gave rise to a branch of domain-specific models for science: SciBERT [Beltagy et al., 2019] and especially models trained on the citation signal— SPECTER [Co-han et al., 2020] uses triplets (citing–cited–non-cited) to learn…”
Hu, Z., Peng, C., Peng, Z., Cai, Z., Zhang, Z., Sun, Z., Hu, J., & Shouyu, L. (2026) · Mathematics · doi:10.3390/math14193479
Panagoulias, D. P., & Tsihrintzis, G. A. (2026) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa71656.2026.11705494
“In parallel, transformer-based scientific NLP systems, biomedical language models, retrieval-based models, and scientific knowledge graph methods can extract study characteristics, outcomes, confidence intervals, coefficients, and associations from text [32]–[37].”
Wang, Y., Zhang, J., Bai, H., Ren, Y., Yang, Y., Wang, J., & Lv, J. (2026) · International Journal of Bioprinting · doi:10.36922/ijb026350375
Madeira, C., Dopke, L., Losch, G., Magnaguagno, M. C., & Griebler, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39898-7_18
Ma, W., Nie, P., Yin, R., Li, L., & Wang, L. (2026) · Biosafety and Health · doi:10.1016/j.bsheal.2026.09.006
Xie, H., Yan, Z., Luo, D., Chen, L., Yu, K., Hui, J., Lookman, T., Rao, Z., Ren, Y., & Zeng, X. (2026) · Progress in Materials Science · doi:10.1016/j.pmatsci.2026.101849
DPBERT: A domain-specific pre-trained model for Chinese data policy texts and its interpretability
Zhang, T., Ce, Z., Hangong, W., Haiqun, M., & Lei, J. (2026) · Journal of Information Science · doi:10.1177/01655515261489505
From OCR to Multimodal Reasoning: A Comprehensive Study of Clinical Document Understanding
Deepak, A., Bhutada, V., Agrawal, R., Patil, P., & Chobe, S. (2026) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea69821.2026.11702339
“…Transformer models such as BioBERT and PubMedBERT have been pre-trained on large-scale biomedical datasets (like PubMed abstracts and PMC documents), showing superior performance in Named Entity Recognition (NER) and Relation Extraction tasks compared to general-purpose models [1], [10], [14].”
Yao, H., & Shu, Y. (2026) · Chinese Journal of Chemical Engineering · doi:10.1016/j.cjche.2026.09.010
Harwella, S., & Supangkat, S. H. (2026) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss71466.2026.11715090
Intelligent Reinforcement Learning-Based Weight Optimization for Enhanced Information Retrieval
Jose, J., & Lydia, D. M. (2026) · International Conference on Innovations in Engineering and Technology (ICIET) · doi:10.1109/iciet70550.2026.11712797
“Such strategies are especially useful in book recommendation systems, where different metadata fields contribute differently to relevance [4], [5], [17].”
Zakiri, H., Muhammad, A. H., & Nasiri, A. (2025) · Journal of Informatics Electrical and Electronics Engineering
Graph-based Knowledge Representation for Natural Language Understanding Relatrice:
Nolano, G. (2022) · ARCHIVIO ISTITUZIONALE DELLA RICERCA-UNIVERSITA DEGLI STUDI DI NAPOLI "LORIENTALE"
Ngetich, K. (2026) · International Journal of Computer Applications Technology and Research · doi:10.7753/ijcatr1510.1006
Jaramillo, K. M. P. (2026) · Repositorio Digital (Escuela Politécnica Nacional) · Cited by 1
A Preview of Computational Animal Linguistics
Dang, T. M., Desai, A., Lekhak, H., Panunto, T., Pike, L., Wang, S., & Zhu, K. Q. (2026) · ACM Computing Surveys · doi:10.1145/3850160 · Cited by 1
“We then apply a SciBERT relevance classiier [16].”
Mamai, K. A., Lazare, W., Amaria, S., Guidedi, K., & Kolyang (2026) · International Journal of Research and Innovation in Applied Science · doi:10.51584/ijrias.2026.11090035
Guan, C., & Cheng, J. Y. C. (2026) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case69030.2026.11704254
The influence of optical character recognition quality on the robustness of semantic encoding
Jiang, M. (2022) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 1
The silence of the labs: Ethical oversight in metabolomics research
Badia, J., Llambrich, M., Rubio, R. L., Saggion, H., & Cumeras, R. (2026) · PLoS ONE · doi:10.1371/journal.pone.0346284
“Domain-specific models like SciBERT are pretrained on scientific corpora, improving performance in scholarly text [33].”
A Comparative Evaluation of Topic Modeling Methods for Researcher-to-Research-Group Recommendation
Bratakusumah, D., Rumadi, & Ichwan, M. (2026) · International Conference on Computer, Control, Informatics and its Applications · doi:10.1109/ic3ina72033.2026.11714643
“Domain-adapted embeddings such as SciB-ERT [17] and the citation-informed SPECTER embeddings used here [18] better capture scientific-publication semantics than general-domain embeddings, motivating BERTopic’s SPECTER-based encoder.”
Classification of NOTAMs for General and Scenario-Specific Applications Using Fine-Tuned BERT
Puca, N., Niermann, C. A., & Guglieri, G. (2026) · Aerospace · doi:10.3390/aerospace13111023
Popello, V., Martinis, M. C., Lazzaro, I., Zucco, C., Milano, M., & Cannataro, M. (2026) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms69103.2026.00264
“Methods developed for scientific text mining and representation learning allow large collections of research articles to be processed and represented in machine-readable form, supporting large-scale exploration of research landscapes [9]–[11].”
Joireman, J. A., Lissa, C. J. V., Lange, P. A. M. V., Kleinberg, B., Norouzi, R., & Balliet, D. (2026) · Current Research in Ecological and Social Psychology · doi:10.1016/j.cresp.2026.100290
Comparative Analysis of Encoder-Based and Generative Models for Symptom-Based Disease Classification
Leong, O. A. P., Djoenarko, N. N., Putri, S. E., & Wiputra, R. (2026) · International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC) · doi:10.1109/assic70037.2026.11717348
Shi, C., Luo, R., Zhao, S., Geng, Q., & Wu, D. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01297-1
“…semantic similarity measurement computationally feasible, while domain-adapted models for scientific and technical text, including SciBERT [10], SPECTER [16], and more recent patent-oriented embedding models [8], suggest that semantic representations may be especially useful when…”
PaperMind AI - Intelligent Research Paper Analysis Platform
Thakare, A. N., Kadam, N. V., Ingle, P. M., Sable, P. H., & Mishra, K. A. (2026) · International Journal for Research Trends and Innovation · doi:10.56975/ijrti.v11i10.301147
EvidenceCell-DL: Counterfactual Evidence Gates for Metadata Repair in Digital Libraries
Constantin, D., & Bălcău, C. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846054
MOTEGI, K., Ito, K., & Matsubara, S. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846499
“We used BERT [7], RoBERTa [14], DeBERTa [9], ModernBERT [31], and SciBERT [4] as pretrained encoders.”
Extracting and Structuring Experimental Knowledge from Electronic Theses and Dissertations
Klair, H. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3847132
“A substantial literature extracts entities, relations, and procedures from scientific text, often with domain-adapted encoders [5, 9].”
Han, B. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3847292
PReFER: A Multi-Proposer Refiner Framework for Named Entity Recognition
Anjum, A., & Krestel, R. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846032
CITATION IMPACT PREDICTION USING MULTI CHANNEL FUSION FRAMEWORK
Al-Mayali, Z. Y. M., & Al-Augby, S. (2026) · Archives for Technical Sciences · doi:10.70102/afts.2026.1836.575
ChemSafe: Hybrid Retrieval-Augmented Decision Support for Laboratory Chemical Hazard Assessment
Elgedawy, R., Das, S. K., Seefried, E., Wiggins, G. M., Burchfield, R., Jeskie, K., Fiscor, S., Pramanik, S., Thomas, B., Schnell, C., et al. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846505
“Domain-adapted models such as ChemBERTa [1, 4] and SciBERT [2] have improved chemical text understanding, and automated SDS indexing [12] has demonstrated high-precision entity extraction.”
How much does the map of science depend on the embedding model?Preprint
Ortega, A. T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.16639
Huang, L., Ren, H., Cao, X., & Zhang, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05826-4
Martinis, M. C., Popello, V., Lazzaro, I., Zucco, C., & Cannataro, M. (2026) · AI · doi:10.3390/ai7100424
Bootstrapping Knowledge Acquisition from Domain Text with a Seed Ontology and Small Language Models
Tan, H., Yang, Y., Yuan, T., & Li, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41614-8_11
Metadata-Driven Scientific Hypothesis Generation via Multi-agent Symbolic–Neural Reasoning
Alotaibi, F., Kulkarni, A., & Zhou, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41086-3_11
Stede, M., Schneider, J., & Wachsmuth, H. (2026) · Synthesis lectures on human language technologies · doi:10.1007/978-3-032-32291-3_4
Li, X., Song, Y., Li, Y., Li, Y., Wu, K., Huang, Y., & Zhang, K. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00479-2
“As a fundamental task in medical information extraction, MRE supports a wide range of downstream applications, including medical knowledge graph construction, medical question answering and clinical decision support [2].”
Siswanto, E., & Danang, D. (2026) · Jurnal Teknik Informatika (Jutif) · doi:10.52436/1.jutif.2026.7.5.6468
Liu, Y., Liu, Z., Zuo, W., Yang, Z., Wu, J., Dai, T., & Shi, S. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728965
“For the downstream PURE evaluator, we follow a standard training configuration with SciBERT [6] (scibert_scivocab_uncased) as the backbone, using a learning rate of 1e−5 for the encoder and a higher task learning rate of 5e−4 for the task -specific layers, a train batch size of 16, and…”
BERTopic-Based Latent Hazards Mining and Topic Database Construction for Gas Pipeline Accidents
Shi, X., Zhang, C., Jiang, W., Jiao, W., & Wang, C. (2026) · Applied System Innovation · doi:10.3390/asi9100217
InvisiTunnel: Causality-Preserving Covert Communication Over DoH Through Request Substitution
Pan, P., Li, J., Zhang, J., Ma, X., Li, W., Zheng, J., Wang, W., Wang, J., & Guan, X. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3744608
Navigating the Imagination-Implementation Divide in Blockchain Adoption
Mäkineste, N., & Angelopoulos, S. (n.d.) · Journal of the Association for Information Systems
Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm WinsPreprint
Fang, Y., Srivastava, G., Ding, S., & Wang, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.24108
Uddin, H. M. W., Progoti, Q. R. M., Mondal, M. C., Malitha, S. B., Saif, T., & Chowdhury, A. M. S. (2026) · Biomedical Materials & Devices · doi:10.1007/s44174-026-00796-w
Vomo-Donfack, K. L., Aberbache, M., Cantez, A., Hoszu, A., GINOT, G., Doblas, V. G., & Morilla, I. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.10.19.760098
“…and scientific natural language processing, pretraining on in-domain text rather than fine-tuning a general model yields consistent and sometimes large gains [27, 28, 29], and in genomics a compact domain model has recently been argued to compete with much larger generalist alternatives [31].”
Ismukanova, A. N., Кощанова, Г., Плескачев, Д. В., Сарсенбаева, Ж., & Daurenova, A. (2026) · INTERNATIONAL JOURNAL OF INFORMATION AND COMMUNICATION TECHNOLOGIES · doi:10.54309/ijict.2026.27.3.016
Văruicu, M. E., Popescu, P. Ș., & Mihăescu, M. C. (2026) · IEEE International Conference on Intelligent Systems · doi:10.1109/is71416.2026.11727411
“The advent of pre-trained transformer models substantially advanced this area, with domain-specific variants such as SciBERT [12] setting new benchmarks on various scientific NLP tasks and demonstrating clear advantages over general-purpose models [13] across tasks including literature similarity…”
ResearchMatch: A Semantic-Bibliometric Prototype for Evidence-Supported Researcher Discovery
Emmanuel, D. S., & Emmanuel, I. (2026) · Journal of Information Systems and Informatics · doi:10.63158/journalisi.v8i5.1841
“SciBERT showed the value of transformer models trained on scientific text [4], while SPECTER demonstrated citation-informed document representations for scientific recommendation, classification and citation prediction [5].”
Han, S., Liu, D., Wang, X., & Jiang, Y. (2026) · Sustainability · doi:10.3390/su182110792
Cheimariotis, I., Costa, J. N., Piazza, F., Lodi, C., Rataj, M., Zawieska, J., Stępniak, M., & Grosso, M. (2026) · European Transport Research Review · doi:10.1186/s12544-026-00854-w
“Supervised taxonomy classification with BERT-family models applied to scholarly corpora extended this line of work to assigning research texts to predefined categories [9].”
Contextual Language Modeling for Domain-Specific Knowledge Generation
Patkar, P., BasiReddy, S. R., Kasimala, K., Pappula, K. K., Enjam, G. R., & Sadat, Q. T. (2026) · doi:10.1109/incect72370.2026.11742603
“To overcome these drawbacks, some domain-specific language models have been proposed, such as BioBERT [7], ClinicalBERT [8], SciBERT [9], FinBERT [10] and Legal-BERT [11].”
Fine-Tuning LLMs for Methodology Extraction from Scientific Papers
Dikilitaş, Y., & Sayar, A. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727939
“Likewise, SciB-ERT showed that models trained on scientific corpora produce more appropriate representations for scientific language processing [2].”
Hybrid retrieval–verification framework for hallucination reduction in medical LLMs
Reddy, K. S., & Samreen, S. (2026) · Scientific Reports · doi:10.1038/s41598-026-75909-y
The golden cage of science: first-last author prestige gap buys impact over disruptionPreprint
Zhang, C., Bi, Y., & Zhou, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.34020
“The SciBERT model [41] is fine-tuned using articles with explicit type labels from PubMed, supplemented by MAG records whose article types can be clearly identified.”
Vasani, V. P., Easwaramoorthy, S. V., Gupta, R. K., Vasani, V. K., & Pawar, S. (2026) · doi:10.1109/icsl-dsga70378.2026.11736652
“SciBERT [12] and PubMedBERT [13] further refined these results by pre-training from scratch on scientific corpora, thereby avoiding the domain mismatch inherent to general-purpose BERT checkpoints.”
Qin, W., Jie, R., Qian, Y., & Srifi, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.35098
“DML added adjustment for a hierarchical full-text SciBERT predictor of mean human ratings (Beltagy et al., 2019; Chernozhukov et al., 2018).”
AI-Driven Research Data Management
Silambarasan, G., & Shivaraj, G. (2026) · IGI Global eBooks · doi:10.4018/979-8-2600-1227-7.ch001
Peñaranda, J., Piñeros-Guerrero, N., Romero-Gélvez, J. I., & Piñeros-Castro, Y. (2026) · Polymers · doi:10.3390/polym18212686
“For the assisted full-text set, the transformer experiment was rerun as a benchmark across MatSciB-ERT, SciBERT, and BGE-small [44–46], using the checkpoints m3rg-iitd/matscibert, al-lenai/scibert_scivocab_uncased, and BAAI/bge-small-en-v1.”
Chokalingam, P., & Varadarajan, S. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5567-2_3
Sentiment Analysis for RRI Disease Prediction - A Comparative Review
Ganesh, S., & Varadarajan, S. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5567-2_4
AI-driven discovery and delivery systems for anti-Parkinsonian agents
Tiwari, P. C., Kartik, S., Rohilla, K., Chaudhary, M. J., Pal, R., & Nath, R. (2026) · Journal of drug targeting · doi:10.1080/1061186x.2026.2756474
Deep Learning Approaches for the RNA–RNA Interaction Prediction Problem
Harithanush, S., & Varadarajan, S. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5567-2_7
Анатольевич, Д. А., Галиев, А. Б., & Алексеевич, Б. О. (2026) · Economics and Management · doi:10.35854/1998-1627-2026-10-1338-1349
Sazlı, B., & Günay, M. (2026) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk71847.2026.11745416
“For end-to-end fine-tuning, distilbert-base-uncased and allenai/ scibert_scivocab_uncased were loaded with newly initialized 379-class output layers [13], [14].”
Тұрсынхан, А., Turdalyuly, M., Yerimbetova, A., Akmetov, B., & Sakenov, B. (2026) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk71847.2026.11745148
“Beltagy et al. introduced SciBERT, a transformer model pre-trained on scientific publications that significantly outperformed general-purpose BERT models on scientific NLP tasks [6].”
RAG-Enhanced ForenGemma-1B: A Systematic Evaluation of Retrieval Strategies Using DeepEval
Karakuş, S., Muhammad, A., Kaya, M., & Eriş, M. (2026) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk71847.2026.11745321
“Of note, BioBERT and SciBERT [3], [4] have obtained over 12% gains over general-purpose BERT for biomedical question answering and around 2 F1 points across scientific NLP benchmarks.”
Lavanya, P., Madhushri, K., Sheema, D., Aiswarya, S., Shanthamalar, J. J., & Jayant, P. G. (2026) · doi:10.1109/icosics70095.2026.11746789
“Domain-specific encoders are also relevant: SciBERT was pretrained on scientific publications (Beltagy et al., 2019), while biomedical language-model variants have been developed for biomedical text mining.”
billa, M. M. (2026) · Research Square · doi:10.21203/rs.3.rs-11264021/v1
A Lightweight Approach for Adaptation of Language Models for Cybersecurity Named Entity Recognition
Srivastava, S., Gupta, D., Paul, B., & Sahoo, S. (2026) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp6060188
Cognitive distance in scientific teams and innovative research outcomes
Yang, A. J., Peng, Z., Hu, H., & Deng, S. (2026) · Journal of Information Science · doi:10.1177/01655515261498142
Zhang, Y., & Zhang, C. (2026) · Scientometrics · doi:10.48550/arxiv.2610.41552
“The marked sequence is then encoded by SciBERT (Beltagy et al., 2019) to generate contextualized representations.”
Residual Orthogonal Projection for Multi-Modal Molecular Retrieval
Zhou, G., Wang, Z., Zhao, G., Cheng, Z., Ji, X., Cai, H., Wei, Z., & Gao, Z. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3841159
“Scientific and biomedical language models [2, 10, 17] provide pretrained representations for domain-specific text.”
Generative-Reflective Agent-Based Method for the Extraction of Chemical Information
Alshehri, A. S. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.09.482
Song, Y., Lu, X., Wan, L., & Zhang, Z. (2026) · Applied Sciences · doi:10.3390/app162110848
Wang, H., Chen, X., Du, B., & Guo, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3749353
Iterative Silver-Label Refinement for Biomedical Temporal Information Extraction
Lee, C., & Arifuzzaman, S. (2026) · International Conference on Machine Learning and Applications · doi:10.1109/icmla69890.2026.00108
“Domain-specific variants such as BioBERT [17], SciBERT [18], and ClinicalBERT [19], pretrained on biomedical and clinical corpora respectively, have shown improvements over general-domain models on biomedical NLP tasks.”
A Multi-Agent Framework for Structured Contribution Extraction from Scientific Papers
Siravuri, H. V., Akella, A. P., & Alhoori, H. (2026) · International Conference on Machine Learning and Applications · doi:10.1109/icmla69890.2026.00097 · Cited by 1
“Earlier scientific information-extraction pipelines used small task-specific models trained on hand-annotated corpora such as SciERC, SciREX, TDMSci, and AxCell [7]–[10], [12].”
Liu, Y., Xue, D., Sun, J., Ding, X., Gao, Z., & Lookman, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.03767
Multimodal Neural Framework for Interpretable Assessment of Scientific Manuscripts
Dong, S., Fan, M., & Zhou, W. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-7738-4_14
A local orchestration harness for molecular design with foundation modelsPreprint
Mostafa, S., Ahamed, M. A., Zou, J., Xing, L., & Islam, M. T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.03676
OdorZero: Zero-Shot Odor-to-Molecule Prediction from Natural Language Descriptions
Liu, Z., & Zhang, X. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3836453
Aaron, S., Newaz, A., Abir, S., Rahman, M., Hossain, M., & Ahosan, N. (2026) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2026.v08i06.90699
“[ 53] Foundations of Inference ANLI (Adversarial Natural Language Inference) collected inference examples through an iterative human-and-model protocol that challenged current state-of-the-art systems.”
Qi, X., Malik, M. I., 王忠静, ZHANG, J., Wang, S., Gan, H., Li, K., & Shi, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-11469328/v1
Huang, C.-Y., Hsu, T.-Y., Rossi, R., Nenkova, A., Kim, S., Chan, G. Y.-Y., Koh, E., Giles, C. L., & Huang, T.-H. '. (2023) · arXiv · doi:10.48550/arxiv.2302.12324
“We used SciBERT (Beltagy et al., 2019) to obtain contextual embeddings and softmax threshold = 0.99 to reduce false alignments.”
MethodInfluentialSciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Singh, A., D'Arcy, M., Cohan, A., Downey, D., & Feldman, S. (2022) · arXiv · doi:10.48550/arxiv.2211.13308 · Cited by 76
“Among the vanilla models, even though E5 v2 and MPNet perform better than SciBERT; SPECTER, SciNCL and SPECTER2 Base outperform them suggesting the need for domain and task specific embeddings to do well on SciRepE-val.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.