SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 101–200 of 3,596
A comprehensive survey on relation extraction: recent advances and new frontiers
Zhao, X., Deng, Y., Yang, M., Wang, L., Zhang, R., Cheng, H., Lam, W., Shen, Y., & Xu, R. (2024) · ACM Computing Surveys · doi:10.1145/3674501 · Cited by 168
“Luan [96] proposed a semi-supervised learning framework for scientific RE.”
A Partition Filter Network for Joint Entity and Relation Extraction
Yan, Z., Zhang, C., Fu, J., Zhang, Q., & Wei, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.17 · Cited by 201
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 247
“PBP [96] used SciBERT [97] as the backbone model and learned natural language prompts that can classify medical texts.”
Improving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161
CrossNER: Evaluating Cross-Domain Named Entity Recognition
Liu, Z., Xu, Y., Yu, T., Dai, W., Ji, Z., Cahyawijaya, S., Madotto, A., & Fung, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17587 · Cited by 261
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.29 · Cited by 91
Global Pointer: Novel Efficient Span-based Approach for Named Entity RecognitionPreprint
Su, J., Ahmed, M., Pan, S., Hou, J., Sun, J., Huang, W., Wen, B., & Liu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.03054 · Cited by 228
AI vs. Human -- Differentiation Analysis of Scientific Content Generation
Ma, Y., Liu, J., Fan, Y., Cheng, Q., Huang, Y., Lu, W., & Liu, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.10416 · Cited by 68
“Black et al., (2021) combine source-domain data with in-domain labeled data to solve the problem of detecting GPT-2-generated technical research text. During the Scholarly Document Processing workshop at COLING 2022, Kashnitsky et al., (2022) proposed the task and corresponding dataset on the detection of automaticall…”
BackgroundMethodInfluentialSciREX: A Challenge Dataset for Document-Level Information Extraction
Jain, S., van Zuylen, M., Hajishirzi, H., & Beltagy, I. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.670 · Cited by 159
Self-citationDifferentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
Zhang, N., Li, L., Chen, X., Deng, S., Bi, Z., Tan, C., Huang, F., & Chen, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13161 · Cited by 74
Large language models: a survey of their development, capabilities, and applications
Yadagiri, A., & Pakray, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02310-4 · Cited by 218
An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics
Koh, H. Y., Ju, J., Liu, M., & Pan, S. (2023) · ACM Computing Surveys · doi:10.1145/3545176 · Cited by 112
“[1] ine-tuned BERT on large-scale scientiic paper datasets and have found its performance to improve in scientiic domains as compared to the BERT-base model.”
Method(2021) · doi:10.1145/3462757.3466088 · Cited by 151
“DS scores may also be readily extended to estimate the domain-specificity of tasks in other domains with existing pretrained models like SciBERT and BioBERT [3, 25].”
BackgroundMethodResultInfluentialCOVID-19 Literature Knowledge Graph Construction and Drug Repurposing Report Generation
Wang, Q., Li, M., Wang, X., Parulian, N., Han, G., Ma, J., Tu, J., Lin, Y., Zhang, R. H., Liu, W., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations · doi:10.18653/v1/2021.naacl-demos.8 · Cited by 121
Combating misinformation in the age of LLMs: Opportunities and challenges
Chen, C., & Shu, K. (2024) · AI Magazine · doi:10.1002/aaai.12188 · Cited by 148
Applications of natural language processing and large language models in materials discovery
Jiang, X., Wang, W., Tian, S., Wang, H., Lookman, T., & Su, Y. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01554-0 · Cited by 145
FedBERT : When Federated Learning Meets Pre-training
Tian, Y., Wan, Y., Lyu, L., Yao, D., Jin, H., & Sun, L. (2022) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3510033 · Cited by 201
“, BioBERT [24] in medical texts, SciBERT in scientific articles [3], and so on.”
BackgroundBiomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 118
“Pre-trained LMs on biomedical corpora, such as BioELMo [80], BioBERT [98], SciBERT [14], clinical BERT [7, 73], and PubMedBERT [58], further improve their in-domain performance.”
MethodInfluentialPacked Levitated Marker for Entity and Relation Extraction
Ye, D., Lin, Y., Li, P., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.337 · Cited by 144
“For SciERC, we use the indomain scibert-scivocab-uncased (Beltagy et al., 2019) encoder.”
Pretrained Language Models for Sequential Sentence Classification
Cohan, A., Beltagy, I., King, D., Dalvi, B., & Weld, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1383 · Cited by 112
Self-citationData cleaning and machine learning: a systematic literature review
Côté, P.-O., Nikanjam, A., Ahmed, N., Humeniuk, D. V., & Khomh, F. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00453-w · Cited by 163
Mining product innovation ideas from online reviews
Zhang, M., Fan, B., Zhang, N., Wang, W., & Fan, W. (2020) · Information Processing & Management · doi:10.1016/j.ipm.2020.102389 · Cited by 140
A survey on extraction of causal relations from natural language text
Yang, J., Han, S. C., & Poon, J. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01665-w · Cited by 189
UniRE: A Unified Label Space for Entity Relation Extraction
Wang, Y., Sun, C., Wu, Y., Zhou, H., Li, L., & Yan, J. (2021) · doi:10.18653/v1/2021.acl-long.19 · Cited by 159
“For fair comparison with previous works, we use three pre-trained language models: bert-base-uncased (Devlin et al., 2019), albert-xxlarge-v1 (Lan et al., 2019) and scibert-scivocab-uncased (Beltagy et al., 2019) as the sentence encoder and fine-tune them in training stage.”
BackgroundMethodLarge language models for reticular chemistry
Zheng, Z., Rampal, N., Inizan, T. J., Borgs, C., Chayes, J. T., & Yaghi, O. M. (2025) · Nature Reviews Materials · doi:10.1038/s41578-025-00772-8 · Cited by 134
“Models that can be fine-tuned include LLaMA 20,21 , Mistral 64 , SciBERT 65 , BART 66 , T5 (ref.”
Zeng, Z., Yao, Y., Liu, Z., & Sun, M. (2022) · Nature Communications · doi:10.1038/s41467-022-28494-3 · Cited by 233
A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108189 · Cited by 106
Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law
Paul, S., Mandal, A., Goyal, P., & Ghosh, S. (2023) · International Conference on Artificial Intelligence and Law · doi:10.1145/3594536.3595165 · Cited by 180
“Although earlier efforts mostly utilized general (open) domain data for pre-training [2, 18, 32], limited ability of such PLMs to fine-tune to end tasks on specialized domains such as biomedical or law has prompted efforts in domain-specific pre-training [1, 16].”
BackgroundWagner, T., Shweta, F., Murugadoss, K., Awasthi, S., Venkatakrishnan, A., Bade, S., Puranik, A., Kang, M., Pickering, B. W., O'Horo, J. C., et al. (2020) · eLife · doi:10.7554/elife.58227 · Cited by 135
BioELECTRA:Pretrained Biomedical text Encoder using Discriminators
Kanakarajan, K. R., Kundumani, B., & Sankarasubbu, M. (2021) · Proceedings of the 20th Workshop on Biomedical Language Processing · doi:10.18653/v1/2021.bionlp-1.16 · Cited by 112
LLM for SoC Security: A Paradigm Shift
Saha, D., Tarek, S., Yahyaei, K., Saha, S. K., Zhou, J., Tehranipoor, M., & Farahmandi, F. (2024) · IEEE Access · doi:10.1109/access.2024.3427369 · Cited by 93
“The remarkable success of these models has catalyzed the development of fine-tuned domain-specific LLMs such as Med-PaLM [18], BloombergGPT [19], AugGPT [20], BioBERT [21], SciBERT [22], etc.”
BackgroundA Span-Based Model for Joint Overlapped and Discontinuous Named Entity Recognition
Li, F., Lin, Z., Zhang, M., & Ji, D. (2021) · doi:10.18653/v1/2021.acl-long.372 · Cited by 154
“Considering the domains of the datasets, we employ clinical BERT 1 (Alsentzer et al., 2019), SciBERT 2 (Beltagy et al., 2019) and Google BERT 3 (Devlin et al., 2019) for the CLEF (and CADEC), GENIA and ACE05 datasets, respectively.”
MethodGREEK-BERT: The Greeks visiting Sesame Street
Koutsikakis, J., Chalkidis, I., Malakasiotis, P., & Androutsopoulos, I. (2020) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3411408.3411440 · Cited by 171
LLMs for science: usage for code generation and data analysis
Nejjar, M., Zacharias, L., Stiehle, F., & Weber, I. (2025) · Journal of Software: Evolution and Process · doi:10.1002/smr.2723 · Cited by 72
A survey of recent methods for addressing AI fairness and bias in biomedicine
Yang, Y., Lin, M., Zhao, H., Peng, Y., Huang, F., & Lu, Z. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104646 · Cited by 146
“In a study by Zhang et al., they used MIMIC-III 26 , a publicly available clinical note database, to finetune a BERT model initialized from SciBERT 27 .”
MethodJust, J. (2023) · Technovation · doi:10.1016/j.technovation.2023.102883 · Cited by 145
LLM-Prop: predicting the properties of crystalline materials using large language models
Rubungo, A. N., Arnold, C. B., Rand, B. P., & Dieng, A. B. (2025) · npj Computational Materials · doi:10.1038/s41524-025-01536-2 · Cited by 144
Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology
Tan, T. F., Thirunavukarasu, A. J., Campbell, J. P., Keane, P. A., Pasquale, L. R., Abramoff, M. D., Kalpathy-Cramer, J., Lum, F., Kim, J. E., Baxter, S. L., et al. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100394 · Cited by 116
He, Y., Zhu, Z., Zhang, Y., Chen, Q., & Caverlee, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.372 · Cited by 103
From Text to Transformation: A Comprehensive Review of Large Language Models' VersatilityPreprint
Kaur, P., Kashyap, G. S., Kumar, A., Nafis, M. T., Kumar, S., & Shokeen, V. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16142 · Cited by 159
“A study by [27] introduced SciBERT to examine the use of LLMs for production of scientific explanations, showcasing its potential in knowledge dissemination and hypothesis generation.”
BackgroundCyBERT: Contextualized Embeddings for the Cybersecurity Domain
Ranade, P., Piplai, A., Joshi, A., & Finin, T. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671824 · Cited by 185
“scratch on a new corpus of domain-specific task, typically as large as the original Wikipedia and Brown corpus used to train the base BERT model [35].”
MethodMSˆ2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.594 · Cited by 83
“…with work spanning the whole NLP pipeline: datasets (S2ORC (Lo et al., 2020), CORD-19 (Wang et al., 2020b)), pretrained transformer models (SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2020), ClinicalBERT (Huang et al., 2019), SPECTER (Cohan et al., 2020)), NLP tasks like NER (Nye et al.,…”
MethodInfluentialSelf-citationCODER: Knowledge-infused cross-lingual medical term embedding for term normalization
Yuan, Z., Zhao, Z., Sun, H., Li, J., Wang, F., & Yu, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103983 · Cited by 111
Zero-shot learning for requirements classification: An exploratory study
Alhoshan, W., Ferrari, A., & Zhao, L. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107202 · Cited by 149
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., & Chua, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.966 · Cited by 224
Yu, Y., Zuo, S., Jiang, H., Ren, W., Zhao, T., & Zhang, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.84 · Cited by 164
Text mining approaches for dealing with the rapidly expanding literature on COVID-19
Wang, L. L., & Lo, K. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa296 · Cited by 116
“Many projects in Section on “Text mining systems” leverage domain-adapted BERT [24] models such as SciBERT [6] and BioBERT [49], which have been fine-tuned to scientific and biomedical text, respectively.”
BackgroundMethodSelf-citationMedICaT: A Dataset of Medical Images, Captions, and Textual References
Subramanian, S., Wang, L. L., Bogin, B., Mehta, S., van Zuylen, M., Parasa, S., Singh, S., Gardner, M., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.191 · Cited by 82
S2ORC: The Semantic Scholar Open Research CorpusPreprint
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.02782 · Cited by 147
Self-citationHunFlair: an easy-to-use tool for state-of-the-art biomedical named entity recognition
Weber, L., Sänger, M., Münchmeyer, J., Habibi, M., Leser, U., & Akbik, A. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab042 · Cited by 106
von der Mosel, J., Trautsch, A., & Herbold, S. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3178469 · Cited by 133
“For example, SciBERT [6], BioBERT [7] and ClinicalBERT [8] are BERTmodels adopted to the bio-medical domain.”
BackgroundMethodInfluentialGIT-Mol: A multi-modal large language model for molecular science with graph, image, and text
Liu, P., Ren, Y., Tao, J., & Ren, Z. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108073 · Cited by 89
“In the pre-training phase, we use frozen image and graph encoders and SciBERT for text encoding.”
MethodInfluentialArtificial intelligence tools expand scientists' impact but contract science's focus
Hao, Q., Xu, F., Li, Y., & Evans, J. (2026) · Nature · doi:10.1038/s41586-025-09922-y · Cited by 117
“To identify AI papers in various fields across eras, we fine-tune BERT 32,33 , an established language model 34–36 , on articles published in explicitly AI-oriented scientific journals and conferences to automatically extract and interpret information from context.”
Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries
Edwards, C., Zhai, C., & Ji, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.47 · Cited by 83
Gangwal, A., & Lavecchia, A. (2025) · Drug Discovery Today · doi:10.1016/j.drudis.2025.104360 · Cited by 130
Reichenpfader, D., Müller, H., & Denecke, K. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01219-0 · Cited by 140
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction
Paranjape, B., Joshi, M., Thickstun, J., Hajishirzi, H., & Zettlemoyer, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.153 · Cited by 86
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Comparison of BERT implementations for natural language processing of narrative medical documents
Turchin, A., Masharsky, S., & Žitnik, M. (2022) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2022.101139 · Cited by 130
Pei, Q., Zhang, W., Zhu, J., Wu, K., Gao, K., Wu, L., Xia, Y., & Yan, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.70 · Cited by 190
Pilán, I., Lison, P., Øvrelid, L., Papadopoulou, A. I., Sánchez, D., & Batet, M. (2022) · Computational Linguistics · doi:10.1162/coli_a_00458 · Cited by 227
Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106
Improving Biomedical Pretrained Language Models with Knowledge
Yuan, Z., Liu, Y., Tan, C., Huang, S., & Huang, F. (2021) · Proceedings of the 20th Workshop on Biomedical Language Processing · doi:10.18653/v1/2021.bionlp-1.20 · Cited by 79
AI-Driven Inverse Design of Materials: Past, Present, and Future
Han, X., Wang, X.-D., 徐, M.-Y. 孟. X., Zhen, F., 姚, B.-W. 博. Y., Guo, P.-J., Gao, Z.-F., & Lu, Z.-Y. (2025) · Chinese Physics Letters · doi:10.1088/0256-307x/42/2/027403 · Cited by 123
Data extraction from polymer literature using large language models
Gupta, S., Mahmood, A. U., Shetty, P., Adeboye, A., & Ramprasad, R. (2024) · Communications Materials · doi:10.1038/s43246-024-00708-9 · Cited by 130
“In recent years, the transformer-based BERT architecture, with its ability to capture contextual and semantic relationships within scienti fi c texts 9 , has especially demonstrated superior performance com-paredtotraditionalneuralnetworkmodels 10 .”
A text-guided protein design framework
Liu, S., Li, Y., Li, Z., Gitter, A., Zhu, Y., Lu, J., Xu, Z., Nie, W., Ramanathan, A., Xiao, C., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01011-z · Cited by 134
Yang, K., Zhang, T., & Ananiadou, S. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.102961 · Cited by 116
RoBERTuito: a pre-trained language model for social media text in Spanish
Pérez, J. M., Furman, D., Alemany, L. A., & Luque, F. M. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4v9eufgmyo6q · Cited by 177
Data science for engineering design: State of the art and future directions
Chiarello, F., Belingheri, P., & Fantoni, G. (2021) · Computers in Industry · doi:10.1016/j.compind.2021.103447 · Cited by 103
Pre-trained language models with domain knowledge for biomedical extractive summarization
Xie, Q., Bishop, J. A., Tiwari, P., & Ananiadou, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109460 · Cited by 90
“However, previous efforts such as BioBERT [20], BlueBERT [21], ClinicalBERT, [22] and SciBERT [23] generally pretrained the PLMs on biomedical corpora, such as the PubMed and de-identified clinical notes from MIMIC-III.”
MethodInfluentialExplainable deep learning in healthcare: A methodological survey from an attribution view
Jin, D., Sergeeva, E., Weng, W., Chauhan, G., & Szolovits, P. (2022) · WIREs Mechanisms of Disease · doi:10.1002/wsbm.1548 · Cited by 72
An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels
Chalkidis, I., Fergadiotis, M., Kotitsas, S., Malakasiotis, P., Αλέτρας, Ν., & Androutsopoulos, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.607 · Cited by 126
Can Language Models be Biomedical Knowledge Bases?
Sung, M., Lee, J., Yi, S., Jeon, M., Kim, S., & Kang, J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.388 · Cited by 66
“While factual probing of LMs has attracted much attention from researchers, a more practical application would be to leverage the power of domainspecific LMs (Beltagy et al., 2019; Lee et al., 2020) as domain knowledge bases (KBs).”
BackgroundCVE2ATT&CK: BERT-Based Mapping of CVEs to MITRE ATT&CK Techniques
Grigorescu, O., Nica, A., Dascălu, M., & Rughinis, R. V. (2022) · Algorithms · doi:10.3390/a15090314 · Cited by 128
“Figure 6 details the proposed model which considers 768-dimensional contextual embeddings from various BERT-based models (i.e., BERT [41], SciBERT [42], and SecBERT [43]) passed through a Dropout layer.”
MethodFormative assessment strategies for students' conceptions—The potential of learning analytics
Stanja, J., Gritz, W., Krugel, J., Hoppe, A., & Dannemann, S. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13288 · Cited by 105
“It would be exciting to test the above proposals using these, or specialised language models such as BioBERT (Lee, Yoon, et al., 2020), SciBERT (Beltagy et al., 2019) or EduBERT (Clavié & Gal, 2019).”
MethodMolXPT: Wrapping Molecules with Text for Generative Pre-training
Liu, Z., Zhang, W., Xia, Y., Wu, L., Xie, S., Qin, T., Zhang, M., & Liu, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.138 · Cited by 142
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
Analyzing Transfer Learning of Vision Transformers for Interpreting Chest Radiography
Usman, M., Zia, T., & Tariq, S. A. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00666-z · Cited by 102
“The use of pre-trained transformers on a large corpus is widely used [60] and in medical field use of transformers on text is also well known like BioBERT [61], SciBERT [62], and ClinicalBERT [63].”
MethodAccelerating materials language processing with large language models
Choi, J., & Lee, B. (2024) · Communications Materials · doi:10.1038/s43246-024-00449-9 · Cited by 118
“For example, domain-speci fi c pretrained language models such as SciBERT 36 , MatBERT 8 , MatSciBERT 30 , and MaterialsBERT 37 were used to extract specialised information from materials science literature, thereby extracting entities on solid-state materials, doping, gold nanoparticles (AuNPs),polymers,electrocatalyt…”
Murali, L., Gopakumar, G., Viswanathan, D. M., & Nedungadi, P. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104403 · Cited by 93
Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge
Chen, Z., Li, G., & Wan, X. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547948 · Cited by 82
Kumar, A., & Starly, B. (2021) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-021-01807-x · Cited by 121
“In an effort to replace word2vec models, a custom pre-trained model meant for scientific text, called ‘SciBERT’ is available (Beltagy et al., 2019).”
MethodCan BERT Dig It? Named Entity Recognition for Information Retrieval in the Archaeology Domain
Brandsen, A., Verberne, S., Lambers, K., & Wansleeben, M. (2022) · Journal on Computing and Cultural Heritage · doi:10.1145/3497842 · Cited by 99
“We decided not to train a model from scratch as previous research showed only a minimal increase in quality compared to further pre-training [4] an existing model, and because our corpus is relatively small and would probably not be enough to train an efective model.”
InfluentialThe landscape of biomedical research
González-Márquez, R., Schmidt, L., Schmidt, B. M., Berens, P., & Kobak, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100968 · Cited by 89
“…biochemistry, bioinformatics, cancer, cardiology, chemistry, computation, dermatology, ecology, education, engineering, environment, ethics, genetics, gynecology, healthcare, immunology, infectious, material, microbiology, neurology, neuroscience, nursing, nutrition, ophthalmology,…”
Song, D., Vold, A., Madan, K., & Schilder, F. (2021) · Information Systems · doi:10.1016/j.is.2021.101718 · Cited by 113
A survey on retrieval-augmentation generation (RAG) models for healthcare applications
El-Enen, M. A. A., Saad, S., & Nazmy, T. M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11666-9 · Cited by 121
BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition
Naseem, U., Khushi, M., Reddy, V. L., Rajendran, S., Razzak, I., & Kim, J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533884 · Cited by 101
A Hierarchical BERT-Based Transfer Learning Approach for Multi-Dimensional Essay Scoring
Xue, J., Tang, X., & Zheng, L. (2021) · IEEE Access · doi:10.1109/access.2021.3110683 · Cited by 99
From Vulnerability to Defense: The Role of Large Language Models in Enhancing Cybersecurity
Kasri, W., Himeur, Y., Alkhazaleh, H. A., Tarapiah, S., Atalla, S., Mansoor, W., & Al-Ahmad, H. (2025) · Computation · doi:10.3390/computation13020030 · Cited by 70
Similarity of Precursors in Solid-State Synthesis as Text-Mined from Scientific Literature
He, T., Sun, W., Huo, H., Kononova, O., Rong, Z., Tshitoyan, V., Botari, T., & Ceder, G. (2020) · Chemistry of Materials · doi:10.1021/acs.chemmater.0c02553 · Cited by 85
Information Retrieval: Advanced Topics and Techniques
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 88
The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain
Friedrich, A., Adel, H., Tomazic, F., Hingerl, J., Benteau, R., Marusczyk, A., & Lange, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.116 · Cited by 113
Task-Specific Transformer-Based Language Models in Health Care: Scoping Review
Cho, H. N., Jun, T. J., Kim, Y., Kang, H. J., Ahn, I., Gwon, H., Kim, Y., Seo, H., Choi, H., Kim, M., et al. (2024) · JMIR Medical Informatics · doi:10.2196/49724 · Cited by 106
ScienceQA: a novel resource for question answering on scholarly articles
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., & Bhattacharyya, P. (2022) · International Journal on Digital Libraries · doi:10.1007/s00799-022-00329-y · Cited by 260
“Combination of SciBERT and Bi-DAF : This model combines both SciBERT and Bi-DAF approaches.”
MethodInfluentialCOVID-19 Knowledge Graph: Accelerating Information Retrieval and Discovery for Scientific Literature
Wise, C., Romero Calvo, M., Bhatia, P., Ioannidis, V., Karypus, G., Price, G., Song, X., Brand, R., & Kulkani, N. (2020) · Proceedings of Knowledgeable NLP: the First Workshop on Integrating Structured Knowledge and Neural Networks for NLP · doi:10.18653/v1/2020.knlp-1.1 · Cited by 11
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 108
“SciBERT [211] is an example of this approach, where a novel vocabulary of 30,000 terms specific to the domain was developed, and the model was trained on a corpus comprising both computer science (18”
FiNER: Financial Numeric Entity Recognition for XBRL Tagging
Loukas, L., Fergadiotis, M., Chalkidis, I., Spyropoulou, E., Malakasiotis, P., Androutsopoulos, I., & Paliouras, G. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-long.303 · Cited by 118
NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data
Bogdanov, S. V., Constantin, A., Bernard, T., Crabbé, B., & Bernard, E. P. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.660 · Cited by 131
Buehler, M. J. (2024) · ACS Engineering Au · doi:10.1021/acsengineeringau.3c00058 · Cited by 82
CyBERT: Cybersecurity Claim Classification by Fine-Tuning the BERT Language Model
Ameri, K., Hempel, M., Sharif, H., Lopez Jr., J., & Perumalla, K. (2021) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp1040031 · Cited by 76
“A pre-trained language model for scientific text (SciBERT) was introduced by Beltagy, I., Lo, K., and Cohan, A. [10].”
Method
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.