SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 301–400 of 3,596
Li, J., Wei, Q., Ghiasvand, O., Chen, M., Lobanov, V. S., Weng, C., & Xu, H. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-01967-7 · Cited by 63
“The Wilcoxon rank sum test is a non-parametric test method that determines whether the means of strict F1 scores (Bold values) from the 10-fold experiments of the PubMedBERT model and each other model (BERT, SpanBERT, BioBERT, SciBERT) are statistically different from each other based on ranks rather than the original…”
MethodInfluentialValidation guidelines for drug-target prediction methods
Tanoli, Z., Schulman, A., & Aittokallio, T. A. (2024) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2024.2430955 · Cited by 60
“For example, recent studies [52,53] have utilized LLMs such as BERT [54], SciBERT [55], BioBERT [56], BioMed-RoBERTa [57] and BlueBERT [58] to identify drug-target interactions from experimental literature.”
Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature
Goldsack, T., Zhang, Z., Lin, C., & Scarton, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.724 · Cited by 138
Deidentification of free-text medical records using pre-trained bidirectional transformers
Johnson, A. E. W., Bulgarelli, L., & Pollard, T. J. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384455 · Cited by 55
“As BERT was trained using a general language corpus, we evaluate model performance when using pretrained weights from scientific corpora (SciBERT) or biomedical corpora (BioBERT) [3, 17].”
MethodContext-Aware Attentive Multilevel Feature Fusion for Named Entity Recognition
Yang, Z., Ma, J., Chen, H., Zhang, J., & Chang, Y. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3178522 · Cited by 57
“Although NER approaches combining word-level embedding and character-level embedding have demonstrated improved results [7]–[9], they may pay little attention to fusing multifeatures and therefore lose a significant amount of information.”
BackgroundMethodResultInfluentialNaseem, U., Musiał, K., Eklund, P., & Prasad, M. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206808 · Cited by 63
Siano, F., & Wysocki, P. D. (2021) · Accounting Horizons · doi:10.2308/horizons-19-161 · Cited by 62
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 44
Gender bias in transformers: A comprehensive review of detection and mitigation strategies
Nemani, P., Joel, Y. D., Vijay, P., & Liza, F. F. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100047 · Cited by 75
Lo, K., Chang, J. C., Head, A., Bragg, J., Zhang, A. X., Trier, C., Anastasiades, C., August, T., Authur, R., Bragg, D., et al. (2024) · Communications of the ACM · doi:10.1145/3659096 · Cited by 61
Self-citationPaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Lála, J., O'Donoghue, O., Shtedritski, A., Cox, S., Rodriques, S. G., & White, A. D. (2023) · arXiv · doi:10.48550/arxiv.2312.07559 · Cited by 55
“Work in the last 10 years has sought to make the space of literature more manageable for scientists, with the introduction of keyword search systems [6, 7], vector similarity embeddings [8, 9, 10, 11] and recommender systems [12, 13].”
BackgroundAutomated Extraction and Presentation of Data Practices in Privacy Policies
Bui, D., Shin, K. G., Choi, J., & Shin, J. (2021) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2021-0019 · Cited by 82
TNT-KID: Transformer-based neural tagger for keyword identification
Martinc, M., Škrlj, B., & Pollak, S. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000127 · Cited by 64
Effective entity matching with transformers
Li, Y., Li, J., Suhara, Y., Doan, A., & Tan, W.-C. (2023) · The VLDB Journal · doi:10.1007/s00778-023-00779-z · Cited by 68
Song, Y., Miret, S., & Liu, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.201 · Cited by 83
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 58
Carrino, C. P., Armengol-Estapé, J., Gutiérrez-Fandiño, A., Palao, J. L., Pàmies, M., González-Agirre, A., & Villegas, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03570 · Cited by 77
“However, in contrast to [6, 13] i) we used a corpus 3-5 times smaller, ii) we performed pretraining from scratch with mixed-domain data from biomedical documents and clinical notes and iii) we assess the suitability of cross-domain transfer from the biomedical to the clinical domain.”
InfluentialAttend to Medical Ontologies: Content Selection for Clinical Abstractive Summarization
Sotudeh, S., Goharian, N., & Filice, R. W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.172 · Cited by 64
SsciBERT: a pre-trained language model for social science texts
Shen, S., Liu, J., Lin, L., Huang, Y., Zhang, L., Liu, C., Feng, Y., & Wang, D. (2022) · Scientometrics · doi:10.1007/s11192-022-04602-4 · Cited by 64
“The emergence of BERT has greatly promoted the development of pre-trained models and has generated a series of pre-trained models based on domain-specific data, such as SciBERT (Beltagy et al., 2019), BioBERT (Lee et al.”
MethodResultInfluentialReadability Controllable Biomedical Document Summarization
Luo, Z., Xie, Q., & Ananiadou, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.343 · Cited by 86
End-to-End Construction of NLP Knowledge Graph
Mondal, I., Hou, Y., & Jochim, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.165 · Cited by 79
The Diminishing Returns of Masked Language Models to Science
Hong, Z., Ajith, A., Pauloski, J., Duede, E., Chard, K., & Foster, I. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.82 · Cited by 24
“Subsequently, other masked language models have since been pretrained on domain-specific scientific corpora (Gu et al., 2021; Huang and Cole, 2022; Beltagy et al., 2019) with the goal of improving downstream task performance.”
BackgroundMethodRAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
Yuan, Z. P., Jin, Q., Tan, C., Zhao, Z., Yuan, H., Huang, F., & Huang, S. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611830 · Cited by 85
“In biomedical NLP, previous research has shown that continue training in the in-domain corpus can enhance the performance of pretrained language models [1, 14, 23, 44, 45].”
BackgroundEckmann, P., & Bandrowski, A. E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281659 · Cited by 54
“While language models that were trained on scientific text exist, like SciBERT [68], they were too slow for our purposes.”
Semantics and Content-Based Recommendations
Musto, C., de Gemmis, M., Lops, P., Narducci, F., & Semeraro, G. M. (2012) · Springer eBooks · doi:10.1007/978-1-0716-2197-4_7 · Cited by 69
ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
Huang, J., Wang, H., Sun, Y., Shi, Y., Huang, Z., Zhuo, A., & Feng, S. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539021 · Cited by 77
“Existing domain-specific PTMs mainly lie in the domain of healthcare [1, 17], biomedical [8, 10, 19], and academic & research [2].”
BackgroundGlobal citation recommendation employing generative adversarial network
Ali, Z., Qi, G., Muhammad, K., Kefalas, P., & Khusro, S. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114888 · Cited by 45
Domain-specific NLP system to support learning path and curriculum design at tech universities
Vo, N. N. Y., Vu, Q. T., Vu, N. H., Vu, T. A., Mach, B. D., & Xu, G. (2021) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2021.100042 · Cited by 52
Advances in materials informatics: a review
Sivan, D., Kumar, K. S., Abdullah, A., Raj, V., Misnon, I. I., Ramakrishna, S., & Jose, R. (2024) · Journal of Materials Science · doi:10.1007/s10853-024-09379-w · Cited by 58
UDALM: Unsupervised Domain Adaptation through Language Modeling
Karouzos, C., Paraskevopoulos, G., & Potamianos, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.203 · Cited by 83
Lightweight transformers for clinical natural language processing
Rohanian, O., Nouriborji, M., Jauncey, H., Kouchaki, S., Nooralahzadeh, F., Clifton, L., Merson, L., & Clifton, D. A. (2024) · Natural Language Engineering · doi:10.1017/s1351324923000542 · Cited by 65
“…is ample evidence that, for certain domains and downstream tasks, models that are pre-trained on specialised data out-perform baselines that have only relied on generic texts (Sanh et al. 2019; Alsentzer et al. 2019; Beltagy, Lo, and Cohan 2019; Nguyen, Vu, and Nguyen 2020; Chalkidis et al. 2020).”
BackgroundPatton: Language Model Pretraining on Text-Rich Networks
Jin, B., Zhang, W., Zhang, Y., Yu, M., Zhang, X., Zhu, Q., & Han, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.387 · Cited by 88
Beyond Citations: Measuring Novel Scientific Ideas and their Impact in Publication Text
Arts, S., Melluso, N., & Veugelers, R. (2025) · The Review of Economics and Statistics · doi:10.1162/rest_a_01561 · Cited by 63
“However, this metric seems better suited for measuring similarity between publication documents rather than identifying new scientific ideas (Beltagy et al., 2019; Cohan et al., 2020; Erhardt et al., 2022).”
BackgroundLu, Z., Sim, J.-A., Wang, J. X., Forrest, C. B., Krull, K. R., Srivastava, D. K., Hudson, M. M., Robison, L. L., Baker, J. N., & Huang, I. (2021) · Journal of Medical Internet Research · doi:10.2196/26777 · Cited by 52
“Similar methods may include SciBERT [62], trained by texts in Semantic Scholar; BioBERT [49], trained by texts in PubMed; and Clinical BERT [51], trained by clinical notes in MIMIC-III [63].”
Quo Vadis ChatGPT? From large language models to Large Knowledge Models
Venkatasubramanian, V., & Chakraborty, A. (2024) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2024.108895 · Cited by 54
“With the rapid increase in the number of applications of LLMs in the scientific and engineering domains, there has been a rise in scientific LLMs such as PubMedBERT[106], MolGPT[107], SciBERT[108] – to name a few.”
BackgroundAn Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction
Zaratiana, U., Tomeh, N., Holat, P., & Charnois, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29919 · Cited by 74
AdaPrompt: Adaptive Model Training for Prompt-based NLP
Chen, Y., Liu, Y., Li, D., Wang, S., Zhu, C., Zeng, M., & Zhang, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.448 · Cited by 61
BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. E., & Rodriques, S. G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.162 · Cited by 74
Watermarking techniques for large language models: a survey
Liang, Y., Xiao, J., Gan, W., & Yu, P. S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11474-6 · Cited by 65
“…of model-as-a-service [83], LLMs have achieved preliminary intelligent applications in fields such as education [82, 114, 254], scientific research [17, 210], healthcare [221, 285], programming [40, 253], law [54, 124], and robotics [271], demonstrating remarkable performance that can efficiently…”
Chen, C., Wang, H., Shapiro, M. A., Xiao, Y., Wang, F., & Shu, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.05289 · Cited by 58
NLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.277 · Cited by 50
Single Model for Organic and Inorganic Chemical Named Entity Recognition in ChemDataExtractor
Isazawa, T., & Cole, J. M. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.1c01199 · Cited by 50
MTTLADE: A multi-task transfer learning-based method for adverse drug events extraction
El-allaly, E.-D., Sarrouti, M., En-Nahnahi, N., & Alaoui, S. O. E. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2020.102473 · Cited by 54
Venkatesan, V. K., Mahesh, T. R., Batyuk, A. Y., Barna, A., & Havrysh, B. (2023) · Systems · doi:10.3390/systems11020081 · Cited by 50
Document-level Entity-based Extraction as Template Generation
Huang, K.-H., Tang, S., & Peng, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.426 · Cited by 80
Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media
Dai, X., Karimi, S., Hachey, B., & Paris, C. L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.151 · Cited by 57
Study of Word Embeddings for Enhanced Cyber Security Named Entity Recognition
Srivastava, S., Paul, B., & Gupta, D. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.01.027 · Cited by 60
Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context
Ruan, K., Wang, X., Hong, J., Wang, P., Liu, Y., & Sun, H. (2026) · Nature Communications · doi:10.1038/s41467-026-70245-1 · Cited by 68
PTM4Tag: sharpening tag recommendation of stack overflow posts with pre-trained models
He, J., Xu, B., Yang, Z., Han, D., Yang, C., & Lo, D. (2022) · Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension · doi:10.1145/3524610.3527897 · Cited by 43
“Recent works [2, 8, 13] have shown that the in-domain knowledge acquired by PTMs is valuable in improving the performance on domain-specific tasks, such as ClinicalBERT [8] for clinical text, SciBERT [2] for scientific text, and BioBERT [13] for biomedical text.”
BackgroundAdapting vs. Pre-training Language Models for Historical Languages
Manjavacas, E., & Fonteyn, L. (2022) · Journal of Data Mining & Digital Humanities · doi:10.46298/jdmdh.9152 · Cited by 77
Cao, L., & Liu, Q. (2022) · doi:10.1101/2022.08.22.22279022 · Cited by 49
Esteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., Radev, D., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09595 · Cited by 48
Revisiting Electrocatalyst Design by a Knowledge Graph of Cu-Based Catalysts for CO 2 Reduction
Gao, Y., Wang, L., Chen, X., Du, Y., & Wang, B. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c00759 · Cited by 48
Foppiano, L., Lambard, G., Amagasa, T., & Ishii, M. (2024) · Science and Technology of Advanced Materials Methods · doi:10.1080/27660400.2024.2356506 · Cited by 48
“This reduces the risk that its annotations had been used during the pre-training of any of the LLMs. Baseline scores were established using a SciBERT-based [32] encoder and RE rule-based algorithm [27] for material-related extractions.”
MethodLEAP: LLM instruction-example adaptive prompting framework for biomedical relation extraction
Zhou, H., Li, M., Xiao, Y., Yang, H., & Zhang, R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae147 · Cited by 53
MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., & Lee, D. (2021) · arXiv · doi:10.48550/arxiv.2106.07340 · Cited by 47
“Inspired by SciBERT [2], we create a custom mathematical vocabulary (mathVocab) usingHugging Face BertWordPieceTokenizer4 with a size of 30,522 from the BASE BERT.”
BackgroundMethodInfluentialA Supervised Multi-Head Self-Attention Network for Nested Named Entity Recognition
Xu, Y., Huang, H., Feng, C., & Hu, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17669 · Cited by 54
“…model, we apply SciBERT base on GENIA and JNLPBA datasets and apply BERT base on other datasets and their detailed model size are referred to (Beltagy, Lo, and Cohan 2019) and (De-vlin et al. 2019), respectively; while without pre-trained language models, we use the pretrained embeddings…”
MethodMachine learning for analyses and automation of structural characterization of polymer materials
Lu, S., & Jayaraman, A. (2024) · Progress in Polymer Science · doi:10.1016/j.progpolymsci.2024.101828 · Cited by 49
Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
Su, P., Peng, Y., & Vijay‐Shanker, K. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.1 · Cited by 53
Zhang, T., Cai, Z., Wang, C., Qiu, M., Yang, B., & He, X. (2021) · doi:10.18653/v1/2021.acl-long.457 · Cited by 61
“(1) BioBERT ( Lee et al., 2020), BlueBERT (Peng et al., 2019), SCIBERT ( Beltagy et al., 2019) and ClinicalBert (Huang et al., 2019) apply continual learning on (2) PubMedBERT (Gu et al., 2020) learns weights from scratch using PubMed data to obtain an in-domain vocabulary, alleviating the out-of-vocabulary (OOV) prob…”
fBERT: A Neural Transformer for Identifying Offensive Content
Sarkar, D., Zampieri, M., Ranasinghe, T., & Ororbia, A. G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.154 · Cited by 47
MARTA: Leveraging Human Rationales for Explainable Text Classification
Arous, I., Dolamic, L., Yang, J., Bhardwaj, A., Cuccu, G., & Cudré-Mauroux, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16734 · Cited by 41
“First, we observe that ALBERT and SciBERT perform relatively well compared to the other baseline methods, especially on the Wiki-Tech dataset.”
BackgroundInfluentialAugmenting Scientific Creativity with an Analogical Search Engine
Kang, H. B., Qian, X., Hope, T., Shahaf, D., Chan, J., & Kittur, A. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3530013 · Cited by 40
“, ELMo [90], BERT [29], and SciBERT [7]) will be valuable.”
BackgroundLamsal, R., Rodriguez, M. A., & Karunasekera, S. A. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111916 · Cited by 62
“Similarly, SciBERT [4] also shared the architecture of BERT but was pre-trained on a random”
BackgroundMethodInfluentialEfficient Domain Adaptation of Language Models via Adaptive Tokenization
Sachidananda, V., Kessler, J. S., & Lai, Y.-A. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.16 · Cited by 77
Large language models to process, analyze, and synthesize biomedical texts: a scoping review
Doneva, S. E., Qin, S., Sick, B., Ellendorff, T., Goldman, J.-P., Schneider, G., & Ineichen, B. V. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00197-2 · Cited by 49
“…OR ’machine learning’/de OR ’automated pattern recognition’/exp OR ’artificial intelligence’/de OR ’semi automation’:ti,ab,kw OR automation:ti,ab,kw OR ’artificial intelligence’:ti,ab,kw OR ai:ti,ab,kw OR ’natural language processing’:ti,ab,kw OR ((machine NEXT/1 (intelligence OR learning”
E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commercePreprint
Zhang, D., Yuan, Z., Liu, Y., Zhuang, F., Chen, H., & Xiong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02835 · Cited by 55
Unsupervised Pre-training for Biomedical Question AnsweringPreprint
Kommaraju, V., Gunasekaran, K. P., Li, K., Bansal, T., McCallum, A., Williams, I., & Istrate, A.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12952 · Cited by 51
Wang, J., Zhang, G., Wang, W., Zhang, K., & Sheng, Y. (2021) · Journal of Cloud Computing · doi:10.1186/s13677-020-00218-2 · Cited by 43
“As for domain-oriented pre-trained models, Beltagy [25] et al.”
MethodEnhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations
Zeng, Q., Lin, J., Yu, W., Cleland‐Huang, J., & Jiang, M. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467308 · Cited by 58
“For the masked concept 𝑣 , we finally return the graph-based hidden state v at K-th layer, i.e., v = v ( 𝐾 ) .”
MethodSemantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes
Moro, G., & Ragazzi, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21357 · Cited by 80
Zhang, H., Zhang, C., & Wang, Y. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103574 · Cited by 49
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.134 · Cited by 65
A Search Engine for Discovery of Scientific Challenges and Directions
Lahav, D., Saad-Falcon, J., Kuehl, B., Johnson, S., Parasa, S., Shomron, N., Chau, D. H., Yang, D., Horvitz, E., Weld, D. S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21456 · Cited by 55
“We report results for PubMedBERT-abstract-fulltext (Gu et al. 2020) which was pre-trained on PubMed paper abstracts & full texts, and for SciB-ERT (Beltagy, Lo, and Cohan 2019), trained on a corpus of biomedical and computer science papers.”
MethodScientific Fact-Checking: A Survey of Resources and Approaches
Vladika, J., & Matthes, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.387 · Cited by 96
Natural Language Processing in Pathology
Úbeda, P. L., Martín‐Noguerol, T., Aneiros‐Fernández, J., & Luna, A. (2022) · American Journal Of Pathology · doi:10.1016/j.ajpath.2022.07.012 · Cited by 52
ChatGPT for digital pathology research
Omar, M., Ullanat, V., Loda, M. F., Marchionni, L., & Umeton, R. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00114-6 · Cited by 57
Accurate Clinical and Biomedical Named Entity Recognition at Scale
Kocaman, V., & Talby, D. (2022) · Software Impacts · doi:10.1016/j.simpa.2022.100373 · Cited by 39
KALA: Knowledge-Augmented Language Model Adaptation
Kang, M., Baek, J., & Hwang, S. J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.379 · Cited by 36
KnowLog: Knowledge Enhanced Pre-trained Language Model for Log Understanding
Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B., Liang, J., Zhou, M., & Xiao, Y. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623304 · Cited by 35
A large language model framework to uncover underreporting in traffic crashes
Arteaga, C., & Park, J. (2024) · Journal of Safety Research · doi:10.1016/j.jsr.2024.11.009 · Cited by 45
Text Representations and Word Embeddings: Vectorizing Textual Data
Egger, R. (2022) · Tourism on the verge · doi:10.1007/978-3-030-88389-8_16 · Cited by 65
Fayyazi, R., Taghdimi, R., & Yang, S. J. (2024) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw65225.2024.00036 · Cited by 54
Rise of Generative Artificial Intelligence in Science
Ding, L., Lawson, C., & Shapira, P. (2025) · Scientometrics · doi:10.1007/s11192-025-05413-z · Cited by 53
“This exponential growth pattern reflects the recent surge of engagement with The SciBERT model demonstrated strong performance, achieving high accuracy in the manual check of 150 samples used to assess the search term accuracy. generative AI.”
InfluentialLitLLMs, LLMs for Literature Review: Are we there yet?
Agarwal, S., Sahu, G., Puri, A., Laradji, I., Dvijotham, K., Stanley, J., Charlin, L., & Pal, C. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2412.15249 · Cited by 53
“SPECTER uses a contrastive learning approach to train a SciBERT (Beltagy et al., 2019) Table 1: We created two datasets to measure the efficacy of search using LLM-generated keywords - RollingEval-Aug and RollingEval-Dec.”
Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
Chau, E. C., Lin, L. H., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.118 · Cited by 44
“Following prior work involving these two variants (Beltagy et al., 2019), FT variants perform biaffine attention directly on the outputs of MBERT instead of first passing them through a BiLSTM, as in Dozat and Manning (2017).”
MethodChen, Q., Allot, A., Leaman, R., Islamaj, R., Du, J., Li, F., Wang, K., Xu, S., Zhang, Y., Bagherzadeh, P., et al. (2022) · Database · doi:10.1093/database/baac069 · Cited by 53
Shi, S., Hu, K., Xie, J., Guo, Y., & Wu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103531 · Cited by 50
Licari, D., & Comandé, G. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105908 · Cited by 59
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups
Shen, Z., Lo, K., Wang, L. L., Kuehl, B., Weld, D. S., & Downey, D. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00466 · Cited by 64
Self-citationConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
Hu, Y., Hosseini, M., Parolin, E. S., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.400 · Cited by 90
OAG-BERT: Towards a Unified Backbone Language Model for Academic Knowledge Services
Liu, X., Yin, D., Zheng, J., Zhang, X., Zhang, P., Yang, H., Dong, Y., & Tang, J. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539210 · Cited by 58
“We take SciBERT [3] as our major compared method to demonstrate the importance of our entity-augmented pre-training.”
BackgroundMethodInfluentialMorales-Hernández, R. C., Gutiérrez, J., & Becerra‐Alonso, D. (2022) · IEEE Access · doi:10.1109/access.2022.3223094 · Cited by 49
Automating the removal of obsolete TODO comments
Gao, Z., Xia, X., Lo, D., Grundy, J., & Zimmermann, T. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468553 · Cited by 44
“This is because BERT has been proven to be effective for capturing semantics and context information of sentences in many other works [1, 4, 17, 32, 42].”
BackgroundA meta-analysis of semantic classification of citations
Kunnath, S. N., Herrmannova, D., Pride, D., & Knoth, P. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00159 · Cited by 43
Kedia, N., Sanjeev, S., Ong, J., & Chhablani, J. K. (2024) · Eye · doi:10.1038/s41433-023-02915-z · Cited by 50
“The responses generated when the patient was specified to be black or African American resulted in a worse course of action than when specified as white and Caucasian [62, 63].”
Färber, M., & Ao, L. (2022) · Quantitative Science Studies · doi:10.1162/qss_a_00183 · Cited by 53
“As we can see, SciBERT and BERT base outperform other models significantly, with SciBERT slightly edging ahead in comparison.”
BackgroundInfluentialDrug-Drug Interaction Extraction from Biomedical Text Using Relation BioBERT with BLSTM
KafiKang, M., & Hendawi, A. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5020036 · Cited by 50
Lauscher, A., Ko, B., Kuehl, B., Johnson, S., Cohan, A., Jurgens, D., & Lo, K. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.137 · Cited by 73
Self-citationA Comparison of SVM Against Pre-trained Language Models (PLMs) for Text Classification Tasks
Wahba, Y., Madhavji, N. H., & Steinbacher, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25891-6_23 · Cited by 63
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.