SciBERT: A Pretrained Language Model for Scientific Text
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 9 of the 33 sources its claims cite were checked for support.
Abstract
Iz Beltagy, Kyle Lo, Arman Cohan. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
CoChat Summary
TL;DR
SciBERT is a BERT-based pretrained language model trained on a 3.17B-token corpus of 1.14M scientific papers, with an in-domain WordPiece vocabulary. It improves over BERT-Base across scientific NLP tasks and achieves new state-of-the-art results on several datasets.
Key findings
- SciBERT outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without finetuning.
- Domain breakdowns: biomedical +1.92 F1 with finetuning and +3.59 F1 without; computer science +3.55 F1 with finetuning and +1.13 F1 without; multidomain +0.49 F1 with finetuning and +0.93 F1 without.
- Finetuning beats frozen BERT embeddings by +3.25 F1 with SciBERT and +3.58 F1 with BERT-Base on average.
- Using SCIVOCAB instead of BASEVOCAB gives +0.60 F1 averaged across datasets: +0.76 F1 biomedical, +0.61 F1 computer science, +0.11 F1 multidomain. SciBERT achieves new SOTA on BC5CDR, ChemProt, EBM-NLP, ACL-ARC, the NER part of SciERC, and SciCite.
Methods
- SciBERT follows the BERT-Base architecture and is pretrained on a random sample of 1.14M Semantic Scholar papers: 18% computer science and 82% broad biomedical, using full text, averaging 154 sentences (2,769 tokens) per paper, for 3.17B tokens.
- The authors build SCIVOCAB, a 30K WordPiece vocabulary with 42% token overlap with BASEVOCAB, and train four variants: cased/uncased and BASEVOCAB/SCIVOCAB.
- Evaluation covers NER, PICO extraction, text classification, relation classification, and dependency parsing, using datasets including EBM-NLP, SciERC, ACL-ARC, SciCite, and Paper Field; results are macro F1 for most tasks, micro F1 for ChemProt, and LAS/UAS for DEP, averaged over multiple random seeds.
Limitations
- The paper states future work will release a BERT-Large analogue and experiment with different proportions of papers from each domain, noting these models are costly to train and aiming for a single resource useful across multiple domains.
- The paper notes SciBERT performs slightly worse than SOTA on JNLPBA, NCBI-disease, and GENIA, attributing these to comparison models: a BiLSTM-CRF ensemble trained on multiple NER datasets, BIOBERT trained on 18B biomedical tokens, and a parser using POS features not used here.
- Evident from the text: the pretraining corpus is limited to computer science and biomedical papers, so coverage of other scientific domains is not established; frozen-embedding experiments did not use extensive hyperparameter search; and SciERC relation results are not comparable to prior work because this paper uses gold entities while prior work performs joint entity and relation extraction.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 0 Not valid · 16 Not checked · 29 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
- Partly supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) sourceExperimental — accuracy not yet measured
Section 3.4 says the ELMo biLM was pretrained on the 1B Word Benchmark, and the Introduction says the trained models were made publicly available. The paper does not discuss BERT or Wikipedia, or identify the benchmark text as news articles, so it does not support those parts of the sentence. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 1 Introduction
What the sources say
- Supportedcites Peters et al., 2018 matched to “Deep Contextualized Word Representations” (2018) source
Section 2 describes pretraining ELMo’s bidirectional language model on unlabeled data, and Section 3.4 identifies the large 1B Word Benchmark training corpus. Section 4 reports that adding ELMo improved results across six NLP benchmark tasks, with relative error reductions of 6–20%. This supports the claim as it relates to ELMo; the other cited models are not attributed to this source. (LLM, conf=0.92)
- Not matchedcites Radford et al., 2018
An author–year citation the paper’s reference list doesn’t settle. We don’t search these by name.
- Not checked — no full text of the sourcecites Devlin et al., 2019 matched to “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” (2019) source
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“SCIBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BIOBERT (Lee et al., 2019) results on biomedical tasks.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 7 Conclusion and Future Work
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The citation identifies the source of the reported BioBERT results used as a comparator, rather than attributing the SciBERT comparison to Lee et al. The Experimental results (§4.3) report BioBERT performance on biomedical named-entity recognition, relation extraction, and question-answering tasks. (LLM, conf=0.90)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In Table 2, we compare SCIBERT results with reported BIOBERT results on the subset of datasets included in (Lee et al., 2019) BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.1 Biomedical Domain
What the sources say
- Supportedcites Lee et al., 2019 matched to “BioBERT: a pre-trained biomedical language representation model for biomedical text mining” (2019) source
The results tables in our copy report BioBERT scores for BC5CDR, ChemProt, and JNLPBA (section 6; page unavailable). The citation identifies the source of those reported BioBERT results; the comparison with SciBERT is the citing authors’ own conclusion. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition, SCIBERT outperforms the SOTA on SciCite (Co-han et al., 2019).”
§ SCIBERT: A Pretrained Language Model for Scientific Text > 4.3 Multiple Domains
What the sources say
- Partly supportedcites Cohan et al., 2019 matched to “Structural Scaffolds for Citation Intent Classification in Scientific Publications” (2019) sourceExperimental — accuracy not yet measured
Table 4 (section 4.3 Results; page unavailable) shows that the paper’s BiLSTM-Attn model with ELMo and both scaffolds outperformed the listed previous SOTA on SciCite (84.0 versus 79.6 macro F1). It does not report a SciBERT result, so it does not support attributing that improvement to SciBERT. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
17 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“With their word-piece vocabulary, contextual embeddings, and self-supervised pre-training on large unlabelled corpora these models have essentially removed the need for most pre-processing beyond automatically-applied lowercasing. 14,31”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [31]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SciBERT, introduced by Beltagy et al. [16], was the first language model tailored for scientific texts.”
From Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph (2024) · cites this paper as [16]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Beltagy et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Although the first experiments with BERT ( 39 ) were disappointing, fine-tuning SciBERT ( 40 ) led to the best results and is the current production architecture of the platform.”
From scite: a smart citation index that displays the context of citations and classifies their intent using deep learning (2021) · cites this paper as [40]
The cited source says what the claim attributes to it.
In the citing paper
“Similar observations have been made in prior work, e.g. Beltagy et al. (2019) found that pre-training BERT on text from research papers improved its performance on scientific tasks.”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as Beltagy et al. (2019)
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“3617 Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 5×
“Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
MethodResultInfluentialGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al.”
Method
Cited by
Items 1,101–1,200 of 3,596
Retrieval augmented generation for building datasets from scientific literature
Maharana, P. R., Verma, A., & Joshi, K. (2025) · Journal of Physics Materials · doi:10.1088/2515-7639/ade1fa · Cited by 16
Gangwar, A., Jain, S., Sourav, S., & Modi, A. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.175 · Cited by 13
Handling Extreme Class Imbalance in Technical Logbook Datasets
Akhbardeh, F., Alm, C. O., Zampieri, M., & Desell, T. (2021) · doi:10.18653/v1/2021.acl-long.312 · Cited by 20
Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience
Chrysostomou, G., & Αλέτρας, Ν. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.645 · Cited by 21
“, 2019) for (SST, AG, SEMEVAL); SCIBERT (Beltagy et al., 2019) for”
Cross-lingual citations in English papers: a large-scale analysis of prevalence, usage, and impact
Saier, T., Färber, M., & Tsereteli, T. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00312-z · Cited by 15
Understanding progress in software citation: a study of software citation in the CORD-19 corpus
Du, C., Cohoon, J., Lopez, P., & Howison, J. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1022 · Cited by 15
“…the pipeline obtains PDFs, structures them using GROBID ( GROBID, 2021 ; Lopez, 2009 ), and runs the software mention recognizer ( Softcite Software Mention Recognizer, 2021 ) based on a fine-tuned SciBERT+CRF model ( Beltagy, Lo & Cohan, 2019 ) trained on the softcite-dataset ( Du et al., 2021a ).”
Miko Team: Deep Learning Approach for Legal Question Answering in ALQAC 2022
Van, H. N., Nguyen, D. M.-T., Nguyen, P., & Nguyen, L.-M. (2022) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse56063.2022.9953780 · Cited by 15
“In addition, many studies have shown that further training BERT or completely pre-training the model from scratch using a domain-specific corpus can significantly improve its performance, SciBERT [13] and BioBERT [14] are examples.”
BackgroundLIMEADE: From AI Explanations to Advice Taking
Lee, B. C. G., Downey, D., Lo, K., & Weld, D. S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3589345 · Cited by 14
Self-citationAutomatic Document Selection for Efficient Encoder Pretraining
Feng, Y., Xia, P., Durme, B. V., & Sedoc, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.647 · Cited by 13
Nie, Y., Huang, H., Wei, W., & Mao, X.-L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.336 · Cited by 22
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
Wan, Z., Yin, Y., Zhang, W., Shi, J., Shang, L., Chen, G., Jiang, X., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.441 · Cited by 17
Crema, C., Buonocore, T. M., Fostinelli, S., Parimbelli, E., Verde, F., Fundarò, C., Manera, M. R., Ramusino, M. C., Capelli, M., Costa, A. C., et al. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104557 · Cited by 18
“Another example is SciBERT [18], which exploits the original BERT architecture but trains the model from scratch on a different dataset (1.14 million scientific papers from Semantic Scholar).”
MethodInfluentialTowards an understanding and explanation for mixed-initiative artificial scientific text detection
Weng, L., Liu, S., Zhu, H., Sun, J., Kam-Kwai, W., Han, D., Zhu, M., & Chen, W. (2024) · Information Visualization · doi:10.1177/14738716241240156 · Cited by 14
Explanatory argument extraction of correct answers in resident medical exams
Goenaga, I., Atutxa, A., Gojenola, K., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102985 · Cited by 17
icsBERTs: Optimizing Pre-trained Language Models in Intelligent Customer Service
Liu, S., Peng, C., Wang, C., Chen, X., & Song, S. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.08.150 · Cited by 21
Extracting Summary Knowledge Graphs from Long DocumentsPreprint
Wu, Z., Koncel-Kedziorski, R., Ostendorf, M., & Hajishirzi, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09162 · Cited by 16
CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain SearchPreprint
Xiong, C., Liu, Z., Sun, S. Y., Dai, Z., Zhang, K., Shi, Y., Liu, Z., Poon, H., Gao, J., & Bennett, P. N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01580 · Cited by 16
Roy, K., Lokala, U., Khandelwal, V., & Sheth, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01222 · Cited by 15
“Models have demonstrated state of the art in RE based on the popular BERT language model, BioBERT [40], SciBERT [41], and XLNet Yang et al.”
MethodDiscourse-level Relation Extraction via Graph PoolingPreprint
Hsu, I.-H., Guo, X. J., Natarajan, P., & Peng, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00124 · Cited by 18
“This is a standard practice in distant supervision RE (Jia, Wong, and Considering that the n -ary dataset is in biomedical domain, we use SciBERT (Beltagy, Lo, and Cohan 2019).”
MethodPubGraph: A Large-Scale Scientific Knowledge GraphPreprint
Ahrabian, K., Du, X., Myloth, R. D., Ananthan, A. B. S., & Pujara, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.02231 · Cited by 14
“Recently, many large language models (LLM) have been introduced to tackle the problem of generating representations for scientific documents [1,6].”
BackgroundCan Theoretical Physics Research Benefit from Language Agents?Preprint
Lu, S., Jin, Z., Zhang, T. J., Kos, P., Cirac, J. I., & Schölkopf, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.06214 · Cited by 22
“While our primary focus is on theoretical physics, we acknowledge that in the long term, AI systems might also actively control experimental instrumentation, interpret sensory data in real-time, and adjust experimental parameters accordingly.”
El Morr, C., Maret, P., Muhlenbach, F., Dharmalingam, D., Tadesse, R., Creighton, A., Kundi, B., Buettgen, A., Mgwigwi, T., Dinca-Panaitescu, S., et al. (2021) · JMIR Formative Research · doi:10.2196/33335 · Cited by 12
Automatic extraction of ranked SNP-phenotype associations from text using a BERT-LSTM-based method
Bokharaeian, B., Dehghani, M., & Díaz, A. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05236-w · Cited by 12
“Therefore, recent biomedical text mining models rely primarily on the adapted versions of word representations such as SciBERT [13] for scientific texts and PubMedBERT-LSTM [14] for biomedical texts.”
Split-NER: Named Entity Recognition via Two Question-Answering-based Classifications
Arora, J., & Park, Y. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.36 · Cited by 21
“(2022); Zhu and Li (2022) and use SciBERT-scivocab-uncased (Beltagy et al., 2019) for BioNLP13CG since this dataset has chemical formulas and scientific entities4.”
MethodStöhr, F. (2024) · Journal of Computational Social Science · doi:10.1007/s42001-024-00286-3 · Cited by 11
“The authors trained their model on full texts only [37].”
BERT for Complex Systematic Review Screening to Support the Future of Medical Research
Hasny, M., Vasile, A.-P., Gianni, M., Bannach‐Brown, A., Nasser, M., Mackay, M., donovan, D., Šorli, J., Domocos, I., Dulloo, M., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34344-5_21 · Cited by 12
CRAFT Shared Tasks 2019 Overview –- Integrated Structure, Semantics, and Coreference
Baumgartner, W. A., Bada, M. A., Pyysalo, S., Ciosici, M. R., Hailu, N., Pielke-Lombardo, H., Regan, M., & Hunter, L. (2019) · Workshop on BioNLP Open Shared Tasks (BioNLP-OST) · doi:10.18653/v1/d19-5725 · Cited by 15
Biomedical Question Answering: A Survey of Methods and Datasets
Kaddari, Z., Mellah, Y., Berrich, J., Bouchentouf, T., & Belkasmi, M. G. (2020) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds50568.2020.9268742 · Cited by 18
Attend to Medical Ontologies: Content Selection for Clinical Abstractive SummarizationPreprint
Sotudeh, S., Goharian, N., & Filice, R. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00163 · Cited by 12
Transformer networks of human conceptual knowledge.Preprint
Bhatia, S., & Richie, R. (2020) · Psychology Review · doi:10.31234/osf.io/hs4ra · Cited by 55
Establishing Strong Baselines For TripClick Health Retrieval
Hofstätter, S., Althammer, S., Sertkan, M., & Hanbury, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_17 · Cited by 12
TaughtNet: Learning Multi-Task Biomedical Named Entity Recognition From Single-Task Teachers
Moscato, V., Postiglione, M., Sansone, C., & Sperlí, G. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3244044 · Cited by 15
“Recently, large-scale language models pre-trained on biomedical corpora and fine-tuned over BioNER datasets [7], [8], [9], [10], [11] have shown their remarkable potential to enhance the state-of-the-art of biomedical entity recognition and their promising prospects for improvement as the…”
BackgroundComparative Study of Pre-trained Language Models for Text Classification in Smart Agriculture Domain
Yadav, S., & Kaushik, A. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-0981-0_21 · Cited by 12
Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models
Wang, L., Li, Y., Miller, T., Bethard, S. J., & Savova, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.877 · Cited by 20
ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select
Zhuang, Y., Li, Y., Zhang, J., Yu, Y., Mou, Y., Chen, X., Song, L., & Zhang, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.46 · Cited by 18
“For SciREX and TDMS-NLP, we use SciBERT (Beltagy et al., 2019) as the encoder for all methods; for PubMed, we use ClinicalBERT (Alsentzer et al.”
MethodMa, K., Zheng, S., Tian, M., Qiu, Q., Tan, Y.-J., Hu, X., Li, H., & Xie, Z. (2023) · Earth Science Informatics · doi:10.1007/s12145-023-01112-6 · Cited by 12
“In addition, Beltagy et al. (2019) proposed SciBERT for texts in the scientific domain, and Araci (2019) proposed FinBERT for texts in the financial domain.”
BackgroundMethodEnhancing abusive language detection: A domain-adapted approach leveraging BERT pre-training tasks
Jarquín-Vásquez, H., Escalante, H. J., & Montes-y-Gómez, M. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.05.007 · Cited by 12
KPLLM-STE: Knowledge-enhanced and prompt-aware large language models for short-text expansion
Zhong, H., Zhang, Q., Li, W., Lin, R., & Tang, Y. Z. (2024) · World Wide Web · doi:10.1007/s11280-024-01322-y · Cited by 12
“SciBERT [35] is built through unsupervised pre-training on a broad dataset of scientific publications.”
An, J., Kim, J., Sunwoo, L., Baek, H., Yoo, S., & Lee, S. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-02913-z · Cited by 12
“In addition to KoBERT, we fine-tuned SciBERT [21], which is a BERT-based model trained on large-scale scientific articles.”
Klarák, J., Brito, A. C. M., Moreira, L. F., Silva, F. N., Amancio, D. R., Andok, R., de Oliveira, M. C. F., Bardosova, M., & Oliveira, O. N. (2025) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2025.141565 · Cited by 12
Varma, G., Yenukoti, R. K., Kumar, P. M., Ashrit, B. S., Purushotham, K., Subash, C., Ravi, S. K., Kurien, V., Aman, A., Manoharan, M., et al. (2025) · JMIR Cancer · doi:10.2196/64697 · Cited by 12
“These proprietary models are fine-tuned versions of SciBERT-cased [19], a domain-specific transformer model pretrained on scientific text.”
Using Transformer Based Ensemble Learning to Classify Scientific Articles
Ghosh, S., & Chopra, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-75015-2_11 · Cited by 12
“Finally, we bench-marked our model with one an existing SOTA model which is SciBERT [1].”
BackgroundMethodInfluentialPoetryBERT: Pre-training with Sememe Knowledge for Classical Chinese Poetry
Zhao, J., Bai, T., Wei, Y., & Wu, B. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-8991-9_26 · Cited by 15
SciTweets - A Dataset and Annotation Framework for Detecting Scientific Online Discourse
Hafid, S., Schellhammer, K. S., Bringay, S., Todorov, K., & Dietze, S. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557693 · Cited by 24
“Experimenting with different base models showed that SciBERT [4] provides superior performance on the tasks.”
MethodTowards semantic-driven boolean query formalization for biomedical systematic literature reviews
Pourreza, M., & Ensan, F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104928 · Cited by 17
Exploring Pre-Trained Language Models to Build Knowledge Graph for Metal-Organic Frameworks (MOFs)
An, Y., Greenberg, J., Hu, X. T., Kalinowski, A., Fang, X., Zhao, X., McClellan, S., Uribe‐Romo, F. J., Langlois, K., Furst, J. D., et al. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020568 · Cited by 16
“We also selected the SciBERT PLM [8] trained on general science related text.”
MethodORKG-Leaderboards: a systematic workflow for mining leaderboards as a knowledge graph
Kabongo, S., D’Souza, J., & Auer, S. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00366-1 · Cited by 25
“BERT, SciBERT, XLNet, and BigBird.”
BackgroundMethodInfluentialHierarchical Catalogue Generation for Literature Review: A Benchmark
Zhu, K., Feng, X., Feng, X., Wu, Y., & Qin, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.453 · Cited by 26
Ghosal, T., Varanasi, K. K., & Kordoni, V. (2023) · Scientometrics · doi:10.1007/s11192-023-04811-5 · Cited by 12
“We pass it onto the SciBERT (Beltagy, 2019).”
Neural Passage Quality Estimation for Static Pruning
Chang, X., Mishra, D., Macdonald, C., & MacAvaney, S. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657765 · Cited by 19
Framework for automation of short answer grading based on domain-specific pre-training
Bonthu, S., Sree, S. R., & Prasad, M. H. M. K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109163 · Cited by 15
Nazyrova, N., Chahed, S., Chaussalet, T., & Dwek, M. V. (2024) · International Conference on Pattern Recognition Systems (ICPRS) · doi:10.1109/icprs62101.2024.10677826 · Cited by 19
“Longformer’s attention mechanism is a combination of a windowed local-context self-attention and an end task motivated global attention that encodes inductive bias about the task [23].”
BackgroundTransformer-based active learning for multi-class text annotation and classification
Afzal, M., Hussain, J., Abbas, A., Hussain, M., Attique, M., & Lee, S. (2024) · Digital Health · doi:10.1177/20552076241287357 · Cited by 16
“According to Beltagy et al., 15 the SciBERT outperforms the baseline encoder representations from transformers (BERT) model on biomedical tasks.”
BackgroundMethodInfluentialSemi-automated title-abstract screening using natural language processing and machine learning
Pilz, M., Zimmermann, S., Friedrichs, J., Wördehoff, E., Ronellenfitsch, U., Kieser, M., & Vey, J. A. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02688-w · Cited by 13
“Furthermore, we applied SciBERT [34], a variant of BERT [35] that is specifically trained for scientific text.”
MethodPTM4Tag+: Tag recommendation of stack overflow posts with pre-trained models
He, J., Xu, B., Yang, Z., Han, D., Yang, C., Liu, J., Zhao, Z., & Lo, D. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10576-z · Cited by 13
“Recent works (Beltagy et al., 2019; Huang et al., 2020; Lee et al., 2019) have shown that the in-domain knowledge acquired by PTMs is valuable in improving the performance on domain-specific tasks, such as ClinicalBERT (Huang et al., 2020) for clinical text, SciBERT (Beltagy et al., 2019) for…”
Transformer-Based Tool for Automated Fact-Checking of Online Health Information: Development Study
Bayani, A., Ayotte, A., & Nikiema, J. N. (2025) · JMIR Infodemiology · doi:10.2196/56831 · Cited by 16
“In contrast, BioBERT and SciBERT are trained on more specialized texts, such as scientific publications [36,37].”
SC4ANM: Identifying optimal section combinations for automated novelty prediction in academic papers
Wu, W., Zhang, C., Bao, T., & Zhao, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126778 · Cited by 19
“We then fine-tuned the SciBERT (Beltagy et al., 2019) on this data to train a four-class classifier aimed at determining which section structure a given piece of academic text belongs to.”
InfluentialHou, J., Zheng, B., Li, H., & Li, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04617-1 · Cited by 13
“Moreover, models speci fi cally fi ne-tuned for scienti fi c documents, such as SciBERT (Beltagy, Lo, & Cohan, 2019), SPECTER (Cohan, Feldman, Beltagy, Downey, & Weld, 2020), and SPECTER2 (Singh, D ’ Arcy, Cohan, Downey, & Feldman, 2022), have achieved high performance in tasks closely related to SoS.”
McMinn, D., Grant, T., DeFord-Watts, L., Porkess, V., Lens, M., Rapier, C., Joe, W., Becker, T. A., & Bender, W. R. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf023 · Cited by 16
Özmen, B. B., Berber, İ., & Schwarz, G. S. (2025) · Aesthetic Surgery Journal · doi:10.1093/asj/sjaf049 · Cited by 12
LUK: Empowering Log Understanding With Expert Knowledge From Large Language Models
Ma, L., Yang, W., Jiang, S., Fei, B., Zhou, M., Li, S., Zhao, M., Xu, B., & Xiao, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3594046 · Cited by 17
“As the corpus and parameter scale grows, the capability of the language model is enhanced, and In-Context Learning paradigm [32] is proposed due to the expensive cost of fine-tuning whole models.”
D’Souza, J., & Auer, S. (2020) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/16292 · Cited by 17
GraphConfRec: A Graph Neural Network-Based Conference Recommender System
Iana, A., & Paulheim, H. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00021 · Cited by 16
“Therefore, we embed the papers’ titles and abstracts using the pre-trained transformer-based [31] language model SciBERT [32], with an uncased vocabulary.”
MethodDICE: A Drug Indication Classification and Encyclopedia for AI-Based Indication Extraction
Bhatt, A., Roberts, R. A., Chen, X., Li, T., Connor, S., Hatim, Q., Mikailov, M., Tong, W., & Liu, Z. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.711467 · Cited by 12
“The BioBERT model outperformed BERTbase on some domain-specific tasks such as biomedical named entity recognition (NER), and bio-Questions and answering with a 0.51–9.61% absolute improvement.”
BackgroundInfluentialSupercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer
Xiu, Z., Chen, J., Henao, R., Goldstein, B., Carin, L., & Tao, C. (2020) · PubMed · doi:10.48550/arxiv.2011.12454 · Cited by 15
“5,−1],μ1 = [2, 1],μ2 = [5, 2],μ3 = [1, 3],μ4 = [−2, 1],μ5 = [−3.”
MethodAN ONLINE REPOSITORY OF PYTHON RESOURCE FOR TEACHING MACHINE TRANSLATION TO TRANSLATION STUDENTS
Krüger, R. (2021) · CURRENT TRENDS IN TRANSLATION TEACHING AND LEARNING E · doi:10.51287/cttle20212 · Cited by 12
Biomedical Argument Mining Based on Sequential Multi-Task Learning
Si, J., Sun, L., Zhou, D., Ren, J., & Li, L. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3173447 · Cited by 12
“In [5], the utilization of various contextualized word embeddings were explored to address biomedical argument mining, such as BERT [26], BioBERT [27], SciBERT [28], RoBERTa [29], which are pre-trained on the large corpora in different domains.”
MethodClinical Abbreviation Disambiguation Using Clinical Variants of BERT
Wagh, A., & Khanna, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-36402-0_19 · Cited by 12
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
Hu, D., Hou, X., Du, X., Zhou, M., Jiang, L., Mo, Y., & Shi, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.468 · Cited by 21
Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?
Liu, S., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.459 · Cited by 23
Incorporating Attribution Importance for Improving Faithfulness Metrics
Zhao, Z., & Αλέτρας, Ν. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.261 · Cited by 26
SynKB: Semantic Search for Synthetic Procedures
Bai, F., Ritter, A., Madrid, P. B., Freitag, D., & Niekrasz, J. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.31 · Cited by 15
Query-based Instance Discrimination Network for Relational Triple Extraction
Tan, Z., Shen, Y., Hu, X., Zhang, W., Cheng, X., Lü, W., & Zhuang, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.523 · Cited by 13
SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains
Goswami, K., Lange, L., Araki, J., & Adel, H. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.197 · Cited by 23
Montanelli, L., Venugopal, V., Olivetti, E., & Latypov, M. I. (2024) · Integrating materials and manufacturing innovation · doi:10.1007/s40192-024-00344-8 · Cited by 12
Pan, L., Wang, X., Liang, Q., Shang, J., Liu, W., Xu, L., & Peng, S. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108478 · Cited by 13
“Second, the interpretability of the generalizable framework for cancer subtype analysis is improved through bioinformatics analysis of subtypes.”
MethodA Robust and Domain-Adaptive Approach for Low-Resource Named Entity Recognition
Yu, H., Mao, X.-L., Chi, Z., Wei, W., & Huang, H. (2020) · IEEE International Conference on Knowledge Graph (ICKG) · doi:10.1109/icbk50248.2020.00050 · Cited by 15
Domain-Specific Contextualized Embedding: A Systematic Literature Review
Yunianto, I., Permanasari, A. E., & Widyawan, W. (2020) · International Conferences on Information Technologies and Electrical Engineering · doi:10.1109/icitee49829.2020.9271752 · Cited by 16
Jiang, M., D’Souza, J., Auer, S., & Downie, J. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64452-9_1 · Cited by 12
tWT–WT: A Dataset to Assert the Role of Target Entities for Detecting Stance of Tweets
Kaushal, A., Saha, A., & Ganguly, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.303 · Cited by 26
“Following the previous works demonstrating that domain-specific weights result in improved performance (Gururangan et al., 2020; Lee et al., 2019; Beltagy et al., 2019; Lee and Hsiang, 2019; Liu et al., 2020; Müller et al., 2020), we use the Bert weights fine tuned on tweets (Nguyen et al., 2020)…”
MethodAnalyzing Research Trends in Inorganic Materials Literature Using NLP
Kuniyoshi, F., Ozawa, J., & Miwa, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86517-7_20 · Cited by 15
Position-based Prompting for Health Outcome Generation
Abaho, M., Bollegala, D., Williamson, P. R., & Dodd, S. R. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.3 · Cited by 15
K-12BERT: BERT for K-12 Education
Goel, V., Sahnan, D., Venktesh, V., Sharma, G., Dwivedi, D., & Mohania, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11647-6_123 · Cited by 14
Popovič, N., Laurito, W., & Färber, M. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.232 · Cited by 15
Smirnova, N., & Mayr, P. (2023) · Scientometrics · doi:10.1007/s11192-023-04806-2 · Cited by 18
“Thus, Beltagy et al.. (2019) released SciBERT a BERT-based language model pre-trained on a large number of unlabeled scientific articles from the computer science and biomedical domains.”
Pham-Hong, B.-T., & Chokshi, S. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.280 · Cited by 12
Spanish Biomedical Crawled Corpus: A Large, Diverse Dataset for Spanish Biomedical Language ModelsPreprint
Carrino, C. P., Armengol-Estapé, J., Bonet, O. D. G., Gutiérrez-Fandiño, A., González-Agirre, A., Krallinger, M., & Villegas, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07765 · Cited by 12
“In the case of the biomedical domain in English, BioBERT [5], and SciBERT [2] aggregated and processed different corpora (mostly proceeding from articles) to develop domain-specific BERT models.”
BackgroundMOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11
“MOPRD as a large-scale dataset of review comment is highly suitable for the task of review comment generation.”
BackgroundFinEAS: Financial Embedding Analysis of Sentiment
Gutiérrez-Fandiño, A., Kolm, P. N., Alonso, M. N. I., & Armengol-Estapé, J. (2022) · The Journal of Financial Data Science · doi:10.3905/jfds.2022.1.095 · Cited by 11
Boosting the predictive power of protein representations with a corpus of text annotations
Duan, H., Skreta, M., Cotta, L., Rajaonson, E. M., Dhawan, N., Aspuru‐Guzik, A., & Maddison, C. J. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01088-6 · Cited by 16
Wang, Z., & Liu, Y. (2022) · Journal of Information Science · doi:10.1177/01655515221106651 · Cited by 11
“Beltagy et al. released word embeddings from scientific publications called SciBERT to promote downstream scientific NLP tasks [40].”
Academic Expert Finding via $(k, \mathcal{P})$-Core based Embedding over Heterogeneous Graphs
Xu, X., Liu, J., Wang, Y., & Ke, X. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00030 · Cited by 21
A Joint Entity and Relation Extraction Model based on Efficient Sampling and Explicit Interaction
Li, Q., Yao, N., Zhou, N., Zhao, J. K., & Zhang, Y. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3604811 · Cited by 13
“For SciERC, we use the indomain scibert scivacb uncased encoder [4], a BERT model pretrained on a large corpus of scientiic papers.”
MethodLATTE: Label-efficient incident phenotyping from longitudinal electronic health records
Wen, J., Hou, J., Bonzel, C.-L., Zhao, Y., Castro, V. M., Gainer, V. S., Weisenfeld, D., Cai, T. A., Ho, Y.-L. A., Panickan, V. A., et al. (2023) · Patterns · doi:10.1016/j.patter.2023.100906 · Cited by 17
Liu, X., Fan, J., & Dong, S. (2020) · JMIR Medical Informatics · doi:10.2196/17644 · Cited by 11
Looking inside Noun Compounds: Unsupervised Prepositional and Free Paraphrasing
Ponkiya, G., Murthy, R., Bhattacharyya, P., & Palshikar, G. K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.386 · Cited by 11
Taju, S. W., Shah, S. M. A., & Ou, Y. (2021) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2021.107537 · Cited by 12
Efficient Argument Structure Extraction with Transfer Learning and Active Learning
Hua, X., & Wang, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.36 · Cited by 18
Tsubota, T., Bollegala, D., Zhao, Y., Jin, Y., & Kozu, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104185 · Cited by 11
On the Interpolation of Contextualized Term-based Ranking with BM25 for Query-by-Example Retrieval
Abolghasemi, A., Askari, A., & Verberne, S. (2022) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3539813.3545133 · Cited by 12
“TILDEBERT uses bert-base-uncased, TILDESciBERT uses SciBERT, and TILDEMSMARCO uses a TILDE which is already finetuned onMSMARCO; we use TILDEMSMARCO in a zero-shot setting on our data.”
MethodInfluential
Metadata
- Authors
- Beltagy, I., Lo, K., & Cohan, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- ORCID
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3613-3618
- OpenAlex
- W2970771982
- Semantic Scholar
- 156d217b0a911af97fa1b5a71dc909ccef7a8028
- Cited by
- 3,218
- References
- 33
- Citations by year
- 2026: 328 · 2025: 534 · 2024: 542 · 2023: 626 · 2022: 478 · 2021: 486 · 2020: 203 · 2019: 19
- FWCI
- 113.46
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- SciBERT, pre-trained language models, scientific texts, scientific publications, biomedical text, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.