Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Open accessThe DOI matches its Crossref record; 61 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These research efforts present promising results in extracting true ADE mentions from drug label text, indicating the suitability of using NLP methods for this task.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Healthcare represents a compelling domain for LLM use because the vast majority of actionable patient data is locked within unstructured clinical text, including electronic health records, clinical notes, and biomedical literature [4,5].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“These systems use biomedical embeddings (e.g., BioBERT, PubMedBERT [30]), specialized corpora (e.g., PubMed abstracts and clinical notes), and domain-specific knowledge bases (e.g., unified medical language system (UMLS), MIMIC). By customizing retrieval and generation to medical language, they enable reliable performance in tasks such as diagnostic decision support, clinical report summarization,”
From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [30]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“For example, a number of efforts have fine-tuned BERT models to classify drug label text blocks into predefined sections of the label and outperformed baseline models for this task22–23.”
From OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [2]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“State-of-the-art neural languagemodels are generally based on transformer architectures [53], following the recent success of BERT [16, 38].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“models in GPT [47, 48] and BERT [16, 38]. Their success is fueled by access to large text corpora, advanced hardware such as GPUs, and a culmination of advances in optimization methods, such as Adam [29] and slanted triangular learning rate [22]. Here, transfer learning goes from the pretrained language models to fine…”
MethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×
“nto question [38], but we include it in our pretraining experiments to enable a head-to-head comparison with prior BERT models. 2.4 Advanced Pretraining Techniques In the original formulation of BERT [16], the masked language model (MLM) simply selects random subwords to mask. When a word is only partially masked, it…”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodHanahan, D., & Weinberg, R. A. (2000) · Cell · doi:10.1016/s0092-8674(00)81683-9 · Cited by 29,219
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“This starts with simple stand-alone word embeddings [40, 45], and evolves into more sophisticated pretrained language models, from LSTM in ULMFiT [22] and ELMo [46] to transformer-based models in GPT [47, 48] and BERT [16, 38].”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 7×
“For example, the original BERT model was trained on Wikipedia1 and BookCorpus [59], and subsequent efforts have focused on crawling additional text from the Web to power even larger-scale pretraining [38, 49].”
BackgroundMethodInfluentialConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Research works printed but not linked: 14
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (Eds.). 2018. Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S18-1/.
- 10.5555/2145432.2145474 · doi:10.5555/2145432.2145474
- Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel M. Cer, and David Jurgens (Eds.). 2017. Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver, Canada, August 3-4, 2017. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S17-2/.
- Steven Bethard, Daniel M. Cer, Marine Carpuat, David Jurgens, Preslav Nakov, and Torsten Zesch (Eds.). 2016. Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S16-1/.
- Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (Eds.). 2019. Proceedings of the 18th BioNLP Workshop and Shared Task, BioNLP@ACL 2019, Florence, Italy, August 1, 2019. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/W19-50/.
- Mona T. Diab, Timothy Baldwin, and Marco Baroni (Eds.). 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. Association for Computational Linguistics. https://www.aclweb.org/anthology/volumes/S13-2/.
- 10.5555/2598938.2599127 · doi:10.5555/2598938.2599127
- 10.5555/1567594.1567610 · doi:10.5555/1567594.1567610
- 10.5555/2107691.2107693 · doi:10.5555/2107691.2107693
- Sun Kim, Rezarta Islamaj Dogan, Andrew Chatr-aryamontri, Mike Tyers, W. John Wilbur, and Donald C. Comeau. 2015. Overview of BioCreative V BioC track. In Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, Sevilla, Spain. 1–9.
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, G. P. Rodríguez, G. Tsatsaronis, et al. 2017. Overview of the BioCreative VI Chemical-Protein Interaction Track. In Proceedings of the 6th BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- 10.5555/645530.655813 · doi:10.5555/645530.655813
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- 10.5555/3454287.3454581 · doi:10.5555/3454287.3454581
Cited by
Items 1,001–1,034 of 1,034
Named entity recognition for de-identifying Spanish electronic health records
Moreno-Barea, F. J., López-García, G., Mesa, H., Ribelles, N., Alba, E., Jerez, J. M., & Veredas, F. J. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109576 · Cited by 10
Howell, L., Zarei, A., Wah, T. M., Chandler, J. H., Karthik, S., Court, Z., Ng, H. H. L., & McLaughlan, J. R. (2025) · European Radiology · doi:10.1007/s00330-025-11416-4 · Cited by 8
Infusing Multi-Hop Medical Knowledge Into Smaller Language Models for Biomedical Question Answering
Chen, J., Wei, Z., Shen, W., & Shang, R. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3547444 · Cited by 10
Kumawat, H., Sharan, A., & Verma, S. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.587 · Cited by 8
MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model
Reddy, K. S., Ragavenderan, N., Vasanth, K., Naik, G. N., Vishalakshi Prabhu, H., & Nagaraja, G. S. (2025) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v14.i3.pp2367-2378 · Cited by 7
“We evaluated the benchmark on the hallmarks of cancer (HoC) dataset [24], which is also available in BLURB [20].”
InfluentialBhasuran, B., Murugesan, G., & Natarajan, J. (2025) · Medinformatics · doi:10.47852/bonviewmedin52025348 · Cited by 8
“…success of BioBERT enabled the researchers to develop new transfer learning models using biomedical and clinical literature resulted in PubMedBERT [96], ClinicalBERT [97], MT-clinical BERT [98], Umlsbert [99], ELECTRAMed [95], BioMegatron [100] This final model also gives insights that we can use…”
Wu, W., Wang, C., Chen, L., Yin, M., Zhu, Y., Fu, K., Ye, J., Xiong, H., & Wang, Z. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737138 · Cited by 15
“For evaluation metrics, we employ BLEU score [77], ROUGE score [51], METEOR score [9], BERT score [105] calculated by PubMedBERT [28], and Accuracy to assess performance across two types of tasks: open-ended generation and closed-set answer, respectively.”
Talati, I. A., Chaves, J. M. Z., Das, A., Banerjee, I., & Rubin, D. L. (2025) · American Journal of Roentgenology · doi:10.2214/ajr.25.33469 · Cited by 8
Kong, C., Xu, R., Chen, J., & Yin, Z. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755548 · Cited by 7
“However, traditional PLMs often underperform in the medical domain due to their limited grasp of medical knowledge, prompting the development of specialized Med-PLMs [9, 16] pre-trained on medical domain texts.”
InfluentialA Programmable Ontology Encompassing the Functional Logic of the Drosophila Brain
Lazar, A. A., Türkcan, M. K., & Zhou, Y. (2022) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2022.853098 · Cited by 8
Smooth Operators for Effective Systematic Review Queries
Scells, H., Schlatt, F., & Potthast, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591768 · Cited by 7
“We chose PubMedBERT as it was pre-trained using medical terminology and should have a good representation of documents in PubMed.”
MethodComposition operators for modeling languages: A literature review
Pfeiffer, J., Rumpe⋆, B., Schmalzing, D., & Wortmann, A. (2023) · Journal of Computer Languages · doi:10.1016/j.cola.2023.101226 · Cited by 7
Biomedical Language Models are Robust to Sub-optimal Tokenization
Gutiérrez, B. J., Sun, H., & Su, Y. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.32 · Cited by 7
End-to-end clinical temporal information extraction with multi-head attention
Miller, T., Bethard, S. J., Dligach, D., & Savova, G. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.28 · Cited by 15
“Our model uses a multi-task learning setup built on top of a pre-trained transformer encoder, and specifically the PubmedBERT-base-uncased-abstract model (Gu et al., 2021).”
MethodComparing and combining some popular NER approaches on Biomedical tasks
Verma, H., Bergler, S., & Tahaei, N. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.24 · Cited by 9
Kanakarajan, K. R., & Sankarasubbu, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.137 · Cited by 23
“When applying the same model architecture to domains such as biomedical, they also achieved state-of-the-art results on domain specific tasks; domain specific trained models like BioBERT Lee et al. (2020), PubMedBERT Gu et al. (2021), BioELECTRA raj Kanakarajan et al. (2021) further improved performance.”
BackgroundMultimodal Clinical Prediction with Unified Prompts and Pretrained Large-Language Models
Winston, C., Winston, C., Winston, C., Winston, C., & Winston, C. (2024) · 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00108 · Cited by 6
Wang, Z., Wu, D., Nguyen, Q., & Wang, K. (2026) · Medicine Bulletin · doi:10.1002/mdb2.70033 · Cited by 6
“Prominent examples include Phenolyzer, which integrates functional interaction networks 15 ; Phen2Gene 16 , which is a probabilistic model that incorporates HPO annotations and gene prioritization; Exomiser 17 , VarElect 18 and AMELIE 19 , which combines phenotype similarity and genomic variant filtering; as well as o…”
InfluentialLitFM: A Retrieval Augmented Structure-aware Foundation Model For Citation Graphs
Zhang, J., Maatouk, A., Chen, J., Bui, N., Xie, Q., Tassiulas, L., Xu, H., Shao, J., & Ying, R. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3737028 · Cited by 6
“Despite these advancements, these models tend to generate fabricated content, leading to hallucinations that are particularly problematic in literature tasks.”
Gao, J., Chen, G., O’Rourke, A. P., Caskey, J., Carey, K. A., Oguss, M., Stey, A., Dligach, D., Miller, T., Mayampurath, A., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae071 · Cited by 5
Self-citationDevelopment of a Natural Language Processing Model for Extracting Kidney Biopsy Pathology Diagnoses
Bobart, S. A., Hsu, E., Potter, T., Truong, L., Waterman, A., Jones, S., & Shafi, T. (2025) · Kidney Medicine · doi:10.1016/j.xkme.2025.101047 · Cited by 5
MedRedQA for Medical Consumer Question Answering: Dataset, Tasks, and Neural Baselines
Nguyen, V., Karimi, S., Rybinski, M., & Xing, Z. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.42 · Cited by 5
“We use the PubMEDBERT model by Gu et al. (2021) as our BiEnc, pretrained on the PubMed corpus (NCBI, 2023), without additional finetuning.”
MethodExplanatory argumentation in natural language for correct and incorrect medical diagnoses
Molinet, B., Marro, S., Cabrio, E., & Villata, S. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00306-1 · Cited by 4
“For the entity detection task, we experiment with different transformer-based Language Models such as BERT [87], SciBERT [53], BioBERT [54], PubMed-BERT [55] and UmlsBERT [57] initialized with their respective pre-trained weights.”
MethodAutomatic literature screening using the PAJO deep-learning model for clinical practice guidelines
Lin, Y., Li, J., Xiao, H., Zheng, L., Xiao, Y., Song, H., Fan, J., Xiao, D., Ai, D., Fu, T., et al. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02328-8 · Cited by 4
“Fortunately, because our research field is closely related to biomedicine, we noted that a pretrained PubMedBERT NLP already exists in that field [18].”
BackgroundXu, N. Y., Litake, O., Tully, J. L., Meineke, M. N., Sinha, A., Meyer, M., & Gabriel, R. A. (2025) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-025-01401-z · Cited by 2
“The two domain-specific language models included versions specifically trained on clinical text and biomedical literature, ClinicalBERT and PubMedBERT, respectively [13, 14].”
Liu, X., Chen, Y.-W., Xu, X., Smith, D., & Fan, F. (2026) · Toxicological Sciences · doi:10.1093/toxsci/kfag021 · Cited by 1
Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safety
Kim, Y., Jeong, H., Park, C., Park, E., Zhang, H., Liu, X., Lee, H., McDuff, D., Ghassemi, M., Breazeal, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.12482 · Cited by 1
“One line of work focuses on training with domain-specific corpora to embed medical knowledge directly into model weights (Gu et al., 2021).”
Self-citationXu, D., Chen, S., & Miller, T. (2021) · arXiv · doi:10.48550/arxiv.2111.13726 · Cited by 1
“We select 8 transformer models pre-trained with different settings as our initialization points: BERT-base-cased, BERT-base-uncased, BERT-large-cased, BERT-large-uncased, BioBERT-base-cased, BioRedditBERT-base-uncased, BioClinicalBERT-base-cased, PubMedBERT-base-uncased.”
MethodMichalowski, M., Wilk, S., Bauer, J. M., Carrier, M., Viktor, H., & Michalowski, W. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-95838-0_29 · Cited by 1
Rahgozar, A., & Mortezaagha, P. (2026) · arXiv · doi:10.48550/arxiv.2608.14551
“Each paper is represented by a BERT embedding of its title and abstract (using PubMedBERT; Gu et al., 2021) and a GCN embedding learned over a k -nearest-neighbour similarity graph with optional citation edges.”
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
Pitre, T., Zeraatkar, D., Granton, J., Busse, J., Rochwerg, B., & Guyatt, G. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112514 · Cited by 2
A Curated Dataset for Question Answering on Laboratory Test Reference Ranges and Interpretation
Bhasuran, B., Jin, Q., Deville, A., Wu, Y., Hanna, K., Lu, Z., & He, Z. (2026) · Scientific Data · doi:10.1038/s41597-026-07554-z
“Early efforts primarily involved training smaller, domain-specific models such as BioBERT 25 , PubMedBERT 26 , and ClinicalBERT 27 .”
Artenie, A. C., Morar, C. D., Popa, C.-A., & Moldovan, O. G. (2026) · Applied Sciences · doi:10.3390/app16189296
Metadata
- Authors
- Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H.
- Affiliations
- Microsoft Research, Redmond, WA
- Microsoft (United States) (US)
- ORCID
- Venue
- ACM Transactions on Computing for Healthcare
- Published
- 2021-10-15
- Volume
- 3
- Issue
- 1
- Pages
- 1-23
- ISSN
- 2637-8051 (ISSN-L)
- 2691-1957
- OpenAlex
- W3046375318
- Cited by
- 2,170
- References
- 63
- Citations by year
- 2026: 402 · 2025: 512 · 2024: 456 · 2023: 411 · 2022: 243 · 2021: 125
- FWCI
- 109.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Computing for Healthcare (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CNS-1738645 and DRL-1837446)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- biomedical natural language processing, BERT, fine-tuning, named entity recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.