Ascle—A Python Natural Language Processing Toolkit for Medical Text Generation: Development and Evaluation Study
Open accessPublished version of a preprint
Abstract
Background: Medical texts present significant domain-specific challenges, and manually curating these texts is a time-consuming and labor-intensive process. To address this, natural language processing (NLP) algorithms have been developed to automate text processing. In the biomedical field, various toolkits for text processing exist, which have greatly improved the efficiency of handling unstructured text. However, these existing toolkits tend to emphasize different perspectives, and none of them offer generation capabilities, leaving a significant gap in the current offerings. Objective: This study aims to describe the development and preliminary evaluation of Ascle. Ascle is tailored for biomedical researchers and clinical staff with an easy-to-use, all-in-one solution that requires minimal programming expertise. For the first time, Ascle provides 4 advanced and challenging generative functions: question-answering, text summarization, text simplification, and machine translation. In addition, Ascle integrates 12 essential NLP functions, along with query and search capabilities for clinical databases. Methods: We fine-tuned 32 domain-specific language models and evaluated them thoroughly on 27 established benchmarks. In addition, for the question-answering task, we developed a retrieval-augmented generation (RAG) framework for large language models that incorporated a medical knowledge graph with ranking techniques to enhance the reliability of generated answers. Additionally, we conducted a physician validation to assess the quality of generated content beyond automated metrics. Results: The fine-tuned models and RAG framework consistently enhanced text generation tasks. For example, the fine-tuned models improved the machine translation task by 20.27 in terms of BLEU score. In the question-answering task, the RAG framework raised the ROUGE-L score by 18% over the vanilla models. Physician validation of generated answers showed high scores for readability (4.95/5) and relevancy (4.43/5), with a lower score for accuracy (3.90/5) and completeness (3.31/5). Conclusions: This study introduces the development and evaluation of Ascle, a user-friendly NLP toolkit designed for medical text generation. All code is publicly available through the Ascle GitHub repository. All fine-tuned language models can be accessed through Hugging Face.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Technical limitations inherent in LLMs, including their understanding of complex medical terminologies and nuances, may not match the expertise of experienced clinicians, possibly limiting the scope of applicability.
“Meanwhile, the technical limitations inherent in LLMs, including their understanding of complex medical terminologies and nuances [51], may not match the expertise of experienced clinicians, possibly limiting the scope of applicability.”
From Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: simulation study (2024) · cites this paper as [51]
The Introduction (page not provided) says medical texts present challenges involving ambiguity and specialized terminology, and Discussion > Future Work says LLM-generated content can be unfaithful or inconsistent. Results > Physician Validation also reports incomplete medical answers. The paper does not specifically establish that LLMs understand medical terminology less well than experienced clinicians or that this limits their scope of applicability. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Rosenbloom, L. (2019) · Charleston Advisor · doi:10.5260/chara.21.2.8 · Cited by 16,663 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · see citing works
Kincaid, J. P., Fishburne, Jr., Robert P., R., Richard L., C., & Brad, S. (1975) · doi:10.21236/ada006655 · Cited by 3,497 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · see citing works
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1912.08777 · Cited by 1,206 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5034 · Cited by 772 · see citing works
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., & Goharian, N. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2097 · Cited by 658 · see citing works
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · see citing works
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · see citing works
BioWordVec, improving biomedical word embeddings with subword information and MeSH
Zhang, Y., Chen, Q., Yang, Z., Lin, H., & Lu, Z. (2019) · Scientific Data · doi:10.1038/s41597-019-0055-0 · Cited by 455 · see citing works
Self-citationLarge language models in health care: Development, applications, and challenges
Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., & Liu, N. (2023) · Health Care Science · doi:10.1002/hcs2.61 · Cited by 366 · see citing works
Self-citationOpportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Self-citationTowards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Neural Natural Language Processing for unstructured data in electronic health records: A review
Li, I., Pan, J. N., Goldwasser, J., Verma, N., Wong, W. P., Nuzumlalı, M. Y., Rosand, B., Li, Y., Zhang, M. M., Chang, D., et al. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2022.100511 · Cited by 262 · see citing works
Self-citationBioSentVec: creating sentence embeddings for biomedical texts
Chen, Q.-Y., Peng, Y., & Lu, Z.-Y. (2018) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi.2019.8904728 · Cited by 241 · see citing works
Self-citationSciFive: a text-to-text transformer model for biomedical literaturePreprint
Phan, L., Anibal, J., Tran, H., Chanana, S., Bahadroglu, E., Peltekian, A., & Altan-Bonnet, G. (2021) · arXiv.org · doi:10.48550/arxiv.2106.03598 · Cited by 203 · see citing works
Text summarization in the biomedical domain: A systematic review of recent research
Mishra, R., Bian, J., Marcelo Fiszman, B., Weir, C. R., Jonnalagadda, S., Mostafa, J., & Fiol, G. D. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.009 · Cited by 201 · see citing works
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
Xu, C., Guo, D., Duan, N., & McAuley, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.385 · Cited by 191 · see citing works
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Wen, X., Beltagy, I., Carenini, G., & Cohan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.360 · Cited by 155 · see citing works
Wang, S., McDermott, M. B. A., Chauhan, G., Ghassemi, M., Hughes, M. C., & Naumann, T. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384469 · Cited by 149 · see citing works
BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
Yuan, H., Yuan, Z., Gan, R., Zhang, J., Xie, Y., & Yu, S. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.9 · Cited by 142 · see citing works
Capabilities of Gemini Models in Medicine
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18416 · Cited by 117 · see citing works
Biomedical and clinical English model packages for the Stanza Python NLP library
Zhang, Y., Zhang, Y., Qi, P., Manning, C. D., & Langlotz, C. P. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab090 · Cited by 112 · see citing works
Marian: Fast Neural Machine Translation in C++
Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., et al. (2018) · doi:10.18653/v1/p18-4020 · Cited by 90 · see citing works
Question-driven summarization of answers to consumer health questions
Savery, M. E., Abacha, A. B., Gayen, S., & Demner‐Fushman, D. (2020) · Scientific Data · doi:10.1038/s41597-020-00667-z · Cited by 88 · see citing works
Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python
Eyre, H., Chapman, A. B., Peterson, K., Shi, J., Alba, P. R., Jones, M., Box, T. L., DuVall, S. L., & Patterson, O. V. (2021) · PubMed Central · doi:10.48550/arxiv.2106.07799 · Cited by 81 · see citing works
Paragraph-level Simplification of Medical Texts
Devaraj, A., Marshall, I., Wallace, B., & Li, J. J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.395 · Cited by 79 · see citing works
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30205 · Cited by 66 · see citing works
Faithful AI in Medicine: A Systematic Review with Large Language Models and BeyondPreprint
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · medRxiv · doi:10.1101/2023.04.18.23288752 · Cited by 57 · see citing works
Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature
Goldsack, T., Zhang, Z., Lin, C., & Scarton, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.724 · Cited by 45 · see citing works
Bridging the Gap Between Consumers’ Medication Questions and Trusted Answers
Abacha, A. B., Mrabet, Y., Sharp, M. J., Goodwin, T. R., Shooshan, S. E., & Demner‐Fushman, D. (2019) · Studies in health technology and informatics · doi:10.3233/shti190176 · Cited by 44 · see citing works
Transformers-sklearn: a toolkit for medical language understanding with transformer-based models
Yang, F., Wang, X., Ma, H., & Li, J. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01459-0 · Cited by 40 · see citing works
A Research Agenda for Using Machine Translation in Clinical Medicine
Khoong, E. C., & Rodriguez, J. A. (2022) · Journal of General Internal Medicine · doi:10.1007/s11606-021-07164-y · Cited by 32 · see citing works
Disparities in clinical studies of AI enabled applications from a global perspective
Yang, R., Nair, S. V., Ke, Y., D’Agostino, D., Liu, M., Ning, Y., & Liu, N. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01212-7 · Cited by 29 · see citing works
Self-citationKe, Y., Yang, R., & Liu, N. (2024) · Journal of Medical Internet Research · doi:10.2196/48330 · Cited by 15 · see citing works
Self-citationA Survey for Biomedical Text Summarization: From Pre-trained to Large Language ModelsPreprint
Xie, Q., Luo, Z., Wang, B., & Ananiadou, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08763 · Cited by 10 · see citing works
Survey: Deep Learning Concepts and Techniques for Electronic Health Record
Alaiad, A., Duwairi, R., & Fraihat, M. (2018) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa.2018.8612827 · Cited by 9 · see citing works
Generative Artificial Intelligence in Healthcare: Ethical Considerations and Assessment Checklist
Ning, Y., Teixayavong, S., Shang, Y., Savulescu, J., Nagaraj, V., Miao, D., Mertens, M., Ting, D. S. W., Ong, J. C. L., Liu, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.02107 · Cited by 9 · see citing works
Self-citationRetrieval-Augmented Generation for Generative Artificial Intelligence in MedicinePreprint
Yang, R., Ning, Y., Keppo, E., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.12449 · Cited by 6 · see citing works
Self-citationKe, Y. H., Yang, R., Lie, S. A., Lim, T. X. Y., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024) · arXiv · doi:10.48550/arxiv.2401.14589 · Cited by 5 · see citing works
Self-citationKG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques
Yang, R., Liu, H., Marrese-Taylor, E., Zeng, Q., Ke, Y. H., Li, W., Cheng, L., Chen, Q., Caverlee, J., Matsuo, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05881 · Cited by 5 · see citing works
Self-citationLi, C., Mowery, D. L., Ma, X., Yang, R., Vurgun, U., Hwang, S., Donnelly, H. K., Bandhey, H., Akhtar, Z., Senathirajah, Y., et al. (2024) · medRxiv · doi:10.1101/2024.02.04.24302242 · Cited by 4 · see citing works
Self-citationLeveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP EducationPreprint
Yang, R., Yang, B., Ouyang, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.14293 · Cited by 3 · see citing works
Self-citationExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2023) · arXiv · doi:10.48550/arxiv.2309.07852 · Cited by 2 · see citing works
Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and EvaluationPreprint
Luo, J., Zheng, Z., Ye, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.02420 · Cited by 2 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 7
- [7]Devlin J Chang MW Lee K Toutanova K BERT: pre-training of deep bidirectional transformers for language understanding ArXiv. Preprint posted online on May 24, 2019 2019 1 16 10.5260/chara.21.2.8 · DOI
- [26]Abacha AB Agichtein E Pinter Y Demner-Fushman D Overview of the medical question answering task at TREC 2017 LiveQA 2018 Text REtrieval Conference (TREC) May 2017 Gaithersburg, MD, USA
- [48]Lin CY ROUGE: a package for automatic evaluation of summaries 2004 Text Summarization Branches Out July 2004 Barcelona, Spain Association for Computational Linguistics 74 81
- [49]Zhang T Kishore V Wu F Weinberger KQ Artzi Y BERTScore: evaluating text generation with BERT ArXiv. Preprint posted online on February 24, 2020 2020 10.5260/chara.21.2.8 · DOI
- [52]Rohde T Wu X Liu Y Hierarchical learning for generation with long source sequences ArXiv. Preprint posted online on September 16, 2021 2021 10.5260/chara.21.2.8 · DOI
- [54]Mihalcea R Tarau P TextRank: Bringing Order into Text 2004 Proceedings of the Conference on Empirical Methods in Natural Language Processing August 01, 2004 Barcelona, Spain Association for Computational Linguistics 404 411
- [57]Gao F Jiang H Yang R Zeng Q Lu J Blum M Liu D She T Jiang Y Li I Large Language models on wikipedia-style survey generation: an evaluation in NLP concepts ArXiv. Preprint posted online on May 23, 2024 2024 10.48550/ARXIV.2308.10410 · DOI
Cited by
Sorted by citations
Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024) · Journal of Medical Internet Research · doi:10.2196/59439 · Cited by 64 · see citing works
“Meanwhile, the technical limitations inherent in LLMs, including their understanding of complex medical terminologies and nuances [51], may not match the expertise of experienced clinicians, possibly limiting the scope of applicability.”
Self-citation
- Authors
- Yang, R., Zeng, Q., You, K., Qiao, Y., Huang, L., Hsieh, C.-C., Rosand, B., Goldwasser, J., Dave, A., Keenan, T., Ke, Y., Hong, C., Liu, N., Chew, E., Radev, D., Lu, Z., Xu, H., Chen, Q., & Li, I.
- Affiliations
- Duke-NUS Medical School (SG)
- Northwestern University (US)
- National Institutes of Health (US)
- Yale University (US)
- Yale New Haven Hospital (US)
- Singapore General Hospital (SG)
- Duke University (US)
- National University of Singapore (SG)
- The University of Tokyo (JP)
- Corresponding author
- Li, Irene
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-10-03
- Volume
- 26
- Pages
- e60601
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11487205
- OpenAlex
- W4400644060
- Cited by
- 16
- References
- 56
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 2
- FWCI
- 2.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (7k99lm014024-02, UL1TR001863)
- National Institute of Standards and Technology
- Japan Society for the Promotion of Science (24K20832)
- U.S. National Library of Medicine (K99 LM014024)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- clinical text generation, question answering, text summarization, text simplification, machine translation, retrieval-augmented generation, medical knowledge graph, fine-tuned language models, readability, biomedical text processing
- MeSH terms
- Algorithms, Humans, Natural Language Processing, Software
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
