Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals
Open accessAbstract
Publicly accessible benchmarks that allow for assessing and comparing model performances are important drivers of progress in artificial intelligence (AI). While recent advances in AI capabilities hold the potential to transform medical practice by assisting and augmenting the cognitive processes of healthcare professionals, the coverage of clinically relevant tasks by AI benchmarks is largely unclear. Furthermore, there is a lack of systematized meta-information that allows clinical AI researchers to quickly determine accessibility, scope, content and other characteristics of datasets and benchmark datasets relevant to the clinical domain. To address these issues, we curated and released a comprehensive catalogue of datasets and benchmarks pertaining to the broad domain of clinical and biomedical natural language processing (NLP), based on a systematic review of literature and. A total of 450 NLP datasets were manually systematized and annotated with rich metadata, such as targeted tasks, clinical applicability, data types, performance metrics, accessibility and licensing information, and availability of data splits. We then compared tasks covered by AI benchmark datasets with relevant tasks that medical practitioners reported as highly desirable targets for automation in a previous empirical study. Our analysis indicates that AI benchmarks of direct clinical relevance are scarce and fail to cover most work activities that clinicians want to see addressed. In particular, tasks associated with routine documentation and patient data administration workflows are not represented despite significant associated workloads. Thus, currently available AI benchmarks are improperly aligned with desired targets for AI automation in clinical settings, and novel benchmarks should be created to fill these gaps.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clinical FMs lack the shared evaluation frameworks and datasets that have underpinned progress in other fields such as natural language processing and computer vision.
“Clinical FMs lack the shared evaluation frameworks and datasets20 that have underpinned progress in other fields, such as natural language processing (NLP) and computer vision21.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [20]
The Introduction describes public benchmarks as shared, standardized ways to compare AI models, and the Analysis of benchmark datasets finds that few benchmarks have direct clinical relevance. But the paper examines clinical and biomedical AI benchmarks, not clinical foundation models specifically; it also identifies some existing clinical benchmarks rather than a complete lack of them. Page numbers were not available in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · see citing works
The future of digital health with federated learning
Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 3,119 · cited 1× · see citing works
“[19] However, while such approaches hold promise to unlock the potential value of sensible data, widespread application is yet to be awaited and will require adequate incentivisation.”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402 · see citing works
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
On the Automatic Generation of Medical Imaging Reports
Jing, B., Xie, P., & Xing, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1240 · Cited by 657 · see citing works
EDAM: an ontology of bioinformatics operations, types of data and identifiers, topics and formats
Ison, J., Kalaš, M., Jonassen, I., Bolser, D. M., Uludağ, M., McWilliam, H., MALONE, J. M., López, R., Pettifer, S. R., & Rice, P. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt113 · Cited by 284 · see citing works
Erickson, S. M., Rockwern, B., Koltov, M., McLean, R. M., & for the Medical Practice and Quality Committee of the American College of Physicians* (2017) · Annals of Internal Medicine · doi:10.7326/m16-2697 · Cited by 238 · see citing works
Woolhandler, S., & Himmelstein, D. U. (2014) · International Journal of Health Services · doi:10.2190/hs.44.4.a · Cited by 209 · see citing works
emrQA: A Large Corpus for Question Answering on Electronic Medical Records
Pampari, A., Raghavan, P., Liang, J., & Peng, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1258 · Cited by 191 · see citing works
Measuring the Occupational Impact of AI: Tasks, Cognitive Abilities and AI Benchmarks
Tolan, S., Pesole, A., Martínez‐Plumed, F., Fernández‐Macías, E., Hernández‐Orallo, J., & Gómez, E. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12647 · Cited by 181 · see citing works
Johnson, A., Pollard, T., & Mark, R. (2015) · PhysioNet · doi:10.13026/c2xw26 · Cited by 165
An empirical evaluation of deep learning for ICD-9 code assignment using MIMIC-III clinical notes
Huang, J., Osorio, C., & Sy, L. W. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.05.024 · Cited by 128 · see citing works
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman, S. R., & Dahl, G. E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.385 · Cited by 117 · see citing works
FindZebra: A search engine for rare diseases
Dragusin, R., Petcu, P., Lioma, C., Larsen, B., Jørgensen, H. L., Cox, I. J., Hansen, L. K., Ingwersen, P., & Winther, O. (2013) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.01.005 · Cited by 103 · see citing works
A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images
Messina, P., Pino, P., Parra, D., Soto, Á., Besa, C., Uribe, S. A., Andía, M. E., Cristián Tejos, N., Prieto, C., & Daniel Capurro, N. (2022) · ACM Computing Surveys · doi:10.1145/3522747 · Cited by 103 · see citing works
Research community dynamics behind popular AI benchmarks
Martínez-Plumed, F., Barredo, P., hÉigeartaigh, S. Ó., & Hernández-Orallo, J. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00339-6 · Cited by 86 · see citing works
PathVQA: 30000+ Questions for Medical Visual Question Answering
He, X., Zhang, Y., Mou, L., Xing, E., & Xie, P. (2020) · arXiv · doi:10.48550/arxiv.2003.10286 · Cited by 79 · see citing works
Benchmarking for biomedical natural language processing tasks with a domain specific ALBERT
Naseem, U., Dunn, A. G., Khushi, M., & Kim, J. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04688-w · Cited by 68 · see citing works
Blagec, K., Barbosa-Silva, A., Ott, S., & Matthias Samwald, A. (2022) · Scientific Data · doi:10.1038/s41597-022-01435-x · Cited by 57 · see citing works
Self-citationInterpretable deep learning to map diagnostic texts to ICD-10 codes
Atutxa, A., de Ilarraza, A. D., Gojenola, K., Oronoz, M., & Perez-de-Viñaspre, O. (2019) · Int. J. Medical Informatics · doi:10.1016/j.ijmedinf.2019.05.015 · Cited by 55 · see citing works
Classification of radiology reports using neural attention models
Shin, B., Chokshi, F. H., Lee, T. J., & Choi, J. D. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966408 · Cited by 48 · see citing works
A Comparison of Deep Learning Methods for ICD Coding of Clinical Records
Moons, E., Khanna, A., Akkasi, A., & Moens, M. (2020) · Applied Sciences · doi:10.3390/app10155262 · Cited by 47 · see citing works
Sizing the Problem of Improving Discovery and Access to NIH-Funded Data: A Preliminary Study
Read, K., Sheehan, J. R., Huerta, M. F., Knecht, L. S., Mork, J. G., Humphreys, B. L., & NIH Big Data Annotator Group (2015) · PLoS ONE · doi:10.1371/journal.pone.0132735 · Cited by 36 · see citing works
Narayanan, B. N., Hardie, R. C., Silva, M. S. D., & Kueterman, N. K. (2020) · Journal of medical imaging · doi:10.1117/1.jmi.7.3.034501 · Cited by 33 · see citing works
Towards better healthcare: What could and should be automated?
Fruehwirt, W., & Duckworth, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120967 · Cited by 31 · see citing works
SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search
MacAvaney, S., Cohan, A., & Goharian, N. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.341 · Cited by 23 · see citing works
Nentidis, A., Krithara, A., Παλιούρας, Γ., Gascó, L., & Krallinger, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_53 · Cited by 9 · see citing works
ImageCLEF 2022: Multimedia Retrieval in Medical, Nature, Fusion, and Internet Applications
de Herrera, A. G. S., Ionescu, B. E., Müller, H., Péteri, R., Abacha, A. B., Friedrich, C. M., Rückert, J., Bloch, L., Brüngel, R., Idrissi-Yaghir, A., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_48 · Cited by 3 · see citing works
Repurposing TREC-COVID Annotations to Answer the Key Questions of CORD-19Preprint
Heaton, C., & Mitra, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12353 · Cited by 3 · see citing works
A living catalogue of artificial intelligence datasets and benchmarks for medical decision making
Blagec, K., Jakob, K., & Matthias Samwald, A. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4647823 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- A. Ben Abacha, V.V. Datla, S.A. Hasan, D. Demner-Fushman, H. Müller, Overview of the VQA-Med Task at ImageCLEF 2020: Visual Question Answering and Generation in the Medical Domain., CEUR Workshop Proceedings; 2020.
- V. Nguyen, S. Karimi, S. Falamaki, C. Paris, Benchmarking Clinical Decision Support Search. ArXiv 2018. 10.48550/arxiv.1801.09322. · DOI
- D. Mowery, ShAReCLEF eHealth 2013: Natural Language Processing and Information Retrieval for Clinical Care, PhysioNet 2020. 10.13026/0zsp-0e97. · DOI
- P. Messina, P. Pino, D. Parra, A. Soto, C. Besa, S. Uribe, et al. A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical Images. ArXiv 2020. 10.48550/arxiv.2010.10563. · DOI
Cited by
Sorted by citations
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
Javed, H., El–Sappagh, S., & Abuhmed, T. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-11005-9 · Cited by 205 · see citing works
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · see citing works
Mapping global dynamics of benchmark creation and saturation in artificial intelligence
Ott, S., Barbosa-Silva, A., Blagec, K., Brauner, J., & Samwald, M. (2022) · Nature Communications · doi:10.1038/s41467-022-34591-0 · Cited by 73 · see citing works
Self-citationLarge language models for science and medicine
Telenti, A., Auli, M., Hie, B., Maher, C., Saria, S., & Ioannidis, J. P. A. (2024) · European Journal of Clinical Investigation · doi:10.1111/eci.14183 · Cited by 59 · see citing works
Blagec, K., Barbosa-Silva, A., Ott, S., & Matthias Samwald, A. (2022) · Scientific Data · doi:10.1038/s41597-022-01435-x · Cited by 57 · see citing works
Self-citationArtificial intelligence for high content imaging in drug discovery
Puigvert, J. C., & Spjuth, O. (2024) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2024.102842 · Cited by 51 · see citing works
Verlingue, L., Boyer, C., Olgiati, L., Brutti-Mairesse, C., Morel, D., & Blay, J. (2024) · The Lancet Regional Health - Europe · doi:10.1016/j.lanepe.2024.101064 · Cited by 50 · see citing works
Li, L. T., Haley, L. C., Boyd, A. K., & Bernstam, E. V. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104531 · Cited by 45 · see citing works
Bucher, A., Blazek, E. S., & Symons, C. T. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.05.007 · Cited by 37 · see citing works
Othman, Z. K., Ahmed, M. M., Kasimieh, O., Musa, S. S., Branda, F., Cue, E. G., Ocampo, J. M. A., Lucero‐Prisno, D. E., & Vimolmangkang, S. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100316 · Cited by 29 · see citing works
Adversarial artificial intelligence in radiology: Attacks, defenses, and future considerations
Dietrich, N. T., Gong, B., & Patlas, M. N. (2025) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2025.05.006 · Cited by 28 · see citing works
Agweyu, A., Mwaniki, P., Musau, W., Korom, R., Isaaka, L., Wanyama, C. W., Kiptinness, S., Adan, N., Emmanuel-Fabula, M., & Mateen, B. A. (2026) · Nature Health · doi:10.1038/s44360-026-00082-5 · Cited by 22 · see citing works
Computational hermeneutics: evaluating generative AI as a cultural technology
Kommers, C., Ahnert, R., Antoniak, M., Benetos, E., Benford, S. A., Bunz, M., Caramiaux, B., Concannon, S. J., Disley, M., Dobson, J. E., et al. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1753041 · Cited by 19 · see citing works
van Breugel, M., Greenhawt, M., Eguíluz‐Gracia, I., Torres, M. J., Anagnostou, A., & Koppelman, G. H. (2025) · Journal of Allergy and Clinical Immunology · doi:10.1016/j.jaci.2025.08.022 · Cited by 14 · see citing works
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35 · see citing works
Theodorakopoulos, L., & Theodoropoulou, A. (2026) · Applied Sciences · doi:10.3390/app16104603 · Cited by 12 · see citing works
Artificial Intelligence in Psychological Diagnosis and Intervention
Ramírez, L. Á. (2024) · LatIA · doi:10.62486/latia202326 · Cited by 11 · see citing works
Benítez‐Andrades, J. A., Prada‐García, C., Ordás-Reyes, N., Esteban-Blanco, M., Corcoba, A. M., & García, A. S. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00343-9 · Cited by 11 · see citing works
Faiyazuddin, M., Gholap, A. D., Maqsood, S., Abbas, S., Chaudhari, Y. S., Sharmila, S., & Webster, T. J. (2024) · Journal of Biomedical Nanotechnology · doi:10.1166/jbn.2024.3925 · Cited by 10 · see citing works
Al-Jebrni, A. H., Jiang, Z., Qin, Y., Qiu, Y., Jiang, N., Li, G., Zheng, Y., Wong, T. Y., Kim, S., Jung, Y., et al. (2026) · Computational Visual Media · doi:10.26599/cvm.2026.9450530 · Cited by 9 · see citing works
Gautam, S., Chandra, M., De, A., Chakravorti, T., Malik, G., & De Choudhury, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06849 · Cited by 7 · see citing works
Reporting on artificial intelligence use in entrepreneurship research: Using a model card
Fox, J. D. (2024) · The International Journal of Entrepreneurship and Innovation · doi:10.1177/14657503241238691 · Cited by 7 · see citing works
Park, S. H., & Kim, N. (2024) · Radiology · doi:10.1148/radiol.241703 · Cited by 7 · see citing works
Dataset Debt in Biomedical Language Modeling
Fries, J., Seelam, N., Altay, G., Weber, L., Kang, M., Datta, D., Su, R., Garda, S., Wang, B., Ott, S., et al. (2022) · doi:10.18653/v1/2022.bigscience-1.10 · Cited by 7 · see citing works
Invited commentary: deep learning—methods to amplify epidemiologic data collection and analyses
Quistberg, D. A., Mooney, S. J., Taşdizen, T., Arbeláez, P., & Nguyen, Q. C. (2024) · American Journal of Epidemiology · doi:10.1093/aje/kwae215 · Cited by 6 · see citing works
The Algorithmic Construction of Epistemic Injustice
McInerney, T. (2025) · Palgrave socio-legal studies · doi:10.1007/978-3-032-07581-9_5 · Cited by 4 · see citing works
Structured reasoning failures compromise LLM interpretation of clinical oncology notes
Kenaston, M. W., Ayub, U., Parmar, M., Kumar, P., Anjum, M. U., Naqvi, S. A. A., Chaudhuri, A. A., Zakharia, Y., Heath, E. I., Bekaii-Saab, T. S., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02951-5 · Cited by 2 · see citing works
Sul, A.-R., & Kim, S. (2025) · Healthcare Informatics Research · doi:10.4258/hir.2025.31.2.146 · Cited by 2 · see citing works
Understanding generative AI to harness its potentials and mini- mize risks: A perspective
Caselli, T. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.111951 · Cited by 2 · see citing works
Developing AI Models Using a Code-Free Platform
Aksyonov, K. A., Sun, L., & Kalinin, I. A. (2025) · International Russian Smart Industry Conference (SmartIndustryCon) · doi:10.1109/smartindustrycon65166.2025.10986074 · Cited by 1 · see citing works
Babkina, T. M., Bondarenko, A. O., & Zemskova, O. V. (2025) · Український радіологічний та онкологічний журнал · doi:10.46879/ukroj.3.2025.392-411 · Cited by 1 · see citing works
Lee, J., Shin, J., & Cho, B. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02567-9 · Cited by 1 · see citing works
Wang, Y., Bowditch, L., Molloy, C. J., Ying-hua, Y., Hibbert, P., & Magrabi, F. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag068 · Cited by 1 · see citing works
Hu, H., Wan, Z., Ng, P. H. F., Liu, K. P. Y., Fu, E. Y., & Cheng, A. S. K. (2026) · Frontiers in Rehabilitation Sciences · doi:10.3389/fresc.2026.1725150 · Cited by 1 · see citing works
Plasek, J. M., Li, Y., Amato, M. G., Foer, D., Seger, D. L., Alzaidi, S., Zhou, H., Jackson, G. P., Bates, D. W., & Zhou, L. (2026) · EBioMedicine · doi:10.1016/j.ebiom.2026.106516 · Cited by 1 · see citing works
Li, G., Lu, Y., Wu, G., Wang, L., & Ma, R. (2026) · Scientific Data · doi:10.1038/s41597-026-06620-w · Cited by 1 · see citing works
Bao, T. (2024) · International Conference on Cyber Security and Information Engineering (ICCSIE) · doi:10.1145/3689236.3696743
Domain Knowledge Empowered Language Models: A SurveyPreprint
Palikhe, A., Wang, Z., Hu, S., Yin, Z., Zhao, X., Liu, J., Cai, Y., & Zhang, W. (2025) · doi:10.36227/techrxiv.176403151.11257350/v1
Performance assessment strategies for language model applications in healthcare
García, V., Sidulova, M., & Badano, A. G. (2026) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2026.100162
Multimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Hueso, M., Alvarez, R., Saggion, H., & Vellido, A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-26466-5.00014-3
Agustavada, A. A. P., Wibawa, A. P., Sholum, A., Hilmi, D. F., & Dwiyanto, F. A. (2026) · Indonesian Journal of Electronics Electromedical Engineering and Medical Informatics · doi:10.35882/ijeeemi.v8i2.325
Okutan, H. E., & Baykara, M. (2026) · Applied Sciences · doi:10.3390/app16104637
Artificial Intelligence in Rare Diseases: Workflow-Integrated Precision Kidney Care
Thongprayoon, C., Pesce, F., & Cheungpasitporn, W. (2026) · Clinics and Practice · doi:10.3390/clinpract16060101
Use of generative artificial intelligence models in healthcare – a scoping review
Nooruddin, A., Lal, A., Ahmed, S. M. F., Ghori, M. H., Adnan, N., & Umer, F. (2026) · BMC Artificial Intelligence · doi:10.1186/s44398-026-00029-6
COLLECT: A Counterfactual Oral Lesion Library for Trustworthy AI Evaluation
Tafti, K. T., D’Souza, Z., Ajam, A., Chauvin, P., Magalhães, M. A. O., Fuoco, J., Sperandio, F. F., Al-Shehri, M. Y., Tichý, A., Tamimi, F., et al. (2026) · Journal of Dental Research · doi:10.1177/00220345261466240
Aimyshev, T., Arupzhanov, I., Zhakhina, G., Рахманкулова, А., Smagul, S., & Gaipov, A. (2026) · medRxiv · doi:10.64898/2026.09.03.26362149
liu, Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7438211
Lavecchia, A. (2026) · Springer eBooks · doi:10.1007/978-981-96-7448-0_68-1
Early Detection of Potato Leaf Disease Using Deep Learning
Purnama, M. O. S., Hafizh, G. A., Tirta, N., & Rangkuti, A. H. (2026) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss71466.2026.11715357
Radar-based cuffless blood pressure monitoring: a comprehensive review
Yen, H. T., Thanh, N. D., Hoàng, V. P., & Sun, G. (2026) · Measurement · doi:10.1016/j.measurement.2026.123660
- Authors
- Blagec, K., Kraiger, J., Frühwirt, W., & Samwald, M.
- Affiliations
- Medical University of Vienna (AT)
- ORCID
- Blagec, Kathrin
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2023-01
- Volume
- 137
- Pages
- 104274
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W4312055891
- Cited by
- 47
- References
- 34
- Citations by year
- 2026: 19 · 2025: 11 · 2024: 11 · 2023: 3 · 2022: 3
- FWCI
- 0.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Horizon 2020 (668353)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- AI benchmarks, clinical natural language processing, dataset metadata, clinical AI, clinician workload, clinical documentation automation, AI-assisted clinical decision support
- MeSH terms
- Artificial Intelligence, Humans, Natural Language Processing, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
