Evaluation of artificial intelligence on a reference standard based on subjective interpretation
Open accessAbstract
Rapid progress has been made in artificial intelligence (AI) models for medical applications, especially over the past 5 years, with substantial efforts focusing on diagnosis from medical images. An essential aspect of evaluating the performance of AI models and their potential clinical utility is the rigor of the reference standard.1 A reference standard is “the best available method for establishing the presence or absence of the target condition”,2 and is thus equivalent to what is commonly referred to as the ground truth in AI literature.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some reference standards are highly accurate but costly and poorly available, whereas others neglect intermediate findings or are operator-dependent, potentially reducing label applicability and reliability.
“Some could be highly accurate but also costly and poorly available, whereas others could neglect intermediate findings or be operator-dependent,27 potentially reducing label applicability and reliability.”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570 · see citing works
Bulten, W., Pinckaers, H., van Boven, H., Vink, R., de Bel, T., van Ginneken, B., van der Laak, J., Hulsbergen-van de Kaa, C., & Litjens, G. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30739-9 · Cited by 705 · see citing works
Ström, P., Kartasalo, K., Olsson, H., Solorzano, L., Delahunt, B., Berney, D. M., Bostwick, D. G., Evans, A. J., Grignon, D. J., Humphrey, P. A., et al. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30738-7 · Cited by 620 · see citing works
How to Read Articles That Use Machine Learning
Liu, Y., Chen, P.-H. C., Krause, J., & Peng, L. H. (2019) · JAMA · doi:10.1001/jama.2019.16489 · Cited by 530 · see citing works
Self-citationPantanowitz, L., Quiroga‐Garza, G. M., Bien, L., Heled, R., Laifenfeld, D., Linhart, C., Sandbank, J., Shach, A. A., Shalev, V., Vecsler, M., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30159-x · Cited by 361 · see citing works
Nagpal, K., Foote, D., Tan, F. E., Liu, Y., Chen, P.-H. C., Steiner, D. F., Manoj, N. S., Olson, N. H., Smith, J. L., Mohtashamian, A., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.2485 · Cited by 223 · see citing works
Self-citationBertens, L. C. M., Broekhuizen, B. D., Naaktgeboren, C. A., Rutten, F. H., Hoes, A. W., van Mourik, Y., Moons, K. G. M., & Reitsma, J. B. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001531 · Cited by 158 · see citing works
Bulten, W., Pinckaers, H., van Boven, H., Vink, R., de Bel, T., van Ginneken, B., van der Laak, J. A. W. M., van de Kaa, C. A. H. ‐., & Litjens, G. J. S. (2020) · arXiv (Cornell University) · Cited by 106 · see citing works
Predicting prostate cancer specific-mortality with artificial intelligence-based Gleason grading
Wulczyn, E., Nagpal, K., Symonds, M., Moran, M., Plass, M., Reihs, R., Nader, F., Tan, F., Cai, Y., Brown, T., et al. (2021) · Communications Medicine · doi:10.1038/s43856-021-00005-3 · Cited by 51 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
“26 Some could be highly accurate but also costly and poorly available, whereas others could neglect intermediate findings or be operator-dependent, 27 potentially reducing label applicability and reliability.”
Wysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2022.103839 · Cited by 161 · see citing works
APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support
Kwong, J. C. C., Khondker, A., Lajkosz, K., McDermott, M. B. A., Borrat, X., McCradden, M. D., Mamdani, M., Kulkarni, G. S., & Johnson, A. E. W. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.35377 · Cited by 151 · see citing works
Deep learning: A primer for dentists and dental researchers
Mohammad‐Rahimi, H., Rokhshad, R., Bencharit, S., Krois, J., & Schwendicke, F. (2023) · Journal of Dentistry · doi:10.1016/j.jdent.2023.104430 · Cited by 97 · see citing works
Deep learning models for histologic grading of breast cancer and association with disease prognosis
Jaroensri, R., Wulczyn, E., Hegde, N., Brown, T., Flament-Auvigne, I., Tan, F. E., Cai, Y., Nagpal, K., Rakha, E. A., Dabbs, D. J., et al. (2022) · npj Breast Cancer · doi:10.1038/s41523-022-00478-y · Cited by 84 · see citing works
Self-citationArtificial Intelligence Enables Quantitative Assessment of Ulcerative Colitis Histology
Najdawi, F., Sucipto, K., Mistry, P., Hennek, S. R., Jayson, C., Lin, M., Fahy, D., Kinsey, S., Wapinski, I. N., Beck, A. H., et al. (2023) · Modern Pathology · doi:10.1016/j.modpat.2023.100124 · Cited by 68 · see citing works
Mohammad‐Rahimi, H., Sohrabniya, F., Ourang, S. A., Dianat, O., Aminoshariae, A., Nagendrababu, V., Dummer, P. M. H., Duncan, H. F., & Nosrat, A. (2024) · International Endodontic Journal · doi:10.1111/iej.14128 · Cited by 70 · see citing works
Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs
Lee, J. H., Hong, H., Nam, G., Hwang, E. J., & Park, C. M. (2023) · Radiology · doi:10.1148/radiol.222976 · Cited by 52 · see citing works
Zhang, J., Whebell, S. F., Gallifant, J., Budhdeo, S., Mattie, H., Lertvittayakumjorn, P., Lopez, M. D. P. A., Tiangco, B. J., Gichoya, J. W., Ashrafian, H., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00032-2 · Cited by 52 · see citing works
Is Artificial Intelligence Replacing Our Radiology Stars? Not Yet!
Cacciamani, G. E., Sanford, D. I., Chu, T. N., Kaneko, M., Abreu, A. L., Duddalwar, V. A., & Gill, I. S. (2022) · European Urology Open Science · doi:10.1016/j.euros.2022.09.024 · Cited by 43 · see citing works
Montezuma, D., Oliveira, S. P., Neto, P. C., Oliveira, D., Monteiro, A., Cardoso, J. S., & Macedo-Pinto, I. (2023) · Modern Pathology · doi:10.1016/j.modpat.2022.100086 · Cited by 40 · see citing works
Artificial Intelligence in Infection Management in the ICU
Corte, T. D., Hoecke, S. V., & Waele, J. J. D. (2022) · Critical Care · doi:10.1186/s13054-022-03916-2 · Cited by 38 · see citing works
Ong, A. Y., Taribagil, P., Sevgi, M., Kale, A. U., Dow, E. R., Macdonald, T., Kras, A., Maniatopoulos, G., Liu, X., Keane, P. A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01726-8 · Cited by 39 · see citing works
Mohammad‐Rahimi, H., Dianat, O., Abbasi, R., Zahedrozegar, S., Ashkan, A., Motamedian, S. R., Rohban, M. H., & Nosrat, A. (2023) · Journal of Endodontics · doi:10.1016/j.joen.2023.11.004 · Cited by 38 · see citing works
Sharan, R. V., & Rahimi‐Ardabili, H. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105093 · Cited by 32 · see citing works
Wagner, S. K., Liefers, B. J., Radia, M., Zhang, G., Struyven, R., Faes, L., Than, J., Balal, S., Hennings, C., Kilduff, C. L. S., et al. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00050-x · Cited by 29 · see citing works
Meinert, E., Milne‐Ives, M., Lim, E., Higham, A., Boege, S., de Pennington, N., Bajre, M., Mole, G., Normando, E. M., & Xue, K. (2024) · EClinicalMedicine · doi:10.1016/j.eclinm.2024.102692 · Cited by 27 · see citing works
Masters, K., & Salcedo, D. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2298756 · Cited by 26 · see citing works
Automated Cough Sound Analysis for Detecting Childhood Pneumonia
Sharan, R. V., Qian, K., & Yamamoto, Y. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3327292 · Cited by 25 · see citing works
Rojas-Carabali, W. A., Cifuentes‐González, C., Gutiérrez-Sinisterra, L., Heng, L. Y., Tsui, E. Y. K., Gangaputra, S., Sadda, S. R., Nguyen, Q. D., Kempen, J. H., Pavesio, C. E., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100082 · Cited by 21 · see citing works
González, R., Saha, A., Campbell, C. J. V., Nejat, P., Lokker, C., & Norgan, A. P. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100347 · Cited by 21 · see citing works
Use of artificial intelligence in forecasting glaucoma progression
Thakur, S., Dinh, L. L., Lavanya, R., Quek, T. C., Liu, Y., & Cheng, C. (2023) · Taiwan Journal of Ophthalmology · doi:10.4103/tjo.tjo-d-23-00022 · Cited by 14 · see citing works
Self-citationHuh, J. E., Lee, J. H., Hwang, E. J., & Park, C. M. (2023) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0548 · Cited by 13 · see citing works
Montezuma, D., Oliveira, S. P., Tolkach, Y., Boor, P., Haragan, A., Carvalho, R., Mea, V. D., Kiehl, T., Leh, S. M., Yousif, M. Q., et al. (2024) · Laboratory Investigation · doi:10.1016/j.labinv.2024.102203 · Cited by 12 · see citing works
Oliveira, L. D., Lu, J., Erak, E., Mendes, A. A. D. A., Dairo, O., Ertunç, O., Kulaç, İ., Valle, J. A. B.-D., Jones, T., Hicks, J. L., et al. (2024) · European Urology Oncology · doi:10.1016/j.euo.2024.08.004 · Cited by 12 · see citing works
Huemer, J. C., Kronschläger, M., Ruiß, M., Sim, D. A., Keane, P. A., Findl, O., & Wagner, S. K. (2022) · BMJ Open Ophthalmology · doi:10.1136/bmjophth-2022-000992 · Cited by 11 · see citing works
Chang, J., & Hatfield, B. S. (2024) · Advances in cancer research · doi:10.1016/bs.acr.2024.05.006 · Cited by 11 · see citing works
Shen, N., Zhang, Y., Ding, Z., Li, R., Quan, X., Liu, X., Zhang, Y., Xiang, T., Zhang, Y., Yin, C., et al. (2025) · International Journal of Surgery · doi:10.1097/js9.0000000000002730 · Cited by 12 · see citing works
CALSTER, B. V., van Smeden, M., van Amsterdam, W. A. C., Coemans, M., Wynants, L., & Steyerberg, E. W. (2025) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042324-123749 · Cited by 12 · see citing works
Deep learning methods for diagnosis of graves’ ophthalmopathy using magnetic resonance imaging
Ma, Z.-C., Lin, J., Li, S.-K., Liu, H.-J., & Zhang, Y. (2024) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-24-80 · Cited by 9 · see citing works
Scodellaro, R., Zschüntzsch, J., Hell, A. K., & Alves, F. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108733 · Cited by 9 · see citing works
Aanestad, E., Beniczky, S., Olberg, H. K., & Brøgger, J. C. (2024) · Epileptic Disorders · doi:10.1002/epd2.20291 · Cited by 8 · see citing works
Yuan, L., Zhou, H., Xiao, X., Zhang, X., Chen, F., Liu, L., Liu, J., Bao, S., & Tao, K. (2024) · Frontiers in Oncology · doi:10.3389/fonc.2024.1365364 · Cited by 6 · see citing works
Cochrane corner: artificial intelligence for diagnosing exudative age-related macular degeneration
Ong, A. Y., Hogg, H. D. J., & Keane, P. A. (2025) · Eye · doi:10.1038/s41433-025-03599-3 · Cited by 6 · see citing works
Duncan, S. F., Kidd, A. C., Perdomo-Lampignano, J. A., Cannon, P., Hall, M., Stobo, D., Maclay, J. D., Blyth, K. G., & Lowe, D. J. (2025) · European Journal of Radiology · doi:10.1016/j.ejrad.2025.112409 · Cited by 7 · see citing works
Clinical ground truth in machine learning for early sepsis diagnosis
Lindner, H. A., Thiel, M., & Schneider‐Lindner, V. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00070-5 · Cited by 5 · see citing works
Glass, M., Ji, Z., Davis, R., Pavlisko, E. N., DiBernardo, L. R., Carney, J., Fishbein, G. A., Luthringer, D., Miller, D. V., Mitchell, R. N., et al. (2024) · Cardiovascular Pathology · doi:10.1016/j.carpath.2024.107646 · Cited by 4 · see citing works
Sharan, R. V., & Xiong, H. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105912 · Cited by 4 · see citing works
Hu, B., He, H., Shi, Z., Wang, L., Liu, Q., Sun, Z., & Zhang, L. J. (2025) · Neuroradiology · doi:10.1007/s00234-024-03536-3 · Cited by 4 · see citing works
Conformity towards humans versus AI in different task domains: the type of task matters
Thorp, S. O., Slupphaug, K. S., Rimol, L. M., Lervik, S., Hoel, S. K. B., & Grassini, S. (2025) · Journal of Psychology and AI · doi:10.1080/29974100.2025.2540762 · Cited by 3 · see citing works
de la Calle, C. M., Baras, A. S., & Lotan, T. L. (2026) · British Journal of Urology · doi:10.1111/bju.70164 · Cited by 3 · see citing works
Medical multimodal large language models: A survey
Xiao, H., Hui, N., Xu, Y., Li, Z., & Peng, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104386 · Cited by 2 · see citing works
Artificial Intelligence in Infection Management in the ICU
Corte, T. D., Hoecke, S. V., & Waele, J. J. D. (2022) · Annual update in intensive care and emergency medicine · doi:10.1007/978-3-030-93433-0_28 · Cited by 2 · see citing works
TRIAGE: Trustworthy Reporting and Assessment for Clinical Gain and Effectiveness of AI Models
Fazilati, F., Rajabi, M. T., Alihosseini, N., Farsani, M. E., Sandid, S. H., Zamani, S., Farahani, M. A., Biriaei, F., Sadeghipour, F., Mirshamsi, M. T., et al. (2026) · Diagnostics · doi:10.3390/diagnostics16050666 · Cited by 5 · see citing works
Shih, Y.-C., Ko, C., Wang, S.-Y., Chang, C.-Y., Lin, S.-S., Huang, C.-W., Cheng, M., Chen, C., & Wu, Y. (2025) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-025-07243-w · Cited by 1 · see citing works
Tzirita, S., King, C., McCollum, E. D., Baker, K. S., López, M. B., & Schedwin, M. (2026) · BMJ Paediatrics Open · doi:10.1136/bmjpo-2025-003506 · Cited by 1 · see citing works
Artificial intelligence for screening drug resistance in tuberculosis
Mohanvel, S. K., Radhakrishnan, R., Balraj, P., Singh, N., Danisetty, S., Vats, H., Chaudhury, A., Choudhary, H., Rajendran, P., Madhumathi, J., et al. (2026) · The Indian Journal of Medical Research · doi:10.25259/ijmr_1546_2025 · Cited by 1 · see citing works
AI‐Assisted Fungal Detection in Dermatopathology: A Study on Accuracy, Efficiency, and Usability
Schmidle, P., Lang, T., Springenberg, S., Calvopina, D. A., Köhler, T., Getova, V., Gebing, A.-M., Hetzer, S., Pierrone, J., Aebischer, V., et al. (2026) · JDDG Journal der Deutschen Dermatologischen Gesellschaft · doi:10.1111/ddg.70242 · Cited by 1 · see citing works
Uveal Melanoma Ground Truth Labeling in Machine Learning
Kao, E., Ganesh, S., Chadwick, W. F., Alahmadi, R., Yao, X., & Heiferman, M. J. (2026) · Cancers · doi:10.3390/cancers18091357 · Cited by 1 · see citing works
Delvaux, B. V., Elmaleh, Y. D., Chuan, A., Sia, A. T., GUPTA, R. K., Schroeder, K. M., Guessous, K., & Bowness, J. S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1823745 · Cited by 1 · see citing works
Artificial intelligence for dental caries diagnosis: translating algorithms to clinical practice
Wang, Z., Zhu, Z., Jiang, T., & Gu, Y. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1879708 · Cited by 1 · see citing works
Awasthi, R., Nagori, A., Mishra, S. V., Mathur, A., Mathur, P., & Nasri, B. (2023) · medRxiv · doi:10.1101/2023.06.20.23291653 · Cited by 1 · see citing works
Farhat, M., Dagher, S. A., Ozkara, B. B., Ozcan, B. B., Karabacak, M., Chen, M., Yedavalli, V. S., & Wintermark, M. (2025) · The Neuroradiology Journal · doi:10.1177/19714009251407175
Hashim, A., Giamperoli, A., Indre, M. G., & Piscaglia, F. (2026) · Ultraschall in der Medizin - European Journal of Ultrasound · doi:10.1055/a-2809-9211
Artificial Intelligence (Pattern Recognition) in Musculoskeletal Imaging: The Future or Hype?
Tordjman, M., Regnard, N.-E., Bouvier, F. M., Rizk, B., & Guermazi, A. (2026) · Seminars in Musculoskeletal Radiology · doi:10.1055/a-2818-3827
White, S. J., Barker, T. H., Merlin, T., & Munn, Z. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112333
Lu, L., Li, H., Liu, Y.-M., Du, H.-X., Qin, J., Yeung, W.-F., Zhong, C. C., Xiong, L., & Chen, S.-C. (2026) · Information · doi:10.3390/info17060554
Self-citationArtificial Intelligence in Orthodontics: Part 2—Data Preparation and Performance Evaluation
Naveed, K., Sohrabniya, F., Mohammad‐Rahimi, H., Woolley, J., Pauwels, R., & Stoustrup, P. B. (2026) · Orthodontics and Craniofacial Research · doi:10.1111/ocr.70171
Delvaux, B. V., Elmaleh, Y. D., Guessous, K., Garnier, T., Morel, B., Yao, K., Giral, T., Bowness, J. S., & Herbelet, S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1879747
Gao, Q., Kuo, D. C., Amason, J., Lee, T., Rathinavelu, J. K., Pajić, M., & Hadziahmetovic, M. (2026) · British Journal of Ophthalmology · doi:10.1136/bjo-2025-328327
Kumar, R., Sharma, N., Deshmukh, A., Nair, A., & Pillai, M. (2022) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/l244649356
On the Behavior of Calibration Error Under Rater Disagreement
Kumar, H., Vu, Y. N. T., Su, J., & Matthews, T. P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38470-6_21
Automatic Posterior Capsule Opacification detection with a custom deep learning model
Lisy, M., Peer, L., Schartmüller, D., Leydolt, C., & Saffer, Z. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100503
- Authors
- Chen, P.-H. C., Mermel, C. H., & Liu, Y.
- Corresponding author
- Chen, Po-Hsuan Cameron
- Venue
- The Lancet Digital Health
- Publisher
- Elsevier BV
- Published
- 2021-11
- Volume
- 3
- Issue
- 11
- Pages
- e693-e695
- ISSN
- 2589-7500 (ISSN-L)
- OpenAlex
- W3201000121
- Cited by
- 63
- References
- 9
- Citations by year
- 2026: 18 · 2025: 11 · 2024: 12 · 2023: 14 · 2022: 8
- FWCI
- 2.46
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- The Lancet Digital Health (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- AI evaluation, reference materials, ground truth, subjective interpretation, medical image diagnosis, clinical utility, diagnostic accuracy
- MeSH terms
- Clinical Decision-Making, Artificial Intelligence, Humans, Models, Biological, Reference Standards, Observer Variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review