CoChat
Sign Up
Cited by 26Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareExplainable Artificial Intelligence (XAI)

A framework for evaluating clinical artificial intelligence systems without ground-truth annotations

Open access
Kiyasseh, Dani, Cohen, Aaron, Jiang, Chengsheng, Altieri, Nicholas·2024·Nature Communications
DOI 10.1038/s41467-024-46000-9PMC PMC10902352

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

A clinical artificial intelligence (AI) system is often validated on data withheld during its development. This provides an estimate of its performance upon future deployment on data in the wild; those currently unseen but are expected to be encountered in a clinical setting. However, estimating performance on data in the wild is complicated by distribution shift between data in the wild and withheld data and the absence of ground-truth annotations. Here, we introduce SUDO, a framework for evaluating AI systems on data in the wild. Through experiments on AI systems developed for dermatology images, histopathology patches, and clinical notes, we show that SUDO can identify unreliable predictions, inform the selection of models, and allow for the previously out-of-reach assessment of algorithmic bias for data in the wild without ground-truth annotations. These capabilities can contribute to the deployment of trustworthy and ethical AI systems in medicine.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-024-46000-9

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These limitations reinforce existing healthcare disparities, as benefits of AI solutions are predominantly confined to demographic groups already well-represented in training datasets.

    “Secondly, these limitations reinforce existing healthcare disparities, as the benefits of AI solutions are predominantly confined to demographic groups already well-represented in training datasets [37,38].”

    From Frameworks encompassing intersectional perspective of artificial intelligence in healthcare. Scoping review (2026) · cites this paper as [38]

    The Results section reports that a dermatology AI system performed better for patients with Fitzpatrick skin tones I–II than for those with V–VI, supporting the point that benefits can differ across demographic groups. The paper does not establish that the better-performing group was better represented in training data, or that AI benefits are predominantly confined to well-represented groups generally. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [1]Bulten, W. et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature Medicine1-10 (2022). PMC8799467 35027755
  2. [2]Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L. & Ranganath, R. Opportunities in machine learning for healthcare. CoRRabs/1806.00388 (2018). PMC7233077 32477638 · arXiv
  3. [3]Koh, P. W. et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 5637-5664 (PMLR, 2021).
  4. [4]Mehrabi N Morstatter F Saxena N Lerman K Galstyan A A survey on bias and fairness in machine learning ACM Comput. Surv. (CSUR) 2021 54 1 35 10.1145/3457607 · DOI · Paper page
  5. [5]Hendrycks, D. & Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations (2017).
  6. [7]Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning, 1321-1330 (PMLR, 2017).
  7. [8]Platt, J. C. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, 61-74 (MIT Press, 1999).
  8. [9]Bibas K Feder M Hassner T Single layer predictive normalized maximum likelihood for out-of-distribution detection Adv. Neural Inform. Process. Syst. 2021 34 1179 1191
  9. [10]Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 30 (2017).
  10. [12]Claesen, M., Davis, J., De Smet, F. & De Moor, B. Assessing binary classifiers using only positive and unlabeled data. arXiv preprint arXiv:1504.06837 (2015). · arXiv
  11. [13]Zhou, D., Liu, M., Li, M. & Cai, T. Doubly robust augmented model accuracy transfer inference with high dimensional features. arXiv preprint arXiv:2208.05134 (2022). PMC12037143 40302926 · arXiv
  12. [15]Esteva A Dermatologist-level classification of skin cancer with deep neural networks Nature 2017 542 115 118 10.1038/nature21056 28117445 PMC8382232 · DOI · Paper page
  13. [16]Tschandl P Rosendahl C Kittler H The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions Sci. Data 2018 5 1 9 10.1038/sdata.2018.161 30106392 PMC6091241 · DOI · Paper page
  14. [17]Blitzer, J., Dredze, M. & Pereira, F. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 440-447 (2007).
  15. [18]Cohen AB A natural language processing algorithm to improve completeness of ecog performance status in real-world data Appl. Sci. 2023 13 6209 10.3390/app13106209 · DOI
  16. [19]Buccheri G Ferrigno D Tamburini M Karnofsky and ecog performance status scoring in lung cancer: a prospective, longitudinal study of 536 patients from a single institution Eur. J. Cancer 1996 32 1135 1141 10.1016/0959-8049(95)00664-8 8758243 · DOI
  17. [20]Manola J Atkins M Ibrahim J Kirkwood J Prognostic factors in metastatic melanoma: a pooled analysis of eastern cooperative oncology group trials J. Clin. Oncol. 2000 18 3782 3793 10.1200/JCO.2000.18.22.3782 11078491 · DOI
  18. [21]Gronsbell JL Cai T Semi-supervised approaches to efficient evaluation of model prediction performance J. Royal Stat. Soc. Ser. B (Stat. Methodol.) 2018 80 579 594 10.1111/rssb.12264 PMC9586151 36275859 · DOI
  19. [22]Wang, L., Wang, X., Liao, K. P. & Cai, T. Semi-supervised transfer learning for evaluation of model classification performance. arXiv preprint arXiv:2208.07927 (2022). PMC10926267 38465982 · arXiv
  20. [23]Ji D Smyth P Steyvers M Can i trust my fairness metric? assessing fairness with unlabeled data and bayesian inference Adv. Neural Inf. Process. Syst. 2020 33 18600 18612
  21. [24]Fluss R Reiser B Faraggi D Rotnitzky A Estimation of the roc curve under verification bias Biometrical J.: J. Math. Methods in Biosci. 2009 51 475 490 10.1002/bimj.200800128 PMC3475535 19588455 · DOI
  22. [25]Umemneku Chikere CM Wilson K Graziadio S Vale L Allen AJ Diagnostic test evaluation methodology: a systematic review of methods employed to evaluate diagnostic tests in the absence of gold standard–an update PLoS One 2019 14 e0223832 10.1371/journal.pone.0223832 31603953 PMC6788703 · DOI
  23. [26]Fan, W. & Davidson, I. Reverse testing: an efficient framework to select amongst classifiers under sample selection bias. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 147-156 (2006).
  24. [27]Zhong, E., Fan, W., Yang, Q., Verscheure, O. & Ren, J. Cross validation framework to choose amongst models and datasets for transfer learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 547-562 (Springer, 2010).
  25. [29]Bandi P From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge IEEE Trans. Med. imaging 2018 38 550 560 10.1109/TMI.2018.2867350 30716025 · DOI · Paper page
  26. [30]Ma, X., Long, L., Moon, S., Adamson, B. J. & Baxi, S. S. Comparison of population characteristics in real-world clinical oncology databases in the us: Flatiron Health, SEER, and NPCR. Medrxiv (2020).
  27. [31]Oken MM Toxicity and response criteria of the eastern cooperative oncology group Am. J. Clin. Oncol. 1982 5 649 656 10.1097/00000421-198212000-00014 7165009 · DOI
  28. [32]Jang RW Simple prognostic model for patients with advanced cancer based on performance status J. Oncol. Pract. 2014 10 e335 e341 10.1200/JOP.2014.001457 25118208 · DOI
  29. [34]Sørensen J Klee M Palshof T Hansen H Performance status assessment in cancer patients. an inter-observer variability study Br. J. Cancer 1993 67 773 775 10.1038/bjc.1993.140 8471434 PMC1968363 · DOI
  30. [35]Hardt M Price E Srebro N Equality of opportunity in supervised learning Adv. Neural Inf. Process. Syst. 2016 29 3315 3323
  31. [36]Curtis MD Development and validation of a high-quality composite real-world mortality endpoint Health Serv. Res. 2018 53 4460 4476 10.1111/1475-6773.12872 29756355 PMC6232402 · DOI
  32. [37]Davidson-Pilon C lifelines: survival analysis in python J. Open Source Softw. 2019 4 1317 10.21105/joss.01317 · DOI

Cited by

Sorted by citations

Authors
Kiyasseh, D., Cohen, A., Jiang, C., & Altieri, N.
Affiliations
  • Cedars-Sinai Medical Center (US)
  • Flatiron Health (United States) (US)
Corresponding author
Kiyasseh, Dani
ORCID
Venue
Nature Communications
Publisher
Springer Science and Business Media LLC
Published
2024-02-28
Volume
15
Issue
1
Pages
1808-1808
ISSN
2041-1723 (ISSN-L)
DOI
10.1038/s41467-024-46000-9
PMC
PMC10902352
OpenAlex
W4392237966
Cited by
26
References
5
Citations by year
2026: 13 · 2025: 6 · 2024: 7
FWCI
2.43
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
clinical AI, distribution shift, dermatological image analysis, clinical notes, algorithmic bias, model selection, trustworthy AI, ethical AI
MeSH terms
Artificial Intelligence, Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review