CoChat
Sign Up
PreprintCited by 12Machine Learning in HealthcareComputational Drug Discovery MethodsArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Reliable machine learning models in genomic medicine using conformal prediction

Open access
Papangelou, Christina, Kyriakidis, Konstantinos, Natsiavas, Pantelis, Chouvarda, Ioanna, Malousi, Andigoni·2024·Frontiers in Bioinformatics
DOI 10.1101/2024.09.09.24312995PMC PMC11891349

Preprint — published as 10.3389/fbinf.2025.1507448 · Paper page

Open this paper in CoChat — free account

Abstract

Machine learning and genomic medicine are the mainstays of research in delivering personalized healthcare services for disease diagnosis, risk stratification, tailored treatment, and prediction of adverse effects. However, potential prediction errors in healthcare services can have life-threatening impact, raising reasonable skepticism about whether these applications are beneficial in real-world clinical practices. Conformal prediction is a versatile method that mitigates the risks of singleton predictions by estimating the uncertainty of a predictive model. In this study, we investigate potential applications of conformalized models in genomic medicine and discuss the challenges towards bridging genomic medicine applications with clinical practice. We also demonstrate the impact of a binary transductive model and a regression-based inductive model in predicting drug response and the performance of a multi-class inductive predictor in addressing distribution shifts in molecular subtyping. The main conclusion is that as machine learning and genomic medicine are increasingly infiltrating healthcare services, conformal prediction has the potential to overcome the safety limitations of current methods and could be effectively integrated into uncertainty-informed applications within clinical environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.09.09.24312995

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Figures

FIGURE 1 CP-based frameworks. Inductive CP (ICP), involves a static split of the dataset into training and calibration sets, using a single model for all predictions, while Transductive CP (TCP), builds a new model for each test instance, offering potentially more tailored predictions at the cost of increased computational effort.
FIGURE 1 CP-based frameworks. Inductive CP (ICP), involves a static split of the dataset into training and calibration sets, using a single model for all predictions, while Transductive CP (TCP), builds a new model for each test instance, offering potentially more tailored predictions at the cost of increased computational effort.CC BY · PubMed Central
FIGURE 2 Overview of the classification study design. First, the ML model produces singleton predictions for the binary (responder/non-responder patients to infliximab) and the multiclass (MHG, GCB and ABC subtypes of diffuse large B-cell lymphomas) problems. Singleton predictions are given without any indication of their accuracy and reliability. In the second step, CP is applied on the results of the ML models to estimate the uncertainty of each prediction. In the binary classification, TCP produces a prediction region that contains the true class with high probability and detects the uncertain (UNC) predictions. In the multiclass setting ICP identifies unreliable predictions among the samples classified by the non-conformalized model.
FIGURE 2 Overview of the classification study design. First, the ML model produces singleton predictions for the binary (responder/non-responder patients to infliximab) and the multiclass (MHG, GCB and ABC subtypes of diffuse large B-cell lymphomas) problems. Singleton predictions are given without any indication of their accuracy and reliability. In the second step, CP is applied on the results of the ML models to estimate the uncertainty of each prediction. In the binary classification, TCP produces a prediction region that contains the true class with high probability and detects the uncertain (UNC) predictions. In the multiclass setting ICP identifies unreliable predictions among the samples classified by the non-conformalized model.CC BY · PubMed Central
FIGURE 3 Principal Component Analysis (PCA). Left: PCA analysis of the samples without unclassified cases (UNC), revealing a small overlap between subtypes ABC and GCB. Right: PCA analysis of the dataset including unclassified cases (UNC), shows the UNC class overlapping with subtypes ABC and GCB.
FIGURE 3 Principal Component Analysis (PCA). Left: PCA analysis of the samples without unclassified cases (UNC), revealing a small overlap between subtypes ABC and GCB. Right: PCA analysis of the dataset including unclassified cases (UNC), shows the UNC class overlapping with subtypes ABC and GCB.CC BY · PubMed Central
FIGURE 4 Visualization of the distribution shift between the two diffuse large B-cell lymphoma datasets based on the maximum mean discrepancy (MMD).
FIGURE 4 Visualization of the distribution shift between the two diffuse large B-cell lymphoma datasets based on the maximum mean discrepancy (MMD).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 71

  1. 10.1016/j.metabol.2017.01.011 · DOI · Paper page
  2. 10.7861/clinmedicine.17-6-545 · DOI
  3. 10.1007/s41666-021-00113-8 · DOI
  4. 2022, Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction, Nature communications, 13, 7761, 10.1038/s41467-022-34945-8 · DOI · Paper page
  5. Urban Fagerholm , Jonathan Alvarsson , Sven Hellberg , and Ola Spjuth . (2022) Validation of predicted conformal intervals for prediction of human clinical pharmacokinetics. bioRxiv, pages 2022–11. · DOI
  6. 2017, Conformal prediction of biological activity of chemical compounds, Annals of Mathematics and Artificial Intelligence, 81, 105, 10.1007/s10472-017-9556-8 · DOI
  7. 2021, Deep learning-based conformal prediction of toxicity, Journal of chemical information and modeling, 61, 2648, 10.1021/acs.jcim.1c00208 · DOI
  8. 2021, Predicting with confidence: Using conformal prediction in drug discovery, Journal of Pharmaceutical Sciences, 110, 42, 10.1016/j.xphs.2020.09.055 · DOI · Paper page
  9. Antonis Lambrou , Harris Papadopoulos , and Alex Gammerman .(2009) Evolutionary conformal prediction for breast cancer diagnosis. In 2009 9th international conference on information technology and applications in biomedicine, pages 1–4. IEEE. · DOI
  10. Vladimir Vovk , Alexander Gammerman , and Glenn Shafer . (2005) Algorithmic learning in a random world. Springer, volume 29.
  11. 10.1093/comjnl/bxl065 · DOI
  12. Harris Papadopoulos . Inductive conformal prediction: Theory and application to neural networks. (2008) In Tools in Artificial Intelligence. Citeseer.. · DOI
  13. 2013, Peripheral blood derived gene panels predict response to infliximab in rheumatoid arthritis and Crohn’s disease, Genome medicine, 5, 1
  14. 2010, Peripheral blood gene expression patterns discriminate among chronic inflammatory diseases and healthy controls and identify novel targets, BMC medical genomics, 3, 1
  15. 2020, Targeted sequencing in DLBCL, molecular subtypes, and outcomes: A Haematological Malignancy Research Network report. Blood, The Journal of the American Society of Hematology, 135, 1759
  16. 10.1200/JCO.18.01314 · DOI
  17. 2015, Transferring genomics to the clinic: distinguishing Burkitt and diffuse large B cell lymphomas, Genome medicine, 7, 1
  18. Tianqi Chen and Carlos Guestrin . (2016) Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 785–794. · DOI · Paper page
  19. Daniel Painter , Sharon Barrans , Stuart Lacy , Alexandra Smith , Simon Crouch , David Westhead , Chulin Sha , Russell Patmore , Reuben Tooze , Cathy Burton , et al. Cell-of-origin in diffuse large B-cell lymphoma: findings from the UK’s population-based Haematological Malignancy Research Network. British journal of haematology, 185(4):781–784. · DOI
  20. 2019, Measuring domain shift for deep learning in histopathology, IEEE journal of biomedical and health informatics, 25, 325
  21. 2022, Fair conformal predictors for applications in medical imaging, Proceedings of the AAAI Conference on Artificial Intelligence, 36, 12008, 10.1609/aaai.v36i11.21459 · DOI
  22. Charles Lu , Anastasios N Angelopoulos , and Stuart Pomerantz .(2022) Improving trustworthiness of AI disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 545–554. Springer. · DOI
  23. 2020, Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images, IEEE Journal of Biomedical and Health Informatics, 25, 371
  24. 10.1186/s13321-019-0358-3 · DOI
  25. 10.1073/pnas.2204569119 · DOI
  26. 2017, Modelling compound cytotoxicity using conformal prediction and PubChem HTS data, Toxicology Research, 6, 73, 10.1039/C6TX00252H · DOI
  27. 2018, Predicting skin sensitizers with confidence—using conformal prediction to determine applicability domain of GARD, Toxicology in Vitro, 48, 179, 10.1016/j.tiv.2018.01.021 · DOI
  28. 2018, The certainty of uncertainty in genomic medicine: managing the challenge, Journal of Healthcare Communications, 3, 1
  29. Eric D Sun , Rong Ma , Paloma Navarro Negredo , Anne Brunet , and James Zou . (2024) TISSUE: uncertainty-calibrated prediction of single-cell spatial transcriptomics improves downstream analyses. Nature Methods, pages 1–11. · DOI
  30. 2021, Translating polygenic risk scores for clinical use by estimating the confidence bounds of risk prediction, Nature Communications, 12, 5276, 10.1038/s41467-021-25014-7 · DOI
  31. 2022, PrecisionFDA Truth Challenge V2: Calling variants from short and long reads in difficultto-map regions, Cell Genomics, 2, 5
  32. 2020, Machine learning approaches to drug response prediction: challenges and recent progress, NPJ Precision Oncology, 4, 19, 10.1038/s41698-020-0122-1 · DOI
  33. 2022, Unraveling drug response from pharmacogenomic data to advance systems pharmacology decisions in tumor therapeutics, Future Pharmacology, 2, 31, 10.3390/futurepharmacol2010003 · DOI
  34. Kerstin Lenhof , Lea Eckhart , Lisa-Marie Rolli , Andrea Volkamer , and Hans-Peter Lenhof . (2023). Reliable anti-cancer drug sensitivity prediction and prioritization. · DOI
  35. John Hammersley. (2013). Monte Carlo methods. Springer Science & Business Media.
  36. Ryan J Tibshirani , Rina Foygel Barber , Emmanuel Candes , and Aaditya Ramdas . Conformal prediction under covariate shift. Advances in Neural Information Processing Systems, 32, 2019.
  37. 2021, Critical assessment of conformal prediction methods applied in binary classification settings, Journal of Chemical Information and Modeling, 61, 4823, 10.1021/acs.jcim.1c00549 · DOI
  38. 10.1093/bioinformatics/btl242 · DOI · Paper page
  39. 2007, Covariate shift adaptation by importance weighted cross validation, Journal of Machine Learning Research, 8, 5
  40. 10.1038/nbt.4235 · DOI · Paper page
  41. 2012, Reliable detection of subclonal single-nucleotide variants in tumour cell populations, Nature Communications, 3, 811, 10.1038/ncomms1814 · DOI
  42. Hendrik Mehrtens , Tabea Bucher , and Titus J Brinker . (2023) Pitfalls of conformal predictions for medical image classification. In International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, pages 198–207. Springer Nature Switzerlandr. · DOI
  43. 2017, Variational inference: A review for statisticians, Journal of the American Statistical Association, 112, 859, 10.1080/01621459.2017.1285773 · DOI
  44. 10.1109/TPAMI.2005.159 · DOI
  45. Sangwoo Park , Kfir M Cohen , and Osvaldo Simeone . (2023) Few-shot calibration of set predictors via meta-learned cross-validation-based conformal prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence. · DOI
  46. 2009, Cancer informatics by prototype networks in mass spectrometry, Artificial Intelligence in Medicine, 45, 215, 10.1016/j.artmed.2008.07.018 · DOI
  47. 2022, Providing post-hoc explanation for node representation learning models through inductive conformal predictions, IEEE Access, 11, 1202
  48. 2022, CPSC: Conformal prediction with shrunken centroids for efficient prediction reliability quantification and data augmentation, a case in alternative herbal medicine classification with electronic nose, IEEE Transactions on Instrumentation and Measurement, 71, 1
  49. Andrea Campagner , Marília Barandas , Duarte Folgado , Hugo Gamboa , and Federico Cabitza . (2024) Ensemble predictors: Possibilistic combination of conformal predictors for multivariate time series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence. · DOI
  50. 2017, Applying mondrian cross-conformal prediction to estimate prediction confidence on large imbalanced bioactivity data sets, Journal of Chemical Information and Modeling, 57, 1591, 10.1021/acs.jcim.7b00159 · DOI
  51. Henrik Boström and Ulf Johansson . (2020) Mondrian conformal regressors. In Conformal and Probabilistic Prediction and Applications, pages 114–133.
  52. Henrik Boström , Ulf Johansson , and Tuwe Löfström . (2021) Mondrian conformal predictive distributions. In Conformal and Probabilistic Prediction and Applications, pages 24–38.
  53. Feiyang Cai , Ali I Ozdagli , Nicholas Potteiger , and Xenofon Koutsoukos . (2021) Inductive conformal out-of-distribution detection based on adversarial autoencoders. In 2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS), pages 1–6. · DOI
  54. Mohamed Farag , Jana Kierdorf , and Ribana Roscher . (2023) Inductive conformal prediction for harvest-readiness classification of cauliflower plants: A comparative study of uncertainty quantification methods. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 651–659. · DOI
  55. Dongping Zhang , Angelos Chatzimparmpas , Negar Kamali , and Jessica Hullman . (2024) Evaluating the utility of conformal prediction sets for AI-advised image labeling. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–19. · DOI
  56. Gilad Kutiel , Regev Cohen , Michael Elad , Daniel Freedman , and Ehud Rivlin . (2023) Conformal prediction masks: Visualizing uncertainty in medical imaging. In International Workshop on Trustworthy Machine Learning for Healthcare, pages 163–176. Springer. · DOI
  57. 2022, Predicting Amazon customer reviews with deep confidence using deep learning and conformal prediction, Journal of Management Analytics, 9, 1, 10.1080/23270012.2022.2031324 · DOI
  58. Siddhartha Laghuvarapu , Zhen Lin , and Jimeng Sun . (2024) CoDrug: Conformal drug property prediction with density estimation under covariate shift. Advances in Neural Information Processing Systems, 36.
  59. Alexander S Millar , John Arnn , Sam Himes , and Julio C Facelli . (2024) Uncertainty in breast cancer risk prediction: A conformal prediction study of race stratification. In MEDINFO 2023—The Future Is Accessible, pages 991–995. IOS Press. · DOI
  60. Siddhartha Kapuria , Patrick Minot , Ariel Kapusta , Naruhiko Ikoma , and Farshid Alambeigi . (2024) A novel dual layer cascade reliability framework for an informed and intuitive clinician-ai interaction in diagnosis of colorectal cancer polyps. IEEE Journal of Biomedical and Health Informatics. · DOI
  61. 2024, Reliable anti-cancer drug sensitivity prediction and prioritization, Scientific Reports, 14, 12303, 10.1038/s41598-024-62956-6 · DOI
  62. 2022, Global burden of bacterial antimicrobial resistance in 2019: a systematic analysis, The lancet, 399, 629, 10.1016/S0140-6736(21)02724-0 · DOI · Paper page
  63. 2022, Machine learning for antimicrobial resistance prediction: current practice, limitations, and clinical perspective, Clinical microbiology reviews, 35, e00179
  64. 2018, Developing an in silico minimum inhibitory concentration panel test for klebsiella pneumoniae, Scientific reports, 8, 421, 10.1038/s41598-017-18972-w · DOI
  65. 2020, A biochemically-interpretable machine learning classifier for microbial gwas, Nature communications, 11, 2580, 10.1038/s41467-020-16310-9 · DOI
  66. 10.1002/pro.4497 · DOI · Paper page
  67. 2017, Netmhcpan-4.0: improved peptide–mhc class i interaction predictions integrating eluted ligand and peptide binding affinity data, The Journal of Immunology, 199, 3360, 10.4049/jimmunol.1700893 · DOI
  68. 10.1093/bioinformatics/btaa119 · DOI
  69. 2023, A guide to current methodology and usage of reverse vaccinology towards in silico vaccine discovery, FEMS Microbiology Reviews, 47, fuad004, 10.1093/femsre/fuad004 · DOI
  70. Balaji Lakshminarayanan , Alexander Pritzel , and Charles Blundell . (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
  71. Yonatan Geifman and Ran El-Yaniv . (2019) Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning, pages 2151–2159. PMLR.

Metadata

Authors
Papangelou, C., Kyriakidis, K., Natsiavas, P., Chouvarda, I., & Malousi, A.
Affiliations
  • Aristotle University of Thessaloniki (GR)
  • University of California, Santa Cruz (US)
  • Centre for Research and Technology Hellas (GR)
Corresponding author
Malousi, Andigoni
ORCID
Venue
Frontiers in Bioinformatics
Publisher
openRxiv
Published
2024-09-10
Volume
5
Pages
1507448-1507448
ISSN
2673-7647 (ISSN-L)
DOI
10.1101/2024.09.09.24312995
PMC
PMC11891349
OpenAlex
W4407895095
Cited by
12
References
2
Citations by year
2026: 8 · 2025: 3 · 2024: 1
FWCI
1.05
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Bioinformatics (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
conformal prediction, genomic medicine, uncertainty quantification, drug response prediction, molecular subtyping, distribution shift, clinical translation, personalized medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review