The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Predicting With Confidence: Using Conformal Prediction in Drug Discovery
Open accessAbstract
One of the challenges with predictive modeling is how to quantify the reliability of the models' predictions on new objects. In this work we give an introduction to conformal prediction, a framework that sits on top of traditional machine learning algorithms and which outputs valid confidence estimates to predictions from QSAR models in the form of prediction intervals that are specific to each predicted object. For regression, a prediction interval consists of an upper and a lower bound. For classification, a prediction interval is a set that contains none, one, or many of the potential classes. The size of the prediction interval is affected by a user-specified confidence/significance level, and by the nonconformity of the predicted object; i.e., the strangeness as defined by a nonconformity function. Conformal prediction provides a rigorous and mathematically proven framework for in silico modeling with guarantees on error rates as well as a consistent handling of the models' applicability domain intrinsically linked to the underlying machine learning model. Apart from introducing the concepts and types of conformal prediction, we also provide an example application for modeling ABC transporters using conformal prediction, as well as a discussion on general implications for drug discovery.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Another limitation of most AI/ML models is that they do not assign a confidence interval to the predictions.277,278”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [278]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Data Mining and Knowledge Discovery Handbook
(2010) · doi:10.1007/978-0-387-09823-4 · Cited by 2,190
Data Mining for Imbalanced Datasets: An Overview
Chawla, N. V. (2009) · Springer eBooks · doi:10.1007/978-0-387-09823-4_45 · Cited by 1,197
On a simple approach for determining applicability domain of QSAR models
Roy, K., Kar, S., & Ambure, P. S. (2015) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2015.04.013 · Cited by 800
Role of the Breast Cancer Resistance Protein (BCRP/ABCG2) in Drug Transport—an Update
Mao, Q., & Unadkat, J. D. (2014) · The AAPS Journal · doi:10.1208/s12248-014-9668-6 · Cited by 708
A tutorial on conformal prediction
Shafer, G., & Vovk, V. (2007) · arXiv · doi:10.48550/arxiv.0706.3188 · Cited by 655
The Role of ABC Transporters in Drug Resistance, Metabolism and Toxicity
Glavinas, H., Krajcsi, P., Cserepes, J., & Sarkadi, B. (2004) · Current Drug Delivery · doi:10.2174/1567201043480036 · Cited by 559
The P-glycoprotein multidrug transporter
Sharom, F. J. (2011) · Essays in Biochemistry · doi:10.1042/bse0500161 · Cited by 528
Norinder, U., Carlsson, L., Boyer, S. K., & Eklund, M. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci5001168 · Cited by 208
Self-citationVovk, V. (2013) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-013-9368-4 · Cited by 174
Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications
Balasubramanian, V. N., Ho, S.-S., & Vovk, V. (2014) · Cited by 155
Role of the bile salt export pump, BSEP, in acquired forms of cholestasis
Stieger, B. (2009) · Drug Metabolism Reviews · doi:10.3109/03602530903492004 · Cited by 134
Principles of Clinical Pharmacology
Atkinson, A. J. (2007) · Elsevier eBooks · doi:10.1016/b978-0-12-369417-1.x5039-1 · Cited by 129
Principles of Clinical Pharmacology
Zakusov, V. V. (2012) · Elsevier eBooks · doi:10.1016/c2009-0-63932-5 · Cited by 96
The application of conformal prediction to the drug discovery process
Eklund, M., Norinder, U., Boyer, S. K., & Carlsson, L. (2013) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-013-9378-2 · Cited by 73
Self-citationSvensson, F., Aniceto, N., Norinder, U., Cortés‐Ciriano, I., Spjuth, O., Carlsson, L., & Bender, A. (2018) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.8b00054 · Cited by 72
Self-citationSun, J., Carlsson, L., Ahlberg, E., Norinder, U., Engkvist, O., & Chen, H. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00159 · Cited by 69
Self-citationModelling compound cytotoxicity using conformal prediction and PubChem HTS data
Svensson, F., Norinder, U., & Bender, A. (2016) · Toxicology Research · doi:10.1039/c6tx00252h · Cited by 63
Self-citationResampling methods improve the predictive power of modeling in class-imbalanced datasets
Lee, P. (2014) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph110909776 · Cited by 62
Binary classification of imbalanced datasets using conformal prediction
Norinder, U., & Boyer, S. K. (2017) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2017.01.008 · Cited by 59
Self-citationA confidence predictor for logD using conformal regression and a support-vector machine
Lapins, M., Arvidsson, S., Lampa, S., Berg, A., Schaal, W., Alvarsson, J., & Spjuth, O. (2018) · Journal of Cheminformatics · doi:10.1186/s13321-018-0271-1 · Cited by 58
Self-citationATP binding cassette systems: structures, mechanisms, and functions
Licht, A., & Schneider, E. (2011) · Open Life Sciences · doi:10.2478/s11535-011-0054-4 · Cited by 54
Aggregated Conformal Prediction
Carlsson, L., Eklund, M., & Norinder, U. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44722-2_25 · Cited by 50
Self-citationA New Imbalanced Learning and Dictions Tree Method for Breast Cancer Diagnosis
Parvïn, H., Bidgoli, B. M., & Alinejad‐Rokny, H. (2013) · Journal of Bionanoscience · doi:10.1166/jbns.2013.1162 · Cited by 46
Montanari, F., Knasmuller, B., Kohlbacher, S. M., Hillisch, C., Baierová, C., Grandits, M., & Ecker, G. F. (2020) · Frontiers in Chemistry · doi:10.3389/fchem.2019.00899 · Cited by 35
Integrated Decision Support for Assessing Chemical Liabilities
Spjuth, O., Eklund, M., Helgee, E. A., Boyer, S. K., & Carlsson, L. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci200242c · Cited by 33
Self-citationPredicting Off-Target Binding Profiles With Confidence Using Conformal Prediction
Lampa, S., Alvarsson, J., Arvidsson, S., Berg, A., Ahlberg, E., & Spjuth, O. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.01256 · Cited by 33
Self-citationReliable Confidence Predictions Using Conformal Prediction
Linusson, H., Johansson, U., Boström, H., & Löfström, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-31753-3_7 · Cited by 13
Improving peptide-MHC class I binding prediction for unbalanced datasets
Sales, A. P., Tomaras, G. D., & Kepler, T. B. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-385 · Cited by 11
Training conformal predictorsPreprint
Colombo, N., & Vovk, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07037 · Cited by 3
Flagged first, then by citations
Research works printed but not linked: 1
- Kubinyi, 1993, vol. 1
Cited by
Sorted by citations
Lambert, B., Forbes, F., Doyle, S., Dehaene, H., & Michel Dojat, D. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102830 · Cited by 353
GHOST: Adjusting the Decision Threshold to Handle Imbalanced Data in Machine Learning
Esposito, C., Landrum, G. A., Schneider, N., Stiefl, N., & Riniker, S. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.1c00160 · Cited by 229
Olsson, H., Kartasalo, K., Mulliqi, N., Capuccini, M., Ruusuvuori, P., Samaratunga, H., Delahunt, B., Lindskog, C., Janssen, E. A. M., Blilie, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34945-8 · Cited by 111
Self-citationGraph Neural Networks in Modern AI-Aided Drug Discovery
Zhang, O., Lin, H., Zhang, X., Wang, X., Wu, Z., Ye, Q., Zhao, W., Wang, J., Ying, K., Kang, Y., et al. (2025) · Chemical Reviews · doi:10.1021/acs.chemrev.5c00461 · Cited by 110
Predictive Modeling in Medicine
Toma, M., & Ong, C. W. (2023) · Encyclopedia · doi:10.3390/encyclopedia3020042 · Cited by 70
Metadata
- Authors
- Alvarsson, J., Arvidsson McShane, S., Norinder, U., & Spjuth, O.
- Affiliations
- Uppsala University (SE)
- Corresponding author
- Spjuth, Ola
- Venue
- Journal of Pharmaceutical Sciences
- Publisher
- Elsevier BV
- Published
- 2021-01
- Volume
- 110
- Issue
- 1
- Pages
- 42-49
- ISSN
- 0022-3549 (ISSN-L)
- OpenAlex
- W3093152954
- Cited by
- 116
- References
- 30
- Citations by year
- 2026: 16 · 2025: 30 · 2024: 24 · 2023: 16 · 2022: 15 · 2021: 15
- FWCI
- 5.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of Pharmaceutical Sciences (publishedVersion, cc-by)
- PubMed (publishedVersion)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Funders
- Svenska Forskningsrådet Formas (2018-00924)
- Stiftelsen för Strategisk Forskning (BD15-0008SB16-0046)
- Topics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Keywords
- conformal prediction, prediction intervals, QSAR, confidence estimation, applicability domain, ABC transporters, drug discovery, in silico modeling, forecast uncertainty
- MeSH terms
- Machine Learning, Algorithms, Molecular Conformation, Reproducibility of Results, Quantitative Structure-Activity Relationship, Drug Discovery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
