Conformal Prediction: A Gentle Introduction
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction (a.k.a. conformal inference) is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, timeseries, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run by following the code footnotes.</>
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Koenker, R., & Bassett, G. (1978) · Econometrica · doi:10.2307/1913643 · Cited by 13,316
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Koenker, R. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511754098 · Cited by 4,551
Pimentel, M. A. F., Clifton, D. A., Clifton, L., & Tarassenko, L. (2014) · Signal Processing · doi:10.1016/j.sigpro.2013.12.026 · Cited by 1,649
Algorithmic Learning in a Random World
(2005) · doi:10.1007/b106715 · Cited by 1,119
Distribution-Free Predictive Inference for Regression
Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1307116 · Cited by 1,049
Determination of Sample Sizes for Setting Tolerance Limits
Wilks, S. S. (1941) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731788 · Cited by 851
Symmetric measures on Cartesian products
Hewitt, E., & Savage, L. J. (1955) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1955-0076206-8 · Cited by 710
Cited by
Conformal prediction beyond exchangeability
Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2023) · The Annals of Statistics · doi:10.1214/23-aos2276 · Cited by 305
Lambert, B., Forbes, F., Doyle, S., Dehaene, H., & Dojat, M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102830 · Cited by 269
Pauly, L., Rharbaoui, W., Shneider, C., Rathinam, A., Gaudillière, V., & Aouada, D. (2023) · Acta Astronautica · doi:10.1016/j.actaastro.2023.08.001 · Cited by 121
Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., & Lei, L. (2021) · arXiv · doi:10.48550/arxiv.2110.01052 · Cited by 100
Self-citationSuperintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i5.15569 · Cited by 41
Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions
Lekeufack, J., Angelopoulos, A. N., Bajcsy, A., Jordan, M. I., & Malik, J. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610041 · Cited by 18
Self-citationReliable Text-to-SQL with Adaptive Abstention
Chen, K., Chen, Y., Koudas, N., & Yu, X. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3709719 · Cited by 12
CONSeg: Voxelwise Uncertainty Quantification for Glioma Segmentation Using Conformal Prediction
Elyassirad, D., Gheiji, B., Vatanparast, M., Ahmadzadeh, A. M., & Faghani, S. (2025) · American Journal of Neuroradiology · doi:10.3174/ajnr.a8914 · Cited by 10
Berk, R. A., & Braverman, A. (2025) · arXiv · doi:10.48550/arxiv.2503.16118
Conformal Risk Training: End-to-End Optimization of Conformal Risk Control
Yeh, C., Christianson, N., Wierman, A., & Yue, Y. (2025) · arXiv · doi:10.48550/arxiv.2510.08748
Metadata
- Authors
- Angelopoulos, A. N., & Bates, S.
- Venue
- Foundations and Trends in Machine Learning
- Publisher
- Emerald
- Published
- 2023-03-27
- Volume
- 16
- Issue
- 4
- Pages
- 494-591
- ISSN
- 1935-8237 (ISSN-L)
- 1935-8245
- OpenAlex
- W4361012985
- Cited by
- 740
- References
- 43
- FWCI
- 65.01
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.