The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Conformal Prediction: A Gentle Introduction
Abstract
Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction (a.k.a. conformal inference) is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, timeseries, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run by following the code footnotes.</>
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Koenker, R., & Bassett, G. (1978) · Econometrica · doi:10.2307/1913643 · Cited by 13,316
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Koenker, R. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511754098 · Cited by 4,551
Pimentel, M. A. F., Clifton, D. A., Clifton, L., & Tarassenko, L. (2014) · Signal Processing · doi:10.1016/j.sigpro.2013.12.026 · Cited by 1,649
Algorithmic Learning in a Random World
(2005) · doi:10.1007/b106715 · Cited by 1,119
Distribution-Free Predictive Inference for Regression
Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1307116 · Cited by 1,049
Determination of Sample Sizes for Setting Tolerance Limits
Wilks, S. S. (1941) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731788 · Cited by 851
Symmetric measures on Cartesian products
Hewitt, E., & Savage, L. J. (1955) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1955-0076206-8 · Cited by 710
Logical basis for information theory and probability theory
Kolmogorov, A. (1968) · IEEE Transactions on Information Theory · doi:10.1109/tit.1968.1054210 · Cited by 592
The limits of distribution-free conditional predictive inference
Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2020) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaaa017 · Cited by 510
A graphical approach to sequentially rejective multiple test procedures
Bretz, F., Maurer, W., Brannath, W., & Posch, M. (2009) · Statistics in Medicine · doi:10.1002/sim.3495 · Cited by 420
On the concept of a random sequence
Church, A. (1940) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1940-07154-x · Cited by 390
Predictive inference with the jackknife+
Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2021) · The Annals of Statistics · doi:10.1214/20-aos1965 · Cited by 351
Statistical Prediction with Special Reference to the Problem of Tolerance Limits
Wilks, S. S. (1942) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731537 · Cited by 334
Grundlagen der Wahrscheinlichkeitsrechnung
Mises, R. V. (1919) · Mathematische Zeitschrift · doi:10.1007/bf01203155 · Cited by 308
Combinatorial foundations of information theory and the calculus of probabilities
Kolmogorov, A. N. (1983) · Russian Mathematical Surveys · doi:10.1070/rm1983v038n04abeh004203 · Cited by 307
Distribution-Free Prediction Sets
Lei, J., Robins, J. M., & Wasserman, L. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.751873 · Cited by 306
Kingmán, J. F. C. (1978) · The Annals of Probability · doi:10.1214/aop/1176995566 · Cited by 247
Exact and asymptotically robust permutation tests
Chung, E., & Romano, J. P. (2013) · The Annals of Statistics · doi:10.1214/13-aos1090 · Cited by 236
Conditional validity of inductive conformal predictors
Vovk, V. (2013) · Machine Learning · doi:10.1007/s10994-013-5355-6 · Cited by 225
An Extension of Wilks' Method for Setting Tolerance Limits
Wald, A. (1943) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731491 · Cited by 193
Regression conformal prediction with random forests
Johansson, U., Boström, H., Löfström, T., & Linusson, H. (2014) · Machine Learning · doi:10.1007/s10994-014-5453-0 · Cited by 184
A conformal prediction approach to explore functional data
Lei, J., Rinaldo, A., & Wasserman, L. (2013) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-013-9366-6 · Cited by 179
Tukey, J. W. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730343 · Cited by 173
Vovk, V. (2015) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-013-9368-4 · Cited by 166
Conformal Inference of Counterfactuals and Individual Treatment Effects
Lei, L., & Candès, E. J. (2021) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12445 · Cited by 166
Distribution-free cumulative sum control charts using bootstrap-based control limits
Chatterjee, S., & Qiu, P. (2009) · The Annals of Applied Statistics · doi:10.1214/08-aoas197 · Cited by 146
Distribution-free, Risk-controlling Prediction Sets
Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M. (2021) · Journal of the ACM · doi:10.1145/3478535 · Cited by 144
Self-citationClassification with confidence
Lei, J. (2014) · Biometrika · doi:10.1093/biomet/asu038 · Cited by 139
With Malice Toward None: Assessing Uncertainty via Equalized Coverage
Romano, Y., Barber, R. F., Sabatti, C., & Candès, E. J. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.03f00592 · Cited by 125
Additive models for quantile regression: Model selection and confidence bandaids
Koenker, R. W. (2011) · Brazilian Journal of Probability and Statistics · doi:10.1214/10-bjps131 · Cited by 105
Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., & Lei, L. (2021) · arXiv · doi:10.48550/arxiv.2110.01052 · Cited by 100
Self-citationConsistent Accelerated Inference via Confident Adaptive Transformers
Schuster, T., Fisch, A., Jaakkola, T., & Barzilay, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.406 · Cited by 80
Vovk, V. (2021) · Statistical Science · doi:10.1214/20-sts817 · Cited by 78
Fast exact conformalization of the lasso using piecewise linear homotopy
Lei, J. (2019) · Biometrika · doi:10.1093/biomet/asz046 · Cited by 55
Accelerating difficulty estimation for conformal regression forests
Boström, H., Linusson, H., Löfström, T., & Johansson, U. (2017) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-017-9539-9 · Cited by 50
Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
Malinin, A., Band, N., Ganshin, A., Chesnokov, G., Gal, Y., Gales, M. J. F., Noskov, A., Ploskonosov, A., Prokhorenkova, L., Provilkov, I., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07455 · Cited by 48
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T. (2022) · arXiv · doi:10.48550/arxiv.2208.02814 · Cited by 37
Self-citationKnowing what you know: valid and validated confidence sets in multiclass and multilabel prediction
Cauchois, M., Gupta, S., & Duchi, J. (2020) · arXiv · doi:10.48550/arxiv.2004.10181 · Cited by 31
Root-finding approaches for computing conformal prediction set
Ndiaye, E., & Takeuchi, I. (2022) · Machine Learning · doi:10.1007/s10994-022-06233-5 · Cited by 20
Cited by
Conformal prediction beyond exchangeability
Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2023) · The Annals of Statistics · doi:10.1214/23-aos2276 · Cited by 305
Lambert, B., Forbes, F., Doyle, S., Dehaene, H., & Dojat, M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102830 · Cited by 269
Pauly, L., Rharbaoui, W., Shneider, C., Rathinam, A., Gaudillière, V., & Aouada, D. (2023) · Acta Astronautica · doi:10.1016/j.actaastro.2023.08.001 · Cited by 121
Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., & Lei, L. (2021) · arXiv · doi:10.48550/arxiv.2110.01052 · Cited by 100
Self-citationSuperintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Mindermann, S., Oberman, A., Richardson, J., Richardson, O., et al. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i5.15569 · Cited by 41
Metadata
- Authors
- Angelopoulos, A. N., & Bates, S.
- Venue
- Foundations and Trends in Machine Learning
- Publisher
- Emerald
- Published
- 2023-03-27
- Volume
- 16
- Issue
- 4
- Pages
- 494-591
- ISSN
- 1935-8237 (ISSN-L)
- 1935-8245
- OpenAlex
- W4361012985
- Cited by
- 740
- References
- 42
- FWCI
- 65.01
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.