A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“Conformal prediction provides distribution-free marginal coverage guarantees (Gammerman et al., 1998; Saunders et al., 1999; Vovk et al., 2005), with extensive work on regression (Shafer & Vovk, 2008; Lei & Wasserman, 2014).”

“Conformal prediction is a flexible, model-agnostic technique for measuring uncertainty in predictions (Shafer and Vovk, 2008a).”

“Elements in an exchangeable sequence are identically distributed, but not necessarily independent [40].”

“In conformal prediction, calibration data are used to build prediction sets with finite-sample coverage under weak distributional assumptions; see [1, 16, 18, 21, 23].”

“Conformal prediction is a technique that uses nonconformity scores computed on a held-out calibration set to construct a prediction set containing the ground truth solution with high probability (Angelopoulos and Bates, 2023; Angelopoulos et al., 2024; Shafer and Vovk, 2008; Vovk et al., 2005).”

“Given a miscoverage level ϵ , standard conformal methods provide finite-sample marginal coverage of at least 1 − ϵ under exchangeability and the conditions of the chosen conformal procedure [161].”

“Conformal prediction (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopou-los and Bates, 2023; Romano et al., 2019; Lei and Candès, 2021) and distribution-free risk con-trol (Bates et al., 2021; Angelopoulos et al., 2022, 2025; Gibbs and Candes, 2021) provide principled frameworks for calibrating…”

“We supplemented this search with backward citation search for foundational work in classical forecasting [19, 71], proper scoring rules [20, 54, 55], conformal prediction [169], forecasting competitions [122, 123], and human superforecasting [181].”

“Classical conformal prediction obtains finite-sample marginal coverage from exchangeability; standard regression treatments include both split and candidate-wise conformal constructions (Vovk et al., 2005; Lei et al., 2018; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023).”

“The guarantee is finite-sample, distribution-free and model-agnostic, which accounts for the method’s wide adoption; Shafer and Vovk (2008) give a tutorial treatment and Angelopoulos and Bates (2023) and Fontana et al. (2023) recent surveys.”

“Conformal prediction provides distribution-free thresholding from calibration data (Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008), and functional conformal methods extend this idea to curve-or trajectory-valued scores (Lei, Rinaldo, and Wasserman 2015).”

“Linde-mann et al. [141] addressed this issue by using conformal prediction [153] to quantify the uncertainty of learning-based trajectory predictions.”

“Conformal prediction offers distribution-free, finite-sample predictive inference under exchangeability [23], [24], [25].”

“Conformal prediction gives finite-sample predictive coverage under exchangeability (Vovk et al., 2005; Papadopoulos et al., 2002; Shafer and Vovk, 2008; Barber et al., 2023; Angelopoulos and Bates, 2023).”

“…dataset and a user-defined nonconformity score, conformal methods compute an empirical quantile of the calibration scores to construct prediction sets with distribution-free marginal coverage, where a future test sample is contained in the conformal set with a prescribed probability [45].”

“Conformal prediction provides a model-agnostic way to construct calibrated prediction intervals with a finite-sample coverage interpretation under exchangeability assumptions (Vovk et al., 2005; Shafer and Vovk, 2008; Stephen et al., 2021).”

“However, CP assumes exchangeability between the data used to calibrate the framework and the data it is applied to [58], which may not hold if there are changes to the search pipelines or detectors, raising the question of whether the conditional confidence scores shown in Fig.”

“In statistics and machine learning, conformal prediction of Shafer & Vovk (2008)”

“When calibration and deployment covariates differ, importance-weighted conformal prediction preserves distribution-free guarantees under a weighted exchangeability assumption [39].”

“Conformal prediction (Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008) offers a complementary approach that provides distribution-free coverage guarantees, and recent work has begun applying conformal methods to LLM outputs (Wang et al. 2024; Vishwakarma et al. 2024).”

“To address the limitation in ESO-based CBF designs for a known disturbance-variation bound, we further develop a sliding-window conformal prediction mechanism [10] (blue block) to estimate this bound online in a data-driven and less conservative manner.”

“A key advantage of CP is that it can be applied as a post-hoc wrapper around arbitrary forecasting models while providing finite-sample coverage guarantees under relatively mild assumptions (Shafer & Vovk, 2008).”

“Since s RPS is a real-valued noncon-formity score and the calibration scores are exchangeable with the test score, the standard quantile-based construction (3) guarantees marginal coverage [Vovk et al., 2005, Shafer and Vovk, 2008].”

“In parallel, conformal prediction (CP) has emerged as a powerful framework for distribution-free uncertainty quantification in machine learning [21, 22, 23, 24, 25].”

“…rather than diversity: a model that produces many visibly different futures is not trust-worthy unless its predicted distribution achieves nominal coverage of held-out outcomes (CRPS, reliability curves), with conformal prediction [49] offering one route to finite-sample coverage guarantees.”

“…be expressed as follows: Motivated by conformal prediction, which uses past experience to determine precise levels of confi-dence in new predictions (Shafer and Vovk 2008), we assemble a data set { (ˆ s j , q j ) } Jj =1 where each q j ∈ R T such that q tj := Q t/ ( T +1) (ˆ s j ) (˜ ξ ) , and Q α…”

“We assume that the calibration graphs and the test graph are i.i.d. samples from the same underlying distribution (Liao et al., 2021; Veliˇckovi´c et al., 2018), and therefore exchangeable as required for conformal calibration (Shafer & Vovk, 2008; Vovk et al., 2005).”

“From an AI point of view, formalising such a multi-task problem implies deciding the mathematical form of the outputs the model will generate 176 (e.g., a vector, a set, as in conformal learning 177 , a tensor 178 , a probability distribution 179 , a credal set 180,181,182 , a random set 183,184 ).”

“Conformal prediction [19, 20] offers exactly such a framework.”

“…et al., 2018) unify aleatoric and epistemic components under a single Dirichlet predictive distribution, while conformal prediction frameworks (Shafer & Vovk, 2008; Angelopoulos & Bates, 2021; Barber et al., 2023) provide distribution-free uncertainty sets with finite-sample coverage…”

“Conformal prediction (CP) is a versatile methodology for constructing prediction sets that contain the unknown label of a test point with finite-sample, distribution-free guarantees [3, 12, 32, 39, 44, 45, 50, 51].”

“…uses a fully connected generator with hidden layers [ 128, 96, 64, 32, 16 ] and a fully connected discriminator with hidden layers [ 64, 48, 32, 16 ] , with batch normalization, LeakyReLU activations, latent dimension 32, label smoothing { 0.9, 0.1 } and a curriculum-supervision schedule…”

“CP [40, 29] provides a path toward bridging this gap, which is a model-agnostic and distribution-free framework characterized by its rigorous finite-sample marginal coverage guarantee.”

“…prediction interval is then Under the usual exchangeability assumption between the calibration and test samples, this conformal construction yields finite-sample marginal coverage approximately 90% for the binary response Y , regardless of whether the working model is correctly specified [19, 30].”

“To ensure high-fidelity supervision, we employ Conformal Prediction (Shafer & Vovk, 2008) to derive statistically valid, attribute-specific thresholds.”

“Conformal prediction has emerged as a powerful tool for constructing prediction sets with distribution-free coverage guarantees (Papadopoulos et al., 2002; Vovk et al., 2005; Shafer and Vovk, 2008).”

“Conformal prediction (Shafer & Vovk, 2008; Papadopoulos et al., 2008; Balasubramanian et al., 2014; Vovk, 2012; Angelopoulos & Bates, 2021) provides distribution-free prediction sets but primarily captures aleatoric uncertainty and is sensitive to miscalibration (Bethell et al., 2024).”

“Vovk et al., 2005, Shafer and Vovk, 2008].”

“Quantile Regression (Koenker, 2005) and Lower Upper Bound Estimation (Khosravi et al., 2011) estimate interval bounds without explicit distributional assumptions, while conformal prediction (Vovk et al., 2005; Shafer & Vovk, 2008) ensures statistically valid coverage for arbitrary models.”

“Conformal prediction provides distribution-free uncertainty sets under exchangeability (Lei et al., 2018; Shafer and Vovk, 2008; Vovk et al., 2005) and has been extended in several directions for dependent or shifting data.”

“and its embedded data form a point cloud in the phase space.”

“Conformal prediction (Vovk et al, 2005; Shafer and Vovk, 2008; Romano et al, 2019; Barber et al, 2021; Barber et al, 2023; 2023) is a forecasting method rather than a checking method; because its outputs are sets without underlying densities, the coverage half of our pipeline applies directly (the…”

“Conformal prediction (CP) is a distribution-free framework for constructing prediction sets with finite-sample marginal coverage guarantees, requiring only that the data are exchangeable [Shafer and Vovk, 2008, Papadopoulos et al., 2008, Vovk et al., 2022].”

“Conformal prediction constructs prediction regions with finite-sample coverage guarantees under minimal assumptions on the data distribution [32, 39].”

“We test both via split conformal prediction (CP) [27], calibrated on 100 held-out trajectories ( v g ∈ [0 .”

“One method that has had some success is CP, a distribution-free framework for uncertainty quantification that offers formal statistical guarantees (Shafer and Vovk, 2008; Lei et al., 2013).”

“…as temperature scaling and Platt scaling could map raw model scores into more reliable probabilities before a gating decision (Guo et al., 2017; Platt, 1999; Kuleshov et al., 2018), and conformal methods offer another route to coverage guarantees under stronger assumptions (Shafer & Vovk, 2008).”

“…a decoder-only Transformer [23] on certificate-assisted tightened labels to replace multi-step propagation with autoregressive set-valued prediction, and apply split conformal prediction [24], [25] to obtain distribution-free finite-sample coverage guarantees for realized output trajectory points.”

“Conformal Prediction (CP) [30], [31] is a widely used statistical inference that constructs prediction sets with coverage guarantees for the true target, while the set size provides an informative measure that dynamically reflects model uncertainty.”

“…expository resources on conformal prediction already exist, including books (Vovk et al., 2022a, Angelopoulos et al., 2024), literature surveys (Shafer and Vovk, 2008, Tian et al., 2022, Fontana et al., 2023, Zhou et al., 2025), and practitioner-oriented tutorials (Angelopoulos and Bates,…”

“See, for example, Gammerman et al. (1998), Vovk and Shafer (2008), Toccaceli and Gammerman (2019), Angelopoulos and Bates (2021) and Vovk (2022).”

“…forecasts into distribution-free prediction sets and sees growing use in motion planning and predictive control with learned trajectory models (Shafer and Vovk, 2008; Lei, G’Sell, Rinaldo, Tibshirani, and Wasserman, 2018; Angelopoulos and Bates, 2023; Dixit, Lindemann, Wei, Cleaveland,…”

“Uncertainty arising from the learned model’s confidence can be quantified through conformal prediction approaches, which allow us to obtain confidence intervals over predictions [43]; these can then be aggregated across the test set to determine the uncertainty of the overall measured performance.”

“It can be applied at inference time to any base predictor (Shafer & Vovk, 2008; Saunders et al., 1999; Vovk et al., 2017).”

“We use conformal inference [66, 78] to develop a distribution-agnostic algorithm that counts points in D t whose causal factors fall within the distribution of the source-learned set K s from D s .”

“Conformal prediction (Shafer and Vovk, 2008 ; Angelopoulos and Bates, 2023 ) provides distribution-free coverage guarantees by wrapping any black-box predictor with a quantile-based calibration step.”

“By the fundamental property of exchangeable sequences (see Shafer and Vovk [2008, Lemma 1] or Vovk et al. [2005, Proposition 1]), the test score R Step 3 (Combining).”

“%) without assuming any specific error distribution or model structure [32, 33].”

“Recent work has demonstrated that Conformal Prediction (CP) [16, 5, 12] can provide finite-sample valid coverage guarantees for scalar metrics derived from black-box imaging pipelines [3].”

“…2023; Ravi et al., 2024; Friel and Sanyal, 2023) and uncertainty measures (Tomani et al., 2024; Shelmanov et al., 2025), struggle because models can be confidently wrong (Kuhn et al., 2023; Simhi et al., 2025) and because “hallucination” lacks a consistent operational definition (Qi et al., 2024).”

“To address the challenges of distribution assumptions and the absence of theoretical guarantees, we propose to leverage Conformal Prediction (CP) [25, 21, 10, 3, 29] to calibrate a global acceptance threshold on a calibration set.”

“Given any predictor g and an i.i.d. calibration set, split conformal constructs intervals C ( X ) with finite-sample, distribution-free marginal coverage, P ( Y ∈ C ( X )) ≥ 1 − α , under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos et al., 2023; Zhou et al., 2025).”

“…a split conformal (or inductive conformal) approach, where the dataset is divided into a training set for model fitting and a disjoint calibration set for uncertainty quantification (Papadopoulos et al., 2002b; Vovk et al., 2005; Shafer & Vovk, 2008; Angelopoulos & Bates, 2021; Lei et al., 2015).”

“Conformal prediction addresses this limitation by providing distribution-free coverage guarantees [35, 36].”

“Among available approaches, conformal prediction has emerged as a particularly attractive framework (Vovk, Gammerman and Shafer, 2005; Shafer and Vovk, 2008; Lei et al., 2018; Angelopoulos and Bates, 2023).”

“∼ P XY and a test point ( X n +1 , Y n +1 ) , the goal is to ensure for any α ∈ (0 , 1) under an exchangeability assumption between the calibration and test point, i.e., their joint distribution is invariant under permutations [24, Section 3].”

“Conformal prediction (CP) has become the dominant paradigm for distribution-free uncertainty quantification (Shafer and Vovk, 2008, Vovk et al., 2005).”

“…a rich history in both conventional modeling and the more recent deep learning wave, with techniques for the latter including Bayesian neural networks (Neal, 1992), neural network uncertainty via dropout (Gal & Ghahramani, 2016), and conformal prediction (Vovk et al., 2005; Shafer & Vovk, 2008).”

“For a general treatment of conformal prediction, see Vovk et al. (2005) and Shafer and Vovk (2008); for applications of conformal prediction in insurance, see Hong and Martin (2021) and Hong (2026).”

“Among all the uncertainty quantification methods, conformal prediction (CP) stands out due to its simplicity and distribution-free characteristics (Vovk et al., 2005; Shafer & Vovk, 2008; Angelopoulos & Bates, 2021).”

“Prior CAD works [36], [37] Conformal Prediction (CP) [5], [6], [15] filtering to discard suboptimal candidate architectures before incurring expensive evaluation costs.”

“Conformal prediction provides a principled framework for constructing prediction sets with finite-sample marginal coverage guarantees (Shafer & Vovk, 2008; Angelopoulos et al., 2024).”

“CP is a generic framework to post-calibrate any arbitrary (possibly imperfect) predictor and ensures a valid finite sample statistical guarantee under the weak assumptions of data exchangeability [30; 31].”

“Developed by Vovk, Gammerman, and Shafer (Shafer & Vovk, 2008; Vovk et al., 2022), CP transforms the point predictions of any underlying algorithm into prediction sets (for classification) or intervals (for regression).”

“Over the past decade, conformal prediction has emerged as a powerful distribution-free framework for calibration of probabilistic predictions with rigorous theoretical guarantees (Angelopoulos & Bates 2023; Fontana et al. 2023; Shafer & Vovk 2008).”

“Conformal prediction (CP) (Shafer and Vovk 2008; Fontana, Zeni, and Vantini 2023) is a statistical framework for distribution-free, finite-sample coverage guarantees.”

“Following the hypothesis testing interpretation of conformal prediction, each candidate time t ∈ T can be viewed as testing a null hypothesis H 0 : T n +1 = t (Shafer and Vovk, 2008).”

“Conformal prediction provides a general framework for constructing prediction sets with guaranteed finite-sample coverage under minimal distributional assumptions [9, 10, 11].”

“The conformal prediction paradigm delivers model-independent, assumption-free machinery for systematic uncertainty characterization [25], [26], [27].”

“…sets or intervals with finite-sample coverage guarantees under the mild assumption of exchangeable data (Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008; An-gelopoulos, Bates et al. 2023; Romano, Patterson, and Can-des 2019; Ghosh et al. 2023a,b; Shi et al. 2024; Shahrokhi et al.…”

“A variety of scoring functions have been introduced in the literature (Shafer and Vovk 2008; Angelopoulos et al. 2021; Huang et al. 2023).”

“Conformal Prediction (CP) (Vovk, Gammerman, and Saunders 1999; Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008; Angelopoulos and Bates 2021) is a post-hoc framework that provides distribution-free prediction sets with guaranteed marginal coverage (aka. statistically valid coverage),…”

“To mitigate this, recent works have leveraged Conformal Prediction (CP) (Angelopoulos and Bates, 2023; Shafer and Vovk, 2008) to quantify uncertainty in LLM-based planners (Ren et al., 2023; Wang et al., 2025b).”

“Second, we employ conformal prediction (CP), a statistical framework for uncertainty quantification in black-box models [9], [10], to quantify uncertainty of these path predictors and construct prediction sets around The authors are with Department of Electrical and Systems Engineering, Washington…”