A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“We assume the standard split CP exchangeability condition on the data and choices [31, 33, 34].”

“We will explore more systematic approaches to further improve calibration in our future studies, for instance, by adding a conformal prediction layer, which provides distribution-free guarantees for UQ (Shafer and Vovk 2008).”

“Conformal prediction (CP) [27, 22] provides a model-agnostic solution for quantifying uncertainty over machine learning predictions.”

“Conformal Prediction (CP) provides such a principled, distribution-free framework for producing prediction sets with finite-sample coverage guarantees (Vovk et al., 2005; Shafer & Vovk, 2008).”

“Conformal prediction provides finite-sample distribution-free predictive guarantees under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023; Angelopoulos et al., 2026).”

“CE methods offer statistically grounded mechanisms for re-jecting low-confidence predictions, making them ideal for drift detection in streaming, real-time environments such as IoT security monitoring [3], [6], [17], [22], [30].”

“Recently, conformal prediction has emerged as a model-agnostic framework for constructing prediction intervals with finite-sample coverage guarantees under an exchangeability assumption (Shafer & Vovk, 2008; Angelopoulos & Bates, 2023).”

“This can be made formal using conformal prediction (CP, Shafer & Vovk, 2008; Jazbec et al., 2024; Ringel et al., 2024).”

“• Conformal Prediction (Shafer and Vovk, 2008).”

“Incorporating open ended action generation and structured medical guidelines into the latent path formulation remains an important direction for future research(Shafer and Vovk, 2008; Vovk et al., 2005).”

“This gap between accuracy and reliability has motivated extensive work on uncertainty quantification, including post-hoc calibration methods [64–66], conformal prediction [67,68], Bayesian neural networks [69], and ensemble approaches [70,71].”

“To address these challenges, we develop a conformal framework [27] that provides valid coverage guarantees while capturing meaningful variation across ignition scenarios and remaining computationally tractable.”

“Conditional conformal prediction is implemented through Mondrian (group-conditional) conformal prediction, 12,34,36 with related extensions studying conditional guarantees under covariate shift, 21 selection-conditional coverage, 37 and localized conformal methods.”

“Conformal prediction provides distribution-free finite-sample uncertainty sets under exchangeability [21].”

“It uses nonconformity scores to construct prediction intervals that are valid up to the user-defined probability level while being distribution agnostic [19], [42].”

“CP [Shafer and Vovk, 2008, Fontana et al., 2023] provides finite-sample coverage guarantees under minimal assumptions, requiring only exchangeability of the data, and can be applied on top of any pre-trained predictor without modifying training.”

“Conformal prediction (CP) (Vovk et al., 2005; Shafer & Vovk, 2008) provides finite-sample, distribution-free statistical guarantees through a well-defined procedure; yet, whether these procedural guarantees translate into equitable outcomes in downstream decision-making remains unclear.”

“Conformal prediction [10] provides distribution-free and informative bounds on prediction error.”

“Conformal Prediction (CP) [8], [9] provides such guarantees, but applying it to contingency screening is nontrivial due to the heterogeneous severity of N − k events and element-wise nature of grid violations.”

“CP constructs prediction regions with guaranteed coverage under exchangeability [Vovk et al., 2005, Shafer and Vovk, 2008].”

“Under exchangeability, calibration scores computed on past data are representative of the score that would be observed for a new test point, enabling valid predictive inference without strong parametric assumptions [Shafer and Vovk, 2008, Lei et al., 2018].”

“While we focus on regression here, the same framework extends to classification and other prediction tasks through appropriate choices of conformity scores (Shafer and Vovk, 2008; Angelopoulos and Bates, 2021), as detailed in Algorithm 1 .”

“In addition, to quantify predictive confidence in a principled and model-agnostic way, we adopt conformal prediction (CP) [31,32], which wraps around any trained classifier and produces statistically grounded measures of uncertainty.”

“Conformal Prediction (CP) is a lightweight statistical tool for uncertainty quantification that can enable practical safety guarantees for autonomous systems [23].”

“Design implications involve Multi-perspective AI outputs (Davani et al., 2022) and Conformal prediction (Shafer and Vovk, 2008): instead of presenting a single deterministic decision, AI should present multiple plausible classifications for the case at hand, highlighting the range of expert…”

“Moreover, these approaches do not typically provide finite-sample distribution-free marginal coverage guarantees under minimal assumptions [26, 27].”

“Our work proposes convex cost functions that will lead to conditional coverage in the infinite sample limit, and thus leads to good candidates for procedures like conformal prediction that can provably get marginal coverage [3, 4, 5].”

“In addition, we implemented uncertainty quantification via conformal prediction [61].”

“Proper calibration methodology, as formalized in conformal prediction [27, 28], requires exchangeability assumptions that are violated when train and test data share the same underlying source.”

“The popular framework known as Conformal Prediction (CP) [24, 25, 26] precisely aims to offer such guarantees.”

“Finally, besides typical density, local fit, and UQ methodologies, other approaches have been developed, namely conformal prediction [54] and metamorphic tests [55].”

“Early work introduced transductive and inductive conformal prediction, nonconformity scores, and rank-based calibration as model-agnostic tools for predictive inference (Saunders et al., 1999; Papadopoulos et al., 2002; Gammerman & Vovk, 2007; Shafer & Vovk, 2008; Papadopoulos et al., 2011).”

“2) Conformal Prediction: We obtain keypoint uncertainty bounds, the input to SLUE through (3), using split conformal prediction [21], [28].”

“Conformal prediction [17, 18] is a statistical approach for uncertainty quantification that operates without distributional assumptions.”

“Contextual bandits [9], [10], [35] and conformal prediction [36], [37] leverage features to improve prediction, but typically focus on single-agent settings.”

“Among these, conformal prediction stands out for its ability to provide distribution-free coverage guarantees [24].”

“Assuming only the exchangeability of the data, this offers a versatile framework that converts the outputs from any black-box prediction algorithm into a valid prediction set (Vovk et al., 2005; Shafer and Vovk, 2008; Lei et al., 2018).”

“Specifically, I is the set of interval inclusion functions where each M ∈ I is an interval function M : IR n → IR .”

“…goes one step in this direction by issuing a set of probabilistic predictions (i.e. a credal set) that is guaranteed to contain an out-of-sample calibrated prediction, under the assumption of exchangeable data (Gammerman et al., 1998; Shafer and Vovk, 2008; Vovk et al., 2017; Fontana et al., 2023).”

“Originally introduced by [6] and later extended by [7] and [126], CP constructs PIs with valid coverage guarantees, regardless of the underlying data distribution.”

“Separately, Conformal Prediction (CP) [10], [11] provides reliable uncertainty estimates by generating prediction sets with guaranteed coverage.”

“Rather than producing a single point value with estimated confidence, CP constructs a prediction set that contains the true label with a user-specified confidence level [7, 8, 9].”

“Compared to standard residual-based scores [Shafer and Vovk, 2008, Lei et al., 2018, Tibshirani et al., 2019], our choice yields smaller and more informative prediction sets.”

“Conformal prediction (CP) provides statistically rigorous prediction intervals with guaranteed coverage independent of model or data [15], [16].”

“Conformal prediction (CP) constructs prediction sets with a pre-specified probability 1 − α of containing the true label, without distributional assumptions beyond exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008).”

“Conformal prediction (Shafer & Vovk, 2008; Angelopou-los & Bates, 2022; Vovk et al., 2022) is a general-purpose method that constructs distribution-free prediction sets ˆ C α such that P ( ϑ ∈ ˆ C α ) ≥ 1 − α .”

“A common baseline is to clip the neural network output so that ˆ u ( ·; w) remains within admissible bounds.”

“Conformal prediction (CP) [12, 13, 14] is a distribution-free and model-agnostic methodology that generates prediction sets with theoretically guaranteed coverage probabilities.”

“Overviews of the field are provided by Vovk et al. [2005], Shafer and Vovk [2008], and Angelopoulos and Bates [2023].”

“Reducing this model deficiency through advanced statistical techniques [50] could improve empirical coverage and enhance the reliability of the estimates.”

“These lenient requirements make CP broadly applicable across various Machine Learning models, with use-cases including classi # cation, regression, anomaly detection, and active learning [43].”

“In this context, CP has emerged as an advantageous UQ framework due to its mathematical rigor and distribution-free nature [19, 13].”

“(1) achieved by [13], [15].”

“The significance of this technique cannot be overstated, given the unpredictable dynamics of hydrocarbon production; it’s crucial to have forecasts that not only pinpoint accuracy but also offer a defined level of confidence [19].”

“The challenge of ensuring reliable predictions for individual cases is tackled through the CP framework [7, 51].”

“Within this prediction framework, conformal prediction (Shafer and Vovk, 2008; Papadopoulos et al., 2008; Vovk et al., 2022) offers a machine learning approach for constructing prediction intervals with finite-sample marginal validity and asymptotic conditional validity.”

“The prediction intervals for the prediction of points made by the three ANN models are computed by inductive conformal prediction [49].”

“More formally, let us first define data exchangeability following the CP framework: Definition 3.1 (Exchangeability [Shafer and Vovk, 2008]) .”

“For more details, see [VGS05] and the surveys by [SV08, AB21].”

“CP is a model-agnostic, distribution-free uncertainty quantification (UQ) framework that provides set-valued predictions or intervals with finite-sample marginal coverage guarantees [2, 30].”

“CP is a statistical method that converts any model’s empirical conference value to a statistic rigorously guaranteed assurance [14], [15].”

“3. is intractable, so much focus has been devoted to studying performative stability, i.e., finding θ PS = arg min θ E z ∼D ( θ PS ) ℓ ( z ; θ ) .”

“Foundational works, such as [25], introduced the framework with error bounds for predictions, and [26] developed its theoretical foundations for probabilistic prediction.”

“Quite recently, Lei and Candès (2021) proposed to quantify the uncertainty of the predictive ITE using the conformal inference method (Vovk et al. 2005; Shafer and Vovk 2008).”

“Conformalpredictionisageneralframeworkfordistribution-free uncertainty quantification (Vovk et al. (2005); Shafer and Vovk (2008); Lei et al. (2018); Tibshirani et al. (2019); Angelopoulos et al. (2023); Barber et al. (2023)).”

“Conformal prediction (CP) (Shafer and Vovk 2008; Sesia and Romano 2021; Ndiaye 2022; Li et al. 2024; Sun et al. 2024) has emerged as a widely used technique for uncertainty quantification in deep neural networks (DNNs), offering statistically rigorous uncertainty sets for model predictions.”

“To ensure probabilistic safety guarantees under uncertainty, we employ conformal prediction (Shafer and Vovk (2008)).”

“…uncertainty quantification (UQ) framework is introduced based on deep ensembles (Lakshminarayanan et al., 2017) combined with conformal prediction (Shafer and Vovk, 2007; Angelopoulos and Bates, 2021), that provides prediction intervals over the predicted fields and enables local assessment of…”

“Conformal Prediction (CP) is a distribution-free framework for generating prediction intervals with guaranteed coverage probability 1 − α , requiring no assumptions about the data distribution [11].”

“Given a bag of conformity (or non-conformity) scores, {s 1 , … s N }, because of the exchangeability property of the data and symmetry, the score for any future sample, s n + 1 has an equal probability of occurring between any two successive scores (quanta) in an ordered set [7, 8].”

“This concept was first introduced in [60], and a more comprehensive theoretical background is discussed in [61].”

“Conformal prediction [73, 4, 63, 51] has gained much attention in recent years in the machine learning community.”

“The approach, termed Conformalized Distillation for Credal Inference (CD-CI), builds on conformal prediction (CP) [6], [7] and imprecise probability (IP) [8], [9].”

“More technical introductory accounts of the conformal prediction logic can be found in Gammerman and Vovk [ 101 ] and Shafer and Vovk [ 102 ].”

“To counter the aforementioned challenges, we first take in-spiration from an important tool in statistical learning, conformal prediction (CP) (Shafer & Vovk, 2008; Angelopou-los & Bates, 2021), which uses a post-training calibration step to guarantee a user-specified coverage.”

“One particular model is a machine learning algorithm model, Conformal Prediction[9][10][11], which can be used to produce interval predictions with valid edge coverage without making any distributional assumptions.”

“Weighted coverage measures the proportion of instances that are correctly classified and included within prediction sets, thereby indicating the model’s reliability in identifying uncertain regions relative to class distribution [34].”

“Split conformal prediction (SCP) Shafer and Vovk [2007], Jing et al. [2018] is an efficient alternative to FCP, in which data is split between training and validation sets, the latter being used to calibrate the model after training.”

“Conformal prediction is an emerging technique in machine learning that provides a way to quantify the uncertainty of model predictions [18].”

“One of the most widely used and robust frameworks for quantifying uncertainty in statistical and machine learning models is conformal inference [59].”

“Moreover, we adopt two prevalent metrics—coverage and efficiency—defined below to assess the performance of conformal prediction for node-wise traffic prediction (Shafer and Vovk 2008): where 𝑈 𝑖,𝑡 and 𝐿 𝑖,𝑡 are the upper and lower bounds produced by conformal prediction, respectively.”

“We can look at conformal prediction [1, 16, 23], quantile regression [14] and tools in deep learning with uncertainty (TensorFlow Probability, Bayesian Neural Networks, Epistemic Neural Networks etc.) see [5, 19, 20].”

“Shafer et al. [5] presented non-conformity measures for nearest neighbors, which measure how well a data point in a dataset deviates from the rest of the dataset.”

“…offers a robust statistical framework for assessing the uncertainty in predictive models, and has gained widespread applications in areas such as bioinformatics, financial modeling, and climate science (Vovk et al., 2005; Shafer and Vovk, 2008; Balasubramanian et al., 2014; Zeni et al., 2020).”

“Nevertheless, the proper selection of the estimator h ( · ) has a significant effect in conformal prediction efficiency [23].”

“Conformal prediction gives distribution-free finite-sample coverage under exchangeability [37, 38], while adaptive conformal inference updates the miscoverage level online to handle distribution shift [18].”

“Conformal prediction constructs prediction sets by calibrating nonconformity scores, which measure disagreement between predictions and observations (Vovk et al., 2005; Shafer and Vovk, 2008).”