A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“Conformal prediction, originally introduced by Vovk et al. 10,13 provides a principled and model-agnostic framework for uncertainty quantification.”

“[10], [11] provide only marginal coverage.”

“Conformal prediction is a post-hoc uncertainty quantification framework that produces prediction sets with guaranteed statistical coverage, with no assumptions about the underlying model [28].”

“For prediction in the regression setting, conformal prediction (e.g., Vovk et al. 2005, 2009; Shafer and Vovk 2008; Barber et al. 2021) is a model-free machine learning method for generating finite-sample valid prediction regions at a given confidence level.”

“As an alternative, a popular methodology known as conformal prediction (Shafer & Vovk 2008) can be used to construct probabilistic forecasts calibrated on out-of-sample interval forecast errors.”

“This problem, known as novelty detection (e.g., [1]), has attracted much attention recently through the lens of conformal p-values [2, 3, 4, 5, 6, 7, 8].”

“…)} 𝑚𝑗 = 1 , we construct a conformal prediction set for the 𝑡 -th horizon by including all labels whose predicted probabilities reach at least the threshold 1 − 𝑞 ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑠 : Again, this prediction set is guaranteed to include the true horizon label with probability at least 1 − 𝛼 [20].”

“By using an external calibration set, CP produce prediction sets whose size adapts to the data while guaranteeing finite-sample, model-agnostic, and distribution-free coverage [Shafer and Vovk, 2008, Balasubramanian et al., 2014, Tibshirani et al., 2019, Ding et al., 2023].”

“16 19 Conformal prediction provides guarantees on predictions by constructing calibrated prediction regions from past data, but it focuses on producing valid prediction sets rather than characterizing specific patients associated with higher uncertainty.”

“(Shafer and Vovk, 2008) explored its theoretical advantages for clinical decision support, highlighting how distribution-free guarantees can address the variability inherent in biological systems.”

“To address this challenge, we adopt an algorithm with provable guarantees based on the framework of conformal prediction ( CP ) which allows us to determine the transition epoch in a reliable manner [ 10 , 11 ].”

“To address the lack of uncertainty estimation in regular DNNs, various methods have been proposed (e.g., [21, 22, 33, 53, 62, 79, 87, 95, 96]).”

“Set-valued predictions are well studied in the single-domain framework, often through the lens of conformal prediction - a widely used method for constructing set-valued predictions with a predefined level of coverage guarantee (Shafer & Vovk, 2008; Vovk et al., 2005); see also Section 4.2.”

“Crucially, relying on a data exchangeability argument ( i.e. relaxed i.i.d. ’ness) a probabilistic coverage guarantee for unseen test samples can be provided, attaching a notion of reliability to obtained uncertainties [Shafer and Vovk, 2008].”

“Finally, we apply split conformal prediction (Shafer and Vovk, 2008) to the multi-label probabilities, generating valid confidence sets at a desired coverage level even under complex code co-occurrences.”

“Conformal Prediction (CP) (see, e.g., [Gammerman et al., 1998, Papadopoulos et al., 2002, Shafer and Vovk, 2008] or [Angelopoulos and Bates, 2023] for a modern introduction) has been designed from the ground up to be an uncertainty quantification statistical framework that provides marginal…”

“(1) Proposition 3.2 (Conformal Prediction for Regression (Shafer & Vovk, 2008)) .”

“Conformal prediction constructs prediction sets with guaranteed marginal coverage under the data exchangeability assumption [Shafer and Vovk, 2008, Angelopoulos et al., 2023, Plassier et al., 2024a,b].”

“That is Where E t,i,j = { e k,i,j | k ∈ [0 , 1 , ..., t ] } is a bag Shafer & Vovk (2008) of errors in the past times.”

“To address these challenges, we propose ConfEviSurrogate, a novel architecture that combines evidential deep regression (EDR) [2] and conformal prediction [3, 21, 23, 29] for a robust and well-calibrated UQ.”

“In Appendix C, we also present the conformal prediction interval approach of Shafer and Vovk (2008), which does not require a tuning parameter.”

“Conformal prediction has emerged as a powerful framework for quantifying uncertainty in machine learning models [10, 11].”

“As reported in Shafer and Vovk (2008), the variables z 1 , . . . , z N are exchangeable if for every permutation τ of the integers 1, . . . , N , the variables w 1 , . . . , w N , where w i = z τ( i ) , have the same joint probability distribution as z 1 , . . . , z N .”

“A popular alternative to models that output predictive probabilities are ones that perform conformal prediction (5; 54; 96; 148).”

“Given a desired error probability ϵ , conformal prediction uses the available data to generate a set of labels , typically containing ˆ y , containing the true label y corresponding to the feature x with a probability not smaller than 1 − ϵ (Shafer & Vovk, 2008).”

“To tackle this issue, this study employed methods from the conformal prediction framework 7,14 , using a calibration dataset of 1000 observations.”

“5.”

“RISE [55] proposed a non-i.i.d. index based on Conformal Prediction (CP) theory [56] for traditional machine learning models.”

“In recent years, the Conformal Prediction (CP) (Shafer & Vovk, 2007) has garnered significant attention within energy forecast models.”

“Conformal prediction (CP) is a general framework that can be employed to control the error rate of any classifier [8].”

“…operators from Linear Temporal Logic, whose Boolean satisfaction semantics can informally be defined as: Conformal prediction was introduced in [5, 6] to obtain valid prediction regions for complex predictive models such as neural networks without making assumptions on the distribution of the…”

“Conformal prediction (Shafer and Vovk, 2008; Angelopoulos et al., 2023) is a general uncertainty quantification framework that, independent of the underlying regression function g employed, provides non-asymptotic marginal (global) guarantees of the type In this case, the probability P is over the…”

“During the operation phase, each node computes a conformal p -value [13] from the previously learned nominal distribution and the present local summary statistic.”

“This can be used to test or by inverting the ac-5 ceptance region generating prediction-intervals [Shafer and Vovk, 2008, Haroush et al., 2021].”

“When building a prediction interval, the most efficient valid interval possible is desired (Shafer and Vovk, 2008).”

“A non-conformity score (a measure of how unusual a prediction looks relative to the response (Shafer & Vovk, 2008), for example, the absolute difference between the response and the prediction is a non-conformity score.) is then computed on the with-held point.”

“20 It is a distribution-free, post-hoc framework that operates on saved model predictions,so it can be applied to any auto contouring algorithm without touching the model.”

“To address these challenges, recent advances in conformal prediction (CP) have introduced prediction sets that provide distribution-free coverage guarantees [7].”

“Within the class of bounds on expected losses, conformal prediction (CP) [49, 45, 4] bounds expected miscoverage loss for set-valued predictions, whereas conformal risk control (CRC) [6] generalizes CP to bound the expectation of any bounded monotone loss function.”

“Given any underlying regres - sor, it converts residuals from a calibration set into prediction sets with finite-sample coverage guarantees under exchangeability, without distri - butional assumptions (Lei et al., 2018; Shafer and Vovk, 2008; Vovk et al., 2005).”

“Conformal prediction Conformal prediction (CP) is a data-based, distribution-free technique for constructing prediction sets, using a calibration set D calib = { ( x i , y i ) } n calib i =1 that is different from the train or test set [41].”

“Conformal prediction constructs prediction sets for future responses [Shafer and Vovk, 2008], whereas TabCUQ constructs confidence intervals for conditional means and CATEs.”

“Predictive uncertainty and term-wise support: When a separate calibration dataset is available, we quantify predictive uncertainty using split conformal prediction ( 57 ).”

“Conformal prediction provides finite-sample marginal coverage under exchangeability without a correctly specified prediction model (Shafer and Vovk, 2008; Angelopoulos and Bates, 2023).”

“Conformal methods provide finite-sample set coverage under exchangeability [50], [51].”

“Our method integrates conformal prediction, a distribution-free framework that produces statistically valid prediction intervals with formal coverage guarantees under the assumption of exchangeability between calibration and test data [10].”

“Conformal prediction uses calibration residuals or noncon-formity scores to construct prediction sets without specifying a parametric error distribution [6, 7].”

“(Shafer and Vovk (2008)) The features of online testing are summarized in Appendix A.”

“Instead, we aim to calculate the statistical discrepancy bounds between two controllers based on conformal prediction [18, 46, 50].”

“Selective classification with a reject option [42, 43] trades coverage for accuracy; conformal prediction supplies distribution-free finite-sample guarantees under exchangeability [44, 45], with label-conditional variants [46, 47] and conformal risk control for bounded losses [48].”

“Conformal prediction turns any trained classifier into such a procedure, with a finite-sample coverage guarantee and no assumptions on the data distribution [1, 39, 46].”

“Conformal prediction (Shafer & Vovk, 2008) is a general, distribution-free framework for uncertainty quantification that produces prediction sets with finite-sample coverage guarantees under mere exchangeability.”

“Under exchangeability, it offers finite-sample, distribution-free marginal coverage guarantees (Shafer & Vovk 2008, Lei et al. 2018), and it can be applied on top of any point forecast, whether from a statistical/machine learning model, or a judgmental process, without specifying a predictive…”

“Conformal prediction provides a framework for quantifying predictive uncertainty with finite-sample guarantees under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008).”

“Conformal prediction converts fixed model scores into finite-sample marginal prediction sets (Vovk et al., 2005; Shafer & Vovk, 2008; Papadopoulos et al., 2002; Lei et al., 2018).”

“Conformal prediction (CP) is a distribution-free framework for constructing prediction sets with finite-sample coverage guarantees [3].”

“Split conformal prediction (Vovk et al., 2005; Shafer and Vovk, 2008; Lei et al., 2018; Angelopoulos and Bates, 2023) turns such a heuristic uncertainty into intervals with a finite-sample, distribution-free coverage guarantee, provided that calibration examples with known truth are available.”

“Conformal prediction provides uncertainty sets with finite-sample, distribution-free coverage under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008).”

“The appeal of distribution-free prediction is that this guarantee can be obtained without imposing strong assumptions on the underlying data distribution [20, 19, 1], namely, While this guarantee is highly appealing, it has an important limitation.”

“We therefore use conformal prediction (CP) [17], [18], [46], a distribution-free statistical method, to calibrate two quantities: the KL-ball radius ϵ KL and the OOD threshold ϵ OOD .”

“Conformal prediction provides a general framework for converting the output of a predictive model into a prediction set with finite-sample coverage guarantees (Vovk et al., 2005; Shafer & Vovk, 2008; Angelopoulos et al., 2025).”

“The conformal approach (Shafer and Vovk, 2008) can be used to test the non-conformity of the data with resampled data.”

“Split conformal prediction (SCP) enables prediction confidence intervals to be calculated in unseen data samples [16], [17].”

“Conformal prediction provides a general framework for constructing prediction sets with finite-sample, distribution-free validity under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008).”

“Conformal Prediction for LLMs. Conformal prediction (CP) provides distribution-free prediction sets with finite-sample coverage under exchangeability [44, 34].”

“Conformal prediction also provides prediction-set coverage guarantees, including class-conditional coverage (Shafer and Vovk, 2008; Vovk, 2013; Romano et al., 2020).”

“Instead, it provides prediction intervals with finite-sample coverage guarantees under the mild assumption of exchangeable observations, meaning the joint distribution of the observation sequence is invariant under permutations, a condition implied by i.i.d. sampling [15].”

“…any predictor into set-or interval-valued predictions with finite-sample marginal coverage, assuming only exchangeability (Vovk et al., 2022; Shafer and Vovk, 2008); the split (inductive) construction makes it cheap enough to deploy around learned models (Papadopoulos et al., 2002; Lei et…”

“Conformal prediction is particularly attractive because it can provide distribution-free finite-sample coverage guarantees under exchangeability assumptions (Vovk et al. 2005; Shafer and Vovk 2008; Angelopoulos and Bates 2023).”

“Conformal prediction provide an alternative framework for constructing statistically valid prediction intervals with minimal distributional assumptions, offering a promising approach for uncertainty-aware AI systems in engineering applications [11 , 109 , 125].”