A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“…new candidate a , the true reward satisfies: ε i ← (cid:12)(cid:12) R ( a i ) − g ( x ( a i )) (cid:12)(cid:12) 4: end for 5: Determine α 1 − δ as the (1 − δ ) quantile of { ε 1 , ε 2 , . . . , ε M } 6: return g ( · ) and α 1 − δ where δ ∈ (0 , 1) is the user-specified miscoverage rate [5], [6].”

“While these methods provide intuitive uncertainty representations, their calibration often fails in practice due to overfitting or distributional mismatches, leading to overconfident or overly conservative predictions [27].”

“Non-exchangeable CP [126]; DDCRP [127];Exchangeability [72], [128]; KnowNo [129]; Rejection/stopping rule [96]; Reflexion [130]; UALA [131]; CRITIC [132]; DebUnc [73]; ReConcile [133]; MAD [134]; Human uncertainty Fluency/Adecuacy [135]; Perceptual similarity [136];DistCE [137]; LID [138];…”

“To measure the significance of an individual event within the CP framework, we can use the confidence [38].”

“Robust CP is an extension of CP which is a statistical tool for uncertainty quantification [2, 34, 60, 66].”

“In this paper, we propose a method to detect evolving HTs using uncertainty-aware unimodal and multimodal learning which leverages the algorithm-agnostic statistical inference technique of conformal prediction [6] as shown in Fig.1.”

“To enhance the effectiveness of the CSQ framework, we propose to construct candidate sets guided by prediction uncertainty from a trained model using conformal prediction (Shafer & Vovk, 2008; Angelopoulos et al., 2023).”

“Conformal prediction addresses this limitation by establishing a systematic framework to construct prediction sets with provable coverage guarantee [5, 6, 7, 8].”

“Over recent years, significant progress in reliable ML has been achieved through the use of conformal prediction (Vovk, Gammerman, & Shafe, 2005; Shafer & Vovk, 2008; Angelopoulos & Bates, 2023).”

“The design of our framework is inspired by conformal prediction (Shafer and Vovk 2008, Angelopoulos et al. 2023, Fontana et al. 2023).”

“This function can be defined arbitrarily as long as it returns a real-valued number that is representative of the non-conformity of a sample [24].”

“…inference methods is to produce prediction intervals that provably achieve the optimal level of coverage without making any distributional assumptions on the underlying data generating process and using the predictions of any algorithm as input (Shafer and Vovk, 2008; Angelopoulos and Bates, 2023).”

“Alternatively, conformal prediction 49 models could be established to estimate the aleatoric uncertainty and by that give the medicinal chemists a better idea of the reliability of an implemented model.”

“CP has emerged as an effective method to quantify uncertainty in ML models by providing prediction sets instead of single predictions [27].”

“This calibration process relies on the theory of conformal prediction [29]. a) Driving Perception Model: Recall that a driving perception model M p takes images and returns predictions on object category Cl , attribute At , and tracking Tr .”

“…r as a constraint rather than a soft objective, it is not affected by scaling the adversarial objective function; and 2) because r is always a scalar, we can use conformal prediction Shafer and Vovk (2008); Angelopoulos et al. (2023) to determine its value in practice with a small calibration set.”

“Conformal prediction, proposed by Vovk et al. (2005); Shafer and Vovk (2008), provides a framework for distribution-free uncertainty quantification, which constructs confidence intervals that are valid with exchangeable data from any underlying distribution and with any “black-box” algorithm.”

“Conformal prediction [82] is used to generate valid prediction regions (orange circles) that quantify the uncertainty of neural network–based trajectory forecasts (orange dots) [81]. reason about safety with formal uncertainty guarantees.”

“Conformal prediction provides a general, model-agnostic framework for constructing prediction sets with finite-sample, distribution-free coverage guarantees [Vovk et al., 2005, Shafer and Vovk, 2008, Angelopoulos and Bates, 2023].”

“This coverage guarantee is distribution-free and valid in finite samples, while also allowing the user to set their own error tolerance α [27, 26].”

“Unlike traditional Bayesian or error-variance methods, it provides distribution-free and finite-sample tools to construct rigorous prediction intervals that bound the surrogate’s approximation error with a pre-specified confidence level [17].”

“139 This indicates that standardized protocols (>80% ASTM C157 specimens) and overwhelming compositional factor influence effectively masked geometry-related effects.”

“Conformal prediction (Vovk et al., 2005; Shafer & Vovk, 2008) provides distribution-free prediction sets with finite-sample coverage guarantees under exchangeability.”

“Conformal Prediction (CP) (Vovk, Gammerman, and Saunders 1999; Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008; Angelopoulos and Bates 2021) is a post-hoc framework that provides distribution-free prediction sets with guaranteed marginal coverage (aka. statistically valid coverage),…”

“Even though conformal prediction (CP) [47, 43, 30, 2] provides a distribution-free recipe for constructing prediction sets with finite-sample marginal coverage under exchangeability, valid prediction intervals should further be as short as the conditional distribution permits while adapting to…”

“In response, conformal prediction (CP) was recently used to construct prediction regions that capture uncertainty in predicted future trajectories of uncontrollable agents with a desired probability (Shafer and Vovk, 2008; Cleaveland et al., 2024).”

“2) Uncertainty Quantification: Conformal prediction (CP) [36] is a distribution-free uncertainty estimation method that constructsvalidpredictionintervals.”

“Conformal inference provides distribution-free, finite-sample guarantees for uncertainty quantification without requiring assumptions on the data distribution (Shafer and Vovk, 2008).”

“Sequential data X 1 , ⋯ , X N under exchangeability are prevalent in modern statistics, such as in permutation testing and conformal prediction [37, 40, 27, 2].”

“…to provide appropriate predictive intervals is often as valuable However, standard split conformal prediction only provides a marginal guarantee over the population and cannot guarantee conditional coverage for specific instances— precisely what practitioners require in high-stakes scenarios.”

“Reliability-focused learning spans conformal prediction [22], selective prediction [3], and robust objectives [15]; these emphasize coverage/sets or worst-case risk rather than selectively releasing point refinements after ˆ Y base is fixed.”

“Test-time augmentation (36), Conformal prediction (37), DUQ (38), and SNGP (39) offer lightweight alternatives, showing promise on MedMNIST (16) and VinDr-Mammo (20).”

“Conformal prediction (CP; [25, 21]) has emerged as a promising statistical framework to address this need.”

“This theorem is a direct application of conformal prediction guarantees [AB23, SV08].”

“Recently, conformal prediction (CP) methods have been widely employed for constructing prediction regions under weak assumptions Vovk et al. (2005); Shafer & Vovk (2008).”

“Conformal prediction (CP) (Gammerman et al., 1998; Vovk et al., 2005; Shafer and Vovk, 2008) provides a rigorous, model-agnostic framework for quantifying uncertainty.”

“A total of 8000 training and 2000 test points are sampled uniformly from x To evaluate the CLFs on this task, we use techniques from conformal prediction [60].”

“The SCAPE framework introduces a novel approach to uncertainty quantification by leveraging a high-dimensional non-conformity score space [5, 35], contrasting with the previously introduced methods that rely on independent thresholds for binary classifiers [2, 6, 30].”

“Its core idea is to determine precise confidence levels in new predictions from past experience [23].”

“These raw confidence scores { ˆ 𝑝 ( 𝑦 | 𝑜 𝑖 )} = are calibrated using conformal prediction [2, 27, 29], providing probabilistic guarantees on these class predictions matching the ground truth labels 𝑌 ∗ 𝑘 .”

“A central focus of conformal prediction is efficiency , commonly quantified by the expected measure of the prediction set (Shafer & Vovk, 2008).”

“In the scalar case, conformalized quantile regression (CQR; Shafer & Vovk, 2008; Romano et al., 2019; Angelopoulos et al., 2023) adjusts quantile estimates to achieve valid intervals.”

“An example of this approach would be conformal prediction (CP) [10].”

“Conformal prediction (CP) turns arbitrary model scores into uncertainty sets with distribution-free, finite-sample guarantees under minimal assumptions, spanning classical tutorials and modern variants such as split CP for regression and conformalized quantile regression [18, 32, 34, 39].”

“One of its key strengths is its ability to maintain robustness under distribution shifts, a property that has led to its widespread adoption in fields such as time series analysis (Shafer and Vovk, 2008; Papadopoulos, 2008).”

“A promising solution to the problem of uncertainty quantification for individual counterfactual outcomes comes from conformal prediction [1,2], a post-hoc calibration method that provides statistically valid prediction intervals without strong distributional assumptions.”

“Therefore, we propose a new statistically rigorous uncertainty quantification technique for NO models, based on locally adaptive conformal inference [44, 22], called Local Spectral Conformal Inference (LSCI).”

“A good tutorial of this resource can also be found in [34].”

“It was formally introduced in [44], and has been shown to provide valid marginal coverage under the assumption of exchangeability in data [42].”

“CP [31] has emerged as a promising UQ method, offering statistically valid prediction sets without stringent data distribution assumptions [17].”

“In particular, we use the conformal inference framework (Shafer and Vovk 2008, Lei et al. 2018) to provide prediction regions for new observations.”

“Conformal prediction is a powerful framework for uncertainty quantification that has been widely used to predict a set of candidate outcomes for input data (Shafer & Vovk, 2008; Vovk, 2015; Pa-padopoulos et al., 2002).”

“To achieve this, we adopt a conformal inference procedure (Vovk et al., 1999; 2005; Shafer & Vovk, 2008) that selects a subset of plausible prediction candidates with a statistical coverage guarantee.”

“We designed AURC[F1]* to detect class-imbalanced rejection behavior masked by good aggregate error ranking.”

“As a popular CP framework, split conformal prediction [42] introduces a hold-out calibration set D cal = { ( x i , y i ) } n i =1 consisting of labeled examples drawn from P XY , which are exchangeable.”

“Currently, there are di ff erent methodologies addressing this topic: conformal prediction (CP) [25], which deals with the discovery of the confidence interval of each prediction, thus giving a clear indication of the quality of the prediction.”

“For more information on conformal prediction, see Shafer & Vovk (2008); Gibbs & Candès (2021).”

“Unlike traditional methods, which may provide a point estimate with no accompanying measure of reliability, conformal prediction offers a statistically valid way to associate each prediction with a confidence level (Shafer and Vovk, 2008).”

“CP is introduced to provide a robust statistical framework for quantifying prediction uncertainty (e.g., [2, 46, 55]).”

“This motivates the usage of split conformal regression (Shafer & Vovk, 2008), which uses a hold-out calibration set to provide statistically guaranteed bounds in probability (under the assumption of exchangeability).”

“Shafer and Vovk (2008) give a user-friendly introduction to conformal prediction.”

“Theorem 3.1 (SCP Coverage Guarantee (Shafer and Vovk, 2008)) .”

“Additional examples can be found in [2, 48].”

“LLMs EDL[69],DER[70],ConformalFactuality[71],Bayesian[72,73],Calibration[74,75], ConformalPrediction[76,77,78],UaIT[79],LUQ[80] ExternalTools SUE[81],CalibrateMath[82] Alignment(§4.2) DataCleaning andCuration JoSEC[83],StochasticParrots[84],DeepSoftDebias[85] RL-based Alignment PPO[3],DPO[86],GRPO[87],RLAIF[88] In-con…”

“Conformal prediction [32], [33] starts with the question: Given N + 1 random variables (scores) s 1 , . . . , s N , s N +1 which are exchangeable, equally likely under permutation, 1 what can be said about the value of s N +1 relative to s 1 , . . . , s N ?”

“Conformal methods have recently emerged as an approach to using data to obtain valid prediction regions under minimal assumptions (Vovk et al., 2005; Shafer and Vovk, 2008; An-gelopoulos et al., 2023).”

“This means that conformal prediction can be applied on top of machine learning algorithms such as random forests, boosted tree models, and support vector machines without requiring any further adaptation (Fontana, Zeni, and Vantini, 2023; Shafer and Vovk, 2008).”

“Conformal prediction (Shafer and Vovk 2008; Vovk, Gammerman, and Shafer 2005; An-gelopoulos, Bates et al. 2023) is a method that provides valid measures of uncertainty for predictions in a variety of machine learning tasks, ensuring that the coverage probability of prediction intervals/sets is…”

“Our causal inference framework, designed for predicting missing counterfactuals, allows us to construct inference procedures based on conformal prediction, as introduced by Shafer and Vovk (2008), to ensure robustness against misspecification.”

“The solution is based upon conformal prediction [Vovk et al., 2005, Shafer and Vovk, 2008, Lei and Wasserman, 2014].”

“CP (Shafer & Vovk, 2008) gives an answer to the following question: Given some arbitrary dataset ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , ..., ( X n , Y n ) , and some machine learn-1 Prediction Sets Figure 1: Inductive CP Framework over a Deterministic Model (see RES in Section 3.1.2): ( 1) Perform…”

“Conformal Prediction (CP) methods [Shafer and Vovk, 2008], provide a set of plausible values for the output Y with a given input X .”

“…in their most commonly used form - split conformal inference - work by calibrating the predictions of the model on a held-out dataset in order to provide sets which contain the output with a given probability, see Shafer and Vovk (2008) and Angelopou-los and Bates (2021) for good introductions.”

“Conformal prediction (CP) converts a point predictor into a set-valued predictor with finite-sample, distribution-free marginal coverage (Vovk et al., 2005; Shafer & Vovk, 2008; Lei et al., 2018; Angelopoulos & Bates, 2023).”

“Conformal prediction (CP) uses a calibration sample to convert any predictive model into a set predictor with marginal coverage guarantees, under the assumption that the calibration and test data are exchangeable [11], [12].”

“Furthermore, these authors frequently motivate conformal prediction by drawing connections to the work of Gosset, Fisher, Neyman, and others in the early to mid 1900s; see, e.g., Chapter 13.3 of Vovk et al. (2022), and also Shafer and Vovk (2008) for related discussion.”

“Given a predictor fixed before calibration and calibration observations exchangeable with the test observation, split conformal prediction constructs finite-sample valid prediction sets (Vovk et al., 2005; Shafer and Vovk, 2008; Lei et al., 2018).”

“This study lies at the intersection of conformal prediction [Shafer and Vovk, 2008, Lei et al., 2018, Angelopoulos and Bates, 2021] and latent-variable inference [Bartholomew et al., 2011, Everett, 2013].”

“Conformal prediction (CP) [30, 31, 25, 1] provides a general framework for doing so, where given a feature vector X , one constructs a prediction set for the corresponding response variable Y .”

“Split conformal prediction (Vovk et al., 2005; Papadopoulos et al., 2002; Angelopoulos and Bates, 2023; Shafer and Vovk, 2008) produces prediction sets with distribution-free finite-sample coverage and has been extended to regression (Romano et al., 2019; Lei et al., 2018), classification with…”

“No distri - butional assumptions on residuals are imposed—CP is valid for any model family, from linear regression to gradient-boosted trees [51].”

“The key advantage of CP is its model-agnostic nature: it can be applied to any base classifier without requiring modifications to the underlying model architecture [13].”

“Conformal prediction provides a general, distribution-free framework for constructing predictive sets with finite-sample coverage guarantees [35, 42].”

“The observed marginal coverage of 0.77 at a nominal 80% confidence level is consistent with finite-sample variability, as empirical coverage fluctuates around the nominal rate under full exchangeability [34, 85].”

“Conformal prediction (CP) and conformal risk control (CRC) provide distribution-free, finite-sample guarantees on user-specified loss functions (Shafer and Vovk, 2007; Angelopoulos et al., 2024).”

“regression and classification tasks (Shafer & Vovk, 2008; Angelopoulos et al., 2023; Papadopoulos et al., 2002; Vovk et al., 2005).”