A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“It can be shown [29] that is the p -th smallest nonconformity score is an upper bound for R 0 ; note that if p = N + 1 , inequality (3) is trivially valid but uninformative.”

“Conformal prediction has been adapted to various traditional machine learning models, like linear regression, support vector machines, and nearest neighbors [37], as well as neural networks [32, 33].”

“More recently, conformal prediction techniques (Shafer & Vovk, 2008) have gained popularity for providing confidence intervals in predictions.”

“Conformal Prediction (CP) is a powerful and flexible distribution-free uncertainty quantification technique that can be applied to any machine learning model (Angelopoulos & Bates, 2021; Shafer & Vovk, 2008).”

“Conformal prediction methods have been recently applied in the field of regression problems to address the point prediction limitation (Shafer and Vovk, 2008).”

“Conformal prediction is a principled, distribution-free approach of uncertainty quantification (Angelopoulos and Bates, 2021; Shafer and Vovk, 2008).”

“Confor-mal prediction (Lei et al., 2018b; Romano et al., 2019; Lei et al., 2018a; Shafer & Vovk, 2008) offers distribution-free coverage guarantees for uncertainty intervals, but classical variants focus on marginal coverage and are not readily adapted to the conditional, sample-specific…”

“• Independent per-stage CP (Shafer & Vovk, 2008): cal-ibrates each stage separately at 1 − α .”

“Some approaches, such as conformal prediction, provide finite-sample coverage guarantees under minimal assumptions [ 67, 68 ] .”

“To address this, we incorporate a conformal prediction (CP) calibration step that adjusts the safety threshold using held-out calibration errors, yielding finite-sample probabilistic coverage guarantees (Shafer & Vovk, 2008; Lindemann et al., 2025).”

“To mitigate potential miscalibration in LLM-generated lists of targets and relevant objects, which leads to irrelevant information, we employ conformal prediction (CP) [11] to calibrate object lists for message generation.”

“CI has become a popular technique for quantifying model uncertainty in the form of interval predictions (Shafer & Vovk, 2008).”

“Conformal Prediction (CP) (Zhou et al., 2025; Shafer and Vovk, 2008; Angelopoulos and Bates, 2021) offers such a framework: a model-agnostic method that provides formal statistical guarantees that the correct answer is included with a user-specified probability.”

“To address this issue, we build on conformal prediction [14, 15], a flexible statistical framework for uncertainty quantification that can construct principled prediction sets based on the output of any model.”

“A variety of post-hoc methods enforce this property on the outputs of a predictive model, including Platt scaling (Platt, 1999) and conformal prediction (Shafer & Vovk, 2008).”

“Introduced by Vovk et al. (2005); Shafer & Vovk (2008) and reviewed in An-gelopoulos et al. (2024), conformal p -values are now central to two-sample tests (Hu & Lei, 2024) and conditional testing (Wu et al., 2024a), outlier detection (Bates et al., 2023; Zhang et al., 2022), and data selection…”

“We hypothesized that information lift could distinguish high-quality, coherent responses from generic, incoherent, or repetitive ones.”

“Expand use of stochastic and probabilistic modeling approaches, particularly Bayesian frameworks and machine learning (Shafer and Vovk 2008), to support robust uncertainty partitioning of model projections and improve representation of natural variability versus model error.”

“Conformal prediction provides a powerful, model-agnostic framework for constructing predictive sets with guaranteed finite-sample marginal coverage (Vovk et al., 2005; Shafer and Vovk, 2008).”

“Conformal prediction (Shafer and Vovk, 2007) is a robust statistical method for uncertainty quantification that provides predictive intervals with a specified confidence level.”

“ML-based OPF solvers trained on such data may generalize poorly, leading to inaccurate predictions and erroneous uncertainty quantifi-cation [28], [29].”

“Conformal Prediction (CP) [30] is a distribution-free framework to provide confidence intervals of predictions with reliable coverage.”

“Conformal prediction (CP) fulfills this need by producing finite-sample valid prediction intervals for virtually any ML model under broad assumptions on the data-generating process [58].”

“To rigorously guide the sampling process and determine how many samples to use for the estimation, we draw inspiration from the conformal prediction (CP) (Shafer & Vovk, 2008; Fontana et al., 2023) that can efficiently estimate the sampling deviation.”

“Among UQ methods, we choose Conformal Prediction (CP) [23] as the approach for quantifying uncertainty in multi-step time series forecasting.”

“…prediction set is obtained for each data instance that, with a pre-determined probability, covers the true response value (this probability is also known as the coverage rate) (Shafer and Vovk, 2008; Vovk et al., 2009; Vovk, 2013; Burnaev and Vovk, 2014; Lei and Wasserman, 2014; Lei et al., 2018).”

“Conformal prediction enables the construction of confidence sets that contain the true outcome with a specified probability (Shafer & Vovk, 2007; Angelopoulos & Bates, 2022; Barber et al., 2023).”

“( iii) Finally, we need an approach to quantify the confidence we have in the captions by leveraging techniques from learning theory, such as conformal prediction [45].”

“Then a small calibration dataset will be utilized to calculate the nonconformity score, which describes the similarity between the given sample and the ground truth of the prediction [18].”

“Introduced and developed by Vovk, see (Vovk et al., 2005; Gammerman & Vovk, 2007; Shafer & Vovk, 2008), Conformal Prediction (CP) is a robust and flexible tool to quantify and handle the inherent uncertainty of these models.”

“One method that is widely used to quantify uncertainty is conformal prediction [3, 32, 36].”

“It has different names, such as predictive interval [18], interval predictor model (IPM) [14], [39], and conformal predictor [41].”

“Conformal inference, introduced by [39], is designed to quantify uncertainty in predictive modeling and has become an important framework in both statistics and machine learning.”

“A u t ho r M anu sc r i p t validity and efficiency 33–35 , were evaluated.”

“Introduced in the field of machine learning, the conformal prediction of Shafer and Vovk (2008) and Fontana et al. (2023) is used to construct probabilistic forecasts calibrated on out-of-sample errors.”

“The conformal curve is therefore essential: it separates the question “does the model rank anomalies above normals?” from “how many alerts will we actually send?” and follows the broader conformal literature on calibrated outlier scori ng, p-values, and reliable anomaly control [18 – 22][27].”

“Predictive uncertainty has been studied through Bayesian neural networks, Monte Carlo dropout, ensembles, conformal prediction, and related meth-ods [15, 16, 17, 18, 19].”

“Foundations of Split CP. CP is a distribution-free framework for constructing prediction sets with finite-sample coverage guarantees (Shafer & Vovk, 2008; Angelopoulos & Bates, 2021 , where x i is the input question, y i is its corresponding answer, n ∗ denotes the size of each respective subset,…”

“Model-agnostic methods such as conformal prediction (Shafer and Vovk [2008]) are often used to create post-hoc prediction intervals from a trained time series model (Barman et al. [2025], Panja et al. [2026]).”

“Conformal prediction is a statistical methodology that enhances the reliability of ML models by generating prediction intervals that are guaranteed to contain the true outcome with a specified confidence level [79].”

“Conformal Prediction for Structured and Hierarchical Data Conformal prediction methods have gained widespread adoption due to their minimal distributional assumptions and mathematically rigorous coverage guarantees 21–23 , 42 .”

“We therefore wrap probe scores in split conformal prediction (Vovk et al., 2005, Shafer and Vovk, 2008, Angelopoulos and Bates, 2021), turning them into an adaptive stop/continue rule with a user-chosen miscoverage level.”

“This is considered by Mohri and Hashimoto [2024]; Cherian et al. [2024]; Rubin-Toles et al. [2025], who use p-values based conformal prediction [Shafer and Vovk, 2008; Vovk et al., 2022a] to achieve this.”

“For instance, recent papers focus on how conformal prediction [34] can improve the joint decision-making of human-AI teams [10, 37].”

“Overviews of the field are provided by Vovk et al. (2005), Shafer & Vovk (2008), and Angelopoulos et al. (2023).”

“To combat learning errors, we use conformal prediction (CP) [5] to calibrate the FRS.”

“Conformal prediction [63] is a general framework for constructing set predictors that offer formal, distribution-free calibration guarantees and has been applied to problems such as symbol demodulation and modulation clas-sification [64].”

“We refer the reader to [1, 38] for more thorough introductions to conformal prediction.”

“Proof: The results of Shafer and Vovk [31] ( cf . pp. 382 and Appendix A) imply that the definitions of ˆ L i ( w ) and ˆ T i ( w ) in Lines 14 and 15, respectively, of Algorithm 1 guarantee that (5) and (6) are satisfied with probability at least 1 − α .”

“Given the VLM M vl and a labeled calibration dataset that is distributed identically to the task domain, using conformal prediction [22], we learn a calibration function f C : [0 , 1] (cid:55)→ [0 , 1] that maps a confidence score, c ∈ [0 , 1] to a lower bound for the probability of correct…”

“( 4) Conformal prediction methods [32] use a nonconfor-mity measure for calibration and derive the prediction interval from a set of predictions with residual errors [7].”

“Among various uncertainty estimation approaches, conformal prediction is recognized as an approach that offers finite-sample coverage guarantees for constructing prediction sets through distribution-free calibration [2, 24].”

“Theoretical guarantees for conformal prediction stem from the notion of exchangeability, which ensures that the predictions made on unseen data are valid, meaning they will cover the true values with the desired probability 1 − 𝛼 [39].”

“Conformal prediction is a distribution-free framework for constructing prediction sets that provide valid coverage guarantees [Papadopoulos et al., 2002, Vovk et al., 2005, Shafer and Vovk, 2008].”

“Typical calibration techniques include Platt scaling [29], temperature scaling [12], Bayesian binning [26], and conformal prediction [31, 35].”

“Another technique which has received interest recently is conformal prediction, which uses past data to determine a prediction interval or set in which future points are predicted to fall with high probability (Shafer & Vovk, 2008; Vovk et al., 2005).”

“Split conformal prediction is a method for uncertainty quantification that generates prediction intervals with statistical guarantees (Vovk et al., 2005; Shafer and Vovk, 2008).”

“Hence, taking for granted exchangeability [54], mixing the dataset, shuffling the sessions and trials is at least a challenging assumption.”

“Another method that became popular in the last couple of years is conformal prediction [69, 70], which addresses the problem of constructing prediction bands ˆ C m : X → { subsets of B } for a given level α with the property that for a new i.i.d. pair ( a i , b i ) , we get P ( b m +1 ∈ ˆ C n ( a m…”

“Conformal prediction (CP) has emerged as a popular method for such uncertainty quantification, providing distribution-free, finite-sample coverage guarantees under assumptions on data exchange-ability (Shafer and Vovk, 2008; Vovk et al., 2005).”

“The nonconformity measure quantifies the “strangeness” of a given object by comparing it to the objects previously encountered by the model during training and calibration [35].”

“Here, notable methods include quantile regression (Koenker & Bassett, 1978), conformal prediction (Shafer & Vovk, 2008), or direct variance prediction (Nix & Weigend, 1994).”

“Conformal prediction (CP) constructs prediction intervals with finite-sample coverage guarantees under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023).”

“Conformal prediction provides finite-sample marginal coverage under exchangeability and can be implemented through full or split calibration schemes (Vovk et al., 2005; Papadopoulos et al., 2002; Shafer and Vovk, 2008; Barber et al., 2023; Angelopoulos and Bates, 2023).”

“For general background on conformal prediction, we refer the reader to the monographs (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023; Angelopoulos et al., 2024) for introductions to both practical and theoretical aspects of the subject.”

“Conformal prediction (Shafer & Vovk, 2008; Angelopoulos et al., 2023) provides distribution-free uncertainty guarantees, but its worst-case nature often leads to overly conservative uncertainty regions and does not align with the fine-grained per-point 3D geometry (Mo et al., 2019; Fillioux et al.,…”

“Conformal Risk Control (Angelopoulos et al., 2024) extends conformal prediction (Shafer & Vovk, 2008) to control the expected value of a monotone risk function.”

“Conformal prediction (Shafer & Vovk, 2008; Angelopoulos & Bates, 2021) and calibration methods provide finite-sample coverage guarantees, and they may provide confidence estimates of whether a set of LLM outputs contains the correct answer.”

“Let q Inspired by conformal prediction [25], [26], this quantile provides a finite-sample calibration threshold to define the importance region as: Intuitively, T ( k ) captures states where the monitor’s prediction error exceeds the calibrated tolerance level, focusing refinement on systematically…”

“Conformal Prediction (CP) is a lightweight statistical tool for uncertainty quantification that can enable practical safety guarantees for autonomous systems [63], [64].”

“…to addressing this problem is to draw inspiration from existing inferential paradigms that can accommodate black-box machine learning algorithms, such as double machine learning [23] and conformal inference [102], and extend these ideas to settings where synthetic data play a central role.”

“12,14,32 .”

“Conformal prediction is a general, distribution-free framework for post-hoc uncertainty quantification (UQ) [41, 38, 25].”

“Conformal prediction [Vovk et al., 2022, Shafer and Vovk, 2008] provides distribution-free coverage guarantees: prediction sets that contain the true label with probability at least 1 − α , without distributional assumptions beyond exchangeability.”

“Conformal Prediction is a set of tools introduced in [13], [11] and often used in regression and classification tasks to construct guaranteed prediction sets that contain the true outcome with a specified probability.”

“Below we provide the formal definition of exchangeability from [10].”

“A conformal prediction framework (Shafer & Vovk, 2008 ) has been adopted, which provides distribution-free predictive intervals with valid coverage guarantees.”

“…make a step in this direction by introducing a broadly applicable methodology grounded in the theories of conformal prediction (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023) and density-based clustering (Rodriguez and Laio, 2014; Ester et al., 1996; Kriegel et al.,…”

“Conformal Prediction (CP) is an attractive framework to produce prediction regions with finite-sample guarantees of validity Vovk et al. [2005], Shafer and Vovk [2008].”

“One promising avenue is to adapt model-agnostic approaches such as conformal inference (Shafer and Vovk 2008), which can provide finite-sample guarantees under minimal assumptions.”

“Conformal Prediction (CP) [10], [14] is a framework for generating prediction sets for any model with statistical guarantees.”

“Such noise can violate the underlying exchangeability assumption required by conformal prediction (Shafer and Vovk, 2008).”

“We show that the resulting decomposition terms provide information that is not available from simply comparing the average length of unconditionally calibrated prediction intervals, which is commonly performed in practice (Shafer and Vovk, 2008; Fontana et al., 2023).”

“All those variants are valid choices (Vovk et al., 2005; Shafer & Vovk, 2008), and the literature is not conclusive regarding which one is to be preferred (Fong & Holmes, 2021; Fontana et al., 2023).”

“One of the most widely used and robust frameworks for quantifying uncertainty in statistical and machine learning models is conformal inference [Shafer and Vovk, 2008].”