A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“We then calibrate the probes using methods from conformal prediction [81].”

“This has led to increased interest in uncertainty quantification methods, particularly conformal prediction, which constructs predictive regions with finite-sample validity under mild i.i.d. assumptions [Vovk et al., 2005, Shafer and Vovk, 2008].”

“Uncertainty Quantification (UQ) has been extensively studied in conventional ML (Gupta et al., 2006; Shafer and Vovk, 2008; Vaicenavicius et al., 2019; Tibshirani et al., 2019; Abdar et al., 2021), and is receiving increasing attention due to its potential to mitigate hallucinations in foundation…”

“In general, we can assign a label to each pixel by taking the maximum probability value; however, alternative techniques can be utilized, including multi-hypothesis prediction methods, e.g., conformal prediction [34].”

“The i.i.d. assumption is stronger than exchangeability, as the latter can hold with dependence (Shafer & Vovk, 2008).”

“11 CP provides a formal and structured approach to addressing questions that were often approached in a vague and abstract way in machine learning (ML) models, such as determining 2 . the confidence level of a model’s predictions.”

“For each task T , the upper threshold is estimated using calibration datasets S 1 and S 2 derived from successful expert demonstrations[37, 47]: where µ T is the mean anomaly score from S 1 , and Anomaly Handling.”

“Conformal prediction (CP) offers a general framework for constructing prediction sets with finite-sample coverage guarantees (Vovk et al., 2005; Shafer & Vovk, 2008).”

“There are several excellent, shorter introductions to the field of CP, which include Toccaceli (2022) and Shafer and Vovk (2008).”

“Conformal prediction (CP) (Vovk et al., 2005; Shafer & Vovk, 2008) is a statistical framework that addresses the accuracy of a predictive model.”

“To address this, conformal prediction (CP) [36, 33, 4], uses the (pre-trained) model π θ to construct, for an input X , a prediction set C ( X ) ⊆ Y that contains the true label with high probability, satisfying the desired coverage guarantee.”

“CP [11] [24] is a statistical framework designed to quantify uncertainty in the predictions of any deterministic algorithm.”

“Its credibility and potential make it widely applicable for quantifying the uncertainty of predictions produced by any black-box machine learning model (Shafer & Vovk 2008, Papadopoulos 2008, Barber et al. 2021) or non-parametric model (Lei & Wasserman 2014).”

“Conformal Prediction (CP) is a general framework for uncertainty quantification that provides statistically valid prediction regions for any predictive model under the mild assumption of data exchangeability [13, 34, 35].”

“Conformal prediction : Conformal prediction (CP) is a statistical framework [40], [41] that aims at calibrating the output of black-box models via data-driven post-processing.”

“Conformal prediction has emerged as a popular and modern technique for uncertainty quantification by providing valid predictive inference for those black-box models [8, 2].”

“Conformal selection is based on the conformal prediction (CP) method, 15 − 17 which is initially developed by Vovk et al. 18 It is a model-free framework for uncertainty quantification, offering probabilistic guarantees on range estimates of the target value.”

“To unleash the power of the foundational segmentation model in a target domain when there are limited labels, we propose ConformalSAM , an SSSS framework based on conformal prediction (CP) [1, 2, 4, 26].”

“D cal ∪ ( X n +1 , Y n +1 ) , ˆ C ( X n +1 ) attains the following marginal probabilistic guarantee, for some tolerated miscoverage rate α [41].”

“Conformal prediction, advanced by Vovk and co-authors [17], and later expanded on by various researchers in the Statistics community [18–26], is an increasingly popular paradigm for constructing provably valid prediction sets from a ‘black-box’ algorithm with minimal assumptions and at low cost.”

“Conformal prediction is one such method [23, 24].”

“…we guarantee coverage for the final policy π K assuming π k converges, we perform a batch calibration on π K (i.e., use traditional conformal prediction (Shafer and Vovk 2008) on a held-out calibration set to select a final value κ K +1 ) to ensure coverage for π K even without this assumption.”

“The conformal prediction introduced by Shafer and Vovk (2008) is a popular methodology in machine learning and is used to construct probabilistic forecasts calibrated on out-of-sample errors.”

“It does so by replacing the prediction of the most likely class with an uncertainty set –a set of classes accompanied by a probabilistic guarantee that the true class appears in the set [36].”

“In recent years, the CP method has attracted interest for quantifying the prediction uncertainty of ML models in the ML community [47,48].”

“Although we are the first to study calibration for algorithms with predictions, Sun et al. (2024) proposed using conformal prediction in this setting—a common tool in uncertainty quantification (Vovk et al., 2005; Shafer & Vovk, 2008).”

“In Table IX, we compare uncertainty estimation using cross-entropy (CE) and conformal prediction (CP) [79].”

“Conformal prediction (Vovk et al., 2005; Shafer and Vovk, 2008) provides a black-box, distribution-free method for generating prediction regions from online observations, making it ideal for wide-ranging applications, e.g., in automated decision support systems (Straitouri et al., 2023; Babbar et…”

“To mitigate the issues of miscalibrated posterior distributions, recent work draws on the success of Conformal prediction (CP) for prediction methods [Shafer and Vovk, 2008, Angelopoulos et al., 2023] to construct credible regions with finite-sample coverage guarantees in an SBI setting.”

“Key contributions include laying the foundational principles in CP (Vovk et al., 2005; 1999; Shafer & Vovk, 2008).”

“Framing long-form text generation as a conformal prediction (Shafer and Vovk, 2008) problem, Mohri and Hashimoto (2024) provide high probability guarantees for factuality.”

“We focus on conformal prediction (Shafer & Vovk, 2008; Vovk et al., 2005) as uncertainty measure since it provides strong theoretical guarantees for the coverage.”

“We begin by defining score functions; for more details on conformal prediction, see Shafer & Vovk (2008).”

“Conformal prediction, originally introduced by Vovk et al. (2005) and Shafer and Vovk (2008), is an increasingly popular method for obtaining data-driven uncertainty bounds.”

“This approach, initially introduced by Shafer [22], has been applied in diverse fields, including pedestrian localisation scenarios [23], and, more recently, time-series prediction interval detection [24,25].”

“This section is based on Cella and Martin (2022, Section 4.1), who summarize work by Shafer and Vovk (2008) and Vovk et al. (2005) on transductive conformal prediction.”

“Conformal prediction (CP) has gained popularity as a prominent technique for providing valid predictive inference in such models [25, 30].”

“In this paper, we pursue a method based on conformal prediction [Vovk et al., 2005, Shafer and Vovk, 2008], which is less committal than purely probabilistic approaches, but can yield such guarantees.”

“As a post-processing method, CP can be coupled with any learning model and can provide valid coverage set as long as data are (pseudo)exchangeable [2], [3], [6], [7].”

“Overviews of the field are provided by Vovk et al. [2005], Shafer and Vovk [2008], and Angelopoulos et al. [2023].”

“This variability hampers the ability to make deterministic predictions, and directly applying traditional uncertainty quantification methods such as conformal prediction [ Shafer and Vovk, 2008 ] can result in inequities across sensitive features.”

“To address these challenges, we introduce conformal prediction (CP) [17, 18], as a post-hoc module to our framework.”

“Conformal prediction is a framework for constructing distribution-free predictive confi-dence regions as long as the training and test data are exchangeable Vovk et al. [2005] (also see Shafer and Vovk [2008], Vovk et al. [2009], Vovk [2012]).”

“Conformal prediction (Vovk et al., 2005; Angelopoulos & Bates, 2023) is a model-agnostic framework that constructs prediction sets with finite-sample coverage guarantees from exchangeable data, free of any distributional assumption.”

“As such, this methodology is applied only to illustrate the value of using con fi dence intervals from probabilistic forecasts to determine operating reserves.”

“) is a set of techniques introduced in [20, 21], commonly applied in regression and classification tasks to derive prediction regions with certified guarantees.”

“All other models uncertainty quantification is based on a 50/50 split conformal prediction to estimate ˆ σ 2 i [ 91].”

“One promising framework for UQ is conformal prediction (CP), which provides finite-sample and distribution-free coverage guarantees without requiring strong modeling assumptions (Vovk et al., 2005; Shafer & Vovk, 2008; An-gelopoulos & Bates, 2023).”

“CP 18 offers a non-Bayesian alternative that provides valid confidence intervals under fewer assumptions.”

“Uncertainty modeling has long relied on Bayesian inference [13], ensembles [14], and conformal prediction [15] to produce confidence estimates.”

“Conformal Prediction (CP) [ 6 ] is a powerful framework for quantifying uncertainty, offering key advantages over traditional probabilistic methods such as Bayesian infer - ence and deep ensembles.”

“Conformal Prediction (CP) provides a principled framework for uncertainty quantification by constructing prediction sets with guaranteed marginal coverage [1–3].”

“Background Conformal factuality (Mohri & Hashimoto, 2024) is a method for providing statistical guarantees on the correctness of LLM-generated answers to open-ended questions based on conformal prediction (CP) (Vovk et al., 2005; Shafer & Vovk, 2008).”

“This formulation ensures valid finite-sample coverage guarantees [7], [33], [34].”

“For a detailed discussion on conformal prediction, we refer interested readers to the canonical tutorial paper of Shafer and Volk [67].”

“Conformal prediction is a rigorous statistical calibration technique for uncertainty quantification of predictive models [21].”

“For more information on these techniques, refer to Shafer & Vovk (2008) or An-gelopoulos & Bates (2023).”

“An overview of this area is provided in Vovk et al. [2005], Shafer and Vovk [2008], Angelopoulos and Bates [2021], and Angelopoulos et al. [2024].”

“Conformal prediction, a statistic framework for uncertainty estimation, converts an algorithm’s predictions into prediction sets containing the true class with a user-specified coverage rate (Balasubramanian et al., 2014; Shafer & Vovk, 2008).”

“Conformal prediction (CP), described in detail in Shafer and Vovk (2008) and Vovk et al. (2022), is a general method for distribution-free uncertainty quantification, using past experience to determine precise levels of confidence in a new prediction.”

“In this section, we recall some basics from (Vovk et al., 2005; Shafer and Vovk, 2008).”

“To address these challenges, this paper proposes combining conformal inference [3, 16, 24] and evidential learning [2, 8, 15, 23], two promising, complementary methodologies for UQ in edge-deployed AI models, which is illustrated in Fig.”

“Conformal prediction provides distribution-free uncertainty quantification with finite-sample guarantees (Vovk et al., 2005; Shafer & Vovk, 2008; Angelopoulos & Bates, 2021).”

“See the monograph of [VGS05] and the surveys [SV08, AB21] for further background.”

“Conformal Prediction (CP) (Shafer and Vovk, 2008) provides a non-parametric, distribution-free approach for constructing confidence sets with formal guarantees.”

“Conformal prediction (Gammerman et al., 1998; Vovk et al., 2005; Shafer and Vovk, 2008) is a powerful framework for quantifying uncertainty that is particularly useful in machine-learning applications (Papadopoulos et al., 2002; Balasubramanian et al., 2014; Laxham-mar and Falkman, 2015; Lei et…”

“We first formalize the nestedness condition, which is a fundamental requirement for the validity of any conformal procedure [Vovk et al., 2005, Shafer and Vovk, 2008].”

“Conformal prediction [62, 59] has recently risen in popularity as it provides distribution-free, finite-sample coverage guarantees [2], and has shown promise in classification [56, 3, 33], regression [10, 43, 55], and language tasks such as factual question answering [37, 52, 45, 14, 23].”

“· 100 (1)”

“Existing UQ approaches rely predominantly on heuristic consistency checks.”

“Conformal Prediction (CP) (Gammerman et al., 1998; Shafer and Vovk, 2008) offers a model-agnostic framework for uncertainty quantification with finite-sample coverage guarantees.”

“To establish the foundation, we introduce the basic principles of CP [4, 38, 40, 45].”

“…us to invert this relationship and define the precise set of outcomes for which a given decision is optimal; and ( ii ) using conformal prediction [39] with generative models, we estimate the probability mass of this region, yielding valid, data-driven upper bounds on decision risk with…”

“To achieve this, we introduce a multiple choice question-answering (MCQA) procedure that leverages a second LLM and conformal prediction (CP) (Shafer and Vovk, 2008), to estimate uncertainty and guarantee that the refined set of paths includes, with a user-defined probability, at least one path…”

“Conformal prediction [21] provides guarantees of finite-sample coverage under exchangeability.”

“In particular, conformal prediction (CP) has emerged as a powerful UQ framework, providing finite-sample validity guarantees under mild assumptions [35].”

“Conformal prediction has emerged as a general and versatile framework for constructing prediction sets in regression and classification tasks (Shafer and Vovk, 2008).”

“O-CRC builds upon the conformal prediction (CP) framework [11], [14], and it is applicable to scenarios where the AI decisions take the form of a prediction set .”

“Conformal prediction is a statistical framework for constructing prediction intervals or sets with guaranteed coverage probabilities under minimal distributional assumptions (Shafer and Vovk 2008).”

“Unlike these approaches, Conformal Prediction (CP) (Angelopoulos and Bates, 2022; Vovk et al., 2005; Shafer and Vovk, 2007) offers a distribution-free frequentist methodology for uncertainty quantification with finite-sample coverage guarantees.”