A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“In addition to calculating the average zn−1, calculate s2n−1 := 1 n − 2 n−1 ∑ i=1 (zi − zn−1) 2, which is sometimes called the sample variance.”

“But the efficiency of conformal prediction will depend on Q and the nonconformity measure.”

“This definition makes it easy to see that independent and identically distributed random variables are exchangeable.”

“The probability is valid if the offer does not put the person making it at a disadvantage, inasmuch as a long sequence of equally reasonable offers will not allow an opponent to multiply the capital he or she risks by a large factor (Shafer and Vovk, 2001).”

“But this turned out not to be the case (Robinson, 1975).”

“This allows us to express the empirical interpretation of given probabilities in terms of betting, using what we call Cournot’s principle: the odds determined by the probabilities will not allow a bettor to multiply the capital he or she risks by a large factor [23].”

“Exchangeability is closely related to the idea that examples are drawn independently from a probability distribution.”

“Conformal prediction (Vovk et al., 2003; Shafer and Vovk, 2008; Balasubramanian et al., 2014) is a framework for reliable prediction that is rooted in classical frequentist statistics, more specifically in hypothesis testing.”

“Training-based calibration methods comprise supervised techniques that modify the network’s weights and various types of self-verbalization, where the model qualifies and refines its outputs based on its own reasoning or feedback about uncertainty.”

“Several of these proposed methods, like Bayesian deep learning [68] or conformal inference methods [143, 1, 162] can be applied to spacecraft pose estimation improving their explainability, which are interesting directions for future research.”

“In the second stage , we use conformal prediction (CP) [52, 47] to construct a time-varying threshold to sequentially determine when a score indicates failure with statistical guarantees on false positive rates.”

“When data are generated via a hierarchical sampling procedure, it creates a simple form of heterogeneity within data.”

“This necessitates the development of specialized techniques tailored to the unique structure and requirements of RAG models.”

“Future work will consider how to better choose a threshold for a given environment, e.g., using conformal prediction [47, 48] to bound the false positive rate of detecting sensitive regions.”

“Detailed tutorials on the principles and implementation of conformal prediction can be found in Shafer and Vovk (2008) and Cordier et al. (2023).”

“Most related, conformal prediction (Shafer & Vovk, 2008) has been popu-larized as an effective method for uncertainty quantification with safety assurances on the miscoverage risk (Fontana et al., 2023; Angelopoulos et al., 2023).”

“Survey articles on the topic of conformal prediction were presented in [19, 21, 22].”

“Under a slightly weaker exchangibility assumption (i.e. that the joint distribution remains the same under permutations of the training and test data), the conformal set of predictions is guaranteed to contain the true label with error probability less than some ϵ [35].”

“Recent approaches such as isotonic regression (Kuleshov et al., 2018) and conformal prediction (Shafer & Vovk, 2008) have made progress in providing calibrated uncertainty estimates from deep learning models, but such methods are typically applied post-hoc to trained models and are therefore…”

“Conformal prediction (Shafer and Vovk 2008; Zhao et al. 2024) is a statistical tool for uncertainty quantification that provides valid prediction regions with a user-specified risk tolerance, making it particularly useful for ensuring safety in dynamic environments.”

“Conformal prediction [59, 49, 2] provides a model-agnostic framework for constructing such sets with finite-sample validity, ensuring that the true response is captured at least 1 − 𝛼 fraction of the time without requiring strong distributional assumptions.”

“A special case of threshold-based classifier is conformal prediction [Shafer and Vovk, 2008], which is a general framework that provides provably valid confidence intervals for a black-box scoring function.”

“Prior work has considered the design of good non-conformity scoring functions, e.g., [2, 50, 47].”

“Under minimal assumptions, they offer finite-sample validity (Vovk et al., 2005; Shafer & Vovk, 2008).”

“Conformal prediction (CP) (Gammerman et al., 1998; Vovk et al., 2005; Shafer & Vovk, 2008) has emerged as a simple framework to quantify the prediction uncertainty of machine learning algorithms without relying on distributional assumptions on the data.”

“Initially introduced in (Vovk et al., 1999) and further elaborated in (Shafer & Vovk, 2008) and (Balasubrama-nian et al., 2014), conformal prediction offers a distribution-free uncertainty quantification technique that has been effectively applied in various applications (Angelopoulos & Bates,…”

“26–35) or oncogene amplification across various tumor types, such as amplification of CDK4 , MDM2 , CCNE1 and MYC in osteosarcomas and MYC and EGFR in glioblastomas 1 (Fig.”

“The idea of conformal prediction is to use past experience to determine precise levels of confidence in new predictions (Shafer and Vovk, 2008).”

“Conformal prediction quantifies the uncertainty in machine learning models by generating set-valued predictions (Shafer and Vovk, 2008; Angelopoulos and Bates, 2021).”

“Conformal prediction constructs prediction intervals that guarantee a specified level of confidence, allowing users to assess the uncertainty associated with individual predictions [183, 184].”

“CP (Vovk et al 2005, Shafer and Vovk 2008) addresses a fundamental question in machine learning: given a dataset ( X 1 , Y 1 ) , ( X 2 , Y 2 ) ,... , ( X n , Y n ) and a trained model ˆ f : X → Y , how accurate is ˆ f at predicting the true label Y n + 1 for a new query point X n + 1 ?”

“Conformal prediction [40,42] is a general uncertainty quantification framework that provides non-asymptotic marginal (global) guarantees, independent of the underlying regression function g .”

“Split Conformal Prediction Vovk et al. [66], Shafer and Vovk [60], Papadopoulos [46] provide standardized tools to construct prediction sets that provide coverage guarantees.”

“CP (Vovk et al., 2005; Shafer & Vovk, 2008) is one of the leading approaches to uncertainty quantification in machine learning (Soize, 2017; Abdar et al., 2021), and is aligned with efforts to make models more trustworthy.”

“Conformal prediction is a statistical uncertainty quantification technique which yields marginal coverage guarantees over a confidence set in a distribution-free manner, traditionally only assuming exchangeability of the data (Gammerman et al., 1998; Shafer & Vovk, 2008; Angelopoulos & Bates, 2022).”

“Conformal prediction (CP; Vovk et al., 2005; Shafer & Vovk, 2008) is a robust framework for constructing distribution-free predictions with finite-sample validity guarantees; see (An-* Furthermore, classical CP methods ensure marginal validity but do not guarantee conditional validity, a stricter…”

“Standard split conformal prediction typically assumes instead a calibration set of examples with “crisp” ground truth labels { ( X i , Y i ) } ni =1 [49].”

“Our approach relies on pre-trained LLMs as the back-bone translator, augmented by Conformal Prediction (CP) [26], [27], an uncertainty quantification method, which plays a critical role to achieve user-specified translation success rates.”

“We refer the audience to predictive inference, especially conformal inferences, to [35–37].”

“Conformal prediction (CP) addresses prediction uncertainty by generating a set of possible targets instead of a single prediction (Vovk et al., 2005; Shafer & Vovk, 2007; Angelopoulos & Bates, 2021).”

“Conformal prediction (Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008) is a framework for constructing prediction intervals with guaranteed marginal coverage in regression problems.”

“Several textbooks and surveys provide in-depth discussions [Vovk et al., 2005, Shafer and Vovk, 2008, Balasubramanian et al., 2014, Angelopoulos et al., 2024].”

“Unlike the aforementioned interval construction methods, CP relaxes the assumptions on the underlying model and can still ensure the coverage rate of the prediction intervals under limited samples [29], which has attracted widespread attention from scholars in recent years.”

“Another line of work tries to add formal statistical controls to LLMs. Conformal prediction methods [48] quantify uncertainty in individual predictions while providing finite-sample and black-box guarantees in those uncertainties.”

“A very common method in the statistics and machine learning literature these days is conformal prediction (e.g., Vovk, Gammerman, and Shafer 2005; Shafer and Vovk 2008).”

“CPcomputesthepredictionintervalsina few steps 110,111 asindicated in the example in Fig.”

“This letter focuses on conformal prediction (CP), a statistical tool based on rank statistics for quantifying uncertainties in data-driven prediction models [1], [2], [3].”

“30) to evaluate the reliability of individual predictions by generating nonconformity scores and subsequently removed single-drug response predictions with a nonconformity score lower than 0.8.”

“One technique that can provide such a statistical guarantee to assess AI code assistants’ trustworthiness is conformal prediction [64].”

“Unlike traditional methods that rely on specific distributional assumptions, CP provides valid prediction sets under the minimal assumption of exchangeability, making it widely applicable across various domains [36].”

“Conformal prediction (Shafer & Vovk, 2008; Bal-asubramanian et al., 2014; Romano et al., 2020) demonstrates the capacity to provide such statistical guarantees for any black-box DNN with exchangeable data.”

“To achieve this, we introduced conformal prediction, a widely used uncertainty quantification framework in combination with AI methods (Shafer and Vovk 2008).”

“Specifically, we employ Conformal Prediction (CP), a distribution-free statistical method [10], [11].”

“Finite-sample statistical guarantees on inferential outputs can be obtained via conformal prediction methods (Shafer & Vovk, 2008; Angelopoulos et al., 2023) and, more generally, via conformal risk control and risk-controlling set-valued predictions (Angelopoulos et al., 2022; Bates et al., 2021).”

“We approach problem (1) via calibration , in which one wishes to output a calibrated score for each claim, while for problem (2) , we apply conformal prediction [Shafer and Vovk, 2008], selecting a subset of claims that—with high probability—are all correct.”

“36,38 In regression tasks, the PIs constructed by CP achieve valid marginal coverage with the general i.i.d. (independent and identically distributed) data assumption.”

“Finally, exploring uncertainty quantification techniques [1], such as conformal prediction [113], can provide valuable information on the reliability and robustness of CI under latent variables, facilitate more informed decision-making, and provide pessimistic/optimistic bounds when exact causal…”

“Specifically, methods such as split conformal prediction (CP) [ Shafer and Vovk, 2008; Lindemann et al. , 2023; Lekeufack et al. , 2023 ] provide confidence intervals that guarantee to contain the ground truth target data with a predefined probability.”

“Conformal Prediction (CP) is a frequentist approach for producing valid PIs without making assumptions on the data-generating distribution, P XY , or the prediction model, f [Vovk et al., 2005, Shafer and Vovk, 2008].”

“In contrast, CP has recently emerged as a promising alternative, which provides marginal coverage guarantees over unseen test samples (Vovk et al., 2005; Shafer & Vovk, 2008).”

“…X | − | ∈ Given a miscoverage rate α ∈ [0 , 1] and a test input X ∗ ∈ X , the conformal set gathers outcomes yielding the smallest scores: where Q 1 − α is the ⌈ (1 − α )( n + 1) ⌉ -th smallest of the scores evaluated on the calibration set, where − − We refer readers to [38, 39] for more details.”

“Several excellent shorter introductions to CP exist, including [6] and [7].”

“Therefore, we incorporate uncertainty quantification into our model’s predictions using Conformal Prediction (Shafer & Vovk, 2008).”

“1 estimation based on conformal prediction (CP) [26], [27].”

“Comprehensive overviews of this field can be found in [2, 25], which review both theoretical foundations and practical applications across regression, classification, and structured prediction tasks.”

“…1], here again we obtain the familiar guarantee To compare with existing results, we begin by noting that standard results for conformal prediction (Shafer and Vovk, 2008; Lei et al., 2018; Angelopoulos et al., 2024) do not allow for memory-based predictors even when the process Z is exchangeable,…”

“CP has been extensively studied for classical tasks such as classification and regression [9–12].”

“The volume of the prediction set in conformal prediction is also sometimes referred to as ‘ef-ficiency’ has been stated as an important consideration in many prior works (see e.g., Shafer and Vovk, 2008; Angelopoulos and Bates, 2023).”

“Originally introduced by Vovk and Shafer [12], [13], CP uses a calibration dataset to infer prediction regions for a test dataset with a specified probability, while remaining distribution-agnostic.”

“More recently, conformal prediction [37], [38] has gained popularity due to its ability to provide probabilistic guarantees on coverage.”

“A prominent line of work within this field is called conformal prediction (Vovk et al., 2005; Shafer & Vovk, 2008).”

“Within the batch regime, these are often termed ‘distribution free,’ as in the case of conformal prediction and related techniques [4, 66, 76].”

“The validity guarantees of conformal prediction hold under the even weaker assumption of exchangeability (Shafer and Vovk, 2008; Vovk et al., 2005).”