A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“Distribution-free alternatives such as conformal prediction offer calibrated coverage guar - antees without a fully specified generative model (Shafer and Vovk 2008).”

“Conformal prediction converts nonconformity into finite-sample predictive sets or values under exchange-ability (Vovk et al. 2005; Shafer and Vovk 2008).”

“Conformal prediction provides a related framework for distribution-free uncertainty quantification under appropriate assumptions [36,37], and recent sensor studies have used conformal calibration to construct reliable prediction intervals [38].”

“Conformal prediction (CP) provides a rigorous statistical framework for constructing prediction sets with distribution-free marginal coverage guarantees under appropriate exchangeability assumptions [11–15].”

“When the calibration set is obtained by splitting the available labeled data, this procedure is commonly referred to as Split Conformal Prediction (SCP) [37, 54, 44, 53].”

“Conformal prediction (CP) is attractive because it is distribution free, model agnostic, and gives finite-sample guarantees that a prediction set contains the future outcome at a user-specified rate Vovk et al. (2005); Shafer and Vovk (2008).”

“If the calibration targets and test target are exchangeable conditional on the fitting set, the split-conformal argument gives P ( A ℓ test ≤ (cid:98) k α ) ≥ 1 − α (Vovk et al., 2005; Shafer and Vovk, 2008; An-gelopoulos and Bates, 2023).”

“Conformal methods can use surrogate outputs to construct prediction sets with finite sample calibration guarantees [10, 11, 12].”

“Classical methods provide distribution-free predictive coverage under exchangeability [20]–[23], with later extensions to counterfactual outcomes [24], [25], continuous-treatment causal and dose–response inference [26], [27], nonexchangeable data [28], [29], localized calibration [30], [31], and…”

“Conformal prediction transformed nonconformity scores into p-values under exchangeability assumptions [1, 28, 30].”

“Unlike this line of work, however, OCSP does not aim to calibrate a predictor [10] or to defer from cloud to edge [5], but to certify and control the reliability of a closed-loop decision-making mechanism that jointly governs which actions to take and when to probe.”

“Conformal prediction [64, 55] provides a generic and flexible framework for converting the outputs of a black-box model into prediction sets, in such a way as to guarantee (or closely approximate) the marginal coverage property where C n ( X n + 1 ) is a prediction interval or prediction set for…”

“To address this challenge and provide robust uncertainty quantification, certain works have leveraged the conformal prediction (CP) framework [Shafer and Vovk, 2008, Vovk et al., 2022] to the SBI context.”

“The full conformal reference is classical (Shafer and Vovk, 2008); label conditional ranks (Vovk, 2012; Ding et al., 2023) and stability-based outer sets (Ndiaye, 2022) are established alternatives.”

“(6) is related to reject-option and selective classification [22, 23, 24], as well as set-valued and conformal prediction [27, 28].”

“It ensures that the assigned p -values reflect the plausibility of each forecast, enhancing the reliability of predictions and providing a quantifiable measure of uncertainty crucial for decision-making processes [25,20].”

“…a general framework for uncertainty quantification that transforms the output of an arbitrary predictive model into prediction sets or intervals with finite-sample coverage guarantees under the assumption of exchangeable data [Vovk et al., 2005, Shafer and Vovk, 2008, Angelopoulos and Bates, 2023].”

“Conformal prediction has emerged as a powerful framework for this purpose, providing model-agnostic prediction sets with finite-sample marginal coverage guarantees for missing potential outcomes and individual treatment effects (Shafer and Vovk, 2008).”

“Thelatter transformsstandardpointpredictionsintointervalpredictionswith rigorousfinite-samplecoverageguarantees,withtheonlyassump-tionbeingdataexchangeability(Shafer&Vovk2008;Angelopoulos &Bates2021).”

“To provide astronomers with a single interpretable significance measure, we use the conditional confidence defined in Ashton et al. (2024), an extension of the CP confidence (Shafer and Vovk, 2008) that can always be efficiently computed in practice.”

“Focusing specifically on the distribution-free objective of our work, conformal prediction, originally introduced by Vovk et al. (1999); Shafer and Vovk (2008), provides a general framework for distribution-free predictive inference.”

“Given inputs x ∈ X and ground-truth values y ∗ ∈ Y drawn jointly from a distribution ( x, y ∗ ) ∼ P , split conformal prediction (Vovk et al., 2005; Shafer and Vovk, 2008) constructs a prediction set C ˆ q ( x test ) ⊆ Y for a new datapoint x test that come with a finite-sample coverage guarantee…”

“This line of research originates from conformal prediction [29, 30, 31], which generates prediction sets that contain ground-truth labels with a desired coverage guarantee.”

“…which help a user trace back a judgment to specific questions, and (2) a representation of confidence, either in the form of a calibrated probability [Platt, 1999, Guo et al., 2017, Niculescu-Mizil and Caruana, 2005] or a high-confidence interval via conformal prediction [Shafer and Vovk, 2008].”

“…is the setting adopted here. b) Conformal prediction: Conformal prediction gives finite-sample coverage under exchangeability (Vovk et al., 2005; Shafer & Vovk, 2008); adaptive-width extensions include conformalized quantile regression and conditional calibration (Romano et al., 2019;…”

“Conformal prediction [15], [31] gives distribution-free coverage; adaptive conformal inference [32] and online risk control [17], [19], [33], [34], [35] retain validity under shift.”

“Under exchangeability, there is a principled approach to create exchangeable conformal prediction sets with coverage guarantees that are distribution-free [3] and [4] and [11].”

“While the concept of prediction regions is initially rooted in theoretical and heuristic frameworks suggesting that the world cannot always be represented by single-point predictions (Shafer & Vovk, 2008), recent work (Jesse C. Cresswell & Vouitsis, 2024) have demonstrated its practical value.”

“In the standard split conformal prediction (SCP) framework [81, 2], this threshold is the empirical quantile which calibrates the model’s confidence scores to achieve the desired marginal coverage (6).”

“Unlike Bayesian or bootstrap-based approaches that require distributional assumptions, conformal prediction is distribution-free and model-agnostic and provides finite-sample validity guarantees that hold for any sample size[21].”

“Conformal prediction has emerged as a flexible framework for constructing prediction intervals with finite-sample coverage guarantees under relatively weak assumptions [48, 49].”

“Multi-output conformal reconstruction from finitely many search directions also precedes it; in that directional framework, star-shapedness is imposed as a structural condition and sampled boundary points may be fit by convex objects such as ellipses or convex hulls [Johnstone and Ndiaye, 2025].”

“Our audit is orthogonal, and adds a conformal-calibrated [9] meta-metric tested as a would-be combiner.”

“First, telemetry is serially dependent and nonstationary, whereas simple rank arguments are sharpest under ex-changeability [19].”

“Finally, by incorporating a conformal prediction strategy [21], SPADE produces statistically calibrated prediction intervals for each spot’s location, allowing robust and interpretable identification of spatially aberrant cells with controlled false discovery rates.”

“To handle the resulting estimation error, we use conformal prediction (CP) to calibrate an error bound from data at a user-chosen miscoverage level [7,8].”

“(24) satisfies, for every fixed j ∈ I , This is the standard split-conformal rank argument [52, 65]: exchangeability of the trajectories implies exchangeability of the N cal calibration scores and the corresponding test score at the same index j .”

“To quantify predictive uncertainty, we use conformal prediction, which constructs distribution-free prediction intervals from calibration residuals [21, 22].”

“Among the post-processing methods considered in this study, CP (Shafer and Vovk, 2008; Kath and Ziel, 2021; Za ff ran et al., 2022) and QR (Nowotarski and Weron, 2015; Uniejewski, 2022; Lipiecki et al., 2024) are typically, although not exclusively (Taylor and Bunn, 1999), applied to out-of-sample…”

“…Carlo dropout provides an approximate Bayesian mechanism through repeated stochastic predictions [60]; deep ensembles estimate predictive dispersion across independently trained models [61]; and conformal prediction constructs finite-sample prediction sets under exchangeability assumptions [62].”

“To overcome these limitations, non-parametric and deep learning-based alternatives have emerged, including error prediction methods, 25 end-to-end QR, 26,27 and CP. 28 Furthermore, state-of-the-art (SOTA) temporal architectures, such as Temporal Convolutional Networks (TCNs), 29 have been increasingly deployed in time…”

“…the mean pinball (quantile) loss averaged over the predicted quantile levels, so minimizing it during training is a distribution-free way to fit calibrated bands – connecting our objective to quantile regression (Koenker and Hallock 2001) and conformal-style calibration (Shafer and Vovk 2008).”

“Split conformal prediction [7,60,61] fixes a nonconformity score s ( x , y ) (for example, the absolute residual of any fitted model), evaluates it on a held-out calibration set to obtain s 1 , . . . , s n , and forms the prediction set from the empirical quantile of those scores, Exchangeability…”

“For more comprehensive discussions, we refer to Shafer and Vovk (2008) and Zhou et al. (2025).”

“The NCM is calculated for each individual in both the calibration and test sets and serves as the basis for determining which labels should be included in the prediction region [18].”

“Split (inductive) conformal prediction [21,22] converts any trained classifier into a set-valued predictor with a finite-sample coverage guarantee.”

“Among the numerous approaches to uncertainty quantification, Conformal Prediction (CP) [9] receives significant attention due to its appealing properties, such as distribution-free, model-agnostic, and guaranteed coverage under finite samples [10].”

“Conformal prediction provides finite-sample, distribution-free predictive guarantees under exchangeability [10, 8, 5] and is summarized in the overview of Angelopoulos and Bates [1].”

“Conformal methods (Vovk et al., 2005; Shafer and Vovk, 2008; Romano et al., 2019; Angelopoulos and Bates, 2023) supply finite-sample marginal guarantees, and Mondrian variants restore per-stratum validity (Vovk, 2013).”

“Conformal prediction (CP) (Vovk et al 2005, Shafer and Vovk 2008, Angelopoulos and Bates 2023) provides a distribution-free framework that converts any model’s output into prediction sets with finite-sample marginal coverage guarantees.”

“CP provides distribution-free, finite-sample valid uncertainty quantification for the predictions of any fixed model [34], [35].”

“Conformal prediction provides a distribution-free calibration layer that can wrap almost any regression model and deliver finite-sample marginal coverage under exchangeability [12]- [15].”

“The foundations were developed by Vovk, Gammerman, and Shafer [14], and later tutorials and split-conformal formulations made the approach accessible for practical regression tasks [15], [16].”

“Conformal prediction provides distribution-free predictive inference with finite-sample marginal validity under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008; Lei et al., 2018).”

“Conformal prediction allows for a model-agnostic framework for producing prediction sets or intervals with user-specified coverage guarantees under exchangeability assumptions [8], [9].”