A tutorial on conformal prediction

“408–410 of his Statistical Learning Theory [27], support-vector machines grew out of the idea of separating two groups of examples with a hyperplane in a way that makes as few mistakes as possible—i.”

“1 Example: Classifying iris flowers In 1936 [11], R.”

“, zn−1) can be framed as an application of the widely accepted Neyman-Pearson theory for hypothesis testing and confidence intervals [17].”

“Readers familiar with Bayes nets [4] will recognize this diagram as an example; conditional on each variable, a joint probability distribution is given for its children (the variables to which arrows from it point), and given the variable, its descendants are independent of its ancestors.”

“This gives the following protocol: The Backward-Looking Betting Protocol Players: Joe, Bill KN := 1.”

“82; [21], pp.”

“Our concept of validity is consistent with a tradition that can be traced back to Jerzy Neyman’s introduction of confidence intervals for parameters in 1937 [19] and even to work by Laplace and others in the late 18th century.”

“On the other hand, as we have already said, it will make no difference if we replace the absolute difference d(z, z′) = |z − z′| with the squared difference d(z, z′) = (z − z′)2, thus squaring A.”

“• Uncertainty Quantification and Validation of Theoretical results via Simulation: We complement point forecasting with uncertainty quantification (Shafer & Vovk, 2008), robustness analysis, and simulation diagnostics.”

“Conformal prediction (Vovk et al., 2005) provides exactly this guarantee under exchangeability (Shafer & Vovk, 2008).”

“Conformal prediction (CP) provides a statistically principled UQ framework with finite-sample guarantees under exchangeability [19, 1, 14, 7].”

“Specifically, LogiCP integrates Conformal Prediction (CP) (Shafer and Vovk 2008) with Signal Temporal Logic (STL)-based inference (An et al. 2024; Ji et al. 2023; Ma, Gao, et al. 2020) to address data heterogeneity and model uncertainty.”

“Risk control can be viewed as a natural extension of conformal prediction (Shafer & Vovk, 2008), which is primarily designed to ensure coverage guarantees, whereas risk control applies to a broader class of performance metrics beyond coverage.”

“An additional dimension to calibration is conformal prediction , which provides prediction intervals under mild distributional assumptions [86].”

“Conformal prediction provides distribution-free prediction sets under exchangeability and has been used to quantify uncertainty in load and building-performance settings [13–15].”

“CP was selected for its ability to generate prediction intervals with finite-sample coverage guarantees without requiring specific distributional assumptions, rendering it suitable for complex energy forecasting tasks where error distributions are often unknown [12], [13].”

“See [54, 46, 5, 4] for further background on full conformal prediction and related methods.”

“Their approach and, more broadly, distribution-free calibration via Conformal Prediction (CP) [6] and risk control [7, 8] certifies a generic loss against an abstract, user-chosen level rather than a normative safety target.”

“For production deployment, conformal prediction [26] provides coverage-guaranteed interval estimates without distributional assumptions and represents a natural extension of the present framework.”

“Conformal prediction [17], [18] is a framework for constructing prediction intervals with guaranteed marginal coverage.”

“In response to these limitations, conformal prediction (CP) [12] has gained increasing attention [13] as a model-agnostic framework that provides statistically valid PIs without requiring prior assumptions.”

“SHAP [15] is widely used for post hoc attribution, but is rarely paired with uncertainty frameworks such as Conformal Prediction [16], which can provide distribution-free error control.”

“Split conformal prediction provides a general mechanism for constructing distribution-free prediction intervals with finite-sample marginal coverage [1], [2], [4].”

“The classical conformal prediction framework was first introduced in (Shafer and Vovk 2008); later, split conformal prediction improved its computational practicality by separating model fitting from calibration (Vovk 2012).”

“…control, and learning-to-defer formalize when a system should abstain or hand off to a human expert [Chow, 1970, El-Yaniv and Wiener, 2010, Geifman and El-Yaniv, 2017, Shafer and Vovk, 2008, Bates et al., 2021, Madras et al., 2018, Mozannar and Sontag, 2020, Raghu et al., 2019, Kompa et al., 2021].”

“Conformal prediction (CP) turns any fixed nonconformity score into prediction sets with distribution-free finite-sample marginal coverage under exchangeability [36, 33, 1].”

“Conformal prediction [24], [53], [54] has emerged as a tool for providing finite-sample probabilistic safety coverage.”

“To calibrate an acceptance threshold at level (1 − α ), FACTER relies on split conformal prediction (Shafer & Vovk, 2008).”

“Split conformal prediction (Vovk et al., 2005; Shafer & Vovk, 2008) provides finite-sample, distribution-free marginal coverage under exchangeability.”

“Starting with the work of [8], there has been increased interest in PIs obtained via conformal prediction ; see also [9].”

“The conformal screen is connected to distribution-free prediction and risk control [Shafer and Vovk, 2008, Angelopoulos and Bates, 2021, Bates et al., 2021].”

“Split conformal prediction (SCP) constructs prediction intervals from calibration residuals and achieves marginal coverage under exchangeability (Lei et al., 2018; Shafer & Vovk, 2008).”

“The Adaptive Routing employs a lightweight decision model to evaluate the first-token statistics, with its decision threshold calibrated via conformal prediction (Shafer and Vovk 2008) to provide a controllable lower bound on the recall of difficult samples.”

“…we employ monotone submodular maximization with cardinality constraints [Nemhauser et al., 1978, Krause and Golovin, 2014] and integrate a calibration layer inspired by conformal prediction coverage guarantees [Vovk et al., 2005, Shafer and Vovk, 2008, Lei et al., 2018, Romano et al., 2019].”

“Conformal prediction yields distribution-free sets under exchangeability (Lei et al., 2018; Shafer and Vovk, 2008; Vovk et al., 2005), with split (inductive) conformal prediction the practical workhorse (Papadopoulos et al., 2002).”

“Conformal prediction [Vovk et al., 2005, Shafer and Vovk, 2008, Angelopoulos and Bates, 2022] offers an attractive alternative: it provides distribution-free, finite-sample coverage guarantees without requiring any assumptions about the data-generating process beyond exchangeability.”

“However, more robust methods like conformal prediction have gained prominence, primarily because they offer rigorous statistical guarantees [Shafer and Vovk, 2008].”

“Conformal prediction provides finite-sample, distribution-free coverage guarantees under exchangeability [Vovk et al., 2005, Shafer and Vovk, 2008, Angelopoulos and Bates, 2023].”

“We thus reduce the certification problem to lower-bounding p A ( x ) from q A ( x ) , and address it with a one-time conformal calibration [Shafer and Vovk, 2008, Lei et al., 2018] that determines a single parameter δ ≥ 0 on a held-out set such that q A ( x ) − δ ≤ p A ( x ) with high probability.”

“Conformal prediction provides statistically calibrated confidence estimation by comparing nonconformity scores with a calibration distribution [21].”

“In this study, we aim to address these limitations by developing a multilabel classification model capable of predicting multiple infection foci and combining it with the conformal prediction framework 3 to provide well-calibrated uncertainty estimates.”

“This guarantee holds for any data distribution, any base model, and any finite sample size [10].”

“Conformal prediction is an alternate UQ paradigm that makes no distributional assumptions, and instead, constructs prediction intervals with finite sample coverage guarantees [10].”

“Our method produces prediction regions containing the true registration result with user-specified probability [13], [14].”

“Conformal prediction [38, 43] offers an elegant resolution: given any base predictor and an exchangeable calibration sample, conformal prediction constructs prediction sets that contain the true outcome with a user-specified probability, without distributional assumptions on the data-generating…”

“Conformal Prediction (CP) provides valid uncertainty quantification for arbitrary predictive models under the assumption of exchangeability (Vovk et al., 2005; Shafer & Vovk, 2008).”

“Accuracy, calibration behavior [20], and confidence intervals for key paired effects and pass-rate estimates [21, 22] are interpreted alongside conformal and predictive-uncertainty literature [23–26] so that local gains are assessed together with deployment reliability.”

“Finally, distribution-free uncertainty quantification methods, such as conformal prediction, provide finite-sample coverage guarantees with minimal assumptions, thereby aligning predictive performance with reliability in deployment [37].”

“Split conformal prediction (also known as inductive conformal prediction) (Vovk et al., 2005; Shafer & Vovk, 2008; Lei et al., 2018; Angelopoulos & Bates, 2023) is a standard approach for this purpose.”

“Lemma 1 ( Marginal coverage [17, 13]).”

“Conformal prediction provides distribution-free uncertainty quanti fi cation with fi nite-sample statistical guarantees, offering an alternative paradigm that constructs prediction sets rather than point estimates with con fi dence scores (Shafer and Vovk, 2008).”

“[3] and in [5] it is used in classification.”

“A comprehensive review of these developments is given by Shafer and Vovk [2008] and, more recently, by Angelopoulos and Bates [2023].”

“Conformal prediction (CP) wraps a fixed predictive model with set-valued outputs whose marginal miscoverage is controlled in finite samples under exchangeability (Vovk et al., 2005; Shafer and Vovk, 2008; Angelopoulos and Bates, 2023).”

“Conformal prediction relates to the capacity of a model to establish levels of confidence for its estimations, which can in turn be used to characterize said answers with respect to their certainty or associated risk [22].”

“While conformal prediction (CP) provides distribution-free coverage guarantees for such models (Vovk et al., 2005; Shafer & Vovk, 2008), it typically treats the regression stage in isolation, regardless of first-stage uncertainty.”

“Conformal prediction: Conformal prediction (CP) [37, 40] is a model-agnostic and distribution-free framework for uncertainty quantification, providing prediction sets with guaranteed marginal coverage properties.”

“CP produces prediction sets containing the ground-truth value with a pre-specified coverage [12].”

“For example, Shafer and Vovk (2008) introduced the CP classifiers for vector-valued covariates.”

“Split conformal prediction supplies distribution-free intervals and prediction sets (Vovk et al. 2005, Shafer & Vovk 2008, Lei et al. 2018), with extensions to heteroscedastic regression (Romano et al. 2019) and localized calibration (Guan 2023).”

“This is consistent with prior work showing that post-hoc calibration can meaningfully improve regression uncertainty estimates, especially when the underlying predictive uncertainty is initially poorly calibrated [46, 47, 48].”

“Thus, (3) gives a high-probability risk-control statement over the calibration sample, rather than the standard marginal coverage guarantee of split CP (Vovk et al., 2005; Shafer and Vovk, 2008).”

“Such a confidence set may be formed by residual calibration or conservative uncertainty bounds [27], [28].”

“Conformal prediction (CP): Originally introduced in [1], [2], [3], CP has recently attracted significant interest in the machine learning community as a robust framework for uncertainty quantification.”

“4) Conformal prediction and distribution-free intervals: CP provides finite-sample, distribution-free guarantees for prediction sets under exchangeability [8], [10], [11], [29].”

“CP is a model-agnostic statistical framework providing distribution-free theoretical guarantees—reliable predictions regardless of underlying data distribution, assuming exchangeability [32], [33].”

“Conformal prediction [20] provides distribution-free guarantees, while recent meth-ods explore streaming adaptation [21].”

“To improve prediction reliability, we incorporate conformal prediction [38] into our framework to provide principled uncertainty quantification and set-valued predictions.”

“Conformal Prediction (CP) is an effective distribution-free framework for constructing set-valued predictions that guarantees covering the ground-truth values using a finite sample of data points [36, 1].”

“Specifically, we leverage a split conformal framework (Shafer and Vovk, 2008) using the normalized scores 𝐺 ( 𝑋, 𝑌 ) /𝑞 𝜏 ( 𝑋 ).”

“It has finite-sample coverage under the exchangeability assumption (Vovk et al., 2005; Shafer & Vovk, 2008).”

“Conformal prediction provides a general framework for constructing prediction sets with finite-sample, distribution-free coverage guarantees [9–12], and has recently been extended to causal inference [13–16].”

“The Conformal Prediction [17] is a distribution-free framework that provides predictive sets with statistical guarantees.”

“This property has made conformal methods widely used for uncertainty quantification in regression problems [Vovk et al., 2005, 2009, Shafer and Vovk, 2008].”

“Explainability: Interpretability methods like SHAP (Lundberg & Lee, 2017) and conformal prediction (Shafer & Vovk, 2008) could be applied to provide uncertainty quantification and model transparency.”

“For the purpose of uncertainty quantification, conformal prediction (CP) (Shafer and Vovk (2008)) has been introduced for risk-aware navigation in several aspects (Yang et al. (2023); Kwon et al. (2025); Mei et al. (2024)), where CP could provide valid prediction regions with a user-specified…”

“Conformal prediction (CP) is a distribution-free uncertainty quantification method [31,6,2] based on the assumption of the exchangeability of data.”

“Conformal Prediction (CP) yields a set of possible targets rather than a single prediction to accommodate prediction uncertainty [1, 2].”

“Second, test-conditional coverage (Shafer & Vovk, 2008) is considered because a marginally valid prediction set may still undercover in certain regions of the data space.”

“While Conformal Prediction (CP) [Vovk et al., 2005, Shafer and Vovk, 2008] provides powerful prediction intervals with finite-sample marginal coverage guarantees, standard CP assumes data exchangeability—a property severely violated by continual distribution shifts and abrupt regime shifts in…”

“Conformal prediction [25, 29] provides a distribution-free way to wrap a forecaster with calibrated prediction sets.”

“Furthermore, conformal prediction offers distribution-free risk control, enabling the conversion of model scores into calibrated decisions under mild assumptions and providing an effective mechanism for false-alarm control in the presence of drift [16].”

“We therefore use a separate calibration dataset to convert this disagreement into uncertainty bounds (Angelopoulos & Bates, 2023; Vovk et al., 1999; 2022; Shafer & Vovk, 2008; Papadopoulos et al., 2002).”

“CP offers a robust approach to construct prediction sets that obtain valid coverage for exchangeable data [see Shafer and Vovk, 2008, Vovk et al., 2022, for more on CP].”

“Conformal prediction constructs prediction intervals with finite-sample coverage guarantees under exchangeability [21,31].”

“Conformal prediction provides a distribution-free framework for constructing valid prediction intervals from empirical prediction errors computed on a held-out validation set [Shafer and Vovk, 2008].”

“By calibrating prediction intervals based on the sorted residuals from a calibration set, CP provides statistically valid coverage guarantees under the mild assumption of exchangeability (Shafer and Vovk 2008).”

“While Conformal Prediction (CP) (Vovk et al., 2005; Shafer and Vovk, 2008) offers a distribution-free and model-agnostic framework to address this, traditional CP methods are insufficient for LRMs. Existing approaches either calibrate the final answer while ignoring the reasoning trace (Ye et al.,…”

“Conformal prediction offers a distribution-free means of generating prediction sets with guaranteed coverage levels for uncertainty quantification [11] and has recently been studied to increase decision reliability in fault diagnosis [12]–[14].”

“We compare our uncertainty representation using ACCRUE to the performance of deterministic forecasts without uncertainty quantification, as well as two state-of-the-art uncertainty methods: conformal prediction (CP) [23, 20] and EasyUQ [24].”

“These parametric uncertainties are determined via Conformal Prediction (CP), 22,23 as illustrated in Fig.”

“Conformal prediction [11] provides coverage guaranteed sets/intervals, but typically lacks a structured posterior and does not naturally connect uncertainties to pro-totypical image patterns.”