The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index; none of the 22 sources its claims cite is checked for support yet.
Distribution-free, Risk-controlling Prediction Sets
Open accessAbstract
While improving prediction accuracy has been the focus of machine learning in recent years, this alone does not suffice for reliable decision-making. Deploying learning systems in consequential settings also requires calibrating and communicating the uncertainty of predictions. To convey instance-wise uncertainty for prediction tasks, we show how to generate set-valued predictions from a black-box predictor that controls the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets. This framework enables simple, distribution-free, rigorous error control for many tasks, and we demonstrate it in five large-scale machine learning problems: (1) classification problems where some mistakes are more costly than others; (2) multi-label classification, where each observation has multiple associated labels; (3) classification problems where the labels have a hierarchical structure; (4) image segmentation, where we wish to predict a set of pixels containing an object of interest; and (5) protein structure prediction. Last, we discuss extensions to uncertainty quantification for ranking, metric learning, and distributionally robust learning.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces risk-controlling prediction sets (RCPS), a distribution-free, finite-sample method to turn any black-box predictor into set-valued predictions whose expected loss is below a user-specified level with high probability. It develops upper-confidence-bound calibration, concentration inequalities for bounded and unbounded losses, and demonstrates the framework on five large-scale prediction tasks.
Key findings
- RCPS calibration chooses the smallest nested set parameter \(\lambda\) such that a pointwise upper confidence bound on risk stays below \(\alpha\) for all larger \(\lambda\), yielding a \((\alpha,\delta)\)-RCPS guarantee (Theorem 1).
- For bounded losses, the Waudby-Smith–Ramdas bound is recommended for non-binary losses; for binary losses, the exact binomial bound should be used. For unbounded losses, the Pinelis–Utev bound gives valid coverage under bounded coefficient of variation, while the CLT bound is tighter but lacks finite-sample guarantees and undercovers heavy-tailed distributions.
- The greedy set construction is approximately optimal: under loss forms \(L(y,S)=L_y 1\{y\notin S\}\) and more general \(L(y;S)=\int_{z\in S^c}\ell(y,z)d\mu(z)\), it has smallest average set size among risk-controlling predictors (Theorems 7 and 8).
- Experiments report risk control on ImageNet classification with class-varying loss (30,000 calibration, 20,000 evaluation points, 100 splits, \(\alpha=\delta=0.1\)), MS COCO multi-label classification (4,000 calibration, 1,000 test, \(\alpha=\delta=0.1\)), hierarchical ImageNet classification (\(\alpha=0.05,\delta=0.1\)), polyp segmentation (1,781 examples, 1,000 calibration, \(\alpha=\delta=0.1\)), and protein structure prediction on 71 CASP-13 test points (35 calibration, 36 test, \(\alpha=2\)Å, \(\delta=0.1\)).
Methods
- Split data into a training set and a calibration set; fit an arbitrary black-box predictor \(f\) on the training set, then use only calibration points to calibrate a nested family of set-valued predictors \(T_\lambda\) with monotone loss.
- Construct upper confidence bounds for risk via concentration inequalities: Hoeffding, Hoeffding–Bentkus, empirical Bernstein, Waudby-Smith–Ramdas, exact binomial for binary losses, Pinelis–Utev for unbounded losses, and an asymptotic CLT bound; numerical simulations use one million replicates to compare coverage and median gap.
- Evaluate on five tasks using ImageNet with ResNet-152/ResNet-18, MS COCO with TResNet, polyp segmentation datasets with PraNet, and AlphaFoldv1 distograms for protein structure, reporting risk and set sizes over random splits.
Limitations
- The paper states that unbounded losses require regularity conditions such as bounded coefficient of variation; Proposition A.1 shows no nontrivial finite-sample upper confidence bound exists for nonnegative random variables without restrictions.
- The CLT bound lacks finite-sample coverage guarantees and undercovers in simulations, especially for small means and heavy-tailed distributions; the protein experiment uses only 35 calibration and 36 test points and reports only the CLT bound because exact concentration results are “hopelessly conservative” at that sample size.
- The protein experiment uses semi-synthetic data because ground-truth distance data was unavailable, and the greedy optimality results assume exact conditional density and specific loss forms, whereas practice uses plug-in estimates.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
70 Own interpretation · 17 Sources not found · 15 Own result · 4 No basis given
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“due to the small sample size (35 calibration and 36 test points), we only report results using the CLT bound, because the exact concentration results are hopelessly conservative with only 35 calibration points.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.5 Protein Structure Prediction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 15 shows the quantitative performance of the CLT, which nearly controls the risk.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.5 Protein Structure Prediction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By contrast, the PU bound has valid coverage in these settings.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.2.2 Numerical Comparisons of Upper Confidence Bounds.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In Figure 8, we find that the conformal baseline returns larger prediction sets.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.2 Multi-label Classification
What the sources say
- Not matchedcites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we observe that the CLT does not always have correct coverage, especially when the true mean is small, unless the sample size is large.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.1.4 Numerical Experiments for Bounded Losses.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The loss is bounded above by one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The CLT bound is excluded in (b), because it does not achieve the target coverage in most of the cases.”
- Also stated in: § E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION · plain
“The CLT bound is excluded in (b) because it does not achieve the target coverage in most of the cases.”
- Also stated in: § E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION · plain
“The CLT bound is excluded in (b) because it does not achieve the target coverage in most of the cases.”
§ Distribution-free, Risk-controlling Prediction Sets > E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results show that the WSR bound is not uniformly better than the HB bound, although it is still the best all-around choice for bounded losses.”
§ Distribution-free, Risk-controlling Prediction Sets > D FURTHER COMPARISONS OF UPPER CONFIDENCE BOUNDS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The HB bound is more likely to be tighter than the WSR bound, especially when the mean μ or the level δ is small.”
§ Distribution-free, Risk-controlling Prediction Sets > D FURTHER COMPARISONS OF UPPER CONFIDENCE BOUNDS
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we show how to generate set-valued predictions from a black-box predictor that controls the expected loss on future test points at a user-specified level.”
§ Distribution-free, Risk-controlling Prediction Sets > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: For each λ, a pointwise upper confidence bound R+(λ) for R(λ) is available such that P(R(λ) ≤ R+(λ)) ≥ 1-δ.
- The authors state this as an assumption: The loss function satisfies the nesting property: S ⊂ S' ⇒ L(y,S) ≥ L(y,S').
- The authors state this as an assumption: The collection of set-valued predictors {Tλ} is nested: λ1 < λ2 ⇒ Tλ1(x) ⊂ Tλ2(x).
- The authors state this as an assumption: The data (Xi, Yi) are i.i.d.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This framework enables simple, distribution-free, rigorous error control for many tasks”
§ Distribution-free, Risk-controlling Prediction Sets > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 22
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Next, as shown in Figure 4(b), the WSR bound outperforms the others for all Beta distributions and has a similar performance to the HB bound for Bernoulli distributions.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.1.4 Numerical Experiments for Bounded Losses.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The loss is bounded above by one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From Figure 5(a), we see that the CLT bound nearly achieves the desired coverage for light-tailed distributions but drastically undercovers for heavy-tailed distributions.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.2.2 Numerical Comparisons of Upper Confidence Bounds.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As before, we find that RCPS controls the risk at the desired level, and the predictions are generally relatively precise (i.e., of low depth in the tree).”
§ Distribution-free, Risk-controlling Prediction Sets > 5.3 Hierarchical Classification
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From Figure 5(b), we see that the CLT bound is much tighter in all cases, though the gap between two bounds shrink as the sample size grows.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.2.2 Numerical Comparisons of Upper Confidence Bounds.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“this work explicitly considers classification with different penalties for different misclassification events, multi-label classification, classification with hierarchically structured classes, image segmentation, prediction problems where the response is a 3D structure, ranking, and metric learning.”
§ Distribution-free, Risk-controlling Prediction Sets > Our Contribution
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“RCPS again control the risk at the desired level, and the average prediction set size size is comparable to the average polyp size.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.4 Image Segmentation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our method controls the risk and gives sets with reasonable sizes.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.2 Multi-label Classification
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The RCPS procedure controls the risk at the correct level and the sets have reasonable sizes.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.1 Classification with a Class-varying Loss
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The left column shows that when we seek to control the risk at the relatively loose α = 0.1 level, around 1, 000 calibration points suffice; the middle panel shows that when we seek to control the risk at level α = 0.01, a few thousand calibration points suffice; and the right column shows that for the strict risk level α = 0.001, about 10,000 calibration points suffice.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.4 How Large Should the Calibration Set Be?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The confidence bound of Waudby-Smith and Ramdas [59] is the clear winner, and ultimately we recommend this bound for use in all cases with bounded loss.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.1 Bounded Losses
What the sources say
- Not matchedcites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- The authors state this as an assumption: The loss is bounded above by one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The set sizes for all three methods overlap.”
§ Distribution-free, Risk-controlling Prediction Sets > E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our prediction is an ancestor in the WordNet hierarchy of both the true class and the model's top-1 prediction.”
§ Distribution-free, Risk-controlling Prediction Sets > E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The reason is that the notion of coverage used by conformal prediction is more strict, requiring that all classes are covered.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.2 Multi-label Classification
What the sources say
- Not matchedcites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Because of the explicit guarantee on our output, a doctor could safely exclude diagnoses outside the set and test for those within.”
§ Distribution-free, Risk-controlling Prediction Sets > 1 INTRODUCTION
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“In parallel, conformal prediction [55,56] has been recognized as an attractive way of producing predictive sets with finite-sample guarantees.”
§ Distribution-free, Risk-controlling Prediction Sets > Related Work
What the sources say
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [56]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To do this, the size of the output set is chosen based on the accuracy of the classifier and the desired risk level-a lower accuracy classifier or a more strict risk level will require larger sets to guarantee risk control.”
§ Distribution-free, Risk-controlling Prediction Sets > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In view of Equation ( 6), UCB calibration with this version of Hoeffding's bound results in a procedure that is simple to state-one selects the smallest set size such that the empirical risk on the calibration set is below αlog(1/δ )/2n.”
§ Distribution-free, Risk-controlling Prediction Sets > R(λ)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The loss is bounded above by one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Returning a set of responses is a useful way to represent uncertainty, since such sets can be readily constructed from any existing predictor and, moreover, they are often interpretable.”
§ Distribution-free, Risk-controlling Prediction Sets > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Tolerance regions are sets that contain a desired fraction of the population distribution with high probability.”
§ Distribution-free, Risk-controlling Prediction Sets > Related Work
What the sources say
- Not matchedcites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [60]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [61]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We assumed that R(•) is continuous for simplicity, but this condition can be removed with minor modifications.”
§ Distribution-free, Risk-controlling Prediction Sets > Remark 2.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Unlike conformal prediction, however, we pursue the high-probability error guarantees of tolerance regions and thus rely on entirely different proof techniques-see Reference [53] for a discussion of their relationship.”
§ Distribution-free, Risk-controlling Prediction Sets > Related Work
What the sources say
- Not matchedcites [53]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The required number of samples will increase slightly if we ask for a higher confidence level (i.e., smaller δ ), but the dependence on δ is minimal, since the bounds will roughly scale as log(1/δ )”
§ Distribution-free, Risk-controlling Prediction Sets > 3.4 How Large Should the Calibration Set Be?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Then, T λPU is a (α, δ )-RCPS.”
§ Distribution-free, Risk-controlling Prediction Sets > Table 1. Distributions Considered for the Unbounded Case
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- The authors state this as an assumption: For each λ, c_v(λ) ≤ c_v for some constant c_v.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This result is only presented for illustration purposes, however.Much tighter concentration results are available, so in practice, we recommend using the better bounds described next.”
§ Distribution-free, Risk-controlling Prediction Sets > R(λ)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We next present a bound that is adaptive to the variance and improves upon the previous result in most settings.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.1.3 Waudby-Smith-Ramdas Bound.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the case considered in Section 4.2, E[ℓ(Y ; z) | X = x] = L z p(z | x ), so we see Theorem 8 includes Theorem 7 as a special case.”
§ Distribution-free, Risk-controlling Prediction Sets > 4.3 Optimality in a More General Setting
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The present work concerns the construction of tolerance regions with a user-specified coverage guarantee, and we do not pursue this latter formulation here.”
§ Distribution-free, Risk-controlling Prediction Sets > Related Work
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is not surprising that the HB bound performs well for binary variables, since the Bentkus inequality is nearly tight here.”
§ Distribution-free, Risk-controlling Prediction Sets > 3.1.4 Numerical Experiments for Bounded Losses.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When c v (λ) is unknown, we can treat it as a sensitivity parameter or estimate it based on the sample moments.”
§ Distribution-free, Risk-controlling Prediction Sets > Table 1. Distributions Considered for the Unbounded Case
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since our procedure finds the last point where the UCB R + (λ) is above the desired level α, it will produce sets that are nearly as small as possible when R + (λ) is close to the true risk R(λ).”
§ Distribution-free, Risk-controlling Prediction Sets > 3.4 How Large Should the Calibration Set Be?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our algorithm includes the construction of tolerance regions as special case, but applies to many other problems”
§ Distribution-free, Risk-controlling Prediction Sets > Our Contribution
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our proposed algorithm is somewhat similar to split conformal prediction.”
§ Distribution-free, Risk-controlling Prediction Sets > Related Work
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the Hoeffding-Bentkus bound is nearly tight for binary loss functions”
§ Distribution-free, Risk-controlling Prediction Sets > 3.1.3 Waudby-Smith-Ramdas Bound.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The remainder of this article shows how to subsequently create set-valued predictors from f that control a certain statistical error notion, regardless of the quality of the initial model fit or the distribution of the data.”
§ Distribution-free, Risk-controlling Prediction Sets > 2.1 Setting and Notation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, our approach can incorporate many loss functions, whereas conformal prediction controls the coverage-i.e., binary risk.”
§ Distribution-free, Risk-controlling Prediction Sets > 7 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“this is the well-known case of split conformal prediction; see Reference [22].”
§ Distribution-free, Risk-controlling Prediction Sets > C CONFORMAL CALIBRATION
What the sources say
- Not matchedcites [22]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“The binary loss case case results in a classical tolerance region, as discussed previously in References [53] and [40].”
§ Distribution-free, Risk-controlling Prediction Sets > B AN EXACT BOUND FOR BINARY LOSS
What the sources say
- Not matchedcites [53]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“When the loss takes values in {0, 1}, for a fixed λ the loss at each point is a Bernoulli random variable, and the risk is simply the mean of this random variable.”
§ Distribution-free, Risk-controlling Prediction Sets > B AN EXACT BOUND FOR BINARY LOSS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, risk-controlling prediction sets apply whenever one has access to a concentration result, whereas conformal prediction relies on exchangeability, a particular combinatorial structure.”
§ Distribution-free, Risk-controlling Prediction Sets > 7 DISCUSSION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“For sums of i.i.d.random variables, the Hoeffding's inequality (Proposition 3) is derived by setting ψ (z) = exp{νz} [26], and the Bentkus inequality (Proposition 4) is derived by setting ψ (z) = (zν ) + .”
§ Distribution-free, Risk-controlling Prediction Sets > APPENDICES A PROOFS
What the sources say
- Not matchedcites [26]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While carrying out this procedure would require computational innovations, our results establish that it is statistically valid.”
§ Distribution-free, Risk-controlling Prediction Sets > 6.3 Adversarially Robust Uncertainty Quantification
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
There exists λmax ∈ Λ such that R(λmax)=0.
“We further assume that there exists an element λ max ∈ Λ such that R(λ max ) = 0.”
The authors state this as an assumption.
For each λ, a pointwise upper confidence bound R+(λ) for R(λ) is available such that P(R(λ) ≤ R+(λ)) ≥ 1-δ.
“We assume that we have access to a pointwise upper confidence bound (UCB) for the risk function for each λ: where R + (λ) may depend on (X 1 , Y 1 ), . . ., (X n , Y n ).”
The authors state this as an assumption.
The loss function satisfies the nesting property: S ⊂ S' ⇒ L(y,S) ≥ L(y,S').
“We require that the loss function respects the following nesting property: S ⊂ S =⇒ L(y, S) ≥ L(y, S ).(2)”
The authors state this as an assumption.
The collection of set-valued predictors {Tλ} is nested: λ1 < λ2 ⇒ Tλ1(x) ⊂ Tλ2(x).
“We further suppose we have a collection of such set-valued predictors indexed by a one-dimensional parameter λ taking values in a closed set Λ ⊂ R ∪ {±∞} that are nested, meaning that larger values of λ lead to larger sets: λ 1 < λ 2 =⇒ T λ 1 (x ) ⊂ T λ 2 (x ).(1)”
The authors state this as an assumption.
R(λ) is continuous.
“Suppose Equation (3) holds pointwise for each λ, and that R(λ) is continuous.”
The authors state this as an assumption.
The calibration data and test data come from the same distribution.
“The only requirement is that the calibration data and the test data come from the same distribution.”
The authors state this as an assumption.
The data (Xi, Yi) are i.i.d.
“Let (X i , Y i ) i=1, ...,m be an independent and identically distributed (i.i.d.) set of variables”
The authors state this as an assumption.
For each λ, c_v(λ) ≤ c_v for some constant c_v.
“suppose in addition that for each λ ∈ Λ, c v (λ) ≤ c v for some constant c v .”
The authors state this as an assumption.
The loss L(Y, Tλ(X)) has a finite second moment for each λ.
“assume additionally that L(Y , T λ (X )) has a finite second moment for each λ.”
The authors state this as an assumption.
The loss is bounded above by one.
“We begin with the case where our loss is bounded above and, without loss of generality, we take the bound to be one.”
The paper relies on this without support.
Black-box predictive algorithms are rarely accompanied by reliable uncertainty quantification.
“Problematically, however, these algorithms are rarely accompanied by reliable uncertainty quantification.”
The paper relies on this without support.
Prediction failures will often be silent ones, particularly in high-consequence settings.
“Thus, prediction failures will often be silent ones, which is particularly alarming in high-consequence settings.”
The paper relies on this without support.
Black-box prediction will be with us for some time to come.
“it is clear that black-box prediction will be with us for some time to come.”
The authors state this as an assumption.
The predictor px(y) is an estimate of p_{Y|X=x}(y).
“We assume that our predictor is px (y), an estimate of p Y |X =x (y), and we let ρx (y, S) = L(y, S) px (y).”
The authors state this as an assumption.
The loss function has the simple form L(y,S)=L_y 1{y not in S} for constants L_y.
“Suppose our loss function has the simple form L(y, S) = L y 1 {y S} , for constants L y .”
The authors state this as an assumption.
The loss takes the form L(y;S)=∫_{z∈S^c} ℓ(y,z)dμ(z) for some nonnegative ℓ and a finite measure μ.
“Suppose our loss takes the form L(y; S) = z ∈S c ℓ(y, z)dμ (z), for some nonnegative ℓ and a finite measure μ.”
The authors state this as an assumption.
In the ranking setting, we have an i.i.d. sequence of points (X1,Y1),...,(Xm,Ym), where Y∈{1,...,k}.
“We assume we have an i.i.d.sequence of points, (X 1 , Y 1 ), . . ., (X m , Y m ), where Y ∈ {1, . . ., k }.”
The authors state this as an assumption.
In the ranking setting, the family of set-valued predictors satisfies the monotonicity property λ<λ' implies Tλ ⊂ Tλ'.
“We assume that we have a family of such predictors satsifying the following monotonicity property: λ < λ =⇒ T λ (x 1 , x 2 ) ⊂ T λ (x 1 , x 2 ).(13)”
The authors state this as an assumption.
In the metric learning setting, we have an i.i.d. sequence of points (X1,Y1),...,(Xm,Ym), with Y={1,...,k}.
“We next consider the problem of supervised metric learning, where we have an i.i.d.sequence of points (X 1 , Y 1 ), . . ., (X m , Y m ), with Y = {1, . . ., k }.”
The authors state this as an assumption.
In the metric learning setting, the family of set-valued metrics has the monotonicity property in Equation (13).
“We assume that we have a family of such set-valued metrics that have the monotonicity property in Equation (13).”
The authors state this as an assumption.
In the metric learning setting, the loss function is bounded by 1.
“In the setting above, suppose the loss function is bounded by 1.”
The authors state this as an assumption.
In the ranking setting, the loss function is bounded by one.
“Consider the setting above with any loss function bounded by one.”
The authors state this as an assumption.
The family of sets Tλ is such that the minimal element in the score definition exists with probability one.
“where we assume that the family of sets T λ is such that the minimal element exists with probability one.”
The paper relies on this without support.
The general case where Y is not binary has not been explicitly dealt with in prior work.
“To the best of our knowledge, the general case where Y 2 Y has not been explicitly dealt with, so we record this mild generalization here.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, RCPS focuses exclusively on the average risk, which fails to capture critical aspects of risk-sensitive decisionmaking, particularly the tail behavior of the loss distribution.”
From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [8]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The WSR UCB in (12) satisfies the inequality”
From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“[39] generalizes [18] to control the quantile of a monotone loss function conditional on ( X i , Y i ) n i =1 with probability 1 -δ over the calibration dataset for any user-specified tolerance parameter δ .”
From Conformal Risk Control (2022) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“risk-controlling prediction sets (RCPS) [8] offer a principled approach to control the expected loss of the prediction set with high probability.”
From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“SCP offers distribution-free, model-agnostic guarantees on groundtruth coverage, assuming data exchangeability.”
From Conformal Lesion Segmentation for 3D Medical Images (2025) · cites this paper as Bates et al. 2021
The cited source says what the claim attributes to it.
In the citing paper
“the algorithm presented herein differs greatly: it has a guarantee in expectation, and neither the algorithm nor its analysis share much technical similarity with these previous works.”
From Conformal Risk Control (2022) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Distribution-free, Risk-controlling Prediction Sets > 5.5 Protein Structure Prediction
“The label space Y is the set of all symmetric matrices with positive elements of any side length.”
§ Distribution-free, Risk-controlling Prediction Sets > 5.5 Protein Structure Prediction
“the entry y i, j defines the distance in 3D space of residues x i and x j ; hence, y ∈ R |x |×|x | , and y i, j = y j,i .”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Distribution-free, Risk-controlling Prediction Sets > 5.4 Image Segmentation
“In Figure 12, we report on our method's performance on 20 randomly selected images from the polyp datasets that contain at least two polyps”
§ Distribution-free, Risk-controlling Prediction Sets > E ADAPTIVE SCORE RENORMALIZATION FOR POLYP SEGMENTATION
“The top two rows show examples with a single polyp per image, and the second two rows show examples with two polyps per image.”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Here, we conduct an experiment on Imagenet—the gold-standard computer vision classification dataset— comprised of one thousand classes of natural images [42].”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 1×
“To evaluate our method, we use the Microsoft Common Objects in Context (MS COCO) dataset, a large-scale, eighty-category dataset of natural images in realistic and often complicated contexts [45].”
MethodProbability Inequalities for sums of Bounded Random Variables
Hoeffding, W. (1994) · Springer series in statistics · doi:10.1007/978-1-4612-0865-5_26 · Cited by 6,683
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
Probability Inequalities for Sums of Bounded Random Variables
Hoeffding, W. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500830 · Cited by 4,869
Interval Estimation for a Binomial Proportion
Brown, L. D., Cai, T., & Dasgupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,746
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378 · cited 1×
“Because the AlphaFoldv2 codebase was not released at the time this paper was written, we use AlphaFoldv1 here [55].”
MethodPraNet: Parallel Reverse Attention Network for Polyp Segmentation
Fan, D.-P., Ji, G.-P., Zhou, T., Chen, G., Fu, H., Shen, J.-B., & Shao, L. (2020) · International Conference on Medical Image Computing and Computer-Assisted Intervention · doi:10.1007/978-3-030-59725-2_26 · Cited by 2,003 · cited 1×
“We used PraNet [54] as our base segmentation model.”
MethodToward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer
Silva, J. C., Histace, A., Romain, O., Dray, X., & Granado, B. (2013) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-013-0926-3 · Cited by 1,318 · cited 2×
“We evaluated our method with an experiment combining several open-source polyp segmentation datasets: Kvasir [45], Hyper-Kvasir [46], CVC-ColonDB and CVC-ClinicDB [47], and ETIS-Larib [48].”
Method
Research works printed but not linked: 22
- Angelopoulos Anastasios Nikolas, International Conference on Learning Representations (ICLR).
- 10.5555/3454287.3455291 · DOI
- Cauchois Maxime, 2020, Duchi
- 10.5555/2354409.2355033 · DOI
- Grycko Eugen, Information and Classification
- Gupta Chirag, 2020, Ramdas
- Izbicki Rafael, 2019, Stern
- Lei Lihua, 2020, Candès
- Lin Tsung-Yi, European Conference on Computer Vision. Springer, 740–755
- Madry Aleksander, 2018, International Conference on Learning Representations (ICLR).
- 10.5555/1148466.1148472 · DOI
- Maurer Andreas, 2009, Conference on Learning Theory (COLT).
- Park Sangdon, 2020, International Conference on Learning Representations (ICLR).
- Park Sangdon, 2021, International Conference on Learning Representations (ICLR).
- 10.5555/3454287.3454605 · DOI
- Romano Yaniv, 2020, International Conference on Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), 33
- Senior Andrew W., 2020, et al
- 10.5555/3454287.3454514 · DOI
- Vovk Vladimir, 2012, Asian Conference on Machine Learning, 25
- 10.5555/645528.657641 · DOI
- 10.5555/1062391 · DOI
- Vovk Vladimir, 2020, Ninth Symposium on Conformal and Probabilistic Prediction and Applications, Alexander Gammerman, Vladimir Vovk, Zhiyuan Luo, Evgueni Smirnov, and Giovanni Cherubin (Eds.), 128
Cited by
Sorted by citations
Conformal Prediction: A Gentle Introduction
Angelopoulos, A. N., & Bates, S. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000101 · Cited by 2,636
Self-citationTesting for outliers with conformal p-values
Bates, S., Candès, E. J., Lei, L., Romano, Y., & Sesia, M. (2023) · The Annals of Statistics · doi:10.1214/22-aos2244 · Cited by 112
“See [64] for a overview of the subject, [38] for a discussion of their connection with conformal inference, and [65, 66] for modern examples using tolerance regions for predictive inference with neural networks.”
BackgroundSelf-citationFair Conformal Predictors for Applications in Medical Imaging
Lu, C., Lemay, A., Chang, K., Hoebel, K., & Kalpathy–Cramer, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21459 · Cited by 79
“Although we focus on multi-class classification, the conformal framework is easily extended to other prediction tasks such as regression, multi-label classification, and image segmentation (Romano, Patterson, and Candès 2019; Cauchois, Gupta, and Duchi 2020; Bates et al. 2021).”
MethodLearn then test: Calibrating predictive algorithms to achieve risk control
Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., & Lei, L. (2025) · The Annals of Applied Statistics · doi:10.1214/24-aoas1998 · Cited by 309
“The resulting guarantees are the first of their kind; all previous work in distribution-free uncertainty quantification, such as conformal prediction [1] and risk-controlling prediction sets [2], have required that λ is one-dimensional and that the risk function be monotonic in λ .”
InfluentialSelf-citationTomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Bulian, J., Buck, C., Gajewski, W., Börschinger, B., & Schuster, T. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.20 · Cited by 44
Conformal Prediction: A Data Perspective
Zhou, X., Chen, B., Gui, Y., & Cheng, L. (2025) · ACM Computing Surveys · doi:10.1145/3736575 · Cited by 69
“For example, [6] proposes a distribution-free pipeline for image-to-image regression based on risk control [15].”
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T. (2022) · arXiv · doi:10.48550/arxiv.2208.02814 · Cited by 37
Self-citationYang, H., & Pavone, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00864 · Cited by 43
“Existing works focus on image classifi-cation [1, 76], tumor segmentation [3, 8, 94], and bounding box detection [2, 23, 53], which are classification or low-dimensional regression problems.”
BackgroundConsistent Accelerated Inference via Confident Adaptive Transformers
Schuster, T., Fisch, A., Jaakkola, T., & Barzilay, R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.406 · Cited by 46
Sensitivity analysis of individual treatment effects: A robust conformal inference approach
Jin, Y., Ren, Z., & Candès, E. J. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2214889120 · Cited by 33
Conformal Prediction for Natural Language Processing: A Survey
Campos, M., Farinhas, A., Zerva, C., Figueiredo, M. A. T., & Martins, A. F. T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00715 · Cited by 63
“This is also related to (but different from) previous work by Bates et al. (2021) and Angelopoulos et al. (2022), who prove bounds of the form where δ is a parameter and ℓ does not need to be monotone.”
Reliability engineering, risk management, and trustworthiness assurance for AI systems
Zhang, X., Wang, T., Ma, L., & Mahadevan, S. (2025) · Journal of Reliability Science and Engineering · doi:10.1088/3050-2454/adcef2 · Cited by 29
Active deep learning on entity resolution by risk sampling
Nafa, Y., Chen, Q., Chen, Z., Lu, X., He, H., Duan, T., & Li, Z. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107729 · Cited by 20
“It is noteworthy that the concept of risk has also been leveraged in other works for different purposes as demonstrated in [49, 48, 2].”
Ghosh, S., Belkhouja, T., Yan, Y., & Doppa, J. R. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25936 · Cited by 27
Quantum Conformal Prediction for Reliable Uncertainty Quantification in Quantum Machine Learning
Park, S., & Simeone, O. (2023) · IEEE Transactions on Quantum Engineering · doi:10.1109/tqe.2023.3333224 · Cited by 27
“Future directions for research include the generalization of the QCP framework to more general form of risk control beyond coverage [53]–[55].”
BackgroundConformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertainty
Mossina, L., Dalmau, J., & Andéol, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00361 · Cited by 18
“Also based on risk control, albeit us-ing different mathematical frameworks, the contributions of [3, 8, 46] extend the concept of tolerance regions to ML problems.”
BackgroundMethodLearning Optimal Conformal ClassifiersPreprint
Stutz, D., Krishnamurthy, Dvijotham, Cemgil, A. T., & Arnaud Doucet, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.09192 · Cited by 70
“Nevertheless, extensions to regression, or other CP formulations such as (Bates et al., 2021) during training, are possible.”
InfluentialTan, W., Cao, Y., Ma, X., Ru, G., Li, J., Zhang, J., Gao, Y. J., Yang, J., Huang, G., Yan, B., et al. (2023) · Engineering · doi:10.1016/j.eng.2022.10.015 · Cited by 18
Principal Uncertainty Quantification With Spatial Correlation for Image Restoration Problems
Belhasin, O., Romano, Y., Freedman, D., Rivlin, E., & Elad, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3343031 · Cited by 26
“One key paradigm in this context is conformal prediction (CP) [12], [13], [14] and risk-controlling methods [15], [33], [34], which allow to rigorously quantify the prediction uncertainty of a machine learning model with a user-specified probability guarantee.”
BackgroundMethodFew-shot Conformal Prediction with Auxiliary TasksPreprint
Fisch, A., Schuster, T., Jaakkola, T., & Barzilay, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08898 · Cited by 22
“…conformal prediction in the context of various validity conditions, such as conditional, risk-controlling, admissible, or equalized coverage (Chernozhukov et al., 2019; Cauchois et al., 2020; Kivaranovic et al., 2020; Romano et al., 2019; 2020; Bates et al., 2020; Fisch et al., 2021, inter alia ).”
BackgroundInfluentialConformalized Link Prediction on Graph Neural Networks
Zhao, T., Kang, J. L., & Cheng, L. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672061 · Cited by 15
Hajibabaee, P., Pourkamali‐Anaraki, F., & Hariri‐Ardebili, M. A. (2024) · IEEE Access · doi:10.1109/access.2024.3387858 · Cited by 15
“This work is ongoing and is an active area of research within the field of conformal prediction [18], [20], [38], [39].”
BackgroundExchangeability, Conformal Prediction, and Rank TestsPreprint
Kuchibhotla, A. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06095 · Cited by 14
“Prediction problems in general spaces occur in applications such as functional data analysis (Lei et al., 2015) and image prediction (Bates et al., 2021).”
BackgroundCalibrated Multiple-Output Quantile Regression with Representation LearningPreprint
Feldman, S., Bates, S., & Romano, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.00816 · Cited by 14
“Extensions of our procedure to control other error rates would be possible in combination with generalizations of conformal prediction (Bates et al., 2021; Angelopoulos et al., 2021), and we view this as an important next step.”
MethodSelf-citationVishwakarma, R., & Rezaei, A. (2023) · doi:10.1109/iccad57390.2023.10323655 · Cited by 13
“Furthermore, we also explored variations of conformal predictors as described in [50].”
MethodConformal Prediction Masks: Visualizing Uncertainty in Medical Imaging
Kutiel, G., Cohen, R., Elad, M., Freedman, D., & Rivlin, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39539-0_14 · Cited by 12
“, 2022b), conformal risk control (Angelopoulos et al., 2022a; Bates et al., 2021; Angelopoulos et al., 2021) and semantic uncertainty intervals for generative adversarial networks (Sankaranarayanan et al.”
BackgroundImproving Conditional Coverage via Orthogonal Quantile RegressionPreprint
Feldman, S., Bates, S., & Romano, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00394 · Cited by 12
“Next, HSIC is a more sophisticated, nonlinear complement to the Pearson’s correlation measure, which can detect arbitrary complex relationships between the coverage identifier and the interval length.”
MethodSelf-citationRisk verification of stochastic systems with neural network controllers
Cleaveland, M., Lindemann, L., Ivanov, R., & Pappas, G. J. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103782 · Cited by 11
“In [43] and [44], the authors used statistical testing to derive high confidence bounds on the probability of a CPS with ML components satisfying an STL specification.”
MethodObject Detection with Probabilistic Guarantees: A Conformal Prediction Approach
de Grancey, F., Adam, J.-L., Alecu, L., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14862-0_23 · Cited by 11
Posterior-Variance–Based Error Quantification for Inverse Problems in Imaging
Narnhofer, D., Habring, A., Höller, M., & Pock, T. (2024) · SIAM Journal on Imaging Sciences · doi:10.1137/23m1546129 · Cited by 18
Selection and Aggregation of Conformal Prediction Sets
Yang, Y., & Kuchibhotla, A. K. (2024) · Journal of the American Statistical Association · doi:10.1080/01621459.2024.2344700 · Cited by 22
“This could be useful when the domain Z of the observations is more complicated, for example, the space of images or the space of functions (Lei, Rinaldo, and Wasserman 2015; Bates et al. 2021).”
Functional protein mining with conformal guarantees
Boger, R. S., Chithrananda, S., Angelopoulos, A. N., Yoon, P. H., Jordan, M. I., & Doudna, J. A. (2025) · Nature Communications · doi:10.1038/s41467-024-55676-y · Cited by 14
“Conformal prediction is a technique for calibrating arbitrary black-box prediction algorithms in order to satisfy statistical guarantees of marginal coverage (and notably, not conditional coverage [10, 47]).”
Self-citationConformalized survival analysis with adaptive cut-offs
Gui, Y., Hore, R., Ren, Z., & Barber, R. F. (2023) · Biometrika · doi:10.1093/biomet/asad076 · Cited by 19
Selection and Aggregation of Conformal Prediction SetsPreprint
Yang, Y., & Kuchibhotla, A. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13871 · Cited by 9
“This could be useful when the domain Z of the observations is more complicated, for example, the space of images or the space of functions (Lei, Rinaldo, and Wasserman 2015; Bates et al. 2021).”
Learning from learning machines: a new generation of AI technology to meet the needs of sciencePreprint
Pion-Tonachini, L., Bouchard, K. E., Martín, H. G., Peisert, S., Holtz, W. B., Aswani, A. J., Dwivedi, D., Wainwright, H. M., Pilania, G., Nachman, B. P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13786 · Cited by 14
“Conformal prediction techniques now make it possible to obtain accurate prediction intervals (measures of uncertainty) in such settings under only the assumption of access to statistically exchangeable observations [11].”
BackgroundAngelopoulos, A. N., Bates, S., Zrnic, T., & Jordan, M. I. (2022) · Harvard Data Science Review · doi:10.1162/99608f92.16c71dad · Cited by 8
“Further extensions address problems in distribution estimation [47, 48], handling or testing distribution shift [49, 50, 51], causal inference [52], and controlling other notions of statistical error [53].”
BackgroundSelf-citationCross-Validation Conformal Risk Control
Cohen, K. M., Park, S., Simeone, O., & Shamai, S. S. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619172 · Cited by 11
“CRC generalizes CP to address the calibration criterion (1) for a wider class of risks, with the only constraints that the risk function be bounded and monotonic in the predicted set size [7], [22], [25], [26].”
BackgroundRoot-finding approaches for computing conformal prediction set
Ndiaye, E., & Takeuchi, I. (2022) · Machine Learning · doi:10.1007/s10994-022-06233-5 · Cited by 9
“…have been developed for designing uncertainty sets in active learning (Ho & Wechsler, 2008), anomaly detection (Laxhammar & Falkman, 2015; Bates et al., 2021b), image classification (Angelopoulos et al., 2020), few shot learning (Fisch et al., 2021), time series (Xu & Xie, 2021) or to…”
BackgroundTraining-conditional coverage for distribution-free predictive inference
Bian, M., & Barber, R. F. (2023) · Electronic Journal of Statistics · doi:10.1214/23-ejs2145 · Cited by 8
“[2020], Bates et al. [2021], Yang and Kuchibhotla [2021]; see also Park et al.”
BackgroundMethodInfluentialAdaptive Bounding Box Uncertainties via Two-Step Conformal Prediction
Timans, A., Straehle, C.-N., Sakmann, K., & Nalisnick, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73223-2_21 · Cited by 13
AmnioML: Amniotic Fluid Segmentation and Volume Prediction with Uncertainty Quantification
Csillag, D., Paes, L. M., Ramos, T. R., Romano, J. V., Schuller, R., Seixas, R. D. B., Oliveira, R. I., & Orenstein, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26837 · Cited by 8
“In this paper, we make use of tools from the field of conformal prediction (Bates et al. 2021; Barber et al. 2020; Lei et al. 2018).”
MethodIntegrative conformal p-values for out-of-distribution testing with labelled outliers
Liang, Z., Sesia, M., & Sun, W. (2024) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad138 · Cited by 37
Tracking the risk of a deployed model and detecting harmful distribution shiftsPreprint
Podkopaev, A., & Ramdas, A. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2110.06177 · Cited by 57
“…could target covering the correct label of a test point with high probability (Vovk et al., 2005) or controlling other notions of risk (Bates et al., 2021) like the miscoverage loss: `miscov(y, S(x)) = 1 {y /∈ S(x)} , (14) where S(x) denotes the output of a set-valued predictor on any…”
BackgroundMethodInfluentialConformal Loss-Controlling Prediction
Wang, D., Wang, P., Ji, Z., Yang, X., & Li, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3356512 · Cited by 8
“The spirit is to employ calibration data to obtain the information of the upper bound of the expected value of the loss function at hand and control the expected value for the test object, whose main idea was originally proposed from their pioneer work named risk-controlling prediction sets [16].”
BackgroundMethodInfluential(2025) · doi:10.1007/978-3-032-04965-0_8 · Cited by 6
Conformal prediction for trustworthy detection of railway signals
Andéol, L., Fel, T., de Grancey, F., & Mossina, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00400-7 · Cited by 8
Semantic Meta-Split Learning: A TinyML Scheme for Few-Shot Wireless Image Classification
Eldeeb, E., Shehab, M. A., Alves, H., & Alouini, M. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3557734 · Cited by 5
“However, these metrics lack measuring the degree of uncertainty and the model’s trustworthiness, which is usually common in deep learning models with limited access to data points [42].”
Semi-Supervised Risk Control via Prediction-Powered Inference
Einbinder, B.-S., Ringel, L., & Romano, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3594263 · Cited by 5
“Other methods for constructing a UCB include Waudby-Smith– Ramdas (WSR) bound [53], Hoeffding–Bentkus bound [11], [54], Clopper–Pearson bound [55], and more.”
InfluentialYou Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Control
Toni, G. D., Purificato, E., G'omez, E., Passerini, A., Lepri, B., & Consonni, C. (2025) · ACM Conference on Recommender Systems · doi:10.1145/3705328.3748054 · Cited by 7
Segal, G., Keidar, N., Lotan, R. M., Romano, Y., Herskovitz, M., & Yaniv, Y. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1184990 · Cited by 5
“, 1999), risk-controlling prediction sets (Bates et al., 2021), and the Learn Then Test method (LTT) (Angelopoulos et al.”
MethodDiffusion Models for Generative Histopathology
Sridhar, N., Elad, M., McNeil, C., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53767-7_15 · Cited by 5
Non-Exchangeable Conformal Risk ControlPreprint
Farinhas, A., Zerva, C., Ulmer, D., & Martins, A. F. T. (2023) · IT University Of Copenhagen (IT University of Copenhagen) · doi:10.48550/arxiv.2310.01262 · Cited by 24
Exploring Uncertainty in Medical Federated Learning: A Survey
Zeng, X., Ahmed, A., & Tunio, M. H. (2025) · Electronics · doi:10.3390/electronics14204072 · Cited by 5
Adaptive deep learning for entity disambiguation via knowledge-based risk analysis
Nafa, Y., Chen, Q., Hou, B., & Li, Z. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122342 · Cited by 4
Wadia, N. S., Dandi, Y., & Jordan, M. I. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad3194 · Cited by 4
“…sets (in many problems, Y ′ = 2 Y ) and λ ∈ R is a scalar parameter; given some user-specified γ, δ ∈ R , for an appropriate choice ˆ λ of λ , we require the risk R of T ˆ λ ( X ) to be bounded above with high probability: In [5], the authors provide a procedure by which to build T and choose ˆ λ .”
BackgroundMethodInfluentialSelf-citationUnderstanding the Under-Coverage Bias in Uncertainty EstimationPreprint
Bai, Y., Mei, S., Wang, H., & Xiong, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05515 · Cited by 4
“Uncertainty quantification for classification For classification problems, two main types of uncertainty quantification methods have been considered: outputting discrete prediction sets with guarantees of covering the true (discrete) label [70, 71, 38, 7, 12, 18, 17], or calibrating the predicted probabilities [51, 72…”
MethodSensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference ApproachPreprint
Jin, Y., Ren, Z., & Candès, E. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12161 · Cited by 4
Random Projection Ensemble Conformal Prediction for High-dimensional Classification
Qian, X., Wu, J., Wei, L., & Lin, Y. (2024) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2024.105225 · Cited by 4
coverforest: Conformal predictions with random forest in python
Meehinkong, P., & Ponnoprat, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132362 · Cited by 4
“Our future plans include implementing multilabel classification with finite-sample coverage guarantee via conformal risk control [25, 26, 27].”
Uncertainty-Aware Unimodal and Multimodal Learning for Evolving Hardware Trojan Detection
Vishwakarma, R., & Rezaei, A. (2025) · Journal of Hardware and Systems Security · doi:10.1007/s41635-025-00160-2 · Cited by 4
“In addition, we explore variations of conformal predictors as described in [68].”
A critical perspective on finite sample conformal prediction theory in medical applications
Kladny, K.-R., Schölkopf, B., Koch, L. M., Baumgartner, C. F., & Muehlebach, M. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103462 · Cited by 5
Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization
Farzaneh, A., Park, S., & Simeone, O. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3486238 · Cited by 3
“For example, risk-controlling prediction sets (RCPS) ensure control over the average risk, with the option to extend this control to other risk functionals [15].”
BackgroundLeterme, H., Fadili, J. M., & Starck, J. (2025) · Astronomy and Astrophysics · doi:10.1051/0004-6361/202451756 · Cited by 3
A Fast and Generalizable ML-Assisted Framework for Full-Wave Inverse Scattering
Huang, S., Yang, S., Wu, H., et al. (2025) · IEEE Transactions on Antennas and Propagation · doi:10.1109/tap.2025.3577780 · Cited by 3
“This means that no more than α of the true permittivity values will lie outside of T , except with probability δ [39].”
Leveraging conformal prediction to annotate enzyme function space with limited false positives
Ding, K., Luo, J., & Luo, Y. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012135 · Cited by 5
“The commonly used p-values include Hoeffding’s inequality-based p-values [25] and Hoeffding-Bentkus inequality-based p-values [26].”
Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking
Li, M., Zhang, X., Xin, J., Zhang, H., & Lin, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.23 · Cited by 5
Distribution-Free Federated Learning with Conformal PredictionsPreprint
Lu, C., & Kalpathy-Cramer, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07661 · Cited by 7
“Conformal inference was introduced by [5] as a general framework to incorporate a rigour notion of confidence into machine learning models and are flexible enough to have been adapted for tasks such as quantile regression, multi-label classification, and image segmentation[6, 7, 8].”
MethodAndéol, L., Fel, T., de Grancey, F., & Mossina, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06052 · Cited by 9
Wang, Y. H., & Politis, D. N. (2025) · Journal of nonparametric statistics · doi:10.1080/10485252.2025.2603398 · Cited by 2
“…inference problems, new findings are recently discovered regarding the impact of conditioning for prediction problems under a regression setup; see e.g. Rosset and Tibshirani (2020) and Bates et al. (2021), where new tools are also developed for better predictive inference under conditioning.”
Uncertainty Visualization via Low-Dimensional Posterior Projections
Yair, O., Nehme, E., & Michaeli, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01050 · Cited by 3
“Distribution-free methods such as risk-controlling prediction sets (RCPS) [3] treat all pixels jointly by aggregating and controlling their risk ( e.g., in image segmentation).”
Task-Driven Uncertainty Quantification in Inverse Problems via Conformal Prediction
Wen, J., Ahmad, R., & Schniter, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73027-6_11 · Cited by 3
Conjunction subspaces test for conformal and selective classification
He, Z., Li, Z., Dong, J., et al. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122037 · Cited by 1
SarRec: Statistically-guaranteed Augmented Retrieval for Recommendation
Zhang, T., Bisht, N., Li, Z.-H., Xu, G., & Wang, X.-Z. (2025) · International Conference on Information and Knowledge Management · doi:10.1145/3746252.3761054 · Cited by 4
Dai, M., Luo, W., & Li, T. (2025) · International Conference on Computers, Information Processing and Advanced Education (CIPAE) · doi:10.1109/cipae66821.2025.00090 · Cited by 1
“Its core innovation lies in the “blender” module, which dynamically combines global and local information to produce more accurate masks, particularly excelling in complex scenes and small object segmentation [22-23].”
InfluentialUncertainty Quantification and Risk Control for Multiple Sound Source Localization
Rozenfeld, V., & Laufer-Goldshtein, B. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3725584 · Cited by 1
“Indeed, Risk-Controlling Prediction Set (RCPS) [33], [34] offers a rigorous method for constructing prediction functions that achieves the property in (16).”
InfluentialUncertainty-aware large language models: a scoping review of conformal prediction methods
Ashby, A. E., Nguyen, K. A., & Luo, Z. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2026.0061 · Cited by 1
The Battle with Data: Realities of Bringing Artificial Intelligence to the Battlefield
Whitty, R. (2022) · Expeditions with MCUP · doi:10.36304/expwmcup.2022.09 · Cited by 1
Reliable Classifications with Guaranteed Confidence Using the Dempster-Shafer Theory of Evidence
Kempkes, M., Dunjko, V., van Nieuwenburg, E., & Spiegelberg, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70344-7_6 · Cited by 1
Subgroup-Specific Risk-Controlled Dose Estimation in Radiotherapy
Fischer, P., Willms, H., Schneider, M. J., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72117-5_65 · Cited by 1
Single Trajectory Conformal Prediction
Lee, B. W., & Matni, N. (2024) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc56724.2024.10886644 · Cited by 2
“An example of this approach is conformal prediction (CP), a family of assumption-light methods that post-process an arbitrary point predictor, like a black-box neural network, into a set predictor that provably covers the true label at test time with high probability [1]–[3].”
Risk-Based Thresholding for Reliable Anomaly Detection in Concentrated Solar Power Plants
Estievenart, Y., Patra, S., & Taieb, S. B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06129-4_7 · Cited by 1
CondSR: Conditional Shift-Robust Conformal Prediction for Graph Neural Networks
Akansha, S. (2026) · Soft Computing · doi:10.1007/s00500-025-11152-3 · Cited by 1
“…across various domains, including medicine (Lei and Candès 2021; Kompa et al. 2021; Dolezal et al. 2022; Chua et al. 2023), risk control (Bates et al. 2021; Angelopoulos et al. 2022), language modeling (Quach et al. 2023; Deutschmann et al. 2024), and time series analysis (Gibbs and…”
Predictive modeling of microbial data with interaction effectsPreprint
Stadler, M., Olayo-Alarcón, R., Bien, J., & Müller, C. L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.29.591596 · Cited by 1
“This can be more rigorously assessed using the concept of (conformal) prediction sets [85].”
Conformal Learning for Quantifying Uncertainty in Psychiatric Condition PredictionsPreprint
Stewart, C., Millar, A., Facelli, J. C., & Gouripeddi, R. (2025) · Research Square · doi:10.21203/rs.3.rs-6458806/v1 · Cited by 1
Optimized Certainty Equivalent Risk-Controlling Prediction Sets
Huang, J., Farzaneh, A., & Simeone, O. (2026) · arXiv · doi:10.48550/arxiv.2602.13660
“Other examples include the FNR in multi-label classification and image segmentation [8].”
InfluentialConsistent Accelerated Inference via Confident Adaptive TransformersPreprint
Schuster, T., Fisch, A., Jaakkola, T., & Barzilay, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08803 · Cited by 7
Artificial Intelligence Techniques for Fetal Medicine
Werner, H., Tonni, G., & Lopes, J. (2023) · Springer eBooks · doi:10.1007/978-3-031-14855-2_6
Angelopoulos, A. N. (2023) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad115
Self-citationAuthors' reply to the Discussion of ‘Estimating means of bounded random variables by betting’
Waudby-Smith, I., & Ramdas, A. (2023) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkad127
Conditional Shift-Robust Conformal Prediction for Graph Neural NetworkPreprint
Singh, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4862107
E-DM: Evaluating Diffusion Model by Conformal Prediction
Wang, C., Gu, Y., Liu, P., Zhao, S., & Zhou, A. S. K. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635216
“Conformal Prediction: Constructing statistically rigorous confidence intervals for ML model is a popular research field, known as trust computing [10, 13–15].”
BackgroundMethodInfluentialBlot, V., de Brionne, A. L., Sellami, I., Trassard, O., Beau, I., Sonigo, C., & Brunel, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73158-7_17 · Cited by 1
Uniform Mean Estimation for Monotonic ProcessesPreprint
Clerico, E., Flynn, H., & Rebeschini, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5139382
SymmPI: predictive inference for data with group symmetries
Dobriban, E., & Yu, M. (2025) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkaf022 · Cited by 6
Aggregating Image Segmentation Predictions with Probabilistic Risk Control Guarantees
Alvarez, J., & Román-Rangel, E. (2025) · Mathematics · doi:10.3390/math13111711
“Thus, we can use our framework and hybridize it via the uniform concentration bounds (UCBs) presented in [17].”
InfluentialAl-Arbo, Y., & Alkawaz, M. H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5749866
Legé, D., Balança, B., Kazimierska, A., et al. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108603
“In contrast, p 1 λ has to be calculated using the more conservative Hoeffding-Bentkus inequality p -value [24] due to its conditional component.”
Conformal Risk Control for Semantic Uncertainty Quantification in Computed Tomography
Teneggi, J., Stayman, J. W., & Sulam, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05185-1_5
Uniform mean estimation for monotonic processes
Clerico, E., Flynn, H., & Rebeschini, P. (2025) · Statistics & Probability Letters · doi:10.1016/j.spl.2025.110558
Huang, C., Wu, Y., & Wang, K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6131846
Metadata
- Authors
- Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M.
- Affiliations
- University of California, Berkeley (US)
- Stanford University (US)
- Venue
- Journal of the ACM
- Published
- 2021-09-30
- Volume
- 68
- Issue
- 6
- Pages
- 1-34
- ISSN
- 0004-5411 (ISSN-L)
- 1557-735X
- OpenAlex
- W3118640283
- Semantic Scholar
- e343f1517f038da090a78159363d14cdad5ddc35
- Cited by
- 144
- References
- 63
- Citations by year
- 2026: 48 · 2025: 27 · 2024: 25 · 2023: 21 · 2022: 8 · 2021: 14
- FWCI
- 5.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the ACM (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation Graduate Research Fellowship Program and a Berkeley Fellowship
- Army Research Office (W911NF-16-1-0368)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- prediction sets, uncertainty quantification, finite-sample guarantees, risk control, multi-label classification, hierarchical classification, image segmentation, protein structure prediction, metric learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
