CoChat
Sign Up
PreprintCited by 37Machine Learning and AlgorithmsMachine Learning and Data ClassificationStatistical Methods and Inference

Conformal Risk Control

Open access
Angelopoulos, Anastasios N., Bates, Stephen, Fisch, Adam, Lei, Lihua, Schuster, Tal·2022·arXiv
DOI 10.48550/arxiv.2208.02814arXiv 2208.02814
Open this paper in CoChat — free accountOpen PDF

Abstract

We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper extends conformal prediction to control the expected value of any monotone loss, calling the procedure conformal risk control. It provides finite-sample risk guarantees, a lower bound showing tightness up to \(O(1/n)\), extensions to distribution shift, quantile, multiple, adversarial, and U-statistic risks, and worked examples in vision and NLP.

Key findings

  • Conformal risk control chooses \(\hat\lambda\) from calibration data so that \(\mathbb{E}[L_{n+1}(\hat\lambda)] \le \alpha\) for non-increasing, right-continuous losses, recovering split conformal prediction when the loss is miscoverage.
  • The procedure is tight up to a factor \(2B/(n+1)\) under continuity assumptions; for conformal prediction this gives \(P(Y_{n+1}\notin C_{\hat\lambda}(X_{n+1})) \ge \alpha - 2/(n+1)\), slightly worse than the known \(1/(n+1)\) constant.
  • If losses are non-monotone, the algorithm does not control risk; the paper gives a counterexample where the expectation can be arbitrarily close to \(B\).
  • Empirical examples report mean risks near target \(\alpha\): tumor segmentation FNR \(0.0987 \pm 0.0114\) at \(\alpha=0.1\); MS COCO FNR \(0.0996 \pm 0.0052\) at \(\alpha=0.1\); hierarchical ImageNet graph distance \(0.0499 \pm 0.0011\) at \(\alpha=0.05\); Natural Questions F1 risk \(0.2996 \pm 0.0150\) at \(\alpha=0.3\).

Methods

  • Design: theoretical extension of split conformal prediction to monotone loss functions, with proofs of risk control, a lower bound, and extensions to distribution shift, quantile risk, multiple risks, adversarial risks, and U-statistics.
  • Data and sample: four worked examples—tumor segmentation using pooled Kvasir, Hyper-Kvasir, CVC-ColonDB, CVC-ClinicDB, and ETIS-Larib with PraNet, \(n=1000\), 781 validation points; MS COCO multilabel classification with TResNet, \(n=4000\), 1000 validation points; hierarchical ImageNet with ResNet152, \(n=30000\), 20000 validation points; Natural Questions open-domain QA with DPR Retriever-Reader, \(n=2500\), 1110 validation points.
  • Main analysis: for each task, choose \(\hat\lambda\) by the conformal risk control rule and evaluate risk and set size over 1000 independent random data splits.

Limitations

  • The paper states that the algorithm does not control risk for non-monotone losses; a monotonization procedure gives asymptotic control, but finite-sample control for that procedure is unknown.
  • The paper states that the factor of 2 in the lower bound may not be tight and that its tightness remains an open question.
  • The paper leaves full investigation of weighted conformal risk control with an estimated covariate shift for future research, and notes it is unclear whether more general quantile-based risks such as CVaR can be controlled unconditionally.
  • Evident from the text: the empirical validation is limited to four illustrative tasks with specific models, datasets, and chosen \(\alpha\) values, including \(\alpha=0.3\) chosen empirically for Natural Questions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2208.02814

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 11 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Findingreasoning
    § 4.1 Risk control under distributional shift · p. 10

    In the covariate shift setting with transductive learning, risk control can be achieved even when the weight w is unbounded.

    “In this case, we can achieve the risk control even when w is unbounded.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The conditional distribution of Y given X remains the same in the training and test domains.
    • The authors state this as an assumption: L_i ∈ [0,B] for all i.
  • Findingreasoning
    § 4.1 Risk control under distributional shift · p. 10

    Under distributional shift, if P_test is absolutely continuous with respect to P_train, the weighted objective (9) can be rewritten as (10).

    “Assuming that P test is absolutely continuous with respect to P train , the weighted objective (9) can be rewritten as”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: P_test is absolutely continuous with respect to P_train.
  • Findingreasoning
    § 4.1 Risk control under distributional shift · p. 10

    When the covariate shift weight w is known and bounded, applying the conformal risk control procedure on the weighted loss ̃L_{n+1}(λ)=w(X_{n+1},Y_{n+1})L_{n+1}(λ) yields a λ that satisfies the risk-control guarantee (10).

    “When w is known and bounded, we can apply our procedure on the loss function ˜ L n +1 ( λ ) = w ( X n +1 , Y n +1 ) L n +1 ( λ ), which is non-decreasing, bounded, and right-continuous in λ whenever L n +1 is. Thus, Theorem 1 guarantees that the resulting ˆ λ satisfies (10).”
    Quote in the paperSubstantial claim
  • Findingnone given
    § 4.3 Controlling multiple risks · p. 11

    Proposition 5: In the setting of Theorem 1, the multiple-risk control guarantee (12) is satisfied.

    “Proposition 5. In the setting of Theorem 1, (12) is satisfied.”
    Quote in the paperNot a substantial claim
  • Findingreasoning
    § 4.3 Controlling multiple risks · p. 11

    The worst-case risk over a family of loss functions indexed by γ can still be controlled.

    “Though the above worst-case risk is not an expectation, it can still be controlled.”
    Quote in the paperSubstantial claim
  • May combine two claims. 18
  • Classified as probably not a substantial claim. 7
  • The quote only partly states this claim. 9

  • May combine two claims.

    Interpretationnone given
    § 1.2 Related work · p. 2

    The algorithm presented herein differs from previous high-probability risk control work: it has a guarantee in expectation, and neither the algorithm nor its analysis share much technical similarity with these previous works.

    “the algorithm presented herein differs greatly: it has a guarantee in expectation, and neither the algorithm nor its analysis share much technical similarity with these previous works.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [17] [17] V. Vovk, 'Conditional validity of inductive conformal predictors,' in Proceedings of the Asian Conference on Machine Learning , vol. 25, 2012, pp. 475-490. - source
    • cites [18] [18] S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. I. Jordan, 'Distribution-free, risk-controlling prediction sets,' Journal of the ACM , vol. 68, no. 6, Sep. 2021. - source
    • cites [19] [19] A. N. Angelopoulos, S. Bates, E. J. Cand` es, M. I. Jordan, and L. Lei, 'Learn then Test: Calibrating predictive algorithms to achieve risk control,' arXiv preprint arXiv:2110.01052 , 2021. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.2 Quantile risk control · p. 11

    [39] generalizes [18] to control the quantile of a monotone loss function conditional on (X_i,Y_i)_{i=1}^n with probability 1-δ over the calibration dataset for any user-specified tolerance parameter δ.

    “[39] generalizes [18] to control the quantile of a monotone loss function conditional on ( X i , Y i ) n i =1 with probability 1 -δ over the calibration dataset for any user-specified tolerance parameter δ .”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [39] [39] J. C. Snell, T. P. Zollo, Z. Deng, T. Pitassi, and R. Zemel, 'Quantile risk control: A flexible framework for bounding the probability of high-loss predictions,' arXiv preprint arXiv:2212.13629 , 2022. - source
    • cites [18] [18] S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. I. Jordan, 'Distribution-free, risk-controlling prediction sets,' Journal of the ACM , vol. 68, no. 6, Sep. 2021. - source

      Section 2.2 and Appendix A establish that calibration can control a monotone risk with probability at least 1−δ over the calibration data. But Section 6 describes the paper’s main risk as the mean loss, while Appendix B gives a binary-loss tolerance-region case; the cited passages do not establish the claimed guarantee for the quantile of an arbitrary monotone loss. The paper’s Related Work section supports the characterization of its method as an expansion of tolerance regions. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • The quote only partly states this claim.

    Findingreasoning
    § Abstract · p. 1

    Under i.i.d. L_i ≥ 0 and no discontinuities, the conformal risk control procedure is tight up to a factor 2B/(n+1), satisfying E[L_{n+1}(λ̂)] ≥ α - 2B/(n+1).

    “Like conformal prediction, the conformal risk control procedure is tight up to an O (1 /n ) factor.”
    • Also stated in: § 2.2 A risk lower bound · p. 4 · plain
      “Next we show that the conformal risk control procedure is tight up to a factor 2 B/ ( n +1).”
    • Also stated in: § 2.2 A risk lower bound · p. 4 · plain
      “Theorem 2. In the setting of Theorem 1, further assume that the L i are i.i.d., L i ≥ 0 , and for any λ , P ( J ( L i , λ ) > 0) = 0 . Then”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The loss functions L_i are i.i.d.
    • The authors state this as an assumption: The loss functions L_i are nonnegative.
    • The authors state this as an assumption: For any λ, P(J(L_i,λ)>0)=0.

  • The quote only partly states this claim.

    Findingreasoning
    § 1.1 Algorithm and preview of main results · p. 2

    For exchangeable non-increasing right-continuous loss functions L_i, the conformal risk control algorithm choosing λ̂ by (4) satisfies the risk control guarantee E[L_{n+1}(λ̂)] ≤ α.

    “Our main result is that this algorithm satisfies (3).”
    • Also stated in: § 2.1 Risk control · p. 3 · plain
      “We first show that the proposed algorithm leads to risk control when the loss is monotone.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The calibration and test loss functions are exchangeable.
    • The authors state this as an assumption: The loss functions L_i are non-increasing in λ.
    • The authors state this as an assumption: The loss functions L_i are right-continuous.
    • The authors state this as an assumption: The loss functions L_i are bounded above by B < ∞.

  • May combine two claims. The quote only partly states this claim.

    Findingreasoning
    § Total variation bound · p. 10

    Proposition 3: Under the conditions in Theorem 1, the total variation bound holds for a sequence Z=(Z_1,...,Z_{n+1}); if further the assumptions of Theorem 2 hold, a second bound holds.

    “Proposition 3. Let Z = ( Z 1 , . . . , Z n +1 ) be a sequence of random variables. Then, under the conditions in Theorem 1,”
    • Also stated in: § Total variation bound · p. 11 · plain
      “If further the assumptions of Theorem 2 hold,”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.4 F1-score control in open-domain question answering · p. 9

    In Natural Questions open-domain QA, the mean and standard deviation of the risk over 1000 trials are 0.2996 and 0.0150, respectively, with α=0.3.

    “The mean and standard deviation of the risk over 1000 trials are 0.2996 and 0.0150, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 3.1 FNR control in tumor segmentation · p. 6

    In tumor segmentation, the mean and standard deviation of the risk over 1000 trials are 0.0987 and 0.0114, respectively, with α=0.1.

    “The mean and standard deviation of the risk over 1000 trials are 0.0987 and 0.0114, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 2.4 Controlling general loss functions · p. 5

    If the loss functions L_i are not monotone, the conformal risk control algorithm does not control the risk; for any ε there exists a non-monotone loss function such that the expected risk can be arbitrarily close to B.

    “We next show that the conformal risk control algorithm does not control the risk if the L i are not assumed to be monotone. In particular, (3) does not hold. We show this by example.”
    • Also stated in: § 2.4 Controlling general loss functions · p. 5 · plain
      “Notice that for any desired level α , the expectation in (3) can be arbitrarily close to B .”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingnone given
    § 4.5 U-risk control · p. 12

    Proposition 7: Assume L_S(λ) is non-increasing in λ, right-continuous, and satisfies the stated condition. Then the U-risk control guarantee (15) is achieved.

    “Proposition 7. Assume that L S ( λ ) is non-increasing in λ , right-continuous, and”
    • Also stated in: § 4.5 U-risk control · p. 12 · plain
      “Then (15) is achieved.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: L_S(λ) is non-increasing in λ, right-continuous, and satisfies the stated condition.

  • The quote only partly states this claim.

    Findingnone given
    § A Monotonizing non-monotone risks · p. 15

    Theorem A.1: Let L_i(λ) be right-continuous, i.i.d., bounded functions satisfying (5). Then the stated asymptotic risk-control result holds.

    “Theorem A.1. Let the L i ( λ ) be right-continuous, i.i.d., bounded (both above and below) functions satisfying (5) . Then,”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: L_i(λ) are right-continuous, i.i.d., bounded (both above and below) functions satisfying (5).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § A Monotonizing non-monotone risks · p. 15

    In the case of a monotone loss function, ̃λ↑ = ˆλ.

    “Note that in the case of a monotone loss function, ˜ λ ↑ = ˆ λ .”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § 3.3 Control of graph distance in hierarchical image classification · p. 8

    In hierarchical ImageNet classification, the mean and standard deviation of the risk over 1000 trials are 0.0499 and 0.0011, respectively, with α=0.05.

    “The mean and standard deviation of the risk over 1000 trials are 0.0499 and 0.0011, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 3.2 FNR control in multilabel classification · p. 6

    In MS COCO multilabel classification, the mean and standard deviation of the risk over 1000 trials are 0.0996 and 0.0052, respectively, with α=0.1.

    “The mean and standard deviation of the risk over 1000 trials are 0.0996 and 0.0052, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.4 Adversarial risks · p. 12

    Adversarial risks defined by perturbations can be recast as a conformal risk control problem by taking ̃L_i(λ)=sup_{γ∈Γ} L_i(λ;γ), and the following choice of λ leads to risk control.

    “This can be recast as a conformal risk control problem by taking ˜ L i ( λ ) = sup γ ∈ Γ L i ( λ ; γ ). Then, the following choice of λ leads to risk control:”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 5 Conclusion · p. 12

    The generalization of conformal prediction broadens its scope to new applications, as shown in Section 3.

    “This generalization of conformal prediction broadens its scope to new applications, as shown in Section 3.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § 4.1 Risk control under distributional shift · p. 10

    In transductive learning, if the conditional distribution of Y given X remains the same in the training and test domains, the distributional shift reduces to a covariate shift.

    “In the setting of transductive learning, X n +1 is available to the user. If the conditional distribution of Y given X remains the same in the training and test domains, the distributional shift reduces to a covariate shift and”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The conditional distribution of Y given X remains the same in the training and test domains.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § 3 Examples · p. 5

    The four loss functions used in the examples are non-binary, monotone losses bounded by 1.

    “All four loss functions are non-binary, monotone losses bounded by 1.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.4 F1-score control in open-domain question answering · p. 8

    The token-level F1 loss is monotone and upper bounded by 1, so it can be controlled by choosing λ̂ as in Section 2.1 to achieve the risk-control guarantee.

    “Since L F1 i , as defined in this way, is monotone and upper bounded by 1, it can be controlled by choosing ˆ λ as in Section 2.1 to achieve the following guarantee:”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3.3 Control of graph distance in hierarchical image classification · p. 7

    The hierarchical distance loss is monotone and can be controlled by choosing λ̂ as in (4) to achieve the risk-control guarantee.

    “Thus, it is a monotone loss function and can be controlled by choosing ˆ λ as in (4) to achieve the guarantee”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 2.4 Controlling general loss functions · p. 5

    The further algorithm in Appendix A provides an asymptotic risk-control guarantee for non-monotone loss functions, but is only powerful when the risk is near-monotone and reduces to standard conformal risk control when the loss is monotone.

    “We propose a further algorithm for picking λ in Appendix A that provides an asymptotic risk-control guarantee for non-monotone loss functions. However, this algorithm again is only powerful when the risk E [ L n +1 ( λ )] is near-monotone and reduces to the standard conformal risk control algorithm when the loss is monotone.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data
    § 3 Examples · p. 5

    The results over 1000 random data splits demonstrate that the procedure provides valid risk control (mean risk ≤ α) and gives useful outputs to the end-user.

    “Our results validate that the procedure bounds the risk as desired and gives useful outputs to the end-user.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 5 Conclusion · p. 12

    The mathematical tools developed in Section 2, Section 4, and the Appendix provide a new and more general language for studying conformal prediction along with new results about its validity.

    “The mathematical tools developed in Section 2, Section 4, and the Appendix may be of independent technical interest, since they provide a new and more general language for studying conformal prediction along with new results about its validity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given
    § 1.2 Related work · p. 2

    The guarantee available through conformal risk control strictly subsumes that of conformal prediction; it is generally impossible to recast risk control as coverage control.

    “The guarantee available through conformal risk control, (3), strictly subsumes that of conformal prediction; it is generally impossible to recast risk control as coverage control.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 2.4 Controlling general loss functions · p. 5

    Monotonizing all loss functions L_i and running conformal risk control guarantees (3), but this strategy is only powerful if the loss is near-monotone.

    “Simply 'monotonizing' all loss functions L i and running conformal risk control will guarantee (3), but this strategy will only be powerful if the loss is near-monotone.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 2.3 Conformal prediction reduces to risk control · p. 4

    The risk lower bound in Theorem 2 has a slightly worse constant than the usual conformal guarantee; it recovers the conformal lower bound P(Y_{n+1} ∉ C_{λ̂}(X_{n+1})) ≥ α - 1/(n+1) as P(Y_{n+1} ∉ C_{λ̂}(X_{n+1})) ≥ α - 2/(n+1), a factor of two worse.

    “However, the risk lower bound in Theorem 2 has a slightly worse constant than the usual conformal guarantee.”
    • Also stated in: § 2.3 Conformal prediction reduces to risk control · p. 4 · plain
      “However, the bound for the conformal case is better than the bound for the general case in Theorem 2 by a factor of two.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] [8] J. Lei, M. G'Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman, 'Distribution-free predictive inference for regression,' Journal of the American Statistical Association , vol. 113, no. 523, pp. 1094-1111, 2018. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § 3 Examples · p. 5

    The choices of C λ used in the examples are only for illustration; any nested family of sets will work.

    “We note that the choices of C λ used herein are only for the purposes of illustration ; any nested family of sets will work.”
    Quote in the paperNot a substantial claim
26 of 26

  • The authors state this as an assumption.

    The calibration and test loss functions are exchangeable.

    “Consider an exchangeable collection of non-increasing, random functions L i : Λ → ( -∞ , B ], i = 1 , . . . , n +1.”

  • The authors state this as an assumption.

    The loss functions L_i are non-increasing in λ.

    “We require the loss function to be non-increasing as a function of λ .”

  • The authors state this as an assumption.

    The loss functions L_i are right-continuous.

    “Assume that L i ( λ ) is non-increasing in λ , right-continuous, and”

  • The authors state this as an assumption.

    The loss functions L_i are bounded above by B < ∞.

    “we consider a loss function ℓ ( C λ ( x ) , y ) ∈ ( -∞ , B ] for some B &lt; ∞ .”

  • The authors state this as an assumption.

    The loss functions L_i are i.i.d.

    “further assume that the L i are i.i.d., L i ≥ 0 , and for any λ , P ( J ( L i , λ ) &gt; 0) = 0 .”

  • The authors state this as an assumption.

    The loss functions L_i are nonnegative.

    “further assume that the L i are i.i.d., L i ≥ 0 , and for any λ , P ( J ( L i , λ ) &gt; 0) = 0 .”

  • The authors state this as an assumption.

    For any λ, P(J(L_i,λ)>0)=0.

    “further assume that the L i are i.i.d., L i ≥ 0 , and for any λ , P ( J ( L i , λ ) &gt; 0) = 0 .”

  • The paper relies on this without support.

    The top several hundred candidate answers retrieved from the top 100 passages are sufficient to control all risks of interest.

    “we retrieve the top several hundred candidate answers, extracted from the top 100 passages (which is sufficient to control all risks of interest).”

  • The authors state this as an assumption.

    P_test is absolutely continuous with respect to P_train.

    “Assuming that P test is absolutely continuous with respect to P train , the weighted objective (9) can be rewritten as”

  • The authors state this as an assumption.

    The conditional distribution of Y given X remains the same in the training and test domains.

    “If the conditional distribution of Y given X remains the same in the training and test domains, the distributional shift reduces to a covariate shift and”

  • The authors state this as an assumption.

    L_i ∈ [0,B] for all i.

    “In particular, assuming L i ∈ [0 , B ], for any potential value x of the covariate, we define”

  • The authors state this as an assumption.

    L_S(λ) is non-increasing in λ, right-continuous, and satisfies the stated condition.

    “Proposition 7. Assume that L S ( λ ) is non-increasing in λ , right-continuous, and”

  • The authors state this as an assumption.

    L_i(λ) are right-continuous, i.i.d., bounded (both above and below) functions satisfying (5).

    “Theorem A.1. Let the L i ( λ ) be right-continuous, i.i.d., bounded (both above and below) functions satisfying (5) .”

  • The authors state this as an assumption.

    P(J(L_i,λ) > 0) = 0.

    “applying the assumption P ( J ( L i , λ ) &gt; 0) = 0,”
14 of 14

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Multilabel FNR example says it counts false positives

    Minor

    The multilabel example says it calculates false positives, although it describes its result as FNR control. Earlier, the paper defines the FNR loss using false negatives.

  • Weighted loss is described with the opposite monotonicity

    Minor

    The paper requires the loss to be non-increasing in λ, but says the weighted loss is non-decreasing whenever the original loss has that property.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. [4][4] A. Fisch, T. Schuster, T. S. Jaakkola, and R. Barzilay, 'Efficient conformal prediction via cascaded inference with expanded admission,' in International Conference on Learning Representations , 2021. -
  2. [5][5] V. Vovk, A. Gammerman, and C. Saunders, 'Machine-learning applications of algorithmic randomness,' in International Conference on Machine Learning , 1999, pp. 444-453. -
  3. [9][9] R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas, 'Conformal prediction under covariate shift,' in Advances in Neural Information Processing Systems , vol. 32, 2019, pp. 2530-2540. -
  4. [16][16] Y. Romano, M. Sesia, and E. Cand` es, 'Classification with valid and adaptive coverage,' in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., 2020, pp. 3581-3591. -
  5. [20][20] S. Park, O. Bastani, N. Matni, and I. Lee, 'PAC confidence sets for deep neural networks via calibrated prediction,' in International Conference on Learning Representations (ICLR) , 2020. -
  6. [38][38] E. Cand` es, L. Lei, and Z. Ren, 'Conformalized survival analysis,' Journal of the Royal Statistical Society Series B: Statistical Methodology , vol. 85, no. 1, pp. 24-45, 2023. -

Cited by

Sorted by citations

Related

Authors
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T.
Affiliations
  • University of California, Berkeley (US)
  • Massachusetts Institute of Technology (US)
  • Stanford University (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2208.02814
arXiv
2208.02814
OpenAlex
W4290055968
Semantic Scholar
af2fba1911f0d0977c731e3918c51428e08741da
Cited by
37
References
32
Citations by year
2026: 2 · 2025: 10 · 2024: 16 · 2023: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
Keywords
conformal prediction, risk control, split conformal prediction, coverage guarantees, distribution shift, U-statistics, false negative rate, graph distance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-01.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-01.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review