CoChat
Sign Up
PreprintCited by 1Risk and Portfolio OptimizationAdversarial Robustness in Machine LearningMachine Learning and Algorithms

Conformal Risk Training: End-to-End Optimization of Conformal Risk Control

Open access
Yeh, Christopher, Christianson, Nicolas, Wierman, Adam, Yue, Yisong·2025·arXiv
DOI 10.48550/arxiv.2510.08748arXiv 2510.08748
Open this paper in CoChat — free accountOpen PDF

Abstract

While deep learning models often achieve high predictive accuracy, their predictions typically do not come with any provable guarantees on risk or reliability, which are critical for deployment in high-stakes applications. The framework of conformal risk control (CRC) provides a distribution-free, finite-sample method for controlling the expected value of any bounded monotone loss function and can be conveniently applied post-hoc to any pre-trained deep learning model. However, many real-world applications are sensitive to tail risks, as opposed to just expected loss. In this work, we develop a method for controlling the general class of Optimized Certainty-Equivalent (OCE) risks, a broad class of risk measures which includes as special cases the expected loss (generalizing the original CRC method) and common tail risks like the conditional value-at-risk (CVaR). Furthermore, standard post-hoc CRC can degrade average-case performance due to its lack of feedback to the model. To address this, we introduce "conformal risk training," an end-to-end approach that differentiates through conformal OCE risk control during model training or fine-tuning. Our method achieves provable risk guarantees while demonstrating significantly improved average-case performance over post-hoc approaches on applications to controlling classifiers' false negative rate and controlling financial risk in battery storage operation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2510.08748

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • OCE-CRC guarantees that the OCE risk remains below a target level α only on average over the possible held-out datasets used for calibration.

    “The recently introduced OCE conformal risk control (OCE-CRC) [1] guarantees that the OCE risk remains below a target level α . However, this guarantee holds only on average over the possible held-out datasets used for calibration.”

    From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [1]

  • OCE-CRC can fail to produce prediction sets that satisfy the risk constraint for a high fraction of calibration datasets.

    “this implies that OCE-CRC can fail to produce prediction sets that satisfy the risk constraint for a high fraction of calibration datasets, leading to unreliable decisions in practice.”

    From Optimized Certainty Equivalent Risk-Controlling Prediction Sets (2026) · cites this paper as [1]

    Section 2 (p. 3) describes CRC as controlling marginal loss, and the Conclusion (p. 10) describes conformal OCE risk control as providing a risk guarantee under its assumptions. Neither passage, nor the passages found by searching for conditional risk and calibration-dataset failures, says that the risk constraint fails for a high fraction of calibration datasets. A marginal guarantee alone does not establish that frequency. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • A tutorial on conformal prediction

    Shafer, G., & Vovk, V. (2007) · arXiv · doi:10.48550/arxiv.0706.3188 · Cited by 655 · cited 1× · see citing works

    “Within the class of bounds on expected losses, conformal prediction (CP) [49, 45, 4] bounds expected miscoverage loss for set-valued predictions, whereas conformal risk control (CRC) [6] generalizes CP to bound the expectation of any bounded monotone loss function.”

Flagged first, then by citations

Research works printed but not linked: 49

  1. [1][1] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable Convex Optimization Layers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d. AlchéBuc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://papers.nips.cc/paper\_files/ paper/2019/hash/9ce3c52fc54362e22053399d3181c638-Abstract.html . -
  2. [2][2] A. Agrawal, S. Barratt, S. Boyd, and W. M. Moursi. Differentiating through a cone program. Journal of Applied and Numerical Optimization , 1(2):107-115, Aug. 2019. doi: 10.23952/jano. 1.2019.2.02. URL https://jano.biemdas.com/archives/931 . - · DOI
  3. [3][3] B. Amos and J. Z. Kolter. OptNet: Differentiable Optimization as a Layer in Neural Networks. In Proceedings of the 34th International Conference on Machine Learning , pages 136-145. PMLR, July 2017. URL https://proceedings.mlr.press/v70/amos17a.html . ISSN: 2640-3498. -
  4. [4][4] A. N. Angelopoulos and S. Bates. Conformal Prediction: A Gentle Introduction. Foundations and Trends® in Machine Learning , 16(4):494-591, 2023. ISSN 1935-8237. doi: 10.1561/ 2200000101. URL http://dx.doi.org/10.1561/2200000101 . - · DOI
  5. [5][5] A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. Technical report, Apr. 2022. URL http://arxiv.org/abs/2110.01052 . arXiv:2110.01052 [cs, stat] type: article. - · arXiv
  6. [6][6] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster. Conformal Risk Control, Apr. 2023. URL http://arxiv.org/abs/2208.02814 . arXiv:2208.02814 [cs, math, stat]. - · arXiv
  7. [7][7] P. Artzner, F. Delbaen, J. Eber, and D. Heath. Coherent Measures of Risk. Mathematical Finance , 9(3):203-228, July 1999. ISSN 0960-1627, 1467-9965. doi: 10.1111/1467-9965.00068. URL https://onlinelibrary.wiley.com/doi/10.1111/1467-9965.00068 . - · DOI
  8. [8][8] S. Barratt. On the Differentiability of the Solution to Convex Optimization Problems, Nov. 2019. URL http://arxiv.org/abs/1804.05098 . arXiv:1804.05098 [math]. - · arXiv
  9. [9][9] S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan. Distribution-free, Risk-controlling Prediction Sets. Journal of the ACM , 68(6):43:1-43:34, Sept. 2021. ISSN 0004-5411. doi: 10.1145/3478535. URL https://doi.org/10.1145/3478535 . - · DOI · Paper page
  10. [10][10] A. Ben-Tal and M. Teboulle. Expected Utility, Penalty Functions, and Duality in Stochastic Nonlinear Programming. Management Science , 32(11):1445-1466, Nov. 1986. ISSN 00251909, 1526-5501. doi: 10.1287/mnsc.32.11.1445. URL https://pubsonline.informs. org/doi/10.1287/mnsc.32.11.1445 . - · DOI
  11. [11][11] A. Ben-Tal and M. Teboulle. An Old-New Concept of Convex Risk Measures: The Optimized Certainty Equivalent. Mathematical Finance , 17(3):449-476, 2007. ISSN 1467-9965. doi: 10.1111/j.1467-9965.2007.00311.x. URL https:// onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9965.2007.00311.x . \_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9965.2007.00311.x. - · DOI · Paper page
  12. [12][12] J. Bernal, J. Sánchez, and F. Vilariño. Towards automatic polyp detection with a polyp appearance model. Pattern Recognition , 45(9):3166-3182, Sept. 2012. ISSN 0031-3203. doi: 10.1016/j.patcog.2012.03.002. URL https://www.sciencedirect.com/science/ article/pii/S0031320312001185 . - · DOI
  13. [13][13] J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilariño. WMDOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics , 43:99-111, July 2015. ISSN 0895-6111. doi: 10.1016/j.compmedimag.2015.02.007. URL https://www.sciencedirect. com/science/article/pii/S0895611115000567 . - · DOI · Paper page
  14. [15][15] D. Bessis and F. Clarke. Partial subdifferentials, derivates and Rademacher's Theorem. Transactions of the American Mathematical Society , 351(7):2899-2926, 1999. ISSN 0002-9947, 1088-6850. doi: 10.1090/S0002-9947-99-02203-5. URL https://www.ams.org/tran/ 1999-351-07/S0002-9947-99-02203-5/ . - · DOI
  15. [16][16] C. Y.-C. Chen, J. Shen, Z. Deng, and L. Lei. Conformal Tail Risk Control for Large Language Model Alignment, Feb. 2025. URL http://arxiv.org/abs/2502.20285 . arXiv:2502.20285 [cs]. - · arXiv
  16. [17][17] A. H. Correia, F. V . Massoli, C. Louizos, and A. Behboodi. An Information Theoretic Perspective on Conformal Prediction. Advances in Neural Information Processing Systems , 37:101000101041, Dec. 2024. URL https://proceedings.neurips.cc/paper\_files/paper/ 2024/hash/b6fa3ed9624c184bd73e435123bd576a-Abstract-Conference.html . -
  17. [18][18] S. Cortes-Gomez, C. M. Patiño, Y. Byun, S. Wu, E. Horvitz, and B. Wilder. Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification. Oct. 2024. URL https://openreview.net/forum?id=iOMnn1hSBO . -
  18. [19][19] P. L. Donti, B. Amos, and J. Z. Kolter. Task-based End-to-end Model Learning in Stochastic Optimization. In Advances in Neural Information Processing Systems , volume 30, Long Beach, CA, USA, Dec. 2017. Curran Associates, Inc. URL http://papers.nips.cc/paper/ 7132-task-based-end-to-end-model-learning-in-stochastic-optimization . -
  19. [20][20] J. C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49 (3), June 2021. ISSN 0090-5364. doi: 10.1214/20-AOS2004. URL https: //projecteuclid.org/journals/annals-of-statistics/volume-49/issue-3/ Learning-models-with-uniform-performance-via-distributionally-robust-optimization/ 10.1214/20-AOS2004.full . - · DOI
  20. [21][21] B.-S. Einbinder, Y. Romano, M. Sesia, and Y. Zhou. Training Uncertainty-Aware Classifiers with Conformalized Deep Learning. In Advances in Neural Information Processing Systems , New Orleans, LA, USA, Nov. 2022. doi: 10.48550/arXiv.2205.05878. URL http://arxiv. org/abs/2205.05878 . arXiv:2205.05878 [cs, stat]. - · DOI
  21. [22][22] A. N. Elmachtoub and P. Grigas. Smart 'Predict, then Optimize'. Management Science , 68(1):9-26, Jan. 2022. ISSN 0025-1909. doi: 10.1287/mnsc.2020.3922. URL https:// pubsonline.informs.org/doi/10.1287/mnsc.2020.3922 . Publisher: INFORMS. - · DOI
  22. [23][23] D.-P. Fan, G.-P. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao. PraNet: Parallel Reverse Attention Network for Polyp Segmentation, July 2020. URL http://arxiv.org/abs/2006. 11392 . arXiv:2006.11392 [eess]. - · arXiv
  23. [24][24] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr. Res2Net: A New Multi-Scale Backbone Architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(2):652-662, Feb. 2021. ISSN 1939-3539. doi: 10.1109/TPAMI.2019.2938758. URL https://ieeexplore.ieee.org/document/8821313 . - · DOI
  24. [25][25] K. He, D. Adam, S. Han-Oh, and A. Liu. Training-Aware Risk Control for Intensity Modulated Radiation Therapies Quality Assurance with Conformal Prediction, Jan. 2025. URL http: //arxiv.org/abs/2501.08963 . arXiv:2501.08963 [cs]. - · arXiv
  25. [26][26] L. J. Hong. Estimating Quantile Sensitivities. Operations Research , 57(1):118-130, Feb. 2009. ISSN 0030-364X. doi: 10.1287/opre.1080.0531. URL https://pubsonline.informs.org/ doi/10.1287/opre.1080.0531 . Publisher: INFORMS. - · DOI
  26. [27][27] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37, pages 448-456, Lille, France, July 2015. PMLR. URL http://arxiv.org/abs/1502.03167 . Series Title: Proceedings of Machine Learning Research. - · arXiv
  27. [28][28] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. de Lange, D. Johansen, and H. D. Johansen. Kvasir-SEG: A Segmented Polyp Dataset. In Y. M. Ro, W.-H. Cheng, J. Kim, W.-T. Chu, P. Cui, J.-W. Choi, M.-C. Hu, and W. De Neve, editors, MultiMedia Modeling , pages 451-462, Cham, 2020. Springer International Publishing. ISBN 978-3-030-37734-2. doi: 10.1007/978-3-030-37734-2\_37. - · DOI
  28. [29][29] Z. Jiang, A. Liu, and B. V. Durme. Conformal Linguistic Calibration: Trading-off between Factuality and Specificity, Feb. 2025. URL http://arxiv.org/abs/2502.19110 . arXiv:2502.19110 [cs]. - · arXiv
  29. [30][30] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , San Diego, CA, USA, May 2015. URL http: //arxiv.org/abs/1412.6980 . -
  30. [31][31] S. Kiyani, G. Pappas, A. Roth, and H. Hassani. Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents, Feb. 2025. URL http://arxiv.org/abs/2502.02561 . arXiv:2502.02561 [cs]. - · arXiv
  31. [32][32] P. Krokhmal, T. Uryasev, and J. Palmquist. Portfolio optimization with conditional value-at-risk objective and constraints. The Journal of Risk , 4(2): 43-68, Mar. 2001. ISSN 14651211. doi: 10.21314/JOR.2002.057. URL http://www.risk.net/journal-of-risk/technical-paper/2161203/ portfolio-optimization-conditional-value-risk-objective-constraints . - · DOI
  32. [33][33] J. Lee, S. Park, and J. Shin. Learning Bounds for Risk-sensitive Learning. In Advances in Neural Information Processing Systems , volume 33, pages 13867-13879. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 9f60ab2b55468f104055b16df8f69e81-Abstract.html . -
  33. [34][34] J. Lekeufack, A. N. Angelopoulos, A. Bajcsy, M. I. Jordan, and J. Malik. Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions, May 2024. URL http: //arxiv.org/abs/2310.05921 . arXiv:2310.05921. - · arXiv
  34. [35][35] L. Leqi, A. Huang, Z. Lipton, and K. Azizzadenesheli. Supervised Learning with General Risk Functionals. In Proceedings of the 39th International Conference on Machine Learning , pages 12570-12592. PMLR, June 2022. URL https://proceedings.mlr.press/v162/ leqi22a.html . ISSN: 2640-3498. -
  35. [36][36] A. N. Madavan, N. Dahlin, S. Bose, and L. Tong. Risk-Based Hosting Capacity Analysis in Distribution Systems. IEEE Transactions on Power Systems , 39(1):355-365, Jan. 2024. ISSN 1558-0679. doi: 10.1109/TPWRS.2023.3238846. URL https://ieeexplore.ieee.org/ document/10040563 . Conference Name: IEEE Transactions on Power Systems. - · DOI
  36. [37][37] J. Mandi, J. Kotary, S. Berden, M. Mulamba, V. Bucarey, T. Guns, and F. Fioretto. DecisionFocused Learning: Foundations, State of the Art, Benchmark and Future Opportunities. Journal of Artificial Intelligence Research , 80:1623-1701, Aug. 2024. ISSN 1076-9757. doi: 10.1613/ jair.1.15320. URL https://www.jair.org/index.php/jair/article/view/15320 . -
  37. [38][38] M. Ndrio, A. N. Madavan, and S. Bose. Pricing Conditional Value at Risk-Sensitive Economic Dispatch. In 2021 IEEE Power & Energy Society General Meeting (PESGM) , pages 01-05, Washington, DC, USA, July 2021. IEEE. ISBN 978-1-66540-507-2. doi: 10.1109/PESGM46819.2021.9637845. URL https://ieeexplore.ieee.org/document/ 9637845/ . - · DOI
  38. [39][39] S. Noorani, O. Romero, N. D. Fabbro, H. Hassani, and G. J. Pappas. Conformal Risk Minimization with Variance Reduction, Feb. 2025. URL http://arxiv.org/abs/2411.01696 . arXiv:2411.01696 [cs]. - · arXiv
  39. [40][40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , number 721, pages 8026-8037. Curran Associates Inc., Red Hook, NY, USA, Dec. 2019. -
  40. [41][41] R. T. Rockafellar and S. Uryasev. Optimization of conditional value-at-risk. The Journal of Risk , 2(3):21-41, 2000. ISSN 14651211. doi: 10.21314/JOR.2000. 038. URL http://www.risk.net/journal-of-risk/technical-paper/2161159/ optimization-conditional-value-risk . - · DOI
  41. [42][42] R. T. Rockafellar and S. Uryasev. Conditional value-at-risk for general loss distributions. Journal of Banking & Finance , 26(7):1443-1471, July 2002. ISSN 0378-4266. doi: 10.1016/ S0378-4266(02)00271-6. URL https://www.sciencedirect.com/science/article/ pii/S0378426602002716 . -
  42. [43][43] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015 , Lecture Notes in Computer Science, pages 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-4. doi: 10.1007/978-3-319-24574-4\_28. - · DOI · Paper page
  43. [44][44] U. Sadana, A. Chenreddy, E. Delage, A. Forel, E. Frejinger, and T. Vidal. A survey of contextual optimization methods for decision-making under uncertainty. European Journal of Operational Research , Mar. 2024. ISSN 0377-2217. doi: 10.1016/j.ejor.2024.03.020. URL https://www.sciencedirect.com/science/article/pii/S0377221724002200 . - · DOI
  44. [46][46] J. Silva, A. Histace, O. Romain, X. Dray, and B. Granado. Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. International Journal of Computer Assisted Radiology and Surgery , 9(2):283-293, Mar. 2014. ISSN 1861-6429. doi: 10.1007/s11548-013-0926-3. URL https://doi.org/10.1007/s11548-013-0926-3 . - · DOI
  45. [47][47] D. Stutz, Krishnamurthy, Dvijotham, A. T. Cemgil, and A. Doucet. Learning Optimal Conformal Classifiers, May 2022. URL http://arxiv.org/abs/2110.09192 . arXiv:2110.09192 [cs, stat]. - · arXiv
  46. [48][48] J. Teneggi, M. Tivnan, W. Stayman, and J. Sulam. How to Trust Your Diffusion Model: A Convex Optimization Approach to Conformal Risk Control. In Proceedings of the 40th International Conference on Machine Learning , pages 33940-33960. PMLR, July 2023. URL https://proceedings.mlr.press/v202/teneggi23a.html . ISSN: 2640-3498. -
  47. [49][49] V. Vovk, A. Gammerman, and G. Shafer. Algorithmic Learning in a Random World . SpringerVerlag, New York, 1st edition, 2005. ISBN 978-0-387-00152-4. doi: 10.1007/b106715. URL http://link.springer.com/10.1007/b106715 . - · DOI · Paper page
  48. [50][50] W. Xu, X. Gao, and X. He. Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents. In Proceedings of the 40th International Conference on Machine Learning , pages 38400-38427. PMLR, July 2023. URL https://proceedings. mlr.press/v202/xu23d.html . ISSN: 2640-3498. -
  49. [51][51] C. Yeh, N. Christianson, A. Wu, A. Wierman, and Y. Yue. End-to-End Conformal Calibration for Optimization Under Uncertainty, Sept. 2024. URL http://arxiv.org/abs/2409.20534 . arXiv:2409.20534 [cs, math]. <!-- image --> (c) α = 0 . 1 Figure 3: Predictions on 8 randomly selected images from the test set for the tumor image segmentation problem (Section 5.1). Black and white pixels indicate correct outputs. False positives are shaded teal; false negatives are shaded red. · arXiv

Cited by

Sorted by citations

Authors
Yeh, C., Christianson, N., Wierman, A., & Yue, Y.
Affiliations
  • California Institute of Technology (US)
  • Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
78067-78103
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2510.08748
arXiv
2510.08748
OpenAlex
W4417115480
Cited by
0
References
1
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Risk and Portfolio Optimization
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
conformal risk control, conditional value at risk, tail risk, risk measures, end-to-end optimization, financial risk control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review