CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Conformal Tail Risk Control for Large Language Model Alignment

Chen, Catherine Yu-Chi, Shen, Jingyan, Deng, Zhun, Lei, Lihua·2025·arXiv
DOI 10.48550/arxiv.2502.20285arXiv 2502.20285
Open this paper in CoChat — free account

Abstract

Recent developments in large language models (LLMs) have led to their widespread usage for various tasks. The prevalence of LLMs in society implores the assurance on the reliability of their performance. In particular, risk-sensitive applications demand meticulous attention to unexpectedly poor outcomes, i.e., tail events, for instance, toxic answers, humiliating language, and offensive outputs. Due to the costly nature of acquiring human annotations, general-purpose scoring models have been created to automate the process of quantifying these tail events. This phenomenon introduces potential human-machine misalignment between the respective scoring mechanisms. In this work, we present a lightweight calibration framework for blackbox models that ensures the alignment of humans and machines with provable guarantees. Our framework provides a rigorous approach to controlling any distortion risk measure that is characterized by a weighted average of quantiles of the loss incurred by the LLM with high confidence. The theoretical foundation of our method relies on the connection between conformal risk control and a traditional family of statistics, i.e., L-statistics. To demonstrate the utility of our framework, we conduct comprehensive experiments that address the issue of human-machine misalignment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.20285

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Examples of distortion risk measures: Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR).
Figure 1 Examples of distortion risk measures: Value-at-Risk (VaR) and Conditional Value-at-Risk (CVaR).CC-BY · From the PDF
Figure 2 Illustration of ˆ λ . We choose ˆ λ as the last λ such that the (asymptotic) upper confidence bound ˆ R + ψ ( λ ) falls below α .
Figure 2 Illustration of ˆ λ . We choose ˆ λ as the last λ such that the (asymptotic) upper confidence bound ˆ R + ψ ( λ ) falls below α .CC-BY · From the PDF
Figure 3 Illustration of the process to generate C ( x ) , C λ ( x ) , and r λ ( x ) . We sample N responses y i , i = 1 , . . . , N from the LLM p ( y | x ) . Each response is associated with a machine disutility score r m ( y i ( x )) , and a human-rated disutility score r ( y i ( x )) . To construct C λ j ( x ) , we keep the responses such that the machine toxicity score satisfies r m ( y i ( x )) ≤ λ j for each λ j ∈ Λ . Finally, we compute the induced score r λ ( x ) by taking the maximum human disutility score of each C λ j ( x ) .
Figure 3 Illustration of the process to generate C ( x ) , C λ ( x ) , and r λ ( x ) . We sample N responses y i , i = 1 , . . . , N from the LLM p ( y | x ) . Each response is associated with a machine disutility score r m ( y i ( x )) , and a human-rated disutility score r ( y i ( x )) . To construct C λ j ( x ) , we keep the responses such that the machine toxicity score satisfies r m ( y i ( x )) ≤ λ j for each λ j ∈ Λ . Finally, we compute the induced score r λ ( x ) by taking the maximum human disutility score of each C λ j ( x ) .CC-BY · From the PDF
Figure 4 Average sampling cost vs. α (row 1), and realized CVaR vs. α (row 2) for Spearman correlation between human and machine toxicity scores at ρ = 0 . 57 evaluated on held-out dataset . The confidence band is computed by taking the mean estimate plus/minus one standard error estimated from the results across independent experiments. Each subplot in the respective rows illustrates a different setting of β ∈ { 0 . 5 , 0 . 75 , 0 . 9 } . From the panels in the first row, we observe that our method, CDRC-L (orange), is an improvement to CDRC-DKW (green) and CDRC-BJ (blue), as it is able to achieve risk control (shown in the black dotted line) while being less conservative than both baseline methods. Evident from the panels in the second row, our method is more efficient in generating a risk-controlled LLM response than DKW or BJ.
Figure 4 Average sampling cost vs. α (row 1), and realized CVaR vs. α (row 2) for Spearman correlation between human and machine toxicity scores at ρ = 0 . 57 evaluated on held-out dataset . The confidence band is computed by taking the mean estimate plus/minus one standard error estimated from the results across independent experiments. Each subplot in the respective rows illustrates a different setting of β ∈ { 0 . 5 , 0 . 75 , 0 . 9 } . From the panels in the first row, we observe that our method, CDRC-L (orange), is an improvement to CDRC-DKW (green) and CDRC-BJ (blue), as it is able to achieve risk control (shown in the black dotted line) while being less conservative than both baseline methods. Evident from the panels in the second row, our method is more efficient in generating a risk-controlled LLM response than DKW or BJ.CC-BY · From the PDF
Figure 5 Average sampling cost vs. α (row 1), and realized VaR vs. α (row 2) for Spearman correlation between human and machine toxicity scores at ρ = 0 . 57 evaluated on held-out dataset . The confidence band is computed by taking the mean estimate plus/minus one standard error estimated from the results across the independent experiments. Each panel in the respective rows illustrates a different setting of β ∈ { 0 . 5 , 0 . 75 , 0 . 9 } . CDRC-L (orange), is an improvement to CDRC-DKW (green) and CDRC-BJ (blue), as it is able to achieve risk control within the margin of error (shown in the black dotted line) while being more efficient than both baseline methods.
Figure 5 Average sampling cost vs. α (row 1), and realized VaR vs. α (row 2) for Spearman correlation between human and machine toxicity scores at ρ = 0 . 57 evaluated on held-out dataset . The confidence band is computed by taking the mean estimate plus/minus one standard error estimated from the results across the independent experiments. Each panel in the respective rows illustrates a different setting of β ∈ { 0 . 5 , 0 . 75 , 0 . 9 } . CDRC-L (orange), is an improvement to CDRC-DKW (green) and CDRC-BJ (blue), as it is able to achieve risk control within the margin of error (shown in the black dotted line) while being more efficient than both baseline methods.CC-BY · From the PDF
Figure 6 Spearman correlation vs. average sampling cost for CVaR .
Figure 6 Spearman correlation vs. average sampling cost for CVaR .CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Figure 8 Distribution of toxicity scores assigned by humans (i.e., original Detoxify model) and machine (i.e., fine-tuned Detoxify model) for responses generated by the LLaMA 2 7B model across all prompts. The main plot shows the density of scores, highlighting a long-tailed distribution in both human and machine evaluations. The zoomed-in plot illustrates that human scores have a heavier tail than machine scores.
Figure 8 Distribution of toxicity scores assigned by humans (i.e., original Detoxify model) and machine (i.e., fine-tuned Detoxify model) for responses generated by the LLaMA 2 7B model across all prompts. The main plot shows the density of scores, highlighting a long-tailed distribution in both human and machine evaluations. The zoomed-in plot illustrates that human scores have a heavier tail than machine scores.CC-BY · From the PDF
Figure 9 Average cost and realized CVaR on held-out dataset for ρ = 0 . 68 .
Figure 9 Average cost and realized CVaR on held-out dataset for ρ = 0 . 68 .CC-BY · From the PDF
Figure 10 Average cost and realized CVaR on held-out dataset for ρ = 0 . 78 .
Figure 10 Average cost and realized CVaR on held-out dataset for ρ = 0 . 78 .CC-BY · From the PDF
Figure 11 Average cost and realized VaR on held-out dataset for ρ = 0 . 68 .
Figure 11 Average cost and realized VaR on held-out dataset for ρ = 0 . 68 .CC-BY · From the PDF
Figure 12 Average cost and realized VaR on held-out dataset for ρ = 0 . 78 .
Figure 12 Average cost and realized VaR on held-out dataset for ρ = 0 . 78 .CC-BY · From the PDF
Figure 13 Spearman correlation vs. average sampling cost for VaR .
Figure 13 Spearman correlation vs. average sampling cost for VaR .CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 18

  1. A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. CoRR, abs/2110.01052, 2021. · arXiv
  2. A. Balbás, J. Garrido, and S. Mayoral. Properties of distortion risk measures. Methodology and Computing in Applied Probability, 11(3):385-399, 2009.
  3. S. Bates, E. Candès, L. Lei, Y. Romano, and M. Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149-178, 2023.
  4. E. Candès, L. Lei, and Z. Ren. Conformalized survival analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(1):24-45, 2023.
  5. S. Casper, S. Zhang, A. Holtzman, J. Kreutzer, C. Zhu, M. Ranzato, and N. Stiennon. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. · arXiv
  6. M. Csörgő and P. Révész. Strong approximations of the quantile process. The Annals of Statistics, pages 882-894, 1978.
  7. J. C. Gardiner and P. Kumar Sen. Asymptotic normality of a variance estimator of a linear combination of a function of order statistics. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 50(2):205-221, 1979.
  8. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020. · arXiv
  9. J. Kiefer. Deviations between the sample quantile process and the sample df. Non-parametric Techniques in Statistical Inference, pages 299-319, 1970.
  10. L. Lei and E. J. Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911- 938, 2021.
  11. A. Moscovich and B. Nadler. Fast calculation of boundary crossing probabilities for poisson processes. Statistics & Probability Letters, 123:177-182, 2017.
  12. F. Mosteller. On some useful" inefficient" statistics. Annals of Mathematical Statisitcs, 17:377-408, 1946.
  13. S. M. Stigler. Linear functions of order statistics with smooth weight functions. The Annals of Statistics, pages 676-693, 1974.
  14. R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
  15. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  16. J. W. Tukey. The future of data analysis. In Breakthroughs in Statistics: Methodology and Distribution, pages 408-452. Springer, 1962.
  17. A. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
  18. D. M. Ziegler, N. Stiennon, J. Wu, T. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. · arXiv

Cited by

Sorted by citations

  • Calibrating Conservatism for Scalable Oversight

    Overman, W., & Bayati, M. (2026) · arXiv · doi:10.48550/arxiv.2605.28807

    “Recent work has begun applying conformal prediction meth-ods to LLM safety, controlling tail risks (Chen et al., 2025), improving factuality (Cherian et al., 2024), and mediating between helpfulness and harmlessness (Overman & Bayati, 2025), but these approaches predominantly address single-turn…”

See all 1

Metadata

Authors
Chen, C. Y.-C., Shen, J., Deng, Z., & Lei, L.
Venue
arXiv
Publisher
arXiv
Published
2025
DOI
10.48550/arxiv.2502.20285
arXiv
2502.20285
References
1
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review