CoChat
Sign Up
PreprintCited by 0Privacy-Preserving Technologies in DataMachine Learning in HealthcareArtificial Intelligence in Healthcare and Education

When Average Calibration Fails: Site-Conditional Federated Conformal Risk Control

Open access
Shahid, Nafis Fuad·2026·arXiv
DOI 10.48550/arxiv.2606.20115arXiv 2606.20115
Open this paper in CoChat — free accountOpen PDF

Abstract

Conformal risk control (CRC) provides distribution-free segmentation guarantees by calibrating a prediction-set threshold on held-out data. In federated deployments, the standard approach pools calibration scores into a single threshold. We quantify, on real multi-institutional brain tumor data (FeTS-2022, 1,251 subjects, 20 institutions), a critical failure: naive pooled CRC protects the average hospital but violates coverage at 40% of individual institutions, with the worst site exceeding the target false-negative rate by 7.8 percentage points. We trace this failure to a hidden design choice: the aggregation weights implicitly determine whose coverage is protected. Sample-size weighting optimizes patient-level validity but can sacrifice institution-level reliability; equal-site weighting improves institution-level reliability on this benchmark at comparable efficiency, using only a single scalar per site. We propose risk-curve shrinkage as a principled mechanism: each site transmits its empirical risk curve (G scalars) and a single hyperparameter n0 smoothly interpolates between site-specific local calibration and sample-size-weighted pooled calibration. Leave-one-site-out sensitivity analysis identifies n0=19, achieving 2.7/20 violations at 2.0x stretch. Direct Lagrangian budget optimization fails by concentrating risk on vulnerable hospitals; the finite-sample correction term is essential: removing it triples violations. No patient-level images, masks, or per-volume scores leave any site.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper quantifies a failure mode of naive pooled federated conformal risk control (CRC) on real multi-institutional brain tumor segmentation data: marginal calibration can look fine while 40% of hospitals violate the target false-negative rate. It proposes a shrinkage-based federated CRC protocol that transmits only per-site empirical risk curves and uses a single hyperparameter \(n_0\) to trade worst-case coverage against prediction-set efficiency.

Key findings

  • Naive pooled CRC (B3) achieves marginal FNR near \(\alpha=0.10\) but violates coverage at \(8.0 \pm 2.4\) of 20 sites, with worst-site FNR \(0.178\); per-site local CRC (B2) reduces violations to \(1.3 \pm 1.2\) but inflates stretch to \(83.2\times\).
  • The shrinkage method at \(n_0=9\) matches B2’s violations (\(1.3 \pm 1.2\)) while reducing stretch to \(28.8\times\); at \(n_0=15\) it gives \(2.7 \pm 1.7\) violations at \(4.4\times\) stretch; leave-one-site-out selection identifies \(n_0=19\), achieving \(2.7 \pm 1.7\) violations at \(2.0\times\) stretch.
  • Direct Lagrangian budget optimization fails: uncapped allocation reaches \(1.4\times\) stretch but violates \(12.3 \pm 0.9\) sites with worst FNR \(0.351\); adding a per-site cap gives \(2.7\) violations but \(71.6\times\) stretch.
  • Removing the finite-sample correction term triples violations to \(8.0\)–\(9.3\), and deploying the conservative \(\lambda_{\text{fed}}\) from Theorem 1 gives 0 violations at \(n_0=9\) but \(67.3\times\) stretch.

Methods

  • Data: FeTS-2022 training set, 1,251 multi-modal brain MRI volumes from 23 institutions; retain 20 institutions with \(\ge 6\) subjects (range 6–511), per-site 50/50 calibration/test splits with seeds {42, 1337, 2024}; pre-trained SegResNet from MONAI, trained on BraTS-2021.
  • CRC setup: per-volume pixel-FNR loss, prediction sets \(C_\lambda=\{v: f(X)_v \ge 1-\lambda\}\), \(G=200\) uniformly spaced \(\lambda \in [0,1]\), target \(\alpha=0.10\) swept over \(\{0.05,0.10,0.15,0.20\}\); metrics are violations, worst-site FNR, and stretch \(|C_\lambda|/|Y|\).
  • Shrinkage protocol: each site computes its local empirical risk curve \(R_k(\lambda)\) on the grid and sends only \(G\) scalars; the server computes \(R^{\text{shrink}}_k(\lambda)=w_k R_k(\lambda)+(1-w_k)R^{\text{global}}(\lambda)+\text{corr}_k\), with \(w_k=n_k/(n_k+n_0)\) and \(\text{corr}_k\) interpolating per-site and pooled finite-sample corrections; thresholds are \(\lambda_k=\inf\{\lambda: R^{\text{shrink}}_k(\lambda)\le\alpha\}\), with LOSO sensitivity analysis for \(n_0\).

Limitations

  • Stated: Theorem 1 provides only a marginal guarantee under the stated site-mixture assumption; per-site deployment is empirically validated but formally open, and if the test-site mixture differs from calibration-site proportions the marginal guarantee may not transfer.
  • Stated: the correction term \(\text{corr}_k\) is a heuristic interpolation, motivated by but not formally derived from CRC theory; its formal finite-sample analysis is future work.
  • Stated/evident: evaluation uses a single pre-trained model on one dataset, so validation with additional anatomies, backbones, and federally trained models is needed; adapting federated conformal baselines to pixel-level CRC is future work.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2606.20115

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 6 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    A single hyperparameter n0 smoothly trades worst-case coverage for prediction-set efficiency.

    “A single hyperparameter n0 smoothly trades worst-case coverage for prediction-set efficiency”
    • Also stated in: § Algorithm 1 Shrinkage-Based Federated CRC · plain
      “Intermediate n 0 interpolates smoothly.”
    • Also stated in: § 4.3 The n 0 Dial and LOSO Sensitivity Analysis · plain
      “Fig. 3 shows how n 0 controls the coverage-efficiency frontier.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    Shrinkage uses a blended risk estimator that implicitly regularizes toward the global curve, an inductive bias the constrained optimizer cannot replicate.

    “Shrinkage uses a blended risk estimator that implicitly regularizes toward the global curve, an inductive bias the constrained optimizer cannot replicate.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    Theorem 1: Under partial exchangeability and the site-mixture assumption, λfed = max{λpool, max_k λshrink_k} yields E[l(C λfed(Xtest),Ytest)] ≤ α.

    “Theorem 1 (Site-Marginal Federated CRC).Assume partial exchangeability: for each site k, (Z k 1 , . . ., Z k n k , Z k test ) is exchangeable given H=k, with P(H=k) = (n k +1)/(N +K).Define λfed = max{ λpool , max k λshrink k }.Then E[ℓ(C λfed (X test ), Y test )] ≤ α.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The site mixture at test time matches calibration-site proportions and partial exchangeability holds per site.
  • Premisereasoning

    The cell federated × CRC × pixel-level segmentation is empty.

    “The cell federated × CRC × pixel-level segmentation is empty.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    In radiation therapy planning, missed tumor voxels at specific hospitals could leave cancerous tissue untreated while aggregate metrics suggest everything is calibrated.

    “In radiation therapy planning, this means missed tumor voxels at specific hospitals-potentially leaving cancerous tissue untreated-while aggregate metrics suggest everything is calibrated.”
    Quote in the paperSubstantial claim
  • May combine two claims. 24
  • Classified as probably not a substantial claim. 2
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    The failure is not a small-sample artifact: Institution 18 (382 patients, 30% of dataset) is miscovered at FNR=0.150, and B2's stretch explodes to 100-250× at small sites while ours stays below 5×.

    “Critically, the failure is not a small-sample artifact: Fig. 2 shows that Institution 18 (382 patients, 30% of the dataset) is miscovered at FNR=0.150, and B2's stretch explodes to 100-250× at small sites while ours stays below 5×.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Naive pooled federated CRC satisfies marginal coverage but violates coverage at 40% (8/20) of individual institutions on FeTS-2022, with the worst site FNR 0.178 versus target α=0.10.

    “showing that this naive pooled CRC protects the average hospital but violates coverage at 40% of individual institutions, with the worst site exceeding the target false-negative rate by 7.8 percentage points.”
    • Also stated in: § 1 Introduction · plain
      “On FeTS-2022 brain tumor data [12] with 20 real institutions, 40% of sites exceed the target α=0.10,with the worst reaching FNR=0.178.”
    • Also stated in: § 4.2 Main Results · plain
      “B3 achieves marginal FNR≈ α but violates coverage at 8.0±2.4 of 20 sites, with the worst site reaching FNR = 0.178-nearly double the target.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The value of shrinkage is entirely empirical: conservative global λfed achieves 0 violations at n0=9 but at 67× stretch, whereas per-site thresholds yield practical stretch while maintaining near-nominal site-level coverage.

    “The value of shrinkage is therefore entirely empirical: the conservative global λfed achieves 0 violations at n 0 =9 but at 67× stretch (Table 2), whereas per-site thresholds λshrink k yield practical stretch while maintaining near-nominal site-level coverage in all tested configurations.”
    • Also stated in: § 4.5 Ablations and Sensitivity · boosted
      “Conservative λfed (Theorem 1).Deploying the single global threshold λfed = max{ λpool , max k λshrink k } dramatically reduces violations (0 at n 0 =9) but at extreme stretch (67×).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At n0=15, stretch drops to 4.4× with only 2.7 violations.

    “At n 0 =15: stretch drops to 4.4× with only 2.7 violations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Adding a per-site cap (max_k L_k ≤ α+0.03) reduces violations to 2.7 but inflates stretch to 71.6×, worse than shrinkage n0=9 on both dimensions.

    “Adding a persite cap (max k L k ≤ α + 0.03) reduces violations to 2.7 but inflates stretch to 71.6×-worse than shrinkage n 0 =9 on both dimensions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The failure mode persists across all tested α∈{0.05,0.10,0.15,0.20}: B3 violates 5-8 sites at each target while ours (n0=9) reduces violations to 0-1.

    “The failure mode persists across all tested α ∈ {0.05, 0.10, 0.15, 0.20}: B3 violates 5-8 sites at each target while ours (n 0 =9) reduces violations to 0-1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At n0=9, the method matches B2's violations (1.3) while reducing stretch by 65% (83× to 29×).

    “At n 0 =9: matches B2's violations (1.3) while reducing stretch by 65% (83× to 29×).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Grid resolution G: at n0=15, G=50 yields 35.7× stretch; G=200 and G=500 yield 4.4× and 3.0× respectively with similar violations; G=200 is a stable, practical choice.

    “At n 0 =15: G=50 yields 35.7× stretch (discretization too coarse); G=200 and G=500 yield 4.4× and 3.0× respectively with similar violations.G=200 is a stable, practical choice.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Per-site local CRC largely restores coverage but inflates prediction sets by 83× (stretch 83.2×), rendering them clinically useless.

    “The naive alternative-per-site local CRClargely restores coverage but inflates prediction sets by 83×, rendering them clinically useless.”
    • Also stated in: § 4.2 Main Results · plain
      “B2 reduces violations to 1.3 but inflates stretch to 83×, rendering predictions clinically useless.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Leave-one-site-out sensitivity analysis identifies n0=19 as an operating point achieving 2.7/20 violations at 2.0× stretch, the same violation count as n0=15 and a 97.5% reduction from B2.

    “leave-one-site-out sensitivity analysis identifies n0=19, achieving 2.7/20 violations at 2.0× stretch.”
    • Also stated in: § 4.3 The n 0 Dial and LOSO Sensitivity Analysis · plain
      “LOSO identifies n 0 =19 as the lowest-stretch operating point with ≤3 mean violations, achieving 2.7 violations at 2.0× stretch-squarely within the sweet spot of Fig. 3.”
    • Also stated in: § 4.2 Main Results · plain
      “Leave-one-site-out sensitivity analysis (Sec.4.3) identifies n 0 =19 as a favorable operating point, achieving the same violation count as n 0 =15 at just 2.0× stretch-a 97.5% reduction from B2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The finite-sample correction term is essential: removing it triples violations, with violations jumping from 1.3-2.7 to 8.0-9.3 and worst FNR rising to 0.173-0.181.

    “the finite-sample correction term is essential: removing it triples violations.”
    • Also stated in: § 4.5 Ablations and Sensitivity · boosted
      “Removing corr k is catastrophic: violations jump from 1.3-2.7 to 8.0-9.3 across all n 0 values, with worst FNR rising to 0.173-0.181-comparableto naive pooled CRC.”
    • Also stated in: § 6 Conclusion · boosted
      “the finite-sample correction, though heuristic, is essential-removing it triples violations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Fig. 3 shows small n0 (≤9) gives few violations and high stretch, large n0 (≥30) gives low stretch and many violations, and the knee at n0∈[10,20] offers the best trade-off.

    “Fig. 3 shows how n 0 controls the coverage-efficiency frontier.Small n 0 (≤ 9): few violations, high stretch (per-site CRC regime).Large n 0 (≥ 30): low stretch, many violations (pooled regime).The knee at n 0 ∈ [10, 20] (shaded) offers the best trade-off.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Direct Lagrangian optimization of coverage budgets fails, concentrating risk on vulnerable hospitals; uncapped budget allocation achieves 1.4× stretch but fails 12.3/20 sites with worst FNR=0.351.

    “We further show that direct Lagrangian optimization of coverage budgets fails-concentrating risk on vulnerable hospitals”
    • Also stated in: § 4.4 Why Direct Optimization Fails · plain
      “Uncapped budget allocation achieves minimal stretch (1.4×) but catastrophically fails 12.3/20 sites (worst FNR=0.351): the optimizer concentrates risk on small, hard institutions to satisfy the marginal constraint cheaply.”
    • Also stated in: § 6 Conclusion · plain
      “Direct optimization of coverage budgets fails by exploiting vulnerable hospitals”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    When sites differ in tumor prevalence, scanner protocols, or difficulty, the pooled threshold is dominated by majority-site behavior, and minority sites can have E[l | H=k] ≫ α.

    “When sites differ in tumor prevalence, scanner protocols, or difficulty, the pooled threshold is dominated by majority-site behavior, and minority sites can have E[ℓ | H=k] ≫ α.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Theorem 1 provides only a marginal guarantee under the stated site-mixture assumption; per-site deployment is empirically validated but formally open.

    “Theorem 1 provides only a marginal guarantee under the stated site-mixture assumption; the per-site deployment is empirically validated but formally open.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The site mixture at test time matches calibration-site proportions and partial exchangeability holds per site.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study uses a single pretrained model on one dataset; validation with additional anatomies, backbones, and federally trained models is needed.

    “We use a single pretrained model on one dataset; validation with additional anatomies, backbones, and federally trained models is needed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Under partial exchangeability, pooled federated CRC controls marginal risk E[l test] ≤ α but not site-conditional risk E[l | H=k].

    “Under partial exchangeability [7], this controls the marginal risk E[ℓ test ] ≤ α but not site-conditional risk E[ℓ | H=k].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    The marginal-vs-conditional coverage gap is well studied in conformal prediction theory, but its magnitude in federated medical segmentation has not been quantified.

    “This marginal-vs-conditional coverage gap is well studied in conformal prediction theory [16,4], but its magnitude in federated medical segmentation has not been quantified.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [16]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [4]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    Per-site coverage is empirically near-nominal across all tested configurations, validated across four targets with three seeds.

    “per-site coverage is validated across four targets with three seeds.”
    • Also stated in: § Contributions. · plain
      “per-site coverage is empirically near-nominal across all tested configurations.”
    • Also stated in: § 4.3 The n 0 Dial and LOSO Sensitivity Analysis · plain
      “The failure mode persists across all tested α ∈ {0.05, 0.10, 0.15, 0.20}: B3 violates 5-8 sites at each target while ours (n 0 =9) reduces violations to 0-1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    n0→0 recovers per-site CRC; n0→∞ recovers pooled CRC; intermediate n0 interpolates smoothly.

    “n 0 → 0: recovers per-site CRC (high coverage, high stretch).n 0 → ∞: recovers pooled CRC (low stretch, poor worst-site coverage).Intermediate n 0 interpolates smoothly.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    No patient-level images, masks, or per-volume scores leave any site; each site transmits only its empirical risk curve (G scalars).

    “No patient-level images, masks, or per-volume scores leave any site.”
    • Also stated in: § Algorithm 1 Shrinkage-Based Federated CRC · plain
      “Each site transmits G real numbers summarizing aggregate loss statistics.No individual volumes, masks, or per-volume scores leave the site.”
    • Also stated in: § 6 Conclusion · plain
      “No patient-level images, masks, or per-volume scores leave any site.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    This quantifies the gap between the formal guarantee and practical deployment, confirming that per-site thresholds are necessary for clinical utility.

    “This quantifies the gap between the formal guarantee and practical deployment, confirming that per-site thresholds are necessary for clinical utility.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Per-site local CRC guarantees site-conditional coverage but for small nk the correction B/(nk+1) is prohibitively large; with nk=5 and B=1, correction alone is 0.167 > α=0.10, making the constraint unsatisfiable, forcing λk=λmax and stretch exceeding 80×.

    “This guarantees site-conditional coverage but, for small n k , the correction B/(n k +1) is prohibitively large.With n k =5 and B=1, the correction alone is 0.167 > α=0.10: the constraint becomes unsatisfiable, forcing λk = λ max (predict everything as tumor) and stretch exceeding 80×.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Vulnerability arises from case-mix and outcome heterogeneity across sites, not merely from small sample size: even the largest institution is miscovered under naive pooling.

    “Vulnerability here arises from case-mix and outcome heterogeneity across sites, not merely from small sample size: even our largest institution is miscovered under naive pooling.”
    Quote in the paperSubstantial claim
24 of 24

  • The authors state this as an assumption.

    The site mixture at test time matches calibration-site proportions and partial exchangeability holds per site.

    “Assume partial exchangeability: for each site k, (Z k 1 , . . ., Z k n k , Z k test ) is exchangeable given H=k, with P(H=k) = (n k +1)/(N +K).”
    “under the stated site-mixture assumption”
1 of 1

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The reported 1,251 subjects cannot all belong to the retained 20 institutions

    Minor

    The setup says 1,251 volumes came from 23 institutions, then excludes three institutions. The retained 20 therefore cannot still contain all 1,251 subjects, as the abstract implies.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. Angelopoulos, A.N., Bates, S., Fisch, A., Lei, L., Schuster, T.: Conformal risk control. In: ICLR (2024)

Authors
Shahid, N. F.
Corresponding author
Shahid, Nafis Fuad
ORCID
Shahid, Nafis Fuad
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2606.20115
arXiv
2606.20115
OpenAlex
W7165371612
Semantic Scholar
3b4bd648aaccdda25a20294a9691dcd509585cc6
Cited by
0
References
0
Type
posted content
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
conformal risk control, federated learning, site-specific calibration, aggregation weights, brain tumor segmentation, false negative rate, finite-sample correction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review