CoChat
Sign Up
Cited by 12Medical Image Segmentation TechniquesAdvanced Neural Network ApplicationsAI in cancer detection

Uncertainty-aware segmentation quality prediction via deep learning Bayesian Modeling: Comprehensive evaluation and interpretation on skin cancer and liver segmentation

Open access
O.K., Sikha, Riera-Marín, Meritxell, Galdran, Adrian, López, Javier García, Rodríguez-Comas, Júlia, Piella, Gemma, Ballester, Miguel A. González·2025·Computerized Medical Imaging and Graphics
DOI 10.1016/j.compmedimag.2025.102547
Open this paper in CoChat — free account

Abstract

score of 85.03 and a Pearson correlation of 65.02, demonstrating cross-modality robustness. Additionally, we propose an aggregation strategy that combines multiple uncertainty estimates into a single score per image, offering a more robust and comprehensive assessment of segmentation quality compared to evaluating each measure independently. The proposed uncertainty-aware segmentation quality prediction network is interpreted using gradient-based methods such as Grad-CAM and feature embedding analysis through UMAP. These techniques provide insights into the model's behavior and reliability, helping to assess the impact of incorporating uncertainty into the segmentation quality prediction pipeline. The code is available at: https://github.com/sikha2552/Uncertainty-Aware-Segmentation-Quality-Prediction-Bayesian-Modeling-with-Comprehensive-Evaluation-.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes uncertainty-aware segmentation quality prediction without test-time ground truth, using Bayesian approximations (Monte Carlo Dropout, ensembles, test-time augmentation) and uncertainty maps to regress Dice scores. Across HAM10000 skin lesion and liver CT datasets, confidence maps and three-input CNNs improve prediction, with best HAM10000 R² 93.99/PCC 97.01 and liver R² 85.03/PCC 65.02.

Key findings

  • For the two-way CNN using uncertainty estimates plus predicted segmentation maps, MCD + confidence achieved the best HAM10000 result: R² 93.25 and Pearson correlation 96.58; Ensemble + confidence reached R² 86.07/PCC 92.83, while TTA + confidence reached R² 71.7/PCC 85.74. On liver, Ensemble + confidence reached R² 72.94/PCC 85.88 and MCD + confidence R² 61.63/PCC 78.81.
  • The three-way CNN incorporating original image, uncertainty map, and predicted segmentation map improved performance: HAM10000 MCD + entropy reached R² 93.99/PCC 97.01, TTA + entropy R² 89.13/PCC 94.49, and Ensemble + entropy R² 86.49/PCC 93.2. For liver, TTA + entropy reached R² 85.03/PCC 65.02.
  • Confidence maps consistently improved two-way predictions across uncertainty models, while entropy maps showed better correlation in the three-way setup. The proposed aggregate uncertainty score combines confidence, entropy, mutual information, and expected pairwise KL divergence with weights 0.4, 0.2, 0.2, and 0.2, respectively, and was used to flag ambiguous or noisy labels.
  • In RMSE comparison on skin cancer segmentation, Ensemble-Confidence achieved the lowest RMSE (0.132 ± 0.039), followed by TTA-Confidence (0.153 ± 0.049), LCE (0.167 ± 0.019), Max Probability (0.168 ± 0.014), HCNN (0.196 ± 0.023), No Uncertainty (0.198 ± 0.011), MCD-Confidence (0.209 ± 0.071), Quality Net (0.213 ± 0.009), and RCA (0.438 ± 0.007); the paper states uncertainty reduced RMSE by up to 0.066.

Methods

  • The framework uses a fixed segmentation backbone, uncertainty models, uncertainty estimates, and a CNN regressor for segmentation quality. Backbones were FPN with ResNet50 for HAM10000 skin lesion segmentation and SwinUNet for liver segmentation. Uncertainty models were Monte Carlo Dropout (10 passes), ensembles (reported as 3 models in the Method and 5 models in Results), and test-time augmentation (10 random rotations). Uncertainty estimates were confidence map, entropy (Renyi with α = 2), mutual information, and expected pairwise KL divergence.
  • Data: HAM10000 contains 2500 dermatoscopic images, with 50% used to train the segmentation model and 50% reserved for testing. The liver dataset contains 130 CT scans from Kaggle/LiTS17, with tumor labels merged with liver labels for binary segmentation.
  • The quality prediction models were trained supervised on Dice scores using either two input branches (uncertainty estimates plus predicted segmentation maps, or uncertainty estimates plus original images) or three input branches (original image, uncertainty map, predicted segmentation map). Evaluation used R², Pearson correlation, and RMSE, with Grad-CAM and UMAP for interpretation.

Limitations

  • The paper does not include a dedicated limitations section.
  • Evident from the text: evaluation is limited to two datasets/modalities (2D skin lesion and 3D liver CT), and the quality prediction network is trained using ground-truth Dice scores from the training set, so ground truth is still required during training even though not at test time.
  • Evident from the text: TTA performance was variable and sometimes lower, liver results showed high R² but lower Pearson correlation (65.02), and the paper contains reporting inconsistencies, including ensemble size (3 vs. 5 models) and reversed backbone-dataset assignments in the Conclusion.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.compmedimag.2025.102547

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 14 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1. Introduction

    Traditional approaches to assessing segmentation quality heavily depend on the availability of ground truth data.

    “Traditional approaches to assessing segmen- tation quality, such as boundary-based, region-based, and hy- brid metrics, heavily depend on the availability of ground truth data [7], [8], [9].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [7] DeVries, 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [8] DeVries, 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [9] Esteva, 2017, Dermatologist-level classification of skin cancer with deep neural networks, Nature source

      The Abstract discusses lesion segmentation as preprocessing used by earlier classification methods, but does not discuss how segmentation quality is assessed. A search for boundary-based, region-based, and hybrid metrics and segmentation ground truth found no passage addressing the claimed dependence on ground-truth data. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data

    Traditional methods like RCA exhibit higher RMSE than uncertainty-aware approaches on the skin cancer segmentation dataset.

    “In contrast, traditional methods like RCA [3] exhibit higher RMSE, underscoring the advantages of uncertainty-aware approaches.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] Bilic, 2019 source

      The Results (Meta analysis and Results of individual challenges; page not provided) concern liver tumor segmentation and report Dice and ASD scores, not RMSE comparisons on a skin cancer dataset. Searches for RCA, RMSE, skin lesions, and uncertainty-aware methods found no passage addressing the attributed comparison. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingown data
    § Abstract

    The framework achieves a Pearson correlation of 96.58 on the HAM10000 dataset.

    “Pearson correlation of 96.58 on the HAM10000 dataset”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4. Expected pairwise Kullback-Leibler (KL) diver-

    In the authors' experiments, confidence maps consistently showed strong correlation with the prediction task across different models and datasets.

    “In our experiments, confidence maps consistently showed strong correlation with the prediction task across different models and datasets.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract

    For 3D liver segmentation, Test Time Augmentation with entropy achieves a Pearson correlation of 65.02.

    “a Pearson correla- tion of 65.02”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 13
  • The quote only partly states this claim. 2
  • May combine two claims. 12

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction

    Robinson et al. proposed a CNN-based regression model to predict Dice Similarity Coefficients using original images and segmentation maps.

    “Robinson et al. [10] proposed a CNN-based regression model to predict Dice Similarity Co- efficients (DSC) using original images and segmentation maps”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [10] Gal, 2016, Dropout as a bayesian approximation: Representing model uncertainty in deep learning source

      Section 5.2 (p. 6) describes a CNN used to assess classification confidence on MNIST, while Section 5.1 (p. 5) describes regression experiments on CO₂ data. Searches for Robinson, Dice Similarity Coefficients, and segmentation maps found no passage addressing the attributed model. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2. Ensemble models: Ensemble models involve com-

    Ensemble models combine predictions from multiple individual models to improve overall performance.

    “Ensemble models involve com- bining predictions from multiple individual models to improve overall performance[32].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [32] Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., Xu, D., Nath, V., Hatamizadeh, A., 2022. Self-supervised pre-training of swin transformers for 3d medical image analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20730–20740. source

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1. Monte Carlo Dropout (MCD): Monte Carlo

    Monte Carlo Dropout estimates model uncertainty in neural networks by enabling dropout during test time.

    “Monte Carlo Dropout (MCD) estimate model uncertainty in neu- ral networks [21] by enabling dropout during test time.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] Liu, 2019, Accurate uncertainty estimation and decomposition in ensemble learning, Adv. Neural Inf. Process. Syst. source

      The Introduction (pp. 1–2) discusses epistemic uncertainty, and the Experiments section (p. 8) describes a deep-ensemble comparator, but neither explains Monte Carlo Dropout. Searches for “dropout” and “test time” found no passage saying that enabling dropout at test time estimates model uncertainty. (LLM, conf=0.75)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5. Datasets and Segmentation Backbones

    The HAM10000 dataset contains 2500 dermatoscopic images of pigmented skin lesions.

    “The HAM10000 dataset [34] contains 2500 dermatoscopic images of pigmented skin lesions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [34] Valindria, 2017, Reverse classification accuracy: predicting segmentation performance in the absence of ground truth, IEEE Trans. Med. Imaging source

      The Introduction (p. 1) discusses medical-image segmentation, and the Discussion (p. 8) describes a liver-image example. Searches for HAM10000, 2500 dermatoscopic images, and skin-lesion datasets found no passage in our copy addressing the claimed dataset or image count. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • The quote only partly states this claim.

    Findingown data
    § Abstract

    The framework achieves an R2 score of 93.25 on the HAM10000 dataset.

    “Our framework achieves an R² score of 93.25”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The proposed aggregated uncertainty score effectively identifies and quantifies discrepancies between the ground truth and model predictions.

    “The proposed aggregated uncertainty score effectively identifies and quantifies these discrepancies between the ground truth and model predictions.”
    • Also stated in: § 7. Conclusion · boosted
      “the results demonstrated that the aggregated uncertainty score accurately identifies and quantifies discrepancies between the ground truth and model predictions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the Liver dataset, MCD with confidence map had an R2 score of 61.63 and a Pearson correlation of 78.81.

    “while MCD with confidence map had an R² score of 61.63 and a Pearson correlation of 78.81.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For samples with the lowest Dice score predictions, the Grad-CAMs from the second sub-branch concentrate on very small regions inside the lesions, reflecting high uncertainty across most pixels.

    “The 4th column, representing the Grad-CAM from the second sub-branch, concentrates on very small regions inside the lesions, reflecting high uncertainty across most pixels as seen in the uncertainty map.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The top three images with the highest aggregated uncertainty scores have ambiguous or noisy labels, where a significant portion of pixels are labeled as lesions, and the model predictions are notably inaccurate.

    “It is evident from the images that all three have ambiguous or noisy labels, where a significant portion of pixels are labeled as lesions, and the model predictions are notably inaccurate.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    The 3-way CNN UMAP embedding shows the highest degree of feature separation among the three configurations, followed by the 2-way CNN with segmentation map and uncertainty map, then the 2-way CNN with original image and uncertainty map.

    “In Figure.10 column (a), the 2-way CNN embedding with the original image and uncertainty map shows a moderate level of feature separation”
    • Also stated in: § 6.3. Visualization and Interpretation of Grad-CAMs and · boosted
      “Column (c) demonstrates the UMAP embedding for the 3-way CNN that includes the original image, segmentation map, and uncertainty map. It shows the highest degree of feature separation among the three configurations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning
    § 3. Method

    Full Bayesian inference in deep learning models is computationally infeasible due to the complexity of calculating the posterior distribution.

    “However, in practice, performing full Bayesian inference in deep learning models is compu- tationally infeasible due to the complexity of calculating the posterior distribution.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3. Test Time Augmentation (TTA): TTA involves ap-

    Test Time Augmentation involves applying data augmentation transformations to the input image during test time and generating predictions for each augmented version.

    “TTA involves ap- plying data augmentation transformations, such as rotations, flips, or color adjustments, to the input im- age during test time and generating predictions for each augmented version [24].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Movahedi, 2010, Design and perceptual validation of performance measures for salient object segmentation source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1. Introduction

    Zhou et al. combined deep reconstruction networks with regression models for DSC prediction.

    “Zhou et al. [12] combined deep reconstruction networks with regression models for DSC prediction.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] Huang, 2016, Qualitynet: Segmentation quality evaluation with deep convolutional networks source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3. Method

    FPN with ResNet50 has been widely used for skin cancer segmentation, with Esteva et al. and Codella et al. highlighting its effectiveness in capturing fine-grained details in dermoscopic images.

    “Similarly, FPN with ResNet50 has been widely used for skin cancer segmentation, with Esteva et al. [30] and Codella et al. [31] highlighting its effective- ness in capturing fine-grained details in dermoscopic im- ages.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [30] Sikha, 2024, Uncertainty aware segmentation quality assessment in medical images source
    • cites [31] Somasundaram, 2019, Current trends on deep learning models for brain tumor segmentation and detection–A review source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1. Introduction

    DeVries et al. introduced a DCNN-based model that uses uncertainty maps alongside original images and segmentation maps to predict DSC scores and flag cases for expert review.

    “De- Vries et al.[14] introduced a DCNN-based model that uses uncertainty maps alongside original images and segmentation maps to predict DSC scores and flag cases for expert review.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Kendall, 2017, What uncertainties do we need in bayesian deep learning for computer vision?, Adv. Neural Inf. Process. Syst. source

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1. Introduction

    No studies have systematically evaluated and benchmarked multiple uncertainty models alongside various uncertainty estimates in a multimodal setting.

    “No studies have systematically eval- uated and benchmarked multiple uncertainty models (such as Monte Carlo Dropout, Ensembles, and Test Time Augmenta- tion) alongside various uncertainty estimates (including confi- dence maps, entropy, mutual information, and expected pair- wise Kullback-Leibler divergence) in a multimodal setting.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § Abstract

    The aggregation strategy combines multiple uncertainty estimates into a single score per image, offering a more robust and comprehensive assessment of segmentation quality compared to evaluating each measure independently.

    “Additionally, we propose an aggregation strategy that combines mul- tiple uncertainty estimates into a single score per image, offering a more robust and comprehensive assessment of segmenta- tion quality compared to evaluating each measure independently.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 5. Datasets and Segmentation Backbones

    SwinUnet achieves superior segmentation performance on complex volumetric medical datasets, effectively balancing broad contextual understanding with fine-grained local delineation.

    “This synergy between global context captured by the Swin Transformer and the precise spatial details from UNet en- ables SwinUnet to achieve superior segmentation performance on complex volumetric medical datasets, effectively balancing broad contextual understanding with fine-grained local delin- eation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The reduced effectiveness of TTA may be due to noise or irrelevant variations introduced by augmentations, suggesting that TTA's effectiveness is highly dependent on the dataset and augmentation type.

    “The reduced effectiveness may be due to noise or irrelevant variations introduced by augmentations, suggesting that TTA's effectiveness is highly dependent on the dataset and augmentation type.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The improved clustering in the 3-way CNN signifies that it is better equipped to push the Dice scores further apart, thereby enhancing its predictive capabilities and robustness in segmentation tasks.

    “The improved clustering in (c) signifies that the 3-way CNN is better equipped to push the Dice scores further apart, thereby enhancing its predictive capabilities and robustness in segmentation tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § 5. Datasets and Segmentation Backbones

    The liver dataset is part of the Liver Tumor Segmentation Challenge (LiTS17).

    “This dataset is part of the Liver Tu- mor Segmentation Challenge (LiTS17), organized in conjunc- tion with ISBI 2017 and MICCAI 2017.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Ensemble models and MCD, which address epistemic uncertainty, significantly benefit from confidence maps, whereas TTA, capturing aleatoric uncertainty, showed variable performance due to the potential introduction of noise or artifacts.

    “Ensemble models and MCD, which address epistemic uncertainty, significantly benefit from confidence maps, whereas TTA, capturing aleatoric uncertainty, showed variable performance due to the potential introduction of noise or artifacts.”
    Quote in the paperSubstantial claim
22 of 22

  • The paper relies on this without support.

    Ground truth annotations are often impractical to obtain for large medical image datasets.

    “particularly in medical image analysis, where obtaining ground truth for large datasets is often impractical.”

  • The authors state this as an assumption.

    Test Time Augmentation can be viewed as a Bayesian mixture model for uncertainty estimation.

    “While not explicitly Bayesian, TTA can be viewed as a Bayesian mixture model for uncertainty es- timation.”

  • The paper relies on this without support.

    The HAM10000 and liver datasets are sufficiently representative of different modalities to demonstrate cross-modality robustness.

    “For 3D liver segmentation, Test Time Augmentation with entropy achieves an R² score of 85.03 and a Pearson correla- tion of 65.02, demonstrating cross-modality robustness.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Ensemble size changes from three to five models

    Major

    The methods specify an ensemble of three models, but the results describe the evaluated ensemble as having five models.

  • Conclusion reverses the segmentation backbones assigned to the datasets

    Major

    The methods section assigns FPN to HAM10000 and Swin UNET to the liver dataset, while the conclusion assigns them the other way around.

  • The stated highest HAM10000 three-input R2 is below another reported result

    Major

    The text calls Ensemble + Entropy's R2 of 86.49 the highest for HAM10000, then reports 93.99 for MCD + Entropy in the same three-input comparison.

  • The segmentation model is described as requiring a label at input

    Minor

    The stated segmentation function takes both image x and label y as inputs, although the framework is said to operate without ground truth at test time. As written, the function would require the unavailable label to generate a test-time segmentation.

  • The stated best liver R2 in Table 2 is not the largest

    Minor

    The narrative identifies Ensemble + Confidence at 63.72 as the best liver R2 for this input setup, whereas Table 2 reports 63.75 for TTA + Entropy.

  • Entropy is not the best-correlating three-input estimate for every model

    Minor

    The text says entropy maps showed better correlation with all uncertainty models, but Table 3 reports a higher liver Pearson correlation for Ensemble + MI (85.31) than for Ensemble + Entropy (84.76).

  • TTA is not uniformly worse on the liver two-input task

    Minor

    The narrative says TTA performed comparatively worse with all uncertainty estimates on the liver dataset. In Table 1, TTA + Confidence has a higher liver R2 (71.6) than MCD + Confidence (61.63), with a higher PCC as well.

7 of 7

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Konyushkova, 2017, Learning active learning from data, Adv. Neural Inf. Process. Syst., 30

Cited by

Sorted by citations

Related

Authors
O.K., S., Riera-Marín, M., Galdran, A., López, J. G., Rodríguez-Comas, J., Piella, G., & Ballester, M. A. G.
Affiliations
Universitat Pompeu Fabra (ES)
ORCID
Venue
Computerized Medical Imaging and Graphics
Publisher
Elsevier BV
Published
2025-07
Volume
123
Pages
102547
ISSN
  • 0730-4862 (ISSN-L)
  • 0895-6111
DOI
10.1016/j.compmedimag.2025.102547
OpenAlex
W4409432992
Semantic Scholar
ceb476c2f3c016b379ea6b5873428231105dca11
Cited by
11
References
43
Citations by year
2026: 7 · 2025: 4
FWCI
3.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Computerized Medical Imaging and Graphics (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • TECNALIA Publications (Fundación TECNALIA Research & Innovation) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (cc-by)
Funders
Spain Ministry of Science and Innovation (CPP2023-010970)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Keywords
uncertainty estimation, Bayesian deep learning, liver segmentation, uncertainty aggregation, Grad-CAM, interpretability
MeSH terms
Deep Learning, Bayes Theorem, Humans, Image Interpretation, Computer-Assisted, Image Processing, Computer-Assisted, Liver Neoplasms, Skin Neoplasms, Reproducibility of Results, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review