CoChat
Sign Up
Cited by 207Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Discrimination and Equality Law

A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices

Open access
Speicher, Till, Heidari, Hoda, Grgic-Hlaca, Nina, Gummadi, Krishna P., Singla, Adish, Weller, Adrian, Zafar, Muhammad Bilal·2018·Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
DOI 10.1145/3219819.3220046
Open this paper in CoChat — free accountOpen PDF

Abstract

Discrimination via algorithmic decision making has received considerable attention. Prior work largely focuses on defining conditions for fairness, but does not define satisfactory measures of algorithmic unfairness. In this paper, we focus on the following question: Given two unfair algorithms, how should we determine which of the two is more unfair? Our core idea is to use existing inequality indices from economics to measure how unequally the outcomes of an algorithm benefit different individuals or groups in a population. Our work offers a justified and general framework to compare and contrast the (un)fairness of algorithmic predictors. This unifying approach enables us to quantify unfairness both at the individual and the group level. Further, our work reveals overlooked tradeoffs between different fairness notions: using our proposed measures, the overall individual-level unfairness of an algorithm can be decomposed into a between-group and a within-group component. Earlier methods are typically designed to tackle only between-group un- fairness, which may be justified for legal or other reasons. However, we demonstrate that minimizing exclusively the between-group component may, in fact, increase the within-group, and hence the overall unfairness. We characterize and illustrate the tradeoffs between our measures of (un)fairness and the prediction accuracy.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes using inequality indices from economics, especially generalized entropy, to quantify algorithmic unfairness as inequality in benefits across individuals and groups. It shows that overall individual unfairness decomposes into between-group and within-group components, and that minimizing only between-group unfairness can increase within-group and overall unfairness.

Key findings

  • In the illustrative example, both classifiers are individually unfair, but with benefits from Eq. 1 and generalized entropy with α = 2, C1 has unfairness 0.2 while C2 has unfairness 0.3, so C2 is more individually unfair.
  • Between-group unfairness can be a small fraction of overall unfairness: for logistic regression on COMPAS, the maximum overall unfairness is close to 0.6 while the maximum between-group unfairness is merely 0.01.
  • Applying Zafar et al.’s false-negative-rate constraints to reduce between-group unfairness increased within-group unfairness for Whites almost monotonically, and for covariance multiplicative factors between 0.8 and 0.6 the overall unfairness went up.
  • On COMPAS, logistic regression, SVM, and random forests achieved similar accuracies of 66%, 67%, and 65% and similar overall individual unfairness of 0.145, 0.151, and 0.134, respectively.

Methods

  • Theoretical framework: define a benefit function for binary classification, b_i = ŷ_i − y_i + 1, then apply inequality indices such as generalized entropy E_α; the paper argues these indices satisfy anonymity, population invariance, transfer principle, zero-normalization, and subgroup decomposability.
  • Empirical data: Adult income dataset after filtering (44,434 subjects, 11 features; sensitive features gender and race) and ProPublica COMPAS dataset (5,786 subjects, 5 features; sensitive features gender, race, and age).
  • Empirical procedure: repeatedly split data into 70%–30% train-test sets 10 times, use logistic regression, SVM with RBF kernel, and random forest, vary a decision ranking threshold τ, and decompose E_2 into between-group and within-group unfairness; also train constrained logistic regression classifiers using Zafar et al.’s covariance multiplicative factors from 1.00 to 0.00.

Limitations

  • Stated: proofs are omitted due to space constraints and can be found in the long version; the paper focuses on supervised learning and binary classification, leaving unsupervised tasks for future work.
  • Evident: the benefit values are context- and task-dependent, and the paper notes that determining them may vary in practice.
  • Evident: empirical validation uses only two real-world datasets (Adult and COMPAS), and the exposition assumes one sensitive feature for simplicity, though the discussion is said to extend to multiple sensitive features.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3219819.3220046

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 24 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Existing fair learning methods, while successful in eliminating between-group unfairness, may be targeting only a small fraction of the overall unfairness in decision making algorithms.

    “Our empirical evaluations show that existing fair learning methods [17,40], while successful in eliminating between-group unfairness, (a) may be targeting only a small fraction of the overall unfairness in the decision making algorithms”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [17]

      Not checked in the free run.

    • cites [40] source

      Not checked in the free run.

    • The authors state this as an assumption: Labels in the training data reflect ground truth, so y_i is the label deserved by individual i.
  • Findingnone given

    Recently proposed learning mechanisms for mitigating (between-)group unfairness can lead to high within-group unfairness and consequently, high individual unfairness.

    “we study the relationship between individual-and group-level unfairness, showing that recently proposed learning mechanisms for mitigating (between-)group unfairness can lead to high within-group unfairness and consequently, high individual unfairness.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Labels in the training data reflect ground truth, so y_i is the label deserved by individual i.
  • Findingnone given

    If an inequality measure I satisfies the transfer principle and is population- and scale-invariant, and there exists a feature vector x such that 0 < P[y = 1 | x = x] < 0.5, then there exists a training data set D for which the accuracy-optimal classifier θ_B^D and the fairness-optimal classifier θ_F^D do not coincide.

    “Proposition 3.2.Suppose I (.) satisfies the transfer principle and is population-and scale-invariant.If there exists a feature vector x such that 0 < P[y = 1|x = x] < 0.5, then there exists a training data set D for which θ B D θ F D .”
    Quote in the paperSubstantial claim
  • Findingown data

    Overall unfairness is not always a monotone function of the decision ranking threshold.

    “We note that the overall unfairness is not always a monotone function of the decision ranking threshold (an example is provided in the long version of the paper).”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    As the number of sensitive feature groups increases, between-group unfairness contributes more and more towards overall unfairness.

    “Figure 6 shows that as the number of sensitive feature groups increases, the between-group unfairness contributes more and more towards the overall unfairness.This result is also in line with the implications of Proposition 3.4.”
    Quote in the paperSubstantial claim
  • May combine two claims. 23
  • Classified as probably not a substantial claim. 4

  • May combine two claims.

    Findingown data

    Existing fair learning methods can result in an increase in within-group unfairness, which can lead to training algorithms whose overall unfairness is worse than those trained using traditional learning methods.

    “and (b) can result in an increase in within-group unfairness, which paradoxically can lead to training algorithms whose overall unfairness is worse than those trained using traditional learning methods.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [17]

      Not checked in the free run.

    • cites [40] source

      Not checked in the free run.

    • The authors state this as an assumption: Labels in the training data reflect ground truth, so y_i is the label deserved by individual i.

  • May combine two claims.

    Findingnone given

    Using generalized entropy with α=2, a classifier C1 achieving a false positive rate of 0.8 on group A and 0.6 on group B has a between-group inequality of 0.06, whereas a classifier C2 with false positive rates of 0.6 on A and 0.8 on B results in a lower between-group inequality of 0.04.

    “Using generalized entropy with α = 2, a classifier C 1 achieving a false positive rate of 0.8 on A and 0.6 on B has a betweengroup inequality of 0.06, whereas a classifier C 2 with false positive rates of 0.6 on A and 0.8 on B results in a lower between-group inequality of 0.04.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the ProPublica COMPAS dataset, logistic regression, SVM, and random forests achieve similar accuracies of 66%, 67%, and 65% and similar overall individual unfairness of 0.145, 0.151, and 0.134, respectively.

    “Logistic regression, SVM and Random Forests achieve similar accuracies of 66%, 67% and 65% as well as similar overall individual unfairness of 0.145, 0.151 and 0.134 respectively on the ProPublica Compas dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Reducing between-group unfairness can increase within-group unfairness; for Whites, within-group unfairness almost monotonically increases as between-group unfairness is reduced.

    “(i) Reducing the between-group unfairness can in fact increase the within-group unfairness: the withingroup unfairness for Whites almost monotonically increases as the between-group unfairness is reduced.This observation also follows Proposition 3.3.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: No prior work in fairness-aware machine learning has studied the effect of constraints designed to remove between-group unfairness on overall and within-group unfairness.

  • May combine two claims.

    Findingown data

    There is no definitive correlation between accuracy and between-group unfairness; in the Adult dataset, the highest accuracy around τ = 0.8 corresponds to one of the lowest between-group unfairness values for all classifiers, but this does not hold for the COMPAS dataset.

    “We notice that there is no definitive correlation between the accuracy and the between-group unfairness: In the Adult dataset, the highest level of accuracy (around τ = 0.8) corresponds to one of the lowest values of between-group unfairness for all classifiers.However, this doesn't hold in the case of COMPAS dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    For the oracle classifier, perfect accuracy corresponds to zero unfairness; with increasing decision ranking threshold τ, accuracy increases and unfairness decreases until an optimal threshold close to 0.75 in the Adult data and 0.45 in the COMPAS data, after which the trend reverses.

    “For the oracle, as expected from Proposition 3.1, a perfect accuracy corresponds to zero unfairness: with an increasing τ , the accuracy increases while the unfairness decreases.After a certain optimal value of threshold τ (close to 0.75 in the Adult data and 0.45 in the COMPAS data), the trend reverses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the same value of τ, between-group unfairness is a very small fraction of overall unfairness; for example, for logistic regression on COMPAS, the maximum overall unfairness is close to 0.6 while the maximum between-group unfairness is merely 0.01.

    “Comparing the between-group unfairness and overall unfairness in Figure 5 reveals a very interesting insight: for the same value of τ , the between-group unfairness (solid lines) is a very small fraction of the overall unfairness (dotted lines).For example, considering the performance of the logistic regression classifier on the COMPAS dataset, the maximum value of the overall unfairness is close”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    For a zero-normalized and additively decomposable inequality measure I with I(b) > 0, for any partition G of the population into disjoint subgroups, the between-group share of overall unfairness lies between 0 and 1; further, there exist benefit distributions b and b′ such that the share equals 1 and equals 0.

    “Proposition 3 . 5 . Suppose I (.) is zero-normalized and additively decomposable.Suppose I (b) 0. For any partition G of the population to disjoint subgroups, 0 ≤ I G β (b) I (b) ≤ 1.Further, there exist benefit distributions b and b ′ such that I G β (b) I (b) = 1 and”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    For non-oracle classifiers, the optimal threshold for imperfect accuracy is far from the optimal threshold for unfairness; with increasing τ, unfairness continually increases while accuracy initially increases and then drops.

    “However, for all other (non-oracle) classifiers, as expected via Proposition 3.2, the trend is very different: the optimal threshold for (imperfect) accuracy is far from the optimal threshold for unfairness.Moreover, with increasing τ , while unfairness continually increases, for accuracy we initially see an increase followed by a drop.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the illustrative example, classifier C1 has individual unfairness 0.2 and classifier C2 has individual unfairness 0.3, so C2 is more individually unfair than C1.

    “We note that while both C 1 and C 2 are individually unfair according to our unfairness measure (with benefits defined as in Eq. 1 and Generalized Entropy in Eq. 2 used with α = 2), the unfairness of C 1 is 0.2 whereas the unfairness of C 2 is 0.3.Hence, C 2 is more individually unfair than C 1 .”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Prior work on algorithmic fairness largely focuses on formally defining conditions for fairness, but does not precisely define suitable measures for unfairness.

    “Prior work largely focuses on defining conditions for fairness, but does not define satisfactory measures of algorithmic unfairness.”
    • Also stated in: § 1 INTRODUCTION · plain
      “Prior works on algorithmic fairness largely focus on formally defining conditions for fairness, but do not precisely define suitable measures for unfairness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Inequality indices offer a unifying approach to quantifying unfairness at the levels of both individuals and groups, and reveal previously overlooked tradeoffs between individual-level and group-level fairness.

    “Thus, inequality indices not only offer a unifying approach to quantifying unfairness at the levels of both individuals and groups, but they also reveal previously overlooked tradeoffs between individual-level and group-level fairness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Dwork et al.'s definition does not take into account the difference in social desirability of various outcomes; flipping the labels predicted by a fair classifier yields a classifier considered equally fair.

    “Second, the definition does not take into account the difference in social desirability of various outcomes.For instance, if one flips the (binary) labels predicted by a fair classifier, the resulting classifier will be considered equally fair”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12] source

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Dwork et al.'s notion of individual fairness only provides a yes/no answer to whether fairness conditions are satisfied, but does not provide a meaningful measure of algorithmic fairness when considered independent of prediction accuracy.

    “Furthermore, Dwork et al. 's notion of individual fairness only provides a 'yes/no' answer to whether fairness conditions are satisfied, but does not provide a meaningful measure of algorithmic fairness when considered independent of prediction accuracy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [12] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The choice of benefit function determines the relative desirability of different outcome types and captures different notions of fairness.

    “the choice of our benefit function crucially determines the relative desirability of these different types of outcomes and captures different notions of fairness.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Labels in the training data reflect ground truth, so y_i is the label deserved by individual i.

  • May combine two claims.

    Premisenone given

    Existing group fairness notions usually do not take into account the size of different groups, whereas the proposed between-group measure considers the proportion of the groups relative to the total population.

    “However, these notions usually do not take into account the size of different groups, whereas our between-group measure considers the proportion of the groups relative to the total population as illustrated in Figure 3.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Dwork et al. first formalized the notion of individual fairness for classification tasks using Lipschitz conditions on the classifier outcomes.

    “Dwork et al. [12] first formalized the notion of individual fairness for classification tasks using Lipschitz conditions on the classifier outcomes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The proposed measure is independent of the specifics of any learning model and can be used to compare unfairness of different algorithms.

    “Since we quantify inequality of algorithmic outcomes, our measure is independent of the specifics of any learning model and can be used to compare unfairness of different algorithms.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Generalized entropy satisfies anonymity, population-invariance, the Pigou-Dalton transfer principle, zero-normalization, subgroup decomposability, and scale-invariance.

    “Generalized entropy satisfies the earlier properties of anonymity, population-invariance, the Pigou-Dalton transfer principle, and zero-normalization.Further it is subgroup decomposable [10], and also scale-invariant.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Existing group fairness notions cannot guarantee fairness at the individual level, or when groups are further refined.

    “Generally, these notions cannot guarantee fairness at the individual level, or when groups are further refined (see Kearns et al. [22] for an illustrative example).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [22]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The proposed measure addresses these issues by offering a merit-based metric of fairness that seeks to equalize the benefit individuals receive as the result of being subject to algorithmic decision making.

    “The measure we propose in equations 1 and 2 addresses these issues by offering a merit-based metric of fairness that seeks to equalize the benefit individuals receive as the result of being subject to algorithmic decision making.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Labels in the training data reflect ground truth, so y_i is the label deserved by individual i.

  • May combine two claims.

    Interpretationreasoning

    The resulting measures of total unfairness unify previous concepts of group and individual fairness and allow quantitative study of the behavior of earlier methods to mitigate unfairness.

    “The resulting measures of total unfairness unify previous concepts of group and individual fairness, and allow us to study quantitatively the behavior of earlier methods to mitigate unfairness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The empirical and theoretical analysis brings out inherent tensions between fairness and accuracy, as well as between different between-group and within-group components of fairness.

    “The empirical (as well as the theoretical) analysis brings out the inherent tensions between fairness and accuracy, as well as between different (between-and within-group) components of fairness.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    As the number of groups increases or treatment within groups becomes more uniform, the between-group component plays a larger role in overall unfairness, but it also becomes computationally harder to control and limit between-group unfairness.

    “On the other hand, as the number of groups increases or the treatment of people within a group becomes more uniform, the role that the between-group component plays in overall unfairness grows -but, as noted by Kearns et al. [22], it also becomes computationally harder to control and limit the between-group unfairness.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kearns et al. [22]

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    The proposed approach is based on well-justified principles (axioms) and is general enough that, by varying the benefit function, it can capture all previous notions of algorithmic fairness conditions as special cases while also admitting interesting generalizations.

    “The approach is based on well-justified principles (axioms), and is general enough so that by varying the benefit function, we can capture all previous notions of algorithmic fairness conditions as special cases, while also admitting interesting generalizations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    For all experiments, the data were repeatedly split into 70%-30% train-test sets 10 times and average statistics were reported; all hyperparameters were validated using a further 70%-30% split of the train set into train and validation sets.

    “For all experiments, we repeatedly split the data into 70%-30% train-test sets 10 times and report average statistics.All hyperparameters are validated using a further 70%-30% split of the train set into train and validation sets.”
    Quote in the paperNot a substantial claim
26 of 26

  • The authors state this as an assumption.

    Labels in the training data reflect ground truth, so y_i is the label deserved by individual i.

    “We assume that labels in the training data reflect ground truth, and thus, y i is the label deserved by individual i.”

  • The paper relies on this without support.

    No prior work in fairness-aware machine learning has studied the effect of constraints designed to remove between-group unfairness on overall and within-group unfairness.

    “Our study is motivated by the fact that while several methods focus on designing constraints to remove the between-group unfairness (e.g., see [17,40,42]), to the best of our knowledge, no prior work in fairness-aware machine learning has studied the effect of these constraints on the overall and the within-group unfairness.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Many approaches to group fairness tackle only between-group issues, as a consequence worsening within-group fairness.

    “[270] note that many approaches to group fairness tackle only between-group issues, as a consequence worsening within-group fairness.”

    From Fairness in machine learning: A survey (2024) · cites this paper as [270]

  • Computing threshold values by hand can introduce new biases without training, as fairness is typically not monotonic, thus assigning a threshold value may be quite arbitrary.

    “Computing by hand can introduce new biases without training [286], as fairness is typically not monotonic, thus assigning a threshold value may be quite arbitrary [270].”

    From Fairness in machine learning: A survey (2024) · cites this paper as [270]

  • Often approaches to group fairness tackle between-group issues, worsening within-group issues through this choice.

    “[270] also note that often approaches to group fairness tackle between-group issues, worsening within-group issues through this choice.”

    From Fairness in machine learning: A survey (2024) · cites this paper as [270]

  • The literature typically hints toward 'optimizing' fairness without transparency of the root(s) of (un)fairness, rarely extending beyond '(un)fair', typically to mirror current legal thought.

    “The literature typically hints toward łoptimizingž fairness without transparency of the root(s) of (un)fairness [192] rarely extending beyond ł(un)fairž [80,270] typically to mirror current legal thought [98].”

    From Fairness in machine learning: A survey (2024) · cites this paper as [270]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Negative true labels are identified as y = 1

    Minor

    The text identifies negative true labels as y = 1 when describing false positive rates, but elsewhere defines false positives as having y = 0.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. Bellù Lorenzo Giovanni, 2006, Inequality analysis: The gini index
  2. Corrado Gini, 1912, Variabilità e mutabilità. Reprinted in Memorie di metodologica statistica
  3. Hardt Moritz, 2016, Proceedings of NIPS. 3315--3323
  4. Scott, 2017, Lundberg and Su-In Lee
  5. Pigou Arthur Cecil, Wealth and welfare
  6. Podesta John, 2014, Big data: Seizing opportunities, preserving values. Executive Office of the President
  7. Amartya Sen, On economic inequality
  8. Sundararajan Mukund, 2017, Proceedings of ICML. 3319--3328
  9. Zemel Rich, 2013, Proceedings of ICML. 325--333

Cited by

Sorted by citations

Related

Authors
Speicher, T., Heidari, H., Grgic-Hlaca, N., Gummadi, K. P., Singla, A., Weller, A., & Zafar, M. B.
Affiliations
  • Max Planck Institute for Software Systems (DE)
  • ETH Zurich (CH)
  • MPI-SWS, Saarbruecken, Germany
  • University of Cambridge (GB)
ORCID
Venue
Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Publisher
ACM
Published
2018-07-19
Pages
2239-2248
DOI
10.1145/3219819.3220046
OpenAlex
W4289751798
Semantic Scholar
d9a62ae14d511362998b1645c1398d3d51202af6
Cited by
207
References
39
Citations by year
2026: 19 · 2025: 27 · 2024: 31 · 2023: 34 · 2022: 30 · 2021: 26 · 2020: 22 · 2019: 18
FWCI
29.20
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Cambridge University Engineering Department Publications Database (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Darwin College, University of Cambridge (David MacKay Newton research fellowship)
  • Leverhulme Trust
  • Alan Turing Institute (EP/N510129/1, TU/B/000074)
  • Innosuisse (27248.1 PFES-ES)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Discrimination and Equality Law
Keywords
algorithmic fairness, algorithmic discrimination, inequality indices, prediction accuracy, algorithmic decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review