CoChat
Sign Up
Cited by 179Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Game Theory and Voting Systems

Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning

Open access
Srivastava, Megha, Heidari, Hoda, Krause, Andreas·2019·Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
DOI 10.1145/3292500.3330664
Open this paper in CoChat — free accountOpen PDF

Abstract

Fairness for Machine Learning has received considerable attention, recently. Various mathematical formulations of fairness have been proposed, and it has been shown that it is impossible to satisfy all of them simultaneously. The literature so far has dealt with these impossibility results by quantifying the tradeoffs between different formulations of fairness. Our work takes a different perspective on this issue. Rather than requiring all notions of fairness to (partially) hold at the same time, we ask which one of them is the most appropriate given the societal domain in which the decision-making model is to be deployed. We take a descriptive approach and set out to identify the notion of fairness that best captures lay people's perception of fairness. We run adaptive experiments designed to pinpoint the most compatible notion of fairness with each participant's choices through a small number of tests. Perhaps surprisingly, we find that the most simplistic mathematical definition of fairness---namely, demographic parity---most closely matches people's idea of fairness in two distinct application scenarios. This conclusion remains intact even when we explicitly tell the participants about the alternative, more complicated definitions of fairness, and we reduce the cognitive burden of evaluating those notions for them. Our findings have important implications for the Fair ML literature and the discourse on formalizing algorithmic fairness.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper uses adaptive human-subject experiments to ask which mathematical fairness notion best matches lay people’s fairness perceptions in two contexts: criminal risk prediction and skin cancer risk prediction. It finds that demographic parity, the simplest notion tested, most closely matches participants’ choices in both scenarios, even when alternative notions are explicitly explained.

Key findings

  • Demographic parity (DP) best captured the majority of participants’ choices in both scenarios: for crime risk prediction, 80% of participants were matched with DP at high likelihood (>80%); for cancer risk prediction, 73% were matched with DP.
  • In a smaller structured-explanation interface condition, DP still dominated: 17 out of 20 participants for crime prediction and 9 out of 20 for cancer prediction were matched with DP.
  • Participants reported no significant variation across demographic attributes; for crime risk prediction, high-likelihood DP rates were 78% for female and 79% for male participants, 82% for Caucasian and 72% for non-Caucasian participants, 80% for liberal and 74% for conservative participants, 77% for those without a college/university degree and 79% for the rest, and 78% for young participants (age <40) and 81% for older participants.
  • In the separate survey testing H3, when stakes were high, participants gave higher weight to accuracy and lower weight to inequality, as hypothesized.

Methods

  • Adaptive experiments on Amazon Mechanical Turk (AMT) with 100 participants per scenario for criminal risk prediction and skin cancer risk prediction; each participant answered at most 20 adaptively chosen tests.
  • Each test showed predictions from two hypothetical algorithms for ten hypothetical individuals along with true labels; algorithms had equal overall accuracy, and participants chose which algorithm they considered more discriminatory.
  • The EC² active-learning algorithm selected tests to identify the most compatible fairness notion among Demographic Parity, Error Parity, False Discovery rate Parity, and False Negative rate Parity, using a Generalized Entropy inequality index; a separate survey presented three algorithms with accuracy/equality tradeoffs to test H3.

Limitations

  • The paper states that it assumes at least one existing mathematical fairness notion represents human judgment in a highly stylized setting, while real-world scenarios are more complex and existing notions are simplistic.
  • The paper states concerns about AMT engagement and attention, mitigated by restricting to 99% approval rate, limiting tests to 20, requiring explanations, and allowing participation only once; it also notes framing effects cannot be completely ruled out, participants had no personal stakes, and tests used models of similar accuracy.
  • The paper states that AMT workers are not a representative sample of the U.S. population; its participants were younger and more liberal than the average U.S. population, as shown in Table 2.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3292500.3330664

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 8 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the structured explanation interface condition for the crime prediction scenario, 17 out of 20 participants were matched with demographic parity.

    “For the crime prediction scenario, 17 out of 20 participants were matched with DP, and for the cancer prediction scenario, this number was 9 out of 20.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is a uniform prior over the four fairness notions h1, h2, h3, h4.
    • The authors state this as an assumption: The cost is uniform across all tests.
    • The authors state this as an assumption: The probability of an individual following fairness notion h_i selecting algorithm A1 is given by a softmax over the Generalized Entropy indices of the two algorithms' benefit vectors.
  • Findingown data

    With a random test sequence, at least 600 tests are needed before a high likelihood for one hypothesis can be obtained.

    “With a random test sequence, at least 600 tests are needed before we can obtain a high likelihood for one hypothesis.”
    Quote in the paperSubstantial claim
  • Findingown data

    With adaptive test selection, only around 20 tests are needed to assign a high likelihood to the fairness notion the participant is following.

    “With our adaptive test selection method, only around 20 tests are needed to assign a high likelihood to the fairness notion the participant is following.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the structured explanation interface condition for the cancer prediction scenario, 9 out of 20 participants were matched with demographic parity.

    “For the crime prediction scenario, 17 out of 20 participants were matched with DP, and for the cancer prediction scenario, this number was 9 out of 20.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is a uniform prior over the four fairness notions h1, h2, h3, h4.
    • The authors state this as an assumption: The cost is uniform across all tests.
    • The authors state this as an assumption: The probability of an individual following fairness notion h_i selecting algorithm A1 is given by a softmax over the Generalized Entropy indices of the two algorithms' benefit vectors.
  • Findingown data

    The conclusion that demographic parity best matches participants' fairness perceptions remains intact when participants are explicitly told about alternative, more complicated definitions of fairness and the cognitive burden of evaluating those notions is reduced.

    “This conclusion remains intact even when we explicitly tell the participants about the alternative, more complicated definitions of fairness, and we reduce the cognitive burden of evaluating those notions for them.”
    • Also stated in: § 1 INTRODUCTION · plain
      “This remains the case even when subjects are explicitly told about the alternative, more complicated definitions of fairness and the cognitive burden of evaluating those notions is reduced for them.”
    • Also stated in: § 0% 24% · plain
      “We observe that even under this new condition the majority of participants made choices that are best captured by demographic parity.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is a uniform prior over the four fairness notions h1, h2, h3, h4.
    • The authors state this as an assumption: The cost is uniform across all tests.
    • The authors state this as an assumption: The probability of an individual following fairness notion h_i selecting algorithm A1 is given by a softmax over the Generalized Entropy indices of the two algorithms' benefit vectors.
  • May combine two claims. 10
  • The quote only partly states this claim. 2
  • Classified as probably not a substantial claim. 12

  • May combine two claims.

    Findingown data

    The AMT pilot participants found the first user interface, with text explanations, less restrictive and easier to work with than the structured interface.

    “The AMT pilot participants found our first user interface (the one with text explanations) less restrictive and easier to work with, so we scaled up our experiments to 100 crowd workers per scenario using that interface.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In the skin cancer risk prediction scenario, trends are similar to the crime risk scenario, with demographic parity best capturing participants' choices.

    “Trends are similar for the cancer risk prediction scenario.See Table 3 for the summary.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: There is a uniform prior over the four fairness notions h1, h2, h3, h4.
    • The authors state this as an assumption: The cost is uniform across all tests.
    • The authors state this as an assumption: The probability of an individual following fairness notion h_i selecting algorithm A1 is given by a softmax over the Generalized Entropy indices of the two algorithms' benefit vectors.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participants in the experiments were younger and more liberal compared to the average U.S. population.

    “In particular, for our experiments, participants were younger and more liberal compared to the average U.S. population.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some participants thought more context about the algorithm and decision subjects would have helped them make a more informed choice.

    “some thought more context about the algorithm and decision subjects would have helped them make a more informed choice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The task encouraged users to think about what factors algorithmic decision-making systems should take into account and to reflect about the use of discriminatory algorithms in society.

    “Feedback from the users, such as the examples shown in Table 6, demonstrate that our task encouraged users to think about what factors algorithmic decision-making systems should take into account, as well as reflect about the use of discriminatory algorithms in society.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Demographic parity captures the choices made by the majority of participants in the cancer risk prediction scenario.

    “Demographic parity captures the choices made by the majority of participants.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The conditional independence assumption used in the Bayesian model holds.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participants on Amazon Mechanical Turk found the task engaging and informative.

    “participants on Amazon Mechanical Turk found the task engaging and informative”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The study limited attention to tests consisting of algorithms with equal overall accuracy to control for the impact of accuracy on people's perception.

    “We limited attention to tests that consist of algorithms with equal overall accuracy to control for the impact of accuracy on people's perception.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Mathematical definitions of fairness are incompatible with one another and cannot hold simultaneously.

    “While each of these notions is appealing on its own, it has been shown that they are incompatible with one another and cannot hold simultaneously [6,19].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [6]

      Not checked in the free run.

    • cites [19] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The EC2 algorithm can handle noisy observations and its expected cost is competitive with that of the optimal sequential policy.

    “The algorithm, called EC 2 (for Equivalence Class Edge Cutting algorithm), can handle noisy observations and its expected cost is competitive with that of the optimal sequential policy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Golovin et al. [12]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The interface design was validated through two rounds of pilot studies, one internal and one on AMT among 20 crowd workers.

    “We validated our interface design through two rounds of pilot studies-one internal (among members of our research group) and one on AMT (among 20 crowd workers).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The EC2 algorithm allowed the study to determine the most compatible fairness notion with each participant's choices, employing at most 20 tests per participant.

    “This allowed us to determine the most compatible fairness notion with each participant's choices, employing at most 20 tests per participant (see Figure 3).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: There is a uniform prior over the four fairness notions h1, h2, h3, h4.
    • The authors state this as an assumption: The cost is uniform across all tests.
    • The authors state this as an assumption: The probability of an individual following fairness notion h_i selecting algorithm A1 is given by a softmax over the Generalized Entropy indices of the two algorithms' benefit vectors.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Saxena et al. investigated ordinary people's attitude toward three notions of individual fairness in loan decisions and showed that people exhibit a preference for the fairness definition similar in essence to calibration.

    “They show that people exhibit a preference for the last fairness definition-which is similar, in essence, to calibration.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Saxena et al. [26]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The literature so far has dealt with impossibility results by quantifying the tradeoffs between different formulations of fairness.

    “The literature so far has dealt with these impossibility results by attempting to quantify the tradeoffs between different formulations of fairness, hoping that practitioners will be better positioned to determine the extent to which the violation of each fairness criterion can be socially tolerated (see, e.g., [7]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All hypotheses were defined and registered with the authors' institution in advance of the experiments.

    “All hypotheses were defined and registered with our institution in advance of the experiments.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The authors prevented participants from choosing answers without careful consideration by restricting participation to turkers with 99% approval rate, limiting the number of tests to at most 20, requiring an explanation for their choice, and restricting participants to complete the task at most once.

    “We prevented this by • Restricting participation to turkers with 99% approval rate. • Limiting the number of tests to at most 20. • Requiring the participant to provide an explanation for their choice. • Restricting participants to complete the task at most once.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    There is no consensus on a precise definition of (un)fairness.

    “Despite the recent surge of interest in Fair ML, there is no consensus on a precise definition of (un)fairness.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [9] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [19] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The paper's framework can be readily adapted to scenarios beyond the ones studied here.

    “Our framework can be readily adapted to scenarios beyond the ones studied here.”
    Quote in the paperNot a substantial claim
18 of 18

  • The authors state this as an assumption.

    There is a uniform prior over the four fairness notions h1, h2, h3, h4.

    “We assume a uniform prior over h 1 , • • • , h 4 .”

  • The authors state this as an assumption.

    The cost is uniform across all tests.

    “We assume the cost is uniform across all tests.”

  • The authors state this as an assumption.

    The probability of an individual following fairness notion h_i selecting algorithm A1 is given by a softmax over the Generalized Entropy indices of the two algorithms' benefit vectors.

    “We assume 6 that the probability of an individual following h i selecting A 1 is as follows:”

  • The authors state this as an assumption.

    Fairness is a highly context-dependent ideal, and depending on the societal domain in which the decision-making model is deployed, one mathematical notion of fairness may be considered ethically more desirable than other alternatives.

    “We posit that fairness is a highly context-dependent ideal and depending on the societal domain in which the decision-making model is deployed, one mathematical notion of fairness may be considered ethically more desirable than other alternatives.”

  • The paper relies on this without support.

    There is no simple, widely-accepted, normative principle to settle the ethical problem of algorithmic fairness.

    “We emphasize that there is no simple, widely-accepted, normative principle to settle the ethical problem of algorithmic fairness.”

  • The authors state this as an assumption.

    At least one of the existing mathematical notions of fairness is a good representation of human judgment, at least in the highly stylized setting presented to participants.

    “We took it as a given that at least one of these notions is a good representation of human judgment-at least in the highly stylized setting we presented them with.”

  • The authors state this as an assumption.

    The conditional independence assumption used in the Bayesian model holds.

    “We assume conditional independence, that is, P(o|h) = o ∈o P(o|h).”

  • The paper relies on this without support.

    The Amazon Mechanical Turk participants are representative of the general population for the purpose of inferring people's perception of fairness.

    “Demographics of our AMT participants compared to the 2016 U.S. census data.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There are differing views of fairness and bias among people.

    “Further adding to this discussion is the difering views of fairness and bias.[131,236,237,271] study the difering views of people in this regard and observe that this is not a trivial challenge to address.”

    From Fairness in machine learning: A survey (2024) · cites this paper as [271]

The citation lists haven’t been fetched yet

References 30 · Cited by 179

Research works printed but not linked: 11

  1. Angwin Julia, 2016, Machine Bias. Propublica
  2. Awad Edmond, 2018, Jean-Francc ois Bonnefon, and Iyad Rahwan
  3. Barry-Jester Anna, 2015, The New Science of Sentencing. The Marshall Project (August
  4. Buolamwini Joy, 2018, In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT*). 77--91
  5. Chouldechova Alexandra, 2017, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. arXiv preprint arXiv:1703.00056 · arXiv
  6. Daniel Golovin Andreas Krause and Debajyoti Ray. 2010. Near-optimal bayesian active learning with noisy observations. In NIPS . 766--774. Daniel Golovin Andreas Krause and Debajyoti Ray. 2010. Near-optimal bayesian active learning with noisy observations. In NIPS . 766--774.
  7. Hart Robert David, 2017, If you're not a white male, artificial intelligence's use in healthcare could be dangerous. Quartz (July
  8. Lee Min Kyung, 2018, Xinran Yuan, Allissa Chan, Ritesh Noothigattu, Daniel See, Siheon Lee, and Christos-Alexandros Psomas.
  9. Noothigattu Ritesh, 2018, Procaccia
  10. Petrasic Kevin, 2017, Algorithms and bias: What lenders need to know. White & Case
  11. Rudin Cynthia, 2013, Predictive Policing Using Machine Learning to Detect Patterns of Crime. Wired Magazine (August

Authors
Srivastava, M., Heidari, H., & Krause, A.
Affiliations
  • Stanford University (US)
  • ETH Zurich (CH)
ORCID
Venue
Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Publisher
ACM
Published
2019-07-25
Pages
2459-2468
DOI
10.1145/3292500.3330664
OpenAlex
W2914202940
Semantic Scholar
784c51d2a4f51d67219bef617cce40102bb513e7
Cited by
179
References
30
Citations by year
2026: 11 · 2025: 31 · 2024: 33 · 2023: 25 · 2022: 28 · 2021: 39 · 2020: 10 · 2019: 2
FWCI
20.61
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
Funders
  • National Science Foundation (DGE-1656518)
  • Eidgenössische Technische Hochschule Zürich
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
Keywords
algorithmic fairness, fairness definitions, demographic parity, impossibility results, adaptive experiments

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review