The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index; none of the 45 sources its claims cite is checked for support yet.
Fairness testing: testing software for discrimination
Open accessAbstract
This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper defines causal software discrimination and presents Themis, a testing method that automatically generates test suites to measure group and causal discrimination without an oracle. Evaluating 20 instances of 8 financial-decision systems, it finds discrimination even in systems designed to avoid it and shows sound pruning reduces test suites by two-to-three orders of magnitude.
Key findings
- Themis detected causal discrimination missed by group discrimination: for `B census gender`, the causal score was 11.27% versus a group score of 0.52%, more than 21× higher.
- Discrimination was present even in systems designed to avoid it: `B census gender` had a causal discrimination score over 11% for gender, and discriminated against gender, marital status, and race with a causal score of 77.2%; `F census` discriminated against country and race with a causal score of 98.1%.
- There was no clear evidence that discrimination-aware methods outperformed discrimination-unaware ones; the discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.
- Pruning reduced the number of test cases by, on average, a factor of 2,849 for group and 148 for causal discrimination; Themis was more efficient when systems discriminated more, e.g., `B credit gender` needed 3,413 tests at threshold θ = 0.7 but only 10 tests at θ = 0.6.
Methods
- The paper formally defines group, causal, and apparent discrimination, then introduces Themis, which uses an input schema, requires no oracle, and employs test caching, adaptive confidence-driven sampling, and sound pruning based on monotonicity properties.
- Evaluation used 20 instances of 8 software systems making financial decisions: 12 instances from prior discrimination-aware work (4 focused on race, 8 on gender) and 8 standard machine-learning instances that made no attempt to avoid discrimination.
- The datasets were the Adult/Census Income dataset (45K individuals, 14 attributes) and the Statlog German Credit dataset (1,000 individuals, 20 pieces of data); systems were trained on balanced samples of 15.5K census individuals and 600 credit individuals, and Themis measured group and causal discrimination with respect to race and gender, including discrimination-checking experiments with thresholds and pruning comparisons at 99% confidence and 0.05 error bound.
Limitations
- The paper states simplifying assumptions: software is modeled as a black box mapping categorical input characteristics to a categorical output characteristic, and non-categorical inputs/outputs are handled via binning; extending the measures directly to broader data types is left to future work.
- The paper states Themis requires an input schema, and its apparent-discrimination method does not compute confidence because it is only as strong as the developer’s confidence that the test suite or operational profile is representative; apparent discrimination was not directly evaluated.
- Evident from the text: the evaluation is limited to 20 instances of 8 financial-decision systems on two datasets, so the findings may not generalize to other software domains; the paper also leaves integration with combinatorial testing and other automated test-generation improvements to future work.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Own interpretation · 23 Not checked yet · 21 Own result · 5 No basis given · 2 Sources not found · 1 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) Themis optimizations are effective at producing efficient test suites for measuring discrimination”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that (1) Themis is effective at discovering software discrimination”
- Also stated in: § ACMReference format: · p. 2 · plain
“Themis is an effective fairness testing tool.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § ACMReference format: · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, a discrimination-aware decision tree approach trained not to discriminate against gender, B census gender , had a causal discrimination score over 11%: more than 11% of the individuals had the output flipped just by altering the individual's gender.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“demonstrating that (i) even when fairness is a design goal, developers can easily introduce discrimination in software”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As Section 5.4 will evaluate empirically, pruning leads to, on average, a two-to-three order of magnitude reduction in test suite size.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Discrimination is present even in instances designed to avoid discrimination.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for B census gender , the causal score was more than 21 × higher than the group score (11.27% vs. 0.52%).”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Reducing discrimination (CV score) in classifiers [18, 40, 88, 91], as our evaluation has shown, often fails to remove causal discrimination and discrimination against certain groups.”
What the sources say
- Not checked yetcites [18] [18] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In Proceedings of the 2009 IEEE International Conference on Data Mining (ICDM) Workshops , pages 13-18, Miami, FL, USA, December 2009. - source
- Not checked yetcites [40] [40] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In International Conference on Data Mining (ICDM) , pages 869-874, Sydney, Australia, December 2010. - source
- Not checked yetcites [88] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis was able to discover significant discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 32
- Probably not a substantial claim. 30
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, the discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pruning reduces the number of test cases by, on average, a factor of 2 , 849 for group and 148 for causal discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, B census gender discriminates against gender, marital status, and race with a causal score of 77.2%. That means for 77.2% of the individuals, changing only the gender, marital status, or race causes the output of the algorithm to flip.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is the case with B census gender , for example, as its causal discrimination score against gender is 11.2%, but against gender, marital status, and race is 77.2%.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“show that Themis effectively measures discriminations and produces efficient test suites to do so.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Themis is effective at finding both group and causal discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Themis needed 3,413 tests to find sets of characteristics that B credit gender discriminated with a score of more than θ = 0 . 7, but only 10 tests when we reduced θ to 0.6.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle.”
- Also stated in: § ACMReference format: · p. 2 · plain
“Themis requires a schema for generating inputs, but does not require an oracle.”
- Also stated in: § ACMReference format: · p. 2 · plain
“Unlike prior work that requires manually written tests [79], Themis automatically generates test suites that measure discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § ACMReference format: · p. 2 · plain
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Amazon.com, Inc. software that determined where to offer free same-day delivery did not explicitly consider race but made racecorrelated decisions because of correlations between race and other input characteristics [36, 52].”
What the sources say
- Not matchedcites [36] [36] David Ingold and Spencer Soper. Amazon doesn't consider the race of its customers. Should it? Bloomberg , April 21, 2016. http://www.bloomberg.com/ graphics/2016-amazon-same-day. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Test caching has low storage overhead and offers significant runtime gains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The output domain distance function generalizes our work beyond binary outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These approaches do not capture causality and can miss discrimination that our causal approach detects, e.g., when the software discriminates negatively with respect to a group in one settings, but positively in another.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis is effective. Themis is able to (1) verify that many of the software instances do not discriminate against race and gender, and (2) identify the software that does.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In forecasting who would reoffend, the software is 'particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants; white defendants were mislabeled as low risk more often than black defendants' [5].”
What the sources say
- Not checked yetcites [5] [5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica , May 23, 2016. https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing. -
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, for example, while the group and apparent discrimination scores penalize software that gives loans to different fractions of individuals of different races, the causal discrimination score penalizes software that gives loans to individuals of one race but not to otherwise identical individuals of another race.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Our causal fairness definition is designed specifically to be testable, unlike definitions that require probabilistic estimation or knowledge of the future [38].”
What the sources say
- Not checked yetcites [38] [38] Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. CoRR , abs/1207.0016, 2012. https: //arxiv.org/abs/1605.07139. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The causal discrimination score identifies changing which characteristics directly affects the output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our definition extends to the more broad categorical input types, reflecting the relative complexity of arbitrary decision software.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our definition captures causal relationships between inputs and outputs”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis could have analyzed the software and detected the discrimination prior to deployment.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Operational profiles [58] describe the input distributions likely to be observed in the field.”
What the sources say
- Not checked yetcites [58] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Our group discrimination score generalizes the CV score to the software domain with more complex inputs.”
What the sources say
- Not checked yetcites [19] [19] Toon Calders and Sicco Verwer. Three naive Bayes approaches for discriminationfree classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“operational profiles can significantly improve the effectiveness of testing by more accurately representing real-world system use [7, 50]”
What the sources say
- Not checked yetcites [7] [7] Andrea Arcuri and Lionel Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE) , pages 1-10, Honolulu, HI, USA, 2011. -
- Not checked yetcites [50] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“the use of operational profiles has been shown to more accurately measure system properties, such as reliability [35].”
What the sources say
- Not matchedcites [35] [35] Herman Hartmann. A statistical analysis of operational profile driven testing. In International Conference on Software Quality, Reliability, and Security Companion (QRS-C) , pages 109-116, Viena, Austria, August 2016. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis measures apparent discrimination by either executing a provided test suite, or by generating a test suite following a provided operational profile.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis can also measure apparent discrimination (Definition 3.8) via a developer-provided test suite or operational profile.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our work does not attempt to remove discrimination but offers developers a tool to identify and measure discrimination”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“input characteristics may correlate, e.g., education correlates with age, so perturbing some characteristics without perturbing others may create inputs not representative of the real world.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have further described Themis, an approach and its open-source implementation for measuring discrimination in software and for generating efficient test suites to perform these measurements.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, focusing on avoiding discriminating against one characteristic may create discrimination against another, e.g., B census gender limits discrimination against gender but discriminates against race with a causal score of 38.40%.”
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“FairTest, an implementation of the unwarranted associations framework [79] uses manually written tests to measure four kinds of discrimination scores: the CV score and a related ratio, mutual information, Pearson correlation, and a regression between the output and sensitive inputs.”
What the sources say
- Not checked yetcites [79]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The B approach was designed to avoid a variant of group discrimination (as are other discrimination-aware approaches), but this design is, at least in some conditions, insufficient to prevent causal discrimination.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“FairML [1] uses orthogonal projection to co-perturb characteristics, which can mask some discrimination, but find discrimination that is more likely to be observed in real-world scenarios”
What the sources say
- Not checked yetcites [1] [1] Julius Adebayo and Lalana Kagal. Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR , abs/1611.04967, 2016. -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Counterfactual fairness is related to causal fairness but can miss some instances of discrimination, e.g., if loan shows preferential treatment for some purple inputs, but at the same time against some other similar purple inputs.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This experiment demonstrates that Themis effectively discovers discrimination and can test software for unexpected software discrimination effects across a wide variety of input partitions.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The work on operational profiles is complementary to ours: Themis uses operational profiles and work on more efficient test generation from operational profiles can directly benefit discrimination testing.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have formally defined software fairness testing and introduced a causality-based measure of discrimination.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The number of tests needed to evaluate every possible value pair can be significantly smaller than the exhaustive testing alternative since each test can simultaneously contribute to multiple value pairs [22, 44, 80].”
What the sources say
- Not checked yetcites [22] [22] Marcelo d'Amorim, Steven Lauterburg, and Darko Marinov. Delta execution for efficient state-space exploration of object-oriented programs. In International Symposium on Software Testing and Analysis (ISSTA) , pages 50-60, London, UK, 2007. - source
- Not checked yetcites [44] [44] Christian Kästner, Alexander von Rhein, Sebastian Erdweg, Jonas Pusch, Sven Apel, Tillmann Rendel, and Klaus Ostermann. Toward variability-aware testing. In International Workshop on Feature-Oriented Software Development (FOSD) , pages 1-8, Dresden, Germany, 2012. - source
- Not checked yetcites [80] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Counterfactual fairness requires output probability distributions to match for input populations that differ only in the label value of a sensitive input characteristic [51].”
What the sources say
- Not checked yetcites [51] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The goal of the generated tests is typically finding bugs [29] or generating specifications [21].”
What the sources say
- Not checked yetcites [29] [29] Gordon Fraser and Andrea Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering (TSE) , 39(2):276-291, February 2013. - source
- Not checked yetcites [21] [21] Valentin Dallmeier, Nikolai Knopp, Christoph Mallon, Gordon Fraser, Sebastian Hack, and Andreas Zeller. Automatically generating test cases for specification mining. IEEE Transactions on Software Engineering (TSE) , 38(2):243-257, March 2012. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our approach generates tests automatically and measures causal discrimination.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“AdFisher computes a variant of group discrimination, but it could be integrated with Themis and its algorithms to measure causal discrimination.”
What the sources say
- Not checked yetcites [23] [23] Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. Proceedings on Privacy Enhancing Technologies (PoPETs) , 1:92-112, 2015. - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our main goal is to develop a method to process test executions to measure software discrimination, whereas that is not a goal of combinatorial testing.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Causal relationships in data management systems [54, 55] can help explain query results [57] and debug errors [82-84] by tracking and using data provenance [56].”
What the sources say
- Not checked yetcites [54]
- Not checked yetcites [55]
- Not checked yetcites [57]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Our prior work on testing software that relies on data management systems has focused on data errors [59, 60], whereas this work focuses on testing fairness.”
What the sources say
- Not checked yetcites [59]
- Not checked yetcites [60] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These tools deal with more complex input spaces than Themis, but none of them focus on testing fairness and they require oracles whereas Themis does not need oracles as it measures discrimination by comparing tests' outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: None of the automated test generation tools focus on testing fairness.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our oracle is more precise and easier to compute, but is only applicable to fairness testing.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The authors state this as an assumption.
Software can be modeled as a black box that maps input characteristics to an output characteristic.
“First, we define software as a black box that maps input characteristics to an output characteristic.”
The authors state this as an assumption.
Input characteristics and the output characteristic are categorical variables.
“Second, we assume that the input characteristics and the output characteristic are categorical variables, each having a set of possible values (e.g., race, gender, eye color, age ranges, income ranges).”
The authors state this as an assumption.
For the remainder of the paper, software outputs binary decisions.
“For simplicity of exposition, for the remainder of this paper, we assume software outputs binary decisions - a natural domain for fairness testing.”
The paper relies on this without support.
No prior work automatically generates test suites to measure causal discrimination in software.
“To the best of our knowledge, this work is the first to automatically generate test suites to measure causal discrimination in software.”
“We are aware of no prior testing technique, neither automated nor manual, that produces a measure of a software system's causal discrimination.”
The authors state this as an assumption.
Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
“decreasing θ effectively simulates increasing system discrimination.”
The paper relies on this without support.
No prior work on operational profile testing has measured software discrimination.
“Meanwhile no prior work on operational profile testing has measured software discrimination.”
The paper relies on this without support.
None of the automated test generation tools focus on testing fairness.
“none of them focus on testing fairness”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Thus, causal methods are speciically well suited to identifying 1) proxies of sensitive variables as discussed in subsection 2.1; 2) which subgroups are most (un)fairly treated and diferentiate the types of bias exhibited [118]; and 3) provide transparency with respect to how (classiication) decisions were made [143].”
From Fairness in machine learning: A survey (2024) · cites this paper as [113]
In the citing paper
“As a result, a classifier trained without regard to the protected attribute not only fails to remove discrimination, but it can complicate the detection and mitigation of discrimination downstream via in-processing or post-processing techniques [9,10,18,24,25,34,43,51], such as those we describe next.”
From Interventional Fairness: Causal Database Repair for Algorithmic Fairness (2019) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.2 Race and Gender Discrimination · p. 9
“| A census race | 2.10% | 1.13% | 8.90% | 7.20% |”
§ 4.2 Sound Pruning · p. 7
“Theorem 4.3. Let X be an input type and let S be a decision software with input type X . Then for all sets of characteristics X ′ ⊆ X , ˜ d X ′ ( S ) ≤ ⃗ d X ′ ( S ) .”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“nition of discrimination is that a “better” input is never deprived of the “better” output [24].”
BackgroundInternational Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 4×
“Our causal discrimination score is a stronger measure of discrimination, as it seeks out causality in software, measuring the fraction of inputs for which changing specific input characteristics causes the output to change [67].”
BackgroundInfluential(2013) · Cited by 1,126
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104 · cited 1×
“While prior work has tackled the oracle problem [10, 26, 30] typically using inferred pre- and post-conditions or documentation, our oracle is more precise and easier to compute, but is only applicable to fairness testing.”
BackgroundKusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Arcuri, A., & Briand, L. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985795 · Cited by 989 · cited 4×
“Apparent discrimination captures this notion, applying group or causal discrimination score measurement to a subset of the input domain, which can be described by an operational profile [7, 58].”
BackgroundMethodInfluentialFeedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2007.37 · Cited by 879 · cited 1×
“Automated testing research has produced tools to generate tests, including random testing, such as Randoop [63, 64], NightHawk [4], JCrasher [20], CarFast [65], and T3 [69]; search-based testing, such as EvoSuite [29], TestFul [9], and eToc [78]; dynamic symbolic execution tools, such as DSC [37], Symbolic PathFinder…”
Method
Research works printed but not linked: 8
- Agitar Technologies . AgitarOne. http://www.agitar.com/solutions/products/ automated_junit_generation.html , 2016 . Agitar Technologies. AgitarOne. http://www.agitar.com/solutions/products/ automated_junit_generation.html, 2016.
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . Machine bias . ProPublica , May 23, 2016 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May 23, 2016.
- 10.5555/1792786.1792789 · DOI
- Katelyn Ferral . Wisconsin supreme court allows state to continue using computer program to assist in sentencing . The Capital Times , July 13, 2016 . http://host.madison.com/ct/news/local/govt-and-politics/ wisconsin-supreme-court-allows-state-to-continue-using-computer-program/ article_7eb67874-bf40-59e3-b62a-923d1626fa0f.html. Katelyn Ferral. Wisconsin supreme court allows state to continue using computer program to assist in sentencing. The Capital Times, July 13, 2016. http://host.madison.com/ct/news/local/govt-and-politics/ wisconsin-supreme-court-allows-state-to-continue-using-computer-program/ article_7eb67874-bf40-59e3-b62a-923d1626fa0f.html.
- Rafi Letzter . Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist . TECH Insider , April 21, 2016 . http://www.techinsider.io/ how-algorithms-can-be-racist- 2016 - 2014 . Rafi Letzter. Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist. TECH Insider, April 21, 2016. http://www.techinsider.io/ how-algorithms-can-be-racist-2016-4.
- Satya Nadella . The partnership of the future . Slate , June 28, 2016 . http: //www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_ satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html. Satya Nadella. The partnership of the future. Slate, June 28, 2016. http: //www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_ satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html.
- Aarti Shahani . Now algorithms are deciding whom to hire, based on voice . NPR All Things Considered , March 2015 . Aarti Shahani. Now algorithms are deciding whom to hire, based on voice. NPR All Things Considered, March 2015.
- 10.5555/1792786.1792798 · DOI
Cited by
Sorted by citations
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124
“Then, statistical parity will deem the classifier fair, despite a discrepancy in how the applications are processed based on gender [13].”
BackgroundMachine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
“As the field matures, integrated toolkits are being developed, with the aim of making these methods more widely accessible and usable [7, 28, 35, 42, 70].”
BackgroundInfluentialAlgorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
“…in technical review articles (Berk et al. 2017, Chouldechova & Roth 2018, Friedler et al. 2018, Verma & Rubin 2018) and an in-progress textbook (Barocas et al. 2018), and has inspired a number of software packages (Galhotra et al. 2017, Zehlike et al. 2017, Angell et al. 2018, Wexler 2018).”
Background
Metadata
- Authors
- Galhotra, S., Brun, Y., & Meliou, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2017-08-21
- Pages
- 498-510
- OpenAlex
- W2730550703
- Cited by
- 405
- References
- 91
- Citations by year
- 2026: 26 · 2025: 55 · 2024: 46 · 2023: 56 · 2022: 50 · 2021: 53 · 2020: 57 · 2019: 37 · 2018: 22 · 2017: 3
- FWCI
- 23.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-1453474, IIS-1453543, CNS-1744471)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic discrimination, fairness testing, automated test generation, THEMIS, algorithmic fairness, automated decision systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
