The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index; none of the 45 sources its claims cite is checked for support yet.
Fairness testing: testing software for discrimination
Open accessAbstract
This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper defines causal software discrimination and presents Themis, a testing method that automatically generates test suites to measure group and causal discrimination without an oracle. Evaluating 20 instances of 8 financial-decision systems, it finds discrimination even in systems designed to avoid it and shows sound pruning reduces test suites by two-to-three orders of magnitude.
Key findings
- Themis detected causal discrimination missed by group discrimination: for `B census gender`, the causal score was 11.27% versus a group score of 0.52%, more than 21× higher.
- Discrimination was present even in systems designed to avoid it: `B census gender` had a causal discrimination score over 11% for gender, and discriminated against gender, marital status, and race with a causal score of 77.2%; `F census` discriminated against country and race with a causal score of 98.1%.
- There was no clear evidence that discrimination-aware methods outperformed discrimination-unaware ones; the discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.
- Pruning reduced the number of test cases by, on average, a factor of 2,849 for group and 148 for causal discrimination; Themis was more efficient when systems discriminated more, e.g., `B credit gender` needed 3,413 tests at threshold θ = 0.7 but only 10 tests at θ = 0.6.
Methods
- The paper formally defines group, causal, and apparent discrimination, then introduces Themis, which uses an input schema, requires no oracle, and employs test caching, adaptive confidence-driven sampling, and sound pruning based on monotonicity properties.
- Evaluation used 20 instances of 8 software systems making financial decisions: 12 instances from prior discrimination-aware work (4 focused on race, 8 on gender) and 8 standard machine-learning instances that made no attempt to avoid discrimination.
- The datasets were the Adult/Census Income dataset (45K individuals, 14 attributes) and the Statlog German Credit dataset (1,000 individuals, 20 pieces of data); systems were trained on balanced samples of 15.5K census individuals and 600 credit individuals, and Themis measured group and causal discrimination with respect to race and gender, including discrimination-checking experiments with thresholds and pruning comparisons at 99% confidence and 0.05 error bound.
Limitations
- The paper states simplifying assumptions: software is modeled as a black box mapping categorical input characteristics to a categorical output characteristic, and non-categorical inputs/outputs are handled via binning; extending the measures directly to broader data types is left to future work.
- The paper states Themis requires an input schema, and its apparent-discrimination method does not compute confidence because it is only as strong as the developer’s confidence that the test suite or operational profile is representative; apparent discrimination was not directly evaluated.
- Evident from the text: the evaluation is limited to 20 instances of 8 financial-decision systems on two datasets, so the findings may not generalize to other software domains; the paper also leaves integration with combinatorial testing and other automated test-generation improvements to future work.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Own interpretation · 23 Not checked yet · 21 Own result · 5 No basis given · 2 Sources not found · 1 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) Themis optimizations are effective at producing efficient test suites for measuring discrimination”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that (1) Themis is effective at discovering software discrimination”
- Also stated in: § ACMReference format: · p. 2 · plain
“Themis is an effective fairness testing tool.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § ACMReference format: · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, a discrimination-aware decision tree approach trained not to discriminate against gender, B census gender , had a causal discrimination score over 11%: more than 11% of the individuals had the output flipped just by altering the individual's gender.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“demonstrating that (i) even when fairness is a design goal, developers can easily introduce discrimination in software”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As Section 5.4 will evaluate empirically, pruning leads to, on average, a two-to-three order of magnitude reduction in test suite size.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Discrimination is present even in instances designed to avoid discrimination.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for B census gender , the causal score was more than 21 × higher than the group score (11.27% vs. 0.52%).”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Reducing discrimination (CV score) in classifiers [18, 40, 88, 91], as our evaluation has shown, often fails to remove causal discrimination and discrimination against certain groups.”
What the sources say
- Not checked yetcites [18] [18] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In Proceedings of the 2009 IEEE International Conference on Data Mining (ICDM) Workshops , pages 13-18, Miami, FL, USA, December 2009. - source
- Not checked yetcites [40] [40] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In International Conference on Data Mining (ICDM) , pages 869-874, Sydney, Australia, December 2010. - source
- Not checked yetcites [88] source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis was able to discover significant discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 32
- Probably not a substantial claim. 30
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, the discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pruning reduces the number of test cases by, on average, a factor of 2 , 849 for group and 148 for causal discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, B census gender discriminates against gender, marital status, and race with a causal score of 77.2%. That means for 77.2% of the individuals, changing only the gender, marital status, or race causes the output of the algorithm to flip.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is the case with B census gender , for example, as its causal discrimination score against gender is 11.2%, but against gender, marital status, and race is 77.2%.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“show that Themis effectively measures discriminations and produces efficient test suites to do so.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Themis is effective at finding both group and causal discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Themis needed 3,413 tests to find sets of characteristics that B credit gender discriminated with a score of more than θ = 0 . 7, but only 10 tests when we reduced θ to 0.6.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle.”
- Also stated in: § ACMReference format: · p. 2 · plain
“Themis requires a schema for generating inputs, but does not require an oracle.”
- Also stated in: § ACMReference format: · p. 2 · plain
“Unlike prior work that requires manually written tests [79], Themis automatically generates test suites that measure discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § ACMReference format: · p. 2 · plain
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Amazon.com, Inc. software that determined where to offer free same-day delivery did not explicitly consider race but made racecorrelated decisions because of correlations between race and other input characteristics [36, 52].”
What the sources say
- Not matchedcites [36] [36] David Ingold and Spencer Soper. Amazon doesn't consider the race of its customers. Should it? Bloomberg , April 21, 2016. http://www.bloomberg.com/ graphics/2016-amazon-same-day. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Test caching has low storage overhead and offers significant runtime gains.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The output domain distance function generalizes our work beyond binary outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These approaches do not capture causality and can miss discrimination that our causal approach detects, e.g., when the software discriminates negatively with respect to a group in one settings, but positively in another.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis is effective. Themis is able to (1) verify that many of the software instances do not discriminate against race and gender, and (2) identify the software that does.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In forecasting who would reoffend, the software is 'particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants; white defendants were mislabeled as low risk more often than black defendants' [5].”
What the sources say
- Not checked yetcites [5] [5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica , May 23, 2016. https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing. -
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, for example, while the group and apparent discrimination scores penalize software that gives loans to different fractions of individuals of different races, the causal discrimination score penalizes software that gives loans to individuals of one race but not to otherwise identical individuals of another race.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Our causal fairness definition is designed specifically to be testable, unlike definitions that require probabilistic estimation or knowledge of the future [38].”
What the sources say
- Not checked yetcites [38] [38] Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. CoRR , abs/1207.0016, 2012. https: //arxiv.org/abs/1605.07139. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The causal discrimination score identifies changing which characteristics directly affects the output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our definition extends to the more broad categorical input types, reflecting the relative complexity of arbitrary decision software.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our definition captures causal relationships between inputs and outputs”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis could have analyzed the software and detected the discrimination prior to deployment.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Operational profiles [58] describe the input distributions likely to be observed in the field.”
What the sources say
- Not checked yetcites [58] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Our group discrimination score generalizes the CV score to the software domain with more complex inputs.”
What the sources say
- Not checked yetcites [19] [19] Toon Calders and Sicco Verwer. Three naive Bayes approaches for discriminationfree classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“operational profiles can significantly improve the effectiveness of testing by more accurately representing real-world system use [7, 50]”
What the sources say
- Not checked yetcites [7] [7] Andrea Arcuri and Lionel Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE) , pages 1-10, Honolulu, HI, USA, 2011. -
- Not checked yetcites [50] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“the use of operational profiles has been shown to more accurately measure system properties, such as reliability [35].”
What the sources say
- Not matchedcites [35] [35] Herman Hartmann. A statistical analysis of operational profile driven testing. In International Conference on Software Quality, Reliability, and Security Companion (QRS-C) , pages 109-116, Viena, Austria, August 2016. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis measures apparent discrimination by either executing a provided test suite, or by generating a test suite following a provided operational profile.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Themis can also measure apparent discrimination (Definition 3.8) via a developer-provided test suite or operational profile.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our work does not attempt to remove discrimination but offers developers a tool to identify and measure discrimination”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“input characteristics may correlate, e.g., education correlates with age, so perturbing some characteristics without perturbing others may create inputs not representative of the real world.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have further described Themis, an approach and its open-source implementation for measuring discrimination in software and for generating efficient test suites to perform these measurements.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further, focusing on avoiding discriminating against one characteristic may create discrimination against another, e.g., B census gender limits discrimination against gender but discriminates against race with a causal score of 38.40%.”
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“FairTest, an implementation of the unwarranted associations framework [79] uses manually written tests to measure four kinds of discrimination scores: the CV score and a related ratio, mutual information, Pearson correlation, and a regression between the output and sensitive inputs.”
What the sources say
- Not checked yetcites [79]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The B approach was designed to avoid a variant of group discrimination (as are other discrimination-aware approaches), but this design is, at least in some conditions, insufficient to prevent causal discrimination.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“FairML [1] uses orthogonal projection to co-perturb characteristics, which can mask some discrimination, but find discrimination that is more likely to be observed in real-world scenarios”
What the sources say
- Not checked yetcites [1] [1] Julius Adebayo and Lalana Kagal. Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR , abs/1611.04967, 2016. -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Counterfactual fairness is related to causal fairness but can miss some instances of discrimination, e.g., if loan shows preferential treatment for some purple inputs, but at the same time against some other similar purple inputs.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This experiment demonstrates that Themis effectively discovers discrimination and can test software for unexpected software discrimination effects across a wide variety of input partitions.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The work on operational profiles is complementary to ours: Themis uses operational profiles and work on more efficient test generation from operational profiles can directly benefit discrimination testing.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have formally defined software fairness testing and introduced a causality-based measure of discrimination.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The number of tests needed to evaluate every possible value pair can be significantly smaller than the exhaustive testing alternative since each test can simultaneously contribute to multiple value pairs [22, 44, 80].”
What the sources say
- Not checked yetcites [22] [22] Marcelo d'Amorim, Steven Lauterburg, and Darko Marinov. Delta execution for efficient state-space exploration of object-oriented programs. In International Symposium on Software Testing and Analysis (ISSTA) , pages 50-60, London, UK, 2007. - source
- Not checked yetcites [44] [44] Christian Kästner, Alexander von Rhein, Sebastian Erdweg, Jonas Pusch, Sven Apel, Tillmann Rendel, and Klaus Ostermann. Toward variability-aware testing. In International Workshop on Feature-Oriented Software Development (FOSD) , pages 1-8, Dresden, Germany, 2012. - source
- Not checked yetcites [80] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Counterfactual fairness requires output probability distributions to match for input populations that differ only in the label value of a sensitive input characteristic [51].”
What the sources say
- Not checked yetcites [51] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The goal of the generated tests is typically finding bugs [29] or generating specifications [21].”
What the sources say
- Not checked yetcites [29] [29] Gordon Fraser and Andrea Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering (TSE) , 39(2):276-291, February 2013. - source
- Not checked yetcites [21] [21] Valentin Dallmeier, Nikolai Knopp, Christoph Mallon, Gordon Fraser, Sebastian Hack, and Andreas Zeller. Automatically generating test cases for specification mining. IEEE Transactions on Software Engineering (TSE) , 38(2):243-257, March 2012. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our approach generates tests automatically and measures causal discrimination.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“AdFisher computes a variant of group discrimination, but it could be integrated with Themis and its algorithms to measure causal discrimination.”
What the sources say
- Not checked yetcites [23] [23] Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. Proceedings on Privacy Enhancing Technologies (PoPETs) , 1:92-112, 2015. - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our main goal is to develop a method to process test executions to measure software discrimination, whereas that is not a goal of combinatorial testing.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Causal relationships in data management systems [54, 55] can help explain query results [57] and debug errors [82-84] by tracking and using data provenance [56].”
What the sources say
- Not checked yetcites [54]
- Not checked yetcites [55]
- Not checked yetcites [57]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Our prior work on testing software that relies on data management systems has focused on data errors [59, 60], whereas this work focuses on testing fairness.”
What the sources say
- Not checked yetcites [59]
- Not checked yetcites [60] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These tools deal with more complex input spaces than Themis, but none of them focus on testing fairness and they require oracles whereas Themis does not need oracles as it measures discrimination by comparing tests' outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: None of the automated test generation tools focus on testing fairness.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our oracle is more precise and easier to compute, but is only applicable to fairness testing.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The authors state this as an assumption.
Software can be modeled as a black box that maps input characteristics to an output characteristic.
“First, we define software as a black box that maps input characteristics to an output characteristic.”
The authors state this as an assumption.
Input characteristics and the output characteristic are categorical variables.
“Second, we assume that the input characteristics and the output characteristic are categorical variables, each having a set of possible values (e.g., race, gender, eye color, age ranges, income ranges).”
The authors state this as an assumption.
For the remainder of the paper, software outputs binary decisions.
“For simplicity of exposition, for the remainder of this paper, we assume software outputs binary decisions - a natural domain for fairness testing.”
The paper relies on this without support.
No prior work automatically generates test suites to measure causal discrimination in software.
“To the best of our knowledge, this work is the first to automatically generate test suites to measure causal discrimination in software.”
“We are aware of no prior testing technique, neither automated nor manual, that produces a measure of a software system's causal discrimination.”
The authors state this as an assumption.
Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
“decreasing θ effectively simulates increasing system discrimination.”
The paper relies on this without support.
No prior work on operational profile testing has measured software discrimination.
“Meanwhile no prior work on operational profile testing has measured software discrimination.”
The paper relies on this without support.
None of the automated test generation tools focus on testing fairness.
“none of them focus on testing fairness”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Thus, causal methods are speciically well suited to identifying 1) proxies of sensitive variables as discussed in subsection 2.1; 2) which subgroups are most (un)fairly treated and diferentiate the types of bias exhibited [118]; and 3) provide transparency with respect to how (classiication) decisions were made [143].”
From Fairness in machine learning: A survey (2024) · cites this paper as [113]
In the citing paper
“As a result, a classifier trained without regard to the protected attribute not only fails to remove discrimination, but it can complicate the detection and mitigation of discrimination downstream via in-processing or post-processing techniques [9,10,18,24,25,34,43,51], such as those we describe next.”
From Interventional Fairness: Causal Database Repair for Algorithmic Fairness (2019) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.2 Race and Gender Discrimination · p. 9
“| A census race | 2.10% | 1.13% | 8.90% | 7.20% |”
§ 4.2 Sound Pruning · p. 7
“Theorem 4.3. Let X be an input type and let S be a decision software with input type X . Then for all sets of characteristics X ′ ⊆ X , ˜ d X ′ ( S ) ≤ ⃗ d X ′ ( S ) .”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“nition of discrimination is that a “better” input is never deprived of the “better” output [24].”
BackgroundInternational Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,558
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 4×
“Our causal discrimination score is a stronger measure of discrimination, as it seeks out causality in software, measuring the fraction of inputs for which changing specific input characteristics causes the output to change [67].”
BackgroundInfluential(2013) · Cited by 1,126
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104 · cited 1×
“While prior work has tackled the oracle problem [10, 26, 30] typically using inferred pre- and post-conditions or documentation, our oracle is more precise and easier to compute, but is only applicable to fairness testing.”
BackgroundKusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Arcuri, A., & Briand, L. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985795 · Cited by 989 · cited 4×
“Apparent discrimination captures this notion, applying group or causal discrimination score measurement to a subset of the input domain, which can be described by an operational profile [7, 58].”
BackgroundMethodInfluentialFeedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2007.37 · Cited by 879 · cited 1×
“Automated testing research has produced tools to generate tests, including random testing, such as Randoop [63, 64], NightHawk [4], JCrasher [20], CarFast [65], and T3 [69]; search-based testing, such as EvoSuite [29], TestFul [9], and eToc [78]; dynamic symbolic execution tools, such as DSC [37], Symbolic PathFinder…”
Method
Research works printed but not linked: 8
- Agitar Technologies . AgitarOne. http://www.agitar.com/solutions/products/ automated_junit_generation.html , 2016 . Agitar Technologies. AgitarOne. http://www.agitar.com/solutions/products/ automated_junit_generation.html, 2016.
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . Machine bias . ProPublica , May 23, 2016 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May 23, 2016.
- 10.5555/1792786.1792789 · DOI
- Katelyn Ferral . Wisconsin supreme court allows state to continue using computer program to assist in sentencing . The Capital Times , July 13, 2016 . http://host.madison.com/ct/news/local/govt-and-politics/ wisconsin-supreme-court-allows-state-to-continue-using-computer-program/ article_7eb67874-bf40-59e3-b62a-923d1626fa0f.html. Katelyn Ferral. Wisconsin supreme court allows state to continue using computer program to assist in sentencing. The Capital Times, July 13, 2016. http://host.madison.com/ct/news/local/govt-and-politics/ wisconsin-supreme-court-allows-state-to-continue-using-computer-program/ article_7eb67874-bf40-59e3-b62a-923d1626fa0f.html.
- Rafi Letzter . Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist . TECH Insider , April 21, 2016 . http://www.techinsider.io/ how-algorithms-can-be-racist- 2016 - 2014 . Rafi Letzter. Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist. TECH Insider, April 21, 2016. http://www.techinsider.io/ how-algorithms-can-be-racist-2016-4.
- Satya Nadella . The partnership of the future . Slate , June 28, 2016 . http: //www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_ satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html. Satya Nadella. The partnership of the future. Slate, June 28, 2016. http: //www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_ satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html.
- Aarti Shahani . Now algorithms are deciding whom to hire, based on voice . NPR All Things Considered , March 2015 . Aarti Shahani. Now algorithms are deciding whom to hire, based on voice. NPR All Things Considered, March 2015.
- 10.5555/1792786.1792798 · DOI
Cited by
Sorted by citations
Items 301–399 of 399
GIST : Generated Inputs Sets Transferability in Deep Learning
Tambon, F., Khomh, F., & Antoniol, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672457 · Cited by 2
“…among others, we can cite DLFuzz [16] using neuron-coverage guide fuzzing, DeepRoad [61] which used a GAN based approach, DeepJanus [48] using the notion of frontier behavior, DeepEvolution [5] using metamorphic transformations to generate images or Themis [15] a fairness testing technique.”
Data vs. Model Machine Learning Fairness Testing: An Empirical Study
Shome, A., Cruz, L., & Arie Van Deursen, A. (2024) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1145/3643786.3648022 · Cited by 2
Ramanaik, C. K., Willmann, A., Cardona, J.-E. S., Hanfeld, P., Hoffmann, N., & Hecht, M. (2024) · Axioms · doi:10.3390/axioms13080535 · Cited by 2
Roads Ahead to Diversity and Inclusion by Software Engineering
Rastogi, A. (2024) · Apress eBooks · doi:10.1007/978-1-4842-9651-6_1 · Cited by 2
FIPSER: Improving Fairness Testing of DNN by Seed Prioritization
Chen, J., Zhang, Y., Zhang, L., Zhang, M., Wan, C., Su, T., & Pu, G. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695486 · Cited by 2
“Fairness testing aims to measure fairness and find out fairness defects in softwares [2, 3, 10, 17, 46, 54].”
MobilityMirror: Bias-Adjusted Transportation Datasets
Rodriguez, L., Salimi, B., Ping, H., Stoyanovich, J., & Howe, B. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-11238-7_2 · Cited by 2
“use a causal framework to develop a software testing framework for fairness [17].”
MethodA New Reinforcement Learning-Based Framework for Unbiased Autonomous Software Systems
Sulaimon, I. A., Ghoneim, A., & Alrashoud, M. R. (2019) · International Conference on Modeling, Simulation, and Applied Optimization · doi:10.1109/icmsao.2019.8880288 · Cited by 3
“on the insight provided by the detection tools [6] [11] [13]”
BackgroundMethodFairway: SE Principles for Building Fairer Software.
Chakraborty, J., Majumder, S., Wu, Z., & Menzies, T. (2020) · Cited by 2
Data Synthesis for Testing Black-Box Machine Learning Models
Saha, D., Aggarwal, A., & Hans, S. (2022) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3493700.3493704 · Cited by 2
Proposing an Interactive Audit Pipeline for Visual Privacy Research
DeHart, J., Xu, C., Grant, C., & Egede, L. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671478 · Cited by 3
The promise and perils of using machine learning when engineering software (keynote paper)
Brun, Y. (2022) · doi:10.1145/3549034.3570200 · Cited by 2
“Coupled with an emergence of tools that support fairness-aware decision-making in data-driven systems [2, 4, 8, 16, 22, 23], these”
BackgroundSelf-citationHunt, L., & Ferrario, M. A. (2022) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1145/3510458.3513013 · Cited by 2
“For example, this could be achieved by enhancing existing SE practices with values-led tools as described in [115] or through new automated SE values detection techniques [28].”
MethodFairLay-ML: Intuitive Debugging of Fairness in Data-Driven Social-Critical Software
Yu, N., Carreon, L., Tan, G., & Tizpaz-Niari, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00016 · Cited by 4
“This notion is the primary oracle of fairness testing [12], [22], [23], [13].”
InfluentialA Case Study on a Sustainable Framework for Ethically Aware Predictive Modeling
Lux, T. C. H., Nagy, S., Almanaa, M., Yao, S., & Bixler, R. (2019) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas48451.2019.8937885 · Cited by 2
“Many works have since expanded on CV scoring for preventing discrimination [26], [31], [32].”
BackgroundProbabilistic Verification of Fairness Properties via ConcentrationPreprint
Bastani, O., Zhang, X., & Solar-Lezama, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02573 · Cited by 2
“For example, we can straightforwardly support other kinds of fairness for supervised learning [Galhotra et al. 2017; Kleinberg et al. 2017; Zafar et al. 2017].”
BackgroundMethodSOCRATES: Towards a Unified Platform for Neural Network AnalysisPreprint
Pham, L. H., Li, J., & Sun, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.11206 · Cited by 2
“Galhotra et al. proposed THEMIS [44], [45], a causality based algorithm utilizing the random test generation to evaluate a model’s fairness, i.e., the frequency of individual discriminatory instances.”
How does Software Change?Preprint
Rastogi, A., & Gousios, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01885 · Cited by 3
Verifying Global Neural Network Specifications using Hyperproperties
Boetius, D., & Leue, S. (2023) · Kalpa publications in computing · doi:10.29007/pvtn · Cited by 1
The Fair Game: Auditing & debiasing AI algorithms over time
Basu, D., & Das, U. (2025) · Cambridge Forum on AI Law and Governance · doi:10.1017/cfl.2025.8 · Cited by 2
“Though initial auditors used to evaluate the bias over full input data (Galhotra et al., 2017; Bellamy et al., 2019), distributional auditors have been developed to estimate bias over whole input data distribution (Yan and Zhang, 2022; Ajarra et al., 2024).”
Exploring Fairness Debt Through Evidence from Studies on Algorithmic Discrimination
Aryan, F., Valença, L., & Santos, R. D. S. (2025) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1109/techdebt66644.2025.00011 · Cited by 1
“Today, despite attempts to mitigate these issues, the “black-box” nature of many algorithms—where decision-making logic is neither transparent nor interpretable—makes it chal-lenginge to identify and correct biases, leading to unintended yet far-reaching consequences in society [11], [12].”
GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
Xiao, Y., Liu, X., Cheng, Q., Yin, Z., Liang, S., Li, J., Shao, J., Liu, A., & Tao, D. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02556-7 · Cited by 24
“GenderBias-VL supports measuring bias in two ways: probability difference (under coun-terfactual fairness [22]) and outcome difference (under causal fairness [21]).”
Exploring Software Fairness Debt in Gray Literature
Sotolani, R. S., Freire, S., Fronchetti, F., Santos, R. D. S., & Spinola, R. O. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04207-1_7 · Cited by 1
Shafqat, Z., Jilani, A. A. A., Butt, N. A., Rehman, S. U., Khalid, M. U., & Gruhn, V. (2025) · IEEE Access · doi:10.1109/access.2025.3618868 · Cited by 1
“Prompts are short sentences that tell conversational AI systems what they should do [1].”
Causally Perturbed Fairness Testing
Du, C., & Chen, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773088 · Cited by 1
A Study of Floating-Point Precision Tuning in Deep Learning Operators Implementations
Wen, Z., Liu, H., Zhu, T., Pan, M., Wang, S., Lin, Y. K., Liu, K., Zhang, T., & Li, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773992 · Cited by 2
“Themis [25], a fairness testing technique designed to detect causal discrimination, identified substantial model bias based on gender, marital status, and race.”
Software Fairness Testing in Practice
Santos, R. D. S., Leça, M. D. M., Santos, R., & de Magalhães, C. V. C. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00012 · Cited by 1
“By identifying and mitigating discriminatory risks based on factors such as race, gender, or socioeconomic status, fairness testing plays a key role in ensuring ethical AI development and promoting fairness in automated decision-making [9], [10].”
InfluentialTowards User-Focused Cross-Domain Testing: Disentangling Accessibility, Usability, and Fairness
Leça, M. D. M., & Santos, R. D. S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c66551.2025.00029 · Cited by 1
“…emerges as a non-functional requirement and a quality attribute of systems, especially those reliant on data-driven processes and powered by artificial intelligence, to ensure just and unbiased treatment of users or groups of users affected by the outcomes of algorithms and applications [2]–[4].”
InfluentialTesting Individual Fairness in Graph Neural Networks
Nasiri, R. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3757048 · Cited by 2
“Key studies include THEMIS [14], which generates test cases using the random sampling strategy to detect discrimination.”
Is Diversity a Meaningful Metric in Fairness Testing?
Funamoto, K., Kitamura, T., & Takada, S. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00040 · Cited by 1
“We begin by reviewing the I FT framework proposed by [1], as illustrated in Figure 1.”
InfluentialCoverage-Based Harmfulness Testing for LLM Code Transformation
Tan, H., Wang, H., Pressato, D., Xu, Y., & Tan, S. H. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00086 · Cited by 2
The role of team diversity in AI systems development
Santos, R. D. S., Baldassarre, M. T., & de Magalhães, C. V. C. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10755-6 · Cited by 1
“This includes fairness-aware requirements engineering (Zhang and Harman, 2021), which identifies stakeholder values and potential sources of bias early, as well as fairness testing (Galhotra et al., 2017; Chen et al., 2024), which evaluates system outputs for discriminatory behavior.”
Influentialde Magalhães, C. V. C., Santos, R. D. S., Ayoola, B., Stuart-Verner, B., & Santos, Í. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805545 · Cited by 1
“Unaddressed fairness problems can reinforce structural inequalities, excluding or disadvantaging certain populations [9, 14, 22].”
Automated Fairness Testing with Representative Sampling
Karakas, U., & Tosun, A. (2023) · doi:10.1145/3617555.3617871 · Cited by 1
“Galhotra et al. [14] defined group and causal fairness first.”
BackgroundYuan, B., Gui, S., Zhang, Q., Wang, Z., Wen, J., Mao, B., Liu, J., & Yao, X. (2024) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2024.3364430 · Cited by 3
“FairerML determines and measures fairness in data via group fairness analysis, conditional group fairness analysis, and individual fairness analysis, implemented by statistical parity [18], conditional statistical parity [19], and causal discrimination [22], respectively.”
BackgroundMethodMBFair: a model-based verification methodology for detecting violations of individual fairness
Ramadan, Q., Konersmann, M., Ahmadian, A. S., Jürjens, J., & Staab, S. (2024) · Software & Systems Modeling · doi:10.1007/s10270-024-01184-y · Cited by 1
“Existing approaches for analyzing software fairness do not analyze fairness ex ante but ex post, namely during the testing phase of the software development life cycle (e.g., [5, 16]) or during the run-time of the software (e.g., [17, 18]).”
InfluentialBoundary-Guided Black-Box Fairness Testing
Yin, Z., Zhao, W., & Song, T. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00163 · Cited by 2
“In contrast, black-box methods [7], [11], [12], [15], [20] assess the model solely based on its inputs and outputs.”
BackgroundMethodInfluentialRSUTT: Robust Search Using T-Way Testing
Matsukawa, C., & Takada, S. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70245-7_3 · Cited by 1
Research on Artificial Intelligence-Assisted Software Test Automation Methods
Hu, Y. (2024) · Applied Mathematics and Nonlinear Sciences · doi:10.2478/amns-2024-2874 · Cited by 1
Du, C., & Chen, T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686673 · Cited by 1
Fairness Testing of Machine Learning Models using Combinatorial Testing in Latent Space
Dahal, A., Shree, S., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962484 · Cited by 2
Proxy Attribute Discovery in Machine Learning Datasets via Inductive Logic Programming
Gonçalves, R. S. D. M., Gouveia, F., Lynce, I., & Santos, J. F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90653-4_17 · Cited by 2
HIFI: Explaining and Mitigating Algorithmic Bias Through the Lens of Game-Theoretic Interactions
Zhang, L., Wang, Z., Zhang, Y., Zhang, M., & Wang, J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00221 · Cited by 1
Cherekar, R. (2023) · International Journal of Emerging Research in Engineering and Technology · doi:10.63282/3050-922x.ijeret-v4i2p105 · Cited by 4
Correct-by-Construction: Certified Individual Fairness through Neural Network Training
Zhang, R.-H., & Sun, J. (2025) · Proc. ACM Program. Lang. · doi:10.1145/3763107 · Cited by 2
Rondina, M., Vetrò, A., Coppola, R., Regragui, O., Fabris, A., Silvello, G., Susto, G. A., & Martin, J. C. D. (2025) · Frontiers of artificial intelligence, ethics and multidisciplinary applications. · doi:10.1007/978-981-96-7945-4_15 · Cited by 1
Data processing: Reflections on ethics
Firmani, D., Tanca, L., & Torlone, R. (2019) · Cited by 1
Lexicographically Ordered Multi-Objective ClusteringPreprint
Galhotra, S., Saisubramanian, S., & Shlomo Zilberstein, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.00750 · Cited by 1
“Another setting studied in the literature considers individual fairness, where two nodes possessing similar attributes but different color should not be discriminated [ Galhotra et al. , 2017 ] .”
Self-citationFairness-guided SMT-based Rectification of Decision Trees and Random ForestsPreprint
Zhang, J., Beschastnikh, I., Mechtaev, S., & Roychoudhury, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.11001 · Cited by 1
“fairness in machine learning in recent years. Various fairness definitions have been proposed [6], [17], [20], [22]. In our paper, we used group fairness, which is used in the fairness testing work of [15]. Other recent works on fairness testing includes black-box fairness testing [1] and individual fairness testing [3…”
InfluentialTowards Integrating Fairness Transparently in Industrial ApplicationsPreprint
Dodwell, E., Flynn, C., Krishnamurthy, B., Majumdar, S., & Mitra, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06082 · Cited by 2
“Among other similar packages, which are mostly open-source, FairML [1], Themis [18], FairTest [33], Aequitas 5 , and Fair-ness Measures offer bias detection, while Themis-ml [4] offers both detection and mitigation through an expandable platform.”
Unlawful-by-Default:Toward Ex Ante Justification for AiPreprint
Malgieri, G., & Pasquale, F. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4346120 · Cited by 1
Castro, L. M. (2020) · IntechOpen eBooks · doi:10.5772/intechopen.86538
New foundations of ethical multiagent systems blue sky ideas track
Murukannaiah, P. K., Ajmeri, N., Jonker, C. M., & Singh, M. P. (2020) · Research Repository (Delft University of Technology)
Ensuring Fairness Despite Differences in Environment
Singh, K. D. (2021) · York University Digital Library (York University)
Dill, K. (2022) · Sozialwissenschaften und Berufspraxis · doi:10.1007/978-3-658-36514-1_7
On Retrofitting Provenance for Transparent and Fair Software - Drivers and Challenges
Dietrich, J. B., Galster, M., & Luczak–Roesch, M. (2023) · doi:10.1109/fairware59297.2023.00007
“The work of Galhotra et al. [31] generates test cases to reveal discriminatory behaviour.”
MethodTesting, Debugging, and Repairing Individual Discrimination in Machine Learning Models
Saha, D., Agarwal, A., Hans, S., & Haldar, S. (2023) · Studies in computational intelligence · doi:10.1007/978-981-99-7184-8_1
Interpretable White-Box Fairness Testing Through Biased Neuron Identification
Chen, J., Zhang, X., & Zheng, H. (2024) · Springer eBooks · doi:10.1007/978-981-97-0425-5_19
Efficient DNN-Powered Software with Fair Sparse Models
Gao, X., Jiang, W., Zhai, J., Ma, S., Zhang, X., & Shen, C. Q. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680336
“Inspired by ethics-aware software engineering [10, 19, 28], we present Ballot, a novel fairness-aware pruning framework by revised winning ticket finding and training for deep neural networks.”
MethodJohnson, B., & Menzies, T. (2025) · IEEE Software · doi:10.1109/ms.2025.3539402
Ethical Considerations and Challenges of Leveraging IoT and Machine Learning in Smart Urban Planning
Mohtashim, M. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-9030-6.ch007
Automated Data Bias Mitigation Technique for Algorithmic Fairness
Shi, J. M., & Rong, C. (2025) · The European Journal on Artificial Intelligence · doi:10.1177/30504554251328351
“Usually, there are two main bias reduction algorithms, causality based methods (Galhotra et al., 2017; Nabi & Shpitser, 2018) and association based methods (Calders & Verwer, 2010; Dwork et al., 2012).”
Boosting Generalizable Fairness With Mahalanobis Distances Guided Boltzmann Exploratory Testing
Dong, K., Wu, P., & Chen, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3581402 · Cited by 1
“Galhotra et al. proposed THEMIS [78], which for the first time defines software fairness testing, introduces the metric of fairness score, and designs a causality-based testing algorithm to evaluate software fairness.”
Wu, Z., Wang, Z., Luo, C. W., Du, X., & Chen, J. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728948
“For example, on DeepFM with LFM360K under the AUC metric, FairAS spends 773 seconds to achieve the M AUC value of 0.9142 (the same as the ideal effectiveness achieved by Themis, taking 8,733 seconds), while FairRec spends 1,798 seconds to achieve the M AUC value of 0.9178.”
InfluentialFrom fair graphs to fair data: a DAG-based approach to mitigating bias in AI systems
Jiang, V. W., Batista, G., & Bain, M. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02817-y
Teaching Software Engineering for Artificial Intelligence: An Experience Report
Palomba, F., Voria, G., Parziale, A., Pentangelo, V., Porta, A. D., Martino, V. D., Recupito, G., & Giordano, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04207-1_15
Liu, Y., Cao, Q., Shen, H., Zhang, K., Wu, Y., & Cheng, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757414
“Existing fairness testing methods mostly examine “causal fairness” [28], which is a simplified definition of individual fairness and requires a software system to produce the same outcome for every two individual samples that differ only in sensitive attributes.”
Fairness set and forgotten: Mining fairness toolkit usage in open-source machine learning projects
Cannavale, A., Voria, G., Scognamiglio, A., Giordano, G., Catolino, G., & Palomba, F. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107957
Contextual Fairness-Aware Practices in ML: A Cost-Effective Empirical Evaluation
Parziale, A., Voria, G., Giordano, G., Catolino, G., Robles, G., & Palomba, F. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c66551.2025.00008
“Galhotra et al. [28] proposed T HEMIS , which identifies bias using input perturbation, and Udeshi et al. [29] introduced A EQUITAS to enhance bias detection efficiency.”
Towards Fairness in AI: A Systematic Mapping Study on Software Engineering Solutions
Nepomuceno, K. T. C., & Petrillo, F. D. S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering - Companion (SANER-C) · doi:10.1109/saner-c66551.2025.00014
“Notably, the first studies explicitly using terms like “Software Fairness” or directly addressing fairness in software emerged around 2017, marking a foundational period in this field [12].”
Towards Practical Software Discrimination Testing
Mim, S. A. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00061
“Despite their widespread adoption, numerous instances of bias and discriminatory behavior have been observed in software systems [1].”
Fairness Testing: Review of the Relevance and Methodologies of Ensuring Software Fairness
Zia, I., Raj, A., & Dabu, J. B. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-08977-9_48
On the Robustness of Fairness Practices: A Causal Framework for Systematic Evaluation
Monjezi, V., Kumar, A., Trivedi, A., Tan, G., & Tizpaz-Niari, S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773175
“The notion of individual discrimination [22] has been significantly used to test software for discrimination [7–9, 20, 51, 60, 61, 64].”
Responsible DevOps for Cyber-Physical Systems
Trubiani, C., Aleti, A., Böhm, B., Zhou, Z., & Ma, L. (2026) · Springer eBooks · doi:10.1007/978-981-95-1786-2_7
Bias Smells: Exploring the Roots of Discrimination and Societal Inequities in Software SystemsPreprint
Santos, R. D. S., Spinola, R. O., & de Magalhães, C. V. C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6215818
Latent Search-Based Boundary Aware Fairness Testing for Deployed Deep Models
Sun, Q., Zhao, W., & Song, T. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00041
“Galhotra et al. [5] introduced THEMIS, which formalises software fairness testing, proposes fairness scores as assessment criteria, and employs a causality based algorithm that constructs test inputs through unguided random sampling to estimate the prevalence of discriminatory cases.”
Cobweb: Enhanced Generation Diversity for Black-box Fairness Testing
Guo, Y., Zhao, W., & Song, T. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00044
“Prior works have explored two complementary notions: group fairness [7], which evaluates statistical parity across demographic groups, and individual fairness [8], which requires that similar individuals receive similar outcomes.”
Caetano, S. P., & Lee, S.-W. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113066
FairCG: Mitigate Counterfactual and Group Bias in Machine Learning
Zhang, M., Zhang, L., & Zhang, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461316
Muzaisham, P. A. H. B., Balogun, A. O., Basri, S. B., Mamman, H., Capretz, L. F., & Lawal, A. B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20749-4_30
Alaghbari, M. A., Al-Azazi, S. A., Alawag, A. M., Al-Ghazali, B. M., Alnadish, A. M., & Alzubi, K. (2025) · International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa68193.2025.11498905
“The experimental design and evaluation metrics are grounded in established methodologies for fairness testing and evaluating human-AI interaction [34], [36].”
Muzaisham, P. A. H. B., Balogun, A. O., Basri, S. B., Mamman, H., Capretz, L. F., & Lawal, A. B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20746-3_30
Based on past experience: highlighting potential human value issues in domain modelling
Kaur, J. (2023) · eScholarship@McGill (McGill) · doi:10.82308/14932
Measuring Database Unfairness via Dependency Quantification Under Differential Privacy
Vologdin, M., Tao, Y., & Gilad, A. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3819518.3819560
Lambiase, S., Voria, G., Schiavone, M. C., Catolino, G., & Palomba, F. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10902-7
“Post-processing techniques instead adjust the model outputs after training to correct for unfair outcomes, with Galhotra et al. (2017) introducing Themis and Udeshi et al. (2018) developing Aequitas .”
Individual Fairness Testing in Fairness through Unawareness
Kuma, T., Kitamura, T., & Takada, S. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00077
“The framework is based on the individual fairness testing framework for FTA proposed by Galhotra et al. [5], and is adapted to the FTU setting by incorporating mechanisms such as inferring individuals’ protected attributes and redefining similarity between individuals.”
InfluentialQuantifying Sensitivity for Tree Ensembles: A Symbolic and Compositional Approach
Naik, A., Garg, C., Akshay, S., Gupta, A., & Meel, K. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-32526-6_26
Alharbi, R., & Abbadeni, N. (2026) · IEEE Access · doi:10.1109/access.2026.3717516
“Galhotra et al. [31] designed it to evaluate software discrimination through causal testing by generating input pairs that differed only in their sensitive attributes.”
Racismo Algorítmico em Sistemas de Inteligência Artificial: Uma Revisão Sistemática da Literatura
Campos, A. M. D. S., dos Santos, N. D. S., da Silva, D. T. A., & Cunha, L. D. S. B. (2026) · Anais do Workshop sobre as Implicações da Computação na Sociedade (WICS) · doi:10.5753/wics.2026.23828
Evaluating the Impact of Situation Testing on SMOTE-Based Sampling Techniques for Bias Mitigation
Shahrezaei, M. H., McDaid, K., & Loughran, R. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-25809-0_22
FIFT: Feature Importance-Guided Fairness Testing for machine learning software
Mamman, H., Balogun, A. O., Maidawa, M., Basri, S. B., Adamu, S., Garba, A., & Capretz, L. F. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01130-6
“…interest in individual fairness testing (IFT), which aims to identify individual discriminatory instances (IDIs) produced by a classifier under test (CuT) through systematic examination of input pairs that differ only in protected attributes yet receive different predictions (Galhotra et al. 2017).”
InfluentialFAIR-SHEPHERD: Fairness beyond Statistical Parity toward Structural Alignment
Mohole, S., Choi, H., de Witt, C. S., Sakla, W., Klymko, C., Glatt, R., & Galhotra, S. (2026) · ACM Journal on Responsible Computing · doi:10.1145/3844729
Self-citationSoftware Fairness Analysis and Repair via Causal Model-Guided Data Mutation
Xiao, Y., Chen, Z., Liu, Y., Mousavi, M. R., & Zhang, J. M. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3846174
“Furthermore, the development of Fairabel reinforces ongoing efforts by SE researchers to fulfill ethical responsibilities and software quality control [35, 79] in delivering fair ML software services [22, 23].”
ProofCoop: Collaborative Automated Formal Verification
Kaufman, Z., First, E., Sanchez-Stern, A., Thompson, K., Lerner, S., & Brun, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787783
“Our work demonstrates the potential benefits of collaboration between models developed for automated proof synthesis and supports continuing exploration of robust and effective collaboration methods.”
Self-citationBased on Past Experience: Highlighting Potential Human Value Issues in Domain Modelling
Kaur, J., & Mussbacher, G. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00054
Parziale, A., D'Auria, A., Scaparra, D. P., & Pontillo, V. (2026) · doi:10.1145/3843782.3844655
“Existing research spans empirical studies of model bias [23], bias mitigation techniques [1, 20], and testing approaches to identify biased behavior [3, 5, 24].”
Metadata
- Authors
- Galhotra, S., Brun, Y., & Meliou, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2017-08-21
- Pages
- 498-510
- OpenAlex
- W2730550703
- Cited by
- 405
- References
- 91
- Citations by year
- 2026: 26 · 2025: 55 · 2024: 46 · 2023: 56 · 2022: 50 · 2021: 53 · 2020: 57 · 2019: 37 · 2018: 22 · 2017: 3
- FWCI
- 23.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-1453474, IIS-1453543, CNS-1744471)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic discrimination, fairness testing, automated test generation, THEMIS, algorithmic fairness, automated decision systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
