Fairness testing: testing software for discrimination
Open accessAbstract
This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in modern software systems that recommend criminal sentences, grant access to financial products, and determine who is allowed to participate in promotions. Our approach, Themis, generates efficient test suites to measure discrimination. Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle. We evaluate Themis on 20 software systems, 12 of which come from prior work with explicit focus on avoiding discrimination. We find that (1) Themis is effective at discovering software discrimination, (2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain, (3) Themis optimizations are effective at producing efficient test suites for measuring discrimination, and (4) Themis is more efficient on systems that exhibit more discrimination. We thus demonstrate that fairness testing is a critical aspect of the software development cycle in domains with possible discrimination and provide initial tools for measuring software discrimination.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper defines causal software discrimination and presents Themis, a testing method that automatically generates test suites to measure group and causal discrimination without an oracle. Evaluating 20 instances of 8 financial-decision systems, it finds discrimination even in systems designed to avoid it and shows sound pruning reduces test suites by two-to-three orders of magnitude.
Key findings
- Themis detected causal discrimination missed by group discrimination: for `B census gender`, the causal score was 11.27% versus a group score of 0.52%, more than 21× higher.
- Discrimination was present even in systems designed to avoid it: `B census gender` had a causal discrimination score over 11% for gender, and discriminated against gender, marital status, and race with a causal score of 77.2%; `F census` discriminated against country and race with a causal score of 98.1%.
- There was no clear evidence that discrimination-aware methods outperformed discrimination-unaware ones; the discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.
- Pruning reduced the number of test cases by, on average, a factor of 2,849 for group and 148 for causal discrimination; Themis was more efficient when systems discriminated more, e.g., `B credit gender` needed 3,413 tests at threshold θ = 0.7 but only 10 tests at θ = 0.6.
Methods
- The paper formally defines group, causal, and apparent discrimination, then introduces Themis, which uses an input schema, requires no oracle, and employs test caching, adaptive confidence-driven sampling, and sound pruning based on monotonicity properties.
- Evaluation used 20 instances of 8 software systems making financial decisions: 12 instances from prior discrimination-aware work (4 focused on race, 8 on gender) and 8 standard machine-learning instances that made no attempt to avoid discrimination.
- The datasets were the Adult/Census Income dataset (45K individuals, 14 attributes) and the Statlog German Credit dataset (1,000 individuals, 20 pieces of data); systems were trained on balanced samples of 15.5K census individuals and 600 credit individuals, and Themis measured group and causal discrimination with respect to race and gender, including discrimination-checking experiments with thresholds and pruning comparisons at 99% confidence and 0.05 error bound.
Limitations
- The paper states simplifying assumptions: software is modeled as a black box mapping categorical input characteristics to a categorical output characteristic, and non-categorical inputs/outputs are handled via binning; extending the measures directly to broader data types is left to future work.
- The paper states Themis requires an input schema, and its apparent-discrimination method does not compute confidence because it is only as strong as the developer’s confidence that the test suite or operational profile is representative; apparent discrimination was not directly evaluated.
- Evident from the text: the evaluation is limited to 20 instances of 8 financial-decision systems on two datasets, so the findings may not generalize to other software domains; the paper also leaves integration with combinatorial testing and other automated test-generation improvements to future work.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 45 cited sources checked · 0 supported · 0 flagged
- Findingown data§ 1 INTRODUCTION · p. 1
Themis optimizations are effective at producing efficient test suites for measuring discrimination.
“(3) Themis optimizations are effective at producing efficient test suites for measuring discrimination”
Quote in the paperSubstantial claim - Findingown data§ 1 INTRODUCTION · p. 1
Themis is effective at discovering software discrimination.
“We find that (1) Themis is effective at discovering software discrimination”
- Also stated in: § ACMReference format: · p. 2 · plain
“Themis is an effective fairness testing tool.”
Quote in the paperSubstantial claim - Also stated in: § ACMReference format: · p. 2 · plain
- Findingown data§ 1 INTRODUCTION · p. 1
State-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain.
“(2) state-of-the-art techniques for removing discrimination from algorithms fail in many situations, at times discriminating against as much as 98% of an input subdomain”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Race and Gender Discrimination · p. 8
The discrimination-aware decision tree instance B census gender, trained not to discriminate against gender, had a causal discrimination score over 11%, meaning more than 11% of individuals had their output flipped by altering only gender.
“For example, a discrimination-aware decision tree approach trained not to discriminate against gender, B census gender , had a causal discrimination score over 11%: more than 11% of the individuals had the output flipped just by altering the individual's gender.”
Quote in the paperSubstantial claim - Findingown data§ ACMReference format: · p. 2
Even when fairness is a design goal, developers can easily introduce discrimination in software.
“demonstrating that (i) even when fairness is a design goal, developers can easily introduce discrimination in software”
Quote in the paperSubstantial claim
- Findingown data§ 4.2 Sound Pruning · p. 6
Pruning leads to, on average, a two-to-three order of magnitude reduction in test suite size.
“As Section 5.4 will evaluate empirically, pruning leads to, on average, a two-to-three order of magnitude reduction in test suite size.”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Race and Gender Discrimination · p. 8
Discrimination is present even in software instances designed to avoid discrimination.
“Discrimination is present even in instances designed to avoid discrimination.”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Race and Gender Discrimination · p. 9
For the B census gender instance, the causal discrimination score was more than 21 times higher than the group discrimination score (11.27% vs. 0.52%).
“For example, for B census gender , the causal score was more than 21 × higher than the group score (11.27% vs. 0.52%).”
Quote in the paperSubstantial claim - Findingown data§ 6 RELATED WORK · p. 10
Reducing discrimination (CV score) in classifiers often fails to remove causal discrimination and discrimination against certain groups.
“Reducing discrimination (CV score) in classifiers [18, 40, 88, 91], as our evaluation has shown, often fails to remove causal discrimination and discrimination against certain groups.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [18] [18] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In Proceedings of the 2009 IEEE International Conference on Data Mining (ICDM) Workshops , pages 13-18, Miami, FL, USA, December 2009. - source
Not checked in the free run.
- cites [40] [40] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In International Conference on Data Mining (ICDM) , pages 869-874, Sydney, Australia, December 2010. - source
Not checked in the free run.
- cites [88] source
Not checked in the free run.
- Findingown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9
Themis was able to discover significant discrimination.
“Themis was able to discover significant discrimination.”
Quote in the paperSubstantial claim - Findingnone given§ 5.5 Discussion · p. 10
State-of-the-art approaches for designing fair systems often miss discrimination.
“Stateof-the-art approaches for designing fair systems often miss discrimination”
Quote in the paperSubstantial claim - Findingown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9
F census discriminates against country and race with a causal score of 98.1%.
“Even worse, F census discriminates against country and race with a causal score of 98.1%.”
Quote in the paperSubstantial claim - Findingown data§ 5.5 Discussion · p. 10
Themis can detect such discrimination via fairness testing.
“Themis can detect such discrimination via fairness testing.”
Quote in the paperSubstantial claim - Findingown data§ 5.5 Discussion · p. 10
Overall, pruning reduced test suite sizes, on average, two to three orders of magnitude.
“Overall, pruning reduced test suite sizes, on average, two to three orders of magnitude.”
Quote in the paperSubstantial claim - Findingown data§ 5.4 Themis Efficiency and the Pruning Effect · p. 10
Pruning increased when θ decreased (equivalently, when discrimination increased).
“We verified that pruning increased when θ decreased (or equivalently, when discrimination increased).”
Quote in the paperSubstantial claim- The authors state this as an assumption: Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
- Findingown data§ 5.4 Themis Efficiency and the Pruning Effect · p. 10
The number of tests for F credit dropped from 920 to 10 when lowering θ from 0.6 to 0.5.
“Similarly, the number of tests for F credit dropped from 920 to 10 when lowering θ from 0.6 to 0.5.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
- Findingown data§ 7 CONTRIBUTIONS · p. 11
Our evaluation demonstrates that discrimination in software is common, even when fairness is an explicit design goal.
“Our evaluation demonstrates that discrimination in software is common, even when fairness is an explicit design goal”
Quote in the paperSubstantial claim - Interpretationreasoning§ 1 INTRODUCTION · p. 1
Themis is more efficient on systems that exhibit more discrimination.
“(4) Themis is more efficient on systems that exhibit more discrimination.”
- Also stated in: § ACMReference format: · p. 2 · boosted
“a proof that Themis is more efficient on systems that exhibit more discrimination.”
Quote in the paperSubstantial claim - Also stated in: § ACMReference format: · p. 2 · boosted
- Interpretationreasoning§ 4.2 Sound Pruning · p. 7
For any decision software S with input type X, and for all sets of characteristics X′ ⊆ X, the group discrimination score with respect to X′ is less than or equal to the causal discrimination score with respect to X′.
“Theorem 4.3. Let X be an input type and let S be a decision software with input type X . Then for all sets of characteristics X ′ ⊆ X , ˜ d X ′ ( S ) ≤ ⃗ d X ′ ( S ) .”
Quote in the paperSubstantial claim - Interpretationreasoning§ ACMReference format: · p. 2
Themis can detect when sentencing software behaves such that changing only the applicant's race affects the software's sentence recommendations for 13% of possible applicants.
“can, for example, detect when sentencing software behaves such that 'changing only the applicant's race affects the software's sentence recommendations for 13% of possible applicants.'”
Quote in the paperNot a substantial claim - Interpretationreasoning§ ACMReference format: · p. 2
Restricting the input space to real-world inputs may hide software discrimination that causal testing can reveal.
“Restricting the input space to real-world inputs [1] may similarly hide software discrimination that causal testing can reveal.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [1] [1] Julius Adebayo and Lalana Kagal. Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR , abs/1611.04967, 2016. -
Not checked in the free run.
- Interpretationreasoning§ ACMReference format: · p. 2
Themis optimizations lead to provably sound two-to-three orders of magnitude improvements in test suite size.
“including proofs of monotonicity of discrimination that lead to provably sound two-to-three orders of magnitude improvements in test suite size”
Quote in the paperSubstantial claim - Interpretationcitation§ 3 FORMAL FAIRNESS DEFINITIONS · p. 4
The causal discrimination score is a stronger measure of discrimination than group and apparent discrimination scores.
“Our causal discrimination score is a stronger measure of discrimination, as it seeks out causality in software, measuring the fraction of inputs for which changing specific input characteristics causes the output to change [67].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [67] source
Not checked in the free run.
- Interpretationnone given§ 4.2 Sound Pruning · p. 7
For any decision software S with input type X, and for all sets of characteristics X′ and X′′ such that X′′ ⊇ X′, the causal discrimination score with respect to X′′ is greater than or equal to the causal discrimination score with respect to X′.
“Theorem 4.2 (Causal discrimination monotonicity). Let X be an input type and let S be a decision software with input type X . Then for all sets of characteristics X ′ , X ′′ ⊆ X , X ′′ ⊇ X ′ = ⇒ ⃗ d X ′′ ( S ) ≥ ⃗ d X ′ ( S ) .”
Quote in the paperSubstantial claim - Study design and conductcitation§ 5.1 Subject Software Systems · p. 8
The reimplementation of system B produced results consistent with the evaluation of the original system.
“We verified that our reimplementation produced results consistent with the the evaluation of the original system [40].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [40] [40] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In International Conference on Data Mining (ICDM) , pages 869-874, Sydney, Australia, December 2010. - source
Not checked in the free run.
- Interpretationreasoning§ 5.2 Race and Gender Discrimination · p. 9
Today's discrimination-aware approaches are insufficient.
“Today's discrimination-aware approaches are insufficient.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Sound Pruning · p. 6
For any decision software S with input type X, and for all sets of characteristics X′ and X′′ such that X′′ ⊇ X′, the group discrimination score with respect to X′′ is greater than or equal to the group discrimination score with respect to X′.
“Theorem 4.1 (Group discrimination monotonicity). Let X be an input type and let S be a decision software with input type X . Then for all sets of characteristics X ′ , X ′′ ⊆ X , X ′′ ⊇ X ′ = ⇒ ˜ d X ′′ ( S ) ≥ ˜ d X ′ ( S ) .”
Quote in the paperSubstantial claim - Premisecitation§ ACMReference format: · p. 2
Software testing offers a unique opportunity to conduct causal experiments to determine statistical causality.
“Software testing offers a unique opportunity to conduct causal experiments to determine statistical causality [67]”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [67] source
Not checked in the free run.
- Interpretationreasoning§ 4.2 Sound Pruning · p. 6
The monotonicity results guarantee that Themis's pruning strategy is sound.
“These results guarantee that Themis pruning strategy is sound.”
Quote in the paperSubstantial claim - Premisecitation§ ACMReference format: · p. 2
Prior work on detecting discrimination has focused on measuring correlation or mutual information between inputs and outputs, discrepancies in fractions of inputs that produce a given output, or discrepancies in output probability distributions.
“Prior work on detecting discrimination has focused on measuring correlation or mutual information between inputs and outputs [79], discrepancies in the fractions of inputs that produce a given output [18, 19, 39-42, 87-89, 91], or discrepancies in output probability distributions [51].”
Quote in the paperSubstantial claim0 of 10 sources checked- cites [79]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [18] [18] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In Proceedings of the 2009 IEEE International Conference on Data Mining (ICDM) Workshops , pages 13-18, Miami, FL, USA, December 2009. - source
Not checked in the free run.
- cites [19] [19] Toon Calders and Sicco Verwer. Three naive Bayes approaches for discriminationfree classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010. - source
Not checked in the free run.
- cites [39] [39] Faisal Kamiran and Toon Calders. Classifying without discriminating. In International Conference on Computer, Control, and Communication (IC4) , pages 1-6, Karachi, Pakistan, February 2009. - source
Not checked in the free run.
- cites [40] [40] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In International Conference on Data Mining (ICDM) , pages 869-874, Sydney, Australia, December 2010. - source
Not checked in the free run.
- cites [41] [41] Faisal Kamiran, Asim Karim, and Xiangliang Zhang. Decision theory for discrimination-aware classification. In International Conference on Data Mining (ICDM) , pages 924-929, Brussels, Belgium, December 2012. - source
Not checked in the free run.
- cites [42] [42] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairnessaware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD) , pages 35-50, Bristol, England, UK, September 2012. - source
Not checked in the free run.
- cites [87] source
Not checked in the free run.
- cites [88] source
Not checked in the free run.
- cites [89]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation§ ACMReference format: · p. 1
Controlling for criminal history, recidivism, age, and gender shows that the software predicts black defendants to be 77% more likely to be pegged as at higher risk of committing a future violent crime than white defendants.
“Prior criminal history does not explain this difference: Controlling for criminal history, recidivism, age, and gender shows that the software predicts black defendants to be 77% more likely to be pegged as at higher risk of committing a future violent crime than white defendants [5].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5] [5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica , May 23, 2016. https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing. -
Not checked in the free run.
- Interpretationreasoning§ 4.2 Sound Pruning · p. 6
Pruning in Themis is based on the monotonicity property that if a software S discriminates over threshold θ with respect to a set of characteristics X′, then S also discriminates over threshold θ with respect to all supersets of X′.
“Pruning is based on a fundamental monotonicity property of group and causal discrimination: if a software S discriminates over threshold θ with respect to a set of characteristics X ′ , then S also discriminates over threshold θ with respect to all superset of X ′ .”
Quote in the paperSubstantial claim - Interpretationreasoning§ 2 SOFTWARE FAIRNESS MEASURES · p. 3
Group discrimination may fail to observe some discrimination.
“First, group discrimination may fail to observe some discrimination.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4 THE THEMIS SOLUTION · p. 5
Themis addresses the challenges of exhaustive testing and measuring discrimination over all subsets of characteristics by employing three optimizations: test caching, adaptive confidence-driven sampling, and sound pruning.
“Themis addresses these challenges by employing three optimizations: (1) test caching, (2) adaptive, confidence-driven sampling, and (3) sound pruning.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 2 SOFTWARE FAIRNESS MEASURES · p. 3
Software may circumvent group discrimination detection.
“Second, software may circumvent discrimination detection.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.1 Caching and Approximation · p. 6
Sampling in Themis is adaptive, using the ongoing score computation to determine if a specified margin of error ε with a desired confidence level conf has been reached.
“Sampling in Themis is adaptive, using the ongoing score computation to determine if a specified margin of error ϵ with a desired confidence level conf has been reached.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.1 Caching and Approximation · p. 6
Because exhaustive testing is infeasible, Themis computes approximate group and causal discrimination scores through sampling.
“Since exhaustive testing is infeasible, Themis computes approximate group and causal discrimination scores through sampling.”
Quote in the paperSubstantial claim - Premisecitation§ 2 SOFTWARE FAIRNESS MEASURES · p. 3
Apparent discrimination depends on the operational profile of the system's use.
“Apparent discrimination depends on the operational profile of the system system's use [7, 58].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [7] [7] Andrea Arcuri and Lionel Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE) , pages 1-10, Honolulu, HI, USA, 2011. -
Not checked in the free run.
- cites [58] source
Not checked in the free run.
- Study design and conductreasoning§ 5.5 Discussion · p. 10
Themis employs provably sound pruning to reduce test suite size.
“Themis employs provably sound pruning to reduce test suite size”
- Also stated in: § 7 CONTRIBUTIONS · p. 11 · boosted
“we formally prove soundness of our approach”
Quote in the paperSubstantial claim - Also stated in: § 7 CONTRIBUTIONS · p. 11 · boosted
- Interpretationown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9
It is possible to build an algorithm that appears fair with respect to a characteristic in general, but discriminates heavily against that characteristic when the input space is partitioned by another characteristic.
“It is possible to build an algorithm that appears to be fair with respect to a characteristic in general, but discriminates heavily against that characteristic when the input space is partitioned by another characteristic.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9
The findings suggest that software designed to produce fair results sometimes achieved fairness at the global scale by creating severe discrimination for certain groups of inputs.
“these findings suggest that the software designed to produce fair results sometimes achieved fairness at the global scale by creating severe discrimination for certain groups of inputs.”
Quote in the paperSubstantial claim - Interpretationcitation§ 6 RELATED WORK · p. 10
Our causal discrimination score goes beyond prior work by measuring causality.
“Our causal discrimination score goes beyond prior work by measuring causality [67].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [67] source
Not checked in the free run.
- Interpretationown data§ 5.4 Themis Efficiency and the Pruning Effect · p. 10
Themis is more efficient when the benefits of fairness testing increase because the software discriminates more.
“This confirms that Themis is more efficient when the benefits of fairness testing increase because the software discriminates more.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
- Premisecitation§ 6 RELATED WORK · p. 11
Developer-written test suites are often not as representative of field executions as developers would like.
“developer-written test suites are often not as representative of field executions as developers would like [81]”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [81] source
Not checked in the free run.
- May combine two claims. 32
- Classified as probably not a substantial claim. 30
May combine two claims.
Findingown data§ 5.2 Race and Gender Discrimination · p. 9The discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.
“In fact, the discrimination-unaware approaches typically discriminated less than their discrimination-aware counterparts, with the exception of logistic regression.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.4 Themis Efficiency and the Pruning Effect · p. 9Pruning reduces the number of test cases needed to compute group and causal discrimination by, on average, a factor of 2,849 for group and 148 for causal discrimination.
“Pruning reduces the number of test cases by, on average, a factor of 2 , 849 for group and 148 for causal discrimination.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9B census gender discriminates against gender, marital status, and race with a causal score of 77.2%, meaning that for 77.2% of individuals, changing only gender, marital status, or race causes the output to flip.
“For example, B census gender discriminates against gender, marital status, and race with a causal score of 77.2%. That means for 77.2% of the individuals, changing only the gender, marital status, or race causes the output of the algorithm to flip.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9For B census gender, the causal discrimination score against gender is 11.2%, but against gender, marital status, and race is 77.2%.
“This is the case with B census gender , for example, as its causal discrimination score against gender is 11.2%, but against gender, marital status, and race is 77.2%.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 7 CONTRIBUTIONS · p. 11Themis effectively measures discriminations and produces efficient test suites to do so.
“show that Themis effectively measures discriminations and produces efficient test suites to do so.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone given§ 5.5 Discussion · p. 10Themis is effective at finding both group and causal discrimination.
“Themis is effective at finding both group and causal discrimination.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5.4 Themis Efficiency and the Pruning Effect · p. 10Themis needed 3,413 tests to find sets of characteristics that B credit gender discriminated with a score of more than θ = 0.7, but only 10 tests when θ was reduced to 0.6.
“For example, Themis needed 3,413 tests to find sets of characteristics that B credit gender discriminated with a score of more than θ = 0 . 7, but only 10 tests when we reduced θ to 0.6.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
May combine two claims.
Interpretationreasoning§ 1 INTRODUCTION · p. 1Themis automatically generates test suites that measure discrimination and does not require an oracle.
“Given a schema describing valid system inputs, Themis generates discrimination tests automatically and does not require an oracle.”
- Also stated in: § ACMReference format: · p. 2 · plain
“Themis requires a schema for generating inputs, but does not require an oracle.”
- Also stated in: § ACMReference format: · p. 2 · plain
“Unlike prior work that requires manually written tests [79], Themis automatically generates test suites that measure discrimination.”
Quote in the paperSubstantial claim- Also stated in: § ACMReference format: · p. 2 · plain
May combine two claims.
Premisecitation§ 2 SOFTWARE FAIRNESS MEASURES · p. 3Amazon.com's same-day delivery software did not explicitly consider race but made race-correlated decisions because of correlations between race and other input characteristics.
“Amazon.com, Inc. software that determined where to offer free same-day delivery did not explicitly consider race but made racecorrelated decisions because of correlations between race and other input characteristics [36, 52].”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [36] [36] David Ingold and Spencer Soper. Amazon doesn't consider the race of its customers. Should it? Bloomberg , April 21, 2016. http://www.bloomberg.com/ graphics/2016-amazon-same-day. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationreasoning§ 4.1 Caching and Approximation · p. 6Test caching in Themis has low storage overhead and offers significant runtime gains.
“Test caching has low storage overhead and offers significant runtime gains.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 3 FORMAL FAIRNESS DEFINITIONS · p. 4The output domain distance function generalizes our work beyond binary outputs.
“The output domain distance function generalizes our work beyond binary outputs.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ ACMReference format: · p. 2These prior approaches do not capture causality and can miss discrimination that our causal approach detects.
“These approaches do not capture causality and can miss discrimination that our causal approach detects, e.g., when the software discriminates negatively with respect to a group in one settings, but positively in another.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 5.2 Race and Gender Discrimination · p. 8Themis is effective: it can verify that many software instances do not discriminate against race and gender, and identify the software that does.
“Themis is effective. Themis is able to (1) verify that many of the software instances do not discriminate against race and gender, and (2) identify the software that does.”
Quote in the paperSubstantial claimMay combine two claims.
Premisecitation§ ACMReference format: · p. 1The COMPAS software is particularly likely to falsely flag black defendants as future criminals, wrongly labeling them at almost twice the rate as white defendants; white defendants were mislabeled as low risk more often than black defendants.
“In forecasting who would reoffend, the software is 'particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants; white defendants were mislabeled as low risk more often than black defendants' [5].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [5] [5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica , May 23, 2016. https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing. -
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ 3 FORMAL FAIRNESS DEFINITIONS · p. 4The causal discrimination score penalizes software that gives loans to individuals of one race but not to otherwise identical individuals of another race, while group and apparent discrimination scores penalize software that gives loans to different fractions of individuals of different races.
“As a result, for example, while the group and apparent discrimination scores penalize software that gives loans to different fractions of individuals of different races, the causal discrimination score penalizes software that gives loans to individuals of one race but not to otherwise identical individuals of another race.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationcitation§ ACMReference format: · p. 2Our causal fairness definition is designed specifically to be testable, unlike definitions that require probabilistic estimation or knowledge of the future.
“Our causal fairness definition is designed specifically to be testable, unlike definitions that require probabilistic estimation or knowledge of the future [38].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [38] [38] Matthew Joseph, Michael Kearns, Jamie Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. CoRR , abs/1207.0016, 2012. https: //arxiv.org/abs/1605.07139. - source
Not checked in the free run.
Classified as probably not a substantial claim.
Interpretationreasoning§ 3 FORMAL FAIRNESS DEFINITIONS · p. 4The causal discrimination score identifies changing which characteristics directly affects the output.
“The causal discrimination score identifies changing which characteristics directly affects the output.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 3 FORMAL FAIRNESS DEFINITIONS · p. 4Our definition extends to the more broad categorical input types.
“Our definition extends to the more broad categorical input types, reflecting the relative complexity of arbitrary decision software.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone given§ ACMReference format: · p. 2Our definition captures causal relationships between inputs and outputs.
“Our definition captures causal relationships between inputs and outputs”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ ACMReference format: · p. 2Themis could have analyzed the Amazon software and detected the discrimination prior to deployment.
“Themis could have analyzed the software and detected the discrimination prior to deployment.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11Operational profiles describe the input distributions likely to be observed in the field.
“Operational profiles [58] describe the input distributions likely to be observed in the field.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [58] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 6 RELATED WORK · p. 10Our group discrimination score generalizes the CV score to the software domain with more complex inputs.
“Our group discrimination score generalizes the CV score to the software domain with more complex inputs.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [19] [19] Toon Calders and Sicco Verwer. Three naive Bayes approaches for discriminationfree classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010. - source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11Operational profiles can significantly improve the effectiveness of testing by more accurately representing real-world system use.
“operational profiles can significantly improve the effectiveness of testing by more accurately representing real-world system use [7, 50]”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [7] [7] Andrea Arcuri and Lionel Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE) , pages 1-10, Honolulu, HI, USA, 2011. -
Not checked in the free run.
- cites [50] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11The use of operational profiles has been shown to more accurately measure system properties, such as reliability.
“the use of operational profiles has been shown to more accurately measure system properties, such as reliability [35].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [35] [35] Herman Hartmann. A statistical analysis of operational profile driven testing. In International Conference on Software Quality, Reliability, and Security Companion (QRS-C) , pages 109-116, Viena, Austria, August 2016. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone given§ 6 RELATED WORK · p. 11Themis measures apparent discrimination by either executing a provided test suite, or by generating a test suite following a provided operational profile.
“Themis measures apparent discrimination by either executing a provided test suite, or by generating a test suite following a provided operational profile.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 5.5 Discussion · p. 10Themis can also measure apparent discrimination (Definition 3.8) via a developer-provided test suite or operational profile.
“Themis can also measure apparent discrimination (Definition 3.8) via a developer-provided test suite or operational profile.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ 6 RELATED WORK · p. 10Our work does not attempt to remove discrimination but offers developers a tool to identify and measure discrimination.
“our work does not attempt to remove discrimination but offers developers a tool to identify and measure discrimination”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Premisereasoning§ 6 RELATED WORK · p. 11Input characteristics may correlate, so perturbing some characteristics without perturbing others may create inputs not representative of the real world.
“input characteristics may correlate, e.g., education correlates with age, so perturbing some characteristics without perturbing others may create inputs not representative of the real world.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationnone given§ 7 CONTRIBUTIONS · p. 11We have described Themis, an approach and its open-source implementation for measuring discrimination in software and for generating efficient test suites to perform these measurements.
“We have further described Themis, an approach and its open-source implementation for measuring discrimination in software and for generating efficient test suites to perform these measurements.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown data§ 5.2 Race and Gender Discrimination · p. 9Focusing on avoiding discrimination against one characteristic may create discrimination against another; for example, B census gender limits discrimination against gender but discriminates against race with a causal score of 38.40%.
“Further, focusing on avoiding discriminating against one characteristic may create discrimination against another, e.g., B census gender limits discrimination against gender but discriminates against race with a causal score of 38.40%.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11FairTest uses manually written tests to measure four kinds of discrimination scores: the CV score and a related ratio, mutual information, Pearson correlation, and a regression between the output and sensitive inputs.
“FairTest, an implementation of the unwarranted associations framework [79] uses manually written tests to measure four kinds of discrimination scores: the CV score and a related ratio, mutual information, Pearson correlation, and a regression between the output and sensitive inputs.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [79]
Not checked in the free run.
May combine two claims.
Interpretationown data§ 5.2 Race and Gender Discrimination · p. 9The B approach was designed to avoid a variant of group discrimination, but this design is, at least in some conditions, insufficient to prevent causal discrimination.
“The B approach was designed to avoid a variant of group discrimination (as are other discrimination-aware approaches), but this design is, at least in some conditions, insufficient to prevent causal discrimination.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 6 RELATED WORK · p. 11FairML uses orthogonal projection to co-perturb characteristics, which can mask some discrimination, but find discrimination that is more likely to be observed in real-world scenarios.
“FairML [1] uses orthogonal projection to co-perturb characteristics, which can mask some discrimination, but find discrimination that is more likely to be observed in real-world scenarios”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [1] [1] Julius Adebayo and Lalana Kagal. Iterative orthogonal feature projection for diagnosing bias in black-box models. CoRR , abs/1611.04967, 2016. -
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisereasoning§ 6 RELATED WORK · p. 11Counterfactual fairness can miss some instances of discrimination.
“Counterfactual fairness is related to causal fairness but can miss some instances of discrimination, e.g., if loan shows preferential treatment for some purple inputs, but at the same time against some other similar purple inputs.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 5.3 Computing Discriminated-Against Characteristics · p. 9The experiment demonstrates that Themis effectively discovers discrimination and can test software for unexpected discrimination effects across a wide variety of input partitions.
“This experiment demonstrates that Themis effectively discovers discrimination and can test software for unexpected software discrimination effects across a wide variety of input partitions.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationreasoning§ 6 RELATED WORK · p. 11The work on operational profiles is complementary to ours: Themis uses operational profiles and work on more efficient test generation from operational profiles can directly benefit discrimination testing.
“The work on operational profiles is complementary to ours: Themis uses operational profiles and work on more efficient test generation from operational profiles can directly benefit discrimination testing.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationnone given§ 7 CONTRIBUTIONS · p. 11We have formally defined software fairness testing and introduced a causality-based measure of discrimination.
“We have formally defined software fairness testing and introduced a causality-based measure of discrimination.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11The number of tests needed to evaluate every possible value pair can be significantly smaller than the exhaustive testing alternative.
“The number of tests needed to evaluate every possible value pair can be significantly smaller than the exhaustive testing alternative since each test can simultaneously contribute to multiple value pairs [22, 44, 80].”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [22] [22] Marcelo d'Amorim, Steven Lauterburg, and Darko Marinov. Delta execution for efficient state-space exploration of object-oriented programs. In International Symposium on Software Testing and Analysis (ISSTA) , pages 50-60, London, UK, 2007. - source
Not checked in the free run.
- cites [44] [44] Christian Kästner, Alexander von Rhein, Sebastian Erdweg, Jonas Pusch, Sven Apel, Tillmann Rendel, and Klaus Ostermann. Toward variability-aware testing. In International Workshop on Feature-Oriented Software Development (FOSD) , pages 1-8, Dresden, Germany, 2012. - source
Not checked in the free run.
- cites [80] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11Counterfactual fairness requires output probability distributions to match for input populations that differ only in the label value of a sensitive input characteristic.
“Counterfactual fairness requires output probability distributions to match for input populations that differ only in the label value of a sensitive input characteristic [51].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [51] source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 6 RELATED WORK · p. 11The goal of the generated tests is typically finding bugs or generating specifications.
“The goal of the generated tests is typically finding bugs [29] or generating specifications [21].”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [29] [29] Gordon Fraser and Andrea Arcuri. Whole test suite generation. IEEE Transactions on Software Engineering (TSE) , 39(2):276-291, February 2013. - source
Not checked in the free run.
- cites [21] [21] Valentin Dallmeier, Nikolai Knopp, Christoph Mallon, Gordon Fraser, Sebastian Hack, and Andreas Zeller. Automatically generating test cases for specification mining. IEEE Transactions on Software Engineering (TSE) , 38(2):243-257, March 2012. - source
Not checked in the free run.
May combine two claims.
Interpretationnone given§ 6 RELATED WORK · p. 11Our approach generates tests automatically and measures causal discrimination.
“our approach generates tests automatically and measures causal discrimination.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 6 RELATED WORK · p. 11AdFisher computes a variant of group discrimination, but it could be integrated with Themis and its algorithms to measure causal discrimination.
“AdFisher computes a variant of group discrimination, but it could be integrated with Themis and its algorithms to measure causal discrimination.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [23] [23] Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. Proceedings on Privacy Enhancing Technologies (PoPETs) , 1:92-112, 2015. - source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationreasoning§ 6 RELATED WORK · p. 11Our main goal is to develop a method to process test executions to measure software discrimination, whereas that is not a goal of combinatorial testing.
“Our main goal is to develop a method to process test executions to measure software discrimination, whereas that is not a goal of combinatorial testing.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 6 RELATED WORK · p. 11Causal relationships in data management systems can help explain query results and debug errors by tracking and using data provenance.
“Causal relationships in data management systems [54, 55] can help explain query results [57] and debug errors [82-84] by tracking and using data provenance [56].”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [54]
Not checked in the free run.
- cites [55]
Not checked in the free run.
- cites [57]
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 6 RELATED WORK · p. 11Our prior work on testing software that relies on data management systems has focused on data errors, whereas this work focuses on testing fairness.
“Our prior work on testing software that relies on data management systems has focused on data errors [59, 60], whereas this work focuses on testing fairness.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [59]
Not checked in the free run.
- cites [60] source
Not checked in the free run.
May combine two claims.
Premisereasoning§ 6 RELATED WORK · p. 11These tools deal with more complex input spaces than Themis, but none of them focus on testing fairness and they require oracles whereas Themis does not need oracles as it measures discrimination by comparing tests' outputs.
“These tools deal with more complex input spaces than Themis, but none of them focus on testing fairness and they require oracles whereas Themis does not need oracles as it measures discrimination by comparing tests' outputs.”
Quote in the paperNot a substantial claim- Rests on an assumption: None of the automated test generation tools focus on testing fairness.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone given§ 6 RELATED WORK · p. 11Our oracle is more precise and easier to compute, but is only applicable to fairness testing.
“our oracle is more precise and easier to compute, but is only applicable to fairness testing.”
Quote in the paperNot a substantial claim
The authors state this as an assumption.
Software can be modeled as a black box that maps input characteristics to an output characteristic.
“First, we define software as a black box that maps input characteristics to an output characteristic.”
The authors state this as an assumption.
Input characteristics and the output characteristic are categorical variables.
“Second, we assume that the input characteristics and the output characteristic are categorical variables, each having a set of possible values (e.g., race, gender, eye color, age ranges, income ranges).”
The authors state this as an assumption.
For the remainder of the paper, software outputs binary decisions.
“For simplicity of exposition, for the remainder of this paper, we assume software outputs binary decisions - a natural domain for fairness testing.”
The paper relies on this without support.
No prior work automatically generates test suites to measure causal discrimination in software.
“To the best of our knowledge, this work is the first to automatically generate test suites to measure causal discrimination in software.”
“We are aware of no prior testing technique, neither automated nor manual, that produces a measure of a software system's causal discrimination.”
The authors state this as an assumption.
Decreasing the discrimination threshold θ effectively simulates increasing system discrimination.
“decreasing θ effectively simulates increasing system discrimination.”
The paper relies on this without support.
No prior work on operational profile testing has measured software discrimination.
“Meanwhile no prior work on operational profile testing has measured software discrimination.”
The paper relies on this without support.
None of the automated test generation tools focus on testing fairness.
“none of them focus on testing fairness”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Causal methods are specifically well suited to identifying proxies of sensitive variables.
“Thus, causal methods are speciically well suited to identifying 1) proxies of sensitive variables as discussed in subsection 2.1; 2) which subgroups are most (un)fairly treated and diferentiate the types of bias exhibited [118]; and 3) provide transparency with respect to how (classiication) decisions were made [143].”
From Fairness in machine learning: A survey (2024) · cites this paper as [113]
A classifier trained without regard to the protected attribute fails to remove discrimination and can complicate detection and mitigation of discrimination downstream via in-processing or post-processing techniques.
“As a result, a classifier trained without regard to the protected attribute not only fails to remove discrimination, but it can complicate the detection and mitigation of discrimination downstream via in-processing or post-processing techniques [9,10,18,24,25,34,43,51], such as those we describe next.”
From Interventional Fairness: Causal Database Repair for Algorithmic Fairness (2019) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Reported group score exceeds causal score despite the stated bound
MinorTheorem 4.3 states that the group discrimination score cannot exceed the causal score for the same characteristics. For A census race, the reported gender group score is 8.90%, above the gender causal score of 7.20%. Sampling uncertainty may explain the estimates, but the displayed scores do not satisfy the stated bound.
“| A census race | 2.10% | 1.13% | 8.90% | 7.20% |”
“Theorem 4.3. Let X be an input type and let S be a decision software with input type X . Then for all sets of characteristics X ′ ⊆ X , ˜ d X ′ ( S ) ≤ ⃗ d X ′ ( S ) .”
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
International Conference on Machine Learning (ICML)-2005
津田, 宏. (2006) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 2,482 · see citing works
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 4× · see citing works
“Our causal discrimination score is a stronger measure of discrimination, as it seeks out causality in software, measuring the fraction of inputs for which changing specific input characteristics causes the output to change [67].”
BackgroundInfluential(2013) · Cited by 1,126 · see citing works
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104 · see citing works
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · see citing works
Arcuri, A., & Briand, L. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985795 · Cited by 989 · see citing works
Feedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2007.37 · Cited by 879 · see citing works
Dong, X.-N. (2004) · Current opinion in plant biology · doi:10.1016/j.pbi.2004.07.005 · Cited by 841 · see citing works
Pex–White Box Test Generation for .NET
Tillmann, N., & de Halleux, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-79124-9_10 · Cited by 805 · see citing works
Learning and evaluating classifiers under sample selection bias
Zadrozny, B. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015425 · Cited by 800 · see citing works
Discrimination in online ad delivery
Sweeney, L. (2013) · Communications of the ACM · doi:10.1145/2447976.2447990 · Cited by 783 · see citing works
Operational profiles in software-reliability engineering
Musa, J. D. (1993) · IEEE Software · doi:10.1109/52.199724 · Cited by 765 · see citing works
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · see citing works
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694 · see citing works
Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
Datta, A., Tschantz, M. C., & Datta, A. (2015) · Proceedings on Privacy Enhancing Technologies · doi:10.1515/popets-2015-0007 · Cited by 676 · see citing works
Fraser, G., & Arcuri, A. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.14 · Cited by 586 · see citing works
Pacheco, C., & Ernst, M. D. (2007) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/1297846.1297902 · Cited by 558 · see citing works
Unequal Representation and Gender Stereotypes in Image Search Results for Occupations
Kay, M., Matuszek, C., & Munson, S. A. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702520 · Cited by 509 · see citing works
Sweeney, L. (2013) · Queue · doi:10.1145/2460276.2460278 · Cited by 477 · see citing works
Building Classifiers with Independency Constraints
Calders, T., Kamiran, F., & Pechenizkiy, M. (2009) · 2009 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2009.83 · Cited by 468 · see citing works
JCrasher: an automatic robustness tester for Java
Csallner, C., & Smaragdakis, Y. (2004) · Software: Practice and Experience · doi:10.1002/spe.602 · Cited by 426 · see citing works
Evolutionary testing of classes
Tonella, P. (2004) · Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis · doi:10.1145/1007512.1007528 · Cited by 410 · see citing works
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · see citing works
CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools
Sen, K., & Agha, G. (2006) · Lecture notes in computer science · doi:10.1007/11817963_38 · Cited by 393 · see citing works
Discrimination Aware Decision Tree Learning
Kamiran, F., Calders, T., & Pechenizkiy, M. (2010) · 2010 IEEE International Conference on Data Mining · doi:10.1109/icdm.2010.50 · Cited by 303 · see citing works
Decision Theory for Discrimination-Aware Classification
Kamiran, F., Karim, A., & Zhang, X. (2012) · 2012 IEEE 12th International Conference on Data Mining · doi:10.1109/icdm.2012.45 · Cited by 300 · see citing works
Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution
Xie, T., Marinov, D., Schulte, W., & Notkin, D. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31980-1_24 · Cited by 287 · see citing works
Păsăreanu, C. S., & Rungta, N. (2010) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/1858996.1859035 · Cited by 258 · see citing works
Agrawal, R., Srikant, R., & Thomas, D. (2005) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1066157.1066187 · Cited by 241 · see citing works
Software Engineering for Self-Adaptive Systems II
de Lemos, R., Giese, H., Müller, H. A., & Shaw, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-35813-5 · Cited by 218 · see citing works
The complexity of causality and responsibility for query answers and non-answers
Meliou, A., Gatterbauer, W., Moore, K. F., & Suciu, D. M. (2010) · Proceedings of the VLDB Endowment · doi:10.14778/1880172.1880176 · Cited by 199 · see citing works
Self-citationFairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171 · see citing works
Strategies for product-line verification: case studies and experiments
Apel, S., von Rhein, A., Wendler, P., Größlinger, A., & Beyer, D. (2013) · doi:10.5555/2486788.2486852 · Cited by 149 · see citing works
Reducing combinatorics in testing product lines
Kim, C. H. P., Batory, D. S., & Khurshid, S. (2011) · International Conference on Aspect-Oriented Software Development (AOSD) · doi:10.1145/1960275.1960284 · Cited by 148 · see citing works
Can you program ethics into a self-driving car?
Goodall, N. J. (2016) · IEEE Spectrum · doi:10.1109/mspec.2016.7473149 · Cited by 141 · see citing works
Handling Conditional Discrimination
liobaite, I., Kamiran, F., & Calders, T. (2011) · 2011 IEEE 11th International Conference on Data Mining · doi:10.1109/icdm.2011.72 · Cited by 125 · see citing works
Automatic generation of oracles for exceptional behaviors
Goffi, A., Gorla, A., Ernst, M. D., & Pezzè, M. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931061 · Cited by 114 · see citing works
Genetic Programming for data classification
Eggermont, J., Kok, J. N., & Kosters, W. A. (2004) · ACM Symposium on Applied Computing · doi:10.1145/967900.968104 · Cited by 107 · see citing works
SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems
Kim, C. H. P., Marinov, D., Sarfraz Khurshid, D., Batory, D. S., Souto, S., Barros, P., & d’Amorim, M. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2491459 · Cited by 99 · see citing works
Meliou, A., Gatterbauer, W., Halpern, J. Y., Koch, C., Moore, K. F., & Suciu, D. M. (2010) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 93 · see citing works
Self-citationStrategies for product-line verification: Case studies and experiments
Apel, S., von Rhein, A., Wendler, P., Größlinger, A., & Beyer, D. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606594 · Cited by 84 · see citing works
Synthesizing method sequences for high-coverage testing
Thummalapenta, S., Xie, T., Tillmann, N., de Halleux, J., & Su, Z. (2011) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2048066.2048083 · Cited by 81 · see citing works
TestFul: An Evolutionary Test Approach for Java
Baresi, L., Lanzi, P. L., & Miraz, M. (2010) · International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2010.54 · Cited by 81 · see citing works
Wang, X., Dong, X. L., & Meliou, A. (2015) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2723372.2750549 · Cited by 78 · see citing works
Self-citationZafar, M. B., Valera, I., Gomez-Rodriguez, M., & Gummadi, K. P. (2015) · MPG.PuRe (Max Planck Society) · Cited by 75 · see citing works
Causality and explanations in databases
Meliou, A., Roy, S., & Suciu, D. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2733004.2733070 · Cited by 73 · see citing works
Self-citationGenerating parameterized unit tests
Fraser, G., & Zeller, A. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001464 · Cited by 68 · see citing works
Automatically Generating Test Cases for Specification Mining
Dallmeier, V., Knopp, N., Mallon, C., Fraser, G., Hack, S., & Zeller, A. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.105 · Cited by 67 · see citing works
A Design Space for Self-Adaptive Systems
Brun, Y., Desmarais, R., Geihs, K., Litoiu, M., Lopes, A., Shaw, M., & Smit, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-35813-5_2 · Cited by 67 · see citing works
Self-citationToward variability-aware testing
Kästner, C., von Rhein, A., Erdweg, S., Pusch, J., Apel, S., Rendel, T., & Ostermann, K. (2012) · Proceedings of the 4th International Workshop on Feature-Oriented Software Development - FOSD '12 · doi:10.1145/2377816.2377817 · Cited by 63 · see citing works
CarFast: achieving higher statement coverage faster
Park, S., Hossain, B. M. M., Hussain, I., Csallner, C., Grechanik, M., Taneja, K., Fu, C., & Xie, Q. (2012) · Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering · doi:10.1145/2393596.2393636 · Cited by 55 · see citing works
Decomposition Methods for Linear Support Vector Machines
Kao, W.-C., Chung, K.-M., Sun, C.-L., & Lin, C.-J. (2004) · Neural Computation · doi:10.1162/089976604774201640 · Cited by 54 · see citing works
Shared Execution for Efficiently Testing Product Lines
Kim, C. H. P., Khurshid, S., & Batory, D. S. (2012) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2012.23 · Cited by 53 · see citing works
Andrews, J. H., Li, F. C. H., & Menzies, T. (2007) · doi:10.1145/1321631.1321654 · Cited by 49 · see citing works
Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box ModelsPreprint
Adebayo, J., & Kagal, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04967 · Cited by 44 · see citing works
Smart Redundancy for Distributed Computation
Brun, Y., Edwards, G., Bang, J. Y., & Medvidović, N. (2011) · Proceedings of the ... International Conference on Distributed Computing Systems/Proceedings of the International Conference on Distributed Computing Systems · doi:10.1109/icdcs.2011.25 · Cited by 43 · see citing works
Self-citationSynthesizing partial component-level behavior models from system specifications
Krka, I., Brun, Y., Edwards, G., & Medvidovic, N. (2009) · Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1595696.1595756 · Cited by 40 · see citing works
Self-citationFault and adversary tolerance as an emergent property of distributed systems' software architectures
Brun, Y., & Medvidović, N. (2007) · doi:10.1145/1316550.1316557 · Cited by 36 · see citing works
Self-citationData debugging with continuous testing
Muşlu, K., Brun, Y., & Meliou, A. (2013) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2491411.2494580 · Cited by 36 · see citing works
Self-citationAn Architectural Style for Solving Computationally Intensive Problems on Large Networks
Brun, Y., & Medvidović, N. (2007) · doi:10.1109/seams.2007.4 · Cited by 35 · see citing works
Self-citationPreventing data errors with continuous testing
Muşlu, K., Brun, Y., & Meliou, A. (2015) · International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771792 · Cited by 34 · see citing works
Self-citationGraph-Based Relational Concept Learning
González, J. A., Holder, L. B., & Cook, D. J. (2002) · Cited by 31 · see citing works
Dsc+Mock: a test case + mock class generator in support of coding against interfaces
Islam, M., & Csallner, C. (2010) · Proceedings of the Eighth International Workshop on Dynamic Analysis · doi:10.1145/1868321.1868326 · Cited by 30 · see citing works
Behavioral Execution Comparison: Are Tests Representative of Field Behavior?
Wang, Q., Brun, Y., & Orso, A. (2017) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2017.36 · Cited by 29 · see citing works
Self-citationFairness Constraints: A Mechanism for Fair Classification
Zafar, M. B., Valera, I., Gomez-Rodriguez, M., & Gummadi, K. P. (2015) · Cited by 28 · see citing works
Wang, X., Meliou, A., & Wu, E. (2017) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3035918.3035925 · Cited by 28 · see citing works
Self-citationDelta execution for efficient state-space exploration of object-oriented programs
d'Amorim, M., Lauterburg, S., & Marinov, D. (2007) · Proceedings of the 2007 international symposium on Software testing and analysis · doi:10.1145/1273463.1273472 · Cited by 26 · see citing works
Synthesizing method sequences for high-coverage testing
Thummalapenta, S., Xie, T., Tillmann, N., de Halleux, J., & Su, Z. (2011) · ACM SIGPLAN Notices · doi:10.1145/2076021.2048083 · Cited by 24 · see citing works
Introducing Binary Decision Diagrams in the explicit-state verification of Java code
von Rhein, A., Apel, S., & Raimondi, F. (2011) · Middlesex University Research Repository (Middlesex University Of London) · Cited by 23 · see citing works
Keeping Data Private while Computing in the Cloud
Brun, Y., & Medvidović, N. (2012) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2012.126 · Cited by 23 · see citing works
Self-citationBringing Provenance to its Full Potential Using Causal Reasoning
Meliou, A., Gatterbauer, W., & Suciu, D. M. (2011) · Cited by 19 · see citing works
Self-citationBall, T., & Kupferman, O. (2008) · Springer eBooks · doi:10.1007/978-3-540-79124-9_2 · Cited by 18 · see citing works
Strickland, E. (2016) · IEEE Spectrum · doi:10.1109/mspec.2016.7473150 · Cited by 17 · see citing works
Delta Execution for Efficient State-Space Exploration of Object-Oriented Programs
d’Amorim, M., Lauterburg, S., & Marinov, D. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.37 · Cited by 16 · see citing works
Self-Adapting Reliability in Distributed Software Systems
Brun, Y., Bang, J. Y., Edwards, G., & Medvidovic, N. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2412134 · Cited by 16 · see citing works
Self-citationDecomposition methods for linear support vector machines
Chung, K.-M., Kao, W.-C., Sun, T., & Lin, C. (2004) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2003.1202781 · Cited by 15 · see citing works
Budget-aware random testing with T3
Prasetya, I. S. W. B. (2016) · doi:10.1145/2897010.2897019 · Cited by 13 · see citing works
Software operational profile based test case allocation using fuzzy logic
Kumar, K. S., & Misra, R. B. (2007) · International Journal of Automation and Computing · doi:10.1007/s11633-007-0388-6 · Cited by 12 · see citing works
Entrusting Private Computation and Data to Untrusted Networks
Brun, Y., & Medvidović, N. (2013) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2013.13 · Cited by 10 · see citing works
Self-citationGuizzo, E., & Ackerman, E. (2016) · IEEE Spectrum · doi:10.1109/mspec.2016.7473151 · Cited by 9 · cited 1× · see citing works
“history of warfare: For the first time, a collection of microchips and software will decide whether a human being lives or dies” [34].”
BackgroundVacuity analysis for property qualification by mutation of checkers
Guglielmo, L. D., Fummi, F., & Pravadelli, G. (2010) · Design, Automation, and Test in Europe · doi:10.5555/1870926.1871041 · Cited by 8 · see citing works
Wang, X., Meliou, A., & Wu, E. (2016) · Proceedings - ACM-SIGMOD International Conference on Management of Data · doi:10.1145/2882903.2899388 · Cited by 7 · see citing works
Self-citationQFix: Diagnosing errors through query historiesPreprint
Wang, X., Meliou, A., & Wu, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07539 · Cited by 5 · see citing works
Self-citationA Statistical Analysis of Operational Profile Driven Testing
Hartmann, H. (2016) · 2016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c.2016.18 · Cited by 4 · see citing works
Vacuity analysis for property qualification by mutation of checkers
Guglielmo, L. D., Fummi, F., & Pravadelli, G. (2010) · Design, Automation, and Test in Europe · doi:10.1109/date.2010.5457158 · Cited by 4 · see citing works
Bounds and Capacity Theorems for Cognitive Interference Channels with State
Liang, R. D. Y. (2012) · arXiv · doi:10.48550/arxiv.1207.0016 · Cited by 3 · see citing works
A Design Space for Adaptive Systems
Brun, Y., Desmarais, R., Geihs, K., Litoiu, M., Lopes, A., Shaw, M., & Smit, M. (2018) · KiltHub Repository · doi:10.1184/r1/6620630 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 8
- Agitar Technologies . AgitarOne. http://www.agitar.com/solutions/products/ automated_junit_generation.html , 2016 . Agitar Technologies. AgitarOne. http://www.agitar.com/solutions/products/ automated_junit_generation.html, 2016.
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . Machine bias . ProPublica , May 23, 2016 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May 23, 2016.
- 10.5555/1792786.1792789 · DOI
- Katelyn Ferral . Wisconsin supreme court allows state to continue using computer program to assist in sentencing . The Capital Times , July 13, 2016 . http://host.madison.com/ct/news/local/govt-and-politics/ wisconsin-supreme-court-allows-state-to-continue-using-computer-program/ article_7eb67874-bf40-59e3-b62a-923d1626fa0f.html. Katelyn Ferral. Wisconsin supreme court allows state to continue using computer program to assist in sentencing. The Capital Times, July 13, 2016. http://host.madison.com/ct/news/local/govt-and-politics/ wisconsin-supreme-court-allows-state-to-continue-using-computer-program/ article_7eb67874-bf40-59e3-b62a-923d1626fa0f.html.
- Rafi Letzter . Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist . TECH Insider , April 21, 2016 . http://www.techinsider.io/ how-algorithms-can-be-racist- 2016 - 2014 . Rafi Letzter. Amazon just showed us that ‘unbiased’ algorithms can be inadvertently racist. TECH Insider, April 21, 2016. http://www.techinsider.io/ how-algorithms-can-be-racist-2016-4.
- Satya Nadella . The partnership of the future . Slate , June 28, 2016 . http: //www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_ satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html. Satya Nadella. The partnership of the future. Slate, June 28, 2016. http: //www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_ satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html.
- Aarti Shahani . Now algorithms are deciding whom to hire, based on voice . NPR All Things Considered , March 2015 . Aarti Shahani. Now algorithms are deciding whom to hire, based on voice. NPR All Things Considered, March 2015.
- 10.5555/1792786.1792798 · DOI
Cited by
Sorted by citations
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · see citing works
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundFairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581 · see citing works
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 326 · see citing works
Socially Responsible AI Algorithms: Issues, Purposes, and Challenges
Cheng, L., Varshney, K. R., & Liu, H. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12814 · Cited by 261 · see citing works
On testing machine learning programs
Braiek, H. B., & Khomh, F. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110542 · Cited by 203 · see citing works
Automated directed fairness testing
Udeshi, S., Arora, P., & Chattopadhyay, S. (2018) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3238147.3238165 · Cited by 200 · see citing works
Bias in machine learning software: why? how? what to do?
Chakraborty, J., Majumder, S., & Menzies, T. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468537 · Cited by 224 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
Black box fairness testing of machine learning models
Aggarwal, A., Lohia, P., Nagar, S., Dey, K., & Saha, D. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338937 · Cited by 180 · see citing works
Perceived diversity in software engineering: a systematic literature review
Rodríguez-Pérez, G., Nadri, R., & Nagappan, M. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09992-2 · Cited by 148 · see citing works
Is neuron coverage a meaningful measure for testing deep neural networks?
Harel-Canada, F., Wang, L., Gulzar, M. A., Gu, Q., & Kim, M. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409754 · Cited by 150 · see citing works
Interventional Fairness: Causal Database Repair for Algorithmic Fairness
Salimi, B., Rodriguez, L., Howe, B., & Suciu, D. (2019) · Proceedings of the 2019 International Conference on Management of Data · doi:10.1145/3299869.3319901 · Cited by 145 · see citing works
Preventing undesirable behavior of intelligent machines
Thomas, P. S., da Silva, B. C., Barto, A. G., Giguere, S., Brun, Y., & Brunskill, E. (2019) · Science · doi:10.1126/science.aag3311 · Cited by 142 · see citing works
Self-citationFairness in rankings and recommendations: an overview
Pitoura, E., Stefanidis, K., & Koutrika, G. (2021) · The VLDB Journal · doi:10.1007/s00778-021-00697-y · Cited by 150 · see citing works
Brun, Y., & Meliou, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264838 · Cited by 138 · see citing works
Self-citationWhite-box fairness testing through adversarial sampling
Zhang, P., Wang, J., Sun, J., Dong, G., Wang, X., Wang, X., Dong, J. S., & Dai, T. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380331 · Cited by 139 · see citing works
Bias Mitigation Post-processing for Individual and Group Fairness
Lohia, P., Ramamurthy, K. N., Bhide, M. A., Saha, D., Varshney, K. R., & Puri, R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682620 · Cited by 135 · see citing works
Algorithmic Decision-Making in AVs: Understanding Ethical and Technical Concerns for Smart Cities
Lim, H. S. M., & Taeihagh, A. (2019) · Sustainability · doi:10.3390/su11205791 · Cited by 136 · see citing works
Corbett, C. J. (2018) · Production and Operations Management · doi:10.1111/poms.12837 · Cited by 128 · see citing works
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
From Gender Biases to Gender-Inclusive Design
Vorvoreanu, M., Zhang, L., Huang, Y., Hilderbrand, C., Steine-Hanson, Z., & Burnett, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300283 · Cited by 119 · see citing works
Biswas, S., & Rajan, H. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468536 · Cited by 114 · see citing works
A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifiers
Chen, Z., Zhang, J. M., Sarro, F., & Harman, M. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583561 · Cited by 105 · see citing works
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Chen, Z., Zhang, J. M., Hort, M., Harman, M., & Sarro, F. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652155 · Cited by 100 · see citing works
Biswas, S., & Rajan, H. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409704 · Cited by 99 · see citing works
Baldur: Whole-Proof Generation and Repair with Large Language Models
First, E., Rabe, M. N., Ringer, T., & Brun, Y. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616243 · Cited by 94 · see citing works
Self-citationThemis: automatically testing software for discrimination
Angell, R., Johnson, B., Brun, Y., & Meliou, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264590 · Cited by 94 · see citing works
Self-citationGender-Inclusive HCI Research and Design: A Conceptual Review
Stumpf, S., Peters, A. N., Bardzell, S., Burnett, M., Busse, D. K., Cauchard, J. R., & Churchill, E. F. (2020) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000056 · Cited by 107 · see citing works
Balayn, A., Lofi, C., & Houben, G.-J. (2021) · The VLDB Journal · doi:10.1007/s00778-021-00671-8 · Cited by 98 · see citing works
Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactuals
Galhotra, S., Pradhan, R., & Salimi, B. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3458455 · Cited by 87 · see citing works
Self-citationKraska, T. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3240493 · Cited by 83 · see citing works
Metamorphic object insertion for testing object detection systems
Wang, S., & Su, Z. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416584 · Cited by 83 · see citing works
Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models
Ma, P., Wang, S., & Liu, J. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/64 · Cited by 80 · see citing works
Fairea: a model behaviour mutation approach to benchmarking bias mitigation methods
Hort, M., Zhang, J. M., Sarro, F., & Harman, M. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468565 · Cited by 80 · see citing works
Improving fairness in machine learning systems: What do industry practitioners need?Preprint
Holstein, K., Vaughan, J. W., Daumé, H., Dudík, M., & Wallach, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05239 · Cited by 81 · see citing works
"Ignorance and Prejudice" in Software Fairness
Zhang, J. M., & Harman, M. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00129 · Cited by 80 · see citing works
Implicit Gender Biases in Professional Software Development: An Empirical Study
Wang, Y., & Redmiles, D. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1109/icse-seis.2019.00009 · Cited by 79 · see citing works
Operationalizing human values in software: a research roadmap
Mougouei, D., Perera, H., Hussain, W., Shams, R. A., & Whittle, J. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3264843 · Cited by 78 · see citing works
A Case for Human Values in Software Engineering
Whittle, J., Ferrario, M. A., Simm, W., & Hussain, W. (2019) · IEEE Software · doi:10.1109/ms.2019.2956701 · Cited by 78 · see citing works
A roadmap for ethics-aware software engineering
Aydemir, F. B., & Dalpiaz, F. (2018) · doi:10.1145/3194770.3194778 · Cited by 76 · see citing works
The concept of fairness in the GDPR
Malgieri, G. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372868 · Cited by 75 · see citing works
Biswas, S., & Rajan, H. (2021) · Iowa State University Digital Repository (Iowa State University) · Cited by 75 · see citing works
Chen, Z., Zhang, J. M., Sarro, F., & Harman, M. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549093 · Cited by 74 · see citing works
Measuring human values in software engineering
Winter, E., Forshaw, S., & Ferrario, M. A. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3267427 · Cited by 73 · see citing works
On the Applicability of Machine Learning Fairness Notions
Makhlouf, K., Zhioua, S., & Catuscia Palamidessi, A. (2021) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3468507.3468511 · Cited by 83 · see citing works
Shen, H., Deng, W. H., Chattopadhyay, A., Wu, Z. S., Wang, X., & Zhu, H. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445971 · Cited by 71 · see citing works
Toward involving end-users in interactive human-in-the-loop AI fairness
Nakao, Y., Stumpf, S., Ahmed, S., Naseer, A., & Strappelli, L. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514258 · Cited by 69 · see citing works
Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Text
Madaan, N., Padhi, I., Panwar, N., & Saha, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17594 · Cited by 69 · see citing works
BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systems
Asyrofi, M. H., Yang, Z., Yusuf, I. N. B., Kang, H. J., Thung, F., & Lo, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3136169 · Cited by 69 · see citing works
Zheng, H., Chen, Z., Du, T., Zhang, X., Cheng, Y., Ji, S., Wang, J., Yu, Y., & Chen, J. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510123 · Cited by 63 · see citing works
Fairness-Aware Machine Learning Engineering: How Far Are We?
Ferrara, C., Sellitto, G., Ferrucci, F., Palomba, F., & De Lucia, A. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10402-y · Cited by 60 · see citing works
“Galhotra et al. (2017) recently proposed the Themis approach to assess whether and how machine learning-based solutions tend to be biased.”
BackgroundPerfectly parallel fairness certification of neural networks
Urban, C., Christakis, M., Wüstholz, V., & Zhang, F. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428253 · Cited by 62 · see citing works
Black, E., Yeom, S., & Fredrikson, M. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372845 · Cited by 61 · see citing works
Vasudevan, S., & Kenthapadi, K. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412705 · Cited by 60 · see citing works
Machine learning fairness notions: Bridging the gap with real-world applications
Makhlouf, K., Zhioua, S., & Catuscia Palamidessi, A. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102642 · Cited by 76 · see citing works
Information-Theoretic Testing and Debugging of Fairness Defects in Deep Neural Networks
Monjezi, V., Trivedi, A., Tan, G., & Tizpaz-Niari, S. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00136 · Cited by 59 · see citing works
Probabilistic verification of fairness properties via concentration
Bastani, O., Zhang, X., & Solar-Lezama, A. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360544 · Cited by 57 · see citing works
Efficient white-box fairness testing through gradient search
Zhang, L., Zhang, Y., & Zhang, M. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464820 · Cited by 58 · see citing works
Nanda, V., Dooley, S. W., Singla, S., Feizi, S., & Dickerson, J. P. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445910 · Cited by 58 · see citing works
Metamorphic Testing of Deep Learning Compilers
Xiao, D., Liu, Z., Yuan, Y., Pang, Q., & Wang, S. (2022) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3508035 · Cited by 57 · see citing works
Transport-based Counterfactual ModelsPreprint
de Lara, L., González-Sanz, A., Asher, N. M., Risser, L., & Loubes, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13025 · Cited by 60 · see citing works
SOSRepair: Expressive Semantic Search for Real-World Program Repair
Afzal, A., Motwani, M., Stolee, K. T., Brun, Y., & Goues, C. L. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2944914 · Cited by 56 · see citing works
Self-citationCCTEST: Testing and Repairing Code Completion Systems
Li, Z., Wang, C., Liu, Z., Wang, H., Chen, D., Wang, S., & Gao, C. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00110 · Cited by 62 · see citing works
Shrestha, Y. R., & Yang, Y. (2019) · Algorithms · doi:10.3390/a12090199 · Cited by 55 · see citing works
Do automated program repair techniques repair hard and important bugs?
Motwani, M., Sankaranarayanan, S., Just, R., & Brun, Y. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9550-0 · Cited by 54 · see citing works
Self-citationFairness-aware configuration of machine learning libraries
Tizpaz-Niari, S., Kumar, A., Tan, G., & Trivedi, A. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510202 · Cited by 54 · see citing works
Shen, H., Jin, H., Cabrera, Á. A., Perer, A., Zhu, H., & Hong, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415224 · Cited by 57 · see citing works
Teaching software engineering for AI-enabled systems
Kästner, C., & Kang, E. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3377814.3381714 · Cited by 53 · see citing works
Albarghouthi, A., & Vinitsky, S. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287588 · Cited by 52 · see citing works
Explanation-guided fairness testing through genetic algorithm
Fan, M. Y., Wei, W., Jin, W., Yang, Z., & Liu, T. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510137 · Cited by 51 · see citing works
Quality of Automated Program Repair on Real-World Defects
Motwani, M., Soto, M., Brun, Y., Just, R., & Goues, C. L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2998785 · Cited by 50 · see citing works
Self-citationAutomatically Generating Precise Oracles from Structured Natural Language Specifications
Motwani, M., & Brun, Y. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00035 · Cited by 47 · see citing works
Self-citationTowards Integrating Human Values into Software: Mapping Principles and Rights of GDPR to Values
Perera, H., Hussain, W., Mougouei, D., Shams, R. A., Nurwidyantoro, A., & Whittle, J. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00053 · Cited by 45 · see citing works
MDPFuzz: testing models solving Markov decision processes
Pang, Q., Yuan, Y., & Wang, S. (2022) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3533767.3534388 · Cited by 47 · see citing works
Wang, Z., Wang, Y., & Redmiles, D. (2018) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1145/3183428.3183437 · Cited by 44 · see citing works
Who's Learning? Using Demographics in EDM Research
Paquette, L., Ocumpaugh, J. L., Li, Z., Andres, A., & Baker, R. S. (2020) · Journal of Educational Data Mining · doi:10.5281/zenodo.4143611 · Cited by 46 · see citing works
The Model Card Authoring Toolkit: Toward Community-centered, Deliberation-driven AI Design
Shen, H., Wang, L., Deng, W. H., Brusse, C., Velgersdijk, R., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533110 · Cited by 45 · see citing works
Malgieri, G., & Pasquale, F. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105899 · Cited by 43 · see citing works
Johnson, B., Brun, Y., & Meliou, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380377 · Cited by 46 · see citing works
Self-citationFairMask: Better Fairness via Model-Based Rebalancing of Protected Attributes
Peng, K., Chakraborty, J., & Menzies, T. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3220713 · Cited by 42 · see citing works
Assessing trustworthy AI: Technical and legal perspectives of fairness in AI
Kattnig, M., Angerschmid, A., Reichel, T., & Kern, R. (2024) · Computer law & security review · doi:10.1016/j.clsr.2024.106053 · Cited by 53 · see citing works
Young, M., Rodriguez, L., Keller, E., Sun, F., Sa, B., Whittington, J., & Howe, B. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287577 · Cited by 41 · see citing works
Are Machine Learning Cloud APIs Used Correctly?
Wan, C., Liu, S., Hoffmann, H., Maire, M., & Lu, S. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00024 · Cited by 41 · see citing works
Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing
Yuan, Y., Wang, S., Jiang, M., & Chen, T. Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01663 · Cited by 41 · see citing works
A Review of Formal Methods applied to Machine LearningPreprint
Urban, C., & Min'e, A. (2021) · arXiv.org · doi:10.48550/arxiv.2104.02466 · Cited by 81 · see citing works
Quality assurance strategies for machine learning applications in big data analytics: an overview
Ogrizović, M., Drašković, D., & Bojić, D. M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01028-y · Cited by 40 · see citing works
Advancing the Study of Human Values in Software Engineering
Winter, E., Forshaw, S., Hunt, L., & Ferrario, M. A. (2019) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2019.00012 · Cited by 40 · see citing works
Operationalizing Human Values in Software Engineering: A Survey
Shahin, M., Hussain, W., Nurwidyantoro, A., Perera, H., Shams, R. A., Grundy, J., & Whittle, J. (2022) · IEEE Access · doi:10.1109/access.2022.3190975 · Cited by 40 · see citing works
Testing DNN image classifiers for confusion & bias errors
Tian, Y., Zhong, Z., Ordonez, V., Kaiser, G. E., & Ray, B. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380400 · Cited by 39 · see citing works
How Can Human Values Be Addressed in AgileMethods A Case Study on SAFe
Hussain, W., Shahin, M., Hoda, R., Whittle, J., Perera, H., Nurwidyantoro, A., Shams, R. A., & Oliver, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3140230 · Cited by 39 · see citing works
The favela effect: Spatial inequalities and firm strategies in disadvantaged urban communities
Pongeluppe, L. (2022) · Strategic Management Journal · doi:10.1002/smj.3414 · Cited by 38 · see citing works
Personalized chatbot trustworthiness ratings
Srivastava, B., Rossi, F., Usmani, S., & Bernagozzi, M. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.3023919 · Cited by 40 · see citing works
- Authors
- Galhotra, S., Brun, Y., & Meliou, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2017-08-21
- Pages
- 498-510
- OpenAlex
- W2730550703
- Cited by
- 405
- References
- 91
- Citations by year
- 2026: 26 · 2025: 55 · 2024: 46 · 2023: 56 · 2022: 50 · 2021: 53 · 2020: 57 · 2019: 37 · 2018: 22 · 2017: 3
- FWCI
- 23.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (CCF-1453474, IIS-1453543, CNS-1744471)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic discrimination, fairness testing, automated test generation, THEMIS, algorithmic fairness, automated decision systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
