The DOI matches its DataCite record; 49 of the 50 references it cites were found in a registry or an index.
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Open accessAbstract
In a world where traditional notions of privacy are increasingly challenged by the myriad companies that collect and analyze our data, it is important that decision-making entities are held accountable for unfair treatments arising from irresponsible data usage. Unfortunately, a lack of appropriate methodologies and tools means that even identifying unfair or discriminatory effects can be a challenge in practice. We introduce the unwarranted associations (UA) framework, a principled methodology for the discovery of unfair, discriminatory, or offensive user treatment in data-driven applications. The UA framework unifies and rationalizes a number of prior attempts at formalizing algorithmic fairness. It uniquely combines multiple investigative primitives and fairness metrics with broad applicability, granular exploration of unfair treatment in user subgroups, and incorporation of natural notions of utility that may account for observed disparities. We instantiate the UA framework in FairTest, the first comprehensive tool that helps developers check data-driven applications for unfair user treatment. It enables scalable and statistically rigorous investigation of associations between application outcomes (such as prices or premiums) and sensitive user attributes (such as race or gender). Furthermore, FairTest provides debugging capabilities that let programmers rule out potential confounders for observed unfair effects. We report on use of FairTest to investigate and in some cases address disparate impact, offensive labeling, and uneven rates of algorithmic error in four data-driven applications. As examples, our results reveal subtle biases against older populations in the distribution of error in a predictive health application and offensive racial labeling in an image tagger.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Even a very accurate model may introduce systematic errors concentrated on certain classes of messages or of users (Cohen and Ruths, 2013; Pavalanathan and Eisenstein, 2015; Tramer et al., 2015).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Tramer et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190
(1992) · Cited by 23,431
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
Matrix Factorization Techniques for Recommender Systems
Koren, Y., Bell, R. M., & Volinsky, C. (2009) · Computer · doi:10.1109/mc.2009.263 · Cited by 11,232
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
The MovieLens Datasets: History and Context
Harper, F., & Konstan, J. (2016) · TIIS · doi:10.1145/2827872 · Cited by 5,938
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
Research works printed but not linked: 2
- [24][24] P. Bickel, E. Hammel, and J. O'Connell, 'Sex bias in graduate admissions: Data from Berkeley,' Science , vol. 187, no. 4175, pp. 398-404, 1975. -
- [29][29] B. Liu, A. Sheth, U. Weinsberg, J. Chandrashekar, and R. Govindan, 'AdReveal: improving transparency into online targeted advertising,' in HotNets-XII . ACM, Nov. 2013. -
Cited by
Sorted by citations
Items 101–170 of 170
Exploring Server-side Blocking of RegionsPreprint
Afroz, S., Tschantz, M. C., Sajid, S., Qazi, S. A., Javed, M., & Paxson, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11606 · Cited by 9
Fairness, explainability, privacy, and robustness for trustworthy algorithmic decision-making
Majumdar, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-85713-0.00017-7 · Cited by 10
Democratizing Quality-Based Machine Learning Development through Extended Feature Models
d’Aloisio, G., Marco, A. D., & Stilo, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30826-0_5 · Cited by 8
Seldonian Toolkit: Building Software with Safe and Fair Machine Learning
Hoag, A., Kostas, J., da Silva, B. C., Thomas, P. S., & Brun, Y. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion58688.2023.00035 · Cited by 8
E-FAIR-DB: Functional Dependencies to Discover Data Bias and Enhance Data Equity
Azzalini, F., Criscuolo, C., & Tanca, L. (2022) · Journal of Data and Information Quality · doi:10.1145/3552433 · Cited by 8
Assessing Fairness in Classification Parity of Machine Learning Models in HealthcarePreprint
Yuan, M., Kumar, V., Ahmad, M. A., & Teredesai, A. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.03717 · Cited by 8
BiasTrap: Runtime Detection of Biased Prediction in Machine Learning Systems
Mamman, H., Basri, S. B., Balogun, A. O., Imam, A. A., Kumar, G., & Capretz, L. F. (2024) · Journal of Advanced Research in Applied Sciences and Engineering Technology · doi:10.37934/araset.40.2.127139 · Cited by 9
Zaridis, D. I., Pezoulas, V. C., Mylona, E., Kalantzopoulos, C., Tachos, N. S., Tsiknakis, N., Matsopoulos, G. K., Regge, D., Papanikolaou, N. K., Tsiknakis, M., et al. (2025) · Bioengineering · doi:10.3390/bioengineering12030242 · Cited by 7
Johnson, B., Bartola, J., Angell, R., Witty, S., Giguere, S., & Brun, Y. (2023) · EURO Journal on Decision Processes · doi:10.1016/j.ejdp.2023.100031 · Cited by 18
Andrews, P. (2019) · Online Information Review · doi:10.1108/oir-09-2018-0284 · Cited by 10
Attesting Digital Discrimination Using Norms.
Criado, N., Ferrer, X., & Such, J. M. (2021) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2021.02.008 · Cited by 15
Johnson, B., Bartola, J., Angell, R., Keith, K. A., Witty, S., Giguere, S., & Brun, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09951 · Cited by 7
PCIV method for Indirect Bias Quantification in AI and ML Models
Garg, A., & Dr. Rajesh, S. L. (2021) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit217251 · Cited by 6
A Normative approach to Attest Digital DiscriminationPreprint
Criado, N., Ferrer, X., & Such, J. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07092 · Cited by 6
Learning Fair Naive Bayes Classifiers by Discovering and Eliminating Discrimination Patterns
Choi, Y., Farnadi, G., Babaki, B., & Van den Broeck, G. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i06.6565 · Cited by 5
An empirical study of testing practices in open source AI agent frameworks and agentic applications
Hasan, M. M., Li, H., Fallahzadeh, E., Rajbahadur, G. K., Adams, B., & Hassan, A. E. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10857-9 · Cited by 25
Bias-aware guidelines and fairness-preserving Taxonomy in software engineering education
Spoletini, P., & Parizi, R. M. (2018) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8659178 · Cited by 5
Privileged and unprivileged groups
Hort, M., & Sarro, F. (2022) · doi:10.1145/3524491.3527308 · Cited by 8
Specifying and Testing k-Safety Properties for Machine-Learning Models
Christakis, M., Enişer, H. F., Hoffmann, J., Singla, A. K., & Wüstholz, V. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/528 · Cited by 8
Fairness and Explainability for Enabling Trust in AI Systems
Sacharidis, D. (2024) · Human-computer interaction series · doi:10.1007/978-3-031-55109-3_3 · Cited by 5
AutoRIC: Automated Neural Network Repairing Based on Constrained Optimization
Sun, X., Liu, W., Wang, S., Chen, T., Tao, Y., & Mao, X. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3690634 · Cited by 7
Optimizing Fairness in Machine Learning: A Hyperparameter Tuning Approach
Nadeem, M. A. (2025) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins65080.2025.11125753 · Cited by 4
Shrestha, H., Cachel, K., Alkhathlan, M., Rundensteiner, E. A., & Harrison, L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594108 · Cited by 4
Stream-Based Monitoring of Algorithmic Fairness
Baumeister, J., Finkbeiner, B., Scheerer, F., Siber, J., & Wagenpfeil, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90643-5_4 · Cited by 4
Detection and Mitigation of Rare Subclasses in Deep Neural Network Classifiers
Paterson, C., Călinescu, R., & Picardi, C. (2021) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest52744.2021.00012 · Cited by 8
FlipTest: Fairness Auditing via Optimal Transport.Preprint
Black, E., Yeom, S., & Fredrikson, M. (2019) · arXiv (Cornell University) · Cited by 4
The Impact of Presentation Style on Human-In-The-Loop Detection of Algorithmic BiasPreprint
Law, P.-M., Malik, S., Du, F., & Sinha, M. (2020) · arXiv (Cornell University) · doi:10.20380/gi2020.30 · Cited by 4
Black-Box Testing and Auditing of Bias in ADM Systems
Krafft, T. D., Hauer, M. P., & Zweig, K. A. (2024) · Minds and Machines · doi:10.1007/s11023-024-09666-0 · Cited by 7
Beyond the Seeds: Fairness Testing via Counterfactual Analysis of Non-Seed Instances
Mamman, H., Basri, S. B., Balogun, A. O., Gilal, A. R., Imam, A. A., Kumar, G., & Capretz, L. F. (2024) · IEEE Access · doi:10.1109/access.2024.3502164 · Cited by 3
Guerdan, L., Saxena, D., Chancellor, S., Wu, Z. S., & Holstein, K. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757628 · Cited by 4
Towards Assessing Data Bias in Clinical Trials
Criscuolo, C., Dolci, T., & Salnitri, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-23905-2_5 · Cited by 3
Black-Box Fairness Testing with Shadow Models
Jiang, W., Shen, C. Q., Lin, C., Wang, J., Sun, J., & Gao, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-7356-9_28 · Cited by 3
Fundamental Approaches to Software Engineering
Lambers, L., & Uchitel, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-30826-0 · Cited by 5
Towards Interactively Improving ML Data Preparation Code via "Shadow Pipelines"
Grafberger, S., Groth, P., & Schelter, S. (2024) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3650203.3663327 · Cited by 3
Nahar, N., Zhou, S., Lewis, G. A., & Kästner, C. (2021) · arXiv (Cornell University) · Cited by 3
Unbiasing on the Fly: Explanation-Guided Human Oversight of Machine Learning Decisions
Mamman, H., Basri, S. B., Balogun, A. O., Imam, A. A., Kumar, G., & Capretz, L. F. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-70300-3_20 · Cited by 5
FairZK: A Scalable System to Prove Machine Learning Fairness in Zero-Knowledge
Zhang, T., Dong, S., Kose, O. D., Shen, Y., & Zhang, Y. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00205 · Cited by 5
FairSense: Long-Term Fairness Analysis of ML-Enabled Systems
She, Y., Biswas, S., Kästner, C., & Kang, E. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00159 · Cited by 2
Locating disparities in machine learning
von Zahn, M., Hinz, O. F., & Feuerriegel, S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386485 · Cited by 6
FIPSER: Improving Fairness Testing of DNN by Seed Prioritization
Chen, J., Zhang, Y., Zhang, L., Zhang, M., Wan, C., Su, T., & Pu, G. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695486 · Cited by 2
A New Reinforcement Learning-Based Framework for Unbiased Autonomous Software Systems
Sulaimon, I. A., Ghoneim, A., & Alrashoud, M. R. (2019) · International Conference on Modeling, Simulation, and Applied Optimization · doi:10.1109/icmsao.2019.8880288 · Cited by 3
Fairway: SE Principles for Building Fairer Software.
Chakraborty, J., Majumder, S., Wu, Z., & Menzies, T. (2020) · Cited by 2
A Review of Bias and Discrimination In Digital Platforms
Dabo, A. M., Sarki, A. M., Fapohunda, S. E., & Longe, E. O. (2022) · Advances in Multidisciplinary & Scientific Research Journal Publication · doi:10.22624/aims/digital/v10n1p5 · Cited by 2
Effects of Adversarial Training on the Safety of Classification Models
Kim, H., & Han, J. (2022) · Symmetry · doi:10.3390/sym14071338 · Cited by 2
PanViz 2.0: Intregating AI into Visual analytics to adapt to the novel challenges of COVID-19
Snyder, L. S., Reinert, A., & Ebert, D. S. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.176 · Cited by 2
Detection and Mitigation of Rare Subclasses in Neural Network ClassifiersPreprint
Paterson, C., & Călinescu, R. (2019) · arXiv (Cornell University) · Cited by 2
An Analysis of Pairwise PreferencePreprint
Kronovet, D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3359677 · Cited by 2
FALE: Fairness-Aware ALE Plots for Auditing Bias in Subgroups
Giannopoulos, G., Sacharidis, D., Theologitis, N., Kavouras, L., & Emiris, I. Z. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-74627-7_37 · Cited by 1
The Cost of Balanced Training-Data Production in an Online Data Market
Chaintreau, A., Maio, R., & Ziani, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714882 · Cited by 1
Fair-FLIP: Fair Deepfake Detection with Fairness-Oriented Final Layer Input Prioritising
Szandała, T., Ezzeddine, F., Rusin, N., Giordano, S., & Ayoub, O. (2025) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds66131.2025.00021 · Cited by 3
Causally Perturbed Fairness Testing
Du, C., & Chen, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773088 · Cited by 1
Automated Fairness Testing with Representative Sampling
Karakas, U., & Tosun, A. (2023) · doi:10.1145/3617555.3617871 · Cited by 1
Yuan, B., Gui, S., Zhang, Q., Wang, Z., Wen, J., Mao, B., Liu, J., & Yao, X. (2024) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2024.3364430 · Cited by 3
Du, C., & Chen, T. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3686673 · Cited by 1
Proxy Attribute Discovery in Machine Learning Datasets via Inductive Logic Programming
Gonçalves, R. S. D. M., Gouveia, F., Lynce, I., & Santos, J. F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90653-4_17 · Cited by 2
FAIR-DB: FunctionAl DependencIes to discoveR Data Bias.
Azzalini, F., Criscuolo, C., & Tanca, L. (2021) · EDBT/ICDT Workshops · Cited by 1
Towards Integrating Fairness Transparently in Industrial ApplicationsPreprint
Dodwell, E., Flynn, C., Krishnamurthy, B., Majumdar, S., & Mitra, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06082 · Cited by 2
Is it ethical to avoid error analysis?Preprint
García-Martín, E., & Lavesson, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.10237
Comparing Explanations between Random Forests and Artificial Neural Networks
Harris, L., & Grześ, M. (2019) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2019.8914321
Testing, Debugging, and Repairing Individual Discrimination in Machine Learning Models
Saha, D., Agarwal, A., Hans, S., & Haldar, S. (2023) · Studies in computational intelligence · doi:10.1007/978-981-99-7184-8_1
Interpretable White-Box Fairness Testing Through Biased Neuron Identification
Chen, J., Zhang, X., & Zheng, H. (2024) · Springer eBooks · doi:10.1007/978-981-97-0425-5_19
OxonFair: A Flexible Toolkit for Algorithmic FairnessPreprint
Delaney, E. D., Fu, Z., Wachter, S., Mittelstadt, B., & Russell, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4894794 · Cited by 8
Towards Assessing Integrated Differential Privacy and Fairness Mechanisms in Supervised Learning
Aldairi, M., & Joshi, J. B. D. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00034
Balancing Accuracy and Safety in AI: A Novel Adversarial Training Approach
Atkuri, V. V. N. K. C. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-83793-7_21
Ethical Considerations and Challenges of Leveraging IoT and Machine Learning in Smart Urban Planning
Mohtashim, M. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-9030-6.ch007
Towards Practical Software Discrimination Testing
Mim, S. A. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00061
Muzaisham, P. A. H. B., Balogun, A. O., Basri, S. B., Mamman, H., Capretz, L. F., & Lawal, A. B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20749-4_30
Muzaisham, P. A. H. B., Balogun, A. O., Basri, S. B., Mamman, H., Capretz, L. F., & Lawal, A. B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20746-3_30
Measuring Database Unfairness via Dependency Quantification Under Differential Privacy
Vologdin, M., Tao, Y., & Gilad, A. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3819518.3819560
Metadata
- Authors
- Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- École Polytechnique Fédérale de Lausanne (CH)
- Saarland University (DE)
- Jacobs Institute (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- Pages
- 401-416
- arXiv
- 1510.02377
- OpenAlex
- W2964116855
- Cited by
- 171
- References
- 50
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 20 · 2023: 21 · 2022: 33 · 2021: 23 · 2020: 21 · 2019: 20 · 2018: 10 · 2017: 4
- FWCI
- 37.48
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE European Symposium on Security and Privacy (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS (submittedVersion)
- SERVAL (Université de Lausanne) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- algorithmic fairness, algorithmic bias detection, indirect discrimination, sensitive attributes, fairness metrics, data-driven applications, image tagging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
