The DOI matches its DataCite record; 49 of the 50 references it cites were found in a registry or an index.
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Open accessAbstract
In a world where traditional notions of privacy are increasingly challenged by the myriad companies that collect and analyze our data, it is important that decision-making entities are held accountable for unfair treatments arising from irresponsible data usage. Unfortunately, a lack of appropriate methodologies and tools means that even identifying unfair or discriminatory effects can be a challenge in practice. We introduce the unwarranted associations (UA) framework, a principled methodology for the discovery of unfair, discriminatory, or offensive user treatment in data-driven applications. The UA framework unifies and rationalizes a number of prior attempts at formalizing algorithmic fairness. It uniquely combines multiple investigative primitives and fairness metrics with broad applicability, granular exploration of unfair treatment in user subgroups, and incorporation of natural notions of utility that may account for observed disparities. We instantiate the UA framework in FairTest, the first comprehensive tool that helps developers check data-driven applications for unfair user treatment. It enables scalable and statistically rigorous investigation of associations between application outcomes (such as prices or premiums) and sensitive user attributes (such as race or gender). Furthermore, FairTest provides debugging capabilities that let programmers rule out potential confounders for observed unfair effects. We report on use of FairTest to investigate and in some cases address disparate impact, offensive labeling, and uneven rates of algorithmic error in four data-driven applications. As examples, our results reveal subtle biases against older populations in the distribution of error in a predictive health application and offensive racial labeling in an image tagger.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Even a very accurate model may introduce systematic errors concentrated on certain classes of messages or of users (Cohen and Ruths, 2013; Pavalanathan and Eisenstein, 2015; Tramer et al., 2015).”
From Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019) · cites this paper as Tramer et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190
(1992) · Cited by 23,431
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
Matrix Factorization Techniques for Recommender Systems
Koren, Y., Bell, R. M., & Volinsky, C. (2009) · Computer · doi:10.1109/mc.2009.263 · Cited by 11,232
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
The MovieLens Datasets: History and Context
Harper, F., & Konstan, J. (2016) · TIIS · doi:10.1145/2827872 · Cited by 5,938
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Thirteen Ways to Look at the Correlation Coefficient
Rodgers, J. L., & Nicewander, W. A. (1988) · The American Statistician · doi:10.1080/00031305.1988.10475524 · Cited by 3,004
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
Measuring and testing dependence by correlation of distances
Székely, G. J., Rizzo, M. L., & Bakirov, N. K. (2007) · The Annals of Statistics · doi:10.1214/009053607000000505 · Cited by 2,684
The interpretation of interaction in contingency tables
Simpson, E. H. (1951) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1951.tb00088.x · Cited by 2,301 · cited 1×
“Yet prior work has shown that discrim inatory effects in a population may differ from, and even contradict, those exhibited in smaller subsets, an effect known as Simpson’s paradox [41].”
BackgroundMLlib: Machine Learning in Apache SparkPreprint
Meng, X., Bradley, J. K., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D., Amde, M., Owen, S., et al. (2015) · Journal of machine learning research · doi:10.48550/arxiv.1505.06807 · Cited by 1,898
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Measuring Statistical Dependence with Hilbert-Schmidt Norms
Gretton, A., Bousquet, O., Smola, A. J., & Schölkopf, B. (2005) · Lecture notes in computer science · doi:10.1007/11564089_7 · Cited by 1,668
Estimation of entropy and mutual information
Paninski, L. (2003) · Neural Computation · doi:10.1162/089976603321780272 · Cited by 1,442
(2013) · Cited by 1,126
Permutation Methods: A Basis for Exact Inference
Ernst, M. (2004) · Statistical Science · doi:10.1214/088342304000000396 · Cited by 858
Three naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764
Fairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676
Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
Datta, A., Tschantz, M. C., & Datta, A. (2015) · Proceedings on Privacy Enhancing Technologies · doi:10.1515/popets-2015-0007 · Cited by 676
Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975) · Science · doi:10.1126/science.187.4175.398 · Cited by 659
UPTON, G. J. G., & Cook, I. (2008) · Oxford University Press eBooks · doi:10.1093/acref/9780199541454.001.0001 · Cited by 493
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 405 · cited 1×
“3), the field of web transparency [1,3,6,17,18,26,27,29,47,48] is closest to our work; indeed, our team contributed two scalable and generic system designs [26, 27].”
BackgroundMeasuring personalization of web search
Hannak, A., Sapiezynski, P., Molavi Kakhki, A., Krishnamurthy, B., Lazer, D., Mislove, A., & Wilson, C. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488435 · Cited by 396
Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife
Wager, S., Hastie, T. J., & Efron, B. (2013) · PubMed · doi:10.48550/arxiv.1311.4555 · Cited by 375
Scott, A. J., Hosmer, D. W., & Lemeshow, S. A. (1991) · Biometrics · doi:10.2307/2532419 · Cited by 372
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312
The reusable holdout: Preserving validity in adaptive data analysis
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015) · Science · doi:10.1126/science.aaa9375 · Cited by 307
Discrimination Aware Decision Tree Learning
Kamiran, F., Calders, T., & Pechenizkiy, M. (2010) · 2010 IEEE International Conference on Data Mining · doi:10.1109/icdm.2010.50 · Cited by 303
Kroll, J. A., Huey, J., Barocas, S., Felten, E. W., Reidenberg, J. R., Robinson, D. G., & Yu, H. (2017) · FLASH - Fordham Law Archive of Scholarship & History (Fordham University) · Cited by 299
k-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263
Data mining for discrimination discovery
Ruggieri, S., Pedreschi, D., & Turini, F. (2010) · TKDD · doi:10.1145/1754428.1754432 · Cited by 250
Handling Conditional Discrimination
liobaite, I., Kamiran, F., & Calders, T. (2011) · 2011 IEEE 11th International Conference on Data Mining · doi:10.1109/icdm.2011.72 · Cited by 125
XRay: Enhancing the Web's Transparency with Differential CorrelationPreprint
Lécuyer, M., Ducoffe, G., Lan, F., Papancea, A., Petsios, T., Spahn, R., Chaintreau, A., & Geambasu, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.2323 · Cited by 87
Self-citationBootstrapping Privacy Compliance in Big Data Systems
Sen, S., Guha, S., Datta, A., Rajamani, S. K., Tsai, J., & Wing, J. M. (2014) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2014.28 · Cited by 81
Barford, P., Canadi, I., Krushevskaja, D., Ma, Q., & Muthukrishnan, S. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2567992 · Cited by 75
Liu, B., Sheth, A. N., Weinsberg, U., Chandrashekar, J., & Govindan, R. (2013) · ACM Workshop on Hot Topics in Networks (HotNets) · doi:10.1145/2535771.2535783 · Cited by 68
Nonparametric Divergence Estimation with Applications to Machine Learning on DistributionsPreprint
Póczos, B., Xiong, L., & Schneider, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.3758 · Cited by 66
(2014) · Lirias · Cited by 61
Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence
Lecuyer, M., Spahn, R., Spiliopolous, Y., Chaintreau, A., Geambasu, R., & Hsu, D. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813614 · Cited by 48
Self-citationAlgorithmic Transparency via Quantitative Input Influence
Datta, A., Sen, S., & Zick, Y. (2017) · Studies in big data · doi:10.1007/978-3-319-54024-5_4 · Cited by 43
Integrating induction and deduction for finding evidence of discrimination
Ruggieri, S., Pedreschi, D., & Turini, F. (2010) · Artificial Intelligence and Law · doi:10.1007/s10506-010-9089-5 · Cited by 41
Understanding classifier errors by examining influential neighbors
Kabra, M., Robie, A. A., & Branson, K. M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299017 · Cited by 24
Research works printed but not linked: 2
- [24][24] P. Bickel, E. Hammel, and J. O'Connell, 'Sex bias in graduate admissions: Data from Berkeley,' Science , vol. 187, no. 4175, pp. 398-404, 1975. -
- [29][29] B. Liu, A. Sheth, U. Weinsberg, J. Chandrashekar, and R. Govindan, 'AdReveal: improving transparency into online targeted advertising,' in HotNets-XII . ACM, Nov. 2013. -
Cited by
Sorted by citations
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646
Metadata
- Authors
- Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H.
- Affiliations
- Stanford University (US)
- Columbia University (US)
- École Polytechnique Fédérale de Lausanne (CH)
- Saarland University (DE)
- Jacobs Institute (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- Pages
- 401-416
- arXiv
- 1510.02377
- OpenAlex
- W2964116855
- Cited by
- 171
- References
- 50
- Citations by year
- 2026: 4 · 2025: 15 · 2024: 20 · 2023: 21 · 2022: 33 · 2021: 23 · 2020: 21 · 2019: 20 · 2018: 10 · 2017: 4
- FWCI
- 37.48
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE European Symposium on Security and Privacy (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- IRIS (submittedVersion)
- SERVAL (Université de Lausanne) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- algorithmic fairness, algorithmic bias detection, indirect discrimination, sensitive attributes, fairness metrics, data-driven applications, image tagging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
