FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
The growing capability and accessibility of machine learning has led to its application to many real-world domains and data about people. Despite the benefits algorithmic systems may bring, models can reflect, inject, or exacerbate implicit and explicit societal biases into their outputs, disadvantaging certain demographic subgroups. Discovering which biases a machine learning model has introduced is a great challenge, due to the numerous definitions of fairness and the large number of potentially impacted subgroups. We present FairVis, a mixed-initiative visual analytics system that integrates a novel subgroup discovery technique for users to audit the fairness of machine learning models. Through FairVis, users can apply domain knowledge to generate and investigate known subgroups, and explore suggested and similar subgroups. FairVis' coordinated views enable users to explore a high-level overview of subgroup performance and subsequently drill down into detailed investigation of specific subgroups. We show how FairVis helps to discover biases in two real datasets used in predicting income and recidivism. As a visual analytics system devoted to discovering bias in machine learning, FairVis demonstrates how interactive visualization may help data scientists and the general public understand and create more equitable algorithmic systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Some research groups have developed open-source toolkits allowing technology providers to test and evaluate the performance of AI systems on different tasks and datasets (Cabrera et al., 2019;Saleiro et al., 2018).”
From Auditing of AI: Legal, Ethical and Technical Approaches (2023) · cites this paper as Cabrera et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,410
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
(2007) · doi:10.5555/1283383.1283494 · Cited by 7,364
(2018) · Cited by 3,510
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“When a biased subgroup has been identified, it can be informative to look at the performance of similar subgroups to draw conclusions about feature importance or to create more general groups [13, 38].”
BackgroundBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“The potential disparity in performance between populations may have many sources; an ML model can naturally encode implicit and explicit societal biases [5], which is often referred to as algorithmic bias.”
BackgroundEquality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 3×
“These range from techniques for obfuscating sensitive variables in training data [37], to new regularization parameters for training [6] and post-processing outcomes by adding noise to predictions [17].”
BackgroundFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Subspace clustering for high dimensional data: a review
Parsons, L., Haque, E., & Liu, H. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007731 · Cited by 1,346 · cited 1×
“Additionally, we plan to experiment with more clustering techniques, such as subspace clustering methods [29] to future versions of F AIR V IS so that users can see how the groups compare.”
Method(2013) · Cited by 1,126
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995 · cited 1×
“exploration tool that combines dataset exploration with counterfactual explanations and fairness modifications. Users can explore a dataset using the Facets interface, and then look for counterfactual [27] explanations for specific instances. There is also a feature that allows users to modify a classifier’s threshold to…”
MethodInherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943 · cited 1×
“Two papers [14, 24] simultaneously proved that if groups have different base rates in their labels, it is statistically impossible to ensure fairness across three base fairness metrics — balance for the positive class, balance for the negative class, and calibration of the model.”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941 · cited 1×
“Two papers [14, 24] simultaneously proved that if groups have different base rates in their labels, it is statistically impossible to ensure fairness across three base fairness metrics — balance for the positive class, balance for the negative class, and calibration of the model.”
BackgroundImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 1×
“A system should enable users to generate any type of subgroup they want to investigate, and efficiently generate and calculate metrics for it [20].”
BackgroundVisual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582
Self-citationVisual comparison for information visualization
Gleicher, M., Albers, D., Walker, R., Jusufi, I., Hansen, C. D., & Roberts, J. C. (2011) · Information Visualization · doi:10.1177/1473871611416549 · Cited by 567 · cited 1×
“By juxtaposing plots, users are able to see how different metrics are spread out [15].”
BackgroundVeale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 469 · cited 2×
“When investigating intersectional bias, there could be hundreds or thousands of subgroups a user may need to look at [23].”
BackgroundClustering cancer gene expression data: a comparative study
de Souto, M. C., Costa, I. G., de Araujo, D. S., Ludermir, T. B., & Schliep, A. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-497 · Cited by 444
FairGAN: Fairness-aware Generative Adversarial Networks
Xu, D., Yuan, S., Zhang, L., & Wu, X. (2018) · 2018 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2018.8622525 · Cited by 379
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Kahng, M., Andrews, P. Y., Kalro, A., & Chau, D. H. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744718 · Cited by 337 · cited 1×
“Additionally, systems have been introduced that enable users to analyze production-level models [17] and the full ML workflow from training to production [2, 21, 23].”
Self-citationInherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Application of k Means Clustering algorithm for prediction of Students Academic Performance
Oyelade, J. O., Oladipupo, O. O., & Obagbuwa, I. C. (2010) · Covenant University Repository (Covenant University) · doi:10.48550/arxiv.1002.2425 · Cited by 282 · cited 1×
“Given prior successful application of K-means to a variety of problems and tasks with both categorical and numerical features, we decided to first adapt K-means for F AIR V IS [10, 28].”
MethodModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702509 · Cited by 262 · cited 2×
“Additionally, systems have been introduced that enable users to analyze production-level models [21] and the full ML workflow from training to production [2, 25, 30].”
BackgroundOn the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247
Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
Strobelt, H., Gehrmann, S., Behrisch, M., Perer, A., Pfister, H., & Rush, A. M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865044 · Cited by 239
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Ren, D., Amershi, S., Lee, B., Suh, J., & Williams, J. D. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598828 · Cited by 200 · cited 2×
“Squares [32] introduces a novel encoding for visually understanding the performance of a multi-class classifier.”
BackgroundPredictive Inequity in Object Detection
Wilson, B., Hoffman, J., & Morgenstern, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.11097 · Cited by 149 · cited 1×
“The latter two cases are examples where differing levels of predictive accuracy have been observed for different demographic groups [8, 36].”
BackgroundSelf-citationGestalt: integrated support for implementation and analysis in machine learning
Patel, K., Bancroft, N., Drucker, S. M., Fogarty, J., Ko, A. J., & Landay, J. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866038 · Cited by 112 · cited 2×
“Additionally, systems have been introduced that enable users to analyze production-level models [21] and the full ML workflow from training to production [2, 25, 30].”
BackgroundPutting Fairness Principles into Practice: Challenges, Metrics, and Improvements
Beutel, A., Chen, J., Doshi, T., Qian, H., Woodruff, A., Luu, C., Kreitmann, P., Bischof, J., & Chi, E. H. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314234 · Cited by 111 · cited 1×
“These range from techniques for obfuscating sensitive variables in training data [37], to new regularization parameters for training [6] and post-processing outcomes by adding noise to predictions [17].”
BackgroundA Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations
Krause, J., Dasgupta, A., Swartz, J., Aphinyanaphongs, Y., & Bertini, E. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585720 · Cited by 86
Slice Finder: Automated Data Slicing for Model Validation
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019) · 2019 IEEE 35th International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00139 · Cited by 83
Visual exploration of machine learning results using data cube analysis
Kahng, M., Fang, D., & Chau, D. H. (. (2016) · Proceedings of the Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/2939502.2939503 · Cited by 77 · cited 2×
“MLCube [22] is a visualization technique that allows users to compare the accuracy of groups defined by at most two features.”
BackgroundMethodSelf-citationVeale, M., Van Kleek, M., & Binns, R. (2018) · doi:10.31235/osf.io/8kvf4 · Cited by 60
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2011) · arXiv · doi:10.48550/arxiv.1104.3913 · Cited by 55 · cited 1×
“When a biased subgroup has been identified, it can be informative to look at the performance of similar subgroups to draw conclusions about feature importance or to create more general groups [13, 38].”
BackgroundPutting Fairness Principles into Practice: Challenges, Metrics, and ImprovementsPreprint
Beutel, A., Chen, J., Doshi, T. S., Qian, H., Woodruff, A., Luu, C., Kreitmann, P., Bischof, J., & Ed, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04562 · Cited by 29 · cited 1×
“These range from techniques for obfuscating sensitive variables in training data [37], to new regularization parameters for training [6] and post-processing outcomes by adding noise to predictions [17].”
Background
Research works printed but not linked: 7
- 2019, COMPAS recidivism risk score data and analysis
- 10.2139/ssrn.3063289 · DOI · Paper page
- dieterich, 2016, COMPAS risk scales Demonstrating accuracy equity and predictive parity
- 2018, What-If Tool
- angwin, 2016, Machine Bias
- 2017, Facets - visualizations for ML datasets
- krause, 0, A user study on the effect of aggregating explanations for interpreting machine learning models, Interactive Data Exploration and Analytics Workshop at KDD
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“There has been insufficient research into intersectionality in educational work on algorithmic bias, and indeed in algorithmic bias more generally (Cabrera et al., 2019).”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
“Examples including allowing data scientists and model experts to interactively explore model representations [86], analyze model training processes [129], and detect learning biases [27].”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
Metadata
- Authors
- Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H.
- Affiliations
- Georgia Institute of Technology (US)
- ORCID
- Venue
- 2019 IEEE Conference on Visual Analytics Science and Technology (VAST)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 46-56
- OpenAlex
- W2939675332
- Cited by
- 14
- References
- 36
- Citations by year
- 2024: 1 · 2023: 2 · 2022: 2 · 2021: 4 · 2020: 4 · 2019: 1
- FWCI
- 2.87
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Conference on Visual Analytics Science and Technology (VAST) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- SMARTech Repository (Georgia Institute of Technology) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- intersectional bias, algorithmic fairness, subgroup discovery, fairness auditing, visual analytics, mixed-initiative interaction, coordinated views, income prediction, recidivism prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.