The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Explaining models: an empirical study of how explanations impact fairness judgment
Open accessAbstract
Ensuring fairness of machine learning systems is a human-in-the-loop process. It relies on developers, users, and the general public to identify fairness problems and make improvements. To facilitate the process we need effective, unbiased, and user-friendly explanations that people can confidently rely on. Towards that end, we conducted an empirical study with four types of programmatically generated explanations to understand how they impact people's fairness judgments of ML systems. With an experiment involving more than 160 Mechanical Turk workers, we show that: 1) Certain explanations are considered inherently less fair, while others can enhance people's confidence in the fairness of the algorithm; 2) Different fairness problems-such as model-wide fairness issues versus case-specific fairness discrepancies-may be more effectively exposed through different styles of explanation; 3) Individual differences, including prior positions and judgment criteria of algorithmic fairness, impact how people react to different styles of explanation. We conclude with a discussion on providing personalized and adaptive explanations to support fairness judgments of ML systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They found that local explanations, by highlighting unfair features used for individual predictions, appear to be more alarming than global explanations when used to explain an unfair model's decisions, and thus more effective in calibrating people's fairness judgment of ML models.”
From Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making (2020) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 2×
“Much recent research is dedicated to the generation of explanations in various styles, includingmodel-agnostic approaches [26, 30] applicable to any ML algorithm.”
BackgroundMethod"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 3×
“For example, LIME explains feature contribution by what happens to the prediction when the feature-values change (perturbing data) [30].”
BackgroundMethodGauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Explanation in Artificial Intelligence: Insights from the Social Sciences
Miller, T. (2017) · arXiv · doi:10.48550/arxiv.1706.07269 · Cited by 5,338 · cited 1×
“However, this growing body of research is criticized for łapproaching this [XAI] challenge in a vacuum considering only the computational problemsž [27] without the quintessential understanding of how people perceive and use the explanations.”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial NetsPreprint
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 3,557
The Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913 · cited 2×
“For cognitive style, we measure individual’s need for cognition [7].”
MethodLee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 1×
“There has been a recent resurgence of XAI work driven by the challenge to interpret increasingly complex ML models, such as multi-layered neural networks, and by the evidence that ethical concerns and lack of trust hampers adoption of AI applications [15, 22].”
Background(2013) · Cited by 1,126
Counterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1×
“Research shows that by optimizing for the unitary goal of accuracy, ML algorithms trained on historical data not only replicate, but may amplify existing biases or discrimination [40].”
BackgroundAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
The epistemology of a rule-based expert system —a framework for explanation
Clancey, W. J. (1983) · Artificial Intelligence · doi:10.1016/0004-3702(83)90008-5 · Cited by 712 · cited 1×
“Work on model explanations can be traced to early work on expert systems [11, 33], which often explicitly revealed reasoning rules to end-users.”
BackgroundFairness-Aware Classifier with Prejudice Remover Regularizer
Kamishima, T., Akaho, S., Asoh, H., & Sakuma, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33486-3_3 · Cited by 694 · cited 1×
“, [16, 17, 38]), equal attention has not been paid to the human scrutiny necessary to identify and remedy fairness issues.”
Background'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691
Zafar, M. B., Valera, I., Gomez Rodriguez, M., & Gummadi, K. P. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052660 · Cited by 632 · cited 1×
“, [16, 17, 38]), equal attention has not been paid to the human scrutiny necessary to identify and remedy fairness issues.”
Background(2016) · neural information processing systems · Cited by 556
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1×
“The use of a regression model is ecologically valid—many current decision support systems use such simple and interpretable models [35].”
BackgroundOptimized Pre-Processing for Discrimination Prevention
Calmon, F. P., Wei, D., Vinzamuri, B., Ramamurthy, K. N., & Varshney, K. R. (2017) · neural information processing systems · Cited by 472
Tickle, A. B., Andrews, R., Golea, M., & Diederich, J. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.728352 · Cited by 437 · cited 1×
“Given the complexity of current ML models, explanations are often pedagogical [34], meaning that they reveal information about how the model works without faithfully representing the algorithms.”
BackgroundToo much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · cited 3×
“by considering two dimensions of explanation fidelity: soundness (how truthful each element in an explanation is with respect to the underlying system) and completeness (the extent to which an explanation describes all of the underlying system) [20].”
BackgroundAlgorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334
Evaluating the predictive validity of the COMPAS risk and needs assessment system
Brennan, T., Dieterich, W., & Ehret, B. (2009) · Criminal Justice and Behavior · doi:10.1177/0093854808326545 · Cited by 272 · cited 1×
“6% on processed data, comparable to reports on the accuracy of COMPAS system [6, 21].”
MethodArtificial Intelligence's White Guy Problem
Crawford, K. (2016) · The New York Times · Cited by 212
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Fairness in Learning: Classic and Contextual Bandits
Joseph, M., Kearns, M., Morgenstern, J., & Roth, A. (2016) · arXiv · doi:10.48550/arxiv.1605.07139 · Cited by 199 · cited 2×
“, [16, 17, 38]), equal attention has not been paid to the human scrutiny necessary to identify and remedy fairness issues.”
BackgroundActive Learning with Feedback on Features and Instances
Raghavan, H., Madani, O., & Jones, R. (2006) · Journal of Machine Learning Research · Cited by 184
Settles, B. (2011) · Cited by 167
Reconstructive expert system explanation
Wick, M. R., & Thompson, W. B. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90087-e · Cited by 158 · cited 1×
“Accordingly, Wick and Thompson discussed the taxonomy of global-local explanations [37].”
BackgroundNarayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153
Toward harnessing user feedback for machine learning
Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T. G., Sullivan, E., Drummond, R., & Herlocker, J. L. (2007) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1216295.1216316 · Cited by 134 · cited 1×
“Work on how people perceive explanations of ML systems is a growing area [4, 20, 25, 32] which aims to inform the choices and design of explanations for particular systems or tasks.”
BackgroundWhy Unbiased Computational Processes Can Lead to Discriminative Decision Procedures
Calders, T., & Žliobaitė, I. (2013) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-642-30487-3_3 · Cited by 128 · cited 5×
“For example Calders and Žliobaitė suggested techniques to de-bias data [8], includingmodifying labels of the training data, duplicating or deleting instances, adding synthetic instances, and transforming data into a new representation space.”
BackgroundMethodInfluentialSage Dictionary of Statistics: A Practical Resource for Students in the Social Sciences
Broughton, V. (2005) · Reference Reviews · doi:10.1108/09504120510580208 · Cited by 123
Explaining and Justifying Expert Consulting Programs
Swartout, W. (1985) · Computers and medicine · doi:10.1007/978-1-4612-5108-8_15 · Cited by 102
Interpretable Structure-Evolving LSTM
Liang, X., Lin, L., Shen, X., Feng, J., Yan, S., & Xing, E. P. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.234 · Cited by 99 · cited 1×
“[10, 23]), or on methods to automatically generate explanations.”
MethodExplanation Fiends and Foes: How Mechanistic Detail Determines Understanding and Preference
Fernbach, P. M., Sloman, S. A., Louis, R. D. S., & Shube, J. N. (2012) · Journal of Consumer Research · doi:10.1086/667782 · Cited by 91 · cited 2×
“For example, research shows that some prefer simple, supericial explanations and others are more deliberative and relective in their reasoning [13, 19].”
BackgroundActive Learning with Feedback on Features and Instances
RaghavanHema, MadaniOmid, & JonesRosie (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248608 · Cited by 71
Expert voices in echo chambers
Liao, Q. V., & Fu, W. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557240 · Cited by 47 · cited 1×
“We use a semantic diferential scale that is often used to measure attitude on controversial topics [24].”
MethodSelf-citation2023 IEEE Symposium on Visual Languages and Human-Centric Computing
(2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00002 · Cited by 28
Computer assisted medical decision making.
Brooks, G. (1983) · BMJ · doi:10.1136/bmj.287.6401.1299-a · Cited by 23
Toward Harnessing User Feedback For Machine Learning
Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T. G., Sullivan, E., Drummond, R., & Herlocker, J. L. (2006) · doi:10.21236/ada471484 · Cited by 20
Influencing Preferences for Different Types of Causal Explanation of Complex Events
Klein, G. A., Rasmussen, L. J., Lin, M., Hoffman, R. R., & Case, J. L. (2014) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814530427 · Cited by 16 · cited 2×
“For example, research shows that some prefer simple, supericial explanations and others are more deliberative and relective in their reasoning [13, 19].”
Background
Cited by
Sorted by citations
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Algorithmic bias: review, synthesis, and future research directions
Kordzadeh, N., & Ghasemaghaei, M. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2021.1927212 · Cited by 770
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Self-citationZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Self-citationManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
Metadata
- Authors
- Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C.
- Affiliations
- Oregon State University (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 24th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2019-03-17
- Pages
- 275-285
- OpenAlex
- W4288414189
- Cited by
- 286
- References
- 43
- Citations by year
- 2026: 24 · 2025: 44 · 2024: 52 · 2023: 48 · 2022: 56 · 2021: 41 · 2020: 18 · 2019: 3
- FWCI
- 11.04
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Army Research Laboratory and UK Ministry of Defence (W911NF-16-3-0001)
- Defense Advanced Research Projects Agency (N66001-17-2-4030)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- explainable AI, algorithmic fairness, fairness judgments, human-in-the-loop, explanation styles, individual differences, personalized explanations, adaptive explanations, trust in machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
