CoChat
Sign Up
Cited by 728Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIPrivacy-Preserving Technologies in Data

Algorithmic Transparency via Quantitative Input Influence: Theory and Experiments with Learning Systems

Datta, Anupam, Sen, Shayak, Zick, Yair·2016·2016 IEEE Symposium on Security and Privacy (SP)
DOI 10.1109/sp.2016.42
Open this paper in CoChat — free account

Abstract

Algorithmic systems that employ machine learning play an increasing role in making substantive decisions in modern society, ranging from online personalization to insurance and credit decisions to predictive policing. But their decision-making processes are often opaque-it is difficult to explain why a certain decision was made. We develop a formal foundation to improve the transparency of such decision-making systems. Specifically, we introduce a family of Quantitative Input Influence (QII) measures that capture the degree of influence of inputs on outputs of systems. These measures provide a foundation for the design of transparency reports that accompany system decisions (e.g., explaining a specific credit decision) and for testing tools useful for internal and external oversight (e.g., to detect algorithmic discrimination). Distinctively, our causal QII measures carefully account for correlated inputs while measuring influence. They support a general class of transparency queries and can, in particular, explain decisions about individuals (e.g., a loan decision) and groups (e.g., disparate impact based on gender). Finally, since single inputs may not always have high influence, the QII measures also quantify the joint influence of a set of inputs (e.g., age and income) on outcomes (e.g. loan decisions) and the marginal influence of individual inputs within such a set (e.g., income). Since a single input may be part of multiple influential sets, the average marginal influence of the input is computed using principled aggregation measures, such as the Shapley value, previously applied to measure influence in voting. Further, since transparency reports could compromise privacy, we explore the transparency-privacy tradeoff and prove that a number of useful transparency reports can be made differentially private with very little addition of noise. Our empirical validation with standard machine learning algorithms demonstrates that QII measures are a useful transparency mechanism when black box access to the learning system is available. In particular, they provide better explanations than standard associative measures for a host of scenarios that we consider. Further, we show that in the situations we consider, QII is efficiently approximable and can be made differentially private while preserving accuracy.

Integrity signals

Registry

Crossref · 10.1109/sp.2016.42

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even in cases of complex multi-layered models, the problem of opacity may be overestimated; there are promising approaches to explaining the specific outputs of a model without attempting to open its 'black box'.

    “But even in cases of complex multi-layered models, the problem of opacity may be overestimated; there are various promising approaches to explaining the specific outputs of a model without attempting to open its ‘black box’ (Ribeiro et al., 2016; Datta, Sen, & Zick, 2016).”

    From Algorithmic Accountability and Public Reason (2018) · cites this paper as Datta, Sen, & Zick, 2016

  • It is not possible to understand deep architectures as the loss landscape might be opaque since it cannot be fully observed and the solution has to be approximated through heuristic optimization.

    “it is not possible to understand it in deep architectures as the loss landscape might be opaque [172, 173] since it cannot be fully observed and the solution has to be approximated through heuristic optimization (e.g. through stochastic gradient descent).”

    From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [173]

  • Blinding (or partial blinding) has been used as a fairness audit mechanism.

    “Blinding (or partial blinding) has also been used as a fairness audit mechanism [2,82,143].”

    From Fairness in machine learning: A survey (2024) · cites this paper as [82]

  • Blinding audit approaches explore how partially blinding features (sensitive or otherwise) affect model performance.

    “Speciically, such approaches explore how partially blinding features (sensitive or otherwise) afect model performance.”

    From Fairness in machine learning: A survey (2024) · cites this paper as [82]

  • Prior works suggest making a probabilistic intervention on a feature by replacing it with an independent random variable.

    “To measure the influence of a certain feature 𝑊 𝑖 prior works suggest to make a probabilistic intervention on that variable by replacing it with an independent random variable [13, 23, 34].”

    From Marrying Fairness and Explainability in Supervised Learning (2022) · cites this paper as [13]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Datta, A., Sen, S., & Zick, Y.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
2016 IEEE Symposium on Security and Privacy (SP)
Publisher
IEEE
Published
2016-05
Pages
598-617
ISSN
1081-6011 (ISSN-L)
DOI
10.1109/sp.2016.42
OpenAlex
W2510508396
Cited by
728
References
56
Citations by year
2026: 38 · 2025: 58 · 2024: 73 · 2023: 96 · 2022: 80 · 2021: 121 · 2020: 107 · 2019: 80 · 2018: 38 · 2017: 30 · 2016: 7
FWCI
27.84
Type
conference paper
Language
en
Open access
closed
Locations
IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
Keywords
algorithmic transparency, explainable AI, indirect discrimination, Shapley value, differential privacy, transparency reports, correlated inputs, credit decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review