CoChat
Sign Up
PreprintCited by 273Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AISoftware Testing and Debugging Techniques

One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques

Open access
Arya, Vijay, Bellamy, Rachel K. E., Chen, Pin-Yu, Dhurandhar, Amit, Hind, Michael, Hoffman, Samuel C., Houde, Stephanie, Liao, Q. Vera +12 more·2019·arXiv
DOI 10.48550/arxiv.1909.03012arXiv 1909.03012
Open this paper in CoChat — free accountOpen PDF

Abstract

As artificial intelligence and machine learning algorithms make further inroads into society, calls are increasing from multiple stakeholders for these algorithms to explain their outputs. At the same time, these stakeholders, whether they be affected citizens, government regulators, domain experts, or system developers, present different requirements for explanations. Toward addressing these needs, we introduce AI Explainability 360 (http://aix360.mybluemix.net/), an open-source software toolkit featuring eight diverse and state-of-the-art explainability methods and two evaluation metrics. Equally important, we provide a taxonomy to help entities requiring explanations to navigate the space of explanation methods, not only those in the toolkit but also in the broader literature on explainability. For data scientists and other users of the toolkit, we have implemented an extensible software architecture that organizes methods according to their place in the AI modeling pipeline. We also discuss enhancements to bring research innovations closer to consumers of explanations, ranging from simplified, more accessible versions of algorithms, to tutorials and an interactive web demo to introduce AI explainability to different audiences and application domains. Together, our toolkit and taxonomy can help identify gaps where more explainability methods are needed and provide a platform to incorporate them as they are developed.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces AI Explainability 360 (AIX360), an open-source toolkit with eight explainability algorithms and two evaluation metrics, plus a taxonomy to help different explanation consumers choose methods. It describes algorithmic enhancements, educational materials, and comparisons showing BRCG-light trades accuracy for reduced computation.

Key findings

  • AIX360 contains eight explainability algorithms—BRCG, GLRM, ProtoDash, ProfWeight, TED, CEM, CEM-MAF, and DIP-VAE—and two metrics, Faithfulness and Monotonicity.
  • The proposed taxonomy organizes explanations by static/interactive, data/model, local/global, direct/post-hoc, and other distinctions; Table 1 indicates AIX360 uniquely covers data explanations, directly interpretable methods, local post-hoc, global post-hoc, persona-specific explanations, and metrics among the compared toolkits.
  • BRCG-light replaces integer programming with beam search; on 15 datasets, it is worse on average than BRCG-IP but better than BRS, AM, and BCD, and beam width B=5 gives better results than smaller widths on most datasets, particularly larger ones.
  • The TED employee-retention synthetic dataset has 10,000 entries, 8 features, 25 rules, and a feature space of over 80 million possibilities; the toolkit also includes a FICO-based web demo, five tutorials, and 13 Jupyter notebooks.

Methods

  • Design: proposes a taxonomy decision tree and implements an extensible Python class hierarchy organized by the AI modeling pipeline, including data explainers, directly interpretable explainers, local post-hoc explainers, and global post-hoc explainers.
  • Data or sample: BRCG-light is evaluated on the same 15 datasets as [17]; the TED use case uses a synthetic employee-retention dataset with 10,000 entries; the web demo uses the FICO Explainable Machine Learning Challenge dataset, and tutorials cover credit approval, healthcare cost prediction, dermoscopic images, NHANES data, and employee retention.
  • Main analysis: descriptive toolkit comparison in Table 1 and accuracy-complexity trade-off curves for BRCG-light versus BRCG-IP, BRS, AM, and BCD; no formal user study is reported in the text.

Limitations

  • The paper states the taxonomy is “by no means” claimed to be perfect or complete, and the toolkit lacks methods in three taxonomy categories marked with “?”: interactive, static → data → distributions, and static → model → global → post-hoc → visualize.
  • The paper states BRCG-light is inferior to BRCG-IP on average, and that if beam search fails to find a negative-objective solution, it cannot certify that none exists, unlike integer programming run to completion.
  • The paper states explanations are difficult to quantitatively evaluate, human evaluation is the gold standard, and the two included metrics are proxies and not novel; evident from the text, the toolkit and taxonomy are evaluated mainly through descriptive comparison and trade-off curves rather than user studies.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1909.03012

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 35 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § ABSTRACT · p. 1

    AIX360 features eight diverse and state-of-the-art explainability methods.

    “we introduce AI Explainability 360 1 , an open-source software toolkit featuring eight diverse and state-of-the-art explainability methods and two evaluation metrics.”
    • Also stated in: § 1 INTRODUCTION · p. 2 · plain
      “The current version of the toolkit contains eight explainability algorithms:”
    Quote in the paperSubstantial claim
  • Findingcitation
    § ABSTRACT · p. 1

    AIX360 includes two evaluation metrics: Faithfulness and Monotonicity.

    “two evaluation metrics”
    • Also stated in: § 1 INTRODUCTION · p. 2 · plain
      “The toolkit also includes two metrics from the explainability literature: Faithfulness [12] and Monotonicity [41].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [12] [12] D. Alvarez Melis and T. Jaakkola. Towards robust interpretability with selfexplaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 7775-7784. 2018. -

      Not checked in the free run.

    • cites [41] [41] R. Luss, P.-Y. Chen, A. Dhurandhar, P. Sattigeri, K. Shanmugam, and C.-C. Tu. Generating contrastive explanations with monotonic attribute functions. 2019. -

      Not checked in the free run.

  • Findingown data
    § One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques · p. 1

    AIX360 is the most comprehensive explainability toolkit across different ways of explaining.

    “It is the most comprehensive explainability toolkit across different ways of explaining (see Table 1 and Section 5).”
    Quote in the paperSubstantial claim
  • Findingnone given
    § ABSTRACT · p. 1

    The paper implemented an extensible software architecture that organizes methods according to their place in the AI modeling pipeline.

    “we have implemented an extensible software architecture that organizes methods according to their place in the AI modeling pipeline.”
    Quote in the paperSubstantial claim
  • Findingreasoning
    § ABSTRACT · p. 1

    AIX360 is an open-source software toolkit.

    “we introduce AI Explainability 360 1 , an open-source software toolkit featuring eight diverse and state-of-the-art explainability methods and two evaluation metrics.”
    • Also stated in: § One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques · p. 1 · plain
      “We have released the toolkit into the open source community under the name AI Explainability 360 (AIX360).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 54
  • May combine two claims. 20
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    The taxonomy includes most of the methods reviewed by Guidotti et al. [25].

    “Again, we do not claim that this is a complete listing, although most of the methods reviewed by Guidotti et al. [25] are included.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] [25] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR) , 51(5):93, 2018. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 4.6 Web demo, tutorials and resources · p. 9

    AIX360 includes multiple educational materials: a general guidance for key concepts, a web demo, and multiple tutorials.

    “These educational materials include a general guidance for the key concepts of explainability, a web demo that illustrates the usage of different explainability methods, and multiple tutorials.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2 TAXONOMY FOR AI EXPLAINABILITY · p. 3

    The decision tree leaves indicate the methods currently available through the toolkit along with the modalities they can work with.

    “The decision tree leaves indicate the methods currently available through the toolkit along with the modalities they can work with.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 2 TAXONOMY FOR AI EXPLAINABILITY · p. 2

    AIX360 focuses on static explanations.

    “AIX360 focuses on static explanations, like much of the literature to date.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation
    § 4.6 Web demo, tutorials and resources · p. 9

    The web demo was created based on the FICO Explainable Machine Learning Challenge dataset, a real-world scenario where a machine learning system is used to support decisions on loan applications by predicting the repayment risk of the applicants.

    “The web demo was created based on the FICO Explainable Machine Learning Challenge dataset [22], a real-world scenario where a machine learning system is used to support decisions on loan applications by predicting the repayment risk of the applicants.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] [22] FICO. FICO Explainable Machine Learning Challenge. https://community.fico. com/s/explainable-machine-learning-challenge, 2018. Last accessed 2019-08. -

      Not checked in the free run.

  • May combine two claims.

    Findingown data
    § 4.1 Boolean Decision Rules via Column Generation: 'Light' version · p. 7

    BRCG-light occupies an intermediate position in the trade-off between classification accuracy and rule complexity, inferior to BRCG-IP but superior to earlier methods.

    “These comparisons show that BRCG-light occupies an intermediate position, inferior to BRCG-IP as may be expected, but superior to the earlier methods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2 Teaching Explanations for Decisions: Data synthesis · p. 7

    AIX360 contains an employee retention dataset with 10,000 entries.

    “AIX360 contains such a dataset for employee retention with 10,000 entries, created by an available python script that can be modified by changing the distribution of features, changing the rules, or the size of the dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.6 Web demo, tutorials and resources · p. 9

    AIX360 was developed with the goal of providing accessible resources on explainability to non-technical stakeholders.

    “AIX360 was developed with the goal of providing accessible resources on explainability to non-technical stakeholders.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data
    § 4.2 Teaching Explanations for Decisions: Data synthesis · p. 7

    The synthetic employee retention dataset was created using 8 features and 25 rules.

    “We created 25 rules, motivated by common retention scenarios, such as not getting a promotion in a while, not being paid competitively, receiving a disappointing evaluation, being a new employee in organizations with inherently high attrition, not having a salary that is consistent with positive evaluations, mid-career crisis, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2 TAXONOMY FOR AI EXPLAINABILITY · p. 3

    The current version of AIX360 does not have a method available in some categories of the taxonomy.

    “The leaves in the decision tree are labeled with algorithms in the AIX360 toolkit or a question mark ('?') if the category is not represented in the current toolkit.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 4.1 Boolean Decision Rules via Column Generation: 'Light' version · p. 6

    BRCG-light replaces the integer programming components of BRCG with a heuristic beam search algorithm.

    “In AIX360, we have replaced the integer programming components with a heuristic beam search algorithm and have modified the problem formulation accordingly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2 Teaching Explanations for Decisions: Data synthesis · p. 7

    The application of the rules varies depending on positions and organizations.

    “In the current implementation of this use case, we further vary the application of these rules depending on various positions and organizations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.1 Boolean Decision Rules via Column Generation: 'Light' version · p. 7

    The module implementing BRCG-light is novel.

    “The modules implementing GLRM follow [59] plus the enhancement of Section 4.3 below while the module implementing BRCG-light is novel.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 3 IMPLEMENTATION OF TAXONOMY · p. 5

    AIX360 includes dataset classes to facilitate loading and processing of commonly used datasets.

    “In addition to the above base classes, AIX360 includes dataset classes to facilitate loading and processing of commonly used datasets so that users can easily experiment with the implemented algorithms.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.2 Teaching Explanations for Decisions: Data synthesis · p. 7

    The rules for the employee retention dataset were motivated by common retention scenarios.

    “We created 25 rules, motivated by common retention scenarios, such as not getting a promotion in a while, not being paid competitively, receiving a disappointing evaluation, being a new employee in organizations with inherently high attrition, not having a salary that is consistent with positive evaluations, mid-career crisis, etc.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 2 TAXONOMY FOR AI EXPLAINABILITY · p. 3

    The proposed taxonomy is based on questions about what is explained, how it is explained, and at what level.

    “The proposed taxonomy based on questions about what is explained (e.g., data or model), how it is explained (e.g., direct/post-hoc, static/interactive) and at what level (i.e. local/global).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data
    § 6 DISCUSSION · p. 10

    The taxonomy of Figure 1 has three categories not present in the toolkit: interactive, static → data → distributions, and static → model → global → post-hoc → visualize.

    “These are indicated in Figure 1 by '?' in three cases: interactive, static → data → distributions, and static → model → global → post-hoc → visualize.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given
    § 6 DISCUSSION · p. 10

    The paper provides a simple yet comprehensive taxonomy that structures the explainability space and serves multiple purposes, including guiding practitioners, revealing algorithmic gaps to researchers, and informing design choices related to explainability software.

    “We have also provided a simple yet comprehensive taxonomy that structures the explainability space and serves multiple purposes, including guiding practitioners, revealing algorithmic gaps to researchers, and informing design choices related to explainability software (e.g., class hierarchy) for data scientists and developers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 4.6 Web demo, tutorials and resources · p. 9

    AIX360 currently includes five tutorials in the form of Jupyter notebooks that show data scientists and other developers how to use different explanation methods across several application domains.

    “The AIX360 toolkit currently includes five tutorials in the form of Jupyter notebooks that show data scientists and other developers how to use different explanation methods across several application domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 6 DISCUSSION · p. 10

    Preliminary work in AIX360 initiated framework-independent explainability by defining a class for Keras-based classifiers utilized by CEMExplainer and CEMMAFImageExplainer.

    “Preliminary work in this direction was initiated in AIX360 by defining a class for Keras-based classifiers that is utilized by CEMExplainer and CEMMAFImageExplainer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 6 DISCUSSION · p. 10

    Some of the algorithms were enhanced from their published versions for AIX360 to make them more usable and their outputs easier to consume.

    “Some of the algorithms were enhanced from their published versions for AIX360 to make them more usable and their outputs easier to consume.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 6 DISCUSSION · p. 10

    The toolkit also contains two explainability metrics, a credit approval demonstration, five elaborate tutorials covering different real-world use cases, and 13 Jupyter notebooks, making it accessible to practitioners and non-experts.

    “The toolkit also contains two explainability metrics, a credit approval demonstration, five elaborate tutorials covering different real-world use cases, and 13 Jupyter notebooks, making it accessible to practitioners and non-experts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § A ACCURACY-COMPLEXITY TRADE-OFFS FOR BRCG-LIGHT · p. 13

    BRCG-light provides trade-offs between classification accuracy and DNF rule complexity that are worse on average than BRCG-IP, but better than BRS, AM, and BCD.

    “Overall, the comparisons show that BRCG-light provides trade-offs that are worse on average than BRCG-IP, as may be expected given the dramatic reduction in computation, but better than the other methods.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 5 RELATED WORK · p. 10

    AIX360 also targets industry-specific users with the tutorials and web demo.

    “AIX360 also targets industry-specific users with the tutorials and web demo in Section 4.6.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 6 DISCUSSION · p. 10

    The AIX360 toolkit solicits contributions related to explainable AI to enable it to grow.

    “The AIX360 toolkit solicits contributions related to explainable AI to enable it to grow.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques · p. 1

    There is no single approach to explainable AI that always works best.

    “Since there is no single approach to explainable AI that always works best, we require organizing principles for the space of possibilities and tools that bridge the gap from research to practice.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § ABSTRACT · p. 1

    The paper provides a taxonomy to help entities requiring explanations navigate the space of explanation methods, not only those in the toolkit but also in the broader literature on explainability.

    “Equally important, we provide a taxonomy to help entities requiring explanations to navigate the space of explanation methods, not only those in the toolkit but also in the broader literature on explainability.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques · p. 1

    Different people in different settings may require different kinds of explanations.

    “different people in different settings may require different kinds of explanations.”
    • Also stated in: § ABSTRACT · p. 1 · plain
      “these stakeholders, whether they be affected citizens, government regulators, domain experts, or system developers, present different requirements for explanations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 2.1 Navigating the Taxonomy · p. 3

    The taxonomy allows different consumers to navigate the space of explanations and find the right algorithm or class of algorithms appropriate for their task.

    “We show that the taxonomy in Figure 1 allows different consumers to navigate the space of explanations and find the right algorithm or class of algorithms appropriate for their task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    Knowledge distillation methods are global interpretations learned using a post-hoc surrogate model.

    “Knowledge distillation-type approaches [13, 14, 31], which learn a simpler model based on a complex model's predictions, would be considered as global interpretations that are learned using a post-hoc surrogate model.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [13] [13] L. J. Ba and R. Caurana. Do deep nets really need to be deep? CoRR , abs/1312.6184, 2013. -

      Not checked in the free run.

    • cites [14] [14] C. Buciluˇ a, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2006. -

      Not checked in the free run.

    • cites [31] [31] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In https://arxiv.org/abs/1503.02531 , 2015. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Generalized Linear Rule Models: Visualization · p. 8

    A GAM can be visualized as a series of plots of the functions f_j, thus summarizing the effects of all first-degree rules and linear terms.

    “The benefit of having a GAM is that it can be visualized as a series of plots of the functions f j , thus summarizing the effects of all first-degree rules and linear terms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.3 Generalized Linear Rule Models: Visualization · p. 7

    GLRM models can be interpreted like linear models, and their constituent rules are readily understood individually.

    “GLRM models can therefore be interpreted like linear models and their constituent rules are also readily understood individually.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.4 ProtoDash and Contrastive Explanations: Feature importance · p. 8

    For CEM, a pertinent positive (PP) informs us of what minimal values for different features not exceeding those in x would be sufficient to obtain the same prediction y.

    “A PP informs us of what minimal values for different features not exceeding those in x would be sufficient to obtain the same prediction y .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] [18] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, and P. Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems , 2018. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques · p. 1

    The taxonomy is actionable in that it aids users in choosing an approach for a given application and may also reveal gaps in available explainability techniques.

    “This taxonomy is actionable in that it aids users in choosing an approach for a given application and may also reveal gaps in available explainability techniques.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 4.5 Explainability metrics · p. 9

    If features are (independent) positively correlated with the decision, then the class probabilities should monotonically increase.

    “If the features are (independent) positively correlated with the decision, then the class probabilities should monotonically increase.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    Methods that provide rationales are local self explanations.

    “Work in the natural language processing and computer vision domains that generates rationales/explanations derived from input text [11, 28, 39] would be considered as local self explanations.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [11] [11] Y. Ainur, Y. Choi, and C. Cardie. Automatically generating annotator rationales to improve sentiment classification. In Proceedings of the ACL 2010 Conference Short Papers , 2010. -

      Not checked in the free run.

    • cites [28] [28] L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell. Generating visual explanations. In European Conference on Computer Vision , 2016. -

      Not checked in the free run.

    • cites [39] [39] T. Lei, R. Barzilay, and T. Jaakkola. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155 , 2016. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning
    § ABSTRACT · p. 1

    Together, the toolkit and taxonomy can help identify gaps where more explainability methods are needed and provide a platform to incorporate them as they are developed.

    “Together, our toolkit and taxonomy can help identify gaps where more explainability methods are needed and provide a platform to incorporate them as they are developed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation
    § 4.3 Generalized Linear Rule Models: Visualization · p. 7

    A GLRM model with tens of rules may require a non-trivial investment of effort to interpret, though interpretation is still feasible.

    “Nevertheless, the discussion in [59, Sec. 6] as well as our experience in putting together the AIX360 web demo and tutorials (Section 4.6) raise the question of the degree to which a GLRM model with (say) tens of rules is interpretable. While interpretation is still feasible, it may require a non-trivial investment of effort.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] [59] D. Wei, S. Dash, T. Gao, and O. Günlük. Generalized linear rule models. In Proceedings of the 36th International Conference on Machine Learning (ICML) , 2019. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    Restricted neural network architectures fall under the global directly interpretable category.

    “Methods that propose certain restrictions on the neural network architecture [12, 62] to make it interpretable, yet maintain richness of the hypothesis space to model complicated decision boundaries would fall under the global directly interpretable category.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [12] [12] D. Alvarez Melis and T. Jaakkola. Towards robust interpretability with selfexplaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 7775-7784. 2018. -

      Not checked in the free run.

    • cites [62] [62] Q. Zhang, Y. N. Wu, and S.-C. Zhu. Interpretable convolutional neural networks. In Proceedings of Computer Vision and Pattern Recognition , 2018.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.5 Explainability metrics · p. 9

    The second metric is Monotonicity, which tries to measure if adding more positive evidence increases the probability of classification in the specified class.

    “The second metric is Monotonicity [41], which simply tries to measure if adding more positive evidence increases the probability of classification in the specified class.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41] [41] R. Luss, P.-Y. Chen, A. Dhurandhar, P. Sattigeri, K. Shanmugam, and C.-C. Tu. Generating contrastive explanations with monotonic attribute functions. 2019. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 4.5 Explainability metrics · p. 9

    The intuition behind the Faithfulness metric is that removing more important features should reduce the confidence of the classifier more than removing less important ones.

    “The intuition is that removing more important features should reduce the confidence of the classifier more than removing less important ones.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.5 Explainability metrics · p. 8

    The first metric is a version of the Faithfulness metric, which tries to evaluate if the feature importances returned by an explainability method are the correct ones.

    “The first metric we implement is a version of the Faithfulness metric [12], which tries to evaluate if the feature importances returned by an explainability method are the correct ones.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] [12] D. Alvarez Melis and T. Jaakkola. Towards robust interpretability with selfexplaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 7775-7784. 2018. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Interpretationreasoning
    § 4.3 Generalized Linear Rule Models: Visualization · p. 8

    Higher-degree rules, which involve feature interactions, can be studied separately from the GAM part.

    “Higher-degree rules, which do involve feature interactions, can then be studied separately.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    Saliency methods can be categorized as local explanation methods that are static and provide feature-based explanations.

    “Saliency based methods [51, 52], which highlight different portions in an image whose classification we want to understand, can be categorized as local explanation methods that are static and provide feature-based explanations in terms of highlighting pixels/superpixels.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [51] [51] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Gradcam: Visual explanations from deep networks via gradient-based localization. See https://arxiv. org/abs/1610.02391 v3 , 2016. -

      Not checked in the free run.

    • cites [52] [52] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR , abs/1312.6034, 2013. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 4.4 ProtoDash and Contrastive Explanations: Feature importance · p. 8

    The feature importance for ProtoDash, θ_j, is computed as a value in [0,1], where higher values indicate greater similarity and importance.

    “The above equation creates a feature importance value in [ 0 , 1 ] . The higher the value more similar the feature and hence higher its importance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    Feature relevance methods are used to study the global effects of different input features on output values.

    “Methods such as partial dependence plots (PDP) and sensitivity analysis [43] are used to study the (global) effects of different input features on the output values and are typically consumed through appropriate visualizations (e.g., PDP plots, scatter plots, control charts).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43] [43] C. Molnar. Interpretable machine learning, 2019. -

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 4.4 ProtoDash and Contrastive Explanations: Feature importance · p. 8

    The base algorithms for ProtoDash and CEM do not return feature importances; they return vectors the same dimension as the input.

    “The base algorithms for both ProtoDash [26] and CEM [18] do not return feature importances. What they return are vectors that are the same dimension as the input.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] [26] K. Gurumoorthy, A. Dhurandhar, G. Cecchi, and C. Agarwal. Efficient data representation by selecting prototypes with importance weights. IEEE Intl. Conference on Data Mining , 2019. -

      Not checked in the free run.

    • cites [18] [18] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, and P. Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems , 2018. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.2 Teaching Explanations for Decisions: Data synthesis · p. 7

    The synthetic dataset can serve as an example for the generation of datasets for other classification problems.

    “It can also serve as a example for the generation of datasets for other classification problems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 4.3 Generalized Linear Rule Models: Visualization · p. 8

    Linear functions of original, unbinarized numerical features (linear terms) if included in a GLRM model can be added to the GAM.

    “The same is true of linear functions of original, unbinarized numerical features (so-called 'linear terms') if they are included in the GLRM model, so these can be added to the GAM.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.2 Categorizing Common Classes of Explainability Methods · p. 4

    Exemplar methods are local explanations in the form of samples.

    “Methods to explain the predictions of test instances based on similar or influential training instances [26, 33, 36] would be considered as local explanations in the form of samples.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [26] [26] K. Gurumoorthy, A. Dhurandhar, G. Cecchi, and C. Agarwal. Efficient data representation by selecting prototypes with importance weights. IEEE Intl. Conference on Data Mining , 2019. -

      Not checked in the free run.

    • cites [33] [33] B. Kim, R. Khanna, and O. Koyejo. Examples are not enough, learn to criticize! Criticism for interpretability. In In Advances of Neural Inf. Proc. Systems , 2016. -

      Not checked in the free run.

    • cites [36] [36] P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning , 2017. -

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data
    § 4.2 Teaching Explanations for Decisions: Data synthesis · p. 7

    The synthetic employee retention dataset shows that TED explanations match the terminology of the domain and represent higher-level semantic concepts.

    “the explanations match the terminology of the domain; they represent higher-level semantic concepts, e.g., 'not being paid competitively', created by the domain expert, rather than mathematical combinations of feature values,”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.4 ProtoDash and Contrastive Explanations: Feature importance · p. 8

    For CEM, a pertinent negative (PN) informs us of what minimal features if added or increased in value would change the classification of x to something other than y.

    “While, a PN informs us of what minimal features if added or increased in value would change the classification of x to something other than y .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] [18] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, and P. Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems , 2018. -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 4.6 Web demo, tutorials and resources · p. 9

    The web demo highlights that three groups of people-data scientists, loan officers, and bank customers-are involved in the scenario and their needs are best served by different explainability methods.

    “The demo highlights that three groups of people - data scientists, loan officers, and bank customers - are involved in the scenario and their needs are best served by different explainability methods.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.1 Boolean Decision Rules via Column Generation: 'Light' version · p. 6

    The BRCG-light modifications bring it closer to GLRM in two respects: they conform to the GLRM problem formulation and allow reuse of a GLRM sub-algorithm.

    “These modifications bring BRCG-light closer to GLRM [59] in two respects: 1) They conform to the GLRM problem formulation in which rule complexity is penalized in the same way, giving AIX360 users a common interface; 2) They allow reuse of a GLRM sub-algorithm as explained below.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 4.6 Web demo, tutorials and resources · p. 9

    For example, a data scientist may demand a global understanding of model behavior through an interpretable model, which can be provided by the GLRM algorithm; a bank customer would ask for justification for their loan application results, which can be generated by the CEM algorithm.

    “For example, although the data scientist may demand a global understanding of model behavior through an interpretable model, which can be provided by the GLRM algorithm, a bank customer would ask for justification for their loan application results, which can be generated by the CEM algorithm.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 6 DISCUSSION · p. 10

    These resources, together with the breadth of the toolkit and its common extensible programming interface, help distinguish AIX360 from existing explainability toolkits.

    “These resources, together with the breadth of the toolkit and its common extensible programming interface, help distinguish AIX360 from existing explainability toolkits.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 6 DISCUSSION · p. 10

    The toolkit and taxonomy provide technical, educational, and operational benefits to the community over and above what currently exists.

    “We thus believe that our toolkit and taxonomy provide technical, educational, and operational benefits to the community over and above what currently exists.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 5 RELATED WORK · p. 10

    Another distinctive feature of AIX360 is its common architecture and programming interface.

    “Another distinctive feature of AIX360 is its common architecture and programming interface.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § 5 RELATED WORK · p. 9

    The proposed taxonomy and software architecture add to prior criteria by making broader distinctions (data vs. model) and finer distinctions (features vs. samples, black-box vs. white-box post-hoc).

    “The proposed taxonomy (Figure 1) and software architecture (Section 3) add to the criteria above by making both broader distinctions (data vs. model) as well as finer distinctions (features vs. samples, black-box vs. white-box post-hoc).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 4.6 Web demo, tutorials and resources · p. 9

    Beyond illustrating the application of different methods, the tutorials also provide considerable insight into the datasets that are used and, to the extent that these insights generalize, into the respective problem domains.

    “Beyond illustrating the application of different methods, the tutorials also provide considerable insight into the datasets that are used and, to the extent that these insights generalize, into the respective problem domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § 5 RELATED WORK · p. 10

    The clearest difference between AIX360 and other toolkits is that AIX360 offers a wider spectrum of explainability methods, notably data explanations, metrics, and persona-specific explanations.

    “The clearest difference between AIX360 and these other toolkits is that AIX360 offers a wider spectrum of explainability methods, notably data explanations, metrics, and persona-specific explanations, the last referring to TED's ability to provide explanations in the language of the persona.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 4.6 Web demo, tutorials and resources · p. 9

    These insights are a natural consequence of using explainable machine learning and could be of independent interest.

    “These insights are a natural consequence of using explainable machine learning and could be of independent interest.”
    Quote in the paperNot a substantial claim
62 of 62

  • The paper relies on this without support.

    Many machine learning techniques are not easily explainable, even by experts in the field.

    “However, many machine learning techniques, which are responsible for much of the advances in AI, are not easily explainable, even by experts in the field.”

  • The paper relies on this without support.

    There remains a gap between what society needs and what the research community is producing.

    “growing volume of publications, there remains a gap between what society needs and what the research community is producing.”

  • The paper relies on this without support.

    A lack of a precise definition of an explanation is one reason for the gap between what society needs and what the research community is producing.

    “One reason for this gap is a lack of a precise definition of an explanation.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Measures have to be taken to address risk-related concerns and demonstrate compliance with the law.

    “Measures have to be taken to address the risk-related concerns and demonstrate compliance with the law [8].”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [8]

    The introduction (p. 1) says regulators need to understand an AI system’s behavior to ensure it complies with the law. Related Work (p. 10) lists safety and guarding against discrimination as motivations for explainability, but the paper does not say that specific measures must be taken to address risk-related concerns. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Some XAI toolkits offer XAI evaluations to facilitate quality control and comparability.

    “To facilitate quality control and comparability, some of the toolkits also offer XAI evaluations37,39,42.”

    From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [37]

  • Multiple toolkits offer readily implemented XAI methods for a wide variety of tasks applicable to medical imaging.

    “There are multiple toolkits, such as AIX-36037, Alibi38, Captum39, EthicalML-XAI40, iNNvestigate41, Quantus42, among others, offering readily implemented XAI methods for a wide variety of tasks applicable to medical imaging (Table 1).”

    From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [37]

  • Various toolboxes have been released in the past years to assist algorithm experimentation and software development to enhance AI trustworthiness in different aspects such as fairness [12, 13, 71], XAI [56, 1, 8], and adversarial robustness [55, 2].

    “Various toolboxes have been released in the past years to assist the algorithm experimentation and software development to enhance AI trustworthiness in different aspects such as fairness [12, 13, 71], XAI [56, 1, 8], and adversarial robustness [55, 2].”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [8]

  • Laws for data privacy protection like the GDPR require a data protection impact assessment (DPIA) if any data processing poses a risk to data privacy.

    “Laws for data privacy protection like the GDPR require data protection impact assessment (DPIA) if any data processing poses a risk to data privacy.”

    From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [8]

    The Introduction (p. 1) discusses regulations requiring explanations, but does not address data protection impact assessments. Searches for DPIA and for GDPR-related processing risks found no passage stating when an assessment is required. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

2 of the 46 references couldn’t be matched to a known paper.

Research works printed but not linked: 11

  1. [12][12] D. Alvarez Melis and T. Jaakkola. Towards robust interpretability with selfexplaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 7775-7784. 2018. -
  2. [14][14] C. Buciluˇ a, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2006. -
  3. [18][18] A. Dhurandhar, P.-Y. Chen, R. Luss, C.-C. Tu, P. Ting, K. Shanmugam, and P. Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems , 2018. -
  4. [30][30] M. Hind, D. Wei, M. Campbell, N. C. F. Codella, A. Dhurandhar, A. Mojsilovic, K. N. Ramamurthy, and K. R. Varshney. TED: Teaching AI to explain its decisions. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES) , pages 123-129, 2019. -
  5. [32][32] F. Hohman, A. Head, R. Caruana, R. DeLine, and S. M. Drucker. Gamut: A design probe to understand how data scientists understand machine learning models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , page 579. ACM, 2019. -
  6. [34][34] B. Kim, K. R. Varshney, and A. Weller, editors. Proc. ICML Workshop Human Interpret. Mach. Learn. Stockholm, Sweden, June 2018. -
  7. [38][38] A. Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In Proceedings of the Sixth International Conference on Learning Representations , 2018. -
  8. [40][40] S. Lundberg and S.-I. Lee. Unified framework for interpretable methods. In Advances of Neural Inf. Proc. Systems , 2017. -
  9. [45][45] D. B. Pasternak. Illinois and City of Chicago poised to implement new laws addressing changes in the workplace - signs of things to come? The National Law Review , June 2019. -
  10. [58][58] T. Wang, C. Rudin, F. Doshi-Velez, Y. Liu, E. Klampfl, and P. MacNeille. A Bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research , 18(70):1-37, 2017. -
  11. [60][60] D. Weidele, H. Strobelt, and M. Martino. Deepling: A visual interpretability system for convolutional neural networks. In Proceedings SysML , 2019. -

Cited by

Sorted by citations

Related

Authors
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., Mourad, S., Pedemonte, P., Raghavendra, R., Richards, J., Sattigeri, P., Shanmugam, K., Singh, M., Varshney, K. R., Wei, D., & Zhang, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1909.03012
arXiv
1909.03012
OpenAlex
W2972059645
Semantic Scholar
c57a28759b6e1ddd7be03fa41e82b01130c8f8b2
Cited by
268
References
46
Citations by year
2026: 6 · 2025: 20 · 2024: 46 · 2023: 42 · 2022: 51 · 2021: 71 · 2020: 30 · 2019: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
explainable AI, explainability evaluation metrics, open source software, explainability requirements

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review