Explainable AI in Industry: Practical Challenges and Lessons Learned
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Artificial Intelligence is increasingly playing an integral role in determining our day-to-day experiences. Moreover, with the proliferation of AI based solutions in areas such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of AI have become far-reaching. The dominant role played by AI models in these domains has led to a growing concern regarding potential bias in these models, and a demand for model transparency and interpretability [11]. Model explainability is considered a prerequisite for building trust and adoption of AI systems in high stakes domains such as lending and healthcare [1] which require reliability, safety, and fairness. It is also critical to automated transportation, and other industrial applications with significant socio-economic implications such as predictive maintenance, exploration of natural resources, and climate change modeling.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“As a consequence, AI researchers and practitioners have focused their attention on explainable AI to help them better trust and understand models at scale [14, 15, 25].”
BackgroundAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Self-citationPermutation importance: a corrected feature importance measure
Altmann, A., Toloşi, L., Sander, O., & Lengauer, T. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq134 · Cited by 2,843
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×
“Prominent techniques include: ∗ Methods based on Local surrogatemodels such as LIME [20] and Anchors [21] ∗ Feature attribution methods such as Integrated Gradients [24], DeepLift [23], Layerwise Relevance Propagation [18], Shapley Values [22], SHAP [17], and Quantitative Input Influence [5] ∗ Example based explanatio…”
MethodIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 1×
“…Propagation [18], Shapley Values [22], SHAP [17], and Quantitative Input Influence [5] ∗ Example based explanation methods such as Influence functions [12] and Prototypes and Criticisms [3] – Global Explanations: This class of methods involves ex-plainingorsummarizingthemodel behaviorglobally.”
MethodExplaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357 · cited 1×
“…perspectives [20] and Anchors [21] ∗ Feature attribution methods such as Integrated Gradi-ents [24], DeepLift [23], Layerwise Relevance Propagation [18], Shapley Values [22], SHAP [17], and Quantitative Input Influence [5] ∗ Example based explanation methods such as Influence functions [12] and…”
MethodExplainable artificial intelligence: A survey
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400040 · Cited by 1,147
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · cited 1×
“…Integrated Gradi-ents [24], DeepLift [23], Layerwise Relevance Propagation [18], Shapley Values [22], SHAP [17], and Quantitative Input Influence [5] ∗ Example based explanation methods such as Influence functions [12] and Prototypes and Criticisms [3] – Global Explanations: This class of…”
MethodAccurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564 · cited 1×
“Prominent techniques include: – Generalized linear models (GLMs) – Generalized additive models (GAMs) [4], and variants like GA2Ms [16] – Decision trees, Decision sets, and Decision lists • Evaluation of Explainability: Accuracy, Coverage / Represen-tativeness, Complexity, Human friendliness…”
Method(2016) · neural information processing systems · Cited by 556
Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search
Geyik, S. C., Ambler, S., & Kenthapadi, K. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330691 · Cited by 350
Self-citationExplainable Artificial Intelligence (XAI)
Mohaghegh, S. D. (2024) · Artificial Intelligence for Science and Engineering Applications · doi:10.1201/9781003369356-8 · Cited by 215
Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
Lakkaraju, H., Kamar, E., Caruana, R., & Horvitz, E. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10821 · Cited by 145
Relevance Debugging and Explaining at LinkedIn
Qiu, D., & Qian, Y. (2019) · Cited by 3
Research works printed but not linked: 9
- Ahmad Muhammad Aurangzeb, 2018, Interpretable Machine Learning in Healthcare. In ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics.
- Frosst Nicholas, 2017, International Workshop on Comprehensibility and Explanation in AI and ML.
- Geyik Sahin Cem, 2018, Building Representative Talent Search at LinkedIn. (October
- Rory Mc Grath Luca Costabello Chan Le Van Paul Sweeney Farbod Kamiab Zhao Shen and Freddy Lecue. 2018. Interpretable Credit Application Predictions With Counterfactual Explanations. arXiv preprint arXiv:1811.05245(2018). Rory Mc Grath Luca Costabello Chan Le Van Paul Sweeney Farbod Kamiab Zhao Shen and Freddy Lecue. 2018. Interpretable Credit Application Predictions With Counterfactual Explanations. arXiv preprint arXiv:1811.05245(2018). · arXiv
- Himabindu Lakkaraju Ece Kamar Rich Caruana and Eric Horvitz. 2016. Discovering Unknown Unknowns of Predictive Models. In NeurIPS. Himabindu Lakkaraju Ece Kamar Rich Caruana and Eric Horvitz. 2016. Discovering Unknown Unknowns of Predictive Models. In NeurIPS.
- Lakkaraju Himabindu, 2017, Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML).
- Lipton C, 2018, The mythos of model interpretability. Commun. ACM 61, 10
- Ribeiro Marco Tulio, 2016, ICML Workshop on Human Interpretability in Machine Learning (WHI).
- Tan Sarah, 2018, Learning Global Additive Explanations for Neural Nets Using Model Distillation. In NeurIPS Workshop on Machine Learning for Health (ML4H).
Cited by
Ahmed, I., Jeon, G., & Piccialli, F. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3146552 · Cited by 853
Artificial intelligence applications in Latin American higher education: a systematic review
Salas-Pilco, S. Z., & Yang, Y. (2022) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-022-00326-w · Cited by 330
Neupane, S., Ables, J., Anderson, W., Mittal, S., Rahimi, S., Banicescu, I., & Seale, M. (2022) · IEEE Access · doi:10.1109/access.2022.3216617 · Cited by 236
A Survey on Data-driven Network Intrusion Detection
Chou, D., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3472753 · Cited by 189
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Metadata
- Authors
- Gade, K., Geyik, S., Kenthapadi, K., Mithal, V., & Taly, A.
- Affiliations
- Fiddler Labs
- LinkedIn (United States) (US)
- Amazon (United States) (US)
- Venue
- Companion Proceedings of the Web Conference 2020
- Publisher
- ACM
- Published
- 2020-04-20
- Pages
- 303-304
- OpenAlex
- W3022454434
- Cited by
- 46
- References
- 17
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 9 · 2023: 4 · 2022: 9 · 2021: 6
- FWCI
- 1.81
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International World Wide Web Conference (WWW) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- explainable AI, industrial applications, model interpretability, model transparency, algorithmic bias, trust in AI, high-stakes decision making, predictive maintenance, artificial intelligence adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.