PALM: Machine Learning Explanations For Iterative Debugging
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
When a Deep Neural Network makes a misprediction, it can be challenging for a developer to understand why. While there are many models for interpretability in terms of predictive features, it may be more natural to isolate a small set of training examples that have the greatest influence on the prediction. However, it is often the case that every training example contributes to a prediction in some way but with varying degrees of responsibility. We present Partition Aware Local Model (PALM), which is a tool that learns and summarizes this responsibility structure to aide machine learning debugging. PALM approximates a complex model (e.g., a deep neural network) using a two-part surrogate model: a meta-model that partitions the training data, and a set of sub-models that approximate the patterns within each partition. These sub-models can be arbitrarily complex to capture intricate local patterns. However, the meta-model is constrained to be a decision tree. This way the user can examine the structure of the meta-model, determine whether the rules match intuition, and link problematic test examples to responsible training data efficiently. Queries to PALM are nearly 30x faster than nearest neighbor queries for identifying relevant data, which is a key property for interactive applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In fact, removing the influence of outlier training samples from a model will lead to higher model performance and robustness [16].”
From Machine Unlearning: A Survey (2024) · cites this paper as [16]
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 2×
“Furthermore, highly expressive deep models such as deep neural networks, which in principle can learn any deterministic function, are susceptible to adversarial examples (i.e., imperceptible perturbations to the features that cause a change in prediction)[15].”
BackgroundRationalizing Neural Predictions
Lei, T., Barzilay, R., & Jaakkola, T. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1011 · Cited by 777 · cited 4×
“[13, 14, 16].”
BackgroundInfluentialProvenance in Databases: Why, How, and Where
Cheney, J., Chiticariu, L., & Tan, W.-C. (2009) · Foundations and Trends in Databases · doi:10.1561/1900000006 · Cited by 547 · cited 1×
“This notion is similar to responsibility in the data lineage and explanation literature [4, 17], in the context of black-box machine learning models as opposed to database queries with well defined semantics.”
ResultThe MADlib analytics library: or MAD skills, the SQL
Hellerstein, J. M., Ré, C., Schoppmann, F., Wang, D. Z., Fratkin, E., Gorajek, A., Ng, K. S., Welton, C., Feng, X., Li, K., et al. (2012) · Proceedings of the VLDB Endowment · doi:10.14778/2367502.2367510 · Cited by 373
Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention
Kim, J., & Canny, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.320 · Cited by 366 · cited 1×
“Another more recent approach in computer vision is to use attention models, enforce that the model focuses on certain features, for ex-plainability [9].”
BackgroundMLbase: A Distributed Machine-learning System
Kraska, T., Talwalkar, A., & Duchi, J. C. (2013) · Cited by 324
ActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264 · cited 1×
“Explaining a prediction exactly in terms of features is highly useful for an end-user, but developers also need to be able to trace modeling errors to training data [12].”
BackgroundSelf-citationScorpion: explaining away outliers in aggregate queries
Wu, E., & Madden, S. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536354.2536356 · Cited by 240 · cited 1×
“This notion is similar to responsibility in the data lineage and explanation literature [4, 17], in the context of black-box machine learning models as opposed to database queries with well defined semantics.”
ResultSelf-citationMacroBase: Prioritizing Attention in Fast Data
Bailis, P., Gan, E., Madden, S., Narayanan, D., Rong, K., & Suri, S. (2017) · Proceedings of the 2017 ACM International Conference on Management of Data · doi:10.1145/3035918.3035928 · Cited by 99 · cited 1×
“We believe that coupling explanations with anomaly detection may be a viable next step, as in the MacroBase project [3].”
Method
Research works printed but not linked: 3
- Andrew Crotty , Alex Galakatos , and Tim Kraska . 2014 . Tupleware: Distributed Machine Learning on Small Clusters . In IEEE Data Eng. Bull . http://sites.computer.org/debull/A14sept/p63.pdf Andrew Crotty, Alex Galakatos, and Tim Kraska. 2014. Tupleware: Distributed Machine Learning on Small Clusters. In IEEE Data Eng. Bull. http://sites.computer.org/debull/A14sept/p63.pdf
- Roy Fox* Sanjay Krishnan* Ken Goldberg and Ion Stoica. 2017. Multi-Layer Deep Option Discovery. In Under Review ICML. Roy Fox* Sanjay Krishnan* Ken Goldberg and Ion Stoica. 2017. Multi-Layer Deep Option Discovery. In Under Review ICML.
- Tameru Hailesilassie . 2016. Rule Extraction Algorithm for Deep Neural Networks: A Review. arXiv preprint arXiv:1610.05267 ( 2016 ). Tameru Hailesilassie. 2016. Rule Extraction Algorithm for Deep Neural Networks: A Review. arXiv preprint arXiv:1610.05267 (2016). · arXiv
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
“Many IML methods are surrogate model approaches [89,75,72,95,34,10,18,61] and differ, e.”
MethodA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403
Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267
“Some approaches are model-agnostic, e.g. Golden Eye, PALM [59, 71, 116].”
MethodCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K., Dickerson, J., & Shah, C. (2024) · ACM Computing Surveys · doi:10.1145/3677119 · Cited by 255
Counterfactual Explanations for Machine Learning: A Review.Preprint
Verma, S., Dickerson, J. P., & Hines, K. (2020) · arXiv (Cornell University) · Cited by 332
MODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211
“There are also works [76, 77, 80, 87] aiming at providing provenance information and explanation of models to data engineers to help them”
BackgroundDing, W., Abdel-Basset, M., Hawash, H., & Ali, A. M. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.013 · Cited by 183
Review Study of Interpretation Methods for Future Interpretable Machine Learning
Mi, J.-X., Li, A.-D., & Zhou, L.-F. (2020) · IEEE Access · doi:10.1109/access.2020.3032756 · Cited by 158
“Furthermore, the partition aware local model (PALM) was proposed to dispose of the misprediction in Deep Neural Network (DNN) [31] which helps machine learning to debug the reason for incorrect classifications.”
MethodXu, H., Zhu, T., Zhang, L., Zhou, W., & Yu, P. S. (2024) · ACM Computing Surveys · doi:10.1145/3603620 · Cited by 164
“In fact, removing the inluence of outlier training samples from a model will lead to higher model performance and robustness [16].”
BackgroundClassification of Explainable Artificial Intelligence Methods through Their Output Formats
Vilone, G., & Longo, L. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030032 · Cited by 152
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135
“Krishnan and Wu [112] proposed a technique called PALM, which stands for Partition Aware Local Model.”
MethodPost-hoc explanation of black-box classifiers using confident itemsets
Moradi, M., & Samwald, M. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113941 · Cited by 118
Principles and Practice of Explainable Machine Learning
Belle, V., & Papantonis, I. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.688969 · Cited by 99
“Another perspective in simplification is introduced in [48].”
BackgroundSurvey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
“Model Extraction [168] and Partition Aware Local Model (PALM) [169] utilize decision trees (DTs) to approximate complex models with the assumption that, as long as the approximation quality is good, the statistical properties of the complex model are reflected in the interpretable ones.”
BackgroundThe Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
Balagopalan, A., Zhang, H., Hamidieh, K., Hartvigsen, T., Rudzicz, F., & Ghassemi, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533179 · Cited by 136
“Many explainability methods aim to assist model debugging, while non-engineer users only choose when to accept a blackbox’s decisions [57, 86, 93].”
BackgroundAutomated Data Slicing for Model Validation: A Big Data - AI Integration Approach
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2916074 · Cited by 93
“PALM [7] isolates a small set of training examples that have the greatest influence on the prediction by approximating a complex model into an interpretable meta-model that partitions the training data”
BackgroundMethodInfluentialExplanation-Based Human Debugging of NLP Models: A Survey
Lertvittayakumjorn, P., & Toni, F. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00440 · Cited by 92
“This agrees with the correctability principles proposed by Kulesza et al. (2015) that the system should be actionable and reversible, honor user feedback, and show incremental changes.”
BackgroundMethodInfluentialValente, F., Paredes, S., Henriques, J. M. O., Rocha, T., Carvalho, P., & Morais, J. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00821-8 · Cited by 62
“Several methods have been proposed in this context (Puri et al. 2017; Frosst and Hinton 2017; Krishnan and Wu 2017).”
MethodComplaint-driven Training Data Debugging for Query 2.0
Wu, W., Flokas, L., Wu, E., & Wang, J. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3389696 · Cited by 52
Self-citationExplainable Methods for Image-Based Deep Learning: A Review
Gupta, L. K., Koundal, D., & Mongia, S. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09881-5 · Cited by 39
Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning WorkloadsPreprint
Li, T., Zhong, J., Liu, J., Wu, W., & Zhang, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.07308 · Cited by 33
“As a result, how to make machine learning techniques more accessible and usable to non-computer science users has become a research topic that has attracted intensive interest from the database community [3, 6, 24, 39, 41].”
BackgroundData Management in Machine Learning Systems
Böehm, M., Kumar, A., & Yang, J. (2019) · Synthesis lectures on data management · doi:10.1007/978-3-031-01869-5 · Cited by 41
Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653697 · Cited by 154
PrIU: A Provenance-Based Approach for Incrementally Updating Regression Models
Wu, Y., Tannen, V., & Davidson, S. B. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3380571 · Cited by 42
“xtensive human subjects experiments in [45], as well the many references in these papers). The problem is being studied from several different perspectives (see Sec. 2). The data-driven approaches of [15, 35] discover factors of interpretability by performing repeated retraining of models using multiple different subs…”
MethodExplaining Black-Box Models for Biomedical Text Classification
Moradi, M., & Samwald, M. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3056748 · Cited by 34
Renkhoff, J., Feng, K., Meier-Doernberg, M., Velasquez, A., & Song, H. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3351798 · Cited by 42
“Testing Tools [79], [80] Testing the Input and Oracle Design [81], [82], [83], [84], [85], [86] Searching Data Bugs [87], [88] 1) Properties to Validate: Correctness: Correctness is a fundamental property of a system, representing the probability that it completes a task correctly.”
BackgroundOn the Intersection of Explainable and Reliable AI for Physical Fatigue Prediction
Narteni, S., Orani, V., Cambiaso, E., Rucco, M., & Mongelli, M. (2022) · IEEE Access · doi:10.1109/access.2022.3191907 · Cited by 28
Explainable AI for Cybersecurity
Sindiramutty, S. R., Tan, C. E., Lau, S. P., Thangaveloo, R., Gharib, A. H., Manchuri, A. R., Khan, N. A., Tee, W. J., & Muniandy, L. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-6684-6361-1.ch002 · Cited by 34
Towards Enabling Feedback on Rhetorical Structure with Neural Sequence Models
Fiacco, J., Cotos, E., & Rosé, C. P. (2019) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3303772.3303808 · Cited by 31
“Though advances in interpretability of network architectures in areas like computer vision have inspired similar efforts in the language technologies field [25], progress has been slow, especially in regard to interpreting recurrent neural models [2, 21, 23, 28, 39, 40, 50].”
BackgroundMethodExplanation sets: A general framework for machine learning explainability
Fernández, R. R., de Diego, I. M., Moguerza, J. M., & Herrera, F. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.084 · Cited by 26
XElemNet: towards explainable AI for deep neural networks in materials science
Wang, K., Gupta, V., Lee, C. S. G., Mao, Y., Kilic, M. N. T., Li, Y., Huang, Z., Liao, W., Choudhary, A., & Agrawal, A. (2024) · Scientific Reports · doi:10.1038/s41598-024-76535-2 · Cited by 30
“One common method for approaching the transparency of opaque models is through simplification techniques 31,32 .”
DeepBase: Deep Inspection of Neural Networks
Sellam, T., Lin, K., Huang, I., Yang, M. Y., Vondrick, C., & Wu, E. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3300073 · Cited by 20
Self-citationData Management in Machine Learning Systems
Böehm, M., Kumar, A., & Yang, J. (2019) · Synthesis lectures on data management · doi:10.2200/s00895ed1v01y201901dtm057 · Cited by 25
Adversarial Examples on XAI-Enabled DT for Smart Healthcare Systems
Imam, N. H. (2024) · Sensors · doi:10.3390/s24216891 · Cited by 20
“On the other hand, rule-based ML models, such as decision trees [44] or decision sets [43], interpret a learned structure (e.g., IF-THEN) to understand how the model makes predictions.”
MethodQuantitative Evaluations on Saliency Methods: An Experimental StudyPreprint
Li, X., Shi, Y., Li, H., Bai, W., Song, Y., Cao, C. C., & Chen, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15616 · Cited by 16
“As tools explaining the decision-making process, XAI methods can be applied in human-in-the-loop machine learning systems [11, 21, 23] or model debugging systems [17, 29, 35].”
MethodExplaining Aggregates for Exploratory Analytics
Savva, F., Anagnostopoulos, C. B., & Triantafillou, P. (2018) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2018.8621953 · Cited by 13
“explaining ML models for debugging purposes [23] or to”
MethodExplainable Machine Learning for Autonomous Vehicle Positioning Using SHAP
Onyekpe, U., Lu, Y., Apostolopoulou, E., Palade, V., Eyo, E., & Kanarachos, S. A. (2022) · Intelligent systems reference library · doi:10.1007/978-3-031-12807-3_8 · Cited by 17
Deep Neural Model Inspection and Comparison via Functional Neuron Pathways
Fiacco, J., Choudhary, S., & Rosé, C. P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1575 · Cited by 14
“There have also been approaches that made use of simpler classifiers to predict and then explain mistakes made by more complex models (Ribeiro et al., 2016; Krishnan and Wu, 2017).”
MethodEstimation and Interpretation of Machine Learning Models with Customized Surrogate Model
Ali, M., Khattak, A. M., Ali, Z., Hayat, B., Idrees, M., Pervez, Z., Rizwan, K., Sung, T.-E., & Kim, K. (2021) · Electronics · doi:10.3390/electronics10233045 · Cited by 18
“Partition aware local model (PALM) is another robust surrogate f discussed in [38] wherein the black box b is mimicked with the help of a two-part surrogate model f : the meta-model was used for partitioning the training data D train , then those partitioned data were approximated with the…”
MethodTesting methods of neural systems understanding
Lindsay, G. W., & Bau, D. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101156 · Cited by 26
Explaining Entity Resolution Predictions
Thirumuruganathan, S., Ouzzani, M., & Tang, N. (2019) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3328519.3329130 · Cited by 9
“It is also possible to use other models for partitioning such as [10].”
BackgroundMethodNasir, M., Summerfield, N. S., Simsek, S., & Oztekin, A. (2024) · Decision Sciences · doi:10.1111/deci.12642 · Cited by 8
Toward a System Building Agenda for Data Integration (and Data Science).
Doan, A., Konda, P., Suganthan, G. C. P., Ardalan, A., Ballard, J. R., Das, S., Govind, Y., Li, H., Martinkus, P., Mudgal, S., et al. (2018) · IEEE Data(base) Engineering Bulletin · Cited by 10
Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical ContextPreprint
Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., González-Zelaya, V., Missier, P., Ng, M., & van Moorsel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08911 · Cited by 33
“In addition, there exist solutions that follow an agnostic approach through which any type of model gets to be explained [35, 97, 118, 140, 141].”
BackgroundLarge-scale Data Exploration Using Explanatory Regression Functions
Savva, F., Anagnostopoulos, C. B., Triantafillou, P., & Kolomvatsos, K. (2020) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3410448 · Cited by 9
“A recent trend is in explaining ML models for debugging purposes [31] or to understand how a model makes predictions [40] and conveys trust to users using these predictions.”
BackgroundTooling for Developing Data-Driven Applications: Overview and Outlook
Weber, T., & Hußmann, H. (2022) · Mensch und Computer (MuC) · doi:10.1145/3543758.3543779 · Cited by 6
Interpretability and Explainability of Machine Learning Models: Achievements and Challenges
Henriques, J. M. O., Rocha, T., Carvalho, P., Silva, C., & Paredes, S. (2024) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-59216-4_9 · Cited by 13
Arnaldo, I., & Veeramachaneni, K. K. (2019) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3373464.3373472 · Cited by 9
“Recent works show improved results by focusing on local approximations [15,32].”
BackgroundDeepEverest: accelerating declarative top-K queries for deep neural network interpretation
He, D. J., Daum, M., Cai, W., & Bałazińska, M. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3485450.3485460 · Cited by 11
“While many systems have been developed to enable various forms of DNN interpretation [2, 22, 23, 26, 45, 47], none supports flexible and efficient interpretation by example queries.”
Improving Algorithmic Decision–Making in the Presence of Untrustworthy Training Data
Qi, W., & Chelmis, C. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671677 · Cited by 7
Generating Purpose-Driven Explanations: The Case of Process Predictive Model Inspection
Wickramanayake, B., Ouyang, C., Moreira, C., & Xu, Y. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-07481-3_14 · Cited by 7
Revisiting Lightweight Compiler Provenance Recovery on ARM Binaries
Kim, J. Z., Genkin, D., & Leach, K. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00044 · Cited by 9
Canha, D., Kubler, S., Främling, K., & Fagherazzi, G. (2025) · ACM Computing Surveys · doi:10.1145/3737445 · Cited by 10
Explainable AI for Car Crash Detection using Multivariate Time Series
Tronchin, L., Sicilia, R., Cordelli, E., Celsi, L. R., Maccagnola, D., Natale, M., & Soda, P. (2021) · IEEE International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC) · doi:10.1109/iccicc53683.2021.9811335 · Cited by 6
When debugging encounters artificial intelligence: state of the art and open challenges
Song, Y., Xie, X., & Xu, B. (2024) · Science China Information Sciences · doi:10.1007/s11432-022-3803-9 · Cited by 6
Sharma, A., Leite, D. F., Demir, C., & Ngomo, A.-C. N. (2024) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee60900.2024.10611982 · Cited by 4
Achieving Transparency Report Privacy in Linear Time
Chen, C., Golubchik, L., & Pal, R. (2022) · Journal of Data and Information Quality · doi:10.1145/3460001 · Cited by 5
“Popular methods include Anchors [74] and PALM [57].”
MethodAmortized Generation of Sequential Counterfactual Explanations for Black-box Models.Preprint
Verma, S., Hines, K., & Dickerson, J. P. (2021) · arXiv (Cornell University) · Cited by 8
Alharith, R., zhang, J., Ibrahim, A. O., & Wu, Z. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02647-5 · Cited by 3
“Decision trees and sensitivity analysis can also be used in order to obtain local explanations (Krishnan & Wu, 2017; Adebayo et al., 2018).”
Mann, S. (2024) · AI & Society · doi:10.1007/s00146-023-01837-4 · Cited by 4
“Such a lack of understanding can lead to ethical problems (Mittelstadt et al. 2016), violate laws and regulations (Goodman and Flaxman 2017), or hinder epistemic and practical ends (Boge 2021; Krishnan and Wu 2017).”
Interpretability Index Based on Balanced Volumes for Transparent Models and Agnostic Explainers
Leite, D. F., Sharma, A., Demir, C., & Ngomo, A.-C. N. (2024) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee60900.2024.10611937 · Cited by 2
“The Partition Aware Local Model (PALM) method [29] is founded on the hypothesis that all instances contribute to a prediction, albeit with varying degrees of responsibility.”
MethodDeepEverest: Accelerating Declarative Top-K Queries for Deep Neural Network InterpretationPreprint
He, D. J., Daum, M., Cai, W., & Bałazińska, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02234 · Cited by 2
“While many systems have been developed to enable various forms of DNN interpretation [2, 22, 23, 26, 45, 47], none supports flexible and efficient interpretation by example queries.”
MIRAI: A Modifiable, Interpretable, and Rational AI Decision Support System
Mallia, N., Dingli, A., & Haddod, F. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-61045-6_10 · Cited by 2
Bridging Data and AIOps for Future AI Advancements with Human-in-the-Loop. The AI-DAPT Concept
Koussouris, S., Dalamagas, T., Figueiras, P. A., Pallis, G., Bountouni, N., Gkolemis, V., Perakis, Κ., Bibikas, D., & Agostinho, C. (2024) · IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC) · doi:10.1109/ice/itmc61926.2024.10794334 · Cited by 3
Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers
Tung, L. N., Cho, S. B., Du, X., Neelofar, N., Terragni, V., Ruberto, S., & Aleti, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729376 · Cited by 8
“Relying on prediction uncertainty is ineffective for trustworthiness assessment, overlooking both trustworthy low and untrustworthy high confidence predictions.”
Alaoui, E. A. A., Gamouz, N. E., Sallah, A., Filali, A., Hessane, A., & Agoujil, S. (2025) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc66714.2025.11134869 · Cited by 1
Locally Pareto-Optimal Interpretations for Black-Box Machine Learning Models
Joshi, A. J., Chakraborty, S., Akshay, S., Shah, S., Torfah, H., & Seshia, S. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-08707-2_15 · Cited by 1
A Systematic Review on Human Roles, Solutions, and Methodological Approaches to Address Bias in AI
Hashky, A., & Ragan, E. D. (2026) · ACM Computing Surveys · doi:10.1145/3793667 · Cited by 4
Searching for Meaning in RNNs using Deep Neural Inspection
Lin, K., & Wu, E. (2017) · doi:10.7916/d8h13dgd · Cited by 1
Self-citationLiang, W., Zhang, C.-C., & Xicai, Y. (2005) · PubMed · Cited by 1
Synthesizing Pareto-Optimal Interpretations for Black-Box ModelsPreprint
Torfah, H., Shah, S., Chakraborty, S., Akshay, S., & Seshia, S. A. (2021) · arXiv (Cornell University) · doi:10.34727/2021/isbn.978-3-85448-046-4_24 · Cited by 1
Quantifying Uncertainty in Data Exploration
Chung, Y. (2019) · doi:10.26300/801n-zz34
Forest explanation through pattern discovery
Veneri, A. (2021)
CoSP: co-selection pick for a global explainability of black box machine learning models
Mansouri, D. E. K., Benkabou, S.-E., Meddahi, K., Hadjali, A., Mesmoudi, A., Benabdeslem, K., & Chaib, S. (2023) · World Wide Web · doi:10.1007/s11280-023-01213-8
Thiyagarajan, G., Vijayalakshmi, V., & Viswanathan, R. (2024) · AIP conference proceedings · doi:10.1063/5.0217354
The Balancing Act of Policies in Developing Machine Learning Explanations
Tjaden, J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00076
A Neural-Symbolic learning framework to produce interpretable predictions for image classification
Bennetot, A. (2022) · theses.fr (ABES) · doi:10.70675/2b59f598z2028z49afz8a20z29b21f19f394 · Cited by 1
Metadata
- Authors
- Krishnan, S., & Wu, E.
- Affiliations
- University of California, Berkeley (US)
- Columbia University (US)
- Venue
- Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics
- Publisher
- ACM
- Published
- 2017-05-14
- Pages
- 1-6
- OpenAlex
- W2614087582
- Semantic Scholar
- 6315d7f2619d9ec05846765d1283dee28bce9bad
- Cited by
- 81
- References
- 13
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 15 · 2023: 5 · 2022: 10 · 2021: 15 · 2020: 11 · 2019: 11 · 2018: 4 · 2017: 2
- FWCI
- 3.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Workshop on Human-In-the-Loop Data Analytics (HILDA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- interpretability, training data influence, surrogate models, deep neural network
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.