Explainable Artificial Intelligence (XAI) for Internet of Things: A Survey
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) and machine learning (ML) are widely employed to make the solutions more accurate and autonomous in many smart and intelligent applications in the Internet of Things (IoT). In these IoT applications, the performance and accuracy of AI/ML models are the main concerns; however, the transparency, interpretability, and responsibility of the models’ decisions are often neglected. Moreover, in AI/ML-supported next-generation IoT applications, there is a need for more reliable, transparent, and explainable systems. In particular, regardless of whether the decisions are simple or complex, how the decision is made, which features affect the decision, and their adoption and interpretation by people or experts are crucial issues. Also, people typically perceive unpredictable or opaque AI outcomes with skepticism, which reduces the adoption and proliferation of IoT applications. To that end, explainable AI (XAI) has emerged as a promising research topic that allows ante-hoc and post-hoc functioning and stages of black-box models to be transparent, understandable, and interpretable. In this article, we provide an in-depth and systematic review of recent studies that use XAI models in the scope of the IoT domain. We classify the studies according to their methodology and application areas. Additionally, we highlight the challenges and open issues and provide promising future directions to lead the researchers in future investigations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 6×
“To clarify the similarities and differences among these terms, we will provide brief definitions as follows [7], [12], [28], [29].”
InfluentialBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Saliency Maps [54] and DeepLift [55] can be applied for local explanations.”
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“For example, the studies of [10], [11], and [12] explain XAI and focus on XAI concepts, terminology, taxonomy, and challenges, whereas the study of [13] examines XAI for time-series data.”
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Anchors [39] have also the ability of local explanations by extracting a set of if-then rules.”
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 1×
“Individual conditional explanation (ICE) [33] plot is a more recent method that is similar to PDP. PDP calculates the mean over the marginal distribution, whereas ICE keeps the entire distribution.”
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 1×
“The taxonomy of XAI can be classified from a variety of perspectives [12], [30], as shown in Fig.”
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · cited 1×
“about the inner workings of black-box models and provide transparency to their users [8].”
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1×
“There is ample evidence that humans have over-trusted AI systems in the past, and they still have a long way to go before they can fully trust them today [116].”
Visualizing the effects of predictor variables in black box supervised learning models
Apley, D. W., & Zhu, J. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12377 · Cited by 1,535
Predictive learning via rule ensembles
Friedman, J. H., & Popescu, B. E. (2008) · The Annals of Applied Statistics · doi:10.1214/07-aoas148 · Cited by 1,315 · cited 1×
“Feature Interaction [40] is used for identifying the interaction effect of the pair of feature–output or feature–feature.”
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · cited 1×
“Meaningful perturbation (MP) [52] is a local explanation approach for neural classifier predictions based on a framework of meta-predictors.”
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080
Distribution-Free Predictive Inference for Regression
Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1307116 · Cited by 1,049 · cited 1×
“Also, leave-one-covariate-out (LOCO) [42] is another method that involves dropping each variable one at a time, retraining the model, and comparing the following model error to a baseline model that consists of all features.”
Ahmed, I., Jeon, G., & Piccialli, F. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3146552 · Cited by 853 · cited 3×
“Here, considering the goals of XAI, the following additional reasons motivate us to adopt XAI approaches in the field of IoT [14], [15], [16], as shown in Fig.”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · cited 1×
“As a classifier explanation method, prediction difference analysis (PDA) [51] assigns a significance value to each feature concerning each class and calculates the importance of a feature by observing how prediction changes when the value of a feature is uncertain.”
An Explainable Machine Learning Framework for Intrusion Detection Systems
Wang, M., Zheng, K., Yang, Y., & Wang, X. (2020) · IEEE Access · doi:10.1109/access.2020.2988359 · Cited by 355
Hossain, M. S., Muhammad, G., & Guizani, N. (2020) · IEEE Network · doi:10.1109/mnet.011.2000458 · Cited by 347 · cited 1×
“Hossain et al. [79] proposed a three-tiered (stakeholder layer, edge layer, and cloud layer) smart healthcare framework that uses 5G networking to combat COVID-19-like pandemics.”
Senoner, J., Netland, T., & Feuerriegel, S. (2022) · Management Science · doi:10.1287/mnsc.2021.4190 · Cited by 317 · cited 1×
“Senoner et al. [89] designed a DT-based decision model to improve process quality in the manufacturing process.”
Human gut microbiome aging clock based on taxonomic profiling and deep learning
Galkin, F., Mamoshina, P., Aliper, A., Lane, E., Moskalev, V., Gladyshev, V. N., & Zhavoronkov, A. (2020) · iScience · doi:10.1016/j.isci.2020.101199 · Cited by 292
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
Le, T.-T.-H., Kim, H., Kang, H., & Kim, H. (2022) · Sensors · doi:10.3390/s22031154 · Cited by 208 · cited 1×
“In another study of the same scope, Le et al. [103] aimed to increase the intrusion detection performance of IDS systems working with ML algorithms in IoT-based security system data sets.”
Explainable Artificial Intelligence: An Introduction to Interpretable Machine Learning
Kamath, U., & Liu, J. (2021) · doi:10.1007/978-3-030-83356-5 · Cited by 126 · cited 1×
“These methods adopt different approaches and provide different interpretations [31].”
Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey
Rojat, T., Puget, R., Filliat, D., Del Ser, J., Gelin, R., & Díaz-Rodríguez, N. (2021) · arXiv · doi:10.48550/arxiv.2104.00950 · Cited by 116 · cited 1×
“Additionally, we highlight the challenges and open issues and provide promising future directions to lead the researchers in future investigations.”
Conceptual challenges for interpretable machine learning
Watson, D. S. (2022) · Synthese · doi:10.1007/s11229-022-03485-5 · Cited by 87
Mehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78 · cited 1×
“Mehdiyev and Fettke [90] developed a conceptual framework for predictive process monitoring that provides approaches to guide researchers and practitioners.”
Explainable Activity Recognition for Smart Home Systems
Das, D., Nishimura, Y., Vivek, R. P., Takeda, N., Fish, S. T., Plötz, T., & Chernova, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3561533 · Cited by 70 · cited 1×
“For example, a smart thermostat can use XAI to determine the optimal temperature based on home occupancy, time of day, and weather conditions [17].”
Cited by
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
Zhang, Z., Hamadi, H. A., Damiani, E., Yeun, C. Y., & Taher, F. (2022) · IEEE Access · doi:10.1109/access.2022.3204051 · Cited by 450
Fahim, Y. A., Hasani, I. W., Kabba, S., & Ragab, W. M. (2025) · European Journal of Medical Research · doi:10.1186/s40001-025-03196-w · Cited by 329
“Sensitive medical data are exposed to risks such as sensor spoofing during acquisition, scaling attacks during preprocessing, and adversarial manipulations that subtly alter inputs to trigger incorrect predictions or compromise privacy.”
Chataut, R., Phoummalayvane, A., & Akl, R. (2023) · Sensors · doi:10.3390/s23167194 · Cited by 329
Internet of Things and Wireless Sensor Networks for Smart Agriculture Applications: A Survey
Mowla, M. N., Mowla, N., Shah, A. F. M. S., Rabie, K. M., & Shongwe, T. (2023) · IEEE Access · doi:10.1109/access.2023.3346299 · Cited by 297
Survey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98
Metadata
- Authors
- Kök, İ., Okay, F. Y., Muyanlı, Ö., & Özdemir, S.
- Venue
- IEEE Internet of Things Journal
- Published
- 2023-08-15
- Volume
- 10
- Issue
- 16
- Pages
- 14764-14779
- ISSN
- 2327-4662 (ISSN-L)
- 2372-2541
- OpenAlex
- W4381327855
- Cited by
- 151
- References
- 36
- FWCI
- 13.18
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.