Explainable Machine Learning for Scientific Insights and Discoveries
Open accessThe DOI matches its Crossref record; 135 of the 137 references it cites were found in a registry or an index.
Abstract
Machine learning methods have been remarkably successful for a wide range of application areas in the extraction of essential information from data. An exciting and relatively recent development is the uptake of machine learning in the natural sciences, where the major goal is to obtain novel scientific insights and discoveries from observational or simulated data. A prerequisite for obtaining a scientific outcome is domain knowledge, which is needed to gain explainability, but also to enhance scientific consistency. In this article, we review explainable machine learning in view of applications in the natural sciences and discuss three core elements that we identified as relevant in this context: transparency, interpretability, and explainability. With respect to these core elements, we provide a survey of recent scientific works that incorporate machine learning and the way that explainable machine learning is used in combination with domain knowledge from the application areas.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several powerful AI algorithms are employing a so-called “black box” approach, where it is difficult or even impossible to understand how the obtained results have been achieved.66”
From Addressing bias in big data and AI for health care: A call for open science (2021) · cites this paper as [66]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“image or a video. Lerer et al. [2016] and Li et al. [2016] use video simulations to learn intuitive physics, for example about the stability of wooden block towers. Lerer et al. [2016] use ResNet-34 [He et al., 2016] and Googlenet [Szegedy et al., 2015] to predict the fall of towers of wooden blocks, as well as DeepMa…”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“o clips containing 101 objects of dierent characteristics, which was built for the task of deriving physical parameters such as velocity and mass. In their work, they use the LeNet CNN architecture [LeCun et al., 1998] to capture visual as well as physical characteristics while explicitly integrating physical laws bas…”
MethodGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
(2016) · Cited by 11,679
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“Further details, types of interpretation, and specific realization can be found in recent surveys [7], [8], [20].”
BackgroundDeep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Research works printed but not linked: 3
- 10.1037/0033-2909.110.3.499 · doi:10.1037/0033-2909.110.3.499 · Paper page
- frigg, 2018, Scientific representation, The Stanford Encyclopedia of Philosophy
- 10.1145/3233231 · doi:10.1145/3233231 · Paper page
Cited by
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
“Only recently, the technologies have become available to extract scientific insights from complex nonlinear models [121], [150], [166].”
BackgroundAddressing bias in big data and AI for health care: A call for open science
Norori, N., Hu, Q., Aellen, F. M., Faraci, F. D., & Tzovara, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100347 · Cited by 903
“66 Explainable AI by contrast includes interpretable AI models, where the strengths and weaknesses of a decision-making process are transparent.”
Backgroundvon Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Walczak, M., Pfrommer, J., Pick, A., et al. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3079836 · Cited by 793
“With machine learning models becoming more and more complex, there is also a growing need for models to be interpretable and explainable [9].”
BackgroundSelf-citationMachine learning for digital soil mapping: Applications, challenges and suggested solutions
Wadoux, A. M. J.-C., Minasny, B., & McBratney, A. B. (2020) · Earth-Science Reviews · doi:10.1016/j.earscirev.2020.103359 · Cited by 731
Jusup, M., Holme, P., Kanazawa, K., Takayasu, M., Romić, I., Wang, Z., Geček, S., Lipić, T., Podobnik, B., Wang, L., et al. (2022) · Physics Reports · doi:10.1016/j.physrep.2021.10.005 · Cited by 643
The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support
Razavi, S., Jakeman, A., Saltelli, A., Prieur, C., Iooss, B., Borgonovo, E., Plischke, E., Lo Piano, S., Iwanaga, T., Becker, W., et al. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2020.104954 · Cited by 628
“These two terms, usually used interchangeably in the literature, refer to the ability of a model developer to make sense of why the model functions the way it does and to explain that to a user (Rudin, 2019; Samek and Müller, 2019; Roscher et al., 2020).”
Gao, H., Sun, L., & Wang, J.-X. (2021) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.110079 · Cited by 621
“Physics-informed fully-connected neural networks Generally, training of a DNN model requires a vast amount of labeled data, which are often unavailable in many scientific machine learning (SciML) applications [9–12].”
BackgroundNotions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598
Recent advances in decision trees: an updated survey
Costa, V. G., & Pedreira, C. E. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10275-5 · Cited by 572
“With the growing ubiquity of machine learning and automated decision systems, there has been a rising interest in explainable machine learning : building models that can be, in some sense, understood by humans (Rudin 2019; Roscher et al. 2020).”
BackgroundNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
“Some argue that the terms are closely related but distinguish between them [21, 85, 181, 225], although there is no consensus on what the distinction exactly is [25].”
BackgroundMousavi, S. M., & Beroza, G. C. (2022) · Science · doi:10.1126/science.abm4470 · Cited by 506
“can contribute to scientific goals of the community (139), but presenting the properties of these models in understandable terms to humans (interpretability) and revealing the underlying reasons for their output decision (explainability) (140) would allay such skepticism and lead to wider adoption and new insights.”
Explainable machine learning in materials science
Zhong, X., Gallagher, B., Liu, S., Kailkhura, B., Hiszpanski, A., & Han, T. Y.-J. (2022) · npj Computational Materials · doi:10.1038/s41524-022-00884-7 · Cited by 501
“There is an increasing interest to apply XAI in scienti fi c studies 18,19 .”
BackgroundKöchling, A., & Wehner, M. C. (2020) · Business Research · doi:10.1007/s40685-020-00134-w · Cited by 511
Ekanayake, I. U., Meddage, D. P. P., & Rathnayake, U. (2022) · Case Studies in Construction Materials · doi:10.1016/j.cscm.2022.e01059 · Cited by 498
On scientific understanding with artificial intelligence
Krenn, M., Pollice, R., Guo, S. Y., Aldeghi, M., Cervera-Lierta, A., Friederich, P., dos Passos Gomes, G., Häse, F., Jinich, A., Nigam, A., et al. (2022) · Nature Reviews Physics · doi:10.1038/s42254-022-00518-3 · Cited by 372
“[85] Ribana Roscher, Bastian Bohn, Marco F Duarte,”
BackgroundAutonomous Discovery in the Chemical Sciences Part I: Progress
Coley, C. W., Eyke, N. S., & Jensen, K. F. (2020) · Angewandte Chemie International Edition · doi:10.1002/anie.201909987 · Cited by 317
Wang, J., Li, Y., Gao, R. X., & Zhang, F. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.04.004 · Cited by 317
Ahmad, K., Maabreh, M., Ghaly, M., Khan, K., Qadir, J., & Al-Fuqaha, A. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2021.100452 · Cited by 291
“For instance, in [53, 54, 55, 56], a survey of existing literature on explainable AI is presented.”
BackgroundWang, H., Li, B., Gong, J., & Xuan, F.-Z. (2023) · Engineering Fracture Mechanics · doi:10.1016/j.engfracmech.2023.109242 · Cited by 281
Machine learning in materials science: From explainable predictions to autonomous design
Pilania, G. (2021) · Computational Materials Science · doi:10.1016/j.commatsci.2021.110360 · Cited by 281
Mersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
“Model-agnostic explanations are also used to explain the general behavior of physics-aware AI models to ensure their decisions are understandable in various scenarios regardless of the specific model architecture [208, 209].”
MethodAutonomous Discovery in the Chemical Sciences Part II: Outlook
Coley, C. W., Eyke, N. S., & Jensen, K. F. (2020) · Angewandte Chemie International Edition · doi:10.1002/anie.201909989 · Cited by 245
Neural Additive Models: Interpretable Machine Learning with Neural Nets
Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B. J., Caruana, R., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13912 · Cited by 117
Jiang, S., Zheng, Y., Wang, C., & Babovic, V. (2022) · Water Resources Research · doi:10.1029/2021wr030185 · Cited by 244
“Compared to previous methods, our interpretive DL-based flood attribution approach provides a bottom-up perspective (Roscher et al., 2020) by the predictive relationships revealed from data.”
MethodStructural identification with physics-informed neural ordinary differential equations
Lai, Z., Mylonas, C., Nagarajaiah, S., & Chatzi, E. (2021) · Journal of Sound and Vibration · doi:10.1016/j.jsv.2021.116196 · Cited by 220
A novel method of multiaxial fatigue life prediction based on deep learning
Yang, J., Kang, G., Liu, Y., & Kan, Q. (2021) · International Journal of Fatigue · doi:10.1016/j.ijfatigue.2021.106356 · Cited by 215
Nilsson, S. R., Goodwin, N. L., Choong, J. J., Hwang, S., Wright, H. R., Norville, Z. C., Tong, X., Lin, D., Bentzley, B. S., Eshel, N., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.19.049452 · Cited by 212
“We have conscientiously selected a supervised approach, and incorporated a number of explainability and validation checkpoints 38,39 , to ensure that researchers unfamiliar with these approaches are confident in the accuracy and validity of the classifiers they generate.”
BackgroundMethodPerturbation-based methods for explaining deep neural networks: A survey
Ivanovs, M., Kadikis, R., & Ozols, K. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.030 · Cited by 204
Zhou, X., Wen, H., Li, Z., Zhang, H., & Zhang, W. (2022) · Geocarto International · doi:10.1080/10106049.2022.2076928 · Cited by 200
“Due to the growing demand for methods that can explain complex machine learning algorithms, interpretable machine learning algorithms have attracted much attention as a powerful interpretation tool, which can comprehensively explain the output results of complex machine learning algorithms (Roscher et al. 2020).”
BackgroundBikmukhametov, T., & Jäschke, J. (2020) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2020.106834 · Cited by 194
Zihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231166 · Cited by 187
“Unfortunately, explainability in the modeling context is an illdefined term that can also have other meanings and several other terms such as interpretability and transparency are in use [34].”
BackgroundAutonomous materials synthesis by machine learning and robotics
Shimizu, R., Kobayashi, S., Watanabe, Y., Ando, Y., & Hitosugi, T. (2020) · APL Materials · doi:10.1063/5.0020370 · Cited by 188
Shen, Y., & Pan, Y. (2023) · Applied Energy · doi:10.1016/j.apenergy.2022.120575 · Cited by 179
Garzón, A., Kapelan, Z., Langeveld, J., & Taormina, R. (2022) · Water Resources Research · doi:10.1029/2021wr031808 · Cited by 179
Roadmap on Machine learning in electronic structure
Kulik, H. J., Hammerschmidt, T., Schmidt, J., Botti, S., Marques, M. A. L., Boley, M., Scheffler, M., Todorović, M., Rinke, P., Oses, C., et al. (2022) · Electronic Structure · doi:10.1088/2516-1075/ac572f · Cited by 180
How Interpretable Machine Learning Can Benefit Process Understanding in the Geosciences
Jiang, S., Sweet, L., Blougouras, G., Brenning, A., Li, W., Reichstein, M., Denzler, J., Shangguan, W., Yu, G., Huang, F., et al. (2024) · Earth's Future · doi:10.1029/2024ef004540 · Cited by 182
“Despite the progress made in implementing IML to improve scientific understanding and discovery in many fields (Roscher et al., 2020b), its integration into established geoscientific methodologies still requires both more diverse and careful application.”
von Rueden, L., Mayer, S., Sifa, R., Bauckhage, C., & Garcke, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-44584-3_43 · Cited by 177
“For further examples we refer to a survey about explainable machine learning for scientific discovery [32].”
BackgroundSelf-citationExplainable Artificial Intelligence for Human Decision Support System in the Medical Domain
Knapič, S., Malhi, A., Saluja, R., & Främling, K. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030037 · Cited by 171
Explainable AI Over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directions
Jagatheesaperumal, S. K., Pham, Q.-V., Ruby, R., Yang, Z., Xu, C., & Zhang, Z. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3215676 · Cited by 171
“[39] XAI and ML A brief survey on the usage of XAI and ML for scientific insights and discoveries.”
Intelligent metaphotonics empowered by machine learning
Krasikov, S., Nonlinear Physics Center, R. S. O. P., Tranter, A., Bogdanov, A., Kivshar, Y., School of Physics and Engineering, I. U., & Centre for Quantum Computation and Communication Technology, D. O. Q. S. (2022) · Opto-Electronic Advances · doi:10.29026/oea.2022.210147 · Cited by 169
“[12] Ribana Roscher, Bastian Bohn, Marco F.”
Interpretable and explainable machine learning: A methods‐centric overview with concrete examples
Marcinkevičs, R., & Vogt, J. E. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1493 · Cited by 161
Sundui, B., Ramirez Calderon, O. A., Abdeldayem, O. M., Lázaro-Gil, J., Rene, E. R., & Sambuu, U. (2021) · Clean Technologies and Environmental Policy · doi:10.1007/s10098-020-01993-x · Cited by 160
“After that, a suitable MLA will be selected so that the users (i.e. plant operators and managers) are aware of the internal system dynamics thereby allowing them to thoroughly interpret the plant data and take decisions at the right time (Roscher et al. 2019).”
BackgroundPhysics-informed meta learning for machining tool wear prediction
Li, Y., Wang, J., Huang, Z., & Gao, R. X. (2021) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2021.10.013 · Cited by 171
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 129
“For example, a model that is used in high-stakes settings like clinical decision-making may need to be interpretable or explainable ( 140 ), so it may be more appropriate to choose a model type that is likelier to have these properties.”
Soil Science-Informed Machine Learning
Minasny, B., Bandai, T., Ghezzehei, T. A., Huang, Y., Ma, Y., McBratney, A. B., Ng, W., Norouzi, S., Padarian, J., Rudiyanto, N.-F., et al. (2024) · Geoderma · doi:10.1016/j.geoderma.2024.117094 · Cited by 178
Orynbaikyzy, A., Geßner, U., Mack, B., & Conrad, C. (2020) · Remote Sensing · doi:10.3390/rs12172779 · Cited by 165
“Feature selection is used not only to improve the classification accuracy but also to better understand and interpret the input dataset [29,31,32], leading to explainable classification outcomes [28].”
BackgroundInterpretable Machine Learning: A brief survey from the predictive maintenance perspective
Vollert, S., Atzmueller, M., & Theissler, A. (2021) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa45728.2021.9613467 · Cited by 174
XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process
Clement, T., Kemmerzell, N., Abdelaal, M., & Amberg, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010006 · Cited by 131
Cha, Y., Shin, J., Go, B., Lee, D., Kim, Y. W., Kim, T., & Park, Y. (2021) · Journal of Environmental Management · doi:10.1016/j.jenvman.2021.112719 · Cited by 134
Allal, Z., Noura, H. N., Salman, O., & Chahine, K. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.120392 · Cited by 130
A Review on Interpretable and Explainable Artificial Intelligence in Hydroclimatic Applications
Başağaoğlu, H., Chakraborty, D., do Lago, C. A. F., Gutierrez, L., Şahinli, M. A., Giacomoni, M. H., Furl, C. V., Mirchi, A., Moriasi, D. N., & Şengör, S. S. (2022) · Water · doi:10.3390/w14081230 · Cited by 153
“…that is understandable by a human [6] and is deemed to be explainable if it is capable of justifying the decision made, enhancing control, improving the decision, and revealing new knowledge [8] from a collection of interpretations from an IAI model coupled with contextual information [7].”
BackgroundMethodInfluentialToward Explainable Artificial Intelligence for Regression Models: A methodological perspective
Letzgus, S., Wagner, P., Lederer, J., Samek, W., Müller, K., & Montavon, G. (2022) · IEEE Signal Processing Magazine · doi:10.1109/msp.2022.3153277 · Cited by 166
“accuracy, runtime) [63], an increasingly relevant question is whether these explanations enable the human to truly understand the model at hand (also known as causability [35]) and whether these explanations can be turned into meaningful insights and decisions [25], [58], [12], [82].”
BackgroundArtificial Intelligence and Advanced Materials
López, C. (2023) · Advanced Materials · doi:10.1002/adma.202208683 · Cited by 144
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
“…responses enough to participate in the mixed-initiative execution process" [48], “to attain scientific outcomes with ML one wants an understanding" [102], “understand the ‘algorithmic decision model’" [29], general [3, 11, 21, 26, 27, 35, 38, 47, 89, 121, 130] (16/58) G2: building trust in the…”
BackgroundInfluentialHaghighatlari, M., Li, J., Heidar‐Zadeh, F., Liu, Y., Guan, X., & Head-Gordon, T. L. (2020) · Chem · doi:10.1016/j.chempr.2020.05.014 · Cited by 132
Harnessing the Metal–Insulator Transition of VO 2 in Neuromorphic Computing
Schofield, P., Bradicich, A., Gurrola, R. M., Zhang, Y., Brown, T. D., Pharr, M., Shamberger, P. J., & Banerjee, S. (2022) · Advanced Materials · doi:10.1002/adma.202205294 · Cited by 121
From Treatment to Healing:Envisioning a Decolonial Digital Mental Health
Pendse, S. R., Nkemelu, D., Bidwell, N. J., Jadhav, S., Pathare, S., De Choudhury, M., & Kumar, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501982 · Cited by 115
“” However, given a signifcant risk of surveillance and harm, participatory algorithm design must also incorporate accountability [89], social transparency [94], and explainability [34, 282].”
Faegh, M., Ghungrad, S., Oliveira, J. P., Rao, P., & Haghighi, A. (2024) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2024.11.066 · Cited by 142
Elucidating the Behavior of Nanophotonic Structures through Explainable Machine Learning Algorithms
Yeung, C. J., Tsai, J.-M., King, B., Kawagoe, Y., Ho, D., Knight, M. W., & Raman, A. P. (2020) · ACS Photonics · doi:10.1021/acsphotonics.0c01067 · Cited by 111
“This rapidly developing field aims to analyze and understand ML models in general, with domain-specific demonstrations of scientific insights that might in turn emerge 36,37 .”
BackgroundExplainable AI for earth observation: A review including societal and regulatory perspectives
Gevaert, C. M. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102869 · Cited by 177
“However, others present a distinction where interpretability refers to the ability to understand how models reach certain predictions while explainability links such interpretations to domain knowledge (Roscher et al., 2020b; Tuia et al., 2021; Zhang et al., 2022).”
BackgroundInfluentialSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103
Meddage, D. P. P., Ekanayake, I. U., Weerasuriya, A. U., Lewangamage, C. S., Tse, K. T., Miyanawala, T. P., & Ramanayaka, C. D. E. (2022) · Journal of Wind Engineering and Industrial Aerodynamics · doi:10.1016/j.jweia.2022.105027 · Cited by 98
A federated learning framework for cyberattack detection in vehicular sensor networks
Driss, M., Almomani, I., Huma, Z. E., & Ahmad, J. E. (2022) · Complex & Intelligent Systems · doi:10.1007/s40747-022-00705-w · Cited by 116
“Furthermore, we intend to look into the promising field of explainable FL [45], which refers to strategies for providing human-readable insights into data, variables, and decisions.”
BackgroundComputer-inspired quantum experiments
Krenn, M., Erhard, M., & Zeilinger, A. (2020) · Nature Reviews Physics · doi:10.1038/s42254-020-0230-4 · Cited by 86
Review: application of artificial intelligence in phenomics
Nabwire, S., Suh, H.-K., Kim, M. S., Baek, I., & Cho, B.-K. (2021) · Sensors · doi:10.3390/s21134363 · Cited by 91
“2020 [35], the rise and success of neural networks, coupled with the abundance of data and high-level computational and data processing infrastructure, has led to the comprehensive utilization of machine learning (ML) models and algorithms despite the challenge of model determinations for a given task.”
BackgroundA review of explainable artificial intelligence in smart manufacturing
Madathil, A. P., Luo, X., Liu, Q., Walker, C. T., Madarkar, R., & Qin, Y. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2513574 · Cited by 146
“This means that the role of each feature or parameter is clear, ensuring that no part of the decision-making relies on hidden or opaque factors (Lipton 2018; Roscher et al. 2020; Vilone and Longo 2020).”
Toward a Collective Agenda on AI for Earth Science Data Analysis
Tuia, D., Roscher, R., Wegner, J. D., Jacobs, N., Zhu, X. X., & Camps‐Valls, G. (2021) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2020.3043504 · Cited by 113
“[17], [18] A lack of human-understandable interpretation, with a tendency toward confirmation bias (e.”
BackgroundSelf-citationOpening the Black Box: A systematic review on explainable artificial intelligence in remote sensing
Höhl, A., Obadić, I., Fernández‐Torres, M., Najjar, H., Oliveira, D. A. B., Akata, Z., Dengel, A. R., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3467001 · Cited by 141
Outlook for Exploiting Artificial Intelligence in the Earth and Environmental Sciences
Boukabara, S.-A., Krasnopolsky, V., Penny, S. G., Stewart, J. Q., McGovern, A., Hall, D., Ten Hoeve, J. E., Hickey, J., Allen Huang, H.-L., Williams, J. K., et al. (2021) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-20-0031.1 · Cited by 87
“Reviews and references of different methods to combine data and knowledge into ML are given by von Rueden et al. (2019), while Roscher et al. (2020) survey how these concepts are applied in science.”
de Moura, I. R., Teles, A. S., Viana, D., Marques, J., Coutinho, L., & Silva, F. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104278 · Cited by 109
Lu, L., Daniels, J., Bailey, C. P., Hu, L., Phillips, R., & South, J. (2023) · Environmental Pollution · doi:10.1016/j.envpol.2023.121832 · Cited by 100
“In response to the long-standing black box issues of DL algorithms, explainable artificial intelligence (XAI) has emerged as a promising solution to improve the accountability of model behavior and feature importance estimation (Roscher et al., 2020; Rudin, 2019).”
Machine learning for optical fiber communication systems: An introduction and overview
Nevin, J. W., Nallaperuma, S., Shevchenko, N. A., Li, X., Faruk, M. S., & Savory, S. J. (2021) · APL Photonics · doi:10.1063/5.0070838 · Cited by 95
“Another interesting avenue for future research is the combination of physical models with ML, so as to embed our knowledge of system physics into models such as NNs and GPs, as discussed in Sec.”
BackgroundMethodInfluentialChallenges of Machine Learning Applied to Safety-Critical Cyber-Physical Systems
Pereira, A., & Thomas, C. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make2040031 · Cited by 117
“Due to the lack of understanding of the exact nature between the parameters learned during model training and the performance of the trained model [66], the adjustments and tuning inside the loop are driven by heuristics (i.”
BackgroundThisovithan, P., Aththanayake, H., Meddage, D. P. P., Ekanayake, I. U., & Rathnayake, U. (2023) · Results in Engineering · doi:10.1016/j.rineng.2023.101388 · Cited by 97
Revealing drivers and risks for power grid frequency stability with explainable AI
Kruse, J., Schäfer, B., & Witthaut, D. (2021) · Patterns · doi:10.1016/j.patter.2021.100365 · Cited by 102
“However, complex ML models do not provide insights on the mapping of input to output.(17,18) This is particularly problematic for critical infrastructures such as power systems, where the black box character poses a security risk.”
BackgroundStef, N., Başağaoğlu, H., Chakraborty, D., & Jabeur, S. B. (2023) · Energy Economics · doi:10.1016/j.eneco.2023.106822 · Cited by 81
Medical-informed machine learning: integrating prior knowledge into medical decision systems
Sirocchi, C., Bogliolo, A., & Montagna, S. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02582-4 · Cited by 105
“Interpretability Embedding domain knowledge within ML models can add a layer of interpretability to the model [78] by providing the rationale behind the model’s recommendations [79] and allowing the re-traceability of the model decisions to specific model components [42].”
MethodWang, Z., Gu, Y., Zheng, L., Hou, J., Zheng, H., Sun, S., & Wang, L. (2021) · Advanced Materials · doi:10.1002/adma.202106776 · Cited by 80
Explainable Unsupervised Machine Learning for Cyber-Physical Systems
Wickramasinghe, C. S., Amarasinghe, K., Marino, D., Rieger, C. G., & Manic, M. (2021) · IEEE Access · doi:10.1109/access.2021.3112397 · Cited by 94
Deng, Y., Yao, Y., Wang, Y., Yu., T. S., Cai, W.-H., Zhou, D., Yin, F., Liu, W., Liu, Y., Xie, C., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51433-3 · Cited by 87
“Furthermore, as deep learning methods are usually perceived as “ black-box ” processes, the importance of model interpretability for prediction in the context of biomedical research is increasingly recognised 23 – 25 .”
The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
Balagopalan, A., Zhang, H., Hamidieh, K., Hartvigsen, T., Rudzicz, F., & Ghassemi, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533179 · Cited by 133
“To make such blackbox models more useful, users require that they be understandable, often due to laws [13] or preference [12, 49, 95].”
BackgroundWagan, S. A., Koo, J., Siddiqui, I. F., Attique, M., Shin, D. R., & Qureshi, N. M. F. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.09.005 · Cited by 86
SoK: Explainable Machine Learning for Computer Security Applications
Nadeem, A., Vos, D., Cao, C., Pajola, L., Dieck, S., Baumgartner, R., & Verwer, S. (2023) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp57164.2023.00022 · Cited by 114
“Strictly speaking, black-box models cannot be interpretable [112].”
BackgroundGoing Beyond XAI: A Systematic Survey for Explanation-Guided Learning
Gao, Y., Gu, S., Jiang, J., Hong, S.-S., Yu, D., & Zhao, L. (2024) · ACM Computing Surveys · doi:10.1145/3644073 · Cited by 121
“Several related surveys provide an in-depth classification of machine learning interpretability methods in general [10, 61, 93, 124], while others focus on more specific fields of study.”
BackgroundAdvancing materials discovery through artificial intelligence
Otyepka, M., Pykal, M., & Otyepka, M. (2025) · Applied Materials Today · doi:10.1016/j.apmt.2025.102981 · Cited by 105
An Empirical Study of Refactorings and Technical Debt in Machine Learning Systems
Tang, Y., Khatchadourian, R., Bagherzadeh, M., Singh, R., Stewart, A., & Raja, A. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00033 · Cited by 101
“Plain-old-data type debt occurs when rich information used and produced by ML systems is encoded using primitives, making, for example, the purpose of hyperparameter indecipherable and predictions less explainable [1], [50].”
BackgroundKulasooriya, W. K. V. J. B., Ranasinghe, R. S. S., Perera, U. S., Thisovithan, P., Ekanayake, I. U., & Meddage, D. P. P. (2023) · Scientific Reports · doi:10.1038/s41598-023-40513-x · Cited by 85
“XAI converts models from black box to glass box (transparent) models by exposing underlying reasoning 31–38 .”
MethodTamagusko, T., Correia, M. G., & Ferreira, A. (2024) · Infrastructures · doi:10.3390/infrastructures9120213 · Cited by 99
“An increasing amount of research concentrates on interpretability and explainability in ML models [178–181].”
Sheng, H., Xia, J., Wang, Y., Wang, W., Zeng, S., She, D., & Wang, G. (2022) · Water Resources Research · doi:10.1029/2022wr032183 · Cited by 73
“Therefore, making machine learning models explicable is the focus of current research (Mcgovern et al., 2019; Murdoch et al., 2018; Roscher et al., 2020).”
BackgroundInterpretable Machine Learning for Discovery: Statistical Challenges and Opportunities
Allen, G. I., Gan, L., & Zheng, L. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040120-030919 · Cited by 115
Study becomes insight: Ecological learning from machine learning
Yu, Q., Ji, W., Prihodko, L., Ross, C. W., Anchang, J. Y., & Hanan, N. P. (2021) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13686 · Cited by 85
“However, when prediction is not the primary goal, most ML methods tend to behave like ‘black boxes’, meaning that it can be challenging to derive new understand - ing or ecological insights from these statistical models, irrespective of their predictive abilities (Roscher et al., 2020).”
BackgroundPerspectives on data-driven models and its potentials in metal forming and blanking technologies
Liewald, M., Bergs, T., Groche, P., Behrens, B., Briesenick, D., Müller, M., Niemietz, P., Kubik, C., & Müller, F. (2022) · Production Engineering · doi:10.1007/s11740-022-01115-0 · Cited by 82
Current Trends in Fluid Research in the Era of Artificial Intelligence: A Review
Sofos, F. D., Stavrogiannis, C., Exarchou-Kouveli, K. K., Akabua, D., Charilas, G., & Karakasidis, T. E. (2022) · Fluids · doi:10.3390/fluids7030116 · Cited by 80
Ronco, M., Tárraga, J. M., Muñoz-Marí, J., Piles, M., Marco, E. S., Wang, Q., Espinosa, M. T. M., Ponserre, S., & Camps‐Valls, G. (2023) · Nature Communications · doi:10.1038/s41467-023-43809-8 · Cited by 84
“We focused on new internal displace-mentsinthepresenceofsudden-onsetdisastersandexploitedXAIand causal methods to unravel the main drivers of the phenomenon.”
MethodInfluentialTranslating AI to Clinical Practice: Overcoming Data Shift with Explainability
Choi, Y., Yu, W., Nagarajan, M. B., Teng, P., Goldin, J. G., Raman, S. S., Enzmann, D. R., Kim, G. H. J., & Brown, M. S. (2023) · RadioGraphics · doi:10.1148/rg.220105 · Cited by 65
Integrating scientific knowledge into machine learning using interactive decision trees
Sarailidis, G., Wagener, T., & Pianosi, F. (2022) · Computers & Geosciences · doi:10.1016/j.cageo.2022.105248 · Cited by 73
Explainable AI (XAI) in image segmentation in medicine, industry, and beyond: A survey
Gipiškis, R., Tsai, C.-W., & Kurasova, O. (2024) · ICT Express · doi:10.1016/j.icte.2024.09.008 · Cited by 91
An Outlook for Deep Learning in Ecosystem Science
Perry, G. L. W., Seidl, R., Bellvé, A. M., & Rammer, W. (2022) · Ecosystems · doi:10.1007/s10021-022-00789-y · Cited by 87
Zsidai, B., Hilkert, A., Kaarre, J., Narup, E., Senorski, E. H., Grassi, A., Ley, C., Longo, U. G., Herbst, E., Hirschmann, M. T., et al. (2023) · Journal of Experimental Orthopaedics · doi:10.1186/s40634-023-00683-z · Cited by 73
“Essentially, a white-box approach to AI-intensive research answers how and why predictions are made and ensures that scientifically or clinically relevant building blocks are incorporated in the structure of models [39].”
BackgroundBig data and deep learning for RNA biology
Hwang, H., Jeon, H., Yeo, N., & Baek, D. (2024) · Experimental & Molecular Medicine · doi:10.1038/s12276-024-01243-w · Cited by 73
“Third, the dif fi culty in understanding the prediction of DL models often impedes the conception of scienti fi c hypotheses, which requires understanding the causal relationship between the input and the output 14 .”
Background
Metadata
- Authors
- Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J.
- Affiliations
- University of Bonn (DE)
- University of Massachusetts Amherst (US)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 42200-42216
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2945526235
- Cited by
- 1,040
- References
- 137
- Citations by year
- 2026: 152 · 2025: 178 · 2024: 168 · 2023: 184 · 2022: 174 · 2021: 130 · 2020: 47
- FWCI
- 52.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Fraunhofer Cluster of Excellence Cognitive Internet Technologies
- Deutsche Forschungsgemeinschaft
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Keywords
- explainable machine learning, scientific discovery, transparency, interpretability, explainability, domain knowledge, observational data, simulated data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.