Explainable Machine Learning for Scientific Insights and Discoveries
Open accessThe DOI matches its Crossref record; 135 of the 137 references it cites were found in a registry or an index.
Abstract
Machine learning methods have been remarkably successful for a wide range of application areas in the extraction of essential information from data. An exciting and relatively recent development is the uptake of machine learning in the natural sciences, where the major goal is to obtain novel scientific insights and discoveries from observational or simulated data. A prerequisite for obtaining a scientific outcome is domain knowledge, which is needed to gain explainability, but also to enhance scientific consistency. In this article, we review explainable machine learning in view of applications in the natural sciences and discuss three core elements that we identified as relevant in this context: transparency, interpretability, and explainability. With respect to these core elements, we provide a survey of recent scientific works that incorporate machine learning and the way that explainable machine learning is used in combination with domain knowledge from the application areas.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Several powerful AI algorithms are employing a so-called “black box” approach, where it is difficult or even impossible to understand how the obtained results have been achieved.66”
From Addressing bias in big data and AI for health care: A call for open science (2021) · cites this paper as [66]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“image or a video. Lerer et al. [2016] and Li et al. [2016] use video simulations to learn intuitive physics, for example about the stability of wooden block towers. Lerer et al. [2016] use ResNet-34 [He et al., 2016] and Googlenet [Szegedy et al., 2015] to predict the fall of towers of wooden blocks, as well as DeepMa…”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“o clips containing 101 objects of dierent characteristics, which was built for the task of deriving physical parameters such as velocity and mass. In their work, they use the LeNet CNN architecture [LeCun et al., 1998] to capture visual as well as physical characteristics while explicitly integrating physical laws bas…”
MethodGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
(2016) · Cited by 11,679
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“Further details, types of interpretation, and specific realization can be found in recent surveys [7], [8], [20].”
BackgroundDeep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Research works printed but not linked: 3
- 10.1037/0033-2909.110.3.499 · doi:10.1037/0033-2909.110.3.499 · Paper page
- frigg, 2018, Scientific representation, The Stanford Encyclopedia of Philosophy
- 10.1145/3233231 · doi:10.1145/3233231 · Paper page
Cited by
Items 101–200 of 999
Machine Learning in Chemical Product Engineering: The State of the Art and a Guide for Newcomers
Trinh, C., Meimaroglou, D., & Hoppe, S. (2021) · Processes · doi:10.3390/pr9081456 · Cited by 67
Explainable machine learning for project management control
Santos, J. I., Pereda, M., Ahedo, V., & Galán, J. M. (2023) · Computers & Industrial Engineering · doi:10.1016/j.cie.2023.109261 · Cited by 106
Machine learning in proton exchange membrane water electrolysis — A knowledge-integrated framework
Chen, X., Rex, A., Woelke, J., Eckert, C., Bensmann, B., Hanke‐Rauschenbach, R., & Geyer, P. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123550 · Cited by 69
Trust and stakeholder perspectives on the implementation of AI tools in clinical radiology
Bergquist, M., Rolandsson, B., Gryska, E. A., Laesser, M., Hoefling, N., Heckemann, R. A., Schneiderman, J. F., & Björkman‐Burtscher, I. M. (2023) · European Radiology · doi:10.1007/s00330-023-09967-5 · Cited by 80
“The common demand for AI models is to be explainable and interpretable so that human experts can understand the reasons for the model output [10].”
Multiclass sentiment analysis on COVID-19-related tweets using deep learning models
Vernikou, S., Lyras, A., & Kanavos, A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07650-2 · Cited by 69
“…for future work as the produced explainable models will maintain a high level of learning performance (in terms of prediction accuracy) as well as will explain their rationale, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future [53, 54].”
BackgroundGeospatial Information Research: State of the Art, Case Studies and Future Perspectives
Bill, R., Blankenbach, J., Breunig, M., Haunert, J., Heipke, C., Herlé, S., Maas, H., Mayer, H., Meng, L., Rottensteiner, F., et al. (2022) · PFG – Journal of Photogrammetry Remote Sensing and Geoinformation Science · doi:10.1007/s41064-022-00217-9 · Cited by 95
“There are large databases of annotated terrestrial RGB images such as ImageNet (Russakovsky et al. 2015) which form a good basis for training classifiers that generalize well for a large variety of applications but are restricted to the object types shown in these images and to the type of sensor used for acquiring th…”
BackgroundInfluentialAn Advanced Dirichlet Prior Network for Out-of-Distribution Detection in Remote Sensing
Gawlikowski, J., Saha, S., Kruspe, A., & Zhu, X. X. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3140324 · Cited by 65
“Our work is also a step forward towards building explainable remote sensing model [41], [42].”
BackgroundNguyen, M. H., Mai, H.-V. T., Trinh, S. H., & Ly, H. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-08042-2 · Cited by 59
“These interpretable models clarify how ML models generate their predictions by highlighting the significance of features and the interdependence of variables [55, 56].”
BackgroundGhasemkhani, B., Aktaş, Ö., & Birant, D. (2023) · Machines · doi:10.3390/machines11030322 · Cited by 75
Explainable Security in SDN-Based IoT Networks
Sarica, A. K., & Angın, P. (2020) · Sensors · doi:10.3390/s20247326 · Cited by 65
“Although deep learning models have been shown to be quite successful in intrusion detection, they are usually used as blackboxes, and their decision-making processes are not readily explainable to human experts [1].”
BackgroundLejarza, F., & Bâldea, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-13644-w · Cited by 71
“ML techniques perform well in regression and classification tasks, but the resulting models are “black” or “grey-box” in nature, offering little physical insight [2], and extrapolate poorly to regimes beyond the scope of the training data.”
MethodA review of explainable AI in the satellite data, deep machine learning, and human poverty domain
Hall, O., Ohlsson, M., & Rognvaldsson, T. S. (2022) · Patterns · doi:10.1016/j.patter.2022.100600 · Cited by 78
“Therefore, we use the framework from Roscher et al. to structure and review the concepts of transparency, interpretability, explainability, and domain knowledge (see Tables 1, 2, and 3).”
BackgroundMethodInfluentialTamagusko, T., & Ferreira, A. (2023) · Infrastructures · doi:10.3390/infrastructures8120170 · Cited by 72
“Additionally, the lack of interpretability limits the transparency of the ML models, making it difficult to assess their reliability and validity [81–83].”
BackgroundSitu, G. (2022) · Light Advanced Manufacturing · doi:10.37188/lam.2022.013 · Cited by 66
Meddage, D. P. P., Ekanayake, I. U., Perera, U. S., Azamathulla, H. M., Said, M. A. M., & Rathnayake, U. (2022) · Buildings · doi:10.3390/buildings12060734 · Cited by 65
Huang, F., Teng, Z., Yao, C., Jiang, S., Catani, F., Chen, W., & Huang, J. (2023) · Journal of Rock Mechanics and Geotechnical Engineering · doi:10.1016/j.jrmge.2023.11.001 · Cited by 61
Passos, D., & Mishra, P. (2021) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2021.104354 · Cited by 58
Colour and Texture Descriptors for Visual Recognition: A Historical Overview
Bianconi, F., Fernández, A., Smeraldi, F., & Pascoletti, G. (2021) · Journal of Imaging · doi:10.3390/jimaging7110245 · Cited by 64
Integrated data-driven modeling and experimental optimization of granular hydrogel matrices
Verheyen, C. A., Uzel, S. G. M., Kurum, A., Roche, E. T., & Lewis, J. A. (2023) · Matter · doi:10.1016/j.matt.2023.01.011 · Cited by 56
Comparison of Traffic Accident Prediction with Explainable Machine Learning Methods
Çiçek, E., Akın, M., Uysal, F., & Aytas, R. M. T. (2022) · Transportation Letters · doi:10.1080/19427867.2023.2214758 · Cited by 59
“Explainable machine-learning increases transparency, interpretability, explainability, domain knowledge, and scientific consistency of a machinelearning model (Roscher et al. 2020).”
Anomaly and intrusion detection using deep learning for software-defined networks: A survey
Ruffo, V. G. D. S., Lent, D. M. B., Komarchesqui, M., Schiavon, V. F., de Assis, M. V. O., Carvalho, L. F., & Proença, M. L. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124982 · Cited by 78
Papacharalampous, G., & Tyralis, H. (2022) · Frontiers in Water · doi:10.3389/frwa.2022.961954 · Cited by 64
Ayoub, O., Troìa, S., Andreoletti, D., Bianco, A. G., Tornatore, M., Giordano, S., & Rottondi, C. (2022) · Journal of Optical Communications and Networking · doi:10.1364/jocn.470812 · Cited by 63
“Extracting insights and discoveries [22] (insights).”
Interpretable Long Short-Term Memory Networks for Crop Yield Estimation
Mateo-Sanchis, A., Adsuara, J. E., Piles, M., Muñoz-Marí, J., Pérez-Suay, A., & Camps‐Valls, G. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3244064 · Cited by 65
“Model interpretability is especially important when data-driven deep-learning DL networks are used to support decision-making [11].”
BackgroundKhan, N., Nauman, M. A., Almadhor, A. S., Akhtar, N., Alghuried, A., & Alhudhaif, A. (2024) · IEEE Access · doi:10.1109/access.2024.3420415 · Cited by 73
“Roscher et al. [71] explored the recent scientific works and the contributions towards explainable ML in several application domains.”
BackgroundManthena, H., Shajarian, S., Kimmell, J. C., Abdelsalam, M., Khorsandroo, S., & Gupta, M. (2025) · IEEE Access · doi:10.1109/access.2025.3555926 · Cited by 100
“DL-based models, in particular, are frequently described as black boxes, as their decision-making processes remain largely inscrutable to users and security professionals alike [1].”
Predicting the presence of hazardous materials in buildings using machine learning
Wu, P.-Y., Sandels, C., Mjörnell, K., Mangold, M., & Johansson, T. D. (2022) · Building and Environment · doi:10.1016/j.buildenv.2022.108894 · Cited by 59
Alert-driven Attack Graph Generation using S-PDFA
Nadeem, A., Verwer, S., Moskal, S., & Yang, S. J. (2021) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3117348 · Cited by 49
“These notions are described by Roscher et al. [46], who list the three components of explainable machine learning as: transparency , interpretability , and explainability .”
BackgroundDeep reinforcement learning for quantum multiparameter estimation
Cimini, V., Valeri, M., Polino, E., Piacentini, S., Ceccarelli, F., Corrielli, G., Spagnolo, N., Osellame, R., & Sciarrino, F. (2023) · Advanced Photonics · doi:10.1117/1.ap.5.1.016005 · Cited by 72
“Different supervised and unsupervised learning algorithms have been applied to solve efficiently quantum many-body problems,(28) reconstruct the density matrix of high-dimensional quantum systems,(29) and even to design new quantum experiments and discover new physical concepts.(35,36) Their application to the metrolo…”
BackgroundKim, H., Crow, W. T., Li, X., Wagner, W., Hahn, S., & Lakshmi, V. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113776 · Cited by 57
AI-DARWIN: A first principles-based model discovery engine using machine learning
Chakraborty, A., Sivaram, A., & Venkatasubramanian, V. (2021) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2021.107470 · Cited by 51
Validity Arguments for AI‐Based Automated Scores: Essay Scoring as an Illustration
Ferrara, S., & Qunbar, S. (2022) · Journal of Educational Measurement · doi:10.1111/jedm.12333 · Cited by 72
Understanding electricity prices beyond the merit order principle using explainable AI
Trebbien, J., Gorjão, L. R., Praktiknjo, A., Schäfer, B., & Witthaut, D. (2023) · Energy and AI · doi:10.1016/j.egyai.2023.100250 · Cited by 60
Conceptual Understanding through Efficient Automated Design of Quantum Optical Experiments
Krenn, M., Kottmann, J. S., Tischler, N., & Aspuru‐Guzik, A. (2021) · Physical Review X · doi:10.1103/physrevx.11.031044 · Cited by 95
“The main difference is that these applications so far have been applied to rediscover previously known concepts [26].”
BackgroundHagen, A., & Andersen, T. M. (2024) · Structure and Infrastructure Engineering · doi:10.1080/15732479.2024.2311911 · Cited by 48
“Strategies that are best in catching anomalies due to complex interactions also tend to be more difficult in terms of ‘transparency, interpretability, and explainability’ (Roscher et al., 2020).”
BackgroundWhen Machine Learning Meets Spectrum Sharing Security: Methodologies and Challenges
Wang, Q., Sun, H., Hu, R. Q., & Bhuyan, A. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3146364 · Cited by 64
Singh, A., Amutha, J., Nagar, J., & Sharma, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118588 · Cited by 62
“We have selected Radial Basis Neural Network (RBN), Exact Radial Basis Neural Network (ERBN), Recurrent Neural Network (RNN), LSTM, Boosting (Leastsquare boosting) Ensemble Learning (EL), Bagging EL (Random Forest), Binary Decision Tree (BDT), General Regression Neural Network (GRNN), Gaussian Process Regression (GPR)…”
Advances and gaps in the science and practice of impact‐based forecasting of droughts
Shyrokaya, A., Pappenberger, F., Pechlivanidis, I. G., Messori, G., Khatami, S., Mazzoleni, M., & Baldassarre, G. D. (2023) · Wiley Interdisciplinary Reviews Water · doi:10.1002/wat2.1698 · Cited by 64
“…the performance of process-based (causal) models of drought impacts — also known as model benchmarking predictions and which often focus solely on optimizing predictive accuracy on specific sets of test data without providing explicit explanations for their predicted impacts (Roscher et al., 2020).”
BackgroundMachine learning topological phases in real space
Holanda, N. L., & Griffith, M. A. R. (2020) · Physical review. B./Physical review. B · doi:10.1103/physrevb.102.054107 · Cited by 55
Deep Learning-Driven Data Curation and Model Interpretation for Smart Manufacturing
Zhang, J., & Gao, R. X. (2021) · Chinese Journal of Mechanical Engineering · doi:10.1186/s10033-021-00587-y · Cited by 59
Geometrical Interpretation and Design of Multilayer Perceptrons
Lin, R., Zhou, Z., You, S. O., Rao, R., & Kuo, C. ‐. J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3190364 · Cited by 54
Designing the ultrasonic treatment of nanoparticle-dispersions via machine learning
Glaubitz, C., Rothen‐Rutishauser, B., Lattuada, M., Balog, S., & Petri‐Fink, A. (2022) · Nanoscale · doi:10.1039/d2nr03240f · Cited by 51
Abu, M., Egbueri, J. C., & Agbasi, J. C. (2025) · Environmental Earth Sciences · doi:10.1007/s12665-025-12174-8 · Cited by 51
“In recent times, considerable efforts have been dedicated to addressing this concern (Guidotti et al. 2019), leading to the conception of methods aimed at clarifying the predictions of the model on both a local level and a global level (Roscher et al. 2020).”
Neural hierarchical models of ecological populations
Joseph, M. B. (2020) · Ecology Letters · doi:10.1111/ele.13462 · Cited by 59
Wu, Y., Meng, Y., & Shao, C. (2022) · Journal of Manufacturing Processes · doi:10.1016/j.jmapro.2022.09.011 · Cited by 45
Cooper, C., Zhang, J., Huang, J., Bennett, J. L., Cao, J., & Gao, R. X. (2023) · Journal of Materials Processing Technology · doi:10.1016/j.jmatprotec.2023.117908 · Cited by 51
The promise of explainable deep learning for omics data analysis: Adding new discovery tools to AI
Santorsola, M., & Lescai, F. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.06.002 · Cited by 49
Hwangbo, S., Al, R., & Sin, G. (2020) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2020.107071 · Cited by 44
Shen, C., Krenn, M., Eppel, S., & Aspuru-Guzik, A. (2021) · Machine Learning: Science and Technology · doi:10.1088/2632-2153/ac09d6 · Cited by 45
“In particular, we take interest in the recent progress in the machine-assisted discovery of concepts in the natural sciences [23, 24, 25, 26].”
BackgroundBIM Integration with XAI Using LIME and MOO for Automated Green Building Energy Performance Analysis
Khan, A. M., Tariq, M. A., Rehman, S. K. U., Saeed, T., Alqahtani, F. K., & Sherif, M. (2024) · Energies · doi:10.3390/en17133295 · Cited by 64
Guo, Y., Wang, N., Shao, S., Huang, C., Zhang, Z., Li, X., & Wang, Y. (2024) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2024.114804 · Cited by 50
Geyer, P., Singh, M. M., & Chen, X. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102843 · Cited by 63
Explainability can foster trust in artificial intelligence in geoscience
Dramsch, J. S., Kuglitsch, M. M., Fernández‐Torres, M., Toreti, A., Albayrak, A. R., Nava, L., Ghaffarian, S., Cheng, X., Ma, J., Samek, W., et al. (2025) · Nature Geoscience · doi:10.1038/s41561-025-01639-x · Cited by 65
“These methods can provide insight into an AI system, identifying issues related to data or the model.”
EXPLAINABILITY OF NEURAL NETWORK CLUSTERING IN INTERPRETING THE COVID-19 EMERGENCY DATA
Yu, Z., Sohail, A., Nofal, T. A., & Tavares, J. M. R. S. (2021) · Fractals · doi:10.1142/s0218348x22401223 · Cited by 42
Braun, M., & Kellner, L. (2022) · Fatigue & Fracture of Engineering Materials & Structures · doi:10.1111/ffe.13800 · Cited by 42
Memristive tonotopic mapping with volatile resistive switching memory devices
Milozzi, A., Ricci, S., & Ielmini, D. (2024) · Nature Communications · doi:10.1038/s41467-024-47228-1 · Cited by 52
“Furthermore, thesesystems pose challenges in terms of interpretability, as the biological mapping and meaning are often lost 37 .”
Explainable machine learning models for corn yield prediction using UAV multispectral data
Kumar, C., Dhillon, J. S., Huang, Y., & Reddy, K. N. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.109990 · Cited by 54
Naser, M. Z. (2025) · Computers & Industrial Engineering · doi:10.1016/j.cie.2025.111704 · Cited by 63
The grammar of interactive explanatory model analysis
Baniecki, H., Parzych, D., & Biecek, P. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00924-w · Cited by 73
“While Roscher et al. (2020) surveys XIML use for knowledge discovery, Lipton (2018) and Miller (2019) point out that there is a huge margin for improvement in the area of human-centered XIML.”
BackgroundAsset Management in Machine Learning: State-of-research and State-of-practice
Idowu, S., Strüber, D., & Berger, T. (2022) · ACM Computing Surveys · doi:10.1145/3543847 · Cited by 52
“, model explainability [16, 67] and interpretability [17, 31]) should be considered in future tools for improved result interpretation and decision making.”
BackgroundMethodMachine learning approach towards explaining water quality dynamics in an urbanised river
Schäfer, B., Beck, C., Rhys, H. I., Soteriou, H., Jennings, P., Beechey, A., & Heppell, C. (2022) · Scientific Reports · doi:10.1038/s41598-022-16342-9 · Cited by 51
“When applying machine learning, it is important to avoid black-box solutions, as these do not provide process-based scientific insight 33 .”
MethodAboulola, O. I. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300640 · Cited by 57
“Explanation of models enhances transparency, interpretability, explanatory capacity, domain knowledge integration, and scientific coherence of models [45].”
BackgroundOn the interpretability of quantum neural networks
Pira, L., & Ferrie, C. (2024) · Quantum Machine Intelligence · doi:10.1007/s42484-024-00191-y · Cited by 77
“The interpretability of AI models is a pivotal concern in contemporary AI research, particularly with the ubiquity of deep neural networks.”
Li, X., Yang, Y. J., Ishizaka, J., & Li, X. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113628 · Cited by 48
Stieler, F., Rabe, F., & Bauer, B. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00199 · Cited by 47
Mapping forest in the Swiss Alps treeline ecotone with explainable deep learning
Nguyen, T.-A., Kellenberger, B., & Tuia, D. (2022) · Remote Sensing of Environment · doi:10.1016/j.rse.2022.113217 · Cited by 46
Reinforcement learning explains various conditional cooperation
Geng, Y., Liu, Y., Lu, Y., Shen, C., & Ши, Л. (2022) · Applied Mathematics and Computation · doi:10.1016/j.amc.2022.127182 · Cited by 54
Deep learning-enhanced nuclear medicine SPECT imaging applied to cardiac studies
Apostolopoulos, I. D., Papandrianos, N. I., Feleki, A., Moustakidis, S., & Papageorgiou, E. I. (2023) · EJNMMI Physics · doi:10.1186/s40658-022-00522-7 · Cited by 41
“A feature extraction/selection step (sometimes referred to as feature engineering) is applied first in a standard ML pipeline to extract or identify the most useful features [9].”
MethodComprehensible Artificial Intelligence on Knowledge Graphs: A survey
Schramm, S., Wehner, C., & Schmid, U. (2023) · Journal of Web Semantics · doi:10.1016/j.websem.2023.100806 · Cited by 66
“…of CAI as a conglomerate of methods that reconstruct the decision-making process of a black box AI model with the help of a human-understandable mapping from the model’s input to its output and methods that make the decision-making process of an AI model human-understandable [30, 31, 32, 33].”
MethodChen, H., Yang, L., & Wu, Q. (2023) · Remote Sensing · doi:10.3390/rs15184585 · Cited by 53
Machine learning for beyond Li-ion batteries: Powering the research
Kilic, A., Oral, B., Eroğlu, D., & Yıldırım, R. (2023) · Journal of Energy Storage · doi:10.1016/j.est.2023.109057 · Cited by 41
Shi, K., Gu, D., Liu, H., Chen, Y., & Lin, K. (2023) · Virtual and Physical Prototyping · doi:10.1080/17452759.2023.2266641 · Cited by 43
“Even advanced ML algorithms are not always logically reliable and may show poor extrapolation performance [39,40], greatly affecting the usability of pure data-driven algorithms in small datasets.”
BackgroundFreiesleben, T., König, G., Molnar, C., & Tejero-Cantero, Á. (2024) · Minds and Machines · doi:10.1007/s11023-024-09691-z · Cited by 82
“Instead of the EPE itself, the learning algorithm minimizes the empirical risk on training data and is then evaluated based on the empirical risk on test data (i.e. data not contained in the training data), which constitutes a finite-data estimate of the EPE (Hastie et al. 2009).”
Machine learning approach for spatial modeling of ridesourcing demand
Zhang, X., & Zhao, X. (2022) · Journal of Transport Geography · doi:10.1016/j.jtrangeo.2022.103310 · Cited by 50
Thomas, E., Wilson, D. L., Kathuni, S., Libey, A., Chintalapati, P., & Coyle, J. (2021) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2021.146486 · Cited by 39
Ekanayake, I. U., Palitha, S., Gamage, S., Meddage, D. P. P., Wijesooriya, K., & Mohotti, D. (2023) · Materials Today Communications · doi:10.1016/j.mtcomm.2023.106545 · Cited by 41
Explainable sequence-to-sequence GRU neural network for pollution forecasting
Borujeni, S. M., Arras, L., Srinivasan, V., & Samek, W. (2023) · Scientific Reports · doi:10.1038/s41598-023-35963-2 · Cited by 44
“These methods not only facilitate transparency and trust in the model’s decision, they potentially also lead to new scientific insights and discoveries 40 .”
Leitherer, A., Yeo, B. C., Liebscher, C. H., & Ghiringhelli, L. M. (2023) · npj Computational Materials · doi:10.1038/s41524-023-01133-1 · Cited by 47
Rudin, C. D., & Shaposhnik, Y. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3395422 · Cited by 43
“…focus on this research area and refer the reader to survey papers on interpretable and explainable machine learning to other bodies of work (Rudin et al. 2022, Došilović et al. 2018, Guidotti et al. 2018b, Arya et al. 2020, Tjoa and Guan 2020, Roscher et al. 2020, Burkart and Huber 2021).”
Meddage, D. P. P., Ekanayake, I. U., Herath, S., Gobirahavan, R., Muttil, N., & Rathnayake, U. (2022) · Sensors · doi:10.3390/s22124398 · Cited by 39
“Accordingly, XAI turns a black box model into a glass box model, thereby elucidating the working principle and causality behind predictions [44,45].”
BackgroundAfdhal, A., Jiroušek, O., Palar, P. S., Falta, J., & Dwianto, Y. B. (2023) · Materials & Design · doi:10.1016/j.matdes.2023.112128 · Cited by 44
Use of explainable machine learning models in blast load prediction
Widanage, C., Mohotti, D., Lee, C. K., Wijesooriya, K., & Meddage, D. P. P. (2024) · Engineering Structures · doi:10.1016/j.engstruct.2024.118271 · Cited by 37
Scientific intuition inspired by machine learning-generated hypotheses
Friederich, P., Krenn, M., Tamblyn, I., & Aspuru‐Guzik, A. (2021) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000133179 · Cited by 35
Non-perturbative renormalization for the neural network-QFT correspondence
Erbin, H., Lahoche, V., & Samary, D. O. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac4f69 · Cited by 64
“Moreover, having a black box is also a drawback for scientific discovery since the goal of science is to interpret and explain, and knowledge can grow only from understanding [5].”
BackgroundOpportunities for machine learning to accelerate halide-perovskite commercialization and scale-up
Kumar, R. E., Tiihonen, A., Sun, S., Fenning, D. P., Liu, Z., & Buonassisi, T. (2022) · Matter · doi:10.1016/j.matt.2022.04.016 · Cited by 46
Duarte, J. C., da Rocha, R. D., & Borges, I. (2023) · Physical Chemistry Chemical Physics · doi:10.1039/d2cp05339j · Cited by 39
An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning
Müller, S., Toborek, V., Beckh, K., Jakobs, M., Bauckhage, C., & Welke, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43418-1_28 · Cited by 73
AQ-Bench: a benchmark dataset for machine learning on global air quality metrics
Betancourt, C., Stomberg, T. T., Roscher, R., Schultz, M. G., & Stadtler, S. (2021) · Earth system science data · doi:10.5194/essd-13-3013-2021 · Cited by 55
“Benchmarks can also be used for the development of explainable artificial intelligence approaches (Kierdorf et al., 2020; Roscher et al., 2020) So far, few of such benchmark datasets exist in the field of environmental science, especially re-lated to air quality .”
BackgroundSelf-citationSong, W., Hou, Z., Gu, W., Afgan, M. S., Cui, J., Wang, H., Wang, Y., & Wang, Z. (2022) · Spectrochimica Acta Part B Atomic Spectroscopy · doi:10.1016/j.sab.2022.106490 · Cited by 37
Sivaram, A., & Venkatasubramanian, V. (2022) · AIChE Journal · doi:10.1002/aic.17687 · Cited by 39
Li, R., Zhang, Y., & Chen, H. (2022) · AIAA Journal · doi:10.2514/1.j061673 · Cited by 33
“There are various definitions of interpretability in machine learning, and one popular definition consists of making sense of the obtained model [7].”
BackgroundComparing human and AI expertise in the academic peer review process: towards a hybrid approach
Farber, S. (2025) · Higher Education Research & Development · doi:10.1080/07294360.2024.2445575 · Cited by 33
“This lack of transparency can raise questions about the reliability and accountability of AI-assisted peer review, while detecting and rectifying inaccuracies or biases remains challenging (Roscher et al., 2020).”
Panigrahi, B., Razavi, S., Doig, L. E., Cordell, B., Gupta, H. V., & Liber, K. (2025) · Water Resources Research · doi:10.1029/2024wr037398 · Cited by 38
“…implications for societal and environmental well‐being, ML often faces acceptability and legitimacy challenges (Coyle & Weller, 2020; Lakkaraju & Bastani, 2020; Lakkaraju et al., 2019; Lipton, 2017; Read et al., 2019; Roscher et al., 2020; Rudin, 2019; Samek et al., 2019; Slack et al., 2021, 2023).”
Prediction of Parkinson’s Disease Depression Using LIME-Based Stacking Ensemble Model
Nguyen, H. V., & Byeon, H. (2023) · Mathematics · doi:10.3390/math11030708 · Cited by 40
Cai, X., Li, L., Fisher, J. B., Zeng, Z., Zhou, S., Tan, X., Liu, B., & Chen, X. (2023) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.129696 · Cited by 37
Towards falsifiable interpretability researchPreprint
Leavitt, M. L., & Morcos, A. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12016 · Cited by 109
“A breadth of such approaches have been developed, typically referred to using the umbrella terms “interpretability” or “explainability” (Yosinski et al., 2015; Bau et al., 2017; Olah et al., 2018; Donnelly and Roegiest, 2019; Radford et al., 2017; Erhan et al., 2009; Zeiler and Fergus, 2014; Karpathy et al., 2016; Amj…”
BackgroundFeng, J., Li, J., Xu, C., Wang, Z., Zhang, Z., Wu, X., Lai, C., Zeng, Z., Tong, H., & Jiang, S. (2024) · Water Resources Research · doi:10.1029/2023wr036297 · Cited by 38
“However, the complex network structure and huge parameter space in certain ML algorithms, such as deep learning, have posed challenges in comprehending the underlying mechanics (Jiang et al., 2022; Mamalakis et al., 2022; Roscher et al., 2020).”
Why explainable AI may not be enough: predictions and mispredictions in decision making in education
Saqr, M., & López‐Pernas, S. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00343-4 · Cited by 68
“Interpretability made it clear why students were predicted and what variables were taken into account to produce the results (Došilović et al., 2018; Roscher et al., 2020).”
Molecular Origin of Blood‐Based Infrared Spectroscopic Fingerprints**
Voronina, L., Leonardo, C., Mueller‐Reif, J. B., Geyer, P. E., Huber, M., Trubetskov, M. K., Kepesidis, K. V., Behr, J., Mann, M., Krausz, F., et al. (2021) · Angewandte Chemie International Edition · doi:10.1002/anie.202103272 · Cited by 34
Learning main drivers of crop progress and failure in Europe with interpretable machine learning
Mateo-Sanchis, A., Piles, M., Amorós-López, J., Muñoz-Marí, J., Adsuara, J. E., Moreno‐Martínez, Á., & Camps‐Valls, G. (2021) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2021.102574 · Cited by 38
Metadata
- Authors
- Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J.
- Affiliations
- University of Bonn (DE)
- University of Massachusetts Amherst (US)
- Venue
- IEEE Access
- Published
- 2020
- Volume
- 8
- Pages
- 42200-42216
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2945526235
- Cited by
- 1,040
- References
- 137
- Citations by year
- 2026: 152 · 2025: 178 · 2024: 168 · 2023: 184 · 2022: 174 · 2021: 130 · 2020: 47
- FWCI
- 52.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Fraunhofer Cluster of Excellence Cognitive Internet Technologies
- Deutsche Forschungsgemeinschaft
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Keywords
- explainable machine learning, scientific discovery, transparency, interpretability, explainability, domain knowledge, observational data, simulated data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.