How Interpretable Machine Learning Can Benefit Process Understanding in the Geosciences
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Interpretable Machine Learning (IML) has rapidly advanced in recent years, offering new opportunities to improve our understanding of the complex Earth system. IML goes beyond conventional machine learning by not only making predictions but also seeking to elucidate the reasoning behind those predictions. The combination of predictive power and enhanced transparency makes IML a promising approach for uncovering relationships in data that may be overlooked by traditional analysis. Despite its potential, the broader implications for the field have yet to be fully appreciated. Meanwhile, the rapid proliferation of IML, still in its early stages, has been accompanied by instances of careless application. In response to these challenges, this paper focuses on how IML can effectively and appropriately aid geoscientists in advancing process understanding—areas that are often underexplored in more technical discussions of IML. Specifically, we identify pragmatic application scenarios for IML in typical geoscientific studies, such as quantifying relationships in specific contexts, generating hypotheses about potential mechanisms, and evaluating process‐based models. Moreover, we present a general and practical workflow for using IML to address specific research questions. In particular, we identify several critical and common pitfalls in the use of IML that can lead to misleading conclusions, and propose corresponding good practices. Our goal is to facilitate a broader, yet more careful and thoughtful integration of IML into Earth science research, positioning it as a valuable data science tool capable of enhancing our current understanding of the Earth system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439
Generalized Additive Models: An Introduction with R
Wood, S. N. (2017) · doi:10.1201/9781315370279 · Cited by 13,852
Eyring, V., Bony, S., Meehl, G. A., Senior, C. A., Stevens, B., Stouffer, R. J., & Taylor, K. E. (2016) · Geoscientific Model Development · doi:10.5194/gmd-9-1937-2016 · Cited by 11,981
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 2×
“When features are highly correlated, these perturbations can extrapolate into “uncharted” regions within the feature space that lie outside the observed joint distribution of the variables, leading to biased assessments of feature importance (Hooker et al., 2021).”
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · cited 6×
“…general principles of data preparation for ML models, such as handling missing values or outliers (Zhu et al., 2023), it is necessary to ensure that the data adequately reflect the temporal and spatial scales relevant to the processes under study (Jiang, Bevacqua, & Zscheischler, 2022; W. Li et…”
InfluentialBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 2×
“For readers seeking detailed guidance on method selection, numerous comprehensive reviews and studies (e.g., Barredo Arrieta et al., 2020; Bommer et al., 2024; Graziani et al., 2023; Mamalakis, Barnes, & Ebert‐Uph-off, 2022; McGovern, Lagerquist, et al., 2019; Schwalbe & Finzel, 2023; Zhong et al.,…”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
The Scenario Model Intercomparison Project (ScenarioMIP) for CMIP6
O'Neill, B. C., Tebaldi, C., van Vuuren, D. P., Eyring, V., Friedlingstein, P., Hurtt, G., Knutti, R., Kriegler, E., Lamarque, J.-F., Lowe, J., et al. (2016) · Geoscientific Model Development · doi:10.5194/gmd-9-3461-2016 · Cited by 6,189
Deep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673 · cited 5×
“To this end, the use of IML to evaluate these multifaceted relationships holds promise to provide geoscientists with a tool that complements and enhances traditional evaluation techniques and moves toward pattern‐ and process‐oriented model evaluation (Reichstein et al., 2019).”
InfluentialSelf-citationStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · cited 3×
“In contrast, IML enhances data exploration within large, high‐dimensional data sets that often contain a multitude of interacting factors and patterns that are not readily apparent through traditional statistical analysis (Breiman, 2001b).”
Quinlan, J. R. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80053-6 · Cited by 2,553
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
An Overview of Overfitting and its Solutions
Ying, X. (2019) · Journal of Physics: Conference Series · doi:10.1088/1742-6596/1168/2/022022 · Cited by 2,352
Explaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286 · cited 2×
“Similarly, Monte Carlo integration methods are often used to approximate Shapley values, which are also subject to sampling variability (Goldwasser & Hooker, 2023; Štrumbelj & Kononenko, 2013).”
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 5×
“…Başağaoğlu et al., 2022; Gevaert, 2022; Gilpin et al., 2018; Gunning et al., 2019; Mamalakis, Ebert‐Uphoff, & Barnes, 2022; Molnar et al., 2020; Murdoch et al., 2019; Roscher et al., 2020a, 2020b) that provide in‐depth discussions of various interpretation techniques, their theoretical…”
InfluentialAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×
“Careful evaluation of model performance is essential to derive meaningful interpretations in subsequent steps.”
XAI—Explainable artificial intelligence
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G.-Z. (2019) · Science Robotics · doi:10.1126/scirobotics.aay7120 · Cited by 2,053
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823 · cited 1×
“It was hoped that ML would revolutionize scientific inquiry (H. Wang et al., 2023), but this anticipated transformation has yet to fully manifest.”
Accurate medium-range global weather forecasting with 3D neural networks
Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., & Tian, Q. (2023) · Nature · doi:10.1038/s41586-023-06185-3 · Cited by 1,670 · cited 1×
“The widespread application of machine learning (ML) in the geosciences, particularly for predictive modeling, represents a significant technological advance (e.g., Bi et al., 2023; Ham et al., 2019).”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · cited 3×
“Our concentration is more on the broader concept and practical applications of IML, and thus most terms related to IML can be used interchangeably with XAI in the following discussions.”
The Inter-Sectoral Impact Model Intercomparison Project (ISI–MIP): Project framework
Warszawski, L., Frieler, K., Huber, V., Piontek, F., Serdeczny, O., & Schewe, J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1312330110 · Cited by 1,253
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040 · cited 3×
“Despite the progress made in implementing IML to improve scientific understanding and discovery in many fields (Roscher et al., 2020b), its integration into established geoscientific methodologies still requires both more diverse and careful application.”
What Role Does Hydrological Science Play in the Age of Machine Learning?
Nearing, G. S., Kratzert, F., Sampson, A. K., Pelissier, C. S., Klotz, D., Frame, J. M., Prieto, C., & Gupta, H. V. (2021) · Water Resources Research · doi:10.1029/2020wr028091 · Cited by 894
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880
Artificial intelligence and illusions of understanding in scientific research
Messeri, L., & Crockett, M. J. (2024) · Nature · doi:10.1038/s41586-024-07146-0 · Cited by 707 · cited 2×
“In many cases, applying different interpretation methods to a single model (Mamalakis, Barnes, & Ebert‐Uphoff, 2022), or even applying the same interpretation method but with varying settings or hyperparameters (S. Müller et al., 2023), can lead to different results.”
Model-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699 · cited 2×
“For instance, the LIME method constructs simpler, surrogate models based on perturbed samples to locally approximate the prediction function of complex models (Tulio Ribeiro et al., 2016).”
Mitigating the Multicollinearity Problem and Its Machine Learning Approach: A Review
Chan, J. Y.-L., Leow, S. M. H., Bea, K. T., Cheng, W. K., Phoong, S. W., Hong, Z.-W., & Chen, Y.-L. (2022) · Mathematics · doi:10.3390/math10081283 · Cited by 617 · cited 1×
“In this case, the importance of features may be spread across multiple features, suggesting a weak or negligible association with the response (Brenning, 2023), or it may show high variability and even directional shifts (Chan et al., 2022).”
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · cited 1×
“Several metrics have been developed to evaluate the suitability and effectiveness of interpretation techniques, focusing on comparing key properties between techniques for specific research problems (e.g., Hedström et al., 2024; Nauta et al., 2023).”
Accurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564 · cited 4×
“Alternatively, one could consider Explainable Boosting Machines (Lou et al., 2013), where interaction terms can be explicitly specified during model configuration and each interaction term is modeled and interpreted separately.”
InfluentialRogger, M., Agnoletti, M., Alaoui, A., Bathurst, J. C., Bodner, G., Borga, M., Chaplot, V., Gallart, F., Glatzel, G., Hall, J., et al. (2017) · Water Resources Research · doi:10.1002/2017wr020723 · Cited by 448
Science in the age of large language models
Birhane, A., Kasirzadeh, A., Leslie, D., & Wachter, S. (2023) · Nature Reviews Physics · doi:10.1038/s42254-023-00581-4 · Cited by 344
Machine learning-based global maps of ecological variables and the challenge of assessing them
Meyer, H., & Pebesma, E. (2022) · Nature Communications · doi:10.1038/s41467-022-29838-9 · Cited by 310 · cited 4×
“…2023; Brenning, 2022; Davenport & Diffenbaugh, 2021; de Burgh‐Day & Leeuwenburg, 2023; Lopez‐Gomez et al., 2023; McGovern, Jergensen, et al., 2019; Meyer & Pebesma, 2022) (detailed in Text S3 in Supporting Information S1), and implementing strategies such as regularization and early stopping to…”
InfluentialTowards neural Earth system modelling by integrating artificial intelligence in Earth system science
Irrgang, C., Boers, N., Sonnewald, M., Barnes, E. A., Kadow, C., Staneva, J., & Saynisch-Wagner, J. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00374-3 · Cited by 308
Pathways and pitfalls in extreme event attribution
van Oldenborgh, G. J., van der Wiel, K., Kew, S., Philip, S., Otto, F., Vautard, R., King, A., Lott, F., Arrighi, J., Singh, R., et al. (2021) · Climatic Change · doi:10.1007/s10584-021-03071-7 · Cited by 277
Hooker, G., Mentch, L., & Zhou, S. (2021) · Statistics and Computing · doi:10.1007/s11222-021-10057-z · Cited by 264
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211 · cited 9×
“On the other hand, complex ML models, such as deep neural networks (NNs) and boosting algorithms, often outperform simpler models in terms of accuracy, but lack inherent interpretability (Dramsch, 2020; Molnar et al., 2020).”
InfluentialFrom attribution maps to human-understandable explanations through Concept Relevance Propagation
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00711-8 · Cited by 181
Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Watkins, E. A., Manger, C., Daumé III, H., Riener, A., & Riedl, M. O. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503727 · Cited by 117
Global extreme heat forecasting using neural weather models
Lopez-Gomez, I., McGovern, A., Agrawal, S., & Hickey, J. (2023) · Artificial Intelligence for the Earth Systems · doi:10.1175/aies-d-22-0035.1 · Cited by 41 · cited 2×
“…of the specific characteristics of the data (Bischl et al., 2023; Brenning, 2022; Davenport & Diffenbaugh, 2021; de Burgh‐Day & Leeuwenburg, 2023; Lopez‐Gomez et al., 2023; McGovern, Jergensen, et al., 2019; Meyer & Pebesma, 2022) (detailed in Text S3 in Supporting Information S1), and…”
Spatial machine-learning model diagnostics: a model-agnostic distance-based approach
Brenning, A. (2023) · International Journal of Geographical Information Science · doi:10.1080/13658816.2022.2131789 · Cited by 33 · cited 3×
“…requires careful management of the training and test data sets with consideration of the specific characteristics of the data (Bischl et al., 2023; Brenning, 2022; Davenport & Diffenbaugh, 2021; de Burgh‐Day & Leeuwenburg, 2023; Lopez‐Gomez et al., 2023; McGovern, Jergensen, et al., 2019; Meyer &…”
InfluentialSelf-citationCausal implicatures from correlational statements
Gershman, S. J., & Ullman, T. D. (2023) · PLOS ONE · doi:10.1371/journal.pone.0286067 · Cited by 21 · cited 1×
“However, there remains the possibility that readers may interpret correlational statements as causal (Gershman & Ullman, 2023).”
Metadata
- Authors
- Jiang, S., Sweet, L., Blougouras, G., Brenning, A., Li, W., Reichstein, M., Denzler, J., Shangguan, W., Yu, G., Huang, F., & Zscheischler, J.
- Venue
- Earth's Future
- Publisher
- American Geophysical Union (AGU)
- Published
- 2024-07
- Volume
- 12
- Issue
- 7
- ISSN
- 2328-4277 (ISSN-L)
- OpenAlex
- W4400660198
- Cited by
- 182
- References
- 47
- FWCI
- 27.09
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.