Peeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional Expectation
Open accessAbstract
This article presents individual conditional expectation (ICE) plots, a tool for visualizing the model estimated by any supervised learning algorithm. Classical partial dependence plots (PDPs) help visualize the average partial relationship between the predicted response and one or more features. In the presence of substantial interaction effects, the partial response relationship can be heterogeneous. Thus, an average curve, such as the PDP, can obfuscate the complexity of the modeled relationship. Accordingly, ICE plots refine the PDP by graphing the functional relationship between the predicted response and the feature for individual observations. Specifically, ICE plots highlight the variation in the fitted values across the range of a covariate, suggesting where and to what extent heterogeneities might exist. In addition to providing a plotting suite for exploratory analysis, we include a visual test for additive structure in the data-generating model. Through simulated examples and real datasets, we demonstrate how ICE plots can shed light on estimated models in ways PDPs cannot. Procedures outlined are available in the R package ICEbox.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Individual conditional expectation curves and prediction gradients also allow for the detection of somewhere inaccuracies.
“Other, more direct, explanation methods for detecting monotonicity conflicts include individual conditional expectation curves (Goldstein et al. 2015) or prediction gradients (Wang et al. 2024), both of which also allow for the detection of somewhere inaccuracies.”
From Explainable AI Isn't Enough! Rethinking Algorithmic Contestability (2026) · cites this paper as Goldstein et al. 2015
Adoption of AI in clinical practice is still limited.
“there is no widespread use of AI in health care yet [4, 5].”
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
DeRubeis, R. J., Cohen, Z. D., Forand, N. R., Fournier, J. C., Gelfand, L. A., & Lorenzo-Luaces, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0083875 · Cited by 509 · cited 4× · see citing works
“The goal of the analysis in DeRubeis et al. (2014) is to understand how different subjects respond to different treatments, conditional on their personal covariates.”
BackgroundMethodInfluential
Flagged first, then by citations
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
“Finally, [238] presents ICE (Individual Conditional Expectation) plots as a tool for visualizing the model estimated by any supervised learning algorithm.”
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
“[112] introduced ICE techniques and proved it advantage over PDP.”
MethodExplainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · see citing works
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,062 · see citing works
“Goldstein et al. supported the same [133].”
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · see citing works
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Visualizing the effects of predictor variables in black box supervised learning models
Apley, D. W., & Zhu, J. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12377 · Cited by 1,535 · see citing works
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386 · see citing works
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008 · see citing works
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677 · see citing works
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · see citing works
On the interpretability of artificial intelligence in radiology: challenges and opportunities
Reyes, M., Meier, R., Pereira, S., Silva, C. A., Dahlweid, F.-M., Tengg-Kobligk, H. V., Summers, R. M., & Wiest, R. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190043 · Cited by 484 · see citing works
Opening the black box: interpretable machine learning for geneticists
Azodi, C. B., Tang, J., & Shiu, S.-H. (2020) · Trends in Genetics · doi:10.1016/j.tig.2020.03.005 · Cited by 473 · see citing works
Interpretable and Explainable Machine Learning for Materials Science and Chemistry
Oviedo, F., Ferres, J. L., Buonassisi, T., & Butler, K. T. (2022) · Accounts of Materials Research · doi:10.1021/accountsmr.1c00244 · Cited by 435 · see citing works
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
Krause, J., Perer, A., & Ng, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858529 · Cited by 390 · see citing works
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · see citing works
Weber, P., Carl, K. V., & Hinz, O. (2024) · Management Review Quarterly · doi:10.1007/s11301-023-00320-0 · Cited by 292 · see citing works
“For instance, Fernández (2020) predicts bank solvency in the United States and uses ICE plots for sensitivity analysis.”
InfluentialNazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283 · see citing works
“PDP was extended to propose a toolbox, Individual Conditional Expectation (ICE) plots [68].”
MethodInfluentialCounterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review
Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020) · arXiv · doi:10.48550/arxiv.2010.10596 · Cited by 267 · see citing works
Moustafa, N., Koroniotis, N., Keshk, M., Zomaya, A. Y., & Tari, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3280465 · Cited by 266 · see citing works
Hooker, G., Mentch, L., & Zhou, S. (2021) · Statistics and Computing · doi:10.1007/s11222-021-10057-z · Cited by 264 · see citing works
Causal interpretability for machine learning-problems, methods and evaluation
Moraffah, R., Karami, M., Guo, R., Raglin, A., & Liu, H. (2020) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3400051.3400058 · Cited by 238 · see citing works
A Survey of Data-driven and Knowledge-aware eXplainable AI
Li, X.-H., Cao, C. C., Shi, Y., Bai, W., Gao, H., Qiu, L., Wang, C., Gao, Y., Zhang, S., Xue, X., et al. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2983930 · Cited by 224 · see citing works
Explainable AI for clinical and remote health applications: A survey on tabular and time series data
Di Martino, F., & Delmastro, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10304-3 · Cited by 216 · see citing works
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212 · see citing works
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
“provide in Reference [35] a technique f that extends classical PDP named Individual Conditional Expectation (ICE) to visualize the model approximated by a black box b that helps in visualizing the average partial relationship between the outcome and some features.”
MethodExplainable artificial intelligence in Alzheimer’s disease classification: a systematic review
Viswan, V., Shaffi, N., Mahmud, M., Subramanian, K., & Hajamohideen, F. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10192-x · Cited by 195 · see citing works
Forkel, M., Andela, N., Harrison, S. P., Lasslop, G., van Marle, M., Chuvieco, E., Dorigo, W., Forrest, M., Hantson, S., Heil, A., et al. (2019) · Biogeosciences · doi:10.5194/bg-16-57-2019 · Cited by 184 · see citing works
Guleria, P., & Sood, M. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11221-2 · Cited by 178 · see citing works
Perturbation-Based Explanations of Prediction Models
Robnik-Šikonja, M., & Bohanec, M. (2018) · Human–Computer Interaction Series · doi:10.1007/978-3-319-90403-0_9 · Cited by 171 · see citing works
Sokol, K., & Flach, P. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00637-y · Cited by 186 · see citing works
General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G., Grosse-Wentrup, M., & Bischl, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_4 · Cited by 211 · see citing works
Explainable artificial intelligence for cybersecurity: a literature survey
Charmet, F., Tanuwidjaja, H. C., Ayoubi, S., Gimenez, P.-F., Han, Y., Jmila, H., Blanc, G., Takahashi, T., & Zhang, Z. (2022) · Annals of Telecommunications · doi:10.1007/s12243-022-00926-7 · Cited by 167 · see citing works
Wysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2022.103839 · Cited by 161 · see citing works
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135 · see citing works
“The authors propose to measure the feature importance for a single prediction with the Individual Conditional Expectation (ICE) curves [88]— curves showing how the prediction changes when the feature is changed, while other features are fixed to the values of the considered observation.”
BackgroundEXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications
Ortigossa, E. S., Gonçalves, T., & Nonato, L. G. (2024) · IEEE Access · doi:10.1109/access.2024.3409843 · Cited by 124 · see citing works
Boosting Insights in Insurance Tariff Plans with Tree-Based Machine Learning Methods
Henckaerts, R., Côté, M.-P., Antonio, K., & Verbelen, R. (2021) · North American Actuarial Journal · doi:10.1080/10920277.2020.1745656 · Cited by 109 · see citing works
Enhancing Explainability of Neural Networks Through Architecture Constraints
Yang, Z., Zhang, A., & Sudjianto, A. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3007259 · Cited by 106 · see citing works
Survey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98 · see citing works
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · see citing works
Mehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78 · see citing works
"Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations
Zhang, Y., Song, K., Sun, Y., Tan, S., & Udell, M. (2019) · arXiv · doi:10.48550/arxiv.1904.12991 · Cited by 68 · see citing works
Explainability Methods for Identifying Root-Cause of SLA Violation Prediction in 5G Network
Terra, A., Inam, R., Baskaran, S., Batista, P., Burdick, I., & Fersman, E. (2020) · GLOBECOM 2020 - 2020 IEEE Global Communications Conference · doi:10.1109/globecom42002.2020.9322496 · Cited by 50 · see citing works
Darvishvand, L., Kamkari, B., Huang, M. J., & Hewitt, N. J. (2025) · Sustainable Cities and Society · doi:10.1016/j.scs.2025.106492 · Cited by 42 · see citing works
Trust and Trustworthiness in AI
Durán, J. M., & Pozzi, G. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00843-2 · Cited by 47 · see citing works
Explainable outlier detection: What, for Whom and Why?
Sejr, J. H., & Schneider-Kamp, A. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100172 · Cited by 32 · see citing works
Modelling Motor Insurance Claim Frequency and Severity Using Gradient Boosting
Clemente, C., Guerreiro, G. R., & Bravo, J. M. (2023) · Risks · doi:10.3390/risks11090163 · Cited by 28 · see citing works
“ICE plots allow us to capture heterogeneity in the relationship between the feature variable and the response created by variable interaction, see Goldstein et al. (2015).”
Devanarayan, V., Charil, A., Horie, K., Doherty, T., Llano, D. A., Andreozzi, E., Sachdev, P., Ye, Y., Murali, L. K., Zhou, J., et al. (2025) · Alzheimer's & Dementia · doi:10.1002/alz.14411 · Cited by 19 · see citing works
From bias to black boxes: understanding and managing the risks of AI – an actuarial perspective
du Preez, V., Bennet, S., Byrne, M., Couloumy, A., Das, A., Dessain, J., Galbraith, R., King, P., Mutanga, V., Schiller, F., et al. (2024) · British Actuarial Journal · doi:10.1017/s1357321724000060 · Cited by 15 · see citing works
Causal Learning for Socially Responsible AI
Cheng, L., Mosallanezhad, A., Sheth, P., & Liu, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/598 · Cited by 15 · see citing works
Nomological Deductive Reasoning for Trustworthy, Human-Readable, and Actionable AI Outputs
Hakizimana, G., & Ledezma Espino, A. (2025) · Algorithms · doi:10.3390/a18060306 · Cited by 2 · see citing works
Mohamed, A. S., Fritz, C., Saber, A. M., Ma, Y., Kaur, R., Al-Darwbi, M., Friedrich, D., & Kundur, D. (2026) · arXiv · doi:10.48550/arxiv.2609.00171
“3) Feature Influence Visualization Techniques: PDP [164], Individual Conditional Expectancy (ICE) [165], and ALE [166] are model-agnostic XAI techniques used to visualize the relationship between input features and model predictions.”
Explainable AI Isn't Enough! Rethinking Algorithmic Contestability
Freiesleben, T., Meding, K., & König, G. (2026) · arXiv · doi:10.48550/arxiv.2605.16041
- Authors
- Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E.
- Affiliations
- City University of New York (US)
- University of Pennsylvania (US)
- Venue
- Journal of Computational and Graphical Statistics
- Publisher
- Informa UK Limited
- Published
- 2015-01-02
- Volume
- 24
- Issue
- 1
- Pages
- 44-65
- ISSN
- 1061-8600 (ISSN-L)
- 1537-2715
- OpenAlex
- W2125847307
- Cited by
- 1,861
- References
- 3
- Citations by year
- 2026: 298 · 2025: 319 · 2024: 284 · 2023: 270 · 2022: 221 · 2021: 215 · 2020: 127 · 2019: 71 · 2018: 35 · 2017: 14 · 2016: 5 · 2015: 1 · 2014: 1
- FWCI
- 6.98
- Type
- journal article
- Language
- en
- Open access
- green
- figshare.com
- Licence
- cc-by
- Locations
- Journal of Computational and Graphical Statistics (publishedVersion)
- Figshare (submittedVersion, cc-by)
- Figshare (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Applications
- Keywords
- partial dependence plots, statistical learning, model visualization, interaction effects, supervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
