Why are we using black box models in AI when we don't need to? A lesson from an explainable AI competition
Open accessAbstract
In 2018, a landmark challenge in artificial intelligence (AI) took place, namely, the Explainable Machine Learning Challenge. The goal of the competition was to create a complicated black box model for the dataset and explain how it worked. One team did not follow the rules. Instead of sending in a black box, they created a model that was fully interpretable. This leads to the question of whether the real world of machine learning is similar to the Explainable Machine Learning Challenge, where black box models are used even when they are not needed. We discuss this team’s thought processes during the competition and their implications, which reach far beyond the competition itself.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI management of SAI would introduce new issues such as black box opacity of decision processes or inappropriate generalizations of incomplete data.
“This would introduce a raft of new issues, for instance 'black box' opacity of decision processes (Rudin and Radin, 2019) or inappropriate generalizations of incomplete data (Martin et al., 2012; Christie et al., 2020).”
From A Fate Worse Than Warming? Stratospheric Aerosol Injection and Global Catastrophic Risk (2021) · cites this paper as Rudin and Radin, 2019
Cites
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
The age of secrecy and unfairness in recidivism prediction
Rudin, C., Wang, C., & Coker, B. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.6ed64b30 · Cited by 177 · see citing works
Self-citationLearning Certifiably Optimal Rule Lists for Categorical Data
Angelino, E., Larus-Stone, N., Alabi, D., Seltzer, M., & Rudin, C. (2017) · arXiv · doi:10.48550/arxiv.1704.01701 · Cited by 108 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · see citing works
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958 · see citing works
Self-citationAmann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · see citing works
“, even the humans who design models do not always understand how variables are being combined to make inferences [171], many respondents discussed an end-to-end opacity of inputs, model behaviour, and outcomes in India.”
BackgroundFour Principles of Explainable Artificial Intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Broniatowski, D. A., & Przybocki, M. A. (2020) · doi:10.6028/nist.ir.8312-draft · Cited by 136 · see citing works
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135 · see citing works
“In particular, recent studies [17], [161] argue that post-hoc explanations are not reliable since they are not necessarily faithful to the underlying models and present correlations rather than information about the original computation.”
BackgroundMolina, M. D., & Sundar, S. S. (2024) · New Media & Society · doi:10.1177/14614448221103534 · Cited by 124 · see citing works
Sakai, A., Komatsu, M., Komatsu, R., Matsuoka, R., Yasutomi, S., Dozen, A., Shozu, K., Arakaki, T., Machino, H., Asada, K., et al. (2022) · Biomedicines · doi:10.3390/biomedicines10030551 · Cited by 68 · see citing works
Interpretable generalized additive neural networks
Kraus, M., Tschernutter, D., Weinzierl, S., & Zschech, P. (2024) · European Journal of Operational Research · doi:10.1016/j.ejor.2023.06.032 · Cited by 59 · see citing works
A Fate Worse Than Warming? Stratospheric Aerosol Injection and Global Catastrophic Risk
Tang, A., & Kemp, L. (2021) · Frontiers in Climate · doi:10.3389/fclim.2021.720312 · Cited by 58 · see citing works
“This would introduce a raft of new issues, for instance “black box” opacity of decision processes (Rudin and Radin, 2019) or inappropriate generalizations of incomplete data (Martin et al.”
BackgroundInterpretable algorithmic forensics
Garrett, B. L., & Rudin, C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2301842120 · Cited by 46 · see citing works
“As one of us has put it simply: “Why Are We Using Black Box Models in AI When We Don’t Need To?””
BackgroundSelf-citationClassification in the Wild: The Science and Art of Transparent Decision Making
Katsikopoulos, K. V., Şimşek, Ö., Buckmann, M., & Gigerenzer, G. (2021) · doi:10.7551/mitpress/11790.001.0001 · Cited by 42 · see citing works
Novazi, Z. (2025) · Law, Technology and Humans · doi:10.5204/lthj.4081
- Authors
- Rudin, C., & Radin, J.
- Affiliations
- Duke University (US)
- Yale University (US)
- ORCID
- Rudin, Cynthia
- Venue
- Harvard Data Science Review
- Publisher
- MIT Press
- Published
- 2019-11-01
- Volume
- 1
- Issue
- 2
- ISSN
- 2644-2353 (ISSN-L)
- OpenAlex
- W2991138092
- Cited by
- 432
- References
- 3
- Citations by year
- 2026: 32 · 2025: 72 · 2024: 82 · 2023: 101 · 2022: 78 · 2021: 48 · 2020: 18 · 2019: 1
- FWCI
- 16.73
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Harvard Data Science Review (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- explainable AI, black-box models, model interpretability, model selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
