Sources of Opacity in Computer Systems: Towards a Comprehensive Taxonomy
Open accessAbstract
Modern computer systems are ubiquitous in contemporary life yet many of them remain opaque. This poses significant challenges in domains where desiderata such as fairness or accountability are crucial. We suggest that the best strategy for achieving system transparency varies depending on the specific source of opacity prevalent in a given context. Synthesizing and extending existing discussions, we propose a taxonomy consisting of eight sources of opacity that fall into three main categories: architectural, analytical, and socio-technical. For each source, we provide initial suggestions as to how to address the resulting opacity in practice. The taxonomy provides a starting point for requirements engineers and other practitioners to understand contextually prevalent sources of opacity, and to select or develop appropriate strategies for overcoming them.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some AI systems are opaque to a certain extent, due to proprietary nature, sub-symbolic representations, or complexity.
“We can say that some AI systems are, to a certain extent, opaque. Sometimes the opacity of a system is due merely to the fact that the system is proprietary, but other times the reason is that its model involves sub-symbolic representations incomprehensible even to experts, or that it is too large or too complex to grasp”
From The Epistemic Cost of Opacity: How the Use of Artificial Intelligence Undermines the Knowledge of Medical Doctors in High-Stakes Contexts (2025) · cites this paper as Mann et al., 2023
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,726 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · see citing works
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 3,432 · cited 1× · see citing works
“This is a common phenomenon in computer science, e.g., when programmers rely on programming languages developed others instead of dealing with machine language [19], or on external libraries they treat as black boxes [49].”
BackgroundHow the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 10× · see citing works
“Strategies: The complexity of a model or software can be reduced with methods like feature extraction or by following best practice standards of programming [12].”
BackgroundMethodInfluentialMitchell, M. (2009) · doi:10.1093/oso/9780195124415.001.0001 · Cited by 2,310 · see citing works
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · see citing works
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058 · see citing works
Ananny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · see citing works
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389 · see citing works
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Hardwig, J. (1985) · The Journal of Philosophy · doi:10.2307/2026523 · Cited by 937 · see citing works
Ladyman, J., Lambert, J., & Wiesner, K. (2013) · European Journal for Philosophy of Science · doi:10.1007/s13194-012-0056-8 · Cited by 846 · see citing works
A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods
Speith, T. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534639 · Cited by 377 · see citing works
Self-citationThe philosophical novelty of computer simulation methods
Humphreys, P. (2009) · Synthese · doi:10.1007/s11229-008-9435-2 · Cited by 362 · see citing works
Testimony and Epistemic Autonomy
Fricker, E. (2006) · doi:10.1093/acprof:oso/9780199276011.003.0011 · Cited by 307 · see citing works
Understanding from Machine Learning Models
Sullivan, E. (2019) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axz035 · Cited by 229 · see citing works
Miller, T., Howe, P., & Sonenberg, L. (2017) · arXiv · doi:10.48550/arxiv.1712.00547 · Cited by 218 · see citing works
Transparency in Complex Computational Systems
Creel, K. (2020) · Philosophy of Science · doi:10.1086/709729 · Cited by 195 · see citing works
Holism, entrenchment, and the future of climate model pluralism
Lenhard, J., & Winsberg, E. B. (2010) · Studies in History and Philosophy of Science Part B Studies in History and Philosophy of Modern Physics · doi:10.1016/j.shpsb.2010.07.001 · Cited by 194 · see citing works
The age of secrecy and unfairness in recidivism prediction
Rudin, C., Wang, C., & Coker, B. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.6ed64b30 · Cited by 177 · see citing works
Langer, M., & König, C. J. (2021) · Human Resource Management Review · doi:10.1016/j.hrmr.2021.100881 · Cited by 159 · see citing works
Exploring Explainability: A Definition, a Model, and a Knowledge Catalogue
Chazette, L., Brunotte, W., & Speith, T. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00025 · Cited by 120 · see citing works
Self-citationExplainability as a Non-Functional Requirement
Köhl, M. A., Baum, K. L., Langer, M., Oster, D., Speith, T., & Bohlender, D. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00046 · Cited by 117 · see citing works
Self-citationStrevens, M. (2003) · Harvard University Press eBooks · doi:10.4159/9780674044067 · Cited by 110 · see citing works
From Responsibility to Reason-Giving Explainable Artificial Intelligence
Baum, K., Mantel, S., Schmidt, E., & Speith, T. (2022) · Philosophy & Technology · doi:10.1007/s13347-022-00510-w · Cited by 103 · see citing works
Self-citationPrecise Documentation: The Key to Better Software
Parnas, D. L. (2010) · Springer eBooks · doi:10.1007/978-3-642-15187-3_8 · Cited by 102 · see citing works
Your Understanding Is My Understanding: Evidence for a Community of Knowledge
Sloman, S. A., & Rabb, N. (2016) · Psychological Science · doi:10.1177/0956797616662271 · Cited by 88 · see citing works
Programming Language Design Concepts
Watt, D. A. (2004) · Cited by 64 · see citing works
Requirements of Software Visualization Tools: A Literature Survey
Kienle, H. M., & Müller, H. A. (2007) · Visualizing Software for Understanding and Analysis · doi:10.1109/vissof.2007.4290693 · Cited by 60 · see citing works
Overcomplicated: Technology at the Limits of Comprehension
Konkani, A. (2017) · Biomedical Instrumentation & Technology · doi:10.2345/0899-8205-51.1.81 · Cited by 53 · see citing works
Clark, A. (1987) · Mind & Language · doi:10.1111/j.1468-0017.1987.tb00123.x · Cited by 48 · see citing works
Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI)
Crook, B., Schlüter, M., & Speith, T. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00060 · Cited by 41 · see citing works
Self-citationOPAQUE AND TRANSLUCENT EPISTEMIC DEPENDENCE IN COLLABORATIVE SCIENTIFIC PRACTICE
Wagenknecht, S. (2014) · Episteme · doi:10.1017/epi.2014.25 · Cited by 40 · see citing works
Towards a Taxonomy for the Opacity of AI Systems
Facchini, A., & Termine, A. (2022) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-031-09153-7_7 · Cited by 34 · see citing works
Big Data, Thick Mediation, and Representational Opacity
Alvarado, R., & Humphreys, P. (2017) · New Literary History · doi:10.1353/nlh.2017.0037 · Cited by 32 · see citing works
Opacity thought through: on the intransparency of computer simulations
Beisbart, C. (2021) · Synthese · doi:10.1007/s11229-021-03305-2 · Cited by 30 · see citing works
Computer Simulations and Computational Models in Science
Imbert, C. (2017) · Springer eBooks · doi:10.1007/978-3-319-30526-4_34 · Cited by 29 · see citing works
Schubert, L., Voss, C., & Olah, C. (2021) · Distill · doi:10.23915/distill.00024.005 · Cited by 16 · see citing works
Contemporary COBOL: Developers' Perspectives on Defects and Defect Location
Ciborowska, A., Chakarov, A., & Pandita, R. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00027 · Cited by 13 · see citing works
A machine learning view on profi ling
Brunton, F., & Nissenbaum, H. F. (2013) · Routledge eBooks · doi:10.4324/9780203427644-11 · Cited by 12 · see citing works
Explaining with Simulations: Why Visual Representations Matter
Jebeile, J. (2018) · Perspectives on Science · doi:10.1162/posc_a_00273 · Cited by 12 · see citing works
Mathematische Opazität. Über Rechtfertigung und Reproduzierbarkeit in der Computersimulation
Kaminski, A., Resch, M., & Uwe Küster, A. (2018) · Nomos Verlagsgesellschaft mbH & Co. KG eBooks · doi:10.5771/9783845285429-253 · Cited by 9 · see citing works
Davidsson, P., Klügl, F., & Verhagen, H. (2017) · Springer eBooks · doi:10.1007/978-3-319-30526-4_35 · Cited by 8 · see citing works
Explaining Epistemic OpacityPreprint
Alvarado, R. (2021) · PhilSci-Archive (University of Pittsburgh) · Cited by 6 · see citing works
Bringing COBOL back into the college IT curriculum
Lindoo, E. (2014) · Journal of computing sciences in colleges · Cited by 4 · see citing works
Jebeile, J. (2018) · Philosophia Scientae · doi:10.4000/philosophiascientiae.1483 · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 53 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Revisiting the Performance-Explainability Trade-Off in Explainable Artificial Intelligence (XAI)
Crook, B., Schlüter, M., & Speith, T. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00060 · Cited by 41 · see citing works
Self-citationSpeith, T., Crook, B., Mann, S., Schomäcker, A., & Langer, M. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09769-3 · Cited by 23 · see citing works
Self-citationSchmidt, E., Putora, P. M., & Fijten, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-024-00834-9 · Cited by 14 · see citing works
“…of a system is due merely to the fact that the system is proprietary, but other times the reason is that its model involves sub-symbolic representations incomprehensible even to experts, or that it is too large or too complex to grasp (Bathaee, 2018 ; Ghassemi et al., 2021 ; Mann et al., 2023 ).”
Explainability as a Requirement for Hardware: Introducing Explainable Hardware (XHW)
Speith, T., Speith, J., Becker, S., Zou, Y., Biega, A. J., & Paar, C. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00042 · Cited by 5 · see citing works
Self-citationModeling and Reasoning About Explanation Requirements Using Goal Models
Liaskos, S., Mylopoulos, J., Borgida, A. T., & Khan, S. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75872-0_12 · Cited by 4 · see citing works
Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence
Queloz, M. (2025) · Minds and Machines · doi:10.1007/s11023-025-09738-9 · Cited by 2 · see citing works
Mann, S. (2024) · AI & Society · doi:10.1007/s00146-023-01837-4 · Cited by 2 · see citing works
Self-citationResponsibility Attribution for AI-Mediated Damages with Mechanistic Interpretability
Kästner, L., Cordes, J., & Zech, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01377-4_10 · Cited by 1 · see citing works
Self-citationCrook, B., & Kästner, L. (2025) · Synthese Library/Synthese library · doi:10.1007/978-3-032-03083-2_20
Self-citationExperiencing and Engineering Robotic Art A Three-Axis Framework [Arts and Robotics]
Aktaş, B., & RUŞEN, F. (2026) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2026.3712363
- Authors
- Mann, S., Crook, B., Kästner, L., Schomäcker, A., & Speith, T.
- Affiliations
- TU Dortmund University (DE)
- University of Bayreuth (DE)
- Venue
- 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW)
- Publisher
- IEEE
- Published
- 2023-09
- Pages
- 337-342
- OpenAlex
- W4387123667
- Cited by
- 10
- References
- 53
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 4 · 2023: 1
- FWCI
- 1.77
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Requirements Engineering Conference Workshops (REW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Volkswagen Foundation (AZ 9B830,AZ 98510,AZ 98514)
- DFG (389792660)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- opacity, system transparency, taxonomy (general classification scheme), fairness, accountability, requirements engineering, computer systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
