Two Dimensions of Opacity and the Deep Learning Predicament
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Deep neural networks (DNNs) have become increasingly successful in applications from biology to cosmology to social science. Trained DNNs, moreover, correspond to models that ideally allow the prediction of new phenomena. Building in part on the literature on ‘eXplainable AI’ (XAI), I here argue that these models are instrumental in a sense that makes them non-explanatory, and that their automated generation is opaque in a unique way. This combination implies the possibility of an unprecedented gap between discovery and explanation: When unsupervised models are successfully used in exploratory contexts, scientists face a whole new challenge in forming the concepts required for understanding underlying mechanisms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 4×
“Given the close connection between understanding and explanation (de Regt, 2017; Grimm, 2010; Khalifa, 2017; Rudin, 2019; Strevens, 2008), scientists’ interest in what is usually called ‘eXplainable AI’ (XAI) should be rather high accordingly.”
BackgroundResultInfluentialExplaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“Deep neural networks (DNNs) are currently being praised for their astonishing utility in applications ranging “from biology to cosmology to social science” (Jordan & Mitchell, 2015, p. 255).”
BackgroundDeep learning and process understanding for data-driven Earth system science
Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., & Prabhat (2019) · Nature · doi:10.1038/s41586-019-0912-1 · Cited by 5,673 · cited 3×
“(Reichstein et al., 2019, p. 200; emph. added) Note the crucial role of w-opacity here: It is exactly the ability of DNNs to ‘auto-matically discover’ important, complex features that further prediction and, at the same time, provide a basis for understanding underlying mechanisms (such as the…”
BackgroundInfluentialDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Falsification and the Methodology of Scientific Research Programmes
Lakatos, I. (1970) · Criticism and the Growth of Knowledge · doi:10.1017/cbo9781139171434.009 · Cited by 4,649
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 3×
“However, there is also a major commonality : Both CSs and DL have been recognized for their opacity (cf., in particular, Burrell, 2016; Humphreys, 2009).”
BackgroundStatistics versus machine learning
Bzdok, D., Altman, N., & Krzywinski, M. (2018) · Nature Methods · doi:10.1038/nmeth.4642 · Cited by 1,508 · cited 3×
“Only then is the chosen architecture used to “ find patterns in often rich and unwieldy data” (Bzdok et al., 2018, p. 233; emph. added).”
BackgroundMethodInfluential
Research works printed but not linked: 90
- Albertsson, K., Altoe, P., Anderson, D., Andrews, M., Espinosa, J. P. A., Aurisano, A., Basara, L., Bevan, A., Bhimji, W., Bonacorsi, D., Calafiura, P., Campanelli, M., Capps, L., Carminati, F., Carrazza, S., Childers, T., Coniavitis, E., Cranmer, K., David, C., … Zapata, O. (2018). Machine learning in high energy physics community white paper. Journal of Physics: Conference Series, 1085(2), 022008.
- Alvarez Melis, D., & Jaakkola, T. (2018). Towards robust interpretability with self-explaining neural networks. Advances in Neural Information Processing Systems, 31, 7775–7784.
- Baldi, P., Sadowski, P., & Whiteson, D. (2014). Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 5, 4308. · DOI · Paper page
- Basso, A., Lisciandra, C., & Marchionni, C. (2017). Hypothetical models in social science. In L. Magnani & T. Bertolotti (Eds.), Springer handbook of model-based science (pp. 413–433). Springer. · DOI
- Beisbart, C. (2012). How can computer simulations produce new knowledge? European Journal for Philosophy of Science, 2(3), 395–434. · DOI
- Boge, F. J. (2019a). How to infer explanations from computer simulations. Studies in History and Philosophy of Science Part A. https://doi.org/10.1016/j.shpsa.2019.12.003 · DOI
- Boge, F. J. (2019b). Why computer simulations are not inferences, and in what sense they are experiments. European Journal for Philosophy of Science, 9(1), 13. · DOI
- Boge, F. J., & Zeitnitz, C. (2020). Polycratic hierarchies and networks: What simulation-modeling at the LHC can teach us about the epistemology of simulation. Synthese. https://doi.org/10.1007/s11229-020-02667-3 · DOI
- Bokulich, A. (2011). How scientific models can explain. Synthese, 180(1), 33–45. · DOI · Paper page
- Bokulich, A. (2014). How the tiger bush got its stripes: ‘How possibly’ vs. ‘how actually’ model explanations’. The Monist, 97(3), 321–338. · doi:10.5840/monist201497321
- Buckner, C. (2019). Deep learning: A philosophical introduction. Philosophy Compass, 14(10), e12625. · DOI · Paper page
- Buckner, C. (2020). Understanding adversarial examples requires a theory of artefacts for deep learning. Nature Machine Intelligence, 2(12), 731–736. · DOI
- Buckner, C. J. (2021). Black boxes, or unflattering mirrors? Comparative bias in the science of machine behavior. The British Journal for the Philosophy of Science. https://doi.org/10.1086/714960 · DOI
- Card, D., Mas, A., & Rothstein, J. (2008). Tipping and the dynamics of segregation*. The Quarterly Journal of Economics, 123(1), 177–218. · DOI · Paper page
- Chang, S., Cohen, T., & Ostdiek, B. (2018). What is the machine learning? Physical Review D, 97(5), 6.
- Chattopadhyay, A., Hassanzadeh, P., & Pasha, S. (2020). Predicting clustered weather patterns: A test case for applications of convolutional neural networks to spatio-temporal climate data. Scientific Reports, 10(1), 1–13. · DOI · Paper page
- Chirimuuta, M. (2020). Prediction versus understanding in computationally enhanced neuroscience. Synthese. https://doi.org/10.1007/s11229-020-02713-0 · DOI
- Cho, A. (2017). Ai’s early proving ground: The hunt for new particles. Science, 357(6346), 20. · doi:10.1126/science.357.6346.20
- Clark, W. A. (1991). Residential preferences and neighborhood racial segregation: A test of the schelling segregation model. Demography, 28(1), 1–19. · DOI · Paper page
- Craver, C. F. (2006). When mechanistic models explain. Synthese, 153(3), 355–376. · DOI · Paper page
- Creel, K. A. (2020). Transparency in complex computational systems. Philosophy of Science, 87(4), 568–589. · DOI · Paper page
- Davies, P. L. (2014). Data analysis and approximate models. CRC Press. · doi:10.1201/b17146
- de Regt, H. (2017). Understanding scientific understanding. Oxford University Press. · DOI · Paper page
- Douglas, H. E. (2009). Reintroducing prediction to explanation. Philosophy of Science, 76(4), 444–463. · DOI · Paper page
- Duncan, A., & Janssen, M. (2019). Constructing quantum mechanics (Vol. 1). Oxford University Press. · DOI
- Durán, J. M. (2017). Varying the explanatory span: Scientific explanation for computer simulations. International Studies in the Philosophy of Science, 31(1), 27–45. · DOI
- Durán, J. M. (2018). Computer simulations in science and engineering. Springer Nature. · DOI · Paper page
- Durán, J. M. (2020). What is a simulation model? Minds and Machines, 30(3), 301–323. · DOI
- Farina, M., Nakai, Y., & Shih, D. (2020). Searching for new physics with deep autoencoders. Physical Review D, 101(7), 075021. · DOI · Paper page
- Franklin, L. R. (2005). Exploratory experiments. Philosophy of Science, 72(5), 888–899. · DOI · Paper page
- Frigg, R., & Reiss, J. (2009). The philosophy of simulation: Hot new issues or same old stew? Synthese, 169(3), 593–613. · DOI
- Giere, R. (2006). Scientific perspectivism. University of Chicago Press. · DOI · Paper page
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. The MIT Press.
- Grimm, S. R. (2010). The goal of explanation. Studies in History and Philosophy of Science Part A, 41(4), 337–344. · DOI
- Grüne-Yanoff, T. (2013). Appraising models nonrepresentationally. Philosophy of Science, 80(5), 850–861. · DOI
- Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM Computing Surveys, 51(5), 1–42. · DOI · Paper page
- Hacking, I. (1992). The self-vindication of the laboratory sciences. In A. Pickering (Ed.), Science as practice and culture (pp. 29–64). The University of Chicago Press.
- Harris, T. (2003). Data models and the acquisition and manipulation of data. Philosophy of Science, 70(5), 1508–1517. · doi:10.1086/377426
- Hasse, H., & Lenhard, J. (2017). Boon and bane: On the role of adjustable parameters in simulation models. In J. Lenhard & M. Carrier (Eds.), Mathematics as a tool (pp. 93–116). Springer. · DOI
- Hastie, T., Tibshirani, R., & Friedman, J. (2013). The elements of statistical learning: Data mining, inference, and prediction. Springer.
- Hornik, K., Stinchcombe, M., White, H., et al. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359–366. · DOI · Paper page
- Humphreys, P. (2004). Extending ourselves: Computational science, empiricism, and scientific method. Oxford University Press. · DOI · Paper page
- Humphreys, P. (2013). Data analysis: Models or techniques? Foundations of Science, 18(3), 579–581. · doi:10.1007/s10699-012-9317-4
- Karaca, K. (2013). The strong and weak senses of theory-ladenness of experimentation. Science in Context, 26(01), 93–136. · DOI
- Karaca, K. (2018). Lessons from the large hadron collider for model-based experimentation. Synthese, 195(12), 5431–5452. · DOI
- Kasabov, N. (2019). Time-space, spiking neural networks and brain-inspired artificial intelligence. Springer. · doi:10.1007/978-3-662-57715-8
- Khalifa, K. (2017). Understanding, explanation, and scientific knowledge. Cambridge University Press. · DOI · Paper page
- Lenhard, J. (2006). Surprised by a nanowire: Simulation, control, and understanding. Philosophy of Science, 73(5), 605–616. · doi:10.1086/518330
- Lenhard, J. (2007). Computer simulation: The cooperation between experimenting and modeling. Philosophy of Science, 74(2), 176–194. · DOI
- Lenhard, J. (2019). Calculated surprises. Oxford University Press. · DOI
- Lenhard, J., & Winsberg, E. (2010). Holism, entrenchment, and the future of climate model pluralism. Studies in History and Philosophy of Modern Physics, 41(3), 253–262. · DOI · Paper page
- Leonelli, S. (2016). Data-centric biology. University of Chicago Press. · DOI · Paper page
- Leonelli, S. (2019). What distinguishes data from models? European Journal for Philosophy of Science, 9(2), 22. · DOI
- López-Rubio, E. (2020). Throwing light on black boxes: Emergence of visual categories from deep learning. Synthese. https://doi.org/10.1007/s11229-020-02700-5 · DOI
- Lyre, H. (2008). Does the Higgs mechanism exist? International Studies in the Philosophy of Science, 22(2), 119–133. · doi:10.1080/02698590802496664
- Lyre, H. (2020). The state space of artificial intelligence. Minds and Machines. https://doi.org/10.1007/s11023-020-09538-3 · DOI
- Maher, P. (1988). Prediction, accommodation, and the logic of discovery. Philosophy of Science, 1988(1), 273–285.
- Massimi, M. (2005). Pauli’s exclusion principle: The origin and validation of a scientific principle. Cambridge University Press. · DOI
- McCrea, R. (2009). Explaining sociospatial patterns in South East Queensland, Australia. Economy and Space, 41(9), 2201–2214.
- Minsky, M., & Papert, S. A. (1969). Perceptrons: An introduction to computational geometry. MIT Press.
- Morello, V., Barr, E., Bailes, M., Flynn, C., Keane, E., & van Straten, W. (2014). Spinn: A straightforward machine learning solution to the pulsar candidate selection problem. Monthly Notices of the Royal Astronomical Society, 443(2), 1651–1662. · DOI
- Morrison, M. (1999). Models as autonomous agents. In M. Morrison & M. S. Morgan (Eds.), Models as mediators (pp. 38–65). Cambridge University Press. · DOI · Paper page
- Morrison, M. (2015). Reconstructing reality: Models, mathematics, and simulations. Oxford University Press. · DOI · Paper page
- Morrison, M., & Morgan, M. S. (1999). Models as mediating instruments. In M. Morrison & M. S. Morgan (Eds.), Models as mediators (pp. 10–37). Cambridge University Press. · DOI · Paper page
- Napoletani, D., Panza, M., & Struppa, D. C. (2011). Agnostic science. Towards a philosophy of data analysis. Foundations of Science, 16(1), 1–20. · DOI
- Patterson, J., & Gibson, A. (2017). Deep learning: A practitioner’s approach. O’Reilly Media.
- Poggio, T., Banburski, A., & Liao, Q. (2020). Theoretical issues in deep networks. Proceedings of the National Academy of Sciences, 117(48), 30039–30045. · DOI · Paper page
- Potochnik, A. (2017). Idealization and the aims of science. University of Chicago Press. · DOI · Paper page
- Raghu, M., & Schmidt, E. (2020). A survey of deep learning for scientific discovery. arXiv:2003.11755 (arXiv preprint). · arXiv
- Reutlinger, A., Hangleiter, D., & Hartmann, S. (2018). Understanding (with) toy models. The British Journal for the Philosophy of Science, 69(4), 1069–1099. · DOI
- Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019). On the information bottleneck theory of deep learning. Journal of Statistical Mechanics, 2019(12), 124020. · DOI · Paper page
- Schelling, T. C. (1971). Dynamic models of segregation. Journal of Mathematical Sociology, 1(2), 143–186. · DOI · Paper page
- Schwartz-Ziv, R., & Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv:1703.00810 (arXiv preprint). · arXiv
- Skansi, S. (2018). Introduction to deep learning: From logical calculus to artificial intelligence. Springer International Publishing. · DOI · Paper page
- Smeenk, C. (2006). The elusive Higgs mechanism. Philosophy of Science, 73(5), 487–499. · DOI
- Sober, E. (2002). Instrumentalism, parsimony, and the Akaike framework. Philosophy of Science, 69(S3), S112–S123. · DOI · Paper page
- Steinle, F. (1997). Entering new fields: Exploratory uses of experimentation. Philosophy of Science, 64, S65–S74. · DOI · Paper page
- Steinle, F. (2016). Exploratory experiments: Ampère, Faraday, and the origins of electrodynamics. University of Pittsburgh Press. · doi:10.2307/j.ctt1f89r1p
- Strevens, M. (2008). Depth: An account of scientific explanation. Harvard University Press.
- Strevens, M. (2013). No understanding without explanation. Studies in History and Philosophy of Science Part A, 44(3), 510–515. · DOI · Paper page
- Suárez, M., & Cartwright, N. (2008). Theories: Tools versus models. Studies in History and Philosophy of Modern Physics, 39(1), 62–81. · DOI
- Sullivan, E. (2019). Understanding from machine learning models. The British Journal for the Philosophy of Science. https://doi.org/10.1093/bjps/axz035 · DOI · Paper page
- Suthaharan, S. (2016). Machine learning models and algorithms for big data classification. Springer. · DOI · Paper page
- Voss, H. (2013). Classification. In O. Behnke, K. Kröninger, G. Schott, & T. Schörner-Sadenius (Eds.), Data analysis in high energy physics: A practical guide to statistical methods (pp. 153–186). Wiley. · DOI
- Wallace, D. (2020). On the plurality of quantum theories. In S. French & J. Saatsi (Eds.), Scientific realism and the quantum (pp. 78–102). Oxford University Press. · DOI
- Wilholt, T. (2005). Explaining models: Theoretical and phenomenological models and their role for the first explanation of the hydrogen spectrum. Foundations of Chemistry, 7(2), 149–169. · DOI
- Winsberg, E. (1999). The hierarchy of models in simulation. In L. Magnani, N. Nersessian, & P. Thagard (Eds.), Model-based reasoning in scientific discovery (pp. 255–269). Kluwer Academic/Plenum Publishers. · doi:10.1007/978-1-4615-4813-3_16
- Winsberg, E. (2010). Science in the age of computer simulation. University of Chicago Press. · DOI · Paper page
- Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82. · DOI · Paper page
- Worrall, J. (1985). Scientific discovery and theory-confirmation. In J. C. Pitt (Ed.), Change and progress in modern science (pp. 301–331). Springer. · DOI
Cited by
Alvarado, R. (2023) · Science and Engineering Ethics · doi:10.1007/s11948-023-00451-3 · Cited by 137
Beyond generalization: a theory of robustness in machine learning
Freiesleben, T., & Grote, T. (2023) · Synthese · doi:10.1007/s11229-023-04334-9 · Cited by 106
“…Physics or climate science, the challenge is to determine when we can trust a simulation model to generate credible experimental results and under what conditions we are permitted to infer that a CS model faithfully represents its target system (Boge, 2021; Durán & Formanek, 2018; Karaca, 2022).”
BackgroundMethodInfluentialFerrario, A., Facchini, A., & Termine, A. (2024) · Minds and Machines · doi:10.1007/s11023-024-09681-1 · Cited by 64
“22 See, for instance, (Creel, 2020; Facchini & Termine, 2022; Boge, 2022). individuals (Hollan et al., 2000).”
Trust and Trustworthiness in AI
Durán, J. M., & Pozzi, G. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00843-2 · Cited by 47
Towards a Taxonomy for the Opacity of AI Systems
Facchini, A., & Termine, A. (2022) · Studies in Applied Philosophy, Epistemology and Rational Ethics · doi:10.1007/978-3-031-09153-7_7 · Cited by 34
Metadata
- Authors
- Boge, F. J.
- Venue
- Minds and Machines
- Published
- 2021-09-03
- Volume
- 32
- Issue
- 1
- Pages
- 43-75
- ISSN
- 0924-6495 (ISSN-L)
- 1572-8641
- OpenAlex
- W3197302864
- Cited by
- 110
- References
- 14
- FWCI
- 4.32
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Deutsche Forschungsgemeinschaft (FOR 2063)
- Bergische Universität Wuppertal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Two Dimensions of Opacity and the Deep Learning Predicament. https://app.cochat.ai/papers/10.1007/s11023-021-09569-4
- The paper
- Boge, F. J. (2022). Two Dimensions of Opacity and the Deep Learning Predicament. Minds and Machines. https://doi.org/10.1007/s11023-021-09569-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.