Sociotechnical Envelopment of Artificial Intelligence: An Approach to Organizational Deployment of Inscrutable Artificial Intelligence Systems
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
The paper presents an approach for implementing inscrutable (i.e., nonexplainable) artificial intelligence (AI) such as neural networks in an accountable and safe manner in organizational settings. Drawing on an exploratory case study and the recently proposed concept of envelopment, it describes a case of an organization successfully “enveloping” its AI solutions to balance the performance benefits of flexible AI models with the risks that inscrutable models can entail. The authors present several envelopment methods—establishing clear boundaries within which the AI is to interact with its surroundings, choosing and curating the training data well, and appropriately managing input and output sources—alongside their influence on the choice of AI models within the organization. This work makes two key contributions: It introduces the concept of sociotechnical envelopment by demonstrating the ways in which an organization’s successful AI envelopment depends on the interaction of social and technical factors, thus extending the literature’s focus beyond mere technical issues. Secondly, the empirical examples illustrate how operationalizing a sociotechnical envelopment enables an organization to manage the trade-off between low explainability and high performance presented by inscrutable models. These contributions pave the way for more responsible, accountable AI implementations in organizations, whereby humans can gain better control of even inscrutable machine-learning models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Seeking Qualitative Rigor in Inductive Research: Notes on the Gioia Methodology
Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013) · Organizational Research Methods · doi:10.1177/1094428112452151 · Cited by 13,227
Constructing Grounded Theory. A Practical Guide Through Qualitative Analysis
(2006) · QMiP Bulletin · doi:10.53841/bpsqmip.2006.1.2.36 · Cited by 11,828
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“This assertion stems from the so-called Rashomon set argument (Rudin, 2019), under which there is a good chance that at least one of the acceptable models is interpretable yet still accurate.”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems1
Klein, H. K., & Myers, M. D. (1999) · MIS Quarterly · doi:10.2307/249410 · Cited by 5,263
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796
Digital Twin in manufacturing: A categorical literature review and classification
Kritzinger, W., Karner, M., Traar, G., Henjes, J., & Sihn, W. (2018) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2018.08.474 · Cited by 3,612 · cited 1×
“approach that simplifies envelopment is to strive for “gray-box models,” as exemplified by the creation of “digital twins” that can simulate real, physical processes (see El Saddik, 2018; Kritzinger et al., 2018).”
BackgroundInternational evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · cited 1×
“analysis and decision-making situations (McKinney et al., 2020), their operation logic differs dramatically from humans’ ways of approaching similar problems.”
BackgroundHow the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 3×
“Inscrutability takes many forms, linked to such elements as intentional corporate or state secrecy, technical illiteracy, and innate characteristics of ML models (Burrell, 2016).”
BackgroundKaplan, A., & Haenlein, M. (2019) · Business Horizons · doi:10.1016/j.bushor.2018.08.004 · Cited by 3,140
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making
Jarrahi, M. H. (2018) · Business Horizons · doi:10.1016/j.bushor.2018.03.007 · Cited by 2,772
The qualitative interview in IS research: Examining the craft
Myers, M. D., & Newman, M. (2007) · Information and Organization · doi:10.1016/j.infoandorg.2006.11.001 · Cited by 1,941
Explainable artificial intelligence: A survey
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400040 · Cited by 1,147
Working and organizing in the age of the learning algorithm
Faraj, S., Pachidi, S., & Sayegh, K. (2018) · Information and Organization · doi:10.1016/j.infoandorg.2018.02.005 · Cited by 929 · cited 1×
“The recent emergence and proliferation of new generations of ML tools have reawakened interest in organizational AI research (Faraj et al. 2018; Keding 2021; Sousa et al. 2019).”
BackgroundDigital Twins: The Convergence of Multimedia Technologies
El Saddik, A. (2018) · IEEE MultiMedia · doi:10.1109/mmul.2018.023121167 · Cited by 817
Artificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“These challenges have prompted calls for human-centered and pragmatic approaches to explainability (Mittelstadt et al., 2019; Ribera & Lapedriza, 2019).”
BackgroundThe WyCash portfolio management system
Cunningham, W. (1992) · Addendum to the proceedings on Object-oriented programming systems, languages, and applications (Addendum) - OOPSLA '92 · doi:10.1145/157709.157715 · Cited by 742
Technical Debt: From Metaphor to Theory and Practice
Kruchten, P., Nord, R. L., & Ozkaya, I. (2012) · IEEE Software · doi:10.1109/ms.2012.167 · Cited by 735
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · cited 1×
“6 billion free parameters in its deep neural network (Adiwardana et al., 2020), is an extreme example of an AI system whose inner workings are inscrutable for humans even if they are transparent.”
BackgroundExplanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
Gregor, S., & Benbasat, I. (1999) · MIS Quarterly · doi:10.2307/249487 · Cited by 535
Edwards, L., & Veale, M. (2017) · doi:10.31228/osf.io/97upg · Cited by 524
Sousa, W. G. D., Melo, E. R. P. D., Bermejo, P. H. D. S., Farias, R. A. S., & Gomes, A. O. (2019) · Government Information Quarterly · doi:10.1016/j.giq.2019.07.004 · Cited by 501
Newell, S., & Marabelli, M. (2015) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2015.02.001 · Cited by 494
Algorithms and their others: Algorithmic culture in context
Dourish, P. (2016) · Big Data & Society · doi:10.1177/2053951716665128 · Cited by 481 · cited 1×
“Then, one deems them inscrutable (Dourish, 2016; Martin, 2019), referring to situations wherein the system’s complexity outstrips practical means of analyzing it comprehensively.”
BackgroundLindebaum, D., Vesa, M., & den Hond, F. (2020) · Academy of Management Review · doi:10.5465/amr.2018.0181 · Cited by 290
Slave to the Algorithm? Why a Right to Explanationn is Probably Not the Remedy You are Looking for
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2972855 · Cited by 244
Explanation in Case-Based Reasoning—Perspectives and Goals
Sørmo, F., Cassens, J., & Aamodt, A. (2005) · Artificial Intelligence Review · doi:10.1007/s10462-005-4607-7 · Cited by 239
Keding, C. (2021) · Management Review Quarterly · doi:10.1007/s11301-020-00181-x · Cited by 231 · cited 3×
“Studies indicate that AI is rarely considered a “plug-and-play” technology and that an organization deploying it requires a clear implementation strategy that takes into account the wide spectrum of stakeholders (Keding, 2021).”
BackgroundArtificial intelligence as digital agency
Ågerfalk, P. J. (2020) · European Journal of Information Systems · doi:10.1080/0960085x.2020.1721947 · Cited by 217 · cited 1×
“The papers have increasingly shifted the position of AI research from a largely technical one to a perspective encompassing the social component (Ågerfalk, 2020).”
BackgroundBureaucracy as a Lens for Analyzing and Designing Algorithmic Systems
Pääkkönen, J., Nelimarkka, M., Haapoja, J., & Lampinen, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376780 · Cited by 35 · cited 2×
“These considerations also show that organizational AI deployment entails a significant amount of coordination and mutual adaptation between humans and AI and is thus inescapably a matter of sociotechnical organization design (Pääkkönen et al., 2020).”
Background
Cited by
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
“behavior of humans and AI systems to enable effective and efficient use in AI applications on the one hand and, on the other hand, to ensure envelopment of AI agency and reach (Asatiani et al., 2021).”
Managing artificial intelligence
Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16274 · Cited by 898
Keding, C., & Meissner, P. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2021.120970 · Cited by 168
How to explain AI systems to end users: a systematic literature review and research agenda
Laato, S., Tiainen, M., Najmul Islam, A. K. M., & Mäntymäki, M. (2022) · Internet Research · doi:10.1108/intr-08-2021-0600 · Cited by 155
Designing a feature selection method based on explainable artificial intelligence
Zacharias, J., von Zahn, M., Chen, J., & Hinz, O. (2022) · Electronic Markets · doi:10.1007/s12525-022-00608-1 · Cited by 109
“Such metrics steer the development process and facilitate the objective comparison of competing classifiers (Asatiani et al., 2021).”
Background
Metadata
- Authors
- Asatiani, A., University of Gothenburg, S., Malo, P., Aalto University School of Business, F., Nagbøl, P. R., IT University of Copenhagen, D., Penttinen, E., Aalto University School of Business, F., Rinta-Kahila, T., The University of Queensland, A., Salovaara, A., & Aalto University School of Arts, D. A. A.
- Venue
- Journal of the Association for Information Systems
- Publisher
- Association for Information Systems
- Published
- 2021
- Volume
- 22
- Issue
- 2
- Pages
- 325-352
- ISSN
- 1536-9323 (ISSN-L)
- OpenAlex
- W3142698583
- Cited by
- 173
- References
- 33
- FWCI
- 8.91
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.