Rethinking the Maturity of Artificial Intelligence in Safety‐Critical Settings
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Artificial intelligence, in the form of machine learning, has the potential to transform many safety‐critical applications such as those in transportation and healthcare. However, despite significant investment and impressive demonstrations, such technologies have struggled to live up to their promises. To this end, this article illustrates that machine learning fundamentally lacks the ability to leverage top‐down reasoning, a critical element in safety‐critical systems. This is especially important in situations where uncertainty can grow very quickly, requiring adaption to unknowns. This fundamental lack of contextual reasoning, combined with a lack of understanding of what constitutes maturity in artificial intelligence‐embedded systems, has significantly contributed to the failures of these systems. Demonstrations where safety‐critical artificial intelligence‐enabled systems function as if they were almost operational should not be a substitute for testing. Instead, companies and regulatory agencies need to work together to develop clear criteria and certification protocols before such technologies are made publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rasmussen, J. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313160 · Cited by 3,434
One Pixel Attack for Fooling Deep Neural Networks
Su, J., Vargas, D. V., & Sakurai, K. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2890858 · Cited by 2,406
Robust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167 · cited 1×
“Another major area where AI has been heralded as a success is in healthcare settings, including drug discovery (Morrison 2019) and radiology (Ardila et al. 2019; Park et al. 2019).”
BackgroundZang, S., Ding, M., Smith, D., Tyler, P., Rakotoarivelo, T., & Kaafar, M. A. (2019) · IEEE Vehicular Technology Magazine · doi:10.1109/mvt.2019.2892497 · Cited by 457
IBM Watson, heal thyself: How IBM overpromised and underdelivered on AI health care
Strickland, E. (2019) · IEEE Spectrum · doi:10.1109/mspec.2019.8678513 · Cited by 402 · cited 2×
“IBM’s Watson, the decisionmaking engine behind the Jeopardy! AI success, has been deemed a costly and potentially deadly failure when extended to medical applications such as cancer diagnosis (Strickland 2019).”
BackgroundDeep Learning–Assisted Diagnosis of Cerebral Aneurysms Using the HeadXNet Model
Park, A., Chute, C., Rajpurkar, P., Lou, J., Ball, R. L., Shpanskaya, K., Jabarkheel, R., Kim, L. H., McKenna, E., Tseng, J., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.5600 · Cited by 258
Man versus Machine or Man + Machine?
Cummings, M. M. (2014) · IEEE Intelligent Systems · doi:10.1109/mis.2014.87 · Cited by 146
Self-citation
Cited by
The uncontroversial ‘thingness’ of AI
Suchman, L. (2023) · Big Data & Society · doi:10.1177/20539517231206794 · Cited by 229
“Noting this, Cummings (2021) underscores what she names a ‘ fake-it-til-you-make-it ’ culture pervasive among technology vendors and promoters.”
BackgroundInfluentialArtificial intelligence in safety-critical systems: a systematic review
Wang, Y., & Chung, S. H. (2022) · Industrial Management & Data Systems · doi:10.1108/imds-07-2021-0419 · Cited by 96
Macrae, C. (2022) · Risk Analysis · doi:10.1111/risa.13850 · Cited by 76
Metadata
- Authors
- Cummings, M. L.
- Venue
- AI Magazine
- Publisher
- Wiley
- Published
- 2021-03
- Volume
- 42
- Issue
- 1
- Pages
- 6-15
- ISSN
- 0738-4602 (ISSN-L)
- 2371-9621
- OpenAlex
- W3165581183
- Cited by
- 73
- References
- 8
- FWCI
- 7.97
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Rethinking the Maturity of Artificial Intelligence in Safety‐Critical Settings. https://app.cochat.ai/papers/10.1002/j.2371-9621.2021.tb00005.x
- The paper
- Cummings, M. L. (2021). Rethinking the Maturity of Artificial Intelligence in Safety‐Critical Settings. AI Magazine. https://doi.org/10.1002/j.2371-9621.2021.tb00005.x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.