Developing artificial neural networks for safety critical systems

Kurd, Zeshan; Kelly, Tim; Austin, Jim

doi:10.1007/s00521-006-0039-9

Cited by 55Safety Systems Engineering in AutonomyExplainable Artificial Intelligence (XAI)Adversarial Robustness in Machine Learning

Developing artificial neural networks for safety critical systems

Kurd, Zeshan, Kelly, Tim, Austin, Jim·2006·Neural Computing and Applications

DOI 10.1007/s00521-006-0039-9

Open this paper in CoChat — free account

Integrity signals

Registry

Crossref · 10.1007/s00521-006-0039-9

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Specific fault types and failures have been cataloged for neural networks.
“Specific fault types and failures have been cataloged for neural networks in [139], [140] and [141].”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [139]
Troubleshooting issues of ML applications such as fault characterization, detection and elimination are a software quality challenge.
“-Fault Analysis: Trouble shooting issues of ML applications such as fault characterization, detection and elimination [10,22,36,37,122] (see also 4.5).”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [22]

Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
IEC 61508: functional safety of electrical/electronic/ programme electronic safety-related systems: overview
Bell, R. (1999) · Cited by 1,579 · see citing works
Survey and critique of techniques for extracting rules from trained artificial neural networks
Andrews, R., Diederich, J., & Tickle, A. B. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(96)81920-4 · Cited by 1,225 · see citing works
Knowledge-based artificial neural networks
Towell, G. G., & SHAVLIK, J. W. (1994) · Artificial Intelligence · doi:10.1016/0004-3702(94)90105-8 · Cited by 751 · see citing works
Safety-critical computer systems
Storey, N. (1996) · International Linear Collider · Cited by 605 · see citing works
Arguing Safety - A Systematic Approach to Managing Safety Cases
Kelly, T. (1998) · Cited by 378 · see citing works
Self-citation
A Systematic Approach to Safety Case Maintenance
Kelly, T., & McDermid, J. A. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48249-0_2 · Cited by 114 · see citing works
Self-citation
A Framework for Combining Symbolic and Neural Learning
SHAVLIK, J. W. (1992) · Cited by 48 · see citing works
Combining neural networks and fuzzy controllers
Nauck, D. D., Klawonn, F., & Kruse, R. (1993) · Lecture notes in computer science · doi:10.1007/3-540-56920-0_6 · Cited by 48 · see citing works
Adaptive guidance and control for autonomous launch vehicles
Johnson, E. N., Calise, A. J., & Corban, J. E. (2002) · Proceedings - IEEE Aerospace Conference · doi:10.1109/aero.2001.931288 · Cited by 36 · see citing works
A software development process model for artificial neural networks in critical applications
Rodvold, D. M. (2003) · IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339) · doi:10.1109/ijcnn.1999.836192 · Cited by 34 · see citing works
Establishing Safety Criteria for Artificial Neural Networks
Kurd, Z., & Kelly, T. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45224-9_24 · Cited by 30 · see citing works
Self-citation
Neural nets tell why
Klimasauskas, C. C. (1991) · Dr. Dobb's Journal archive · Cited by 29 · see citing works
Risk and system integrity concepts for safety-related control systems
Bell, R., & Reinert, D. (1993) · Microprocessors and Microsystems · doi:10.1016/0141-9331(93)90088-o · Cited by 25 · see citing works
Neural Nets and Diversity
SHARKEY, A. J. C., SHARKEY, N. E., & Chandroth, G. O. (1995) · Springer eBooks · doi:10.1007/978-1-4471-3054-3_25 · Cited by 21 · see citing works
Safety Lifecycle for Developing Safety Critical Artificial Neural Networks
Kurd, Z., & Kelly, T. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39878-3_7 · Cited by 21 · see citing works
Self-citation
Safety critical neural networks
Morgan, G. (1995) · doi:10.1049/cp:19950556 · Cited by 21 · see citing works
Practical Assessment of Neural Network Applications
Nabney, I. T., Paven, M. J. S., Eldridge, R. C., & Lee, C. (1997) · Springer eBooks · doi:10.1007/978-1-4471-0997-6_28 · Cited by 10 · see citing works
A Hybrid Approach to Breast Cancer Diagnosis
Sordo, M., Buxton, H., & Watson, D. (2001) · International series in intelligent technologies · doi:10.1007/978-94-010-0678-1_10 · Cited by 9 · see citing works

Flagged first, then by citations

1 of the 20 references couldn’t be matched to a known paper.

Sorted by citations

A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
An Abstraction-Refinement Approach to Verification of Artificial Neural Networks
Pulina, L., & Tacchella, A. (2010) · International Conference on Computer Aided Verification · doi:10.1007/978-3-642-14295-6_24 · Cited by 311 · see citing works
Artificial Intelligence for Safety-Critical Systems in Industrial and Transportation Domains: A Survey
Perez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., Englund, C., Tauber, M., Nikolakopoulos, G., & Flores, J. L. (2024) · ACM Computing Surveys · doi:10.1145/3626314 · Cited by 185 · see citing works
An Analysis of ISO 26262: Machine Learning and Safety in Automotive Software
Salay, R., Queiroz, R., & Czarnecki, K. (2018) · SAE Technical Paper Series · doi:10.4271/2018-01-1075 · Cited by 109 · see citing works
Artificial intelligence in safety-critical systems: a systematic review
Wang, Y., & Chung, S. H. (2022) · Industrial Management & Data Systems · doi:10.1108/imds-07-2021-0419 · Cited by 96 · see citing works
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81 · see citing works
Building safer robots: Safety driven control
Woodman, R., Winfield, A. F. T., Harper, C., & Fraser, M. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364912459665 · Cited by 53 · see citing works
Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262Preprint
Salay, R., & Czarnecki, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.01614 · Cited by 47 · see citing works
Safety Assurance of Artificial Intelligence-Based Systems: A Systematic Literature Review on the State of the Art and Guidelines for Future Work
Silva, A. V., Camargo, J., Almeida, J., & Cugnasca, P. (2022) · IEEE Access · doi:10.1109/access.2022.3229233 · Cited by 51 · see citing works
Explainable Online Validation of Machine Learning Models for Practical Applications
Fuhl, W., Rong, Y., Motz, T., Scheidt, M., Hartel, A., Koch, A., & Kasneci, E. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412959 · Cited by 47 · see citing works
Formal verification of input-output mappings of tree ensembles
Törnblom, J., & Nadjm‐Tehrani, S. (2020) · Science of Computer Programming · doi:10.1016/j.scico.2020.102450 · Cited by 36 · see citing works
Hyperdimensional Computing in Industrial Systems: The Use-Case of Distributed Fault Isolation in a Power Plant
Kleyko, D. V., Osipov, E., Papakonstantinou, N., & Vyatkin, V. V. (2018) · IEEE Access · doi:10.1109/access.2018.2840128 · Cited by 34 · see citing works
Application of Artificial Intelligence for COVID-19 Epidemic: An Exploratory Study, Opportunities, Challenges, and Future Prospects
Awotunde, J. B., Folorunso, S. O., Jimoh, R. G., Adeniyi, A. E., Abiodun, M. K., & Ajamu, G. J. (2021) · Studies in systems, decision and control · doi:10.1007/978-3-030-69744-0_4 · Cited by 32 · see citing works
NeVer: a tool for artificial neural networks verification
Pulina, L., & Tacchella, A. (2011) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-011-9243-0 · Cited by 29 · see citing works
Fault detection in the hyperspace: Towards intelligent automation systems
Kleyko, D. V., Osipov, E., Papakonstantinou, N., Vyatkin, V. V., & Mousavi, A. (2015) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin.2015.7281909 · Cited by 25 · see citing works
On the Safety of Automotive Systems Incorporating Machine Learning Based Components: A Position Paper
Gharib, M., Lollini, P., Botta, M., Amparore, E. G., Donatelli, S., & Bondavalli, A. (2018) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w.2018.00074 · Cited by 25 · see citing works
Expanding the Classical V-Model for the Development of Complex Systems Incorporating AI
Ullrich, L., Buchholz, M., Dietmayer, K., & Graichen, K. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3434515 · Cited by 21 · see citing works
Using Fuzzy Self-Organising Maps for Safety Critical Systems
Kurd, Z., & Kelly, T. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30138-7_3 · Cited by 21 · see citing works
Self-citation
Optimization of an artificial neural network for fatigue damage identification using analysis of variance
Sbarufatti, C. (2016) · Structural Control and Health Monitoring · doi:10.1002/stc.1964 · Cited by 20 · see citing works
Neural Network Application in Reservoir Water Level Forecasting and Release Decision
Ishak, W. H. W., Ku‐Mahamud, K. R., & Norwawi, N. M. (2011) · Universiti Utara Malaysia Institutional Repository (Universiti Utara Malaysia) · Cited by 20 · see citing works
Formal Verification of Random Forests in Safety-Critical Applications
Törnblom, J., & Nadjm‐Tehrani, S. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-12988-0_4 · Cited by 19 · see citing works
On the Evaluation Measures for Machine Learning Algorithms for Safety-Critical Systems
Gharib, M., & Bondavalli, A. (2019) · European Dependable Computing Conference · doi:10.1109/edcc.2019.00035 · Cited by 17 · see citing works
Verifying Feedforward Neural Networks for Classification in Isabelle/HOL
Brucker, A. D., & Stell, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-27481-7_24 · Cited by 15 · see citing works
Prediction and classification of diabetes mellitus using genomic data
Awotunde, J. B., Ayo, F. E., Jimoh, R. G., Ogundokun, R. O., Matiluko, O. E., Oladipo, I. D., & AbdulRaheem, M. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-821187-8.00009-5 · Cited by 13 · see citing works
The Missing Link: Developing a Safety Case for Perception Components in Automated Driving
Salay, R., Czarnecki, K., Kuwajima, H., Yasuoka, H., Abdelzad, V., Huang, C. G., Kahn, M., Nguyen, V. D., & Nakae, T. (2022) · SAE International Journal of Advances and Current Practices in Mobility · doi:10.4271/2022-01-0818 · Cited by 13 · see citing works
Verification of Adaptive Systems
Pullum, L., Cui, X., Vassev, E., Hinchey, M., Rouff, C., & Buskens, R. W. (2012) · AIAA Infotech@Aerospace Conference · doi:10.2514/6.2012-2478 · Cited by 13 · see citing works
A cognitive architecture safety design for safety critical systems
Dreany, H. H., & Roncace, R. A. (2019) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2019.106555 · Cited by 12 · see citing works
Modelling reservoir water release decision using temporal data mining and neural network
Ishak, W. H. W., Ku‐Mahamud, K. R., & Norwawi, N. M. (2012) · Universiti Utara Malaysia Institutional Repository (Universiti Utara Malaysia) · Cited by 10 · see citing works
Deep Learning for Autonomous Vehicle Control: Algorithms, State-of-the-Art, and Future Prospects
Kuutti, S., Fallah, S., Bowden, R., & Barber, P. (2019) · Synthesis lectures on advances in automotive technology · doi:10.2200/s00932ed1v01y201906aat008 · Cited by 29 · see citing works
Towards Efficient Hazard Identification in the Concept Phase of Driverless Vehicle Development
Graubohm, R., Stolte, T., Bagschik, G., & Maurer, M. (2020) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv47402.2020.9304780 · Cited by 9 · see citing works
Internet of medical things for enhanced smart healthcare systems
Awotunde, J. B., Chakraborty, C., AbdulRaheem, M., Jimoh, R. G., Oladipo, I. D., & Bhoi, A. K. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91916-6.00009-6 · Cited by 9 · see citing works
A Plant-Wide and Function-Specific Hierarchical Functional Fault Detection and Identification (HFFDI) System for Multiple Fault Scenarios on Complex Systems
Papakonstantinou, N., Proper, S., O’Halloran, B. M., & Tumer, I. Y. (2015) · doi:10.1115/detc2015-46447 · Cited by 7 · see citing works
A Review Study on Functional Safety Verification and Validation for Autonomous Vehicles
Haspolat, C., Celik, E., Aygun, M., Ertaş, I. C., & Karaduman, M. (2023) · National Conference on Electrical and Electronics Engineering (ELECO) · doi:10.1109/eleco60389.2023.10416071 · Cited by 6 · see citing works
Safety engineering of computational cognitive architectures within safety-critical systems
Dreany, H. H., Roncace, R. A., & Young, P. (2017) · Safety Science · doi:10.1016/j.ssci.2017.10.020 · Cited by 6 · see citing works
Safety Challenges of AI in Autonomous Systems Design – Solutions from Human Factors Perspective Emphasizing AI Awareness
Karvonen, H., Heikkilä, E., & Wahlström, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49183-3_12 · Cited by 8 · see citing works
Numerical Discrimination of the Generalisation Model from Learnt Weights in Neural Networks
Rudd-Orthner, R. N. M., & Milhaylova, L. (2019) · Annals of Emerging Technologies in Computing · doi:10.33166/aetic.2019.04.001 · Cited by 5 · see citing works
Self-organisation for Survival in Complex Computer Architectures
Polack, F. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14412-7_4 · Cited by 5 · see citing works
Impact of Machine Learning on Safety Monitors
Terrosi, F., Strigini, L., & Bondavalli, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14835-4_9 · Cited by 5 · see citing works
Establishing Verification and Validation Objectives for Safety-Critical Bayesian Networks
Douthwaite, M., & Kelly, T. (2017) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2017.60 · Cited by 5 · see citing works
Self-citation
Deployment of a Machine Learning System for Predicting Lawsuits Against Power Companies: Lessons Learned from an Agile Testing Experience for Improving Software Quality
Rivero, L., Diniz, J. O. B., Silva, G., Borralho, G., Bráz, G., de Paiva, A. C., Alves, E. W. B. A. L., & Oliveira, M. (2020) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3439961.3439991 · Cited by 5 · see citing works
Foreseeing the Impact of the Proposed AI Act on the Sustainability and Safety of Critical Infrastructures
Sovrano, F., & Masetti, G. (2022) · International Conference on Theory and Practice of Electronic Governance · doi:10.1145/3560107.3560253 · Cited by 4 · see citing works
Design and Assurance of Safety-Critical Systems with Artificial Intelligence in FPGAs: The Safety ArtISt Method and a Case Study of an FPGA-Based Autonomous Vehicle Braking Control System
Neto, A. V. D. S., Silva, H. L., Camargo, J. B., de Almeida, J. R., & Cugnasca, P. S. (2023) · Electronics · doi:10.3390/electronics12244903 · Cited by 4 · see citing works
IT2F-SEDNN: an interval type-2 fuzzy logic-based stacked ensemble deep learning approach for early phase software dependability analysis
Chatterjee, S., & Saha, D. (2024) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-024-00563-4 · Cited by 3 · see citing works
Assurance Case Patterns for Cyber-Physical Systems with Deep Neural Networks
Kaur, R., Ivanov, R., Cleaveland, M., Sokolsky, O. V., & Lee, I. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55583-2_6 · Cited by 3 · see citing works
A New Perspective on AI Safety Through Control Theory Methodologies
Ullrich, L., Zimmer, W. M. X., Greer, R., Graichen, K., Knoll, A., & Trivedi, M. M. (2025) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2025.3585274 · Cited by 3 · see citing works
The State of Solutions for Autonomous Systems Safety
Banks, A., Alexander, R., & Ashmore, R. (2018) · Safety-Critical Systems Symposium (SSS) · doi:10.65391/r920 · Cited by 2 · see citing works
Numerical Discrimination of the Generalisation Model from Learnt Weights in Neural Networks
Rudd-Orthner, R. N. M., & Mihaylova, L. S. (2019) · International Conference on Computing, Electronics & Communications Engineering (iCCECE) · doi:10.1109/iccece46942.2019.8941988 · Cited by 2 · see citing works
A Structured Approach to Trustworthy Autonomous/Cognitive SystemsPreprint
Putzer, H. J., & Woźniak, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08210 · Cited by 2 · see citing works
Learning Verifiable Ensembles for Classification Problems with High Safety Requirements
Nusser, S., Otte, C., Hauptmann, W., & Kruse, R. (2010) · IGI Global eBooks · doi:10.4018/978-1-61520-757-2.ch019 · Cited by 1 · see citing works
On an attempt to explore challenges for Artificial Intelligence and Machine Learning in Indian Military and Defence Sector and Studying the Possible Inter-relationship amongst them using ISM Methodology
Bansal, M., Dinesh, U., Aggarwal, R., & K., V. (2019) · International Journal of Computer Applications · doi:10.5120/ijca2019919695 · Cited by 1 · see citing works
Autonomous Driving using CNN
Amar, V. (2021) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2021.35771 · Cited by 1 · see citing works
The Capture and Evolution of Contextual Requirements: The Case of Adaptive Systems
Knauss, A. (2015)
Dependability Evaluation of an Online Pupillometry-based Feedback System for Optimized Training
Carmegren, E., Ästrand, E., & Tomasic, I. (2022) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro55190.2022.9803542
Arguing Reliability of Machine Learning-based Components
Hu, B. C., Sandro, A. D., Marsso, L., Czarnecki, K., & Chećhik, M. (2025) · IEEE Engineering Reliable Autonomous Systems (ERAS) · doi:10.1109/eras63351.2025.11135634
Modeling of distillation processes using ANNs
Shokri, A., & Ghaemi, A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-32832-9.00008-4

Authors: Kurd, Z., Kelly, T., & Austin, J.
Affiliations: University of York (GB)
Corresponding author: Kurd, Zeshan
ORCID: Kelly, Tim
Venue: Neural Computing and Applications
Publisher: Springer Science and Business Media LLC
Published: 2006-03-30
Volume: 16
Issue: 1
Pages: 11-19
ISSN: 0941-0643 (ISSN-L)
1433-3058
DOI: 10.1007/s00521-006-0039-9
OpenAlex: W2046353767
Cited by: 55
References: 20
Citations by year: 2025: 3 · 2024: 2 · 2023: 5 · 2022: 5 · 2021: 5 · 2020: 7 · 2019: 8 · 2018: 5 · 2017: 2 · 2016: 1 · 2015: 3 · 2012: 3
FWCI: 0.00
Type: journal article
Language: en
Open access: closed
Licence: www.springer.com/tdm (tdm)
Locations: Neural Computing and Applications (publishedVersion)
Topics: Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords: artificial neural networks, safety-critical systems, safety certification, safety assurance, safety requirements, safety lifecycle, dependability, safety criteria

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Developing artificial neural networks for safety critical systems

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks