A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective
Open accessThe DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; 12 of the 71 sources its claims cite were checked for support.
Abstract
The requirements for ensuring functional safety have always been very high. Modern safety-related systems are becoming increasingly complex, making also the safety integrity assessment more complex and time-consuming. This trend is further intensified by the fact that AI-based algorithms are finding their way into safety-related systems or will do so in the future. However, existing and expected standards and regulations for the use of AI methods pose significant challenges for the development of embedded AI software in functional safety-related systems. The consideration of essential requirements from various perspectives necessitates an intensive examination of the subject matter, especially as different standards have to be taken into account depending on the final application. There are also different targets for the “safe behavior” of a system depending on the target application. While stopping all movements of a machine in industrial production plants is likely to be considered a “safe state”, the same condition might not be considered as safe in flying aircraft, driving cars or medicine equipment like heart pacemaker. This overall complexity is operationalized in our approach in such a way that it is straightforward to monitor conformity with the requirements. To support safety integrity assessments and reduce the required effort, a Self-Enforcing Network (SEN) model is presented in which developers or safety experts can indicate the degree of fulfillment of certain requirements with possible impact on the safety integrity of a safety-related system. The result evaluated by the SEN model indicates the achievable safety integrity level of the assessed system, which is additionally provided by an explanatory component.
Overview
CoChat Summary
TL;DR
The paper proposes an explainable Self-Enforcing Network (SEN) requirements model that maps developers’ assessed fulfillment of 44 AI/functional-safety requirements to achievable Safety Integrity Levels (SIL 1–3) or an unsuitable classification. It uses expert-defined reference types and Shapley-value feature importance to show which requirements drive the result.
Key findings
- SEN validation of the reference types produced ranking/distance values of SIL 1: 1.26/0.00, SIL 2: 0.82/0.00, SIL 3: 2.68/0.00, and “Not suitable for SCS”: 2.84/0.00 (Table 1).
- For a deterministic NN with an explanatory component, SEN classified the input as SIL 2 but with lower activation than the validated vector and at a greater distance; feature importance showed “hazard and risk analysis” was not met at all and “coding standards” only partially satisfied.
- DL without xAI was classified as not suitable for SCS, with feature importance showing most requirements not met. DL with beginnings of xAI gave no clear SIL assignment: distance 0.85 assigned SIL 1, while the strongest ranking activation of 1.22 argued it was still unsuitable, lower than 2.84 for “Not suitable for SCS” and near the SIL 1 final activation.
- A rule-based algorithm in a 2-channel architecture produced activation 2.17 and distance 0.61, conforming to SIL 3 requirements; feature importance showed deviations because a 3-channel architecture was expected and the base model was designed primarily for AI methods used redundantly for monitoring or diagnostics.
Methods
- The requirements model was built from standards including EN 61508:2010, EN ISO 13849-1:2023, EN IEC 62061:2021, IEC ISO TR 5469:2024, and the AI Act, with 44 selected requirements and expert-assigned reference types for SIL 1–3 and an unsuitable method; values were normalized between -1 and 1.
- SEN is a two-layer feed-forward self-organized learning network: semantic-matrix values are transformed by the Self-Enforcing Rule \(w_{ao}=c * v_{sm} * cvf_a\), with an Enforcing Activation Function; parameters included learning rate 0.1, one learning iteration, and cue validity factors of 0.5 for general requirements and 1.0 for explainability and system-decisive attributes.
- Analysis used validation by presenting reference types, classification by highest activation ranking and smallest Euclidean distance, and explainability via Shapley-value feature importance, simplified in SEN as \(\Phi_i(v)=v(i)\).
Limitations
- The paper states the basic model was developed primarily for NN-based methods, so requirements and possible applications for other algorithms need to be extended.
- The paper states the selection and evaluation of requirements were not determined by a broad group of experts, so subjective influences cannot be ruled out; more experts should be involved to generalize the model.
- The paper notes the DL-with-beginnings-of-xAI case produced non-equal first-place results and that its feature-importance visualization could be misinterpreted; it also states the model considers only general terms, with concrete requirements still needing identification for specific cases, and no data are associated with the article.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 9 Not valid · 49 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In both cases, experts also determine the most important characteristics of the processes, which are reinforced in the SEN by the cvf.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 4.1 Examples of practical applications of SEN in SCS
What the sources say
- Partly supportedcites [77] Zinkhan D, Greisbach A, Zurmaar B, Klüver C and Klüver J. Intrinsic explainable self-enforcing networks using the icon-d2-ensemble prediction system for runway configurations. Eng Proc 2023; 39: 41. sourceExperimental — accuracy not yet measured
Section 2.2 (SEN) supports the mechanism half of the claim: the cvf is listed as 'the so-called cue validity factor (cvf) for highlighting essential attributes', and 'For each feature, a cue validity factor (cvf) is set when building the model... influences the strength of an attribute's effect on activation', with the farthest measurement points getting the lowest cvf and the median the highest. However, our copy nowhere says that experts determine the most important characteristics of the process: domain experts (air traffic controllers) appear only in the Conclusions (p. '4. Conclusions and Recent Work'), where the requirement is that decisions be retraceable for them, and the paper actually states the modelling process 'can no longer be accomplished by a human alone'. The paper also covers a single application case (runway-direction prediction with ICON-D2), so the 'in both cases' framing is not established here. (LLM, conf=0.75)
- Not matchedcites [78] Klüver C, Werner C, Nowara P, Castel B and Israel R. Self-enforcing networks for monitoring safety-critical systems: A prototype development. In: Klüver C, & Klüver J (eds.) New algorithms for practical problems: Springer Vieweg, 2025 (in German).
Its entry has no title and year that we can search by.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“AI is often not used directly in a safety function, but rather as a diagnostic function.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 2.4 Standardization in functional safety systems and AI-Methods
What the sources say
- Could not verifycites [18] Bacciu D, Carta A, Gallicchio C and Schmittner C. Safety and Robustness for Deep Neural Networks: An Automotive Use Case. In: International Conference on Computer Safety, Reliability, and Security, Springer, 2023, pp. 95-107. sourceExperimental — accuracy not yet measured
Our copy under DOI 10.1007/978-3-031-40953-0_9 contains no text of the paper itself: every retrieved passage (sections 'Abstract', 'Abstract > 1.01 Articolo in rivista', 'Abstract > 789.32 kB') is boilerplate from an Italian institutional repository (IRIS, Università degli Studi di Palermo) landing page, including an ASN 2023-2025 simulation disclaimer, file-size/format listings for a 789.32 kB PDF, and a citation record for an unrelated item ('Carta, A. (2023). IL PURGATORIO: la cantica della nostalgia. IL FARDELLA'). Four searches (the claim's phrasing, 'diagnostic function ISO 26262 safety mechanism', 'ASIL decomposition', 'robustness of neural networks automotive') returned nothing about AI, safety functions, diagnostic functions, deep neural networks or automotive applications, so the claim cannot be checked against this source. This is absence of coverage in our copy, not a contradiction. (LLM, conf=0.72)
- Partly supportedcites [19] Perez-Cerrolaza J, Abella J and Borg M et al. Artificial intelligence for safety-critical systems in industrial and transportation domains: A survey. ACM Comput Surv 2024; 56: 1-40. sourceExperimental — accuracy not yet measured
Section 3.4 (Point of Application, Usage Level and Class) does support the substance of the distinction: under the ISO 5469 UL taxonomy AI may implement a safety function (A), a non-safety-related function (C/D), or serve the development process (B), and it explicitly notes that "AI-based diagnostic functions can be classified as A2 or C". Related passages reinforce the pattern — the safety-bag/run-time-monitor discussion (Section 3.4 and the Data Quality/ASTM F3269 passage) says "the safety bag becomes the safety function ... and the AI item does not require safety standard compliance", and the Introduction states AI techniques "have not been recommended for use in safety-critical systems". What our copy does NOT support is the frequency/dichotomy claim that AI is "often" used as a diagnostic function rather than directly in a safety function: the survey treats both usages as categories and its Table 5 case studies list AI items at UL A performing safety functions in avionics, aerospace and industrial control, so the "often ... rather as a diagnostic function" generalisation goes beyond what the source states. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, IEC 61508:2010 is interpreted such that it does not recommend the use of AI methods, even for diagnostics, solutions as part of safety-related systems are based on performing system decompositions [10] so that e.g., a non-AI-based deterministic monitor adopts its safety requirements together with those of the AI-based component it is monitoring [20].”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 2.4 Standardization in functional safety systems and AI-Methods
What the sources say
- Supportedcites [20] Brando A, Serra I and Mezzetti E et al. On neural networks redundancy and diversity for their use in safety-critical systems. Computer 2023; 56: 41-50. source
The Background on the Development Process for Safety Systems section says that the default approach uses system decomposition so a non-DL-based deterministic monitor inherits its own safety requirements and those of the DL-based component it monitors. This directly supports the portion of the citing sentence attached to [20]. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“As shown in Figure 7, black-box models, like most neural network-based methods, cannot be used in SCS unless they are interpretable by adding an explanatory component.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 3.2 Explainability of AI-systems
What the sources say
- Could not verifycites [45] Ali S, Abuhmed T and El-Sappagh S et al. Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence. Inf Fus 2023; 99: 101805 sourceExperimental — accuracy not yet measured
Section 1.1 (page unavailable) discusses post-hoc explanations and interpretable-by-design models, but does not say black-box or neural-network models cannot be used in safety-critical systems without an explanatory component. Searches for “safety-critical systems,” “SCS,” and neural-network deployment found no passage addressing that requirement. (LLM, conf=0.76)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“No external access to the system: important SCS prerequisite.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > Requirements for AI methods (considering Articles 8 -15 of the AI Act)
What the sources say
- Could not verifycites [64] Wang Y and Chung SH. Artificial intelligence in safety-critical systems: a systematic review. Indus Manag Data Syst 2022; 122: 442-470. sourceExperimental — accuracy not yet measured
Our copy of Wang, Yue and Chung (2022) nowhere states that lack of external access to the system is a prerequisite of a safety-critical system. The Introduction (p. 1) characterises SCS by 'unacceptable consequences' of failure and 'high requirements for dependability, especially for safety, security and reliability', plus certification and IEC 61508 restrictions, and sections 4.2.2.1-4.2.2.5 cover dependability, availability, safety and V&V — none of these assert system isolation or no external access. Searched: 'no external access to the system', 'prerequisite / requirements for a system to be safety-critical', 'closed-world / isolation / connectivity'; no passage addresses the claim, and our copy is not truncated. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
31 more claims
Create a free account to see all 137 claims and to verify the 27 references still open.
- Probably not a substantial claim. 87
- It may state two separate claims as one. 34
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 5 Results
“The visualization of the FI in Figure 20 is also not entirely clear in this case and could be misinterpreted”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 5 Results
“Figure 20.Result of SEN for the input vector "Rule-based algorithm"”
Citations
Cites
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 1×
“HARA (hazard and risk analysis), a known standard framework, defines how to assess and evaluate the risk posed by electronics (in road vehicles).”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 1×
“HARA (hazard and risk analysis), a known standard framework, defines how to assess and evaluate the risk posed by electronics (in road vehicles).”
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495 · cited 3×
“A “MLOps” process (mixed ML application development and operation) supports the continuous development and safety assurance of ML-based systems in the railway sector [31].”
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · cited 1×
“In [22], a corresponding extension is proposed (Figure 6).”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386 · cited 1×
“Several techniques and tools are meanwhile used to support explainability [19, 53, 57, 58], or to define different phases of XAI in a typical machine learning development process [59].”
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983 · cited 1×
“The framework consists of a “world model” that provides a mathematical description of how the AI system affects the outside world, a mathematical description of what effects are acceptable in a “safety specification”, and a “verifier” that should provide a proof that the AI satisfies the safety…”
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“In this case, the learning method can either be transparent (algorithmic transparency), the technical functionality can be clear (simulability), or the algorithm can be decomposed into its individual parts (decomposability) [51, 52].”
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783 · cited 1×
“Quality models for AI products, such as ISO/IEC 25059:2023 (will be replaced by ISO/IEC AWI 25059 Software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Quality model for AI systems) need to be extended, e.g. , to meet the requirements of the AI Act.”
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 2×
“It can refer to the functioning of the entire model ( global xAI ) or to the explanation of the background of a specific decision ( local xAI ) [47–49].”
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504
A., S., & R., S. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100230 · Cited by 432 · cited 1×
“To meet the requirements, for example, libraries for testing deep learning methods [23, 24] and trustworthy ML-based [25], or assurance case-centered methods [23] are proposed for systems engineering.”
Habbal, A., Ali, M. K., & Abuzaraida, M. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122442 · Cited by 407 · cited 1×
“Due to the large number of publications, we focus on recent developments that affect SCS in the selected examples.”
Managing extreme AI risks amid rapid progress
Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Darrell, T., Harari, Y. N., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., et al. (2024) · Science · doi:10.1126/science.adn0117 · Cited by 349 · cited 1×
“In addition, as soon as safety is affected, specific regulations must be followed, especially in functional safety-related systems (ISO/IEC TR 5469:2024), or in SCS respectively, which are meanwhile enshrined in law, e.g. , through the European AI ACT [1–3].”
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
Schwalbe, G., & Finzel, B. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00867-8 · Cited by 328
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 323
Islam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022) · Applied Sciences · doi:10.3390/app12031353 · Cited by 311 · cited 1×
“When using AI methods, special criteria must be considered, such as noise or failures in sensor data, which are essential for the correct detection of problems in machines [26].”
Safety-critical advanced robots: A survey
Guiochet, J., Machin, M., & Waeselynck, H. (2017) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2017.04.004 · Cited by 231 · cited 1×
“• Method for dependability assessment: service that can be trusted [67]; critical for software-intensive SCS. • Method for reliability analysis: the probability that the application will satisfactorily fulfill its expected function without failure.”
Shapley value: from cooperative game to explainable artificial intelligence
Li, M., Sun, H., Huang, Y., & Chen, H. (2024) · Autonomous Intelligent Systems · doi:10.1007/s43684-023-00060-8 · Cited by 205
Perez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., Englund, C., Tauber, M., Nikolakopoulos, G., & Flores, J. L. (2024) · ACM Computing Surveys · doi:10.1145/3626314 · Cited by 185 · cited 3×
“The framework consists of a “world model” that provides a mathematical description of how the AI system affects the outside world, a mathematical description of what effects are acceptable in a “safety specification”, and a “verifier” that should provide a proof that the AI satisfies the safety…”
Artificial Intelligence risk measurement
Giudici, P., Centurelli, M., & Turchetta, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121220 · Cited by 181 · cited 2×
“• AI – Neural Networks: flatten, deterministic.”
Sanneman, L., & Shah, J. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2081282 · Cited by 169
How to certify machine learning based safety-critical systems? A systematic literature review
Tambon, F., Laberge, G., An, L., Nikanjam, A., Mindom, P. S. N., Pequignot, Y., Khomh, F., Antoniol, G., Merlo, E., & Laviolette, F. (2022) · Automated Software Engineering · doi:10.1007/s10515-022-00337-x · Cited by 157 · cited 1×
“Such tools are particularly important for licensing or certifying AI methods [33, 34] for licensing high-risk artificial intelligence].”
Nannini, L., Balayn, A., & Smith, A. L. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594074 · Cited by 138 · cited 1×
“Explainability in SCS is considered from different perspectives, such as regulations and standards [61].”
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Requirements Engineering for Artificial Intelligence Systems: A Systematic Mapping Study
Ahmad, K., Almorsy, M., Arora, C., Bano, M., & Grundy, J. C. (2022) · Information and Software Technology · doi:10.48550/arxiv.2212.10693 · Cited by 128
Wang, H., Shao, W., Sun, C., Yang, K., Cao, D., & Li, J. (2024) · Engineering · doi:10.1016/j.eng.2023.10.011 · Cited by 124 · cited 1×
“…frameworks should ensure trustworthiness for high-risk AI-based systems [37], consistency, reliability, explainability, and safety (CRISP) [38], safety of intended functionality (SOTIF) [19, 39], or improvement for the SCS in terms of uncertainty prediction and performance [40], to name only a few.”
Making Sense of Machine Learning: A Review of Interpretation Techniques and Their Applications
Tursunalieva, A., Alexander, D. L., Dunne, R. A., Li, J., Riera, L., & Zhao, Y. (2024) · Applied Sciences · doi:10.3390/app14020496 · Cited by 119 · cited 1×
“Numerous approaches have emerged, with respect to explainability, in [41], referred to as the “Tower of Babel in Explainable Artificial Intelligence” [42–44].”
Artificial intelligence in safety-critical systems: a systematic review
Wang, Y., & Chung, S. H. (2022) · Industrial Management & Data Systems · doi:10.1108/imds-07-2021-0419 · Cited by 96 · cited 1×
“As mentioned above, many methods are sublimated under AI, which can generally be divided into three classes (Figure 7) in terms of their ability to be explainable or interpretable [45].”
Trustworthiness Assurance Assessment for High-Risk AI-Based Systems
Stettinger, G., Weissensteiner, P., & Khastgir, S. (2024) · IEEE Access · doi:10.1109/access.2024.3364387 · Cited by 90 · cited 1×
“Further, actually developed frameworks should ensure trustworthiness for high-risk AI-based systems [37], consistency, reliability, explainability, and safety (CRISP) [38], safety of intended functionality (SOTIF) [19, 39], or improvement for the SCS in terms of uncertainty prediction and…”
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89 · cited 1×
“• Validation: Model, solution checking [65] (critical in CSC with embedded AI).”
Managing AI Risks in an Era of Rapid Progress
Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Harari, Y., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., Hadfield, G. K., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.17688 · Cited by 89
Malgieri, G., & Pasquale, F. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105899 · Cited by 88 · cited 1×
“Due to the large number of publications, we focus on recent developments that affect SCS in the selected examples.”
Wörsdörfer, M. (2023) · Global Business and Organizational Excellence · doi:10.1002/joe.22238 · Cited by 80
Rech, P. (2024) · IEEE Transactions on Nuclear Science · doi:10.1109/tns.2024.3349956 · Cited by 75 · cited 1×
“CHESSIoT, a model-driven environment for engineering industrial IoT systems (based on Fault Tree Analysis (FTA), CHESS, and Failure Propagation Transformation Calculus (FPTC)), enables the estimation of the failure behavior of the entire system [35].”
Building trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety
Gaur, M., & Sheth, A. (2024) · AI Magazine · doi:10.1002/aaai.12149 · Cited by 65
It is Not “Accuracy vs. Explainability”—We Need Both for Trustworthy AI Systems
Petkovic, D. (2023) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2023.3239921 · Cited by 63 · cited 1×
“Various requirements such as robustness, reliability, traceability, data quality, and other factors must be demonstrated [25, 43, 62, 63].”
Explainable AI Frameworks: Navigating the Present Challenges and Unveiling Innovative Applications
Sharma, N. A., Chand, R. R., Buksh, Z., Ali, A. B. M. S., Hanif, A., & Beheshti, A. (2024) · Algorithms · doi:10.3390/a17060227 · Cited by 61 · cited 1×
“Several techniques and tools are meanwhile used to support explainability [19, 53, 57, 58], or to define different phases of XAI in a typical machine learning development process [59].”
Golpayegani, D., Pandit, H. J., & Lewis, D. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594050 · Cited by 42 · cited 1×
“To describe the safety function, the input and output interface must be specified and the behavior at the output must be described depending on the conditions at the input.”
Is the safety index of modern safety integrity level(SIL) truly appropriate for the railway?
Cao, Y., An, Y., Su, S., et al. (2023) · Accident Analysis & Prevention · doi:10.1016/j.aap.2023.107267 · Cited by 38 · cited 1×
“It specifies the corresponding numerical ranges of the target failure rate [7].”
Explainable deep learning for attack intelligence and combating cyber–physical attacks
Al-Hawawreh, M. S., & Moustafa, N. (2023) · Ad Hoc Networks · doi:10.1016/j.adhoc.2023.103329 · Cited by 35 · cited 1×
“Explainable AI (xAI) is used to improve the traceability of the attack attribution module and to extract attack intelligence [36].”
A software development process model for artificial neural networks in critical applications
Rodvold, D. M. (2003) · IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339) · doi:10.1109/ijcnn.1999.836192 · Cited by 34 · cited 1×
“The example-based methods can include counterfactuals as a possibility to explain predictions, or adversarial examples to provide insight into the internal structure of algorithms, find their weaknesses, and improve interpretability [43].”
Zeller, M., Waschulzik, T., Schmid, R., & Bahlmann, C. (2024) · AI and Ethics · doi:10.1007/s43681-023-00392-4 · Cited by 31 · cited 1×
“A “MLOps” process (mixed ML application development and operation) supports the continuous development and safety assurance of ML-based systems in the railway sector [31].”
Chelouati, M., Boussif, A., Beugin, J., & Koursi, E. M. E. (2022) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2022.108933 · Cited by 30 · cited 1×
“The example-based methods can include counterfactuals as a possibility to explain predictions, or adversarial examples to provide insight into the internal structure of algorithms, find their weaknesses, and improve interpretability [43].”
Navigating the EU AI Act: A Methodological Approach to Compliance for Safety-critical Products
Kelly, J., Zafar, S. A., Heidemann, L., Zacchi, J.-V., Espinoza, D., & Mata, N. (2024) · 2024 IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00179 · Cited by 30 · cited 1×
“In [22], a corresponding extension is proposed (Figure 6).”
Morales‐Forero, A., Bassetto, S., & Coatanéa, É. (2022) · AI & Society · doi:10.1007/s00146-022-01591-z · Cited by 23
An overview of key trustworthiness attributes and KPIs for trusted ML-based systems engineering
Mattioli, J., Sohier, H., Delaborde, A., Amokrane-Ferka, K., Awadid, A., Chihani, Z., Khalfaoui, S., & Pedroza, G. (2024) · AI and Ethics · doi:10.1007/s43681-023-00394-2 · Cited by 22 · cited 2×
“To meet the requirements, for example, libraries for testing deep learning methods [23, 24] and trustworthy ML-based [25], or assurance case-centered methods [23] are proposed for systems engineering.”
The Tower of Babel in Explainable Artificial Intelligence (XAI)
Schneeberger, D. M., Röttger, R., Cabitza, F., Campagner, A., Plass, M., Müller, H., & Holzinger, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40837-3_5 · Cited by 19 · cited 1×
“Quality models for AI products, such as ISO/IEC 25059:2023 (will be replaced by ISO/IEC AWI 25059 Software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Quality model for AI systems) need to be extended, e.g. , to meet the requirements of the AI Act.”
Safety requirements for mining systems controlled in automatic mode
(2020) · Acta Montanistica Slovaca · doi:10.46544/ams.v25i3.13 · Cited by 18 · cited 1×
“In the mining sector, there are demands for updates to ensure the safety of con-trol systems ( e.g. , [79]).”
Gaurav, K., Singh, B. K., & Kumar, V. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17707-6 · Cited by 18 · cited 1×
“• Method for fault diagnosis: detailed fault diagnosis to quickly identify process anomalies and component failures and determine the source of failures ( e.g. , [68]).”
Self-organized Learning by Self-Enforcing Networks
Klüver, C., & Klüver, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-38679-4_52 · Cited by 17 · cited 1×
“• Verification: verification methods and formal verification ( e.g. , rule extraction), dynamic and static verification (tests) (critical in CSC with embedded AI).”
Self-citationFahmy, H., Pastore, F., Briand, L., & Stifter, T. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3569935 · Cited by 17 · cited 1×
“• Hazard and risk analysis: e.g. , HARA; methods for DL are found e.g. , in [74].”
de Koning, M., Machado, T., Ahonen, A., Strokina, N., Dianatfar, M., De Rosa, F., Minav, T., & Ghabcheloo, R. (2024) · Safety Science · doi:10.1016/j.ssci.2024.106517 · Cited by 17 · cited 1×
“Even if the same AI method is used, it must be considered that the standards in the area of e.g. , aerospace are different from those for automated driving or mining.”
Guerreiro Augusto, M., Acar, B., Soto, A. C., Sivrikaya, F., & Albayrak, S. (2024) · Autonomous Intelligent Systems · doi:10.1007/s43684-023-00059-1 · Cited by 15
ACCESS: Assurance Case Centric Engineering of Safety–critical Systems
Wei, R., Foster, S., Mei, H., et al. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112034 · Cited by 15 · cited 1×
“To meet the requirements, for example, libraries for testing deep learning methods [23, 24] and trustworthy ML-based [25], or assurance case-centered methods [23] are proposed for systems engineering.”
Schwalbe, G., & Schels, M. (2020) · doi:10.20378/irb-47276 · Cited by 14
Mapping to IEC 61508 the hardware safety integrity of elements developed to ISO 26262
Okoh, P., & Myklebust, T. (2024) · Safety and Reliability · doi:10.1080/09617353.2024.2343959 · Cited by 14 · cited 1×
“The necessary risk reduction for an application can be defined via PL from EN ISO 13849 or SIL from EN IEC 62061 or IEC 61508 standards [9].”
Ferrucci, F. (2023) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2023.09.318 · Cited by 13 · cited 1×
“In this case, the example is from [81] who used the SISTEMA software to determine the PL in the risk matrix using the specified S , F , and P , for the implementation of an energy storage safety system (Figure 12).”
Baron, C., & Louis, V. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103887 · Cited by 12 · cited 2×
“Prototypes are selected representative instances from the data, and criticisms are instances that are not well represented by these prototypes.”
SAFEXPLAIN: Safe and Explainable Critical Embedded Systems Based on AI
Abella, J., Perez-Cerrolaza, J., Englund, C., Zonooz, B., Giordana, G., Donzella, C., Cazorla, F. J., Mezzetti, E., Serra, I., Brando, A., et al. (2023) · Design, Automation, and Test in Europe · doi:10.23919/date56975.2023.10137128 · Cited by 12 · cited 1×
““SAFEXPLAIN” [32] provides traceability for DL methods, safety patterns to satisfy criticality and fault tolerance requirements, library implementations according to safety requirements, and computer platform configurations to handle non-determinism in Critical Autonomous AI-based Systems (CAIS).”
Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
Dalrymple, D. "., Skalse, J., Bengio, Y., Russell, S., Tegmark, M., Seshia, S., Omohundro, S., Szegedy, C., Goldhaber, B., Ammann, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06624 · Cited by 12
Risk mitigation strategies for automated current and future mine hoists
Galy, B., & Giraud, L. (2023) · Safety Science · doi:10.1016/j.ssci.2023.106267 · Cited by 10 · cited 1×
“For e.g. , mine hoist command systems are safety-related electrical control systems (SRECS) using multiple safety instrumented systems (SIS), the use of a SIL or PL rating is strongly recommended by [80].”
On Neural Networks Redundancy and Diversity for Their Use in Safety-Critical Systems
Brando, A., Serra, I., Mezzetti, E., Cazorla, F. J., Perez-Cerrolaza, J., & Abella, J. (2023) · Computer · doi:10.1109/mc.2023.3236523 · Cited by 9 · cited 1×
“…the use of AI methods, even for diagnostics, solutions as part of safety-related systems are based on performing system decompositions [10] so that e.g. , a non-AI-based deterministic monitor adopts its safety requirements together with those of the AI-based component it is monitoring [20].”
SIL calculation in gas processing plants based on systematic faults and level of maturity
Abbasinejad, R., Hourfar, F., Kacprzak, D., et al. (2023) · Process Safety and Environmental Protection · doi:10.1016/j.psep.2023.04.044 · Cited by 9 · cited 1×
“A further increase in the safety integrity of a system can be achieved by using a system with a fully redundant safety path [13, 14], as shown in Figure 5 on the right.”
Safety Integrity Levels for Artificial Intelligence
Diemert, S., Millet, L., Groves, J., & Joyce, J. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-40953-0_34 · Cited by 8 · cited 2×
“As a result, such systematic faults generally cannot be safely controlled by identical redundant circuits.”
Supporting model-based safety analysis for safety-critical IoT systems
Ihirwe, F., Ruscio, D. D., Blasio, K. D., et al. (2023) · Journal of Computer Languages · doi:10.1016/j.cola.2023.101243 · Cited by 8 · cited 1×
“CHESSIoT, a model-driven environment for engineering industrial IoT systems (based on Fault Tree Analysis (FTA), CHESS, and Failure Propagation Transformation Calculus (FPTC)), enables the estimation of the failure behavior of the entire system [35].”
Safety-critical railway systems
Shubinsky, I. B., Rozenberg, E. N., & Баранов, Л. А. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-99204-6.00006-6 · Cited by 7
Safety and Robustness for Deep Neural Networks: An Automotive Use Case
Bacciu, D., Carta, A., Gallicchio, C., & Schmittner, C. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-40953-0_9 · Cited by 7 · cited 1×
“Another important issue is the classification of usage into “usage levels” [18, 19].”
Concept Enforcement and Modularization for the ISO 26262 Safety Case of Neural Networks
Schwalbe, G., & Schmid, U. (2020) · doi:10.20378/irb-47277 · Cited by 6
Iyenghar, P. (2024) · Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0012683600003687 · Cited by 4 · cited 1×
“When using AI methods, special criteria must be considered, such as noise or failures in sensor data, which are essential for the correct detection of problems in machines [26].”
Cao, S., Sun, X., Widyasari, R., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.14617 · Cited by 4
Determining Feature Importance in Self-Enforcing Networks to achieve Explainable AI (xAI)
Greisbach, A., & Klüver, C. (2022) · Proceedings - 32. Workshop Computational Intelligence: Berlin, 1. - 2. Dezember 2022 · doi:10.58895/ksp/1000151141-16 · Cited by 3
Self-citationSafety Argumentation for Machinery Assembly Control Software
Castellanos-Ardila, J. P., Punnekkat, S., Hansson, H., & Backeman, P. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-68606-1_16 · Cited by 2 · cited 1×
“This creates further challenges for any developer, which are not fully foreseeable at this stage [5, 6].”
Zinkhan, D., Greisbach, A., Zurmaar, B., Klüver, C., & Klüver, J. (2023) · ITISE 2023 · doi:10.3390/engproc2023039041 · Cited by 2
Self-citationFrom risks to requirements: Comparing the assignment of functional safety requirements
Malm, T., Venho-Ahonen, O., Hietikko, M., et al. (2015) · Cited by 1
Bjelica, M. Z. (2023) · Springer eBooks · doi:10.1007/978-3-031-15823-0 · Cited by 1
Correction: Uncertainty Modelling in Performability Prediction for Safety-Critical Systems
Ahamad, S., & Gupta, R. (2024) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-024-09019-0 · Cited by 1 · cited 1×
“…frameworks should ensure trustworthiness for high-risk AI-based systems [37], consistency, reliability, explainability, and safety (CRISP) [38], safety of intended functionality (SOTIF) [19, 39], or improvement for the SCS in terms of uncertainty prediction and performance [40], to name only a few.”
Deep Learning Library Testing: Definition, Methods and ChallengesPreprint
Zhang, X., Jiang, W., Shen, C., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.17871 · Cited by 1
de Koning, M., Machado, T., Ahonen, A., Strokina, N., Dianatfar, M., Rosa, F. D., Minav, T., & ghabcheloo, R. (2023) · doi:10.2139/ssrn.4650221
Determining Feature Importance in Self-Enforcing Networks to achieve Explainable AI (xAI)
Greisbach, A., & Klüver, C. (2022) · Forum Bildverarbeitung 2022 = Image Pocessing Forum 2022 · doi:10.58895/ksp/1000150865-16
Self-citation
Research works printed but not linked: 3
- Oveisi S, Gholamrezaie F and Qajari N et al. Review of artificial intelligence-based systems: evaluation, standards, and methods. Adv. Stand Appl Sci 2024; 2: 4-29.
- Weaver R, McDermid J and Kelly T. Software safety arguments: Towards a systematic categorisation of evidence. In: International System Safety Conference, Denver, CO 2002.
- Klüver C, Werner C, Nowara P, Castel B and Israel R. Self-enforcing networks for monitoring safety-critical systems: A prototype development. In: Klüver C, & Klüver J (eds.) New algorithms for practical problems: Springer Vieweg, 2025 (in German).
Cited by
Fitas, R. (2025) · IEEE Access · doi:10.1109/access.2025.3598909 · Cited by 20
No safe AI standardization in sight at present: an investigation into ISO/IEC TR 5469
Locherer, M., Kordovan, M., Buxbaum, B., & Kever, T. (2025) · International Conference on System Reliability and Safety (ICSRS) · doi:10.1109/icsrs68021.2025.11422167 · Cited by 5
“As noted in [43, Sec.”
Functional Safety Architectural Patterns for AI-Based Critical Systems
Agirre, I., Yarza, I., Abella, J., & Caro, M. (2025) · ACM Transactions on Cyber-Physical Systems · doi:10.1145/3769121 · Cited by 1
A reinforcement-learning-based fall mitigation system for stair-traversing service robots
Ong, J. H., Yang, Z., Samarakoon, S. M. B. P., & Elara, M. R. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.110413 · Cited by 1
Agoi, M. A., Emmanuel Taiwo, A. G. O. I., Oluwanifemi Opeyemi, A. G. O. I., & Ajaga, S. O. (2026) · International Journal of Applied Mathematics Computational Science and Systems Engineering · doi:10.37394/232026.2026.8.9
Metadata
- Authors
- Klüver, C., Greisbach, A., Kindermann, M., & Püttmann, B.
- Affiliations
- Fuchs (Germany) (DE)
- TÜV Nord (Germany) (DE)
- Venue
- Security and Safety
- Publisher
- EDP Sciences
- Published
- 2024
- Volume
- 3
- Pages
- 2024020
- ISSN
- 2826-1275 (ISSN-L)
- OpenAlex
- W4403935496
- Semantic Scholar
- de0d20afbad490814c19722a052f2d954d7a080d
- Cited by
- 8
- References
- 68
- Citations by year
- 2026: 4 · 2025: 4
- FWCI
- 2.08
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Security and Safety (publishedVersion, cc-by)
- Springer Link (Chiba Institute of Technology) (submittedVersion)
- Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Hardware and Architecture › Real-Time Systems Scheduling
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- functional safety, safety-critical systems, artificial intelligence algorithms, explainable AI, requirements models, safety integrity level, developer perspectives, safety standards and regulations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.