A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective
Open accessThe DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index; 11 of the 71 sources its claims cite were checked for support.
Abstract
The requirements for ensuring functional safety have always been very high. Modern safety-related systems are becoming increasingly complex, making also the safety integrity assessment more complex and time-consuming. This trend is further intensified by the fact that AI-based algorithms are finding their way into safety-related systems or will do so in the future. However, existing and expected standards and regulations for the use of AI methods pose significant challenges for the development of embedded AI software in functional safety-related systems. The consideration of essential requirements from various perspectives necessitates an intensive examination of the subject matter, especially as different standards have to be taken into account depending on the final application. There are also different targets for the “safe behavior” of a system depending on the target application. While stopping all movements of a machine in industrial production plants is likely to be considered a “safe state”, the same condition might not be considered as safe in flying aircraft, driving cars or medicine equipment like heart pacemaker. This overall complexity is operationalized in our approach in such a way that it is straightforward to monitor conformity with the requirements. To support safety integrity assessments and reduce the required effort, a Self-Enforcing Network (SEN) model is presented in which developers or safety experts can indicate the degree of fulfillment of certain requirements with possible impact on the safety integrity of a safety-related system. The result evaluated by the SEN model indicates the achievable safety integrity level of the assessed system, which is additionally provided by an explanatory component.
Overview
CoChat Summary
TL;DR
The paper proposes an explainable Self-Enforcing Network (SEN) requirements model that maps developers’ assessed fulfillment of 44 AI/functional-safety requirements to achievable Safety Integrity Levels (SIL 1–3) or an unsuitable classification. It uses expert-defined reference types and Shapley-value feature importance to show which requirements drive the result.
Key findings
- SEN validation of the reference types produced ranking/distance values of SIL 1: 1.26/0.00, SIL 2: 0.82/0.00, SIL 3: 2.68/0.00, and “Not suitable for SCS”: 2.84/0.00 (Table 1).
- For a deterministic NN with an explanatory component, SEN classified the input as SIL 2 but with lower activation than the validated vector and at a greater distance; feature importance showed “hazard and risk analysis” was not met at all and “coding standards” only partially satisfied.
- DL without xAI was classified as not suitable for SCS, with feature importance showing most requirements not met. DL with beginnings of xAI gave no clear SIL assignment: distance 0.85 assigned SIL 1, while the strongest ranking activation of 1.22 argued it was still unsuitable, lower than 2.84 for “Not suitable for SCS” and near the SIL 1 final activation.
- A rule-based algorithm in a 2-channel architecture produced activation 2.17 and distance 0.61, conforming to SIL 3 requirements; feature importance showed deviations because a 3-channel architecture was expected and the base model was designed primarily for AI methods used redundantly for monitoring or diagnostics.
Methods
- The requirements model was built from standards including EN 61508:2010, EN ISO 13849-1:2023, EN IEC 62061:2021, IEC ISO TR 5469:2024, and the AI Act, with 44 selected requirements and expert-assigned reference types for SIL 1–3 and an unsuitable method; values were normalized between -1 and 1.
- SEN is a two-layer feed-forward self-organized learning network: semantic-matrix values are transformed by the Self-Enforcing Rule \(w_{ao}=c * v_{sm} * cvf_a\), with an Enforcing Activation Function; parameters included learning rate 0.1, one learning iteration, and cue validity factors of 0.5 for general requirements and 1.0 for explainability and system-decisive attributes.
- Analysis used validation by presenting reference types, classification by highest activation ranking and smallest Euclidean distance, and explainability via Shapley-value feature importance, simplified in SEN as \(\Phi_i(v)=v(i)\).
Limitations
- The paper states the basic model was developed primarily for NN-based methods, so requirements and possible applications for other algorithms need to be extended.
- The paper states the selection and evaluation of requirements were not determined by a broad group of experts, so subjective influences cannot be ruled out; more experts should be involved to generalize the model.
- The paper notes the DL-with-beginnings-of-xAI case produced non-equal first-place results and that its feature-importance visualization could be misinterpreted; it also states the model considers only general terms, with concrete requirements still needing identification for specific cases, and no data are associated with the article.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Valid · 9 Not valid · 50 Not checked · 72 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“AI is often not used directly in a safety function, but rather as a diagnostic function.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 2.4 Standardization in functional safety systems and AI-Methods
What the sources say
- Could not verifycites [18] Bacciu D, Carta A, Gallicchio C and Schmittner C. Safety and Robustness for Deep Neural Networks: An Automotive Use Case. In: International Conference on Computer Safety, Reliability, and Security, Springer, 2023, pp. 95–107. sourceExperimental — accuracy not yet measured
Our copy under DOI 10.1007/978-3-031-40953-0_9 contains no text of the paper itself: every retrieved passage (sections 'Abstract', 'Abstract > 1.01 Articolo in rivista', 'Abstract > 789.32 kB') is boilerplate from an Italian institutional repository (IRIS, Università degli Studi di Palermo) landing page, including an ASN 2023-2025 simulation disclaimer, file-size/format listings for a 789.32 kB PDF, and a citation record for an unrelated item ('Carta, A. (2023). IL PURGATORIO: la cantica della nostalgia. IL FARDELLA'). Four searches (the claim's phrasing, 'diagnostic function ISO 26262 safety mechanism', 'ASIL decomposition', 'robustness of neural networks automotive') returned nothing about AI, safety functions, diagnostic functions, deep neural networks or automotive applications, so the claim cannot be checked against this source. This is absence of coverage in our copy, not a contradiction. (LLM, conf=0.72)
- Partly supportedcites [19] Perez-Cerrolaza, 2024, ACM Comput Surv sourceExperimental — accuracy not yet measured
Section 3.4 (Point of Application, Usage Level and Class) does support the substance of the distinction: under the ISO 5469 UL taxonomy AI may implement a safety function (A), a non-safety-related function (C/D), or serve the development process (B), and it explicitly notes that "AI-based diagnostic functions can be classified as A2 or C". Related passages reinforce the pattern — the safety-bag/run-time-monitor discussion (Section 3.4 and the Data Quality/ASTM F3269 passage) says "the safety bag becomes the safety function ... and the AI item does not require safety standard compliance", and the Introduction states AI techniques "have not been recommended for use in safety-critical systems". What our copy does NOT support is the frequency/dichotomy claim that AI is "often" used as a diagnostic function rather than directly in a safety function: the survey treats both usages as categories and its Table 5 case studies list AI items at UL A performing safety functions in avionics, aerospace and industrial control, so the "often ... rather as a diagnostic function" generalisation goes beyond what the source states. (LLM, conf=0.68)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, IEC 61508:2010 is interpreted such that it does not recommend the use of AI methods, even for diagnostics, solutions as part of safety-related systems are based on performing system decompositions [10] so that e.g., a non-AI-based deterministic monitor adopts its safety requirements together with those of the AI-based component it is monitoring [20].”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 2.4 Standardization in functional safety systems and AI-Methods
What the sources say
- Supportedcites [20] Brando, 2023, Computer source
The Background on the Development Process for Safety Systems section says that the default approach uses system decomposition so a non-DL-based deterministic monitor inherits its own safety requirements and those of the DL-based component it monitors. This directly supports the portion of the citing sentence attached to [20]. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“As shown in Figure 7, black-box models, like most neural network-based methods, cannot be used in SCS unless they are interpretable by adding an explanatory component.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 3.2 Explainability of AI-systems
What the sources say
- Could not verifycites [45] Ali, 2023, Inf Fus sourceExperimental — accuracy not yet measured
Section 1.1 (page unavailable) discusses post-hoc explanations and interpretable-by-design models, but does not say black-box or neural-network models cannot be used in safety-critical systems without an explanatory component. Searches for “safety-critical systems,” “SCS,” and neural-network deployment found no passage addressing that requirement. (LLM, conf=0.76)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“No external access to the system: important SCS prerequisite.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > Requirements for AI methods (considering Articles 8 -15 of the AI Act)
What the sources say
- Could not verifycites [64] Wang, 2022, Indus Manag Data Syst sourceExperimental — accuracy not yet measured
Our copy of Wang, Yue and Chung (2022) nowhere states that lack of external access to the system is a prerequisite of a safety-critical system. The Introduction (p. 1) characterises SCS by 'unacceptable consequences' of failure and 'high requirements for dependability, especially for safety, security and reliability', plus certification and IEC 61508 restrictions, and sections 4.2.2.1-4.2.2.5 cover dependability, availability, safety and V&V — none of these assert system isolation or no external access. Searched: 'no external access to the system', 'prerequisite / requirements for a system to be safety-critical', 'closed-world / isolation / connectivity'; no passage addresses the claim, and our copy is not truncated. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the base model was primarily designed for applications of AI methods that are currently not allowed to be used for control, but only redundantly for monitoring or diagnostics, the FI shows a deviation in this component.”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 5 Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
31 more claims
Create a free account to see all 137 claims and to verify the 28 references still open.
- Probably not a substantial claim. 87
- It may state two separate claims as one. 34
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 5 Results
“The visualization of the FI in Figure 20 is also not entirely clear in this case and could be misinterpreted”
§ A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective > 5 Results
“Figure 20.Result of SEN for the input vector "Rule-based algorithm"”
Citations
Cites
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 1×
“HARA (hazard and risk analysis), a known standard framework, defines how to assess and evaluate the risk posed by electronics (in road vehicles).”
17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495 · cited 3×
“A “MLOps” process (mixed ML application development and operation) supports the continuous development and safety assurance of ML-based systems in the railway sector [31].”
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762 · cited 1×
“In [22], a corresponding extension is proposed (Figure 6).”
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386 · cited 1×
“Several techniques and tools are meanwhile used to support explainability [19, 53, 57, 58], or to define different phases of XAI in a typical machine learning development process [59].”
Explainable artificial intelligence: a comprehensive review
Minh, D., Wang, H. X., Li, Y. F., & Nguyen, T. N. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10088-y · Cited by 983 · cited 1×
“The framework consists of a “world model” that provides a mathematical description of how the AI system affects the outside world, a mathematical description of what effects are acceptable in a “safety specification”, and a “verifier” that should provide a proof that the AI satisfies the safety…”
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“In this case, the learning method can either be transparent (algorithmic transparency), the technical functionality can be clear (simulability), or the algorithm can be decomposed into its individual parts (decomposability) [51, 52].”
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783 · cited 1×
“Quality models for AI products, such as ISO/IEC 25059:2023 (will be replaced by ISO/IEC AWI 25059 Software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Quality model for AI systems) need to be extended, e.g. , to meet the requirements of the AI Act.”
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 2×
“It can refer to the functioning of the entire model ( global xAI ) or to the explanation of the background of a specific decision ( local xAI ) [47–49].”
Research works printed but not linked: 6
- Oveisi, 2024, Adv. Stand Appl Sci, 2, 4
- Gaur, 2024, AI Mag, 45, 139
- Sanneman, 2022, Int J Human–Comput Inter, 38, 1772
- Weaver R, McDermid J and Kelly T. Software safety arguments: Towards a systematic categorisation of evidence. In: International System Safety Conference, Denver, CO 2002.
- Zinkhan, 2023, Eng Proc, 39, 41
- Klüver C, Werner C, Nowara P, Castel B and Israel R. Self-enforcing networks for monitoring safety-critical systems: A prototype development. In: Klüver C, & Klüver J (eds.) New algorithms for practical problems: Springer Vieweg, 2025 (in German).
Cited by
Fitas, R. (2025) · IEEE Access · doi:10.1109/access.2025.3598909 · Cited by 20
No safe AI standardization in sight at present: an investigation into ISO/IEC TR 5469
Locherer, M., Kordovan, M., Buxbaum, B., & Kever, T. (2025) · International Conference on System Reliability and Safety (ICSRS) · doi:10.1109/icsrs68021.2025.11422167 · Cited by 5
“As noted in [43, Sec.”
Functional Safety Architectural Patterns for AI-Based Critical Systems
Agirre, I., Yarza, I., Abella, J., & Caro, M. (2025) · ACM Transactions on Cyber-Physical Systems · doi:10.1145/3769121 · Cited by 1
A reinforcement-learning-based fall mitigation system for stair-traversing service robots
Ong, J. H., Yang, Z., Samarakoon, S. M. B. P., & Elara, M. R. (2026) · Results in Engineering · doi:10.1016/j.rineng.2026.110413 · Cited by 1
Agoi, M. A., Emmanuel Taiwo, A. G. O. I., Oluwanifemi Opeyemi, A. G. O. I., & Ajaga, S. O. (2026) · International Journal of Applied Mathematics Computational Science and Systems Engineering · doi:10.37394/232026.2026.8.9
Metadata
- Authors
- Klüver, C., Greisbach, A., Kindermann, M., & Püttmann, B.
- Affiliations
- Fuchs (Germany) (DE)
- TÜV Nord (Germany) (DE)
- Venue
- Security and Safety
- Publisher
- EDP Sciences
- Published
- 2024
- Volume
- 3
- Pages
- 2024020
- ISSN
- 2826-1275 (ISSN-L)
- OpenAlex
- W4403935496
- Semantic Scholar
- de0d20afbad490814c19722a052f2d954d7a080d
- Cited by
- 8
- References
- 68
- Citations by year
- 2026: 4 · 2025: 4
- FWCI
- 2.08
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Security and Safety (publishedVersion, cc-by)
- Springer Link (Chiba Institute of Technology) (submittedVersion)
- Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Hardware and Architecture › Real-Time Systems Scheduling
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- functional safety, safety-critical systems, artificial intelligence algorithms, explainable AI, requirements models, safety integrity level, developer perspectives, safety standards and regulations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective. https://app.cochat.ai/papers/10.1051/sands/2024020
- The paper
- Klüver, C., Greisbach, A., Kindermann, M., & Püttmann, B. (2024). A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective. Security and Safety. https://doi.org/10.1051/sands/2024020
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.