The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index; 1 of the 11 sources its claims cite were checked for support.
Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation
Open accessAbstract
Ensuring the safety of Artificial Intelligence-enabled Computer-Aided Diagnosis systems is critical because diagnostic errors can have serious consequences for patient care. However, existing regulatory and risk management frameworks often do not sufficiently address the complex socio-technical interactions between clinicians and Artificial Intelligent systems, leaving key human-centered safety challenges underexplored. This paper presents a systematic and human-centered approach to deriving safety design guidelines for clinician-Artificial Intelligence interaction in Computer-Aided Diagnosis systems using System-Theoretic Process Analysis. Through this analysis, we identify critical hazards associated with clinician–Artificial Intelligence collaboration, including automation bias on system recommendations, and misinterpretation of explanations. Based on the identified unsafe control actions, we formulate a set of actionable and traceable safety design guidelines that promote transparency, coherent explanations, and calibrated trust in Artificial Intelligence-assisted decision-making. To bridge safety analysis and system design, the proposed guidelines are operationalized within a Computer-Aided Diagnosis interaction framework. The framework includes a safety-oriented Graphical User Interface that integrates multiple explanation methods and interactive mechanisms to promote clinician engagement. Furthermore, we introduce a safety-oriented evaluation approach that uses consistency across multiple explanation methods as a quantitative indicator of potentially unreliable or ambiguous explanations. By linking System-Theoretic Process Analysis, interaction design, and explainability evaluation, this work provides a unified and reusable framework for improving the safety and reliability of Artificial Intelligence-driven Computer-Aided Diagnosis systems.
Overview
CoChat Summary
TL;DR
This paper applies System-Theoretic Process Analysis (STPA) to clinician–AI interaction in AI-enabled computer-aided diagnosis (CADx), deriving six safety design guidelines and operationalizing them in a prototype interface. It also proposes explanation consistency across multiple XAI methods as a quantitative safety signal, evaluated on a dermoscopic dataset.
Key findings
- STPA identified 12 unsafe control actions (UCAs) and 22 loss scenarios, leading to six safety design guidelines (G1–G6) covering explanation clarity, transparency, interactive explanation, calibrated trust, XAI consistency, and cognitive load.
- In the evaluation, gradient-based explanations had low spatial consistency: mean IoU = 0.13, variance = 0.01, while concept-based explanations had higher consistency: mean cosine similarity = 0.70, variance = 0.15, with a subset of samples below 0.50.
- A representative atypical nevus case had high CADx confidence but low gradient-based consistency (consistency score = 0.363), which the authors interpret as prompting additional clinician review rather than sole reliance on the AI recommendation.
- The authors conclude that explanation disagreement can serve as a practical signal for detecting potentially unreliable or ambiguous explanations, supporting transparency (G2), calibrated trust (G4), and coherence across XAI outputs (G5).
Methods
- The paper uses STPA to model clinician–AI interaction in CADx, defining losses, system-level hazards, system-level constraints, a control structure, UCAs, and loss scenarios.
- The empirical evaluation uses the PH2 dermoscopic image dataset with 40 test images and AlexNet as the base diagnostic model; XAI methods include Grad-CAM, integrated gradients, TCAV, and SHAP projected onto a shared concept space.
- Consistency is quantified with Intersection over Union (IoU) for gradient-based attribution maps and cosine similarity for concept-based explanations, reporting mean and variance across samples, plus comparative and scenario-based safety validation.
Limitations
- The paper states that evaluation is based on simulated scenarios and retrospective analysis of explanation consistency rather than real-world clinician interaction, and that the interface’s effects on clinician trust calibration and cognitive workload have not yet been empirically evaluated.
- The paper states that explanation consistency has inherent limits: multiple XAI methods may agree while reflecting the same incorrect or biased model behavior, and disagreement does not necessarily mean the diagnosis is erroneous.
- The paper states that the dermoscopic CADx focus may limit generalizability to other clinical domains and that STPA requires detailed knowledge of system behavior and clinical workflows; evident from the text, the evaluation uses only 40 test images, a single dataset, and a single base model, with formal usability evaluation left as future work.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
1 Valid · 0 Not valid · 5 Not checked · 48 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Manual inspection of these low consistency cases revealed that disagreement often occurs in samples with ambiguous or complex visual patterns, where different methods focus on distinct regions or concepts.”
§ Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation > Evaluation and validation > Comparative study
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“UCA-8 reflects failures in XAI, where incomplete or unclear explanations may lead to misinterpretation of AI reasoning.”
§ Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation > System-theoretic process analysis > Unsafe control actions
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The mean IoU across the test set was 0.13, indicating low spatial consistency between gradient-based methods.”
§ Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation > Evaluation and validation > Safety-oriented evaluation of explanation consistency
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through this analysis, we identify critical hazards associated with clinician–Artificial Intelligence collaboration, including automation bias on system recommendations, and misinterpretation of explanations.”
§ Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, from this analysis, recurring patterns are observed, particularly in relation to explanation clarity, system transparency, and the calibration of clinician trust.”
§ Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation > System-theoretic process analysis > Loss scenarios
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
28 more claims
Create a free account to see all 54 claims and every source verdict.
- It may state two separate claims as one. 9
- Probably not a substantial claim. 14
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 1×
“However, several studies have highlighted that individual XAI methods can produce inconsistent or conflicting explanations, which may lead to confusion or misinterpretation by clinicians 20 .”
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
PH2 - A dermoscopic image database for research and benchmarking
Mendonca, T., Ferreira, P. M., Marques, J. S., Marcal, A. R. S., & Rozeira, J. (2013) · 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2013.6610779 · Cited by 1,009
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256
Standard IEC 62304 - Medical device software - Software lifecycle processes
Jordan, P. (2006) · IET Seminar on Software for Medical Devices · doi:10.1049/ic:20060141 · Cited by 244
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Hossain, M. I., Zamzmi, G., Mouton, P. R., Salekin, M. S., Sun, Y., & Goldgof, D. B. (2023) · ACM Computing Surveys · doi:10.1145/3637487 · Cited by 183 · cited 1×
“Methods such as saliency-based approaches, feature attribution techniques, and concept-based explanations have been proposed to make AI decisions more interpretable 19 .”
(2004) · Discovering Knowledge in Data · doi:10.1002/0471687545.ch1 · Cited by 180
Gao, Y., Geras, K. J., Lewin, A. A., & Moy, L. P. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20392 · Cited by 140
Research works printed but not linked: 7
- European Parliament and Council of the European Union: Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. Official Journal of the European Union, L 117, pp. 1–175 (2017). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0745
- European Parliament and Council of the European Union: Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in vitro diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU. Official Journal of the European Union, L 117, pp. 176–332 (2017). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0746
- Leveson, N. G. A systems-theoretic approach to safety and hazard analysis. In Proceedings of the 7th International Workshop on Systems Engineering (2004).
- Krizhevsky, A., Sutskever, I., & Hinton, G.E. Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, 25. Curran Associates, Inc., Red Hook, NY, USA (2012). https://neurips.cc
- Sundararajan, M., Taly, A., & Yan, Q. Axiomatic attribution for deep networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning (ICML 2017), 70, pp. 3319–3328. Proceedings of Machine Learning Research, Sydney, NSW, Australia (2017). PMLR. http://proceedings.mlr.press/v70/sundararajan17a.html
- Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viégas, F.B., & Sayres, R. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In: Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, 80, pp. 2668–2677. PMLR, Stockholm, Sweden (2018). https://proceedings.mlr.press/v80/kim18d.html
- Lundberg, S.M., & Lee, S.-I. A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://neurips.cc
Metadata
- Authors
- Hagiwara, Y., Fitch, K., & Trapp, M.
- Affiliations
- Fraunhofer Institute for Cognitive Systems (DE)
- Corresponding author
- Hagiwara, Yuki
- Venue
- Scientific Reports
- Published
- 2026-07-29
- Volume
- 16
- Issue
- 1
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC13421477
- OpenAlex
- W7171689405
- Semantic Scholar
- 8fb2aa90a830c8659f425a34d5f560c2e125754f
- Cited by
- 0
- References
- 13
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion, cc-by)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft)
- Funders
- Fraunhofer-Institut für Kognitive Systeme IKS
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- computer-aided diagnosis, AI safety, STPA, explainable AI, automation bias, graphical user interface, explanation consistency, human-centered AI
- MeSH terms
- Artificial Intelligence, Diagnosis, Computer-Assisted, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation. https://app.cochat.ai/papers/10.1038/s41598-026-64180-w
- The paper
- Hagiwara, Y., Fitch, K., & Trapp, M. (2026). Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation. Scientific Reports. https://doi.org/10.1038/s41598-026-64180-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.