Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index; 1 of the 11 sources its claims cite were checked for support.

Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation

Open access
Hagiwara, Yuki, Fitch, Katherine, Trapp, Mario · 2026 · Scientific Reports

Abstract

Ensuring the safety of Artificial Intelligence-enabled Computer-Aided Diagnosis systems is critical because diagnostic errors can have serious consequences for patient care. However, existing regulatory and risk management frameworks often do not sufficiently address the complex socio-technical interactions between clinicians and Artificial Intelligent systems, leaving key human-centered safety challenges underexplored. This paper presents a systematic and human-centered approach to deriving safety design guidelines for clinician-Artificial Intelligence interaction in Computer-Aided Diagnosis systems using System-Theoretic Process Analysis. Through this analysis, we identify critical hazards associated with clinician–Artificial Intelligence collaboration, including automation bias on system recommendations, and misinterpretation of explanations. Based on the identified unsafe control actions, we formulate a set of actionable and traceable safety design guidelines that promote transparency, coherent explanations, and calibrated trust in Artificial Intelligence-assisted decision-making. To bridge safety analysis and system design, the proposed guidelines are operationalized within a Computer-Aided Diagnosis interaction framework. The framework includes a safety-oriented Graphical User Interface that integrates multiple explanation methods and interactive mechanisms to promote clinician engagement. Furthermore, we introduce a safety-oriented evaluation approach that uses consistency across multiple explanation methods as a quantitative indicator of potentially unreliable or ambiguous explanations. By linking System-Theoretic Process Analysis, interaction design, and explainability evaluation, this work provides a unified and reusable framework for improving the safety and reliability of Artificial Intelligence-driven Computer-Aided Diagnosis systems.

Overview

CoChat Summary

TL;DR

This paper applies System-Theoretic Process Analysis (STPA) to clinician–AI interaction in AI-enabled computer-aided diagnosis (CADx), deriving six safety design guidelines and operationalizing them in a prototype interface. It also proposes explanation consistency across multiple XAI methods as a quantitative safety signal, evaluated on a dermoscopic dataset.

Key findings

  • STPA identified 12 unsafe control actions (UCAs) and 22 loss scenarios, leading to six safety design guidelines (G1–G6) covering explanation clarity, transparency, interactive explanation, calibrated trust, XAI consistency, and cognitive load.
  • In the evaluation, gradient-based explanations had low spatial consistency: mean IoU = 0.13, variance = 0.01, while concept-based explanations had higher consistency: mean cosine similarity = 0.70, variance = 0.15, with a subset of samples below 0.50.
  • A representative atypical nevus case had high CADx confidence but low gradient-based consistency (consistency score = 0.363), which the authors interpret as prompting additional clinician review rather than sole reliance on the AI recommendation.
  • The authors conclude that explanation disagreement can serve as a practical signal for detecting potentially unreliable or ambiguous explanations, supporting transparency (G2), calibrated trust (G4), and coherence across XAI outputs (G5).

Methods

  • The paper uses STPA to model clinician–AI interaction in CADx, defining losses, system-level hazards, system-level constraints, a control structure, UCAs, and loss scenarios.
  • The empirical evaluation uses the PH2 dermoscopic image dataset with 40 test images and AlexNet as the base diagnostic model; XAI methods include Grad-CAM, integrated gradients, TCAV, and SHAP projected onto a shared concept space.
  • Consistency is quantified with Intersection over Union (IoU) for gradient-based attribution maps and cosine similarity for concept-based explanations, reporting mean and variance across samples, plus comparative and scenario-based safety validation.

Limitations

  • The paper states that evaluation is based on simulated scenarios and retrospective analysis of explanation consistency rather than real-world clinician interaction, and that the interface’s effects on clinician trust calibration and cognitive workload have not yet been empirically evaluated.
  • The paper states that explanation consistency has inherent limits: multiple XAI methods may agree while reflecting the same incorrect or biased model behavior, and disagreement does not necessarily mean the diagnosis is erroneous.
  • The paper states that the dermoscopic CADx focus may limit generalizability to other clinical domains and that STPA requires detailed knowledge of system behavior and clinical workflows; evident from the text, the evaluation uses only 40 test images, a single dataset, and a single base model, with formal usability evaluation left as future work.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41598-026-64180-w

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Four-step process of STPA.
Fig. 1 Four-step process of STPA.CC BY · PubMed Central
Fig. 2 Control structure for CADx-enabled diagnosis centered on the clinician–AI interaction. The clinician, CADx system, and its explainable module form the primary information and feedback loops; arrows denote information/decision flow between these controllers. Medical imaging equipment provides diagnostic inputs; treatment delivery and the patient are shown for context only. Dotted elements indicate non-focus. Responsibilities R1–R6 are labeled.
Fig. 2 Control structure for CADx-enabled diagnosis centered on the clinician–AI interaction. The clinician, CADx system, and its explainable module form the primary information and feedback loops; arrows denote information/decision flow between these controllers. Medical imaging equipment provides diagnostic inputs; treatment delivery and the patient are shown for context only. Dotted elements indicate non-focus. Responsibilities R1–R6 are labeled.CC BY · PubMed Central
Fig. 3 Proof-of-concept implementation of the proposed safety-oriented CADx interaction framework, illustrating how the derived design guidelines are operationalized within the user interface.
Fig. 3 Proof-of-concept implementation of the proposed safety-oriented CADx interaction framework, illustrating how the derived design guidelines are operationalized within the user interface.CC BY · PubMed Central

Verification

Claims the paper makes

1 Valid · 0 Not valid · 5 Not checked · 48 Own results and interpretations

28 more claims

Create a free account to see all 54 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 9
  • Probably not a substantial claim. 14

Create a free account to see these claims.

Citations

Cites

Show all 13 references

Research works printed but not linked: 7

  1. European Parliament and Council of the European Union: Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. Official Journal of the European Union, L 117, pp. 1–175 (2017). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0745
  2. European Parliament and Council of the European Union: Regulation (EU) 2017/746 of the European Parliament and of the Council of 5 April 2017 on in vitro diagnostic medical devices and repealing Directive 98/79/EC and Commission Decision 2010/227/EU. Official Journal of the European Union, L 117, pp. 176–332 (2017). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0746
  3. Leveson, N. G. A systems-theoretic approach to safety and hazard analysis. In Proceedings of the 7th International Workshop on Systems Engineering (2004).
  4. Krizhevsky, A., Sutskever, I., & Hinton, G.E. Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, 25. Curran Associates, Inc., Red Hook, NY, USA (2012). https://neurips.cc
  5. Sundararajan, M., Taly, A., & Yan, Q. Axiomatic attribution for deep networks. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning (ICML 2017), 70, pp. 3319–3328. Proceedings of Machine Learning Research, Sydney, NSW, Australia (2017). PMLR. http://proceedings.mlr.press/v70/sundararajan17a.html
  6. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viégas, F.B., & Sayres, R. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In: Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, 80, pp. 2668–2677. PMLR, Stockholm, Sweden (2018). https://proceedings.mlr.press/v80/kim18d.html
  7. Lundberg, S.M., & Lee, S.-I. A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 4768–4777. Curran Associates Inc., Red Hook, NY, USA (2017). https://neurips.cc

Metadata

Authors
Hagiwara, Y., Fitch, K., & Trapp, M.
Affiliations
Fraunhofer Institute for Cognitive Systems (DE)
Corresponding author
Hagiwara, Yuki
Venue
Scientific Reports
Published
2026-07-29
Volume
16
Issue
1
ISSN
2045-2322 (ISSN-L)
OpenAlex
W7171689405
Cited by
0
References
13
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion, cc-by)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft)
Funders
Fraunhofer-Institut für Kognitive Systeme IKS
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
computer-aided diagnosis, AI safety, STPA, explainable AI, automation bias, graphical user interface, explanation consistency, human-centered AI
MeSH terms
Artificial Intelligence, Diagnosis, Computer-Assisted, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation. https://app.cochat.ai/papers/10.1038/s41598-026-64180-w
The paper
Hagiwara, Y., Fitch, K., & Trapp, M. (2026). Safety design guidelines for clinician–AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation. Scientific Reports. https://doi.org/10.1038/s41598-026-64180-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.