Cited by 507Artificial Intelligence in Healthcare and EducationEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)

The DOI matches its Crossref record; 67 of the 71 references it cites were found in a registry or an index; 2 of the 62 sources its claims cite were checked for support.

Ethics and governance of trustworthy medical artificial intelligence

Open access
Zhang, Jie, Zhang, Zong-ming·2023·BMC Medical Informatics and Decision Making

Abstract

Background: The growing application of artificial intelligence (AI) in healthcare has brought technological breakthroughs to traditional diagnosis and treatment, but it is accompanied by many risks and challenges. These adverse effects are also seen as ethical issues and affect trustworthiness in medical AI and need to be managed through identification, prognosis and monitoring. Methods: We adopted a multidisciplinary approach and summarized five subjects that influence the trustworthiness of medical AI: data quality, algorithmic bias, opacity, safety and security, and responsibility attribution, and discussed these factors from the perspectives of technology, law, and healthcare stakeholders and institutions. The ethical framework of ethical values-ethical principles-ethical norms is used to propose corresponding ethical governance countermeasures for trustworthy medical AI from the ethical, legal, and regulatory aspects. Results: Medical data are primarily unstructured, lacking uniform and standardized annotation, and data quality will directly affect the quality of medical AI algorithm models. Algorithmic bias can affect AI clinical predictions and exacerbate health disparities. The opacity of algorithms affects patients’ and doctors’ trust in medical AI, and algorithmic errors or security vulnerabilities can pose significant risks and harm to patients. The involvement of medical AI in clinical practices may threaten doctors ‘and patients’ autonomy and dignity. When accidents occur with medical AI, the responsibility attribution is not clear. All these factors affect people’s trust in medical AI. Conclusions: In order to make medical AI trustworthy, at the ethical level, the ethical value orientation of promoting human health should first and foremost be considered as the top-level design. At the legal level, current medical AI does not have moral status and humans remain the duty bearers. At the regulatory level, strengthening data quality management, improving algorithm transparency and traceability to reduce algorithm bias, and regulating and reviewing the whole process of the AI industry to control risks are proposed. It is also necessary to encourage multiple parties to discuss and assess AI risks and social impacts, and to strengthen international cooperation and communication.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This paper argues that trustworthy medical AI depends on addressing five ethical/governance factors: data quality, algorithmic bias, opacity, safety and security, and responsibility attribution. It proposes governance through an ethical values–principles–norms framework, with humans remaining the responsible actors and with stronger data management, transparency/traceability, whole-process review, multi-party participation, and international cooperation.

Key findings

  • The paper identifies five factors affecting trustworthiness in medical AI: data quality, algorithmic bias, opacity, safety and security, and responsibility attribution. Poor data quality, including unstructured and inconsistently annotated medical data, directly affects algorithm quality; algorithmic bias can exacerbate health disparities; opacity affects patient and clinician trust; safety/security failures can harm patients; and responsibility attribution remains unclear when accidents occur.
  • It reports illustrative harms and market/regulatory figures: the health AI market was estimated to grow tenfold from 2020 to 2026, reaching $45.2 billion in 2026; U.S. surgical robots were linked to at least 1,391 harm-causing incidents and 144 deaths between 2000 and 2013; 15% of global data breaches came from healthcare in 2017; and as of 2020 the FDA had approved 222 AI medical device products while Europe had approved 240 with European conformity certification.
  • It cites algorithmic bias examples, including an AI for melanoma diagnosis being difficult to apply to darker races because of few medical images of darker races, and a health decision algorithm incorrectly using health care costs rather than disease to represent health need, leading it to conclude Black people were healthier despite being much sicker in the datasets.
  • It argues that current medical AI lacks moral status and should be treated as a tool; humans remain the duty bearers. Governance proposals include strengthening data quality management, promoting data sharing with privacy protections, reducing algorithmic bias, increasing transparency and traceability, regulating and reviewing the whole AI lifecycle, encouraging multi-party risk assessment, and strengthening international cooperation.

Methods

  • The paper uses a multidisciplinary approach and summarizes five subjects influencing trustworthy medical AI: data quality, algorithmic bias, opacity, safety and security, and responsibility attribution.
  • It discusses these factors from technology, law, and healthcare stakeholder/institution perspectives, analyzing both the design level (technical reliability) and application level (human impact).
  • It applies an ethical governance framework of ethical values–ethical principles–ethical norms to propose countermeasures from ethical, legal, and regulatory aspects. No empirical sample, dataset, or statistical analysis is reported.

Limitations

  • The paper does not state limitations in a dedicated limitations section.
  • Evident from the text: it is a conceptual/normative review rather than an empirical study, with no primary data collection, systematic review protocol, or formal evaluation of the proposed governance measures.
  • Evident from the text: many supporting examples are drawn from news reports, regulatory actions, and prior literature rather than from a systematic evidence synthesis, and the governance proposals are not tested for implementation feasibility or effectiveness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s12911-023-02103-9

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Factors affecting trustworthy medical AI
Fig. 1 Factors affecting trustworthy medical AICC BY · PubMed Central
Fig. 2 Ethical governance system
Fig. 2 Ethical governance systemCC BY · PubMed Central

Verification

Claims the paper makes

51 Not checked yet · 43 Own interpretation · 36 No basis given · 1 Supported

Verify all 60 references — free account

Show all 27 claims

  • Probably not a substantial claim. 71
  • It may state two separate claims as one. 66
104 of 104
  • The paper relies on this without support.

    When accidents occur with medical AI, the responsibility attribution is not clear.

    “When accidents occur with medical AI, the responsibility attribution is not clear”
  • The paper relies on this without support.

    What is optimal for one patient in the same clinical situation may not be optimal for other patients.

    “What is optimal for one patient in the same clinical situation may not be so for other patients.”
  • The authors state this as an assumption.

    The purpose of AI development is to serve humans better, not to subject humans to AI.

    “We believe that the purpose of AI development is to serve humans better, not to subject humans to AI.”
  • The authors state this as an assumption.

    Technology should serve humans, not replace human talent and creativity.

    “Technology should serve humans, not replace human talent and creativity.”
  • The paper relies on this without support.

    At present, there is no unified standardized quality standard, access system, evaluation system and guarantee system for the application of AI in the medical field, and the related policy and regulation system has not been completely established.

    “At present, there is no unified standardized quality standard, access system, evaluation system and guarantee system for the application of AI in the medical field, and the related policy and regulation system has not been completely established yet.”
  • The paper relies on this without support.

    The utilization rate of big data in healthcare is low.

    “The utilization rate of big data in healthcare is low.”
  • The paper relies on this without support.

    Most hospital data are unstructured and cannot bring out the value of 'big data'.

    “Although the data in hospitals are enormous, most of them are unstructured data, which cannot bring out the value of “big data”.”
  • The paper relies on this without support.

    Many hospitals have not yet established a unified data management system, which is not conducive to unified analysis of data and impacts the application of AI technology in the medical field.

    “Many hospitals have not yet established a unified data management system, which is not conducive to the unified analysis of data and impacts the application of AI technology in the medical field.”
  • The paper relies on this without support.

    Most scholars believe that AI robots at this stage can only belong to the category of tools.

    “Most scholars believe that AI robots at this stage can only belong to the category of tools yet.”
  • The paper relies on this without support.

    It is the current consensus in the AI industry that AI cannot be a responsible subject.

    “It is also the current consensus in the AI industry.”
  • The paper relies on this without support.

    There is limited transparency in the black box of algorithms, whose inherent logic is hidden even from developers.

    “there is limited transparency in the black box of algorithms, whose inherent logic is hidden even from developers”
  • The paper relies on this without support.

    Clinicians are not always able to explain their inferences perfectly, as they may make decisions based more on experience and intuition than on clear medical criteria.

    “clinicians are also not always able to explain their inferences perfectly, as they may make decisions based more on experience and intuition than on clear medical criteria.”
  • The paper relies on this without support.

    Medical AI will belong to the category of medical devices for a considerable period of time, and its main function is to assist doctors in diagnosis and treatment.

    “medical AI will belong to the category of medical devices for a considerable period of time, and its main function is to assist doctors in diagnosis and treatment.”
  • The paper relies on this without support.

    Countries generally require or encourage medical device applicants to submit appropriate scientific research evidence to explain the scientific process and verify the safety and efficacy of the device at all stages of registration.

    “Countries generally require or encourage medical device applicants to submit appropriate scientific research evidence to explain the scientific process and verify the safety and efficacy of the device at all stages of registration-including premarket approval and postmarketing studies.”
  • The paper relies on this without support.

    International cooperation faces many obstacles, such as cultural and legal systems differences in each country, which may lead to different attitudes and positions in the face of medical AI.

    “international cooperation also faces many obstacles, such as cultural and legal systems differences in each country, which may lead to different attitudes and positions in the face of medical AI.”
  • The paper relies on this without support.

    In the past, scientific and technological work was often the result of scientists setting up projects, the relevant departments or enterprises giving money, the government approving them, the public unaccountably affected, and humanities and social science experts cleaning up the mess.

    “In the past, scientific and technological work was often the result of scientists setting up projects, the relevant departments or enterprises giving money, the government approving them, the public unaccountably affected, and humanities and social science experts cleaning up the mess.”
16 of 16

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 71 references

Research works printed but not linked: 14

  1. AMA (American Medical Association). Report of the Council on Long Range Planning and Development. 2018. https://www.ama-assn.org/system/files/2018-11/a18-clrpd-reports.pdf. Accessed 23 Jul 2022.
  2. Wang H, Meng XF, Wang Q, et al. Strategy for management and quality control of datasets in artificial intelligence medical device. China Med Devices. 2018;33(12):1–5.
  3. Xu Y, Ma XM, Yue Q, et al. Ethics of lung cancer image data and artificial intelligence. Med Soc. 2021;34(5):100–104
  4. Price W, Nicholson II. Medical AI and contextual bias. Harv J Law Technol. 2019;33:65–116.
  5. Liu LL, He YM, Liu XD. A survey on patients perceptions and trust in artificial intelligence healthcare. Chin Med Eth. 2019;32(8):986–90. https://doi.org/10.12026/j.issn.1001-8565.2019.08.07. · DOI
  6. Feldman R, Aldana E, Stein K. Artificial intelligence in the health care space: how we can trust what we cannot know. Stanf Law Policy Rev. 2019;30:399–419.
  7. Kamensky S. Artificial intelligence and technology in health care: overview and possible legal implications. DePaul J Health Care Law. 2020;21(3):1–13.
  8. Gan SP. How can robots have rights. Eth Res. 2017;3:126–30. https://doi.org/10.3969/j.issn.1671-9115.2017.03.025. · DOI
  9. Jessica SA. From Jeopardy to Jaundice: the medical liability implications of Dr. Watson and other artificial intelligence systems. La Law Rev. 2013;73:1049.
  10. Chung J, Zink A. Hey Watson, can i sue you for malpractice? Examining the liability of artificial intelligence in medicine. Asia Pac J Health Law Eth. 2018;11(2):51–80.
  11. Weaver JF, Zheng ZF. How to sue a robot: liability and AI. Law Econ. 2019;1:140–60.
  12. Zhang L, Chen P. Critique of the theory of robot “personality” and the legal regulation of artificial intelligent objects. Academia. 2018;12:53–75. https://doi.org/10.3969/j.issn.1002-1698.2018.12.006. · DOI
  13. IEEE. Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, First Edition. 2016. https://ethicsinaction.ieee.org/. Accessed 2 Dec 2022.
  14. Macnish K, Gauttier S. A pre-occupation with possession: the (non-) ownership of personal data. Big data and democracy. Edinburgh: Edinburgh University Press; 2020. p. 42–56.

Cited by

Sorted by citations

Metadata

Authors
Zhang, J., & Zhang, Z.-M.
Affiliations
Nanjing University of Chinese Medicine (CN)
Venue
BMC Medical Informatics and Decision Making
Published
2023-01-13
Volume
23
Issue
1
Pages
7-7
ISSN
1472-6947 (ISSN-L)
OpenAlex
W4315880904
Cited by
507
References
71
Citations by year
2026: 171 · 2025: 169 · 2024: 131 · 2023: 36
FWCI
20.19
Type
journal article
Language
en
Open access
Locations
  • BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Social Science Fund of China (18ZDA322)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
artificial intelligence in healthcare, trustworthy AI, AI ethics, AI governance, algorithmic bias, data quality, algorithmic opacity, algorithmic transparency, attribution of responsibility, health disparities, patient autonomy, AI regulation, ethical frameworks, legal status of AI, international cooperation
MeSH terms
Data Management, Algorithms, Artificial Intelligence, Delivery of Health Care, Humans, Prognosis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review