The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 10 of the 28 sources its claims cite were checked for support.
Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study
Open accessAbstract
BACKGROUND: Despite the increasing implementation of artificial intelligence (AI) and machine learning (ML) technologies in healthcare, their long-term safety, effectiveness, and equity remain compromised by a lack of sustained oversight. This study explores the phenomenon of a "responsibility vacuum" in AI governance, wherein maintenance and monitoring tasks are poorly defined, inconsistently performed, and undervalued across healthcare systems. METHODS: We conducted semi-structured interviews with 21 experts involved in AI implementation in healthcare, including clinicians, clinical informaticists, computer scientists, and legal/policy professionals. Participants were recruited through purposive and snowball sampling. Interviews were transcribed, coded, and analyzed using abductive qualitative methods to identify themes related to maintenance practices, institutional incentives, and responsibility attribution. RESULTS: Participants widely recognized that AI models degrade over time due to factors such as data drift, changes in clinical practice, and poor generalizability. However, monitoring practices remain ad hoc and fragmented, with few institutions investing in structured oversight infrastructure. This "responsibility vacuum" is perpetuated by institutional incentives favoring rapid innovation and strategic ignorance of AI failures. Despite these challenges, some participants described grassroots efforts to monitor and maintain AI systems, drawing inspiration from fields such as radiology, laboratory medicine, and transportation safety. CONCLUSIONS: Our findings suggest that institutional and cultural forces in healthcare deprioritize the maintenance of AI tools, creating a governance gap that may lead to patient harm and inequitable outcomes. Addressing this responsibility vacuum will require formalized accountability structures, interdisciplinary collaboration, and policy reforms that center long-term safety and equity. Without such changes, AI/ML technologies designed to improve patient health may introduce new forms of harm, ultimately eroding trust in AI and machine learning for healthcare.
Overview
CoChat Summary
TL;DR
This qualitative study argues that healthcare AI/ML suffers from a “responsibility vacuum”: models are known to degrade, but monitoring and maintenance tasks are unclaimed, undervalued, and inconsistently performed. The authors find that institutional incentives favor rapid innovation and strategic ignorance, while grassroots actors creatively improvise maintenance practices drawn from other safety-critical fields.
Key findings
- Participants widely recognized that AI/ML models degrade over time due to data drift, changes in clinical practice, poor generalizability, and related factors, but monitoring remained ad hoc and fragmented, with few institutions investing in structured oversight infrastructure.
- The “responsibility vacuum” is perpetuated by institutional incentives favoring rapid innovation and strategic ignorance of AI failures; participants described a culture of “move fast and break things” and fear of being “left behind.”
- Responsibility for monitoring was widely diffused and deflected: machine learning researchers, hospital IT teams, and clinicians were each described as poorly suited or lacking capacity to take ownership, and no existing actor was seen as a natural fit.
- Despite the lack of formal structures, some participants described grassroots efforts to monitor and maintain AI systems, drawing analogies from radiology, laboratory medicine, and transportation safety, including aviation, car service manuals, and autonomous vehicles.
Methods
- Semi-structured interviews with 21 experts involved in AI implementation in healthcare, including clinical informaticists, clinicians, computer scientists, and legal/policy professionals, recruited through purposive and snowball sampling.
- Interviews lasted 30 min to 1 h, were professionally transcribed, coded by 2 researchers using ATLAS.ti 24, and analyzed using abductive qualitative methods; interviews stopped at thematic saturation.
- Participants were asked about how AI/ML factors into their work, promising and challenging applications, and issues including safety, governance, maintenance, and bias.
Limitations
- The paper states that this is a small, qualitative interview study with limited size and scope, so perspectives may not be generalizable outside the sample; it relies on self-reported experiences that may be shaped by participants’ organizational positions and willingness to speak openly about failures or institutional shortcomings.
- Evident from the text: all participants were engaged in academic research in some capacity, many were clinicians or legal/regulatory practitioners, and the sample was predominantly U.S.-based, which may narrow the range of institutional and regulatory perspectives captured.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 0 Not valid · 17 Not checked · 80 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“a widely-used algorithm that helps to triage scarce medical resources was found to reduce the number of Black patients who should have been eligible for extra services by more than half when the model used predicted health costs and a proxy for health needs [16].”
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Results > AI and problems of drift in healthcare
What the sources say
- Supportedcites [16] Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. S. 2019;366:447–53. source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Data for healthcare modeling is also heavily skewed by geography.”
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Results > AI and problems of drift in healthcare
What the sources say
- Supportedcites [15] Kaushal A, Altman R, Langlotz C. Geographic distribution of us cohorts used to train deep learning algorithms. JAMA. 2020;324:1212–3. source
The Results report that 71% of studies with geographically identifiable US training cohorts used data from California, Massachusetts, or New York, while 34 states contributed no cohorts. The Discussion describes these cohorts as disproportionately concentrated in those three states. The evidence concerns US patient data used to train clinical deep-learning algorithms. (LLM, conf=0.89)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“the AI/ML field also faces a widely-recognized “AI chasm”—a significant gap between the development of AI/ML models and their successful integration into clinical practice [1].”
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Introduction
What the sources say
- Supportedcites [1] Aristidou A, Jena R, Topol EJ. Bridging the chasm between AI and clinical implementation. The Lancet. 2022;399:620. source
On p. 1, the paper calls the gap between advances in health-care AI and implementation in health systems a “chasm.” On p. 2, it describes barriers to wider implementation in routine patient care, supporting the citing sentence’s paraphrase. (LLM, conf=0.92)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Understanding persistence and degradation is perhaps particularly critical for ML models, where dataset shift should be expected to cause declines in performance when the data used to train a model begins to look different from the data in a new context [8].”
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Introduction
What the sources say
- Supportedcites [8] Finlayson SG, Subbaswamy A, Singh K, Bowers J, Kupke A, Zittrain J, et al. The clinician and dataset shift in artificial intelligence. New England J Med. 2021;385:283–6. source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“researchers analyzing 5 years of peer-reviewed articles describing AI algorithms for diagnostic tasks in patient care found that 71% used data from California, Massachusetts, or New York.”
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Results > AI and problems of drift in healthcare
What the sources say
- Partly supportedcites [15] Kaushal A, Altman R, Langlotz C. Geographic distribution of us cohorts used to train deep learning algorithms. JAMA. 2020;324:1212–3. sourceExperimental — accuracy not yet measured
The Methods describe a review of articles published from 2015 through 2019 on image-based diagnostic deep-learning algorithms. The Results report that 40 of 56 studies with geographically identifiable US training cohorts (71%) used a cohort from California, Massachusetts, or New York. The claim omits that denominator: 71% does not apply to all 74 included studies. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
49 more claims · 2 with their cited sources checked
Create a free account to see all 107 claims and to verify the 5 references still open.
- It may state two separate claims as one. 38
- Probably not a substantial claim. 24
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Results > The invisible creativity of AI maintainers in practice
“This type of analogy was captured most succinctly by a computer scientist:”
§ Managing a “responsibility vacuum” in AI monitoring and governance in healthcare: a qualitative study > Results > The invisible creativity of AI maintainers in practice
“Every three months we examine data drift and we examine the performance of this model. - Clinical Informaticist.”
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Most famously, a widely-used algorithm that helps to triage scarce medical resources was found to reduce the number of Black patients who should have been eligible for extra services by more than half when the model used predicted health costs and a proxy for health needs [16].”
Theory Construction in Qualitative Research: From Grounded Theory to Abductive Analysis
Timmermans, S., & Tavory, I. (2012) · Sociological Theory · doi:10.1177/0735275112457914 · Cited by 4,349
Trust in Numbers: The Pursuit of Objectivity in Science and Public Life
Porter, T. M. (1995) · doi:10.1515/9781400821617 · Cited by 1,431
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“The harms that can arise from data drift are patterned, and generally fall upon groups with less representation in model training datasets, such as those with lower socio-economic status [14].”
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855 · cited 1×
“Understanding persistence and degradation is perhaps particularly critical for ML models, where dataset shift should be expected to cause declines in performance when the data used to train a model begins to look different from the data in a new context [8].”
Inclusion: The Politics of Difference in Medical Research
Epstein, S. (2007) · doi:10.7208/chicago/9780226213118.001.0001 · Cited by 792
Biomedicalization: Technoscientific Transformations of Health, Illness, and U.S. Biomedicine
Clarke, A. E., Shim, J. K., Mamo, L., Fosket, J. R., & Fishman, J. R. (2003) · American Sociological Review · doi:10.1177/000312240306800201 · Cited by 560 · cited 1×
“At the same time, modern biomedicine also introduces ambiguity and inertia as responsibility for decision-making is dispersed throughout a range of actors including regulatory agencies, clinical researchers, industry stakeholders, and healthcare institutions [21].”
The logic of strategic ignorance
McGoey, L. (2012) · The British Journal of Sociology · doi:10.1111/j.1468-4446.2012.01424.x · Cited by 526
Daniels, A. K. (1987) · Social Problems · doi:10.1525/sp.1987.34.5.03a00020 · Cited by 490
Regulatory objectivity and the generation and management of evidence in medicine
Cambrosio, A., Keating, P., Schlich, T., & Weisz, G. (2006) · Social Science & Medicine · doi:10.1016/j.socscimed.2005.12.007 · Cited by 258 · cited 1×
“As Cambrosio and colleagues theorized, the emergence of regulatory objectivity in biomedicine emphasizes procedural standardization and collective decision-making over individual expertise or scientific certainty [19].”
Research works printed but not linked: 1
- Vaughan D. Analogy, Cases, and Comparative Social Organization. In: R Swedberg, ed. Theorizing in Social Science: The Context of Discovery. Redwood City, CA: Stanford University Press; 2014;61–84.
Cited by
Bailo, P., Nittari, G., Pesel, G., Basello, E., Spasari, T., & Ricci, G. (2026) · Sci · doi:10.3390/sci8020036 · Cited by 22
“…EU AI Act risk-based controls) increasingly shape design requirements around transparency, data lineage, and non-discrimination, but operationalization in clinics depends on institutional capacity: who is responsible for drift detection, subgroup audits, and corrective action [25,35,40,41]?”
InfluentialBulić, L., Brlek, P., Hrvatin, N., Brenner, E., Škaro, V., Projić, P., Rogan, S. A., Bebek, M., Shah, P. S., & Primorac, D. (2026) · AI · doi:10.3390/ai7010011 · Cited by 17
“Without maintenance of AI tools, AI technologies designed to improve patient health may introduce new forms of harm, ultimately eroding trust in AI and machine learning for healthcare [80].”
Talwar, S., Alsaleh, H., Dernóczi-Polyák, A., & Alkahtani, R. (2026) · Journal of Knowledge Management · doi:10.1108/jkm-03-2026-0433
Chi, H., Liu, X., Yan, Y., Wang, L., Luo, P., Su, L., Jiang, L., Zhou, X., Huang, G., Jiang, Y., et al. (2026) · Genes & Diseases · doi:10.1016/j.gendis.2026.102480 · Cited by 1
AI Use in Dementia Care Requires Engagement with Patient and Caregiver Perspectives
Friend, J., Black, V., & Chiong, W. (2026) · The American Journal of Bioethics · doi:10.1080/15265161.2026.2715910
Metadata
- Authors
- Owens, K., Griffen, Z., & Damaraju, L.
- Affiliations
- NYU Langone Health (US)
- New York University (US)
- Corresponding author
- Owens, Kellie
- Venue
- BMC Health Services Research
- Published
- 2025-09-29
- Volume
- 25
- Issue
- 1
- Pages
- 1217-1217
- ISSN
- 1472-6963 (ISSN-L)
- PMC
- PMC12482494
- OpenAlex
- W4414628640
- Semantic Scholar
- 276964b598a4e4726c0cf675cc87a65c927759d6
- Cited by
- 19
- References
- 20
- Citations by year
- 2026: 19
- FWCI
- 4.53
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- BMC Health Services Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- AI governance, AI monitoring, artificial intelligence in healthcare, data drift, model degradation, institutional incentives, accountability, AI safety, AI equity, patient harm, trust in AI
- MeSH terms
- Machine Learning, Artificial Intelligence, Delivery of Health Care, Humans, Interviews as Topic, Social Responsibility, Qualitative Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study. https://app.cochat.ai/papers/10.1186/s12913-025-13388-z
- The paper
- Owens, K., Griffen, Z., & Damaraju, L. (2025). Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study. BMC Health Services Research. https://doi.org/10.1186/s12913-025-13388-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.