Algorithmovigilance, lessons from pharmacovigilance
Open accessAbstract
Artificial Intelligence (AI) systems are increasingly being deployed across various high-risk applications, especially in healthcare. Despite significant attention to evaluating these systems, post-deployment incidents are not uncommon, and effective mitigation strategies remain challenging. Drug safety has a well-established history of assessing, monitoring, understanding, and preventing adverse effects in real-world usage, known as pharmacovigilance. Drawing inspiration from pharmacovigilance methods, we discuss concepts that can be adapted for monitoring AI systems in healthcare. This discussion aims to improve responses to adverse effects and potential incidents and risks associated with AI deployment in healthcare but also beyond.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clinicians may ignore alerts, over-trust outputs, misunderstand scores, or use the tool outside its intended scope.
“Clinicians may ignore alerts, over-trust outputs, misunderstand scores, or use the tool outside its intended scope.”
From The AI Hospital Formulary: A Practical Governance Framework for Prescribing, Monitoring, and Deprescribing Artificial Intelligence in Hospitals (2026) · cites this paper as [32]
The Post-approval evaluation section (page unavailable) says human factors, including how clinicians make decisions based on model predictions, matter to AI safety. Causality assessment also identifies operators’ skills and experience as possible contributors to errors. The paper does not specifically describe clinicians ignoring alerts, over-trusting outputs, misunderstanding scores, or using a tool outside its intended scope. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 4× · see citing works
“ErrorsrelatedtoAIsystems canoriginatefromvariousfactors,includingthedataquality,theunderlying biasinthedata,acquisitiondevices,othersoftwareusedinthework fl ow, medical treatments, and the skills and experience of the AI operators 16,25,26 .”
BackgroundInfluentialCollins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · cited 1× · see citing works
“First, the current state of AI development has led to a proliferation of models still lacking standardized validation, exacerbated by the absence today of tailored development or reporting guidelines — although this will likely improve over time 48 .”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1× · see citing works
“Othereffortsfocusonpresentingand summarizingkeyinformationaboutAIsystemsfordecision-making 44,45 .”
The Medical Dictionary for Regulatory Activities (MedDRA)
Brown, E. G., Wood, L., & Wood, S. (1999) · Drug Safety · doi:10.2165/00002018-199920020-00002 · Cited by 1,396 · see citing works
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215 · see citing works
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855 · see citing works
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · see citing works
Validation of a common data model for active safety surveillance research
Overhage, J. M., Ryan, P. B., Reich, C. G., Hartzema, A. G., & Stang, P. E. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000376 · Cited by 548 · see citing works
Use of AI-based tools for healthcare purposes: a survey study from consumers’ perspectives
Esmaeilzadeh, P. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01191-1 · Cited by 507 · see citing works
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494 · see citing works
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423 · see citing works
Harmonisation in Pharmacovigilance
Edwards, I. R., & Biriell, C. (1994) · Drug Safety · doi:10.2165/00002018-199410020-00001 · Cited by 396 · see citing works
Methods for Causality Assessment of Adverse Drug Reactions: A Systematic Review
Agbabiaka, T. B., Savović, J., & Ernst, E. (2008) · Drug Safety · doi:10.2165/00002018-200831010-00003 · Cited by 366 · see citing works
Winkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1735 · Cited by 365 · see citing works
Gerke, S., Babic, B., Evgeniou, T., & Cohen, I. G. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0262-2 · Cited by 359 · see citing works
Self-citationLaï, M.-C., Brian, M., & Mamzer, M.-F. (2020) · Journal of Translational Medicine · doi:10.1186/s12967-019-02204-y · Cited by 294 · see citing works
Presenting machine learning model information to clinical end users with model facts labels
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0253-3 · Cited by 279 · see citing works
Beware explanations from AI in health care
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021) · Science · doi:10.1126/science.abg1834 · Cited by 252 · see citing works
Self-citationMeyboom, R. H. B., Hekster, Y. A., Egberts, T. C. G., Gribnau, F. W. J., & Edwards, I. R. (1997) · Drug Safety · doi:10.2165/00002018-199717060-00004 · Cited by 241 · see citing works
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
Zhang, J., Wang, Y., Molino, P., Li, L., & Ebert, D. S. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864499 · Cited by 222 · see citing works
Paracetamol – An old drug with new mechanisms of action
Przybyła, G. W., Szychowski, K. A., & Gmiński, J. (2021) · Clinical and Experimental Pharmacology and Physiology · doi:10.1111/1440-1681.13392 · Cited by 190 · see citing works
Pharmacovigilance: An Overview
Beninger, P. R. (2018) · Clinical Therapeutics · doi:10.1016/j.clinthera.2018.07.012 · Cited by 176 · see citing works
Preventing repeated real world AI failures by cataloging incidents: the AI incident database
McGregor, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17817 · Cited by 156 · see citing works
Hwang, T. J., Kesselheim, A. S., & Vokinger, K. N. (2019) · JAMA · doi:10.1001/jama.2019.16842 · Cited by 145 · see citing works
MLOps: A Taxonomy and a Methodology
Testi, M., Ballabio, M., Frontoni, E., Iannello, G., Moccia, S., Soda, P., & Vessio, G. (2022) · IEEE Access · doi:10.1109/access.2022.3181730 · Cited by 137 · see citing works
How the US Drug Safety System Should Be Changed
Strom, B. L. (2006) · JAMA · doi:10.1001/jama.295.17.2072 · Cited by 134 · see citing works
Algorithms on regulatory lockdown in medicine
Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2019) · Science · doi:10.1126/science.aay9547 · Cited by 123 · cited 2× · see citing works
“However, we notethatcurrentregulationmainlypermitsmarketingapprovalforso-called lockedAIsystems 2 , although the FDA is investigating new regulations for adaptiveAIsystemsinhealthcare 41 .”
BackgroundMethodSelf-citationThe evolution of digital health technologies in cardiovascular disease research
Zwack, C. C., Haghani, M., Hollings, M., Zhang, L., Gauci, S., Gallagher, R. D., & Redfern, J. (2023) · npj Digital Medicine · doi:10.1038/s41746-022-00734-2 · Cited by 123 · see citing works
Duckworth, C., Chmiel, F. P., Burns, D. K., Zlatev, Z. D., White, N. M., Daniels, T. W. V., Kiuber, M., & Boniface, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-02481-y · Cited by 109 · see citing works
Embi, P. J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.4622 · Cited by 99 · see citing works
Perspectives on validation of clinical predictive algorithms
de Hond, A., Shah, V. B., Kant, I. M. J., CALSTER, B. V., Steyerberg, E. W., & Hernandez‐Boussard, T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00832-9 · Cited by 95 · see citing works
Healthsheet: Development of a Transparency Artifact for Health Datasets
Rostamzadeh, N., Mincu, D., Roy, S., Smart, A., Wilcox, L. G., Pushkarna, M., Schrouff, J., Amironesei, R., Moorosi, N., & Heller, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533239 · Cited by 79 · see citing works
Poulain, R., Bin Tarek, M. F., & Beheshti, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594102 · Cited by 63 · see citing works
Pharmacoepidemiology I: A Review of Pharmacoepidemiologic Study Designs
Etminan, M., & Samii, A. (2004) · Pharmacotherapy The Journal of Human Pharmacology and Drug Therapy · doi:10.1592/phco.24.11.964.36143 · Cited by 58 · see citing works
Causality or Relatedness assessment in adverse drug reaction and its relevance in dermatology
Pande, S. (2018) · Indian Journal of Dermatology · doi:10.4103/ijd.ijd_579_17 · Cited by 38 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [18]Rabanser, S., Günnemann, S. & Lipton, Z. C. Failing loudly: an empirical study of methods for detecting dataset shift. In Proc. 33rd International Conference on Neural Information Processing Systems, 1396–1408 (Curran Associates Inc., Red Hook, NY, USA, 2019).
Cited by
Sorted by citations
A scoping review of robustness concepts for machine learning in healthcare
Balendran, A., Beji, C., Bouvier, F., Khalifa, O., Evgeniou, T., Ravaud, P., & Porcher, R. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01420-1 · Cited by 43 · see citing works
Self-citationGabarrón, E., Denecke, K., & López–Campos, G. (2025) · BMC Psychiatry · doi:10.1186/s12888-025-06825-0 · Cited by 27 · see citing works
Bailo, P., Nittari, G., Pesel, G., Basello, E., Spasari, T., & Ricci, G. (2026) · Sci · doi:10.3390/sci8020036 · Cited by 22 · see citing works
Rademakers, F. E., Biasin, E., Bruining, N., Caiani, E. G., Davies, R. H., Gilbert, S., Kamenjašević, E., McGauran, G., O’Connor, G., Rouffet, J.-B., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01459-8 · Cited by 21 · see citing works
Sirago, G., Vinci, A., Visci, P., Solarino, B., Dell'Erba, A. S., & Ferorelli, D. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02382-5 · Cited by 10 · see citing works
Elmaleh, Y. D., Guessous, K., & Delvaux, B. V. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1811197 · Cited by 6 · see citing works
Innovating global regulatory frameworks for generative AI in medical devices is an urgent priority
Ong, J. C. L., Ning, Y., Liu, M., Ma, Y., Zhao, L., Singh, K., Chang, R. T., Vogel, S., Lim, J. C. W., Tan, I. S. K., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02552-2 · Cited by 7 · see citing works
Artificial intelligence and the future of the pharmacist’s role in drug information
Patel, R., Shirk, R., Advani, A., Kar, I., Munir, F. M., Nelson, S. D., Saad, M., & Aungst, T. D. (2025) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf175 · Cited by 4 · see citing works
Advancing sustainability through digitalization: from drug development to patient care
Eugster, R., & Luciani, P. (2026) · RSC Pharmaceutics · doi:10.1039/d5pm00342c · Cited by 4 · see citing works
Adefolalu, A. O. (2026) · Risk Management and Healthcare Policy · doi:10.2147/rmhp.s621924 · Cited by 4 · see citing works
Farahani, S., Ieva, A. D., Coiera, E. W., & Liu, S. (2026) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2026.100092 · Cited by 3 · see citing works
Lipskiy, N., & Flowerday, S. (2026) · Future Internet · doi:10.3390/fi18050232 · Cited by 2 · see citing works
Arab, R. A. E., Mustafa, M. H., Almagharbeh, W. T., Ayoub, M. Y., Alsanawi, F., Almathen, F., Almosabeh, R., & Talaq, M. A. (2026) · Healthcare · doi:10.3390/healthcare14142052 · Cited by 2 · see citing works
Harnessing the power of artificial intelligence for clinical trials in cancer
Sönmez, G., Yazarkan, Y., Sahin, T. K., & Güven, D. C. (2026) · Expert Review of Anticancer Therapy · doi:10.1080/14737140.2026.2642221 · Cited by 1 · see citing works
Informatics Infrastructure for the Learning Health System
Tenenbaum, J. D., Cimino, J. J., Dorr, D. A., Embí, P. J., Melton, G. B., Ritchie, M. D., Weiner, M. G., & Friedman, C. P. (2026) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092624-103225 · Cited by 1 · see citing works
Intelligent Control Towers for Resilient and Compliant Pharmaceutical Supply Chains
Gandu, S., & Manda, V. K. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-8770-3.ch009 · Cited by 1 · see citing works
Artificial Intelligence in Clinical Trials: From Protocol Design to Pharmacovigilance
Schulman, A., Fernández‐Torras, A., Gadiya, Y., Saeed, K., Mestres, J., & Tanoli, Z. (2026) · Springer eBooks · doi:10.1007/978-3-031-98022-0_24 · Cited by 1 · see citing works
George, J., Kumar, A. P., Kalaiselvan, V., Shetty, V., Nair, A. S., Chakraborty, A., Kiran, M., Sindu, M. G., Buddha, S. K., Randeo, S., et al. (2026) · Perspectives in Clinical Research · doi:10.4103/picr.picr_153_25 · Cited by 1 · see citing works
Safeguarding AI-driven digital health – An adaptation of the Swiss cheese model for safety
Denecke, K. (2026) · Digital Health · doi:10.1177/20552076261459551 · Cited by 1 · see citing works
Aliaga, C. A. T., Fuentes, V. G. H., Karim, H. B. A., AlDahoul, N., & Tan, M. J. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1894116 · Cited by 1 · see citing works
Wang, Z., Dai, Q., Yang, M., Shi, S., Liao, J., Li, Z., Dai, X., & Tao, L. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag154 · Cited by 1 · see citing works
Regulating AI in healthcare: a moving target
Ball, M. J., & Vallejos, E. P. (2026) · BMJ · doi:10.1136/bmj-2026-100555 · Cited by 1 · see citing works
Epelde, F. (2026) · Hospitals · doi:10.3390/hospitals3030015
“Algorithmovigilance therefore combines technical surveillance with process, outcome, equity, cybersecurity, and human-factors monitoring [24,31,32].”
Ong, A. Y., Nguyen, Q., Barai, I., Engelmann, J., Antaki, F., Sevgi, M., Merle, D. A., Ju, L., Dow, E. R., Zhou, Y., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03136-w
Costa, B., & Vale, N. (2026) · Pharmaceuticals · doi:10.3390/ph19030450
Ai System Decommissioning: A Systematic Literature Review
Indaryani, L. (2026) · Media Informatika · doi:10.37595/mediainfo.v25i1.505
Caparatta, T., Tsoi, A., Byramji, D. K., Farooq, S., Ruiz, K., Frank, C., Gartner, G., Schwarz, N., Fenn, A., Nazarian, J. P., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-9899999/v1
Amoei, M., Fraser, K., & Poenaru, D. (2026) · doi:10.2196/preprints.107982
Sun, X., Qian, D., Zhao, W., Zhang, J., Jin, P., & Zhang, Y. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-08717-7
Continual Learning without Silent Change in Medication-Support Systems
Kim, D., Park, J., Kang, M.-S., & Jung, H.-W. (2026) · Archives of Pharmacy Practice · doi:10.51847/65q1pxmaoe
Cabrera-Rodríguez, R., Reyes-Castañeda, I., Lorenzo-Sánchez, I., Valenzuela-Fernández, A., & Trujillo‐González, R. (2026) · New Microbes and New Infections · doi:10.1016/j.nmni.2026.101843
Sirago, G., Sabato, S., Sebastiani, N., Zotti, F., Dell'Erba, A., & Ferorelli, D. (2026) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/j.jcjq.2026.09.002
Advances in the Use of Artificial Intelligence in the Pharmaceutical Industry
Montaña, A. F., Mora, J. A., Martínez, J. J., Pulido, D. F., & Brijaldo, M. H. (2026) · BioMedInformatics · doi:10.3390/biomedinformatics6050083
An, R., Meng, B., & Lu, X. (2026) · Journal of International Medical Research · doi:10.1177/03000605261491422
Ecer, N., Kankılıç, A. T., Karakoyun, Ö., Aslancan, B., & Kaldan, B. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02558-5
Peng, M., & Feng, L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1986759
Atiku, S. O., Olakotan, O. O., Agbaje, K., Owolanke, K., & Pillay, T. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1986499
- Authors
- Balendran, A., Benchoufi, M., Evgeniou, T., & Ravaud, P.
- Affiliations
- Inserm (FR)
- INSEAD (FR)
- Corresponding author
- Balendran, Alan
- Venue
- npj Digital Medicine
- Published
- 2024-10-02
- Volume
- 7
- Issue
- 1
- Pages
- 270-270
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11447237
- OpenAlex
- W4403048071
- Cited by
- 34
- References
- 37
- Citations by year
- 2026: 30 · 2025: 4
- FWCI
- 2.34
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Bpifrance
- Topics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Toxicology › Pharmacovigilance and Adverse Drug Reactions
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- pharmacovigilance, artificial intelligence in healthcare, adverse events, risk mitigation, drug safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
