FairLens: Auditing black-box clinical decision support systems
Open accessAbstract
The pervasive application of algorithmic decision-making is raising concerns on the risk of unintended bias in AI systems deployed in critical settings such as healthcare. The detection and mitigation of model bias is a very delicate task that should be tackled with care and involving domain experts in the loop. In this paper we introduce FairLens, a methodology for discovering and explaining biases. We show how this tool can audit a fictional commercial black-box model acting as a clinical decision support system (DSS). In this scenario, the healthcare facility experts can use FairLens on their historical data to discover the biases of the model before incorporating it into the clinical decision flow. FairLens first stratifies the available patient data according to demographic attributes such as age, ethnicity, gender and healthcare insurance; it then assesses the model performance on such groups highlighting the most common misclassifications. Finally, FairLens allows the expert to examine one misclassification of interest by explaining which elements of the affected patients’ clinical history drive the model error in the problematic group. We validate FairLens’ ability to highlight bias in multilabel clinical DSSs introducing a multilabel-appropriate metric of disparity and proving its efficacy against other standard metrics.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Visualization tools such as the What-If Tool and FairLens can help surface potential data outliers and help correct them before they creep into models.
“For example, visualization tools such as the What-If Tool [193] and FairLens [194] can help surface any potential data outliers and help correct them before they creep into models.”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [194]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · see citing works
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1× · see citing works
“rformance. While there is an increasing interest of the AI community to commit to interdisciplinary endeavors to dene, investigate and provide guidelines to tackle biases and fairness-related issues [44,49,41], quantitative and systematic auditing for real-world datasets and ML models is still in its infancy. In this…”
BackgroundArtificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837 · see citing works
The potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · see citing works
DNA methylation-based classification of central nervous system tumours
Capper, D., Jones, D., Sill, M., Hovestadt, V., Schrimpf, D., Sturm, D., Koelsche, C., Sahm, F., Chávez, L., Reuß, D., et al. (2018) · Nature · doi:10.1038/nature26000 · Cited by 3,046 · see citing works
Coudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö, D., Moreira, A. L., Razavian, N., & Tsirigos, A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0177-5 · Cited by 2,930 · see citing works
Artificial intelligence in healthcare
Yu, K.-H., Beam, A. L., & Kohane, I. S. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0305-z · Cited by 2,817 · see citing works
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Recurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · see citing works
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · see citing works
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Abràmoff, M. D., Lavin, P. T., Birch, M., Shah, N., & Folk, J. C. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0040-6 · Cited by 1,638 · see citing works
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581 · see citing works
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy166 · Cited by 1,571 · see citing works
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 1,372 · see citing works
Disease Prediction by Machine Learning Over Big Data From Healthcare Communities
Chen, M., Hao, Y., Hwang, K., Wang, L., & Wang, L. (2017) · IEEE Access · doi:10.1109/access.2017.2694446 · Cited by 1,311 · see citing works
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · see citing works
(2013) · Cited by 1,126 · see citing works
Measuring Diagnoses: ICD Code Accuracy
O’Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005) · Health Services Research · doi:10.1111/j.1475-6773.2005.00444.x · Cited by 1,123 · see citing works
Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study
Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N. G., Venugopal, V. K., Mahajan, V., Rao, P., & Warier, P. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31645-3 · Cited by 1,020 · see citing works
Zhang, J., Gajjala, S., Agrawal, P., Tison, G. H., Hallock, L. A., Beussink-Nelson, L., Lassen, M. H., Fan, E., Aras, M. A., Jordan, C., et al. (2018) · Circulation · doi:10.1161/circulationaha.118.034338 · Cited by 1,006 · see citing works
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · see citing works
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941 · see citing works
Socioeconomic Status and Access to Healthcare: Interrelated Drivers for Healthy Aging
McMaughan, D. J., Oloruntoba, O., & Smith, M. L. (2020) · Frontiers in Public Health · doi:10.3389/fpubh.2020.00231 · Cited by 901 · see citing works
Xiao, C., Choi, E., & Sun, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy068 · Cited by 792 · see citing works
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · see citing works
Representation of the Elderly, Women, and Minorities in Heart Failure Clinical Trials
Heiat, A., Gross, C. P., & Krumholz, H. M. (2002) · Archives of Internal Medicine · doi:10.1001/archinte.162.15.1682 · Cited by 723 · see citing works
Discrimination-aware data mining
Pedreshi, D., Ruggieri, S., & Turini, F. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401959 · Cited by 676 · see citing works
Deep neural network improves fracture detection by clinicians
Lindsey, R., Daluiski, A., Chopra, S., Lachapelle, A. S., Mozer, M. C., Sicular, S., Hanel, D. P., Gardner, M. J., Gupta, A., Hotchkiss, R. J., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806905115 · Cited by 654 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Self-citationUsing electronic health records for population health research: a review of methods and applications
Casey, J. A., Schwartz, B. S., Stewart, W. F., & Adler, N. E. (2016) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032315-021353 · Cited by 590 · see citing works
Fast and accurate view classification of echocardiograms using deep learning
Madani, A., Arnaout, R., Mofrad, M., & Arnaout, R. (2018) · npj Digital Medicine · doi:10.1038/s41746-017-0013-1 · Cited by 585 · see citing works
Nam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., et al. (2019) · Radiology · doi:10.1148/radiol.2018180237 · Cited by 539 · see citing works
Automated deep-neural-network surveillance of cranial images for acute neurologic events
Titano, J. J., Badgeley, M., Schefflein, J., Pain, M., Su, A., Cai, M., Swinburne, N., Zech, J., Kim, J., Bederson, J., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0147-y · Cited by 458 · see citing works
Improving palliative care with deep learning
Avati, A., Jung, K., Harman, S. M., Downing, N. L., Ng, A. Y., & Shah, N. H. (2017) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2017.8217669 · Cited by 445 · see citing works
Controlling Popularity Bias in Learning-to-Rank Recommendation
Abdollahpouri, H., Burke, R., & Mobasher, B. (2017) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3109859.3109912 · Cited by 437 · see citing works
Can AI Help Reduce Disparities in General Medical and Mental Health Care?
(2019) · AMA Journal of Ethics · doi:10.1001/amajethics.2019.167 · Cited by 401 · see citing works
Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 381 · cited 1× · see citing works
“Generally speaking, the most prevalent approach to fairness in machine learning is to solicit for approximate parity of some statistics of the predictions (such as false negative rate) across pre-defined subgroups [33,32,14].”
BackgroundInherent Tradeoffs in the Fair Determination of Risk Scores
Raghavan, M. (2023) · ACM eBooks · doi:10.1145/3603195.3603197 · Cited by 347 · see citing works
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337 · see citing works
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334 · see citing works
Local Rule-Based Explanations of Black Box Decision SystemsPreprint
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., & Giannotti, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10820 · Cited by 323 · see citing works
Self-citationAn algorithmic approach to reducing unexplained pain disparities in underserved populations
Pierson, E., Cutler, D. M., Leskovec, J., Mullainathan, S., & Obermeyer, Z. (2021) · Nature Medicine · doi:10.1038/s41591-020-01192-7 · Cited by 268 · see citing works
k-NN as an implementation of situation testing for discrimination discovery and prevention
Luong, B. T., Ruggieri, S., & Turini, F. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020488 · Cited by 263 · see citing works
A call for deep-learning healthcare
Norgeot, B., Glicksberg, B. S., & Butte, A. J. (2018) · Nature Medicine · doi:10.1038/s41591-018-0320-3 · Cited by 252 · see citing works
Upcoding: Evidence from Medicare on Squishy Risk Adjustment
Geruso, M., & Layton, T. (2019) · Journal of Political Economy · doi:10.1086/704756 · Cited by 251 · see citing works
Data mining for discrimination discovery
Ruggieri, S., Pedreschi, D., & Turini, F. (2010) · TKDD · doi:10.1145/1754428.1754432 · Cited by 250 · see citing works
Self-citationRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171 · see citing works
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171 · see citing works
Panigutti, C., Perotti, A., & Pedreschi, D. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372855 · Cited by 162 · see citing works
Self-citationGLocalX—From Local to Global Explanations of Black Box AI Models
Setzu, M., Guidotti, R., Monreale, A., Turini, F., Pedreschi, D., & Giannotti, F. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103457 · Cited by 140 · see citing works
Self-citationRepresentation of South Asian people in randomised clinical trials: analysis of trials' data
Mason, S., Hussain-Gambles, M., Leese, B., et al. (2003) · BMJ · doi:10.1136/bmj.326.7401.1244 · Cited by 118 · see citing works
Learning Certifiably Optimal Rule Lists for Categorical Data
Angelino, E., Larus-Stone, N., Alabi, D., Seltzer, M., & Rudin, C. (2017) · arXiv · doi:10.48550/arxiv.1704.01701 · Cited by 108 · see citing works
The Effect of Insurance Status on Mortality and Procedural Use in Critically Ill Patients
Lyon, S. M., Benson, N. M., Cooke, C. R., Iwashyna, T. J., Ratcliffe, S. J., & Kahn, J. M. (2011) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201101-0089oc · Cited by 100 · see citing works
Doctor performance assessment in daily practise: does it help doctors or not? A systematic review
Overeem, K., Faber, M. J., Arah, O. A., Elwyn, G. J., Lombarts, K. M. J. M. H., Wollersheim, H. C. H., & Grol, R. P. T. M. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02897.x · Cited by 94 · see citing works
FaiRecSys: mitigating algorithmic bias in recommender systems
Edizel, B., Bonchi, F., Hajian, S., Panisson, A., & Tassa, T. (2019) · International Journal of Data Science and Analytics · doi:10.1007/s41060-019-00181-5 · Cited by 71 · see citing works
Self-citationFairML : ToolBox for diagnosing bias in predictive modeling
Adebayo, J. (2016) · Cited by 70 · see citing works
Moja, L., Friz, H. P., Capobussi, M., Kwag, K. H., Banzi, R., Ruggiero, F. M., González‐Lorenzo, M., Liberati, E. G., Mangia, M., Nyberg, P. M., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.17094 · Cited by 68 · see citing works
Hillson, S. D., Connelly, D. P., & Liu, Y. (1995) · Medical Decision Making · doi:10.1177/0272989x9501500202 · Cited by 47 · see citing works
Lindow, T., Kron, J., Thulesius, H. O., Ljungström, E., & Pahlm, O. (2019) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2019.1684429 · Cited by 28 · see citing works
Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations
Guidotti, R., Monreale, A., Matwin, S., et al. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7116 · Cited by 16 · see citing works
Global Explanations with Local Scoring
Setzu, M., Guidotti, R., Monreale, A., & Turini, F. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-43823-4_14 · Cited by 11 · see citing works
Ethical Machine Learning in HealthPreprint
Chen, I. Y., Pierson, E., Rose, S., Joshi, S. D., Ferryman, K., & Ghassemi, M. (2020) · arXiv (Cornell University) · Cited by 10 · see citing works
Ethical Machine Learning in Health CarePreprint
Chen, I. Y., Pierson, E., Rose, S., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10576 · Cited by 2 · see citing works
Aequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577
Flagged first, then by citations
Research works printed but not linked: 18
- Anjomshoae, S., Kampik, T., & Främling, K. (2020). Py-CIU: A python library for explaining machine learning predictions using contextual importance and utility. In IJCAI-PRICAI 2020 workshop on explainable artificial intelligence (XAI).
- Boag, W., Suresh, H., Celi, L. A., Szolovits, P., & Ghassemi, M. (2018). Racial disparities and mistrust in end-of-life care. In Machine learning for healthcare conference (pp. 587-602). PMLR.
- Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2016). Doctor AI: Predicting clinical events via recurrent neural networks. In Machine learning for healthcare conference (pp. 301-318).
- Frogner, C., Zhang, C., Mobahi, H., Araya, M., & Poggio, T. A. (2015). Learning with a wasserstein loss. In Advances in neural information processing systems (pp. 2053-2061).
- Harrington, K., Allen, A., & Ruchala, L. (2007). Restraining medicare abuse: the case of upcoding. Research in Healthcare Financial Management, 11(1), 1.
- Jiang, R., Pacchiano, A., Stepleton, T., Jiang, H., & Chiappa, S. (2020). Wasserstein fair classification. In Uncertainty in artificial intelligence (pp. 862-872).
- PMLR.
- Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Roger, M. (2020). MIMIC-IV (Version 0.4). PhysioNet, http://dx.doi.org/10.13026/a3wn-hq05. · DOI
- Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in neural information processing systems (pp. 4765-4774).
- Miranda-Escalada, A., Gonzalez-Agirre, A., Armengol-Estapé, J., & Krallinger, M. (2020). Overview of automatic clinical coding: annotations, guidelines, and solutions for non-english clinical cases at codiesp track of CLEF ehealth 2020. In Working notes of conference and labs of the evaluation (clef) forum. ceur workshop proceedings.
- Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., & Eisenstein, J. (2018). Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers) (pp. 1101-1111). New Orleans, Louisiana: Association for Computational Linguistics.
- Panigutti, C., Guidotti, R., Monreale, A., & Pedreschi, D. (2019). Explaining multi-label black-box classifiers for health applications. In International workshop on health intelligence (pp. 97-110). Springer.
- Panigutti, C., Perotti, A., & Pedreschi, D. (2020). Doctor XAI: an ontology-based approach to black-box sequential data classification explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 629-639).
- Polignano, M., Suriano, V., Lops, P., de Gemmis, M., & Semeraro, G. (2020). A study of machine learning models for clinical coding of medical reports at codiesp 2020. In Working notes of conference and labs of the evaluation (clef) forum. ceur workshop proceedings.
- Seyyed-Kalantari, L., Liu, G., McDermott, M., & Ghassemi, M. (2020). Chexclusion: Fairness gaps in deep chest X-ray classifiers. arxiv:2003.00827. · arXiv
- Shameer, K., Johnson, K. W., Yahi, A., Miotto, R., Li, L., Ricks, D., et al. (2017). Predictive modeling of hospital readmission rates using electronic medical record-wide machine learning: a case-study using mount sinai heart failure cohort. In Pacific symposium on biocomputing 2017 (pp. 276-287). World Scientific.
- Wang, T., Rudin, C., Doshi-Velez, F., Liu, Y., Klampfl, E., & MacNeille, P. (2017). A bayesian framework for learning rule sets for interpretable classification. Journal of Machine Learning Research, 18(1), 2357-2393.
- Zhang, X., Tan, S., Koch, P., Lou, Y., Chajewska, U., & Caruana, R. (2019). Interpretability is harder in the multiclass setting: axiomatic interpretability for multiclass additive models. Age, 25(50), 75-100.
Cited by
Sorted by citations
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750 · see citing works
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152 · see citing works
“…systems used for recruitment (Rhea et al., 2022), evaluating the capabilities of language models (Goel et al., 2021; Mökander et al., 2023), providing explanations for black-box AI systems (Pedreschi et al., 2018), and conducting audits of clinical decision support systems (Panigutti et al., 2021).”
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127 · see citing works
Self-citationBias Mitigation in Primary Health Care Artificial Intelligence Models: Scoping Review
Sasseville, M., Ouellet, S., Rhéaume, C., Sahlia, M., Couture, V., Després, P., Paquette, J.-S., Darmon, D., Bergeron, F., & Gagnon, M.-P. (2025) · Journal of Medical Internet Research · doi:10.2196/60269 · Cited by 38 · see citing works
Human-centred artificial intelligence for mobile health sensing: challenges and opportunities
Dang, T., Spathis, D., Ghosh, A., & Mascolo, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.230806 · Cited by 30 · see citing works
- Authors
- Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D.
- Affiliations
- Scuola Normale Superiore (IT)
- Institute for Scientific Interchange (IT)
- University of Pisa (IT)
- Corresponding author
- Panigutti, Cecilia
- Venue
- Information Processing & Management
- Publisher
- Elsevier BV
- Published
- 2021-09
- Volume
- 58
- Issue
- 5
- Pages
- 102657
- ISSN
- 0020-0271 (ISSN-L)
- 0306-4573
- OpenAlex
- W3099868389
- Cited by
- 86
- References
- 82
- Citations by year
- 2026: 9 · 2025: 19 · 2024: 15 · 2023: 20 · 2022: 13 · 2021: 9 · 2020: 1
- FWCI
- 17.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Information Processing & Management (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Intesa Sanpaolo Innovation Center
- European Commission (952026, 834756, 952215, 871042)
- Regione Piemonte
- HORIZON EUROPE Excellent Science
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- algorithmic fairness, bias detection, clinical decision support systems, black-box models, explainable AI, model auditing, demographic disparities, multi-label classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
