On the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Open accessAbstract
Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine learning. Heretofore, the definition of safety has not been formalized in a machine learning context. In this article, we do so by defining machine learning safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. We then use this definition to examine safety in all sorts of applications in cyber-physical systems, decision sciences, and data products. We find that the foundational principle of modern statistical machine learning, empirical risk minimization, is not always a sufficient objective. We discuss how four different categories of strategies for achieving safety in engineering, including inherently safe design, safety reserves, safe fail, and procedural safeguards can be mapped to a machine learning context. We then discuss example techniques that can be adopted in each category, such as considering interpretability and causality of predictive models, objective functions beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software and open data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Safety issues have been studied in regards to processes that depend on the output of AI models.
“Safety issues have also been studied in regards to processes that depend on the output of AI models, such as vehicular perception and self-driving in autonomous vehicles, automated surgery, data-based support for medical diagnosis, insurance risk assessment and cyber-physical systems in manufacturing, among others [369].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [369]
Selected papers for the Software Construction KA are also related to the Software Design KA, except Islam et al., who reported the challenges facing the use of ML libraries.
“Selected papers for this KA are also related to the Software Design KA (except Islam et al. [28], who reported the challenges facing the use of ML libraries).”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1× · see citing works
“In certain prediction pro blems, the risk of harm for members of protected groups should not be much worse (up to a multiplicative factor) than the risk of ha rm for others [36]–[38].”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1× · see citing works
“Moreover, not only is non-harm the first ethical duty, many of the safety issues for machine learning we have discussed in this paper are starting to enter legal obligati ons as well [45].”
BackgroundCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 1× · see citing works
“Highly complex modeling techniques used today, including extreme gradient boostin g and deep neural networks, may pick up on those data vagaries in the learned models they produce to achieve high accuracy, but might fail due to an unknown shift in the data domain [26].”
BackgroundImproving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678 · see citing works
Mining with rarity: a unifying framework
Weiss, G. M. (2004) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1007730.1007734 · Cited by 1,393 · cited 1× · see citing works
“sample [41], a safe fail mechanism is to always go for manual examination.”
BackgroundRobust classification for imprecise environments
Provost, F., & Fawcett, T. (2001) · Machine Learning · doi:10.1023/a:1007601015854 · Cited by 1,250 · see citing works
Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance?Preprint
Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011) · SSRN Electronic Journal · doi:10.2139/ssrn.1819486 · Cited by 842 · cited 1× · see citing works
“Man y emerging application domains are now shifting to data-driv en decision making due to greater capture of information digit ally and the desire to be more scientific rather than relying on (fallible) gut instinct [22].”
BackgroundDomain Adaptation for Statistical Classifiers
Daume III, H., & Marcu, D. (2006) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1872 · Cited by 816 · cited 1× · see citing works
“Further, it may be that the distribution the samples actually come from cannot be known, precluding the use of covariate shift [14] and domain adaptation techniques [15].”
BackgroundChallenges in autonomous vehicle testing and validation
Koopman, P., & Wagner, M. (2016) · SAE International Journal of Transportation Safety · doi:10.4271/2016-01-0128 · Cited by 675 · cited 1× · see citing works
“r less confident predictions, stop the autonomous control software and switch to a backup system or a degraded level of autonomy (e.g., full control by the driver) are considered for self-driving cars [48]. B. Decision Sciences In decision sciences applications, people are in the loop in a different way than in cyberph…”
BackgroundSupervised autonomous robotic soft tissue surgery
Shademan, A., Decker, R., Opfermann, J. D., Léonard, S., Krieger, A., & Kim, P. C. W. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aad9398 · Cited by 626 · cited 1× · see citing works
“, knottying, suturing, or stitching) to reduce the cognitive workload on the surgical team [16], [17].”
BackgroundComprehensible classification models
Freitas, A. A. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2594473.2594475 · Cited by 575 · cited 1× · see citing works
“Two related ways to introduce inherently safe design is by insisting on models that can be interpreted by people and by excluding features that are not causally-related to the outcome [27]– [30].”
BackgroundAdverse Events in Robotic Surgery: A Retrospective Study of 14 Years of FDA Data
Alemzadeh, H., Raman, J., Leveson, N., Kalbarczyk, Z., & Iyer, R. K. (2016) · PLOS ONE · doi:10.1371/journal.pone.0151470 · Cited by 441 · cited 2× · see citing works
“As one example in the medical devices domain, we have mined a large database of adverse events to obtain exactly uch characterizations [11].”
BackgroundMethodUniform Guidelines on Employee Selection Procedures
Scheer, T. J. (2007) · Routledge eBooks · doi:10.1081/e-epap2-407 · Cited by 403 · cited 1× · see citing works
“Then using a rule such as the 80% (or four-fifths) rule advocated in the study of disparate impact [29], we can constraint the relative risk of harm for the protected versus unprotected group to a maximum value such as 5/4: ∫ Xp ∫”
MethodThe Future of Human-in-the-Loop Cyber-Physical Systems
Schirner, G., Deniz Erdogmus, B., Chowdhury, K. R., & Padır, T. (2013) · Computer · doi:10.1109/mc.2013.31 · Cited by 330 · cited 1× · see citing works
“In order to adapt to the constantly changing and uncertain environment, they need to take into account not only the current application but also the operator’s preferences, intent, and past behavior [36]. Autonomous machine learning and artificial intelligence techniques have been applied to several decision-making and…”
A Methodology for Direct and Indirect Discrimination Prevention in Data Mining
Hajian, S., & Domingo-Ferrer, J. (2013) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.72 · Cited by 312 · cited 1× · see citing works
“ring fairness and equitability. In certain predictionproblems, the risk of harm for membersof protected groupsshould not be much worse (up to a multiplicative factor) than the risk of harm for others [26]–[28]. We can partition the feature space X into the sets Xu,Xp ⊂ X, respectively, corresponding to the unprotected…”
Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions
Lin, H. C., Shafran, I., Yuh, D., & Hager, G. D. (2006) · Computer Aided Surgery · doi:10.3109/10929080600989189 · Cited by 254 · see citing works
Kassahun, Y., Yu, B., Tibebu, A. T., Stoyanov, D., Giannarou, S., Metzen, J. H., & Vander Poorten, E. (2016) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-015-1305-z · Cited by 238 · cited 1× · see citing works
“Recent research shows an increasing interest in use of machi ne learning algorithms for modeling surgical skills, workflow , and environment and integration of this knowledge into control and automation of surgical robots [13].”
Wagstaff, K. (2012) · arXiv · doi:10.48550/arxiv.1206.4656 · Cited by 148 · cited 1× · see citing works
“that can be judged harmful or not [6].”
BackgroundReliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty
Senge, R., Bösner, S., Dembczyński, K., Haasenritter, J., Hirsch, O., Donner‐Banzhoff, N., & Hüllermeier, E. (2013) · Information Sciences · doi:10.1016/j.ins.2013.07.030 · Cited by 146 · cited 1× · see citing works
“Epistemic uncertainty results from the lack of knowledge that could be obtained in principle, but may be practically intractable to gather [6].”
BackgroundPrinciples of engineering safety: Risk and uncertainty reduction
Möller, N., & Hansson, O. (2007) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2007.03.031 · Cited by 134 · cited 3× · see citing works
“Four categories of approache s have been identified for promoting safety in general [10]: inherently safe design, safety reserves, safe fail, and pro cedural safeguards.”
BackgroundEngineering safety in machine learning
Varshney, K. R. (2016) · Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2016.7888195 · Cited by 128 · cited 2× · see citing works
“To the best of our knowledge, there is no existing work on analyzing machine learning usin g pr cise decision-theoretic definitions of safety except ou r own preliminary work [4].”
BackgroundBeat the Machine: Challenging Humans to Find a Predictive Model's “Unknown Unknowns”
Attenberg, J., Ipeirotis, P., & Provost, F. (2015) · Journal of Data and Information Quality · doi:10.1145/2700832 · Cited by 103 · cited 1× · see citing works
“In these pa rts of the space, distance from the decision boundary is fairly meaningless and the typical trigger for the reject option sh ould be avoided [40].”
BackgroundMotion generation of robotic surgical tasks: Learning from expert demonstrations
Reiley, C. E., Plaku, E., & Hager, G. D. (2010) · Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2010.5627594 · Cited by 98 · cited 2× · see citing works
“, knottying, suturing, or stitching) to reduce the cognitive workload on the surgical team [16], [17].”
BackgroundMethodAutomatic Detection and Segmentation of Robot-Assisted Surgical Motions
Lin, H. C., Izhak Shafran, J., Murphy, T. E., Okamura, A. M., Yuh, D. D., & Hager, G. D. (2005) · Lecture notes in computer science · doi:10.1007/11566465_99 · Cited by 89 · cited 1× · see citing works
“s knowledge into control and automation of surgical robots [37]. Machine learning techniques have been used for detection and classification of surgical motions for automated surgical skill evaluation [38]– [40] and automating portions of repetitive and time-consuming surgical tasks (e.g., knot-tying, suturing). [40],…”
MethodExact Rule Learning via Boolean Compressed Sensing
Malioutov, D., & Varshney, K. R. (2013) · Cited by 73 · see citing works
The Concepts of Risk and Safety
Möller, N. (2012) · Springer eBooks · doi:10.1007/978-94-007-1433-5_3 · Cited by 69 · cited 4× · see citing works
“In particular, a general definition of safety is the minimization of risk andepistemic uncertainty(understood in the usual decision-theoretic senses of the words) associated w i h unwanted outcomes that are severe enough to be seen as harmful [3].”
BackgroundInfluentialFundamentals of Dependable Computing for Software Engineers
KNIGHT, J. C. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b11667 · Cited by 67 · cited 1× · see citing works
“t rates and costs in motor vehicles (hundreds of thousands of fatalities per year) versus commercial aircraft (tens of fatalities per year) shows the subjectivity of the public’s acceptance of safety [13]. C. Risk and Epistemic Uncertainty The risk minimization approach to machine learning has many strengths, which is…”
BackgroundTuning Support Vector Machines for Minimax and Neyman-Pearson Classification
Davenport, M. A., Baraniuk, R. G., & Scott, C. D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.29 · Cited by 67 · cited 1× · see citing works
“Such formulations can capture uncertainty in the class priors and uncertainty resulting from label noise in classification problems [34], [35].”
BackgroundAlgorithms for interpretable machine learning
Rudin, C. D. (2014) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2630823 · Cited by 61 · see citing works
Systems-Theoretic Safety Assessment of Robotic Telesurgical Systems
Alemzadeh, H., Chen, D., Lewis, A. J., Kalbarczyk, Z., Raman, J., Leveson, N. G., & Iyer, R. K. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24255-2_16 · Cited by 39 · see citing works
The Limits of Abstract Evaluation Metrics
Olteanu, A., Talamadupula, K., & Varshney, K. R. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3098871 · Cited by 30 · cited 1× · see citing works
“ution is a discussion on how to even understand and quantify the desirability and undesirability of outcomes along with their costs. To complement simply eliciting such knowledge directly from people [11], we suggest a data-driven approach for characterizing harms that are particularly relevant for cyber-physical syst…”
BackgroundAn Analytics Approach for Proactively Combating Voluntary Attrition of Employees
Singh, M., Varshney, K. R., Wang, J., Mojsilović, A., Gill, A. R., Faur, P. I., & Ezry, R. (2012) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2012.136 · Cited by 27 · cited 1× · see citing works
“1) Predicting Voluntary Resignation: We recently studied a Type A problem of predicting which IBM employees will voluntarily resign from the company in the next six months based on human resources and compensation data, which required us to develop a classification algorithm to be place d within a larger decision-maki…”
BackgroundDebugging Machine Learning TasksPreprint
Chakarov, A., Nori, A. V., Rajamani, S. K., Sen, S., & Vijaykeerthy, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07292 · Cited by 27 · cited 1× · see citing works
“No competing financial interests exist.”
Recursive Partitioning for Heterogeneous Causal EffectsPreprint
Athey, S. C., & Imbens, G. W. (2015) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1504.01132 · Cited by 20 · see citing works
Causal Falling Rule ListsPreprint
Wang, F., & Rudin, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.05189 · Cited by 15 · cited 1× · see citing works
“[31], and causality may be used to induce interpretability, e.”
BackgroundZhu, Y., Reddi, V. J., Adolf, R., Rama, S., Reagen, B., Wei, G.-Y., & Brooks, D. M. (2017) · IEEE Micro · doi:10.1109/mm.2017.2 · Cited by 11 · cited 1× · see citing works
“To the best of our knowledge, there is no self-driving car system with an inherently safe design that utilizes, e.g., interpretable models [47].”
BackgroundPractical Ensemble Classification Error Bounds for Different Operating Points
Varshney, K. R., Prenger, R., Marlatt, T. L., Chen, B. Y., & Hanley, W. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2012.219 · Cited by 10 · see citing works
Interpretable policies for dynamic product recommendations
Petrik, M., & Luss, R. (2016) · Uncertainty in Artificial Intelligence · Cited by 10 · see citing works
Azimian, H., Naish, M. D., Kiaii, B. B., & Patel, R. V. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2014.2315798 · Cited by 6 · cited 1× · see citing works
“An example of adopting safety reserves (Section III-2) in robotic surgery is robust optimization of preoperative planning to minimize the uncertainty at the task level while maximizing the dexterity [43]. 2) Self-Driving Cars: Self-driving cars are autonomous cyber-physical systems capable of making intelligent naviga…”
BackgroundOpen Civic Data: Of the People, For the People, By the People.
Sahuguet, A., Krauss, J. C., Palacios, L., & Sangokoya, D. (2014) · IEEE Data(base) Engineering Bulletin · Cited by 4 · see citing works
Data-driven resiliency assessment of medical cyber-physical systems
Alemzadeh, H. (2016) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 4 · see citing works
Robust binary hypothesis testing under contaminated likelihoods
Wei, D., & Varshney, K. R. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178603 · Cited by 2 · cited 1× · see citing works
“However, since the prediction problem is to predict resignation in the next six months, our set of negati ve samples will necessarily include employees who should be labeled positively because they will be resigning soon [24] .”
Background
Flagged first, then by citations
Research works printed but not linked: 5
- V. Vapnik, "Principles of risk minimization for learning theory," in Adv. Neur. Inf. Process. Syst. 4, 1992, pp. 831-838.
- J. Stanley and D. Tunkelang, "Doing data science right -your most common questions answered," http://firstround.com/review/doing-data-science-right-your-most-common-questions-answered, 2016.
- S. Kapoor, A. Mojsilović, J. N. Strattner, and K. R. Varshney, "From open data ecosystems to systems of innovation: A journey to realize the promise of open data," in Proc. Data for Good Exchange Conf., New York, NY, Sep. 2015.
- Y. Zhu and V. Janapa Reddi, "Cognitive computing safety: The new horizon for reliability," IEEE Micro, forthcoming.
- H. Gerard, K. Rao, M. Simithraaratchy, K. R. Varshney, K. Kabra, and G. P. Needham, "Predictive modeling of customer repayment for sustainable pay-as-you-go solar power in rural India," in Proc. Data for Good Exchange Conf., New York, NY, Sep. 2015.
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · see citing works
“models stating that dangerous situations are safe [McGough, 2018] and generally poor use of limited valuable resources in criminal justice, medicine, energy reliability, finance, and in other domains [Varshney and Alemzadeh, 2016]. Rather than trying to create models that are inherently interpretable, there has been a…”
BackgroundTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
“…in systems optimized not only for expected task performance but also other important criteria such as safety [Otte, 2013, Amodei et al., 2016, Varshney and Alemzadeh, 2016], nondiscrimination [Bostrom and Yudkowsky, 2014, Ruggieri et al., 2010, Hardt et al., 2016], avoiding technical debt…”
BackgroundMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · see citing works
“incidents in other domains, such as healthcare and finance [74].”
BackgroundA survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
“This uncertainty caused by the instantiation of the test set can have large safety implications on individual test samples [136].”
BackgroundAleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
Hüllermeier, E., & Waegeman, W. (2021) · Machine Learning · doi:10.1007/s10994-021-05946-3 · Cited by 1,834 · see citing works
“On the more practical side, Varshney and Alemzadeh (2016) give an example of a recent accident of a self-driving car, which led to the death of the driver (for the first time after 130 million miles of testing).”
BackgroundFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
“Safety, discussed in [3] and [5] as the minimization of both risk and epistemic uncertainty, will include explainability, algorithmic fairness, and robustness to dataset shift.”
BackgroundInfluentialSelf-citationTo Trust Or Not To Trust A ClassifierPreprint
Jiang, H., Kim, B., Guan, M. Y., & Gupta, M. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11783 · Cited by 320 · see citing works
“As such, ML trust and safety is an important theme [1, 2, 3].”
BackgroundAhmad, K., Maabreh, M., Ghaly, M., Khan, K., Qadir, J., & Al-Fuqaha, A. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2021.100452 · Cited by 291 · see citing works
Socially Responsible AI Algorithms: Issues, Purposes, and Challenges
Cheng, L., Varshney, K. R., & Liu, H. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12814 · Cited by 261 · see citing works
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215 · see citing works
Ellahham, S., Ellahham, N., & Simsekler, M. C. E. (2020) · American Journal of Medical Quality · doi:10.1177/1062860619878515 · Cited by 177 · see citing works
“Safety in health care implies the reduction or minimization of risks and uncertainty of harmful events.(8,9) The dimensions of safety are changing with the adoption of AI in health care.”
Transparency: Motivations and Challenges
Weller, A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_2 · Cited by 198 · see citing works
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · see citing works
“38 As an operation continues, models may be retrained on more relevant data, but the amount of data available will be limited (and possibly conflicting and of low quality).”
BackgroundHuman-centred design in industry 4.0: case study review and opportunities for future research
Nguyen, H. N., Lasa, G., & Iriarte, I. (2021) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-021-01796-x · Cited by 170 · see citing works
Considerations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176 · see citing works
“…in systems optimized not only for expected task performance but also other important criteria such as safety [Otte, 2013, Amodei et al., 2016, Varshney and Alemzadeh, 2016], nondiscrimination [Bostrom and Yudkowsky, 2014, Ruggieri et al., 2010, Hardt et al., 2016], avoiding technical debt…”
BackgroundMathews, S. M. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-22868-2_90 · Cited by 154 · see citing works
Applications of machine learning in addiction studies: A systematic review
Mak, K. K., Lee, K., & Park, C. (2019) · Psychiatry Research · doi:10.1016/j.psychres.2019.03.001 · Cited by 148 · see citing works
Understanding artificial intelligence ethics and safetyPreprint
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 268 · see citing works
Bias Mitigation Post-processing for Individual and Group Fairness
Lohia, P., Ramamurthy, K. N., Bhide, M. A., Saha, D., Varshney, K. R., & Puri, R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682620 · Cited by 135 · see citing works
Chang, Y.-L., & Ke, J. (2023) · Human Resource Development Review · doi:10.1177/15344843231200930 · Cited by 105 · see citing works
“This principle re fl ects the requirement that AI-enabled PA must perform in safely and securely, avoiding malicious use as well as preventing unintentional harm and adverse impacts, even with good intentions (Jobin et al., 2019; Varshney & Alemzadeh, 2017).”
BackgroundProbabilistic Object Detection: Definition and Evaluation
Hall, D., Dayoub, F., Skinner, J., Zhang, H., Miller, D., Corke, P., Carneiro, G., Angelova, A., & Sünderhauf, N. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093599 · Cited by 129 · see citing works
“For safe and trusted operation in robots or autonomous systems, CNNs must express meaningful uncertainty information [1, 14, 15, 19, 43, 49].”
BackgroundMachine learning in medicine: should the pursuit of enhanced interpretability be abandoned?
Yoon, C. H., Torrance, R., & Scheinerman, N. (2021) · Journal of Medical Ethics · doi:10.1136/medethics-2020-107102 · Cited by 117 · see citing works
Characterizing Uncertainty in Machine Learning for Chemistry
Heid, E., McGill, C. J., Vermeire, F. H., & Green, W. H. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00373 · Cited by 115 · see citing works
Detecting Statistical Interactions from Neural Network WeightsPreprint
Tsang, M., Cheng, D., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04977 · Cited by 133 · see citing works
“The interpretability of machine learning models is important due to implications for communicating these models to a variety of practitioners and stakeholders (Varshney & Alemzadeh, 2016).”
BackgroundA Roadmap for a Rigorous Science of Interpretability.Preprint
Doshi‐Velez, F., & Kim, B. (2017) · arXiv (Cornell University) · Cited by 97 · see citing works
Novel Machine Learning Approaches for Modelling the Gully Erosion Susceptibility
Arabameri, A., Nalivan, O. A., Pal, S. C., Chakrabortty, R., Saha, A., Lee, S., Pradhan, B., & Bui, D. T. (2020) · Remote Sensing · doi:10.3390/rs12172833 · Cited by 95 · see citing works
“They sometimes rely on grouping, convergence, and object image processing algorithms [51].”
BackgroundSteimers, A., & Schneider, M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19063641 · Cited by 97 · see citing works
“In the case of a collaborative robot, however, this would also mean that the robot arm would first have to assume a safe state [52].”
BackgroundSoftware engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81 · see citing works
“Varshney et al. [103] discussed the definition of safety from the view of reduction or minimization of risk and epistemic uncertainty associated with unwanted outcomes that are severe enough to be seen as harmful.”
Challenges of Machine Learning Applied to Safety-Critical Cyber-Physical Systems
Pereira, A., & Thomas, C. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make2040031 · Cited by 80 · see citing works
“This issue is called covariate/distribution shift, which is one form of epistemic uncertainty [21].”
BackgroundResultInfluentialDetecting Deepfake Images Using Deep Learning Techniques and Explainable AI Methods
Abir, W. H., Khanam, F. R., Alam, K. N., Hadjouni, M., Elmannai, H., Bourouis, S., Dey, R., & Khan, M. M. (2022) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.029653 · Cited by 79 · see citing works
“Therefore, questions of trust and safety in ML are important [13].”
BackgroundA Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · see citing works
Raab, D., Theissler, A., & Spiliopoulou, M. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07809-x · Cited by 79 · see citing works
“At this, explainability and interpretability are essential for ensuring that a prediction is made based on traceable reasons [13] and for addressing the lack of transparency that already has led to incidents [20].”
BackgroundLook Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 132 · see citing works
“Using ML models without appropriate risk assessment could potentially lead to particular concerns and threats regarding trustworthiness, e.g., safety [63], security [64], and ethics [65].”
A Survey and Perspective on Industrial Cyber-Physical Systems (ICPS): From ICPS to AI-Augmented ICPS
Chae, J., Lee, S., Jang, J., Hong, S., & Park, K. (2023) · IEEE Transactions on Industrial Cyber-Physical Systems · doi:10.1109/ticps.2023.3323600 · Cited by 71 · see citing works
“Within this context, ensuring the stability of AI models emerges as a critical task that needs to be addressed to facilitate the successful establishment of AICPS [100].”
BackgroundA Review of Artificial Intelligence Algorithms Used for Smart Machine Tools
Chang, C., Lee, H.-W., & Liu, C.-H. (2018) · Inventions · doi:10.3390/inventions3030041 · Cited by 72 · see citing works
“Varshney and Alemzadeh [141] discussed machine learning schemes and safety in all sorts of CPS applications.”
BackgroundAleatoric and Epistemic Uncertainty with Random Forests
Shaker, M., & Hüllermeier, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-44584-3_35 · Cited by 68 · see citing works
Uncertainty propagation for dropout-based Bayesian neural networks
Mae, Y., Kumagai, W., & Kanamori, T. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.005 · Cited by 67 · see citing works
Machine learning applications in river research: Trends, opportunities and challenges
Ho, L. T., & Goethals, P. L. M. (2022) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.13992 · Cited by 69 · see citing works
“This bias highlights the lack of transparency and accountability in the results of these black-box models (Varshney & Alemzadeh, 2017).”
BackgroundResultHu, J., Huang, M.-C., & Yu, X. B. (2020) · Accident Analysis & Prevention · doi:10.1016/j.aap.2020.105665 · Cited by 62 · see citing works
Fernández, J., Chiachío, M., Chiachío, J., Muñoz, R., & Herrera, F. (2021) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2021.104511 · Cited by 62 · see citing works
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 374 · see citing works
“We can highlight that in the 11 primary studies discussing roadmaps, they are often related to safety and the standard ISO 26262 [7, 66, 72, 105, 201].”
BackgroundAn interpretable unsupervised Bayesian network model for fault detection and diagnosis
Yang, W.-T., Reis, M. S., Borodin, V., Juge, M., & Roussy, A. (2022) · Control Engineering Practice · doi:10.1016/j.conengprac.2022.105304 · Cited by 54 · see citing works
Favaro, A., Tsai, Y.-T., Butala, A., Thebaud, T., Villalba, J., Dehak, N., & Moro-Velázquez, L. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107559 · Cited by 52 · see citing works
A feature-vector generative adversarial network for evading PDF malware classifiers
Li, Y., Wang, Y., Wang, Y., Ke, L., & Tan, Y. (2020) · Information Sciences · doi:10.1016/j.ins.2020.02.075 · Cited by 43 · see citing works
Mining Campus Big Data: Prediction of Career Choice Using Interpretable Machine Learning Method
Wang, Y., Yang, L., Wu, J. Y., Song, Z., & Shi, L. (2022) · Mathematics · doi:10.3390/math10081289 · Cited by 40 · see citing works
“In addition, it is noteworthy that the application of machine learning needs to be fully understood, and such interventions may have a potentially long-lasting impact on people’s learning, development, and life-long functioning [27].”
BackgroundHagos, D. H., & Rawat, D. B. (2022) · Sensors · doi:10.3390/s22249916 · Cited by 44 · see citing works
“Hence, building explainability and interpretability into AI models and techniques of critical systems also create impacts on safety [88], ethics [89–91], law [92–94], and transferability [95].”
BackgroundARM-Net: Adaptive Relation Modeling Network for Structured Data
Cai, S., Zheng, K., Chen, G., Jagadish, H. V., Ooi, B. C., & Zhang, M. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3457321 · Cited by 40 · see citing works
“Despite their great predictive power, many “black box” models such as DNNs model inputs in an implicit way, which is inexplicable and sometimes can learn unintended patterns [53, 60].”
BackgroundImproving Confidence Estimates for Unfamiliar Examples
Li, Z., & Hoiem, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00276 · Cited by 38 · see citing works
“Scholars that study the impact of AI on society consider differently distributed samples to be a major risk [49]: “This is one form of epistemic uncertainty that is quite relevant to safety because training on a dataset from a different distribution can cause much harm.””
BackgroundWhen something goes wrong: Who is responsible for errors in ML decision-making?
Berber, A., & Srećković, S. (2023) · AI & Society · doi:10.1007/s00146-023-01640-1 · Cited by 44 · see citing works
“Making decisions about people’s lives based on opaque decision-making raises ethical questions of whether this decision-making is trustworthy, justified, and ethical (Wex-ler 2017; Rudin 2019; Varshney and Alemzadeh 2017; Flo-res et al. 2016; Wang et al. 2019; Yeung 2019; Russell and Norvig 2016).”
Machine Learning in Cyber-Physical Systems in Industry 4.0
Ahmed, R. S., Ali, E. S., & Saeed, R. A. (2020) · Advances in systems analysis, software engineering, and high performance computing book series · doi:10.4018/978-1-7998-5101-1.ch002 · Cited by 34 · see citing works
Carvalho, A. M., Sampaio, P. A. D. C. A., Rebentisch, E., & Oehmen, J. (2020) · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · Cited by 34 · see citing works
Robustness of AI-based prognostic and systems health management
Khan, S., Tsutsumi, S., Yairi, T., & Nakasuka, S. (2021) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2021.04.001 · Cited by 33 · see citing works
Socially Responsible AI Algorithms: Issues, Purposes, and ChallengesPreprint
Cheng, L., Varshney, K. R., & Liu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.02032 · Cited by 34 · see citing works
“Because AI systems operate in a world with much uncertainty, volatility, and flux, another objective of SRAs is to be safe, accurate, and reliable [200].”
BackgroundZhou, P., Zuo, D., Hou, K. M., Zhang, Z., Dong, J., Li, J., & Zhou, H. (2019) · Sensors · doi:10.3390/s19051033 · Cited by 31 · see citing works
“(9) Safe “AI” methods and related safety V&V methods are necessary for data driven self-adaptation and fault predication [169].”
BackgroundReview of Applications of Artificial Intelligence Algorithms in Pavement Management
Yang, X., & Zhang, Z. (2022) · Journal of Transportation Engineering Part B Pavements · doi:10.1061/jpeodx.0000373 · Cited by 31 · see citing works
Significance-based decision tree for interpretable categorical data clustering
Hu, L., Jiang, M., Liu, X., & He, Z. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121588 · Cited by 27 · see citing works
Benchmarking Sampling-based Probabilistic Object Detectors.
Miller, D., Suenderhauf, N., Zhang, H., Hall, D., & Dayoub, F. (2019) · QUT ePrints (Queensland University of Technology) · Cited by 27 · see citing works
Metacognition for artificial intelligence system safety – An approach to safe and desired behavior
Johnson, B. (2022) · Safety Science · doi:10.1016/j.ssci.2022.105743 · Cited by 27 · see citing works
Gharib, M., Lollini, P., Botta, M., Amparore, E. G., Donatelli, S., & Bondavalli, A. (2018) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w.2018.00074 · Cited by 25 · see citing works
“Yet the adequacy of such standards for ML remains controversial sin ce they, usually, do not address the special characteristics o f MLbased components such as the presence of non-determinism, limited or missing transparency, uncertain error rate, ins tability of results [3], [8], [9].”
BackgroundReal-time process safety and systems decision-making toward safe and smart chemical manufacturing
Braniff, A., Akundi, S. S., Liu, Y.-X., Dantas, B., Niknezhad, S. S., Khan, F., Pistikopoulos, E., & Tian, Y. (2025) · Digital Chemical Engineering · doi:10.1016/j.dche.2025.100227 · Cited by 26 · see citing works
Towards safe machine learning for CPS
Gu, X., & Easwaran, A. (2019) · ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS) · doi:10.1145/3302509.3311038 · Cited by 25 · see citing works
“We can apply various techniques to improve the safety/reliability of ML models [21].”
MethodDesigning Safety Critical Software Systems to Manage Inherent Uncertainty
Serban, A. C. (2019) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c.2019.00051 · Cited by 25 · see citing works
“These properties of DL algorithms can severely impact the safety of systems using them [6, 7].”
BackgroundHybrid Knowledge and Data Driven Synthesis of Runtime Monitors for Cyber-Physical Systems
Zhou, X., Ahmed, B., Aylor, J. H., Asare, P., & Alemzadeh, H. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2023.3242653 · Cited by 24 · see citing works
“Nevertheless, they suffer from the common problems of ML-based systems, including limited availability of labeled datasets, lack of model transparency, and suboptimal performance for corner cases [20], [21], [22], [23].”
BackgroundKamimura, R. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.09.036 · Cited by 24 · see citing works
Transparency: Motivations and ChallengesPreprint
Weller, A. (2017) · arXiv (Cornell University) · doi:10.17863/cam.101505 · Cited by 29 · see citing works
Bachinger, F., Kronberger, G., & Affenzeller, M. (2021) · IET Collaborative Intelligent Manufacturing · doi:10.1049/cim2.12009 · Cited by 26 · see citing works
“The creation of highly detailed simulation models of the production system—the so‐called digital twins—is therefore essential for evaluation of learned control policies by the agent [4,5].”
BackgroundMacKay, C., Klement, W., Vanberkel, P. T., Lamond, N. W. D., Urquhart, R., & Rigby, M. H. (2023) · Healthcare Analytics · doi:10.1016/j.health.2023.100155 · Cited by 20 · see citing works
Ethics and Bias in Machine Learning: A Technical Study of What Makes Us “Good”
Shadowen, N. (2019) · Springer eBooks · doi:10.1007/978-3-030-16920-6_12 · Cited by 20 · see citing works
Machine Learning Applications in Chemical Engineering
Yan, Y., Borhani, T. N. G., & Clough, P. T. (2020) · Royal Society of Chemistry eBooks · doi:10.1039/9781839160233-00340 · Cited by 20 · see citing works
Deep Mixture Density Network for Probabilistic Object Detection
He, Y., & Wang, J. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340882 · Cited by 20 · see citing works
“Mistakes/uncertainties in object detection could potentially risk the success of the robot’s operation and even human lives [3], [4], [5].”
BackgroundWeakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages
Kuutti, S., Bowden, R., & Fallah, S. (2021) · Sensors · doi:10.3390/s21062032 · Cited by 20 · see citing works
Deep Neural Networks Classification via Binary Error-Detecting Output Codes
Klimo, M., Lukáč, P., & Tarábek, P. (2021) · Applied Sciences · doi:10.3390/app11083563 · Cited by 19 · see citing works
“Trust and safety is an important topic in machine learning [17,18].”
BackgroundCustode, L. L., & Iacca, G. (2021) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci50451.2021.9660048 · Cited by 19 · see citing works
Nord-Bronzyk, A., Savulescu, J., Ballantyne, A., Braunack‐Mayer, A., Krishnaswamy, P., Lysaght, T., Ong, M. E. H., Liu, N., Menikoff, J., Mertens, M., et al. (2025) · Asian Bioethics Review · doi:10.1007/s41649-024-00348-8 · Cited by 18 · see citing works
Conformance Constraint Discovery
Fariha, A., Tiwari, A., Radhakrishna, A., Gulwani, S., & Meliou, A. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3452795 · Cited by 18 · see citing works
“The issue of trust, resilience, and interpretability of artificial intelligence (AI) systems has been a theme of increasing interest recently [33, 35, 55, 71], particularly for safety-critical data-driven AI systems [65, 72].”
BackgroundInterpretability and Explainability in Machine Learning
Tsang, W. K., & Benoit, D. F. (2022) · Intelligent systems reference library · doi:10.1007/978-3-031-11593-6_5 · Cited by 19 · see citing works
Interpretable machine learning with reject option
Brinkrolf, J., & Hammer, B. (2018) · at - Automatisierungstechnik · doi:10.1515/auto-2017-0123 · Cited by 18 · see citing works
“In practice, such adversarial examples constitute one reason for a limited safety of ML models if used in realistic environments [31].”
BackgroundHealth Monitoring for Cyber Physical Systems
Shangguan, L., & Gopalswamy, S. (2019) · IEEE Systems Journal · doi:10.1109/jsyst.2019.2922982 · Cited by 18 · see citing works
“, [10], [11]).”
Adaptive functional testing for autonomous trucks
Elgharbawy, M., Scherhaufer, I., Oberhollenzer, K., Frey, M., & Gauterin, F. (2018) · International Journal of Transportation Science and Technology · doi:10.1016/j.ijtst.2018.11.003 · Cited by 17 · see citing works
Minimization of constraint violation probability in model predictive control
Brüdigam, T., Gaßmann, V., Wollherr, D., & Leibold, M. (2021) · International Journal of Robust and Nonlinear Control · doi:10.1002/rnc.5636 · Cited by 16 · see citing works
Vasilaras, A., Papadoudis, N., & Rizomiliotis, P. (2024) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301737 · Cited by 16 · see citing works
Ahmad, K., Maabreh, M., Ghaly, M., Khan, K., Qadir, J., & Al‐Fuqaha, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09110 · Cited by 16 · see citing works
Kaur, H., Conrad, M. R., Rule, D., Lampe, C. A. C., & Gilbert, É. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637354 · Cited by 15 · see citing works
“These auxiliary criteria include things like “safety [6, 93, 126], nondiscrimination [12, 38, 107], avoiding technical debt [110], or providing the right to explanation [34]” [25, p1].”
BackgroundExtremely randomized neural networks for constructing prediction intervals
Mancini, T., Calvo-Pardo, H., & Olmo, J. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.08.020 · Cited by 15 · see citing works
Redefining health, risk, and safety for occupational settings: A mixed-methods study
Karanikas, N., & Zerguine, H. (2024) · Safety Science · doi:10.1016/j.ssci.2024.106698 · Cited by 14 · see citing works
Kamimura, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.01.056 · Cited by 14 · see citing works
Embedded out-of-distribution detection on an autonomous robot platform
Yuhas, M., Feng, Y., Ng, D. J. X., Rahiminasab, Z., & Easwaran, A. (2021) · IEEE Workshop on Design Automation for CPS and IoT (DESTION) · doi:10.1145/3445034.3460509 · Cited by 13 · see citing works
“Machine learning’s prediction power degrades when testing data shift away from distributions of model training data’s underlying properties [19].”
BackgroundExplainable Fraud Detection with Deep Symbolic Classification
Visbeek, S., Acar, E., & den Hengst, F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63800-8_18 · Cited by 13 · see citing works
Gazos, A., Kahn, J., Kusche, I., Büscher, C., & Götz, M. (2024) · Safety Science · doi:10.1016/j.ssci.2024.106731 · Cited by 16 · see citing works
Morales‐Forero, A., Bassetto, S., & Coatanéa, É. (2022) · AI & Society · doi:10.1007/s00146-022-01591-z · Cited by 13 · see citing works
“The opaque nature of DNN, for instance, can still hide unknown shifts in the data and might provide wrong inferences based on incorrect patterns (Varshney and Alemzadeh 2017).”
BackgroundStatistical robustness of Markov chain Monte Carlo accelerators
Zhang, X., Bashizade, R., Wang, Y., Mukherjee, S., & Lebeck, A. R. (2021) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3445814.3446697 · Cited by 12 · see citing works
The Artificial Intelligence Doctor: Considerations for the Clinical Implementation of Ethical AI
Kernbach, J. M., Hakvoort, K., Ort, J., Clusmann, H., Neuloh, G., & Delev, D. (2021) · Acta neurochirurgica. Supplementum · doi:10.1007/978-3-030-85292-4_29 · Cited by 12 · see citing works
Gharib, M., Zoppi, T., & Bondavalli, A. (2022) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2022.3178631 · Cited by 12 · see citing works
“For example, Varshney and Alemzadeh [28] discussed four main categories of approaches/strategies for increasing safety in ML namely: inherently safe design, safety margins, safe fail, and procedural safeguards.”
BackgroundSun, X., Li, X., Ren, K., Song, J., Xu, Z., & Chen, J. (2019) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2019.101660 · Cited by 12 · see citing works
Varma, Y. (2020) · International Journal of Emerging Trends in Computer Science and Information Technology · doi:10.63282/3050-9246.ijetcsit-v1i1p104 · Cited by 12 · see citing works
Promoting Distributed Trust in Machine Learning and Computational Simulation
Bore, N., Raman, R. K., Markus, I. M., Remy, S. L., Bent, O., Hind, M., Pissadaki, E. K., Srivastava, B., Vaculín, R., Varshney, K. R., et al. (2019) · IEEE International Conference on Blockchain and Cryptocurrency (ICBC) · doi:10.1109/bloc.2019.8751423 · Cited by 12 · see citing works
“The mechanism embedded in these two capabilities is trust, and the need to provide added layers of trust is crucial for technical, practical and ethical or legal reasons [6], [7].”
Machine Learning Algorithms for Financial Asset Price ForecastingPreprint
Ndikum, P. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2004.01504 · Cited by 12 · see citing works
“Additionally Recital 71 of the GDPR affords consumers the rights to ask private institutions to explain ML algorithms [78], [79].”
BackgroundTrustworthy AI in education: Framework, cases, and governance strategies
Ma, Y., Li, X., Hu, S., Liu, S., & Cheong, K. H. (2025) · Innovation and Emerging Technologies · doi:10.1142/s2737599425500264 · Cited by 13 · see citing works
Luthra, A., Porter, K. M., Hinton, A. M., Chao, W., Papachristou, G. I., Conwell, D. L., & Krishna, S. G. (2022) · Pancreas · doi:10.1097/mpa.0000000000002208 · Cited by 11 · see citing works
- Authors
- Varshney, K. R., & Alemzadeh, H.
- Affiliations
- IBM Research - Thomas J. Watson Research Center (US)
- University of Virginia (US)
- Corresponding author
- Varshney, Kush R.
- Venue
- Big Data
- Publisher
- SAGE Publications
- Published
- 2017-09
- Volume
- 5
- Issue
- 3
- Pages
- 246-255
- ISSN
- 2167-6461 (ISSN-L)
- 2167-647X
- OpenAlex
- W2528479067
- Cited by
- 211
- References
- 42
- Citations by year
- 2026: 13 · 2025: 23 · 2024: 19 · 2023: 17 · 2022: 24 · 2021: 28 · 2020: 35 · 2019: 32 · 2018: 15 · 2017: 5
- FWCI
- 54.49
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Big Data (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- machine learning safety, risk, epistemic uncertainty, harm, cyber-physical systems, decision science, data products, empirical risk minimization, inherently safe design, safety reserves, procedural guarantees, interpretability, causality, human-in-the-loop annotation, user experience design, open data
- MeSH terms
- Machine Learning, Algorithms, Decision Making, Safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
