Trustworthy AI: From Principles to Practices
Open accessThe DOI matches its Crossref record; 398 of the 399 references it cites were found in a registry or an index; 74 of the 251 sources its claims cite were checked for support.
Abstract
The rapid development of Artificial Intelligence (AI) technology has enabled the deployment of various systems based on it. However, many current AI systems are found vulnerable to imperceptible attacks, biased against underrepresented groups, lacking in user privacy protection. These shortcomings degrade user experience and erode people’s trust in all AI systems. In this review, we provide AI practitioners with a comprehensive guide for building trustworthy AI systems. We first introduce the theoretical framework of important aspects of AI trustworthiness, including robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, and accountability. To unify currently available but fragmented approaches toward trustworthy AI, we organize them in a systematic approach that considers the entire lifecycle of AI systems, ranging from data acquisition to model development, to system development and deployment, finally to continuous monitoring and governance.
Overview
CoChat Summary
TL;DR
This review provides a comprehensive guide to trustworthy AI, organizing robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, and accountability across the AI system lifecycle. It proposes TrustAIOps as a continuous workflow and identifies open challenges including trade-offs, immature evaluation, and risks from large pre-trained models.
Key findings
- The paper defines eight trustworthiness aspects—robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, and accountability—and argues that predictive accuracy alone is insufficient; trustworthiness must be assessed systematically across data preparation, algorithm design, development, deployment, and management.
- It proposes TrustAIOps, extending MLOps/DevOps with aligned trustworthiness principles, interdisciplinary collaboration, extensive artifact management, and continuous feedback loops across the five lifecycle stages.
- It documents frictions and trade-offs: transparency can leak private data or intellectual property and enable targeted hacking; adversarial robustness can reduce accuracy; robustness and fairness can negatively affect each other; and explanations can be fragile, inconsistent with human interpretation, or hard to judge as faithful.
- In privacy evaluation, it cites work showing that 15 demographic attributes were sufficient to render 99% of participants unique.
Methods
- Design: narrative review/survey, not an empirical study; it synthesizes technical, engineering, and governance literature on trustworthy AI.
- Data or sample: no dataset or sample is used; the paper draws on cited literature and industrial case studies of face recognition, autonomous driving, and NLP in the appendix.
- Main analysis: conceptual organization of approaches by trustworthiness aspect and by lifecycle stage, culminating in a proposed systematic framework and TrustAIOps workflow.
Limitations
- The paper states that explainability, transparency, and accountability are still seldom evaluated quantitatively; approaches to robustness and explainability remain immature; trade-offs and frictions exist between trustworthiness aspects; large pre-trained models are costly and difficult to reproduce and may carry bias or privacy leakage; and end-user awareness is limited.
- Evident from the text: as a survey, it presents no original experiments or quantitative validation of the proposed TrustAIOps workflow, and its recommendations are conceptual rather than empirically tested in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Valid · 28 Not valid · 96 Not checked · 119 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It is widely recognized that AI models might be vulnerable to attacks by adversaries with malicious intentions.”
What the sources say
- Partly supportedcites [68] [68] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1-11, 2019. - sourceExperimental — accuracy not yet measured
The Face Manipulation Methods section says forgery detectors have robustness issues and that compression and resizing can erase traces they use to detect manipulation. That supports vulnerability in these particular models, but the paper does not establish the broader claim that AI models are vulnerable to intentional attacks by malicious adversaries. (LLM, conf=0.78)
- Not matchedcites [304]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [11] [11] Sarah Bankins and Paul Formosa. Ethical ai at work: The social contract for artificial intelligence and its implications for the workplace psychological contract. Redefining the Psychological Contract in the Digital Era: Issues for Research and Practice , pages 55-72, 2021. - source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“many current AI systems are found vulnerable to imperceptible attacks, biased against underrepresented groups, lacking in user privacy protection.”
What the sources say
- Could not verifycites [68] [68] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1-11, 2019. - sourceExperimental — accuracy not yet measured
The Face Manipulation Methods section discusses robustness of facial-forgery detectors to compression and resizing, and §4.2.2 reports lower detection performance on compressed videos. Searches for imperceptible adversarial attacks and perturbations found no passage saying that many current AI systems are vulnerable to such attacks. (LLM, conf=0.74)
- Not matchedcites [304]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [11] [11] Sarah Bankins and Paul Formosa. Ethical ai at work: The social contract for artificial intelligence and its implications for the workplace psychological contract. Redefining the Psychological Contract in the Digital Era: Issues for Research and Practice , pages 55-72, 2021. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, liveness detection is still vulnerable against new types of attacks by fake videos generated by AI techniques like DeepFake [54].”
What the sources say
- Partly supportedcites [54] [54] Thanh Thi Nguyen, Quoc Viet Hung Nguyen, Cuong M Nguyen, Dung Nguyen, Duc Thanh Nguyen, and Saeid Nahavandi. Deep learning for deepfakes creation and detection: A survey. arXiv preprint arXiv:1909.11573 , 2019. - sourceExperimental — accuracy not yet measured
Section 3, “Deepfake Detection,” reports that tested face-recognition systems failed to detect deepfakes effectively and that other methods had high error rates on a newly produced deepfake-video dataset. This supports vulnerability to AI-generated fake videos, but the passage does not specifically assess liveness detection. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Research has shown that face recognition systems can be easily spoofed by impersonation attackers wearing a small printed adversarial patch [90, 30, 43, 73].”
What the sources say
- Supportedcites [90] [90] Zihao Xiao, Xianfeng Gao, Chilin Fu, Yinpeng Dong, Wei Gao, Xiaolu Zhang, Jun Zhou, and Jun Zhu. Improving transferability of adversarial patches on face recognition with generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11845-11854, 2021. - source
The Introduction describes adversarial eyeglass frames used to impersonate another identity and says physical patches can fool face-recognition models. Section 4.5 reports an attacker wearing printed adversarial eyeglass frames and finding that the patches retained high transferability after printing and photographing. The physical-world experiment involved one volunteer and three target identities. (LLM, conf=0.86)
- Partly supportedcites [30] [30] Ying Guo, Xingxing Wei, Guoqiu Wang, and Bo Zhang. Meaningful adversarial stickers for face recognition in physical world. arXiv:2104.06728 , 2021. - sourceExperimental — accuracy not yet measured
Section 5.3 reports successful impersonation attacks using facial stickers and shows examples in the physical world. But Section 3.5 explicitly says the method does not involve printing: subjects wear existing stickers placed according to calculated attack parameters. Thus the paper supports the impersonation point, but not the description of a printed adversarial patch. (LLM, conf=0.89)
- Partly supportedcites [43] [43] Stepan Komkov and Aleksandr Petiushko. Advhat: Real-world adversarial attack on arcface face id system. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 819-826. IEEE, 2021. - sourceExperimental — accuracy not yet measured
The Abstract and Introduction (p. 1) describe an easily reproducible attack using a printed sticker on a hat to confuse a Face ID model. But the paper’s results concern making the wearer less recognizable, not impersonating another person: Section 2.1 (p. 2) distinguishes impersonation from dodging, and the Testing Protocol (p. 5) describes its attacks as dodging attacks. (LLM, conf=0.90)
- Partly supportedcites [73] [73] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security (TOPS) , 22(3):1-30, 2019. - sourceExperimental — accuracy not yet measured
The Introduction and Experiment Setup (pages unavailable in our copy) describe impersonation attacks against face-recognition DNNs using printed adversarial patterns affixed to eyeglasses worn by attackers. The source does not establish that face-recognition systems generally are easily spoofed by a small patch: it tests particular DNNs and eyeglass-based attacks under controlled conditions, and notes that some physical impersonation attempts failed. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Some pre-trained models are reported to output training text data containing private user information such as addresses [63].”
What the sources say
- Could not verifycites [63] [63] Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency , pages 33-44, 2020. - sourceExperimental — accuracy not yet measured
Section 4.4.2 discusses documentation for trained models, and Section 4.6 discusses privacy practices for face data; neither reports a model outputting private information from its training text. Searches for pre-trained models, training-text leakage, and addresses found no passage addressing the claim. The retrieved passages provide section labels but no page numbers. (LLM, conf=0.76)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
72 more claims · 5 with their cited sources checked
Create a free account to see all 282 claims and to verify the 38 references still open.
- Probably not a substantial claim. 191
- It may state two separate claims as one. 43
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An effective automated audit system for authorship and provenance compliance functions as a continuous pipeline rather than a retrospective review tool [28].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reliable provenance begins at the moment content is produced, not after distribution [28].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This high-level understanding will help optimize trust, as operators will be aware of the capabilities and limitations of the AI systems they work with and know the appropriate settings for use104.”
From Adopting and expanding ethical principles for generative artificial intelligence from military to healthcare (2023) · cites this paper as [104]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This may be deliberate, to protect brand perception or avoid licensing obligations, or unintentional, especially in creative interfaces that mask algorithmic influence behind seemingly manual controls [28].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 Trustworthy AI: A Systematic Approach · p. 13
“the lifecycle of the development of a typical AI product can be partitioned into data preparation, algorithm design, development-deployment, and management [25].”
§ 2.6 Privacy protection · p. 12
“the five main stages of the lifecycle of an AI system, i.e., data preparation, algorithm design, development, deployment, and management.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · cited 1×
“For example, in computer vision, an established successful paradigm involves pre-training models on a largescale dataset, such as ImageNet, and then fine-tuning them on target tasks with fewer training data [131, 224, 375].”
MethodRich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · cited 1×
“For example, in computer vision, an established successful paradigm involves pre-training models on a largescale dataset, such as ImageNet, and then fine-tuning them on target tasks with fewer training data [131, 224, 375].”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 2×
“This challenge arises from various practical AI tasks [343, 391] in the area of transfer learning [255, 350].”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 3×
“In NLP or Computer Vision (CV), gradients or their variants are used to back-trace the decision of the model to the location of the most intimately related input, in the form of saliency maps and sentence highlights [250, 302, 314, 375].”
BackgroundMethodKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · cited 1×
“, Momentum Contrast (MoCo) [150] and Sequence Contrastive learning (SeCo) [368]).”
BackgroundGauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 1×
“In the well-known class activation mapping [389], an average pooling layer at the end of a DNN can also be regarded as an explainable linear bottleneck.”
BackgroundSelf-citationArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · cited 9×
“The problem of how DNNs outperform other conventional ML approaches has been puzzling researchers [24].”
BackgroundMethodInfluentialExplaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · cited 3×
“This is typically referred to as adversarial training [134, 211, 346].”
BackgroundTowards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859 · cited 1×
“empirical robustness, on a test dataset has been widely used [65, 312].”
BackgroundArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781
Advances and Open Problems in Federated Learning
Kairouz, P., & McMahan, H. B. (2021) · Foundations and Trends in Machine Learning · doi:10.1561/2200000083 · Cited by 7,772 · cited 5×
“In Federated Learning, pages 240–254.”
BackgroundInfluentialCalibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006) · Lecture Notes in Computer Science · doi:10.1007/11681878_14 · Cited by 7,272
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · cited 1×
“It has also been applied to single- and multi-machine computing environments, and integrated with ML models for protecting model privacy [2, 120, 349].”
MethodFederated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322 · cited 5×
“In Federated Learning, pages 240–254.”
BackgroundInfluentialPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“Explaining ML models has been an active topic not only in ML research but also in psychological research in the past five years [5, 24, 47, 141, 216, 250].”
BackgroundWeiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 7×
“Comprehensive statistics relating to and comparisons between these principles have been provided in References [145, 178].”
BackgroundInfluentialA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · cited 8×
“Fairness can be applicable at multiple granularities of system behavior [66, 242, 339].”
BackgroundInfluentialA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
The Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557 · cited 2×
“Data Preprocessing Anomaly Detection [70, 81, 257, 316] Data Anonymozation [14, 37, 232, 232] Differential Privacy [107, 110, 177]”
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375 · cited 1×
“The augmentation can be carried out in a brute-force manner by feeding both the original data and adversarial samples during training [201], or by using a regularization term to implicitly represent adversarial samples [134].”
BackgroundDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 19×
“From the theoretical aspect, the problem of howDNNmodels outperform other conventional ML approaches has been puzzling researchers.”
BackgroundInfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · cited 1×
“Although the definition of the explainability of an AI model is still an open question, research has sought to address this problem from the perspectives of AI [141, 285] and psychology [144, 245].”
BackgroundMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · cited 3×
“Control systems for robots can be compared and benchmarked in simulated environments such as Gazebo [214], MuJoCo [365], VerifAI [119], etc. Similarly, simulators for self-driving cars have also been widely used, including TORCS [404], CarSim [48], PRESCAN [364], etc.”
MethodPrototypical Networks for Few-shot Learning
Snell, J., Swersky, K., & Zemel, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05175 · Cited by 4,755 · cited 1×
“Feature similarity [190, 308] has been used as a representative type of knowledge prior in works such as the Model-Agnostic Meta-Learning (MAML) [119], reinforcement learning [212], and memory-augmented neural network [38, 291].”
MethodLearning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505
Bertrand, M., & Mullainathan, S. (2004) · American Economic Review · doi:10.1257/0002828042002561 · Cited by 4,504
The future of digital health with federated learning
Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00323-1 · Cited by 4,015 · cited 1×
“It has been applied to numerous domains, such as healthcare [282, 299] and finance [223].”
MethodDeep CORAL: Correlation Alignment for Deep Domain Adaptation
Sun, B., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_35 · Cited by 3,757
Design and use paradigms for gazebo, an open-source multi-robot simulator
Koenig, N., & Howard, A. (2005) · 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566) · doi:10.1109/iros.2004.1389727 · Cited by 3,638
The Future of Digital Health with Federated Learning
Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B., Maier-Hein, K., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.08119 · Cited by 3,495
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450
Self-citationFaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Generalizing from a Few Examples: A Survey on Few-shot Learning
Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2021) · ACM Computing Surveys · doi:10.1145/3386252 · Cited by 3,165 · cited 1×
“of few-shots learning is to generalize model to new tasks using only a few examples [78, 348, 371].”
BackgroundUnderstanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 5×
“Algorithm Design Adversarial Robustness Adversarial Robustness [12, 19, 45, 69, 304, 346, 373] Poisoning defense [213] Explainability ML Explainability ML [24, 47, 101, 101, 141, 250] Model Generalization Model Generalization [1, 23, 38, 124, 133, 183, 377] Domain Generalization [343, 391] Algorithmic Fairness Fairnes…”
BackgroundInfluentialDeep Learning for Anomaly Detection: A Review
Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2022) · ACM Computing Surveys · doi:10.1145/3439950 · Cited by 2,837
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“of qualified candidates among male and female candidates (known as test fairness [80, 147]).”
BackgroundUnsupervised Domain Adaptation by Backpropagation
Ganin, Y., & Lempitsky, V. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.7495 · Cited by 2,521
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“As an extension of human–computer interaction (HCI), human–AI interaction has aroused wide attention in the AI industry [4, 18].”
BackgroundFederated Learning With Differential Privacy: Algorithms and Performance Analysis
Wei, K., Li, J., Ding, M., Ma, C., Yang, H. H., Farokhi, F., Jin, S., Quek, T. Q. S., & Vincent Poor, H. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.2988575 · Cited by 2,400 · cited 1×
“It has also been applied to single- and multi-machine computing environments, and integrated with ML models for protecting model privacy [2, 120, 349].”
MethodCausal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336 · cited 1×
“Some learning paradigms other than conventional models are also considered to be explainable, such as causal inference [197, 259] and the knowledge graph [345].”
BackgroundArtificial intelligence and statistics
Yu, B., & Kumbier, K. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700813 · Cited by 2,329
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 11×
“7, documentation is an effective and important approach to enhance the system’s transparency and accountability by tracking, guiding, and auditing its entire lifecycle [272] and serves as a cornerstone of building a trustworthy AI system.”
BackgroundMethodInfluentialRobust physical-world attacks on deep learning visual classification
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00175 · Cited by 2,173
Self-citationAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2×
“, explainer, such as a linear model [225, 279] and rules [140, 280].”
BackgroundFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 1×
“of qualified candidates among male and female candidates (known as test fairness [80, 147]).”
BackgroundEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“From the perspective of practical systems, there is a demand among users for the right to know the intention, business model, and technological mechanism of AI products [9, 135].”
BackgroundMS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
Guo, Y., Zhang, L., Hu, Y., He, X., & Gao, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_6 · Cited by 2,122
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics 2016
Murray, I., & Graham, M. M. (2016) · Cited by 2,115
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021
Jain, S., & Wallace, B. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1357 · Cited by 1,987
Rademacher and Gaussian Complexities: Risk Bounds and Structural Results
Bartlett, P. L., & Mendelson, S. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44581-1_15 · Cited by 1,949 · cited 1×
“For example, Rademacher complexity [35] is commonly used to determine how well a model can it a random assignment of class labels.”
SecureML: A System for Scalable Privacy-Preserving Machine Learning
Mohassel, P., & Zhang, Y. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.12 · Cited by 1,936 · cited 1×
“, linear regression [128] and logistic regression [300], and generic model learning tasks [247].”
BackgroundThe Ethics of AI Ethics: An Evaluation of Guidelines
Hagendorff, T. (2020) · Minds and Machines · doi:10.1007/s11023-020-09517-8 · Cited by 1,911 · cited 4×
“Comprehensive statistics relating to and comparisons between these principles have been provided in References [145, 178].”
BackgroundInfluentialThe Ethics of AI Ethics -- An Evaluation of Guidelines
Hagendorff, T. (2019) · arXiv · doi:10.48550/arxiv.1903.03425 · Cited by 1,907 · cited 4×
“Comprehensive statistics relating to and comparisons between these principles have been provided in References [145, 178].”
BackgroundInfluentialAdvances in Neural Information Processing Systems 5
Zemel, R. S., Williams, C. K. I., & Mozer, M. C. (1993) · Cited by 1,896
RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
Erlingsson, Ú., Pihur, V., & Korolova, A. (2014) · Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2660267.2660348 · Cited by 1,838 · cited 1×
“In Reference [113], researchers proposed the RAPPOR algorithm that satisfies the definition of DP.”
BackgroundCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · cited 2×
“, disparate impact [117]); for realvalued regression, the difference between mean distances to the true value aggregated over each group might be used instead to indicate whether we model one group significantly better than another [59].”
BackgroundHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 3×
“For deep learning models like the Convolutional Neural Network (CNN) or the transformer, the inspection of intermediate features is a widely used means of explaining model behavior [332, 366].”
BackgroundMethodSelf-citationAdversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · cited 4×
“Comprehensive surveys can be found in the literature such as [12, 19, 45, 69, 213, 304, 373].”
BackgroundInfluentialDeep Learning for Anomaly Detection: A Review
Pang, G., Shen, C., Cao, L., et al. (2020) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · doi:10.48550/arxiv.2007.02500 · Cited by 1,718
Sheller, M. J., Edwards, B., Reina, G. A., Martin, J., Pati, S., Kotrotsou, A., Milchenko, M., Xu, W., Marcus, D., Colen, R. R., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-69250-1 · Cited by 1,646
A Unified Framework of Five Principles for AI in Society
Floridi, L., & Cowls, J. (2019) · Harvard Data Science Review · doi:10.1162/99608f92.8cd550d1 · Cited by 1,631 · cited 1×
“A number of enterprises [57, 62, 136, 166, 254, 338], academia [122, 199, 218, 301, 322], public sectors, and organizations [9, 210, 334] have recently identified these factors and summarized them as principles of AI trustworthiness.”
BackgroundReconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595 · cited 21×
“As a fundamental principle of model generalization theory, the bias–variance tradeoff indicates that a generalized model should maintain a balance between underfitting and overfitting [39, 124].”
BackgroundInfluentialBellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Theoretically principled trade-off between robustness and accuracy
Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., & Jordan, M. I. (2019) · arXiv · doi:10.48550/arxiv.1901.08573 · Cited by 1,560 · cited 3×
“Adversarial robustness increases the model’s generalizability and reduces overfitting, but tends to negatively impact its overall accuracy [331, 380].”
BackgroundNeural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks
Wang, B., Yao, Y., Shan, S., Li, H., Viswanath, B., Zheng, H., & Zhao, B. Y. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00031 · Cited by 1,368 · cited 2×
“Neural Cleanse [342] proactively discovers backdoor patterns in a model.”
BackgroundDARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · cited 5×
“To promote the research on AI trustworthiness, stakeholders are actively setting up various forms of collaboration such as research workshops on trustworthy AI and cooperative projects like DARPA XAI [162].”
BackgroundInfluentialBenchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335
Domain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315 · cited 6×
“A summary of commonly used datasets and benchmarks for domain generalization can be found in Reference [391] and covers the tasks of object recognition, action recognition, segmentation, and face recognition.”
BackgroundMethodInfluentialConcrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
A Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307 · cited 21×
“From the theoretical aspect, the problem of howDNNmodels outperform other conventional ML approaches has been puzzling researchers.”
BackgroundInfluentialDeep Learning for Anomaly Detection: A Survey
Chalapathy, R., & Chawla, S. (2019) · arXiv · doi:10.48550/arxiv.1901.03407 · Cited by 1,240 · cited 5×
“Various approaches have been proposed to address this issue by using anomaly detection [70].”
BackgroundMethodInfluential(2021) · Computer Graphics Forum · doi:10.1111/(issn)1467-8659 · Cited by 1,230
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217 · cited 3×
“The concept of test coverage in software testing has been transplanted into DNN models [226, 260].”
BackgroundMethodExplainable artificial intelligence: A survey
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400040 · Cited by 1,147 · cited 1×
“Algorithm Design Adversarial Robustness Adversarial Robustness [12, 19, 45, 69, 304, 346, 373] Poisoning defense [213] Explainability ML Explainability ML [24, 47, 101, 101, 141, 250] Model Generalization Model Generalization [1, 23, 38, 124, 133, 183, 377] Domain Generalization [343, 391] Algorithmic Fairness Fairnes…”
Software Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135 · cited 2×
“Moreover, various techniques researched for this stage can contribute to the trustworthiness of an AI system [17].”
BackgroundFairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · cited 5×
“Fairness can be applicable at multiple granularities of system behavior [66, 242, 339].”
BackgroundMethodInfluentialThe oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104 · cited 1×
“Similar to canonical software testing, the problem to automatically generate the expected ground truth, known as the oracle problem [34], also occurs in the AI software testing scenario.”
MethodThe accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 1×
“, underprivileged groups might experience systematic disadvantage in hiring decisions [49], or be disproportionally impacted in criminal risk profiling [104, 146, 161]).”
BackgroundCounterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061 · cited 1×
“Counterfactual explanation [13, 185, 341] works in a contrary way by finding the boundary case to flip the outcome.”
BackgroundRETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism
Choi, E., Bahadori, M. T., Kulas, J. A., Schuetz, A., Stewart, W. F., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1608.05745 · Cited by 1,029 · cited 1×
“Attention mechanisms [30, 363] have also attracted recent interest and have been regarded as an explainable bottleneck in a DNN in some studies [79, 237].”
BackgroundEvolution Strategies as a Scalable Alternative to Reinforcement Learning
Salimans, T., Ho, J., Chen, X., Sidor, S., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1703.03864 · Cited by 1,027
Learning Deep Representation for Imbalanced Classification
Huang, C., Li, Y., Loy, C. C., & Tang, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.580 · Cited by 988 · cited 1×
“Aside from helping to balance the classification accuracy, these approaches can be applied to balance the cost of classification errors to improve performance on certain groups [163] (e.”
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 1×
“That is, if the job requires strength qualifications, then the rate of admission for qualified male and female candidates should be equal (known as equal opportunity [147]; see also equal odds [43] and accuracy equity [95]).”
BackgroundOn Early Stopping in Gradient Descent Learning
Yao, Y., Rosasco, L., & Caponnetto, A. (2007) · Constructive Approximation · doi:10.1007/s00365-006-0663-2 · Cited by 970 · cited 1×
“Other than adjusting the architecture of the model, one can mitigate overfitting to obtain better generalization via various explicit or implicit regularizations, such as early stopping [370], batch normalization [167], dropout [309], data augmentation, and weight decay [196].”
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
Carlini, N., & Wagner, D. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140444 · Cited by 966
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 1×
“However, this claim continues to be debated, because attention weights representing different explanations can produce similar final predictions [170, 355].”
BackgroundEvading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks
Dong, Y., Pang, T., Su, H., & Zhu, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00444 · Cited by 942 · cited 1×
“The corresponding defense [328] is later shown to be vulnerable against new attacks [99].”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
(2015) · neural information processing systems · Cited by 912
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906 · cited 1×
“To minimize and make the best use of human effort in such interaction, efficient design of patterns of human–machine cooperation is an emerging topic in interdisciplinary work on HCI and AI and is referred to as human-in-the-loop or interactive machine learning [157] in the literature.”
BackgroundMachine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890 · cited 9×
“This strategy is used in tasks like NLP-based reading comprehension by generating supporting sentences [373, 415].”
MethodInfluentialAI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 3×
“Management Documentation [11, 22, 41, 129, 156, 246] Auditing [58, 228, 272, 277, 290, 292, 356] Cooperation [27, 90] Worklow MLOps [165, 233, 303]”
BackgroundInterpretable Convolutional Neural Networks
Zhang, Q., Wu, Y. N., & Zhu, S.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00920 · Cited by 833 · cited 1×
“In image recognition, part-template masks can be used to regularize feature maps to focus on local semantic parts [383].”
BackgroundEstimating the success of re-identifications in incomplete datasets using generative models
Rocher, L., Hendrickx, J. M., & de Montjoye, Y.-A. (2019) · Nature Communications · doi:10.1038/s41467-019-10933-3 · Cited by 816 · cited 2×
“In practical scenarios, the empirical evaluation of the risk of leakage of privacy is usually considered [283, 360].”
BackgroundData governance: Organizing data for trustworthy Artificial Intelligence
Janssen, M., Brous, P., Estevez, E., Barbosa, L. S., & Janowski, T. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101493 · Cited by 811 · cited 4×
“By tracking the data lineage at the highest resolution, data provenance can enhance the transparency, reproducibility, and accountability of an AI system [154, 172].”
BackgroundInfluentialExplainable Reasoning over Knowledge Graphs for Recommendation
Wang, X., Wang, D., Xu, C., He, X., Cao, Y., & Chua, T.-S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015329 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · cited 2×
“In the context of the psychology of users or the public, user studies or A/B test can provide a useful evaluation based on user satisfaction [249].”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Towards identity anonymization on graphs
Liu, K., & Terzi, E. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376629 · Cited by 773 · cited 1×
“Ways of anonymizing such data have been considered in the literature [37, 220].”
BackgroundShneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765 · cited 1×
“, (1) the explanation is fragile to perturbations [130], (2) the explanation is not always consistent with human interpretation [47], and (3) it is difficult to judge if the explanation is correct or faithful [250].”
BackgroundRacial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · cited 1×
“, recognizing speech in less-spoken accents can be naturally harder due to data scarcity [191]).”
BackgroundThree naive Bayes approaches for discrimination-free classification.
Calders, T., & Verwer, S. (2010) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-010-0190-x · Cited by 764 · cited 2×
“Besides making direct use of the existing data samples, it is possible to introduce additional samples that typically involves making changes to the available samples, including through perturbation and relabeling [60, 85].”
BackgroundAccountability in algorithmic decision making
Diakopoulos, N. (2016) · Communications of the ACM · doi:10.1145/2844110 · Cited by 763 · cited 1×
“Note that in terms of information disclosure, transparency contributes the basic mechanism used to facilitate the accountability of an AI system [94, 100].”
BackgroundFormal methods: state of the art and future directions
Clarke, E. M., & Wing, J. M. (1996) · ACM Computing Surveys · doi:10.1145/242223.242257 · Cited by 763 · cited 1×
“It is also worth noting the recent trend of the intersection of certiied robustness and the perspective of formal veriication, which aims at developing rigorous mathematical speciication and techniques of veriication for assurances of software correctness [83].”
BackgroundCertified Robustness to Adversarial Examples with Differential Privacy
Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D., & Jana, S. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00044 · Cited by 750 · cited 1×
“DP is also used to improve the robustness of AI models against adversarial samples [204].”
MethodAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · cited 1×
“As an extension of human–computer interaction (HCI), human–AI interaction has aroused wide attention in the AI industry [4, 18].”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 2×
“, user satisfaction, mental models, and trust [144, 155, 267].”
BackgroundTrusted Execution Environment: What It is, and What It is Not
Sabt, M., Achemlal, M., & Bouabdallah, A. (2015) · 2015 IEEE Trustcom/BigDataSE/ISPA · doi:10.1109/trustcom.2015.357 · Cited by 724
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709
Fraud detection system: A survey
Abdallah, A., Maarof, M. A., & Zainal, A. (2016) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2016.04.007 · Cited by 708 · cited 1×
“Attack monitoring has been widely adopted in conventional SaaS, such as fraud detection [3] in e-commerce systems.”
MethodThis Looks Like That: Deep Learning for Interpretable Image Recognition
Chen, C., Li, O., Tao, C., Barnett, A. J., Su, J., & Rudin, C. (2018) · arXiv · doi:10.48550/arxiv.1806.10574 · Cited by 702 · cited 19×
“For example, previous research has embedded linear models and prototype selection to the DNN [16, 20, 73].”
MethodInfluentialA Proof of Security of Yao’s Protocol for Two-Party Computation
Lindell, Y., & Pinkas, B. (2009) · Journal of Cryptology · doi:10.1007/s00145-008-9036-8 · Cited by 651
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · cited 8×
“Fairness can be applicable at multiple granularities of system behavior [66, 242, 339].”
BackgroundMethodInfluentialFairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · cited 8×
“Fairness can be applicable at multiple granularities of system behavior [66, 242, 339].”
BackgroundMethodInfluentialA secure federated transfer learning framework
Liu, Y., Kang, Y., Xing, C., Chen, T., & Yang, Q. (2020) · IEEE Intelligent Systems · doi:10.1109/mis.2020.2988525 · Cited by 645
Secure Federated Transfer Learning
Liu, Y., Kang, Y., Xing, C., Chen, T., & Yang, Q. (2018) · arXiv · doi:10.48550/arxiv.1812.03337 · Cited by 645
Karnin, E., Greene, J., & Hellman, M. (1983) · IEEE Transactions on Information Theory · doi:10.1109/tit.1983.1056621 · Cited by 630 · cited 1×
“For the multi-party condition, secret sharing is one of the generic frameworks [181].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 6×
“Algorithm Design Adversarial Robustness Adversarial Robustness [12, 19, 45, 69, 304, 346, 373] Poisoning defense [213] Explainability ML Explainability ML [24, 47, 101, 101, 141, 250] Model Generalization Model Generalization [1, 23, 38, 124, 133, 183, 377] Domain Generalization [343, 391] Algorithmic Fairness Fairnes…”
BackgroundMethodInfluentialZhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 2×
“For example, in the field of autonomous driving, besides developing a perceptual system working in sunny scenes, academia and the industry are using numerous development and testing strategies to enhance the perceptual performance of the vehicles in nighttime/rainy scenes to guarantee the system’s reliability under a…”
BackgroundMethodExtracting Tree-Structured Representations of Trained Networks
Craven, M. W., & SHAVLIK, J. W. (1995) · neural information processing systems · Cited by 602
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundTrustworthy artificial intelligence
Thiebes, S., Lins, S., & Sunyaev, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00441-4 · Cited by 595
Adversarial Robustness Toolbox v1.0.0
Nicolae, M.-I., Sinn, M., Tran, M. N., Buesser, B., Rawat, A., Wistuba, M., Zantedeschi, V., Baracaldo, N., Chen, B., Ludwig, H., et al. (2018) · arXiv · doi:10.48550/arxiv.1807.01069 · Cited by 594
Ross, A., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11504 · Cited by 589 · cited 1×
“A similar regularization on gradients was adopted by Reference [286].”
MethodDeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 4×
“Such coverage criteria are effective for certifying the robustness of a DNN against adversarial attacks [226].”
BackgroundMethodInfluentialSelf-citationDevops: A Software Architect's Perspective
Bass, L., Weber, I. M., & Zhu, L. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 561
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
Background(2016) · neural information processing systems · Cited by 556
DARPA's explainable artificial intelligence (XAI) program
Gunning, D. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3308446 · Cited by 546
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 532 · cited 4×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundInfluentialMetrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518 · cited 1×
“, user satisfaction, mental models, and trust [144, 155, 267].”
BackgroundInterpretable Machine Learning: Moving from mythos to diagnostics
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2021) · Queue · doi:10.1145/3511299 · Cited by 516
HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
Mathew, B., Saha, P., Yimam, S. M., Biemann, C., Goyal, P., & Mukherjee, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17745 · Cited by 513 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundSecure Multi-Party Computation: Theory, practice and applications
Zhao, C., Zhao, S., Zhao, M., Chen, Z., Gao, C.-Z., Li, H., & Tan, Y.-A. (2019) · Information Sciences · doi:10.1016/j.ins.2018.10.024 · Cited by 508 · cited 8×
“A typical SMPC protocol satisfies properties of privacy, correctness, independence of inputs, guaranteed output delivery, and fairness [114, 387].”
BackgroundInfluentialDemystifying Arm TrustZone: A Comprehensive Survey
Pinto, S., & Santos, N. (2019) · ACM Computing Surveys · doi:10.1145/3291047 · Cited by 505
Aequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
Tay, B., Jung, Y., & Park, T. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.05.014 · Cited by 490
Data Cleaning: Overview and Emerging Challenges
Chu, X., Ilyas, I. F., Krishnan, S., & Wang, J. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2912574 · Cited by 488 · cited 3×
“Data Preprocessing Anomaly Detection [70, 81, 257, 316] Data Anonymozation [14, 37, 232, 232] Differential Privacy [107, 110, 177]”
BackgroundMethodA Systematic Review of Perception System and Simulators for Autonomous Vehicles Research
Rosique, F., Navarro, P. J., Fernández, C., & Padilla, A. (2019) · Sensors · doi:10.3390/s19030648 · Cited by 469
Deep Learning for Deepfakes Creation and Detection: A Survey
Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., & Nguyen, C. M. (2019) · arXiv · doi:10.48550/arxiv.1909.11573 · Cited by 464
Accountability of AI Under the Law: The Role of Explanation
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S. J., O'Brien, D., Scott, K., Shieber, S., Waldo, J., Weinberger, D., et al. (2017) · doi:10.2139/ssrn.3064761 · Cited by 457 · cited 2×
“Note that in terms of information disclosure, transparency contributes the basic mechanism used to facilitate the accountability of an AI system [94, 100].”
BackgroundAdversarial Attacks and Defences: A Survey
Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., & Mukhopadhyay, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.00069 · Cited by 446 · cited 4×
“Comprehensive surveys can be found in the literature such as [12, 19, 45, 69, 213, 304, 373].”
BackgroundInfluentialThe Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 8×
“Despite the simplicity of the abstract criteria outlined in the previous section, fairness can manifest in many different forms following these principles (see Reference [66, 356] for comprehensive surveys, and Reference [228] for AI ethics’ checklists).”
BackgroundMethodInfluentialFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
Hardy, S., Henecka, W., Ivey-Law, H., Nock, R., Patrini, G., Smith, G., & Thorne, B. (2017) · arXiv · doi:10.48550/arxiv.1711.10677 · Cited by 431 · cited 1×
“Reference [148] used homomorphic encryption for vertical logistic regression-based model learning.”
MethodA Systematic Review of Re-Identification Attacks on Health Data
El Emam, K., Jonker, E., Arbuckle, L., & Malin, B. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028071 · Cited by 427
A brief survey on anonymization techniques for privacy preserving publishing of social network data
Zhou, B., Pei, J., & Luk, W. (2008) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1540276.1540279 · Cited by 420 · cited 1×
“For example, private information in the form of graph data for social networks can be potentially contained in properties of the vertices of the graph, its link relationships, weights, or other graph metrics [390].”
BackgroundSelf-citationThe ethics of information transparency
Turilli, M., & Floridi, L. (2009) · Ethics and Information Technology · doi:10.1007/s10676-009-9187-9 · Cited by 419 · cited 1×
“1We follow Reference [333] to exclude transparency as an ethical requirement.”
BackgroundDistilling a neural network into a soft decision tree
Frosst, N., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1711.09784 · Cited by 407 · cited 1×
“With the rise of deep learning in the past decade, explainer approximation on the DNN has advanced as the knowledge distillation problem on explainers such as trees [125, 384].”
BackgroundFairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundModel complexity of deep learning: a survey
Hu, X., Chu, L., Pei, J., Liu, W., & Bian, J. (2021) · Knowledge and Information Systems · doi:10.1007/s10115-021-01605-0 · Cited by 402 · cited 1×
“Recent research on the complexity of ML models [162] and their cognitive functional complexity [347] may inspire future research on a unified quantitative evaluation metric.”
BackgroundSelf-citationTowards Robust Interpretability with Self-Explaining Neural Networks
Alvarez-Melis, D., & Jaakkola, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07538 · Cited by 397 · cited 20×
“For example, previous research has embedded linear models and prototype selection to the DNN [16, 20, 73].”
MethodInfluentialSTGAN: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing
Liu, M., Ding, Y., Xia, M., Liu, X., Ding, E., Zuo, W., & Wen, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00379 · Cited by 394 · cited 1×
“Similar ideas have been used in generative networks to gain control over attributes, such as the gender, age, and ethnicity, in a face generator [221].”
BackgroundEstablishing the rules for building trustworthy AI
Floridi, L. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0055-y · Cited by 394 · cited 1×
“• We consider ethical requirements with broad concerns in recent literature [9, 57, 121, 145, 178, 199, 218, 301, 334], including fairness, privacy, and accountability.”
BackgroundCalibrating Noise to Sensitivity in Private Data Analysis
Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2017) · Journal of Privacy and Confidentiality · doi:10.29012/jpc.v7i3.405 · Cited by 390 · cited 1×
“It measures the extent to which a (randomized) statistical function on the dataset reflects whether an element has been removed [108].”
BackgroundGreene, D., Hoffmann, A. L., & Stark, L. (2019) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2019.258 · Cited by 386
An Overview of Concept Drift Applications
Žliobaitė, I., Pechenizkiy, M., & Gama, J. (2016) · Studies in Big Data · doi:10.1007/978-3-319-26989-4_4 · Cited by 379 · cited 3×
“Deployment Anomaly Monitoring [70, 257, 394] Human-AI Interaction [72, 179, 301, 326] Fail-Safe Mechanism [126, 160, 230, 252] Hardware Security [264, 287, 364]”
BackgroundMethodXu, H., & Mannor, S. (2012) · Machine Learning · doi:10.1007/s10994-011-5268-1 · Cited by 377 · cited 1×
“As demonstrated in Reference [362], an algorithm that is robust against small perturbations has better generalization.”
Ai for social good: unlocking the opportunity for positive impact
Tomašev, N., Cornebise, J., Hutter, F., Mohamed, S., Picciariello, A., Connelly, B., Belgrave, D. C. M., Ezer, D., Haert, F. C. V. D., Mugisha, F., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-15871-z · Cited by 357
Racial faces in the wild: Reducing racial bias by information maximization adaptation network
Wang, M., Deng, W., Hu, J., Tao, X., & Huang, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00078 · Cited by 353
Cost-sensitive learning methods for imbalanced data
Thai-Nghe, N., Gantner, Z., & Schmidt-Thieme, L. (2010) · The 2010 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2010.5596486 · Cited by 342 · cited 1×
“, for the screening of highly contagious and severe diseases, false negatives can be costlier than false positives; see cost-sensitive learning [321]).”
BackgroundSu, D., Zhang, H., Chen, H., Yi, J., Chen, P.-Y., & Gao, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01258-8_39 · Cited by 342 · cited 2×
“From the attacker’s perspective, the rate of success of an attack also intuitively measures the robustness of the system [312].”
BackgroundSelf-citationThe Ugly Truth About Ourselves and Our Robot Creations: The Problem of Bias and Social Inequity
Howard, A., & Borenstein, J. (2018) · Science and Engineering Ethics · doi:10.1007/s11948-017-9975-2 · Cited by 337 · cited 1×
“, underprivileged groups might experience systematic disadvantage in hiring decisions [49], or be disproportionally impacted in criminal risk profiling [104, 146, 161]).”
BackgroundPresentation Attack Detection Methods for Face Recognition Systems: A Comprehensive Survey
Ramachandra, R., & Busch, C. (2018) · ACM Computing Surveys · doi:10.1145/3038924 · Cited by 332
Interpreting CNNs via Decision Trees
Zhang, Q., Yang, Y., Ma, H., & Wu, Y. N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00642 · Cited by 330 · cited 1×
“With the rise of deep learning in the past decade, explainer approximation on the DNN has advanced as the knowledge distillation problem on explainers such as trees [125, 384].”
BackgroundRakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
CrypTFlow2: Practical 2-Party Secure Inference
Rathee, D., Rathee, M., Kumar, N., Chandran, N., Gupta, D., Rastogi, A., & Sharma, R. (2020) · Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3372297.3417274 · Cited by 326 · cited 2×
“Several tools have been open sourced, such as MP2ML [48], CryptoSPN [330], CrypTFlow [200, 276], and CrypTen [188].”
MethodPrivacy-preserving anonymization of set-valued data
Terrovitis, M., Mamoulis, N., & Kalnis, P. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1453856.1453874 · Cited by 326 · cited 1×
“Specific DA methods have also been designed for relational data [262], set-valued data [151, 320], and image data [97, 239].”
BackgroundInterpretML: A Unified Framework for Machine Learning Interpretability
Nori, H., Jenkins, S., Koch, P., & Caruana, R. (2019) · arXiv · doi:10.48550/arxiv.1909.09223 · Cited by 322
A Pragmatic Introduction to Secure Multi-Party Computation
Evans, D., Kolesnikov, V., & Rosulek, M. (2018) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000019 · Cited by 321 · cited 8×
“A typical SMPC protocol satisfies properties of privacy, correctness, independence of inputs, guaranteed output delivery, and fairness [114, 387].”
BackgroundInfluentialLindell, Y. (2021) · Communications of the ACM · doi:10.1145/3387108 · Cited by 320
Efficient Neural Network Robustness Certification with General Activation Functions
Zhang, H., Weng, T.-W., Chen, P., Hsieh, C., & Daniel, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00866 · Cited by 316 · cited 3×
“For example, CNN-Cert [51], CROWN [379], Fast-lin, and Fast-lip [352] aim to minimize an upper bound of the worst-case loss under given input perturbations.”
BackgroundFedML: A Research Library and Benchmark for Federated Machine Learning
He, C., Li, S., So, J., Zeng, X., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.13518 · Cited by 314 · cited 1×
“To expedite FL-related research and development, many open source libraries have been released, such as FATE, FedML [149], and FedlearnAlgo [217].”
MethodThe ai-based cyber threat landscape: A survey
Kaloudi, N., & Li, J. (2021) · ACM Computing Surveys · doi:10.1145/3372823 · Cited by 310
Generalizing to Unseen Domains: A Survey on Domain Generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., & Qin, T. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/628 · Cited by 308
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 4×
“• We consider ethical requirements with broad concerns in recent literature [9, 57, 121, 145, 178, 199, 218, 301, 334], including fairness, privacy, and accountability.”
BackgroundInfluential
Research works printed but not linked: 60
- Dalal Al-Azizy, David Millard, Iraklis Symeonidis, Kieron O’Hara, and Nigel Shadbolt. 2015. A literature survey and classifications on data deanonymisation. In Proceedings of the International Conference on Risks and Security of Internet and Systems. Springer, 36–51.
- Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR’15) .
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency. PMLR, 77–91.
- 10.5555/3294996.3295155 · doi:10.5555/3294996.3295155
- Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium (USENIX Security’21). 2633–2650.
- Yu Cheng Mo Yu Xiaoxiao Guo and Bowen Zhou. 2019. Few-shot learning with meta metric learners. In NIPS 2017 Workshop on Meta-Learning .
- Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. 2017. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning. PMLR, 854–863.
- Bo Cowgill and Catherine Tucker. 2017. Algorithmic bias: A counterfactual perspective. NSF Trustworthy Algorithms.
- Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. 2021. RobustBench: A standardized adversarial robustness benchmark. In Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
- Luiz Marcio Cysneiros and Vera Maria Benjamim Werneck. 2009. An initial analysis on how software transparency and trust influence each other. In Proceedings of the Workshop on Requirements Engineering (WER’09). Citeseer.
- Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Conference on Robot Learning. PMLR, 1–16.
- 10.5555/1791834.1791836 · doi:10.5555/1791834.1791836
- Exforsys. 2011. What Is Monkey Testing. Retrieved July 9, 2021 from http://www.exforsys.com/tutorials/testing-types/ monkey-testing.html.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. PMLR, 1126–1135.
- Shixiang Gu and Luca Rigazio. 2015. Towards deep neural network architectures robust to adversarial examples. In 3rd International Conference on Learning Representations (ICLR’15) San Diego CA USA May 7-9 2015 Workshop Track Proceedings .
- IBM. Trusting AI. Retrieved February 20, 2021 from https://www.research.ibm.com/artificial-intellig ence/trusted-ai/.
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. PMLR, 448–456.
- Shouling Ji, Weiqing Li, Prateek Mittal, Xin Hu, and Raheem Beyah. 2015. \(\lbrace\) SecGraph \(\rbrace\) : A uniform and open-source evaluation system for graph data anonymization and de-anonymization. In Proceedings of the 24th USENIX Security Symposium (USENIX Security’15). 303–318.
- Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, Vol. 2.
- Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International Conference on Machine Learning. PMLR, 1885–1894.
- Klas Leino, Zifan Wang, and Matt Fredrikson. 2021. Globally-robust neural networks. In International Conference on Machine Learning. PMLR, 6212–6222.
- Ke Li and Jitendra Malik. 2017. Learning to optimize. In International Conference on Learning Representations .
- Yiming Li Yong Jiang Zhifeng Li and Shu-Tao Xia. 2017. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems .
- Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses. Springer, 273–294.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
- Pratyush Maini, Eric Wong, and Zico Kolter. 2020. Adversarial robustness against the union of multiple perturbation models. In International Conference on Machine Learning. PMLR, 6640–6650.
- Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning. PMLR, 1614–1623.
- Jan Hendrik Metzen Tim Genewein Volker Fischer and Bastian Bischoff. 2017. On detecting adversarial perturbations. In 5th International Conference on Learning Representations (ICLR’17) Toulon France April 24-26 2017 Conference Track Proceedings .
- Silvio Micali, Oded Goldreich, and Avi Wigderson. 1987. How to play any mental game. In Proceedings of the 19th ACM Symposium on Theory of Computing (STOC’87). ACM, 218–229.
- Christoph Molnar. 2020. Interpretable Machine Learning. Retrieved from Lulu.com.
- OpenAI. 2018. OpenAI Charter. Retrieved February 20, 2021 from https://openai.com/charter/.
- Andrea Paudice, Luis Muñoz-González, and Emil C. Lupu. 2018. Label sanitization against label flipping poisoning attacks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 5–15.
- Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified defenses against adversarial examples. In Proceedings of the International Conference on Learning Representations.
- Aditi Raghunathan Sang Michael Xie Fanny Yang John C. Duchi and Percy Liang. 2019. Adversarial training can hurt generalization. In ICML 2019 Workshop Deep Phenomena .
- Yuji Roh, Kangwook Lee, Steven Whang, and Changho Suh. 2020. FR-train: A mutual information-based approach to fair and robust training. In International Conference on Machine Learning. PMLR, 8147–8157.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of the Data and Discrimination: Converting Critical Concerns Into Productive Inquiry 22 (2014), 4349–4357.
- Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning. PMLR, 1842–1850.
- Sebastian Schelter, Joos-Hendrik Boese, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert. 2017. Automatically tracking metadata and provenance of machine learning experiments. In Machine Learning Systems Workshop at NIPS. 27–29.
- Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning. PMLR, 3145–3153.
- Simone Stumpf, Adrian Bussone, and Dympna O’sullivan. 2016. Explanations considered harmful? user interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).
- Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre-training of generic visual-linguistic representations. In Proceedings of the International Conference on Learning Representations.
- Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning. PMLR, 3319–3328.
- Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK. 2020. Auditing Machine Learning Algorithms. Retrieved from https://www.auditingalgorithms.net/index.html.
- Liang Tong, Bo Li, Chen Hajaj, Chaowei Xiao, Ning Zhang, and Yevgeniy Vorobeychik. 2019. Improving robustness of \(\lbrace\) ML \(\rbrace\) classifiers against realizable evasion attacks using conserved features. In Proceedings of the 28th USENIX Security Symposium (USENIX Security’19). 285–302.
- Cecília Torres, Walter Franklin, and Laura Martins. 2018. Accessibility in chatbots: The state of the art in favor of users with visual impairment. In International Conference on Applied Human Factors and Ergonomics. Springer, 623–635.
- Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2018. Ensemble adversarial training: Attacks and defenses. In Proceedings of the International Conference on Learning Representations.
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction \(\lbrace\) APIs \(\rbrace\) . In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Dimitris Tsipras Shibani Santurkar Logan Engstrom Alexander Turner and Aleksander Ma̧dry. 2019. Robustness may be at odds with accuracy. In 7th International Conference on Learning Representations (ICLR’19) New Orleans LA USA May 6-9 2019 .
- UNI Global Union. 2017. Top 10 principles for ethical artificial intelligence. In The Future World of Work.
- Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019. On the convergence and robustness of adversarial training. In Proceedings of the International Conference on Machine Learning (ICML’19), Vol. 1. 2.
- Yingxu Wang and Jingqiu Shao. 2003. Measurement of the cognitive functional complexity of software. In Proceedings of the 2nd IEEE International Conference on Cognitive Informatics. IEEE, 67–74.
- Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. 2018. Evaluating the robustness of neural networks: An extreme value theory approach. In Proceedings of the International Conference on Learning Representations.
- Tong Wu Liang Tong and Yevgeniy Vorobeychik. 2020. Defending against physically realizable attacks on image classification. In 8th International Conference on Learning Representations (ICLR’20) Addis Ababa Ethiopia April 26-30 2020 .
- Weiyi Xia, Yongtai Liu, Zhiyu Wan, Yevgeniy Vorobeychik, Murat Kantacioglu, Steve Nyemba, Ellen Wright Clayton, and Bradley A. Malin. 2021. Enabling realistic health data re-identification risk assessment through adversarial modeling. J. Am. Med. Inf. Assoc. 28, 4 (2021), 744–752.
- Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021. To be robust or to be fair: Towards fairness in adversarial training. In International Conference on Machine Learning. PMLR, 11492–11501.
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning. PMLR, 2048–2057.
- 10.5555/1382436.1382751 · doi:10.5555/1382436.1382751
- Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei. 2021. Seco: Exploring sequence supervision for unsupervised representation learning. In AAAI, Vol. 2. 7.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. PMLR, 325–333.
Cited by
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“Researchers [29, 47, 81, 101, 107, 183] also proposed many other desiderata for interpretations, interpretation algorithms or interpretability, such as fairness, privacy, reliability, robustness, causality, trust, fidelity, faithfulness, transferability, informativeness, transparency, plausibility, satisfaction, accou…”
BackgroundSelf-citationEsmaeilzadeh, P. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102861 · Cited by 377
Joyce, D. W., Kormilitzin, A., Smith, K. A., & Cipriani, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00751-9 · Cited by 238
Post-hoc vs ante-hoc explanations: xAI design guidelines for data scientists
Retzlaff, C. O., Angerschmid, A., Saranti, A., Schneeberger, D., Röttger, R., Müller, H., & Holzinger, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101243 · Cited by 194
“In any case, trustworthy AI encompasses elements of security (Li et al., 2023), safety, privacy, fairness (Angerschmid et al., 2022), accountability, auditability, and robustness (Holzinger, Dehmer et al., 2022).”
Metadata
- Authors
- Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B.
- Affiliations
- Tsinghua University (CN)
- Amazon (United States) (US)
- Walmart Inc., Mountain View, CA, USA
- Jingdong (China) (CN)
- JD Technology, Mountain View, CA, USA
- Frontis.AI, Beijing, China
- Venue
- ACM Computing Surveys
- Published
- 2023-01-16
- Volume
- 55
- Issue
- 9
- Pages
- 1-46
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3202183072
- Cited by
- 612
- References
- 384
- Citations by year
- 2026: 164 · 2025: 207 · 2024: 169 · 2023: 65
- FWCI
- 129.72
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- trustworthy AI, robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, accountability, AI lifecycle, AI governance, continuous monitoring, algorithmic bias, imperceptible attacks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.