Trustworthy AI: From Principles to Practices
Open accessThe DOI matches its Crossref record; 398 of the 399 references it cites were found in a registry or an index; 74 of the 251 sources its claims cite were checked for support.
Abstract
The rapid development of Artificial Intelligence (AI) technology has enabled the deployment of various systems based on it. However, many current AI systems are found vulnerable to imperceptible attacks, biased against underrepresented groups, lacking in user privacy protection. These shortcomings degrade user experience and erode people’s trust in all AI systems. In this review, we provide AI practitioners with a comprehensive guide for building trustworthy AI systems. We first introduce the theoretical framework of important aspects of AI trustworthiness, including robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, and accountability. To unify currently available but fragmented approaches toward trustworthy AI, we organize them in a systematic approach that considers the entire lifecycle of AI systems, ranging from data acquisition to model development, to system development and deployment, finally to continuous monitoring and governance.
Overview
CoChat Summary
TL;DR
This review provides a comprehensive guide to trustworthy AI, organizing robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, and accountability across the AI system lifecycle. It proposes TrustAIOps as a continuous workflow and identifies open challenges including trade-offs, immature evaluation, and risks from large pre-trained models.
Key findings
- The paper defines eight trustworthiness aspects—robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, and accountability—and argues that predictive accuracy alone is insufficient; trustworthiness must be assessed systematically across data preparation, algorithm design, development, deployment, and management.
- It proposes TrustAIOps, extending MLOps/DevOps with aligned trustworthiness principles, interdisciplinary collaboration, extensive artifact management, and continuous feedback loops across the five lifecycle stages.
- It documents frictions and trade-offs: transparency can leak private data or intellectual property and enable targeted hacking; adversarial robustness can reduce accuracy; robustness and fairness can negatively affect each other; and explanations can be fragile, inconsistent with human interpretation, or hard to judge as faithful.
- In privacy evaluation, it cites work showing that 15 demographic attributes were sufficient to render 99% of participants unique.
Methods
- Design: narrative review/survey, not an empirical study; it synthesizes technical, engineering, and governance literature on trustworthy AI.
- Data or sample: no dataset or sample is used; the paper draws on cited literature and industrial case studies of face recognition, autonomous driving, and NLP in the appendix.
- Main analysis: conceptual organization of approaches by trustworthiness aspect and by lifecycle stage, culminating in a proposed systematic framework and TrustAIOps workflow.
Limitations
- The paper states that explainability, transparency, and accountability are still seldom evaluated quantitatively; approaches to robustness and explainability remain immature; trade-offs and frictions exist between trustworthiness aspects; large pre-trained models are costly and difficult to reproduce and may carry bias or privacy leakage; and end-user awareness is limited.
- Evident from the text: as a survey, it presents no original experiments or quantitative validation of the proposed TrustAIOps workflow, and its recommendations are conceptual rather than empirically tested in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Valid · 28 Not valid · 96 Not checked · 119 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It is widely recognized that AI models might be vulnerable to attacks by adversaries with malicious intentions.”
What the sources say
- Partly supportedcites [68] [68] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1-11, 2019. - sourceExperimental — accuracy not yet measured
The Face Manipulation Methods section says forgery detectors have robustness issues and that compression and resizing can erase traces they use to detect manipulation. That supports vulnerability in these particular models, but the paper does not establish the broader claim that AI models are vulnerable to intentional attacks by malicious adversaries. (LLM, conf=0.78)
- Not matchedcites [304]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [11] [11] Sarah Bankins and Paul Formosa. Ethical ai at work: The social contract for artificial intelligence and its implications for the workplace psychological contract. Redefining the Psychological Contract in the Digital Era: Issues for Research and Practice , pages 55-72, 2021. - source
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“many current AI systems are found vulnerable to imperceptible attacks, biased against underrepresented groups, lacking in user privacy protection.”
What the sources say
- Could not verifycites [68] [68] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1-11, 2019. - sourceExperimental — accuracy not yet measured
The Face Manipulation Methods section discusses robustness of facial-forgery detectors to compression and resizing, and §4.2.2 reports lower detection performance on compressed videos. Searches for imperceptible adversarial attacks and perturbations found no passage saying that many current AI systems are vulnerable to such attacks. (LLM, conf=0.74)
- Not matchedcites [304]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [11] [11] Sarah Bankins and Paul Formosa. Ethical ai at work: The social contract for artificial intelligence and its implications for the workplace psychological contract. Redefining the Psychological Contract in the Digital Era: Issues for Research and Practice , pages 55-72, 2021. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, liveness detection is still vulnerable against new types of attacks by fake videos generated by AI techniques like DeepFake [54].”
What the sources say
- Partly supportedcites [54] [54] Thanh Thi Nguyen, Quoc Viet Hung Nguyen, Cuong M Nguyen, Dung Nguyen, Duc Thanh Nguyen, and Saeid Nahavandi. Deep learning for deepfakes creation and detection: A survey. arXiv preprint arXiv:1909.11573 , 2019. - sourceExperimental — accuracy not yet measured
Section 3, “Deepfake Detection,” reports that tested face-recognition systems failed to detect deepfakes effectively and that other methods had high error rates on a newly produced deepfake-video dataset. This supports vulnerability to AI-generated fake videos, but the passage does not specifically assess liveness detection. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Research has shown that face recognition systems can be easily spoofed by impersonation attackers wearing a small printed adversarial patch [90, 30, 43, 73].”
What the sources say
- Supportedcites [90] [90] Zihao Xiao, Xianfeng Gao, Chilin Fu, Yinpeng Dong, Wei Gao, Xiaolu Zhang, Jun Zhou, and Jun Zhu. Improving transferability of adversarial patches on face recognition with generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11845-11854, 2021. - source
The Introduction describes adversarial eyeglass frames used to impersonate another identity and says physical patches can fool face-recognition models. Section 4.5 reports an attacker wearing printed adversarial eyeglass frames and finding that the patches retained high transferability after printing and photographing. The physical-world experiment involved one volunteer and three target identities. (LLM, conf=0.86)
- Partly supportedcites [30] [30] Ying Guo, Xingxing Wei, Guoqiu Wang, and Bo Zhang. Meaningful adversarial stickers for face recognition in physical world. arXiv:2104.06728 , 2021. - sourceExperimental — accuracy not yet measured
Section 5.3 reports successful impersonation attacks using facial stickers and shows examples in the physical world. But Section 3.5 explicitly says the method does not involve printing: subjects wear existing stickers placed according to calculated attack parameters. Thus the paper supports the impersonation point, but not the description of a printed adversarial patch. (LLM, conf=0.89)
- Partly supportedcites [43] [43] Stepan Komkov and Aleksandr Petiushko. Advhat: Real-world adversarial attack on arcface face id system. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 819-826. IEEE, 2021. - sourceExperimental — accuracy not yet measured
The Abstract and Introduction (p. 1) describe an easily reproducible attack using a printed sticker on a hat to confuse a Face ID model. But the paper’s results concern making the wearer less recognizable, not impersonating another person: Section 2.1 (p. 2) distinguishes impersonation from dodging, and the Testing Protocol (p. 5) describes its attacks as dodging attacks. (LLM, conf=0.90)
- Partly supportedcites [73] [73] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security (TOPS) , 22(3):1-30, 2019. - sourceExperimental — accuracy not yet measured
The Introduction and Experiment Setup (pages unavailable in our copy) describe impersonation attacks against face-recognition DNNs using printed adversarial patterns affixed to eyeglasses worn by attackers. The source does not establish that face-recognition systems generally are easily spoofed by a small patch: it tests particular DNNs and eyeglass-based attacks under controlled conditions, and notes that some physical impersonation attempts failed. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Some pre-trained models are reported to output training text data containing private user information such as addresses [63].”
What the sources say
- Could not verifycites [63] [63] Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 conference on fairness, accountability, and transparency , pages 33-44, 2020. - sourceExperimental — accuracy not yet measured
Section 4.4.2 discusses documentation for trained models, and Section 4.6 discusses privacy practices for face data; neither reports a model outputting private information from its training text. Searches for pre-trained models, training-text leakage, and addresses found no passage addressing the claim. The retrieved passages provide section labels but no page numbers. (LLM, conf=0.76)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
72 more claims · 5 with their cited sources checked
Create a free account to see all 282 claims and to verify the 38 references still open.
- Probably not a substantial claim. 191
- It may state two separate claims as one. 43
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An effective automated audit system for authorship and provenance compliance functions as a continuous pipeline rather than a retrospective review tool [28].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reliable provenance begins at the moment content is produced, not after distribution [28].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This high-level understanding will help optimize trust, as operators will be aware of the capabilities and limitations of the AI systems they work with and know the appropriate settings for use104.”
From Adopting and expanding ethical principles for generative artificial intelligence from military to healthcare (2023) · cites this paper as [104]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This may be deliberate, to protect brand perception or avoid licensing obligations, or unintentional, especially in creative interfaces that mask algorithmic influence behind seemingly manual controls [28].”
From Designing automated audit mechanisms to evaluate compliance of generative AI platforms with federal authorship and ownership disclosure requirements. (2023) · cites this paper as [28]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 Trustworthy AI: A Systematic Approach · p. 13
“the lifecycle of the development of a typical AI product can be partitioned into data preparation, algorithm design, development-deployment, and management [25].”
§ 2.6 Privacy protection · p. 12
“the five main stages of the lifecycle of an AI system, i.e., data preparation, algorithm design, development, deployment, and management.”
Citations
Cites
Items 201–400 of 400
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 4×
“• We consider ethical requirements with broad concerns in recent literature [9, 57, 121, 145, 178, 199, 218, 301, 334], including fairness, privacy, and accountability.”
BackgroundInfluentialHiding the presence of individuals from shared databases
Nergiz, M. E., Atzori, M., & Clifton, C. (2007) · Proceedings of the 2007 ACM SIGMOD international conference on Management of data · doi:10.1145/1247480.1247554 · Cited by 304 · cited 1×
“Different principles of quantitative data anonymization have been developed, such as k-anonymity [288], (c,k )-safety [236], and δ -presence [253].”
BackgroundA survey on provenance: What for? What form? What from?
Herschel, M., Diestelkämper, R., & Ben Lahmar, H. (2017) · The VLDB Journal · doi:10.1007/s00778-017-0486-1 · Cited by 300 · cited 5×
“By tracking the data lineage at the highest resolution, data provenance can enhance the transparency, reproducibility, and accountability of an AI system [154, 172].”
BackgroundInfluentialThe relationship between trust in AI and trustworthy machine learning technologies
Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., Zelaya, C. G., & van Moorsel, A. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372834 · Cited by 296
Advhat: Real-world adversarial attack on arcface face id system
Komkov, S., & Petiushko, A. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412236 · Cited by 294
State of the Art: Reproducibility in Artificial Intelligence
Gundersen, O. E., & Kjensmo, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11503 · Cited by 293
Towards explainable deep neural networks (xDNN)
Angelov, P., & Soares, E. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.07.010 · Cited by 292 · cited 19×
“For example, previous research has embedded linear models and prototype selection to the DNN [16, 20, 73].”
MethodInfluentialAnonymization Techniques for Privacy Preserving Data Publishing: A Comprehensive Survey
Majeed, A., & Lee, S. (2021) · IEEE Access · doi:10.1109/access.2020.3045700 · Cited by 287 · cited 2×
“Data Preprocessing Anomaly Detection [70, 81, 257, 316] Data Anonymozation [14, 37, 232, 232] Differential Privacy [107, 110, 177]”
Explaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286
$\textsf{LoPub}$ : High-Dimensional Crowdsourced Data Publication With Local Differential Privacy
Ren, X., Yu, C.-M., Yu, W., Yang, S., Yang, X., McCann, J. A., & Yu, P. S. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2812146 · Cited by 282
Sakaridis, C., Dai, D., & Van Gool, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00747 · Cited by 281
Auditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280 · cited 1×
“Common approaches include resampling [6], which involves selecting a subset of the data, reweighting [60], which involves assigning different importance values to data examples, and adversarial learning [229], which can be achieved through resampling or reweighting with the help of a trained model to find the offendin…”
BackgroundArtificial Intelligence Crime: An Interdisciplinary Analysis of Foreseeable Threats and Solutions
King, T. C., Aggarwal, N., Taddeo, M., & Floridi, L. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-018-00081-0 · Cited by 271
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268 · cited 4×
“For operators like bank executives, explainability helps understand the AI credit system to prevent potential defects in it [25, 184].”
BackgroundInfluentialTowards Stable and Efficient Training of Verifiably Robust Neural NetworksPreprint
Zhang, H., Chen, H., Xiao, C., Gowal, S., Stanforth, R., Li, B., Boning, D. S., & Hsieh, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06316 · Cited by 268 · cited 1×
“To address the issue that the exact computation of such bounds is intractable for large networks, various relaxations or approximations, such as References [352, 378], have been proposed as alternatives to regularization.”
BackgroundSelf-citationBenchmarking and survey of explanation methods for black box models
Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., & Rinzivillo, S. (2023) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-023-00933-9 · Cited by 264 · cited 9×
“, (1) the explanation is fragile to perturbations [130], (2) the explanation is not always consistent with human interpretation [47], and (3) it is difficult to judge if the explanation is correct or faithful [250].”
BackgroundMethodInfluentialWorst-Case Background Knowledge for Privacy-Preserving Data Publishing
Martin, D. J., Kifer, D., Machanavajjhala, A., Gehrke, J., & Halpern, J. Y. (2007) · 2007 IEEE 23rd International Conference on Data Engineering · doi:10.1109/icde.2007.367858 · Cited by 263 · cited 1×
“Different principles of quantitative data anonymization have been developed, such as k-anonymity [288], (c,k )-safety [236], and δ -presence [253].”
BackgroundFormal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
Hein, M., & Andriushchenko, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08475 · Cited by 263 · cited 1×
“Reference [152] instead derives a lower bound on the input manipulation required to change the decision of the classifier, and uses it as a regularization term for robust training.”
BackgroundDemographic Bias in Biometrics: A Survey on an Emerging Challenge
Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., & Busch, C. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.2992344 · Cited by 263
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J. M., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02012 · Cited by 263
I'd blush if I could: closing gender divides in digital skills through education
UNESCO, & Coalition, E. S. (2019) · doi:10.54675/rapc9356 · Cited by 262
Trustworthy AI: A Computational Perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A. K., & Tang, J. (2021) · arXiv · Cited by 261
Trustworthy ai: A computational perspective
Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., & Tang, J. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3546872 · Cited by 261
CrypTen: Secure Multi-Party Computation Meets Machine Learning
Knott, B., Venkataraman, S., Hannun, A., Sengupta, S., Ibrahim, M., & van der Maaten, L. (2021) · arXiv · doi:10.48550/arxiv.2109.00984 · Cited by 257 · cited 2×
“Several tools have been open sourced, such as MP2ML [48], CryptoSPN [330], CrypTFlow [200, 276], and CrypTen [188].”
MethodAssuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · cited 1×
“As shown by the solid-lined boxes in Figure 2, the lifecycle of the development of a typical AI product can be partitioned into data preparation, algorithm design, development–deployment, and management [26].”
The challenge of crafting intelligible intelligence
Weld, D. S., & Bansal, G. (2019) · Communications of the ACM · doi:10.1145/3282486 · Cited by 253 · cited 2×
“Various approaches of interaction have been studied to enhance the explainability of AI, including the visualization of ML models [72] and interactive parameter-tuning [351].”
BackgroundMethodCrypTFlow: Secure TensorFlow Inference
Kumar, N., Rathee, M., Chandran, N., Gupta, D., Rastogi, A., & Sharma, R. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00092 · Cited by 245 · cited 2×
“Several tools have been open sourced, such as MP2ML [48], CryptoSPN [330], CrypTFlow [200, 276], and CrypTen [188].”
MethodFederated Learning for Open Banking
Long, G., Tan, Y., Jiang, J., & Zhang, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-63076-8_17 · Cited by 244 · cited 1×
“It has been applied to numerous domains, such as healthcare [282, 299] and finance [223].”
MethodZhang, X., Yang, L. T., Liu, C., & Chen, J. (2014) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.48 · Cited by 241
McKeen, F., Alexandrovich, I., Anati, I., Caspi, D., Johnson, S., Leslie-Hurd, R., & Rozas, C. (2016) · Proceedings of the Hardware and Architectural Support for Security and Privacy 2016 · doi:10.1145/2948618.2954331 · Cited by 238 · cited 1×
“The Intel Software Guard Extensions implement the TEE by hardware-based memory encryption [240].”
MethodData poisoning attacks on factorization-based collaborative filtering
Li, B., Wang, Y., Singh, A., & Vorobeychik, Y. (2016) · arXiv · doi:10.48550/arxiv.1608.08182 · Cited by 235
Self-citationPineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232 · cited 4×
“Reproducibility checklists have been recently widely adopted in ML conferences to assess the reproducibility of submissions [263].”
BackgroundMethodInfluentialClassification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287586 · Cited by 231 · cited 1×
“For instance, quantitative fairness metrics can be used as regularization [7] or constraints for the optimization of the model parameters [67].”
Explainability of deep vision-based autonomous driving systems: Review and challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2021) · arXiv · doi:10.48550/arxiv.2101.05307 · Cited by 227
Casting out Demons: Sanitizing Training Data for Anomaly Sensors
Cretu, G. F., Stavrou, A., Locasto, M. E., Stolfo, S. J., & Keromytis, A. D. (2008) · 2008 IEEE Symposium on Security and Privacy (sp 2008) · doi:10.1109/sp.2008.11 · Cited by 226 · cited 1×
“The mitigation of adversarial attacks is also referred to as data sanitization [71, 87, 258].”
BackgroundTell me more?: the effects of mental model soundness on personalizing an intelligent agent
Kulesza, T., Stumpf, S., Burnett, M., & Kwan, I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207678 · Cited by 225 · cited 1×
“This evaluation has been used for the development of, for instance, recommendation systems [198] and data analysis [132].”
MethodDiverse Few-Shot Text Classification with Multiple Metrics
Yu, M., Guo, X., Yi, J., Chang, S., Potdar, S., Cheng, Y., Tesauro, G., Wang, H., & Zhou, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1109 · Cited by 225 · cited 1×
“of few-shots learning is to generalize model to new tasks using only a few examples [78, 348, 371].”
BackgroundSelf-citationExplainability in human–agent systems
Rosenfeld, A., & Richardson, A. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09408-y · Cited by 224 · cited 1×
“Although the definition of the explainability of an AI model is still an open question, research has sought to address this problem from the perspectives of AI [141, 285] and psychology [144, 245].”
BackgroundCIAGAN: Conditional Identity Anonymization Generative Adversarial Networks
Maximov, M., Elezi, I., & Leal-Taixe, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00549 · Cited by 222 · cited 1×
“Specific DA methods have also been designed for relational data [262], set-valued data [151, 320], and image data [97, 239].”
BackgroundPublishing set-valued data via differential privacy
Chen, R., Mohammed, N., Fung, B. C. M., Desai, B. C., & Xiong, L. (2011) · Proceedings of the VLDB Endowment · doi:10.14778/3402707.3402744 · Cited by 220 · cited 1×
“DP has been explored in various data publishing tasks, such as log data [159, 385], set-valued data [76], correlated network data [75], and crowdsourced data [278, 344].”
BackgroundCorrelated network data publication via differential privacy
Chen, R., Fung, B. C. M., Yu, P. S., & Desai, B. C. (2014) · The VLDB Journal · doi:10.1007/s00778-013-0344-8 · Cited by 215 · cited 1×
“DP has been explored in various data publishing tasks, such as log data [159, 385], set-valued data [76], correlated network data [75], and crowdsourced data [278, 344].”
BackgroundToward verified artificial intelligence
Seshia, S. A., Sadigh, D., & Sastry, S. S. (2022) · Communications of the ACM · doi:10.1145/3503914 · Cited by 215
Anonymization of set-valued data via top-down, local generalization
He, Y., & Naughton, J. F. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687627.1687733 · Cited by 211 · cited 1×
“Specific DA methods have also been designed for relational data [262], set-valued data [151, 320], and image data [97, 239].”
BackgroundTechnology readiness levels for machine learning systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., Ganju, S., Newman, D., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 209 · cited 1×
“Software engineering in the stages of development and deployment has recently aroused wide concern as an essential condition for reliable AI systems [17, 203].”
BackgroundOpportunities and Challenges in Deep Learning Adversarial Robustness: A SurveyPreprint
Silva, S. H., & Najafirad, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00753 · Cited by 208
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Rabanser, S., Günnemann, S., & Lipton, Z. C. (2018) · arXiv · doi:10.48550/arxiv.1810.11953 · Cited by 204 · cited 1×
“Data drift monitoring [268] provides important means to maintain the generalization of an AI system under concept change [394] caused by dynamic environment such as market change [289].”
BackgroundVerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems
Dreossi, T., Fremont, D. J., Ghosh, S., Kim, E., Ravanbakhsh, H., Vazquez-Chanlatte, M., & Seshia, S. A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-25540-4_25 · Cited by 200 · cited 3×
“Control systems for robots can be compared and benchmarked in simulated environments such as Gazebo [214], MuJoCo [365], VerifAI [119], etc. Similarly, simulators for self-driving cars have also been widely used, including TORCS [404], CarSim [48], PRESCAN [364], etc.”
MethodA general framework for adversarial examples with objectives
Sharif, M., Bhagavatula, S., Bauer, L., & Reiter, M. K. (2019) · ACM Transactions on Privacy and Security · doi:10.1145/3317611 · Cited by 200
Generalization in Deep Learning
Kawaguchi, K., Bengio, Y., & Kaelbling, L. (2022) · Mathematical Aspects of Deep Learning · doi:10.1017/9781009025096.003 · Cited by 197
QUOTIENT: Two-Party Secure Neural Network Training and Prediction
Agrawal, N., Shahin Shamsabadi, A., Kusner, M. J., & Gascón, A. (2019) · Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3319535.3339819 · Cited by 196 · cited 1×
“To reduce the high costs of computation and communication of the SMPC, parameter quantization and function approximation were used together with cryptographic protocols in References [8, 32].”
MethodReal-time and Spatio-temporal Crowd-sourced Social Network Data Publishing with Differential Privacy
Wang, Q., Zhang, Y., Lu, X., Wang, Z., Qin, Z., & Ren, K. (2016) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2016.2599873 · Cited by 196 · cited 1×
“DP has been explored in various data publishing tasks, such as log data [159, 385], set-valued data [76], correlated network data [75], and crowdsourced data [278, 344].”
BackgroundVorobeychik, Y., & Kantarcıoğlu, M. (2018) · Synthesis lectures on artificial intelligence and machine learning · doi:10.2200/s00861ed1v01y201806aim039 · Cited by 194
Privacy-preserving distributed linear regression on high-dimensional data
Gascón, A., Schoppmann, P., Balle, B., Raykova, M., Doerner, J., Zahur, S., & Evans, D. (2017) · Proceedings on Privacy Enhancing Technologies · doi:10.1515/popets-2017-0053 · Cited by 193 · cited 1×
“, linear regression [128] and logistic regression [300], and generic model learning tasks [247].”
BackgroundAutonomous vehicles testing methods review
Huang, W., Wang, K., Lv, Y., & Zhu, F. (2016) · 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795548 · Cited by 191
Why is my classifier discriminatory?
Chen, I., Johansson, F. D., & Sontag, D. (2018) · arXiv · doi:10.48550/arxiv.1805.12002 · Cited by 185
Decision Tree Classification with Differential Privacy: A Survey
Fletcher, S., & Islam, M. Z. (2019) · ACM Computing Surveys · doi:10.1145/3337064 · Cited by 185
The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
Chatzimparmpas, A., Martins, R. M., Jusufi, I., Kucher, K., Rossi, F., & Kerren, A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14034 · Cited by 182
Differential Privacy and Machine Learning: a Survey and Review
Ji, Z., Lipton, Z. C., & Elkan, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.7584 · Cited by 181 · cited 2×
“Data Preprocessing Anomaly Detection [70, 81, 257, 316] Data Anonymozation [14, 37, 232, 232] Differential Privacy [107, 110, 177]”
Grgić-Hlača, N., Zafar, M. B., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11296 · Cited by 181 · cited 1×
“At each granularity, we might be concerned with distributive fairness or fairness of the outcome, or procedural fairness or fairness of the process (we refer the reader to Reference [137] for a more detailed discussion).”
BackgroundAdversarial Machine Learning in Image Classification: A Survey Toward the Defender’s Perspective
Machado, G. R., Silva, E., & Goldschmidt, R. R. (2023) · ACM Computing Surveys · doi:10.1145/3485133 · Cited by 180 · cited 1×
“The objective of defense can be either proactive or reactive [227].”
BackgroundLocal Rule-Based Explanations of Black Box Decision Systems
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., & Giannotti, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10820 · Cited by 179 · cited 2×
“, explainer, such as a linear model [225, 279] and rules [140, 280].”
BackgroundRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs
Kolouri, S., Saha, A., Pirsiavash, H., & Hoffmann, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00038 · Cited by 175 · cited 1×
“The detection of backdoor attacks can also be carried out by analyzing the prediction on the model on specifically designed benchmarking inputs [194].”
BackgroundReverse Engineering the Neural Networks for Rule Extraction in Classification Problems
Augasta, M. G., & Kathirvalavakumar, T. (2012) · Neural Processing Letters · doi:10.1007/s11063-011-9207-8 · Cited by 173 · cited 1×
“been proposed to approximate ML models, such as random forests [317, 392] and neural networks [28, 86, 393].”
MethodNight-Time Scene Parsing With a Large Real Dataset
Tan, X., Xu, K., Cao, Y., Zhang, Y., Ma, L., & Lau, R. W. H. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3122004 · Cited by 173 · cited 1×
“For example, in the field of autonomous driving, besides developing a perceptual system working in sunny scenes, academia and the industry are using numerous development and testing strategies to enhance the perceptual performance of the vehicles in nighttime/rainy scenes to guarantee the system’s reliability under a…”
BackgroundFairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171 · cited 3×
“Criteria for AI systems have been studied to spot issues of unfairness by investigating the correlation among sensitive attributes, system outcomes, and the true label when applicable to well-designed diagnostic datasets [327].”
BackgroundFairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171
Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?Preprint
Mäkinen, S., Skogström, H., Laaksonen, E., & Mikkonen, T. (2021) · arXiv (Cornell University) · Cited by 169
Vanschoren, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.03548 · Cited by 169
Human Language Understanding & Reasoning
Manning, C. D. (2022) · Daedalus · doi:10.1162/daed_a_01905 · Cited by 162 · cited 2×
“They not only show the potential to a more general form of AI [234], but also bring new challenges and opportunities to the establishment of trustworthy AI.”
BackgroundAI Ethics in the Public, Private, and NGO Sectors: A Review of a Global Document Collection
Schiff, D., Borenstein, J., Biddle, J., & Laas, K. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3052127 · Cited by 149 · cited 1×
“Developments in non-technical areas have also been reviewed in a few studies, and involve guidelines [145, 178, 294], standardization [210], and management processes [31, 274, 301].”
BackgroundTeach Me to Explain: A Review of Datasets for Explainable Natural Language ProcessingPreprint
Wiegreffe, S., & Marasović, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12060 · Cited by 149
Wing, J. M. (2021) · Communications of the ACM · doi:10.1145/3448248 · Cited by 148
Leite, J. C. S. D. P., & Cappelli, C. (2010) · Business & Information Systems Engineering · doi:10.1007/s12599-010-0102-z · Cited by 147
Adversarial vulnerability for any classifier
Fawzi, A., Fawzi, H., & Fawzi, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08686 · Cited by 144 · cited 1×
“To explain this phenomenon, Reference [116] has argued that the adversarial robustness corresponds to different data distributions that may hurt a model’s generalization.”
BackgroundCARSIM: CAR-FOLLOWING MODEL FOR SIMULATION OF TRAFFIC IN NORMAL AND STOP-AND-GO CONDITIONS
Benekohal, R. F., & Treiterer, J. (1988) · Transportation Research Record Journal of the Transportation Research Board · Cited by 143
Learning to cluster faces on an affinity graph
Yang, L., Zhan, X., Chen, D., Yan, J., Loy, C. C., & Lin, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00240 · Cited by 142
Towards Verified Artificial Intelligence
Seshia, S. A., Sadigh, D., & Sastry, S. S. (2016) · arXiv · doi:10.48550/arxiv.1606.08514 · Cited by 142 · cited 2×
“Simulation [42, 50, 54, 103, 192, 205, 323, 324, 359] Formal Verification [164, 298, 335]”
Graph Data Anonymization, De-Anonymization Attacks, and De-Anonymizability Quantification: A Survey
Ji, S., Mittal, P., & Beyah, R. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2016.2633620 · Cited by 141 · cited 1×
“For example, Reference [176] introduces several approaches into de-anonymize user information from graph data.”
Brun, Y., & Meliou, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264838 · Cited by 138 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundSelect, Answer and Explain: Interpretable Multi-Hop Reading Comprehension over Multiple Documents
Tu, M., Huang, K., Wang, G., Huang, J., He, X., & Zhou, B. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6441 · Cited by 136 · cited 3×
“For deep learning models like the Convolutional Neural Network (CNN) or the transformer, the inspection of intermediate features is a widely used means of explaining model behavior [332, 366].”
BackgroundMethodSelf-citationLearning Optimal and Fair Decision Trees for Non-Discriminative Decision-Making
Aghaei, S., Azizi, M. J., & Vayanos, P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33011418 · Cited by 135 · cited 1×
“For instance, quantitative fairness metrics can be used as regularization [7] or constraints for the optimization of the model parameters [67].”
Z-Inspection ® : A Process to Assess Trustworthy AI
Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., Moslein, F., et al. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3066209 · Cited by 134
Building and Auditing Fair Algorithms: A Case Study in Candidate Screening
Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445928 · Cited by 133 · cited 9×
“Besides the above representatives, designs of approaches to algorithmic auditing can be found in References [290, 356].”
BackgroundMethodInfluentialAdversarial Sticker: A Stealthy Attack Method in the Physical World
Wei, X., Guo, Y., & Yu, J. (2021) · arXiv · doi:10.48550/arxiv.2104.06728 · Cited by 133
Improving transferability of adversarial patches on face recognition with generative models
Xiao, Z., Gao, X., Fu, C., Dong, Y., Gao, W., Zhang, X., Zhou, J., & Zhu, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01167 · Cited by 132
When Age-Invariant Face Recognition Meets Face Age Synthesis: A Multi-Task Learning Framework
Huang, Z., Zhang, J., & Shan, H. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00720 · Cited by 131
A Review of Formal Methods applied to Machine LearningPreprint
Urban, C., & Miné, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02466 · Cited by 131
Protecting Sensitive Labels in Social Network Data Anonymization
Yuan, M., Chen, L., Yu, P. S., & Yu, T. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.259 · Cited by 130 · cited 1×
“Data format-specific DA approaches have been studied for decades [171, 372, 386].”
BackgroundAdaptive Sensitive Reweighting to Mitigate Bias in Fairness-aware Classification
Krasanakis, E., Spyromitros-Xioufis, E., Papadopoulos, S., & Kompatsiaris, Y. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186133 · Cited by 127 · cited 1×
“Similar to pre-processing methods, reweighting [195] and adversarial learning [68] can be used for in-processing, with the potential of either making use of the model parameters or predictions that are not yet fully optimized to more directly debias the model.”
MethodFairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
Kärkkäinen, K., & Joo, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.04913 · Cited by 126
A Survey of Black-Box Adversarial Attacks on Computer Vision ModelsPreprint
Bhambri, S., Muku, S., Tulasi, A., & Buduru, A. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.01667 · Cited by 126 · cited 4×
“Various improvements have been studied to improve this defense [45, 229, 304].”
BackgroundInfluentialKuang, K., Li, L., Geng, Z., Xu, L., Zhang, K., Liao, B., Huang, H., Ding, P., Miao, W., & Jiang, Z. (2020) · Engineering · doi:10.1016/j.eng.2019.08.016 · Cited by 126
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable ClaimsPreprint
Brundage, M., Avin, S., Wang, J., et al. (2020) · arXiv (Cornell University) · Cited by 124
Controlling Attribute Effect in Linear Regression
Calders, T., Karim, A., Kamiran, F., Ali, W., & Zhang, X. (2013) · 2013 IEEE 13th International Conference on Data Mining · doi:10.1109/icdm.2013.114 · Cited by 121 · cited 1×
“, disparate impact [117]); for realvalued regression, the difference between mean distances to the true value aggregated over each group might be used instead to indicate whether we model one group significantly better than another [59].”
BackgroundMourby, M., Mackey, E., Elliot, M., Gowans, H., Wallace, S. E., Bell, J., Smith, H., Aidinlis, S., & Kaye, J. (2018) · Computer Law & Security Review · doi:10.1016/j.clsr.2018.01.002 · Cited by 117 · cited 1×
“Data pseudonymization [251] is also a relevant technique promoted by the GDPR.”
MethodDecorrelated Adversarial Learning for Age-Invariant Face Recognition
Wang, H., Gong, D., Li, Z., & Liu, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00364 · Cited by 115
Decision Provenance: Harnessing Data Flow for Accountable Systems
Singh, J., Cobbe, J., & Norval, C. (2019) · IEEE Access · doi:10.1109/access.2018.2887201 · Cited by 110
Artificial Intelligence and Blockchain for Transparency in Governance
Alshamsi, M., Salloum, S. A., Alshurideh, M. T., & Abdallah, S. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-51920-9_11 · Cited by 110
Mitigating Poisoning Attacks on Machine Learning Models: A Data Provenance Based Approach
Baracaldo, N., Chen, B., Ludwig, H., & Safavi, J. A. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140450 · Cited by 108 · cited 1×
“Moreover, recent research has shown that data provenance can be used to mitigate data poisoning [33], thus enhancing the robustness and security of an AI system.”
BackgroundArtificial Intelligence for Social Good
Hager, G. D., Drobnis, A., Fang, F., Ghani, R., Greenwald, A., Lyons, T., Parkes, D. C., Schultz, J., Saria, S., Smith, S. F., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.05406 · Cited by 108
Likmeta, A., Metelli, A. M., Tirinzoni, A., Giol, R., Restelli, M., & Romano, D. (2020) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2020.103568 · Cited by 106
ModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI
Hummer, W., Muthusamy, V., Rausch, T., Dube, P., El Maghraoui, K., Murthi, A., & Oum, P. (2019) · 2019 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2019.00025 · Cited by 105 · cited 5×
“MLOps [260] and its variants such as ModelOps [187] and SafetyOps [344] extend DevOps to cover the ML lifecycle of data preparation, training, validation, and deployment, in its workflow.”
BackgroundMethodInfluentialConvLab: Multi-Domain End-to-End Dialog System Platform
Lee, S., Zhu, Q., Takanobu, R., Zhang, Z., Zhang, Y., Li, X., Li, J., Peng, B., Li, X., Huang, M., et al. (2019) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p19-3011 · Cited by 104 · cited 2×
“In NLP, especially conversational AI, simulators are widely used to simulate user behavior to test system ability and fulfill user needs by engaging in a dialog [205].”
BackgroundThe Dataset Nutrition Label: A Framework to Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2020) · Data Protection and Privacy · doi:10.5040/9781509932771.ch-001 · Cited by 103 · cited 5×
“Management Documentation [11, 22, 41, 129, 156, 246] Auditing [58, 228, 272, 277, 290, 292, 356] Cooperation [27, 90] Worklow MLOps [165, 233, 303]”
BackgroundInfluentialAdversarial objects against lidar-based autonomous driving systems
Cao, Y., Xiao, C., Yang, D., Fang, J., Yang, R., Liu, M., & Li, B. (2019) · arXiv · doi:10.48550/arxiv.1907.05418 · Cited by 102 · cited 1×
“2 Human–AI Interaction As an extension of human–computer interaction (HCI), human–AI interaction has aroused wide attention in the AI industry [3, 16].”
BackgroundSelf-citationA Survey on Privacy in Social Media: Identification, Mitigation, and Applications
Beigi, G., & Liu, H. (2020) · ACM/IMS Transactions on Data Science · doi:10.1145/3343038 · Cited by 101
MP2ML: a mixed-protocol machine learning framework for private inference
Boemer, F., Cammarota, R., Demmler, D., Schneider, T. H., & Yalame, H. (2020) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3407023.3407045 · Cited by 100
Gundersen, O. E., Gil, Y., & Aha, D. W. (2018) · AI Magazine · doi:10.1609/aimag.v39i3.2816 · Cited by 99
Fairness Through Computationally-Bounded Awareness
Kim, M. P., Reingold, O., & Rothblum, G. N. (2018) · arXiv · doi:10.48550/arxiv.1803.03239 · Cited by 97 · cited 1×
“, constraint optimization) can be applied to train a smaller model to transform model outputs or calibrate model confidence [186].”
BackgroundStructural Data De-anonymization: Quantification, Practice, and Implications
Ji, S., Li, W., Srivatsa, M., & Beyah, R. (2014) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2660267.2660278 · Cited by 96 · cited 1×
“Desirable data anonymization is expected to be immune from data de-anonymization or reidentification attacks that try to recover private information from anonymized data [111, 175].”
BackgroundOpen Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93
Bokc, T., Maurer, M., & Farber, G. (2007) · 2007 IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2007.4290183 · Cited by 92 · cited 2×
“Hardware-in-the-loop simulations can help developers understand system performance when it is run on chips, sensors, and actuators in a simulated environment, and is particularly helpful for latency- and power-critical systems like autonomous driving systems [50, 54].”
BackgroundMitigating face recognition bias via group adaptive classifier
Gong, S., Liu, X., & Jain, A. K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00342 · Cited by 92
Towards Interpretable Face Recognition
Yin, B., Tran, L., Li, H., Shen, X., & Liu, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00944 · Cited by 90
The Sanction of Authority: Promoting Public Trust in AI
Knowles, B., & Richards, J. T. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445890 · Cited by 86 · cited 1×
“On the whole, transparency serves as a basic requirement to build the public’s trust in AI systems [22, 178, 189].”
BackgroundBlockchain analytics and artificial intelligence
Dillenberger, D. N., Novotny, P., Zhang, Q., Jayachandran, P., Gupta, H., Hans, S., Verma, D., Chakraborty, S., Thomas, J. J., Walli, M. M., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2900638 · Cited by 82 · cited 1×
“To ensure that the provenance is tamper proof, the blockchain has been recently considered as a promising tool to certify data provenance in AI [15, 96].”
MethodOut-of-the-Box Reproducibility: A Survey of Machine Learning Platforms
Isdahl, R., & Gundersen, O. E. (2019) · eScience · doi:10.1109/escience.2019.00017 · Cited by 80 · cited 2×
“Based on this methodology, an increasing number of ML platforms are being developed that assist researchers and developers better track the lifecycle in a reproducible manner [169, 374].”
BackgroundMethodConsistent Instance False Positive Improves Fairness in Face Recognition
Xu, X., Huang, Y., Shen, P., Li, S., Li, J., Huang, F., Li, Y., & Cui, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00064 · Cited by 80
xCos: An Explainable Cosine Metric for Face Verification Task
Lin, Y., Liu, Z.-Y., Chen, Y., Wang, Y.-S., Chang, Y.-L., & Hsu, W. H. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3469288 · Cited by 78
Williford, J. R., May, B. B., & Byrne, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_15 · Cited by 76
Collaborative Search Log Sanitization: Toward Differential Privacy and Boosted Utility
Hong, Y., Vaidya, J., Lu, H., Karras, P., & Goel, S. (2015) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2014.2369034 · Cited by 75 · cited 1×
“DP has been explored in various data publishing tasks, such as log data [159, 385], set-valued data [76], correlated network data [75], and crowdsourced data [278, 344].”
BackgroundTrustworthy machine learning and artificial intelligence
Varshney, K. R. (2019) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/3313109 · Cited by 73 · cited 2×
“Some recent studies have organized these developments from the perspective of either research [182, 218, 357] or engineering [57, 62, 199, 338, 353].”
BackgroundFail-safe motion planning of autonomous vehicles
Magdici, S., & Althoff, M. (2016) · 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795594 · Cited by 71 · cited 3×
“In autonomous driving where safety is critical, a fail-safe mechanism like standing still has become an indispensable component in Advanced Driver-Assistant System (ADAS) products [160], and is being researched at a higher level of automation [230].”
BackgroundMethodTrustworthy AI Development Guidelines for Human System Interaction
Wickramasinghe, C. S., Marino, D. L., Grandio, J., & Manic, M. (2020) · 2020 13th International Conference on Human System Interaction (HSI) · doi:10.1109/hsi49210.2020.9142644 · Cited by 71 · cited 1×
“Some recent studies have organized these developments from the perspective of either research [182, 218, 357] or engineering [57, 62, 199, 338, 353].”
BackgroundIntervention user interfaces: a new interaction paradigm for automated systems
Schmidt, A., & Herrmann, T. (2017) · Interactions · doi:10.1145/3121357 · Cited by 70
Situ: Identifying and Explaining Suspicious Behavior in Networks
Goodall, J. R., Ragan, E. D., Steed, C. A., Reed, J. W., Richardson, G. D., Huffer, K. M. T., Bridges, R. A., & Laska, J. A. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865029 · Cited by 69 · cited 1×
“This evaluation has been used for the development of, for instance, recommendation systems [198] and data analysis [132].”
MethodFederated Doubly Stochastic Kernel Learning for Vertically Partitioned Data
Gu, B., Dang, Z., Li, X., & Huang, H. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403298 · Cited by 68 · cited 1×
“In Reference [138], an efficient method of kernel learning was proposed.”
MethodTrustworthy AI in the Age of Pervasive Computing and Big Data
Kumar, A., Braud, T., Tarkoma, S., & Hui, P. (2020) · 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops48775.2020.9156127 · Cited by 68 · cited 3×
“• We consider ethical requirements with broad concerns in recent literature [9, 57, 121, 145, 178, 199, 218, 301, 334], including fairness, privacy, and accountability.”
BackgroundAn overview of face liveness detection
Chakraborty, S., & Das, D. (2014) · International Journal on Information Theory · doi:10.5121/ijit.2014.3202 · Cited by 68
Fu, C., Olivares-Mendez, M. A., Suarez-Fernandez, R., & Campoy, P. (2013) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-013-9918-3 · Cited by 67
Fedface: Collaborative learning of face recognition model
Aggarwal, D., Zhou, J., & Jain, A. K. (2021) · 2021 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb52358.2021.9484386 · Cited by 67
Akula, A. R., Wang, K., Liu, C., Saba-Sadiya, S., Lu, H., Todorovic, S., Chai, J., & Zhu, S.-C. (2022) · iScience · doi:10.1016/j.isci.2021.103581 · Cited by 64 · cited 1×
“Counterfactual explanation [13, 185, 341] works in a contrary way by inding the boundary case to lip the outcome.”
BackgroundPark, N., Jang, K., Cho, S., & Choi, J. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106795 · Cited by 64
Data Cleaning for Accurate, Fair, and Robust Models: A Big Data - AI Integration Approach
Tae, K. H., Roh, Y., Oh, Y. H., Kim, H., & Whang, S. E. (2019) · Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning · doi:10.1145/3329486.3329493 · Cited by 62 · cited 3×
“Data Preprocessing Anomaly Detection [70, 81, 257, 316] Data Anonymozation [14, 37, 232, 232] Differential Privacy [107, 110, 177]”
BackgroundPerez, A., Garcia, M. I., Nieto, M., Pedraza, J. L., Rodriguez, S., & Zamorano, J. (2010) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2010.2046323 · Cited by 61 · cited 1×
“In many safety-critical systems, such as autonomous driving vehicles, logging systems [29, 261, 369] are widely adopted to trace and analyze the system execution.”
BackgroundMachine learning explainability via microaggregation and shallow decision trees
Blanco-Justicia, A., Domingo-Ferrer, J., Martínez, S., & Sánchez, D. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105532 · Cited by 61 · cited 1×
“Some naive measurements of model complexity, like tree depth [46] and the size of the rule set [202], have been studied as surrogate explainability metrics in previous work.”
ResultSoftware transparency as a key requirement for self-driving cars
Cysneiros, L. M., Raffi, M., & Sampaio do Prado Leite, J. C. (2018) · 2018 IEEE 26th International Requirements Engineering Conference (RE) · doi:10.1109/re.2018.00-21 · Cited by 61
Data sanitization against adversarial label contamination based on data complexity
Chan, P. P. K., He, Z., Li, H., & Hsu, C. (2017) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-016-0629-5 · Cited by 60 · cited 1×
“The mitigation of adversarial attacks is also referred to as data sanitization [71, 87, 258].”
BackgroundFu, C., Olivares-Méndez, M. A., Fernández, R. A. S., & Campoy, P. (2014) · Hispana · Cited by 57
Deep Nets Don't Learn via Memorization
Krueger, D., Ballas, N., Jastrzȩbski, S., et al. (2017) · PolyPublie (École Polytechnique de Montréal) · Cited by 57
Dırık, A. E., Sencar, H. T., & Memon, N. D. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2361200 · Cited by 55 · cited 1×
“Specific DA methods have also been designed for relational data [262], set-valued data [151, 320], and image data [97, 239].”
BackgroundHow to Measure the Reproducibility of System-oriented IR Experiments
Breuer, T., Ferro, N., Fuhr, N., Maistro, M., Sakai, T., Schaer, P., & Soboroff, I. M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401036 · Cited by 54 · cited 1×
“For example, References [53, 118] designed metrics to quantify how closely an information retrieval system can be reproduced to its origins.”
BackgroundDesign and evaluation of a data anonymization pipeline to promote Open Science on COVID-19
Jakob, C., Kohlmayer, F., Meurers, T., Vehreschild, J. J., & Praßer, F. (2020) · Scientific Data · doi:10.1038/s41597-020-00773-y · Cited by 52
Secure Multi-pArty Computation Grid LOgistic REgression (SMAC-GLORE)
Shi, H., Jiang, C., Dai, W., Jiang, X., Tang, Y., Ohno-Machado, L., & Wang, S. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0316-1 · Cited by 51 · cited 1×
“, linear regression [128] and logistic regression [300], and generic model learning tasks [247].”
BackgroundAnonymizing Query Logs by Differential Privacy
Zhang, S., Yang, G. H., & Singh, L. (2016) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2911451.2914732 · Cited by 51 · cited 1×
“DP has been explored in various data publishing tasks, such as log data [159, 385], set-valued data [76], correlated network data [75], and crowdsourced data [278, 344].”
BackgroundFairFace Challenge at ECCV 2020: Analyzing Bias in Face Recognition
Sixta, T., Jacques, J. C. S., Buch-Cardona, P., Vazquez, E., & Escalera, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-65414-6_32 · Cited by 51 · cited 4×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundInfluentialTree Space Prototypes: Another Look at Making Tree Ensembles Interpretable
Tan, S., Soloviev, M., Hooker, G., & Wells, M. T. (2020) · Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference · doi:10.1145/3412815.3416893 · Cited by 49 · cited 1×
“been proposed to approximate ML models, such as random forests [317, 392] and neural networks [28, 86, 393].”
MethodHorwick, M., & Siedersberger, K.-H. (2010) · 2010 IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2010.5548115 · Cited by 49 · cited 3×
“In autonomous driving where safety is critical, a fail-safe mechanism like standing still has become an indispensable component in Advanced Driver-Assistant System (ADAS) products [160], and is being researched at a higher level of automation [230].”
BackgroundMethodFAE: A Fairness-Aware Ensemble Framework
Iosifidis, V., Fetahu, B., & Ntoutsi, E. (2019) · 2019 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata47090.2019.9006487 · Cited by 49 · cited 1×
“Reweighting the predictions of multiple models can also help reduce bias [168].”
BackgroundTrust and Artificial Intelligence
Stanton, B., & Jensen, T. (2021) · doi:10.6028/nist.ir.8332-draft · Cited by 48
Benchmarking and Survey of Explanation Methods for Black Box Models
Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., & Rinzivillo, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.13076 · Cited by 47 · cited 9×
“, (1) the explanation is fragile to perturbations [130], (2) the explanation is not always consistent with human interpretation [47], and (3) it is difficult to judge if the explanation is correct or faithful [250].”
BackgroundMethodInfluentialSarkar, D., Bali, R., & Sharma, T. (2017) · Apress eBooks · doi:10.1007/978-1-4842-3207-1_1 · Cited by 47
A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning
Mohseni, S., Block, J. E., & Ragan, E. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05075 · Cited by 45 · cited 2×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
The Role of Cooperation in Responsible AI Development
Askell, A., Brundage, M., & Hadfield, G. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04534 · Cited by 44 · cited 4×
“Cooperation with regulators certifies the products to appropriately follow principles of trustworthiness.”
BackgroundMethodInfluentialSoftware Testing for Machine Learning
Marijan, D., & Gotlieb, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7084 · Cited by 40 · cited 3×
“Development Functional Testing [164, 235, 381] Performance Benchmarking [40, 56, 88, 93, 123, 127, 153, 238, 248, 307, 327]”
BackgroundAuto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-trainingPreprint
Pan, Y., Li, Y., Luo, J., Xu, J., Yao, T., & Mei, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02375 · Cited by 40 · cited 2×
“In addition, self-supervised learning provides a good mechanism to learn a cross-modality feature representation, such as the vision and language models including VL-BERT [355] and Auto-CapTIONs [284].”
MethodInterpreting Models via Single Tree Approximation
Zhou, Y., & Hooker, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09036 · Cited by 39 · cited 1×
“been proposed to approximate ML models, such as random forests [317, 392] and neural networks [28, 86, 393].”
MethodLearning Security Classifiers with Verified Global Robustness Properties
Chen, Y., Wang, S., Qin, Y., Liao, X., Jana, S., & Wagner, D. (2021) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3460120.3484776 · Cited by 39 · cited 1×
“To achieve certified robustness on unseen inputs as well, global robustness has recently attracted the interest of the AI community [77, 206].”
BackgroundHardware-in-the-Loop Autonomous Driving Simulation Without Real-Time Constraints
Brogle, C., Zhang, C., Lim, K. L., & Bräunl, T. (2019) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2919457 · Cited by 38 · cited 2×
“Hardware-in-the-loop simulations can help developers understand system performance when it is run on chips, sensors, and actuators in a simulated environment, and is particularly helpful for latency- and power-critical systems like autonomous driving systems [50, 54].”
BackgroundAccuracy-Constrained Privacy-Preserving Access Control Mechanism for Relational Data
Pervaiz, Z., Aref, W. G., Ghafoor, A., & Prabhu, N. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2013.71 · Cited by 36
Raji, I. D., & Yang, J. (2019) · arXiv · doi:10.48550/arxiv.1912.06166 · Cited by 36
The Smart Black Box: A Value-Driven High-Bandwidth Automotive Event Data Recorder
Yao, Y., & Atkins, E. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2971385 · Cited by 33 · cited 1×
“In many safety-critical systems, such as autonomous driving vehicles, logging systems [29, 261, 369] are widely adopted to trace and analyze the system execution.”
BackgroundA General Retraining Framework for Scalable Adversarial ClassificationPreprint
Li, B., Vorobeychik, Y., & Chen, X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.02606 · Cited by 33 · cited 1×
“This is typically referred to as adversarial training [134, 211, 346].”
BackgroundSelf-citationSecurity of Neural Networks from Hardware Perspective
Xu, Q., Arafin, M. T., & Qu, G. (2021) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1145/3394885.3431639 · Cited by 33 · cited 3×
“Deployment Anomaly Monitoring [70, 257, 394] Human-AI Interaction [72, 179, 301, 326] Fail-Safe Mechanism [126, 160, 230, 252] Hardware Security [264, 287, 364]”
BackgroundMethodKafle, S., Glasser, A., Al-khazraji, S., Berke, L., Seita, M., & Huenerfauth, M. (2020) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386300 · Cited by 31 · cited 3×
“Mitigating such unfairness has aroused concerns in recent research [179, 326].”
BackgroundMethodMonitoring Misuse for Accountable 'Artificial Intelligence as a Service'
Javadi, S. A., Cloete, R., Cobbe, J., Lee, M. S. A., & Singh, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375873 · Cited by 29
Improved Adversarial Learning for Fair Classification
Celis, L. E., & Keswani, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10443 · Cited by 28 · cited 1×
“Similar to pre-processing methods, reweighting [195] and adversarial learning [68] can be used for in-processing, with the potential of either making use of the model parameters or predictions that are not yet fully optimized to more directly debias the model.”
MethodWho Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?
Makinen, S., Skogstrom, H., Laaksonen, E., & Mikkonen, T. (2021) · 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN) · doi:10.1109/wain52551.2021.00024 · Cited by 28 · cited 12×
“MLOps [260] and its variants such as ModelOps [187] and SafetyOps [344] extend DevOps to cover the ML lifecycle of data preparation, training, validation, and deployment, in its workflow.”
BackgroundMethodInfluentialTechnology Readiness Levels for Machine Learning SystemsPreprint
Lavin, A., Lee, C., Visnjic, A., et al. (2021) · Research Square · doi:10.21203/rs.3.rs-133138/v1 · Cited by 27
TAII Framework for Trustworthy AI systems
Baker-Brunnbauer, J. (2021) · ROBONOMICS: The Journal of the Automated Economy · doi:10.67723/rj.v2i.17 · Cited by 23
Interpretable & Explorable Approximations of Black Box Models
Lakkaraju, H., Kamar, E., Caruana, R., & Leskovec, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01154 · Cited by 23 · cited 1×
“Some naive measurements of model complexity, like tree depth [46] and the size of the rule set [202], have been studied as surrogate explainability metrics in previous work.”
ResultBai, F., Wu, J., Shen, P., Li, S., & Zhou, S. (2021) · arXiv · doi:10.48550/arxiv.2105.02501 · Cited by 21
Bankins, S., & Formosa, P. (2021) · Redefining the Psychological Contract in the Digital Era · doi:10.1007/978-3-030-63864-1_4 · Cited by 19
CNN-Cert: An Efficient Framework for Certifying Robustness of Convolutional Neural Networks
Boopathy, A., Weng, T.-W., Chen, P., Liu, S., & Daniel, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013240 · Cited by 18 · cited 3×
“For example, CNN-Cert [51], CROWN [379], Fast-lin, and Fast-lip [352] aim to minimize an upper bound of the worst-case loss under given input perturbations.”
BackgroundOverview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm
Ferro, N., Maistro, M., Sakai, T., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98932-7_23 · Cited by 17 · cited 2×
“For example, References [53, 118] designed metrics to quantify how closely an information retrieval system can be reproduced to its origins.”
BackgroundMethodxCos: An Explainable Cosine Metric for Face Verification TaskPreprint
Lin, Y., Liu, Z., Chen, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05383 · Cited by 17
Ayvaz, S., & Cetin, S. C. (2019) · International Journal of Intelligent Unmanned Systems · doi:10.1108/ijius-05-2018-0011 · Cited by 16
An Ontology for Standardising Trustworthy AI
Lewis, D., Filip, D., & J. Pandit, H. (2021) · Factoring Ethics in Technology, Policy Making, Regulation and AI · doi:10.5772/intechopen.97478 · Cited by 13
Scenario-Based Simulation Environment for Assistance Systems
Tideman, M. (2010) · ATZ autotechnology · doi:10.1007/bf03247153 · Cited by 13
Federated Robustness Propagation: Sharing Adversarial Robustness in Federated Learning.Preprint
Hong, J., Wang, H., Wang, Z., & Zhou, J. (2021) · arXiv (Cornell University) · Cited by 12
Applying Transparency in Artificial Intelligence based Personalization SystemsPreprint
Schelenz, L., Segal, A., & Gal, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00935 · Cited by 12
Proceedings of the 2009 International Conference on Network-Based Information Systems
Mwebaze, J., Boxhoorn, D., & Valentijn, E. A. (2009) · International Conference on Network-Based Information Systems · Cited by 11
Fedlearn-Algo: A flexible open-source privacy-preserving machine learning platformPreprint
Liu, B., Tan, C., Wang, J., Zeng, T., Shan, H., Yao, H., Huang, H., Dai, P., Bo, L., & Chen, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04129 · Cited by 11 · cited 1×
“To expedite FL-related research and development, many open source libraries have been released, such as FATE, FedML [149], and FedlearnAlgo [217].”
MethodSelf-citationStructured Memory for Neural Turing MachinesPreprint
Zhang, W., Yang, Y., & Zhou, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03931 · Cited by 11
Self-citationTrustworthy AI Inference Systems: An Industry Research View
Cammarota, R., Schunter, M., Rajan, A., Boemer, F., Kiss, Á., Treiber, A., Weinert, C., Schneider, T., Stapf, E., Sadeghi, A.-R., et al. (2020) · arXiv · doi:10.48550/arxiv.2008.04449 · Cited by 9 · cited 2×
“Some recent studies have organized these developments from the perspective of either research [182, 218, 357] or engineering [57, 62, 199, 338, 353].”
BackgroundAn Introduction on Interpretable Machine Learning
(2020) · International Journal of Innovative Technology and Exploring Engineering · doi:10.35940/ijitee.g1023.0597s20 · Cited by 8 · cited 1×
“For operators like bank executives, explainability helps understand the AI credit system to prevent potential defects in it [25, 184].”
BackgroundCryptoSPN: Expanding PPML beyond Neural Networks
Treiber, A., Molina, A., Weinert, C., Schneider, T. H., & Kersting, K. (2020) · doi:10.1145/3411501.3419417 · Cited by 7
How to Keep an Online Learning Chatbot From Being Corrupted
Chai, Y., Liu, G., Jin, Z., & Sun, D. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206897 · Cited by 6
did i say something wrong?': Towards a safe collaborative chatbot
Chkroun, M., & Azaria, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12145 · Cited by 6
Extracting symbolic rules from trained neural network ensembles
ZhouZhi-Hua, JiangYuan, & ChenShi-Fu (2003) · AI Communications · doi:10.5555/859130.859132 · Cited by 5
Towards evaluating and eliciting high-quality documentation for intelligent systems
Piorkowski, D., González, D., Richards, J., & Houde, S. (2020) · arXiv · doi:10.48550/arxiv.2011.08774 · Cited by 5 · cited 2×
“Reference [265] summarized the general qualitative dimensions for more specified evaluations.”
BackgroundSiddique, U. (2020) · arXiv · doi:10.48550/arxiv.2008.04461 · Cited by 5
Conversational AI Systems for Social Good: Opportunities and Challenges
Qi, P., Huang, J., Wu, Y., He, X., & Zhou, B. (2021) · arXiv · doi:10.48550/arxiv.2105.06457 · Cited by 3
Self-citationMuller, M. (2016) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010655275 · Cited by 2 · cited 3×
“In the area of Unmanned Aerial Vehicle (UAV), the fail-safe algorithm has been studied for a long time to avoid frequent collision of quadrocopters [126] and to ensure safe landing upon system failure [252].”
BackgroundMethodUntitled work
Srivastava (2014) · doi:10.5555/2627435.2670313 · cited 1×
“Other than adjusting the architecture of the model, one can mitigate overfitting to obtain better generalization via various explicit or implicit regularizations, such as early stopping [370], batch normalization [167], dropout [309], data augmentation, and weight decay [196].”
Generalization in Deep Learning
Kawaguchi, K., Kaelbling, L. P., & Bengio, Y. (2017) · arXiv · doi:10.48550/arxiv.1710.05468
Technology Readiness Levels for Machine Learning Systems
Lavin, A., Gilligan-Lee, C. M., Visnjic, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.03989
An Overview of Face Liveness Detection
Chakraborty, S., & Das, D. (2014) · arXiv · doi:10.48550/arxiv.1405.2227
AI Ethics in the Public, Private, and NGO Sectors: A Review of a Global Document Collection
Schiff, D., Borenstein, J., Biddle, J., & Laas, K. (2021) · doi:10.36227/techrxiv.14109482
Research works printed but not linked: 60
- Dalal Al-Azizy, David Millard, Iraklis Symeonidis, Kieron O’Hara, and Nigel Shadbolt. 2015. A literature survey and classifications on data deanonymisation. In Proceedings of the International Conference on Risks and Security of Internet and Systems. Springer, 36–51.
- Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR’15) .
- Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency. PMLR, 77–91.
- 10.5555/3294996.3295155 · doi:10.5555/3294996.3295155
- Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In Proceedings of the 30th USENIX Security Symposium (USENIX Security’21). 2633–2650.
- Yu Cheng Mo Yu Xiaoxiao Guo and Bowen Zhou. 2019. Few-shot learning with meta metric learners. In NIPS 2017 Workshop on Meta-Learning .
- Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. 2017. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning. PMLR, 854–863.
- Bo Cowgill and Catherine Tucker. 2017. Algorithmic bias: A counterfactual perspective. NSF Trustworthy Algorithms.
- Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. 2021. RobustBench: A standardized adversarial robustness benchmark. In Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
- Luiz Marcio Cysneiros and Vera Maria Benjamim Werneck. 2009. An initial analysis on how software transparency and trust influence each other. In Proceedings of the Workshop on Requirements Engineering (WER’09). Citeseer.
- Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Conference on Robot Learning. PMLR, 1–16.
- 10.5555/1791834.1791836 · doi:10.5555/1791834.1791836
- Exforsys. 2011. What Is Monkey Testing. Retrieved July 9, 2021 from http://www.exforsys.com/tutorials/testing-types/ monkey-testing.html.
- Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. PMLR, 1126–1135.
- Shixiang Gu and Luca Rigazio. 2015. Towards deep neural network architectures robust to adversarial examples. In 3rd International Conference on Learning Representations (ICLR’15) San Diego CA USA May 7-9 2015 Workshop Track Proceedings .
- IBM. Trusting AI. Retrieved February 20, 2021 from https://www.research.ibm.com/artificial-intellig ence/trusted-ai/.
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. PMLR, 448–456.
- Shouling Ji, Weiqing Li, Prateek Mittal, Xin Hu, and Raheem Beyah. 2015. \(\lbrace\) SecGraph \(\rbrace\) : A uniform and open-source evaluation system for graph data anonymization and de-anonymization. In Proceedings of the 24th USENIX Security Symposium (USENIX Security’15). 303–318.
- Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, Vol. 2.
- Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International Conference on Machine Learning. PMLR, 1885–1894.
- Klas Leino, Zifan Wang, and Matt Fredrikson. 2021. Globally-robust neural networks. In International Conference on Machine Learning. PMLR, 6212–6222.
- Ke Li and Jitendra Malik. 2017. Learning to optimize. In International Conference on Learning Representations .
- Yiming Li Yong Jiang Zhifeng Li and Shu-Tao Xia. 2017. Backdoor learning: A survey. IEEE Transactions on Neural Networks and Learning Systems .
- Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses. Springer, 273–294.
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations.
- Pratyush Maini, Eric Wong, and Zico Kolter. 2020. Adversarial robustness against the union of multiple perturbation models. In International Conference on Machine Learning. PMLR, 6640–6650.
- Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning. PMLR, 1614–1623.
- Jan Hendrik Metzen Tim Genewein Volker Fischer and Bastian Bischoff. 2017. On detecting adversarial perturbations. In 5th International Conference on Learning Representations (ICLR’17) Toulon France April 24-26 2017 Conference Track Proceedings .
- Silvio Micali, Oded Goldreich, and Avi Wigderson. 1987. How to play any mental game. In Proceedings of the 19th ACM Symposium on Theory of Computing (STOC’87). ACM, 218–229.
- Christoph Molnar. 2020. Interpretable Machine Learning. Retrieved from Lulu.com.
- OpenAI. 2018. OpenAI Charter. Retrieved February 20, 2021 from https://openai.com/charter/.
- Andrea Paudice, Luis Muñoz-González, and Emil C. Lupu. 2018. Label sanitization against label flipping poisoning attacks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 5–15.
- Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Certified defenses against adversarial examples. In Proceedings of the International Conference on Learning Representations.
- Aditi Raghunathan Sang Michael Xie Fanny Yang John C. Duchi and Percy Liang. 2019. Adversarial training can hurt generalization. In ICML 2019 Workshop Deep Phenomena .
- Yuji Roh, Kangwook Lee, Steven Whang, and Changho Suh. 2020. FR-train: A mutual information-based approach to fair and robust training. In International Conference on Machine Learning. PMLR, 8147–8157.
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of the Data and Discrimination: Converting Critical Concerns Into Productive Inquiry 22 (2014), 4349–4357.
- Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning. PMLR, 1842–1850.
- Sebastian Schelter, Joos-Hendrik Boese, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert. 2017. Automatically tracking metadata and provenance of machine learning experiments. In Machine Learning Systems Workshop at NIPS. 27–29.
- Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning. PMLR, 3145–3153.
- Simone Stumpf, Adrian Bussone, and Dympna O’sullivan. 2016. Explanations considered harmful? user interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).
- Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre-training of generic visual-linguistic representations. In Proceedings of the International Conference on Learning Representations.
- Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning. PMLR, 3319–3328.
- Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK. 2020. Auditing Machine Learning Algorithms. Retrieved from https://www.auditingalgorithms.net/index.html.
- Liang Tong, Bo Li, Chen Hajaj, Chaowei Xiao, Ning Zhang, and Yevgeniy Vorobeychik. 2019. Improving robustness of \(\lbrace\) ML \(\rbrace\) classifiers against realizable evasion attacks using conserved features. In Proceedings of the 28th USENIX Security Symposium (USENIX Security’19). 285–302.
- Cecília Torres, Walter Franklin, and Laura Martins. 2018. Accessibility in chatbots: The state of the art in favor of users with visual impairment. In International Conference on Applied Human Factors and Ergonomics. Springer, 623–635.
- Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. 2018. Ensemble adversarial training: Attacks and defenses. In Proceedings of the International Conference on Learning Representations.
- Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction \(\lbrace\) APIs \(\rbrace\) . In Proceedings of the 25th USENIX Security Symposium (USENIX Security’16). 601–618.
- Dimitris Tsipras Shibani Santurkar Logan Engstrom Alexander Turner and Aleksander Ma̧dry. 2019. Robustness may be at odds with accuracy. In 7th International Conference on Learning Representations (ICLR’19) New Orleans LA USA May 6-9 2019 .
- UNI Global Union. 2017. Top 10 principles for ethical artificial intelligence. In The Future World of Work.
- Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019. On the convergence and robustness of adversarial training. In Proceedings of the International Conference on Machine Learning (ICML’19), Vol. 1. 2.
- Yingxu Wang and Jingqiu Shao. 2003. Measurement of the cognitive functional complexity of software. In Proceedings of the 2nd IEEE International Conference on Cognitive Informatics. IEEE, 67–74.
- Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. 2018. Evaluating the robustness of neural networks: An extreme value theory approach. In Proceedings of the International Conference on Learning Representations.
- Tong Wu Liang Tong and Yevgeniy Vorobeychik. 2020. Defending against physically realizable attacks on image classification. In 8th International Conference on Learning Representations (ICLR’20) Addis Ababa Ethiopia April 26-30 2020 .
- Weiyi Xia, Yongtai Liu, Zhiyu Wan, Yevgeniy Vorobeychik, Murat Kantacioglu, Steve Nyemba, Ellen Wright Clayton, and Bradley A. Malin. 2021. Enabling realistic health data re-identification risk assessment through adversarial modeling. J. Am. Med. Inf. Assoc. 28, 4 (2021), 744–752.
- Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021. To be robust or to be fair: Towards fairness in adversarial training. In International Conference on Machine Learning. PMLR, 11492–11501.
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning. PMLR, 2048–2057.
- 10.5555/1382436.1382751 · doi:10.5555/1382436.1382751
- Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei. 2021. Seco: Exploring sequence supervision for unsupervised representation learning. In AAAI, Vol. 2. 7.
- Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. 2018. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull. 41, 4 (2018), 39–45.
- Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. PMLR, 325–333.
Cited by
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond
Li, X., Xiong, H., Li, X., Wu, X., Zhang, X., Liu, J., Bian, J., & Dou, D. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01756-8 · Cited by 497
“Researchers [29, 47, 81, 101, 107, 183] also proposed many other desiderata for interpretations, interpretation algorithms or interpretability, such as fairness, privacy, reliability, robustness, causality, trust, fidelity, faithfulness, transferability, informativeness, transparency, plausibility, satisfaction, accou…”
BackgroundSelf-citationEsmaeilzadeh, P. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102861 · Cited by 377
Joyce, D. W., Kormilitzin, A., Smith, K. A., & Cipriani, A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00751-9 · Cited by 238
Post-hoc vs ante-hoc explanations: xAI design guidelines for data scientists
Retzlaff, C. O., Angerschmid, A., Saranti, A., Schneeberger, D., Röttger, R., Müller, H., & Holzinger, A. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101243 · Cited by 194
“In any case, trustworthy AI encompasses elements of security (Li et al., 2023), safety, privacy, fairness (Angerschmid et al., 2022), accountability, auditability, and robustness (Holzinger, Dehmer et al., 2022).”
Metadata
- Authors
- Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B.
- Affiliations
- Tsinghua University (CN)
- Amazon (United States) (US)
- Walmart Inc., Mountain View, CA, USA
- Jingdong (China) (CN)
- JD Technology, Mountain View, CA, USA
- Frontis.AI, Beijing, China
- Venue
- ACM Computing Surveys
- Published
- 2023-01-16
- Volume
- 55
- Issue
- 9
- Pages
- 1-46
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3202183072
- Cited by
- 612
- References
- 384
- Citations by year
- 2026: 164 · 2025: 207 · 2024: 169 · 2023: 65
- FWCI
- 129.72
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- trustworthy AI, robustness, generalization, explainability, transparency, reproducibility, fairness, privacy preservation, accountability, AI lifecycle, AI governance, continuous monitoring, algorithmic bias, imperceptible attacks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.