FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Open accessAbstract
Accuracy is an important concern for suppliers of artificial intelligence (AI) services, but considerations beyond accuracy, such as safety (which includes fairness and explainability), security, and provenance, are also critical elements to engender consumers’ trust in a service. Many industries use transparent, standardized, but often not legally required documents called supplier's declarations of conformity (SDoCs) to describe the lineage of a product along with the safety and performance testing it has undergone. SDoCs may be considered multidimensional fact sheets that capture and quantify various aspects of the product and its development to make it worthy of consumers’ trust. In this article, inspired by this practice, we propose FactSheets to help increase trust in AI services. We envision such documents to contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers. We suggest a comprehensive set of declaration items tailored to AI in the Appendix of this article.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI documentation is advocated to increase the transparency and facilitate understanding of AI, and is embodied in recently proposed features of AI Factsheets and Model Cards.
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35,36,40,75,76,82], and is embodied in recently proposed features of AI Factsheets [8,81] and Model Cards [64].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [8]
There is some work on fairness relating to AI services, though this tends to focus on either exposing or 'auditing' certain AI services, or devising tools, methods and interventions relevant for those building (rather than using) such services.
“There is some work on fairness relating to AI services, though this tends to focus on either exposing or 'auditing' certain AI services [22,38,126,133], or devising tools, methods and interventions relevant for those building (rather than using) such services [12,96,106,128].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [12]
There appears a clear role for adopting and extending various documentation approaches, be they relating to data and models (such as datasheets, model cards and factsheets), their interactions and interconnections (e.g. decision provenance), and broader, holistic socio-technical system reviews (e.g. reviewability and traceability).
“There appears a clear role for adopting and extending various documentation approaches, be they relating to data and models (such as datasheets [53], model cards [106] and factsheets [12]), their interactions and interconnections (e.g.decision provenance [139]), and broader, holistic socio-technical system reviews (e.g.reviewability [27] and traceability [86]).”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Market for "Lemons": Quality Uncertainty and the Market Mechanism
Akerlof, G. A. (1970) · The Quarterly Journal of Economics · doi:10.2307/1879431 · Cited by 23,016 · see citing works
Axford, N. (2017) · Routledge eBooks · doi:10.4324/9781315258324-16 · Cited by 11,329 · see citing works
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,673 · see citing works
Another look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757 · see citing works
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993 · see citing works
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2× · see citing works
“…et al. proposed model cards to convey information that characterizes the evaluation of a machine learning model in a variety of conditions and disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information [11].”
BackgroundMethodInfluentialGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827 · see citing works
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581 · see citing works
Self-citationLee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · see citing works
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884 · see citing works
Self-citationBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
Self-citationCertified Defenses against Adversarial ExamplesPreprint
Raghunathan, A., Steinhardt, J., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09344 · Cited by 665 · see citing works
Li, X., Hess, T. J., & Valacich, J. S. (2008) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2008.01.001 · Cited by 643 · see citing works
Adversarial Robustness Toolbox v1.0.0Preprint
Nicolae, M.-I., Sinn, M., Minh, T. N., Buesser, B., Rawat, A., Wistuba, M., Zantedeschi, V., Baracaldo, N., Chen, B., Ludwig, H., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.01069 · Cited by 467 · see citing works
Constitutional democracy and technology in the age of artificial intelligence
Nemitz, P. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0089 · Cited by 418 · see citing works
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334 · see citing works
On the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Varshney, K. R., & Alemzadeh, H. (2017) · Big Data · doi:10.1089/big.2016.0051 · Cited by 211 · cited 5× · see citing works
“Safety, discussed in [3] and [5] as the minimization of both risk and epistemic uncertainty, will include explainability, algorithmic fairness, and robustness to dataset shift.”
BackgroundInfluentialSelf-citationDisparate Impact in Big Data PolicingPreprint
Selbst, A. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2819182 · Cited by 189 · see citing works
Certifying the reliability of software
Currit, P. A., Dyer, M., & Mills, H. D. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312914 · Cited by 176 · see citing works
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175 · see citing works
Wagstaff, K. (2012) · arXiv · doi:10.48550/arxiv.1206.4656 · Cited by 148 · see citing works
A Brief Review of the Belmont Report
Sims, J. (2010) · Dimensions of Critical Care Nursing · doi:10.1097/dcc.0b013e3181de9ec5 · Cited by 135 · see citing works
Principles of engineering safety: Risk and uncertainty reduction
Möller, N., & Hansson, O. (2007) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2007.03.031 · Cited by 134 · see citing works
Who is trustworthy? Predicting trustworthy intentions and behavior.
Levine, E. E., Bitterly, T. B., Cohen, T. R., & Schweitzer, M. E. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000136 · Cited by 97 · see citing works
The Concepts of Risk and Safety
Möller, N. (2012) · Springer eBooks · doi:10.1007/978-94-007-1433-5_3 · Cited by 69 · see citing works
Algorithms for interpretable machine learning
Rudin, C. D. (2014) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2630823 · Cited by 61 · see citing works
Correction to 'Certifying the reliability of software' (Jan. 1986 3-11)
Curritt, P. A., Dyer, M., & Mills, H. D. (1989) · IEEE Transactions on Software Engineering · doi:10.1109/32.21765 · Cited by 60 · see citing works
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
ben‐Aaron, J., Denny, M., Desmarais, B. A., & Wallach, H. (2016) · Public Administration Review · doi:10.1111/puar.12596 · Cited by 50 · see citing works
Disparate Impact in Big Data Policing
Selbst, A. D. (2018) · Digital Commons (University of Georgia School of Law) · Cited by 44 · see citing works
(2018) · arXiv (Cornell University) · Cited by 41 · see citing works
Self-citationA software product certification model
Heck, P., Klabbers, M., & van Eekelen, M. (2009) · Software Quality Journal · doi:10.1007/s11219-009-9080-0 · Cited by 40 · see citing works
The Limits of Abstract Evaluation Metrics
Olteanu, A., Talamadupula, K., & Varshney, K. R. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3098871 · Cited by 30 · see citing works
Self-citationTowards Composable Bias Rating of AI Services
Srivastava, B., & Rossi, F. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278744 · Cited by 26 · see citing works
An Approach for Certifying Security in Software Components
Ghosh, A. K., & McGraw, G. E. (1998) · Cited by 22 · see citing works
Port, D. N., & Wilf, J. M. (2013) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.54 · Cited by 22 · see citing works
(2018) · Cited by 17 · see citing works
McAllister, L. K. (2014) · Michigan Journal of Environmental & Administrative Law · doi:10.36640/mjeal.3.2.harnessing · Cited by 5 · see citing works
Flagged first, then by citations
1 of the 44 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- [11]11. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, "Model cards for model re- porting," in Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, Atlanta, USA, Jan. 2019.
- [24]24. K. R. Varshney, "Data science of the people, for the people, by the people: A viewpoint on an emerging dichotomy," in Data for Good Ex- change Conference, New York, USA, Sep. 2015.
- [27]27. K. L. Wagstaff, "Machine learning that mat- ters," in Proceedings of the International Con- ference on Machine Learning, Edinburgh, UK, Jun.-Jul. 2012, pp. 529-536.
- [43]43. L. K. McAllister, "Harnessing private regula- tion," Michigan Journal of Environmental & Ad- ministrative Law, vol. 3, no. 2, pp. 291-420, 2014.
- [45]45. Armstrong, J. S. (1978). Long-range forecasting: From crystal ball to computer. John Wiley & Sons.
Cited by
Sorted by citations
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751 · see citing works
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957 · see citing works
“This will increase transparency and accountability by providing detailed and continuous information on the AI tools during their lifetime to clinicians, healthcare organisations, citizens and patients, AI developers, and relevant authorities.”
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545 · see citing works
Trust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 618 · see citing works
The Ethics of Artificial Intelligence
Floridi, L. (2023) · doi:10.1093/oso/9780198883098.001.0001 · Cited by 608 · see citing works
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384 · see citing works
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Sokol, K., & Flach, P. A. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372870 · Cited by 278 · see citing works
Socially Responsible AI Algorithms: Issues, Purposes, and Challenges
Cheng, L., Varshney, K. R., & Liu, H. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12814 · Cited by 261 · see citing works
Self-citationGoverning AI safety through independent audits
Falco, G., Shneiderman, B., Badger, J., Carrier, R., Dahbura, A., Danks, D., Eling, M., Goodloe, A., Gupta, J., Hart, C., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00370-7 · Cited by 250 · see citing works
Interpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Chapman & Hall/CRC eBooks · doi:10.1201/9780367816377-16 · Cited by 271 · see citing works
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 225 · see citing works
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219 · see citing works
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35, 36, 40, 75, 76, 82], and is embodied in recently proposed features of AI Factsheets [8, 81] and Model Cards [64].”
BackgroundSelf-citationLai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204 · see citing works
Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making
Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512930 · Cited by 216 · see citing works
Self-citationLukyanenko, R., Maaß, W., & Storey, V. C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00605-4 · Cited by 194 · see citing works
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169 · see citing works
Designing for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165 · see citing works
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151 · see citing works
Self-citationReviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150 · see citing works
“approaches such as service ‘factsheets’ [3] that describe aspects of their service beyond technical capabilities.”
BackgroundMorley, J., Murphy, L., Mishra, A., Joshi, I., & Karpathakis, K. (2021) · JMIR Formative Research · doi:10.2196/31623 · Cited by 119 · see citing works
Understanding artificial intelligence ethics and safetyPreprint
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 268 · see citing works
Fehr, J., Citro, B., Malpani, R., Lippert, C., & Madai, V. I. (2024) · Frontiers in Digital Health · doi:10.3389/fdgth.2024.1267290 · Cited by 150 · see citing works
Z-Inspection <sup>®</sup> : A Process to Assess Trustworthy AI
Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., Moslein, F., et al. (2021) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2021.3066209 · Cited by 134 · see citing works
XAIR: A Systematic Metareview of Explainable AI (XAI) Aligned to the Software Development Process
Clement, T., Kemmerzell, N., Abdelaal, M., & Amberg, M. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5010006 · Cited by 131 · see citing works
Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering
Lu, Q., Zhu, L., Xu, X., Whittle, J., Zowghi, D., & Jacquet, A. (2023) · ACM Computing Surveys · doi:10.1145/3626234 · Cited by 168 · see citing works
Understanding accountability in algorithmic supply chains
Cobbe, J., Veale, M., & Singh, J. P. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594073 · Cited by 141 · see citing works
Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
Kiritchenko, S., Nejadgholi, I., & Fraser, K. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12590 · Cited by 118 · see citing works
Algorithmic fairness datasets: the story so far
Fabris, A., Messina, S., Silvello, G., & Susto, G. A. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00854-z · Cited by 128 · see citing works
A Survey of Privacy Risks and Mitigation Strategies in the Artificial Intelligence Life Cycle
Shahriar, S., Allana, S., Hazratifard, S. M., & Dara, R. (2023) · IEEE Access · doi:10.1109/access.2023.3287195 · Cited by 114 · see citing works
Xie, J., Sun, L., & Zhao, Y. F. (2025) · Engineering · doi:10.1016/j.eng.2024.04.024 · Cited by 112 · see citing works
Measuring algorithmically infused societies
Wagner, C., Strohmaier, M., Olteanu, A., Kıcıman, E., Contractor, N., & Eliassi-Rad, T. (2021) · Nature · doi:10.1038/s41586-021-03666-1 · Cited by 144 · see citing works
Self-citationModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI
Hummer, W., Muthusamy, V., Rausch, T., Dube, P., El Maghraoui, K., Murthi, A., & Oum, P. (2019) · 2019 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2019.00025 · Cited by 105 · see citing works
The Ethics of Algorithms: Key Problems and SolutionsPreprint
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3662302 · Cited by 101 · see citing works
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · see citing works
Anik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103 · see citing works
A Survey of Mental Modeling Techniques in Human–Robot Teaming
Tabrez, A., Luebbers, M. B., & Hayes, B. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00019-0 · Cited by 98 · see citing works
Explainable Embodied Agents Through Social Cues
Wallkötter, S., Tulli, S., Castellano, G., Paiva, A., & Chétouani, M. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3457188 · Cited by 100 · see citing works
Burr, C., & Leslie, D. (2021) · AI and Ethics · doi:10.1007/s43681-022-00178-0 · Cited by 107 · see citing works
Liao, Q. V., Subramonyam, H., Wang, J., & Vaughan, J. W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 97 · see citing works
Wan, Q., Hu, S., Zhang, Y., Wang, P., Wen, B., & Lu, Z. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637361 · Cited by 88 · see citing works
The Sanction of Authority: Promoting Public Trust in AI
Knowles, B., & Richards, J. T. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445890 · Cited by 86 · see citing works
Self-citationA Checklist to Combat Cognitive Biases in Crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86 · see citing works
Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., Choudhury, M. D., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 92 · see citing works
Human-centred artificial intelligence: a contextual morality perspective
van Berkel, N., Tag, B., Gonçalves, J., & Hosio, S. J. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1818828 · Cited by 83 · see citing works
Documenting Computer Vision Datasets
Miceli, M., Yang, T., Naudts, L., Schuessler, M., Serbanescu, D. A., & Hanna, A. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445880 · Cited by 94 · see citing works
Wang, A. Y., Wang, D., Drozdal, J., Muller, M. J., Park, S., Weisz, J. D., Liu, X., Wu, L., & Dugan, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3489465 · Cited by 82 · see citing works
Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82 · see citing works
A Neural Network-Based On-device Learning Anomaly Detector for Edge Devices
Tsukada, M., Kondo, M., & Matsutani, H. (2020) · IEEE Transactions on Computers · doi:10.1109/tc.2020.2973631 · Cited by 81 · see citing works
Public procurement of artificial intelligence systems: new risks and future proofing
Hickok, M. (2024) · AI & SOCIETY · doi:10.1007/s00146-022-01572-2 · Cited by 81 · see citing works
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIPreprint
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07487 · Cited by 81 · see citing works
How do we assess the trustworthiness of AI? Introducing the trustworthiness assessment model (TrAM)
Schlicker, N., Baum, K. L., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108671 · Cited by 87 · see citing works
Healthsheet: Development of a Transparency Artifact for Health Datasets
Rostamzadeh, N., Mincu, D., Roy, S., Smart, A., Wilcox, L. G., Pushkarna, M., Schrouff, J., Amironesei, R., Moorosi, N., & Heller, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533239 · Cited by 79 · see citing works
Petersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 83 · see citing works
Palladino, N. (2022) · Telecommunications Policy · doi:10.1016/j.telpol.2022.102479 · Cited by 49 · see citing works
Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing
Raji, I. D., Gebru, T., Mitchell, M., Buolamwini, J., Lee, J., & Denton, E. (2020) · arXiv · doi:10.48550/arxiv.2001.00964 · Cited by 69 · see citing works
FederatedTrust: A solution for trustworthy federated learning
Sánchez, P. M. S., Celdrán, A. H., Xie, N., Bovet, G., Pérez, G. M., & Stiller, B. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.10.013 · Cited by 73 · see citing works
Building Trustworthy AI Solutions: A Case for Practical Solutions for Small Businesses
Crockett, K. A., Colyer, E., Gerber, L., & Latham, A. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3137091 · Cited by 76 · see citing works
Satornino, C. B., Du, S., & Grewal, D. (2023) · Industrial Marketing Management · doi:10.1016/j.indmarman.2023.11.011 · Cited by 64 · see citing works
Heger, A., Marquis, L. B., Vorvoreanu, M., Wallach, H., & Vaughan, J. W. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555760 · Cited by 60 · see citing works
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58 · see citing works
Regulatory Aspects of the Use of Artificial Intelligence Medical Software
Zanca, F., Brusasco, C., Pesapane, F., Kwade, Z., Beckers, R., & Avanzo, M. (2022) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2022.06.012 · Cited by 59 · see citing works
Weidele, D. K. I., Weisz, J. D., Oduor, E., Muller, M. J., Andrés, J., Gray, A., & Wang, D. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377538 · Cited by 58 · see citing works
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 374 · see citing works
Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models
Park, S., Wang, A. Y., Kawas, B., Liao, Q. V., Piorkowski, D., & Danilevsky, M. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450637 · Cited by 61 · see citing works
Self-citationMaking It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI Auditability
Li, Y., & Goel, S. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10508-8 · Cited by 65 · see citing works
Behavioral Use Licensing for Responsible AI
Contractor, D., McDuff, D., Haines, J. K., Lee, J., Hines, C., Hecht, B., Vincent, N., & Li, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533143 · Cited by 54 · see citing works
AI Documentation: A path to accountability
Königstorfer, F., & Thalmann, S. (2022) · Journal of Responsible Technology · doi:10.1016/j.jrt.2022.100043 · Cited by 55 · see citing works
Choudhury, A. (2022) · JMIR Human Factors · doi:10.2196/35421 · Cited by 56 · see citing works
Zheng, C., Wang, D., Wang, A. Y., & Ma, X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517615 · Cited by 54 · see citing works
In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making ProcessesPreprint
Ashoori, M., & Weisz, J. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02675 · Cited by 54 · see citing works
A Transparency Index Framework for AI in Education
Chaudhry, M. A., Cukurova, M., & Luckin, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-11647-6_33 · Cited by 53 · see citing works
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 58 · see citing works
Ladu, L., Koch, C., Ashari, P. A., Blind, K., & Castka, P. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102564 · Cited by 63 · see citing works
Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581268 · Cited by 51 · see citing works
Agarwal, A., Agarwal, H., & Agarwal, N. (2022) · AI and Ethics · doi:10.1007/s43681-022-00147-7 · Cited by 49 · see citing works
Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Scharowski, N., Benk, M., Kühne, S. J., Wettstein, L., & Brühlmann, F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593994 · Cited by 53 · see citing works
Symphony: Composing Interactive Interfaces for Machine Learning
Bäuerle, A., Cabrera, Á. A., Hohman, F., Maher, M., Koski, D., Suau, X., Barik, T., & Moritz, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502102 · Cited by 50 · see citing works
Kiuchi, K., Otsu, K., & Hayashi, Y. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2286528 · Cited by 50 · see citing works
Li, J., Zhou, Y., Yao, J., & Liu, X. (2021) · Scientific Reports · doi:10.1038/s41598-021-92904-7 · Cited by 46 · see citing works
Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems
Windmann, A., Wittenberg, P., Schieseck, M., & Niggemann, O. (2024) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin58382.2024.10774364 · Cited by 58 · see citing works
Considerations for a more ethical approach to data in AI: On data representation and infrastructure
Baird, A., & Schuller, B. (2020) · Frontiers in Big Data · doi:10.3389/fdata.2020.00025 · Cited by 46 · see citing works
Workflow provenance in the lifecycle of scientific machine learning
Souza, R., Azevedo, L. G., Lourenço, V., Soares, E. F. D. S., Thiago, R. M., Brandão, R., Civitarese, D., Brazil, E. V., Moreno, M. F., Valduriez, P., et al. (2021) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6544 · Cited by 46 · see citing works
Experiences with Improving the Transparency of AI Models and Services
Hind, M., Houde, S., Martino, J., Mojsilovic, A., Piorkowski, D., Richards, J., & Varshney, K. R. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3383051 · Cited by 44 · see citing works
Self-citationData Hunches: Incorporating Personal Knowledge into Visualizations
Lin, H., Akbaba, D., Meyer, M., & Lex, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209451 · Cited by 43 · see citing works
Researching AI Legibility through Design
Lindley, J., Akmal, H. A., Pilling, F. L., & Coulton, P. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376792 · Cited by 45 · see citing works
Healthcare Voice AI Assistants: Factors Influencing Trust and Intention to Use
Zhan, X., Abdi, N., Seymour, W., & Such, J. (2024) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3637339 · Cited by 65 · see citing works
David, A., Yiğitcanlar, T., Desouza, K. C., Li, R. Y. M., Cheong, P. H., Mehmood, R., & Corchado, J. M. (2024) · Cities · doi:10.1016/j.cities.2024.105502 · Cited by 49 · see citing works
Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems
Lewis, G. A., Bellomo, S., & Özkaya, İ. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00028 · Cited by 41 · see citing works
What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 41 · see citing works
Machine Learning Explainability for External Stakeholders
Bhatt, U., Andrus, M., Weller, A., & Xiang, A. (2020) · arXiv · doi:10.48550/arxiv.2007.05408 · Cited by 41 · see citing works
AI Regulation Is (not) All You Need
Lucaj, L., van der Smagt, P., & Benbouzid, D. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594079 · Cited by 40 · see citing works
Berman, G., Goyal, N., & Madaio, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 45 · see citing works
Moore, S., Liao, Q. V., & Subramonyam, H. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581242 · Cited by 40 · see citing works
Cajueiro, D. O., & Celestino, V. R. R. (2025) · Journal of Economy and Technology · doi:10.1016/j.ject.2025.07.001 · Cited by 52 · see citing works
Tan, M., & Subramonyam, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642592 · Cited by 40 · see citing works
AI Consent Futures: A Case Study on Voice Data Collection with Clinicians
Wilcox, L. G., Brewer, R. N., & Díaz, F. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610107 · Cited by 37 · see citing works
Dataset Reuse: Toward Translating Principles to Practice
Koesten, L. M., Vougiouklis, P., Simperl, E., & Groth, P. (2020) · Patterns · doi:10.1016/j.patter.2020.100136 · Cited by 40 · see citing works
- Authors
- Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., Reimer, D., Richards, J., Tsay, J., & Varshney, K. R.
- Venue
- IBM Journal of Research and Development
- Publisher
- IBM
- Published
- 2019-07-01
- Volume
- 63
- Issue
- 4/5
- Pages
- 6:1-6:13
- ISSN
- 0018-8646 (ISSN-L)
- OpenAlex
- W2911599914
- Cited by
- 434
- References
- 44
- Citations by year
- 2026: 47 · 2025: 44 · 2024: 62 · 2023: 93 · 2022: 58 · 2021: 78 · 2020: 32 · 2019: 18 · 2018: 2
- FWCI
- 81.25
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IBM Journal of Research and Development (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- trust in AI, AI safety, AI security, AI provenance, algorithmic fairness, explainable AI, consumer trust in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
