The DOI matches its Crossref record; 42 of the 44 references it cites were found in a registry or an index.
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Open accessAbstract
Accuracy is an important concern for suppliers of artificial intelligence (AI) services, but considerations beyond accuracy, such as safety (which includes fairness and explainability), security, and provenance, are also critical elements to engender consumers’ trust in a service. Many industries use transparent, standardized, but often not legally required documents called supplier's declarations of conformity (SDoCs) to describe the lineage of a product along with the safety and performance testing it has undergone. SDoCs may be considered multidimensional fact sheets that capture and quantify various aspects of the product and its development to make it worthy of consumers’ trust. In this article, inspired by this practice, we propose FactSheets to help increase trust in AI services. We envision such documents to contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers. We suggest a comprehensive set of declaration items tailored to AI in the Appendix of this article.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“AI documentation is advocated to increase the transparency and facilitate understanding of AI [35,36,40,75,76,82], and is embodied in recently proposed features of AI Factsheets [8,81] and Model Cards [64].”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [8]
In the citing paper
“There is some work on fairness relating to AI services, though this tends to focus on either exposing or 'auditing' certain AI services [22,38,126,133], or devising tools, methods and interventions relevant for those building (rather than using) such services [12,96,106,128].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [12]
In the citing paper
“There appears a clear role for adopting and extending various documentation approaches, be they relating to data and models (such as datasheets [53], model cards [106] and factsheets [12]), their interactions and interconnections (e.g.decision provenance [139]), and broader, holistic socio-technical system reviews (e.g.reviewability [27] and traceability [86]).”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Market for "Lemons": Quality Uncertainty and the Market Mechanism
Akerlof, G. A. (1970) · The Quarterly Journal of Economics · doi:10.2307/1879431 · Cited by 23,016
Axford, N. (2017) · Routledge eBooks · doi:10.4324/9781315258324-16 · Cited by 11,329
Statsmodels: Econometric and Statistical Modeling with Python
Seabold, S., & Perktold, J. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-011 · Cited by 6,968
Another look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 2×
“…et al. proposed model cards to convey information that characterizes the evaluation of a machine learning model in a variety of conditions and disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information [11].”
BackgroundMethodInfluentialGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Self-citationLee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Self-citationBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763
Self-citationCertified Defenses against Adversarial ExamplesPreprint
Raghunathan, A., Steinhardt, J., & Liang, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09344 · Cited by 758
Li, X., Hess, T. J., & Valacich, J. S. (2008) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2008.01.001 · Cited by 687
Adversarial Robustness Toolbox v1.0.0
Nicolae, M.-I., Sinn, M., Tran, M. N., Buesser, B., Rawat, A., Wistuba, M., Zantedeschi, V., Baracaldo, N., Chen, B., Ludwig, H., et al. (2018) · arXiv · doi:10.48550/arxiv.1807.01069 · Cited by 594
Constitutional democracy and technology in the age of artificial intelligence
Nemitz, P. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0089 · Cited by 418
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334
Disparate Impact in Big Data PolicingPreprint
Selbst, A. D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2819182 · Cited by 237
On the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Varshney, K. R., & Alemzadeh, H. (2017) · Big Data · doi:10.1089/big.2016.0051 · Cited by 211 · cited 5×
“Safety, discussed in [3] and [5] as the minimization of both risk and epistemic uncertainty, will include explainability, algorithmic fairness, and robustness to dataset shift.”
BackgroundInfluentialSelf-citationCertifying the reliability of software
Currit, P. A., Dyer, M., & Mills, H. D. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312914 · Cited by 176
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
Wagstaff, K. (2012) · arXiv · doi:10.48550/arxiv.1206.4656 · Cited by 148
A Brief Review of the Belmont Report
Sims, J. (2010) · Dimensions of Critical Care Nursing · doi:10.1097/dcc.0b013e3181de9ec5 · Cited by 135
Principles of engineering safety: Risk and uncertainty reduction
Möller, N., & Hansson, O. (2007) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2007.03.031 · Cited by 134
Who is trustworthy? Predicting trustworthy intentions and behavior.
Levine, E. E., Bitterly, T. B., Cohen, T. R., & Schweitzer, M. E. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspi0000136 · Cited by 97
The Concepts of Risk and Safety
Möller, N. (2012) · Springer eBooks · doi:10.1007/978-94-007-1433-5_3 · Cited by 69
Algorithms for interpretable machine learning
Rudin, C. D. (2014) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2630823 · Cited by 61
Correction to 'Certifying the reliability of software' (Jan. 1986 3-11)
Curritt, P. A., Dyer, M., & Mills, H. D. (1989) · IEEE Transactions on Software Engineering · doi:10.1109/32.21765 · Cited by 60
Transparency by Conformity: A Field Experiment Evaluating Openness in Local Governments
ben‐Aaron, J., Denny, M., Desmarais, B. A., & Wallach, H. (2016) · Public Administration Review · doi:10.1111/puar.12596 · Cited by 50
(2018) · arXiv (Cornell University) · Cited by 41
Self-citationA software product certification model
Heck, P., Klabbers, M., & van Eekelen, M. (2009) · Software Quality Journal · doi:10.1007/s11219-009-9080-0 · Cited by 40
The Limits of Abstract Evaluation Metrics
Olteanu, A., Talamadupula, K., & Varshney, K. R. (2017) · ACM Web Science Conference (WebSci) · doi:10.1145/3091478.3098871 · Cited by 30
Self-citationTowards Composable Bias Rating of AI Services
Srivastava, B., & Rossi, F. (2018) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3278721.3278744 · Cited by 26
An Approach for Certifying Security in Software Components
Ghosh, A. K., & McGraw, G. E. (1998) · Cited by 22
Port, D. N., & Wilf, J. M. (2013) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.54 · Cited by 22
(2018) · Cited by 17
McAllister, L. K. (2014) · Michigan Journal of Environmental & Administrative Law · doi:10.36640/mjeal.3.2.harnessing · Cited by 5
Flagged first, then by citations
2 of the 44 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- varshney, 0, Data science of the people, for the people, by the people: A viewpoint on an emerging dichotomy, Proc Data for Good Exchange Conf
Cited by
Sorted by citations
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957
“This will increase transparency and accountability by providing detailed and continuous information on the AI tools during their lifetime to clinicians, healthcare organisations, citizens and patients, AI developers, and relevant authorities.”
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
Metadata
- Authors
- Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., Reimer, D., Richards, J., Tsay, J., & Varshney, K. R.
- Venue
- IBM Journal of Research and Development
- Publisher
- IBM
- Published
- 2019-07-01
- Volume
- 63
- Issue
- 4/5
- Pages
- 6:1-6:13
- ISSN
- 0018-8646 (ISSN-L)
- OpenAlex
- W2911599914
- Cited by
- 434
- References
- 44
- Citations by year
- 2026: 47 · 2025: 44 · 2024: 62 · 2023: 93 · 2022: 58 · 2021: 78 · 2020: 32 · 2019: 18
- FWCI
- 81.25
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IBM Journal of Research and Development (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- trust in AI, AI safety, AI security, AI provenance, algorithmic fairness, explainable AI, consumer trust in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
