Responsible AI by Design in Practice
Open accessAbstract
Recently, a lot of attention has been given to undesired consequences of Artificial Intelligence (AI), such as unfair bias leading to discrimination, or the lack of explanations of the results of AI systems. There are several important questions to answer before AI can be deployed at scale in our businesses and societies. Most of these issues are being discussed by experts and the wider communities, and it seems there is broad consensus on where they come from. There is, however, less consensus on, and experience with how to practically deal with those issues in organizations that develop and use AI, both from a technical and organizational perspective. In this paper, we discuss the practical case of a large organization that is putting in place a company-wide methodology to minimize the risk of undesired consequences of AI. We hope that other organizations can learn from this and that our experience contributes to making the best of AI while minimizing its risks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Implementing AI principles requires a methodology that breaks down the process into different parts, such as that presented in [386].
“Implementing AI principles requires a methodology such as that presented in [386] that breaks down the process into different parts.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [386]
The EC Guidelines for Trustworthy AI are an example of end-to-end principles, while Telefonica's are more AI-specific.
“The EC Guidelines for Trustworthy AI are an example of end-to-end principles [390], while those of Telefonica (a large Spanish ICT company operating worldwide) are more AI-specific [386].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [386]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Data preprocessing techniques for classification without discrimination
Kamiran, F., & Calders, T. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0463-8 · Cited by 1,305 · see citing works
Samek, W., Wiegand, T., & Müller, K.-R. (2017) · arXiv · doi:10.48550/arxiv.1708.08296 · Cited by 1,080 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
Flagged first, then by citations
Research works printed but not linked: 27
- Anderson, A., Dodge, J., Sadarangani, A., Juozapaitis, Z., Newman, E., Irvine, J., ... & Burnett, M. (2019). Explaining Reinforcement Learning to Mere Mortals: An Empirical Study. arXiv preprint arXiv:1903.09708 . · arXiv
- Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), e0130140.
- Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., ... & Roli, F. (2013, September). Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases (pp. 387-402). Springer, Berlin, Heidelberg.
- Biggio, B., Pillai, I., Bulò, R., S., Ariu, D., Pelillo, M., & Roli, F. (2013, November). Is data clustering in adversarial settings secure?. In Proceedings of the 2013 ACM workshop on Artificial Intelligence and Security (pp. 87-98). ACM.
- Boden, M., Bryson, J., Caldwell, D., Dautenhahn, K., Edwards, L., Kember, S., Newman, P., Parry, V., Pegman, G., Rodden, T., Sorrell, T., Wallis, M., Whitby, B. & Winfield, A. (2017) Principles of robotics: regulating robots in the real world, Connection Science, 29:2, 124-129, DOI: 10.1080/09540091.2016.1271400 · DOI · Paper page
- Bryson, J., & Winfield, A.F. (2017). Standardizing Ethical Design for Artificial Intelligence and Autonomous Systems. IEEE Computer, 50, 116-119.
- Dignum, V. (2017). Responsible Autonomy. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI), AI and autonomy track. Pages 46984704. https://doi.org/10.24963/ijcai.2017/655. · DOI
- Dignum, V. (2018). Ethics in artificial intelligence: introduction to the special issue. Ethics and Information Technology, 20, 1-3. Springer, https://doi.org/10.1007/s10676-018-9450-z · DOI
- Doran, D., Schulz, S., & Besold, T. R. (2017). What does explainable AI really mean? A new conceptualization of perspectives. arXiv preprint arXiv:1710.00794. · arXiv
- Dwork, C., Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. In Foundations and Trends in Theoretical Computer Science, Vol. 9, Nos. 3-4 (2014) 211-407, DOI: 10.1561/0400000042 · DOI · Paper page
- Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., ... & Song, D. (2018). Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 16251634).
- Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018, October). Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) (pp. 80-89). IEEE.
- Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. · arXiv
- Gross-Brown, R., Ficek, M., Agundez, J.L., Dressler, P., & Laoutaris, N. (2015). Data Transparency Lab Kick Off Workshop (DTL 2014) Report. Computer Communication Review, 45, 44-48.
- Hind, M., Wei, D., Campbell, M., Codella, N. C., Dhurandhar, A., Mojsilović, A., ... & Varshney, K. R. (2019). TED: Teaching AI to Explain its Decisions. In AAAI/ACM conference on Artificial Intelligence, Ethics, and Society.
- Kingston, J. (2016). Artificial Intelligence and Legal Liability. SGAI Conf.
- Hind, M., Mehta, S., Mojsilovi, A. ́, Nair, R., Ramamurthy, K.N., Olteanu, A. & Varshney, K.R. (2018) Increasing Trust in AI Services through Supplier's Declarations of Conformity. arXiv preprint arXiv:1808.07261v1 · arXiv
- Diakopoulos, N. & Koliska, M. (2016): Algorithmic Transparency in the News Media, Digital Journalism, DOI: 10.1080/21670811.2016.1208053 · DOI · Paper page
- Papernot, N., Faghri, F., Carlini, N., Goodfellow, I., Feinman, R., Kurakin, A., ... & Matyasko, A. (2016). Technical report on the cleverhans v2. 1.0 adversarial examples library. arXiv preprint arXiv:1610.00768 . · arXiv
- Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Pappalardo, L., Ruggieri, S., & Turini, F. (June 2018). Open the Black Box Data-Driven Explanation of Black Box Decision Systems. 15 pages. arXiv preprint arXiv:1806.09936 · arXiv
- Pistono, F., & Yampolskiy, R.V. How to Create a Malevolent Artificial Intelligence. In proceedings of Ethics for Artificial Intelligence Workshop (AI-Ethics-2016). Pages 1-7. New York, NY. July 9 -- 15, 2016
- Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). Why should I trust you? Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135-1144). ACM.
- Speicher, T., Heidari, H., Grgic-Hlaca, N., P. Gummadi, K., Singla, A., Weller, A. & Zafar, M. (2018). A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices. 10.1145/3219819.3220046. · DOI · Paper page
- Stoyanovich, J., Abiteboul, S., & Miklau, G. (2016). Data Responsibly: Fairness, Neutrality and Transparency in Data Analysis. in Proc. 19th Inter- national Conference on Extending Database Technology (EDBT), March 15-18, 2016
- Theodorou, A., Wortham, R. H., & Bryson, J. J. (2017). Designing and implementing transparency for real time inspection of autonomous robots. Connection Science, 29(3), 230-241.
- Zemel, R., Wu, Y., Swersky, K., Pitassi, T. & Dwork, C. (2013). Learning fair representations. 30th International Conference on Machine Learning, ICML 2013. 1362-1370.
- Zhang, B., Lemoine, B. & Mitchell, M. (2018). Mitigating Unwanted Biases with Adversarial Learning. 335-340. 10.1145/3278721.3278779. · DOI · Paper page
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Self-citationde Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
AI Fairness in Data Management and Analytics: A Review on Challenges, Methodologies and Applications
Chen, P., Wu, L., & Wang, L. (2023) · Applied Sciences · doi:10.3390/app131810258 · Cited by 155 · see citing works
Towards intellectual freedom in an AI Ethics Global Community
Ebell, C., Baeza-Yates, R., Benjamins, R., Cai, H., Coeckelbergh, M., Duarte, T., Hickok, M., Jacquet, A., Kim, A., Krijger, J., et al. (2021) · AI and Ethics · doi:10.1007/s43681-021-00052-5 · Cited by 34 · see citing works
Self-citationA step toward building a unified framework for managing AI bias
Rana, S. A., Azizul, Z. H., & Awan, A. A. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1630 · Cited by 23 · see citing works
Ethical and Responsible AI in Education: Situated Ethics for Democratic Learning
Hummel, S. (2025) · Education Sciences · doi:10.3390/educsci15121594 · Cited by 21 · see citing works
- Authors
- Benjamins, R., Barbado, A., & Sierra, D.
- Affiliations
- Telefónica (Spain) (ES)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.12838
- OpenAlex
- W2995836228
- Cited by
- 38
- References
- 5
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 4 · 2023: 15 · 2022: 5 · 2021: 6 · 2020: 2 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- responsible AI, AI risk management, algorithmic bias, discrimination, explainability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
