Establishing Data Provenance for Responsible Artificial Intelligence Systems
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Data provenance, a record that describes the origins and processing of data, offers new promises in the increasingly important role of artificial intelligence (AI)-based systems in guiding human decision making. To avoid disastrous outcomes that can result from bias-laden AI systems, responsible AI builds on four important characteristics: fairness, accountability, transparency, and explainability. To stimulate further research on data provenance that enables responsible AI, this study outlines existing biases and discusses possible implementations of data provenance to mitigate them. We first review biases stemming from the data's origins and pre-processing. We then discuss the current state of practice, the challenges it presents, and corresponding recommendations to address them. We present a summary highlighting how our recommendations can help establish data provenance and thereby mitigate biases stemming from the data's origins and pre-processing to realize responsible AI-based systems. We conclude with a research agenda suggesting further research avenues.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 2×
“Evidence-based medicine rests on high standards of explainability of both algorithms and data, as medical decision making requires a sound understanding of the underlying disease mechanisms and treatments [88].”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 2×
“For example, data provenance is needed to develop a data information sheet [31] that provides details on the most important variables influencing an AI-based system’s recommendations.”
BackgroundThe practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246 · cited 1×
“With increased transparency, patients are better informed and are therefore more empowered to seek clarification on diagnoses or recommendations [41].”
BackgroundShortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
XAI—Explainable artificial intelligence
Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G.-Z. (2019) · Science Robotics · doi:10.1126/scirobotics.aay7120 · Cited by 2,053
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 1×
“such as incorrect patient treatment, exacerbated poverty [62], wrongful arrest [33], and unjust criminal sentencing [43].”
BackgroundShin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536 · cited 1×
“Every study and every measurement instrument, however, well designed, still generates some errors [72].”
BackgroundPotential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“In contrast to the majority of existing research, which has focused on biases resulting from algorithms (e.g., [28, 35]), we concentrate on the origins of the data and the data pre-processing rather than on the algorithm that uses the data as inputs.”
ResultWhy and Where: A Characterization of Data Provenance
Buneman, P., Khanna, S., & Wang-Chiew, T. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44503-x_20 · Cited by 1,334 · cited 2×
“The importance of provenance has long been recognized [14] in the pharmaceutical, food, and fashion industries.”
BackgroundTechniques for Interpretable Machine Learning
Du, M., Liu, N., & Hu, X. (2018) · arXiv · doi:10.48550/arxiv.1808.00033 · Cited by 1,261
George, G., Haas, M. R., & Pentland, A. (2014) · Academy of Management Journal · doi:10.5465/amj.2014.4002 · Cited by 1,198 · cited 3×
“In every data science project, sampling the right data to ensure represen-tativeness is important [33].”
BackgroundAmazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *
Dastin, J. (2022) · Ethics of Data and Analytics · doi:10.1201/9781003278290-44 · Cited by 1,165
The responsibility gap: Ascribing responsibility for the actions of learning automata
Matthias, A. (2004) · Ethics and Information Technology · doi:10.1007/s10676-004-3422-1 · Cited by 1,101 · cited 1×
“Current ethical and legal frameworks have not been designed for these situations, leading to a responsibility gap [63].”
BackgroundMachine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023 · cited 1×
“An easily ignorable fact is the non-stationary nature of the environment and the population from which all the input data of AI-based systems are generated [59].”
BackgroundFjeld, J., Achten, N., Hilligoss, H., Nagy, A., & Srikumar, M. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3518482 · Cited by 998 · cited 1×
“Scholars have identified different high-level ethical principles that should govern the development of AI systems [25, 48, 97].”
BackgroundWong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995
Toward reference models for requirements traceability
Ramesh, B., & Jarke, M. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.895989 · Cited by 912
Self-citationData governance: Organizing data for trustworthy Artificial Intelligence
Janssen, M., Brous, P., Estevez, E., Barbosa, L. S., & Janowski, T. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101493 · Cited by 811 · cited 3×
“Enhancing data auditing capability in an organization is another approach to establishing data provenance through data governance [44].”
BackgroundInfluentialRole of fairness, accountability, and transparency in algorithmic affordance
Shin, D., & Park, Y. J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.04.019 · Cited by 643 · cited 1×
“For example, regarding the relationship between transparency and explainability, some scholars suggest that explainability enhances the transparency of systems [82], whereas others argue that explainability is a sub-characteristic of transparency [83].”
BackgroundTransparency and trust in artificial intelligence systems
Schmidt, P., Biessmann, F., & Teubner, T. (2020) · Journal of Decision Systems · doi:10.1080/12460125.2020.1819094 · Cited by 440 · cited 1×
“For example, if the adoption and use of the system are concerns, explainability could be one way of increasing the transparency of a system to increase trust [80].”
BackgroundUnderspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398 · cited 1×
“, decision rules that work well based on the training data because of spurious phenomena [32]) of the AI system will lead to unreliable and unfair recommendations [20] that will undermine possible explanations.”
BackgroundArtificial intelligence and the implementation challenge
Shaw, J., Rudzicz, F., Jamieson, T., & Goldfarb, A. (2019) · Journal of Medical Internet Research · doi:10.2196/13659 · Cited by 389 · cited 1×
“The lack of this understanding undermines the implementation of AI in healthcare [81].”
BackgroundWieringa, M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372833 · Cited by 321
Meng, X.-L. (2018) · The Annals of Applied Statistics · doi:10.1214/18-aoas1161sf · Cited by 316 · cited 1×
“), limited attention has been given to data quality [66], in turn undermining the quality of recommendations generated using such data.”
BackgroundTutorial on practical tips of the most influential data preprocessing algorithms in data mining
García, S., Luengo, J., & Herrera, F. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2015.12.006 · Cited by 310
Deep learning predicts hip fracture using confounding patient and healthcare variables
Badgeley, M. A., Zech, J. R., Oakden-Rayner, L., Glicksberg, B. S., Liu, M., Gale, W., McConnell, M. V., Percha, B., Snyder, T. M., & Dudley, J. T. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0105-1 · Cited by 290
Privacy preserving EHR system using attribute-based infrastructure
Narayan, S., Gagné, M., & Safavi-Naini, R. (2010) · Proceedings of the 2010 ACM workshop on Cloud computing security workshop · doi:10.1145/1866835.1866845 · Cited by 257 · cited 1×
“Organizations also need to leverage some privacy-preserving approaches, such as federated learning, to allow the safe sharing of identifiable data or models across entities [69].”
BackgroundTechnology with No Human Responsibility?
Johnson, D. G. (2015) · Journal of Business Ethics · doi:10.1007/s10551-014-2180-1 · Cited by 195
Artificial intelligence explainability: the technical and ethical dimensions
McDermid, J. A., Jia, Y., Porter, Z., & Habli, I. (2021) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0363 · Cited by 136 · cited 2×
“Questions related to the fairness of responsible AI are often at the cross-section of research focused on novel technology and its ethical behavior [64].”
BackgroundThe Last Mile: Where Artificial Intelligence Meets Reality
Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/16323 · Cited by 102
Yee, K., Tantipongpipat, U., & Mishra, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479594 · Cited by 59 · cited 1×
“The case of Twitter’s cropping algorithm shows a conflict in speed and consistency versus the risk of making incorrect predictions [95].”
BackgroundAccuracy of the Epic Sepsis Prediction Model in a Regional Health System
Bennett, T., Russell, S., King, J., Schilling, L., Voong, C., Rogers, N., Adrian, B., Bruce, N., & Ghosh, D. (2019) · arXiv · doi:10.48550/arxiv.1902.07276 · Cited by 19 · cited 1×
“Transfer learning may have introduced biases, negatively affecting the performance of the sepsis prediction model using data from the University of Michigan Hospital [94] in contrast to data from the University of Colorado Hospital [10].”
Method
Cited by
(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
When Internet of Things Meets Metaverse: Convergence of Physical and Cyber Worlds
Li, K., Cui, Y., Li, W., Lv, T., Yuan, X., Li, S., Ni, W., Simsek, M., & Dressler, F. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2022.3232845 · Cited by 266
Explainable artificial intelligence for drug discovery and development: a comprehensive survey
Alizadehsani, R., Oyelere, S. S., Hussain, S., Jagatheesaperumal, S. K., Calixto, R. R., Rahouti, M., Roshanzamir, M., & De Albuquerque, V. H. C. (2024) · IEEE Access · doi:10.1109/access.2024.3373195 · Cited by 124
Alshehhi, K., Cheaitou, A., & Rashid, H. (2024) · Sustainability · doi:10.3390/su16177724 · Cited by 19
Understanding Data Quality in a Data-Driven Industry Context: Insights from the Fundamentals
Fu, Q., Nicholson, G. L., & Easton, J. M. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100729 · Cited by 17
Metadata
- Authors
- Werder, K., Ramesh, B., & Zhang, R. (.
- Venue
- ACM Transactions on Management Information Systems
- Published
- 2022-03-10
- Volume
- 13
- Issue
- 2
- Pages
- 1-23
- ISSN
- 2158-656X (ISSN-L)
- 2158-6578
- OpenAlex
- W4221090794
- Cited by
- 106
- References
- 38
- FWCI
- 22.17
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.