What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability
Open accessAbstract
As research on algorithms and their impact proliferates, so do calls for scrutiny/accountability of algorithms. A systematic review of the work that has been done in the field of 'algorithmic accountability' has so far been lacking. This contribution puts forth such a systematic review, following the PRISMA statement. 242 English articles from the period 2008 up to and including 2018 were collected and extracted from Web of Science and SCOPUS, using a recursive query design coupled with computational methods. The 242 articles were prioritized and ordered using affinity mapping, resulting in 93 'core articles' which are presented in this contribution. The recursive search strategy made it possible to look beyond the term 'algorithmic accountability'. That is, the query also included terms closely connected to the theme (e.g. ethics and AI, regulation of algorithms). This approach allows for a perspective not just from critical algorithm studies, but an interdisciplinary overview drawing on material from data studies to law, and from computer science to governance studies. To structure the material, Bovens's widely accepted definition of accountability serves as a focal point. The material is analyzed on the five points Bovens identified as integral to accountability: its arguments on (1) the actor, (2) the forum, (3) the relationship between the two, (3) the content and criteria of the account, and finally (5) the consequences which may result from the account. The review makes three contributions. First, an integration of accountability theory in the algorithmic accountability discussion. Second, a cross-sectoral overview of the that same discussion viewed in light of accountability theory which pays extra attention to accountability risks in algorithmic systems. Lastly, it provides a definition of algorithmic accountability based on accountability theory and algorithmic accountability literature.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Explanations or other approaches focused on how the model itself works or has reached a particular outcome may miss much of what is important.
“Though explanations can assist some specific concerns, like model engineering [7], explanations or other approaches focused on how the model itself works or has reached a particular outcome may miss much of what is important [75].”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
Experimental — accuracy not yet measuredMeaningful accountability requires a view of the whole socio-technical process, from commissioning of the system; through design of the model, selection of training data, and training and testing procedures; to making individual decisions; and on to the effects of those decisions and any subsequent investigations.
“Meaningful accountability thus requires a view of the whole socio-technical process [2,43,66,75], from commissioning of the system; through design of the model, selection of training data, and training and testing procedures; to making individual decisions; and on to the effects of those decisions and any subsequent investigations.”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
A meaningful account cannot be given unless the forum can understand and critically engage with the subject matter.
“Crucially, a meaningful account cannot be given unless the forum can understand and critically engage with the subject matter [37,75].”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
Accountability is highly contextual-different actors will likely be accountable for different aspects of the ADM process depending on what is to be accounted for, and the kinds, levels, and formats of information needed for a relevant and appropriate account will depend heavily on the forum to whom it is owed.
“Although accountability can be thought of abstractly in general terms, in practice it is highly contextual -different actors will likely be accountable for different aspects of the ADM process depending on what is to be accounted for, and the kinds, levels, and formats of information needed for a relevant and appropriate account will depend heavily on the forum to whom it is owed [8,75].”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
For accountability to be meaningful, information given by the actor should support effective deliberation and discussion by the forum and the imposition of any consequences by the forum on the actor.
“For accountability to be meaningful, information given by the actor should support effective deliberation and discussion by the forum and the imposition of any consequences by the forum on the actor [75] (such as legal remedies or interventions to correct malfunction).”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
Others have discussed ML as a process involving multiple steps.
“Others have also discussed ML as a process involving multiple steps.”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
Accountability of actors is likely to be dispersed, with obligations on multiple actors to explain or justify actions or decisions relating to aspects of the process for which they are responsible.
“Accountability of actors is likely to be dispersed, with obligations on multiple actors to explain or justify actions or decisions relating to aspects of the process for which they are responsible [75].”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
Wieringa applies the Software Development Life Cycle model to divide the process into 'ex ante', 'in media res', and 'ex post' stages.
“Wieringa applies the Software Development Life Cycle model to divide the process into 'ex ante', 'in media res', and 'ex post' stages [75].”
From Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems (2021) · cites this paper as [75]
After these effects were revealed, the backlash led Dutch Prime Minister Mark Rutte to resign.
“Once these effects were revealed, the backlash prompted Mark Rutte, Prime Minister of the Netherlands, to resign.”
From Towards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard (2022) · cites this paper as [60]
Many respondents expressed concern that AI advances in detecting sexuality, criminality, or terrorism could be exported to India and harm minorities.
“Many respondents expressed concern over AI advances in detecting sexuality, criminality, or terrorism (e.g., [187,212]) potentially being exported to India and harming minorities.”
From Re-imagining Algorithmic Fairness in India and Beyond (2021) · cites this paper as [212]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · see citing works
(2014) · Climate Change 2013 – The Physical Science Basis · doi:10.1017/cbo9781107415324.004 · Cited by 23,880 · see citing works
Fast unfolding of communities in large networks
Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/2008/10/p10008 · Cited by 21,424 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning
Barad, K. (2006) · doi:10.2307/j.ctv12101zq · Cited by 10,472 · see citing works
Street-Level Bureaucracy: Dilemmas of the Individual in Public Services
Lipsky, M. (1981) · Michigan Law Review · doi:10.2307/1288305 · Cited by 8,981 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
The Body Multiple: Ontology in Medical Practice
Mol, A. (2002) · doi:10.1215/9780822384151 · Cited by 4,381 · see citing works
The ethics of algorithms: Mapping the debate
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016) · Big Data & Society · doi:10.1177/2053951716679679 · Cited by 3,432 · cited 2× · see citing works
“on different levels of organization, making a single actor hard to pinpoint [106].”
BackgroundAutomating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor
Lebovits, H. (2018) · Public Integrity · doi:10.1080/10999922.2018.1511671 · Cited by 3,318 · see citing works
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · cited 1× · see citing works
“27 academic articles [8, 9, 17, 20, 22, 28, 35, 48, 52, 58, 61, 65, 86, 90, 93, 98, 106, 110, 111, 120, 124, 135, 139–141, 143, 144] which were found to be relevant to the topic prior to the start of the systematic review were assessed for their keywords.”
Jacomy, M., Venturini, T., Heymann, S., & Bastian, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098679 · Cited by 2,909 · see citing works
Analysing and Assessing Accountability: A Conceptual Framework <sup>1</sup>
Bovens, M. (2007) · European Law Journal · doi:10.1111/j.1468-0386.2007.00378.x · Cited by 2,667 · see citing works
Ananny, M., & Crawford, K. (2018) · New Media & Society · doi:10.1177/1461444816676645 · Cited by 1,836 · see citing works
Two Concepts of Accountability: Accountability as a Virtue and as a Mechanism
Bovens, M. (2010) · West European Politics · doi:10.1080/01402382.2010.486119 · Cited by 1,019 · see citing works
Algorithms as culture: Some tactics for the ethnography of algorithmic systems
Seaver, N. (2017) · Big Data & Society · doi:10.1177/2053951717738104 · Cited by 1,010 · see citing works
Bovens, M., & Zouridis, S. (2002) · Public Administration Review · doi:10.1111/0033-3352.00168 · Cited by 964 · see citing works
‘Hypernudge’: big Data as a mode of regulation by design
Yeung, K. (2017) · Information, Communication & Society · doi:10.1080/1369118x.2016.1186713 · Cited by 868 · see citing works
Algorithmic regulation: A critical interrogation
Yeung, K. (2018) · Regulation & Governance · doi:10.1111/rego.12158 · Cited by 842 · see citing works
Accountability in algorithmic decision making
Diakopoulos, N. (2016) · Communications of the ACM · doi:10.1145/2844110 · Cited by 763 · see citing works
Ethical Implications and Accountability of Algorithms
Martin, K. (2019) · Journal of Business Ethics · doi:10.1007/s10551-018-3921-3 · Cited by 757 · see citing works
Algorithmic Accountability: Journalistic investigation of computational power structures
Diakopoulos, N. (2015) · Digital Journalism · doi:10.1080/21670811.2014.976411 · Cited by 733 · see citing works
Artificial Intelligence and the ‘Good Society’: the US, EU, and UK approach
Cath, C., Wachter, S., Mittelstadt, B., Taddeo, M., & Floridi, L. (2017) · Science and Engineering Ethics · doi:10.1007/s11948-017-9901-7 · Cited by 733 · see citing works
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · see citing works
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · see citing works
DOES PUBLIC ACCOUNTABILITY WORK? AN ASSESSMENT TOOL
Bovens, M., Schillemans, T., & Hart, P. ‘. (2008) · Public Administration · doi:10.1111/j.1467-9299.2008.00716.x · Cited by 664 · see citing works
New Materialism: Interviews & Cartographies
Dolphijn, R., & van der Tuin, I. (2012) · Cited by 660 · see citing works
Bias in algorithmic filtering and personalization
Bozdag, E. (2013) · Ethics and Information Technology · doi:10.1007/s10676-013-9321-6 · Cited by 649 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Governance by algorithms: reality construction by algorithmic selection on the Internet
Just, N., & Latzer, M. (2016) · Media Culture & Society · doi:10.1177/0163443716643157 · Cited by 616 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any ClassifierPreprint
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04938 · Cited by 587 · see citing works
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
Maior, H. A., Wilson, M. L., Locke, C., & Swann, D. (2018) · doi:10.1145/3173574 · Cited by 570 · see citing works
Society-in-the-loop: programming the algorithmic social contract
Rahwan, I. (2018) · Ethics and Information Technology · doi:10.1007/s10676-017-9430-8 · Cited by 553 · see citing works
The threat of algocracy: reality, resistance and accommodation
Danaher, J. (2016) · Philosophy & Technology · doi:10.1007/s13347-015-0211-1 · Cited by 519 · see citing works
How the Machine 'Thinks:' Understanding Opacity in Machine Learning AlgorithmsPreprint
Burrell, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2660674 · Cited by 511 · see citing works
What is a flag for? Social media reporting tools and the vocabulary of complaint
Crawford, K., & Gillespie, T. (2016) · New Media & Society · doi:10.1177/1461444814543163 · Cited by 496 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · see citing works
Algorithmic Accountability and Public Reason
Binns, R. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0263-5 · Cited by 457 · cited 1× · see citing works
“27 academic articles [8, 9, 17, 20, 22, 28, 35, 48, 52, 58, 61, 65, 86, 90, 93, 98, 106, 110, 111, 120, 124, 135, 139–141, 143, 144] which were found to be relevant to the topic prior to the start of the systematic review were assessed for their keywords.”
A Layered Model for AI Governance
Gasser, U., & Almeida, V. A. F. (2017) · IEEE Internet Computing · doi:10.1109/mic.2017.4180835 · Cited by 431 · see citing works
Zarsky, T. (2016) · Science, Technology, & Human Values · doi:10.1177/0162243915605575 · Cited by 422 · see citing works
Algorithmic Prediction in Policing: Assumptions, Evaluation, and Accountability
Moses, L. B., & Chan, J. B. L. (2016) · Policing & Society · doi:10.1080/10439463.2016.1253695 · Cited by 416 · see citing works
ACM code of ethics and professional conduct
Anderson, R. E. (1992) · Communications of the ACM · doi:10.1145/129875.129885 · Cited by 404 · see citing works
Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness
Ananny, M. (2016) · Science, Technology, & Human Values · doi:10.1177/0162243915606523 · Cited by 395 · see citing works
Gudiño, J., Grandi, U., & Hidalgo, C. A. (2019) · doi:10.1098/rsta · Cited by 393 · see citing works
Ziewitz, M. (2015) · Science Technology & Human Values · doi:10.1177/0162243915608948 · Cited by 391 · see citing works
Incorporating Ethics into Artificial Intelligence
Etzioni, A., & Etzioni, O. (2017) · The Journal of Ethics · doi:10.1007/s10892-017-9252-2 · Cited by 388 · see citing works
Ethics and Privacy in AI and Big Data: Implementing Responsible Research and Innovation
Stahl, B. C., & Wright, D. (2018) · IEEE Security & Privacy · doi:10.1109/msp.2018.2701164 · Cited by 374 · see citing works
Algorithmic governance: Developing a research agenda through the power of collective intelligence
Danaher, J., Hogan, M. J., Noone, C., Kennedy, R., Behan, A., De Paor, A., Felzmann, H., Haklay, M., Khoo, S.-M., Morison, J., et al. (2017) · Big Data & Society · doi:10.1177/2053951717726554 · Cited by 369 · cited 1× · see citing works
“Thus, one aspect of an account of algorithms should be the measure of human involvement [37, 40, 41, 46, 62, 89].”
BackgroundThe challenges and limits of big data algorithms in technocratic governance
Janssen, M., & Kuk, G. (2016) · Government Information Quarterly · doi:10.1016/j.giq.2016.08.011 · Cited by 363 · see citing works
Algorithms, Governance, and Governmentality: On Governing Academic Writing
Introna, L. D. (2016) · Science, Technology, & Human Values · doi:10.1177/0162243915587360 · Cited by 339 · see citing works
Can an Algorithm be Agonistic? Ten Scenes from Life in Calculated Publics
Crawford, K. (2015) · Science Technology & Human Values · doi:10.1177/0162243915589635 · Cited by 320 · see citing works
Standardizing Ethical Design for Artificial Intelligence and Autonomous Systems
Bryson, J. J., & Winfield, A. F. T. (2017) · Computer · doi:10.1109/mc.2017.154 · Cited by 298 · see citing works
The Accountability Cube: Measuring Accountability
Brandsma, G. J., & Schillemans, T. (2013) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/mus034 · Cited by 278 · see citing works
Transparent to whom? No algorithmic accountability without a critical audience
Kemper, J., & Kolkman, D. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1477967 · Cited by 260 · cited 5× · see citing works
“Kemper and Kolkman [86] argue that one cannot give account without the audience understanding the subject matter and being able to engage with the material in a critical way.”
BackgroundInfluentialde Laat, P. B. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0293-z · Cited by 251 · see citing works
Isomorphism through algorithms: Institutional dependencies in the case of Facebook
Caplan, R., & Boyd, D. (2018) · Big Data & Society · doi:10.1177/2053951718757253 · Cited by 231 · see citing works
ETHICS AS AN ESCAPE FROM REGULATION.: FROM “ETHICS-WASHING” TO ETHICS-SHOPPING?
WAGNER, B. (2018) · BEING PROFILED · doi:10.2307/j.ctvhrd092.18 · Cited by 228 · see citing works
Building Ethics into Artificial Intelligence
Yu, H., Shen, Z., Miao, C., Leung, C., Lesser, V. R., & Yang, Q. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/779 · Cited by 210 · see citing works
A Review of Future and Ethical Perspectives of Robotics and AI
Tørresen, J. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2017.00075 · Cited by 180 · see citing works
Djonov, E. (2010) · Visual Communication · Cited by 174 · see citing works
Andrews, L. (2019) · Public Administration · doi:10.1111/padm.12534 · Cited by 172 · see citing works
Is there an ethics of algorithms?
Kraemer, F., van Overveld, K., & Peterson, M. (2010) · Ethics and Information Technology · doi:10.1007/s10676-010-9233-7 · Cited by 170 · see citing works
Williamson, B. (2014) · Learning Media and Technology · doi:10.1080/17439884.2014.924527 · Cited by 167 · see citing works
DOES THE BUSINESS JUDGMENT RULE HELP PROMOTE CORPORATE SOCIAL RESPONSIBILITY
Neri-Castracane, G. (2015) · Archive ouverte UNIGE (University of Geneva) · doi:10.3868/s050-004-015-0003-8 · Cited by 163 · see citing works
Marda, V. (2018) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2018.0087 · Cited by 161 · see citing works
Social choice ethics in artificial intelligence
Baum, S. D. (2020) · AI & SOCIETY · doi:10.1007/s00146-017-0760-1 · Cited by 158 · see citing works
Explanation and trust: what to tell the user in security and AI?
Pieters, W. (2011) · Ethics and Information Technology · doi:10.1007/s10676-010-9253-3 · Cited by 153 · see citing works
Nissenbaum, H. F. (1994) · Communications of the ACM · doi:10.1145/175222.175228 · Cited by 151 · see citing works
Governance of algorithms: options and limitations
Saurwein, F., Just, N., & Latzer, M. (2015) · Info · doi:10.1108/info-05-2015-0025 · Cited by 144 · see citing works
Diversity in Big Data: A Review
Drosou, M., Jagadish, H. V., Pitoura, E., & Stoyanovich, J. (2017) · Big Data · doi:10.1089/big.2016.0054 · Cited by 140 · see citing works
Bearing Account-able Witness to the Ethical Algorithmic System
Neyland, D. (2015) · Science Technology & Human Values · doi:10.1177/0162243915598056 · Cited by 127 · see citing works
Algorithmic IF … THEN rules and the conditions and consequences of power
Neyland, D., & Möllers, N. (2017) · Information, Communication & Society · doi:10.1080/1369118x.2016.1156141 · Cited by 108 · see citing works
Are explanations always important?: a study of deployed, low-cost intelligent interactive systems
Bunt, A., Lount, M., & Lauzon, C. (2012) · Proceedings of the 2012 ACM international conference on Intelligent User Interfaces · doi:10.1145/2166966.2166996 · Cited by 106 · see citing works
The “big red button” is too late: an alternative model for the ethical evaluation of AI systems
Arnold, T., & Scheutz, M. (2018) · Ethics and Information Technology · doi:10.1007/s10676-018-9447-7 · Cited by 94 · see citing works
Accountability for the use of algorithms in a big data environment
Vedder, A., & Naudts, L. (2017) · International Review of Law Computers & Technology · doi:10.1080/13600869.2017.1298547 · Cited by 86 · see citing works
Lustig, C., Pine, K., Nardi, B. A., Irani, L. C., Lee, M. K., Nafus, D., & Sandvig, C. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2886426 · Cited by 83 · see citing works
Dixon‐Román, E. J. (2016) · Culture Studies ↔ Critical Methodologies · doi:10.1177/1532708616655769 · Cited by 77 · see citing works
Broeders, D., Schrijvers, E., van der Sloot, B., Brakel, R. E. V., de Hoog, J., & Ballin, E. M. H. H. (2017) · Computer law & security review · doi:10.1016/j.clsr.2017.03.002 · Cited by 73 · see citing works
Chatila, R., Firth-Butterflied, K., Havens, J. C., & Karachalios, K. (2017) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2017.2670225 · Cited by 72 · see citing works
People’s Councils for Ethical Machine Learning
McQuillan, D. (2018) · Social Media + Society · doi:10.1177/2056305118768303 · Cited by 63 · see citing works
Opening the government’s black boxes: freedom of information and algorithmic accountability
Fink, K. (2018) · Information, Communication & Society · doi:10.1080/1369118x.2017.1330418 · Cited by 61 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · doi:10.31235/osf.io/8kvf4 · Cited by 60 · see citing works
Operating anew: Queering GIS with good enough software
Gieseking, J. J. (2017) · Canadian Geographies / Géographies canadiennes · doi:10.1111/cag.12397 · Cited by 58 · see citing works
Of algorithms, apps and advice: Digital social policy and service delivery
Henman, P. (2019) · Journal of Asian Public Policy · doi:10.1080/17516234.2018.1495885 · Cited by 53 · see citing works
Enslaving the Algorithm: From a 'Right to an Explanation' to a 'Right to Better Decisions'?
Edwards, L., & Veale, M. (2018) · Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) · Cited by 53 · see citing works
Doneda, D. C. M., & Almeida, V. A. F. (2016) · IEEE Internet Computing · doi:10.1109/mic.2016.79 · Cited by 50 · see citing works
Offenhuber, D. (2017) · The MIT Press eBooks · doi:10.7551/mitpress/10453.001.0001 · Cited by 42 · see citing works
What Can Political Philosophy Teach Us about Algorithmic Fairness?
Binns, R. (2018) · IEEE Security & Privacy · doi:10.1109/msp.2018.2701147 · Cited by 40 · see citing works
Plausible Cause: Explanatory Standards in the Age of Powerful MachinesPreprint
Brennan-Marquez, K. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2827733 · Cited by 35 · see citing works
Diagnostics techniques and dosimetric evaluations for environmental radioactivity investigations
Caridi, F., D’Agostino, M., Belvedere, A., et al. (2016) · Journal of Instrumentation · doi:10.1088/1748-0221/11/10/c10012 · Cited by 33 · see citing works
Waste Is Information: Infrastructure Legibility and Governance
Offenhuber, D. (2017) · Cited by 31 · see citing works
Writing a Moral Code: Algorithms for Ethical Reasoning by Humans and Machines
McGrath, J. F., & Gupta, A. K. (2018) · Religions · doi:10.3390/rel9080240 · Cited by 25 · see citing works
Data science ethics in government
Drew, C. (2016) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2016.0119 · Cited by 24 · see citing works
Crowd-assisted Search for Price Discrimination in E-Commerce: First resultsPreprint
Mikians, J., Gyarmati, L., Erramilli, V., & Nikolaos Laoutaris, B. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1307.4531 · Cited by 24 · see citing works
Rethinking algorithmic regulation
Medina, E. M. (2015) · Kybernetes · doi:10.1108/k-02-2015-0052 · Cited by 23 · see citing works
A Comprehensive Ethical Framework for AI Entities: Foundations
Dameski, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-97676-1_5 · Cited by 18 · see citing works
Emerging trends: I did it, I did it, I did it, but. . .
Church, K. (2017) · Natural Language Engineering · doi:10.1017/s1351324917000067 · Cited by 16 · see citing works
Can An Algorithm Be Disturbed?: Machine Learning, Intrinsic Criticism, and the Digital Humanities
Dobson, J. E. (2015) · College literature · doi:10.1353/lit.2015.0037 · Cited by 15 · see citing works
Minimizing bias in computer systems
Friedman, B., Brok, E., Roth, S. K., & Thomas, J. C. (1996) · ACM SIGCHI Bulletin · doi:10.1145/249170.249184 · Cited by 14 · see citing works
Big Data, Algorithmic Regulation, and the History of the Cybersyn Project in Chile, 1971–1973
Loeber, K. (2018) · Social Sciences · doi:10.3390/socsci7040065 · Cited by 14 · see citing works
Accountability in Algorithmic Decision-making: A view from computational journalism
Diakopoulos, N. (2015) · Queue · doi:10.1145/2857274.2886105 · Cited by 13 · see citing works
Pauleen, D. J., Rooney, D., & Intezari, A. H. (2016) · Social Epistemology · doi:10.1080/02691728.2016.1249436 · Cited by 12 · see citing works
Ethical Dilemma of Artificial Intelligence and its Research Progress
Ma, L., Zhang, Z., & Zhang, N. (2018) · IOP Conference Series Materials Science and Engineering · doi:10.1088/1757-899x/392/6/062188 · Cited by 12 · see citing works
Algorithmic regulation and the global default: Shifting norms in Internet technology
Wagner, B. (2016) · Etikk i praksis - Nordic Journal of Applied Ethics · doi:10.5324/eip.v10i1.1961 · Cited by 10 · see citing works
List of predatory journals and publishers
Shrestha, J. (2020) · E-LIS Repository (University of Naples Federico II) · doi:10.5281/zenodo.4280714 · Cited by 7 · see citing works
Enslaving the Algorithm: From a Right to an Explanationn to a Right to Better Decisionss?Preprint
Edwards, L., & Veale, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3052831 · Cited by 5 · see citing works
Spotnitz, R. M. (1998) · Computers & Chemical Engineering · doi:10.1016/s0098-1354(98)00029-5 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 12
- [6]6. Madeleine Akrich and Bruno Latour. 1992. A summary of a convenient vo- cabulary for the semiotics of human and nonhuman assemblies. In Shaping technology / building society: studies in sociotechnical change, Wiebe E. Bijker and John Law (Eds.). MIT Press, Cambridge/London, Chapter 9, 259-264.
- [8]8. Mike Ananny. 2016. Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness. Science Technology and Human Values 41, 1 (2016), 93-117. https://doi.org/10.1177/0162243915606523 arXiv:arXiv:1011.1669v3 · DOI · Paper page
- [14]14. Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy. 2009. Gephi: an open source software for exploring and manipulating networks. In Proceedings of the Third International ICWSM Conference. http://www.aaai.org/ocs/index. php/ICWSM/09/paper/download/154/1009
- [24]24. Mark Bovens. 2007. Analysing and Assessing Accountability: A Conceptual Framework. European Law Journal 13, 4 (2007), 447-468. https://doi.org/10. 1111/j.1468-0386.2007.00378.x arXiv:1468-0386
- [54]54. Nicholas Diakopoulos. 2015. Algorithmic Accountability: Journalistic investiga- tion of computational power structures. Digital Journalism 3, 3 (2015), 398-415. https://doi.org/10.1080/21670811.2014.976411 · DOI · Paper page
- [56]56. Ezekiel Dixon-Román. 2016. Algo-Ritmo: More-Than-Human Performative Acts and the Racializing Assemblages of Algorithmic Architectures. Cultural Studies <-> Critical Methodologies 16, 5 (2016), 482-490. https://doi.org/10.1177/ 1532708616655769
- [65]65. European Union. 2016. Regulation 2016/679 of the European parliament and the Council of the European Union. arXiv:arXiv:1011.1669v3 · arXiv
- [80]80. IEEE. [n. d.]. IEEE Code of Ethics. https://www.ieee.org/about/corporate/ governance/p7-8.html
- [86]86. Christian Katzenbach. 2012. Technologies as Institutions: Rethinking the Role of Technology in Media Governance Constellations. In Trends in Communi- cation Policy Research, New Theories, Methods & Subjects, Manuel Puppis and Natascha Just (Eds.). Intellect Books, Bristol, 117-138. https://doi.org/10.1017/ CBO9781107415324.004 arXiv:arXiv:1011.1669v3 · arXiv
- [90]90. Utku Köse. 2017. Are We Safe Enough in the Future of Artificial Intelligence? A Discussion on Machine Ethics and Artificial Intelligence Safety. In Scientific Methods in Academic Research and Teaching International Conference. 184-197.
- [92]92. Joshua A. Kroll, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsyl- vania Law Review 165 (2017), 633-705. https://doi.org/10.3868/s050-004-015- 0003-8 arXiv:arXiv:1011.1669v3 · DOI
- [120]120. Craig Plain. 2007. Build an Affinity for K-J Method. Quality Progress 40, 3 (2007), 88.
Cited by
Sorted by citations
Towards transparency by design for artificial intelligence
Felzmann, H., Fosch-Villaronga, E., Lutz, C., & Tamò-Larrieux, A. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00276-4 · Cited by 588 · see citing works
Hutchinson, B., Smart, A., Hanna, A., Denton, R., Greer, C., Kjartansson, O., Barnes, P., & Mitchell, M. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445918 · Cited by 279 · see citing works
Re-imagining Algorithmic Fairness in India and Beyond
Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., & Prabhakaran, V. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445896 · Cited by 228 · see citing works
“made answerable to a forum: information-sharing, deliberation and discussion, and the imposition of consequences [214].”
BackgroundAlgorithmic Impact Assessments and Accountability: The Co-construction of Impacts
Metcalf, J., Moss, E., Watkins, E. A., Singh, R., & Elish, M. C. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445935 · Cited by 195 · see citing works
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193 · see citing works
Singhal, A., Neveditsin, N., Tanveer, H., & Mago, V. (2024) · JMIR Medical Informatics · doi:10.2196/50048 · Cited by 172 · see citing works
Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
Cobbe, J., Lee, M. S. A., & Singh, J. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445921 · Cited by 150 · see citing works
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149 · see citing works
“” And in a systematic review on algorithmic accountability, Wieringa [196] defines algorithms as “basically instructions fed to a computer”.”
BackgroundInfluentialThe Sanction of Authority: Promoting Public Trust in AI
Knowles, B., & Richards, J. T. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445890 · Cited by 86 · see citing works
Towards Accountability in the Use of Artificial Intelligence for Public Administrations
Loi, M., & Spielkamp, M. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462631 · Cited by 81 · see citing works
Establishing and evaluating trustworthy AI: overview and research challenges
Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., et al. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1467222 · Cited by 59 · see citing works
Lewicki, K., Lee, M. S. A., Cobbe, J., & Singh, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581463 · Cited by 30 · see citing works
Towards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard
Kuźniacki, B., Almada, M., Tyliński, K., Górski, L., Winogradska, B., & Zeldenrust, R. (2022) · World Tax Journal · doi:10.59403/2yhh9pa · Cited by 31 · see citing works
Should Users Trust Advanced AI Assistants? Justified Trust As a Function of Competence and Alignment
Manzini, A., Keeling, G., Marchal, N., McKee, K. R., Rieser, V., & Gabriel, I. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658964 · Cited by 28 · see citing works
“This need becomes particularly compelling if we consider that misplaced trust in AI assistants or their developers may impact not just the individual user but also society at large, as a result of competitive scenarios between different AI assistants (see Section 6.2), or due to widening…”
BackgroundInfluentialMonitoring AI Services for Misuse
Javadi, S. A., Norval, C., Cloete, R., & Singh, J. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462566 · Cited by 21 · see citing works
Taking Algorithms to Courts: A Relational Approach to Algorithmic Accountability
Metcalf, J., Singh, R., Moss, E., Tafesse, E., & Watkins, E. A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594092 · Cited by 21 · see citing works
Hirvonen, H. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01731-z · Cited by 19 · see citing works
Accountability in Government Use of <scp>AI</scp> : Citizen Concerns and Preferences
Feng, N., & Chandra, Y. (2025) · Public Administration · doi:10.1111/padm.70030 · Cited by 18 · see citing works
Kuai, J., Brantner, C., Karlsson, M., Van Couvering, E., & Romano, S. (2026) · New Media & Society · doi:10.1177/14614448251321162 · Cited by 16 · see citing works
“These performance discrepancies across languages suggest inequality in technology deployment (Bender et al., 2021; Helm et al., 2024) and high-light challenges in establishing accountability mechanisms across different contexts (Wieringa, 2020).”
InfluentialAgha, Z., Park, J., Wan, R., Ali, N. S., Wang, Y., Difranzo, D., Badillo-Urquiola, K., & Wisniewski, P. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642313 · Cited by 13 · see citing works
Requirements for Tax XAI Under Constitutional Principles and Human Rights
Kuzniacki, B., Almada, M., Tyliński, K., & Górski, Ł. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-15565-9_14 · Cited by 14 · see citing works
AI as a constituted system: accountability lessons from an LLM experiment
Nabben, K. (2024) · Data & Policy · doi:10.1017/dap.2024.58 · Cited by 11 · see citing works
Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems
Kroll, J. A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445937 · Cited by 2 · see citing works
Hoang, V. T., & Nguyen, N. M. (2026) · Digital Policy, Regulation and Governance · doi:10.1108/dprg-05-2026-0286
Disclosure or Marketing? Analyzing the Efficacy of Vendor Self-reports for Vetting Public-sector AI
Kuehnert, B., Johnson, N., Dotan, R., & Heidari, H. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3812220
- Authors
- Wieringa, M.
- Affiliations
- Utrecht University (NL)
- Corresponding author
- Wieringa, Maranke
- ORCID
- Wieringa, Maranke
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 1-18
- OpenAlex
- W3001363766
- Semantic Scholar
- 431abdf3b187497e8ac0ceac21f41ce60aec1499
- Cited by
- 321
- References
- 108
- Citations by year
- 2026: 90 · 2025: 56 · 2024: 40 · 2023: 29 · 2022: 49 · 2021: 48 · 2020: 9
- FWCI
- 35.01
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Law › Digitalization, Law, and Regulation
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- algorithmic accountability, accountability theory, algorithmic governance, algorithmic regulation, AI ethics, critical algorithm studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
