Algorithmic decision-making and system destructiveness: A case of automatic debt recovery
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Governments are increasingly relying on algorithmic decision-making (ADM) to deliver public services. Recent information systems literature has raised concerns regarding ADM’s negative unintended consequences, such as widespread discrimination, which in extreme cases can be destructive to society. The extant empirical literature, however, has not sufficiently examined the destructive effects of governmental ADM. In this paper, we report on a case study of the Australian government’s “Robodebt” programme that was designed to automatically calculate and collect welfare overpayment debts from citizens but ended up causing severe distress to citizens and welfare agency staff. Employing perspectives from systems thinking and organisational limits, we develop a research model that explains how a socially destructive government ADM programme was initiated, sustained, and delegitimized. The model offers a set of generalisable mechanisms that can benefit investigations of ADM’s consequences. Our findings contribute to the literature of unintended consequences of ADM and demonstrate to practitioners the importance of setting up robust governance infrastructures for ADM programmes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Seeking Qualitative Rigor in Inductive Research: Notes on the Gioia Methodology
Gioia, D. A., Corley, K. G., & Hamilton, A. L. (2013) · Organizational Research Methods · doi:10.1177/1094428112452151 · Cited by 13,227
Strategies for Theorizing from Process Data
Langley, A. (1999) · The Academy of Management Review · doi:10.5465/amr.1999.2553248 · Cited by 7,008 · cited 1×
“The logic of temporal bracketing (Langley, 1999) helped us to organise the events of interest into a process with four distinct phases (see Figure 3”
BackgroundBig other: Surveillance Capitalism and the Prospects of an Information Civilization
Zuboff, S. (2015) · Journal of Information Technology · doi:10.1057/jit.2015.5 · Cited by 3,205 · cited 1×
“This trade-off raises questions regarding individuals’ freedom of choice (Zuboff, 2015, 2019).”
BackgroundGeneral Systems Theory—The Skeleton of Science
Boulding, K. E. (1956) · Management Science · doi:10.1287/mnsc.2.3.197 · Cited by 2,333
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 1×
“Third, the use of algorithms to profile individuals – sometimes inadvertently based on their race, ethnic group, gender, and socio-economic status – can lead to biases and discrimination (Chouldechova, 2017), which raises wider ethical concerns (Boyd & Crawford, 2012).”
MethodEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“In Europe, the General Data Protection Regulation (GDPR) is an example of a legislative limit that sets bounds on organisations’ use of ADM in their operations, including rules relating to “a right for explanation” and “a right to be forgotten” (Goodman & Flaxman, 2017).”
BackgroundAsh, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789
The Escalation of Commitment To a Course of Action
Staw, B. M. (1981) · The Academy of Management Review · doi:10.5465/amr.1981.4285694 · Cited by 1,693
Takeover Threats and Managerial Myopia
Stein, J. C. (1988) · Journal of Political Economy · doi:10.1086/261524 · Cited by 1,673
Cited by
Thinking responsibly about responsible AI and ‘the dark side’ of AI
Mikalef, P., Conboy, K., Lundström, J. E., & Popovič, A. (2022) · European Journal of Information Systems · doi:10.1080/0960085x.2022.2026621 · Cited by 435
The Vicious Circles of Skill Erosion: A Case Study of Cognitive Automation
Rinta-Kahila, T., The University of Queensland Business School, Penttinen, E., Aalto University School of Business, Salovaara, A., Aalto University / University of Helsinki, Soliman, W., University of Agder, N. /. U. O. J., Ruissalo, J., & Aalto University School of Business (2023) · Journal of the Association for Information Systems · doi:10.17705/1jais.00829 · Cited by 86
“Consistent with other studies (e.g., Drummond, 2008; Garud & Kumaraswamy, 2005; Rinta-Kahila et al., 2022), our case-study findings indicate that prioritizing IT-enabled short-term efficiency gains over organizational learning may culminate in detrimental long-term outcomes.”
BackgroundResultSelf-citationAI-deploying organizations are key to addressing ‘perfect storm’ of AI risks
Curtis, C., Gillespie, N., & Lockey, S. (2023) · AI and Ethics · doi:10.1007/s43681-022-00163-7 · Cited by 43
“Use of AI systems in the public sector has unique challenges including the potential for unintended consequences on millions of citizens, the potential to disproportionately impact vulnerable communities, and the integration into essential services where there may be little or no opportunity to opt out [39, 40].”
BackgroundSelf-citationExploring citizens’ stances on AI in public services: A social contract perspective
Schmager, S., Grøder, C. H., Parmiggiani, E., Pappas, I., & Vassilakopoulou, P. (2024) · Data & Policy · doi:10.1017/dap.2024.13 · Cited by 35
Bankuoru Egala, S., & Liang, D. (2024) · European Journal of Information Systems · doi:10.1080/0960085x.2023.2251927 · Cited by 30
“The effects have been varied even though algorithmic decision-making has pervaded several fields of endeavour such as healthcare (Liang & Xue, 2022), finance (Castelo et al., 2019) and governance (Rinta-Kahila et al., 2021).”
Metadata
- Authors
- Rinta-Kahila, T., Someh, I., Gillespie, N., Indulska, M., & Gregor, S.
- Venue
- European Journal of Information Systems
- Publisher
- Informa UK Limited
- Published
- 2021-09-08
- Volume
- 31
- Issue
- 3
- Pages
- 313-338
- ISSN
- 0960-085X (ISSN-L)
- 1476-9344
- OpenAlex
- W3196401313
- Cited by
- 108
- References
- 24
- FWCI
- 17.30
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Algorithmic decision-making and system destructiveness: A case of automatic debt recovery. https://app.cochat.ai/papers/10.1080/0960085x.2021.1960905
- The paper
- Rinta-Kahila, T., Someh, I., Gillespie, N., Indulska, M., & Gregor, S. (2022). Algorithmic decision-making and system destructiveness: A case of automatic debt recovery. European Journal of Information Systems. https://doi.org/10.1080/0960085x.2021.1960905
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.