CoChat

The impact of AI errors in a human-in-the-loop process

Open access
Agudo, Ujué, Liberal, Karlos G., Arrese, Miren, Matute, Helena · 2024 · Cognitive Research: Principles and Implications

The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.

Abstract

Automated decision-making is becoming increasingly common in the public sector. As a result, political institutions recommend the presence of humans in these decision-making processes as a safeguard against potentially erroneous or biased algorithmic decisions. However, the scientific literature on human-in-the-loop performance is not conclusive about the benefits and risks of such human presence, nor does it clarify which aspects of this human–computer interaction may influence the final decision. In two experiments, we simulate an automated decision-making process in which participants judge multiple defendants in relation to various crimes, and we manipulate the time in which participants receive support from a supposed automated system with Artificial Intelligence (before or after they make their judgments). Our results show that human judgment is affected when participants receive incorrect algorithmic support, particularly when they receive it before providing their own judgment, resulting in reduced accuracy. The data and materials for these experiments are freely available at the Open Science Framework: https://osf.io/b6p4z/ Experiment 2 was preregistered.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s41235-023-00529-3

Cites

39 of 39 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Example of a Screenshot of Step 0 (Description of a Criminal Case and Witness Testimonies), in Experiment 1 (Translated from Spanish). Note. In Experiment 1, each criminal case showed seven testimonies (of innocence in the example), while in Experiment 2, each case showed five testimonies
Fig. 1 Example of a Screenshot of Step 0 (Description of a Criminal Case and Witness Testimonies), in Experiment 1 (Translated from Spanish). Note. In Experiment 1, each criminal case showed seven testimonies (of innocence in the example), while in Experiment 2, each case showed five testimoniesCC BY · PubMed Central
Fig. 2 Step 1A shows a example of a Screenshot of Step 1 (Confirm or Modify AI’s Assessment) in the AIsupport→Judgment Group, in Experiments 1 and 2 (Translated from Spanish). Note. Step 1B was only shown to participants who clicked the Modify button. In that case, the change from 1A to 1B did not involve a screen change. The new information was displayed below the Confirm and Modify buttons on the same screen. The Judgment→AIsupport group viewed these screenshots in Step 2
Fig. 2 Step 1A shows a example of a Screenshot of Step 1 (Confirm or Modify AI’s Assessment) in the AIsupport→Judgment Group, in Experiments 1 and 2 (Translated from Spanish). Note. Step 1B was only shown to participants who clicked the Modify button. In that case, the change from 1A to 1B did not involve a screen change. The new information was displayed below the Confirm and Modify buttons on the same screen. The Judgment→AIsupport group viewed these screenshots in Step 2CC BY · PubMed Central
Fig. 3 Example of Screenshots in Step 2 (Judgment Without AI Support) in the AIsupport→Judgment Group, in Experiment 1. Note. The Judgment→AIsupport group viewed this screenshot in Step 1. In that case, the phrase “To close this case definitively" did not appear on this screen, but on the corresponding Step 2 screen
Fig. 3 Example of Screenshots in Step 2 (Judgment Without AI Support) in the AIsupport→Judgment Group, in Experiment 1. Note. The Judgment→AIsupport group viewed this screenshot in Step 1. In that case, the phrase “To close this case definitively" did not appear on this screen, but on the corresponding Step 2 screenCC BY · PubMed Central
Fig. 4 Percentage of Participants with Correct Assessments in Each Group by Type of Trial (Incorrect or Correct), in Experiment 1. Note. On the correct trials, the testimonies and the AI assessment that participants viewed in Steps 1 and 2 (according to the group) were congruent; on the incorrect trial, they were incongruent. Data of accuracy in the correct trials represent the percentage of participants whose judgments were accurate on both correct trials
Fig. 4 Percentage of Participants with Correct Assessments in Each Group by Type of Trial (Incorrect or Correct), in Experiment 1. Note. On the correct trials, the testimonies and the AI assessment that participants viewed in Steps 1 and 2 (according to the group) were congruent; on the incorrect trial, they were incongruent. Data of accuracy in the correct trials represent the percentage of participants whose judgments were accurate on both correct trialsCC BY · PubMed Central
Fig. 5 Example of a Screenshot in Step 2 (Judgment Without AI Support Present) in the AIsupport→Judgment Group, in Experiment 2. Note. To indicate when the steps of each case were completed, in this Experiment 2 we changed the literal of the "Save" button from Experiment 1 to "Save and move on to the next case". The Judgment→AIsupport group viewed this screenshot in Step 1. In that case, the phrase “To close this case definitively” of the instructions did not appear on this screen, but on the Step 2 screen
Fig. 5 Example of a Screenshot in Step 2 (Judgment Without AI Support Present) in the AIsupport→Judgment Group, in Experiment 2. Note. To indicate when the steps of each case were completed, in this Experiment 2 we changed the literal of the "Save" button from Experiment 1 to "Save and move on to the next case". The Judgment→AIsupport group viewed this screenshot in Step 1. In that case, the phrase “To close this case definitively” of the instructions did not appear on this screen, but on the Step 2 screenCC BY · PubMed Central
Fig. 6 Mean Judgment of Guilt in Incorrect Trials, by Type of Testimonies and Group, in Experiment 2. Note. Error bars 95% CI
Fig. 6 Mean Judgment of Guilt in Incorrect Trials, by Type of Testimonies and Group, in Experiment 2. Note. Error bars 95% CICC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 39 references

Research works printed but not linked: 2

  1. Parasuraman, R., & Mustapha, M. (1996). Automation and human performance. CRC Press.
  2. Ponce, J. (2022). Reserva de humanidad y supervisión humana de la Inteligencia artificial. El Cronista Del Estado Social y Democrático De Derecho, 100, 58–67.

Cited by

See all 67

Metadata

Authors
Agudo, U., Liberal, K. G., Arrese, M., & Matute, H.
Affiliations
Universidad de Deusto (ES)
Corresponding author
Matute, Helena
Venue
Cognitive Research: Principles and Implications
Published
2024-01-07
Volume
9
Issue
1
Pages
1-1
ISSN
2365-7464 (ISSN-L)
OpenAlex
W4390655872
Cited by
66
References
38
Citations by year
2026: 30 · 2025: 27 · 2024: 9
FWCI
31.31
Type
journal article
Language
en
Open access
Licence
Locations
  • Cognitive Research Principles and Implications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DeustoTeka (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Ministerio de Ciencia e Innovación (PID2021-126320NB-I00/MCIN/AEI/10.13039/501100011033/ERDF, EU)
  • Eusko Jaurlaritza (IT1696-22)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
human-in-the-loop decision making, AI errors, automated decision support, algorithmic bias, human-computer interaction, public sector decision-making
MeSH terms
Artificial Intelligence, Crime, Humans, Judgment, Public Sector

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The impact of AI errors in a human-in-the-loop process. https://app.cochat.ai/papers/10.1186/s41235-023-00529-3
The paper
Agudo, U., Liberal, K. G., Arrese, M., & Matute, H. (2024). The impact of AI errors in a human-in-the-loop process. Cognitive Research: Principles and Implications. https://doi.org/10.1186/s41235-023-00529-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.