CoChat
Sign Up
Cited by 7Usability and User Interface DesignInnovative Human-Technology InteractionInteractive and Immersive Displays

The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures

Open access
Ou, Changkun, Buschek, Daniel, Mayer, Sven, Butz, Andreas·2022·Mensch und Computer 2022
DOI 10.1145/3543758.3543761
Open this paper in CoChat — free accountOpen PDF

Abstract

Interactive AI systems increasingly employ a human-in-the-loop strategy. This creates new challenges for the HCI community when designing such systems. We reveal and investigate some of these challenges in a case study with an industry partner, and developed a prototype human-in-the-loop system for preference-guided 3D model processing. Two 3D artists used it in their daily work for 3 months. We found that the human-AI loop often did not converge towards a satisfactory result and designed a lab study (N=20) to investigate this further. We analyze interaction data and user feedback through the lens of theories of human judgment to explain the observed human-in-the-loop failures with two key insights: 1) optimization using preferential choices lacks mechanisms to deal with inconsistent and contradictory human judgments; 2) machine outcomes, in turn, influence future user inputs via heuristic biases and loss aversion. To mitigate these problems, we propose descriptive UI design guidelines. Our case study draws attention to challenging and practically relevant imperfections in human-AI loops that need to be considered when designing human-in-the-loop systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3543758.3543761

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • One of the most troublesome types of failure modes identified by Ou et al.'s model are infinite loops where humans and AI systems are constantly deferring to one another without ever resolving the issue.

    “Their analysis suggests that one of the most troublesome types of failure modes identified by their model are infinite loops where humans and AI systems are constantly deferring to one another without ever resolving the issue.”

    From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [106]

    Section 5.1, “Human-AI Mutual Interventions” (page unavailable), says that neither human ratings nor machine optimization improve over iterations and that the loop is kept from terminating. It does not describe humans and AI constantly deferring to one another, or identify this as one of the most troublesome failure types in a model. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Ou et al. identified patterns of breakdown that occur when interaction timing, feedback cycles, or task sequencing are poorly designed.

    “Ou et al. conducted a case study on revealing and explaining human–AI interaction loop failures, identifying patterns of breakdown that occur when interaction timing, feedback cycles, or task sequencing are poorly designed [106].”

    From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [106]

    The Summary and Future Work section describes breakdowns in an iterative human–AI loop caused by inconsistent human judgments, the system’s inability to handle them, and effects of system outcomes on later user inputs. The Potential Countermeasures section proposes ways to improve feedback across iterations. Neither section identifies poorly designed interaction timing or task sequencing as causes of the failures, so the cited paper supports only part of the characterization. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The temporal structure of human-AI interaction affects both the quality of human contributions and the efficiency of collaborative workflows.

    “The temporal structure of human–AI interaction affects both the quality of human contributions and the efficiency of collaborative workflows.”

    From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [106]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [7]7. Curran Associates Inc., Red Hook, NY, USA, 409-416. https://dl.acm.org/doi/abs/10.5555/2981562.2981614 · DOI
  2. [21]21. Javier González, Zhenwen Dai, Andreas Damianou, and Neil D. Lawrence. 2017. Preferential Bayesian Optimization. In Proceedings of the 34th International Con- ference on Machine Learning -Volume 70 (Sydney, NSW, Australia) (ICML'17). JMLR.org, Australia, 1282-1291. https://doi.org/10.5555/3305381.3305514 · DOI
  3. [45]45. Petrus Mikkola, Milica Todorović, Jari Järvi, Patrick Rinke, and Samuel Kaski. 2020. Projective Preferential Bayesian Optimization. In International Conference on Machine Learning. PMLR, MLResearchPress, Online, 6884-6892. https:// proceedings.mlr.press/v119/mikkola20a.html

Cited by

Sorted by citations

Related

Authors
Ou, C., Buschek, D., Mayer, S., & Butz, A.
Affiliations
  • Ludwig-Maximilians-Universität München (DE)
  • University of Bayreuth (DE)
ORCID
Venue
Mensch und Computer 2022
Publisher
ACM
Published
2022-09-04
Pages
158-168
DOI
10.1145/3543758.3543761
OpenAlex
W4288099294
Cited by
7
References
69
Citations by year
2026: 3 · 2025: 2 · 2023: 2
FWCI
0.80
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • Mensch und Computer (MuC) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • ERef Bayreuth (University of Bayreuth) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
Keywords
human-in-the-loop systems, human-AI interaction, human judgment, heuristic biases, loss aversion, user interface design guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review