Cited by 17Usability and User Interface DesignHuman-Automation Interaction and SafetyInnovative Human-Technology Interaction

Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance

Open access
García García, Pedro, Costanza, Enrico, Verame, Jhim, Nowacka, Diana, Ramchurn, Sarvapali D.·2021·Human–Computer Interaction

Abstract

In this article, we report on one lab study and seven follow-up studies on a crowdsourcing platform designed to investigate the potential of animation cues to influence users’ perception of two smart systems: a handwriting recognition and a part-of-speech tagging system. Results from the first three studies indicate that animation cues can influence a participant’s perception of both systems’ performance. The subsequent three studies, designed to try and identify an explanation for this effect, suggest that this effect is related to the participants’ mental model of the smart system. The last two studies were designed to characterize the effect more in detail, and they revealed that different amounts of animation do not seem to create substantial differences and that the effect persists even when the system’s performance decreases, but only when the difference in performance level between the systems being compared is small.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1080/07370024.2018.1453815

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some research suggests that the way transparency is presented may account for contradictory results.

    “Some research suggests that the way we present transparency may account for these contradictory results [28,43].”

    From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [28]

    Section 2.2 (page unavailable) describes how explanations can help users understand a system but can also overwhelm them, and how animation can aid understanding yet sometimes cause confusion. This supports the idea that presentation matters, but the paper does not specifically say that differences in presentation account for the contradictory results cited in the sentence. (LLM, conf=0.78)

    Experimental — accuracy not yet measured
  • Providing potential system users with hypothetical scenarios describing system operation and eliciting reactions to those systems offers a way to collect controlled user data at scale.

    “These methods offer ways to collect controlled user data at scale [28,50].”

    From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [28]

    Section 3.4 describes collecting users’ evaluations after presenting systems under different conditions, including in crowdsourcing studies. But participants viewed system outputs rather than hypothetical scenarios; Section 3.2 says the MTurk samples were kept to eight participants per condition, and Section 6 describes crowdsourcing as less controlled than the lab. The paper therefore supports collecting user reactions online, but not the claimed hypothetical-scenario method or a demonstrated ability to collect controlled data at scale. (LLM, conf=0.83)

    Experimental — accuracy not yet measured
  • The way we present transparency may account for contradictory results about transparency's effects.

    “Some research suggests that the way we present transparency may account for these contradictory results [26,42].”

    From Progressive disclosure (2019) · cites this paper as [26]

    Section 2.2 (page unavailable) says explanations can help users understand smart systems but can also overwhelm or confuse them; it explores animation as another way to present system activity. This supports the idea that presentation matters, but the paper does not say that presentation accounts for contradictory findings about transparency’s effects; it also identifies system uncertainty as a reason effects differ. (LLM, conf=0.78)

    Experimental — accuracy not yet measured
  • Scenario-based methods offer ways to collect controlled user data at scale.

    “These methods offer ways to collect controlled user data at scale [26,49].”

    From Progressive disclosure (2019) · cites this paper as [26]

    The Methodology and Procedure sections describe simulated system tasks with standardized conditions, including studies conducted through MTurk, so the paper supports collecting user data through such methods. But Participants says recruitment was deliberately limited to eight people per condition, and Study 2 calls crowdsourcing less controlled than a lab; the paper does not establish collection of controlled data at scale. Page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Users anticipated a preference for transparency before using the system.

    “Consistent with our initial expectations and the prior literature on transparency [28,49,85], users anticipated a preference for transparency before using the system.”

    From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [28]

    Section 2.2 (page unavailable) discusses how explanations can improve understanding and trust, but does not report that users anticipated preferring transparency before using a system. I searched for pre-use expectations and preferences; the Procedure section (3.4, page unavailable) describes an initial questionnaire but reports no such preference. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 59 · Cited by 17

Research works printed but not linked: 4

  1. [5]5. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's mechanical turk. Perspec- tives on Psychological Science, 6 (1), 3-5. Retrieved from http://dx.doi.org/10.1177/ 1745691610393980 (PMID: 26162106) doi:
  2. [22]22. Ju, W., & Takayama, L. (2009). Approachability: How people interpret automatic door movement as gesture. International Journal of Design, 3 (2).
  3. [30]30. Lyons, J. (2013). Being transparent about transparency: A model for human-robot interaction.. Retrieved from http://www.aaai.org/ocs/index.php/SSS/SSS13/paper/view/5712
  4. [31]31. Mason, W., & Watts, D. J. (2010, May). Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl., 11 (2), 100-108. Retrieved from http://doi.acm.org/10.1145/ 1809400.1809422 doi:

Authors
García García, P., Costanza, E., Verame, J., Nowacka, D., & Ramchurn, S. D.
Affiliations
  • University of Southampton, Agents, Interaction and Complexity Research Group, Southampton, United Kingdom
  • University College London (GB)
  • University of Southampton (GB)
Venue
Human–Computer Interaction
Published
2018-04-12
Volume
36
Issue
1
Pages
1-51
ISSN
  • 0737-0024 (ISSN-L)
  • 1532-7051
OpenAlex
W2796522078
Cited by
17
References
59
Citations by year
2026: 2 · 2024: 1 · 2023: 2 · 2022: 4 · 2021: 2 · 2020: 1 · 2019: 2 · 2018: 3
FWCI
2.00
Type
journal article
Language
en
Open access
Locations
  • Human-Computer Interaction (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • ePrints Soton (University of Southampton) (submittedVersion)
Funders
Engineering and Physical Sciences Research Council (EP/N014243/2)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
automatic systems, handwriting recognition, part-of-speech tagging, mental models, crowdsourcing, user perception, intelligent systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review