Seeing (Movement) is Believing: The Effect of Motion on Perception of Automatic Systems Performance
Open accessAbstract
In this article, we report on one lab study and seven follow-up studies on a crowdsourcing platform designed to investigate the potential of animation cues to influence users’ perception of two smart systems: a handwriting recognition and a part-of-speech tagging system. Results from the first three studies indicate that animation cues can influence a participant’s perception of both systems’ performance. The subsequent three studies, designed to try and identify an explanation for this effect, suggest that this effect is related to the participants’ mental model of the smart system. The last two studies were designed to characterize the effect more in detail, and they revealed that different amounts of animation do not seem to create substantial differences and that the effect persists even when the system’s performance decreases, but only when the difference in performance level between the systems being compared is small.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some research suggests that the way transparency is presented may account for contradictory results.
“Some research suggests that the way we present transparency may account for these contradictory results [28,43].”
From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [28]
Section 2.2 (page unavailable) describes how explanations can help users understand a system but can also overwhelm them, and how animation can aid understanding yet sometimes cause confusion. This supports the idea that presentation matters, but the paper does not specifically say that differences in presentation account for the contradictory results cited in the sentence. (LLM, conf=0.78)
Experimental — accuracy not yet measuredProviding potential system users with hypothetical scenarios describing system operation and eliciting reactions to those systems offers a way to collect controlled user data at scale.
“These methods offer ways to collect controlled user data at scale [28,50].”
From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [28]
Section 3.4 describes collecting users’ evaluations after presenting systems under different conditions, including in crowdsourcing studies. But participants viewed system outputs rather than hypothetical scenarios; Section 3.2 says the MTurk samples were kept to eight participants per condition, and Section 6 describes crowdsourcing as less controlled than the lab. The paper therefore supports collecting user reactions online, but not the claimed hypothetical-scenario method or a demonstrated ability to collect controlled data at scale. (LLM, conf=0.83)
Experimental — accuracy not yet measuredThe way we present transparency may account for contradictory results about transparency's effects.
“Some research suggests that the way we present transparency may account for these contradictory results [26,42].”
From Progressive disclosure (2019) · cites this paper as [26]
Section 2.2 (page unavailable) says explanations can help users understand smart systems but can also overwhelm or confuse them; it explores animation as another way to present system activity. This supports the idea that presentation matters, but the paper does not say that presentation accounts for contradictory findings about transparency’s effects; it also identifies system uncertainty as a reason effects differ. (LLM, conf=0.78)
Experimental — accuracy not yet measuredScenario-based methods offer ways to collect controlled user data at scale.
“These methods offer ways to collect controlled user data at scale [26,49].”
From Progressive disclosure (2019) · cites this paper as [26]
The Methodology and Procedure sections describe simulated system tasks with standardized conditions, including studies conducted through MTurk, so the paper supports collecting user data through such methods. But Participants says recruitment was deliberately limited to eight people per condition, and Study 2 calls crowdsourcing less controlled than a lab; the paper does not establish collection of controlled data at scale. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredUsers anticipated a preference for transparency before using the system.
“Consistent with our initial expectations and the prior literature on transparency [28,49,85], users anticipated a preference for transparency before using the system.”
From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [28]
Section 2.2 (page unavailable) discusses how explanations can improve understanding and trust, but does not report that users anticipated preferring transparency before using a system. I searched for pre-use expectations and preferences; the Procedure section (3.4, page unavailable) describes an initial questionnaire but reports no such preference. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Users anticipated a preference for transparency before using the system.
“Consistent with our initial expectations and the prior literature on transparency [26,48], users anticipated a preference for transparency before using the system.”
From Progressive disclosure (2019) · cites this paper as [26]
Section 2.2 discusses how explanations can help users understand smart systems, but it does not report that users anticipated preferring transparency before use. Section 3.4 describes a questionnaire before the task and evaluations after participants saw the systems; it does not report a pre-use transparency preference. Searches for that preference and its timing found no supporting passage in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measuredOne method used in prior studies is to provide potential system users with hypothetical scenarios describing system operation and eliciting reactions to those systems.
“One method used in prior studies is to provide potential system users with hypothetical scenarios describing system operation and eliciting reactions to those systems.”
From Progressive disclosure (2019) · cites this paper as [26]
Section 3.4 (page unavailable) describes participants viewing system interfaces and evaluating their performance, rather than reacting to hypothetical scenarios. Section 2.2 discusses prior studies of explanations and system intelligibility, but does not describe the claimed scenario-based method. Searches for hypothetical scenarios and vignettes found no passage establishing it. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 59 · Cited by 17
Research works printed but not linked: 4
- [5]5. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's mechanical turk. Perspec- tives on Psychological Science, 6 (1), 3-5. Retrieved from http://dx.doi.org/10.1177/ 1745691610393980 (PMID: 26162106) doi:
- [22]22. Ju, W., & Takayama, L. (2009). Approachability: How people interpret automatic door movement as gesture. International Journal of Design, 3 (2).
- [30]30. Lyons, J. (2013). Being transparent about transparency: A model for human-robot interaction.. Retrieved from http://www.aaai.org/ocs/index.php/SSS/SSS13/paper/view/5712
- [31]31. Mason, W., & Watts, D. J. (2010, May). Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl., 11 (2), 100-108. Retrieved from http://doi.acm.org/10.1145/ 1809400.1809422 doi:
- Authors
- García García, P., Costanza, E., Verame, J., Nowacka, D., & Ramchurn, S. D.
- Affiliations
- University of Southampton, Agents, Interaction and Complexity Research Group, Southampton, United Kingdom
- University College London (GB)
- University of Southampton (GB)
- Venue
- Human–Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2018-04-12
- Volume
- 36
- Issue
- 1
- Pages
- 1-51
- ISSN
- 0737-0024 (ISSN-L)
- 1532-7051
- OpenAlex
- W2796522078
- Cited by
- 17
- References
- 59
- Citations by year
- 2026: 2 · 2024: 1 · 2023: 2 · 2022: 4 · 2021: 2 · 2020: 1 · 2019: 2 · 2018: 3
- FWCI
- 2.00
- Type
- journal article
- Language
- en
- Open access
- green
- discovery.ucl.ac.uk
- Locations
- Human-Computer Interaction (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- ePrints Soton (University of Southampton) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/N014243/2)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- automatic systems, handwriting recognition, part-of-speech tagging, mental models, crowdsourcing, user perception, intelligent systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review