The DOI matches its Crossref record; 50 of the 51 references it cites were found in a registry or an index; none of the 21 sources its claims cite is checked for support yet.
Gaze Path Stimulation in Retrospective Think-Aloud
Open accessAbstract
For a long time, eye tracking has been thought of as a promising method for usability testing. During the last couple of years, eye tracking has finally started to live up to these expectations, at least in terms of its use in usability laboratories. We know that the user’s gaze path can reveal usability issues that would otherwise go unnoticed, but a common understanding of how best to make use of eye movement data has not been reached. Many usability practitioners seem to have intuitively started to use gaze path replays to stimulate recall for retrospective walk through of the usability test. We review the research on thinkaloud protocols in usability testing and the use of eye tracking in the context of usability evaluation. We also report our own experiment in which we compared the standard, concurrent think-aloud method with the gaze path stimulated retrospective think-aloud method. Our results suggest that the gaze path stimulated retrospective think-aloud method produces more verbal data, and that the data are more informative and of better quality as the drawbacks of concurrent think-aloud have been avoided.
Overview
CoChat Summary
TL;DR
In a usability test of Autotalli.com with eight participants, gaze path stimulated retrospective think-aloud produced significantly more verbal data and more cognitive comments than concurrent think-aloud. RTA also revealed 11 new usability problems not found in observation and CTA, though the authors call the results preliminary due to small sample size.
Key findings
- Gaze path stimulated RTA produced significantly more verbal data than CTA: CTA yielded 1,148 words / 66 comments, RTA after CTA yielded 3,309 words / 214 comments, and RTA without preceding CTA yielded 4,136 words / 267 comments; the Stage effect on average number of words was significant, F(1,6) = 29.6, p < .0001.
- Comment quality shifted toward cognitive operations: CTA comments were 82% manipulative, 14% visual, and 4% cognitive; RTA without preceding CTA was 42% manipulative, 15% visual, and 43% cognitive; Comment Category effect F(2,6) = 20.2, p < .0001, and Stage × Category interaction F(2,6) = 14.1, p < .0001.
- Of 44 usability problems encountered, 31 were present in both retrospective and concurrent think-aloud protocols; only two problems observed in videotaped sessions were not talked about in RTA, while RTA revealed 11 new problems not found in observation and CTA.
- Three out of four users found retrospective verbalization more pleasant than concurrent thinking aloud, and all users reported it was easy to follow the gaze paths.
Methods
- Traditional usability test of a Finnish car brokerage web site (Autotalli.com) with eight participants (24–33 years, average 30; three male, five female) and eight tasks; half were instructed to think aloud concurrently, half performed tasks without verbalizing, and all had gaze recorded with a Tobii 1750 eye tracker at 50 Hz.
- All participants then gave retrospective verbalizations while viewing a video replay with their gaze path overlaid, shown at half speed; the experimenter prompted think-aloud if participants fell silent.
- Qualitative analysis categorized usability problems from CTA and RTA; quantitative analysis coded verbalizations using Hansen’s (1991) categories—manipulative, visual, cognitive—and used a 2 × 2 × 3 ANOVA with CTA as between-subjects, Stage as within-subjects, and Comment Category as within-subjects.
Limitations
- Stated in the paper: results are preliminary due to small sample size; the increase in verbalizations might be due to more talkative participants in the RTA condition rather than retrospection itself; instructions and probes in RTA need further research.
- Stated in the paper: eye tracking lost data from nearly one third of participants (one recording problem, two calibration failures), calibration and mobility restrictions remain demanding, and RTA lengthens testing—the retrospective walkthrough likely takes at least twice or three times the original task execution, with their retrospective verbalization taking about 20 minutes.
- Evident from the text: the sample was not representative of the site’s real end-user group because the site is dominated by male users, but three originally recruited male participants were omitted; also, no audio was played during playback, and the experimenter had trouble keeping pace with tasks because tasks were read aloud and not shown on screen.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 0 Not valid · 18 Not checked · 31 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In RTA the percentages were almost equal: 42% of manipulative comments vs. 43% cognitive comments.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was noted that the users were clearly more relaxed in the RTA condition than in the CTA condition.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Conclusions
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 44 usability problems encountered by the participants, 31 problems were present in both retrospective and concurrent think-aloud protocols.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Qualitative Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results suggest that the gaze path stimulated retrospective think-aloud method produces more verbal data”
- Also stated in: § Introduction · plain
“We found that users produced significantly more verbal data in RTA than in CTA.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A significant main effect (F(1,6) = 29.6,p < .0001) of Stage (concurrent vs. retrospective verbalizations) on the average number of words was observed”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 49 claims and to verify the 15 references still open.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 14
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(1,6) = 29.6,p < .0001”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
Reported p < .0001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(2,6) = 20.2,p < .0001”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
Reported p < .0001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(2,6) = 14.1, p < .0001”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
Reported p < .0001 · recomputed p = .005
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(1, 6) = 29.6reported p < .0001 · recomputed p = .002
“F(1,6) = 29.6,p < .0001”
- Doesn’t add upF(2, 6) = 20.2reported p < .0001 · recomputed p = .002
“F(2,6) = 20.2,p < .0001”
- Doesn’t add upF(2, 6) = 14.1reported p < .0001 · recomputed p = .005
“F(2,6) = 14.1, p < .0001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
“Participants made significantly more manipulative comments than visual comments and significantly more cognitive comments than manipulative comments.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
“Manipulative comments produced retrospectively CTA Without CTA 28.0 29.8 10.5 14.1 RTA Visual comments produced retrospectively CTA Without CTA 7.5 10.0 6.6 4.1 Cognitive comments produced retrospectively CTA Without CTA 18.0 28.5 7.0 17.3”
Citations
Cites
(1993) · Cited by 9,879
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
Eye Tracking in Human-Computer Interaction and Usability Research
Jacob, R. J. K., & Karn, K. S. (2003) · The Mind's Eye · doi:10.1016/b978-044451020-4/50031-1 · Cited by 1,567
Protocol Analysis: Verbal Reports as Data
Danks, J. H., Ericsson, K. A., & Simon, H. A. (1985) · The American Journal of Psychology · doi:10.2307/1422516 · Cited by 1,218
Computer interface evaluation using eye movements: methods and constructs
Goldberg, J. H., & Kotval, X. P. (1999) · International Journal of Industrial Ergonomics · doi:10.1016/s0169-8141(98)00068-7 · Cited by 1,053
Thinking aloud: reconciling theory and practice
Boren, T., & Ramey, J. (2000) · IEEE Transactions on Professional Communication · doi:10.1109/47.867942 · Cited by 808
The think aloud method: A practical guide to modelling cognitive processes
Solomon, P. (1995) · Information Processing & Management · doi:10.1016/0306-4573(95)90031-4 · Cited by 737
The mind's eye : cognitive and applied aspects of eye movement research
Hyönä, J., Radach, R. R., & Deubel, H. (2003) · Cited by 588
Research works printed but not linked: 6
- Aaltonen, A. (1999). Eye tracking in usability testing: Is it worthwhile? 'Usability & eye tracking' workshop at CHI'99, ACM Press.
- Duchowski, A.T. (2006). High-level eye movement me- trics in the usability context. Position paper, CHI2006 Workshop Getting a Measure of Satisfaction from Eyetracking in Practice, April 23, 2006. Retrieved September 24, 2008, from http://www.amber- light.co.uk/CHI2006/pos_papr_duchowski.pdf
- Lehtinen, M., Hyrskykari, A., & Räihä, K.-J. (2006). Gaze path playback supporting retrospective think- aloud in usability tests. In Proceedings of the 2nd Conference on Communication by Gaze Interaction - COGAIN 2006, Turin, Italy, 88-91. Retrieved Sep- tember 24, 2008, from http://www.cogain.org/cogain2006/COGAIN2006_Pr oceedings.pdf
- Maughan, L., Dodd, J., & Walters, R. (2007). Video rep- lay as a cue in retrospective protocol …Don't make me think aloud! Poster presented at the Second Scan- dinavian Workshop on Applied Eye-Tracking (SWAET 2007), Lund.
- Norman, K.L. & Murphy, E. (2004). Usability testing of an Internet form for the 2004 overseas enumeration test: A comparison of think-aloud and retrospective reports. In Proceedings of the Human Factors Society 48th Annual Meeting, Human Factors Society, New Orleans, LA. Retrieved September 24, 2008, from http://lap.umd.edu/lap/Papers/Tech_Reports/LAP200 4TR04/LAP2004TR04.pdf
- Poole, A. & Ball, L.J. (2006). Eye tracking in human- computer interaction and usability research: Current status and future prospects. In Ghaoui, C. (Ed.). En- cyclopedia of Human Computer Interaction. Idea Group Publishing. Retrieved September 24, 2008 from http://www.alexpoole.info/academic/Poole&Ball%20 EyeTracking.pdf
Cited by
Eye Tracking in User Experience Design
(2014) · Elsevier eBooks · doi:10.1016/c2012-0-06867-6 · Cited by 296
L2 student engagement with automated feedback on writing: Potential for learning and issues of trust
Ranalli, J. (2021) · Journal of Second Language Writing · doi:10.1016/j.jslw.2021.100816 · Cited by 184
Generative Design: An Explorative Study
Buonamici, F., Carfagni, M., Furferi, R., Volpe, Y., & Governi, L. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.144-155 · Cited by 172
Stahnke, R., & Blömeke, S. (2020) · Teaching and Teacher Education · doi:10.1016/j.tate.2020.103243 · Cited by 132
Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2012.10.012 · Cited by 129
Metadata
- Authors
- Hyrskykari, A., Ovaska, S., Majaranta, P., Räihä, K.-J., & Lehtinen, M.
- Affiliations
- University of Tampere (FI)
- ORCID
- Räihä, Kari-Jouko
- Venue
- Journal of Eye Movement Research
- Publisher
- MDPI AG
- Published
- 2008-11-26
- Volume
- 2
- Issue
- 4
- ISSN
- 1995-8692 (ISSN-L)
- OpenAlex
- W2469474566
- Cited by
- 118
- References
- 51
- Citations by year
- 2026: 9 · 2025: 4 · 2024: 6 · 2023: 8 · 2022: 5 · 2021: 11 · 2020: 24 · 2019: 3 · 2018: 5 · 2017: 6 · 2016: 6 · 2015: 5 · 2014: 7 · 2013: 6 · 2012: 6
- FWCI
- 3.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Eye Movement Research (publishedVersion, cc-by)
- Journal of Eye Movement Research (SwissUniversities) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- retrospective think-aloud, think-aloud, eye tracking, usability evaluation, concurrent think-aloud, usability testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Gaze Path Stimulation in Retrospective Think-Aloud. https://app.cochat.ai/papers/10.16910/jemr.2.4.5
- The paper
- Hyrskykari, A., Ovaska, S., Majaranta, P., Räihä, K.-J., & Lehtinen, M. (2008). Gaze Path Stimulation in Retrospective Think-Aloud. Journal of Eye Movement Research. https://doi.org/10.16910/jemr.2.4.5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
