The DOI matches its Crossref record; 50 of the 51 references it cites were found in a registry or an index; none of the 21 sources its claims cite is checked for support yet.
Gaze Path Stimulation in Retrospective Think-Aloud
Open accessAbstract
For a long time, eye tracking has been thought of as a promising method for usability testing. During the last couple of years, eye tracking has finally started to live up to these expectations, at least in terms of its use in usability laboratories. We know that the user’s gaze path can reveal usability issues that would otherwise go unnoticed, but a common understanding of how best to make use of eye movement data has not been reached. Many usability practitioners seem to have intuitively started to use gaze path replays to stimulate recall for retrospective walk through of the usability test. We review the research on thinkaloud protocols in usability testing and the use of eye tracking in the context of usability evaluation. We also report our own experiment in which we compared the standard, concurrent think-aloud method with the gaze path stimulated retrospective think-aloud method. Our results suggest that the gaze path stimulated retrospective think-aloud method produces more verbal data, and that the data are more informative and of better quality as the drawbacks of concurrent think-aloud have been avoided.
Overview
CoChat Summary
TL;DR
In a usability test of Autotalli.com with eight participants, gaze path stimulated retrospective think-aloud produced significantly more verbal data and more cognitive comments than concurrent think-aloud. RTA also revealed 11 new usability problems not found in observation and CTA, though the authors call the results preliminary due to small sample size.
Key findings
- Gaze path stimulated RTA produced significantly more verbal data than CTA: CTA yielded 1,148 words / 66 comments, RTA after CTA yielded 3,309 words / 214 comments, and RTA without preceding CTA yielded 4,136 words / 267 comments; the Stage effect on average number of words was significant, F(1,6) = 29.6, p < .0001.
- Comment quality shifted toward cognitive operations: CTA comments were 82% manipulative, 14% visual, and 4% cognitive; RTA without preceding CTA was 42% manipulative, 15% visual, and 43% cognitive; Comment Category effect F(2,6) = 20.2, p < .0001, and Stage × Category interaction F(2,6) = 14.1, p < .0001.
- Of 44 usability problems encountered, 31 were present in both retrospective and concurrent think-aloud protocols; only two problems observed in videotaped sessions were not talked about in RTA, while RTA revealed 11 new problems not found in observation and CTA.
- Three out of four users found retrospective verbalization more pleasant than concurrent thinking aloud, and all users reported it was easy to follow the gaze paths.
Methods
- Traditional usability test of a Finnish car brokerage web site (Autotalli.com) with eight participants (24–33 years, average 30; three male, five female) and eight tasks; half were instructed to think aloud concurrently, half performed tasks without verbalizing, and all had gaze recorded with a Tobii 1750 eye tracker at 50 Hz.
- All participants then gave retrospective verbalizations while viewing a video replay with their gaze path overlaid, shown at half speed; the experimenter prompted think-aloud if participants fell silent.
- Qualitative analysis categorized usability problems from CTA and RTA; quantitative analysis coded verbalizations using Hansen’s (1991) categories—manipulative, visual, cognitive—and used a 2 × 2 × 3 ANOVA with CTA as between-subjects, Stage as within-subjects, and Comment Category as within-subjects.
Limitations
- Stated in the paper: results are preliminary due to small sample size; the increase in verbalizations might be due to more talkative participants in the RTA condition rather than retrospection itself; instructions and probes in RTA need further research.
- Stated in the paper: eye tracking lost data from nearly one third of participants (one recording problem, two calibration failures), calibration and mobility restrictions remain demanding, and RTA lengthens testing—the retrospective walkthrough likely takes at least twice or three times the original task execution, with their retrospective verbalization taking about 20 minutes.
- Evident from the text: the sample was not representative of the site’s real end-user group because the site is dominated by male users, but three originally recruited male participants were omitted; also, no audio was played during playback, and the experimenter had trouble keeping pace with tasks because tasks were read aloud and not shown on screen.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 0 Not valid · 18 Not checked · 31 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In RTA the percentages were almost equal: 42% of manipulative comments vs. 43% cognitive comments.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was noted that the users were clearly more relaxed in the RTA condition than in the CTA condition.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Conclusions
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 44 usability problems encountered by the participants, 31 problems were present in both retrospective and concurrent think-aloud protocols.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Qualitative Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results suggest that the gaze path stimulated retrospective think-aloud method produces more verbal data”
- Also stated in: § Introduction · plain
“We found that users produced significantly more verbal data in RTA than in CTA.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Abstract
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A significant main effect (F(1,6) = 29.6,p < .0001) of Stage (concurrent vs. retrospective verbalizations) on the average number of words was observed”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 49 claims and to verify the 15 references still open.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 14
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(1,6) = 29.6,p < .0001”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
Reported p < .0001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(2,6) = 20.2,p < .0001”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
Reported p < .0001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F(2,6) = 14.1, p < .0001”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
Reported p < .0001 · recomputed p = .005
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(1, 6) = 29.6reported p < .0001 · recomputed p = .002
“F(1,6) = 29.6,p < .0001”
- Doesn’t add upF(2, 6) = 20.2reported p < .0001 · recomputed p = .002
“F(2,6) = 20.2,p < .0001”
- Doesn’t add upF(2, 6) = 14.1reported p < .0001 · recomputed p = .005
“F(2,6) = 14.1, p < .0001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
“Participants made significantly more manipulative comments than visual comments and significantly more cognitive comments than manipulative comments.”
§ Gaze Path Stimulation in Retrospective Think-Aloud > Quantitative Evaluation
“Manipulative comments produced retrospectively CTA Without CTA 28.0 29.8 10.5 14.1 RTA Visual comments produced retrospectively CTA Without CTA 7.5 10.0 6.6 4.1 Cognitive comments produced retrospectively CTA Without CTA 18.0 28.5 7.0 17.3”
Citations
Cites
(1993) · Cited by 9,879
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
Eye Tracking in Human-Computer Interaction and Usability Research
Jacob, R. J. K., & Karn, K. S. (2003) · The Mind's Eye · doi:10.1016/b978-044451020-4/50031-1 · Cited by 1,567
Protocol Analysis: Verbal Reports as Data
Danks, J. H., Ericsson, K. A., & Simon, H. A. (1985) · The American Journal of Psychology · doi:10.2307/1422516 · Cited by 1,218
Computer interface evaluation using eye movements: methods and constructs
Goldberg, J. H., & Kotval, X. P. (1999) · International Journal of Industrial Ergonomics · doi:10.1016/s0169-8141(98)00068-7 · Cited by 1,053
Thinking aloud: reconciling theory and practice
Boren, T., & Ramey, J. (2000) · IEEE Transactions on Professional Communication · doi:10.1109/47.867942 · Cited by 808
The think aloud method: A practical guide to modelling cognitive processes
Solomon, P. (1995) · Information Processing & Management · doi:10.1016/0306-4573(95)90031-4 · Cited by 737
The mind's eye : cognitive and applied aspects of eye movement research
Hyönä, J., Radach, R. R., & Deubel, H. (2003) · Cited by 588
Research works printed but not linked: 6
- Aaltonen, A. (1999). Eye tracking in usability testing: Is it worthwhile? 'Usability & eye tracking' workshop at CHI'99, ACM Press.
- Duchowski, A.T. (2006). High-level eye movement me- trics in the usability context. Position paper, CHI2006 Workshop Getting a Measure of Satisfaction from Eyetracking in Practice, April 23, 2006. Retrieved September 24, 2008, from http://www.amber- light.co.uk/CHI2006/pos_papr_duchowski.pdf
- Lehtinen, M., Hyrskykari, A., & Räihä, K.-J. (2006). Gaze path playback supporting retrospective think- aloud in usability tests. In Proceedings of the 2nd Conference on Communication by Gaze Interaction - COGAIN 2006, Turin, Italy, 88-91. Retrieved Sep- tember 24, 2008, from http://www.cogain.org/cogain2006/COGAIN2006_Pr oceedings.pdf
- Maughan, L., Dodd, J., & Walters, R. (2007). Video rep- lay as a cue in retrospective protocol …Don't make me think aloud! Poster presented at the Second Scan- dinavian Workshop on Applied Eye-Tracking (SWAET 2007), Lund.
- Norman, K.L. & Murphy, E. (2004). Usability testing of an Internet form for the 2004 overseas enumeration test: A comparison of think-aloud and retrospective reports. In Proceedings of the Human Factors Society 48th Annual Meeting, Human Factors Society, New Orleans, LA. Retrieved September 24, 2008, from http://lap.umd.edu/lap/Papers/Tech_Reports/LAP200 4TR04/LAP2004TR04.pdf
- Poole, A. & Ball, L.J. (2006). Eye tracking in human- computer interaction and usability research: Current status and future prospects. In Ghaoui, C. (Ed.). En- cyclopedia of Human Computer Interaction. Idea Group Publishing. Retrieved September 24, 2008 from http://www.alexpoole.info/academic/Poole&Ball%20 EyeTracking.pdf
Cited by
Eye Tracking in User Experience Design
(2014) · Elsevier eBooks · doi:10.1016/c2012-0-06867-6 · Cited by 296
L2 student engagement with automated feedback on writing: Potential for learning and issues of trust
Ranalli, J. (2021) · Journal of Second Language Writing · doi:10.1016/j.jslw.2021.100816 · Cited by 184
Generative Design: An Explorative Study
Buonamici, F., Carfagni, M., Furferi, R., Volpe, Y., & Governi, L. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.144-155 · Cited by 172
Stahnke, R., & Blömeke, S. (2020) · Teaching and Teacher Education · doi:10.1016/j.tate.2020.103243 · Cited by 132
Kammerer, Y., Bråten, I., Gerjets, P., & Strømsø, H. I. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2012.10.012 · Cited by 129
Eye tracking in computing education
Busjahn, T., Schulte, C., Sharif, B., Simon, Begel, A., Hansen, M., Bednarik, R., Orlov, P. A., Ihantola, P., Shchekotova, G., et al. (2014) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2632320.2632344 · Cited by 129
Elling, S., Lentz, L., & de Jong, M. (2012) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2012.2206190 · Cited by 48
Stahnke, R., & Blömeke, S. (2021) · Learning and Instruction · doi:10.1016/j.learninstruc.2021.101464 · Cited by 86
Attention Adjustment, Renewal, and Equilibrium Seeking in Online Search: An Eye-Tracking Approach
Ahn, J., Bae, Y.-S., Ju, J., & Oh, W. (2018) · Journal of Management Information Systems · doi:10.1080/07421222.2018.1523595 · Cited by 71
Popelka, S., Stachoň, Z., Šašinka, Č., & Doležalová, J. (2016) · Computational Intelligence and Neuroscience · doi:10.1155/2016/9172506 · Cited by 62
Elling, S., Lentz, L. R., & de Jong, M. D. T. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979116 · Cited by 60
Pepe, M., & Costantino, D. D. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.42-65 · Cited by 56
Constraints on organic chemistry students’ reasoning during IR and 1 H NMR spectral interpretation
Connor, M. C., Finkenstaedt-Quinn, S. A., & Shultz, G. V. (2019) · Chemistry Education Research and Practice · doi:10.1039/c9rp00033j · Cited by 47
Conceptual Change, Text Comprehension and Eye Movements During Reading
Penttinen, M., Anto, E., & Mikkilä‐Erdmann, M. (2012) · Research in Science Education · doi:10.1007/s11165-012-9313-2 · Cited by 42
Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic Review
Hertzum, M. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3665327 · Cited by 41
Students’ Procedure When Solving Problem Tasks Based on the Periodic Table: An Eye-Tracking Study
Tóthová, M., Rusek, M., & Chytrý, V. (2021) · Journal of Chemical Education · doi:10.1021/acs.jchemed.1c00167 · Cited by 40
Where Have Eye Been? Observers Can Recognise Their Own Fixations
Foulsham, T., & Kingstone, A. F. (2013) · Perception · doi:10.1068/p7562 · Cited by 38
Developing Expertise in 1H NMR Spectral Interpretation
Connor, M. C., Glass, B. H., Finkenstaedt-Quinn, S. A., & Shultz, G. V. (2020) · The Journal of Organic Chemistry · doi:10.1021/acs.joc.0c01398 · Cited by 37
Muhonen, H., Pakarinen, E., & Lerkkanen, M. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103458 · Cited by 37
Tang, H. N., Kirk, J. S., & Pienta, N. J. (2014) · Journal of Chemical Education · doi:10.1021/ed4004113 · Cited by 37
What do your eyes say? Bridging eye movements to consumer behavior
Rosa, P. J. (2015) · International journal of psychological research · doi:10.21500/20112084.1513 · Cited by 35
Silent music reading: Amateur musicians’ visual processing and descriptive skill
Penttinen, M., Huovinen, E., & Ylitalo, A. (2013) · Musicae Scientiae · doi:10.1177/1029864912474288 · Cited by 35
Professional vision of Grade 1 teachers experiencing different levels of work-related stress
Muhonen, H., Pakarinen, E., & Lerkkanen, M. (2021) · Teaching and Teacher Education · doi:10.1016/j.tate.2021.103585 · Cited by 33
Tanner, S. A., McCarthy, M. B., & O'Reilly, S. J. (2019) · Appetite · doi:10.1016/j.appet.2018.11.015 · Cited by 28
Muhonen, H., Pakarinen, E., & Lerkkanen, M. (2022) · Teaching and Teacher Education · doi:10.1016/j.tate.2022.103907 · Cited by 32
Mitchell, J. D., Maratos, F. A., Giles, D., Taylor, N., Butterworth, A., & Sheffield, D. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01025 · Cited by 30
Lanini-Maggi, S., Ruginski, I. T., Shipley, T. F., Hurter, C., Duchowski, A. T., Briesemeister, B. B., Lee, J., & Fabrikant, S. I. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100127 · Cited by 30
Looking Back: Retrospective Study Methods for HCI
Russell, D. M., & Ed, H. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8_15 · Cited by 30
Becker, S., Knippertz, L., Ruzika, S., & Kühn, J. (2023) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.19.020142 · Cited by 29
A Usability Analysis of Blocks-based Programming Editors using Cognitive Dimensions
Holwerda, R., & Hermans, F. (2018) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2018.8506483 · Cited by 29
Eye tracking for screening design parameters in adjective-based design of yacht hull
Dogan, K. M., Suzuki, H., & Günpınar, E. (2018) · Ocean Engineering · doi:10.1016/j.oceaneng.2018.08.026 · Cited by 27
Olmsted-Hawala, E. L., Holland, T., & Quach, V. (2014) · Elsevier eBooks · doi:10.1016/b978-0-12-408138-3.00003-0 · Cited by 26
Product Function Redesign Based on Extension Theory
Dong, Y., Peng, Q., Tan, R., Zhang, J., Zhang, P., & We, L. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.199-210 · Cited by 23
Karch, J. M., Valles, J. C. G., & Sevian, H. (2019) · Journal of Chemical Education · doi:10.1021/acs.jchemed.9b00014 · Cited by 22
Heinonen, N., Katajavuori, N., & Södervik, I. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1232273 · Cited by 20
Eye Tracking in Retrospective Think-Aloud Usability Testing:Is there Added Value?
Alhadreti, O., Elbabour, F., & Mayhew, P. J. (2017) · UEA Digital Repository (University of East Anglia) · Cited by 20
Protocol Analysis Data Collection Technique Implemented for Artificial Intelligence Design
Rodgers, W., Al-Shaikh, S. S. K., & Khalil, M. M. (2023) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2023.3269291 · Cited by 20
Molina, A. I., Redondo, M. Á., Ortega, M., & Lacave, C. (2014) · Science of Computer Programming · doi:10.1016/j.scico.2014.02.019 · Cited by 20
Dogan, K. M., & Günpınar, E. (2017) · Ocean Engineering · doi:10.1016/j.oceaneng.2017.08.056 · Cited by 19
Stahnke, R., & Friesen, M. E. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1253459 · Cited by 19
Olsen, A., Smolentzov, L., & Strandvall, T. (2010) · Electronic Workshops in Computing · doi:10.14236/ewic/hci2010.8 · Cited by 7
Heyd‐Metzuyanim, E., Haataja, E., Hannula, M. S., & Moreno-Esteva, E. G. (2023) · Instructional Science · doi:10.1007/s11251-023-09617-9 · Cited by 17
Eye-tracking measures as indicators for a local vs. global view of data
Schreiter, S., & Vogel, M. (2023) · Frontiers in Education · doi:10.3389/feduc.2022.1058150 · Cited by 17
Do You Feel the Same? On the Robustness of Cued-Recall Debriefing for User Experience Evaluation
Bruun, A., Law, E. L.-C., Nielsen, T. D., & Heintz, M. (2021) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3453479 · Cited by 16
Stark, L., Korbach, A., Brünken, R., & Park, B. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13051 · Cited by 16
Die Blickregistrierung in der Werbewirkungsforschung: Grundlagen und Ergebnisse
Hofer, N., & Mayerhofer, W. (2010) · der markt · doi:10.1007/s12642-010-0039-2 · Cited by 16
Liu, X., Werder, K., & Maedche, A. (2020) · Decision Support Systems · doi:10.1016/j.dss.2020.113367 · Cited by 15
The potential of eye tracking for visual literacy research
Brumberger, E. R. (2021) · Journal of Visual Literacy · doi:10.1080/1051144x.2021.1902040 · Cited by 14
Using eye tracking to evaluate usability of user interfaces: Is it warranted?
Groen, M., & Noyes, J. (2010) · IFAC Proceedings Volumes · doi:10.3182/20100831-4-fr-2021.00086 · Cited by 14
Dissecting Optional Micro-Decisions in Online Transactions: Perceptions, Deceptions, and Errors
Hogan, M., Barry, C. J., & Lang, M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3531005 · Cited by 13
Lang, F., Kammerer, Y., Stürmer, K., & Gerjets, P. (2021) · European Journal of Psychology of Education · doi:10.1007/s10212-020-00471-8 · Cited by 12
Stahnke, R., & Gegenfurtner, A. (2025) · Learning and Instruction · doi:10.1016/j.learninstruc.2025.102167 · Cited by 12
Semantic content of motorcycle riders’ eye fixations during lane-splitting
Papakostopoulos, V., Nathanael, D., & Psarakis, L. (2019) · Cognition Technology & Work · doi:10.1007/s10111-019-00572-5 · Cited by 11
Eye tracking recognition-based graphical authentication
Mihajlov, M., Marija, T., & Arsenovski, S. (2013) · International Conference on Application of Information and Communication Technologies · doi:10.1109/icaict.2013.6722632 · Cited by 11
A Comparison of Two Cognitive Pretesting Techniques Supported by Eye Tracking
Neuert, C. E., & Lenzner, T. (2015) · Social Science Computer Review · doi:10.1177/0894439315596157 · Cited by 11
Triggered think-aloud protocol: using eye tracking to improve usability test moderation
Freeman, B. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979117 · Cited by 9
Usability Testing in Tourism Object Management System
Arijaya, I. G. N. P., Pradnyana, I. M. A., Wirawan, I. M. A., Suwendra, I. W., Nugraha, I. G. P., & Suparta, I. N. (2020) · International Conference on Innovative Research Across Disciplines (ICIRAD) · doi:10.2991/assehr.k.200115.023 · Cited by 9
Ways of looking: an eye-tracking study of visual literacy expertise
Brumberger, E. R. (2022) · Journal of Visual Literacy · doi:10.1080/1051144x.2022.2053818 · Cited by 9
Schreiter, S., & Vogel, M. (2024) · Educational Studies in Mathematics · doi:10.1007/s10649-024-10352-2 · Cited by 8
Understanding Students' Frustration and Confusion during a Programming Task: A Multimodal Approach
Batra, R., & Atiq, S. Z. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343499 · Cited by 8
Gaze behavior reveals automaticity and attention allocation during music teaching vs. observing
Heinsen, R. S. (2024) · Journal of Eye Movement Research · doi:10.16910/jemr.17.2.3 · Cited by 7
From sight to insight – reflection processes in an eye-gaze-augmented retrospective
Langner, A., & Graulich, N. (2024) · International Journal of Science Education · doi:10.1080/09500693.2024.2430804 · Cited by 7
Framing or Gaming? Constructing a Study to Explore the Impact of Option Presentation on Consumers
Barry, C. J., Hogan, M., & Torres, A. M. (2016) · Lecture notes in information systems and organisation · doi:10.1007/978-3-319-30133-4_8 · Cited by 7
Interpolating Splines of Biarcs from a Sequence of Planar Points
Bertolazzi, E., Frego, M., & Biral, F. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.66-85 · Cited by 7
User Studies by Driving Simulators in the Era of Automated Vehicle
Shi, Y., Bordegoni, M., & Caruso, G. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.211-226 · Cited by 7
Schreiter, S., Vogel, M., Rehm, M., & Dörfler, T. (2022) · Journal für Mathematik-Didaktik · doi:10.1007/s13138-022-00203-z · Cited by 7
Using Gaze Data in Evaluating Interactive Visualizations
Siirtola, H., & Räihä, K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19641-6_9 · Cited by 6
Self-citationTowards blocks-based prototyping of web applications
Holwerda, R., & Hermans, F. (2017) · IEEE Blocks and Beyond Workshop (B&B) · doi:10.1109/blocks.2017.8120408 · Cited by 6
Eliciting Tacit Expertise in 3D Volume Segmentation
West, R. G., Kajihara, M., Parola, M. J., Hays, K., Hillard, L., Carlew, A. R., Deutsch, J., Lane, B., Holloway, M., John, B., et al. (2016) · International Symposium on Visual Information Communication and Interaction (VINCI) · doi:10.1145/2968220.2968235 · Cited by 5
Wong, C. L., Rana, T., & Leung, Y. M. (2020) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000635 · Cited by 5
Muhonen, H., Pakarinen, E., & Lerkkanen, M. (2025) · Learning and Instruction · doi:10.1016/j.learninstruc.2025.102220 · Cited by 4
Michalik, T. (2024) · Archaeologies · doi:10.1007/s11759-024-09510-y · Cited by 4
Using sensor graphs to stimulate recall in retrospective think-aloud protocols
Pätsch, G., Mandl, T., & Womser‐Hacker, C. (2014) · Information Interaction in Context Symposium (IIiX) · doi:10.1145/2637002.2637048 · Cited by 4
Ren, J., Rong, Y. K., & Ma, Y. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.117-129 · Cited by 4
A Domain-Dependent Lexicon to Augment CAD Peer Review
Beasley, Z., & Piegl, L. A. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.186-198 · Cited by 4
Talsma, R., Barendsen, E., & Smetsers, S. (2020) · Computer Science Education Research Conference · doi:10.1145/3442481.3442505 · Cited by 4
Haataja, E. (2021) · Työväentutkimus Vuosikirja · Cited by 4
Automatic Update of Feature Model After Direct Modeling Operation
Qin, X., Tang, Z., & Gao, S. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.170-185 · Cited by 3
Examining undergraduate and graduate student reasoning when interpreting infrared spectra
Hamilton, D. T., Hollingshead, K., & Atkinson, M. B. (2025) · Chemistry Education Research and Practice · doi:10.1039/d4rp00278d · Cited by 3
Usability Evaluation of Gaze Interaction
Heikkilä, H., & Ovaska, S. (2012) · IGI Global eBooks · doi:10.4018/978-1-61350-098-9.ch017 · Cited by 3
Self-citationAn EYE TRACKING PILOT STUDY ON OPTIONALITY – SOME LESSONS LEARNED
Hogan, M., Barry, C. J., & Torres, A. M. (2015) · Cited by 3
Wang, P. (2022) · Open Journal of Social Sciences · doi:10.4236/jss.2022.104032 · Cited by 3
Cognitive Reliability and Error Analysis based on Anticipatory Failure Determination
Bai, Z., Chang, M., Peng, Q., & Xu, B. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.130-143 · Cited by 3
Modeling Cognitive Strategies during Complex Task Performing Process
Mazman, S. G., & Altun, A. (2012) · Turkish Online Journal of Qualitative Inquiry · doi:10.17569/tojqi.78395 · Cited by 3
Identification of Technology Opportunity Based on a Three-Dimensional Technology Effect Model
He, C., Shi, F., Peng, Q., & Tan, R. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.97-116 · Cited by 3
Evaluating website quality: Five studies on user-focused evaluation methods
Elling, S. (2012) · Utrecht University Repository (Utrecht University) · Cited by 3
Peng, Y., Nijhuis, S., Yu, Y., Agugiaro, G., & Zhang, G. (2026) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2026.103144 · Cited by 2
CAD Learning in Mechanical Engineering at Universities
Barbero, B. R., & Pedrosa, C. M. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.24-41 · Cited by 2
EyeCrowdata: Towards a Web-based Crowdsourcing Platform for Web-related Eye-Tracking Data
Khalil, N. S., Dogruer, E., Elosta, A. K. O., Eraslan, S., Yeşilada, Y., & Harper, S. (2020) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3379157.3391304 · Cited by 2
Brante, E. W. (2013) · Acta Didactica Norge · doi:10.5617/adno.1117 · Cited by 2
Yıldızhan, B., & Ertekin, E. (2025) · Eğitim Bilimleri Dergisi · doi:10.15285/maruaebd.1457154 · Cited by 1
Gaze behavior, attention allocation, and automaticity in expert chamber music coaches
Heinsen, R. S., & Duke, R. A. (2026) · Psychology of Music · doi:10.1177/03057356251413254 · Cited by 1
An Eye Tracking Pilot Study of Optional Decision Constructs in B2C Transactional Processes
Hogan, M., Torres, A. M., & Barry, C. J. (2015) · ARAN (University of Galway Research Repository) (Ollscoil na Gaillimhe – University of Galway) · doi:10.13025/s8cc7b · Cited by 1
Keeping an Eye on How Users Perceive Optionality in Purchasing Decisions: A Pilot Study
Hogan, M., Barry, C. J., & Torres, A. M. (2016) · Cited by 1
Liu, X., Werder, K., & Maedche, A. (2020) · MethodsX · doi:10.1016/j.mex.2020.101133 · Cited by 1
Folding Agent: Parameter Visualization in 2D-to-3D Pattern Transformation
Chen, C.-H. (2020) · Computer-Aided Design and Applications · doi:10.14733/cadaps.2021.86-96 · Cited by 1
Pirttilahti, T. (2017) · Tampere University Institutional Repository (Tampere University) · Cited by 1
Knowles, T. H. (2017) · Scholarship & Creative Works - Digital UNC a service of University Libraries (University of Northern Colorado) · Cited by 1
Understanding learning within a commercial video game: A case studyPreprint
Fowler, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.01046
The Dual Pathway to Information Avoidance in Information Systems Use
Neben, T., Heinzl, A., & von der Trenck, A. (2013) · MADOC (University of Mannheim) · doi:10.13140/rg.2.1.5086.8244
Metadata
- Authors
- Hyrskykari, A., Ovaska, S., Majaranta, P., Räihä, K.-J., & Lehtinen, M.
- Affiliations
- University of Tampere (FI)
- ORCID
- Räihä, Kari-Jouko
- Venue
- Journal of Eye Movement Research
- Publisher
- MDPI AG
- Published
- 2008-11-26
- Volume
- 2
- Issue
- 4
- ISSN
- 1995-8692 (ISSN-L)
- OpenAlex
- W2469474566
- Cited by
- 118
- References
- 51
- Citations by year
- 2026: 9 · 2025: 4 · 2024: 6 · 2023: 8 · 2022: 5 · 2021: 11 · 2020: 24 · 2019: 3 · 2018: 5 · 2017: 6 · 2016: 6 · 2015: 5 · 2014: 7 · 2013: 6 · 2012: 6
- FWCI
- 3.23
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Eye Movement Research (publishedVersion, cc-by)
- Journal of Eye Movement Research (SwissUniversities) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- retrospective think-aloud, think-aloud, eye tracking, usability evaluation, concurrent think-aloud, usability testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
