The DOI matches its Crossref record; 80 of the 81 references it cites were found in a registry or an index; none of the 109 sources its claims cite is checked for support yet.
Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment
Open accessAbstract
Simulations and games bring the possibility to research complex processes of managerial decision-making. However, this modern field requires adequate methodological procedures. Many authors recommend the use of a combination of concurrent think-aloud (CTA) or retrospective think-aloud (RTA) with eye-tracking to investigate cognitive processes such as decision-making. Nevertheless, previous studies have little or no consideration of the possible differential impact of both think-aloud methods on data provided by eye-tracking. Therefore, the main aim of this study is to compare and assess if and how these methods differ in terms of their impact on eye-tracking. The experiment was conducted for this purpose. Participants were 14 managers who played a specific simulation game with CTA use and 17 managers who played the same game with RTA use. The results empirically prove that CTA significantly distorts data provided by eye-tracking, whereas data gathered when RTA is used, provide independent pieces of evidence about the participants’ behavior. These findings suggest that RTA is more suitable for combined use with eye-tracking for the purpose of the research of decision-making in the game environment.
Overview
CoChat Summary
TL;DR
In a between-subjects experiment with managers playing FactOrEasy, concurrent think-aloud (CTA) significantly distorted eye-tracking data, whereas retrospective think-aloud (RTA) did not. The authors conclude RTA is more suitable for combined use with eye-tracking in game-based decision-making research.
Key findings
- CTA (14 managers) and RTA (17 managers) showed no significant differences in game performance: net cash t(48) = −0.447, p = 0.657; overall standings U = 632, p = 0.844; bankruptcies z = 0.512, p < 0.610.
- Overall eye-tracking: number of fixations and total fixation duration were higher for CTA than RTA (p < 0.001); average fixation duration and dwell time ratio were lower for CTA than RTA (p < 0.001).
- Over game rounds, CTA had more fixations than RTA in all rounds except the last; CTA trend was linear (R² = 0.75) with a decrease of 6.37 fixations per round, while RTA trend was logarithmic (R² = 0.79) with faster decrease in early rounds.
- Verbal data: CTA verbalized AOI content in 89.1% of cases vs. RTA in 72.4%; omitted information was 27.2% for RTA vs. 10.4% for CTA (valid information z = 8.988, p < 0.001; omitted information z = 9.095, p < 0.001); unidentified information was CTA 0.5%, RTA 0.4%, with no fabricated information found.
Methods
- Between-subjects lab experiment: 34 managers initially, 3 excluded; final CTA n = 14 (9 men, 5 women, average age 34.8 years), RTA n = 17 (14 men, 3 women, average age 30.7 years). Participants played the FactOrEasy simulation game individually; CTA verbalized concurrently, RTA verbalized at the end of each round.
- Eye-tracking used a Tobii X2-60 with 5-point calibration; analysis focused on the Product Sale phase segments (288 CTA, 329 RTA) using number of fixations, average and total fixation duration, and a custom dwell time ratio.
- Statistical comparisons used Shapiro-Wilk for normality, t-test/Mann-Whitney for game results, Mann-Whitney for eye-tracking metrics, and Z-test for proportions for bankruptcy and verbalization categories.
Limitations
- Paper states: results are limited to the specific FactOrEasy game; different games may produce different eye-tracking results. The study provides only a basic assessment of verbal protocols, with no accurate text coding of transcripts, and does not draw conclusions about specific decision-making processes or skills.
- Paper states: CTA prompts were moved to the end of game rounds and RTA verbalizations occurred at the end of each round without replaying recordings, adaptations that may affect comparability.
- Evident: final groups were small (14 and 17 managers) with uneven gender distribution, limiting generalizability; additionally, after the 18th round less than half of CTA segments were included, and the authors consider explanatory power in later rounds low.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 3 Not valid · 43 Not checked · 124 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No statistically significant difference was found between the “net cash” (t(48) = −0.447; p = 0.657), the player’s overall standing (U = 632; p = 0.844), and the number of bankruptcies (z = 0.512; p < 0.610).”
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 3. Results > 3.1. Comparison of Game Results Achieved by RTA and CTA Players
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No statistically significant difference was found between the “net cash” (t(48) = −0.447; p = 0.657), the player’s overall standing (U = 632; p = 0.844), and the number of bankruptcies (z = 0.512; p < 0.610).”
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 3. Results > 3.1. Comparison of Game Results Achieved by RTA and CTA Players
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No statistically significant difference was found between the “net cash” (t(48) = −0.447; p = 0.657), the player’s overall standing (U = 632; p = 0.844), and the number of bankruptcies (z = 0.512; p < 0.610).”
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 3. Results > 3.1. Comparison of Game Results Achieved by RTA and CTA Players
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of fixations and total fixation duration are higher for CTA than for RTA (p < 0.001).”
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 3. Results > 3.2. Overall Comparison of the Impact of CTA and RTA on Eye-Tracking
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No statistically significant difference was found between these two categories (unidentified information: z = 0.189; p = 0.85, fabricated information: z and p are not available).”
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 3. Results > 3.4. Comparison of the Data Gathered by CTA and RTA
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
87 more claims
Create a free account to see all 170 claims and to verify the 109 references still open.
- It may state two separate claims as one. 50
- Probably not a substantial claim. 36
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 3. Results > 3.3. Comparison of the Impact of CTA and RTA on Eye-Tracking over Time (in the Game Progress)
“The CTA group has a lower total fixation duration than the RTA group in all rounds, except the first and the last one.”
§ Impact of Think-Aloud on Eye-Tracking: A Comparison of Concurrent and Retrospective Think-Aloud for Research on Decision-Making in the Game Environment > 4. Discussion > 4.2. Comparison of the Impact of CTA and RTA on Eye-Tracking over Time (in the Game Progress)
“CTA brings higher values of the number of fixations and total fixation duration during the opening rounds than RTA.”
Citations
Cites
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979
A theory of reading: From eye fixations to comprehension.
Just, M. A., & Carpenter, P. A. (1980) · Psychological Review · doi:10.1037/0033-295x.87.4.329 · Cited by 3,887
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Rayner, K. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210902816461 · Cited by 2,682
Attention and choice: A review on eye movements in decision making
Orquin, J. L., & Mueller Loose, S. (2013) · Acta Psychologica · doi:10.1016/j.actpsy.2013.06.003 · Cited by 940
Knowledge management, decision-making style and organizational performance
Abubakar, A. M., Elrehail, H., Alatailat, M. A., & Elçi, A. (2019) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2017.07.003 · Cited by 856
Thinking aloud: reconciling theory and practice
Boren, T., & Ramey, J. (2000) · IEEE Transactions on Professional Communication · doi:10.1109/47.867942 · Cited by 808
Gegenfurtner, A., Lehtinen, E., & Säljö, R. (2011) · Educational Psychology Review · doi:10.1007/s10648-011-9174-7 · Cited by 715
A review of using eye-tracking technology in exploring learning from 2000 to 2012
Lai, M.-L., Tsai, M.-J., Yang, F.-Y., Hsu, C.-Y., Liu, T.-C., Lee, S. W.-Y., Lee, M.-H., Chiou, G.-L., Liang, J.-C., & Tsai, C.-C. (2013) · Educational Research Review · doi:10.1016/j.edurev.2013.10.001 · Cited by 626
Varieties of knowledge elicitation techniques
Cooke, N. J. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1083 · Cited by 579
Research works printed but not linked: 7
- Prokop, 2019, Evaluation of managerial and decision-making skills of small-scale farmers, Agrarian Perspectives XXVIII, Business Scale in Relation to Economics, Proceedings of the 28th International Scientific Conference, Prague, Czech Republic, 18–19 September 2019, Volume 28, 218
- Holmqvist, K., Nystrom, M., Andersson, R., Dewhurst, R., Jarodzka, H., and van de Weijer, J. (2015). Eye Tracking: A Comprehensive Guide to Methods and Measures, Oxford University Press. First Published in Paperback.
- Russo, J.E. (2011). Eye fixations as a process trace. A Handbook of Process Tracing Methods for Decision Research: A Critical Review and User’s Guide, Psychology Press.
- Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., and Carey, T. (1994). Human-Computer Interaction, Addison-Wesley Longman Ltd.
- Švec, V., Pavlicek, J., Ticha, I., and Kreckova, J. (2016, January 2–3). FactOrEasy (c): ART AND CRAFT OF MANAGEMENT?. Proceedings of the 13th International Conference Efficiency and Responsibility in Education, Prague, Czech Republic.
- Prokop, M., and Švec, V. (2018, January 7–8). Flawless Managers: Myth or Reality?. Proceedings of the 15th International Conference Efficiency and Responsibility in Education 2018 (ERIE), Prague, Czech Republic.
- Švec, V., Flegl, M., Pavlicek, J., Prokop, M., and Kreckova, J. (2017, January 8–9). Factoreasy (r): Do Students Take a Risk?. Proceedings of the 14th International Conference Efficiency and Responsibility in Education 2017 (ERIE), Prague, Czech Republic.
Cited by
Prompt engineering as a new 21st century skill
Federiakin, D. A., Molerov, D., Zlatkin‐Troitschanskaia, O., & Maur, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1366434 · Cited by 231
Eye tracking in physics education research: A systematic literature review
Hahn, L., & Klein, P. (2022) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.18.013102 · Cited by 92
Eye Tracking Research in Cartography: Looking into the Future
Krassanakis, V., & Cybulski, P. (2021) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi10060411 · Cited by 47
Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic Review
Hertzum, M. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3665327 · Cited by 41
The Think-Aloud Method for Evaluating the Usability of a Regional Atlas
Vaníček, T., & Popelka, S. (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12030095 · Cited by 37
Metadata
- Authors
- Prokop, M., Pilař, L., & Tichá, I.
- Affiliations
- Czech University of Life Sciences Prague (CZ)
- Corresponding author
- Prokop, Michal
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2020-05-12
- Volume
- 20
- Issue
- 10
- Pages
- 2750
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC7294419
- OpenAlex
- W3024223677
- Cited by
- 48
- References
- 81
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 10 · 2023: 11 · 2022: 5 · 2021: 5 · 2020: 2
- FWCI
- 3.46
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- IGA PEF CZU (CULS) Prague (2019A0001)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- think-aloud, concurrent think-aloud, retrospective think-aloud, eye tracking, management decision-making, simulation games, data distortion, cognitive processes, methodological procedures
- MeSH terms
- Eye-Tracking Technology, Decision Making, Games, Experimental, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
