Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code

Classified as probably not a substantial claim.

We mitigate conclusion-validity threats by following established principles: domain expertise pooling [45], maximum entropy criterion [46], and uninformative priors [36].

“We mitigate them by following established principles, i.e., domain expertise pooling [45], maximum entropy criterion [46], and uninformative priors [36].”

• cites [45] [45] M. M. Glymour and S. Greenland, 'Causal diagrams,' in Modern Epidemiology , 3rd ed., K. J. Rothman, S. Greenland, and T. L. Lash, Eds. Philadelphia, PA: Lippincott Williams & Wilkins, 2008, pp. 183209. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [46] [46] E. T. Jaynes, Probability Theory: The Logic of Science . Cambridge University Press, 2012.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [36] [36] J. S. Wesner and J. P. Pomeranz, 'Choosing priors in Bayesian ecological models by simulating from the prior predictive distribution,' Ecosphere , vol. 12, no. 9, pp. e03 739:1-e03 739:11, 2021. - sourceDisputed

  The citation [36] is attached to “uninformative priors.” In the Introduction, the paper argues against treating non-informative priors as a sound default; the Discussion says that evaluating priors against domain knowledge is better than seeking the least informative prior. (LLM, conf=0.92)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Beller et al. present a taxonomy of changes resulting from modern code review and show that the majority of these changes are evolvability changes, while the remaining 15% are functional changes.

“Beller et al. [12] present a taxonomy of changes resulting from modern code review and show that the majority of these changes are evolvability changes (e.g., structure and documentation), while the remaining 15% are functional changes (e.g., larger defects and logic).”

• cites [12] [12] M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens, 'Modern code reviews in open-source projects: which problems do they fix?' in Proceedings of the 11th Working Conference on Mining Software Repositories , ser. MSR 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 202-211. [Online]. Available: https://doi.org/10.1145/2597073.2597082 - sourcePartly supported

  The Code review fixes section presents a taxonomy distinguishing evolvability from functional changes. Section 5.1 reports that most changes were evolvability-related, but gives the functional share as 25%, not the claimed 15%. (LLM, conf=0.93)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Prior predictive checks confirmed the appropriateness of the priors, and the training process converged with no diagnostic issues.

“Afterwards, we perform prior predictive checks to confirm their appropriateness [25]. Once assessing that the training process converged accordingly [37] and no diagnostic suggested issues [25], we evaluate the model.”

• cites [25] [25] R. McElreath, Statistical rethinking: A Bayesian course with examples in R and Stan . Boca Raton, FL, USA: Chapman and Hall/CRC, 2018. - sourceNot checked — no full text of the source
• cites [37] [37] A. Gelman and D. B. Rubin, 'Inference from iterative simulation using multiple sequences,' Statistical science , vol. 7, no. 4, pp. 457-472, 1992. - sourcePartly supported

  Section 3.7 (pp. 8–9) describes a diagnostic for monitoring convergence, which supports citing this paper for that method. It does not describe prior predictive checks or establish that the citing authors’ priors were appropriate or that their model had no diagnostic issues. The sentence attributes the convergence assessment to a different citation, [37]. (LLM, conf=0.80)

  Experimental — accuracy not yet measured

May combine two claims.

Bayesian data analysis coupled with the causal inference framework allows differentiating correlations from causal effects and drawing more reliable conclusions.

“Moreover, BDA coupled with our causal inference framework allows us to differentiate correlations from causal effects [27] and draw more reliable conclusions [30].”

• cites [27] [27] J. Pearl, Causality , ser. Causality: Models, Reasoning, and Inference. Cambridge, UK: Cambridge University Press, 2009. [Online]. Available: https://books.google.se/books?id=f4nuexsNVZIC - sourceNot checked — no full text of the source
• cites [30] [30] C. A. Furia, R. Torkar, and R. Feldt, 'Towards causal analysis of empirical software engineering data: The impact of programming languages on coding competitions,' ACM Transactions on Software Engineering and Methodology , vol. 33, no. 1, pp. 1-35, 2023. - sourceSupported

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

May combine two claims.

We mitigate crossover-design threats by adjusting for them in our analyses [18], [44], which factors out effects from the phenomenon of interest and allows us to determine that they did not affect our experiment.

“We mitigate these by adjusting for them in our analyses [18], [44], which not only factors out the effects from the phenomenon of interest but also allows us to determine that they did not affect our experiment.”

• cites [18] [18] S. Vegas, C. Apa, and N. Juristo, 'Crossover designs in software engineering experiments: Benefits and perils,' IEEE Transactions on Software Engineering , vol. 42, no. 2, pp. 120-135, 2016. - sourceNot checked — no full text of the source
• cites [44] [44] J. Frattini, D. Fucci, and S. Vegas, 'Crossover designs in software engineering experiments: Review of the state of analysis,' in 2024 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , ACM. New York, NY, USA: Association for Computing Machinery, 2024, pp. 482-488. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Nahar et al. [42] highlighted needs regarding standardising responsible AI evaluation to mitigate ethical and legal issues for organisations.

“Such needs were also highlighted by Nahar et al. [42] regarding standardising responsible AI evaluation to mitigate ethical and legal issues for organisations.”

• cites [42] [42] N. Nahar, C. K¨ astner, J. Butler, C. Parnin, T. Zimmermann., and C. Bird, 'Beyond the comfort zone: Emerging solutions to overcome challenges in integrating llms into software products,' in 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . New York, NY, USA: Association for Computing Machinery, 2025, pp. 516-527. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Research envisions a change in the role of a software developer to focus more on code verification [40], [41].

“This is particularly important since research envisions a change in the role of a software developer to focus more on code verification [40], [41].”

• cites [40] [40] S. Ferino, R. Hoda, J. Grundy, and C. Treude, 'Walking the tightrope of llms for software development: A practitioners' perspective,' 2025. [Online]. Available: https://arxiv.org/abs/2511.06428 - sourcePartly supported

  The paper’s recommendations section (“3. https://h2o.ai”; page unavailable) calls for human review of LLM-generated code before it is merged, supporting the code-verification aspect. Its Implications for Research section also describes developers overseeing autonomous agents. Neither passage says that developers’ roles will shift to focus more on code verification specifically. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [41] [41] A. Simkute, L. Tankelevitch, V. Kewenig, A. E. Scott, A. Sellen, and S. Rintel, 'Ironies of generative ai: Understanding and mitigating productivity loss in human-ai interaction,' International Journal of Human-Computer Interaction , vol. 41, no. 5, pp. 2898-2919, 2025. [Online]. Available: https://doi.org/10.1080/10447318.2024.2405782 - sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

To support construct validity, we use well-established eye-tracking metrics that capture visual attention [8].

“To support construct validity , we use well-established eye-tracking metrics that capture visual attention [8].”

• cites [8] [8] L. Grabinger, N. Al Madi, R. Bednarik, T. Busjahn, F. Engl, T. Ezer, H. Gruber, F. Hauser, J. I. Maletic, U. Obaidellah, K.-i. Park, B. Sharif, Z. Sharafi, L. Shepherd, and J. Mottok, 'A Cookbook for Eye Tracking in Software Engineering,' in Proceedings of the 6th European Conference on Software Engineering Education , ser. ECSEE '25. New York, NY, USA: Association for Computing Machinery, 2025, p. 60-76. [Online]. Available: https://doi.org/10.1145/3723010.3723018 - sourceSupported

May combine two claims.

Review times for labelled code are either comparable to or longer than those for the same unlabelled code.

“Second, review times for labelled code are either comparable to or longer than those for the same unlabelled code.”

May combine two claims.

14 participants used the prompt during their review, treating it either as a requirement to validate against or as contextual documentation.

“Independent of this, 14 participants used the prompt during their review, treating it either as a requirement to validate against or as contextual documentation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For reviewing 6 lines of code in sessions.py (AOI 1) with an average time of 15 seconds, an additional 5 seconds is approximately a 33% increase in reviewing duration.

“For instance, reviewing 6 lines of code in sessions.py (AOI 1) with an average time of 15 seconds, an additional 5 seconds is approximately a 33% increase in reviewing duration.”

May combine two claims.

The difference in review times for labelled code is more pronounced, up to 15 seconds, for code that requires more time to review (more than 20 seconds), and is negligible otherwise.

“This difference is more pronounced (up to 15 seconds) for code that requires more time to review (more than 20 seconds) and is negligible otherwise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants acknowledged that the margin of faults that LLMs produce grows with the size of the code.

“Some participants also acknowledged that the margin of faults that LLMs produce grows with the size of the code.”

May combine two claims.

Many participants reported that they did not treat LLM-labelled code any differently and followed the same reviewing strategy they used for code written by human developers, while being 'careful'.

“In fact, many of them reported that they did not treat LLM-labelled code any differently and followed the same reviewing strategy they used for code written by human developers, just while being 'careful'.”

May combine two claims.

The effect of the LLM labels on saccade length is nonsignificant, as the confidence interval overlaps with zero.

“The CI overlaps with zero, indicating a nonsignificant effect of the LLM labels.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In utils.py, the developer spent around two minutes reviewing the first labelled code segment (seconds 80-200), revisited the second labelled segment around line 37 at second 500, whereas in unlabelled storage.py they returned to the beginning of the file at line 1 at second 350.

“In the utils.py fi le (bottom), the developer spent around two minutes reviewing the first labelled code segment (gaze spanning between seconds 80-200). They also revisited the second labelled segment later in the review (around line 37 at second 500), whereas in the unlabelled storage.py fi le (top), they instead returned to the beginning of the file (line 1 at second 350).”

May combine two claims.

This resulted in a difference in behaviour in terms of review time only for more complex code snippets, with no big difference in saccade lengths, that is, the thoroughness of the review.

“This resulted in a difference in the behaviour (in terms of review time) only for more complex code snippets (see Figure 5) with no big difference in the saccade lengths, that is, the thoroughness of the review.”

May combine two claims.

Participants who expressed relatively higher trust in AI tended to expect fewer logical errors in LLM-generated code, but more issues related to code quality or consistency.

“Participants who expressed relatively higher trust in AI tended to expect fewer logical errors in LLM-generated code, but more issues related to code quality or consistency.”

May combine two claims.

This gaze pattern was consistent across participants in this group and reflects a shift from understanding the logic of the code to verifying whether the code generation worked as intended.

“This gaze pattern was consistent across participants in this group and reflects a shift in the nature of the task: from understanding the logic of the code to verifying whether the code generation worked as intended.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prompt was useful to understand LLM-generated code that tends to be unnecessarily complex.

“The prompt was also useful to understand LLM-generated code that tends to be unnecessarily complex.”

May combine two claims.

These participants often expressed a general mistrust of LLMs, possibly reinforced by previous experiences with LLMs and broad company AI policies that offer little concrete guidance.

“In their interviews, these participants often expressed a general mistrust of LLMs, possibly reinforced by previous experiences with LLMs and the fact that their companies' AI policies are broad and offer little concrete guidance on the use of LLM-generated code.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants described that such faults are more likely to occur at the integration level rather than at the unit level.

“One reason that some participants described is that such faults are more likely to occur at the integration level rather than at the unit level.”

Classified as probably not a substantial claim. May combine two claims.

Participants expressed the importance of knowing the original intent behind generated code, especially because LLMs do not always produce code that clearly reflects what was asked.

“More specifically, our participants expressed the importance of knowing the original intent behind the generated code, especially because LLMs do not always produce code that clearly reflects what was asked.”

• AssumptionRests on an assumption: LLMs do not always produce code that clearly reflects what was asked.

Probably not a substantial claim: may not pass S3 (load-bearing).

These expectations were often based on prior experiences with LLM-based tools, which participants described as prone to misunderstanding prompts and producing semantically incorrect code.

“These expectations were often based on prior experiences with LLMbased tools, which they described as prone to misunderstanding prompts and producing semantically incorrect code.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Storing prompts as code comments or any code artifact is not ideal, as this may conflict with company coding practices or clean code principles [43].

“We also learned that storing prompts as code comments or any code artifact is not ideal (this may, for example, conflict with company coding practices or clean code principles [43]).”

• cites [43] [43] R. C. Martin, Clean code: a handbook of agile software craftsmanship . Hoboken, NJ, USA: Pearson Education, 2009. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For most areas of interest the effect on saccade length is comparable, with a 2-3 pixel difference, but for the two shortest codes (AOI 5 in temperature.py, 4 lines, and AOI 6 in utils.py, 2 lines) there are gaps in opposite directions.

“While the effect is comparable for most areas of interest, with a difference of 2-3 pixels, we notice gaps in the opposite directions for the two shortest codes (AOI 5 in temperature.py , 4 lines, and AOI 6 in utils.py , 2 lines).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Sometimes this meant checking if the code aligns with the intent of the developer, and other times it reflected focusing on a specific aspect related to LLMs' limitations, e.g., code quality.

“Sometimes, this meant checking if the code aligns with the intent of the developer, and other times it reflects in focusing on a specific aspect related to LLMs' limitations, e.g., code quality.”

May combine two claims.

Among those who reported using the prompt in their review (n=14), two groups emerged: participants who treated the prompt as a requirement description (n=5) and those who treated it as documentation (n=7); two participants reported using both.

“From the interviews, among those who reported using the prompt in their review (n=14), two groups emerged: (i) participants who treated the prompt as a requirement description (n=5), and (ii) those who treated it as documentation (n=7). Two participants reported using both.”

Probably not a substantial claim: may not pass S1 (asserted). May combine two claims.

A software developer believed that human logical errors are usually captured by tests, but that LLM mistakes are broader and many can go undetected by unit tests and linters.

“They believe that the logical errors that a human developer can make are usually captured by the tests. However, they believe that 'the range of mistakes [LLMs] do is much broader than a human' , and many of them can go undetected by unit tests and linters.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants reported reviewing PRs from known or more senior colleagues less thoroughly.

“some participants reported reviewing PRs from known or more senior colleagues less thoroughly”

Probably not a substantial claim: may not pass S3 (load-bearing).

The observed 20-25 pixel difference in saccade lengths corresponds roughly to the length of a single word.

“the 20-25 pixel difference we observe corresponds roughly to the length of a single word”

May combine two claims.

This participant's gaze reflects the general tendency among higher-trust participants: a similar reviewing flow across labelled and unlabelled files, moving top-to-bottom, but with more time spent on LLM-labelled segments.

“While individual gaze paths varied, this participant's gaze (Figure 7) reflects the general tendency among highertrust participants: a similar reviewing flow across labelled and unlabelled files, moving from top to bottom through the code, but with more time spent on LLM-labelled segments.”

May combine two claims.

Participants who treated the prompt as a requirement adjusted their review behaviour, repeatedly comparing the code against the prompt to verify whether the generated output matched the intended specification, resulting in a characteristic back-and-forth gaze pattern between the label and the corresponding code segments.

“Participants who treated the prompt as a requirement adjusted their review behaviour accordingly. Rather than reviewing the code in isolation, they repeatedly compared the code against the prompt to verify whether the generated output matched the intended specification. This resulted in a characteristic back-and-forth gaze pattern between the label and the corresponding code segments.”

May combine two claims.

The label-first strategy was common among participants in this group, although the number of passes through the file varied.

“While the number of passes through the file varied, this label-first strategy was common among participants in this group.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The lognormal distribution was selected for saccade lengths because it had a better fit than the gamma distribution.

“As the data-generating process of saccade lengths is less clear ontologically, we compare models using gamma and lognormal distributions and select the latter for its better fit [33].”

• cites [33] [33] A. Vehtari, A. Gelman, and J. Gabry, 'Practical bayesian model evaluation using leave-one-out cross-validation and waic,' Statistics and computing , vol. 27, pp. 1413-1432, 2017. - sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

May combine two claims.

The label details, particularly the prompt and the LLM used, were unexpectedly important to many participants, revealing that prompt-to-code traceability is valuable during code review.

“However, we found that these details, particularly the prompt and the LLM used, were unexpectedly important to many participants, revealing that prompt-to-code traceability is valuable during code review.”

May combine two claims.

The entire code, whether labelled or not, was reviewed with approximately the same thoroughness, with 85-110 pixel length scans.

“The entire code (whether labelled or not) was reviewed in approximately the same thoroughness (with 85-110 pixel length scans).”

May combine two claims.

Gaze data showed that while developers reviewed LLM-generated code as thoroughly as human-written ones, they planned their time for reviewing such code based on perceived complexity.

“Similarly, the gaze data showed that while developers reviewed LLM-generated code as thoroughly as human-written ones, they planned their time for reviewing such code based on perceived complexity.”

May combine two claims.

Participants who treated the prompt as documentation used it to support their understanding of the code rather than as a reference for validation; their review behaviour typically followed a label-first pattern, reading the prompt first and then inspecting the code.

“In contrast, participants who treated the prompt as documentation used it to support their understanding of the code, rather than a reference for validation. Their review behaviour typically followed a label-first pattern , where they first read the prompt to form an understanding of the functionality, then inspected the code.”

May combine two claims.

A security engineer with 15 years of experience exhibited fewer shifts between label and code, fixated on the prompt first and then on a cluster of fixations on the labelled code, suggesting a sequential process of understanding followed by inspection; the prompt was treated as a code artifact and assessed with the rest of the code.

“Figure 10 shows an example of this behaviour for a security engineer with 15 years of experience. Unlike the requirementbased pattern, this participant exhibits fewer shifts between label and code. Instead, they fixate on the prompt first (yellow), followed by a cluster of fixations on the labelled code (purple), suggesting a sequential process of understanding followed by inspection. In this case”

May combine two claims.

A software test manager with strong trust reviewed two unlabelled PRs in less time than two with labelled codes, but focused on reviewing the labelled code in terms of quality over logical correctness.

“For example, one of the participants who expressed a strong trust towards LLMs is a software test manager (gaze paths in Figure 7). Despite their trust, they still reviewed the two unlabelled PRs (presuming they have no LLM-generated content) in less time than the two with labelled codes. However, they focused on reviewing the labelled code in terms of quality over logical correctness.”

Classified as probably not a substantial claim.

By avoiding dependencies across multiple files or mixed changes, potential confounding factors that might lead participants to prioritize certain parts of the code for reasons unrelated to the presence of the LLM label were reduced.

“By avoiding dependencies across multiple files or mixed changes, we reduced potential confounding factors that might lead participants to prioritize certain parts of the code for reasons unrelated to the presence of the LLM label.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Organisations often establish policies that govern the use of LLM-generated code.

“To manage these limitations, organisations often establish policies that govern the use of LLM-generated code [3].”

• cites [3] [3] R. Khojah, M. Mohamad, L. Erlenhov, F. G. de Oliveira Neto, and P. Leitner, 'Large language model company policies and policy implications in software organizations,' IEEE Software , vol. 43, no. 1, pp. 64-72, 2026. - sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Injected faults and code smells were distributed across labelled and unlabelled code segments to mitigate biases caused by unequal concentrations of issues.

“These were distributed across labelled and unlabelled code segments to mitigate biases caused by unequal concentrations of issues.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The experimental environment was controlled for unwanted factors that could disturb eye-tracking measurements, such as screen brightness and external lighting.

“We first set up the experimental environment by controlling unwanted factors that could disturb eye-tracking measurements, such as screen brightness and external lighting.”

May combine two claims.

File review order and prior exposure to labelled code did not influence fixation durations.

“This suggests that file review order and prior exposure to labelled code did not influence fixation durations, supporting the validity of the counterbalanced design.”

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

Probably not a substantial claim: may not pass S3 (load-bearing).

Code comprehension and understandability are a central challenge in effective code review.

“In addition, code comprehension and understandability have been found to be a central challenge in effective code review [5]”

• cites [5] [5] A. Bacchelli and C. Bird, 'Expectations, outcomes, and challenges of modern code review,' in 2013 35th International Conference on Software Engineering (ICSE) . San Francisco, CA, USA: IEEE, 2013, pp. 712721. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Labelling code as machine-generated alters both attention and trust, with reviewers shifting their gaze from code to tests when patches are presented as automated.

“Bertram et al. [17] showed that labelling code as machine-generated alters both attention and trust, with reviewers shifting their gaze from code to tests when patches are presented as automated.”

• cites [17] [17] I. Bertram, J. Hong, Y. Huang, W. Weimer, and Z. Sharafi, 'Trustworthiness perceptions in code review: An eye-tracking study,' in Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , ser. ESEM '20. New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3382494.3422164 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Policies often highlight the need for code review and verification when generating code, but do not specify how the reviews should be conducted.

“More specifically, these policies often highlight the need for code review and verification when generating code, but do not specify how the reviews should be conducted.”

• cites [3] [3] R. Khojah, M. Mohamad, L. Erlenhov, F. G. de Oliveira Neto, and P. Leitner, 'Large language model company policies and policy implications in software organizations,' IEEE Software , vol. 43, no. 1, pp. 64-72, 2026. - sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

The pull request files are representative of real-world code because they are based on Python code from two popular open-source projects.

“To ensure the pull request files are representative of real-world code, we base the files in the pull requests on Python code from two popular opensource projects”

Classified as probably not a substantial claim.

Code provenance plays a role in shaping how developers allocate attention during review.

“These studies suggest that code provenance plays a role in shaping how developers allocate attention during review.”

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

Classified as probably not a substantial claim. May combine two claims.

The code used is entirely written by human developers because it comes from official versions of the repositories before the release of ChatGPT and GitHub Copilot (before October 2021).

“To ensure that the code we use is entirely written by human developers, we used the official version of the repositories before the release of ChatGPT and GitHub Copilot (i.e., before October 2021).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The qualitative analysis involved multiple authors and sessions, with disagreements resolved on the spot.

“Three authors jointly analysed three interviews in an initial session and established initial themes related to trust, reviewing strategy, and self-reported behavioural changes. The remaining interviews were analysed by four authors across six sessions. Each session involved two authors, where any disagreements were discussed and resolved on the spot.”

Classified as probably not a substantial claim.

The order of pull requests and label assignments was rotated across participants so that the same areas of interest were reviewed both with and without a label by different participants.

“The order of pull requests and label assignments was rotated across participants so that the same areas of interest were reviewed both with and without a label by different participants.”

Classified as probably not a substantial claim.

In the context of code review, saccade length is a suitable proxy for the thoroughness of a reviewer's inspection.

“Consequently, in the context of code review, saccade length is a suitable proxy for the thoroughness of a reviewer's inspection.”

• cites [9] [9] R. Biedert, J. Hees, A. Dengel, and G. Buscher, 'A robust realtime reading-skimming classifier,' in Proceedings of the Symposium on Eye Tracking Research and Applications , ser. ETRA '12. New York, NY, USA: Association for Computing Machinery, 2012, p. 123-130. [Online]. Available: https://doi.org/10.1145/2168556.2168575 - sourceNot checked — no full text of the source
• cites [10] [10] K. Rayner, 'Eye movements and attention in reading, scene perception, and visual search,' The quarterly journal of experimental psychology , vol. 62, no. 8, pp. 1457-1506, 2009. - sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

LLMs struggle with code-related tasks, particularly in terms of correctness, code quality, and other context-specific limitations.

“However, LLMs struggle with code-related tasks, particularly in terms of correctness, code quality, and other context-specific limitations [11].”

• cites [11] [11] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, 'Large language models for software engineering: A systematic literature review,' ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 8, Dec. 2024. [Online]. Available: https://doi.org/10.1145/3695988 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a correlation between fixation duration on buggy code and the number of bugs found.

“Chandrika et al. [16] found that there is a correlation between the fixation duration on buggy code and the number of bugs found.”

• cites [16] [16] K. R. Chandrika, J. Amudha, and S. D. Sudarsan, 'Recognizing eye tracking traits for source code review,' in 2017 22nd IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) . San Francisco, CA, USA: IEEE, 2017, pp. 1-8. - sourceNot checked — no full text of the source

May combine two claims.

Practitioners adapted their review strategy for LLM-labelled code by assessing the code based on specific criteria (e.g., logical correctness), or using the prompt to guide their review.

“Practitioners also adapted their review strategy for LLM-labelled code by assessing the code based on specific criteria (e.g., logical correctness), or using the prompt to guide their review.”

May combine two claims.

The counterbalanced design was valid, supported by the lack of credible period and carryover effects.

“This suggests that file review order and prior exposure to labelled code did not influence fixation durations, supporting the validity of the counterbalanced design.”

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

May combine two claims.

There is no meaningful difference when reviewing very short code, but the time tax imposed by reviewing LLM contributions grows significantly the more complex the code becomes.

“Our results suggest that there is no meaningful difference when reviewing very short code, but the time tax imposed by reviewing LLM contributions grows significantly the more complex the code becomes.”

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

Classified as probably not a substantial claim.

The complexity of LLM-generated code may be unnecessary.

“since it is expected that the complexity of LLM-generated code may be unnecessary.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Logical correctness does not impact total review time, but following a readability rule (minimise nesting) decreases the duration of fixations when reviewing code.

“Park et al. [14] conducted an eye tracking study on code comprehension and found that logical correctness does not impact the total review time, but following a readability rule (minimise nesting) decreases the duration of fixations when reviewing the code.”

• cites [14] [14] K.-i. Park, J. Johnson, C. S. Peterson, N. Yedla, I. Baysinger, J. Aponte, and B. Sharif, 'An eye tracking study assessing source code readability rules for program comprehension,' Empirical Software Engineering , vol. 29, no. 6, p. 160, 2024. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Reviewers often require specific information, such as code context and correct understanding, to conduct a thorough review.

“where reviewers often require specific information, such as code context and correct understanding, to conduct a thorough review [13].”

• cites [13] [13] L. Pascarella, D. Spadini, F. Palomba, M. Bruntink, and A. Bacchelli, 'Information needs in contemporary code review,' Proc. ACM Hum.Comput. Interact. , vol. 2, no. CSCW, Nov. 2018. [Online]. Available: https://doi.org/10.1145/3274404 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

To reduce obtrusiveness, participants reviewed in a quiet room with one author seated facing away.

“To reduce obtrusiveness, participants reviewed in a quiet room with one author seated facing away.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such a scenario is not unrealistic, as some introduce entirely new modules or features.

“Despite this, such a scenario is not unrealistic, as some introduce entirely new modules or features.”

May combine two claims.

The period at which a participant reviewed a particular AOI can potentially affect participants' behavior through a maturation or exhaustion effect, making it necessary to condition on it to avoid bias.

“In our study, for example, the period (at what time a participant reviewed a particular AOI) can potentially affect participants' behavior in what is called the maturation or exhaustion effect [18], making it necessary to condition on it to avoid bias.”

• cites [18] [18] S. Vegas, C. Apa, and N. Juristo, 'Crossover designs in software engineering experiments: Benefits and perils,' IEEE Transactions on Software Engineering , vol. 42, no. 2, pp. 120-135, 2016. - sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

May combine two claims.

Although participants perceived their reviewing approach as consistent, the presence of the LLM label subtly influenced their attention.

“This suggests that, although participants perceived their reviewing approach as consistent, the presence of the LLM label subtly influenced their attention.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The distribution of the outcome variable fixation duration is modeled as a gamma distribution because it emerges from the sum of individual measurements.

“We model the distribution of the outcome variable fi xation duration as a gamma distribution, as it emerges from the sum of individual measurements [32].”

• cites [32] [32] B. Mukherjee, A. Gupta, and S. Upadhyay, 'A bayesian study for the comparison of generalized gamma model with its components,' Sankhya B , vol. 72, no. 2, pp. 154-174, 2010. - sourceNot checked — no full text of the source

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

Probably not a substantial claim: may not pass S3 (load-bearing).

We are unable to mitigate the threat from these unmeasurable moderators.

“We are unable to mitigate the threat from these unmeasurable moderators.”

May combine two claims.

Bertram et al. [17] found no significant differences in saccades when reviewing machine-generated versus human-written code, with saccades around 100 pixels on average.

“This also aligns with the work by Bertram et al. [17] on average saccade lengths when reviewing machine-generated code as opposed to human-written code. There were no significant differences in the saccades, and they were around 100 pixels on average.”

• cites [17] [17] I. Bertram, J. Hong, Y. Huang, W. Weimer, and Z. Sharafi, 'Trustworthiness perceptions in code review: An eye-tracking study,' in Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , ser. ESEM '20. New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3382494.3422164 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The credibility intervals are large due to individual differences between participants, such as reading speed, which is expected.

“First, the CIs are large due to individual differences between participants (e.g., with regard to their reading speed), which is expected.”

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

May combine two claims.

When reviewers use the prompt during code review, they adopt two approaches: as a requirement to validate alignment between intent and code, or as documentation to support code comprehension; these interpretations shift the focus of review from code alone to the relationship between code and its perceived origin.

“Takeaway: When reviewers use the prompt during code review, they adopt two approaches: as a requirement to validate alignment between intent and code, or as documentation to support code comprehension. These interpretations shift the focus of review from code alone to the relationship between code and its perceived origin.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These two interpretations reflect different ways of using the prompt during a review.

“These two interpretations reflect different ways of using the prompt during a review.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tang et al. [15] observed similar behaviour when developers repaired code generated by GitHub Copilot.

“A similar behaviour was observed by Tang et al. [15] when developers repaired code generated by GitHub Copilot.”

• cites [15] [15] N. Tang, M. Chen, Z. Ning, A. Bansal, Y. Huang, C. McMillan, and T. J.J. Li, 'Developer behaviors in validating and repairing llm-generated code using ide and eye tracking,' in 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . San Francisco, CA, USA: IEEE, 2024, pp. 40-46. - sourceNot checked — no full text of the source

May combine two claims.

Covariates such as area of interest do not act as confounders but should still be included to improve the precision of the effect estimation by accounting for systematic variation across code segments.

“Additionally, covariates such as area of interest do not act as confounders but AOIs should still be included to improve the precision of the effect estimation by accounting for systematic variation across code segments [31].”

• cites [31] [31] C. Cinelli, A. Forney, and J. Pearl, 'A crash course in good and bad controls,' Sociological Methods & Research , vol. 53, no. 3, pp. 10711104, 2024. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionThe authors state this as an assumption: The causal relationships represented in the directed acyclic graph are assumed.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The PRs used in the study may not fully reflect real-world code review environments, but we selected code from large and active open-source projects to ensure representation of collaborative development observed in the software industry.

“Moreover, while the PRs used in the study may not fully reflect real-world code review environments, we selected code from large and active open-source projects to ensure representation of collaborative development that we observe in the software industry.”

Classified as probably not a substantial claim. May combine two claims.

Software organisations set AI policies that mandate a more 'careful' verification of LLM-generated code, but there is no specification of what this means in practice.

“While software organisations set AI policies that mandate a more 'careful' verification of LLMgenerated code, there is no specification of what this means in practice.”

May combine two claims.

The relationship between trust and reviewing behaviour is asymmetric; practitioners who trust LLMs do not necessarily spend less time reviewing LLM-generated code, nor do they review it less thoroughly.

“We learned that the relationship between trust and the reviewing behaviour is asymmetric. In other words, practitioners who trust LLMs do not necessarily spend less time reviewing LLM-generated code, nor do they review it less thoroughly.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tang et al. [15] observed that developers similarly shifted between prompt and code to resolve mismatches between intended and generated behaviour.

“Which is in line with prior observations by Tang et al. [15], where developers similarly shifted between prompt and code to resolve mismatches between intended and generated behaviour.”

• cites [15] [15] N. Tang, M. Chen, Z. Ning, A. Bansal, Y. Huang, C. McMillan, and T. J.J. Li, 'Developer behaviors in validating and repairing llm-generated code using ide and eye tracking,' in 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . San Francisco, CA, USA: IEEE, 2024, pp. 40-46. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Participants may behave differently under eye-tracking equipment than in natural settings.

“Additionally, participants may behave differently under eyetracking equipment than in natural settings.”

Classified as probably not a substantial claim.

To reduce the risk of hypothesis guessing, participants were invited to take part in a code review study without being informed that LLM-generated code was involved.

“In addition, to reduce the risk of hypothesis guessing, participants were invited to take part in a code review study without being informed that LLM-generated code was involved.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When treated as documentation, the prompt serves as contextual information to support understanding, shifting the focus to comprehension.

“In contrast, when treated as documentation , the prompt serves as contextual information to support understanding, shifting the focus to comprehension , asking: 'What is this code doing?' .”

Probably not a substantial claim: may not pass S2 (checkable).

For multi-turn interactions, the full conversation history, not just a single prompt, needs to be preserved.

“This is particularly relevant for multi-turn interactions, where the full conversation history (not just a single prompt) needs to be preserved.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

This is an inherent trade-off of controlled experiments needed to isolate the effect of interest.

“This is an inherent trade-off of controlled experiments needed to isolate the effect of interest.”

May combine two claims.

Participant statements about more careful reviews of LLM contributions appear to translate into longer reviewing time, but not more thorough reviews.

“Participant statements regarding more careful reviews of LLM contributions appear to translate into longer reviewing time, but not more thorough reviews.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants reviewed a reduced version of the code files and were described as entirely new, whereas real PRs typically contain a mix of added, deleted, and modified lines.

“However, participants still reviewed a reduced version of these code files and were described as entirely new, whereas real PRs typically contain a mix of added, deleted, and modified lines.”

May combine two claims.

Participants following a consistent reviewing strategy still showed an unconscious attentional pull toward LLM-labelled code, highlighting a gap between stated reviewing intent and actual behaviour.

“Takeaway: Participants following a consistent reviewing strategy still showed an unconscious attentional pull toward LLMlabelled code, highlighting a gap between stated reviewing intent and actual behaviour.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We define our DAG to scope our causal assumptions explicitly and confine it to observable variables.

“Therefore, we define our DAG (see Figure 4) to scope our causal assumptions explicitly and confine it to observable variables.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

External validity may be threatened by missing unobserved variables; for example, participants' stance towards LLM-generated code may affect their trust and moderate the effect of the LLM-label.

“The external validity may be threatened by missing unobserved variables. For example, the participants' stance towards LLM-generated code may affect their trust and, hence, moderate the effect of the LLM-label.”

May combine two claims.

Reviewers adjusted what they look for in LLM-labelled code; depending on their trust in AI, they prioritize logical correctness or code quality, showing that prior expectations shape evaluation criteria even when the overall review process remains similar.

“Takeaway: Reviewers adjusted what they look for in LLMlabelled code. Depending on their trust in AI, they prioritize logical correctness or code quality, showing that prior expectations shape evaluation criteria, even when the overall review process remains similar.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Any subjective step in the BDA, such as DAGs, distribution family for outcome variables, and prior probability distributions, implies a potential threat to conclusion validity.

“Any subjective step in the BDA, e.g., DAGs, distribution family for outcome variables, and prior probability distributions, implies a potential threat to conclusion validity .”

Probably not a substantial claim: may not pass S3 (load-bearing).

We triangulate our analysis by combining quantitative measures, gaze paths, and qualitative data from exit interviews.

“We triangulate our analysis by combining quantitative measures, gaze paths, and qualitative data from exit interviews.”

May combine two claims.

All code in the study was written by human developers; we employed a WoZ design in which no code was actually LLM-generated.

“Crucially, all code in our study was written by human developers: we employed a WoZ design in which no code was actually LLM-generated.”

May combine two claims.

While developers are often distrustful of LLM-generated code and spend more time reviewing it, this additional time does not translate into a more thorough review as measured via saccade lengths.

“We conclude that, while developers are often distrustful of LLM-generated code and spend more time reviewing it, this additional time does not translate into a more thorough review as measured via saccade lengths.”

• AssumptionRests on an assumption: Shorter saccades indicate a more thorough line-by-line review, while longer saccades indicate broader scans.