CoChat
Sign Up
PreprintCited by 4Software Engineering ResearchSoftware Engineering Techniques and PracticesUsability and User Interface Design

How do Copilot Suggestions Impact Developers' Frustration and Productivity?

Open access
Guglielmi, Emanuela, Arnoudova, Venera, Bavota, Gabriele, Oliveto, Rocco, Scalabrino, Simone·2025·arXiv
DOI 10.48550/arxiv.2504.06808arXiv 2504.06808
Open this paper in CoChat — free accountOpen PDF

Abstract

Context. AI-based development tools, such as GitHub Copilot, are transforming the software development process by offering real-time code suggestions. These tools promise to improve the productivity by reducing cognitive load and speeding up task completion. Previous exploratory studies, however, show that developers sometimes perceive the automatic suggestions as intrusive. As a result, they feel like their productivity decreased. Theory. We propose two theories on the impact of automatic suggestions on frustration and productivity. First, we hypothesize that experienced developers are frustrated from automatic suggestions (mostly from irrelevant ones), and this also negatively impacts their productivity. Second, we conjecture that novice developers benefit from automatic suggestions, which reduce the frustration caused from being stuck on a technical problem and thus increase their productivity. Objective. We plan to conduct a quasi-experimental study to test our theories. The empirical evidence we will collect will allow us to either corroborate or reject our theories. Method. We will involve at least 32 developers, both experts and novices. We will ask each of them to complete two software development tasks, one with automatic suggestions enabled and one with them disabled, allowing for within-subject comparisons. We will measure independent and dependent variables by monitoring developers' actions through an IDE plugin and screen recording. Besides, we will collect physiological data through a wearable device. We will use statistical hypothesis tests to study the effects of the treatments (i.e., automatic suggestions enabled/disabled) on the outcomes (frustration and productivity).

AI snapshot

Written from the paper’s full text.

TL;DR

This is a registered-style proposal rather than a completed study: the authors lay out two theories about how GitHub Copilot's automatic suggestions affect developers' frustration and productivity, predicting harm for experts and benefit for novices. They then specify a quasi-experimental protocol (AB/BA crossover, ≥32 developers, wearable physiological sensing, IDE instrumentation) intended to test those theories.

Key findings

No empirical results are reported; the paper presents hypotheses to be tested.

  • Theory T_E: expert developers are expected to experience automatic suggestions as interruptions, leading to higher frustration (H1_E), lower actual productivity measured as completed subtasks divided by completion time (H2_E), and lower perceived productivity (H3_E).
  • Theory T_N: novice developers are expected to benefit, with suggestions lowering frustration (H1_N), raising actual productivity (H2_N), and raising perceived productivity provided suggestion correctness is adequate (H3_N).
  • The authors note prior work they build on: Peng et al. found Copilot users finished a JavaScript task 55.8% faster, and Ziegler et al. found suggestion acceptance rate predicts perceived productivity.
  • Planned power analysis: with 50 participants the design targets 80% power to detect a 50% difference, assuming a standard deviation of 50%.

Methods

Planned design, not executed.

  • Quasi-experimental AB/BA crossover: each participant does two maintenance tasks (a Travel Management System REST API extension and a BrickBreaker Java game extension), one with Copilot automatic suggestions enabled and one disabled, with a 15-minute break and counterbalanced order; four groups balance task order and condition.
  • Sample: at least 32 developers (target up to 50), half experts (industrial developers) and half novices (CS bachelor/master students who passed basic programming exams), recruited by convenience sampling; a minimum 3-year experience gap separates the groups, with outliers removed.
  • Measures: frustration via heart-rate variability (SDNN and RMSSD) from an Empatica E4 wristband, analyzed in 30-second windows around each interruption; actual productivity as completion percentage divided by completion time; perceived productivity via post-task survey; suggestion size, alternatives, relevance, and manual Copilot invocations captured through screen recordings and the Tako VS Code plugin. Analysis uses Wilcoxon signed-rank tests with Bonferroni correction (α′ = 0.017), Cliff's delta effect sizes, and regression models if suggestion characteristics vary across developers.

Limitations

As stated by the authors:

  • The two tasks may not represent real-world coding work; they mitigated this by choosing maintenance tasks rather than from-scratch coding.
  • Motion artifacts or poor skin contact can add noise to HRV data; they expect minimal movement and will verify device fit.
  • The sample size may be insufficient to detect the theorized effects if they exist (type II error), though 50 participants is claimed to give 80% power for a 50% difference.
  • Made evident by the text: the study is prospective, so all outcomes, effect sizes, and conclusions remain unverified; carryover effects are acknowledged as a risk of the crossover design and are only mitigated by a break, counterbalancing, and post-hoc statistical checks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.06808

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

19 of 36 cited sources checked · 9 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Higher variations in HRV indicate greater frustration during task execution.

    “Higher variations in HRV indicate greater frustration during task execution [6].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] 6. Y. Ma, Y. Huang, and K. Leach, "Breaking the flow: A study of interruptions during software engineering activities," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1-12. source

      Section 4.2 (page unavailable) interprets increases in the HRV measures SDNN and RMSSD as indicating *decreased* stress, contrary to the claimed direction. The paper assesses stress rather than establishing a relationship between HRV and frustration. (LLM, conf=0.84)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Developers are frustrated by incorrect or overly large suggestions that interrupt their workflow.

    “Prather et al. [11] show that developers are frustrated by incorrect or overly large suggestions that interrupt their workflow”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Prather et al. [11] 11. J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos, ""it's weird that it knows what i want": Usability and interactions with copilot for novice programmers," ACM Trans. Comput.-Hum. Interact., vol. 31, no. 1, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3617367 source

      The Interaction Pattern: Drifting section (p. 11) reports that incorrect code, large suggestions, and a constant stream of suggestions disrupted participants’ problem solving. Design Implications (p. 19) also says unwanted suggestions frustrated students and slowed them down. The study concerns novice programmers, so the unqualified claim about developers is broader than its findings. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Premisecitation

    Interruptions interrupt cognitive flow.

    “Indeed, Ma et al. [6] showed that interruptions significantly increase stress levels, interrupt cognitive flow, and lengthen task completion times.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] 6. Y. Ma, Y. Huang, and K. Leach, "Breaking the flow: A study of interruptions during software engineering activities," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1-12. source

      The Introduction says some developers may be distracted by interruptions and have difficulty resuming work, which supports the idea of disrupted focus. It frames this as a possibility, however, rather than reporting a finding that interruptions disrupt cognitive flow. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation

    Interruptions significantly increase stress levels.

    “Indeed, Ma et al. [6] showed that interruptions significantly increase stress levels, interrupt cognitive flow, and lengthen task completion times.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] 6. Y. Ma, Y. Huang, and K. Leach, "Breaking the flow: A study of interruptions during software engineering activities," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1-12. source

      Section 4.3 reports significantly higher self-reported stress during in-person interruptions than during on-screen interruptions. The claim overgeneralizes that finding: the comparison is between interruption types, and the Conclusion reports that physiological measures indicated reduced stress during in-person interruptions. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    Copilot's suggestions are perceived as invasive by experienced developers.

    “Peng et al. [12] showed that Copilot's suggestions are perceived as invasive by experienced developer.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [12] 12. S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, "The impact of ai on developer productivity: Evidence from github copilot," arXiv preprint arXiv:2302.06590, 2023. source

      The Results section reports that less experienced developers benefited more from Copilot, and the exit survey asked about productivity gains. Neither passage reports whether experienced developers perceived its suggestions as invasive; searches for invasive or intrusive suggestions found no such finding in our copy. The passages have no page numbers. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 33
  • May combine two claims. 11

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    AI-assisted development tools (AIDTS), such as GitHub Copilot, are transforming software development by offering real-time code suggestions.

    “AI-assisted development tools (AIDTS), such as GitHub Copilot, are transforming the way software is developed by offering real-time code suggestions [2].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [2] 2. N. A. Ernst and G. Bavota, "Ai-driven development is here: Should you worry?" IEEE Software, vol. 39, no. 2, pp. 106-110, 2022. source

  • Classified as probably not a substantial claim.

    Premisecitation

    Increased frustration leads to a negative attitude.

    “As pointed out in the literature increased frustration leads to a negative attitude [11] and that negative emotions reduce productivity [13].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [11] 11. J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos, ""it's weird that it knows what i want": Usability and interactions with copilot for novice programmers," ACM Trans. Comput.-Hum. Interact., vol. 31, no. 1, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3617367 source

      Section 4.2 (p. 11) reports that negative sentiment occurred when novice programmers were frustrated or annoyed while using Copilot. Section 4.2.2 (p. 13) gives examples of frustration, but does not establish that increasing frustration causes a negative attitude. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The surveys were designed following the indications by Ziegler et al. [5] for collecting developer information and perceived productivity.

    “We designed such surveys following the indications by Ziegler et al. [5], both for the collection of developer information and perceived productivity.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ziegler et al. [5] 5. A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, "Productivity assessment of neural code completion," in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, ser. MAPS 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 21-29. [Online]. Available: https://doi.org/10.1145/3520312.3534864 source

  • Classified as probably not a substantial claim.

    Premisecitation

    The perceived invasiveness of Copilot's suggestions is influenced by specific aspects of the suggestions, such as the size of suggested code blocks and the availability of alternative suggestions, which affect the ability of developers to choose the most fitting suggestion for their code.

    “They also showed that the perceived invasiveness is also influenced by specific aspects of the suggestions, such as the size of suggested code blocks and the availability of alternative suggestions, which affect the ability of developers to choose the most fitting suggestion for their code [11].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [11] 11. J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos, ""it's weird that it knows what i want": Usability and interactions with copilot for novice programmers," ACM Trans. Comput.-Hum. Interact., vol. 31, no. 1, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3617367 source

      In “Interaction Pattern: Drifting” (p. 11) and “Design Implications” (pp. 19–20), the paper reports that unsolicited suggestions distracted novices and that large code blocks took longer to decipher. It proposes letting users cycle through alternative suggestions, but does not show that the availability of alternatives affected perceived invasiveness or developers’ ability to choose the best-fitting code. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Frustration occurs when suggestions do not satisfy functional or non-functional requirements.

    “Liang et al. [18] found that frustration occurs when the suggestions do not satisfy functional or non-functional requirements.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liang et al. [18] 18. J. T. Liang, C. Yang, and B. A. Myers, "A large-scale survey on the usability of ai programming assistants: Successes and challenges," in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1-13. source

      Section 5.5, “Giving up on outputted code” (page not available in our copy), reports that participants gave up on generated code when it failed to meet functional or non-functional requirements. It does not establish that they experienced frustration, which is the response the citing sentence attributes to the study. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Frustration is caused both by insufficient help and invasive and premature suggestions not aligned with developers' intentions.

    “Wang et al. [34] report that frustration is caused both by insufficient help and invasive and premature suggestions not aligned with developers' intentions.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al. [34] 34. R. Wang, R. Cheng, D. Ford, and T. Zimmermann, "Investigating and designing for trust in ai-powered code generation tools," in The 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024, pp. 1475-1493. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Individuals with more pronounced affective reactivity to stressors had lower levels of SDNN and RMSSD.

    “Punita et al. [23] suggested that SDNN and RMSSD were reduced with increased intensity of frustration, and Sin et al. [24] found that individuals with more pronounced affective reactivity to stressors had lower levels of SDNN and RMSSD.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [24] 24. N. L. Sin, R. P. Sloan, P. S. McKinley, and D. M. Almeida, "Linking daily stress processes and laboratory-based heart rate variability in a national sample of midlife and older adults," Psychosomatic medicine, vol. 78, no. 5, pp. 573-582, 2016. source

      The Abstract Results and Results regression section report that greater affective reactivity to stressors was associated with lower RMSSD and lower SDRR. The cited paper calls the other measure SDRR, not SDNN, so it does not directly establish the claim for SDNN. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A combination of in-person and on-screen interruptions influenced productivity during code review.

    “and a combination of in-person and on-screen interruptions influenced productivity during code review.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ma et al. [6] 6. Y. Ma, Y. Huang, and K. Leach, "Breaking the flow: A study of interruptions during software engineering activities," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1-12. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The acceptance rate of code suggestions provided by GitHub Copilot is a significant predictor of developers' perceived productivity.

    “Their results show the acceptance rate of code suggestions provided by GitHub Copilot is a significant predictor of developers' perceived productivity.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Ziegler et al. [5] 5. A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, "Productivity assessment of neural code completion," in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, ser. MAPS 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 21-29. [Online]. Available: https://doi.org/10.1145/3520312.3534864 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    AI-assisted development tools promise to improve developer productivity by reducing cognitive effort and speeding up task completion.

    “These tools promise to improve developer productivity by reducing cognitive effort and speeding up task completion [3]- [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]- [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Stress plays a crucial role in cognitive performance, task efficiency, and overall well-being of developers.

    “Stress plays a crucial role in cognitive performance, task efficiency, and overall well-being of developers [11], [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11], [13]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The planned sample size (>32 developers) might not be sufficient to observe the theorized phenomena even if they existed (type II error).

    “The planned sample size (>32 developers) might not be sufficient to observe the phenomena we theorized even if it existed (type II error).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Developers are assigned to groups randomly with equal numbers of expert and novice developers in each group.

    “assign developers to the groups beforehand in a random way but making sure that each group contains the same amount of expert and novice developers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Participants often struggled to understand, edit, and debug the code generated by Copilot.

    “and participants often struggled to understand, edit, and debug the generated code.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Vaithilingam et al. [4] 4. P. Vaithilingam, T. Zhang, and E. L. Glassman, "Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models," in Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1-7. source

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The connection of the Empatica E4 device and signal quality are verified through the E4 real-time application to ensure accurate data capture.

    “The connection of the device and the signal quality are verified through the E4 real-time application to ensure accurate data capture.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    Frustration at a given point in time t is affected not only by the immediate interruption, but also by previous interruptions within the considered window.

    “This means that frustration at a given point in time t is affected not only by the immediate interruption, but also by previous interruptions within the considered window.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The post-task survey complements objective performance metrics such as task correctness and completion time by capturing subjective experiences.

    “By capturing these subjective experiences, the survey complements objective performance metrics such as task correctness and completion time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A fifteen-minute break is provided between the two tasks to ensure participants are not fatigued.

    “To ensure participants are not fatigued, a fifteen minute break is provided between the two tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    SDNN and RMSSD were reduced with increased intensity of frustration.

    “Punita et al. [23] suggested that SDNN and RMSSD were reduced with increased intensity of frustration, and Sin et al. [24] found that individuals with more pronounced affective reactivity to stressors had lower levels of SDNN and RMSSD.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] 23. P. Punita, K. Saranya, and S. S. Kumar, "Gender difference in heart rate variability in medical students and association with the level of stress," National Journal of Physiology, Pharmacy and Pharmacology, vol. 6, no. 5, pp. 431-431, 1970.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Automatic suggestions might increase developers' frustration, especially if they are irrelevant or require significant adaptation.

    “As such, paradoxically, they might disrupt developers' cognitive flow, increase their frustration [7]- [9] (especially if the they are irrelevant or require significant adaptation), and reduce their productivity [10], [11].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [7]- [9]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    The soothing video calibration approach has been demonstrated to return participants' physiological features to a baseline level after approximately one minute.

    “This approach was chosen as it has been demonstrated to effectively return participants' physiological features to a baseline level after approximately one minute.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fritz et al. [30] 30. T. Fritz, A. Begel, S. C. Müller, S. Yigit-Elliott, and M. Züger, "Using psycho-physiological measures to assess task difficulty in software development," in Proceedings of the 36th international conference on software engineering, 2014, pp. 402-413. source

  • May combine two claims.

    Premisecitation

    Automatic suggestions can mostly be perceived as interruptions by expert developers, reducing their focus and imposing a cognitive load on them as they review, accept, or reject them.

    “As previously mentioned, automatic suggestions, while potentially beneficial in some cases, can mostly be perceived as interruptions by expert developers, reducing their focus and imposing a cognitive load on them as they review, accept, or reject them [10], [11].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10], [11]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The choice of tasks may not fully represent coding tasks encountered in real work contexts.

    “The first limitation is related to the choice of the tasks, which may not fully represent coding tasks encountered in real work contexts.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The experimental procedure allows evaluation of both objective productivity metrics (correctness and time) and subjective perceptions of productivity.

    “This procedure allows us to evaluate both objective productivity metrics, such as correctness and time, and subjective perceptions of productivity, facilitating a comprehensive analysis of Copilot's impact under different settings.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The post-analysis review of overlapping time windows allows determination of whether residual frustration from prior interruptions influences the results.

    “This allows us to determine whether residual frustration from prior interruptions influences our results.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Negative emotions reduce productivity.

    “As pointed out in the literature increased frustration leads to a negative attitude [11] and that negative emotions reduce productivity [13].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] 13. F. Nizamudeen, L. Gatti, N. Bouali, and F. Ahmed, "Investigating the impact of code generation tools (chatgpt & github copilot) on programming education." in CSEDU (2), 2024, pp. 221-229. source

  • Classified as probably not a substantial claim.

    Premisecitation

    Being in the situation of constantly needing and requesting help might increase the frustration of developers.

    “We expect that being in the situation of constantly needing and requesting help might increase the frustration of developers [9], [10], [15].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [9], [10], [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The study conducts a post-analysis review of overlapping time windows to mitigate potential carryover effects when multiple interactions occur in close succession.

    “To mitigate potential carryover effects when multiple interactions occur in close succession, we conduct a post-analysis review of overlapping time windows.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The study design includes a 15-minute break between tasks and counterbalanced order of treatments to mitigate carryover effects.

    “To mitigate this, we implemented a 15-minute break between tasks and counterbalanced the order of treatments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An early rejection mechanism using a Transformer-based lightweight estimator significantly reduces computational costs and unhelpful completions.

    “Sun et al. [35] proposed an early rejection mechanism using a Transformer-based lightweight estimator, significantly reducing computational costs and unhelpful completions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Sun et al. [35] 35. Z. Sun, X. Du, F. Song, S. Wang, M. Ni, and L. Li, "Don't complete it! preventing unhelpful code completion for productive and sustainable neural code completion systems," in 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE- Companion). IEEE, 2023, pp. 324-325. source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The study administers two maintenance tasks in which participants do not need to write code from scratch, aiming to make the tasks more realistic.

    “To address this limitation, we decided to administer two maintenance tasks in which the participants do not need to write code from scratch, which aims to make the tasks more realistic.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Copilot provides useful starting points and reduces the need for online research, but does not significantly improve task completion times or success rates.

    “Vaithilingam et al. [4] found that Copilot, while providing useful starting points and reducing the need for online research, did not significantly improve task completion times or success rates”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Vaithilingam et al. [4] 4. P. Vaithilingam, T. Zhang, and E. L. Glassman, "Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models," in Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1-7. source

  • Probably not a substantial claim: may not pass S1 (asserted).

    Premisecitation

    Heart-rate measures will be positively correlated with the negative emotions experienced during frustration.

    “Yuan et al. [25] further predict that heart-rate measures will be positively correlated with the negative emotions experienced during frustration.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] 25. J. Yuan, N. Ding, Y. Liu, and J. Yang, "Unconscious emotion regula- tion: Nonconscious reappraisal decreases emotion-related physiological reactivity during frustration," Cognition and Emotion, vol. 29, no. 6, pp. 1042-1053, 2015. source

  • Classified as probably not a substantial claim.

    Premisereasoning

    There is a potential for carryover effects, where the experience from the first condition might influence the second.

    “However, we acknowledge the potential for carryover effects, where the experience from the first condition might influence the second.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In software engineering, interruptions play a crucial role in shaping developers' perceptions of productivity and the quality of their workday.

    “In software engineering, interruptions play a crucial role in shaping developers' perceptions of productivity and the quality of their workday [41], [42].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41], [42]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Interruptions can increase stress, frustration, time pressure, and the effort required to complete core cognitive tasks.

    “Interruptions can increase stress, frustration, time pressure, and the effort required to complete core cognitive tasks [39], [40].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39], [40]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

36 of 36

  • The paper relies on this without support.

    Novice developers might more often be in a blocked state in which they require additional knowledge to keep coding (e.g., APIs or coding patterns).

    “Novice developers might find more often in a blocked state in which they require additional knowledge to keep coding (e.g., APIs or coding patterns).”

  • The authors state this as an assumption.

    The size of suggested code blocks might increase the invasiveness of the suggestion and thus increase novice developers' frustration.

    “Note that, as for expert developers, the size of suggested code blocks might increase the invasiveness of the suggestion and, thus, increase also the novice developers' frustration [11].”

  • The authors state this as an assumption.

    Having many alternative suggestions could increase the frustration of developers because they are required to actively make a choice they might not be able to make.

    “On the other hard, having many alternative suggestions, in this case, could increase the frustration of developers since they are required to actively make a choice they might not be able to make [11].”

  • The paper relies on this without support.

    An increase in RMSSD indicates a decrease in stress measures.

    “An increase in RMSSD indicates a decrease in stress measures.”

  • The paper relies on this without support.

    Stress induced by interruptions during task execution contributes to frustration.

    “Our study specifically focuses on stress induced by interruptions during task execution, which has been shown to contribute to frustration.”

  • The paper relies on this without support.

    Incorrect suggestions might make even novice developers feel like they are wasting their time.

    “Incorrect suggestions might make even novice developers feel like they are wasting their time.”

  • The authors state this as an assumption.

    The suggestions provided in similar code project contexts will not vary much in terms of SUGG size, SUGG alt, and SUGG rel.

    “we expect that from a similar context in terms of code projects the suggestions provided will not vary much in terms of such variables.”

  • The authors state this as an assumption.

    Participants' motion during the task will be minimal, resulting in minimal motion artifacts.

    “Given the task at hand, we expect that participants' motion will be minimal (as so the related artifacts).”

  • The paper relies on this without support.

    Motion artifacts or inconsistent skin contact can cause noise in the HRV measurements.

    “Motion artifacts or inconsistent skin contact can cause noise in the measurements.”
9 of 9

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. [14]14. A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, "Productivity assessment of neural code completion," in Proceedings of the 6th ACM SIGPLAN Interna- tional Symposium on Machine Programming, 2022, pp. 21-29.
  2. [31]31. F. Wilcoxon, S. Katti, R. A. Wilcox et al., "Critical values and probabil- ity levels for the wilcoxon rank sum test and the wilcoxon signed rank test," Selected tables in mathematical statistics, vol. 1, pp. 171-259, 1970.

Cited by

Sorted by citations

Related

Authors
Guglielmi, E., Arnoudova, V., Bavota, G., Oliveto, R., & Scalabrino, S.
Affiliations
  • University of Molise (IT)
  • Washington State University (US)
  • Università della Svizzera italiana (CH)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.06808
arXiv
2504.06808
OpenAlex
W4417247943
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
GitHub Copilot, developer productivity, developer experience, cognitive load, wearable physiological monitoring

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review