How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot
Open accessAbstract
Background: Recent advancements in large language models have motivated the practical use of such models in code generation and program synthesis. However, little is known about the effects of such tools on code readability and visual attention in practice. Objective: In this paper, we focus on GitHub Copilot to address the issues of readability and visual inspection of model generated code. Readability and low complexity are vital aspects of good source code, and visual inspection of generated code is important in light of automation bias. Method: Through a human experiment (n=21) we compare model generated code to code written completely by human programmers. We use a combination of static code analysis and human annotators to assess code readability, and we use eye tracking to assess the visual inspection of code. Results: Our results suggest that model generated code is comparable in complexity and readability to code written by human pair programmers. At the same time, eye tracking data suggests, to a statistically significant level, that programmers direct less visual attention to model generated code. Conclusion: Our findings highlight that reading code is more important than ever, and programmers should beware of complacency and automation bias with model generated code.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · see citing works
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285 · see citing works
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · see citing works
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200 · see citing works
Analyzing individual performance of source code review using reviewers' eye movement
Uwano, H., Nakamura, M., Monden, A., & Matsumoto, K. (2006) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/1117309.1117357 · Cited by 177 · see citing works
How do we read algorithms? A case study
Crosby, M. E., & Stelovsky, J. (1990) · Computer · doi:10.1109/2.48797 · Cited by 176 · see citing works
Source code metrics: A systematic mapping study
Núñez-Varela, A. S., Pérez‐González, H. G., Martínez‐Pérez, F. E., & Soubervielle‐Montalvo, C. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.03.044 · Cited by 172 · see citing works
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 1× · see citing works
“which found no statistical difference in complexity between model generated code and code written by humans [27].”
BackgroundA practical guide on conducting eye tracking studies in software engineering
Sharafi, Z., Sharif, B., Guéhéneuc, Y., Begel, A., Bednarik, R., & Crosby, M. E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09829-4 · Cited by 133 · see citing works
Applications of AI in classical software engineering
Barenkamp, M., Rebstadt, J., & Thomas, O. (2020) · AI Perspectives · doi:10.1186/s42467-020-00005-4 · Cited by 123 · see citing works
Automatically assessing code understandability: How far are we?
Scalabrino, S., Bavota, G., Vendome, C., Linares‐Vásquez, M., Poshyvanyk, D., & Oliveto, R. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115654 · Cited by 101 · see citing works
Choose your programming copilot
Sobania, D., Briesch, M., & Rothlauf, F. (2022) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3512290.3528700 · Cited by 91 · see citing works
Program Comprehension and Code Complexity Metrics: An fMRI Study
Peitek, N., Apel, S., Parnin, C., Brechmann, A., & Siegmund, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00056 · Cited by 84 · see citing works
Integrating pair programming into a software development process
Williams, L. A. (2002) · doi:10.1109/csee.2001.913816 · Cited by 80 · see citing works
From Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study
Madi, N. A., Peterson, C. S., Sharif, B., & Maletic, J. I. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00025 · Cited by 20 · see citing works
iTrace: Overcoming the Limitations of Short Code Examples in Eye Tracking Experiments
Sharif, B., & Maletic, J. I. (2016) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2016.61 · Cited by 14 · see citing works
Morris, B. J., Crane, P., & Boehm, K. (1981) · Palgrave Macmillan UK eBooks · doi:10.1007/978-1-349-04549-5_15 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147 · see citing works
Practices and Challenges of Using GitHub Copilot: An Empirical Study
Zhang, B., Liang, P., Zhou, X., Ahmad, A., & Waseem, M. (2023) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2023-077 · Cited by 75 · see citing works
Cheng, R., Wang, R., Zimmermann, T., & Ford, D. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3651990 · Cited by 47 · see citing works
“Al Madi [ 2 ] assessed the readability of code produced by Copilot, compared to human-written code, and found that it was comparable.”
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 37 · see citing works
Hamer, S., d’Amorim, M., & Williams, L. A. (2024) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00014 · Cited by 37 · see citing works
Zhou, X., Liang, P., Zhang, B., Li, Z., Ahmad, A., Shahin, M., & Waseem, M. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112204 · Cited by 30 · see citing works
“Al Madi (2023) investigated the readability and visual inspection of the code generated by Copilot.”
BackgroundA Map of Exploring Human Interaction Patterns with LLM: Insights into Collaboration and Creativity
Li, J., Li, J., & Su, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60615-1_5 · Cited by 26 · see citing works
Methodology for Code Synthesis Evaluation of LLMs Presented by a Case Study of ChatGPT and Copilot
Ságodi, Z., Siket, I., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3403858 · Cited by 23 · see citing works
Assessing the Readability of ChatGPT Code Snippet Recommendations: A Comparative Study
Dantas, C., Rocha, A. M., & Maia, M. D. A. (2023) · Brazilian Symposium on Software Engineering · doi:10.1145/3613372.3613413 · Cited by 22 · see citing works
Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
Qian, C., & Wexler, J. (2024) · International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645198 · Cited by 43 · see citing works
CoLadder: Manipulating Code Generation via Multi-Level Blocks
Yen, R., Zhu, J., Suh, S., Xia, H., & Zhao, J. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676357 · Cited by 22 · see citing works
Developer Behaviors in Validating and Repairing LLM-Generated Code Using IDE and Eye Tracking
Tang, N., Chen, M., Ning, Z., Bansal, A., Huang, Y., McMillan, C., & Li, T. J.-J. (2024) · 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00015 · Cited by 18 · see citing works
Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation
Gardella, N., Pettit, R., & Riggs, S. L. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653615 · Cited by 15 · see citing works
“Previous studies either allowed internet access [35,37] or did not specify [1,21], so this choice is important to consider.”
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 26 · see citing works
BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks
Cheng, R., Barik, T., Leung, A., Hohman, F., & Nichols, J. W. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00012 · Cited by 15 · see citing works
PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
Peng, Y. F., Gotmare, A. D., Lyu, M. R.-T., Xiong, C., Savarese, S., & Sahoo, D. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00008 · Cited by 15 · see citing works
Impacts of the Usage of Generative Artificial Intelligence on Software Development Process
Santos, P. D. O., Figueiredo, A. C., Moura, P., Diirr, B., Alvim, A. C. F., & dos Santos, R. P. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658337 · Cited by 14 · see citing works
On Eye Tracking in Software Engineering
Grabinger, L., Häuser, F., Wolff, C. G., & Mottok, J. (2024) · SN Computer Science · doi:10.1007/s42979-024-03045-3 · Cited by 14 · see citing works
Investigating the readability of test code
Winkler, D., Urbanke, P., & Ramler, R. (2024) · Empirical Software Engineering · doi:10.1007/s10664-023-10390-z · Cited by 13 · see citing works
Stone, I. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689553 · Cited by 13 · see citing works
Perceived Impact of AI-Based Tooling on Software Development Code Quality
Martinović, B., & Rozić, R. (2025) · SN Computer Science · doi:10.1007/s42979-024-03608-4 · Cited by 12 · see citing works
Qiao, Y., Shihab, M. I. H., & Hundhausen, C. D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3785366 · Cited by 11 · see citing works
Khurana, A., Su, X., Wang, A. Y., & Chilana, P. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713431 · Cited by 10 · see citing works
Analyzing Developer Use of ChatGPT Generated Code in Open Source GitHub Projects
Grewal, B., Lu, W., Nadi, S., & Bezemer, C. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645072 · Cited by 10 · see citing works
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 24 · see citing works
Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects
Nunes, H., Figueiredo, E., Rocha, L., Nadi, S., Ferreira, F., & Esteves, G. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00069 · Cited by 9 · see citing works
Teaching IT Software Fundamentals: Strategies and Techniques for Inclusion of Large Language Models
Gumina, S., Dalton, T., & Gerdes, J. H. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611409 · Cited by 9 · see citing works
PwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 9 · see citing works
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5 · see citing works
Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassian
Takerngsaksiri, W., Tantithamthavorn, C. K., Fu, M., Pasuksmit, J., Chen, K., & Wu, M. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00073 · Cited by 4 · see citing works
Early Results from a Study of GenAI Adoption in a Large Brazilian Company: The Case of Globo
Pereira, G. V., Prikladnicki, R., Jackson, V., van der Hoek, A., Fortes, L., & Macaubas, I. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_13 · Cited by 5 · see citing works
Impact of AI Tools on Software Development Code Quality
Martinović, B., & Rozić, R. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-62058-4_15 · Cited by 3 · see citing works
Where Is Self-admitted Code Generated by Large Language Models on GitHub?
Yu, X., Liu, L., Hu, X., Liu, J., & Xia, X. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00047 · Cited by 3 · see citing works
How much SPACE do metrics have in GenAI assisted software development?
Sikand, S., Phokela, K. K., Sharma, V. S., Singi, K., Kaulgud, V., Tung, T., Sharma, P., & Burden, A. P. (2024) · Innovations in Software Engineering Conference (ISEC) · doi:10.1145/3641399.3641419 · Cited by 3 · see citing works
Artificially Insecure: Examining GitHub Copilot’s AI-based Vulnerability Prevention System
Armstrong, N., & Hartley, T. P. (2025) · Silicon Valley Cybersecurity Conference (SVCC) · doi:10.1109/svcc65277.2025.11133629 · Cited by 2 · see citing works
Mehmood, S., Janjua, U. I., & Ahmed, A. (2023) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit60620.2023.00013 · Cited by 2 · see citing works
Predicting Multiclass Java Code Readability: A Comparative Study of Machine Learning Algorithms
Susanto, B., Ferdiana, R., & Adji, T. B. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.01604102 · Cited by 1 · see citing works
Tives, H. A., Marioti, M. B., & Jaques, P. A. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12361 · Cited by 1 · see citing works
An Eye for AI: Eye-Tracking the Micro-Interruptions of GenAI Code Suggestions
Alakmeh, T., D’Angelo, S., & Fritz, T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773132 · Cited by 1 · see citing works
How Secure is AI-based Coding?: A Security Analysis Using STRIDE and Data Flow Diagrams
Hossain, I., Chowdhury, N. M. I., & Hasan, R. (2023) · IEEE Virtual Conference on Communications (VCC) · doi:10.1109/vcc60689.2023.10474718 · Cited by 1 · see citing works
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4 · see citing works
AthenaLLM: Supporting Experiments with Large Language Models in Software Development
de Oliveira, B., & Castor, F. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644437
Assessing Code Synthesis In LLMS: CodeGeex and Codeium
Sheoran, S., & Kumar, D. (2025) · Journal of Emerging Technologies and Innovative Research · doi:10.56975/jetir.v12i5.563406
Intelligent Virtual Advisor for Software Development
Warmsley, D., Barbieux, A., Strelnikoff, S., Levy, M., Rego, J., Kim, E., Giusto, P., Genslak, R., & Begum, L. (2025) · International Conference on Information and Computer Technologies (ICICT) · doi:10.1109/icict64582.2025.00009
OR for Everyone: Solving OR Problems as Non-experts with Generative AI
Maurischat, J., Bogs, S., Walther, G., & Kirchhof, O. (2025) · Lecture notes in operations research · doi:10.1007/978-3-031-92575-7_18
How Readable Is LLM-Generated Code Snippets? A Comparison of ChatGPT, DeepSeek, and Gemini
Fernandes, G., Maia, M. D. A., & Dantas, C. E. C. (2025) · doi:10.5753/vem.2025.14275
109PwnPilot: could an adversary be pair programming with our most trusted software engineers?
Horne, D. (2025) · De Gruyter eBooks · doi:10.1515/9783111436548-006
Enhancing Open-Source Projects: The Synergy Between Code Readability Metrics and User Experience
Khalid, A., Haneef, F., & Waseem, F. (2025) · International Journal of Agriculture & Sustainable Development · doi:10.33411/ijist/202571129145
Homophone-Based Chinese Natural Language Data Augmentation
Wei, J., Zhang, X., Witold Pedrycz, C., Wang, L., Ding, W., & Qian, X. (2026) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2026.3655163
When Generative AI Writes Test Cases: Scenario-Driven Evaluation of Generated TestsPreprint
Ardıç, B., Yildirim, I. I., Brandt, C., & Zaidman, A. (2026) · Research Square · doi:10.21203/rs.3.rs-9238547/v1
Beyond Coding: How Students Use Generative AI Across the Full Software Development Lifecycle
Matti, S., Szendrei, R., & Pintér, B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32732-1_14
- Authors
- Al Madi, N.
- Affiliations
- Computer Science, Colby College, USA
- ORCID
- Al Madi, Naser
- Venue
- Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2022-10-10
- Pages
- 1-5
- OpenAlex
- W4294432232
- Cited by
- 50
- References
- 17
- Citations by year
- 2026: 5 · 2025: 17 · 2024: 22 · 2023: 6
- FWCI
- 10.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- GitHub Copilot, code readability, visual inspection, eye tracking, automation bias, large language models, code complexity, pair programming, static code analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
