Students' Use of GitHub Copilot for Working with Large Code Bases
Open accessThe DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Large language models (LLMs) are already heavily used by professional software engineers. An important skill for new university graduates to possess will be the ability to use such LLMs to effectively navigate and modify a large code base. While much of the prior work related to LLMs in computing education focuses on novice programmers learning to code, less work has focused on how upper-division students use and trust these tools, especially while working with large code bases. In this study, we taught students about various GitHub Copilot features, including Copilot chat, in an upper-division software engineering course and asked students to add a feature to a large code base using Copilot. Our analysis revealed a novel interaction pattern that we call one-shot prompting, in which students ask Copilot to implement the entire feature at once and spend the next few prompts asking Copilot to debug the code or asking Copilot to regenerate its incorrect response. Finally, students reported significantly more trust in the code comprehension features than code generation features of Copilot, perhaps due to the presence of trust affordances in the Copilot chat that are absent in the code generation features. Our study takes the first steps in understanding how upper-division students use Github Copilot so that our instruction can adequately prepare students for a career in software engineering.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Generative Artificial Intelligence (AI) tools such as ChatGPT, GitHub Copilot, and Gemini have become embedded in the daily workflow of graduate students [12, 17, 19].”
From Students' Practices and Skills in the LLM-Era: "You Can't Outsource the Struggle and Still Get the Skill" (2026) · cites this paper as [19]
In the citing paper
“Furthermore, our finding that Copilot reduced manual coding comports with Shah et. al.'s [36] observation that students perform 'one-shot prompting' to have Copilot implement entire features at once.”
From The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Coding Tasks (2025) · cites this paper as [36]
Citations
Cites
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Ko, A. J., Myers, B. A., Coblenz, M. J., & Aung, H. H. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.116 · Cited by 685 · cited 1×
“Given that prior studies place the amount of time dedicated to code navigation at 35% [11] and program comprehension up to 70% of a developer’s time to a modify a large code base [15], these results illustrate the potential for LLMs to alter the program comprehension process.”
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Ozkaya, I. (2023) · IEEE Software · doi:10.1109/ms.2023.3248401 · Cited by 206 · cited 1×
“Given the importance of LLMs in industry, there is little doubt that students should learn the best practices for working with these tools [16].”
Investigating the skill gap between graduating students and industry expectations
Radermacher, A., Walia, G., & Knudson, D. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591159 · Cited by 199
Struggles of new college graduates in their first software development job
Begel, A., & Simon, B. (2008) · Proceedings of the 39th SIGCSE technical symposium on Computer science education · doi:10.1145/1352135.1352218 · Cited by 193
Research works printed but not linked: 3
- Mateusz Jaworski and Dariusz Piotrkowski. 2023. Study of software developers' experience using the Github Copilot Tool in the software development process. ArXiv, Vol. abs/2301.04991 (2023). https://api.semanticscholar.org/CorpusID:255749277 · arXiv
- Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arxiv: 2302.06590 · arXiv
- 10.1007/978--94--6351-086--8_4 · doi:10.1007/978--94--6351-086--8_4
Cited by
Exploring Student-AI Interactions in Vibe Coding
Geng, F., Shah, A., Li, H., Mulla, N., Swanson, S., Raj, G. S., Zingaro, D., & Porter, L. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786236 · Cited by 51
Self-citationBernstein, S., Rahman, A., Sharifi, N., Terbish, A., & MacNeil, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770036 · Cited by 38
Shihab, M. I. H., Hundhausen, C., Tariq, A., Haque, S., Qiao, Y., & Mulanda, B. W. (2025) · Proceedings of the 2025 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3702652.3744219 · Cited by 28
Kharrufa, A., Alghamdi, S. S., Aziz, A., & Bull, C. (2026) · ACM Transactions on Computing Education · doi:10.1145/3779296 · Cited by 37
“Shah et al. [48], on the other hand, looked at the use and trust of LLMs by “upper-division” students.”
"My productivity is boosted, but …" Demystifying Users’ Perception on AI Coding Assistants
Lyu, Y., Yang, Z., Shi, J., Chang, J., Liu, Y., & Lo, D. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00024 · Cited by 25
“Few studies examine the usability of chat [32], [58], and none systematically study agent interfaces.”
The Rest of the Robots: Generative AI in Post-introductory Computing Education
Bouvier, D. J., Cipriano, B. P., Glassey, R. J., Petrovska, O., Anderson, E., Birillo, A., Dougherty, R. E., Pettit, R., Pombo, N. G. C. C., Rahimi, E., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3760545.3783970 · Cited by 21
“In multiple studies the motivation for the inclusion of GenAI tools was to provide authentic learning for post-introductory students to prepare them for the types of task they may encounter in the workplace [22, 42, 111, 122].”
Chen, N., Qiu, L. K., Wang, A. Z., Wang, Z., & Yang, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790726 · Cited by 17
“Prior work on AI code assistants suggests that these tools can improve productivity and learning [46, 80, 94], while also introducing challenges related to developers’ understanding, control, and trust in AI-generated outputs [13, 31, 52, 78, 79].”
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Evolution of Programmers' Trust in Generative AI Programming Assistants
Shah, A., Rexin, T., Tomson, E., Griswold, W. G., Porter, L., & Raj, A. G. S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770029 · Cited by 13
“Students demonstrated significantly more trust in the code comprehension features than the code generation features [31], though the authors did not include any qualitative analysis to understand why those trust attitudes formed.”
InfluentialSelf-citationPădurean, V.-A., Gotovos, A., Ghosh, A., Denny, P. C., Leinonen, J., Luxton-Reilly, A., Prather, J., & Singla, A. K. (2026) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3803400.3809312 · Cited by 6
Usage, Effects and Requirements for AI Coding Assistants in the Enterprise: An Empirical Study
Merler, M., Pan, R., Krishna, R., Ho, T. K., Pavuluri, R., & Vuković, M. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788727 · Cited by 6
Needles in a Haystack: Student Struggles with Working on Large Code Bases
Shah, A., Rexin, T., Chernova, A., Allen-Perez, G., Griswold, W. G., & Raj, A. G. S. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744218 · Cited by 7
Self-citationObserving Without Doing: Pseudo-Apprenticeship Patterns in Student LLM Use
Hak, J., Johnson, N. L., Amoozadeh, M., Alipour, M. A., & Chattopadhyay, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770027 · Cited by 10
Pacing for Mastery: Optimizing LLM Interactions for Learning
Tran, K., Gao, G., Lombard, A., Yu, T. T., Jiang, H., & Yeh, T. Y. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772501 · Cited by 5
One Line at a Time: Scaffolding Reflective Code Evaluation through Structured AI Assistance
Qiu, V., Parker, L., Riegel, K., Giacaman, N., Denny, P. C., MacNeil, S., & Prather, J. E. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786243 · Cited by 4
“As a result, computing education researchers are increasingly studying how students use these tools to complete programming assignments [32].”
Engineering Students' Usage and Perceptions of GitHub Copilot in Open-Source Projects
Rani, N., Munnangi, J. R., Spangler, A. M., & Honeycutt, D. R. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59352 · Cited by 2
Wang, Y., Rizi, M. B., Khashei, F., & Rubin, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797092 · Cited by 2
“Alpizar-Chacon and Keuning [25] and Shah et al. [63] investigated characteristics of students’ prompts, showing that students often prompt once to generate the entire feature, followed by additional prompts to debug or regenerate code.”
Process Management for Learning from Professional Source Code: Cultivating Experts in the Age of AI
Dasu, G., O’Rourke, E., Zhang, H., Whitely, M., Wilcox, R. G., Rosenberg, L., Mulat, M., Liu, J. L., & Brewster, N. (2026) · ACM Transactions on Computing Education · doi:10.1145/3796525 · Cited by 1
Roy, N., Horielko, O., & Omojokun, O. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772551 · Cited by 1
“For instance, Harvard’s CS50 course deployed a suite of custom AI tools at scale [14], and other studies have examined student use of Copilot in large codebases [10].”
Alwarafy, A. (2026) · European Journal of Education · doi:10.1111/ejed.70761 · Cited by 2
Kohn, T., Antensteiner, T., Hahnel, C., Kern, B., Keuning, H., Kia, E., Kiesler, N., Komm, D., Marinus, E., Šavelka, J., et al. (2026) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3856208.3856233 · Cited by 2
Mapping the Emerging Curriculum for AI-Assisted Software Engineering via Syllabus AnalysisPreprint
Geng, F., Shah, A., Chen, M., Denny, P. C., Leinonen, J., Griswold, B., Raj, G. S., & Porter, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05898 · Cited by 1
Self-citationde Freitas, E. R. S., Pinto, G., & Monteiro, D. (2026) · arXiv · doi:10.48550/arxiv.2607.29519
“Generative Artificial Intelligence (AI) tools such as ChatGPT, GitHub Copilot, and Gemini have become embedded in the daily workflow of graduate students [12, 17, 19].”
Impact of AI Debugging on Student Code Complexity and Readability: A Literature Review
Thang, H. V., & Vi, T. L. (2025) · International STEM Education Conference (iSTEM-Ed) · doi:10.1109/istem-ed65612.2025.11129285
“Shah et al. reported that final-year undergraduates working with very large legacy code repositories relied on Copilot most frequently for boilerplate generation and search-and-replace refactors, behaviours that mirror the time-pressured practices typical of bootcamps [20].”
Богун, Р. І., Одрібець, Н. В., & VEDENIEIEVA, O. (2025) · Інфокомунікаційні та комп’ютерні технології · doi:10.36994/2788-5518-2025-01-09-09
Zuñiga, A. M., Coronado, E. J. D., & Méndez, R. V. M. (2025) · Ciencia Latina Revista Científica Multidisciplinar · doi:10.37811/cl_rcm.v9i6.21442
Evaluating AI-Generated C# Code in Computing Education: Implications for Academic Integrity
Adegbite, A. A., & Kotzé, E. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-12457-9_8
COMPUTER TECHNOLOGIES: EDUCATION AND SCIENCE
Таланчук, П. (2025) · doi:10.36994/978-966-388-727-2-2025-105
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · doi:10.1145/3760545
Carinan, C. J. M., Lucero, H. R., Navarette, M. N., Ortiz, K. U., Paroni, J. K. S., & Duyag, C. B. C. (2026) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2026.150500182
AI or Ally? Understanding Student Reliance on GitHub Copilot and Human Peers Through Eye Tracking
Salib, G., & Sharafi, Z. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3705249
“In Shah et al.’s study [33], students expressed greater trust in Copilot’s comprehension features (e.g., code explanation) than in code generation, potentially due to “trust affordances” such as clickable code references and transparent reasoning.”
Students’ Perception Accuracy of Partners’ AI Use and its Relation to Collaboration Performance
Graf, L., Beinstingel, R., Krusche, S., & Poquet, O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37979-5_16
Hutchison, J., Applebaum, I. T., Angelikas, K., Patel, K., Nguyen, P., Lázaro, A., Rucinski, N., Nabid, R. A., & MacNeil, S. (2026) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3803400.3809394 · Cited by 1
When LLM Tutoring Responses Work: Evidence from Student Programming ConversationsPreprint
Abrar, M. F., Sharmin, S., & Barmaki, R. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.09919
Gardella, N., Bolton, M. L., & Riggs, S. L. (2026) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208261467254 · Cited by 2
Scaffolding Autocomplete: Improving Guidance for Learners using Generative Code Suggestions
Prather, J. E., MacNeil, S., Luxton-Reilly, A., Margulieux, L. E., Reeves, B. N., Denny, P. C., Leinonen, J., Homer, J., Nabid, R. A., & Rossetti, R. L. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811654
Hey, ChatGPT, Look at My Work: Using Conversational AI in Requirements Engineering Education
Badihi, S., Tegegn, M., Riddell, E., Czarnecki, K., & Rubin, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773112 · Cited by 1
“Our analysis of students’ ChatGPT conversations and our own prompting study shows some deficiencies of the underlying AI technology, such as being incomplete, unspecific, or in-feasible; it has factual knowledge gaps and can produce outputs that are internally inconsistent.”
Shahid, R. M. U., Amir, A., Hameed, F., Gull, Z., & Hussain, I. (2026) · Liberal Journal of Language & Literature Review
An Empirical Analysis of LLM Usage in Software Engineering Course Projects
Jeong, H., & Kim, J. (2026) · ACM Transactions on Computing Education · doi:10.1145/3859283
Model Context Protocol (MCP) for AI Systems: A Survey of Architectures and Applications
Upadhyay, D. N. (2026) · Engineering and Technology Journal · doi:10.47191/etj/v11i10.10
Beyond the Final Code: Understanding Student Programming Behaviors through Process Journals
Chao, A., Li, H., Ma, K., Nazzal, A., Kazerouni, A. M., Perlmutter, L., & Raj, A. G. S. (2026) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3856208.3856238
Chao, L., Cho, C., Lu, C., McGrath, M., Yeh, T., Rios, M., Yeh, E., Ganesh, S., & YEH, T. P. (2026) · doi:10.1145/3795867.3830977
Metadata
- Authors
- Shah, A., Chernova, A., Tomson, E., Porter, L., Griswold, W. G., & Soosai Raj, A. G.
- Affiliations
- University of California San Diego (US)
- University of California, San Diego, La Jolla, CA, USA
- ORCID
- Venue
- Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2025-02-12
- Pages
- 1050-1056
- OpenAlex
- W4407771444
- Cited by
- 19
- References
- 21
- Citations by year
- 2026: 12 · 2025: 7
- FWCI
- 27.61
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion, cc-by)
- Funders
- NSF (National Science Foundation) (2417531)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, large language models, large codebases, one-shot prompting, program comprehension, code generation, feature implementation, debugging, software engineering education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.