Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
Open accessAbstract
We analyzed effectiveness of three generative pre-trained transformer (GPT) models in answering multiple-choice question (MCQ) assessments, often involving short snippets of code, from introductory and intermediate programming courses at the postsecondary level. This emerging technology stirs countless discussions of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming education (e.g., cheating). However, the capabilities of GPT models and their limitations to reason about and/or analyze code in educational settings have been under-explored. We evaluated several OpenAI's GPT models on formative and summative MCQ assessments from three Python courses (530 questions). We found that MCQs containing code snippets are not answered as successfully as those that only contain natural language. While questions requiring to fill-in a blank in the code or completing a natural language statement about the snippet are handled rather successfully, MCQs that require analysis and/or reasoning about the code (e.g., what is true/false about the snippet, or what is its output) appear to be the most challenging. These findings can be leveraged by educators to adapt their instructional practices and assessments in programming courses, so that GPT becomes a valuable assistant for a learner as opposed to a source of confusion and/or potential hindrance in the learning process.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There appeared to be a difference in success rate between MCQs that contained a code snippet and those that did not.
“and that there appeared to be a difference in success rate between MCQs that contained a code snippet and those that did not [45, 46].”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [45]
Prior findings [45, 46] no longer hold.
“Overall, the findings reported in prior work [45, 46] no longer hold.”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [45]
An important limitation of the GPT models was their apparent struggle with activities that required multi-hop reasoning.
“They also observed that an important limitation of the GPT models was their apparent struggle with activities that required multi-hop reasoning,”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784 · see citing works
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 759 · see citing works
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Self-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192 · see citing works
Evaluating large language models on a highly-specialized topic, radiation oncology physics
Holmes, J., Liu, Z., Zhang, L., Ding, Y., Sio, T. T. W., McGee, L. A., Ashman, J. B., Li, X., Liu, T., Shen, J., et al. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1219326 · Cited by 166 · see citing works
Singh, H., Tayarani-Najaran, M.-H., & Yaqoob, M. (2023) · Education Sciences · doi:10.3390/educsci13090924 · Cited by 150 · see citing works
“It can assist in completing partially written code blocks, but lacks the execution capabilities required to produce the correct output; after all, it is a language model [8].”
BackgroundSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · see citing works
Self-citationGenerative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136 · see citing works
Lyu, W., Wang, Y., Chung, T. (., Sun, Y., & Zhang, Y. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662036 · Cited by 131 · see citing works
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132 · see citing works
Self-citationSavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · see citing works
Self-citationThe Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97 · see citing works
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88 · see citing works
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pezeshkpour, P., & Hruschka, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.130 · Cited by 85 · see citing works
Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B. E., Liffiton, M., Šavelka, J., & Denny, P. C. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 76 · see citing works
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71 · see citing works
Cambaz, D., & Zhang, X. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630958 · Cited by 62 · see citing works
Feedback-Generation for Programming Exercises With GPT-4
Azaiz, I., Kiesler, N., & Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653594 · Cited by 61 · see citing works
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60 · see citing works
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60 · see citing works
Exploring prompt pattern for generative artificial intelligence in automatic question generation
Wang, L., Song, R., Guo, W., & Yang, H. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2412082 · Cited by 53 · see citing works
Scholl, A., & Kiesler, N. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893442 · Cited by 35 · see citing works
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 37 · see citing works
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 33 · see citing works
Güler, N. K., Dertli, Z. G., Boran, E., & Yıldız, B. (2024) · Pedagogical Research · doi:10.29333/pr/14145 · Cited by 54 · see citing works
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 40 · see citing works
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 40 · see citing works
From GPT-3.5 to GPT-4.o: A Leap in AI’s Medical Exam Performance
Kipp, M. (2024) · Information · doi:10.3390/info15090543 · Cited by 28 · see citing works
GPT-Powered Elicitation Interview Script Generator for Requirements Engineering Training
Görer, B., & Aydemir, F. B. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00044 · Cited by 19 · see citing works
Computer Programming Education in the Age of Generative AI: Insights from Empirical ResearchPreprint
Deriba, F. G., Sanusi, I. T., Campbell, O. O., & Oyelere, S. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4891302 · Cited by 19 · see citing works
Automated Generation and Evaluation of MultipleChoice Quizzes using Langchain and Gemini LLM
Pawar, P., Dube, R., Joshi, A., Gulhane, Z., & Patil, R. (2024) · International Conference on Electrical Electronics and Computing Technologies (ICEECT) · doi:10.1109/iceect61758.2024.10739326 · Cited by 12 · see citing works
Šavelka, J., Agarwal, A., Bogart, C., & Sakr, M. F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-53656-4_8 · Cited by 11 · see citing works
Self-citationLLM Generative AI and Students’ Exam Code Evaluation: Qualitative and Quantitative Analysis
Smolić, E., Pavelić, M., Boras, B., Mekterović, I., & Jagušt, T. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569820 · Cited by 11 · see citing works
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 13 · see citing works
Large Language Models (LLMs) in Programming Learning: The Current Research State and Agenda
Fu, Q., Zhao, Y., Jia, Z., & Zheng, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3622043 · Cited by 10 · see citing works
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9 · see citing works
Gutierrez, S., Hou, I., Lee, J., Angelikas, K., Man, O., Mettille, S., Prather, J. E., Denny, P. C., & MacNeil, S. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716654 · Cited by 7 · see citing works
A Systematic Review of Gen-AI Applications in Education: Rewards, Challenges and Future Prospects
Wanyonyi, E. N., & Murithi, M. (2025) · Pan-African journal of education and social sciences · doi:10.56893/pajes2025v06i01.01 · Cited by 7 · see citing works
Pre‐trained language models: What do they know?
Guimar�ães, N., Campos, R., & Jorge, A. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1518 · Cited by 7 · see citing works
Cognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 7 · see citing works
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7 · see citing works
Self-citationInvestigating Developers' Preferences for Learning and Issue Resolution Resources in the ChatGPT Era
Tayeb, A. J., Alahmadi, M. D., Tajik, E., & Haiduc, S. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00045 · Cited by 6 · see citing works
LLM-empowered literature mining for material substitution studies in sustainable concrete
Duan, Y., Tian, Y., Ghosh, S., Venugopal, V., Chen, J., & Olivetti, E. (2025) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2025.108379 · Cited by 5 · see citing works
The Impact of AI Use in Programming Courses on Critical Thinking Skills
Clarke, C. J. S. F., & Konak, A. (2025) · Journal of Cybersecurity Education Research and Practice · doi:10.62915/2472-2707.1220 · Cited by 5 · see citing works
Evaluating GPT’s Programming Capability Through CodeWars’ Katas
Zhang, Z., Wen, L., Zhang, S., Chen, D., & Jiang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5489-2_2 · Cited by 3 · see citing works
Evaluate Chat‐GPT 's programming capability in Swift through real university exam questions
Zhang, Z., Wen, L., Jiang, Y., & Liu, Y. (2024) · Software Practice and Experience · doi:10.1002/spe.3330 · Cited by 2 · see citing works
Possible Applications of ChatGPT in Higher Education
Jelica, S., Minović, M., & Šošević, U. (2024) · Lecture notes in educational technology · doi:10.1007/978-981-97-1814-6_13 · Cited by 3 · see citing works
Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs
Raimondi, B., Giallorenzo, S., & Gabbrielli, M. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707732 · Cited by 1 · see citing works
Towards Unified Objectives for Self-Reflective AIPreprint
Samwald, M., Praas, R., & Hebenstreit, K. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4446991 · Cited by 1 · see citing works
Edwards, C. (2024) · Communications of the ACM · doi:10.1145/3637208 · Cited by 1 · see citing works
Accelerate Learning with AI Powered Quiz Master Using Llama LLM
Bhatia, G., & Alhajri, R. (2025) · International Conference for Artificial Intelligence, Applications, Innovation and Ethics (AI2E) · doi:10.1109/ai2e64943.2025.10983252
Generative AI in Consumer Behaviour and Marketing: Applications, Opportunities and Challenges
Dar, A. R., & Gul, M. (2025) · Contributions to management science · doi:10.1007/978-3-032-02526-5_14
Multimodal Quiz Generation via RAG with LLM-as-Judge Evaluation
Kunuku, M. T., & Dehbozorgi, N. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328244
Qin, S., Xue, J., Hao, F., Niu, S., & Chang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1947-6_10
Boutaib, M., Farajy, N., Hanine, H., & Moumen, A. (2026) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc71566.2026.11650160
Hafizin, Maisyaroh, Timan, A., Triwiyanto, T., & Ahyar, M. (2026) · Multidisciplinary Science Journal · doi:10.31893/multiscience.2027184
- Authors
- Savelka, J., Agarwal, A., Bogart, C., & Sakr, M.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 47-58
- arXiv
- 2303.08033
- OpenAlex
- W4366808937
- Cited by
- 56
- References
- 0
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 26 · 2023: 15
- FWCI
- 14.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Conference on Computer Supported Education (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, GPT, multiple-choice questions, program comprehension, programming questions, question answering, model limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
