Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Abstract
With the immense interest in ChatGPT worldwide, education has seen a mix of both excitement and skepticism. To properly evaluate its impact on education, it is crucial to understand how far it can help students without prior knowledge answer assessment questions. This study aims to address this question as well as the impact of the question type. We conducted multiple experiments with computer engineering students (experiment group: n =41 to 56), who were asked to use ChatGPT to answer previous test questions before learning about the related topics. Their scores were then compared with the scores of previous-term students who answered the same questions in a quiz or exam setting (control group: n =24 to 61). The results showed a wide range of effect sizes, from -2.55 to 1.23, depending on the question type and content. The experiment group performed best answering code analysis and conceptual questions but struggled with code completion and questions that involved images. However, the performance in code generation tasks was inconsistent. Overall, the ChatGPT group’s answers lagged slightly behind the control group’s answers with an effect size of -0.16. We conclude that ChatGPT, at least in the field of this study, is not yet ready to rely on by students who do not have sufficient background to evaluate generated answers. We suggest that educators try using ChatGPT and educate students on effective questioning techniques and how to assess the generated responses. This study provides insights into the capabilities and limitations of ChatGPT in education and informs future research and development.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT can provide accurate answers that could be exploited in exams.
“Similarly, Shoufan [27,28] underscores both the benefits and risks of using ChatGPT in educational contexts, particularly its ability to provide accurate answers that could be exploited in exams.”
From Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda (2025) · cites this paper as [27]
Students who are not trained in evaluating the responses from chatbots are not as successful as students who learn this skill.
“Shoufan 2023 found that students who are not trained in evaluating the responses from chatbots are not as successful as students who learn this skill, which points to a need for more student training in the use of chatbots in education.”
From Effects of Generative AI on Engagement and Performance Within Asynchronous Online Courses (2026) · cites this paper as Shoufan 2023
Cites
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 4× · see citing works
“It supports group and distance learning and empowers learners with disabilities [15].”
BackgroundInfluentialChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · cited 3× · see citing works
“This can be achieved by implementing various strategies such as in-class assessments, presentations, and performances, allowing for student choice in topic selection, and using authentic assessment methods that reflect real-world situations, as suggested by Rudolph et al. [34].”
BackgroundInfluentialChatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
Interacting with educational chatbots: A systematic review
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11177-3 · Cited by 841 · see citing works
Chatbots for learning: A review of educational chatbots for the Facebook Messenger
Smutny, P., & Schreiberova, P. (2020) · Computers & Education · doi:10.1016/j.compedu.2020.103862 · Cited by 770 · see citing works
Exploring Students’ Perceptions of ChatGPT: Thematic Analysis and Follow-Up Survey
Shoufan, A. (2023) · IEEE Access · doi:10.1109/access.2023.3268224 · Cited by 499 · see citing works
Self-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · see citing works
Essel, H. B., Vlachopoulos, D., Tachie-Menson, A., Johnson, E. E., & Baah, P. K. (2022) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-022-00362-6 · Cited by 428 · see citing works
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191 · see citing works
Sáiz-Manzanares, M. C., Marticorena-Sánchez, R., Martín-Antón, L. J., González Díez, I., & Almeida, L. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e12843 · Cited by 96 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda
Bittle, K., & El-Gayar, O. (2025) · Information · doi:10.3390/info16040296 · Cited by 305 · see citing works
“Similarly, Shoufan [27,28] underscores both the benefits and risks of using ChatGPT in educational contexts, particularly its ability to provide accurate answers that could be exploited in exams.”
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13 · see citing works
Melanou, C., & Beege, M. (2026) · Education Sciences · doi:10.3390/educsci16040651 · Cited by 12 · see citing works
When the Robotic Maths Tutor is Wrong - Can Children Identify Mistakes Generated by ChatGPT?
Helal, M., Holthaus, P., Wood, L., Velmurugan, V., Lakatos, G., Moros, S., & Amirabdollahian, F. (2024) · 2024 5th International Conference on Artificial Intelligence, Robotics and Control (AIRC) · doi:10.1109/airc61399.2024.10672220 · Cited by 6 · see citing works
“While other work has illustrated various shortcomings in LLM-assisted learning of computer science without prior knowledge [13].”
BackgroundEffects of Generative AI on Engagement and Performance Within Asynchronous Online Courses
Flores, F., Norman, K., & Hanshaw, G. (2026) · Open Praxis · doi:10.55982/openpraxis.18.2.980 · Cited by 2 · see citing works
A Scaffolded GenAI Lab in Early Undergraduate CS: A Mixed-Methods, Multi-Course Evaluation
Dickey, E., Bejarano, A., Kuperus, R., & Fagundes, B. (2025) · arXiv
“In a related direction, journal evidence indicates that students without sufficient background can have difficulty judging the quality of AI-generated answers, underscoring the importance of explicit instruction in evaluation rather than treating GenAI access as inherently supportive [26].”
- Authors
- Shoufan, A.
- Affiliations
- Khalifa University of Science and Technology (AE)
- Corresponding author
- Shoufan, Abdulhadi
- ORCID
- Shoufan, Abdulhadi
- Venue
- ACM Transactions on Computing Education
- Published
- 2023-12-11
- Volume
- 23
- Issue
- 4
- Pages
- 1-29
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W4387735186
- Cited by
- 59
- References
- 11
- Citations by year
- 2026: 17 · 2025: 19 · 2024: 22 · 2023: 1
- FWCI
- 13.85
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- ChatGPT, large language models in education, prior knowledge, question types, code generation, code completion, conceptual questions, image-based questions, computer engineering students, student performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
