Tailoring Your Code Companion: Leveraging LLMs and RAG to Develop a Chatbot to Support Students in a Programming Course
The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Students frequently rely on chatbots powered by generative Artificial Intelligence (GenAI), such as ChatGPT, Copilot, Gemini, and Claude, to assist with a wide range of academic tasks. However, these chatbots are not specifically designed for the context of particular courses, which can lead to responses that are sometimes inaccurate or insufficiently relevant. This paper introduces a chatbot specifically designed to support first-year engineering students in a Java programming course. Developed using the Retrieval-Augmented Generation (RAG) technique, the chatbot draws on course-specific resources such as videos, quizzes, programming exercises, and other materials, while using OpenAI’s Large Language Models (LLMs) GPT-4 and GPT-3.5 for information analysis and response generation. The data collected, consisting of logs from 1,059 messages sent by students to the chatbot and 30 responses to a survey, indicate that students primarily used the chatbot to clarify concepts and explain code snippets. Moreover, most of the students reported that the responses provided by the chatbot were well suited to the Java programming course.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Analyzing the student traces with GenAI systems is not something new, especially for research purposes (e.g., [1,12]).”
From GenAI Analytics and Pedagogical Configurations: Featuring New Generative AI Systems in Education (2026) · cites this paper as [1]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 2×
“To assess usability, the System Usability Scale (SUS) [39], a widely recognized tool for measuring usability, was employed.”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“Conversational agents, commonly known as chatbots, have been in existence for nearly six decades, ELIZA being one of the earliest examples [21].”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254 · cited 1×
“The development of chatbots for educational purposes has been accelerated by the rapid advancements in Generative Artificial Intelligence (GenAI) [9], Large Language Models (LLMs) [10], and particularly following the release of Chat-GPT by OpenAI on November 30, 2022 [11].”
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 3,939
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“An approach to improving domain-specific chatbot performance is through fine-tuning techniques [28], where a pre-trained LLM is adapted to perform new, more specific tasks by following certain rules.”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Research works printed but not linked: 8
- ChatGPT
- Microsoft Copilot
- Google Gemini
- Claude
- Vision
- Embedding models
- pgvector. PostgreSQL: pgvector 0.5.0
- Genkinay, 2024, AI Prompt Engineering Is Dead • Long live AI prompt engineering
Cited by
The ALES Platform: State-of-the-Art and Gap Analysis for an Academic LLM Chatbot
Petrillo, A., Awasare, “. K., & Rehman, M. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.12.036 · Cited by 3
Analysis and Evaluation of Gemini for Question Generation in Two Engineering Courses
Moreno-Marcos, P. M., Santiuste, J. G., Muñoz‐Merino, P. J., Alario‐Hoyos, C., & Kloos, C. D. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328589 · Cited by 2
“In addition, Alario-Hoyos et al. [10] developed a chatbot to support first-year engineering students in a course about Java programming using the Retrieval Augmented Generation (RAG) technique.”
Self-citationGenAI Analytics and Pedagogical Configurations: Featuring New Generative AI Systems in Education
Ortega-Arranz, A., Topali, P., García-Zarza, P., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., & Molenaar, I. (2026) · Proceedings of the LAK26: 16th International Learning Analytics and Knowledge Conference · doi:10.1145/3785022.3785054 · Cited by 2
Reimagining Educational Techniques with Artificial Intelligence 1
Kloos, C. D., Alario‐Hoyos, C., Muñoz-Merino, P. J., Ayres, I. E., Moreno-Marcos, P. M., Callejo, P., & Kemcha, R. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574341 · Cited by 1
“The first experiment refers to the development of a 24/7 tutor for a Java programming course [22-23], called CharlieBOT.”
Self-citationWIP: Context-Aware AI in Learning Management Systems for Computer Science Courses
Simmons, A., de Holanda, M. T., & Silva, D. D. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328241
Metadata
- Authors
- Alario-Hoyos, C., Kemcha, R., Kloos, C. D., Callejo, P., Estévez-Ayres, I., Santín-Cristóbal, D., Cruz-Argudo, F., & López-Sánchez, J. L.
- Affiliations
- Universidad Carlos III de Madrid,Madrid,Spain
- Universidad Carlos III de Madrid (ES)
- Venue
- 2024 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)
- Publisher
- IEEE
- Published
- 2024-12-09
- Pages
- 1-8
- OpenAlex
- W4406417704
- Cited by
- 8
- References
- 34
- Citations by year
- 2026: 5 · 2025: 3
- FWCI
- 3.51
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Teaching, Assessment, and Learning for Engineering (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- chatbots, retrieval-augmented generation, large language models, programming education, Java programming, first-year engineering students, GPT-4, GPT-3.5, code explanation, student perceptions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Tailoring Your Code Companion: Leveraging LLMs and RAG to Develop a Chatbot to Support Students in a Programming Course. https://app.cochat.ai/papers/10.1109/tale62452.2024.10834365
- The paper
- Alario-Hoyos, C., Kemcha, R., Kloos, C. D., Callejo, P., Estévez-Ayres, I., Santín-Cristóbal, D., Cruz-Argudo, F., & López-Sánchez, J. L. (2024). Tailoring Your Code Companion: Leveraging LLMs and RAG to Develop a Chatbot to Support Students in a Programming Course. 2024 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE). https://doi.org/10.1109/tale62452.2024.10834365
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.