Toward human–AI collaboration: a recommender system to support CS1 instructors to select problems for assignments and exams
Abstract
Programming online judges (POJs) have been increasingly used in CS1 classes, as they allow students to practice and get quick feedback. For instructors, it is a useful tool for creating assignments and exams. However, selecting problems in POJs is time consuming. First, problems are generally not organized based on topics covered in the CS1 syllabus. Second, assessing whether problems require similar effort to be completed and map onto the same topic is a subjective and expert-dependent task. The difficulty increases if the instructor must create variations of these assessments, e.g., to avoid plagiarism. Thus, here, we research how to support CS1 instructors in the task of selecting problems, to compose one-size-fits-all or personalized assignments/exams. Our solution is to propose a novel intelligent recommender system, based on a fine-grained data-driven analysis of the students' effort on solving problems in the integrated development environment of a POJ system, and automatic detection of topics for CS1 problems, based on problem descriptions. Data collected from 2714 students are processed to support, via our artificial intelligence (AI) method recommendations, the instructors' decision-making process. We evaluated our method against the state of the art in a simple blind experiment with CS1 instructors ($N =$35). Results show that our recommendations are 88% accurate, surpassing our baseline ($p<$0.05). Finally, our work paves the way for novel POJ smart learning envir…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Ten studies demonstrated that AI could support staff in curriculum, instruction, and assessment by reducing their logistical workloads, especially in curriculum design, interactions with students, delivering personalized instruction, and preparing adapted or personalized assignments.
“Ten studies have demonstrated that AI could support staff in curriculum, instruction, and assessment, by reducing their logistical workloads, especially in terms of labor related to curriculum design, interactions with students, delivering personalized instruction, and preparing adapted or personalized assignments (e.g., Holmes et al., 2023;Pereira et al., 2023;Sajja et al., 2023;Devi and Rroy, 2”
From A systematic review of the early impact of artificial intelligence on higher education curriculum, instruction, and assessment (2025) · cites this paper as Pereira et al., 2023
Pereira et al. (2023) described how an emerging recommender system generated equivalent questions for assignments and exams, to enhance the variation of assignments and support instructors in preparing individualized assignments and minimizing plagiarism.
“Pereira et al. (2023) described how an emerging recommender system generated equivalent questions for assignments and exams, to enhance the variation of assignments and support instructors in preparing individualized assignments and minimizing plagiarism.”
From A systematic review of the early impact of artificial intelligence on higher education curriculum, instruction, and assessment (2025) · cites this paper as Pereira et al. (2023)
AI benefits assessments by generating personalized assignments, effectively assessing and predicting students' academic achievement and non-cognitive outcomes, identifying disadvantaged students, and assessing teaching effectiveness.
“Additionally, AI seems to benefit assessments by generating personalized assignments (Pereira et al., 2023), effectively assessing and predicting students' academic achievement (Wang et al., 2025) and non-cognitive outcomes (e.g., soft skills, Novais et al., 2023), identifying disadvantaged students (Saad and Tounkara, 2023), and assessing teaching effectiveness (Wang et al., 2025).”
From A systematic review of the early impact of artificial intelligence on higher education curriculum, instruction, and assessment (2025) · cites this paper as Pereira et al., 2023
Understanding the benefits that AI brings to HE curriculum, instruction, and assessment could help academics make full use of the technology to reduce workloads and improve productivity.
“In this light, understanding the benefits that AI brings to HE curriculum, instruction, and assessment could help academics make full use of the technology to reduce workloads (Holmes et al., 2023;Pereira et al., 2023) and improve productivity.”
From A systematic review of the early impact of artificial intelligence on higher education curriculum, instruction, and assessment (2025) · cites this paper as Pereira et al., 2023
Cites
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Rowland, C. A. (2014) · Psychological Bulletin · doi:10.1037/a0037559 · Cited by 1,066 · cited 1× · see citing works
“Moreover, the testing effect theory has shown to be effective in many fields of education [50].”
BackgroundA Conceptual Framework for Human–AI Hybrid Adaptivity in Education
Holstein, K., Aleven, V., & Rummel, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52237-7_20 · Cited by 168 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Liang, J., Stephens, J. M., & Brown, G. T. L. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1522841 · Cited by 112 · see citing works
- Authors
- Pereira, F. D., Rodrigues, L., Henklain, M. H. O., Freitas, H., Oliveira, D. F., Cristea, A. I., Carvalho, L., Isotani, S., Benedict, A., Dorodchi, M., & de Oliveira, E. H. T.
- Affiliations
- Department of Computer Science, Federal University of Roraima, Boa Vista, Brazil
- University of São Paulo, São Paulo, Brazil
- Institute of Computing, Federal University of Amazonas, Manaus, Brazil
- Department of Computer Science, Durham University, Durham, U.K.
- Institute of Mathematics and Computer Science, University of São Paulo, São Paulo, Brazil
- Department of Computer Science, The University of North Carolina at Charlotte, Charlotte, NC, USA
- ORCID
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2023-06-01
- Volume
- 16
- Issue
- 3
- Pages
- 457-472
- ISSN
- 1939-1382
- 2372-0050
- Cited by
- 22
- References
- 2
- Licence
- Funders
- Brazilian National Council for Scientific and Technological Development (141859/2019-9, 308458/2020-6, 308513/2020-7)
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (001)
- São Paulo State Research Support Foundation (2013/07375-0)
- Research and Development (001/2020)
- Federal University of Amazonas and FAEPI, Brazil
- Samsung, using resources from the Informatics Law for the Western Amazon (8.387/1991)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
