CoChat
Sign Up
PreprintCited by 31Teaching and Learning ProgrammingSoftware Engineering ResearchOnline Learning and Analytics

Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics

Open access
Pathak, Aditya, Gandhi, Rachit, Uttam, Vaibhav, Ramamoorthy, Arnav, Ghosh, Pratyush, Jindal, Aaryan Raj, Verma, Shreyash, Mittal, Aditya +6 more·2025·arXiv
DOI 10.48550/arxiv.2503.23989arXiv 2503.23989

Preprint — published as 10.1145/3702652.3744220

Open this paper in CoChat — free account

Abstract

Since the emergence of Large Language Models (LLMs) popularized by the release of GPT-3 and ChatGPT, LLMs have shown remarkable promise in programming-related tasks. While code generation using LLMs has become a popular field of research, code evaluation using LLMs remains under-explored. In this paper, we focus on LLM-based code evaluation and attempt to fill in the existing gaps. We propose multi-agentic novel approaches using \emph{question-specific rubrics} tailored to the problem statement, arguing that these perform better for logical assessment than the existing approaches that use \emph{question-agnostic rubrics}. To address the lack of suitable evaluation datasets, we introduce two datasets: a Data Structures and Algorithms dataset containing 150 student submissions from a popular Data Structures and Algorithms practice website, and an Object Oriented Programming dataset comprising 80 student submissions from undergraduate computer science courses. In addition to using standard metrics (Spearman Correlation, Cohen's Kappa), we additionally propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment. Our comprehensive analysis demonstrates that \emph{question-specific rubrics} significantly enhance logical assessment of code in educational settings, providing better feedback aligned with instructional goals beyond mere syntactic correctness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.23989

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Rubrics designed by human experts help ensure the consistency of LLM judge scores with human expert preferences to some extent.

    “Rubrics designed by human experts also help ensure the consistency of LLM judge scores with human expert preferences to some extent [Pathak et al., 2025, Hashemi et al., 2024].”

    From DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports (2026) · cites this paper as Pathak et al., 2025

    The Introduction (p. 2) reports that question-specific rubrics improve code-evaluation accuracy, and Data Preprocessing (p. 8) describes comparing LLM marks with human-grader marks for agreement. The paper supports some alignment in grading student code, but it does not establish that human-designed rubrics ensure LLM judge scores match human expert preferences more generally. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 14

  1. [6][6] Anishka, Atharva Mehta, Nipun Gupta, Aarav Balachandran, Dhruv Kumar, and Pankaj Jalote. 2024. Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course? arXiv:2312.07343 [cs.HC] https: //arxiv.org/abs/2312.07343 - · arXiv
  2. [9][9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805 - · arXiv
  3. [10][10] Zhiyuan Fan, Weinong Wang, Xing Wu, and Debing Zhang. 2025. SedarEval: Automated Evaluation using Self-Adaptive Rubrics. arXiv:2501.15595 [cs.CV] https://arxiv.org/abs/2501.15595 - · arXiv
  4. [12][12] Arto Hellas, Juho Leinonen, and Leo Leppänen. 2024. Experiences from Integrating Large Language Model Chatbots into the Classroom. arXiv:2406.04817 [cs.CY] https://arxiv.org/abs/2406.04817 - · arXiv
  5. [14][14] Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. 2023. AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs. arXiv:2311.02775 [cs.LG] https://arxiv.org/abs/2311.02775 - · arXiv
  6. [22][22] Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. 2025. Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation. arXiv:2406.05053 [cs.LG] https://arxiv.org/abs/2406.05053 - · arXiv
  7. [24][24] Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. arXiv:2308.06921 [cs.CY] https://arxiv.org/abs/2308.06921 - · arXiv
  8. [26][26] Richard Lobb and Jenny Harlow. 2016. Coderunner: a tool for assessing computer programming skills. ACM Inroads 7, 1 (Feb. 2016), 47-51. https://doi.org/10. 1145/2810041 -
  9. [30][30] OpenAI. 2023. GPT-4 Technical Report. https://arxiv.org/abs/2303.08774v2 - · arXiv
  10. [37][37] Nishat Raihan, Mohammed Latif Siddiq, Joanna C. S. Santos, and Marcos Zampieri. 2024. Large Language Models in Computer Science Education: A Systematic Literature Review. arXiv:2410.16349 [cs.LG] https://arxiv.org/abs/2410.16349 - · arXiv
  11. [38][38] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE] https://arxiv.org/abs/2009.10297 - · arXiv
  12. [41][41] Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation with Large Language Models. arXiv:2410.02184 [cs.LG] https://arxiv.org/abs/ 2410.02184 - · arXiv
  13. [44][44] Wenjing Xie, Juxin Niu, Chun Jason Xue, and Nan Guan. 2024. Grade Like a Human: Rethinking Automated Assessment with Large Language Models. arXiv:2405.19694 [cs.AI] https://arxiv.org/abs/2405.19694 - · arXiv
  14. [47][47] Terry Yue Zhuo. 2024. ICE-Score: Instructing Large Language Models to Evaluate Code. arXiv:2304.14317 [cs.AI] https://arxiv.org/abs/2304.14317 · arXiv

Cited by

Sorted by citations

Related

Authors
Pathak, A., Gandhi, R., Uttam, V., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., Nakka, Y., Devansh, Challa, J. S., & Kumar, D.
Affiliations
Birla Institute of Technology and Science, Pilani (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
181-195
DOI
10.48550/arxiv.2503.23989
arXiv
2503.23989
OpenAlex
W4412781285
Cited by
30
References
30
Citations by year
2026: 25 · 2025: 5
FWCI
43.45
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Birla Institute of Technology and Science, Pilani (N4/24/1004)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
automated programming assessment, student code submissions, data structures and algorithms, object-oriented programming, evaluation dataset, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review