CoChat
Sign Up
PreprintCited by 7Student Assessment and FeedbackTopic ModelingNatural Language Processing Techniques

Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks

Open access
Rao, Delip, Callison-Burch, Chris·2026·arXiv
DOI 10.48550/arxiv.2603.00077arXiv 2603.00077
Open this paper in CoChat — free accountOpen PDF

Abstract

Rubric-based LLM judges have become indispensable for evaluating and optimizing systems on non-verifiable tasks, where success cannot be reduced to exact programmatic checks. Yet the underlying judges remain vulnerable to position bias, stochastic inconsistency, criterion conflation, forced judgments under uncertainty, and model-dependent calibration. Rubrics structure evaluation but do not eliminate these failures; they add consequential choices about criterion design, scale types, weighting, aggregation, abstention, calibration, and reliability measurement. Despite extensive research across LLM evaluation, educational measurement, and psychometrics, the relevant methods remain scattered across papers and partial implementations. Researchers therefore pay a reinvention tax, repeatedly rebuilding evaluation machinery instead of accumulating knowledge on a common substrate. We introduce Autorubric, an open-source framework that makes rubric and judge choices explicit, reusable, and auditable. Through a unified API, it supports atomic evaluation of mixed criterion types alongside configurable bias mitigations, calibration, ensembling, abstention, and psychometric diagnostics. Evaluations spanning college chemistry grading, deep-research systems, and CHARM-100---a new mixed-criterion chatbot benchmark---reveal criterion-specific failures, systematic judge-family differences, and configuration effects that do not support a universal mitigation stack. We further demonstrate how per-criterion scores and explanations can support agent skill revision and reward modeling for reinforcement learning. By providing shared infrastructure from measurement through optimization, Autorubric gives the community a common basis for comparing methods, reproducing evaluation choices, and accumulating evidence across studies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.00077

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • With R ≥ 2 judges grading each criterion, the standard interjudge statistics are Fleiss' κ and Krippendorff's α.

    “With R ≥ 2 judges grading each criterion (Rao and Callison-Burch, 2026), the standard interjudge statistics are Fleiss' κ (Fleiss, 1971) and Krippendorff's α (Krippendorff, 2004).”

    From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Rao and Callison-Burch, 2026

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

    Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D'arcy, M., et al. (2024) · arXiv · Cited by 7 · cited 1× · see citing works

    “For LLM-based evaluation, the community, and by extension Autorubric , has gravitated towards analytic rubrics (see for example, Asai et al. (2024); Hashemi et al. (2024)) for several reasons: • Transparency : Each criterion’s contribution to the final score is explicit and au-ditable.”

Flagged first, then by citations

Research works printed but not linked: 48

  1. Zahra Ashktorab, Werner Geyer, Michael Desmond, Elizabeth M. Daly, Martin Santillan Cooper, Qian Pan, Erik Miehling, Tejaswini Pedapati, and Hyo Jin Do. EvalAssist: A human-centered tool for LLM-as-a-judge. arXiv preprint, 2025.
  2. H John Bernardin, HW Hennessey Jr, and Joseph Peyrefitte. Age, racial, and gender bias as a function criterion specificity: A test of expert testimony. Human Resource Management Review, 5(1):63-77, 1995.
  3. Jarosław Błasiok and Preetum Nakkiran. Smooth ece: Principled reliability diagrams via kernel smoothing. arXiv preprint arXiv:2309.12236, 2023. · arXiv
  4. Susan M. Brookhart. Appropriate criteria: Key to effective rubrics. Frontiers in Education, 3: 22, 2018. doi: 10.3389/feduc.2018.00022. · DOI · Paper page
  5. Jodi M. Casabianca, Daniel F. McCaffrey, Matthew S. Johnson, Naim Alper, and Vladimir Zubenko. Validity arguments for constructed response scoring using generative artificial intelligence applications. arXiv preprint, 2025.
  6. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. ChatEval: Towards better LLM-based evaluators through multi-agent debate. arXiv preprint, 2023.
  7. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length- controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint, 2024.
  8. Jonathan St. B. T. Evans. Questions and challenges for the new psychology of reasoning. Thinking & Reasoning, 18(1):5-31, 2012. doi: 10.1080/13546783.2011.637674. URL https: //doi.org/10.1080/13546783.2011.637674. · DOI
  9. Benjamin Feuer, Chiung-Yi Tseng, Astitwa Sarthak Lathe, Oussama Elachqar, and John P. Dickerson. When judgment becomes noise: How design failures in llm judge benchmarks silently undermine validity. arXiv preprint, 2025.
  10. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge. arXiv preprint, 2024.
  11. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint, 2025.
  12. Rajarshi Haldar and Julia Hockenmaier. Rating roulette: Self-inconsistency in llm-as-a-judge frameworks. arXiv preprint, 2025.
  13. Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. Llm- rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13806-13834. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.745. URL http://dx.doi.org/10. 18653/v1/2024.acl-long.745. · DOI · Paper page
  14. Alex B Haynes, Thomas G Weiser, William R Berry, Stuart R Lipsitz, Abdel-Hadi S Breizat, E Patchen Dellinger, Teodoro Herbosa, Sudhir Joseph, Pascience L Kibatala, Marie Carmela M Lapitan, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. New England journal of medicine, 360(5):491-499, 2009.
  15. Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. LLM-as-a-judge for software engineering: Literature review, vision, and the road ahead. arXiv preprint, 2025.
  16. Yihan Hong, Huaiyuan Yao, Bolin Shen, Wanpeng Xu, Hua Wei, and Yushun Dong. Rulers: Locked rubrics and evidence-anchored scoring for robust llm evaluation. arXiv preprint, 2026.
  17. Hui Huang, Yancheng He, Hongli Zhou, Rui Zhang, Wei Liu, Weixun Wang, Jiaheng Liu, and Wenbo Su. Think-j: Learning to think for generative llm-as-a-judge, 2026. URL https://arxiv.org/abs/2505.14268. · arXiv
  18. Anders Jonsson and Ernesto Panadero. The use and design of rubrics to support assessment for learning. In David Carless, Susan M. Bridges, Cecilia Ka Yuk Chan, and Rick Glofcheski (eds.), Scaling up Assessment for Learning in Higher Education, volume 5 of The Enabling Power of Assessment, pp. 99-111. Springer, Singapore, 2017. ISBN 978-981-10-3043-7. doi: 10.1007/978-981-10-3045-1_7. · DOI
  19. Ahmed Karim, Qiao Wang, and Zheng Yuan. Beyond the score: Uncertainty-calibrated llms for automated essay assessment. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 19642-19647, 2025.
  20. Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think, 2025. URL https://arxiv.org/abs/2504.16828. · arXiv
  21. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4334-4353, 2024. doi: 10.18653/v1/2024.emnlp-main.248. · DOI
  22. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp. 611-626, 2023.
  23. Yukyung Lee, JoongHoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 15782-15809, 2025. doi: 10.18653/v1/2025.emnlp-main. 796. · DOI
  24. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena- Hard and BenchBuilder pipeline. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 34209-34231.
  25. PMLR, 2025. URL https://proceedings.mlr.press/v267/li25h.html.
  26. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmark- ing llms with challenging tasks from real users in the wild. In International Conference on Learning Representations (ICLR), 2025. doi: 10.48550/arXiv.2406.04770. · DOI · Paper page
  27. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. arXiv preprint, 2023.
  28. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100, 2023. doi: 10.18653/v1/2023.emnlp-main.741. · DOI · Paper page
  29. Leah Mrangu. Rubric as assessment tool for lecturers and students in higher education institution. Acta Pedagogia Asiana, 1(1):26-33, 2022. doi: 10.53623/apga.v1i1. · DOI
  30. Qian Pan, Zahra Ashktorab, Michael Desmond, Martin Santillan Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. Human-centered design recommendations for llm-as-a-judge. arXiv preprint, 2024.
  31. Ernesto Panadero and Anders Jonsson. A critical review of the arguments against the use of rubrics. Educational Research Review, 30:100329, 2020. doi: 10.1016/j.edurev.2020.100329. · DOI · Paper page
  32. Arjun Panickssery, Samuel Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. Neural Information Processing Systems (NeurIPS), 2024. doi: 10. 48550/arXiv.2404.13076. · DOI · Paper page
  33. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2025. URL https://arxiv.org/abs/2310.13548. · arXiv
  34. Shashank Sonkar, Kangqi Ni, Lesa Tran Lu, Kristi Kincaid, John S. Hutchinson, and Richard G. Baraniuk. Automated long answer grading with ricechem dataset, 2024. URL https://arxiv.org/abs/2404.14316. · arXiv
  35. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
  36. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. URL https://arxiv.org/abs/2305.04388. · arXiv
  37. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint, 2024.
  38. Peter Villanova, H John Bernardin, Sue A Dahmus, and Randi L Sims. Rater leniency and performance appraisal discomfort. Educational and psychological measurement, 53(3): 789-799, 1993.
  39. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint, 2023.
  40. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer 2: Open-source dataset for training top-performing reward models. In Advances in Neural Information Processing Systems, volume 37, pp. 1474-1501.
  41. P.C. Wason and J.ST.B.T. Evans. Dual processes in reasoning? Cognition, 3(2):141-154, 1974. ISSN 0010-0277. doi: https://doi.org/10.1016/0010-0277(74)90017-1. URL https: //www.sciencedirect.com/science/article/pii/0010027774900171. · DOI · Paper page
  42. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in LLM-as-a-judge. In NeurIPS Safe Generative AI Workshop, 2024. doi: 10.48550/arXiv.2410.21819. · DOI · Paper page
  43. Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. Rocketeval: Efficient automated llm evaluation via grading checklist. arXiv preprint, 2025.
  44. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swar- nadeep Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning, 2025. URL https://arxiv.org/abs/2505.10320. · arXiv
  45. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020. URL https://arxiv. org/abs/1910.03771.
  46. Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry, 2025. URL https://arxiv.org/abs/2507.16280. · arXiv
  47. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
  48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint, 2023.

Cited by

Sorted by citations

Authors
Rao, D., & Callison-Burch, C.
Affiliations
University of Pennsylvania (US)
ORCID
Callison-Burch, Chris
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.00077
arXiv
2603.00077
OpenAlex
W7133351461
Cited by
7
References
1
Citations by year
2026: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • U.S. Department of Defense
  • Defense Advanced Research Projects Agency
  • Advanced Research Projects Agency
Topics
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
LLM evaluation, position bias, bias mitigation, ensembling, abstention, reward modeling, reinforcement learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review