LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts
Open accessAbstract
This paper introduces a framework for the automated evaluation of natural language texts.A manually constructed rubric describes how to assess multiple dimensions of interest.To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses.The LLM predictions often fail to agree well with human judges-indeed, the humans do not fully agree with one another.However, the multiple LLM distributions can be combined to predict each human judge's annotations on all questions, including a summary question that assesses overall quality or relevance.LLM-RUBRIC accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters.When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-RUBRIC with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges' assessment of overall user satisfaction, on a scale of 1-4, with RMS error ă 0.5, a 2ˆ improvement over the uncalibrated baseline.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Rubrics designed by human experts help ensure the consistency of LLM judge scores with human expert preferences to some extent.
“Rubrics designed by human experts also help ensure the consistency of LLM judge scores with human expert preferences to some extent [Pathak et al., 2025, Hashemi et al., 2024].”
From DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports (2026) · cites this paper as Hashemi et al., 2024
The Evaluation Rubric Construction section (page not provided) says the authors wrote the rubric questions, but does not identify their designers as human experts. The Introduction and §2 (pages not provided) report that using those questions with judge-specific calibration improves agreement with human judgments; they do not show that expert design of the rubric itself ensures that agreement. (LLM, conf=0.84)
Experimental — accuracy not yet measuredA common case is rubric-based evaluation, where a holistic judgment is decomposed into independently graded criteria, each typically binary (MET/UNMET).
“A common case is rubricbased evaluation, where a holistic judgment is decomposed into independently graded criteria, each typically binary (MET/UNMET) (Hashemi et al., 2024;Saad-Falcon et al., 2023).”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Hashemi et al., 2024
The LLM-RUBRIC Method describes assessing an overall judgment through separate questions about evaluation criteria. But Evaluation Rubric Construction says the criteria usually have scores of 1–4, not binary MET/UNMET responses; it also does not establish that criteria are graded independently. Page numbers are unavailable in our copy. (LLM, conf=0.89)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 15
- Laurent Charlin and Richard S. Zemel. 2013. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In Proceedings of the ICML Workshop on Peer Reviewing and Pub- lishing Models (PEER).
- William Gantt, Lelia Glass, and Aaron Steven White. 2022. Decomposing and recomposing event struc- ture. Transactions of the Association for Computa- tional Linguistics, 10:17-34. · DOI
- Donna K. Harman. 1996. Overview of the Fourth Text REtrieval Conference (TREC-4). Special Publication (NIST SP) 500-236, National Institute of Standards and Technology, Gaithersburg, Maryland.
- He He, Hal Daumé III, and Jason Eisner. 2012. Cost- sensitive dynamic feature selection. In ICML Work- shop on Inferning: Interactions between Inference and Learning, Edinburgh. 6 pages.
- Úrsula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. 2018. Multicalibration: Cali- bration for the (computationally-identifiable) masses. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Pro- ceedings of Machine Learning Research, pages 1939- 1948.
- Andy S. Huang, Kyle Hirabayashi, Laura Barna, Deep Parikh, and Louis R. Pasquale. 2024. Assessment of a Large Language Model's Responses to Questions and Cases About Glaucoma and Retina Management. JAMA Ophthalmology, 142(4):371-375. · DOI
- Jiepu Jiang and James Allan. 2016. Reducing click and skip errors in search result ranking. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, San Francisco, CA, USA, February 22-25, 2016, pages 183-192. ACM. · DOI
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459- 9474.
- Mike Quartararo, Matt Poplawski, Adam Strayer, et al. 2019. Technology Assisted Review (TAR) guide- lines. Technical report, Bolch Judicial Institute of Duke Law School.
- Aviv Shamsian, Aviv Navon, Neta Glazer, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. 2023. Auxiliary learning as an asymmetric bargaining game. In Proceedings of the 40th International Conference on Machine Learning.
- Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle. 2016. Neural autore- gressive distribution estimation. Journal of Machine Learning Research, 17(1):7184-7220.
- Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine- grained human feedback gives better rewards for lan- guage model training. In Advances in Neural Infor- mation Processing Systems (NeurIPS).
- Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghas- semi, Anind K. Dey, and Dakuo Wang. 2024. Mental- LLM: Leveraging large language models for mental health prediction via online text data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 8(1).
- Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. 2019. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 14636-14647. · DOI
- Hiyori Yoshikawa, Tomoya Iwakura, Kimi Kaneko, Hi- roaki Yoshida, Yasutaka Kumano, Kazutaka Shimada, Rafal Rzepka, and Patrycja Swieczkowska. 2021. Tell me what you read: Automatic expertise-based annotator assignment for text annotation in expert domains. In Proceedings of the International Con- ference on Recent Advances in Natural Language Processing (RANLP), pages 1575-1585.
Cited by
Sorted by citations
A Zero-Shot LLM Framework for Automated Assignment Grading in Introductory Higher Education Courses
Yeung, C., Yu, J., Cheung, K. C., Wong, T. W., Chan, C. M., Wong, K. C., & Fujii, K. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99264-3_3 · Cited by 19 · see citing works
Diagnosing Bias and Instability in LLM Evaluation: A Scalable Pairwise Meta-Evaluator
Anghel, C., Anghel, A., Pecheanu, E., Cocu, A., Istrate, A., & Andrei, C. (2025) · Inf. · doi:10.3390/info16080652 · Cited by 20 · see citing works
Anghel, C., Anghel, A. A., Pecheanu, E., Șușnea, I., Cocu, A., & Istrate, A. (2025) · Informatics · doi:10.3390/informatics12030076 · Cited by 13 · see citing works
GradeAgentOps: A Verification-First Framework for Evidence-Anchored LLM Exam Grading
Anghel, C., Anghel, A. A., Crăciun, M., Cocu, A., Vulpe, D. E., Andrei, C. A., Maier, C., Scheau, C., Dragosloveanu, Ş., & Cergan, R. (2026) · AI · doi:10.3390/ai7060198 · Cited by 8 · see citing works
PEARL: A Rubric-Driven Multi-Metric Framework for LLM Evaluation
Anghel, C., Anghel, A. A., Pecheanu, E., Crăciun, M., Cocu, A., & Niculita, C. (2025) · Information · doi:10.3390/info16110926 · Cited by 6 · see citing works
Anghel, C., Anghel, A. A., Pecheanu, E., Cocu, A., Crăciun, M., Iacobescu, P., Balau, A. S., & Andrei, C. A. (2025) · Informatics · doi:10.3390/informatics12040123 · Cited by 6 · see citing works
Bianchi, O., Willey, M., Alvarado, C. X., Danek, B. P., Khani, M., Kuznetsov, N., Dadu, A., Shah, S. M. T., Koretsky, M. J., Makarious, M. B., et al. (2026) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100943 · Cited by 7 · see citing works
Suicide- and crisis-risk detection using large language models in mental-health chatbotsPreprint
Weber, S., Klebe, D., Wolf, L., Aeberli, C., Homan, S., Psathakis, N., Casanova, A., Macias, J. A. G., Sykstus, J., Li, J., et al. (2026) · medRxiv · doi:10.64898/2026.01.12.26343914 · Cited by 6 · see citing works
RUBRA: An Agentic AI System for Automatic Short Answer Grading Using LLMs and RAG
Haldar, S., Sengupta, S., & Das, A. K. (2025) · International Conference on Computing, Intelligence, and Application (CIACON) · doi:10.1109/ciacon65473.2025.11189329 · Cited by 4 · see citing works
Chen, Z., Wang, J., Li, Y., Li, H., Shi, C., Zhang, R., & Qu, H. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747670 · Cited by 4 · see citing works
EvalCouncil: A Committee-Based LLM Framework for Reliable and Unbiased Automated Grading
Anghel, C., Crăciun, M., Anghel, A. A., Cocu, A., Balau, A. S., Andrei, C. A., Maier, C., Dragosloveanu, Ş., Nedelea, D.-G., & Scheau, C. (2025) · Computers · doi:10.3390/computers14120530 · Cited by 4 · see citing works
EvalHack: Answer-Side Prompt Injection for Probing LLM Exam-Grading Panel Stability
Anghel, C., Crăciun, M., Cocu, A., Anghel, A. A., Balau, A. S., Istrate, A., & Anghele, A.-D. (2026) · Information · doi:10.3390/info17030297 · Cited by 3 · see citing works
RubiSCoT: A Framework for AI-Supported Academic Assessment
Fröhlich, T., & Schlippe, T. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-95-4423-3_20 · Cited by 4 · see citing works
Axioms for Retrieval-Augmented Generation
Reimer, J. H., Fröbe, M., Stein, B., Potthast, M., & Hagen, M. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744601 · Cited by 2 · see citing works
Anghel, C., Pecheanu, E., Anghel, A. A., Crăciun, M., & Cocu, A. (2026) · Computers · doi:10.3390/computers15030177 · Cited by 2 · see citing works
Anghel, C., Anghel, A. A., Crăciun, M., Balau, A. S., Istrate, A., Cocu, A., Andrei, C. A., & Anghele, A.-D. (2026) · Computers · doi:10.3390/computers15090571 · Cited by 2 · see citing works
Approach to Evaluating AI-Generated Educational Content: A Case Study on AI-Generated Tests
Maiatskaia, E., & Zabokritskaya, L. (2025) · 2025 IEEE Ural-Siberian Conference on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit65494.2025.11054236 · Cited by 1 · see citing works
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1 · see citing works
“However, Inflection AI (2024) noted that many of the reference answers used in MT-Bench were actually incorrect.”
SteadyEval: Robust LLM Exam Graders via Adversarial Training and Distillation
Anghel, C., Crăciun, M., Cocu, A., Anghel, A. A., & Istrate, A. (2026) · Computers · doi:10.3390/computers15010055 · Cited by 1 · see citing works
RO-FIN-LLM: A Benchmark with LLM-as-a-Judge and Human Evaluators for Romanian Tax and Accounting
Olariu, M.-E., Buinceanu, V.-G., Simionescu, C., Dospinescu, O., Georgescu, R., Tudor, C., Iftene, A., & Bores, A.-M. (2026) · Systems · doi:10.3390/systems14030244 · Cited by 1 · see citing works
Liu, T., Ye, L., & Yan, W. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100565 · Cited by 3 · see citing works
A Modular Approach to Automated News Generation Using Large Language Models
Gambino, O. J., García-Mendoza, C.-V., Minutti, B. H., Zapata-Manilla, C.-M., Bernal-Trani, M.-A., & Calvo, H. (2026) · Information · doi:10.3390/info17040319 · Cited by 1 · see citing works
Anghel, C., Anghel, A. A., Cocu, A., Crăciun, M., Moldovanu, S., Vlase, M., & Sandu, C. (2026) · Informatics · doi:10.3390/informatics13090149 · Cited by 1 · see citing works
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Li, R., Du, M., Xu, B., Zhu, C., Wang, X., & Mao, Z. (2026) · arXiv · doi:10.48550/arxiv.2601.08536
“Rubrics designed by human experts also help ensure the consistency of LLM judge scores with human expert preferences to some extent [Pathak et al., 2025, Hashemi et al., 2024].”
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Yang, J., Hu, Z., Qiu, C., Deng, Z., Jiao, X., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2604.22891
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Liu, M. T.-Y., Harrington, A., Darrell, T., & Bai, Y. (2026) · arXiv · doi:10.48550/arxiv.2610.21179
Exploring and Enhancing Advanced MoE Models: From Deepspeed-MoE to DeepSeek-V3
Zhang, Z., Gao, Y., & Chen, D. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035928
Automated Criterion-Based Thai Text Evaluation with Rationale and Improvement Guidance Using LLMs
Luantangsrisuk, V., Saengthongpattana, K., Chai, W. N., Boonsri, N., Choorat, T., Chunu, S., & Ruangrajitpakorn, T. (2025) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp66160.2025.11320478
MERCI: A Multimodal Dataset for Personalised and Emotionally-Aware Dialogues
Althubyani, M., Meng, Z., Xie, S., Cruz, F., Razzak, I., Prasad, M., Sandoval, E. B., & Kocaballi, B. (2025) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi66578.2025.11339324
Optimizing automated scoring in ILSAs with prompt compression
Jung, J. Y., Bezirhan, U., & von Davier, M. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100558
ADSC: LLM-Augmented Dual-Stream Cooperative Learning for Robust Automated Essay Scoring
Zhang, K., Lai, P., Hu, J., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7072-0_18
Xuan, T. N., Thanh, T. N., Viet, V. H., & Quynh, T. D. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18316-3_27
Palani, S., & Setlur, V. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790735
EvaluAId: Human-AI Collaborative Evaluation of Open-Ended Student Essays
Zhang, C., Ju, K. P., Lu, X., Yen, Y.-C. G., & Rzeszotarski, J. M. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790814
MADCAP: A Multi-agent Deliberative Framework for Robust Assessment of Open-Ended Questions
Lin, S., He, W., & Li, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_32
Lou, Y., Luo, Y., Song, Y., Li, J., Li, X., & Zhou, X. Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0372-7_24
KBase Research Agent: Automated Multi-Agent Workflow Construction for Reproducible Genome Analysis *Preprint
Gupta, P., Riehl, W. J., Cashman, M., Chivian, D. C., Neely, C. J., Canon, R. S., Cottingham, R., Henry, C., Arkin, A. P., & Dehal, P. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.01.729336
The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing
Shergadwala, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806536
AmbiTab-Sports: An Ambiguity-Aware Benchmark for Tabular Question AnsweringPreprint
Billah, M. A. M., & Sarker, T. (2026) · Research Square · doi:10.21203/rs.3.rs-9539202/v1
Scaria, N., Kennedy, S. J. J., & Subramani, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_27
A Survey of AI Misconception and Adoption Among Indonesian K-12 Teachers
Aji, A. F., Amriani, A., Chevi, R., Purwarianti, A., & Wijaya, D. T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29773-0_20
KBase Research Agent: Automated Multi-Agent Workflow Construction for Reproducible Genome Analysis
Gupta, P., Riehl, W. J., Cashman, M., Chivian, D. C., Neely, C. J., Canon, R. S., Cottingham, R. W., Henry, C. S., Arkin, A. P., & Dehal, P. (2026) · Platform for Advanced Scientific Computing Conference (PASC) · doi:10.1145/3815572.3815744
EduArt-Bench: A Benchmark and Lightweight Scoring Calibration for K-12 Art Education
Huang, Z., Chen, S., Chen, Y., Zhou, X. Y., & Xu, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29760-0_18
KI-gestützte Bewertung akademischer Arbeiten
Fröhlich, T. (2026) · Springer eBooks · doi:10.1007/978-3-658-47898-8_15-1
Anghel, C., Anghel, A. A., Crăciun, M., Cocu, A., Moldovanu, S., & Sandu, C. (2026) · AI · doi:10.3390/ai7080286
iBRAHIMOĞLU, N., Demi̇rezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1 · see citing works
A Multi-agent AI System for Automated Patent Deep Research and Analysis
Sofean, M., & Aras, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-34374-1_20
Austria, D., Williams, G. L., Girardo, C., Arowolo, M. O., Mishra, M., Pope, C. A., Axon, R. N., & Hill, J. B. (2026) · JMIR Formative Research · doi:10.2196/100423
Bushra, F., Hassan, M., Gupta, A. B., & Hassan, M. M. (2026) · International Conference on Power, Electronics, Communications, Computing, and Intelligent Infrastructure (PECCII) · doi:10.1109/peccii70991.2026.11661998
"Great! The Next Step Is...": In Pursuit of Proactive Assistance with Multimodal Foundation Models
Andrist, S., Bohus, D., Stiber, M., Bao, Y., Schoonbeek, T. J., & Horvitz, E. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831228
NormaScore: Intelligent Database Normalization Assessment Framework for DBMS Education
Ramachandran, R., Veena, G., Prasad, A., Sujitha, A., & Ravikumar, B. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10100343
- Authors
- Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C.
- Affiliations
- Microsoft (United States) (US)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 13806-13834
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671506
- Cited by
- 47
- References
- 0
- Citations by year
- 2026: 34 · 2025: 13
- FWCI
- 13.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- rubric-based assessment, large language models, LLM calibration, human-AI agreement, user satisfaction prediction, dialogue systems, citation quality, naturalness, brevity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
