Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Open accessAbstract
Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent forms of assessment. Additionally, they do not possess the ability to evaluate based on custom evaluation criteria, focusing instead on general attributes like helpfulness and harmlessness. To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements. Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges among all tested open evaluator LMs. Our models, code, and data are all publicly available at https://github.com/prometheus-eval/prometheus-eval.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most prior evaluation work relies on proprietary GPT models, raising issues of transparency and high costs at scale.
“Most prior evaluation work relies on proprietary GPT models (Wang et al., 2023a;Dubois et al., 2023;Liu et al., 2023;Fu et al., 2024), raising issues of transparency and high costs at scale (Kim et al., 2023(Kim et al., , 2024)).”
From YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering (2025) · cites this paper as Kim et al., 2024
Specialized models like UniEval and Prometheus 2, and LLMs like GPT-4 can be used to judge the factuality of a passage.
“Specialized models like UniEval (Zhong et al., 2022) and Prometheus 2 (Kim et al., 2024), and LLMs like GPT-4 (OpenAI et al., 2024b) can be used to judge the”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Kim et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 42
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
- Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based eval- uators through multi-agent debate. arXiv preprint arXiv:2308.07201. · arXiv
- Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting lan- guage models with high-quality feedback. arXiv preprint arXiv:2310.01377. · arXiv
- Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2022. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378. · arXiv
- Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Al- pacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387. · arXiv
- Markus Freitag, David Grangier, and Isaac Caswell. 2020. Bleu might be guilty but references are not innocent. arXiv preprint arXiv:2004.06063. · arXiv
- Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383. · arXiv
- Sebastian Gehrmann, Tosin Adewumi, Karmanya Ag- garwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021. The gem benchmark: Natural lan- guage generation, its evaluation and metrics. arXiv preprint arXiv:2102.01672. · arXiv
- Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, et al. 2022. Gemv2: Multilingual nlg benchmarking in a single line of code. arXiv preprint arXiv:2206.11249. · arXiv
- Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee's mergekit: A toolkit for merging large lan- guage models. arXiv preprint arXiv:2403.13257. · arXiv
- Suchin Gururangan, Margaret Li, Mike Lewis, Wei- jia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2023. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177. · arXiv
- Michael Hanna and Ondřej Bojar. 2021. A fine-grained analysis of bertscore. In Proceedings of the Sixth Conference on Machine Translation, pages 507-517.
- Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi. 2022. Edit- ing models with task arithmetic. arXiv preprint arXiv:2212.04089. · arXiv
- Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023a. Personalized soups: Personalized large lan- guage model alignment via post-hoc parameter merg- ing. arXiv preprint arXiv:2310.11564. · arXiv
- Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyung- jae Lee, and Minjoon Seo. 2023b. Exploring the benefits of training expert language models over in- struction tuning. arXiv preprint arXiv:2302.03202. · arXiv
- Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
- Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. · arXiv
- Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2023b. Tigerscore: Towards building explainable met- ric for all text generation tasks. arXiv preprint arXiv:2310.00752. · arXiv
- Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023c. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561. · arXiv
- Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evalua- tion capability in language models. arXiv preprint arXiv:2310.08491. · arXiv
- Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. 2024. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models. arXiv preprint arXiv:2406.05761. · arXiv
- Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. · arXiv
- Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. 2024. Prometheus- vision: Vision-language model as a judge for fine-grained evaluation. arXiv preprint arXiv:2401.06591. · arXiv
- Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470. · arXiv
- Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettle- moyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03306. · arXiv
- Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. 2024. Leveraging large language models for nlg evaluation: A survey. arXiv preprint arXiv:2401.07103. · arXiv
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. Gpte- val: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. · arXiv
- Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703- 17716.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
- Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2024. Rewarded soups: towards pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards. Ad- vances in Neural Information Processing Systems, 36.
- Natalie Schluter. 2017. The limits of automatic sum- marisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 41-45. Association for Computational Linguistics.
- Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Ja- cob Kahn, Daniel Li, Wen-tau Yih, Jason We- ston, et al. 2024. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm. arXiv preprint arXiv:2403.07816. · arXiv
- Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024. Arithmetic control of llms for di- verse user preferences: Directional preference align- ment with multi-objective rewards. arXiv preprint arXiv:2402.18571. · arXiv
- Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. 2023a. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. arXiv preprint arXiv:2312.08935. · arXiv
- Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023b. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087. · arXiv
- Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Re- solving interference when merging models. Ad- vances in Neural Information Processing Systems, 36.
- Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928. · arXiv
- Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. Language models are super mario: Absorb- ing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675. · arXiv
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. · arXiv
- Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631. · arXiv
Cited by
Sorted by citations
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Towards lifelong dialogue agents via timeline-based memory management
Ong, K. T.-I., Kim, N., Gwak, M., Chae, H., Kwon, T., Jo, Y., Hwang, S.-W., Lee, D., & Yeo, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.435 · Cited by 18 · see citing works
RewardBench: Evaluating Reward Models for Language Modeling
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL 2025 · doi:10.18653/v1/2025.findings-naacl.96 · Cited by 13 · see citing works
Self-citationYao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., & Yu, H. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.538 · Cited by 12 · see citing works
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.911 · Cited by 9 · see citing works
Palmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3 · see citing works
“Our results in Sections 5.1 and 5.3 demonstrate improved performance by directing the model to specifically address constraints, even when it is refining its responses using only its own feedback.”
InfluentialAdversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text
Cheng, Y., Sadasivan, V. S., Saberi, M., Saha, S., & Feizi, S. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-1588 · Cited by 3 · see citing works
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 2 · see citing works
Huang, H., DeLucia, A., Tiyyala, V. M., & Dredze, M. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.693 · Cited by 2 · see citing works
“{Passage Snippet} mentions that ..., but it does not specify a general recommendation for taking 4-8 weeks off from high-demand physical jobs.”
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., & Berant, J. (2024) · arXiv · doi:10.48550/arxiv.2407.15711 · Cited by 11 · see citing works
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06709
“…al., 2023, Bai et al., 2022b, Ye et al., 2024] to models fine-tuned specifically for evaluation tasks [Wang et al., 2024b, Zhu et al., Li et al., a, Kim et al., 2024]—as well as in the evaluation attributes they target (e.g., faithfulness, relevance) and in their scoring mechanisms, either using…”
Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Goo, T., Kim, J., Han, K., Jo, G., Kim, J.-S., & Kim, T.-H. (2026) · arXiv · doi:10.48550/arxiv.2607.20527
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Liu, Y., James, J., Wang, Y., Zhao, K., Xiao, C., & Lin, C. (2026) · arXiv · doi:10.48550/arxiv.2610.07109
“Other work improves individual judges through specialized evaluator models (Kim et al., 2024; Huang et al., 2025a), self-rationalization (Trivedi et al., 2024), check-list decomposition (Lee et al., 2025), and adaptive debiasing (Yang et al., 2026).”
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
RoPoLL: Robust Panel of LLM Judges
Acharya, A., Pan, K. W., & Verkhovsky, B. (2026) · arXiv · doi:10.48550/arxiv.2606.30931
LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
Li, S., Xu, C., Wang, J., Gong, X., Chen, C., Zhang, J., Wang, J., Lam, K.-Y., & Ji, S. (2025) · arXiv · doi:10.48550/arxiv.2506.09443
“Kim et al. released Prometheus 2 [24], an open-source evaluator achieving top alignment with human and GPT-4 judgments.”
- Authors
- Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., & Seo, M.
- Affiliations
- LG AI Research (South Korea) (KR)
- Massachusetts Institute of Technology (US)
- Allen Institute for Artificial Intelligence (US)
- Carnegie Mellon University (US)
- University of Illinois Chicago (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2405.01535
- OpenAlex
- W4396651266
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Korea Advanced Institute of Science and Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- open-source language models, language model evaluation, pairwise ranking, direct assessment, human evaluation alignment, GPT-4 comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
