CoChat
Sign Up
PreprintCited by 2

Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation

Huang, Tzu-Heng, Vishwakarma, Harit, Sala, Frederic·2025·arXiv
DOI 10.48550/arxiv.2506.10403arXiv 2506.10403
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA (Program-As-a-Judge for Automated Model Assessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.10403

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We follow the suggested criteria from Huang et al. (2025), which are studied that can be translated into executable code.

    “We follow the suggested criteria from Huang et al. (2025), which are studied that can be translated into executable code.”

    From CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (2026) · cites this paper as Huang et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 30

  1. Wei, H.; He, S.; Xia, T.; Liu, F.; Wong, A.; Lin, J.; Han, M. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.13006 2024, · arXiv
  2. Chiang, W.-L.; Zheng, L.; Sheng, Y.; Angelopoulos, A. N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M.; Gonzalez, J. E.; others Chatbot arena: An open platform for evaluating llms by human preference. Forty-first International Conference on Machine Learning. 2024.
  3. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; others Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 2023, 36, 46595-46623.
  4. Zhuge, M.; Zhao, C.; Ashley, D.; Wang, W.; Khizbullin, D.; Xiong, Y.; Liu, Z.; Chang, E.; Krishnamoorthi, R.; Tian, Y.; others Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934 2024, · arXiv
  5. Wettig, A.; Gupta, A.; Malik, S.; Chen, D. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739 2024, · arXiv
  6. Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems 2017, 30.
  7. Zhu, L.; Wang, X.; Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631 2023, · arXiv
  8. Yu, Q.; Zheng, Z.; Song, S.; Li, Z.; Xiong, F.; Tang, B.; Chen, D. xfinder: Robust and pinpoint answer extraction for large language models. arXiv preprint arXiv:2405.11874 2024, · arXiv
  9. Adila, D.; Zhang, S.; Han, B.; Wang, Y. Discovering bias in latent space: An unsupervised debiasing approach. arXiv preprint arXiv:2406.03631 2024, · arXiv
  10. Chen, G. H.; Chen, S.; Liu, Z.; Jiang, F.; Wang, B. Humans or LLMs as the Judge? A Study on Judgement Bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA, 2024; pp 8301-8327.
  11. Ye, J.; Wang, Y.; Huang, Y.; Chen, D.; Zhang, Q.; Moniz, N.; Gao, T.; Geyer, W.; Huang, C.; Chen, P.-Y.; others Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736 2024, · arXiv
  12. Ratner, A. J.; De Sa, C. M.; Wu, S.; Selsam, D.; Ré, C. Data programming: Creating large training sets, quickly. Advances in neural information processing systems 2016, 29.
  13. Ratner, A.; Bach, S. H.; Ehrenberg, H.; Fries, J.; Wu, S.; Ré, C. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases. 2017; p 269.
  14. Ratner, A.; Hancock, B.; Dunnmon, J.; Sala, F.; Pandey, S.; Ré, C. Training complex models with multi-task weak supervision. Proceedings of the AAAI Conference on Artificial Intelligence. 2019; pp 4763-4771.
  15. Fu, D.; Chen, M.; Sala, F.; Hooper, S.; Fatahalian, K.; Ré, C. Fast and three-rious: Speeding up weak supervision with triplet methods. International Conference on Machine Learning. 2020; pp 3280-3291.
  16. Lambert, N.; Pyatkin, V.; Morrison, J.; Miranda, L.; Lin, B. Y.; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y.; others Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787 2024, · arXiv
  17. Kim, S.; Shin, J.; Cho, Y.; Jang, J.; Longpre, S.; Lee, H.; Yun, S.; Shin, S.; Kim, S.; Thorne, J.; others Prometheus: Inducing fine-grained evaluation capability in language models. The Twelfth International Conference on Learning Representations. 2023.
  18. Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; others Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024, · arXiv
  19. Feng, S.; Balachandran, V.; Bai, Y.; Tsvetkov, Y. FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore, 2023; pp 933-952.
  20. Vidgen, B.; Thrush, T.; Waseem, Z.; Kiela, D. Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection. ACL. 2021.
  21. Wang, Y.; Yu, Z.; Zeng, Z.; Yang, L.; Wang, C.; Chen, H.; Jiang, C.; Xie, R.; Wang, J.; Xie, X.; Ye, W.; Zhang, S.; Zhang, Y. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. International Conference on Learning Representations (ICLR) 2024,
  22. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; others Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023, · arXiv
  23. Team, G.; Riviere, M.; Pathak, S.; Sessa, P. G.; Hardin, C.; Bhupatiraju, S.; Hussenot, L.; Mesnard, T.; Shahriari, B.; Ramé, A.; others Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 2024, · arXiv
  24. Li, H.; Dong, Q.; Chen, J.; Su, H.; Zhou, Y.; Ai, Q.; Ye, Z.; Liu, Y. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 2024, · arXiv
  25. Badshah, S.; Sajjad, H. Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text. arXiv preprint arXiv:2408.09235 2024, · arXiv
  26. Huang, T.-H.; Cao, C.; Bhargava, V.; Sala, F. The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. Neural Information Processing Systems (NeurIPS). 2024.
  27. Huang, T.-H.; Cao, C.; Schoenberg, S.; Vishwakarma, H.; Roberts, N.; Sala, F. Scriptoriumws: A code generation assistant for weak supervision. arXiv preprint arXiv:2502.12366 2025, · arXiv
  28. Roberts, N.; Li, X.; Huang, T.-H.; Adila, D.; Schoenberg, S.; Liu, C.-Y.; Pick, L.; Ma, H.; Albarghouthi, A.; Sala, F. Autows-bench-101: Benchmarking automated weak supervision with 100 labels. Advances in Neural Information Processing Systems 2022, 35, 8912-8925.
  29. Huang, T.-H.; Bilkhu, M.; Sala, F.; Movellan, J. Evaluating Sample Utility for Data Selection by Mimicking Model Weights. arXiv preprint arXiv:2501.06708 2025, · arXiv
  30. Vishwakarma, H.; Sala, F. Lifting weak supervision to structured prediction. Advances in Neural Information Processing Systems 2022, 35, 37563-37574.

Cited by

Sorted by citations

Authors
Huang, T.-H., Vishwakarma, H., & Sala, F.
Venue
arXiv
Publisher
arXiv
Published
2025
DOI
10.48550/arxiv.2506.10403
arXiv
2506.10403
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review