Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
Open accessAbstract
The potential of using Large Language Models (LLMs) themselves to evaluate LLM outputs offers a promising method for assessing model performance across various contexts. Previous research indicates that LLM-as-a-judge exhibits a strong correlation with human judges in the context of general instruction following. However, for instructions that require specialized knowledge, the validity of using LLMs as judges remains uncertain. In our study, we applied a mixed-methods approach, conducting pairwise comparisons in which both subject matter experts (SMEs) and LLMs evaluated outputs from domain-specific tasks. We focused on two distinct fields: dietetics, with registered dietitian experts, and mental health, with clinical psychologist experts. Our results showed that SMEs agreed with LLM judges 68% of the time in the dietetics domain and 64% in mental health when evaluating overall preference. Additionally, the results indicated variations in SME-LLM agreement across domain-specific aspect questions. Our findings emphasize the importance of keeping human experts in the evaluation process, as LLMs alone may not provide the depth of understanding required for complex, knowledge specific tasks. We also explore the implications of LLM evaluations across different domains and discuss how these insights can inform the design of evaluation workflows that ensure better alignment between human experts and LLMs in interactive systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM-as-a-judge evaluation is known to be unreliable for expert-domain quality judgment.
“LLM-as-a-judge evaluation is known to be unreliable for expert-domain quality judgment (Dorner et al., 2025;Szymanski et al., 2025).”
From On Epistemic Diversity in Large Language Models (2026) · cites this paper as Szymanski et al., 2025
The paper reports that LLM judges agreed with subject-matter experts 68% of the time in dietetics and 64% in mental health (p. 1). Its Discussion (p. 13) warns against relying solely on LLMs for expert-knowledge tasks. These findings support a concern about reliability, but the claim states it categorically across expert domains, while the study examines two domains and describes the broader validity question as uncertain. (LLM, conf=0.83)
Experimental — accuracy not yet measuredSzymanski et al. (2025) found only moderate agreement between GPT-4's ratings and domain professionals on a dataset of 25 questions in dietetics and mental health.
“Szymanski et al. (2025) examines LLM-as-a-judge in the domains of dietetics and mental health by comparing GPT-4's ratings to those of domain professionals, finding only moderate agreement on a dataset of 25 questions.”
From No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding (2025) · cites this paper as Szymanski et al. (2025)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,805 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data
Xu, X., Yao, B., Dong, Y., Gabriel, S., Yu, H., Hendler, J., Ghassemi, M., Dey, A. K., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3643540 · Cited by 278 · see citing works
Precision nutrition: A systematic literature review
Kirk, D., Catal, C., & Tekinerdogan, B. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104365 · Cited by 238 · see citing works
The now and future of ChatGPT and GPT in psychiatry
Cheng, S., Chang, C., Chang, W., Wang, H., Liang, C., Kishimoto, T., Chang, J. P., Kuo, J. S., & Su, K. (2023) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13588 · Cited by 197 · see citing works
Design and evaluation guidelines for mental health technologies
Doherty, G., Coyle, D., & Matthews, M. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.02.006 · Cited by 179 · see citing works
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A., & Arawjo, I. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676450 · Cited by 151 · cited 1× · see citing works
“An average LLM user may not recognize inaccuracies or the importance of certain dietary restrictions and mental health protocols and is less likely to be aware of the latest research and clinical guidelines.”
Suh, S., Chen, M., Min, B., Li, T. J.-J., & Xia, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642400 · Cited by 148 · see citing works
Self-citationThe credibility of dietary advice formulated by ChatGPT: Robo-diets for people with food allergies
Niszczota, P., & Rybicka, I. (2023) · Nutrition · doi:10.1016/j.nut.2023.112076 · Cited by 141 · see citing works
Yang, Z., Khatibi, E., Nagesh, N., Abbasian, M., Azimi, I., Jain, R., & Rahmani, A. M. (2024) · Smart Health · doi:10.1016/j.smhl.2024.100465 · Cited by 121 · see citing works
Zhang, Z., Gao, J., Dhaliwal, R. S., & Li, T. J.-J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606800 · Cited by 108 · see citing works
Self-citationIs ChatGPT an Effective Tool for Providing Dietary Advice?
Ponzo, V., Goitre, I., Favaro, E., Merlo, F. D., Mancino, M. V., Riso, S., & Bo, S. (2024) · Nutrients · doi:10.3390/nu16040469 · Cited by 100 · see citing works
Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
Tseng, Y.-M., Huang, Y.-C., Hsiao, T.-Y., Chen, W.-L., Huang, C.-W., Meng, Y., & Chen, Y.-N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.969 · Cited by 87 · see citing works
ChatGPT as a Virtual Dietitian: Exploring Its Potential as a Tool for Improving Nutrition Knowledge
Garcia, M. B. (2023) · Applied System Innovation · doi:10.3390/asi6050096 · Cited by 84 · see citing works
Gao, J., Guo, Y., Lim, G., Zhang, T., Zhang, Z., Li, T. J.-J., & Perrault, S. T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642002 · Cited by 78 · see citing works
Self-citationPaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis
Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L., & Li, T. J.-J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581352 · Cited by 72 · see citing works
Self-citationComparison of Answers between ChatGPT and Human Dieticians to Common Nutrition Questions
Kirk, D., van Eijnatten, E. J. M., & Camps, G. (2023) · Journal of Nutrition and Metabolism · doi:10.1155/2023/5548684 · Cited by 67 · see citing works
King, D. R., Nanda, G., Stoddard, J., Dempsey, A., Hergert, S., Shore, J. H., & Torous, J. (2023) · Current Psychiatry Reports · doi:10.1007/s11920-023-01477-x · Cited by 63 · see citing works
Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools
Lu, Y., Zhang, C., Zhang, I., & Li, T. J.-J. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519809 · Cited by 63 · see citing works
Self-citationLouie, R., Nandi, A., Fang, W., Chang, C. K., Brunskill, E., & Yang, D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.591 · Cited by 61 · see citing works
Large Language Models in Mental Health Care: A Systematic Scoping Review (Preprint)
hua, Y., Liu, F., Yang, K., Li, Z., Sheu, Y.-H., Zhou, P., Moran, L. V., Ananiadou, S., Clifton, D. A., & Beam, A. (2024) · doi:10.2196/preprints.64088 · Cited by 46 · see citing works
EvaluLLM: LLM assisted evaluation of generative outputs
Desmond, M. J., Ashktorab, Z., Pan, Q., Dugan, C., & Johnson, J. M. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640544.3645216 · Cited by 34 · see citing works
Szymanski, A., Wimer, B. L., Anuyah, O., Eicher‐Miller, H. A., & Metoyer, R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641924 · Cited by 30 · see citing works
Self-citationChainForge: An open-source visual programming environment for prompt engineering
Arawjo, I., Vaithilingam, P., WATTENBERG, M. P., & Glassman, E. L. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586182.3616660 · Cited by 16 · see citing works
Vo, R., Smith, M., & Patton, N. (2020) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.12820 · Cited by 14 · see citing works
Influences on clinical judgement in mental health nursing
Martin, P. J. (1999) · Journal of research in nursing · doi:10.1177/136140969900400405 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [2][2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). - · arXiv
- [3][3] Jaewoo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, and Gunhee Kim. 2024. TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models. arXiv preprint arXiv:2405.18027 (2024). - · arXiv
- [6][6] Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589-597. -
- [8][8] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669 (2024). - · arXiv
- [10][10] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 (2024). Manuscript submitted to ACM · arXiv
Cited by
Sorted by citations
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445 · see citing works
Retrieval-augmented generation elevates local LLM quality in radiology contrast media consultation
Wada, A., Tanaka, Y., Nishizawa, M., Yamamoto, A., Akashi, T., Hagiwara, A., Hayakawa, Y. K., Kikuta, J., Shimoji, K., Sano, K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01802-z · Cited by 58 · see citing works
Chai, Y., Xie, H., & Qin, J. S. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11405-5 · Cited by 34 · see citing works
A comprehensive survey of AI agents in healthcare
Xu, G., Li, X., Chen, Y., Duan, Y., Wu, S., Yu, H., Chiu, C.-H., Ni, J., Tang, N., Li, T. J.-J., et al. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105045 · Cited by 32 · see citing works
Self-citationKizilkaya, D., Sajja, R., Sermet, Y., & Demir, İ. (2025) · Environmental Data Science · doi:10.1017/eds.2025.10006 · Cited by 19 · see citing works
Nawara, D., & Kashef, R. F. (2025) · IEEE Access · doi:10.1109/access.2025.3599832 · Cited by 18 · see citing works
Tariq, S., Woo, S. S., Singh, P., Irmalasari, I., Gupta, S., & Gupta, D. (2025) · Proceedings of the 33rd ACM International Conference on Multimedia · doi:10.1145/3746027.3755786 · Cited by 15 · see citing works
Large language models for mental health diagnosis and treatment: a survey
Ghorbian, M., & Ghobaei‐Arani, M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11418-0 · Cited by 15 · see citing works
Qiao, S., Fang, X., Wang, J., Zhang, R., Li, X., & Kang, Y. (2024) · medRxiv · doi:10.1101/2024.09.16.24313707 · Cited by 13 · see citing works
MetricMate: An Interactive Tool for Generating Evaluation Criteria for LLM-as-a-Judge Workflow
Gebreegziabher, S. A., Chiang, C., Wang, Z., Ashktorab, Z., Brachman, M., Geyer, W., Li, T. J.-J., & Gómez-Zará, D. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729199 · Cited by 10 · see citing works
Self-citationHow well do DeepSeek, ChatGPT, and Gemini respond to water science questions?
Hosseini, S. H., & Pourzangbar, A. (2025) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2025.106772 · Cited by 7 · see citing works
Kairat, M., Adilmetova, G., Ibraimova, I., Gaipov, A., Varol, H. A., & Chan, M. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14228033 · Cited by 6 · see citing works
MedReadCtrl: Personalizing medical text generation with readability-controlled instruction learningPreprint
Tran, H., Yao, Z., Jang, W. S., Sultana, S., Chang, A., Zhang, Y., & Hong, Y. (2025) · medRxiv · doi:10.1101/2025.07.09.25331239 · Cited by 9 · see citing works
Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences
Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00843-4 · Cited by 13 · see citing works
Zhang, J., Xia, Y., Zhong, H., Lu, W., Deng, Q., & Wan, C. (2025) · Mathematics · doi:10.3390/math14010090 · Cited by 4 · see citing works
Bourgeois, J. P., & Ellingson, H. (2025) · Medical Reference Services Quarterly · doi:10.1080/02763869.2025.2537075 · Cited by 6 · see citing works
Janusko, T., Bochnia, R., Harnisch, M., Krauß, A.-M., Richter, D., Hempel, G., Tomschke, S., Anke, J., & Thiele, M. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96235-6_30 · Cited by 3 · see citing works
Meisenbacher, S., Klymenko, A., & Matthes, F. (2025) · Proceedings of the 2025 Workshop on Human-Centered AI Privacy and Security · doi:10.1145/3733816.3760760 · Cited by 4 · see citing works
Roumeliotis, K. I., Margaris, D., Spiliotopoulos, D., & Vassilakis, C. (2026) · Future Internet · doi:10.3390/fi18020112 · Cited by 3 · see citing works
Ghorbian, M., & Ghobaei-Arani, M. (2025) · Journal of Big Data · doi:10.1186/s40537-025-01304-5 · Cited by 4 · see citing works
Zhang, T., Patterson, L., Webb, B., Jin, Z., & Beiting-Parrish, M. (2026) · Large-scale Assessments in Education · doi:10.1186/s40536-026-00287-w · Cited by 2 · see citing works
Can LLMs Automate Fact-Checking Article Writing?
Sahnan, D., Corney, D. P. A., Larraz, I., Zagni, G., Míguez, R., Xie, Z., Gurevych, I., Churchill, E. F., Chakraborty, T., & Nakov, P. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.644 · Cited by 2 · see citing works
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1 · see citing works
“This issue can be masked by aggregate statistics, especially when the judge performs well on the underlying benchmark itself.”
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Li, C., Akhtar, Z., Kwak, M., Ji, Y., Zhang, H., Obi, T., Ren, Y., Wu, X., Sivarajkumar, S., Lehmann, H. P., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25933 · Cited by 1 · see citing works
Michalowski, M., Wilk, S., Bauer, J. M., Carrier, M., Viktor, H., & Michalowski, W. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-95838-0_29 · Cited by 1 · see citing works
Chen, Y., Shen, H., Xie, B., Wang, J., Zhao, W., Pan, Z., Fu, C., Wang, X., Zhang, Z., & Cai, C. (2025) · International Conference on Artificial Intelligence Technologies and Applications (ICAITA) · doi:10.1109/icaita67588.2025.11137870 · Cited by 1 · see citing works
Koenitz, H., Barbara, J., & Eladhari, M. P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-12408-1_14 · Cited by 1 · see citing works
Huang, R., Cecil, J., Freedman, M., & Chattopadhyay, S. (2026) · Journal of Medical Internet Research · doi:10.2196/88126 · Cited by 1 · see citing works
Chang, M., Srinivasan, A., & Palani, S. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789077 · Cited by 1 · see citing works
Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study
Szymanski, A., Anuyah, O., Li, T. J.-J., & Metoyer, R. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789105 · Cited by 1 · see citing works
Self-citationRoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics
Kim, C. Y., White, N. T., He, E. C. L., Sala, F., & Mutlu, B. (2026) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3757279.3785550 · Cited by 1 · see citing works
PolicyPad: Collaborative Prototyping of LLM Policies
Feng, K. J. K., Kuo, T.-S., Chen, Q. Z., Cheong, I., Holstein, K., & Zhang, A. X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791689 · Cited by 3 · see citing works
Gao, J., Chen, L., Lin, W., Peng, P., Xu, Y., Lan, T., Lu, F., & Claramunt, C. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104829 · Cited by 1 · see citing works
Evaluating the Evaluator: LLM Evaluators for Patient-Facing AI Agents
Mastrianni, A., Andreadis, K., Chen, J., Rodriguez, D. V., & Mann, D. (2026) · ACM Interactive Health Conference · doi:10.1145/3786579.3804927 · Cited by 1 · see citing works
Yajnik, A., & Pushkala, S. A. (2026) · IEEE Access · doi:10.1109/access.2026.3719291 · Cited by 1 · see citing works
Leveraging Large Language Models for Early Study Optimization in Educational Research
Friday, M., Vaccaro, M., & Zaghi, A. E. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--56927 · Cited by 1 · see citing works
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Deng, H., Ke, X., Li, Y., Hu, R., Huang, D., Wong, D. F., Wang, Y., Liu, X., & Zhang, M. (2026) · arXiv · doi:10.48550/arxiv.2605.07905
On Epistemic Diversity in Large Language Models
Kirsten, E., Krämer, N., & Zafar, M. B. (2026) · arXiv · doi:10.48550/arxiv.2609.04835
“LLM-as-a-judge evaluation is known to be unreliable for expert-domain quality judgment (Dorner et al., 2025; Szymanski et al., 2025).”
Chao, C., Kumar, A., Mishra, A., Wang, Y.-C., Tsai, C.-M., Ali, N. B., Sharma, S., Farina, J., Arsanjani, R., Jiang, Y., et al. (2025) · medRxiv · doi:10.1101/2025.05.17.25327814
Beyond the Mask: Benchmarking Multimodal Large Language Models to Expose Phishing WebsitesPreprint
López-Fueyo, A., & deAndrés‐Galiana, E. J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5428153
Steering Semantic Data Processing With DocWrangler
Shankar, S., Chopra, B., Hasan, M., Lee, S., Hartmann, B., Hellerstein, J. M., Parameswaran, A., & Wu, E. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747625 · Cited by 2 · see citing works
When Context Grows, So Does the Challenge: Human Oversight in LLM Evaluation of Financial Tables
Sehanobish, A., Anderson, S., Michel, G., & Arov, M. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746058.3758457
Wang, Y., & Chen, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-11442-6_20
de Sacy, A. S., Faci, A., Maronet, L., & Pouyllau, S. (2025) · Humanités numériques · doi:10.4000/15icj
Han, L., Wei, M., Chen, Q., Wang, A., Pang, R., & Yip, D. K. M. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789085
The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes
Gebreegziabher, S. A., Yang, Y., Chiang, C., Yoo, H., Chen, C., Do, H. J., Ashktorab, Z., Geyer, W., Gómez-Zará, D., & Li, T. J.-J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789219
Self-citationHou, Y., 侯凱傑, Du, Y., & Han, H. (2025) · International Conference on Electronic Information Engineering and Computer Communication (EIECC) · doi:10.1109/eiecc67963.2025.11409491
EvalAgent: Interactive Comparative Evaluation of Computer-Using GUI Agents
Yang, Y., Gebreegziabher, S. A., Yoo, H., Chiang, C., Chen, C., Szymanski, A., Do, H. J., Ashktorab, Z., Geyer, W., Gómez-Zará, D., et al. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3795096
Self-citationDesigning for Social Collaboration and Human-AI Alignment
Chiang, C. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742414.3789236
Karp, M., Kubaszewska, A., Król, M., Król, R., Wydmański, W., Smywiński-Pohl, A., & Szymański, M. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09505-w · Cited by 2 · see citing works
PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading
Wu, Y., Liu, X., Feng, Y., Ding, J., & Ma, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792597
Beyond Accuracy: Experts See AI Fact-Checks as Accurate but Less Useful
Jia, C., Gondimalla, A., Zhang, A., Mullings, D. J., Boltz, A., & Lee, M. K. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791625 · Cited by 1 · see citing works
Anand, A. S., Ganesan, D., & Karkar, R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791866
iRULER: Intelligible Rubric-Based User-Defined LLM Evaluation for Revision
Bai, J., Cheong, W. S., Müller, P., & Lim, B. Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790539 · Cited by 1 · see citing works
LLM-as-a-Judge for mental support: A meta-evaluation using domain-specific platform data
Li, Y., Chen, X., & Dolata, M. (2026) · Electronic Markets · doi:10.1007/s12525-026-00890-3
Windisch, P., Weyrich, J., Dennstädt, F., Zwahlen, D. R., Förster, R., & Schröder, C. (2026) · Cureus · doi:10.7759/cureus.108666
Li, Z., & Qian, Q. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519324
Scaling Expert Feedback with Reflective Edit Propagation in Compositional Knowledge Bases
Guo, J., Li, X., Ono, J. P., He, W., & Liu, R. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813201
Martinenghi, A., Guidotti, S., Donabauer, G., Koyutürk, C., Ortiz-Beltrán, A., Theophilou, E., Chimisso, R., Bink, M., Garzotto, F., Taibi, D., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29763-1_29
Retrieval-augmented generation for medication safety: A case study using drug package inserts
Sun, Y., Wang, W., Cheng, W., Li, R., Liu, X., Pei, Q., Tang, J., & Yang, G. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106599
LLMs for Venture Capital Investment: Approaches and Open Problems
Galang, J. A. C., Mosca, E., Malberg, S., & Groh, G. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30809-2_12
Slany, E., Buchmann, S., Föhr, T. L., Amling, J., Frummet, A., Lang, M., & Scheele, S. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30809-2_28
Kalinich, M., Luccarelli, J., Maria, J. P. S., Flathers, M., Nguyen, A., Song, S. H., Makhoul, K. G., Criado, M. J. R., Ginapp, C. M., Hill, B., et al. (2026) · medRxiv · doi:10.64898/2026.07.10.26357132
LLMs as judges: Toward the LLM-assisted review of GSN-compliant assurance cases
Yu, G., Sivakumar, M., Belle, A. B., Ghari, S., Wang, S., & Lethbridge, T. C. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113036
Structure Matters: Evaluating Multi-Agents Orchestration in Generative Therapeutic Chatbots
Elahimanesh, S., Mohammadkhani, M., Movahed, S., Abootorabi, M. M., Salehi, S., & Edalat, A. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816273
Bieberstein, A., Schnitzer, B. L., Gampe, S., & Korn, O. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816221
Ein KI-Bewertungsagent für Qualitätssicherung von generierten Inhalten im E-Commerce
Bochnia, R., Janusko, T., Hempel, G., Tomschke, S., Anke, J., & Thiele, M. (2026) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-026-01315-7
Supporting Context-Aware AI-Augmented Data Extraction Features for Industrial Technical Documents
Nguyen, N. N. T., & Truong, H. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00392
Fowler, H. M., Xie, R., Saberi, M., & Braytee, A. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787607
A hybrid quantitative–qualitative RAG approach for territorial well-being decision support
Comiti, G., Bisgambiglia, P.-A., & Lameta, N. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.101005
Thinking Slow, Deciding Better? LRM versus LLM Performance on Organisational Decision Tasks
Brândaș, C., Andrei, A. L. B. U., Mariutac, G., Reychav, I., McHaney, R., Sharma, M., & Debons, N. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.08.153
Evaluating RAG-Based Course Advising Chatbots Using LLM-as-a-Judge Methodology
Buti, R., & Mohamed, A. (2026) · Western Canadian Conference on Computing Education · doi:10.1145/3828792.3828802
Windisch, P., Weyrich, J., Dennstädt, F., Zwahlen, D. R., Foerster, R., & Schroeder, C. (2026) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2026.06.1305
Byun, Y., Lee, D., Kwak, C., Yoon, C. Y., & Seo, Y. J. (2026) · Digital Health · doi:10.1177/20552076261490211
Generating UML class diagrams as conceptual models through large language models
Calamo, M., Mecella, M., & Snoeck, M. (2026) · Software & Systems Modeling · doi:10.1007/s10270-026-01427-0
A Framework for Assessing Hard-Constraint Compliance in LLMs
Furtado, L., Araújo, J., Oliveira, J., Pinheiro, V., Caminha, C., & Furtado, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39895-6_12
The Road to Knowledge Hell is Paved with AI Good Intentions
Celino, I. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41614-8_22
PETALE: A Fine-Grained Evaluation Framework for Process Modeling Methods
Kowalczyk, C., Brissard, A., Cuppens, F., & Zouaq, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40795-5_33
Agent-assisted modelling for urban drainage systems
Tobias, M. M., Marchionda, J., & Kerkez, B. (2026) · Digital Water · doi:10.1080/28375807.2026.2745819
On the Effects of Reasoning Effort and Prompt-Based Diversification on Scientific Ideation Diversity
Chinen, Y., & Ozaki, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-41086-3_18
When is an LLM-as-a-Judge Good Enough?
Wefers, S., & Breitenbach, T. (2026) · The European Journal on Artificial Intelligence · doi:10.1177/30504554261495282
- Authors
- Szymanski, A., Ziems, N., Eicher-Miller, H. A., Li, T. J.-J., Jiang, M., & Metoyer, R. A.
- Affiliations
- University of Notre Dame (US)
- Nutrition Science, Purdue University, West Lafayette, Indiana, USA,
- ORCID
- Venue
- Proceedings of the 30th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2025-03-24
- Pages
- 952-966
- OpenAlex
- W4408615073
- Cited by
- 77
- References
- 27
- Citations by year
- 2026: 48 · 2025: 28 · 2024: 1
- FWCI
- 469.83
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- USDA National Institute of Food and Agriculture (2021-67022-33447)
- Office of Naval Research (N00014-22-1-2507)
- Topics
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, LLM-as-a-judge, evaluation limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
