Humans or LLMs as the Judge? A Study on Judgement Bias
Open accessAbstract
Adopting human and large language models (LLM) as judges (a.k.a human-and LLM-as-ajudge) for evaluating the performance of LLMs has recently gained attention.Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results.In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias on LLM and human judges.We curate a dataset referring to the revised Bloom's Taxonomy and conduct thousands of evaluations.Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases.We further exploit these biases to conduct attacks on LLM judges.We hope that our work can notify the community of the bias and vulnerability of human-and LLMas-a-judge, as well as the urgency of developing robust evaluation systems 1 .Warning: we provide illustrative attack protocols to reveal the vulnerabilities of LLM judges, aiming to develop more robust ones.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The faithfulness of LLMs in establishing judgment is established in previous works.
“The faithfulness of LLMs in establishing judgment is established in previous works33–35.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [35]
The Abstract (page unavailable) says LLM judges have biases that question the reliability of their evaluations; it reports that even cutting-edge judges possess considerable biases. Section 2.2 (page unavailable) likewise says their annotation results require validation, rather than treating the faithfulness of their judgments as established. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredAlthough the 3-vote majority consensus protocol reduces stochastic fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family, and may be more lenient on verbose near-misses than a human annotator would be.
“Although the 3-vote majority consensus protocol reduces stochastic fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family (Zheng et al., 2023;Ye et al., 2024), and may be more lenient on verbose near-misses than a human annotator would be (Zheng et al., 2023;Ye et al., 2024;Chen et al., 2024).”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Chen et al., 2024
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredChen et al. (2024) shows that some biases are present in both LLM judges and human evaluators.
“Chen et al. (2024) shows that some of these biases are not only present in LLM judges, but also often in human evaluators.”
From Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges (2025) · cites this paper as Chen et al., 2024
Length bias is a known issue of evaluations that rely on model-based graders.
“Length bias is a known issue of evaluations that rely on model-based graders (Ye et al., 2024; Chen et al., 2024).”
From HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025) · cites this paper as Chen et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 7
- Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. 2021. All that's' human'is not gold: Evaluating hu- man evaluation of generated text. arXiv preprint arXiv:2107.00061. · arXiv
- Michael J. Keough Gerry P. Quinn. 2002. Experimental Design and Data Analysis for Biologists, 1 edition. Cambridge University Press.
- Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in gener- ating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-19.
- David M Howcroft, Anya Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel Van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In 13th International Conference on Natural Language Generation 2020, pages 169-182. Association for Computational Lin- guistics.
- Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Infor- mation Processing Systems.
- Kelly Weidner, Frederik Beuk, and Anjali Bal. 2020. Fake news and the willingness to share: a schemer schema and confirmatory bias perspective. Journal of Product & Brand Management, 29(2):180-187.
- Wangchunshu Zhou and Ke Xu. 2020. Learning to compare for better training and evaluation of open domain natural language generation models. In Pro- ceedings of the 34th AAAI Conference on Artificial Intelligence, pages 9717-9724.
Cited by
Sorted by citations
Large Language Models as Evaluators in Education: Verification of Feedback Consistency and Accuracy
Seo, H., Hwang, T., Jung, J., Kang, H., Namgoong, H., Lee, Y. H., & Jung, S. (2025) · Applied Sciences · doi:10.3390/app15020671 · Cited by 51 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
“Length bias is a known issue of evaluations that rely on model-based graders (Ye et al., 2024; Chen et al., 2024).”
Chai, Y., Xie, H., & Qin, J. S. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11405-5 · Cited by 34 · see citing works
Don’t Be Fooled: The Misinformation Effect of Explanations in Human–AI Collaboration
Spitzer, P., Holstein, J., Morrison, K., Holstein, K., Satzger, G., & Kühl, N. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2574511 · Cited by 12 · see citing works
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
“3%inEnglish,althoughit performedrelativelywellinKorean(35.4%).”
InfluentialMoëll, B., Farestam, F., & Beskow, J. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1557920 · Cited by 12 · see citing works
Hu, Z., Pan, Y., Li, Z., Huang, J., Nakagawa, S., Deng, J., Cai, S., & Ren, F. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104354 · Cited by 10 · see citing works
LLM-as-a-Judge Approaches as Proxies for Mathematical Coherence in Narrative Extraction
Keith, B. (2025) · Electronics · doi:10.3390/electronics14132735 · Cited by 8 · see citing works
Evaluation of cell type annotation reliability using a large language model-based identifier
Ye, W., Ma, Y., Xiang, J., Liang, H., Luo, J., Li, Y., Wang, T., Xiang, Q., Song, W., Li, W., et al. (2025) · Communications Biology · doi:10.1038/s42003-025-08745-x · Cited by 7 · see citing works
Zhang, W., Chen, Y., Liu, X., & Zhang, H. (2025) · Land Use Policy · doi:10.1016/j.landusepol.2025.107807 · Cited by 7 · see citing works
Peasley, D., Kuplicki, R., Sen, S., & Paulus, M. P. (2024) · JMIR Mental Health · doi:10.2196/68135 · Cited by 7 · see citing works
Don’t Do That! Reverse Role Prompting Helps Large Language Models Stay in Personality Traits
Chen, S., Taveekitworachai, P., Xia, Y., Li, X., Gursesli, M. C., Lanatà, A., Guazzini, A., & Thawonmas, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78453-8_7 · Cited by 6 · see citing works
Self-citationLessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
“(Devlin et al., 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), and ChatGPT, used as approximate stand-ins for shifts in how the community uses and therefore evaluates LMs. Common practice for evaluating autoregressive language models today diverges from the method described in the…”
Ahmadov, N., Muradova, A., & Aliyeva, A. (2026) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-026-10245-5 · Cited by 5 · see citing works
Assis, G., Freitas, C., & Paes, A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5814 · Cited by 5 · see citing works
Wang, M., Chen, Y., Huang, X., & Lai, Y. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100568 · Cited by 6 · see citing works
Gender inclusive language generation framework: A reasoning approach with RAG and CoT
Chinnan, S. P. M., Drury‐Grogan, M. L., & Chakravarthi, B. R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114092 · Cited by 4 · see citing works
Large Language Models in Smart Grid: Applications and Risks
Li, J., Xu, J., Yang, S., & Ngai, E. C. ‐. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-96146-5_1 · Cited by 3 · see citing works
Meisenbacher, S., Klymenko, A., & Matthes, F. (2025) · Proceedings of the 2025 Workshop on Human-Centered AI Privacy and Security · doi:10.1145/3733816.3760760 · Cited by 4 · see citing works
Aligning LLMs for the Classroom with Knowledge-Based Retrieval: A Comparative RAG Study
Jain, A. D., Cui, L., & Chen, S. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346682 · Cited by 6 · see citing works
Self-citationEvaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research
Madden, E. R. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0022 · Cited by 3 · see citing works
Preference Leakage: A Contamination Problem in LLM-as-a-judge
Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., & Liu, H. (2025) · arXiv · doi:10.48550/arxiv.2502.01534 · Cited by 2 · see citing works
“Despite its promising potential, recent studies have highlighted the vulnerabilities and biases of LLM-as-a-Judge [79, 72, 28, 4, 79, 21, 59, 54].”
Improving Open-Response Assessment with LearnLM
Thomas, D. R., Borchers, C., Bhushan, S., Kakarla, S., Houk, A., Abboud, R., Gupta, S., Gatz, E. E., & Koedinger, K. R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98462-4_28 · Cited by 2 · see citing works
IberBench: LLM evaluation on Iberian languages
González, J. Á., Borrego-Obrador, I., Herrero, Á. R., Sarvazyan, A. M., Chinea-Ríos, M., Basile, A., & Franco-Salvador, M. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101899 · Cited by 3 · see citing works
DxHF: Providing High-Quality Human Feedback for LLM Alignment with Interactive Decomposition
Shi, D., Cheng, F., Weinkauf, T., Oulasvirta, A., & El‐Assady, M. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747600 · Cited by 2 · see citing works
Benchmarking LLM-as-a-Judge Models for 5W1H Extraction Evaluation
Cassola-Bacallao, J. M., Morales-Donaire, J., Hernández-Montoya, P., & Keith-Norambuena, B. (2026) · Electronics · doi:10.3390/electronics15030659 · Cited by 2 · see citing works
You, J., Hong, S.-K., Liu, L., Lee, K., & Kwon, H.-Y. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777984 · Cited by 3 · see citing works
Zhang, T., Patterson, L., Webb, B., Jin, Z., & Beiting-Parrish, M. (2026) · Large-scale Assessments in Education · doi:10.1186/s40536-026-00287-w · Cited by 2 · see citing works
Anghel, C., Anghel, A. A., Crăciun, M., Balau, A. S., Istrate, A., Cocu, A., Andrei, C. A., & Anghele, A.-D. (2026) · Computers · doi:10.3390/computers15090571 · Cited by 2 · see citing works
Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
Eiras, F., Zemour, E., Lin, E., & Mugunthan, V. (2025) · arXiv · doi:10.48550/arxiv.2503.04474 · Cited by 1 · see citing works
“Chen et al. (2024) shows that some of these biases are not only present in LLM judges, but also often in human evaluators.”
Search Arena: Analyzing Search-Augmented LLMs
Miroyan, M., Wu, T.-H., King, L., Li, T., Pan, J., Hu, X., Chiang, W.-L., Angelopoulos, A. N., Darrell, T., Norouzi, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.05334 · Cited by 1 · see citing works
“First, the collected datamayreflectdemographicskewsandmaynotbefullyrepresentativeofthebroaderpopulation, asnot all users choose to vote and human judgments are inherently subjective [7, 12].”
Less Hallucination and More Factuality: Human Values Alignment in Text Summarization
Qiu, Z., & Xu, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-9671-7_14 · Cited by 1 · see citing works
Yang, B., Yu, X., Zheng, X., Nong, J., Liu, Z., Dai, X., & Xie, X. (2025) · Applied Sciences · doi:10.3390/app151910447 · Cited by 1 · see citing works
Self-citationLanguage Models for Automated Market Commentary from Corporate Disclosures
Assis, G., Dutra, H., Vianna, D., Meira, W., da Silva, A. S., & Paes, A. (2025) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3768292.3770438 · Cited by 1 · see citing works
Power Echoes: Investigating Moderation Biases in Online Power-Asymmetric Conflicts
Li, Y., Zhang, P., Hou, P., Tu, K., Zhang, G., Qu, S., Chen, W., Chen, Y., Gu, N., & Lu, T. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791694 · Cited by 1 · see citing works
Keith-Norambuena, B. F., & Flores-Bustos, C. (2026) · Analytics · doi:10.3390/analytics5020018 · Cited by 1 · see citing works
From Prediction to Stewardship: Framing Educational Data Science in the Age of Generative AI
McNamara, D. S., & Huynh, L. (2026) · Information · doi:10.3390/info17060610 · Cited by 1 · see citing works
Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education
Thomas, D. R., Borchers, C., Vanacore, K., Koedinger, K. R., & Kizilcec, R. F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29773-0_9 · Cited by 1 · see citing works
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06709
“…these findings, Park et al. [2024] identify seven distinct bias types using a meta-evaluation framework based on hand-crafted test cases, and Chen et al. [2024] demonstrate that biases such as misinformation oversight, gender, authority, and style bias are common in both LLM-as-a-judge and…”
Pollertlam, N., & Kornsuwannawit, W. (2026) · arXiv · doi:10.48550/arxiv.2603.04814
“…fluctuation, the judge model may exhibit systematic biases toward responses generated by models in the same model family (Zheng et al., 2023; Ye et al., 2024), and may be more lenient on verbose near-misses than a human annotator would be (Zheng et al., 2023; Ye et al., 2024; Chen et al., 2024).”
MAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1351
“Given this limitation, recent work has explored the use of LLMs as scalable evaluators to score open-ended outputs or generate preference labels (Ding et al., 2023; Zheng et al., 2023; Chen et al., 2024; Liu et al., 2023).”
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Zhao, J., Shin, C., Huang, T.-H., GNVV, S. S. S. N., & Sala, F. (2026) · arXiv · doi:10.48550/arxiv.2603.00039
La evaluación de la escritura en español en la educación superior: una conversación con ChatGPT
Uribe, S. Á. (2025) · Lenguaje · doi:10.25100/lenguaje.v53i1s.14433
Yang, D., Zhang, X., Han, T., & Liu, Y. (2025) · Natural language processing. · doi:10.1017/nlp.2025.10005
When Context Grows, So Does the Challenge: Human Oversight in LLM Evaluation of Financial Tables
Sehanobish, A., Anderson, S., Michel, G., & Arov, M. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746058.3758457
How Real is Your Jailbreak? Fine-Grained Jailbreak Evaluation with Anchored Reference
Liu, S., Li, C., Pu, R., Zhang, L., Wang, C., Chen, Z., Zhang, Y., & Hei, Y. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-37274-1_1
MAARV: Multi-Agent Architecture for Runtime Verification
Deb, A., Jauhari, A., Goswami, M., Usha Nandhini, G., & Vinoth, N. A. S. (2025) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1109/aimlsystems67835.2025.11331130
PRECISE: Reducing the bias of LLM evaluations using prediction‐powered ranking estimation
Divekar, A., & Majumder, A. (2026) · AI Magazine · doi:10.1002/aaai.70066
Tsourma, M., Michail, D., Varlamis, I., Drosou, A., & Tzovaras, D. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391043
NCRAgent: A Neuro-Inspired Multiagent Framework for Transparent and Explainable Medical VQA
Liu, S., Hou, K., Anaissi, A., & Braytee, A. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3668173
Mizuno, T. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401804
Karp, M., Kubaszewska, A., Król, M., Król, R., Wydmański, W., Smywiński-Pohl, A., & Szymański, M. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09505-w · Cited by 2 · see citing works
Diagnosing LLM Hallucinations in Process Mining Tasks: a Taxonomy and a Benchmark
Berti, A., & Kourani, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-17618-9_41
Barghorn, J., Sotnikova, A., Friedli, S., & Bosselut, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791236 · Cited by 1 · see citing works
Fu, Y., Lyu, T., Chen, Y., & Lv, Z. (2026) · ENGINEERING Management · doi:10.1007/s42524-026-4237-0
Probing subjective judgment variance in LLM evaluators: A framework for robust IR evaluation
Kumar, R. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07244-8
Section-Weighted Hybrid Approach for Legal Case Retrieval
Arulanandam, R., & de Silva, N. (2026) · doi:10.1109/icnlp69856.2026.11527762
Hassan, A., Saeed, M. S., Senouci, B., & Benatallah, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-28110-4_21
Shi, W., Herremans, D., & Choo, K. T. W. (2026) · Designing Interactive Systems Conference · doi:10.1145/3802974.3809463
Automated bias blind spots: Examining stereotype associations in VLM-as-a-judge paradigms
Hong, R. S., Ovalle, A., Ung, M., Spiliopoulou, E., Sagun, L., Williams, A., & Ross, C. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806459
Tang, W., Gui, T., Ma, H., & Chen, M. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581585
Analyzing and Mitigating Bias in LLMs through Cross-Model Consensus
Xiao, K., Yang, L., Zhang, D.-Y., Tulajiang, P., & Lin, H.-F. (2026) · International Conference on Computer Supported Cooperative Work in Design · doi:10.1109/cscwd68734.2026.11582018
Mirabi, S., Ofoghi, B., Yearwood, J. L., & Mollá, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-25035-3_12
On the Generalization of Reward Models and LLM Judges under Out-of-Distributions in RLHF
Zhang, Y., Sun, F., Zhang, X., Zhou, Y. L., & Li, M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3381-6_24
Dong, Q., & Qi, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3734-0_13
Bieberstein, A., Schnitzer, B. L., Gampe, S., & Korn, O. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816221
Purk, J. (2026) · Logistics Research · doi:10.1108/lore-12-2025-0184
Han, T., Xu, B., Zhang, H., & Lu, Y. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-4529-1_14
Alharbi, N., Albassam, E., & Alkhuraiji, S. (2026) · IEEE Access · doi:10.1109/access.2026.3723027
Alqahtani, M. Q., & Abumelha, M. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01068-9
Cao, Y., Keung, J., Wang, Y., Sun, Y., & Mao, Z. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00062
VADER: Validation-Driven DSL Generation with Small Language Models
Baber, J., González-Sáez, G., Hili, N., & Schwab, D. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00068
Mehta, M., Sharma, R., Kalluru, V., & Kotwal, A. (2026) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3800935.3830864
Lee, G., Kim, Y., Youn, H., Roh, H., & Borrmann, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7450622
Kawalerowicz, M., & Pietranik, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36871-3_27
Collaço, B. G., Wood, N. G., Yu, Y., Malena, C. R., Srinivasagam, P., Boon, A. L., Fang, Z., Bhagra, A., Leibovich, B. C., & Forte, A. J. (2026) · Bioengineering · doi:10.3390/bioengineering13101115
Costa, R. F. M., Cordão, C. D. P. D. S., Brunet, J., Morais, F., & Maia, D. C. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_17
Alamillo, C. A. H., & Abbas, A. (2026) · IEEE Digital Education and MOOCS Conference (DEMOcon) · doi:10.1109/democon70840.2026.11707912
Autenrieth, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37979-5_5
Anghel, C., Anghel, A. A., Balau, A. S., Crăciun, M., Cocu, A., Moldovanu, S., & Sandu, C. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100327
Putting Large Language Models into Environmental Context
Jiang, J., Zhu, J., & Ren, Z. J. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.6c06045
Do LLM Judges Agree with Humans? Evaluating Financial Commentaries from Material Facts
Assis, G., Vianna, D., Real, L., Masid, M. R., Nepomuceno, J., Rottschaefer, E., da Silva, A. S., & Paes, A. (2026) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2026.26484
Textbook-Enriched Training for Language Models: Boosting Answer Quality in Specialized Contexts
Mota, L. B. B., Dantas, L., Claro, D. B., Paes, A., Freitas, C., Souza, M., Caseli, H. M., & Real, L. (2026) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2026.26566
AI for Grading from the Teaching Perspective (with an EssayGrader Case Comparison)
Shepherd, I. J. (2026) · Journal of Higher Education Theory and Practice · doi:10.33423/pant6n19
A Taylorizable Process for Textual Detector Development for Research-Practice Partnerships
Baker, R. S., Mills, C., Lan, A., Lehman, B. A., & Barany, A. (2026) · Journal of Learning Engineering · doi:10.68597/a.142838
- Authors
- Chen, G. H., Chen, S., Liu, Z., Jiang, F., & Wang, B.
- Affiliations
- Shenzhen Research Institute of Big Data (CN)
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 8301-8327
- OpenAlex
- W4404782878
- Cited by
- 76
- References
- 0
- Citations by year
- 2026: 48 · 2025: 25 · 2024: 3
- FWCI
- 21.94
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Natural Science Foundation of China (12326608)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM-as-a-judge, LLM evaluation, judgment bias, gender bias, authority bias, beauty bias, Bloom's taxonomy, evaluation robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
