State of What Art? A Call for Multi-Prompt LLM Evaluation

Mizrahi, Moran, Kaplan, Guy, Malkin, Dan, Dror, Rotem, Shahaf, Dafna, Stanovsky, Gabriel·2024·Transactions of the Association for Computational Linguistics
DOI 10.1162/tacl_a_00681

Cited by (154)

Items 101–154 of 154

  1. Zh-Paral: Benchmark Dataset for Comprehension of Chinese Paralinguistic Speech
    Dekun Chen, Zhizheng Wu · 2024 · International Symposium on Chinese Spoken Language Processing (ISCSLP)
  2. Bias Mitigation and Shifting Ideologies of LLM by Prompting
    Yuji Ueda, Hirohiko Suwa · 2025 · Communications in computer and information science
  3. On the Attitudes of GameDev Industry Artists towards GenAI. Preliminary Results
    Łukasz Sikorski, Jacek Matulewski, Małgorzata Czerwonka · 2025 · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR)
  4. LLM-CD: A Cognitive Diagnosis Framework for Large Language Models(LLMs) ——Experimental Research Based on Green and Low-Carbon Knowledge
    Zhiping Huang, Ruyin Long, Hong Chen, Jingwen Na, Qian Li, Shiyan Jiang, Qingqing Sun · 2025 · Research Square
  5. Do LLMs fail in bridging generation?
    Natalia Skachkova, Simon Ostermann, Josef van Genabith, Bernd Kiefer · 2025 · LDV-Forum/Journal for language technology and computational linguistics
  6. Preaching to the ChoIR: Lessons IR Should Share with AI
    Gianluca Demartini, Claudia Hauff, Matthew Lease, Stefano Mizzaro, Kevin Roitero, Mark Sanderson, Falk Scholer, Chirag Suresh Shah, Damiano Spina, Paul Thomas et al. · 2025 · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR)
  7. A Conversational Multi‑Agent Framework for Prompt Evaluation across Large Language Models
    Abhishek Palavancha · 2025 · International Journal of Computer Applications
  8. Vertrauenswürdige Künstliche Intelligenz
    Ute Schmid · 2025 · Springer eBooks
  9. Personality Matters: User Traits Predict LLM Preferences in Multi-Turn Collaborative Tasks
    Sarfaroz Yunusov, Kaige Chen, Kazi Nishat Anwar, Ali Emami · 2025 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
  10. Adaptive Dynamic Scheduling Strategy of Multiple Large Language Models for Scientific Literature Classification Tasks
    Xiangyu Li, Yujie Deng, Qiyu Liu, Yingfan Gao, Gexiang Zhang, Ming Qiao Zhu · 2025 · International Symposium on Artificial Intelligence and Intelligent Manufacturing (AIIM)
  11. A Unifying Scheme for Extractive Content Selection Tasks
    Shmuel Amar, Ori Shapira, Aviv Slobodkin, Ido Dagan · 2025 · Transactions of the Association for Computational Linguistics
  12. Evaluation Reporting in AI Model Releases should be Standardized
    Ruchira Dhar, Danae Sánchez Villegas, Antonia Karamolegkou, Alice Schiavone, Yifei Yuan, Xinyi Chen, Jiaang Li, Stella Frank, Laura De Grazia, Monorama Swain et al. · 2025 · SSRN Electronic Journal
  13. Automated Judging of LLM-based Smart Contract Security Auditors
    Viraaji Mothukuri, Reza M. Parizi · 2025 · IEEE International Conference on Blockchain and Cryptocurrency (ICBC)
  14. ConVart: A Framework for Evaluating Confidence and Variability in Large Language Models for Domain-Specific Reliability
    Diya Saha, Manjira Sinha, Tirthankar Dasgupta · 2025 · Annual Meeting of the Forum for Information Retrieval Evaluation (FIRE)
  15. Knowledge Automation and Requirements Analysis in E-Commerce Fraud Detection
    Davis C. Loose, Megan E. Gunn, Igor Linkov, Matthew C. Gunn, Negin Moghadasi, Zachary A. Collier, James H. Lambert · 2025 · IEEE International Symposium on Systems Engineering
  16. Understanding the Interplay between LLMs' Utilisation of Parametric and Contextual Knowledge: A Keynote at ECIR 2025
    Isabelle Augenstein · 2025 · ACM SIGIR Forum
  17. The Illusion of Trust in AI: Behavioural Differences Between Humans and Large Language Models
    Yuzhan Hang, Zhenhua Ling, Quan Liu, Xiaosong He, Wei Wu · 2026 · Human Behavior and Emerging Technologies
  18. Aligned Probing: Relating Toxic Behavior and Model Internals
    Andreas Waldis, Vagrant Gautam, Anne Lauscher, Dietrich Klakow, Iryna Gurevych · 2026 · Transactions of the Association for Computational Linguistics
  19. Discovering the Significance of LLM’s in Medical Science for Disease Detection
    Bhargav R. Vyas, Mittal N. Desai, Heta Patel, Anjali Mahavar, Riya Panchal · 2026 · Communications in computer and information science
  20. Accelerating real-world data collection using large language models in rare neoplasms: a bone sarcoma example
    Paweł Teterycz, S. Rynkun, Bartłomiej Szostakowski, Michał Wągrodzki, Piotr Rutkowski, M. Rosinska · 2026 · ESMO Real World Data and Digital Oncology
  21. What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scale
    Rameen Mahmood, Sai Teja Peddinti, Tousif Ahmed, Danny Yuxing Huang · 2026 · Proceedings of the ACM on Networking
  22. Cognitive Biases in Large Language Models: A Systematic Quantitative Assessment and Debiasing Analysis
    Antonio Pagliaro · 2026 · Electronics
  23. MEDYANIN OTOMATİKLEŞEN FORMU: YENİ MEDYA İLKELERİNİN BÜYÜK DİL MODELLERİ İLE YENİDEN DEĞERLENDİRİLMESİ
    Duygu Ergün · 2026 · İNİF E - Dergi
  24. PromptCOVER: Prompt Copyright Protection with Value Differentiation by Homomorphic Encryption and Watermarking
    Ruijie Sun, Peng Li, Xin Zheng, Y Zheng · 2026 · SSRN Electronic Journal
  25. Prompt Perturbation for reliable LLM evaluation over comparison graphs
    Dong Huang, Jianbo Sun, Pengkun Yang · 2026 · Statistical Learning and Data Science
  26. Latent Confidence Alignment for LLM Self-Assessment
    T X Chen, Tingting Yu, Pei-Cing Huang, Chan Hsu, Ming-Yen Lin, Yihuang Kang · 2026 · IEEE International Conference on Information Reuse and Integration (IRI)
  27. Auditing LLM Responses in a Complex Policy Landscape: Abortion Law in the United States
    Ro Encarnación, Christen Jones, Danaë Metaxa · 2026 · ACM Conference on Fairness, Accountability, and Transparency (FAccT)
  28. Too long; didn’t solve
    Lucía M. Cabrera, Jocelyn D'Arcy, Isaac Saxton-Knight · 2026 · Workshop on Evaluating Evaluations (EvalEval)
  29. Evaluating perturbation robustness of generative systems that use COBOL code inputs
    Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici · 2026
  30. Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilitie
    Florian Kankowski, Torgrim Solstad, Sina Zarrieß, Oliver Bott · 2026 · Dialogue & Discourse
  31. Preliminary Study of an Evaluation Benchmark for Vision–Language Models in Fashion E-Commerce
    Ryotaro Shimizu, Sai HtaungKham, Shion Sakurai, Yuki Shimizu · 2026 · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)
  32. The LLM as variety transducer: Ashby's intelligence-amplifier architecture and Beer's viable system model as a diagnostic and predictive framework
    Richard Sean O'Rourke · 2026 · Kybernetes
  33. Diagnosing the Measurement Problem in AI Evaluation: Four Structural Failure Modes across Six Benchmarks
    Ngọc Bảo Hân Nguyễn · 2026 · SSRN Electronic Journal
  34. Multi-Agent Evaluation Framework for Government Affairs Large Language Models: A Case Study on Government Service Hotline Scenarios
    Zengding Bai, Shuhui Ai, Qihang Gong, Dong Luo, Tianming Shao · 2026 · SSRN Electronic Journal
  35. SkyAgent: A lightweight LLM-driven reinforcement learning framework for adaptive cooperative path planning of two UAVs
    Yuting Cao, Zheng Zhao, Jiekai Wu, Chia Hau Lee, Qi Wei Oung, Hoi Leong Lee · 2026 · Journal of King Saud University - Computer and Information Sciences
  36. Autodesign? Large Language Model Evaluations for UX Assistants
    Weynner Kenneth Bezerra Santos, Felipe de Castro Souza, Bárbara Coutinho da Silva, Paula Peres Gimenez, Ana Karoline Valadão Mendes, Giovanna de Souza Leal Paixão, Walter Franklin Marques Correia, Alexandre Gomes de Siqueira · 2026 · DAT Journal
  37. There Are No Neutral Instruments: How Measurement Choices Shape What We Believe About Language Models
    Pragna Mallappa · 2026 · SSRN Electronic Journal
  38. Response Stability of Large Language Models Under Meaning-Preserving Prompt Variation
    Idrees, Yongzhi Liu · 2026 · International Journal of Research and Innovation in Social Science
  39. From Hand-crafted to LLM-based Variation Operators in Metaheuristics: A Tutorial
    Camilo Chacón Sartori, Guillem Rodríguez Corominas, Christian Blum · 2026 · SSRN Electronic Journal
  40. Measuring GEO Visibility: Prompt Corpora Define the Answer Market A Critical Survey of Validity, Partial Identification, and Source Attribution
    Olivier Martinez · 2026 · SSRN Electronic Journal
  41. Evaluating psychiatric conference posters: Benchmarking a custom generative pre-trained transformer against human inter-rater variability using a structured assessment framework
    Anusa Arunachalam Mohandoss, Rooban Thavarajah, Raj Kiran Donthu, Swapnajeet Sahoo, Sai Krishna Tikka, Shahul Ameen · 2026 · Indian Journal of Psychiatry
  42. Evaluating large language models for lay summaries of radiology reports using tailored prompting strategies and mixed-method assessment
    Nanziba Tasneem, Christian B. van der Pol, Ambreen Zahoor, Nitin Juggath, Kyle McGowan, Cynthia Lokker, Ashirbani Saha · 2026 · PLOS Digital Health
  43. Towards reliable LLM evaluators: Discovering and internalizing explicit scoring logic
    Jie Meng, Jin Mao, Mingchang Ma, Gang Li · 2026 · Information Processing & Management
  44. Prompt Sensitivity or Evaluation Artifact? A Task-Aware Analysis for Large Language Models
    Sayumi Muthukumarana, Buddhi Wijenayake, Roshan Godaliyadda, Parakrama Ekanayake · 2026 · Moratuwa Engineering Research Conference (MERCon)
  45. Testing Frontier LLMs on Indian Statutory Interpretation: The Indian Construction Canon Benchmarking (ICCB) -Pilot Benchmark
    Yashu Bansal, Gaurav Dahiya, Aditi Tiwari, Akshobhya N Saralaya, Dana Susan Kurian, Devyani Singh, Nidhi Singh K V · 2026 · Research Square
  46. LLM-Driven Context-Aware Trash Recognition and Disposal with a Mobile Manipulator
    Janak Panthi, Caleb Horan, Mitch Pryor · 2026 · IEEE RO-MAN
  47. Humanistic-Writing Bias in AI-Powered Grading: An Analysis of Linguistic and Rhetorical Features Related to High- and Low-Scoring Student Papers
    Michael Laudenbach, Ben Markey · 2026 · IEEE International Professional Communication Conference (ProComm)
  48. Translating Product Requirement Documents into structured Use Case Specifications with large language models
    Zesong Xu, 车明亮, Zezheng Gong, Fuke Shen, Tongquan Wei · 2026 · Engineering Applications of Artificial Intelligence
  49. The first token decides: token-identity gating of LoRA task-vector interference
    Ruofan Yang, Lixin Liang, Bingding Huang, Xin Wang · 2026 · SSRN Electronic Journal
  50. Effective Prompt Engineering and AI-Supported Analysis in Educational Research
    Bülent Döş · 2026
  51. Artificial Intelligence Tools in Qualitative Research II
    Demirkol, Mehmet, Candan Çiftçi, Feyza, Başaran, Mehmet, Kalender, Barış, Deniz, İslim Derya, Arıcı, Mehmet, Polat, Önder, Döş, Bülent, Yokuş, Erhan, Kalınkara, Yusuf et al. · 2026
  52. Auditing and Safeguarding Tool-Using Language Agents: Evaluation Validity, Error Propagation, and a Runtime Interceptor (AgentProp-Bench)
    Bhaskar Gurram · 2026 · SSRN Electronic Journal
  53. Protocol-dependent conformity in open-weight large language models under fixed wrong-peer signals
    Ling Xu, Xucheng Xue, Ye Zhu, Junwen Zhang, Dechang Xu · 2026 · AI Agent
  54. The Evaluation Scaling Law: Toward More Reliable LLM Assessment via Multi-Prompt Diversity
    Hongzhi Luan, Zhaoxin Huan, Linfeng Shi, Longfei Zheng, Xiaolu Zhang, Jun Zhou · 2026 · Conference on Empirical Methods in Natural Language Processing (EMNLP)