Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

In the paper

“Of PE tools, only EvalLM offers a way to help users calculate the alignment of LLM evaluators with their expectations, but this feature is mentioned only in authors' Design section and is absent from their user study.”

What the sources say

• Partly supported

  Section 5.1.6 (p. 7) describes a validation feature that lets users compare the automatic evaluator’s predictions with their own ground-truth evaluations. The user-study measures in Section 7.1 (p. 11) do not include that feature. The paper does not establish that EvalLM is the only prompt-engineering tool offering it, and the feature appears in the Interface section, not only in a Design section. (LLM, conf=0.84)

  cites [27] [27] Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. arXiv preprint arXiv:2309.13633 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation.”

• Also stated in: § 1 INTRODUCTION · p. 1 · plain

  “The problem is that, just like the LLMs they evaluate, LLMs that perform evaluations cannot be trusted.”

What the sources say

• ?Could not verify

  The Introduction reports that LLM performance is sensitive to prompt formatting, and the Discussion recommends evaluating across plausible formats (page numbers unavailable). Neither passage addresses LLM-generated evaluators inheriting the problems of models they evaluate or requiring human validation. Searches for those claims found no supporting passage in our copy. (LLM, conf=0.75)

  cites [43] [43] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Many approaches rely on custom model training or finetuning [6], but all strategies heavily rely on humans to identify examples of desirable and undesirable outputs.”

What the sources say

• Partly supported

  The citation is attached to the first clause, about approaches relying on custom model training or finetuning. The Introduction (p. 1) describes training a deep reinforcement-learning system using a learned reward function, which supports custom model training as an example. It does not establish that many approaches to aligning LLMs use such training or finetuning; the paper concerns RL tasks rather than LLMs. (LLM, conf=0.84)

  cites [6] [6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017). - sourceExperimental — accuracy not yet measured

In the paper

“Liu et al. [34] demonstrated using annotated LLM outputs-judged on criteria like consistency and relevance-as 'few-shot examples' for calibrating LLM-based evaluators.”

What the sources say

• Partly supported

  Section 2.3 describes using human-labeled sample–score pairs as few-shot exemplars to infer scoring criteria for calibrating an LLM-based evaluator. Section 4.1 evaluates criteria including consistency and relevance. The copy does not establish that the annotated outputs used as examples were specifically generated by LLMs, rather than other text-generation systems. (LLM, conf=0.84)

  cites [34] [34] Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023. Calibrating LLM-based evaluator. arXiv preprint arXiv:2309.13308 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“One recent LLM-assisted approach, SPADE [45], makes headway on these issues, helping developers generate Python assertion functions for LLM outputs from prompt history.”

What the sources say

• Supported

  The Abstract says SPADE analyzes histories of prompt versions to create candidate assertion functions that identify bad LLM outputs. Component 1 specifies that it synthesizes Python functions as candidate assertions from prompt deltas, and Appendix C says it uses GPT-4 to generate them. (LLM, conf=0.94)

  cites [45] [45] Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, JD Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G Parameswaran, and Eugene Wu. 2024. SPADE: Synthesizing Assertions for Large Language Model Pipelines. arXiv preprint arXiv:2401.03038 (2024). - source

In the paper

“Interfaces that facilitate seamless transitions between these activities result in fewer errors and outputs that better match users' expectations [38, 46].”

What the sources say

• ?Not checked — no full text of the sourcecites [38] [38] Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Amy J. Ko, and James Landay. 2010. Gestalt: integrated support for implementation and analysis in machine learning. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST '10) . Association for Computing Machinery, New York, NY, USA, 37-46. https: //doi.org/10.1145/1866029.1866038 - source
• Partly supported

  Section 2.2 (p. 3) says ICE integrates labeling and feature editing in one interface and lets teachers inspect model errors to decide what to do next. Section 2.3 (p. 4) describes tools for reviewing errors and revising labels. These passages support the integrated workflow, but do not establish that seamless transitions result in fewer errors or outputs that better match users’ expectations. (LLM, conf=0.83)

  cites [46] [46] Patrice Simard, David Chickering, Aparna Lakshmiratan, Denis Charles, Léon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, and Jina Suh. 2014. Ice: enabling non-experts to build models interactively for large-scale lopsided problems. arXiv preprint arXiv:1409.4814 (2014). - sourceExperimental — accuracy not yet measured

In the paper

“Similarly, Arawjo et al. [1] found that even people familiar with LLMs (developers, academics in ML) struggled to scale up their evaluations, appearing to over-generalize from a limited number of outputs even after an automated evaluation pipeline was setup.”

What the sources say

• Supported

  Section 7.4 (page unavailable) reports that a participant with prior LLM research experience built an evaluation pipeline but sent only one prompt to multiple models. Appendix A.3 (page unavailable) gives a further example: after checking a small number of outputs with an evaluator, a participant acknowledged the results were not statistically significant yet drew a general conclusion about a model’s behavior. The Discussion also says that some users with prior AI or LLM API experience struggled to systematize their tests. (LLM, conf=0.88)

  cites [1] [1] Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. arXiv preprint arXiv:2309.09128 (2023). - source

In the paper

“For instance, AutoCalibrate is a method to calibrate LLM evaluators with human preferences that requires large expert-labelled datasets with settled (i.e., established upfront) criteria [34].”

What the sources say

• Partly supported

  Sections 2.1 and 2.3 describe AutoCalibrate as aligning LLM evaluators with human expert labels. But the Effect of Few-Shot Example Size section says 8–12 examples suffice for inferring effective criteria, rather than establishing a need for large labelled datasets. Section 2.3 and Criteria Revisiting say the method drafts and refines criteria from examples, rather than requiring criteria settled upfront. Page numbers are unavailable in our copy. (LLM, conf=0.89)

  cites [34] [34] Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023. Calibrating LLM-based evaluator. arXiv preprint arXiv:2309.13308 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“This suggests an optimization loop akin to ConstitutionMaker and DSPy [26, 39], but for assertion generation and validation , rather than prompt optimization.”

What the sources say

• Supported

  The Introduction (p. 2) describes DSPy’s compiler as simulating program versions, using a validation metric, and bootstrapping examples to improve prompts or fine-tune modules. The DSPy Compiler section (pp. 6–7) describes candidate generation and metric-based filtering, supporting the comparison to an optimization loop. Assertion generation and validation are the citing authors’ proposed application, not a feature attributed to DSPy. (LLM, conf=0.90)

  cites [26] [26] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023. Dspy: Compiling declarative language model calls 27. into self-improving pipelines. arXiv preprint arXiv:2310.03714 (2023). - source
• Supported

  Section 2.2 describes ConstitutionMaker as a process in which users critique outputs, their feedback becomes principles, and those principles are incorporated back into the prompt. The cited comparison is to that feedback-driven prompt-refinement process; assertion generation and validation are the citing authors’ proposed adaptation, not a feature attributed to ConstitutionMaker. (LLM, conf=0.90)

  cites [39] [39] Savvas Petridis, Ben Wedin, James Wexler, Aaron Donsbach, Mahima Pushkarna, Nitesh Goyal, Carrie J. Cai, and Michael Terry. 2023. ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles. arXiv:2310.15428 [cs.HC] - source

In the paper

“This suggests users want data visualization plots between the original prompt and its revision, akin to LLM Comparator and EvalLM [24, 27].”

What the sources say

• Partly supported

  The Visualization Design and Development section describes LLM Comparator as a side-by-side comparison tool with a visualization summary, supporting its use as an example of visual comparison. But Section 3.1 says it compares two models’ responses to a prompt, not an original prompt with its revision. The cited paper therefore supports the analogy, not that specific comparison. (LLM, conf=0.84)

  cites [24] [24] Minsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon. 2024. LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models. arXiv:2402.10524 [cs.HC] - sourceExperimental — accuracy not yet measured
• Supported

  The Interface section (p. 6) says EvalLM lets users compare prompts before and after specific edits. Its history visualization (Figure 6, p. 8) uses color-coded bars to show which prompt performed better for each sample, making EvalLM an example of visualizing comparisons between a prompt and its revision. (LLM, conf=0.86)

  cites [27] [27] Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. arXiv preprint arXiv:2309.13633 (2023). - source

In the paper

“In particular, the dependence of criteria on a specific prompt and LLM's outputs means that any change to the LLM pipeline can cause criteria drift: for example, LLM APIs can be replaced with new models, resulting in prompt drift [4].”

What the sources say

• Partly supported

  The Introduction (page unavailable) says that changes in an LLM’s response to a prompt can break a downstream pipeline. Section 3.6 and the accompanying discussion (pages unavailable) describe prompt-engineered LangChain modules as brittle when LLM behavior drifts. The paper does not establish that any pipeline change causes criteria drift, or describe replacing an API with a new model as causing prompt drift. (LLM, conf=0.84)

  cites [4] [4] Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is ChatGPT's behavior changing over time? arXiv preprint arXiv:2307.09009 (2023). - sourceExperimental — accuracy not yet measured

In the paper

“Second, many users write their LLM pipelines as chains of multiple LLM calls [1, 11, 14, 53].”

What the sources say

• Partly supported

  Section 8.1 reports that five interviewees prototyped LLM data-processing pipelines and sent many parametrized prompts. The paper also describes chaining Prompt Nodes (Section 5), but it does not establish that many users wrote pipelines as chains of multiple sequential LLM calls; sending multiple prompts is not necessarily chaining them. (LLM, conf=0.82)

  cites [1] [1] Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. arXiv preprint arXiv:2309.09128 (2023). - sourceExperimental — accuracy not yet measured
• Not checked yetcites [11] [11] Harrison Chase et al. 2023. LangChain. https://pypi.org/project/langchain/. -
• Not checked yetcites [14] [14] FlowiseAI, Inc. 2023. FlowiseAI Build LLMs Apps Easily. flowiseai.com. -
• Partly supported

  The Introduction (p. 2) describes chains of multiple LLM steps and reports that four designers and developers built chains in the study. This supports the practice described, but the paper does not establish that many users write their pipelines this way. (LLM, conf=0.86)

  cites [53] [53] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA '22) . Association for Computing Machinery, New York, NY, USA, Article 359, 10 pages. https://doi.org/10.1145/3491101.3519729 - sourceExperimental — accuracy not yet measured

In the paper

“In fact, a number of LLMOps tools exist for users to implement such code-based guardrails [19, 22, 42].”

What the sources say

• Not checked yetcites [19] [19] Guardrails 2023. Guardrails AI. https://github.com/guardrails-ai/guardrails. -
• Not checked yetcites [22] [22] Instructor 2023. Instructor, Generating Structure from LLMs. https://jxnl.github.io/instructor/. -
• Supported

  The Introduction identifies NeMo Guardrails as an open-source toolkit that lets users define programmable rails at runtime. Section 3.3 says its execution rails can be custom actions defined in Python. This supports the citation as an example of a tool for implementing code-based guardrails; the other citations need not be established by this paper. (LLM, conf=0.91)

  cites [42] [42] Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501 (2023). - source

In the paper

“Prior work also underscores the importance of soliciting a representative graded sample of LLM outputs [2, 45].”

What the sources say

• Supported

  The Data section assumes a small human-annotated sample and specifies that both the labeled and unlabeled datasets are i.i.d., supporting the need for a representative graded sample. Section 3.3 applies the approach to human preferences over LLM responses, and the Introduction explains that the human data are used to correct bias in synthetic evaluations. No page numbers are available in our copy. (LLM, conf=0.87)

  cites [2] [2] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. 2024. AutoEval Done Right: Using Synthetic Data for Model Evaluation. arXiv preprint arXiv:2403.07008 (2024). - source
• Supported

  In “Component 2: Filtering Candidate Assertions with Limited Data” (page unavailable), the paper warns that developer-labeled LLM outputs may miss failure modes and suggests sampling and labeling new input-output pairs to expand coverage. Its discussion of which labeled examples would best select assertions in “6 Conclusion and Future Work” reinforces the importance of the sample’s composition. (LLM, conf=0.87)

  cites [45] [45] Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, JD Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G Parameswaran, and Eugene Wu. 2024. SPADE: Synthesizing Assertions for Large Language Model Pipelines. arXiv preprint arXiv:2401.03038 (2024). - source

In the paper

“Some frameworks already experiment with using feedback from assertions or user grades to refine prompts [35, 39, 48].”

What the sources say

• Partly supported

  Section 2, “Iterative Refinement with SELF-REFINE” (page unavailable), supports the general idea of a framework that uses feedback for refinement. But it says the model generates feedback on its own outputs and refines those outputs—not prompts—while the “Refine Feedback” section says the process does not involve human assistance. The paper does not support feedback from assertions or user grades as described in the claim. (LLM, conf=0.88)

  cites [35] [35] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024). - sourceExperimental — accuracy not yet measured
• Partly supported

  Section 2.2 says ConstitutionMaker converts users’ feedback on prompt outputs into principles incorporated back into the prompt. Section 5.2 describes kudos, critiques, and rewrites as feedback, but does not describe feedback from assertions or user grades. The prompt-refinement point is supported; the specified feedback forms are not. (LLM, conf=0.85)

  cites [39] [39] Savvas Petridis, Ben Wedin, James Wexler, Aaron Donsbach, Mahima Pushkarna, Nitesh Goyal, Carrie J. Cai, and Michael Terry. 2023. ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles. arXiv:2310.15428 [cs.HC] - sourceExperimental — accuracy not yet measured
• Supported

  Section 4.2 describes using LM Assertions to filter examples during prompt optimization. Section 4.3 says the optimizer incorporates feedback from assertion failures as demonstrations, supporting the assertion-feedback part of this multi-citation claim. (LLM, conf=0.92)

  cites [48] [48] Arnav Singhvi, Manish Shetty, Shangyin Tan, Christopher Potts, Koushik Sen, Matei Zaharia, and Omar Khattab. 2023. DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines. arXiv preprint arXiv:2312.13382 (2023). - source

In the paper

“A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative process of alignment.”

In the paper

“we identify a phenomenon we dub criteria drift : users need criteria to grade outputs, but grading outputs helps users define criteria.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “We observed a 'catch-22' situation: to grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.”

In the paper

“Common issues included the presence of personal information in the medical pipeline outputs and bad reviews or lengthy content in the product pipeline outputs.”

In the paper

“Only one participant did not understand what the overall set's coverage and false failure rate meant (P9), but did once the researcher explained it.”

In the paper

“3 out of 9 participants again opened the EvalGen wizard to iterate on their criteria and assertions (back to step 2), or stopped because they were satisfied or tired.”

In the paper

“6 participants clicked the autogenerate button; 1 participant wrote a criterion themselves and then clicked the auto-generate button. The remaining 2 participants wanted to grade first (P4, P9).”

In the paper

“All participants ( 𝑛 = 9) used the provided task (a prompt template for NER of a dataset of 100 tweets, described in 6). Three (3) participants changed the prompt: of these three, one (1) person changed the task from NER to sentiment analysis.”

In the paper

“Our results in Table 1 show that EvalGen, by incorporating human judgment during criteria selection, achieved equal or better alignment than SPADE with fewer assertions for both pipelines.”

• Also stated in: § 5.2 Impact of Human Input in the Criteria Generation Step · p. 7 · plain

  “With user input at the criteria stage, EvalGen achieves the same or greater alignment with fewer functions.”

In the paper

“More than half of the participants expressed interest in iterating on their assertions if they had more time.”

In the paper

“Common issues included the presence of personal information in the medical pipeline outputs and bad reviews or lengthy content in the product pipeline outputs.”

In the paper

“They usually left evaluation type (code-based or LLM-based) unchanged from what EvalGen suggested, even if EvalGen suggested a type that did not make sense (e.g., checking word count with an LLM API call, rather than code), which happened rarely.”

In the paper

“After receiving criteria suggestions from EvalGen, participants removed some suggestions and added 1-2 criteria of their own.”

In the paper

“Participants felt that EvalGen was a great starting point for assertions, and wanted to-and could-exercise control over EvalGen's assistance.”

In the paper

“Participants who graded LLM outputs first graded between 5 and 10 outputs, with 2 to 4 'thumbsdown' grades.”

In the paper

“In contrast, EvalGen returned a more pragmatic assertion for this criterion-an LLM-based validator to ensure the product descriptions remained entirely positive.”

In the paper

“Participants graded outputs while EvalGen generated and evaluated different candidate assertions. Some participants graded continuously for up to 10 minutes; others stopped after 10 grades.”

In the paper

“Participants really liked viewing the table of assertion results on all LLM outputs, with all participants expressing interest upon first viewing it (Figure 3)-but they verified the results differently.”

In the paper

“Interestingly, only one participant assessed alignment in this table via coverage-by inspecting each row in the table, one at a time-and everyone else assessed alignment via false failure rate: for each column (assertion), they typically looked for rows (LLM outputs) that the assertion returned False.”

In the paper

“However, P7 then acknowledged that their ability to edit depends on code length and complexity:”

In the paper

“Participants who graded before selecting criteria (P4 and P9) expressed that they found the grading process very useful. Both participants were happy that EvalGen prompted them to give feedback on why an LLM output was bad.”

In the paper

“Yet participants reasoned about code-based assertions and LLM-based assertions differently: while they liked having assertions of both types, they wanted to construct and iterate on them differently in order to feel like the resulting set of assertions aligned with their preferences.”

In the paper

“Some participants wanted this table to automatically recompute, with an accompanying visualization, whenever there were changes to the LLM pipeline; for example, P6 said:”

In the paper

“P3 excitedly realized this when looking at the table of results ('I can take these good and bad examples and use them as few-shot exemplars in the prompt for the LLM evaluators...').”

In the paper

“Some participants were skeptical of how LLM-based assertions might transfer to monitoring LLM outputs in a production pipeline (P3, P6, P8).”

In the paper

“Several participants realized they needed to go back and change their grades in EvalGen and were happy they could do this.”

In the paper

“P8 desired to grade for only one criteria, reasoning that it might improve efficiency ('I generally want to be in the loop for these tests...but I want to put myself in the loop in a way that is efficient.').”

In the paper

“P2 said that this table 'earns trust'; P5 said that it was 'cool to see all the results on examples [they] didn't grade.'”

In the paper

“Some participants said that they had not graded enough responses to get their ideal alignment numbers, so they wanted to go back and grade more (P1, P7).”

In the paper

“Another place where participants wanted to exercise control but were limited by EvalGen's interface was in the selected assertions: P2 and P5 wanted to change a selected assertion to another candidate assertion.”

In the paper

“P8 said, 'I cannot begin to think about how LLMs as validators [in production] can work, I'm very skeptical.'”

In the paper

“Participants also liked seeing the coverage and false failure rate of the set, and how each assertion contributed to these overall statistics.”

In the paper

“P9 asked, 'How do I maintain my evals over time; do I have to rerun this entire process?'”

In the paper

“P5 expressed the same uncertainty. 'I think it's hard to know until you see it,' P7 said.”

In the paper

“While P9 selected the grading option initially because it seemed like the 'option that required the least thinking,' they later realized that it was the correct thing to do:”

In the paper

“P7 initially expressed dissatisfaction that 7 criteria were generated, as they had expected less. But after deleting some criteria, they realized that the suggestions covered all the criteria they wanted (i.e., they did not have to add criteria themselves), and stated, 'this auto-generation is sweet'.”

In the paper

“P9 appreciated the ability to provide feedback on unsatisfactory outputs before finalizing the initial criteria set, as this process helped them articulate their criteria.”

In the paper

“P7 wanted to iterate on code-based assertions simply by providing feedback to make the assertion more or less 'fancy'-for example, in checking for a pattern in the LLM output, they might want the regex to be more or less complicated depending on the sample of LLM outputs they've seen.”

In the paper

“Both P4 and P6 mentioned that they liked the ability to correct EvalGen's suggested type.”

In the paper

“Other participants who did not grade before selecting criteria regretted this decision: for example, P2 said, 'you should enforce that we all look at at least 20 examples first.'”

In the paper

“P6 and P8 expressed that they wanted to use LLM-based assertions for their own LLM pipelines because 'most of their criteria was fuzzy.'”

In the paper

“For example, while all participants had a criterion to enforce that there were no entities with hashtags in the output, some participants interpreted this as any hashtags representing entities should not be extracted as entities: e.g., if the output included the hashtag #Nike, P5 did not want Nike to be extracted at all. On the other hand, P9 wanted Nike to be extracted as an entity, but they did”

In the paper

“P5 initially did not like using a GUI to come up with assertions, and then said that they 'prefer looking at this table to [their] current workflow in Jupyter notebooks.'”

In the paper

“While participants had lots of ideas for improvement, participants overall felt that EvalGen provided a good start to code-based assertions, and it was easy for them to edit the code if they wanted: two participants even asked for the ability to export code-based assertions as unit tests or in a Python file (P4, P6).”

In the paper

“We also observed a participant (P8) preferring LLM-based to code-based assertions: LLMs are relatively forgiving when encountering outputs with an unexpected quality or format, whereas code assertions check exact constraints.”

In the paper

“While participants found EvalGen's suggested code-based assertions to be more obviously misaligned than the LLM-based assertions (Section 7.3.3), they also acknowledged that LLM-based assertions are harder for them to trust-since they can edit the code-based assertions more easily than the LLM-based assertions (P2, P5, P6, P8, P9).”

In the paper

“For example, P3 expressed that they would trust the assertions more if the EvalGen process allowed them to set different false failure rates per criteria (since LLMs might be bad at evaluating some criteria), instead of one global false failure rate constraint for the entire assertion set:”

In the paper

“Both P2 and P5 wanted to change code-based assertions: for example, P5 said, 'this is a weird implementation of entity count' and preferred another assertion with a small modification.”

In the paper

“P3 found that some EvalGen-suggested criteria did not make sense given that assertions only operate on LLMpipeline input-output pairs, like response latency and response stability over time.”

In the paper

“While participants generally liked this approach for LLM-based assertions, they did not like this for code-based assertions.”

In the paper

“For another criterion, P5 felt that EvalGen could not find a good assertion that aligned with their grades, but also said that they were 'lost at what would be a good implementation.'”

In the paper

“However, for some criteria, EvalGen Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences generated no good assertions, and participants deleted the criteria without complaints.”

In the paper

“In the EvalGen interface, they could not go back and add new criteria; they had to wait for all candidate assertions to finish executing, move past the report card screen, and start a new EvalGen process.”

In the paper

“All participants liked EvalGen's ability to assist in generating both code-based and LLM-based assertions, expressing the need for both types of assertions.”

In the paper

“The participant who did not agree said that they would find it useful if, on the grading screen, EvalGen showed what it was doing with the grades.”

In the paper

“P3 did not feel that this was a failure of EvalGen, but rather expressed that LLMs are not perfect and stressed the importance of having control over selected criteria.”

In the paper

“However, in practice, we found that developers rapidly iterate over criteria, and furthermore that cognitively engaging with LLM outputs helps them to refine their criteria.”

In the paper

“P7 noticed that in some outputs included hashtags from the original tweet inputs (e.g., #justdoit), while in other cases, the outputs did not use the hashtag symbol but still mentioned the entities referred to by the hashtags (e.g., 'Nike' instead of #Nike). This inconsistency led P7 to rethink the criteria definition for including hashtags; specifically, whether it was acceptable for the LLM to”

In the paper

“While participants liked that EvalGen tried multiple candidate assertions for each criterion, they wanted EvalGen to use their grades, as well as natural language feedback on thumbs-down grades, in prompting LLMs for candidate assertions (P3, P4, P5, P7).”

In the paper

“Overall, participants generally liked the process of grading LLM responses and feeling like the grades were useful, but they wanted to prioritize grading criteria they felt needed their alignment -especially for LLM-based assertions (P3, P5, P7, P8).”

In the paper

“However, in this screen, some participants expressed interest in seeing per-criterion alignment and asked to provide grades for each criterion (P5, P6, P7, P8).”

In the paper

“In fact, specifically for an output length criterion, P3 mentioned that they would look at a histogram of word lengths for a batch of outputs and see if there were some buckets of anomalous word lengths that exhibited other failure modes (e.g., a 'rambling' response from the LLM).”

In the paper

“Our study confirmed the dual necessity for both code-based and LLM-based assertions among participants, but further showed that participants felt each type required distinct treatment, both in implementation selection and in how they perceive alignment.”

In the paper

“Others wanted a more passive deployment-one participant requested for some form of report sent to their email inbox daily, containing results of the assertion set executed on a sample of LLM outputs from that day.”

In the paper

“Some participants mentioned that they wanted their collaborators, such as product managers, to grade outputs in EvalGen.”

In the paper

“Participants expressed the desire to deploy their assertions in production.”

In the paper

“Notably, while the alternating policy didn't consistently outperform, our results suggest that any non-random policy implemented in EvalGen may achieve satisfactory outcomes.”

In the paper

“Some wanted to deploy assertions in the critical path of the LLM pipeline, to catch bad outputs before they are shown to end-users.”

In the paper

“Thus, our findings suggest that users need evaluation assistants to support rapid iteration over criteria and implementations simultaneously .”

In the paper

“Two of the paper authors manually graded all LLM outputs to establish ground-truth labels.”

In the paper

“This illustrates the challenge of balancing coverage with false failures, underscoring the need for evaluator systems to effectively navigate these trade-offs.”

In the paper

“We employed open and axial coding [13] to identify common themes across the transcripts and notes for each participant.”

What the sources say

• ?Not checked — no full text of the sourcecites [13] [13] Uwe Flick. 2013. The SAGE handbook of qualitative data analysis . Sage. - source

In the paper

“Our study was approved by our institutional review board (IRB), and participants generously volunteered their time.”

In the paper

“The human cannot completely validate an LLM-based evaluator: the point of LLM evaluators is to reduce the effort required by the developer, who would otherwise have to grade outputs manually.”

In the paper

“Moreover, for code-based assertions, LLMs occasionally synthesize erroneous functions (e.g., hallucinating a function in a Python library), requiring several candidate assertions.”

In the paper

“Kocielnik et al. [29] found that AI systems that showcase subsets of errors, like false positives or false negatives, that have the same accuracy, can lead to vastly different perceptions of accuracy.”

What the sources say

• ?Not checked — no full text of the sourcecites [29] [29] Rafal Kocielnik, Saleema Amershi, and Paul N. Bennett. 2019. Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3290605.3300641 - source

In the paper

“The only way to fully align an LLM evaluator would be to ask the user to label all outputs; obviously, this defeats the purpose.”

In the paper

“Asking the developer to grade some outputs, using some time they would have spent waiting anyway, is the key idea behind our design.”

In the paper

“Generating multiple candidate assertions improves the probability that there is at least one implementation that aligns with developer expectations.”

In the paper

“It also reveals that auditing LLM outputs is far from easy, with humans prone to the dual biases of over-generalization and overreliance.”

In the paper

“It thus remains unclear how to support developers in their prototyping of evaluations, with the problem becoming even more pressing as the popularity of prompt optimization increases.”

In the paper

“Looking more closely at their interactions, we observed a catch-22 situation: participants needed to externalize criteria in order to grade outputs, but they also needed to grade outputs-providing feedback on why bad outputs were bad-in order to externalize criteria.”

In the paper

“This suggests criteria refinement and grading should happen in tandem in interactive settings, and poses challenges to alignment methods that presume settled, expert labels.”

In the paper

“A criterion like word count is hard for humans to assess but easy for a good Python function to evaluate.”

In the paper

“This observation may be specific to users who are experts in coding (as all of our participants are).”

In the paper

“Moreover, participants did not have enough time to iterate many times in EvalGen, and our setup did not cover the deployment phase of LLM workflows.”

In the paper

“Recognizing the dependency of criteria on LLM outputs highlights new directions for designing future evaluation assistants.”

In the paper

“Initially, when users start grading outputs in EvalGen, they might effectively be grading random outputs for the first one or two outputs, as the 𝜎 scores update and stabilize.”

In the paper

“A score of 0 means no assertions have failed the output, indicating a higher likelihood of quality, while lower scores, resulting from failures by non-selective assertions, point to uncertainty or potential issues with the output.”

In the paper

“In practice, assertions have different operational requirements.”

In the paper

“A score of 0 means no assertions have failed the output, indicating a higher likelihood of quality, while lower scores, resulting from failures by non-selective assertions, point to uncertainty or potential issues with the output.”

In the paper

“third, 'NER for tweets' is a problem that many people have studied [15, 33, 49].”

What the sources say

• Not checked yetcites [15] [15] Michel Naim Gerguis, Cherif Salama, and M Watheq El-Kharashi. 2016. ASU: An Experimental Study on Applying Deep Learning in Twitter Named Entity Recognition.. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT) . 188-196. -
• ?Not checked — no full text of the sourcecites [33] [33] Xiaohua Liu, Furu Wei, Shaodian Zhang, and Ming Zhou. 2013. Named entity recognition for tweets. ACM Transactions on Intelligent Systems and Technology (TIST) 4, 1 (2013), 1-15. - source
• ?Not checked — no full text of the sourcecites [49] [49] Chanchal Suman, Saichethan Miriyala Reddy, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Why pay more? A simple and efficient named entity recognition system for tweets. Expert Systems with Applications 167 (2021), 114101. - source

In the paper

“Second, our qualitative study focused on a small sample of developers, all of whom have significant experience deploying LLMs.”

In the paper

“Moreover, participants did not have enough time to iterate many times in EvalGen, and our setup did not cover the deployment phase of LLM workflows.”

In the paper

“This inconsistency could lead to user frustration, particularly if the effort spent in grading outputs results in assertion sets with unpredictable relevance to their specified criteria.”

In the paper

“First, our off-line evaluation only focused on two pipelines.”

In the paper

“Our criteria drift finding echoes prior work in educational settings, where instructors often update their grading rubrics to reflect common errors as they grade more assignments [47].”

What the sources say

• ?Not checked — no full text of the sourcecites [47] [47] Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017. Gradescope: a fast, flexible, and fair system for scalable assessment of handwritten work. In Proceedings of the fourth (2017) acm conference on learning@ scale . 81-88. - source