Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Of PE tools, only EvalLM offers a way to help users calculate the alignment of LLM evaluators with their expectations, but this feature is mentioned only in authors' Design section and is absent from their user study.

“Of PE tools, only EvalLM offers a way to help users calculate the alignment of LLM evaluators with their expectations, but this feature is mentioned only in authors' Design section and is absent from their user study.”

• cites [27] [27] Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. arXiv preprint arXiv:2309.13633 (2023). - sourcePartly supported

  Section 5.1.6 (p. 7) describes a validation feature that lets users compare the automatic evaluator’s predictions with their own ground-truth evaluations. The user-study measures in Section 7.1 (p. 11) do not include that feature. The paper does not establish that EvalLM is the only prompt-engineering tool offering it, and the feature appears in the Interface section, not only in a Design section. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

LLM-generated evaluators inherit all the problems of the LLMs they evaluate, requiring further human validation.

“Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation.”

• Also stated in: § 1 INTRODUCTION · p. 1 · plain

  “The problem is that, just like the LLMs they evaluate, LLMs that perform evaluations cannot be trusted.”

• cites [43] [43] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324 (2023). - sourceCould not verify

  The Introduction reports that LLM performance is sensitive to prompt formatting, and the Discussion recommends evaluating across plausible formats (page numbers unavailable). Neither passage addresses LLM-generated evaluators inheriting the problems of models they evaluate or requiring human validation. Searches for those claims found no supporting passage in our copy. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

May combine two claims.

Many approaches to align LLMs rely on custom model training or finetuning, but all strategies heavily rely on humans to identify examples of desirable and undesirable outputs.

“Many approaches rely on custom model training or finetuning [6], but all strategies heavily rely on humans to identify examples of desirable and undesirable outputs.”

• cites [6] [6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017). - sourcePartly supported

  The citation is attached to the first clause, about approaches relying on custom model training or finetuning. The Introduction (p. 1) describes training a deep reinforcement-learning system using a learned reward function, which supports custom model training as an example. It does not establish that many approaches to aligning LLMs use such training or finetuning; the paper concerns RL tasks rather than LLMs. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Liu et al. demonstrated using annotated LLM outputs judged on criteria like consistency and relevance as few-shot examples for calibrating LLM-based evaluators.

“Liu et al. [34] demonstrated using annotated LLM outputs-judged on criteria like consistency and relevance-as 'few-shot examples' for calibrating LLM-based evaluators.”

• cites [34] [34] Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023. Calibrating LLM-based evaluator. arXiv preprint arXiv:2309.13308 (2023). - sourcePartly supported

  Section 2.3 describes using human-labeled sample–score pairs as few-shot exemplars to infer scoring criteria for calibrating an LLM-based evaluator. Section 4.1 evaluates criteria including consistency and relevance. The copy does not establish that the annotated outputs used as examples were specifically generated by LLMs, rather than other text-generation systems. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

SPADE helps developers generate Python assertion functions for LLM outputs from prompt history.

“One recent LLM-assisted approach, SPADE [45], makes headway on these issues, helping developers generate Python assertion functions for LLM outputs from prompt history.”

• cites [45] [45] Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, JD Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G Parameswaran, and Eugene Wu. 2024. SPADE: Synthesizing Assertions for Large Language Model Pipelines. arXiv preprint arXiv:2401.03038 (2024). - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Interfaces that facilitate seamless transitions between iML activities result in fewer errors and outputs that better match users' expectations.

“Interfaces that facilitate seamless transitions between these activities result in fewer errors and outputs that better match users' expectations [38, 46].”

• cites [38] [38] Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Amy J. Ko, and James Landay. 2010. Gestalt: integrated support for implementation and analysis in machine learning. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST '10) . Association for Computing Machinery, New York, NY, USA, 37-46. https: //doi.org/10.1145/1866029.1866038 - sourceNot checked — no full text of the source
• cites [46] [46] Patrice Simard, David Chickering, Aparna Lakshmiratan, Denis Charles, Léon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, and Jina Suh. 2014. Ice: enabling non-experts to build models interactively for large-scale lopsided problems. arXiv preprint arXiv:1409.4814 (2014). - sourcePartly supported

  Section 2.2 (p. 3) says ICE integrates labeling and feature editing in one interface and lets teachers inspect model errors to decide what to do next. Section 2.3 (p. 4) describes tools for reviewing errors and revising labels. These passages support the integrated workflow, but do not establish that seamless transitions result in fewer errors or outputs that better match users’ expectations. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Even people familiar with LLMs struggled to scale up their evaluations, appearing to over-generalize from a limited number of outputs even after an automated evaluation pipeline was setup.

“Similarly, Arawjo et al. [1] found that even people familiar with LLMs (developers, academics in ML) struggled to scale up their evaluations, appearing to over-generalize from a limited number of outputs even after an automated evaluation pipeline was setup.”

• cites [1] [1] Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. arXiv preprint arXiv:2309.09128 (2023). - sourceSupported

Classified as probably not a substantial claim.

AutoCalibrate is a method to calibrate LLM evaluators with human preferences that requires large expert-labelled datasets with settled criteria.

“For instance, AutoCalibrate is a method to calibrate LLM evaluators with human preferences that requires large expert-labelled datasets with settled (i.e., established upfront) criteria [34].”

• cites [34] [34] Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023. Calibrating LLM-based evaluator. arXiv preprint arXiv:2309.13308 (2023). - sourcePartly supported

  Sections 2.1 and 2.3 describe AutoCalibrate as aligning LLM evaluators with human expert labels. But the Effect of Few-Shot Example Size section says 8–12 examples suffice for inferring effective criteria, rather than establishing a need for large labelled datasets. Section 2.3 and Criteria Revisiting say the method drafts and refines criteria from examples, rather than requiring criteria settled upfront. Page numbers are unavailable in our copy. (LLM, conf=0.89)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

This suggests an optimization loop akin to ConstitutionMaker and DSPy, but for assertion generation and validation rather than prompt optimization.

“This suggests an optimization loop akin to ConstitutionMaker and DSPy [26, 39], but for assertion generation and validation , rather than prompt optimization.”

• cites [26] [26] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. 2023. Dspy: Compiling declarative language model calls 27. into self-improving pipelines. arXiv preprint arXiv:2310.03714 (2023). - sourceSupported
• cites [39] [39] Savvas Petridis, Ben Wedin, James Wexler, Aaron Donsbach, Mahima Pushkarna, Nitesh Goyal, Carrie J. Cai, and Michael Terry. 2023. ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles. arXiv:2310.15428 [cs.HC] - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Users want data visualization plots between the original prompt and its revision, akin to LLM Comparator and EvalLM.

“This suggests users want data visualization plots between the original prompt and its revision, akin to LLM Comparator and EvalLM [24, 27].”

• cites [24] [24] Minsuk Kahng, Ian Tenney, Mahima Pushkarna, Michael Xieyang Liu, James Wexler, Emily Reif, Krystal Kallarackal, Minsuk Chang, Michael Terry, and Lucas Dixon. 2024. LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models. arXiv:2402.10524 [cs.HC] - sourcePartly supported

  The Visualization Design and Development section describes LLM Comparator as a side-by-side comparison tool with a visualization summary, supporting its use as an example of visual comparison. But Section 3.1 says it compares two models’ responses to a prompt, not an original prompt with its revision. The cited paper therefore supports the analogy, not that specific comparison. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [27] [27] Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. arXiv preprint arXiv:2309.13633 (2023). - sourceSupported

May combine two claims.

The dependence of criteria on a specific prompt and LLM's outputs means that any change to the LLM pipeline can cause criteria drift; for example, LLM APIs can be replaced with new models, resulting in prompt drift.

“In particular, the dependence of criteria on a specific prompt and LLM's outputs means that any change to the LLM pipeline can cause criteria drift: for example, LLM APIs can be replaced with new models, resulting in prompt drift [4].”

• cites [4] [4] Lingjiao Chen, Matei Zaharia, and James Zou. 2023. How is ChatGPT's behavior changing over time? arXiv preprint arXiv:2307.09009 (2023). - sourcePartly supported

  The Introduction (page unavailable) says that changes in an LLM’s response to a prompt can break a downstream pipeline. Section 3.6 and the accompanying discussion (pages unavailable) describe prompt-engineered LangChain modules as brittle when LLM behavior drifts. The paper does not establish that any pipeline change causes criteria drift, or describe replacing an API with a new model as causing prompt drift. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The practice of benchmarks in ML and NLP presumes a world of well-defined criteria (and well-labeled data) on which to judge LLM outputs.

Probably not a substantial claim: may not pass S3 (load-bearing).

Many users write their LLM pipelines as chains of multiple LLM calls.

“Second, many users write their LLM pipelines as chains of multiple LLM calls [1, 11, 14, 53].”

• cites [1] [1] Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. arXiv preprint arXiv:2309.09128 (2023). - sourcePartly supported

  Section 8.1 reports that five interviewees prototyped LLM data-processing pipelines and sent many parametrized prompts. The paper also describes chaining Prompt Nodes (Section 5), but it does not establish that many users wrote pipelines as chains of multiple sequential LLM calls; sending multiple prompts is not necessarily chaining them. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [11] [11] Harrison Chase et al. 2023. LangChain. https://pypi.org/project/langchain/. -Not checked yet

  Not checked in the free run.

• cites [14] [14] FlowiseAI, Inc. 2023. FlowiseAI Build LLMs Apps Easily. flowiseai.com. -Not checked yet

  Not checked in the free run.

• cites [53] [53] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA '22) . Association for Computing Machinery, New York, NY, USA, Article 359, 10 pages. https://doi.org/10.1145/3491101.3519729 - sourcePartly supported

  The Introduction (p. 2) describes chains of multiple LLM steps and reports that four designers and developers built chains in the study. This supports the practice described, but the paper does not establish that many users write their pipelines this way. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

A number of LLMOps tools exist for users to implement code-based guardrails.

“In fact, a number of LLMOps tools exist for users to implement such code-based guardrails [19, 22, 42].”

• cites [19] [19] Guardrails 2023. Guardrails AI. https://github.com/guardrails-ai/guardrails. -Not checked yet

  Not checked in the free run.

• cites [22] [22] Instructor 2023. Instructor, Generating Structure from LLMs. https://jxnl.github.io/instructor/. -Not checked yet

  Not checked in the free run.

• cites [42] [42] Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501 (2023). - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior work underscores the importance of soliciting a representative graded sample of LLM outputs.

“Prior work also underscores the importance of soliciting a representative graded sample of LLM outputs [2, 45].”

• cites [2] [2] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. 2024. AutoEval Done Right: Using Synthetic Data for Model Evaluation. arXiv preprint arXiv:2403.07008 (2024). - sourceSupported
• cites [45] [45] Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, JD Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G Parameswaran, and Eugene Wu. 2024. SPADE: Synthesizing Assertions for Large Language Model Pipelines. arXiv preprint arXiv:2401.03038 (2024). - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Some frameworks already experiment with using feedback from assertions or user grades to refine prompts.

“Some frameworks already experiment with using feedback from assertions or user grades to refine prompts [35, 39, 48].”

• cites [35] [35] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024). - sourcePartly supported

  Section 2, “Iterative Refinement with SELF-REFINE” (page unavailable), supports the general idea of a framework that uses feedback for refinement. But it says the model generates feedback on its own outputs and refines those outputs—not prompts—while the “Refine Feedback” section says the process does not involve human assistance. The paper does not support feedback from assertions or user grades as described in the claim. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [39] [39] Savvas Petridis, Ben Wedin, James Wexler, Aaron Donsbach, Mahima Pushkarna, Nitesh Goyal, Carrie J. Cai, and Michael Terry. 2023. ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles. arXiv:2310.15428 [cs.HC] - sourcePartly supported

  Section 2.2 says ConstitutionMaker converts users’ feedback on prompt outputs into principles incorporated back into the prompt. Section 5.2 describes kudos, critiques, and rewrites as feedback, but does not describe feedback from assertions or user grades. The prompt-refinement point is supported; the specified feedback forms are not. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [48] [48] Arnav Singhvi, Manish Shetty, Shangyin Tan, Christopher Potts, Koushik Sen, Matei Zaharia, and Omar Khattab. 2023. DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines. arXiv preprint arXiv:2312.13382 (2023). - sourceSupported

May combine two claims.

The alignment process is subjective and iterative.

“A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative process of alignment.”

May combine two claims.

Criteria drift occurs: users need criteria to grade outputs, but grading outputs helps users define criteria.

“we identify a phenomenon we dub criteria drift : users need criteria to grade outputs, but grading outputs helps users define criteria.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “We observed a 'catch-22' situation: to grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Common issues included bad reviews or lengthy content in the product pipeline outputs.

“Common issues included the presence of personal information in the medical pipeline outputs and bad reviews or lengthy content in the product pipeline outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Only one participant (P9) did not understand what the overall set's coverage and false failure rate meant, but did once the researcher explained it.

“Only one participant did not understand what the overall set's coverage and false failure rate meant (P9), but did once the researcher explained it.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

3 out of 9 participants again opened the EvalGen wizard to iterate on their criteria and assertions, or stopped because they were satisfied or tired.

“3 out of 9 participants again opened the EvalGen wizard to iterate on their criteria and assertions (back to step 2), or stopped because they were satisfied or tired.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In starting EvalGen, 6 participants clicked autogenerate criteria; 1 wrote a criterion then clicked autogenerate; 2 wanted to grade first (P4, P9).

“6 participants clicked the autogenerate button; 1 participant wrote a criterion themselves and then clicked the auto-generate button. The remaining 2 participants wanted to grade first (P4, P9).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All 9 participants used the provided NER task; 3 changed the prompt; 1 changed the task from NER to sentiment analysis.

“All participants ( 𝑛 = 9) used the provided task (a prompt template for NER of a dataset of 100 tweets, described in 6). Three (3) participants changed the prompt: of these three, one (1) person changed the task from NER to sentiment analysis.”

May combine two claims.

EvalGen achieved equal or better alignment than SPADE with fewer assertions for both pipelines.

“Our results in Table 1 show that EvalGen, by incorporating human judgment during criteria selection, achieved equal or better alignment than SPADE with fewer assertions for both pipelines.”

• Also stated in: § 5.2 Impact of Human Input in the Criteria Generation Step · p. 7 · plain

  “With user input at the criteria stage, EvalGen achieves the same or greater alignment with fewer functions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

More than half of the participants expressed interest in iterating on their assertions if they had more time.

“More than half of the participants expressed interest in iterating on their assertions if they had more time.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Common issues included the presence of personal information in the medical pipeline outputs.

“Common issues included the presence of personal information in the medical pipeline outputs and bad reviews or lengthy content in the product pipeline outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants usually left evaluation type unchanged from what EvalGen suggested, even if EvalGen suggested a type that did not make sense (e.g., checking word count with an LLM API call), which happened rarely.

“They usually left evaluation type (code-based or LLM-based) unchanged from what EvalGen suggested, even if EvalGen suggested a type that did not make sense (e.g., checking word count with an LLM API call, rather than code), which happened rarely.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

After receiving criteria suggestions, participants removed some suggestions and added 1-2 criteria of their own.

“After receiving criteria suggestions from EvalGen, participants removed some suggestions and added 1-2 criteria of their own.”

May combine two claims.

Participants felt EvalGen was a great starting point for assertions and wanted to and could exercise control over EvalGen's assistance.

“Participants felt that EvalGen was a great starting point for assertions, and wanted to-and could-exercise control over EvalGen's assistance.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants who graded LLM outputs first graded between 5 and 10 outputs, with 2 to 4 thumbs-down grades.

“Participants who graded LLM outputs first graded between 5 and 10 outputs, with 2 to 4 'thumbsdown' grades.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the negative phrases criterion, EvalGen returned a more pragmatic assertion: an LLM-based validator to ensure product descriptions remained entirely positive.

“In contrast, EvalGen returned a more pragmatic assertion for this criterion-an LLM-based validator to ensure the product descriptions remained entirely positive.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants graded outputs while EvalGen generated and evaluated candidate assertions; some graded continuously for up to 10 minutes, others stopped after 10 grades.

“Participants graded outputs while EvalGen generated and evaluated different candidate assertions. Some participants graded continuously for up to 10 minutes; others stopped after 10 grades.”

May combine two claims.

All participants liked viewing the table of assertion results on all LLM outputs and expressed interest upon first viewing it, but they verified the results differently.

“Participants really liked viewing the table of assertion results on all LLM outputs, with all participants expressing interest upon first viewing it (Figure 3)-but they verified the results differently.”

May combine two claims.

Only one participant assessed alignment in the table via coverage by inspecting each row one at a time; everyone else assessed alignment via false failure rate by looking for rows where the assertion returned False.

“Interestingly, only one participant assessed alignment in this table via coverage-by inspecting each row in the table, one at a time-and everyone else assessed alignment via false failure rate: for each column (assertion), they typically looked for rows (LLM outputs) that the assertion returned False.”

Probably not a substantial claim: may not pass S3 (load-bearing).

P7 acknowledged that their ability to edit depends on code length and complexity.

“However, P7 then acknowledged that their ability to edit depends on code length and complexity:”

May combine two claims.

Participants who graded before selecting criteria (P4 and P9) found the grading process very useful and were happy that EvalGen prompted them to give feedback on why an LLM output was bad.

“Participants who graded before selecting criteria (P4 and P9) expressed that they found the grading process very useful. Both participants were happy that EvalGen prompted them to give feedback on why an LLM output was bad.”

May combine two claims.

Participants reasoned about code-based assertions and LLM-based assertions differently: while they liked having assertions of both types, they wanted to construct and iterate on them differently in order to feel like the resulting set of assertions aligned with their preferences.

“Yet participants reasoned about code-based assertions and LLM-based assertions differently: while they liked having assertions of both types, they wanted to construct and iterate on them differently in order to feel like the resulting set of assertions aligned with their preferences.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants wanted the table to automatically recompute, with an accompanying visualization, whenever there were changes to the LLM pipeline.

“Some participants wanted this table to automatically recompute, with an accompanying visualization, whenever there were changes to the LLM pipeline; for example, P6 said:”

Probably not a substantial claim: may not pass S3 (load-bearing).

P3 realized they could take good and bad examples and use them as few-shot exemplars in the prompt for the LLM evaluators.

“P3 excitedly realized this when looking at the table of results ('I can take these good and bad examples and use them as few-shot exemplars in the prompt for the LLM evaluators...').”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants (P3, P6, P8) were skeptical of how LLM-based assertions might transfer to monitoring LLM outputs in a production pipeline.

“Some participants were skeptical of how LLM-based assertions might transfer to monitoring LLM outputs in a production pipeline (P3, P6, P8).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Several participants realized they needed to go back and change their grades in EvalGen and were happy they could do this.

“Several participants realized they needed to go back and change their grades in EvalGen and were happy they could do this.”

Probably not a substantial claim: may not pass S3 (load-bearing).

P8 desired to grade for only one criteria, reasoning that it might improve efficiency.

“P8 desired to grade for only one criteria, reasoning that it might improve efficiency ('I generally want to be in the loop for these tests...but I want to put myself in the loop in a way that is efficient.').”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P2 said the table 'earns trust'; P5 said it was 'cool to see all the results on examples [they] didn't grade.'

“P2 said that this table 'earns trust'; P5 said that it was 'cool to see all the results on examples [they] didn't grade.'”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants (P1, P7) wanted to go back and grade more because they had not graded enough responses to get their ideal alignment numbers.

“Some participants said that they had not graded enough responses to get their ideal alignment numbers, so they wanted to go back and grade more (P1, P7).”

Probably not a substantial claim: may not pass S3 (load-bearing).

P2 and P5 wanted to change a selected assertion to another candidate assertion.

“Another place where participants wanted to exercise control but were limited by EvalGen's interface was in the selected assertions: P2 and P5 wanted to change a selected assertion to another candidate assertion.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

P8 said they cannot begin to think about how LLMs as validators in production can work and are very skeptical.

“P8 said, 'I cannot begin to think about how LLMs as validators [in production] can work, I'm very skeptical.'”

May combine two claims.

Participants liked seeing the coverage and false failure rate of the set, and how each assertion contributed to these overall statistics.

“Participants also liked seeing the coverage and false failure rate of the set, and how each assertion contributed to these overall statistics.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P9 asked how to maintain evals over time and whether they have to rerun the entire process.

“P9 asked, 'How do I maintain my evals over time; do I have to rerun this entire process?'”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P5 expressed the same uncertainty as P7; P7 said 'I think it's hard to know until you see it.'

“P5 expressed the same uncertainty. 'I think it's hard to know until you see it,' P7 said.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P9 initially selected the grading option because it seemed like the 'option that required the least thinking,' but later realized it was the correct thing to do.

“While P9 selected the grading option initially because it seemed like the 'option that required the least thinking,' they later realized that it was the correct thing to do:”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P7 initially expressed dissatisfaction that 7 criteria were generated, but after deleting some criteria realized the suggestions covered all the criteria they wanted and stated 'this auto-generation is sweet'.

“P7 initially expressed dissatisfaction that 7 criteria were generated, as they had expected less. But after deleting some criteria, they realized that the suggestions covered all the criteria they wanted (i.e., they did not have to add criteria themselves), and stated, 'this auto-generation is sweet'.”

Probably not a substantial claim: may not pass S3 (load-bearing).

P9 appreciated the ability to provide feedback on unsatisfactory outputs before finalizing the initial criteria set, as this process helped them articulate their criteria.

“P9 appreciated the ability to provide feedback on unsatisfactory outputs before finalizing the initial criteria set, as this process helped them articulate their criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing).

P7 wanted to iterate on code-based assertions by providing feedback to make the assertion more or less 'fancy', for example, making regex more or less complicated depending on the sample of LLM outputs seen.

“P7 wanted to iterate on code-based assertions simply by providing feedback to make the assertion more or less 'fancy'-for example, in checking for a pattern in the LLM output, they might want the regex to be more or less complicated depending on the sample of LLM outputs they've seen.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Both P4 and P6 mentioned that they liked the ability to correct EvalGen's suggested type.

“Both P4 and P6 mentioned that they liked the ability to correct EvalGen's suggested type.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Other participants who did not grade before selecting criteria regretted this decision; P2 said 'you should enforce that we all look at at least 20 examples first.'

“Other participants who did not grade before selecting criteria regretted this decision: for example, P2 said, 'you should enforce that we all look at at least 20 examples first.'”

Probably not a substantial claim: may not pass S3 (load-bearing).

P6 and P8 expressed that they wanted to use LLM-based assertions for their own LLM pipelines because 'most of their criteria was fuzzy.'

“P6 and P8 expressed that they wanted to use LLM-based assertions for their own LLM pipelines because 'most of their criteria was fuzzy.'”

May combine two claims. The quote only partly states this claim.

All participants had a criterion to enforce that there were no entities with hashtags in the output; P5 and P9 interpreted this differently; both got the same code-based assertion checking for the presence of the hashtag character, which did not align with P5's grades but did with P9's.

“For example, while all participants had a criterion to enforce that there were no entities with hashtags in the output, some participants interpreted this as any hashtags representing entities should not be extracted as entities: e.g., if the output included the hashtag #Nike, P5 did not want Nike to be extracted at all. On the other hand, P9 wanted Nike to be extracted as an entity, but they did”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P5 initially did not like using a GUI to come up with assertions, and then said they 'prefer looking at this table to [their] current workflow in Jupyter notebooks.'

“P5 initially did not like using a GUI to come up with assertions, and then said that they 'prefer looking at this table to [their] current workflow in Jupyter notebooks.'”

May combine two claims.

Participants overall felt that EvalGen provided a good start to code-based assertions, and it was easy for them to edit the code if they wanted; two participants (P4, P6) asked for the ability to export code-based assertions as unit tests or in a Python file.

“While participants had lots of ideas for improvement, participants overall felt that EvalGen provided a good start to code-based assertions, and it was easy for them to edit the code if they wanted: two participants even asked for the ability to export code-based assertions as unit tests or in a Python file (P4, P6).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P8 preferred LLM-based to code-based assertions because LLMs are relatively forgiving when encountering outputs with an unexpected quality or format, whereas code assertions check exact constraints.

“We also observed a participant (P8) preferring LLM-based to code-based assertions: LLMs are relatively forgiving when encountering outputs with an unexpected quality or format, whereas code assertions check exact constraints.”

May combine two claims.

Participants found EvalGen's suggested code-based assertions to be more obviously misaligned than the LLM-based assertions, but acknowledged that LLM-based assertions are harder for them to trust since they can edit the code-based assertions more easily than the LLM-based assertions (P2, P5, P6, P8, P9).

“While participants found EvalGen's suggested code-based assertions to be more obviously misaligned than the LLM-based assertions (Section 7.3.3), they also acknowledged that LLM-based assertions are harder for them to trust-since they can edit the code-based assertions more easily than the LLM-based assertions (P2, P5, P6, P8, P9).”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

P3 would trust the assertions more if the EvalGen process allowed them to set different false failure rates per criteria instead of one global false failure rate constraint for the entire assertion set.

“For example, P3 expressed that they would trust the assertions more if the EvalGen process allowed them to set different false failure rates per criteria (since LLMs might be bad at evaluating some criteria), instead of one global false failure rate constraint for the entire assertion set:”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both P2 and P5 wanted to change code-based assertions; P5 said 'this is a weird implementation of entity count' and preferred another assertion with a small modification.

“Both P2 and P5 wanted to change code-based assertions: for example, P5 said, 'this is a weird implementation of entity count' and preferred another assertion with a small modification.”

Probably not a substantial claim: may not pass S3 (load-bearing).

P3 found that some EvalGen-suggested criteria did not make sense given that assertions only operate on LLM pipeline input-output pairs, like response latency and response stability over time.

“P3 found that some EvalGen-suggested criteria did not make sense given that assertions only operate on LLMpipeline input-output pairs, like response latency and response stability over time.”

May combine two claims.

Participants generally liked the approach of exploring multiple candidate assertions and selecting the most aligned for LLM-based assertions, but did not like this for code-based assertions.

“While participants generally liked this approach for LLM-based assertions, they did not like this for code-based assertions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P5 felt that EvalGen could not find a good assertion that aligned with their grades for another criterion, and said they were 'lost at what would be a good implementation.'

“For another criterion, P5 felt that EvalGen could not find a good assertion that aligned with their grades, but also said that they were 'lost at what would be a good implementation.'”

May combine two claims.

For some criteria, EvalGen generated no good assertions, and participants deleted the criteria without complaints.

“However, for some criteria, EvalGen Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences generated no good assertions, and participants deleted the criteria without complaints.”

May combine two claims.

In the EvalGen interface, participants could not go back and add new criteria; they had to wait for all candidate assertions to finish executing, move past the report card screen, and start a new EvalGen process.

“In the EvalGen interface, they could not go back and add new criteria; they had to wait for all candidate assertions to finish executing, move past the report card screen, and start a new EvalGen process.”

May combine two claims.

All participants liked EvalGen's ability to assist in generating both code-based and LLM-based assertions, expressing the need for both types of assertions.

“All participants liked EvalGen's ability to assist in generating both code-based and LLM-based assertions, expressing the need for both types of assertions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The participant who did not agree (P7) said they would find it useful if, on the grading screen, EvalGen showed what it was doing with the grades.

“The participant who did not agree said that they would find it useful if, on the grading screen, EvalGen showed what it was doing with the grades.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P3 did not feel that this was a failure of EvalGen, but expressed that LLMs are not perfect and stressed the importance of having control over selected criteria.

“P3 did not feel that this was a failure of EvalGen, but rather expressed that LLMs are not perfect and stressed the importance of having control over selected criteria.”

May combine two claims.

In practice, developers rapidly iterate over criteria, and cognitively engaging with LLM outputs helps them to refine their criteria.

“However, in practice, we found that developers rapidly iterate over criteria, and furthermore that cognitively engaging with LLM outputs helps them to refine their criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

P7 noticed inconsistency in hashtag handling in outputs, which led P7 to rethink the criteria definition for including hashtags, specifically whether it was acceptable for the LLM to replicate entities from hashtags provided the hashtag was removed.

“P7 noticed that in some outputs included hashtags from the original tweet inputs (e.g., #justdoit), while in other cases, the outputs did not use the hashtag symbol but still mentioned the entities referred to by the hashtags (e.g., 'Nike' instead of #Nike). This inconsistency led P7 to rethink the criteria definition for including hashtags; specifically, whether it was acceptable for the LLM to”

May combine two claims.

Participants wanted EvalGen to use their grades, as well as natural language feedback on thumbs-down grades, in prompting LLMs for candidate assertions (P3, P4, P5, P7).

“While participants liked that EvalGen tried multiple candidate assertions for each criterion, they wanted EvalGen to use their grades, as well as natural language feedback on thumbs-down grades, in prompting LLMs for candidate assertions (P3, P4, P5, P7).”

May combine two claims.

Participants generally liked the process of grading LLM responses and feeling like the grades were useful, but they wanted to prioritize grading criteria they felt needed their alignment, especially for LLM-based assertions (P3, P5, P7, P8).

“Overall, participants generally liked the process of grading LLM responses and feeling like the grades were useful, but they wanted to prioritize grading criteria they felt needed their alignment -especially for LLM-based assertions (P3, P5, P7, P8).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants (P5, P6, P7, P8) expressed interest in seeing per-criterion alignment and asked to provide grades for each criterion.

“However, in this screen, some participants expressed interest in seeing per-criterion alignment and asked to provide grades for each criterion (P5, P6, P7, P8).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For an output length criterion, P3 mentioned they would look at a histogram of word lengths for a batch of outputs and see if there were buckets of anomalous word lengths that exhibited other failure modes.

“In fact, specifically for an output length criterion, P3 mentioned that they would look at a histogram of word lengths for a batch of outputs and see if there were some buckets of anomalous word lengths that exhibited other failure modes (e.g., a 'rambling' response from the LLM).”

May combine two claims.

Participants felt each type of assertion required distinct treatment, both in implementation selection and in how they perceive alignment.

“Our study confirmed the dual necessity for both code-based and LLM-based assertions among participants, but further showed that participants felt each type required distinct treatment, both in implementation selection and in how they perceive alignment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One participant requested a passive deployment where a daily report of assertion results on a sample of LLM outputs is sent to their email inbox.

“Others wanted a more passive deployment-one participant requested for some form of report sent to their email inbox daily, containing results of the assertion set executed on a sample of LLM outputs from that day.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants wanted their collaborators, such as product managers, to grade outputs in EvalGen.

“Some participants mentioned that they wanted their collaborators, such as product managers, to grade outputs in EvalGen.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants expressed the desire to deploy their assertions in production.

“Participants expressed the desire to deploy their assertions in production.”

May combine two claims.

While the alternating policy didn't consistently outperform, the results suggest that any non-random policy implemented in EvalGen may achieve satisfactory outcomes.

“Notably, while the alternating policy didn't consistently outperform, our results suggest that any non-random policy implemented in EvalGen may achieve satisfactory outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some participants wanted to deploy assertions in the critical path of the LLM pipeline to catch bad outputs before they are shown to end-users.

“Some wanted to deploy assertions in the critical path of the LLM pipeline, to catch bad outputs before they are shown to end-users.”

Probably not a substantial claim: may not pass S2 (checkable).

Users need evaluation assistants to support rapid iteration over criteria and implementations simultaneously.

“Thus, our findings suggest that users need evaluation assistants to support rapid iteration over criteria and implementations simultaneously .”

Classified as probably not a substantial claim.

Two authors manually graded all LLM outputs to establish ground-truth labels.

“Two of the paper authors manually graded all LLM outputs to establish ground-truth labels.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

This illustrates the challenge of balancing coverage with false failures, underscoring the need for evaluator systems to effectively navigate these trade-offs.

“This illustrates the challenge of balancing coverage with false failures, underscoring the need for evaluator systems to effectively navigate these trade-offs.”

Classified as probably not a substantial claim.

The analysis employed open and axial coding to identify common themes across transcripts and notes.

“We employed open and axial coding [13] to identify common themes across the transcripts and notes for each participant.”

• cites [13] [13] Uwe Flick. 2013. The SAGE handbook of qualitative data analysis . Sage. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The study was approved by the institutional review board (IRB).

“Our study was approved by our institutional review board (IRB), and participants generously volunteered their time.”

May combine two claims.

The human cannot completely validate an LLM-based evaluator because the point of LLM evaluators is to reduce the effort required by the developer.

“The human cannot completely validate an LLM-based evaluator: the point of LLM evaluators is to reduce the effort required by the developer, who would otherwise have to grade outputs manually.”

Classified as probably not a substantial claim. May combine two claims.

For code-based assertions, LLMs occasionally synthesize erroneous functions, requiring several candidate assertions.

“Moreover, for code-based assertions, LLMs occasionally synthesize erroneous functions (e.g., hallucinating a function in a Python library), requiring several candidate assertions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AI systems that showcase subsets of errors with the same accuracy can lead to vastly different perceptions of accuracy.

“Kocielnik et al. [29] found that AI systems that showcase subsets of errors, like false positives or false negatives, that have the same accuracy, can lead to vastly different perceptions of accuracy.”

• cites [29] [29] Rafal Kocielnik, Saleema Amershi, and Paul N. Bennett. 2019. Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3290605.3300641 - sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

The only way to fully align an LLM evaluator would be to ask the user to label all outputs, which defeats the purpose.

“The only way to fully align an LLM evaluator would be to ask the user to label all outputs; obviously, this defeats the purpose.”

Classified as probably not a substantial claim.

Asking the developer to grade some outputs, using some time they would have spent waiting anyway, is the key idea behind the design.

“Asking the developer to grade some outputs, using some time they would have spent waiting anyway, is the key idea behind our design.”

Probably not a substantial claim: may not pass S4 (based).

Generating multiple candidate assertions improves the probability that there is at least one implementation that aligns with developer expectations.

“Generating multiple candidate assertions improves the probability that there is at least one implementation that aligns with developer expectations.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Auditing LLM outputs is far from easy, with humans prone to the dual biases of over-generalization and overreliance.

“It also reveals that auditing LLM outputs is far from easy, with humans prone to the dual biases of over-generalization and overreliance.”

Classified as probably not a substantial claim.

It remains unclear how to support developers in their prototyping of evaluations.

“It thus remains unclear how to support developers in their prototyping of evaluations, with the problem becoming even more pressing as the popularity of prompt optimization increases.”

May combine two claims.

There is a catch-22: participants needed to externalize criteria in order to grade outputs, but they also needed to grade outputs to externalize criteria.

“Looking more closely at their interactions, we observed a catch-22 situation: participants needed to externalize criteria in order to grade outputs, but they also needed to grade outputs-providing feedback on why bad outputs were bad-in order to externalize criteria.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Criteria refinement and grading should happen in tandem in interactive settings, and this poses challenges to alignment methods that presume settled, expert labels.

“This suggests criteria refinement and grading should happen in tandem in interactive settings, and poses challenges to alignment methods that presume settled, expert labels.”

• AssumptionRests on an assumption: The practice of benchmarks in ML and NLP presumes a world of well-defined criteria (and well-labeled data) on which to judge LLM outputs.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

A criterion like word count is hard for humans to assess but easy for a good Python function to evaluate.

“A criterion like word count is hard for humans to assess but easy for a good Python function to evaluate.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The observation that participants did not like the approach for code-based assertions may be specific to users who are experts in coding (as all participants were).

“This observation may be specific to users who are experts in coding (as all of our participants are).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study setup did not cover the deployment phase of LLM workflows.

“Moreover, participants did not have enough time to iterate many times in EvalGen, and our setup did not cover the deployment phase of LLM workflows.”

Probably not a substantial claim: may not pass S2 (checkable).

Recognizing the dependency of criteria on LLM outputs highlights new directions for designing future evaluation assistants.

“Recognizing the dependency of criteria on LLM outputs highlights new directions for designing future evaluation assistants.”

• AssumptionRests on an assumption: Human preferences are the appropriate standard for evaluating LLM outputs.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Initially, when users start grading outputs in EvalGen, they might effectively be grading random outputs for the first one or two outputs, as the sigma scores update and stabilize.

“Initially, when users start grading outputs in EvalGen, they might effectively be grading random outputs for the first one or two outputs, as the 𝜎 scores update and stabilize.”

Classified as probably not a substantial claim.

Lower scores, resulting from failures by non-selective assertions, point to uncertainty or potential issues with the output.

“A score of 0 means no assertions have failed the output, indicating a higher likelihood of quality, while lower scores, resulting from failures by non-selective assertions, point to uncertainty or potential issues with the output.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In practice, assertions have different operational requirements.

“In practice, assertions have different operational requirements.”

Classified as probably not a substantial claim.

A score of 0 means no assertions have failed the output, indicating a higher likelihood of quality.

“A score of 0 means no assertions have failed the output, indicating a higher likelihood of quality, while lower scores, resulting from failures by non-selective assertions, point to uncertainty or potential issues with the output.”

Classified as probably not a substantial claim.

'NER for tweets' is a problem that many people have studied.

“third, 'NER for tweets' is a problem that many people have studied [15, 33, 49].”

• cites [15] [15] Michel Naim Gerguis, Cherif Salama, and M Watheq El-Kharashi. 2016. ASU: An Experimental Study on Applying Deep Learning in Twitter Named Entity Recognition.. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT) . 188-196. -Not checked yet

  Not checked in the free run.

• cites [33] [33] Xiaohua Liu, Furu Wei, Shaodian Zhang, and Ming Zhou. 2013. Named entity recognition for tweets. ACM Transactions on Intelligent Systems and Technology (TIST) 4, 1 (2013), 1-15. - sourceNot checked — no full text of the source
• cites [49] [49] Chanchal Suman, Saichethan Miriyala Reddy, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Why pay more? A simple and efficient named entity recognition system for tweets. Expert Systems with Applications 167 (2021), 114101. - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The qualitative study focused on a small sample of developers, all of whom have significant experience deploying LLMs.

“Second, our qualitative study focused on a small sample of developers, all of whom have significant experience deploying LLMs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants did not have enough time to iterate many times in EvalGen.

“Moreover, participants did not have enough time to iterate many times in EvalGen, and our setup did not cover the deployment phase of LLM workflows.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This inconsistency could lead to user frustration, particularly if the effort spent in grading outputs results in assertion sets with unpredictable relevance to their specified criteria.

“This inconsistency could lead to user frustration, particularly if the effort spent in grading outputs results in assertion sets with unpredictable relevance to their specified criteria.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The off-line evaluation only focused on two pipelines.

“First, our off-line evaluation only focused on two pipelines.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The criteria drift finding echoes prior work in educational settings, where instructors often update their grading rubrics to reflect common errors as they grade more assignments.

“Our criteria drift finding echoes prior work in educational settings, where instructors often update their grading rubrics to reflect common errors as they grade more assignments [47].”

• cites [47] [47] Arjun Singh, Sergey Karayev, Kevin Gutowski, and Pieter Abbeel. 2017. Gradescope: a fast, flexible, and fair system for scalable assessment of handwritten work. In Proceedings of the fourth (2017) acm conference on learning@ scale . 81-88. - sourceNot checked — no full text of the source