Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)

Kenthapadi, Krishnaram; Sameki, Mehrnoosh; Taly, Ankur

doi:10.1145/3637528.3671467

Cited by 36Ethics and Social Impacts of AIAdversarial Robustness in Machine LearningExplainable Artificial Intelligence (XAI)

Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)

Open access

Kenthapadi, Krishnaram, Sameki, Mehrnoosh, Taly, Ankur·2024·Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

DOI 10.1145/3637528.3671467

Open this paper in CoChat — free account Open PDF

Abstract

With the ongoing rapid adoption of Artificial Intelligence (AI)-based systems in high-stakes domains, ensuring the trustworthiness, safety, and observability of these systems has become crucial. It is essential to evaluate and monitor AI systems not only for accuracy and quality-related metrics but also for robustness, bias, security, interpretability, and other responsible AI dimensions. We focus on large language models (LLMs) and other generative AI models, which present additional challenges such as hallucinations, harmful and manipulative content, and copyright infringement. In this survey article accompanying our tutorial, we highlight a wide range of harms associated with generative AI systems, and survey state of the art approaches (along with open challenges) to address these harms.

AI snapshot

Written from the paper’s full text.

TL;DR

This survey/tutorial reviews harms and practical challenges for grounding and evaluating large language models, with emphasis on hallucinations, safety, bias, robustness, privacy, calibration, and transparency. It surveys solution approaches—especially retrieval augmented generation, constrained decoding, guardrails, revision, and corpus tuning—and frames deployment around identification, measurement, mitigation, and operationalization.

Key findings

Grounding is defined as ensuring every claim in an LLM response can be attributed to a document in a user-specified knowledge base; it differs from factuality, which seeks attribution to commonly agreed world knowledge.
The paper organizes responsible AI evaluation around dimensions including truthfulness, safety and alignment, bias and fairness, robustness and security, privacy/unlearning/copyright, calibration and confidence, and transparency/causal interventions, and highlights harms such as hallucinations, jailbreaks, prompt injection, harmful content, and copyright infringement.
For grounding, it surveys retrieval augmented generation (RAG), constrained decoding, evaluation/guardrails/revision, and corpus tuning; RAG systems are evaluated on context relevance, answer faithfulness, and answer relevance.
For LLM operations, it proposes identification, measurement, mitigation (with four mitigation layers at model, safety system, application, and positioning levels), and operationalization; open challenges include imperfect retrieval, balancing grounding with creativity/helpfulness, multimodal RAG, robust guardrails, scalable privacy/unlearning, and out-of-distribution calibration.

Methods

Survey/tutorial article accompanying a tutorial; no original empirical study, dataset, or quantitative results are reported.
Synthesizes literature and industry lessons across LLM evaluation dimensions, grounding, and LLM operations/observability.
Presents business problems, solution approaches, and open challenges for each dimension rather than a formal systematic review protocol.

Limitations

The paper states many open challenges rather than resolved limitations, including imperfect retrieval, balancing grounding with creativity/helpfulness, extending RAG to multimodal settings, robust guardrails against attacks, scalable privacy/unlearning, and calibration in out-of-distribution settings.
Evident from the text: it is a survey/tutorial without primary experiments, quantitative benchmarks, or sample sizes, so claims are synthesized from cited work and industry practice rather than tested here.
Evident from the text: coverage is broad and high-level; for example, the RAG description is “only a brief overview,” and readers are referred to dedicated surveys.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3637528.3671467

Cites

89 of 89 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 88 cited sources checked · 0 supported · 0 flagged

Premisecitation
Fine-tuning large language models on new information that was not acquired during pretraining can encourage the model to hallucinate.
“Finally, a recent work shows that fine-tuning LLMs on new information that was not acquired during pretraining can encourage the model to hallucinate [38].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [38] source
  Not checked in the free run.
Premisecitation
Various methods have been proposed to detect hallucinations, from sampling-based approaches to those that use the internal states of the model.
“A variety of methods have been proposed to detect hallucinations, ranging from sampling based approaches [75] to approaches leveraging internal states of the LLM [104].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [75] source
  Not checked in the free run.
- cites [104]
  Not checked in the free run.
Premisecitation
One line of research trains or fine-tunes large language models on highly curated, textbook-like datasets.
“One line of work involves training or fine-tuning LLMs on highly curated textbook-like datasets [40,134].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [40]
  Not checked in the free run.
- cites [134]
  Not checked in the free run.
Premisecitation
Several methods have been proposed to fundamentally reduce hallucinations by tuning large language models.
“A number of methods have been proposed to fundamentally reduce hallucinations by tuning models.”
Quote in the paperNot a substantial claim0 of 3 sources checked
- cites [40]
  Not checked in the free run.
- cites [134]
  Not checked in the free run.
- cites [109]
  Not checked in the free run.
Interpretationcitation
Curating fine-tuning datasets to avoid introducing new information that was not in pretraining offers another way to reduce hallucinations.
“Curating fine-tuning sets to avoid this issue paves another path to reducing hallucinations.”
Quote in the paperSubstantial claim

Premisecitation
Recent research indicates that guardrail models based on large language models can themselves be attacked.
“Recent research has shown that LLM based guardrail models could themselves be attacked.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [76]
  Not checked in the free run.
Premisecitation
Benchmarks have been introduced for red-teaming or testing large language models against adversarial attacks and related problems.
“Several benchmarks have been proposed for red-teaming / testing LLMs against adversarial attacks and related issues [31,87,135].”
Quote in the paperNot a substantial claim0 of 3 sources checked
- cites [31]
  Not checked in the free run.
- cites [87] source
  Not checked in the free run.
- cites [135]
  Not checked in the free run.
Premisecitation
Large language models are susceptible to universal and transferable adversarial attacks that target alignment.
“LLMs have been shown to be vulnerable to adversarial perturbations in prompts [135], prompt injection attacks [121], data poisoning attacks [116], and universal and transferable adversarial attacks on alignment [137].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [137]
  Not checked in the free run.
Premisecitation
Several recent studies have produced adversarial prompts designed to circumvent current safeguards against toxic output.
“There has a been a bunch of recent work on generating adversarial prompts to bypass existing mechanisms for mitigating toxic content generation [119,137].”
Quote in the paperSubstantial claim0 of 2 sources checked
- cites [119]
  Not checked in the free run.
- cites [137]
  Not checked in the free run.
Premisecitation
Large language models are susceptible to data poisoning attacks.
“LLMs have been shown to be vulnerable to adversarial perturbations in prompts [135], prompt injection attacks [121], data poisoning attacks [116], and universal and transferable adversarial attacks on alignment [137].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [116] source
  Not checked in the free run.
Premisecitation
A two-step attack method that builds a universal adversarial prefix for the guardrail and then propagates it to the response has been shown to work across multiple threat models, including ones where the adversary cannot access the guardrail.
“For instance, a two-step prefixbased attack procedure -that operates by (a) constructing a universal adversarial prefix for the guardrail model, and (b) propagating this prefix to the response -has been shown to be effective across multiple threat models, including ones in which the adversary has no access to the guardrail model at all [76].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [76]
  Not checked in the free run.
Premisecitation
Large language models are susceptible to prompt injection attacks.
“LLMs have been shown to be vulnerable to adversarial perturbations in prompts [135], prompt injection attacks [121], data poisoning attacks [116], and universal and transferable adversarial attacks on alignment [137].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [121]
  Not checked in the free run.
Premisecitation
Large language models are susceptible to adversarial perturbations introduced in prompts.
“LLMs have been shown to be vulnerable to adversarial perturbations in prompts [135], prompt injection attacks [121], data poisoning attacks [116], and universal and transferable adversarial attacks on alignment [137].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [135]
  Not checked in the free run.
Premisecitation
Metrics have been introduced for quantifying cybersecurity risks of large language models.
“Metrics for quantifying LLM cybersecurity risks, tools to evaluate the frequency of insecure code suggestions, and tools to evaluate LLMs to make it harder to generate malicious code or aid in carrying out cyberattacks have also been proposed [11].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [11]
  Not checked in the free run.
Premisecitation
Learning to defer requires uncertainty quantification and confidence estimation for the underlying AI models.
“Learning to defer in human-AI settings is an active area of research [61], necessitating uncertainty quantification and confidence estimation for the underlying AI models.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [61] source
  Not checked in the free run.
Premisecitation
Tools have been introduced to evaluate large language models in ways that make generating malicious code or assisting cyberattacks harder.
“Metrics for quantifying LLM cybersecurity risks, tools to evaluate the frequency of insecure code suggestions, and tools to evaluate LLMs to make it harder to generate malicious code or aid in carrying out cyberattacks have also been proposed [11].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [11]
  Not checked in the free run.
Premisecitation
Adversaries can exploit combinations of models by splitting a malicious task into subtasks, using aligned frontier models for difficult benign subtasks, and using weaker non-aligned models for easy malicious subtasks.
“For example, it has been shown that adversaries can misuse combinations of models by decomposing a malicious task into subtasks, leveraging aligned frontier models to solve hard but benign subtasks, and leveraging weaker non-aligned models to solve easy but malicious subtasks [56].”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [56]
  Not checked in the free run.
Premisecitation
Learning to defer in human-AI settings is an active research area.
“Learning to defer in human-AI settings is an active area of research [61], necessitating uncertainty quantification and confidence estimation for the underlying AI models.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [61] source
  Not checked in the free run.
Premisecitation
Training data can be extracted from large language models.
“Recent studies have shown that training data can be extracted from LLMs [17] and from diffusion models [16] (which could have copyright implications in case the model is perceived as a database from which the original images or other copyrighted data can be approximately retrieved).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [17]
  Not checked in the free run.
Premisecitation
Training data can be extracted from diffusion models.
“Recent studies have shown that training data can be extracted from LLMs [17] and from diffusion models [16] (which could have copyright implications in case the model is perceived as a database from which the original images or other copyrighted data can be approximately retrieved).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [16]
  Not checked in the free run.
Premisecitation
Tools have been introduced to assess how often insecure code suggestions occur.
“Metrics for quantifying LLM cybersecurity risks, tools to evaluate the frequency of insecure code suggestions, and tools to evaluate LLMs to make it harder to generate malicious code or aid in carrying out cyberattacks have also been proposed [11].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [11]
  Not checked in the free run.
Premisecitation
Models struggle to balance grounding with other response attributes.
“and balancing grounding with other response attributes [34,67].”
Quote in the paperSubstantial claim0 of 2 sources checked
- cites [34]
  Not checked in the free run.
- cites [67]
  Not checked in the free run.
Premisecitation
Causal tracing approaches have been proposed to locate and edit factual associations in large language models.
“Causal tracing approaches have been proposed to locate and edit factual associations in LLMs.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [78]
  Not checked in the free run.
Premisecitation
There is research on unfaithful explanations in chain-of-thought prompting.
“However, there is work on unfaithful explanations in chain-of-thought prompting [113], with connections to language model alignment through externalized reasoning (getting models to do as much processing/reasoning through natural language as possible).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [113]
  Not checked in the free run.
Premisecitation
Including contexts with varying noise levels in the tuning set improves robustness of fine-tuned LLMs.
“In order to improve robustness, it is important to include a variety of contexts with varying levels of noise in the tuning set [72].”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [72]
  Not checked in the free run.
Premisecitation
For prompts that ask about time-varying information, models have difficulty producing grounded answers.
“multiple recent evaluations find that models struggle to generate grounded responses for prompts seeking time-varying information (e.g., “Who won the latest soccer match between Liverpool and Manchester United?”) [115]”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [115] source
  Not checked in the free run.
Premisecitation
NLI-based grounding checks are increasingly deployed as guardrails.
“While NLI based checks are getting rapidly deployed as guardrails for grounding [3,4], they often struggle with performing grounding checks that involve reasoning.”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [3]
  Not checked in the free run.
- cites [4]
  Not checked in the free run.
Premisecitation
LLM responses may not stay grounded in the supplied context, particularly when that context conflicts with the model's parametric knowledge.
“This is not a given, especially when the context conflicts with the LLM's own parameteric knowledge [123].”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [123]
  Not checked in the free run.
Premisecitation
Mitigation approaches include learning invariant representations and preventing models from relying on spurious patterns through methods such as augmentation, reweighting, ensembles, inductive priors, and causal interventions.
“Mitigation approaches involve learning invariant representations, and ensuring models do not rely on spurious patterns using techniques like data augmentation, reweighting, ensembling, inductiveprior design, and causal intervention [117].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [117] source
  Not checked in the free run.
Premisecitation
Reinforcement learning can reward grounded responses.
“Besides fine-tuning, it is also possible to use reinforcement learning (RL) to reward grounded responses [79].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [79]
  Not checked in the free run.
Interpretationcitation
Because smaller T5-family models can be trained for NLI checks, NLI is attractive for verifying grounding at inference time.
“The key advantage of this approach is that smaller T5-family [91] models can be trained to perform NLI checks, making this approach attractive for inference-time grounding verification.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites [91]
  Not checked in the free run.
Interpretationnone given
A key practical challenge for RAG is imperfect retrieval.
“A key practical challenge for RAG frameworks is grappling with imperfect retrieval.”
Quote in the paperSubstantial claim
- Rests on an assumption: For any retrieval system, some out-of-domain, adversarial, or nonsensical prompts will still retrieve irrelevant context.
Interpretationreasoning
NLI-based grounding checks often have difficulty with reasoning, including temporal statements, negation, and quantifiers.
“While NLI based checks are getting rapidly deployed as guardrails for grounding [3,4], they often struggle with performing grounding checks that involve reasoning.”
Quote in the paperSubstantial claim
Interpretationnone given
LLM-based grounding checks perform better when responses are abstract and do not quote directly from the context.
“In general, LLM based approaches excel when the response is more abstract and does not quote directly from the context.”
Quote in the paperNot a substantial claim
Interpretationreasoning
LLM-based grounding checks can use the model's world knowledge and reasoning to make entailment judgments.
“This allows leveraging the LLM's superior world knowledge and reasoning abilities in making entailment judgements.”
Quote in the paperNot a substantial claim
Interpretationreasoning
Domain-specific tuning is particularly helpful for niche domains with novel terms, as is common in medical and healthcare.
“This is particularly helpful when the knowledge base falls in a niche domain and/or involves novel terms not in the model's vocabulary; this is commonly the case for medical and healthcare domains.”
Quote in the paperNot a substantial claim
Premisecitation
Commercial generative AI search engines have been observed to quote verbatim from provided context.
“For instance, they may quote verbatim from the provided context, which was recently observed for a number of commercial generative AI search engines [67].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [67]
  Not checked in the free run.
Interpretationcitation
Frontier AI systems are expected to face a broader risk profile as LLMs gain abilities such as multimodal input, tool use, planning, memory, and inter-system interaction.
“More broadly, the risk profile associated with frontier AI systems is expected to expand in light of extensions of existing LLMs, e.g., multimodality, tool use, deeper reasoning and planning, larger and more capable memory, and increased interaction between AI systems [102,111].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [102]
  Not checked in the free run.
- cites [111]
  Not checked in the free run.
Interpretationcitation
Tuning LLMs to generate grounded responses may cause them to lose creativity and helpfulness.
“A key challenge in tuning LLMs towards generating grounded responses is that the models may optimize for grounding at the expense of losing creativity and helpfulness.”
Quote in the paperSubstantial claim
Interpretationcitation
Domain-specific tuning is expected to benefit both closed-book question answering and RAG.
“Such domain-specific tuning is expected to benefit both closed book question-answering [97] as well as RAG approaches [44].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [97]
  Not checked in the free run.
- cites [44]
  Not checked in the free run.

Classified as probably not a substantial claim. 22
May combine two claims. 11

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation
In addition to response quality, practitioners must consider training data privacy, model stealing, copyright violations, and security risks like jailbreaking and prompt injection.
“Besides evaluations of response quality, practitioners also have to worry about training data privacy, model stealing, copyright violations, and security risks such as jailbreaking [137] and prompt injection [121].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [137]
  Not checked in the free run.
- cites [121]
  Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone given
Large language models and other generative AI models introduce extra difficulties, including hallucinations, harmful and manipulative content, and copyright infringement.
“We focus on large language models (LLMs) and other generative AI models, which present additional challenges such as hallucinations, harmful and manipulative content, and copyright infringement.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
In large language models, a lack of truthfulness can span from subtle misrepresentations to blatant false statements, commonly called hallucinations.
“For instance, lack of truthfulness can range from subtle misrepresentations to making blatant false statements (colloquially known as “hallucinations”) [48].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [48]
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Reducing hallucinations on a synthetic task has been explored as a means to reduce hallucinations on real-world downstream tasks.
“Reducing hallucination on a synthetic task has been explored as a way to reduce hallucination on real-world downstream tasks [57].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [57]
  Not checked in the free run.
May combine two claims.
Premisecitation
Extensive research exists on hallucinations in large language models, covering their causes and sources, and methods to evaluate how prone models are to producing them.
“There is extensive work on hallucinations in LLMs [48,54], including, the causes and sources of hallucinations [77], and measures for evaluating LLMs based on their vulnerability to producing hallucinations [94].”
Quote in the paperNot a substantial claim0 of 4 sources checked
- cites [48]
  Not checked in the free run.
- cites [54] source
  Not checked in the free run.
- cites [77] source
  Not checked in the free run.
- cites [94] source
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Several dedicated frameworks exist that describe evaluation datasets, metrics, and benchmarks for various dimensions of LLM evaluation.
“Evaluation of LLMs is an important topic and there have been a number of dedicated frameworks [33,66,83] describing evaluation datasets, metrics, and benchmarks for various dimensions.”
Quote in the paperNot a substantial claim0 of 3 sources checked
- cites [33]
  Not checked in the free run.
- cites [66]
  Not checked in the free run.
- cites [83]
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
A growing set of tools and resources has been proposed for different phases of developing large language models.
“A growing collection of tools and resources have been proposed across different phases of LLM development [71].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [71]
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Early work exists on detecting and preventing hallucinations in large vision-language models and other multimodal foundation models.
“There is also early work on detecting and preventing hallucinations in large vision language models [41] and other multimodal foundation models [128].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [41] source
  Not checked in the free run.
- cites [128]
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Another approach fine-tunes large language models on preference data for factuality, using response pairs ranked by factuality.
“Another approach involves fine-tuning LLMs on preference data for factuality, i.e., response pairs ranked by factuality [109].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [109]
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation
Alignment research is an active area, with approaches including data-efficient alignment, alternatives to RLHF, and aligning cross-modal representations.
“Overall, alignment is an active area of research, with approaches ranging from data-efficient alignment [55] to alternatives to RLHF [25] to aligning cross-modal representations [84].”
Quote in the paperNot a substantial claim0 of 3 sources checked
- cites [55]
  Not checked in the free run.
- cites [25] source
  Not checked in the free run.
- cites [84]
  Not checked in the free run.
May combine two claims.
Interpretationreasoning
In some settings, the evaluation task may be too difficult even for human experts, which requires using powerful LLMs in a way that aligns with human intent.
“In certain settings, the task to be evaluated could be too hard for even human experts (e.g., comparing two different summaries of a very large collection of documents or judging the quality of hypotheses generated based on a large volume of medical literature), necessitating the use of powerful LLMs in a manner that aligns with human intent.”
Quote in the paperNot a substantial claim
May combine two claims.
Premisecitation
Current approaches to AI alignment depend on human supervision, such as reinforcement learning from human feedback, and would not be feasible once AI systems surpass human intelligence.
“While current approaches for AI alignment rely on human ability to supervise AI (using approaches such as reinforcement learning from human feedback), these approaches would not be feasible when AI systems become smarter than humans [13].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [13]
  Not checked in the free run.
May combine two claims.
Premisecitation
LLMs and other generative AI models exhibit novel bias categories, including gender stereotypes, exclusionary norms, disability-related biases, religious stereotypes, and sexual objectification.
“In addition to known categories of bias observed in predictive ML models, new types of bias arise in LLMs and other generative AI models, e.g., gender stereotypes, exclusionary norms, undesirable biases towards mentions of disability, religious stereotypes, and sexual objectification [10,30,106,122].”
Quote in the paperNot a substantial claim0 of 4 sources checked
- cites [10] source
  Not checked in the free run.
- cites [30]
  Not checked in the free run.
- cites [106]
  Not checked in the free run.
- cites [122] source
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
A large body of research addresses bias detection and mitigation in NLP models.
“There is extensive work on detecting and mitigating bias in NLP models [12,14,15,22,36,98].”
Quote in the paperNot a substantial claim0 of 6 sources checked
- cites [12]
  Not checked in the free run.
- cites [14]
  Not checked in the free run.
- cites [15]
  Not checked in the free run.
- cites [22] source
  Not checked in the free run.
- cites [36] source
  Not checked in the free run.
- cites [98]
  Not checked in the free run.
May combine two claims.
Interpretationreasoning
Because training datasets are enormous, auditing and updating them, or foreseeing which biases they contain, is hard.
“Additionally, due to the sheer size of datasets used, it is difficult to audit and update the training data or even anticipate different kinds of biases that may be present.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Methods for watermarking or otherwise identifying or detecting AI-generated content have been proposed.
“Several approaches for watermarking [28,39,62] (or otherwise identifying / detecting [81]) AI generated content have been proposed.”
Quote in the paperNot a substantial claim0 of 4 sources checked
- cites [28]
  Not checked in the free run.
- cites [39]
  Not checked in the free run.
- cites [62]
  Not checked in the free run.
- cites [81]
  Not checked in the free run.
May combine two claims.
Interpretationreasoning
Attacks that misuse combinations of models do not require aligned frontier models to produce malicious outputs and can therefore escape detection.
“As such attacks do not require the aligned frontier models to generate malicious outputs and hence can go undetected, there is a need to extend red-teaming efforts beyond single models in isolation.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Explainability methods for large language models have been extensively studied.
“Explainability methods for LLMs have been well studied [131], including techniques such as Chain-of-Thought Prompting [120] and variants.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [131] source
  Not checked in the free run.
May combine two claims.
Premisecitation
Recent approaches for large language models include selective prediction, self-evaluation and calibration, semantic uncertainty, and self-evaluation-based selective prediction.
“In the context of LLMs, recent approaches such as selective prediction, self-evaluation and calibration, semantic uncertainty, and self-evaluation-based selective prediction have been proposed [20] (see references there-in).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [20] source
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Learning to defer involves understanding when humans can effectively complement AI models.
“It also involves understanding the conditions under which humans can effectively complement AI models [24].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [24] source
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given
Detecting personally identifiable information in large language model prompts or responses may require domain-specific modifications because what counts as personally identifiable information can differ by application.
“Detecting PII in LLM prompts / responses can be done using off-the-shelf packages, but may require domain-specific modifications since what is considered as PII could vary based on the application.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Chain-of-Thought Prompting and its variants are among the explainability methods for large language models.
“Explainability methods for LLMs have been well studied [131], including techniques such as Chain-of-Thought Prompting [120] and variants.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [120]
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Causal tracing involves first finding neuron activations that are decisive for a model's factual predictions and then altering those activations to update specific factual associations.
“This involves first identifying neuron activations that are decisive in the model's factual predictions, and then modifying these neuron activations to update specific factual associations [78].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [78]
  Not checked in the free run.
Classified as probably not a substantial claim.
Premisecitation
Training with demonstrations of out-of-domain, adversarial, or nonsensical prompts enables a model to generate an "I don't know" response.
“In such cases, it is crucial to train the model to generate an “I don't know” response by including demonstrations of such scenarios in the tuning set [29,130].”
Quote in the paperSubstantial claim0 of 2 sources checked
- cites [29]
  Not checked in the free run.
- cites [130] source
  Not checked in the free run.
- Rests on an assumption: For any retrieval system, some out-of-domain, adversarial, or nonsensical prompts will still retrieve irrelevant context.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
Mechanistic interpretability is an active research area.
“Mechanistic interpretability [93] is another active area of research, which has the potential to be further accelerated by the availability of small language models like phi-2.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [93] source
  Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
RAG systems are evaluated along context relevance, answer faithfulness, and answer relevance.
“Consequently, RAG systems are evaluated based on dimensions such as context relevance (that is, whether the retrieved context is relevant to the given prompt), answer faithfulness (that is, whether the response generated by the LLM is properly grounded in the retrieved context), and answer relevance (that is, whether the response is relevant to the user question) [27,100].”
Quote in the paperNot a substantial claim0 of 2 sources checked
- cites [27]
  Not checked in the free run.
- cites [100] source
  Not checked in the free run.
- Rests on an assumption: The success of a RAG system depends on both the retrieval step and the generation step.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationnone given
Comprehensive evaluations are necessary for trust in LLM applications to be achievable.
“Without comprehensive evaluations, establishing trust in LLM-based applications becomes exceedingly difficult.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
A new harm category-"Disparaging, Existential, and Argumentative"-was added by Microsoft to its responsible AI evaluation for conversational AI after the Bing chatbot behaved unexpectedly.
“As an example, Microsoft introduced a new category of harms called “Disparaging, Existential, and Argumentative” harms as part of the responsible AI evaluation for conversational AI applications in response to the unexpected behavior of the Bing AI chatbot as reported by a New York Times journalist [99].”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites [99]
  Not checked in the free run.
May combine two claims.
Interpretationnone given
LLM-based grounding checks are more computationally expensive and therefore less viable as inference-time guardrails.
“However, LLM based approaches are more computationally expensive making them less viable as inference time guardrails.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given
A common caveat for constrained decoding is balancing groundedness with coherence, fluency, and helpfulness.
“A common caveat for constrained decoding approaches is balancing groundedness with other desirable attributes like coherence, fluency, and helpfulness.”
Quote in the paperNot a substantial claim

30 of 30

The paper relies on this without support.
Harmful responses from large language models can vary from racially biased responses to violent, hateful, and other inappropriate responses, to responses causing social harm.
“Similarly, harmful responses can vary from racially biased responses, to violent, hateful, and other inappropriate responses, to responses causing social harm (e.g., instruction on how to cheat in an examination without getting caught).”
The paper relies on this without support.
A frequent cause of grounding failures is that the LLM's parametric memory does not contain all knowledge-base information.
“Grounding failures often occur because not all information in the knowledge base is stored in the LLM's parametric memory.”
The paper relies on this without support.
For any retrieval system, some out-of-domain, adversarial, or nonsensical prompts will still retrieve irrelevant context.
“no matter how effective the retrieval system is, there will always be instances of out-of-domain, adversarial, or nonsensical prompts where the retrieved context remains irrelevant.”
The paper relies on this without support.
The success of a RAG system depends on both the retrieval step and the generation step.
“The success of a RAG system relies on the success of the retrieval step and the generation step.”
The paper relies on this without support.
Metrics like weight updates, activations, and robustness statistics are not currently available as part of LLM APIs.
“leveraging metrics like weight updates, activations, and robustness statistics, which are not currently available as part of LLM APIs.”

5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks.
“The aforementioned factors require a multidisciplinary approach to LLM benchmarking, integrating insights from linguistics, ethics, and social sciences to create robust evaluation frameworks [3], [13], [30].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [30]

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Usenix Security Symposium
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038 · see citing works
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
(2021) · doi:10.18653/v1/2021.naacl-main · Cited by 1,723 · see citing works
Extracting Training Data from Diffusion ModelsPreprint
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramèr, F., Balle, B., Ippolito, D., & Wallace, E. (2023) · USENIX Security Symposium · doi:10.48550/arxiv.2301.13188 · Cited by 1,112 · see citing works
Word embeddings quantify 100 years of gender and ethnic stereotypes
Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1720347115 · Cited by 1,065 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
Self-citation
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM CollaborationPreprint
Feng, S., Shi, W., Wang, Y., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.00367 · Cited by 250 · see citing works
Retrieval-Augmented Generation for AI-Generated Content: A Survey
Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., Jiang, J., & Cui, B. (2024) · arXiv · doi:10.48550/arxiv.2402.19473 · Cited by 240 · see citing works
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230 · see citing works
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
RQ-RAG: Learning to Refine Queries for Retrieval Augmented GenerationPreprint
Chan, C.-M., Xu, C., Yuan, R., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2404.00610 · Cited by 216 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · see citing works
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161 · see citing works
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12813 · Cited by 156 · see citing works
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.20 · Cited by 148 · see citing works
A General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146 · see citing works
Evaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143 · cited 2× · see citing works
“This enables transparency and allows the end-user to corroborate all claims in the response.”
Sources of Hallucination by Large Language Models on Inference Tasks
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.182 · Cited by 142 · see citing works
Textbooks Are All You Need
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11644 · Cited by 138 · see citing works
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.40 · Cited by 130 · see citing works
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., & Manning, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.330 · Cited by 123 · see citing works
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Räuker, T., Ho, A., Casper, S., & Hadfield-Menell, D. (2023) · 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00039 · Cited by 122 · see citing works
The Troubling Emergence of Hallucination in Large Language Models - An Extensive Definition, Quantification, and Prescriptive Remediations
Rawte, V., Chakraborty, S., Pathak, A., Sarkar, A., Tonmoy, S. M. T. I., Chadha, A., Sheth, A., & Das, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.155 · Cited by 119 · see citing works
Concealed Data Poisoning Attacks on NLP Models
Wallace, E., Zhao, T., Feng, S., & Singh, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.13 · Cited by 115 · see citing works
Rethinking search: making domain experts out of dilettantes
Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021) · ACM SIGIR Forum · doi:10.1145/3476415.3476428 · Cited by 112 · see citing works
Publicly Detectable Watermarking for Language ModelsPreprint
Fairoze, J., Garg, S., Jha, S., Mahloujifar, S., Mahmoody, M., & Wang, M. (2023) · IACR Cryptology ePrint Archive · doi:10.48550/arxiv.2310.18491 · Cited by 108 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal ReasoningPreprint
Panagopoulou, A., Xue, L., Ning, Y.-X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.18799 · Cited by 92 · see citing works
Measure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., & Herzig, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.444 · Cited by 86 · see citing works
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., & Herzig, J. (2024) · arXiv · doi:10.48550/arxiv.2405.05904 · Cited by 86 · see citing works
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., & Yih, W.-T. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-short.69 · Cited by 80 · see citing works
Dense X Retrieval: What Retrieval Granularity Should We Use?
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., & Yu, D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.845 · Cited by 74 · see citing works
Dense X Retrieval: What Retrieval Granularity Should We Use?
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Zhang, H., & Yu, D. (2023) · arXiv · doi:10.48550/arxiv.2312.06648 · Cited by 74 · see citing works
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.813 · Cited by 73 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71 · see citing works
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66 · cited 1× · see citing works
“This is an emerging area, and we refer interested readers to a recent survey by Zhao et al. [133].”
Evaluating Large Language Models: A Comprehensive SurveyPreprint
Guo, Z., Jin, R., Chuang, L. I. U., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19736 · Cited by 64 · see citing works
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · cited 1× · see citing works
“Additionally, can we develop more eﬃcient and practical causal intervention approaches?”
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.05566 · Cited by 46 · see citing works
Towards Unbiased and Accurate Deferral to Multiple Experts
Keswani, V., Lease, M., & Kenthapadi, K. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462516 · Cited by 45 · see citing works
Self-citation
Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias
Wolfe, R., Yang, Y., Howe, B., & Caliskan, A. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594072 · Cited by 42 · see citing works
Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration
Feng, S., Shi, W., Wang, Y., Ding, W., Balachandran, V., & Tsvetkov, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.786 · Cited by 40 · see citing works
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.09390 · Cited by 34 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2021) · arXiv · doi:10.48550/arxiv.2112.12870 · Cited by 34 · see citing works
Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness
Donahue, K., Chouldechova, A., & Kenthapadi, K. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533221 · Cited by 33 · see citing works
Self-citation
Dialect prejudice predicts AI decisions about people's character, employability, and criminality
Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024) · arXiv · doi:10.48550/arxiv.2403.00742 · Cited by 33 · see citing works
The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models
Qi, J., Xu, Z., Shen, Y., Liu, M., Jin, D., Wang, Q., & Huang, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.255 · Cited by 31 · see citing works
CGKPN: Cross-Graph Knowledge Propagation Network with Adaptive Connection for Reasoning-Based Machine Reading Comprehension
Zhao, Z., Zhou, G., Xie, Z., Wu, L., & Huang, J. X. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3658673 · Cited by 29 · see citing works
R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’
Zhang, H., Diao, S., Lin, Y., Fung, Y., Lian, Q., Wang, X., Chen, Y., Ji, H., & Zhang, T. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.394 · Cited by 28 · see citing works
Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
Bhatt, M., Chennabasappa, S., Nikolaidis, C., Wan, S., Evtimov, I., Gabi, D., Song, D., Ahmad, F., Aschermann, C., Fontana, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.04724 · Cited by 27 · see citing works
Benchmarking Differentially Private Synthetic Data Generation AlgorithmsPreprint
Tao, Y., McKenna, R., Hay, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.09238 · Cited by 24 · see citing works
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & ResourcesPreprint
Longpre, S., Biderman, S., Albalak, A., et al. (2024) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2406.16746 · Cited by 21 · see citing works
Measuring Implicit Bias in Explicitly Unbiased Large Language Models
Bai, X., Wang, A., Sucholutsky, I., & Griffiths, T. L. (2024) · arXiv · doi:10.48550/arxiv.2402.04105 · Cited by 17 · see citing works
Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
(2023) · doi:10.18653/v1/2023.eacl-demo · Cited by 16 · see citing works
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Dong, Y., Wang, Z., Sreedhar, M., Wu, X., & Kuchaiev, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.754 · Cited by 15 · see citing works
Watermarking Pre-trained Language Models with Backdooring
Gu, C., Huang, C., Zheng, X., Chang, K.-W., & Hsieh, C.-J. (2022) · arXiv · doi:10.48550/arxiv.2210.07543 · Cited by 15 · see citing works
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., & Jha, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.345 · Cited by 13 · see citing works
AI model disgorgement: Methods and choices
Achille, A., Kearns, M. J., Klingenberg, C., & Soatto, S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2307304121 · Cited by 13 · see citing works
PromptBench: A Unified Library for Evaluation of Large Language ModelsPreprint
Zhu, K., Zhao, Q., Chen, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07910 · Cited by 13 · see citing works
Iterative Methods for Private Synthetic Data: Unifying Framework and New MethodsPreprint
Liu, T., Vietri, G., & Wu, Z. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07153 · Cited by 11 · see citing works
Evaluating and Mitigating Discrimination in Language Model Decisions
Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., & Ganguli, D. (2023) · arXiv · doi:10.48550/arxiv.2312.03689 · Cited by 11 · cited 2× · see citing works
“A key challenge in tuning LLMs towards generating grounded responses is that the models may optimize for grounding at the expense of losing creativity and helpfulness.”
Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. C. (2024) · arXiv · doi:10.48550/arxiv.2403.14403 · Cited by 11 · see citing works
Learning to Filter Context for Retrieval-Augmented Generation
Wang, Z., Araki, J., Jiang, Z., Parvez, M. R., & Neubig, G. (2023) · arXiv · doi:10.48550/arxiv.2311.08377 · Cited by 9 · see citing works
What's in a Name? Auditing Large Language Models for Race and Gender BiasPreprint
Salinas, A., Haim, A., & Nyarko, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.14875 · Cited by 8 · see citing works
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2023) · arXiv · Cited by 7 · see citing works
Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural NetworksPreprint
Lohaus, M., Kleindeßner, M., Kenthapadi, K., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.04440 · Cited by 4 · see citing works
Self-citation
ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencePreprint
Wu, K., Wu, E., & Zou, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.10198 · Cited by 4 · see citing works
Rethinking Machine Unlearning for Large Language ModelsPreprint
Liu, S., Yao, Y., Jia, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.08787 · Cited by 4 · see citing works
Controlled Decoding from Language ModelsPreprint
Mudgal, S., Lee, J., Ganapathy, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17022 · Cited by 2 · see citing works
A Framework for Automated Measurement of Responsible AI Harms in Generative AI ApplicationsPreprint
Magooda, A., Helyar, A., Jackson, K. L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.17750 · Cited by 2 · see citing works
Large Language Models are Biased Because They Are Large Language ModelsPreprint
Resnik, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.13138 · Cited by 2 · see citing works
Adversaries Can Misuse Combinations of Safe Models
Jones, E., Dragan, A., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2406.14595 · Cited by 1
Hallucination Augmented Recitations for Language ModelsPreprint
Köksal, A., Aksitov, R., & Chang, C.-C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07424 · Cited by 1 · see citing works
Personas as a Way to Model Truthfulness in Language ModelsPreprint
Joshi, N., Rando, J., Saparov, A., Kim, N., & He, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.18168
Measuring Distributional Shifts in Text: The Advantage of Language Model-Based EmbeddingsPreprint
Gupta, G., Rastegarpanah, B., Iyer, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.02337
PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-RailsPreprint
Mangaokar, N., Hooda, A., Choi, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.15911

Flagged first, then by citations

Research works printed but not linked: 49

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations.
Sergul Aydore, William Brown, Michael Kearns, Krishnaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva. 2021. Differentially private query release through adaptive projection. In International Conference on Machine Learning. PMLR, 457--467.
Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. 2023. Identifying and mitigating the security risks of generative AI. Foundations and Trends® in Privacy and Security 6, 1 (2023), 1--52.
Tolga Bolukbasi Kai-Wei Chang James Y Zou Venkatesh Saligrama and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In NeurIPS.
Aylin Caliskan. 2021. Detecting and mitigating bias in natural language processing. Brookings Institution (2021).
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017).
Matthijs Douze Alexandr Guzhva Chengqi Deng Jeff Johnson Gergely Szilvasy Pierre-Emmanuel Mazaré Maria Lomeli Lucas Hosseini and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:cs.LG/2401.08281
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2023. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770 (2023). · arXiv
10.18653/v1 · DOI
10.18653/v1 · DOI
10.18653/v1 · DOI
10.18653/v1 · DOI
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674 (2023). · arXiv
10.18653/v1 · DOI
Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, and Mahdi Namazifar. 2023. Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
Erik Jones, Hamid Palangi, Clarisse Simões Ribeiro, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Hassan Awadallah, and Ece Kamar. 2024. Teaching Language Models to Hallucinate Less with Synthetic Tasks. In The Twelfth International Conference on Learning Representations (ICLR).
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning The Difference That Makes A Difference With Counterfactually-Augmented Data. In International Conference on Learning Representations. https://openreview. net/forum?id=Sklgs0NFvr
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning. PMLR, 17061--17084.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis,Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 9459--9474. https://proceedings.neurips. cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
Hongyin Luo, Tianhua Zhang, Yung-Sung Chuang, Yuan Gong, Yoon Kim, Xixin Wu, Helen M. Meng, and James R. Glass. 2023. Search Augmented Instruction Learning. In The 2023 Conference on Empirical Methods in Natural Language Processing. https://openreview.net/forum?id=noIvPGG8P1
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, SeanWelleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. https://arxiv.org/abs/2303.17651. arXiv:cs.CL/2303.17651 · arXiv
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning. PMLR, 24950--24962.
Arshaan Nazir, Thadaka Kalyan Chakravarthy, David Amore Cecchini, Rakshit Khajuria, Prikshit Sharma, Ali Tarik Mirik, Veysel Kocaman, and David Talby. 2024. LangTest: A comprehensive evaluation library for custom LLM and NLP models. Software Impacts (2024), 100619.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson. 2018. Scalable Private Learning with PATE. In International Conference on Learning Representations.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. In The Twelfth International Conference on Learning Representations.
10.18653/v1 · DOI
Alexey Romanov, Maria De-Arteaga, HannaWallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Kalai. 2019. What's in a Name? Reducing Bias in Bios without Access to Protected Attributes. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4187--4195.
Kevin Roose. 2023. A Conversation With Bings Chatbot Left Me Deeply Unsettled. New York Times (2023).
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect?. In International Conference on Machine Learning. PMLR, 29971--30004.
Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. 2023. On Early Detection of Hallucinations in Factual Question Answering. arXiv preprint arXiv:2312.14183 (2023). · arXiv
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 3008--3021. https://proceedings.neurips.cc/ paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf
Yi Tay Vinh Q. Tran Mostafa Dehghani Jianmo Ni Dara Bahri Harsh Mehta Zhen Qin Kai Hui Zhe Zhao Jai Gupta Tal Schuster William W. Cohen and Donald Metzler. 2022. Transformer Memory as a Differentiable Search Index. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.). https://openreview.net/forum? id=Vu-B0clPfq
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. Fine-Tuning Language Models for Factuality. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum? id=WPZ2yPag4K
10.18653/v1 · DOI
Helen Toner, Jessica Ji, John Bansemer, Lucy Lim, Chris Painter, Courtney Corley, Jess Whittlestone, Matt Botvinick, Mikel Rodriguez, and Ram Shankar Siva Kumar. 2023. Skating to Where the Puck is Going: Anticipating and Managing Risks from Frontier AI Systems. Report from the July 2023 Roundtable hosted by the Center for Security and Emerging Technology (CSET) at Georgetown University and Google DeepMind.
Lifu Tu Semih Yavuz Jin Qu Jiacheng Xu Rui Meng Caiming Xiong and Yingbo Zhou. 2024. Unlocking Anticipatory Text Generation: A Constrained Approach for Faithful Decoding with Large Language Models. https://openreview.net/ forum?id=774elYc5tw
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don't always say what they think: Unfaithful explanations in chainof- thought prompting. Advances in Neural Information Processing Systems 36 (2023).
Giorgos Vernikos, Arthur Brazinskas, Jakub Adamek, Jonathan Mallinson, Aliaksei Severyn, and Eric Malmi. 2024. Small Language Models Improve Giants by Rewriting Their Outputs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Yvette Graham and Matthew Purver (Eds.). Association for Computational Linguistics, St. Julian's, Malta, 2703--2718. https://aclanthology.org/2024.eacllong. 165
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail?. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=jA235JGM09
Simon Willison. 2022. Prompt injection attacks against GPT-3. Simon Willison's Weblog (2022).
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. 2023. Data Selection for Language Models via Importance Resampling. In Thirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/ forum?id=uPSQv0leAu
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Retrieval meets Long Context Large Language Models. In The Twelfth International Conference on Learning Representations. https://openreview. net/forum?id=xw5nxFWMlo
Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (LLM) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045 (2023). · arXiv
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than Retrieve: Large Language Models are Strong Context Generators. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=fB0hRu9GZUS
10.18653/v1 · DOI
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment. In Thirty-seventh Conference on Neural Information Processing Systems. https: //openreview.net/forum?id=KBMOKmX2he
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107 (2023). · arXiv
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). · arXiv

Sorted by citations

Industrial applications of large language models
Raza, M., Jahangir, Z., Riaz, M. B., Saeed, M. J., & Sattar, M. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-98483-1 · Cited by 220 · see citing works
“• Holistic evaluation of language model (helm) : HELM was introduced in 2021 by a team of researchers from the University of California, Berkeley, who wanted to provide a more holistic and actionable evaluation of LLMs in real-world applications 113 .”
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“LLMs can inadvertently learn and propagate biases present in training data, requiring benchmarks that can detect and mitigate such issues [32].”
Background
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Tan, Z., Yan, J., Hsu, I.-H., Han, R., Wang, Z., Le, L., Song, Y., Chen, Y., Palangi, H., Lee, G., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.413 · Cited by 16 · see citing works
Tres escenarios para la IA en educación: del apoyo responsable a la cocreación
García‐Peñalvo, F. J. (2025) · Education in the Knowledge Society (EKS) · doi:10.14201/eks.32932 · Cited by 16 · see citing works
Evaluating the quality of GenAI applications in software engineering: a multi-case study
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10759-2 · Cited by 7 · see citing works
“These domains were identified through engagement with industrial engineers and are also highlighted in recent literature on GenAI adoption in industry, such as Esposito et al. (2025) and Kenthapadi et al. (2024).”
Influential
Are Teachers Assessing Work Written by Students or by AI ? A Rapid Literature Review of Research on Detecting Content Generated by Generative AI
Han, J., Yang, Y., & Liu, G. (2025) · European Journal of Education · doi:10.1111/ejed.70240 · Cited by 8 · see citing works
A novel large language model enhanced joint learning framework for fine-grained sentiment analysis on drug reviews
Zou, H., & Wang, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129589 · Cited by 5 · see citing works
Benchmarking Large Language Models for Drug Combination Alerts: Achieving Expert-Level Reliability via Knowledge Grounding and Contextual Reasoning
Hu, H., Wang, L., Chen, L., Lin, H., Huang, L.-T., Wang, Z.-W., & Hou, X. (2026) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.5c03511 · Cited by 5 · see citing works
Artificial intelligence in academia: opportunities, challenges, and ethical considerations
Molligan, J., & Pérez‐López, E. (2024) · Biochemistry and Cell Biology · doi:10.1139/bcb-2024-0216 · Cited by 5 · see citing works
Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis
Brokman, J., Hofman, O., Rachmil, O., Singh, I., Pahuja, V., Sabapathy, R., Priya, A., Giloni, A., Vainshtein, R., & Kojima, H. (2025) · doi:10.1109/raie66699.2025.00005 · Cited by 8 · see citing works
“Kenthapadi et. al., [36] reviewed LLM evaluation metrics focused on aspects of responsible AI such as robustness, bias and security.”
Axioms for Retrieval-Augmented Generation
Reimer, J. H., Fröbe, M., Stein, B., Potthast, M., & Hagen, M. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744601 · Cited by 2 · see citing works
Using LLMs to Enhance Democracy
Lazar, S., & Manuali, L. (2026) · Minds and Machines · doi:10.1007/s11023-026-09767-y · Cited by 12 · see citing works
“Whereas the other examples in this category involve LLMs facilitating judgment aggregation, this approach uses the LLM to represent individual agents’ inputs into the final decision procedure.”
Can LLMs Talk “Sex”? Exploring How AI Models Handle Intimate Conversations
Lai, H. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1326 · Cited by 1 · see citing works
“Additionally, concerns about bias in LLMs — especially those related to gender and culture — are well-documented (Kenthapadi et al., 2024).”
Bridging Context, Statistics, and Practice: A Multi-Dimensional Framework for Responsible LLM Evaluation and Selection
Kumar, N. D., Ramavath, S. K., & Venkatesh, V. (2025) · International Generative AI and Computational Language Modelling Conference (GACLM) · doi:10.1109/gaclm67198.2025.11232293 · Cited by 2 · see citing works
Revisiting data analysis with Pre-trained foundation models
Liang, C., Yang, D., Liang, Z., Liang, Z., Zhang, T., Xiao, B., Yang, Y., Wang, W., & Wang, H. (2026) · The VLDB Journal · doi:10.1007/s00778-025-00953-5 · Cited by 1 · see citing works
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 3 · see citing works
Precision Grounding: Augmenting Large Language Models with Evidence-Based Databases for Trustworthy Genetic Variant SummarizationPreprint
Du, X., Nagy, A. A., Oates, M., Wang, Y., Wang, X., Plasek, J. M., Aronson, S., Lebo, M. S., & Zhou, L. N. (2025) · medRxiv · doi:10.1101/2025.06.09.25329279 · Cited by 1 · see citing works
Precision Grounding: Augmenting Large Language Models with Evidence-Based Databases for Trustworthy Genetic Variant SummarizationPreprint
Du, X., Nagy, A. A., Oates, M., Wang, Y., Wang, X., Plasek, J. M., Aronson, S., Lebo, M. S., & Zhou, L. N. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5385545 · Cited by 2 · see citing works
Sustainable Innovation in Healthcare
Upadhyaya, A., Mishra, M. K., Nandal, S., Whig, A., & Whig, P. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-3952-8.ch012
Assessing the Maturity of Generative AI Systems: A Framework for Education and Public Engagement
Petrillo, A., Rehman, M., & Awasare, K. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.12.018
Unifying Knowledge in Agentic LLMs: Concepts, Methods, and Recent Advancements
Liu, L., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3787470.3787478
“Finally, ensuring reliable grounding is non-trivial: models may ignore retrieved evidence or selectively use it in ways that do not guarantee factual correctness [18].”
Natural-Language Mediation Versus Numerical Aggregation in Multi-Stakeholder AI Governance: Capability Boundaries and Architectural Requirements
Uchoa, A. P., Oliveira, C., Motta, C. L. R., & Schneider, D. (2026) · Computers · doi:10.3390/computers15010024 · Cited by 1 · see citing works
IRSDA: An Agent-Orchestrated Framework for Enterprise Intrusion Response
Panigrahi, D., Patel, R., Mitra, S., Mittal, S., & Rahimi, S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402393 · Cited by 1 · see citing works
“IRSLLM is grounded in high quality enterprise knowledge to reduce hallucinations and to improve factual fidelity in line with guidance on LLM grounding [38].”
LLM-Based Triple Extraction with Class-Constrained Structural Prompting for Phase-Aware Knowledge Graphs in Aviation Incident Analysis
Xin, S., Li, H., Yang, Z., Zhou, Y., Dong, L., & Chen, C. L. P. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423063
“An LLM is used to evaluate extracted triples along three dimensions (factual accuracy, structural coherence, and groundedness) [23].”
Saying More Than They Know: A Framework for Quantifying Epistemic-Rhetorical Miscalibration in Large Language ModelsPreprint
Bakhshi, A. D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6669973
An Instrument for Human-Based Evaluation of GenAI Educational Alignment
Ahmed, H., López‐Pernas, S., Vogelsmeier, L. V. D. E., Tukiainen, M., & Saqr, M. (2026) · Technology Knowledge and Learning · doi:10.1007/s10758-026-10005-8
LLM-Based Agentic AI, Social Contract, and Cognitive Disabilities
McNair, D. (2026) · Springer eBooks · doi:10.1007/978-3-032-11375-7_525
As It Was: Aligning LLM Search Evaluation with Historical User Preferences
Vardasbi, A., Penha, G., Palumbo, E., Hauff, C., Bouchard, H., & Lalmas, M. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808488
Production LLM Deployment: A Practitioner's Field Guide to Patterns and Pitfalls
Mandadi, A. R. (2024) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v3i11.1567
Beyond silos: Anchoring trustworthy AI in cross-domain knowledge graphs for rural health resilience
Li, S., Guo, X., Juarez, F. G., et al. (2026) · Smart Health · doi:10.1016/j.smhl.2026.100681
A Multi-Agent Collaborative and Controllable Reasoning Approach for Ship Building Process
Li, N., Yang, G., Cai, Y., Chen, C., Wang, T., & Cheng, L. (2026) · Youth Academic Annual Conference of Chinese Association of Automation (YAC) · doi:10.1109/yac71005.2026.11616153
“…conﬁdent conclusions without citing veriﬁable source text, making review and accountability difﬁcult [2], [3]; (2) Long documents and noise sensitivity : technical manuals are lengthy and include substantial irrelevant content, which may cause unstable retrieval or off-topic generation [4], [5].”
Grounding Techniques in LLM-Based Recommender Systems: A Systematic Literature Mapping
Pino, A. F. S., Méndez, N. D. D., Ruiz, P. H., Mon, A., & Collazos, C. A. (2026) · Applied Sciences · doi:10.3390/app16189335
In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents
Tan, Z., Yan, J., Hsu, I.-H., Han, R., Wang, Z., Le, L. T., Song, Y., Chen, Y., Palangi, H., Lee, G., et al. (2025) · arXiv
“Given the user query with context in the current session, and the retrieved memories, we prompt the LLM (prompt in Appendix D.2) to generate both the response and the associated citations to each individual memory in the context (Kenthapadi et al., 2024).”

Authors: Kenthapadi, K., Sameki, M., & Taly, A.
Affiliations: Oracle Health AI, Redwood City, CA, USA
Microsoft (United States) (US)
Google (United States) (US)
ORCID: Kenthapadi, Krishnaram
Sameki, Mehrnoosh
Taly, Ankur
Venue: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Publisher: ACM
Published: 2024-08-24
Pages: 6523-6533
DOI: 10.1145/3637528.3671467
OpenAlex: W4401863475
Semantic Scholar: 509d4c6778baf9d42abd86e56be69488804ddeda
Cited by: 36
References: 47
Citations by year: 2026: 13 · 2025: 21 · 2024: 2
FWCI: 16.36
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Social Sciences › Safety Research › Ethics and Social Impacts of AI
Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords: large language models, generative AI, AI safety, AI evaluation, hallucinations, harmful content, copyright infringement, bias, robustness, interpretability, trustworthy AI, AI monitoring

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)

Abstract

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks