Developing a Framework for Auditing Large Language Models using Human-in-the-loop

Abstract

As LLMs become more widely adopted, detecting inconsistencies like bias and hallucination is increasingly important. Auditing LLMs for these inconsistencies is crucial but often challenging. An effective method for auditing an LLM involves using variations of the same question, referred to as probes, where consistent responses to these probes are expected. Deviations in the responses can indicate flaws in the model's knowledge representation or operational behavior. However, producing high-quality probes at scale remains challenging, primarily because it requires human experts to ensure the reliability of the probes. Prior work has relied on human experts to manually verify each individual probe, making the process expensive, resource-intensive, and prone to subjectivity. o address these limitations, we introduce LLMAuditor, a framework that uses a human-in-the-loop (HIL) validated prompt template to guide an LLM in generating probes. This approach eliminates the need for exhaustive human verification of every probe while maintaining high standards of quality and reliability. LLMAuditor operates in two phases: first, a helper LLM generates probes using a HIL-validated prompt template; second, these probes are used to audit the target LLM. This dual-LLM approach ensures verifiability, avoids circular reliance on a single model, and enhances the rigor and generalizability of the auditing process. Case studies on different LLMs show LLMAuditor reliably identifies inconsistencies. The framework's novelty lies in its use of a HIL-validated prompt template for probe generation, which enhances both the transparency and effectiveness of LLM evaluation.