Reducing Hallucinations in Legal AI: A Retrieval Augmented Generation-Based Model for Accurate Legal Guidance

Abstract

The expansion of generative Artificial Intelligence (AI), particularly Large Language Models (LLMs), has significantly transformed various industries, including law, by automating complex tasks and enhancing overall productivity. However, these models are susceptible to generating fabricated or inaccurate information, commonly referred to as hallucinations. This phenomenon poses substantial risks in fields like legal practice, where precision and accuracy are of utmost importance. To mitigate these challenges, a Retrieval-Augmented Generation (RAG) model integrated with LLMs is proposed. This approach grounds AI-generated responses in verified, authoritative sources, such as legal practice guides and rulebooks, by utilizing a customized vectorized database of legal documents. By doing so, the RAG model effectively minimizes the likelihood of errors in legal advice and case references, thereby improving the accuracy, reliability, and ethical application of AI in legal decision-making processes. This is particularly crucial in high-stakes environments where the cost of misinformation can be significant. In this study, a total of 17 recent Indian legal documents sourced from the official Indian government legislation website were employed to create a comprehensive vector database using an advanced embedding model. Six combinations of core and embedding models were rigorously tested and evaluated to determine their effectiveness. Notably, the combination of Llama2 as the core model and All-mpnet-base-v2 as the embedding model demonstrated superior performance, achieving a BERTScore F1 of 56.87%. This finding underscores the potential of RAG models to enhance the reliability of AI-driven legal assistance, ensuring that users receive contextually relevant and trustworthy legal information.