Agentic AI Meets Data Engineering: Toward Self-Directed, Interpretable, and Balanced Pipelines

Abstract

The Agentic AI framework represents a significant advancement in data engineering by unifying automation, interpretability, and adaptability within a single intelligent system. Unlike traditional approaches, which struggle to autonomously adjust to dynamic data environments or provide transparent reasoning, Agentic AI integrates four synergistic agents: an AutoML agent for model selection and tuning, a neuro-symbolic agent for interpretable inference, a generative agent leveraging GANs for rare event synthesis, and an agentic planner that dynamically orchestrates decisions using reinforcement learning. Experimental evaluation on diverse datasets including credit card fraud detection, breast cancer diagnosis, and industrial sensor failure demonstrated the framework’s superior performance, achieving an F1-score of 0.91, 94% rule fidelity, and a reduced adaptation time of 85 seconds. These results significantly surpass baseline AutoML, standalone neuro-symbolic systems, and GAN-based models. The generative component improved minority class representation, while the neuro-symbolic engine provided rule-based explanations closely aligned with the model’s predictions. The agentic planner enabled real-time model drift detection and automatic retraining, ensuring continuous optimization. Collectively, this framework offers a powerful, self-improving pipeline capable of transforming static data processes into adaptive, goal-directed, and explainable workflows suitable for real-world, high-stakes applications.