Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?

Abstract

Large Language Models (LLMs) have significantly advanced Software Engineering (SE) tasks, with prompt engineering techniques enhancing their performance in code-related areas. However, the rapid development of foundational LLMs such as the non-reasoning models (GPT-4o and Claude 3.5 Sonnet) and the reasoning model o1 raises questions about the continued effectiveness of these prompt engineering techniques. This article presents an extensive empirical study that re-evaluates various prompt engineering techniques within the context of these advanced LLMs. Focusing on five representative SE tasks, i.e., code generation, code translation, program repair, code summarization, and commit message generation, we assess whether prompt engineering techniques still yield improvements with advanced models, the actual effectiveness of reasoning models compared to non-reasoning models, and whether the benefits of using these advanced models justify their increased costs. Our findings reveal that some novel prompt engineering techniques developed for earlier LLMs may provide diminished benefits or even hinder performance when applied to advanced models. In reasoning LLMs, the ability of sophisticated built-in reasoning reduces the impact of complex prompts, sometimes making simple zero-shot prompting more effective in some specific tasks. Prompt strategies that utilize execution feedback or precise task-specific guidance remain effective and are essential for improving performance on complex code-related problems. Furthermore, while reasoning models outperform non-reasoning models in tasks requiring complex reasoning, they offer minimal advantages in tasks that do not need reasoning and may incur unnecessary costs. Based on our study, we provide practical guidance for practitioners on selecting appropriate prompt engineering techniques and foundational LLMs, considering factors such as task requirements, operational costs, and environmental impact. Our work contributes to a deeper understanding of effectively harnessing advanced LLMs in SE tasks, informing future research and application development.