Using LLMs for Market Research

Abstract

Large language models (LLMs) have rapidly gained popularity as labor-augmenting tools for programming, writing, and many other processes that benefit from quick text generation. In this paper we explore the uses and limitations of LLMs for researchers and practitioners who aim to understand consumer preferences. We focus on the distributional nature of LLM responses, and query the different LLMs to generate dozens of responses to each survey question. We offer two sets of results to illustrate and assess our approach. First, we show that estimates of willingness-to-pay for products and features derived from LLM responses are sometimes comparable to estimates from human studies, but are often inaccurate and in some cases wrong-signed. Second, we demonstrate a practical method for market researchers to enhance LLMs' responses by incorporating previous survey data from similar contexts via fine-tuning. This method improves alignment with human responses for existing and, importantly, new product features \emph{within} the same product category and population. We do not find similar improvements in the alignment for new product categories or for differences between customer segments. Taken together, our results suggest that the appropriate use case for LLMs in market research is as a supplement to---not a substitute for---human studies, with fine-tuning providing the most reliable gains when prior human survey data from the relevant category and population are already in hand.