Generalized Additive Models

Abstract

Regression models play an important role in many applied settings, by enabling predictive analysis, revealing classification rules, and providing data-analytic tools for understanding the interactions between different variables. Although attractively simple, traditional linear regression often fails in practical situations because real-life effects are usually nonlinear. This article describes statistical methods that may be used to identify and characterize general nonlinear regressions, without requiring the analyst to prespecify the form of the nonlinear relationship. These methods form the basis of the generalized additive models approach to data analysis. The choice of the link function g permits the analyst to incorporate many of the benefits of the methodology of generalized linear models. The logit link is widely viewed as highly appropriate for the analysis of binary data, and other standard link functions are similarly useful. Besides the identity link g(μ) = μ, which yields the additive model corresponding to ordinary multiple regression, people often use g(μ) = lnμ to give a log-additive model for Poisson count data (See Poisson Distribution). These link functions arise from exponential family sampling models, which also include the gamma and negative‐binomial distributions. These families generate the well-known class of generalized linear models 12, which may all be extended in the same way to generalized additive models by replacing linear terms with nonparametric functions. In this section we describe a modular algorithm for fitting additive models and their generalizations. The basic component is the scatterplot smoother for fitting nonlinear curves in a way that permits a great variety of functional forms. Suppose that we have a scatterplot of points (x i , y i ) like that shown in Fig. 1. Here y is a response or outcome variable, and x is a predictor. We wish to fit a smooth curve f(x) that summarizes the dependence of y on x (see Curve Fitting). More formally, we want to fit the model y = f(x) + ε, where f(x) is specified in a flexible way, yet required to be reasonably smooth. If we were to find the curve that simply minimized ∑[y i − f(x i )]2, the result would be an interpolating curve that would not be smooth at all. Notice that ∫f″(x)2 dx measures the “wiggliness” of the function f: linear functions f have ∫f″(x)2 dx = 0, while nonlinear functions produce values larger than zero. Here λ is a nonnegative smoothing parameter that must be chosen by the data analyst. It governs the trade-off between the goodness of fit* to the data (as measured by ∑[y i − f(x i )]2) and the wiggliness of the function. Larger values of λ force f to be smoother. In fact the interpolating curve corresponds to λ = 0 at one extreme, and the straight line fit is the limit as λ → ∞. For any value of λ, the solution to 1 is a cubic spline, i.e., a piecewise cubic polynomial with pieces joined smoothly at the values of x in the data set. Fast and stable numerical procedures are available for computation of the fitted curve. The right panel of Fig. 1 shows a cubic-spline fit to the data. What value of λ did we use in Fig. 1 ? In fact it is not convenient to express the desired smoothness of f in terms of λ, since the meaning of λ depends on the units of the explanatory variable x. Instead, it is possible to define an “effective number of parameters” or “degrees of freedom” for a cubic spline smoother, and then use a numerical search to determine the value of λ that yields this number. In Fig. 1 we chose the degrees of freedom to be five. Roughly speaking, this means that the complexity of the curve is about the same as that of a polynomial regression of degree four. However, the cubic spline smoother “spreads out” its parameters in a more even manner, and hence allows more diverse curves than a polynomial regression. Note that the degrees of freedom of a smoother need not be an integer. The preceding discussion sought to minimize the least squares fit to the data, subject to a penalty on the roughness of the fitted function. For the logistic regression model and other generalized additive models, the appropriate criterion is penalized maximum likelihood or penalized maximum log partial likelihood. To maximize these, the backfitting procedure is used in conjunction with a maximum likelihood or maximum partial likelihood algorithm. The usual Newton–Raphson method for maximizing log likelihoods in these models can be cast in an iteratively reweighted least squares form. This involves repeated use of weighted linear regression of a constructed response variable on the covariates: each regression yields a new value of the parameter estimates which give a new constructed variable, and the process is iterated. In the generalized additive model, the weighted linear regression is simply replaced by a weighted backfitting algorithm. We describe the algorithm in more detail for logistic regression below, and in more genera