6.3 Why not use linear regression for a binary outcome?

Example 6.1: Figure 6.1 displays the linear regression fit for simulated data from 60 individuals aged 20 to 80 years where $Y$ = coronary heart disease (CHD) status and $X$ = age. The possible values for $Y$ are “yes” and “no” coded as $Y = 1$ and $0$, respectively. The gray dots at $Y = 1$ (along the top of the figure) are plotted at the ages of individuals with CHD, while the dots at $Y = 0$ (along the bottom of the figure) are for those without CHD. The solid line is the linear regression line, which attempts to estimate the proportion of CHD = Yes (the mean outcome) as a function of age using a straight line. The dashed line is a smoother which tracks the observed proportion without any constraint on its shape.

Figure 6.1: Linear regression does not fit well for a binary outcome

A straight line is not a good fit to this data in the sense that most of the points are very far from the line. However, it turns out to not be a terrible fit when comparing it to the relationship between the proportion of CHD = Yes and age (the dashed line). However, in general, logistic regression does even better at estimating the proportion.

Logistic regression models the mean (the probability of $Y = 1$) using the complicated looking logit function $\ln{(p/(1-p))}$ on the left-hand side of the equation. Why not have $Y$ or $p$ on the left-hand side? One reason is that, ideally, the range of possible values on the left-hand side should match that of the right-hand side. The right-hand side of the model equation is $\beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_K X_K$ and can take on any value from $-\infty$ to $\infty$, whereas $Y$ can only be 0 or 1 and $p$ must be in the range 0 to 1. However, the logit of $p$, $\ln{(p/(1-p))}$, can take on any value, matching the range of the right-hand side.

When transformed back to the probability scale, logistic regression fits an S-shaped curve that estimates the proportion of 1s at a given value of the predictor. Figure 6.2 demonstrates how the logistic regression fit (solid line) closely tracks the smoother (dashed line) and the predicted probabilities are all in [0, 1]. At the highest observed ages, however, the height of the estimated linear regression line (Figure 6.1) is greater than 1, resulting in predictions outside of [0, 1].

Figure 6.2: Logistic regression fits an S-shaped curve modeling P(Y = 1)