5.1.1 Fitting a line to data

Figure 5.1 shows two variables whose relationship can be modeled perfectly with a straight line. The equation for the line is \(y = 5 + 64.96 x.\) Consider what a perfect linear relationship means: we know the exact value of \(y\) just by knowing the value of \(x.\) A perfect linear relationship is unrealistic in almost any natural process. For example, if we took family income (\(x\)), this value would provide some useful information about how much financial support a college may offer a prospective student (\(y\)). However, the prediction would be far from perfect, since other factors play a role in financial support beyond a family’s finances.

Linear regression is the statistical method for fitting a line to data where the relationship between two variables, \(x\) and \(y,\) can be modeled by a straight line with some error:

\[ y = b_0 + b_1 \ x + e\]

The values \(b_0\) and \(b_1\) represent the model’s intercept and slope, respectively, and the error is represented by \(e\). These values are calculated based on the data, i.e., they are sample statistics. If the observed data is a random sample from a target population that we are interested in making inferences about, these values are considered to be point estimates for the population parameters \(\beta_0\) and \(\beta_1\).

When we use \(x\) to predict \(y,\) we call \(x\) the explanatory variable and we call \(y\) the response variable. We also often drop the \(e\) term when writing down the model since our main focus is often on the prediction of the average response.

It is rare for all of the data to fall perfectly on a straight line. Instead, it’s more common for data to appear as a cloud of points, such as those examples shown in Figure 5.2. In each case, the data fall around a straight line, even if none of the observations fall exactly on the line. The first plot shows a relatively strong downward linear trend, where the remaining variability in the data around the line is minor relative to the strength of the relationship between \(x\) and \(y.\) The second plot shows an upward trend that, while evident, is not as strong as the first. The last plot shows a very weak downward trend in the data, so slight we can hardly notice it. In each of these examples, we will have some uncertainty regarding our estimates of the model parameters, \(\beta_0\) and \(\beta_1.\) For instance, we might wonder, should we move the line up or down a little, or should we tilt it more or less? As we move forward in this chapter, we will learn about criteria for line-fitting, and we will also learn about the uncertainty associated with estimates of model parameters.

There are also cases where fitting a straight line to the data, even if there is a clear relationship between the variables, is not helpful. One such case is shown in Figure 5.3 where there is a very clear relationship between the variables even though the trend is not linear. We discuss nonlinear trends in this chapter and the next, but details of fitting nonlinear models are saved for a later course.

5.1.2 Using linear regression to predict possum head lengths

Brushtail possums are marsupials that live in Australia, and a photo of one is shown in Figure 5.4. Researchers captured 104 of these animals and took body measurements before releasing the animals back into the wild. We consider two of these measurements: the total length of each possum, from head to tail, and the length of each possum’s head.

Figure 5.5 shows a scatterplot for the head length (mm) and total length (cm) of the possums. Each point represents a single possum from the data. The head and total length variables are associated: possums with an above average total length also tend to have above average head lengths. While the relationship is not perfectly linear, it could be helpful to partially explain the connection between these variables with a straight line.

We want to describe the relationship between the head length and total length variables in the possum dataset using a line. In this example, we will use the total length as the explanatory variable, \(x,\) to predict a possum’s head length, \(y.\) We could fit the linear relationship by eye, as in Figure 5.6.

The equation for this line is

\[\hat{y} = 41 + 0.59x\]

A “hat” on \(y\) is used to signify that this is an estimate. We can use this line to discuss properties of possums. For instance, the equation predicts a possum with a total length of 80 cm will have a head length of

\[\hat{y} = 41 + 0.59 \times 80 = 88.2\]

The estimate may be viewed as an average: the equation predicts that possums with a total length of 80 cm will have an average head length of 88.2 mm. Absent further information about an 80 cm possum, the prediction for head length that uses the average is a reasonable estimate.

There may be other variables that could help us predict the head length of a possum besides its length. Perhaps the relationship would be a little different for male possums than female possums, or perhaps it would differ for possums from one region of Australia versus another region. Plot A in Figure 5.7 shows the relationship between total length and head length of brushtail possums, taking into consideration their sex. Male possums (represented by blue triangles) seem to be larger in terms of total length and head length than female possums (represented by red circles). Plot B in Figure 5.7 shows the same relationship, taking into consideration their age. It’s harder to tell if age changes the relationship between total length and head length for these possums.

5.1.3 Residuals

Residuals are the leftover variation in the data after accounting for the model fit:

\[\text{Data} = \text{Fit} + \text{Residual}\]

Each observation will have a residual, and three of the residuals for the linear model we fit for the possum data are shown in Figure 5.8. If an observation is above the regression line, then its residual, the vertical distance from the observation to the line, is positive. Observations below the line have negative residuals. One goal in picking the right linear model is for these residuals to be as small as possible.

Figure 5.8 is almost a replica of Figure 5.6, with three points from the data highlighted. The observation marked by a red circle has a small, negative residual of about -1; the observation marked by a gray diamond has a large positive residual of about +7; and the observation marked by a pink triangle has a moderate negative residual of about -4. The size of a residual is usually discussed in terms of its absolute value. For example, the residual for the observation marked by a pink triangle is larger than that of the observation marked by a red circle because \(|-4|\) is larger than \(|-1|.\)

Residuals are helpful in evaluating how well a linear model fits a dataset. We often display them in a scatterplot such as the one shown in Figure 5.9 for the regression line in Figure 5.8. The residuals are plotted with their predicted response variable value as the horizontal coordinate, and the vertical coordinate as the residual. For instance, the point \((85.0, 98.6)\) (marked by the blue diamond) had a predicted value of 91.4 mm and had a residual of 7.45 mm, so in the residual plot it is placed at \((91.4, 7.45).\) Creating a residual plot is sort of like tipping the scatterplot over so the regression line is horizontal, as indicated by the dashed line.

5.1.4 Describing linear relationships with correlation

We’ve seen plots with strong linear relationships and others with very weak linear relationships. It would be useful if we could quantify the strength of these linear relationships with a statistic.

We can compute the correlation using a formula, just as we did with the sample mean and standard deviation. The formula for correlation, however, is rather complex⁷², and like with other statistics, we generally perform the calculations on a computer or calculator.

Figure 5.11 shows eight plots and their corresponding correlations. Only when the relationship is perfectly linear is the correlation either -1 or 1. If the relationship is strong and positive, the correlation will be near +1. If it is strong and negative, it will be near -1. If there is no apparent linear relationship between the variables, then the correlation will be near zero.

The correlation is intended to quantify the strength of a linear trend. Nonlinear trends, even when strong, sometimes produce correlations that do not reflect the strength of the relationship; see three such examples in Figure 5.12.

One important aspect of the correlation is that it’s unitless. That is, unlike a measurement of the slope of a line (see the next section) which provides an increase in the y-coordinate for a one unit increase in the x-coordinate (in units of the x and y variable), there are no units associated with the correlation of x and y. Figure 5.13 shows the relationship between weights and heights of 507 physically active individuals. In Plot A, weight is measured in kilograms (kg) and height in centimeters (cm). In Plot B, weight has been converted to pounds (lbs) and height to inches (in). The correlation coefficient (\(r = 0.72\)) is also noted on both plots. We can see that the shape of the relationship has not changed, and neither has the correlation coefficient. The only visual change to the plot is the axis labeling of the points.

5.2.1 Gift aid for freshman at Elmhurst College

This section considers a dataset on family income and gift aid data from a random sample of fifty students in the freshman class of Elmhurst College in Illinois. Gift aid is financial aid that does not need to be paid back, as opposed to a loan. A scatterplot of these data is shown in Figure 5.14 along with a linear fit. The line follows a negative trend in the data; students who have higher family incomes tended to have lower gift aid from the university.

5.2.2 An objective measure for finding the best line

We begin by thinking about what we mean by the “best” line. Mathematically, we want a line that has small residuals. But beyond the mathematical reasons, hopefully it also makes sense intuitively that whatever line we fit, the residuals should be small (i.e., the points should be close to the line). The first option that may come to mind is to minimize the sum of the absolute values of the residuals:

\[|e_1| + |e_2| + \dots + |e_n|\]

which we could accomplish with a computer program. The resulting dashed line shown in Figure 5.15 demonstrates this fit can be quite reasonable.

However, a more common practice is to choose the line that minimizes the sum of the squared residuals:

\[e_{1}^2 + e_{2}^2 + \dots + e_{n}^2\]

The line that minimizes this least squares criterion is represented as the solid line in Figure 5.15 and is commonly called the least squares line. The following are three possible reasons to choose the least squares option instead of trying to minimize the sum of residual magnitudes without any squaring:

1. It is the most commonly used method.
2. Computing the least squares line is widely supported in statistical software.
3. In many applications, a residual twice as large as another residual is more than twice as bad. For example, being off by 4 is usually more than twice as bad as being off by 2. Squaring the residuals accounts for this discrepancy.
4. The analyses which link the model to inference about a population are most straightforward when the line is fit through least squares.

The first two reasons are largely for tradition and convenience; the third and fourth reasons explain why the least squares criterion is typically most helpful when working with real data.⁷⁵

5.2.3 Finding and interpreting the least squares line

For the Elmhurst data, we could write the equation of the least squares regression line as \[\widehat{\texttt{aid}} = \beta_0 + \beta_{1}\times \texttt{family_income}\]

Here the equation is set up to predict gift aid based on a student’s family income, which would be useful to students considering Elmhurst. These two values, \(\beta_0\) and \(\beta_1,\) are the parameters of the regression line.

The parameters are estimated using the observed data. In practice, this estimation is done using a computer in the same way that other estimates, like a sample mean, can be estimated using a computer or calculator.

The dataset where these data are stored is called elmhurst. The first 5 rows of this dataset are given in Table 5.1.

We can see that family income is recorded in a variable called family_income and gift aid from university is recorded in a variable called gift_aid. For now, we won’t worry about the price_paid variable. We should also note that these data are from the 2011-2012 academic year, and all monetary amounts are given in $1,000s, i.e., the family income of the first student in the data shown in Table 5.1 is $92,920 and they received a gift aid of $21,700. (The data source states that all numbers have been rounded to the nearest whole dollar.)

Statistical software is usually used to compute the least squares line and the typical output generated as a result of fitting regression models looks like the one shown in Table 5.2. For now we will focus on the first column of the output, which lists \({b}_0\) and \({b}_1.\) The remaining columns give us information on how accurate and precise these values of intercept and slope that are calculated from a sample of 50 students are in estimating the population parameters of intercept and slope for all students.

The model output tells us that the intercept is approximately 24.319 and the slope on family_income is approximately -0.043.

But what do these values mean? Interpreting parameters in a regression model is often one of the most important steps in the analysis.

An alternative way of calculating the values of intercept and slope of a least squares line is manual calculations using formulas. While manual calculations are not commonly used by practicing statisticians and data scientists, it is useful to work through the first time you’re learning about the least squares line and modeling in general. Calculating the values by hand leverages two properties of the least squares line:

1. The slope of the least squares line can be estimated by

\[b_1 = \frac{s_y}{s_x} r \]

where \(r\) is the correlation between the two variables, and \(s_x\) and \(s_y\) are the sample standard deviations of the explanatory and response variables, respectively.

2. If \(\bar{x}\) is the sample mean of the explanatory variable and \(\bar{y}\) is the sample mean of the response variable, then the point \((\bar{x}, \bar{y})\) falls on the least squares line.

Table 5.3 shows the sample means for the family income and gift aid as $101,780 and $19,940, respectively. We could plot the point \((102, 19.9)\) on Figure 5.14 to verify it falls on the least squares line (the solid line).

Next, we formally find the point estimates \(b_0\) and \(b_1\) of the parameters \(\beta_0\) and \(\beta_1.\)

You might recall the form of a line from math class, which we can use to find the model fit, including the estimate of \(b_0.\) Given the slope of a line and a point on the line, \((x_0, y_0),\) the equation for the line can be written as

\[y - y_0 = slope\times (x - x_0)\]

5.2.4 Extrapolation is treacherous

When those blizzards hit the East Coast this winter, it proved to my satisfaction that global warming was a fraud. That snow was freezing cold. But in an alarming trend, temperatures this spring have risen. Consider this: On February 6 it was 10 degrees. Today it hit almost 80. At this rate, by August it will be 220 degrees. So clearly folks the climate debate rages on.⁷⁶

Stephen Colbert April 6th, 2010

Linear models can be used to approximate the relationship between two variables. However, like any model, they have real limitations. Linear regression is simply a modeling framework. The truth is almost always much more complex than a simple line. For example, we do not know how the data outside of our limited window will behave.

Applying a model estimate to values outside of the realm of the original data is called extrapolation. Generally, a linear model is only an approximation of the real relationship between two variables. If we extrapolate, we are making an unreliable bet that the approximate linear relationship will be valid in places where it has not been analyzed.

5.2.5 Describing the strength of a fit

We evaluated the strength of the linear relationship between two variables earlier using the correlation, \(r.\) However, it is more common to explain the strength of a linear fit using \(R^2,\) called R-squared. If provided with a linear model, we might like to describe how closely the data cluster around the linear fit.

The \(R^2\) of a linear model describes the amount of variation in the response variable that is explained by the least squares line. For example, consider the Elmhurst data, shown in Figure 5.14. The variance of the response variable, aid received, is about \(s_{aid}^2 \approx 29.8\) million (calculated from the data, some of which is shown in Table 5.1). However, if we apply our least squares line, then this model reduces our uncertainty in predicting aid using a student’s family income. The variability in the residuals describes how much variation remains after using the model: \(s_{_{RES}}^2 \approx 22.4\) million. In short, there was a reduction of \[\frac{s_{aid}^2 - s_{_{RES}}^2}{s_{aid}^2} = \frac{29800 - 22400}{29800} = \frac{7500}{29800} \approx 0.25,\] or about 25%, of the response variable’s variation by using information about family income for predicting aid using a linear model. It turns out that \(R^2\) corresponds exactly to the squared value of the correlation:

\[r = -0.499 \rightarrow R^2 = 0.25\]

\(R^2\) is also called the coefficient of determination.

More generally, \(R^2\) can be calculated as a ratio of a measure of variability around the line divided by a measure of total variability.

5.2.6 Categorical explanatory variables with two levels

Categorical variables are also useful in predicting the average of a response variable. Here we consider a categorical explanatory variable with two levels (recall that a level is the same as a category). We’ll consider Ebay auctions for a video game, Mario Kart for the Nintendo Wii, where both the total price of the auction and the condition of the game were recorded. Here we want to predict total price based on game condition, which takes values used and new.

A plot of the auction data is shown in Figure 5.16. Note that the original dataset contains some Mario Kart games being sold at prices above $100 but for this analysis we have limited our focus to the 141 Mario Kart games that were sold below $100.

To incorporate the game condition variable into a regression equation, we must convert the categories into a numerical form. We will do so using an indicator variable called condnew, which takes value 1 when the game is new and 0 when the game is used. Using this indicator variable, the linear model may be written as

\[\widehat{\texttt{price}} = b_0 + b_1 \times \texttt{condnew}\]

The parameter estimates are given in Table 5.4.

Using values from Table 5.4, the model equation can be summarized as

\[\widehat{\texttt{price}} = 42.87 + 10.9 \times \texttt{condnew}\]

Note that, fundamentally, the intercept and slope interpretations don’t change when modeling categorical variables with two levels. However, when the explanatory variable is binary, the coefficient estimates (\(b_0\) and \(b_1\)) are directly interpretable with respect to the dataset at hand.

5.4.1 Summary

Throughout this chapter, the nuances of the linear model have been described. You have learned how to create a linear model with explanatory variables that are numerical (e.g., total possum length) and those that are categorical (e.g., whether or not a video game was new). The residuals in a linear model are an important metric used to understand how well a model fits; high leverage points, influential points, and other types of outliers can impact the fit of a model. Correlation is a measure of the strength and direction of the linear relationship of two variables, without specifying which variable is the explanatory and which is the response.

5.4.2 Terms

We introduced the following terms in the chapter. If you’re not sure what some of these terms mean, we recommend you go back in the text and review their definitions. We are purposefully presenting them in alphabetical order, instead of in order of appearance, so they will be a little more challenging to locate. However you should be able to easily spot them as bolded text.