## 4.2 Examples and applications

### 4.2.1 Case study: *The Challenger disaster*

The *Challenger* disaster occurred on the 28th January of 1986, when the NASA Space Shuttle orbiter *Challenger* broke apart and disintegrated at 73 seconds into its flight, leading to the deaths of its seven crew members. The accident deeply shocked the US society, in part due to the attention the mission had received because of the presence of Christa McAuliffe, who would have been the first astronaut-teacher. Because of this, NASA TV broadcasted live the launch to US public schools, which resulted in millions of school children witnessing the accident. The accident had serious consequences for the NASA credibility and resulted in an interruption of 32 months in the shuttle program. The Presidential *Rogers Commission* (formed by astronaut Neil A. Armstrong and Nobel laureate Richard P. Feynman, among others) was created to investigate the disaster.

The Rogers Commission elaborated a report (Presidential Commission on the Space Shuttle Challenger Accident 1986) with all the findings. The commission determined that the disintegration began with the **failure of an O-ring seal in the solid rocket motor due to the unusual cold temperatures (-0.6 Celsius degrees)** during the launch. This failure produced a breach of burning gas through the solid rocket motor that compromised the whole shuttle structure, resulting in its disintegration due to the extreme aerodynamic forces. The **problematic with O-rings was something known**: the night before the launch, there was a three-hour teleconference between motor engineers and NASA management, discussing the effect of low temperature forecasted for the launch on the O-ring performance. The conclusion, influenced by Figure 4.2a, was:

“Temperature data [are] not conclusive on predicting primary O-ring blowby.”

The Rogers Commission noted a major flaw in Figure 4.2a: the **flights with zero incidents were excluded** from the plot because *it was felt* that **these flights did not contribute any information about the temperature effect** (Figure 4.2b). The Rogers Commission concluded:

“A careful analysis of the flight history of O-ring performance would have revealed the correlation of O-ring damage in low temperature”.

The purpose of this case study, inspired by Dalal, Fowlkes, and Hoadley (1989), is to quantify what was the influence of the temperature in the probability of having at least one incident related with the O-rings. Specifically, we want to address the following questions:

- Q1.
*Is the temperature associated with O-ring incidents?* - Q2.
*In which way was the temperature affecting the probability of O-ring incidents?* - Q3.
*What was the predicted probability of an incidient in an O-ring for the temperature of the launch day?*

To try to answer these questions we have the `challenger`

dataset (download). The dataset contains (shown in Table 4.1) information regarding the state of the solid rocket boosters after launch^{27} for 23 flights. Each row has, among others, the following variables:

`fail.field`

,`fail.nozzle`

: binary variables indicating whether there was an incident with the O-rings in the field joints or in the nozzles of the solid rocket boosters.`1`

codifies an incident and`0`

its absence. On the analysis, we focus on the O-rings of the field joint as being the most determinants for the accident.`temp`

: temperature in the day of launch. Measured in Celsius degrees.`pres.field`

,`pres.nozzle`

: leak-check pressure tests of the O-rings. These tests assured that the rings would seal the joint.

flight | date | fail.field | fail.nozzle | temp |
---|---|---|---|---|

1 | 12/04/81 | 0 | 0 | 18.9 |

2 | 12/11/81 | 1 | 0 | 21.1 |

3 | 22/03/82 | 0 | 0 | 20.6 |

5 | 11/11/82 | 0 | 0 | 20.0 |

6 | 04/04/83 | 0 | 1 | 19.4 |

7 | 18/06/83 | 0 | 0 | 22.2 |

8 | 30/08/83 | 0 | 0 | 22.8 |

9 | 28/11/83 | 0 | 0 | 21.1 |

41-B | 03/02/84 | 1 | 1 | 13.9 |

41-C | 06/04/84 | 1 | 1 | 17.2 |

41-D | 30/08/84 | 1 | 1 | 21.1 |

41-G | 05/10/84 | 0 | 0 | 25.6 |

51-A | 08/11/84 | 0 | 0 | 19.4 |

51-C | 24/01/85 | 1 | 1 | 11.7 |

51-D | 12/04/85 | 0 | 1 | 19.4 |

51-B | 29/04/85 | 0 | 1 | 23.9 |

51-G | 17/06/85 | 0 | 1 | 21.1 |

51-F | 29/07/85 | 0 | 0 | 27.2 |

51-I | 27/08/85 | 0 | 0 | 24.4 |

51-J | 03/10/85 | 0 | 0 | 26.1 |

61-A | 30/10/85 | 1 | 0 | 23.9 |

61-B | 26/11/85 | 0 | 1 | 24.4 |

61-C | 12/01/86 | 1 | 1 | 14.4 |

Let’s begin the analysis by replicating Figures 4.2a and 4.2b and checking that linear regression is not the right tool for answering Q1–Q3. For that, we make two scatterplots of `nfails.field`

(number of total incidents in the field joints) versus `temp`

, the first one excluding the launches without incidents (`subset = nfails.field > 0`

) and the second one for all the data. Doing it through `R Commander`

as we saw in Chapter 2, you should get something similar to:

```
scatterplot(nfails.field ~ temp, reg.line = lm, smooth = FALSE, spread = FALSE,
boxplots = FALSE, data = challenger, subset = nfails.field > 0)
```

```
scatterplot(nfails.field ~ temp, reg.line = lm, smooth = FALSE, spread = FALSE,
boxplots = FALSE, data = challenger)
```

There is a fundamental problem in using linear regression for this data: **the response is not continuous**. As a consequence, there is no linearity and the errors around the mean are not normal (indeed, they are strongly non normal). Let’s check this with the corresponding diagnostic plots:

```
mod <- lm(nfails.field ~ temp, data = challenger)
par(mfrow = 1:2)
plot(mod, 1)
plot(mod, 2)
```

Albeit linear regression is not the adequate tool for this data, it is able to detect the obvious difference between the two plots:

**The trend for launches with incidents is flat, hence suggesting there is no dependence on the temperature**(Figure 4.2a). This was one of the arguments behind NASA’s decision of launching the rocket at a temperature of -0.6 degrees.- However,
**the trend for**(Figure 4.2b). Think about it in this way: the minimum temperature for a launch without incidents ever recorded was above 18 degrees, and the Challenger was launched at -0.6 without clearly knowing the effects of such low temperatures.*all*launches indicates a clear negative dependence between temperature and number of incidents!

*probability of expecting at least one incident given the temperature*, a simpler but also revealing approach. In other words, we look to estimate the following curve: \[ p(x)=\mathbb{P}(\text{incident}=1|\text{temperature}=x) \] from

`fail.field`

and `temp`

. This probability can not be properly modeled as a linear function like \(\beta_0+\beta_1x\), since inevitably will fall outside \([0,1]\) for some value of \(x\) (some will have negative probabilities or probabilities larger than one). The technique that solves this problem is the **logistic regression**. The idea behind is quite simple: transform a linear model \(\beta_0+\beta_1x\) – which is aimed for a response in \(\mathbb{R}\) – so that it yields a value in \([0,1]\). This is achieved by the

*logistic function*\[\begin{align} \text{logistic}(t)=\frac{e^t}{1+e^t}=\frac{1}{1+e^{-t}}.\tag{4.1} \end{align}\]

The logistic model in this case is \[ \mathbb{P}(\text{incident}=1|\text{temperature}=x)=\text{logistic}\left(\beta_0+\beta_1x\right)=\frac{1}{1+e^{-(\beta_0+\beta_1x)}}, \] with \(\beta_0\) and \(\beta_1\) unknown. Let’s fit the model to the data by estimating \(\beta_0\) and \(\beta_1\).

In order to fit a logistic regression to the data, go to `'Statistics' -> 'Fit models' -> 'Generalized linear model...'`

. A window like Figure 4.3 will pop-up, which you should fill as indicated.

A code like this will be generated:

```
nasa <- glm(fail.field ~ temp, family = "binomial", data = challenger)
summary(nasa)
##
## Call:
## glm(formula = fail.field ~ temp, family = "binomial", data = challenger)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.0566 -0.7575 -0.3818 0.4571 2.2195
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 7.5837 3.9146 1.937 0.0527 .
## temp -0.4166 0.1940 -2.147 0.0318 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 28.267 on 22 degrees of freedom
## Residual deviance: 20.335 on 21 degrees of freedom
## AIC: 24.335
##
## Number of Fisher Scoring iterations: 5
exp(coef(nasa)) # Exponentiated coefficients ("odds ratios")
## (Intercept) temp
## 1965.9743592 0.6592539
```

The summary of the logistic model is notably different from the linear regression, as the methodology behind is quite different. Nevertheless, we have tests for the significance of each coefficient. Here we obtain that `temp`

is significantly different from zero, at least at a level \(\alpha=0.05\). Therefore we can conclude that **the temperature is indeed affecting the probability of an incident with the O-rings** (answers Q1).

The precise interpretation of the coefficients will be given in the next section. For now, the coefficient of `temp`

, \(\hat\beta_1\), can be regarded the “correlation between the temperature and the probability of having at least one incident”. These correlation, as evidenced by the sign of \(\hat\beta_1\), is negative. Let’s plot the fitted logistic curve to see that indeed the probability of incident and temperature are negatively correlated:

```
# Plot data
plot(challenger$temp, challenger$fail.field, xlim = c(-1, 30), xlab = "Temperature",
ylab = "Incident probability")
# Draw the fitted logistic curve
x <- seq(-1, 30, l = 200)
y <- exp(-(nasa$coefficients[1] + nasa$coefficients[2] * x))
y <- 1 / (1 + y)
lines(x, y, col = 2, lwd = 2)
# The Challenger
points(-0.6, 1, pch = 16)
text(-0.6, 1, labels = "Challenger", pos = 4)
```

At the sight of this curve and the summary of the model we can conclude that **the temperature was increasing the probability of an O-ring incident** (Q2). Indeed, the confidence intervals for the coefficients show a significative negative correlation at level \(\alpha=0.05\):

```
confint(nasa, level = 0.95)
## 2.5 % 97.5 %
## (Intercept) 1.3364047 17.7834329
## temp -0.9237721 -0.1089953
```

Finally, **the probability of having at least one incident with the O-rings in the launch day was \(0.9996\) according to the fitted logistic model** (Q3). This is easily obtained:

```
predict(nasa, newdata = data.frame(temp = -0.6), type = "response")
## 1
## 0.999604
```

Be aware that `type = "response"`

has a different meaning in logistic regression. As you can see it does not return a CI for the prediction as in linear models. Instead, `type = "response"`

means that the *probability* should be returned, instead of the value of the link function, which is returned with `type = "link"`

(the default).

Recall that there is a serious problem of **extrapolation** in the prediction, which makes it less precise (or more variable). But this extrapolation, together with the evidences raised by a simple analysis like we did, should have been strong arguments for postponing the launch.

### References

Presidential Commission on the Space Shuttle Challenger Accident. 1986. *Report of the Presidential Commission on the Space Shuttle Challenger Accident (Vols. 1 & 2)*. Washington, DC. http://history.nasa.gov/rogersrep/genindex.htm.

Dalal, Siddhartha R., Edward B. Fowlkes, and Bruce Hoadley. 1989. “Risk Analysis of the Space Shuttle: Pre-Challenger Prediction of Failure.” *Journal of the American Statistical Association* 84 (408): 945–57. doi:10.1080/01621459.1989.10478858.

After the shuttle exits the atmosphere, the solid rocket boosters separate and descend to land using a parachute where they are carefully analysed.↩