Chapter 6 Logistic Regression
6.1 Logistic Regression
6.1.1 Modeling Binary Response
So far, we have modeled only quantitative response variables.
The normal error regression model makes the assumption that the response variable is normally distributed, given the value(s) of the explanatory variables.
Now, we’ll look at how to model a categorical response variable. We’ll consider only situations where the response is binary (i.e. has 2 categories)
6.1.2 Credit Card Dataset
We’ll consider a dataset pertaining to 10,000 credit cards. The goal is to predict whether or not the user will default on the payment, using information on the credit card balance, user’s annual income, and whether or not the user is a student. Data come from Introduction to Statistical Learning by James, Witten, Hastie, Tibshirani.
library(ISLR)
data(Default)
summary(Default)
## default student balance income
## No :9667 No :7056 Min. : 0.0 Min. : 772
## Yes: 333 Yes:2944 1st Qu.: 481.7 1st Qu.:21340
## Median : 823.6 Median :34553
## Mean : 835.4 Mean :33517
## 3rd Qu.:1166.3 3rd Qu.:43808
## Max. :2654.3 Max. :73554
6.1.4 Linear Regression Model for Education Level
#convert default from yes/no to 0/1
$default <- as.numeric(Default$default=="Yes") Default
ggplot(data=Default, aes(y=default, x= balance)) + geom_point(alpha=0.2) + stat_smooth(method="lm", se=FALSE)
There are a lot of problems with this model!
6.1.5 Transforming into interval (0,1)
Starting with our linear model E(Yi)=β0+β1xi1, we need to transform β0+β1xi1 into (0,1).
Let πi=eβ0+β1xi11+eβ0+β1xi1.
Then 0≤πi≤1, and πi represents an estimate of P(Yi=1).
This function maps the values of β0+β1xi1 into the interval (0,1).
The logistic regression model assumes that:
- Yi∈{0,1}
- E(Yi)=P(Yi=1)=πi=eβ0+β1xi1+…βpxip1+eβ0+β1xi1+…βpxip
- Yi∈{0,1}
i.e. β0+β1xi1+…βpxip=log(πi1−πi). (This is called the logit function and can be written logit(πi).
- Instead of assuming that the expected response is a linear function of the explanatory variables, we are assuming that it is a function of a linear function of the explanatory variables.
6.1.6 Logistic Regression Model for Balance
ggplot(data=Default, aes(y=default, x= balance)) + geom_point(alpha=0.2) +
stat_smooth(method="glm", se=FALSE, method.args = list(family=binomial))
6.1.7 Fitting the Logistic Regression Model in R
<- glm(data=Default, default ~ balance, family = binomial(link = "logit"))
CCDefault_M summary(CCDefault_M)
##
## Call:
## glm(formula = default ~ balance, family = binomial(link = "logit"),
## data = Default)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.2697 -0.1465 -0.0589 -0.0221 3.7589
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -10.6513306 0.3611574 -29.49 <0.0000000000000002 ***
## balance 0.0054989 0.0002204 24.95 <0.0000000000000002 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 2920.6 on 9999 degrees of freedom
## Residual deviance: 1596.5 on 9998 degrees of freedom
## AIC: 1600.5
##
## Number of Fisher Scoring iterations: 8
6.1.8 The Logistic Regression Equation
The regression equation is:
P(Default)=ˆπi=e−10.65+0.0055×balance1+e−10.65+0.0055×balance
For a $1,000 balance, the estimated default probability is e−10.65+0.0055(1000)1+e−10.65+0.0055(1000)≈0.006
For a $1,500 balance, the estimated default probability is e−10.65+0.0055(1500)1+e−10.65+0.0055(1500)≈0.08
For a $2,000 balance, the estimated default probability is e−10.65+0.0055(2000)1+e−10.65+0.0055(2000)≈0.59
6.1.9 Predict in R
predict(CCDefault_M, newdata=data.frame((balance=1000)), type="response")
## 1
## 0.005752145
predict(CCDefault_M, newdata=data.frame((balance=1500)), type="response")
## 1
## 0.08294762
predict(CCDefault_M, newdata=data.frame((balance=2000)), type="response")
## 1
## 0.5857694
6.1.10 Where do the b’s come from?
Recall that for a quantitative response variable, the values of b1,b2,…,bp are chosen in a way that minimizes n∑i=1(yi−(β0+β1xi1+…+βpxip)2).
Least squares does not work well in this generalized setting. Instead, the b’s are calculated using a more advanced technique, known as maximum likelihood estimation.
6.2 Interpretations in a Logistic Regression Model
6.2.1 Recall Logistic Regression Curve for Credit Card Data
ggplot(data=Default, aes(y=default, x= balance)) + geom_point(alpha=0.2) +
stat_smooth(method="glm", se=FALSE, method.args = list(family=binomial))
6.2.2 Recall Credit Card Model Output
<- glm(data=Default, default ~ balance, family = binomial(link = "logit"))
M summary(M)
##
## Call:
## glm(formula = default ~ balance, family = binomial(link = "logit"),
## data = Default)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.2697 -0.1465 -0.0589 -0.0221 3.7589
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -10.6513306 0.3611574 -29.49 <0.0000000000000002 ***
## balance 0.0054989 0.0002204 24.95 <0.0000000000000002 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 2920.6 on 9999 degrees of freedom
## Residual deviance: 1596.5 on 9998 degrees of freedom
## AIC: 1600.5
##
## Number of Fisher Scoring iterations: 8
6.2.3 Balance Logistic Model Equation
The regression equation is:
P(Default)=ˆπi=e−10.65+0.0055×balance1+e−10.65+0.0055×balance
For a $1,000 balance, the estimated default probability is ˆπi=e10.65+0.0055(1000)1+e10.65+0.0055(1000)≈0.005752145
For a $1,500 balance, the estimated default probability is ˆπi=e10.65+0.0055(1500)1+e10.65+0.0055(1500)≈0.08294762
For a $2,000 balance, the estimated default probability is ˆπi=e10.65+0.0055(2000)1+e10.65+0.0055(2000)≈0.5857694
6.2.4 Odds
For an event with probability p, the odds of the event occurring are p1−p.
The odds of default are given by πi1−πi.
Examples:
The estimated odds of default for a $1,000 balance are 0.0057521451−0.005752145≈1:173.
The estimated odds of default for a $1,500 balance are 0.082947621−0.08294762≈1:11.
The estimated odds of default for a $2,000 balance are 0.58576941−0.5857694≈1.414:1.
6.2.5 Odds Ratio
The quantity πi1−πiπj1−πj represents the odds ratio of a default for user i, compared to user j. This quantity is called the odds ratio.
Example:
The default odds ratio for a $1,000 payment, compared to a $2,000 payment is
The odds ratio is 11731.4141≈1:244.
The odds of a default are about 244 times larger for a $2,000 payment than a $1,000 payment.
6.2.6 Interpretation of β1
Consider the odds ratio for a case j with explanatory variable x+1, compared to case i with explanatory variable x.
That is log(πi1−πi)=β0+β1x, and log(πj1−πj)=β0+β1(x+1).
log(πj1−πjπi1−πi)=log(πj1−πj)−log(πi1−πj)=β0+β1(x+1)−(β0+β1(x))=β1.
Thus a 1-unit increase in x is associated with a multiplicative change in the log odds by a factor of β1.
A 1-unit increase in x is associated with a multiplicative change in the odds by a factor of eβ1.
6.2.7 Intrepretation in Credit Card Example
b1=0.0055
The odds of default are estimated to multiply by e0.0055≈1.0055 for each 1-dollar increase in balance on the credit card.
That is, the odds of default are estimated to increase by 0.55% for each additional dollar on the card balance.
An increase of d dollars in credit card balance is estimated to be associated with an increase odds of default by a factor of e0.0055d.
The odds of default for a balance of $2,000 are estimated to be e0.0055×1000≈244 times as great as the odds of default for a $1,000 balance.
6.2.8 Hypothesis Tests in Logistic Regression
The p-value on the “balance” line of the regression output is associated with the null hypothesis β1=0, that is that there is no relationship between balance and the odds of defaulting on the payment.
The fact that the p-value is so small tells us that there is strong evidence of a relationship between balance and odds of default.
6.2.9 Confidence Intervals for β1
confint(M, level = 0.95)
## 2.5 % 97.5 %
## (Intercept) -11.383288936 -9.966565064
## balance 0.005078926 0.005943365
We are 95% confident that a 1 dollar increase in credit card balance is associate with an increased odds of default by a factor between e0.00508≈1.0051 and e0.00594≈1.0060.
This is a profile-likelihood interval, which you can read more about here.
6.3 Multiple Logistic Regression
6.3.1 Logistic Regression Models with Multiple Explanatory Variables
We can also perform logistic regression in situations where there are multiple explanatory variables.
6.3.2 Logistic Model with Multiple Predictors
<- glm(data=Default, default ~ balance + student, family = binomial(link = "logit"))
CCDefault_M2 summary(CCDefault_M2)
##
## Call:
## glm(formula = default ~ balance + student, family = binomial(link = "logit"),
## data = Default)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.4578 -0.1422 -0.0559 -0.0203 3.7435
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -10.7494959 0.3691914 -29.116 < 0.0000000000000002 ***
## balance 0.0057381 0.0002318 24.750 < 0.0000000000000002 ***
## studentYes -0.7148776 0.1475190 -4.846 0.00000126 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 2920.6 on 9999 degrees of freedom
## Residual deviance: 1571.7 on 9997 degrees of freedom
## AIC: 1577.7
##
## Number of Fisher Scoring iterations: 8
6.3.3 Multiple Logistic Model Illustration
ggplot(data=Default, aes(y=default, x= balance, color=student)) + geom_point(alpha=0.2) + stat_smooth(method="glm", se=FALSE, method.args = list(family=binomial))
6.3.4 Multiple Logistic Regression Interpretation
The regression equation is:
P(Default)=ˆπi=e−10.75+0.00575×balance−0.7149×Istudent1+e−10.75+0.00575×balance−0.7149×Istudent
The odds of default are estimated to multiply by e0.005738≈1.00575 for each 1 dollar increase in balance whether the user is a student or nonstudent. Thus, the estimated odds of default increase by about 0.05%.
We might be more interested in what would happen with a larger increase, say $100. The odds of default are estimated to multiply by e0.005738×100≈1.775 for each $100 increase in balance for students as well as nonstudents. Thus, the estimated odds of default increase by about 77.5%.
The odds of default for students are estimated to be e−0.7149≈0.49 as high for students as non-students, assuming balance amount is held constant.
6.3.5 Hypothesis Tests in Multiple Logistic Regression Model
There is strong evidence of a relationship between balance and odds of default, provided we are comparing students to students, or nonstudents to nonstudents.
There is evidence that students are less likely to default than nonstudents, provided the balance on the card is the same.
6.3.6 Multiple Logistic Regression Model with Interaction
<- glm(data=Default, default ~ balance * student, family = binomial(link = "logit"))
CCDefault_M_Int summary(CCDefault_M_Int)
##
## Call:
## glm(formula = default ~ balance * student, family = binomial(link = "logit"),
## data = Default)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.4839 -0.1415 -0.0553 -0.0202 3.7628
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -10.8746818 0.4639679 -23.438 <0.0000000000000002 ***
## balance 0.0058188 0.0002937 19.812 <0.0000000000000002 ***
## studentYes -0.3512310 0.8037333 -0.437 0.662
## balance:studentYes -0.0002196 0.0004781 -0.459 0.646
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 2920.6 on 9999 degrees of freedom
## Residual deviance: 1571.5 on 9996 degrees of freedom
## AIC: 1579.5
##
## Number of Fisher Scoring iterations: 8
6.3.7 Interpretations for Logistic Model with Interaction
- The regression equation is:
P(Default)=ˆπi=e−10.87+0.0058×balance−0.35×Istudent−0.0002×balance×Istudent1+e−10.87+0.0058×balance−0.35×Istudent−0.0002×balance×Istudent
- Since estimate of the interaction effect is so small and the p-value on this estimate is large, it is plausible that there is no interaction at all. Thus, the simpler non-interaction model is preferable.
6.3.8 Logistic Regression Key Points
Y is a binary response variable.
πi is a function of explanatory variables xi1,…xip.
E(Yi)=πi=eβ0+β1xi+…βpxip1+eβ0+β1xi+…βpxip
β0+β1xi+…βpxip=log(πi1−πi)
For quantitative xj, when all other explanatory variables are held constant, the odds of “success” multiply be a factor of eβj for each 1 unit increase in xj
For categorical xj, when all other explanatory variables are held constant, the odds of “success” are eβj times higher for category j than for the “baseline category.”
For models with interaction, we can only interpret βj when the values of all other explanatory variables are given (since the effect of xj depends on the other variables.)