Chapter 6 Logistic Regression

6.1 Logistic Regression

6.1.1 Modeling Binary Response

  • So far, we have modeled only quantitative response variables.

  • The normal error regression model makes the assumption that the response variable is normally distributed, given the value(s) of the explanatory variables.

  • Now, we’ll look at how to model a categorical response variable. We’ll consider only situations where the response is binary (i.e. has 2 categories)

6.1.2 Credit Card Dataset

We’ll consider a dataset pertaining to 10,000 credit cards. The goal is to predict whether or not the user will default on the payment, using information on the credit card balance, user’s annual income, and whether or not the user is a student. Data come from Introduction to Statistical Learning by James, Witten, Hastie, Tibshirani.

library(ISLR)
data(Default)
summary(Default)
##  default    student       balance           income     
##  No :9667   No :7056   Min.   :   0.0   Min.   :  772  
##  Yes: 333   Yes:2944   1st Qu.: 481.7   1st Qu.:21340  
##                        Median : 823.6   Median :34553  
##                        Mean   : 835.4   Mean   :33517  
##                        3rd Qu.:1166.3   3rd Qu.:43808  
##                        Max.   :2654.3   Max.   :73554

6.1.3 Default and Balance

ggplot(data=Default, aes(y=default, x=balance)) + geom_point(alpha=0.2) 

6.1.4 Linear Regression Model for Education Level

#convert default from yes/no to 0/1
Default$default <- as.numeric(Default$default=="Yes") 
ggplot(data=Default, aes(y=default, x= balance)) + geom_point(alpha=0.2)  + stat_smooth(method="lm", se=FALSE)

There are a lot of problems with this model!

6.1.5 Transforming into interval (0,1)

  • Starting with our linear model E(Yi)=β0+β1xi1, we need to transform β0+β1xi1 into (0,1).

  • Let πi=eβ0+β1xi11+eβ0+β1xi1.

  • Then 0πi1, and πi represents an estimate of P(Yi=1).

  • This function maps the values of β0+β1xi1 into the interval (0,1).

  • The logistic regression model assumes that:

    • Yi{0,1}
    • E(Yi)=P(Yi=1)=πi=eβ0+β1xi1+βpxip1+eβ0+β1xi1+βpxip

i.e. β0+β1xi1+βpxip=log(πi1πi). (This is called the logit function and can be written logit(πi).

  • Instead of assuming that the expected response is a linear function of the explanatory variables, we are assuming that it is a function of a linear function of the explanatory variables.

6.1.6 Logistic Regression Model for Balance

ggplot(data=Default, aes(y=default, x= balance)) + geom_point(alpha=0.2) + 
  stat_smooth(method="glm", se=FALSE, method.args = list(family=binomial)) 

6.1.7 Fitting the Logistic Regression Model in R

CCDefault_M <- glm(data=Default, default ~ balance, family = binomial(link = "logit"))
summary(CCDefault_M)
## 
## Call:
## glm(formula = default ~ balance, family = binomial(link = "logit"), 
##     data = Default)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.2697  -0.1465  -0.0589  -0.0221   3.7589  
## 
## Coefficients:
##                Estimate  Std. Error z value            Pr(>|z|)    
## (Intercept) -10.6513306   0.3611574  -29.49 <0.0000000000000002 ***
## balance       0.0054989   0.0002204   24.95 <0.0000000000000002 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1596.5  on 9998  degrees of freedom
## AIC: 1600.5
## 
## Number of Fisher Scoring iterations: 8

6.1.8 The Logistic Regression Equation

The regression equation is:

P(Default)=ˆπi=e10.65+0.0055×balance1+e10.65+0.0055×balance

  • For a $1,000 balance, the estimated default probability is e10.65+0.0055(1000)1+e10.65+0.0055(1000)0.006

  • For a $1,500 balance, the estimated default probability is e10.65+0.0055(1500)1+e10.65+0.0055(1500)0.08

  • For a $2,000 balance, the estimated default probability is e10.65+0.0055(2000)1+e10.65+0.0055(2000)0.59

6.1.9 Predict in R

predict(CCDefault_M, newdata=data.frame((balance=1000)), type="response")
##           1 
## 0.005752145
predict(CCDefault_M, newdata=data.frame((balance=1500)), type="response")
##          1 
## 0.08294762
predict(CCDefault_M, newdata=data.frame((balance=2000)), type="response")
##         1 
## 0.5857694

6.1.10 Where do the b’s come from?

  • Recall that for a quantitative response variable, the values of b1,b2,,bp are chosen in a way that minimizes ni=1(yi(β0+β1xi1++βpxip)2).

  • Least squares does not work well in this generalized setting. Instead, the b’s are calculated using a more advanced technique, known as maximum likelihood estimation.

6.2 Interpretations in a Logistic Regression Model

6.2.1 Recall Logistic Regression Curve for Credit Card Data

ggplot(data=Default, aes(y=default, x= balance)) + geom_point(alpha=0.2) + 
  stat_smooth(method="glm", se=FALSE, method.args = list(family=binomial)) 

6.2.2 Recall Credit Card Model Output

M <- glm(data=Default, default ~ balance, family = binomial(link = "logit"))
summary(M)
## 
## Call:
## glm(formula = default ~ balance, family = binomial(link = "logit"), 
##     data = Default)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.2697  -0.1465  -0.0589  -0.0221   3.7589  
## 
## Coefficients:
##                Estimate  Std. Error z value            Pr(>|z|)    
## (Intercept) -10.6513306   0.3611574  -29.49 <0.0000000000000002 ***
## balance       0.0054989   0.0002204   24.95 <0.0000000000000002 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1596.5  on 9998  degrees of freedom
## AIC: 1600.5
## 
## Number of Fisher Scoring iterations: 8

6.2.3 Balance Logistic Model Equation

The regression equation is:

P(Default)=ˆπi=e10.65+0.0055×balance1+e10.65+0.0055×balance

  • For a $1,000 balance, the estimated default probability is ˆπi=e10.65+0.0055(1000)1+e10.65+0.0055(1000)0.005752145

  • For a $1,500 balance, the estimated default probability is ˆπi=e10.65+0.0055(1500)1+e10.65+0.0055(1500)0.08294762

  • For a $2,000 balance, the estimated default probability is ˆπi=e10.65+0.0055(2000)1+e10.65+0.0055(2000)0.5857694

6.2.4 Odds

For an event with probability p, the odds of the event occurring are p1p.

The odds of default are given by πi1πi.

Examples:

  • The estimated odds of default for a $1,000 balance are 0.00575214510.0057521451:173.

  • The estimated odds of default for a $1,500 balance are 0.0829476210.082947621:11.

  • The estimated odds of default for a $2,000 balance are 0.585769410.58576941.414:1.

6.2.5 Odds Ratio

The quantity πi1πiπj1πj represents the odds ratio of a default for user i, compared to user j. This quantity is called the odds ratio.

Example:

The default odds ratio for a $1,000 payment, compared to a $2,000 payment is

The odds ratio is 11731.41411:244.

The odds of a default are about 244 times larger for a $2,000 payment than a $1,000 payment.

6.2.6 Interpretation of β1

Consider the odds ratio for a case j with explanatory variable x+1, compared to case i with explanatory variable x.

That is log(πi1πi)=β0+β1x, and log(πj1πj)=β0+β1(x+1).

log(πj1πjπi1πi)=log(πj1πj)log(πi1πj)=β0+β1(x+1)(β0+β1(x))=β1.

Thus a 1-unit increase in x is associated with a multiplicative change in the log odds by a factor of β1.

A 1-unit increase in x is associated with a multiplicative change in the odds by a factor of eβ1.

6.2.7 Intrepretation in Credit Card Example

b1=0.0055

The odds of default are estimated to multiply by e0.00551.0055 for each 1-dollar increase in balance on the credit card.

That is, the odds of default are estimated to increase by 0.55% for each additional dollar on the card balance.

An increase of d dollars in credit card balance is estimated to be associated with an increase odds of default by a factor of e0.0055d.

The odds of default for a balance of $2,000 are estimated to be e0.0055×1000244 times as great as the odds of default for a $1,000 balance.

6.2.8 Hypothesis Tests in Logistic Regression

  • The p-value on the “balance” line of the regression output is associated with the null hypothesis β1=0, that is that there is no relationship between balance and the odds of defaulting on the payment.

  • The fact that the p-value is so small tells us that there is strong evidence of a relationship between balance and odds of default.

6.2.9 Confidence Intervals for β1

confint(M, level = 0.95)
##                     2.5 %       97.5 %
## (Intercept) -11.383288936 -9.966565064
## balance       0.005078926  0.005943365

We are 95% confident that a 1 dollar increase in credit card balance is associate with an increased odds of default by a factor between e0.005081.0051 and e0.005941.0060.

This is a profile-likelihood interval, which you can read more about here.

6.3 Multiple Logistic Regression

6.3.1 Logistic Regression Models with Multiple Explanatory Variables

We can also perform logistic regression in situations where there are multiple explanatory variables.

6.3.2 Logistic Model with Multiple Predictors

CCDefault_M2 <- glm(data=Default, default ~ balance + student, family = binomial(link = "logit"))
summary(CCDefault_M2)
## 
## Call:
## glm(formula = default ~ balance + student, family = binomial(link = "logit"), 
##     data = Default)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.4578  -0.1422  -0.0559  -0.0203   3.7435  
## 
## Coefficients:
##                Estimate  Std. Error z value             Pr(>|z|)    
## (Intercept) -10.7494959   0.3691914 -29.116 < 0.0000000000000002 ***
## balance       0.0057381   0.0002318  24.750 < 0.0000000000000002 ***
## studentYes   -0.7148776   0.1475190  -4.846           0.00000126 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1571.7  on 9997  degrees of freedom
## AIC: 1577.7
## 
## Number of Fisher Scoring iterations: 8

6.3.3 Multiple Logistic Model Illustration

ggplot(data=Default, aes(y=default, x= balance, color=student)) + geom_point(alpha=0.2) + stat_smooth(method="glm", se=FALSE, method.args = list(family=binomial)) 

6.3.4 Multiple Logistic Regression Interpretation

The regression equation is:

P(Default)=ˆπi=e10.75+0.00575×balance0.7149×Istudent1+e10.75+0.00575×balance0.7149×Istudent

  • The odds of default are estimated to multiply by e0.0057381.00575 for each 1 dollar increase in balance whether the user is a student or nonstudent. Thus, the estimated odds of default increase by about 0.05%.

  • We might be more interested in what would happen with a larger increase, say $100. The odds of default are estimated to multiply by e0.005738×1001.775 for each $100 increase in balance for students as well as nonstudents. Thus, the estimated odds of default increase by about 77.5%.

The odds of default for students are estimated to be e0.71490.49 as high for students as non-students, assuming balance amount is held constant.

6.3.5 Hypothesis Tests in Multiple Logistic Regression Model

  • There is strong evidence of a relationship between balance and odds of default, provided we are comparing students to students, or nonstudents to nonstudents.

  • There is evidence that students are less likely to default than nonstudents, provided the balance on the card is the same.

6.3.6 Multiple Logistic Regression Model with Interaction

CCDefault_M_Int <- glm(data=Default, default ~ balance * student, family = binomial(link = "logit"))
summary(CCDefault_M_Int)
## 
## Call:
## glm(formula = default ~ balance * student, family = binomial(link = "logit"), 
##     data = Default)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -2.4839  -0.1415  -0.0553  -0.0202   3.7628  
## 
## Coefficients:
##                       Estimate  Std. Error z value            Pr(>|z|)    
## (Intercept)        -10.8746818   0.4639679 -23.438 <0.0000000000000002 ***
## balance              0.0058188   0.0002937  19.812 <0.0000000000000002 ***
## studentYes          -0.3512310   0.8037333  -0.437               0.662    
## balance:studentYes  -0.0002196   0.0004781  -0.459               0.646    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2920.6  on 9999  degrees of freedom
## Residual deviance: 1571.5  on 9996  degrees of freedom
## AIC: 1579.5
## 
## Number of Fisher Scoring iterations: 8

6.3.7 Interpretations for Logistic Model with Interaction

  • The regression equation is:

P(Default)=ˆπi=e10.87+0.0058×balance0.35×Istudent0.0002×balance×Istudent1+e10.87+0.0058×balance0.35×Istudent0.0002×balance×Istudent

  • Since estimate of the interaction effect is so small and the p-value on this estimate is large, it is plausible that there is no interaction at all. Thus, the simpler non-interaction model is preferable.

6.3.8 Logistic Regression Key Points

  • Y is a binary response variable.

  • πi is a function of explanatory variables xi1,xip.

  • E(Yi)=πi=eβ0+β1xi+βpxip1+eβ0+β1xi+βpxip

  • β0+β1xi+βpxip=log(πi1πi)

  • For quantitative xj, when all other explanatory variables are held constant, the odds of “success” multiply be a factor of eβj for each 1 unit increase in xj

  • For categorical xj, when all other explanatory variables are held constant, the odds of “success” are eβj times higher for category j than for the “baseline category.”

  • For models with interaction, we can only interpret βj when the values of all other explanatory variables are given (since the effect of xj depends on the other variables.)