Chapter 3 Synchronous seminar
In this week of HE930, we will meet synchronously on Zoom to examine the predictive analytics process, discuss the ethics of using predictive analytics in HPEd, and apply predictive analytics together to a dataset.
Note that there is no assignment to complete and turn in this week. The only required work this week is to attend the synchronous seminar sessions on Zoom. Please try your best to attend these sessions. If you cannot attend, we will arrange for make-up activities for you to complete.
MOST IMPORTANTLY, please complete the following preparation before you attend the synchronous seminar sessions.
3.1 Preparation for synchronous seminar
Please watch the following two videos, which we will discuss in our synchronous seminar:
- “Predictive Analytics in Education: An introductory example”. https://youtu.be/wGE7C5w6hb4. Available on YouTube and also embedded below.
The slides for the video above are available at https://rpubs.com/anshulkumar/ClassificationExample2020.
The Social Dilemma on Netflix. This link might take you there: https://www.netflix.com/title/81254224. If you do not have access to this on your own, inform the course instructors and we will arrange for you to watch.
Finally, in addition to watching the two videos above, please review all Week 1 readings, if you have not done so already.
Once you complete the items above, you do not need to do anything else other than attending the scheduled synchronous sessions on Zoom.
3.2 Seminar details
Schedule as of May 21 2024:
The Analytics Process. Monday May 27 2024, 12:00 – 2:00 p.m. Boston time. To prepare for this session, please review all Week 1 readings and watch this 26-minute video: https://youtu.be/wGE7C5w6hb4.
Machine Learning in R. Tuesday May 28 2024, 2:00 – 4:00 p.m. Boston time. During this session, we will do a machine learning tutorial in R and RStudio on our computers. Please be prepared to run RStudio on your computer and share your screen. No other preparation for this session is needed.
The Social Dilemma and Ethical Concerns of Machine Learning. Friday May 31 2024, 2:00 – 4:00 p.m. To prepare for this session, please watch The Social Dilemma on Netflix, which you can access at https://www.netflix.com/title/81254224.
You should have received calendar invitations to your email address for the sessions above. We will see who all is able to attend the sessions above. Then, for those who are not able to attend, we will schedule additional sessions in which the topics above will be repeated.
3.3 Session 1: The Analytics Process
3.3.1 Goal 1: Discuss the PA process and its applications
Reminder:
- PA = predictive analytics
- ML = machine learning
- AI = artificial intelligence
This discussion will primarily be driven by questions and topics raised by students during the sessions itself. The items listed below are meant to supplement our discussion.
3.3.3 Discussion questions from PA video/slides
How can predictive/learning analytics help you in your own work as an educator or at your organization/institution? What predictions would be useful for you to make?
How can you leverage data that your institution already collects (or that it is well-positioned to collect) using predictive analytic methods?
What would be the benefits and detriments of incorporating predictive analytics into your institution’s practices and processes?
Could predictive analysis complement any already-ongoing initiatives at your institution?
What would the ethical implications be of using predictive analytics at your institution? Would it cause unfair discrimination against particular learners? Would it help level the playing field for all learners?
3.3.4 Goal 2: Practice the PA process using simple examples (if time permits)
To achieve this goal, we will go through the Worksheet Packet used in the 2023 Quantitative Methods Workshop in the HE-942 seminar course. These are available in this week’s content module in D2L under the title “Worksheets for quantitative methods workshop from Anshul Kumar.”
3.3.5 Goal 3: Generate and discuss PA research questions (if time permits)
Discuss how PA can practically be used in HPEd and/or healthcare, especially in the settings in which we all work.
Generate research questions for using PA in HPEd.
Share and discuss answers to Task 16 in Week 1 discussions. Task 16: Find a case or example in the Ekowo & Palmer (2019) report that shows a way in which you wish that you could potentially use analytics within your own work or at your own institution. How is it (the case/example) similar to what you would like to do? How is it different?
3.4 Session 2: Machine Learning in R
During this session, we will work together to complete and discuss the machine learning tutorial below.
3.4.1 Goals
Practice using decision tree and logistic regression models to predict which students are going to pass or fail, using the
student-por.csvdata.Fine-tune predictive models.
Compare the results of different predictive models and choose the best one.
Brainstorm about how the predictions would be used in an educational setting.
3.4.2 Important reminders
Anywhere you see the word
MODIFYis one place where you might consider making changes to the code.If you are not certain about any interpretations of results—especially confusion matrices, accuracy, sensitivity, and specificity—stop and ask an instructor for assistance.
For most of this tutorial, you will run the code below in an R script or R markdown file in RStudio on your own computer. You will also make minor modifications to this code.
3.4.3 Load relevant packages
Step 1: Load packages
if (!require(PerformanceAnalytics)) install.packages('PerformanceAnalytics')
if (!require(rpart)) install.packages('rpart')
if (!require(rpart.plot)) install.packages('rpart.plot')
if (!require(car)) install.packages('car')
if (!require(rattle)) install.packages('rattle')
library(PerformanceAnalytics)
library(rpart)
library(rpart.plot)
library(car)
library(rattle)3.4.4 Import and describe data
We will use the student-por.csv data.
Step 2: Import data
d <- read.csv("student-por.csv")The best place to download this data is from D2L.
Data source and details:
P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. Available at https://archive.ics.uci.edu/ml/datasets/Student+Performance.
Alternate source: https://www.kaggle.com/larsen0966/student-performance-data-set
If the code above does not work to load the data, you can try this code instead:
d <- read.csv(file = "student-por.csv", sep = ";")Step 3: List variables
names(d)## [1] "school" "sex" "age" "address"
## [5] "famsize" "Pstatus" "Medu" "Fedu"
## [9] "Mjob" "Fjob" "reason" "guardian"
## [13] "traveltime" "studytime" "failures" "schoolsup"
## [17] "famsup" "paid" "activities" "nursery"
## [21] "higher" "internet" "romantic" "famrel"
## [25] "freetime" "goout" "Dalc" "Walc"
## [29] "health" "absences" "G1" "G2"
## [33] "G3"Description of dataset and all variables: https://archive.ics.uci.edu/ml/datasets/Student+Performance.
All variables in formula (for easy copying and pasting):
(b <- paste(names(d), collapse="+"))## [1] "school+sex+age+address+famsize+Pstatus+Medu+Fedu+Mjob+Fjob+reason+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+G1+G2+G3"Step 4: Calculate number of observations
nrow(d)## [1] 649Step 5: Generate binary version of dependent variable, G3 (final grade, 0 to 20).
d$passed <- ifelse(d$G3 > 9.99, 1, 0)We’re assuming that a score of over 9.99 is a passing score, and below that is failing.
Step 6: Descriptive statistics for G3 continuous numeric variable.
summary(d$G3)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 10.00 12.00 11.91 14.00 19.00sd(d$G3)## [1] 3.230656Step 7: Who all passed and failed, binary qualitative categorical variable.
with(d, table(passed, useNA = "always"))## passed
## 0 1 <NA>
## 100 549 0Step 8: Histogram
hist(as.numeric(d$G3))
Step 9: Scatterplots
plot(d$G1 , d$G3) # MODIFY which variables you plot
plot(d$health, d$G3)
Step 10: Selected correlations (optional)
chart.Correlation(d[c("G3","G1","Medu","failures")], histogram=TRUE, pch=19) ## Warning in par(usr): argument 1 does not name a graphical
## parameter
## Warning in par(usr): argument 1 does not name a graphical
## parameter
## Warning in par(usr): argument 1 does not name a graphical
## parameter
## Warning in par(usr): argument 1 does not name a graphical
## parameter
## Warning in par(usr): argument 1 does not name a graphical
## parameter
## Warning in par(usr): argument 1 does not name a graphical
## parameter
3.4.5 Divide training and testing data
Step 11: Divide data
trainingRowIndex <- sample(1:nrow(d), 0.75*nrow(d)) # row indices for training data
dtrain <- d[trainingRowIndex, ] # model training data
dtest <- d[-trainingRowIndex, ] # test data3.4.6 Decision tree model – regression tree
Activity summary:
- Goal: predict continuous outcome
G3, using a regression tree. - Start by using all variables to make decision tree. Check predictive capability.
- Remove variables to make predictions with less information.
- Modify cutoff thresholds and see how confusion matrix changes.
- Anywhere you see the word
MODIFYis one place where you might consider making changes to the code. - Figure out which students to remediate.
3.4.6.1 Train and inspect model
Step 14: Train a decision tree model
tree1 <- rpart(G3 ~ school+sex+age+address+famsize+Pstatus+Medu+Fedu+Mjob+Fjob+reason+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+G1+G2, data=dtrain, method = 'anova')
# MODIFY. Complete the next step (visualize the tree) and
# then come back to this and run the model without G1 and
# G2. Then try other combinations.
summary(tree1)## Call:
## rpart(formula = G3 ~ school + sex + age + address + famsize +
## Pstatus + Medu + Fedu + Mjob + Fjob + reason + guardian +
## traveltime + studytime + failures + schoolsup + famsup +
## paid + activities + nursery + higher + internet + romantic +
## famrel + freetime + goout + Dalc + Walc + health + absences +
## G1 + G2, data = dtrain, method = "anova")
## n= 486
##
## CP nsplit rel error xerror xstd
## 1 0.52265775 0 1.0000000 1.0058050 0.09869478
## 2 0.13509387 1 0.4773423 0.4791046 0.06386679
## 3 0.09601004 2 0.3422484 0.3534168 0.03867491
## 4 0.03415689 3 0.2462383 0.2571165 0.03717264
## 5 0.03235626 4 0.2120815 0.2462798 0.03698815
## 6 0.01914289 5 0.1797252 0.1910548 0.03250928
## 7 0.01000000 6 0.1605823 0.1738477 0.03242947
##
## Variable importance
## G2 G1 Fedu Medu failures school absences
## 44 26 7 6 5 5 2
## Mjob address famsup internet
## 1 1 1 1
##
## Node number 1: 486 observations, complexity param=0.5226577
## mean=11.85185, MSE=10.04801
## left son=2 (256 obs) right son=3 (230 obs)
## Primary splits:
## G2 < 11.5 to the left, improve=0.52265770, (0 missing)
## G1 < 11.5 to the left, improve=0.43865020, (0 missing)
## failures < 0.5 to the right, improve=0.19521860, (0 missing)
## higher splits as LR, improve=0.11898040, (0 missing)
## school splits as RL, improve=0.08024918, (0 missing)
## Surrogate splits:
## G1 < 11.5 to the left, agree=0.872, adj=0.730, (0 split)
## Fedu < 2.5 to the left, agree=0.621, adj=0.200, (0 split)
## Medu < 3.5 to the left, agree=0.617, adj=0.191, (0 split)
## school splits as RL, agree=0.611, adj=0.178, (0 split)
## failures < 0.5 to the right, agree=0.611, adj=0.178, (0 split)
##
## Node number 2: 256 observations, complexity param=0.1350939
## mean=9.679688, MSE=6.131775
## left son=4 (25 obs) right son=5 (231 obs)
## Primary splits:
## G2 < 7.5 to the left, improve=0.42026750, (0 missing)
## G1 < 8.5 to the left, improve=0.29929460, (0 missing)
## failures < 0.5 to the right, improve=0.11683160, (0 missing)
## school splits as RL, improve=0.08217415, (0 missing)
## higher splits as LR, improve=0.05697087, (0 missing)
## Surrogate splits:
## G1 < 5.5 to the left, agree=0.914, adj=0.12, (0 split)
##
## Node number 3: 230 observations, complexity param=0.09601004
## mean=14.26957, MSE=3.309943
## left son=6 (122 obs) right son=7 (108 obs)
## Primary splits:
## G2 < 13.5 to the left, improve=0.61586370, (0 missing)
## G1 < 14.5 to the left, improve=0.45375010, (0 missing)
## schoolsup splits as RL, improve=0.06845436, (0 missing)
## Mjob splits as LRLLR, improve=0.05976642, (0 missing)
## Medu < 2.5 to the left, improve=0.03657314, (0 missing)
## Surrogate splits:
## G1 < 13.5 to the left, agree=0.843, adj=0.667, (0 split)
## Mjob splits as LRLLR, agree=0.635, adj=0.222, (0 split)
## Medu < 3.5 to the left, agree=0.609, adj=0.167, (0 split)
## Fedu < 3.5 to the left, agree=0.583, adj=0.111, (0 split)
## Fjob splits as LRLRR, agree=0.570, adj=0.083, (0 split)
##
## Node number 4: 25 observations, complexity param=0.03235626
## mean=4.8, MSE=13.36
## left son=8 (13 obs) right son=9 (12 obs)
## Primary splits:
## absences < 1 to the left, improve=0.4730731, (0 missing)
## famsup splits as LR, improve=0.2526635, (0 missing)
## reason splits as RLLR, improve=0.2071372, (0 missing)
## internet splits as LR, improve=0.2055175, (0 missing)
## G2 < 5.5 to the left, improve=0.2055175, (0 missing)
## Surrogate splits:
## address splits as LR, agree=0.68, adj=0.333, (0 split)
## Fedu < 1.5 to the left, agree=0.68, adj=0.333, (0 split)
## famsup splits as LR, agree=0.68, adj=0.333, (0 split)
## internet splits as LR, agree=0.68, adj=0.333, (0 split)
## G2 < 2.5 to the left, agree=0.68, adj=0.333, (0 split)
##
## Node number 5: 231 observations, complexity param=0.03415689
## mean=10.20779, MSE=2.493619
## left son=10 (85 obs) right son=11 (146 obs)
## Primary splits:
## G2 < 9.5 to the left, improve=0.28956930, (0 missing)
## G1 < 8.5 to the left, improve=0.17691170, (0 missing)
## failures < 1.5 to the right, improve=0.09614057, (0 missing)
## higher splits as LR, improve=0.04207345, (0 missing)
## school splits as RL, improve=0.02994832, (0 missing)
## Surrogate splits:
## G1 < 8.5 to the left, agree=0.740, adj=0.294, (0 split)
## failures < 0.5 to the right, agree=0.684, adj=0.141, (0 split)
## higher splits as LR, agree=0.658, adj=0.071, (0 split)
## famrel < 1.5 to the left, agree=0.658, adj=0.071, (0 split)
## age < 18.5 to the right, agree=0.641, adj=0.024, (0 split)
##
## Node number 6: 122 observations
## mean=12.92623, MSE=0.9207874
##
## Node number 7: 108 observations, complexity param=0.01914289
## mean=15.78704, MSE=1.66761
## left son=14 (64 obs) right son=15 (44 obs)
## Primary splits:
## G2 < 15.5 to the left, improve=0.51904580, (0 missing)
## G1 < 15.5 to the left, improve=0.40331220, (0 missing)
## age < 15.5 to the left, improve=0.09025786, (0 missing)
## studytime < 3.5 to the left, improve=0.07579506, (0 missing)
## reason splits as LLLR, improve=0.05838160, (0 missing)
## Surrogate splits:
## G1 < 15.5 to the left, agree=0.824, adj=0.568, (0 split)
## reason splits as LLLR, agree=0.648, adj=0.136, (0 split)
## age < 17.5 to the left, agree=0.630, adj=0.091, (0 split)
## studytime < 3.5 to the left, agree=0.630, adj=0.091, (0 split)
## nursery splits as RL, agree=0.630, adj=0.091, (0 split)
##
## Node number 8: 13 observations
## mean=2.384615, MSE=12.85207
##
## Node number 9: 12 observations
## mean=7.416667, MSE=0.7430556
##
## Node number 10: 85 observations
## mean=9.094118, MSE=3.308789
##
## Node number 11: 146 observations
## mean=10.85616, MSE=0.8765716
##
## Node number 14: 64 observations
## mean=15.01562, MSE=0.7966309
##
## Node number 15: 44 observations
## mean=16.90909, MSE=0.80991743.4.6.2 Tree visualization
Step 15: Visualize decision tree model in two ways.
prp(tree1)
fancyRpartPlot(tree1, caption = "Regression Tree")
3.4.6.3 Test model
Step 16: Make predictions on testing data, using trained model
dtest$tree1.pred <- predict(tree1, newdata = dtest)Step 17: Visualize predictions
with(dtest, plot(G3,tree1.pred, main="Actual vs Predicted, testing data",xlab = "Actual G3",ylab = "Predicted G3"))
Step 18: Make confusion matrix.
PredictionCutoff <- 9.99 # MODIFY. Compare values in 9-11 range.
dtest$tree1.pred.passed <- ifelse(dtest$tree1.pred > PredictionCutoff, 1, 0)
(cm1 <- with(dtest,table(tree1.pred.passed,passed)))## passed
## tree1.pred.passed 0 1
## 0 22 13
## 1 4 124Step 19: Calculate accuracy
CorrectPredictions1 <- cm1[1,1] + cm1[2,2]
TotalStudents1 <- nrow(dtest)
(Accuracy1 <- CorrectPredictions1/TotalStudents1)## [1] 0.8957055Step 20: Sensitivity (proportion of people who actually failed that were correctly predicted to fail).
(Sensitivity1 <- cm1[1,1]/(cm1[1,1]+cm1[2,1]))## [1] 0.8461538Step 21: Specificity (proportion of people who actually passed that were correctly predicted to pass).
(Specificity1 <- cm1[2,2]/(cm1[1,2]+cm1[2,2]))## [1] 0.9051095BE SURE TO DOUBLE-CHECK THE CALCULATIONS ABOVE MANUALLY!
Step 22: It is very important for you, the data analyst, to modify the 9.99 cutoff assigned as PredictionCutoff above to see how you can change the predictions made by the model. Write down what you observe as you change this value and re-run the confusion matrix, accuracy, sensitivity, and specificity code above. What are the implications of your manual modification of this cutoff? Remind your instructors to discuss this, in case they forget!
3.4.7 Decision tree model – classification tree
Activity summary:
- Goal: predict binary outcome
passed, using a classification tree. - Start by using all variables to make decision tree. Check predictive capability.
- Remove variables to make predictions with less information.
- Modify cutoff thresholds and see how confusion matrix changes.
- Anywhere you see the word
MODIFYis one place where you might consider making changes to the code. - Figure out which students to remediate.
3.4.7.1 Train and inspect model
Step 23: Train a decision tree model
tree2 <- rpart(passed ~ school+sex+age+address+famsize+Pstatus+Medu+Fedu+Mjob+Fjob+reason+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+G1+G2, data=dtrain, method = "class")
# MODIFY. Try without G1 and G2. Then try other combinations.
summary(tree2)## Call:
## rpart(formula = passed ~ school + sex + age + address + famsize +
## Pstatus + Medu + Fedu + Mjob + Fjob + reason + guardian +
## traveltime + studytime + failures + schoolsup + famsup +
## paid + activities + nursery + higher + internet + romantic +
## famrel + freetime + goout + Dalc + Walc + health + absences +
## G1 + G2, data = dtrain, method = "class")
## n= 486
##
## CP nsplit rel error xerror xstd
## 1 0.60810811 0 1.0000000 1.0000000 0.10703224
## 2 0.02702703 1 0.3918919 0.3918919 0.07056791
## 3 0.01000000 3 0.3378378 0.4459459 0.07494734
##
## Variable importance
## G2 G1 goout absences
## 75 19 4 1
##
## Node number 1: 486 observations, complexity param=0.6081081
## predicted class=1 expected loss=0.1522634 P(node) =1
## class counts: 74 412
## probabilities: 0.152 0.848
## left son=2 (59 obs) right son=3 (427 obs)
## Primary splits:
## G2 < 8.5 to the left, improve=71.39302, (0 missing)
## G1 < 8.5 to the left, improve=55.96656, (0 missing)
## failures < 0.5 to the right, improve=21.60141, (0 missing)
## school splits as RL, improve=11.93134, (0 missing)
## higher splits as LR, improve=10.37243, (0 missing)
## Surrogate splits:
## G1 < 7.5 to the left, agree=0.912, adj=0.271, (0 split)
## absences < 21.5 to the right, agree=0.881, adj=0.017, (0 split)
##
## Node number 2: 59 observations
## predicted class=0 expected loss=0.1186441 P(node) =0.1213992
## class counts: 52 7
## probabilities: 0.881 0.119
##
## Node number 3: 427 observations, complexity param=0.02702703
## predicted class=1 expected loss=0.05152225 P(node) =0.8786008
## class counts: 22 405
## probabilities: 0.052 0.948
## left son=6 (51 obs) right son=7 (376 obs)
## Primary splits:
## G2 < 9.5 to the left, improve=6.817189, (0 missing)
## G1 < 8.5 to the left, improve=5.597683, (0 missing)
## failures < 1.5 to the right, improve=3.292459, (0 missing)
## school splits as RL, improve=1.943624, (0 missing)
## goout < 4.5 to the right, improve=1.159030, (0 missing)
## Surrogate splits:
## G1 < 7.5 to the left, agree=0.890, adj=0.078, (0 split)
## failures < 1.5 to the right, agree=0.885, adj=0.039, (0 split)
##
## Node number 6: 51 observations, complexity param=0.02702703
## predicted class=1 expected loss=0.2941176 P(node) =0.1049383
## class counts: 15 36
## probabilities: 0.294 0.706
## left son=12 (10 obs) right son=13 (41 obs)
## Primary splits:
## goout < 4.5 to the right, improve=4.098422, (0 missing)
## famrel < 4.5 to the right, improve=3.601572, (0 missing)
## Mjob splits as LLRLR, improve=2.366947, (0 missing)
## Walc < 1.5 to the right, improve=1.588235, (0 missing)
## age < 15.5 to the left, improve=1.493931, (0 missing)
##
## Node number 7: 376 observations
## predicted class=1 expected loss=0.01861702 P(node) =0.7736626
## class counts: 7 369
## probabilities: 0.019 0.981
##
## Node number 12: 10 observations
## predicted class=0 expected loss=0.3 P(node) =0.02057613
## class counts: 7 3
## probabilities: 0.700 0.300
##
## Node number 13: 41 observations
## predicted class=1 expected loss=0.195122 P(node) =0.08436214
## class counts: 8 33
## probabilities: 0.195 0.8053.4.7.2 Tree Visualization
Step 24: Visualize decision tree model in two ways
prp(tree2)
fancyRpartPlot(tree2, caption = "Classification Tree")
3.4.7.3 Test model
Step 25: Make predictions and confusion matrix on testing data classes, using trained model.
dtest$tree2.pred <- predict(tree2, newdata = dtest, type = 'class')
# MODIFY. change 'class' to 'prob'
(cm2 <- with(dtest,table(tree2.pred,passed)))## passed
## tree2.pred 0 1
## 0 16 6
## 1 10 131Step 26: Make predictions and confusion matrix on testing data using probability cutoffs. Optional; results not shown.
dtest$tree2.pred <- predict(tree2, newdata = dtest, type = 'prob')
ProbabilityCutoff <- 0.5 # MODIFY. Compare different probability values.
dtest$tree2.pred.probs <- 1-dtest$tree2.pred[,1]
dtest$tree2.pred.passed <- ifelse(dtest$tree2.pred.probs > ProbabilityCutoff, 1, 0)
(cm2b <- with(dtest,table(tree2.pred.passed,passed)))Step 27: Calculate accuracy
CorrectPredictions2 <- cm2[1,1] + cm2[2,2]
TotalStudents2 <- nrow(dtest)
(Accuracy2 <- CorrectPredictions2/TotalStudents2)## [1] 0.9018405Step 28: Sensitivity (proportion of people who actually failed that were correctly predicted to fail)
(Sensitivity2 <- cm2[1,1]/(cm2[1,1]+cm2[2,1]))## [1] 0.6153846Step 29: Specificity (proportion of people who actually passed that were correctly predicted to pass):
(Specificity2 <- cm2[2,2]/(cm2[1,2]+cm2[2,2]))## [1] 0.9562044ALSO DOUBLE-CHECK THE CALCULATIONS ABOVE MANUALLY!
3.4.8 Logistic regression model – classification
Activity summary:
- Goal: predict binary outcome
passed, using logistic regression. - Start by using all variables to make a logistic regression model. Check predictive capability.
- Remove variables to make predictions with less information.
- Modify cutoff thresholds and see how confusion matrix changes.
- Anywhere you see the word
MODIFYis one place where you might consider making changes to the code. - Figure out which students to remediate.
3.4.8.1 Train and inspect model
Step 30: Train a logistic regression model
blr1 <- glm(passed ~ school+sex+age+address+famsize+Pstatus+Medu+Fedu+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+Mjob+reason+Fjob+G1+G2, data=dtrain, family = "binomial")
# MODIFY. Try without G1 and G2. Then try other combinations.
# also remove variables causing multicollinearity and see if it makes a difference!
summary(blr1)##
## Call:
## glm(formula = passed ~ school + sex + age + address + famsize +
## Pstatus + Medu + Fedu + guardian + traveltime + studytime +
## failures + schoolsup + famsup + paid + activities + nursery +
## higher + internet + romantic + famrel + freetime + goout +
## Dalc + Walc + health + absences + Mjob + reason + Fjob +
## G1 + G2, family = "binomial", data = dtrain)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -3.8661 0.0002 0.0082 0.0990 1.8686
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -23.040984 6.887785 -3.345 0.000822 ***
## schoolMS -2.001589 0.837455 -2.390 0.016844 *
## sexM -0.375397 0.781008 -0.481 0.630760
## age 0.339504 0.279343 1.215 0.224227
## addressU -0.004627 0.639516 -0.007 0.994227
## famsizeLE3 0.733613 0.637131 1.151 0.249555
## PstatusT -0.174664 1.020227 -0.171 0.864065
## Medu -0.166469 0.378993 -0.439 0.660487
## Fedu -0.234892 0.360151 -0.652 0.514270
## guardianmother -0.543305 0.877613 -0.619 0.535870
## guardianother -0.680017 1.388191 -0.490 0.624234
## traveltime 0.072141 0.357331 0.202 0.840005
## studytime 0.387251 0.403021 0.961 0.336617
## failures -0.280919 0.383309 -0.733 0.463632
## schoolsupyes -1.045999 1.010617 -1.035 0.300664
## famsupyes 0.174679 0.636691 0.274 0.783812
## paidyes -1.376655 1.350185 -1.020 0.307916
## activitiesyes -0.350516 0.659548 -0.531 0.595108
## nurseryyes -0.514131 0.691681 -0.743 0.457296
## higheryes 0.599830 0.727741 0.824 0.409806
## internetyes 0.271653 0.746568 0.364 0.715956
## romanticyes -0.009714 0.681448 -0.014 0.988627
## famrel -0.208882 0.292422 -0.714 0.475032
## freetime 0.172662 0.328729 0.525 0.599416
## goout -0.183948 0.290576 -0.633 0.526703
## Dalc 0.210777 0.398147 0.529 0.596531
## Walc -0.140283 0.335146 -0.419 0.675528
## health -0.112965 0.250815 -0.450 0.652430
## absences -0.077681 0.062396 -1.245 0.213139
## Mjobhealth -1.133978 1.352049 -0.839 0.401632
## Mjobother 0.484169 0.723450 0.669 0.503337
## Mjobservices -0.426972 1.068876 -0.399 0.689555
## Mjobteacher 2.048168 1.589345 1.289 0.197507
## reasonhome 0.417396 0.846756 0.493 0.622059
## reasonother 0.299751 0.949328 0.316 0.752192
## reasonreputation 0.929403 0.902950 1.029 0.303341
## Fjobhealth -3.086336 2.340498 -1.319 0.187281
## Fjobother -2.298034 1.423163 -1.615 0.106368
## Fjobservices -1.099519 1.425432 -0.771 0.440495
## Fjobteacher -2.600208 2.292015 -1.134 0.256600
## G1 0.352955 0.206878 1.706 0.087989 .
## G2 2.165515 0.400888 5.402 6.6e-08 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 414.67 on 485 degrees of freedom
## Residual deviance: 114.65 on 444 degrees of freedom
## AIC: 198.65
##
## Number of Fisher Scoring iterations: 9car::vif(blr1)## GVIF Df GVIF^(1/(2*Df))
## school 3.055055 1 1.747872
## sex 2.677012 1 1.636158
## age 2.620530 1 1.618805
## address 1.745089 1 1.321018
## famsize 1.491143 1 1.221124
## Pstatus 2.187087 1 1.478880
## Medu 2.736422 1 1.654214
## Fedu 2.240528 1 1.496839
## guardian 3.629598 2 1.380272
## traveltime 1.766750 1 1.329192
## studytime 1.695953 1 1.302287
## failures 1.885413 1 1.373103
## schoolsup 2.107672 1 1.451782
## famsup 1.752538 1 1.323835
## paid 1.406720 1 1.186052
## activities 1.793292 1 1.339138
## nursery 1.682246 1 1.297014
## higher 1.642264 1 1.281509
## internet 2.321050 1 1.523499
## romantic 1.946589 1 1.395202
## famrel 2.112690 1 1.453510
## freetime 2.084860 1 1.443905
## goout 2.434667 1 1.560342
## Dalc 2.863053 1 1.692056
## Walc 3.854881 1 1.963385
## health 2.125608 1 1.457946
## absences 1.960477 1 1.400170
## Mjob 9.506182 4 1.325106
## reason 5.062362 3 1.310365
## Fjob 10.222706 4 1.337198
## G1 2.000219 1 1.414291
## G2 2.333219 1 1.5274883.4.8.2 Test model
Step 31: Make predictions on testing data, using trained model.
Predicting probabilities…
dtest$blr1.pred <- predict(blr1, newdata = dtest, type = 'response')
ProbabilityCutoff <- 0.5 # MODIFY. Compare different probability values.
dtest$blr1.pred.probs <- 1-dtest$blr1.pred
dtest$blr1.pred.passed <- ifelse(dtest$blr1.pred > ProbabilityCutoff, 1, 0)
(cm3 <- with(dtest,table(blr1.pred.passed,passed)))## passed
## blr1.pred.passed 0 1
## 0 18 4
## 1 8 133Step 32: Make confusion matrix
(cm3 <- with(dtest,table(blr1.pred.passed,passed)))## passed
## blr1.pred.passed 0 1
## 0 18 4
## 1 8 133Step 33: Calculate accuracy
CorrectPredictions3 <- cm3[1,1] + cm3[2,2]
TotalStudents3 <- nrow(dtest)
(Accuracy3 <- CorrectPredictions3/TotalStudents3)## [1] 0.9263804Step 34: Sensitivity (proportion of people who actually failed that were correctly predicted to fail)
(Sensitivity3 <- cm3[1,1]/(cm3[1,1]+cm3[2,1]))## [1] 0.6923077Step 35: Specificity (proportion of people who actually passed that were correctly predicted to pass)
(Specificity3 <- cm3[2,2]/(cm3[1,2]+cm3[2,2]))## [1] 0.9708029ALSO DOUBLE-CHECK THE CALCULATIONS ABOVE MANUALLY!
Step 36: It is very important for you, the data analyst, to modify the 0.5 cutoff assigned as ProbabilityCutoff above to see how you can change the predictions made by the model. Write down what you observe as you change this value and re-run the confusion matrix, accuracy, sensitivity, and specificity code above. What are the implications of your manual modification of this cutoff? Remind your instructors to discuss this, in case they forget!
3.5 Session 3: The Social Dilemma and Ethical Concerns of ML
3.5.1 Goals
Examine similarities and differences between social media analytics and HPEd analytics
Discuss ethics of predictive analytics
Practice predictive analytics research design (if time permits)
3.5.2 Discussion topics
Week 1 discussion questions related to ethics and PA/ML.
Will people start to behave differently just because they know that these models are being used to analyze them, subconsciously or consciously?
The same uses of predictive analytics shown in the movie, to make profit for the tech companies and for the people who advertise with detective companies, are also coming to Health Professions education at a very high speed. Are we ready for this?
Can we use the same technology to manipulate users for good purposes? Not only predict their outcomes but also cause them to do things to make their outcomes better? How appropriate or inappropriate is it for us to do this?
An interviewee in The Social Dilemma gave the example of Wikipedia showing different users different content based on what they were paid to show. As educators, we might show different materials to different students based on having analyzed their data. Is this the right or wrong thing to do? Who should control this capability? Should it even be possible or not?
How do the social media marketing frameworks on the previous slides relate to the relationships between people and technology in HPEd?
