Chapter 3 May 30–Jun 4: Synchronous seminar
In this week of HE930, we will meet synchronously on Zoom to examine the predictive analytics process, discuss the ethics of using predictive analytics in HPEd, and apply predictive analytics together to a dataset.
Note that there is no assignment to complete and turn in this week. The only required work this week is to attend the synchronous seminar sessions on Zoom. Please try your best to attend these sessions. If you cannot attend, we will arrange for make-up activities for you to complete.
MOST IMPORTANTLY, please complete the following preparation before you attend the synchronous seminar sessions.
3.1 Preparation for synchronous seminar
Please watch the following two videos, which we will discuss in our synchronous seminar:
- “Predictive Analytics in Education: An introductory example”. https://youtu.be/wGE7C5w6hb4. Available on YouTube and also embedded below.
The slides for the video above are available at https://rpubs.com/anshulkumar/ClassificationExample2020.
The Social Dilemma on Netflix. This link might take you there: https://www.netflix.com/title/81254224. If you do not have access to this on your own, inform the course instructors and we will arrange for you to watch.
Finally, in addition to watching the two videos above, please review all Week 1 readings, if you have not done so already.
Once you complete the items above, you do not need to do anything else other than attending the scheduled synchronous sessions on Zoom.
3.2 Seminar details
Schedule as of May 29 2023:
The Analytics Process. Thursday June 1 2023, 11 a.m. – 1 p.m. Boston time. To prepare for this session, please review all Week 1 readings and watch this 26-minute video: https://youtu.be/wGE7C5w6hb4.
Machine Learning in R. Part 1: Thursday June 1 2023, 1–2 p.m. Boston time; Part 2: Friday June 2 2023, 1–2 p.m. Boston time. During this session, we will do a machine learning tutorial in R and RStudio on our computers. Please be prepared to run RStudio on your computer and share your screen. No other preparation for this session is needed.
The Social Dilemma and Ethical Concerns of ML. Friday June 2 2023, 11 a.m. – 1 p.m. Boston time. To prepare for this session, please watch The Social Dilemma on Netflix, which you can access at https://www.netflix.com/title/81254224.
You should have received calendar invitations to your email address for the sessions above. We will see who all is able to attend the sessions above. Then, for those who are not able to attend, we will schedule additional sessions in which the topics above will be repeated.
3.3 Session 1: The Analytics Process
3.3.1 Goal 1: Discuss the PA process and its applications
Reminder:
- PA = predictive analytics
- ML = machine learning
- AI = artificial intelligence
This discussion will primarily be driven by questions and topics raised by students during the sessions itself. The items listed below are meant to supplement our discussion.
3.3.2 Questions & topics from weeks 1 & 2 discussion posts
3.3.2.1 What is PA/ML and how do we conduct it?
Item 1: “I would like to learn more about the early alert systems these schools were able to use and how to build something like that.” (DR)
Item 2: “Recent work on learning analytics based on students’ online engagement highlights the importance of students’ internal conditions as well as external factors as indicators of their performance.” (SA)
3.3.2.2 Using PA/ML in HPEd
Item 3: “The article makes use of a single course in a single medical university and hence the findings may not be generalizable.” (SA)
Item 4: “how are predictive analytics being used at MGIHP and in the PhD? I am also curious about Nicole’s and Anshul’s thoughts about the ethicality, bias and the role disclosure to learners when predictive analytics are being used.” (FS)
3.3.2.3 Ethics of PA/ML
Item 5: “I would have liked more transparency about how and which data were used from assignments/discussions, however: ‘The process of establishing features from raw data in the database was realized automatically via the help of an in-house developed tool.’ [(Akçapınar 2019)] That said, there is a good Appendix (Table 14) that spells out which data were used.” (DR)
Item 6: “several of the metrics are related to number or amount of time spent visiting a particular part of the course site. As we all know, it is easy to access a page and then minimize it without interacting with the content. If the numbers are very high in this area, it may skew results.” (DR)
Item 7: “As described in the beginning of the Ekowo and Palmer article, this type of information can be used for less than altruistic purposes- to deter students who are unlikely to pass and therefore will reflect poorly on the metrics published about the university.” (DR)
Item 8: Possible risks of using PA/ML in education…
- “Data selection bias - unintentional stereotyping” (DT)
- “Breach of confidentiality” / “Misuse of student information” (DT)
- “The use of analytic process to predict college academic success has huge potentials for discrimination, labeling and stigma, and if not implemented correctly are potential sources of ethical and legal issues in higher education.” (RB)
- “May be used to perpetuate bias and/or structural racism in academia.” (FS)
- “There is also the issue of mistrust and potential lack of transparency regarding how the data will be stored, shared, and used outside of the stated intent.” (FS)
Item 9: “As we know, there are different types of learners. Some students would thrive using the online platform methods of learning. Some students may not. It does not mean they are less intelligent. It simply means that those individual students retain and utilize knowledge differently. So, I do worry about creating a system where the number of logins, the time spent, number of contents created, etc. are used as markers for student success. It gives an arbitrary meaning to the term ‘success,’ if we are only defining success by their performance on the online platform and the final course grade.” (MM)
Item 10: “With course signals, leaners are assigned a color based on their risk level, this could lead to labeling and foster a class system within the learning community.” (JM)
Item 11: “I find this topic to be fascinating as I reflect back on my academic experience in undergraduate studies. The common sentiment to someone heading into college can be, ‘You need to be accountable for your work and your time, no one is going to be checking in to see if you’re handing in your work or not.’ This can be true, however, to note how universities do care for the wellbeing of their students (and not just for performative reasons) is an essential piece for not only students in the classroom, but also how they then take what they learn and bring it into their careers.” (ED)
Item 12: “The vulnerable period of adjustment, especially for certain groups with most factors for academic failure is a critical period for colleges/universities for identification and provision of academic support such as targeted advising and adaptive learning.” (RB)
3.3.2.4 Disseminating/communicating results
Item 13: JM pointed out that many forms of intervention were used in the Arnold and Pistilli study, including traffic signal indicator in LMS, emails, text messages, referrals to advisors or resource centers, face to face meetings with instructor. (JM)
Item 14: “Please share why these 3 articles were selected vs. others?” (FS)
3.3.3 Discussion questions from PA video/slides
How can predictive/learning analytics help you in your own work as an educator or at your organization/institution? What predictions would be useful for you to make?
How can you leverage data that your institution already collects (or that it is well-positioned to collect) using predictive analytic methods?
What would be the benefits and detriments of incorporating predictive analytics into your institution’s practices and processes?
Could predictive analysis complement any already-ongoing initiatives at your institution?
What would the ethical implications be of using predictive analytics at your institution? Would it cause unfair discrimination against particular learners? Would it help level the playing field for all learners?
3.4 Session 2: Machine Learning in R
During this session, we will work together to complete and discuss the machine learning tutorial below.
3.4.1 Goals
Practice using decision tree and logistic regression models to predict which students are going to pass or fail, using the
student-por.csvdata.Fine-tune predictive models.
Compare the results of different predictive models and choose the best one.
Brainstorm about how the predictions would be used in an educational setting.
3.4.2 Important reminders
Anywhere you see the word
MODIFYis one place where you might consider making changes to the code.If you are not certain about any interpretations of results—especially confusion matrices, accuracy, sensitivity, and specificity—stop and ask an instructor for assistance.
For most of this tutorial, you will run the code below in an R script or R markdown file in RStudio on your own computer. You will also make minor modifications to this code.
3.4.3 Load relevant packages
Step 1: Load packages
if (!require(PerformanceAnalytics)) install.packages('PerformanceAnalytics')
if (!require(rpart)) install.packages('rpart')
if (!require(rpart.plot)) install.packages('rpart.plot')
if (!require(car)) install.packages('car')
if (!require(rattle)) install.packages('rattle')
library(PerformanceAnalytics)
library(rpart)
library(rpart.plot)
library(car)
library(rattle)3.4.4 Import and describe data
We will use the student-por.csv data.
Step 2: Import data
d <- read.csv("student-por.csv")The best place to download this data is from D2L.
Data source and details:
P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. Available at https://archive.ics.uci.edu/ml/datasets/Student+Performance.
Alternate source: https://www.kaggle.com/larsen0966/student-performance-data-set
If the code above does not work to load the data, you can try this code instead:
d <- read.csv(file = "student-por.csv", sep = ";")Step 3: List variables
names(d)## [1] "school" "sex" "age"
## [4] "address" "famsize" "Pstatus"
## [7] "Medu" "Fedu" "Mjob"
## [10] "Fjob" "reason" "guardian"
## [13] "traveltime" "studytime" "failures"
## [16] "schoolsup" "famsup" "paid"
## [19] "activities" "nursery" "higher"
## [22] "internet" "romantic" "famrel"
## [25] "freetime" "goout" "Dalc"
## [28] "Walc" "health" "absences"
## [31] "G1" "G2" "G3"Description of dataset and all variables: https://archive.ics.uci.edu/ml/datasets/Student+Performance.
All variables in formula (for easy copying and pasting):
(b <- paste(names(d), collapse="+"))## [1] "school+sex+age+address+famsize+Pstatus+Medu+Fedu+Mjob+Fjob+reason+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+G1+G2+G3"Step 4: Calculate number of observations
nrow(d)## [1] 649Step 5: Generate binary version of dependent variable, G3 (final grade, 0 to 20).
d$passed <- ifelse(d$G3 > 9.99, 1, 0)We’re assuming that a score of over 9.99 is a passing score, and below that is failing.
Step 6: Descriptive statistics for G3 continuous numeric variable.
summary(d$G3)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 0.00 10.00 12.00 11.91 14.00 19.00sd(d$G3)## [1] 3.230656Step 7: Who all passed and failed, binary qualitative categorical variable.
with(d, table(passed, useNA = "always"))## passed
## 0 1 <NA>
## 100 549 0Step 8: Histogram
hist(as.numeric(d$G3))
Step 9: Scatterplots
plot(d$G1 , d$G3) # MODIFY which variables you plot
plot(d$health, d$G3)
Step 10: Selected correlations (optional)
chart.Correlation(d[c("G3","G1","Medu","failures")], histogram=TRUE, pch=19) ## Warning in par(usr): argument 1 does not name a
## graphical parameter
## Warning in par(usr): argument 1 does not name a
## graphical parameter
## Warning in par(usr): argument 1 does not name a
## graphical parameter
## Warning in par(usr): argument 1 does not name a
## graphical parameter
## Warning in par(usr): argument 1 does not name a
## graphical parameter
## Warning in par(usr): argument 1 does not name a
## graphical parameter
3.4.5 Divide training and testing data
Step 11: Divide data
trainingRowIndex <- sample(1:nrow(d), 0.75*nrow(d)) # row indices for training data
dtrain <- d[trainingRowIndex, ] # model training data
dtest <- d[-trainingRowIndex, ] # test data3.4.6 Decision tree model – regression tree
Activity summary:
- Goal: predict continuous outcome
G3, using a regression tree. - Start by using all variables to make decision tree. Check predictive capability.
- Remove variables to make predictions with less information.
- Modify cutoff thresholds and see how confusion matrix changes.
- Anywhere you see the word
MODIFYis one place where you might consider making changes to the code. - Figure out which students to remediate.
3.4.6.1 Train and inspect model
Step 14: Train a decision tree model
tree1 <- rpart(G3 ~ school+sex+age+address+famsize+Pstatus+Medu+Fedu+Mjob+Fjob+reason+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+G1+G2, data=dtrain, method = 'anova')
# MODIFY. Complete the next step (visualize the tree) and
# then come back to this and run the model without G1 and
# G2. Then try other combinations.
summary(tree1)## Call:
## rpart(formula = G3 ~ school + sex + age + address + famsize +
## Pstatus + Medu + Fedu + Mjob + Fjob + reason + guardian +
## traveltime + studytime + failures + schoolsup + famsup +
## paid + activities + nursery + higher + internet + romantic +
## famrel + freetime + goout + Dalc + Walc + health + absences +
## G1 + G2, data = dtrain, method = "anova")
## n= 486
##
## CP nsplit rel error xerror
## 1 0.52083647 0 1.0000000 1.0049734
## 2 0.14446906 1 0.4791635 0.4861773
## 3 0.09621072 2 0.3346945 0.3860498
## 4 0.04873478 3 0.2384838 0.2498483
## 5 0.02269069 4 0.1897490 0.2202419
## 6 0.02065619 5 0.1670583 0.2010360
## 7 0.01389335 6 0.1464021 0.1889762
## 8 0.01000000 7 0.1325087 0.1678254
## xstd
## 1 0.09694661
## 2 0.06118933
## 3 0.04487333
## 4 0.03205810
## 5 0.02768125
## 6 0.02504732
## 7 0.02498770
## 8 0.02401157
##
## Variable importance
## G2 G1 Medu school
## 41 26 8 6
## failures traveltime absences age
## 6 5 3 1
## Mjob famrel schoolsup
## 1 1 1
##
## Node number 1: 486 observations, complexity param=0.5208365
## mean=11.93621, MSE=11.06383
## left son=2 (247 obs) right son=3 (239 obs)
## Primary splits:
## G2 < 11.5 to the left, improve=0.52083650, (0 missing)
## G1 < 11.5 to the left, improve=0.45540180, (0 missing)
## failures < 0.5 to the right, improve=0.16729010, (0 missing)
## higher splits as LR, improve=0.10402620, (0 missing)
## school splits as RL, improve=0.08877125, (0 missing)
## Surrogate splits:
## G1 < 11.5 to the left, agree=0.883, adj=0.762, (0 split)
## school splits as RL, agree=0.628, adj=0.243, (0 split)
## failures < 0.5 to the right, agree=0.623, adj=0.234, (0 split)
## Medu < 3.5 to the left, agree=0.621, adj=0.230, (0 split)
## traveltime < 1.5 to the right, agree=0.603, adj=0.192, (0 split)
##
## Node number 2: 247 observations, complexity param=0.1444691
## mean=9.574899, MSE=7.037912
## left son=4 (54 obs) right son=5 (193 obs)
## Primary splits:
## G2 < 8.5 to the left, improve=0.44686460, (0 missing)
## G1 < 8.5 to the left, improve=0.29815780, (0 missing)
## failures < 0.5 to the right, improve=0.08333001, (0 missing)
## school splits as RL, improve=0.07690047, (0 missing)
## absences < 0.5 to the left, improve=0.05888545, (0 missing)
## Surrogate splits:
## G1 < 7.5 to the left, agree=0.842, adj=0.278, (0 split)
## famrel < 1.5 to the left, agree=0.794, adj=0.056, (0 split)
## failures < 1.5 to the right, agree=0.789, adj=0.037, (0 split)
## absences < 21.5 to the right, agree=0.789, adj=0.037, (0 split)
##
## Node number 3: 239 observations, complexity param=0.09621072
## mean=14.37657, MSE=3.506731
## left son=6 (119 obs) right son=7 (120 obs)
## Primary splits:
## G2 < 13.5 to the left, improve=0.61725540, (0 missing)
## G1 < 14.5 to the left, improve=0.45640120, (0 missing)
## schoolsup splits as RL, improve=0.06896048, (0 missing)
## Medu < 2.5 to the left, improve=0.05083120, (0 missing)
## Mjob splits as LRLLR, improve=0.04596584, (0 missing)
## Surrogate splits:
## G1 < 13.5 to the left, agree=0.833, adj=0.664, (0 split)
## Medu < 2.5 to the left, agree=0.611, adj=0.218, (0 split)
## Mjob splits as LRLLR, agree=0.594, adj=0.185, (0 split)
## schoolsup splits as RL, agree=0.573, adj=0.143, (0 split)
## age < 16.5 to the left, agree=0.565, adj=0.126, (0 split)
##
## Node number 4: 54 observations, complexity param=0.04873478
## mean=6.222222, MSE=12.06173
## left son=8 (20 obs) right son=9 (34 obs)
## Primary splits:
## absences < 1 to the left, improve=0.4023255, (0 missing)
## G2 < 5.5 to the left, improve=0.2724873, (0 missing)
## reason splits as RLLR, improve=0.1727322, (0 missing)
## school splits as RL, improve=0.1507900, (0 missing)
## G1 < 5.5 to the left, improve=0.1282040, (0 missing)
## Surrogate splits:
## G2 < 2.5 to the left, agree=0.722, adj=0.25, (0 split)
## address splits as LR, agree=0.685, adj=0.15, (0 split)
## Medu < 3.5 to the right, agree=0.685, adj=0.15, (0 split)
## internet splits as LR, agree=0.685, adj=0.15, (0 split)
## age < 18.5 to the right, agree=0.648, adj=0.05, (0 split)
##
## Node number 5: 193 observations, complexity param=0.01389335
## mean=10.51295, MSE=1.607345
## left son=10 (118 obs) right son=11 (75 obs)
## Primary splits:
## G2 < 10.5 to the left, improve=0.24081430, (0 missing)
## G1 < 9.5 to the left, improve=0.12838150, (0 missing)
## failures < 0.5 to the right, improve=0.07005277, (0 missing)
## Mjob splits as LLLRR, improve=0.06961104, (0 missing)
## freetime < 4.5 to the right, improve=0.03487091, (0 missing)
## Surrogate splits:
## G1 < 10.5 to the left, agree=0.684, adj=0.187, (0 split)
## Mjob splits as LLLRR, agree=0.658, adj=0.120, (0 split)
## Medu < 3.5 to the left, agree=0.648, adj=0.093, (0 split)
## studytime < 2.5 to the left, agree=0.632, adj=0.053, (0 split)
## Fedu < 2.5 to the left, agree=0.627, adj=0.040, (0 split)
##
## Node number 6: 119 observations
## mean=12.89916, MSE=0.9310077
##
## Node number 7: 120 observations, complexity param=0.02065619
## mean=15.84167, MSE=1.749931
## left son=14 (70 obs) right son=15 (50 obs)
## Primary splits:
## G2 < 15.5 to the left, improve=0.52892010, (0 missing)
## G1 < 15.5 to the left, improve=0.41717930, (0 missing)
## age < 15.5 to the left, improve=0.05026133, (0 missing)
## studytime < 3.5 to the left, improve=0.04771113, (0 missing)
## goout < 4.5 to the right, improve=0.03361244, (0 missing)
## Surrogate splits:
## G1 < 15.5 to the left, agree=0.817, adj=0.56, (0 split)
## absences < 0.5 to the right, agree=0.633, adj=0.12, (0 split)
## school splits as LR, agree=0.617, adj=0.08, (0 split)
## Dalc < 2.5 to the left, agree=0.608, adj=0.06, (0 split)
## reason splits as LLLR, agree=0.600, adj=0.04, (0 split)
##
## Node number 8: 20 observations, complexity param=0.02269069
## mean=3.35, MSE=17.1275
## left son=16 (12 obs) right son=17 (8 obs)
## Primary splits:
## G2 < 7.5 to the left, improve=0.3561767, (0 missing)
## reason splits as RLLR, improve=0.2638102, (0 missing)
## age < 17.5 to the right, improve=0.2323763, (0 missing)
## famrel < 3.5 to the right, improve=0.2207763, (0 missing)
## freetime < 3.5 to the right, improve=0.1736187, (0 missing)
## Surrogate splits:
## Medu < 1.5 to the right, agree=0.75, adj=0.375, (0 split)
## famrel < 1.5 to the right, agree=0.75, adj=0.375, (0 split)
## age < 18.5 to the left, agree=0.70, adj=0.250, (0 split)
## famsize splits as LR, agree=0.70, adj=0.250, (0 split)
## Fjob splits as --LRL, agree=0.70, adj=0.250, (0 split)
##
## Node number 9: 34 observations
## mean=7.911765, MSE=1.374567
##
## Node number 10: 118 observations
## mean=10.01695, MSE=1.677679
##
## Node number 11: 75 observations
## mean=11.29333, MSE=0.5006222
##
## Node number 14: 70 observations
## mean=15.02857, MSE=0.7991837
##
## Node number 15: 50 observations
## mean=16.98, MSE=0.8596
##
## Node number 16: 12 observations
## mean=1.333333, MSE=8.888889
##
## Node number 17: 8 observations
## mean=6.375, MSE=14.234383.4.6.2 Tree visualization
Step 15: Visualize decision tree model in two ways.
prp(tree1)
fancyRpartPlot(tree1, caption = "Regression Tree")
3.4.6.3 Test model
Step 16: Make predictions on testing data, using trained model
dtest$tree1.pred <- predict(tree1, newdata = dtest)Step 17: Visualize predictions
with(dtest, plot(G3,tree1.pred, main="Actual vs Predicted, testing data",xlab = "Actual G3",ylab = "Predicted G3"))
Step 18: Make confusion matrix.
PredictionCutoff <- 9.99 # MODIFY. Compare values in 9-11 range.
dtest$tree1.pred.passed <- ifelse(dtest$tree1.pred > PredictionCutoff, 1, 0)
(cm1 <- with(dtest,table(tree1.pred.passed,passed)))## passed
## tree1.pred.passed 0 1
## 0 16 3
## 1 8 136Step 19: Calculate accuracy
CorrectPredictions1 <- cm1[1,1] + cm1[2,2]
TotalStudents1 <- nrow(dtest)
(Accuracy1 <- CorrectPredictions1/TotalStudents1)## [1] 0.9325153Step 20: Sensitivity (proportion of people who actually failed that were correctly predicted to fail).
(Sensitivity1 <- cm1[1,1]/(cm1[1,1]+cm1[2,1]))## [1] 0.6666667Step 21: Specificity (proportion of people who actually passed that were correctly predicted to pass).
(Specificity1 <- cm1[2,2]/(cm1[1,2]+cm1[2,2]))## [1] 0.9784173BE SURE TO DOUBLE-CHECK THE CALCULATIONS ABOVE MANUALLY!
Step 22: It is very important for you, the data analyst, to modify the 9.99 cutoff assigned as PredictionCutoff above to see how you can change the predictions made by the model. Write down what you observe as you change this value and re-run the confusion matrix, accuracy, sensitivity, and specificity code above. What are the implications of your manual modification of this cutoff? Remind your instructors to discuss this, in case they forget!
3.4.7 Decision tree model – classification tree
Activity summary:
- Goal: predict binary outcome
passed, using a classification tree. - Start by using all variables to make decision tree. Check predictive capability.
- Remove variables to make predictions with less information.
- Modify cutoff thresholds and see how confusion matrix changes.
- Anywhere you see the word
MODIFYis one place where you might consider making changes to the code. - Figure out which students to remediate.
3.4.7.1 Train and inspect model
Step 23: Train a decision tree model
tree2 <- rpart(passed ~ school+sex+age+address+famsize+Pstatus+Medu+Fedu+Mjob+Fjob+reason+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+G1+G2, data=dtrain, method = "class")
# MODIFY. Try without G1 and G2. Then try other combinations.
summary(tree2)## Call:
## rpart(formula = passed ~ school + sex + age + address + famsize +
## Pstatus + Medu + Fedu + Mjob + Fjob + reason + guardian +
## traveltime + studytime + failures + schoolsup + famsup +
## paid + activities + nursery + higher + internet + romantic +
## famrel + freetime + goout + Dalc + Walc + health + absences +
## G1 + G2, data = dtrain, method = "class")
## n= 486
##
## CP nsplit rel error xerror
## 1 0.60526316 0 1.0000000 1.0000000
## 2 0.01973684 1 0.3947368 0.3947368
## 3 0.01000000 3 0.3552632 0.5263158
## xstd
## 1 0.10535787
## 2 0.06980898
## 3 0.07971971
##
## Variable importance
## G2 G1 Mjob health studytime
## 68 27 2 1 1
## Fedu goout
## 1 1
##
## Node number 1: 486 observations, complexity param=0.6052632
## predicted class=1 expected loss=0.1563786 P(node) =1
## class counts: 76 410
## probabilities: 0.156 0.844
## left son=2 (54 obs) right son=3 (432 obs)
## Primary splits:
## G2 < 8.5 to the left, improve=71.95267, (0 missing)
## G1 < 8.5 to the left, improve=60.70383, (0 missing)
## failures < 0.5 to the right, improve=16.66729, (0 missing)
## higher splits as LR, improve=12.48079, (0 missing)
## school splits as RL, improve=11.47405, (0 missing)
## Surrogate splits:
## G1 < 7.5 to the left, agree=0.92, adj=0.278, (0 split)
##
## Node number 2: 54 observations
## predicted class=0 expected loss=0.07407407 P(node) =0.1111111
## class counts: 50 4
## probabilities: 0.926 0.074
##
## Node number 3: 432 observations, complexity param=0.01973684
## predicted class=1 expected loss=0.06018519 P(node) =0.8888889
## class counts: 26 406
## probabilities: 0.060 0.940
## left son=6 (30 obs) right son=7 (402 obs)
## Primary splits:
## G1 < 8.5 to the left, improve=8.9778330, (0 missing)
## G2 < 9.5 to the left, improve=8.1965850, (0 missing)
## school splits as RL, improve=1.8207900, (0 missing)
## freetime < 4.5 to the right, improve=1.5129240, (0 missing)
## Mjob splits as LRRRR, improve=0.9563274, (0 missing)
##
## Node number 6: 30 observations, complexity param=0.01973684
## predicted class=1 expected loss=0.4333333 P(node) =0.0617284
## class counts: 13 17
## probabilities: 0.433 0.567
## left son=12 (17 obs) right son=13 (13 obs)
## Primary splits:
## Mjob splits as LLRLR, improve=1.8826550, (0 missing)
## age < 17.5 to the left, improve=1.3444440, (0 missing)
## G2 < 9.5 to the left, improve=1.3444440, (0 missing)
## Walc < 2.5 to the right, improve=1.0011900, (0 missing)
## absences < 5.5 to the right, improve=0.8333333, (0 missing)
## Surrogate splits:
## health < 3.5 to the left, agree=0.800, adj=0.538, (0 split)
## studytime < 1.5 to the right, agree=0.733, adj=0.385, (0 split)
## Fedu < 1.5 to the right, agree=0.700, adj=0.308, (0 split)
## goout < 4.5 to the left, agree=0.700, adj=0.308, (0 split)
## Fjob splits as R-RLL, agree=0.667, adj=0.231, (0 split)
##
## Node number 7: 402 observations
## predicted class=1 expected loss=0.03233831 P(node) =0.8271605
## class counts: 13 389
## probabilities: 0.032 0.968
##
## Node number 12: 17 observations
## predicted class=0 expected loss=0.4117647 P(node) =0.03497942
## class counts: 10 7
## probabilities: 0.588 0.412
##
## Node number 13: 13 observations
## predicted class=1 expected loss=0.2307692 P(node) =0.02674897
## class counts: 3 10
## probabilities: 0.231 0.7693.4.7.2 Tree Visualization
Step 24: Visualize decision tree model in two ways
prp(tree2)
fancyRpartPlot(tree2, caption = "Classification Tree")
3.4.7.3 Test model
Step 25: Make predictions and confusion matrix on testing data classes, using trained model.
dtest$tree2.pred <- predict(tree2, newdata = dtest, type = 'class')
# MODIFY. change 'class' to 'prob'
(cm2 <- with(dtest,table(tree2.pred,passed)))## passed
## tree2.pred 0 1
## 0 18 3
## 1 6 136Step 26: Make predictions and confusion matrix on testing data using probability cutoffs. Optional; results not shown.
dtest$tree2.pred <- predict(tree2, newdata = dtest, type = 'prob')
ProbabilityCutoff <- 0.5 # MODIFY. Compare different probability values.
dtest$tree2.pred.probs <- 1-dtest$tree2.pred[,1]
dtest$tree2.pred.passed <- ifelse(dtest$tree2.pred.probs > ProbabilityCutoff, 1, 0)
(cm2b <- with(dtest,table(tree2.pred.passed,passed)))Step 27: Calculate accuracy
CorrectPredictions2 <- cm2[1,1] + cm2[2,2]
TotalStudents2 <- nrow(dtest)
(Accuracy2 <- CorrectPredictions2/TotalStudents2)## [1] 0.9447853Step 28: Sensitivity (proportion of people who actually failed that were correctly predicted to fail)
(Sensitivity2 <- cm2[1,1]/(cm2[1,1]+cm2[2,1]))## [1] 0.75Step 29: Specificity (proportion of people who actually passed that were correctly predicted to pass):
(Specificity2 <- cm2[2,2]/(cm2[1,2]+cm2[2,2]))## [1] 0.9784173ALSO DOUBLE-CHECK THE CALCULATIONS ABOVE MANUALLY!
3.4.8 Logistic regression model – classification
Activity summary:
- Goal: predict binary outcome
passed, using logistic regression. - Start by using all variables to make a logistic regression model. Check predictive capability.
- Remove variables to make predictions with less information.
- Modify cutoff thresholds and see how confusion matrix changes.
- Anywhere you see the word
MODIFYis one place where you might consider making changes to the code. - Figure out which students to remediate.
3.4.8.1 Train and inspect model
Step 30: Train a logistic regression model
blr1 <- glm(passed ~ school+sex+age+address+famsize+Pstatus+Medu+Fedu+guardian+traveltime+studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+internet+romantic+famrel+freetime+goout+Dalc+Walc+health+absences+Mjob+reason+Fjob+G1+G2, data=dtrain, family = "binomial")## Warning: glm.fit: fitted probabilities
## numerically 0 or 1 occurred# MODIFY. Try without G1 and G2. Then try other combinations.
# also remove variables causing multicollinearity and see if it makes a difference!
summary(blr1)##
## Call:
## glm(formula = passed ~ school + sex + age + address + famsize +
## Pstatus + Medu + Fedu + guardian + traveltime + studytime +
## failures + schoolsup + famsup + paid + activities + nursery +
## higher + internet + romantic + famrel + freetime + goout +
## Dalc + Walc + health + absences + Mjob + reason + Fjob +
## G1 + G2, family = "binomial", data = dtrain)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -4.7993 0.0000 0.0016 0.0523 1.8413
##
## Coefficients:
## Estimate Std. Error z value
## (Intercept) -42.824891 10.098604 -4.241
## schoolMS -1.156079 0.795052 -1.454
## sexM 0.152463 0.852354 0.179
## age 0.975521 0.327271 2.981
## addressU -0.313691 0.742143 -0.423
## famsizeLE3 -0.549077 0.691410 -0.794
## PstatusT -1.310276 1.256844 -1.043
## Medu 0.188860 0.388875 0.486
## Fedu 0.011102 0.370558 0.030
## guardianmother -0.982600 0.957471 -1.026
## guardianother -2.445557 1.583061 -1.545
## traveltime 0.328649 0.371401 0.885
## studytime 0.098952 0.404642 0.245
## failures 0.556032 0.417010 1.333
## schoolsupyes 0.343042 0.987826 0.347
## famsupyes 0.029035 0.615844 0.047
## paidyes -2.177930 1.302603 -1.672
## activitiesyes 0.348121 0.643352 0.541
## nurseryyes 0.252142 0.684242 0.368
## higheryes 1.042597 0.854447 1.220
## internetyes -0.662391 0.901126 -0.735
## romanticyes -0.422175 0.816231 -0.517
## famrel 0.018146 0.340701 0.053
## freetime 0.006694 0.312094 0.021
## goout -0.265564 0.320924 -0.827
## Dalc -0.686682 0.471170 -1.457
## Walc 0.119416 0.403527 0.296
## health -0.157890 0.241023 -0.655
## absences 0.034672 0.076359 0.454
## Mjobhealth -0.084283 1.384958 -0.061
## Mjobother 0.749795 0.880302 0.852
## Mjobservices 0.753968 1.282110 0.588
## Mjobteacher 0.801697 2.002931 0.400
## reasonhome 0.739129 0.890404 0.830
## reasonother 0.500404 0.926494 0.540
## reasonreputation 1.523593 1.121164 1.359
## Fjobhealth -2.965392 2.484205 -1.194
## Fjobother -1.739182 1.549686 -1.122
## Fjobservices -1.841519 1.619700 -1.137
## Fjobteacher -2.375808 2.506956 -0.948
## G1 0.864247 0.244045 3.541
## G2 2.559910 0.516052 4.961
## Pr(>|z|)
## (Intercept) 2.23e-05 ***
## schoolMS 0.145921
## sexM 0.858037
## age 0.002875 **
## addressU 0.672527
## famsizeLE3 0.427113
## PstatusT 0.297174
## Medu 0.627210
## Fedu 0.976098
## guardianmother 0.304776
## guardianother 0.122388
## traveltime 0.376217
## studytime 0.806812
## failures 0.182408
## schoolsupyes 0.728388
## famsupyes 0.962397
## paidyes 0.094528 .
## activitiesyes 0.588435
## nurseryyes 0.712501
## higheryes 0.222389
## internetyes 0.462297
## romanticyes 0.604999
## famrel 0.957524
## freetime 0.982888
## goout 0.407955
## Dalc 0.145007
## Walc 0.767284
## health 0.512416
## absences 0.649778
## Mjobhealth 0.951474
## Mjobother 0.394354
## Mjobservices 0.556487
## Mjobteacher 0.688964
## reasonhome 0.406479
## reasonother 0.589125
## reasonreputation 0.174166
## Fjobhealth 0.232596
## Fjobother 0.261743
## Fjobservices 0.255559
## Fjobteacher 0.343289
## G1 0.000398 ***
## G2 7.03e-07 ***
## ---
## Signif. codes:
## 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 421.47 on 485 degrees of freedom
## Residual deviance: 103.04 on 444 degrees of freedom
## AIC: 187.04
##
## Number of Fisher Scoring iterations: 10car::vif(blr1)## GVIF Df GVIF^(1/(2*Df))
## school 2.525752 1 1.589262
## sex 2.874376 1 1.695398
## age 3.251407 1 1.803166
## address 2.150724 1 1.466535
## famsize 1.517740 1 1.231966
## Pstatus 2.359969 1 1.536219
## Medu 2.720539 1 1.649406
## Fedu 2.540723 1 1.593965
## guardian 3.885633 2 1.403995
## traveltime 1.700634 1 1.304083
## studytime 1.476679 1 1.215187
## failures 2.057658 1 1.434454
## schoolsup 1.892572 1 1.375708
## famsup 1.484314 1 1.218324
## paid 1.588343 1 1.260295
## activities 1.575232 1 1.255083
## nursery 1.535420 1 1.239121
## higher 1.683875 1 1.297642
## internet 2.847155 1 1.687352
## romantic 2.439613 1 1.561926
## famrel 1.628762 1 1.276229
## freetime 2.108918 1 1.452211
## goout 2.878852 1 1.696718
## Dalc 5.526191 1 2.350785
## Walc 5.309723 1 2.304284
## health 1.592563 1 1.261968
## absences 1.853560 1 1.361455
## Mjob 19.871706 4 1.453046
## reason 6.622551 3 1.370369
## Fjob 19.920151 4 1.453488
## G1 2.361472 1 1.536708
## G2 2.615314 1 1.6171933.4.8.2 Test model
Step 31: Make predictions on testing data, using trained model.
Predicting probabilities…
dtest$blr1.pred <- predict(blr1, newdata = dtest, type = 'response')
# MODIFY. change 'class' to 'prob'
ProbabilityCutoff <- 0.5 # MODIFY. Compare different probability values.
dtest$blr1.pred.probs <- 1-dtest$blr1.pred
dtest$blr1.pred.passed <- ifelse(dtest$blr1.pred > ProbabilityCutoff, 1, 0)
(cm3 <- with(dtest,table(blr1.pred.passed,passed)))## passed
## blr1.pred.passed 0 1
## 0 20 7
## 1 4 132Step 32: Make confusion matrix
(cm3 <- with(dtest,table(blr1.pred.passed,passed)))## passed
## blr1.pred.passed 0 1
## 0 20 7
## 1 4 132Step 33: Calculate accuracy
CorrectPredictions3 <- cm3[1,1] + cm3[2,2]
TotalStudents3 <- nrow(dtest)
(Accuracy3 <- CorrectPredictions3/TotalStudents3)## [1] 0.9325153Step 34: Sensitivity (proportion of people who actually failed that were correctly predicted to fail)
(Sensitivity3 <- cm3[1,1]/(cm3[1,1]+cm3[2,1]))## [1] 0.8333333Step 35: Specificity (proportion of people who actually passed that were correctly predicted to pass)
(Specificity3 <- cm3[2,2]/(cm3[1,2]+cm3[2,2]))## [1] 0.9496403ALSO DOUBLE-CHECK THE CALCULATIONS ABOVE MANUALLY!
Step 36: It is very important for you, the data analyst, to modify the 0.5 cutoff assigned as ProbabilityCutoff above to see how you can change the predictions made by the model. Write down what you observe as you change this value and re-run the confusion matrix, accuracy, sensitivity, and specificity code above. What are the implications of your manual modification of this cutoff? Remind your instructors to discuss this, in case they forget!
3.5 Session 3: The Social Dilemma and Ethical Concerns of ML
3.5.1 Goals
Examine similarities and differences between social media analytics and HPEd analytics
Discuss ethics of predictive analytics
Practice predictive analytics research design (if time permits)
3.5.2 Discussion topics
Week 1 discussion questions related to ethics and PA/ML.
Will people start to behave differently just because they know that these models are being used to analyze them, subconsciously or consciously?
The same uses of predictive analytics shown in the movie, to make profit for the tech companies and for the people who advertise with detective companies, are also coming to Health Professions education at a very high speed. Are we ready for this?
Can we use the same technology to manipulate users for good purposes? Not only predict their outcomes but also cause them to do things to make their outcomes better? How appropriate or inappropriate is it for us to do this?
An interviewee in The Social Dilemma gave the example of Wikipedia showing different users different content based on what they were paid to show. As educators, we might show different materials to different students based on having analyzed their data. Is this the right or wrong thing to do? Who should control this capability? Should it even be possible or not?
How do the social media marketing frameworks on the previous slides relate to the relationships between people and technology in HPEd?
