Chapter 11 Model Evaluation

This week, our goals are to…

Continue to develop your final project plans.
Using peer review, practice giving and receiving evaluations of predictive analytics work and proposals.
Recognize the capabilities and limitations of selected model evaluation metrics, including the basics of how these metrics are calculated.

11.1 Evaluation metrics

Below, please read and watch about a variety of evaluation metrics that can be used to evaluate the results and utility of a machine learning model. Some of these metrics are ones that we have already used before in our class, while others are new.

Start by reading the following:

Erickson, B. J., & Kitamura, F. (2021). Magician’s corner: 9. Performance metrics for machine learning models. Radiology: Artificial Intelligence, 3(3), e200126. https://doi.org/10.1148/ryai.2021200126. PDF: https://pubs.rsna.org/doi/epdf/10.1148/ryai.2021200126.

Continue by watching the following:

Starmer, J. (2018). Machine Learning Fundamentals: The Confusion Matrix. StatQuest. https://www.youtube.com/watch?v=Kdsp6soqA7o.
Starmer, J. (2019). Machine Learning Fundamentals: Sensitivity and Specificity. StatQuest. https://www.youtube.com/watch?v=vP06aMoz4v8.
Data Science Dojo. Introduction to Precision, Recall and F1 | Classification Models. https://www.youtube.com/watch?v=jJ7ff7Gcq34.
Dragonfly Statistics. (2013). Statistics : The F Score. https://www.youtube.com/watch?v=fcO9820wCXE.
Starmer, J. (2019). ROC and AUC, Clearly Explained! StatQuest. https://www.youtube.com/watch?v=4jRBRDbJemM.
Starmer, J. Machine Learning Fundamentals: Cross Validation. StatQuest. https://www.youtube.com/watch?v=fSytzGwwBVw.

11.2 Assignment

This week’s assignment involves three parts:

Peer review – submit over both email and in D2L
Draft abstract – submit in D2L only
Discussion response – submit in D2L only

11.2.1 Peer review

This week, you will do a peer review of a classmate’s final project preparation presentation from earlier in the course. And you will also receive one peer review about your own presentation from one of your classmates.

Task 1: In this week’s D2L section, find the name of the classmate who you will peer review. Then, in the same D2L section, locate that classmate’s presentation—which might be in two separate files for audio and video—and make sure that you can both hear and view it.

Task 2: Listen to and read each slide in the presentation that you are reviewing. Write at least four sentences or more of feedback, suggestions, or commentary for each slide in the presentation, for the five main slides that were required. That’s a total of 20 sentences that you should write, at least.

Task 3: Write an additional paragraph with any overall feedback for the project. Remember to provide the type of thoughtful feedback and review that you yourself would want to receive.

Task 4: E-mail your review directly to the student whose presentation you reviewed. Please copy all instructors on this email.

Task 5: In addition to the e-mail you will send, please also submit a copy of your peer review in the D2L dropbox called “Week 11 peer review.”

11.2.2 Draft abstract

In this part of the assignment, please imagine that it is one year into the future. In this scenario, imagine that you successfully finished your final project in this class and you continued working on the project and published it in a peer-reviewed academic journal. That publication has an abstract at the top. What does that abstract say?

Note that published articles are arguably not the most important outcome and use of machine learning models (compared to other practical applications), but since this is a PhD course, it is worthwhile for us to also deliberately practice a more formal research skill like writing an abstract in the context of machine learning projects. The abstract you submit might be shared with the rest of the class and/or peer-reviewed later in the class.

Task 6: Write a one-page abstract based on your final project. More details are below.

Additional details:

Submit the abstract in D2L to the dropbox called “Week 11 project abstract.”
Your abstract should contain the following sections: purpose, methods, results, conclusions.
Remember that we are pretending that you have already completed your project and written a publication-quality manuscript about it. This means that you will have results to report, even if your final project in this class is only a proposal. We are pretending that you went on to do the whole project, meaning that you ran all of the analytics and got results that you can share.
Since you likely do not know the results of your final project yet, just write “X” in place of any numbers that you do not yet know. Or you can make up a number that you think is reasonable. For example: “The best predictive model has a sensitivity of X, which is inadequate for practical use.”
You might find it useful to look at the abstracts of scholarly articles that we have read earlier in this course, for examples and ideas about how to write.

11.2.3 Discussion post

Below, you will prepare a discussion response and submit it in D2L.

Task 7: Give an example of an analytics scenario in which prioritizing high sensitivity would be most important.

Task 8: Give an example of an analytics scenario in which prioritizing high specificity would be most important.

Task 9: Give an example of an analytics scenario in which prioritizing high F1 score would be most important.

Task 10: What are the benefits of using cross validation? Please include at least one example of one of the benefits that you describe.

Task 11: Even though it’s a pretty smart approach, what are some of the limitations of cross validation? Please include at least one example of one of the limitations that you describe.

You have reached the end of this week’s assignment. Please be sure to both e-mail and upload your peer review, upload your draft abstract, and post your discussion response in D2L. Also, don’t forget to complete your weekly flashcard requirement in the Adaptive Learner App!