Cognitive Diagnosis Modeling
1
Introduction
2
An Introduction to Diagnostic Assessment and Modeling
2.1
Assessments of Learning
2.2
Assessments for Learning
2.3
A comparion
2.4
Educational assessments: reasoning from evidence
2.4.1
IRT as psychometric models
2.4.2
CDM as psychometric models
2.4.3
IRT vs CDM
2.5
An example: A proportional reasoning test
2.5.1
Data
2.5.2
CTT/IRT analysis
2.5.3
CDM analysis
2.6
Some Terms
2.6.1
Attribute
2.6.2
Diagnosis
2.6.3
Latent class
2.7
CDM inputs and outputs
2.7.1
Input: Item responses
2.7.2
Input: Q-matrix
2.7.3
Output: item parameter estimates
2.7.4
Output: Population-level profile estimates
2.7.5
Output: Individual-level profile estimates
3
CDMs for dichotomous data
3.1
MIRT vs CDMs
3.1.1
Multidimensional item response theory (IRT) models
3.1.2
Cognitive Diagnosis models
3.2
CDM Notations
3.3
Attribute Profiles
3.4
Condensation Rules
3.5
The DINA model
3.6
The DINO model
3.7
Reduced RUM
3.8
The LLM
3.9
The Generalized DINA model
3.10
G-DINA model: Link functions and reduced models
3.10.1
G-DINA and DINA
3.10.2
GDINA vs DINO
3.10.3
G-DINA vs R-RUM
3.10.4
G-DINA vs LLM and A-CDM
3.11
Joint Attribute Distribution
3.12
The saturated model for joint attribute distribution
3.13
The independent model for joint attribute distribution
3.14
The multivariate normal model for joint attribute distribution
3.15
The higher-order model for joint attribute distribution
3.16
The hierarchical structure model for joint attribute distribution
3.17
Assignment I
4
R Lab I: CDM Analysis
4.1
Resources for Learning the Package
4.2
Features of the GDINA R package
4.3
Data and Q-matrix
4.4
GDINA model estimation
4.5
DINA model estimation
4.6
ACDM estimation
4.7
R-RUM Estimation
4.8
Estimation of A combination of models
4.9
Estimation of a higher-order model
4.10
Estimation of CDMs with attribute hierarchy
4.11
Exercises
5
Model Estimation
5.1
Model Estimation Approaches
5.2
Likelihood of an individual response vector
5.3
Likelihood of an individual response vector (Cont’d)
5.4
Joint maximum likelihood estimation
5.5
Marginalized likelihood of an individual response vector
5.6
Marginalized maximum likelihood estimation via EM algorithm
5.7
E-step
5.8
M-step
5.9
Joint attribute distribution parameters
5.10
Quanitities in GDINA R package
5.11
Bayesian approach for parameter estimation
5.12
MCMC for DINA model in Nimble R package
5.13
MCMC for DINA model in JAGS and R2jags R package
5.14
MCMC for DINA model in STAN
6
Model Identifiability
6.1
Global Identifiability
6.1.1
What parameters we need to consider in CDMs?
6.1.2
Global identifiability in CDMs
6.2
Global identifiability of the DINA and DINO model
6.3
Global Identifiability for General CDMs
6.4
Generic Identifiability for More General CDMs
6.5
Partial Identifiability of Attribute Profiles
6.6
Partial identifiability: Completeness of the Q-matrix
6.7
Local Identifiability
6.8
Identifiability for MCMC
6.9
Assignment 2
7
Model-data Fit Evaluation
7.1
Relative Model-Data Fit at Test Level
7.1.1
Information criterions as a function of Maximum Likelihood(ML) of CDM models
(Chen et al., 2013)
7.2
Relative Model-Data Fit at Test Level (Cont’d)
7.3
Relative Model-Data Fit at Test Level (Cont’d)
7.3.1
Calculating number of model paraters for CDMs
7.4
Relative Model-Data Fit at Test Level (Cont’d)
7.5
Relative Model-Data Fit at Test Level : Practice Session in R
7.6
Test-level Absolute Fit measures
7.7
Full information statistics
7.8
Full information statistics- Test level (Cont’d)
7.8.1
Limitations:
7.9
Limited information statistics
7.10
Limited information statistics (Cont’d)
7.11
Limited information statistics (Cont’d)
7.12
Standardized Root Mean Squared Residual (SRMSR)
7.13
Practice Session: Calculating Absolute Fit Measures
7.14
Absolute Fit Measures in the Bayesian Approach of CDM estimation
7.14.1
Posterior Predictive Model Checking
7.15
Relative Fit Measures in the Bayesian Approach of CDM estimation
7.15.1
Deviance Information Criterion(DIC)
7.15.2
Watanabe-Akaike Information Criterion(WAIC)
7.16
Exercises
8
Item FIt Measures
8.0.1
Learnng objectives for this chapter
8.0.2
Item level fit measures
8.0.3
Types of Item-level fit Measures
8.0.4
Implications of item-level misfit
8.1
Item-level Absolute Fit measures
8.1.1
Performing hypothesis tests:
8.2
Item-level Absolute Fit measures (Cont’d)
8.3
Item-level Relative Fit measures
8.4
Item-level Relative Fit measures (Cont’d)
8.5
Item-level Relative Fit measures (Cont’d)
8.6
Item-level Relative Fit measures (Cont’d)
8.7
Item-level Relative Fit measures (Cont’d)
8.8
Item-level Relative Fit measures (Cont’d)
8.9
Item-level Relative Fit measures (Cont’d)
8.10
Item-level Relative Fit measures (Cont’d)
8.11
Item-level Relative Fit measures (Cont’d)
8.12
Exercise 1
9
ATTRIBUTE PROFILE ESTIMATION
9.1
Attribute Profile Estimation
9.2
Attribute Estimation Process
9.3
Maximum Likelihood Estimation (MLE)
9.4
Maximum a Posterior (MAP) Estimation
9.5
Expected a Posterior (EAP) Estimation
9.6
Attribute Estimation in GDINA R package
9.7
Exercises
10
CLASSIFICATION ACCURACY
10.1
Classification Accuracy(CA)
10.2
Classification Consistency
10.3
The Monte Carlo Approach
10.4
Classification Accuracey : Monte Carlo Approach Using R
10.5
The Analytic Approach
10.6
\(2\times 2\)
Contingency Table in Principle
10.7
\(2\times 2\)
Contingency Table in Practice
10.8
Estimating
\(2\times 2\)
Contingency Table
10.9
Pattern Classification Accuracy
10.10
Estimating Classification Accuracy using GDINA R package
10.11
Estimating Classficiation Accuracy using Multiple Imputations.
10.12
Assignment 3
11
CONDUCTING SIMULATION STUDY
11.1
Simulation study in CDM
11.2
Some Important Terms for Simulation Study
11.2.1
Factor
11.2.2
Level/Condition
11.2.3
Combination of Condition
11.2.4
Replication
11.3
How to Conduct Simulations?
11.4
An Example: Background
11.4.1
Research Question
11.4.2
Factors
11.4.3
Conditions
11.4.4
Design
11.5
How to simulation data using a CDM
11.6
An Example: Estimating Parameters
11.7
An example: comparing true and estimated parameters
11.8
Replicating the process: An example
12
Q-matrix Validation
12.1
Introduction
12.2
Introduction (Cont’d)
12.3
GDINA discrimination index (GDI)
12.4
Q-matrix validation using the PVAF method
12.5
Q-matrix validation using the PVAF method
12.6
Q-matrix validation using the PVAF method
12.7
Modified PVAF approach and mesa plot
12.8
Stepwise method
12.9
Exercise
References
Published with bookdown
Handout for Cognitive Diagnosis Modeling
10.5
The Analytic Approach
The monte carlo approach has two limitations:
it assumes that the CDM fits the data well
it tends to be slow. We can also use some analytic methods to estimate classification accuracy