Chapter 7 Model-data Fit Evaluation
We have a pool of different types of CDMs, and we can apply various CDMs to different data sets. However, we need to be cautious about fitting a data to CDMs, as different CDMs are different in nature(Chen et al., 2013) i.e.,
- Some CDMs are highly constrained model e.g., DINA, DINO
- Some are additive in nature e.g., Additive CDM (ACDM), Linear Logistic Model (LLM), R-RUM
- Some are more saturated in formulation e.g., log-linear CDM, GDINA.
- As suggested by Chen et al. (2013), we also need to be careful about the Q-matrix associated with our CDM model, as the Q-matrix potentially contributes to the model-data misfit.
Hence, we have a few challenges in performing CDM analysis to ensure validity of our inference.
- Choosing the right CDM: Which CDM should we use for our analysis?
- How well the model that we selected fit the data?
- Are we using the right Q-matrix? (We are planning to discuss about Q-matrix validation on October 24)
So in this session, we will try to explore more about:
- How to select the most appropriate CDM for our analysis.
- Assess the test-level model data fit.
Important: To make valid inference, we need to assess whether model can fit data adequately.
Model-data fit can be assessed both at test level and item level, and either for relative and absolute sense. In this class, we will focus on both absolute and relative fit measures for CDM models.
Relative fit
Relative fit can be defined as the process of identifying the best-fitting model in a set of competing models. We can check relative fit for both test level data and item level data.
- whether model A fits data better than model B for the whole test
- whether model A fits data better than model B for an item
Absolute fit
Absolute fit evaluation is the process of assessing how well a given model matches the actual data. It involves analyzing whether the model’s predictions closely align with the observed values.
whether a set of models fits the data
whether a model fits the data of an item