7.3 Relative Model-Data Fit at Test Level (Cont’d)

Let $P$ be the number of model parameters, several information criteria can be defined:

Akaike (1974) Information Criterion (AIC) adjusts the -2 log likelihood by twice the number of parameters in the model: $$AIC=-2\log L(\mathbf{Y})+2P$$

Schwarz (1978) Bayesian Criterion (BIC) has a stronger penalty than the AIC for overparametrized models, and adjusts the -2 log likelihood by the number of parameters times the log of the number of cases. It is also known as the Bayesian Information Criterion. $$BIC=-2\log L(\mathbf{Y})+P\log(N)$$ Bozdogan (1987) Consistent Akaike’s Information Criterion (CAIC) has a stronger penalty than the AIC for overparametrized models, and adjusts the -2 log likelihood by the number of parameters times one plus the log of the number of cases. As the sample size increases, the CAIC converges to the BIC.

$$CAIC=-2\log L(\mathbf{Y})+P\big[\log(N)+1\big]$$

The sample-size-adjusted BIC (SABIC) is proposed by Sclove (1987) to reduce the penalty in BIC. $$SABIC=-2\log L(\mathbf{Y})+P\bigg[\log(\frac{N+2}{24})\bigg]$$

7.3.1 Calculating number of model paraters for CDMs

Number of model parameters $(P)$ can be calculated in different ways. For instance,

• For DINA model, $P = 2J+2^k-1$, here $J$ is the number of items.

• For ACDM model, $\sum_{j=1}^J K_j^*+J+2^k-1$, here $K_j^*$ is the reduced attribute profiles.

• For Saturated models (e.g., GDINA), $\sum_{j=1}^j 2^{K_j^k}+2^k-1$

We can also identify the number of model parameters from GDINA R-package, as demonstrated in Model-Identifiability class.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705

Bozdogan, H. (1987). Model selection and akaike’s information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370. https://doi.org/10.1007/BF02294361

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. https://doi.org/10.1214/aos/1176344136

Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333–343. https://doi.org/10.1007/BF02294360