Week 5 Validity

A test has validity if it measures what it purports to measure. There are three types of validities; content validity, criterion-related validity, and construct validity. In this chapter, we will focus on the criterion-validity and demonstrate how to obtain a validity coefficient \(\rho_{XY}\).

5.1 read.table

Let’s download the two datasets from the following link and save them in your local computer:

• test1.txt
• test2.txt

Before importing a data file from a local directory, you need to set the working directory of RStudio. On the View pane, click on ... and set the directory where your data file is located.

Then click on More -> Set as Working Directory.

Then RStudio will be able to import data sets located under your working directory. Let’s import the two data files using read.table() function.

test1 <- read.table("test1.txt")
test2 <- read.table("test2.txt")

Let’s check the number of rows and columns.

nrow(test1)

## [1] 100

nrow(test2)

## [1] 100

ncol(test1)

## [1] 20

ncol(test2)

## [1] 20

Both data sets contain responses from 100 subjects and 20 items.

5.2 Criterion Related Validity

Suppose that Test1 is developed to measure some current trait, and it is used to predict a future criterion. The future criterion is measured by another test, Test2.

In this case, a subject’s total score on Test 1 is the “Test score”, \(X\). A subject’s total score on Test 2 is the “Criterion score”, \(Y\). Then, the correlation between observed score on Test 1 (\(X\)) and observed score on Test 2 (\(Y\)) is a measure of how well Test 1 predicts the criterion, namely the criterion-related validity.

Total scores of the two tests, \(X\) and \(Y\), can be obtained by rowSums().

total_X <- rowSums(test1)
total_Y <- rowSums(test2)

Then the Pearson correlation between two variables, \(X\) and \(Y\), can be obtained by the cor() function.

rho_xy <- cor(total_X, total_Y)

The validity coefficient is 0.6024.

5.2.1 Correction for attenuation

The validity \(\rho_{XY}\) of a test is limited by its reliability \(\rho_{XX'}\). That is, the validity of \(X\) is attenuated by its reliability unless \(X\) has perfect reliability.

\[\rho_{XY} \leq \rho_{XT} = \sqrt{\rho_{XX'}}\]

If both tests are perfectly reliable, then the validity of Test 1 will be equal to the correlation between the true scores on the two tests.

The correlation between the true scores on the two tests, \(T_X\) and \(T_Y\) is:

\[\rho_{T_X T_Y} = \frac{\rho_{XY}}{\sqrt{\rho_{XX'}} \sqrt{\rho_{YY'}}}\]

Using the above formula, we can obtain the true-score correlation between the two sets, after correcting for attenuation.

First, let’s calculate the coefficient \(\alpha\) of the two tests as estimates of \(\rho_{XX'}\) and \(\rho_{YY'}\).

coeff_alpha <- function(responses){
  # Get number of items (N) and individuals
  n_items <- ncol(responses)
  n_persons <- nrow(responses)
  # Get individual total scores
  x <- rowSums(responses)
  # Get observed-score variance of whole test (X)
  var_x <- var(x) * (n_persons - 1) / n_persons
  # Get observed-score variance of each item (Y_j)
  var_y <- numeric(n_items)
  for (i in 1:n_items) {
    var_y[i] <- var(responses[, i]) * (n_persons - 1) / n_persons
  }
  # Apply the alpha formula
  alpha <- (n_items / (n_items - 1)) * (1 - sum(var_y) / var_x)
  
  alpha
}

rho_xx <- coeff_alpha(test1)
rho_xx

## [1] 0.6061799

rho_yy <- coeff_alpha(test2)
rho_yy

## [1] 0.6439105

Now we can obtain the true score correlation.

rho_txty <- rho_xy / (sqrt(rho_xx) * sqrt(rho_yy))
rho_txty

## [1] 0.9641909