6 Tutorial 6: Data Collection: Testing Intercoder Reliability

After working through Tutorial 6, you’ll…

  • understand how to calculate intercoder reliability values for your manual content analysis

6.1 Calculating Intercoder Reliability in R

For this tutorial, we’ll use the data sets “Codings_Coder1.csv” and “Codings_Coder2.csv” (via Moodle/Data for R).

The data contains codings by two coders who both coded the same N = 50 social media posts. Both Excel files include the following four variables…

  • the ID of each unit of analysis coded by both coders: PostID
  • the ID of each coder coding each post: CoderID
  • the binary variable V1 describing whether or not the post contains negative sentiment (0 = no, 1 = yes): V1
  • the categorical variable V2 describing which topic each post deals with (1 = “Society”, 2 = “Economy”, 3 = “International Politics”, 4 = “National Politics”): V2

First, we read both files into R.

Using the arrange() function, we make sure that the posts are sorted by ID:

library("tidyverse")
library("tidycomm")
coder1 <- read.csv2("Codings_Coder1.csv", header = TRUE) %>%
  arrange(PostID)
coder2 <- read.csv2("Codings_Coder2.csv", header = TRUE) %>%
  arrange(PostID)

Let’s have a look at the data:

head(coder1)
##   PostID CoderID V1 V2
## 1   ID01       1  0  1
## 2   ID02       1  0  2
## 3   ID03       1  0  4
## 4   ID04       1  0  4
## 5   ID05       1  1  4
## 6   ID06       1  0  4
head(coder2)
##   PostID CoderID V1 V2
## 1   ID01       2  0  1
## 2   ID02       2  0  2
## 3   ID03       2  0  4
## 4   ID04       2  0  4
## 5   ID05       2  1  3
## 6   ID06       2  0  4

We’ll now combine codings by both coders into a single data set using the bind_rows() function. This basically stacks both dataframes - coder1 and coder2 - on top of each other and into a single data frame:

intercoder.codings <- bind_rows(coder1, coder2)
head(intercoder.codings)
##   PostID CoderID V1 V2
## 1   ID01       1  0  1
## 2   ID02       1  0  2
## 3   ID03       1  0  4
## 4   ID04       1  0  4
## 5   ID05       1  1  4
## 6   ID06       1  0  4

Now, we’ll calculate intercoder reliability coefficients.

To do so, we use the fantastic “tidycomm” package developed by Dr. Julian Unkel. If you’re interested, Julian provides a Vignette with even more info on how to use tidycomm for intercoder reliability calculation here.

Using the test_icr() function and providing arguments for…

  • the variable containing the unit identifiers, i.e., the unique ID of each post: PostID
  • the variable containing the coder identifiers, i.e., the unique ID of each coder: CoderID

Important: By default, test_icr() treats all variables as being on a nominal level. Should variables be scaled differently, e.g., on an ordinal or a metric scale, we could define this using the levels argument.

intercoder.codings %>% 
  test_icr(unit_var = PostID,
           coder_var = CoderID)
## # A tibble: 2 x 8
##   Variable n_Units n_Coders n_Categories Level   Agreement Holstis_CR Krippendorffs_Alpha
##   <chr>      <int>    <int>        <int> <chr>       <dbl>      <dbl>               <dbl>
## 1 V1            50        2            2 nominal      0.84       0.84               0.588
## 2 V2            50        2            4 nominal      0.84       0.84               0.788

What is the result?

  • Our binary variables V1 seems to not reach sufficient reliability scores: Holsti’s CR lies at .84, but Krippendorff’s Alpha is far below an acceptable threshold with .59.
  • Our binary variables V2 seems to reach somehwat sufficient reliability score: Holsti’s CR lies at .84 and Krippendorff’s Alpha reaches .79, which is close to an agreeable level of .8

6.2 Take Aways

  • Calculating intercoder reliability:
    • calculate different coefficients viy the tidycomm package: test_icr()

6.3 More tutorials on this

You still have questions? The following tutorials & papers can help you with that:

Let’s keep going: Tutorial 7: Drawing Random Samples.