29.1 Anchoring Vignettes

  • Problem of interpersonal incomparability

  • Resources:

  • Help with 2 questions:

    • Different respondents understand the same question differently: Incomparaability in Survey Responses (“DIF”). Agreement on theoretical concept is almost nearly impossible.

    • How can we measure concepts that can only be defined by examples

  • Measure like usually, then subtract the incomparable portion. (i.e., using the assessment from the same respondents for a particular example/case to correct/adjust for the self-assessment).

  • Varying vignette assessments give us DIF (i.e., differential item functioning)

  • Since we created the anchors (i.e., examples), we know the true vignette assessments are fixed over respondents

29.1.1 Nonparametric method

Code the relative ranking of self-assessment in accordance to vignettes.

Inconsistencies would be considered ties.

Measurement Assumptions:

  • Response consistency: Each responder approaches the self-assessment and vignette categories in a same manner across questions.

  • Vignette Equivalence: For every vignette, the real level is the same for all respondents.

Used Ordered Probit to estimate.

29.1.2 Parametric method

The more vignettes that we have better identification. But it will introduce measurement errors.

  • Also use an ordinal probit model

  • with varying thresholds and a random effect.

# install.packages("anchors")
library(anchors)

# Example from the package's authors
data("freedom")
head(freedom)
a1 <-
    anchors(self ~ vign2 + vign3 + vign4 + vign5 + vign6, freedom, method = "C")
summary(a1)

References

Hopkins, D. J., and G. King. 2010. “Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability.” Public Opinion Quarterly 74 (2): 201–22. https://doi.org/10.1093/poq/nfq011.
KING, GARY, CHRISTOPHER J. L. MURRAY, JOSHUA A. SALOMON, and AJAY TANDON. 2004. “Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research.” American Political Science Review 98 (1): 191–207. https://doi.org/10.1017/s000305540400108x.
King, Gary, and Jonathan Wand. 2007. “Comparing Incomparable Survey Responses: Evaluating and Selecting Anchoring Vignettes.” Political Analysis 15 (1): 46–66. https://doi.org/10.1093/pan/mpl011.