7.8 Examples & Further reading

  • Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments (Shadish, Clark, and Steiner 2008; see also Steiner et al. 2010)
    • “A key justification for using nonrandomized experiments is that, with proper adjustment, their results can well approximate results from randomized experiments. This hypothesis has not been consistently supported by empirical studies; however, previous methods used to study this hypothesis have confounded assignment method with other study features. To avoid these confounding factors, this study randomly assigned participants to be in a randomized experiment or a nonrandomized experiment.”
  • A hard unsolved problem? Post-treatment bias in big social science questions (G. King 2010b)
  • How conditioning on post-treatment variables can ruin your experiment and what to do about it (Montgomery and Nyhan 2016)
  • Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effect (Acharya, Blackwell, and Sen 2015)
  • Does Regression Produce Representative Estimates of Causal Effects? (Aronow and Samii 2015)
  • Statistical Models and Shoe Leather (Freedman 1991) (e.g. Snow & Cholera example)

References

Acharya, Avidit, Matthew Blackwell, and Maya Sen. 2015. “Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects.” The American Political Science Review.

Aronow, Peter M, and Cyrus Samii. 2015. “Does Regression Produce Representative Estimates of Causal Effects?” American Journal of Political Science.

Freedman, David A. 1991. “Statistical Models and Shoe Leather.” Sociological Methodology 21 (2): 291–313.

King, Gary. 2010b. “A Hard Unsolved Problem? Post-Treatment Bias in Big Social Science Questions.” In Hard Problems in Social Science” Symposium, Harvard University.

Montgomery, Jacob M, and Brendan Nyhan. 2016. “How Conditioning on Post-Treatment Variables Can Ruin Your Experiment and What to Do About It.”

Shadish, William R, M H Clark, and Peter M Steiner. 2008. “Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments.” J. Am. Stat. Assoc. 103 (484): 1334–44.

Steiner, Peter M, Thomas D Cook, William R Shadish, and M H Clark. 2010. “The Importance of Covariate Selection in Controlling for Selection Bias in Observational Studies.” Psychol. Methods 15 (3): 250–67.