26.12 Assumptions

  • Parallel Trends: Difference between the treatment and control groups remain constant if there were no treatment.

    • should be used in cases where

      • you observe before and after an event

      • you have treatment and control groups

    • not in cases where

      • treatment is not random

      • confounders.

    • To support we use

  • Linear additive effects (of group/unit specific and time-specific):

  • No anticipation: There is no causal effect of the treatment before its implementation.

Possible issues

  • Estimate dependent on functional form:

    • When the size of the response depends (nonlinearly) on the size of the intervention, we might want to look at the the difference in the group with high intensity vs. low.
  • Selection on (time–varying) unobservables

    • Can use the overall sensitivity of coefficient estimates to hidden bias using Rosenbaum Bounds
  • Long-term effects

    • Parallel trends are more likely to be observed over shorter period (window of observation)
  • Heterogeneous effects

    • Different intensity (e.g., doses) for different groups.
  • Ashenfelter dip (Ashenfelter and Card 1985) (job training program participant are more likely to experience an earning drop prior enrolling in these programs)

  • Response to event might not be immediate (can’t be observed right away in the dependent variable)

  • Other factors that affect the difference in trends between the two groups (i.e., treatment and control) will bias your estimation.

  • Correlated observations within a group or time

  • Incidental parameters problems (Lancaster 2000): it’s always better to use individual and time fixed effect.

  • When examining the effects of variation in treatment timing, we have to be careful because negative weights (per group) can be negative if there is a heterogeneity in the treatment effects over time. Example: [Athey and Imbens (2022)](Borusyak, Jaravel, and Spiess 2021)(Goodman-Bacon 2021). In this case you should use new estimands proposed by @callaway2021difference(Clément De Chaisemartin and d’Haultfoeuille 2020), in the did package. If you expect lags and leads, see (L. Sun and Abraham 2021)

  • (Gibbons, Suárez Serrato, and Urbancic 2018) caution when we suspect the treatment effect and treatment variance vary across groups

26.12.2 Placebo Test

Procedure:

  1. Sample data only in the period before the treatment in time.
  2. Consider different fake cutoff in time, either
    1. Try the whole sequence in time

    2. Generate random treatment period, and use randomization inference to account for sampling distribution of the fake effect.

  3. Estimate the DiD model but with the post-time = 1 with the fake cutoff
  4. A significant DiD coefficient means that you violate the parallel trends! You have a big problem.

Alternatively,

  • When data have multiple control groups, drop the treated group, and assign another control group as a “fake” treated group. But even if it fails (i.e., you find a significant DiD effect) among the control groups, it can still be fine. However, this method is used under Synthetic Control

Code by theeffectbook.net

library(tidyverse)
library(fixest)

od <- causaldata::organ_donations %>%
    # Use only pre-treatment data
    dplyr::filter(Quarter_Num <= 3) %>%
    
    # Create fake treatment variables
    dplyr::mutate(
        FakeTreat1 = State == 'California' &
            Quarter %in% c('Q12011', 'Q22011'),
        FakeTreat2 = State == 'California' &
            Quarter == 'Q22011'
    )


clfe1 <- fixest::feols(Rate ~ FakeTreat1 | State + Quarter,
               data = od)
clfe2 <- fixest::feols(Rate ~ FakeTreat2 | State + Quarter,
               data = od)

fixest::etable(clfe1,clfe2)
#>                           clfe1            clfe2
#> Dependent Var.:            Rate             Rate
#>                                                 
#> FakeTreat1TRUE  0.0061 (0.0051)                 
#> FakeTreat2TRUE                  -0.0017 (0.0028)
#> Fixed-Effects:  --------------- ----------------
#> State                       Yes              Yes
#> Quarter                     Yes              Yes
#> _______________ _______________ ________________
#> S.E.: Clustered       by: State        by: State
#> Observations                 81               81
#> R2                      0.99377          0.99376
#> Within R2               0.00192          0.00015
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We would like the “supposed” DiD to be insignificant.

26.12.3 Assumption Violations

  1. Endogenous Timing

If the timing of units can be influenced by strategic decisions in a DID analysis, an instrumental variable approach with a control function can be used to control for endogeneity in timing.

  1. Questionable Counterfactuals

In situations where the control units may not serve as a reliable counterfactual for the treated units, matching methods such as propensity score matching or generalized random forest can be utilized. Additional methods can be found in Matching Methods.

26.12.4 Robustness Checks

  • Placebo DiD (if the DiD estimate \(\neq 0\), parallel trend is violated, and original DiD is biased):

    • Group: Use fake treatment groups: A population that was not affect by the treatment

    • Time: Redo the DiD analysis for period before the treatment (expected treatment effect is 0) (e.g., for previous year or period).

  • Possible alternative control group: Expected results should be similar

  • Try different windows (further away from the treatment point, other factors can creep in and nullify your effect).

  • Treatment Reversal (what if we don’t see the treatment event)

  • Higher-order polynomial time trend (to relax linearity assumption)

  • Test whether other dependent variables that should not be affected by the event are indeed unaffected.

    • Use the same control and treatment period (DiD \(\neq0\), there is a problem)
  • The triple-difference strategy involves examining the interaction between the treatment variable and the probability of being affected by the program, and the group-level participation rate. The identification assumption is that there are no differential trends between high and low participation groups in early versus late implementing countries.

References

Ashenfelter, Orley, and David Card. 1985. “Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs.” The Review of Economics and Statistics 67 (4): 648. https://doi.org/10.2307/1924810.
———. 2022. “Design-Based Analysis in Difference-in-Differences Settings with Staggered Adoption.” Journal of Econometrics 226 (1): 62–79.
Ban, Kyunghoon, and Desire Kedagni. 2022. “Generalized Difference-in-Differences Models: Robust Bounds.” arXiv Preprint arXiv:2211.06710.
Blundell, Richard, and Stephen Bond. 1998. “Initial Conditions and Moment Restrictions in Dynamic Panel Data Models.” Journal of Econometrics 87 (1): 115–43.
Borusyak, Kirill, Xavier Jaravel, and Jann Spiess. 2021. “Revisiting Event Study Designs: Robust and Efficient Estimation.” arXiv Preprint arXiv:2108.12419.
De Chaisemartin, Clément, and Xavier d’Haultfoeuille. 2020. “Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects.” American Economic Review 110 (9): 2964–96.
Gibbons, Charles E, Juan Carlos Suárez Serrato, and Michael B Urbancic. 2018. “Broken or Fixed Effects?” Journal of Econometric Methods 8 (1): 20170002.
Goodman-Bacon, Andrew. 2021. “Difference-in-Differences with Variation in Treatment Timing.” Journal of Econometrics 225 (2): 254–77.
Heckman, James J., and Jeffrey A. Smith. 1999. “The Pre-Programme Earnings Dip and the Determinants of Participation in a Social Programme. Implications for Simple Programme Evaluation Strategies.” The Economic Journal 109 (457): 313–48. https://doi.org/10.1111/1468-0297.00451.
———. 2021. “On the Use of Two-Way Fixed Effects Regression Models for Causal Inference with Panel Data.” Political Analysis 29 (3): 405–15.
Jepsen, Christopher, Kenneth Troske, and Paul Coomes. 2014. “The Labor-Market Returns to Community College Degrees, Diplomas, and Certificates.” Journal of Labor Economics 32 (1): 95–121. https://doi.org/10.1086/671809.
Kahn-Lang, Ariella, and Kevin Lang. 2020. “The Promise and Pitfalls of Differences-in-Differences: Reflections on 16 and Pregnant and Other Applications.” Journal of Business & Economic Statistics 38 (3): 613–20.
Lancaster, Tony. 2000. “The Incidental Parameter Problem Since 1948.” Journal of Econometrics 95 (2): 391–413.
Li, Xia, Christopher Gan, and Baiding Hu. 2011. “The Welfare Impact of Microcredit on Rural Households in China.” The Journal of Socio-Economics 40 (4): 404–11. https://doi.org/10.1016/j.socec.2011.04.012.
———. 2017b. “Online Reputation Management: Estimating the Impact of Management Responses on Consumer Reviews.” Marketing Science 36 (5): 645–65. https://doi.org/10.1287/mksc.2017.1043.
Rambachan, Ashesh, and Jonathan Roth. 2023. “A More Credible Approach to Parallel Trends.” Review of Economic Studies, rdad018.
Roth, Jonathan. 2022. “Pretest with Caution: Event-Study Estimates After Testing for Parallel Trends.” American Economic Review 4 (3): 305–22.
Roth, Jonathan, and Pedro HC Sant’Anna. 2023. “When Is Parallel Trends Sensitive to Functional Form?” Econometrica 91 (2): 737–47.
Ryan, Andrew M, Evangelos Kontopantelis, Ariel Linden, and James F Burgess Jr. 2019. “Now Trending: Coping with Non-Parallel Trends in Difference-in-Differences Analysis.” Statistical Methods in Medical Research 28 (12): 3697–3711.
Sun, Liyang, and Sarah Abraham. 2021. “Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects.” Journal of Econometrics 225 (2): 175–99.
Wolfers, Justin. 2003. “Is Business Cycle Volatility Costly? Evidence from Surveys of Subjective Well-Being.” International Finance 6 (1): 1–26.