## 24.2 Specification Checks

### 24.2.1 Balance Checks

• Also known as checking for Discontinuities in Average Covariates

• Null Hypothesis: The average effect of covariates on pseudo outcomes (i.e., those qualitatively cannot be affected by the treatment) is 0.

• If this hypothesis is rejected, you better have a good reason to why because it can cast serious doubt on your RD design.

### 24.2.2 Sorting/Bunching/Manipulation

• Also known as checking for A Discontinuity in the Distribution of the Forcing Variable

• Also known as clustering or density test

• Formal test is McCrary sorting test or

• Since human subjects can manipulate the running variable to be just above or below the cutoff (assuming that the running variable is manipulable), especially when the cutoff point is known in advance for all subjects, this can result in a discontinuity in the distribution of the running variable at the cutoff (i.e., we will see “bunching” behavior right before or after the cutoff)>

• People would like to sort into treatment if it’s desirable. The density of the running variable would be 0 just below the threshold

• People would like to be out of treatment if it’s undesirable

• proposes a density test (i.e., a formal test for manipulation of the assignment variable).

• $$H_0$$: The continuity of the density of the running variable (i.e., the covariate that underlies the assignment at the discontinuity point)

• $$H_a$$: A jump in the density function at that point

• Even though it’s not a requirement that the density of the running must be continuous at the cutoff, but a discontinuity can suggest manipulations.

• offers a guide to know when you should warrant the manipulation

• Usually it’s better to know your research design inside out so that you can suspect any manipulation attempts.

• We would suspect the direction of the manipulation. And typically, it’s one-way manipulation. In cases where we might have both ways, theoretically they would cancel each other out.
• We could also observe partial manipulation in reality (e.g., when subjects can only imperfectly manipulate). But typically, as we treat it like fuzzy RD, we would not have identification problems. But complete manipulation would lead to serious identification issues.

• Remember: even in cases where we fail to reject the null hypothesis for the density test, we could not rule out completely that identification problem exists (just like any other hypotheses)

• Bunching happens when people self-select to a specific value in the range of a variable (e.g., key policy thresholds).

• Review paper

• This test can only detect manipulation that changes the distribution of the running variable. If you can choose the cutoff point or you have 2-sided manipulation, this test will fail to detect it.

• Histogram in bunching is similar to a density curve (we want narrower bins, wider bins bias elasticity estimates)

• We can also use bunching method to study individuals’ or firm’s responsiveness to changes in policy.

• Under RD, we assume that we don’t have any manipulation in the running variable. However, bunching behavior is a manipulation by firms or individuals. Thus, violating this assumption.

• Bunching can fix this problem by estimating what densities of individuals would have been without manipulation (i.e., manipulation-free counterfactual).

• The fraction of persons who manipulated is then calculated by comparing the observed distribution to manipulation-free counterfactual distributions.

• Under RD, we do not need this step because the observed and manipulation-free counterfactual distributions are assumed to be the same. RD assume there is no manipulation (i.e., assume the manipulation-free counterfactual distribution)

When running variable and outcome variable are simultaneously determined, we can use a modified RDD estimator to have consistent estimate.

• Assumptions:

• Manipulation is one-sided: People move one way (i.e., either below the threshold to above the threshold or vice versa, but not to or away the threshold), which is similar to the monotonicity assumption under instrumental variable 33.1.3.1

• Manipulation is bounded (also known as regularity assumption): so that we can use people far away from this threshold to derive at our counterfactual distribution [Blomquist et al. (2021)]

Steps:

1. Identify the window in which the running variable contains bunching behavior. We can do this step empirically based on Bosch, Dekker, and Strohmaier (2020). Additionally robustness test is needed (i.e., varying the manipulation window).
2. Estimate the manipulation-free counterfactual
3. Calculating the standard errors for inference can follow where we bootstrap re-sampling residuals in the estimation of the counts of individuals within bins (large data can render this step unnecessary).

If we pass the bunching test, we can move on to the Placebo Test

McCrary (2008) test

A jump in the density at the threshold (i.e., discontinuity) hold can serve as evidence for sorting around the cutoff point

library(rdd)

# you only need the runing variable and the cutoff point

# Example by the package's authors
#No discontinuity
x<-runif(1000,-1,1)
DCdensity(x,0)

#> [1] 0.6126802

#Discontinuity
x<-runif(1000,-1,1)
x<-x+2*(runif(1000,-1,1)>0&x<0)
DCdensity(x,0)

#> [1] 0.0008519227

Cattaneo, Idrobo, and Titiunik (2019) test

library(rddensity)

# Example by the package's authors
# Continuous Density
set.seed(1)
x <- rnorm(2000, mean = -0.5)
rdd <- rddensity(X = x, vce = "jackknife")
summary(rdd)
#>
#> Manipulation testing using local polynomial density estimation.
#>
#> Number of obs =       2000
#> Model =               unrestricted
#> Kernel =              triangular
#> BW method =           estimated
#> VCE method =          jackknife
#>
#> c = 0                 Left of c           Right of c
#> Number of obs         1376                624
#> Eff. Number of obs    354                 345
#> Order est. (p)        2                   2
#> Order bias (q)        3                   3
#> BW est. (h)           0.514               0.609
#>
#> Method                T                   P > |T|
#> Robust                -0.6798             0.4966
#>
#>
#> P-values of binomial tests (H0: p=0.5).
#>
#> Window Length / 2          <c     >=c    P>|T|
#> 0.036                      28      20    0.3123
#> 0.072                      46      39    0.5154
#> 0.107                      68      59    0.4779
#> 0.143                      94      79    0.2871
#> 0.179                     122     103    0.2301
#> 0.215                     145     130    0.3986
#> 0.250                     163     156    0.7370
#> 0.286                     190     176    0.4969
#> 0.322                     214     200    0.5229
#> 0.358                     249     218    0.1650

# you have to specify your own plot (read package manual)

### 24.2.3 Placebo Tests

• Also known as Discontinuities in Average Outcomes at Other Values

• We should not see any jumps at other values (either $$X_i <c$$ or $$X_i \ge c$$)

• Use the same bandwidth you use for the cutoff, and move it along the running variable: testing for a jump in the conditional mean of the outcome at the median of the running variable.
• Also known as falsification checks

• Before and after the cutoff point, we can run the placebo test to see whether X’s are different).

• The placebo test is where you expect your coefficients to be not different from 0.

• This test can be used for

• Testing no discontinuity in predetermined variables:

• Testing other discontinuities

• Placebo outcomes: we should see any changes in other outcomes that shouldn’t have changed.

• Inclusion and exclusion of covariates: RDD parameter estimates should not be sensitive to the inclusion or exclusion of other covariates.

• This is analogous to Experimental Design where we cannot only test whether the observables are similar in both treatment and control groups (if we reject this, then we don’t have random assignment), but we cannot test unobservables.

Balance on observable characteristics on both sides

$Z_i = \alpha_0 + \alpha_1 f(x_i) + [I(x_i \ge c)] \alpha_2 + [f(x_i) \times I(x_i \ge c)]\alpha_3 + u_i$

where

• $$x_i$$ is the running variable

• $$Z_i$$ is other characteristics of people (e.g., age, etc)

Theoretically, $$Z_i$$ should no be affected by treatment. Hence, $$E(\alpha_2) = 0$$

Moreover, when you have multiple $$Z_i$$, you typically have to simulate joint distribution (to avoid having significant coefficient based on chance).

The only way that you don’t need to generate joint distribution is when all $$Z_i$$’s are independent (unlikely in reality).

Under RD, you shouldn’t have to do any Matching Methods. Because just like when you have random assignment, there is no need to make balanced dataset before and after the cutoff. If you have to do balancing, then your RD assumptions are probably wrong in the first place.

### 24.2.4 Sensitivity to Bandwidth Choice

• Methods for bandwidth selection

• Data driven: cross validation

• Conservative approach:

• The objective is to minimize the mean squared error between the estimated and actual treatment effects.

• Then, we need to see how sensitive our results will be dependent on the choice of bandwidth.

• In some cases, the best bandwidth for testing covariates may not be the best bandwidth for treating them, but it may be close.

# find optimal bandwidth by Imbens-Kalyanaraman
rdd::IKbandwidth(running_var,
outcome_var,
cutpoint = "",
kernel = "triangular") # can also pick other kernels

### 24.2.5 Manipulation Robust Regression Discontinuity Bounds

• McCrary (2008) linked density jumps at cutoffs in RD studies to potential manipulation.

• If no jump is detected, researchers proceed with RD analysis; if detected, they halt using the cutoff for inference.

• Some studies use the “doughnut-hole” method, excluding near-cutoff observations and extrapolating, which contradicts RD principles.

• False negative could be due to a small sample size and can lead to biased estimates, as units near the cutoff may still differ in unobserved ways.

• Even correct rejections of no manipulation may overlook that the data can still be informative despite modest manipulation.

• Gerard, Rokkanen, and Rothe (2020) introduces a systematic approach to handle potentially manipulated variables in RD designs, addressing both concerns.

• The model introduces two types of unobservable units in RD designs:

• always-assigned units, which are always on one side of the cutoff,

• potentially-assigned units, which fit traditional RD assumptions.

• The standard RD model is a subset of this broader model, which assumes no always-assigned units.
• Identifying assumption: manipulation occurs through one-sided selection.

• The approach does not make a binary decision on manipulation in RD designs but assesses its extent and worst-case impact.

Two steps are used:

1. Determining the proportion of always-assigned units using the discontinuity at the cutoff
2. Bounding treatment effects based on the most extreme feasible outcomes for these units.
• For sharp RD designs, bounds are established by trimming extreme outcomes near the cutoff; for fuzzy designs, the process involves more complex adjustments due to additional model constraints.

• Extensions of the study use covariate information and economic behavior assumptions to refine these bounds and identify covariate distributions among unit types at the cutoff.

Setup

Independent data points $$(X_i, Y_i, D_i)$$, where $$X_i$$ is the running variable, $$Y_i$$ is the outcome, and $$D_i$$ indicates treatment status (1 if treated, 0 otherwise). Treatment is assigned based on $$X_i \geq c$$.

The design is sharp if $$D_i = I(X_i \geq c)$$ and fuzzy otherwise.

The population is divided into:

• Potentially-assigned units ($$M_i = 0$$): Follow the standard RD framework, with potential outcomes $$Y_i(d)$$ and potential treatment states $$D_i(x)$$.

• Always-assigned units ($$M_i = 1$$): These units do not require potential outcomes or states, and always have $$X_i$$ values beyond the cutoff.

Assumptions

1. Local Independence and Continuity:
• $$P(D = 1|X = c^+, M = 0) > P(D = 1|X = c^-, M = 0)$$
• No defiers: $$P(D^+ \geq D^-|X = c, M = 0) = 1$$
• Continuity in potential outcomes and states at $$c$$.
• $$F_{X|M=0}(x)$$ is differentiable at $$c$$, with a positive derivative.
2. Smoothness of the Running Variable among Potentially-Assigned Units:
• The derivative of $$F_{X|M=0}(x)$$ is continuous at $$c$$.
3. Restrictions on Always-Assigned Units:
• $$P(X \geq c|M = 1) = 1$$ and $$F_{X|M=1}(x)$$ is right-differentiable (or left-differentiable) at $$c$$.
• This (local) one-sided manipulation assumption allows identification of the proportion of always-assigned units among all units close to the cutoff.

When always-assigned unit exist, the RD design is fuzzy because we have

1. Treated and untreated units among the potentially-assigned (below and above the cutoff)
2. Always-assigned units (above the cutoff).

Causal Effects of Interest

causal effects among potentially-assigned units:

$\Gamma = E[Y(1) - Y(0) | X = c, D^+ > D^-, M = 0]$

This parameter represents the local average treatment effect (LATE) for the subgroup of “compliers”—units that receive treatment if and only if their running variable $$X_i$$ exceeds a certain cutoff.

The parameter $$\Gamma$$ captures the causal effect of changes in the cutoff level on treatment status among potentially-assigned compliers.

RD designs with a manipulated running variable “Doughnut-Hole” RD Designs:
• Focuses on actual observations at the cutoff, not hypothetical true values.
• Provides a direct and observable estimate of causal effects, without reliance on hypothetical constructs.
• Exclude observations around the cutoff and use extrapolation from the trends outside this excluded range to infer causal effects at the cutoff
• Assumes a hypothetical population existing in a counterfactual scenario without manipulation.
• Requires strong assumptions about the nature of manipulation and the minimal impact of extrapolation biases.

Identification of $$\tau$$ in RD Designs

• Identification challenges arise due to the inability to distinguish always-assigned from potentially-assigned units, thus Γ is not point identified. We establish sharp bounds on Γ

• These bounds are supported by the stochastic dominance of the potential outcome CDFs over observed distributions.

Unit Types and Notation:

• $$C_0$$: Potentially-assigned compliers.
• $$A_0$$: Potentially-assigned always-takers.
• $$N_0$$: Potentially-assigned never-takers.
• $$T_1$$: Always-assigned treated units.
• $$U_1$$: Always-assigned untreated units.

The measure $$\tau$$ , representing the proportion of always-assigned units near the cutoff, is point identified by the discontinuity in the observed running variable density $$f_X$$ at the cutoff

Sharp RD:

• Units to the left of the cutoff are potentially assigned units. The distribution of their observed outcomes ($$Y$$) are the outcomes $$Y(0)$$ of potentially-assigned compliers ($$C_0$$) at the cutoff.

• To determine the bounds on the treatment effect ($$\Gamma$$), we need to assess the distribution of treated outcomes ($$Y(1)$$) for the same potentially-assigned compliers at the cutoff.

• Information regarding the treated outcomes ($$Y(1)$$) comes exclusively from the subpopulation of treated units, which includes both potentially-assigned compliers ($$C_0$$) and those always assigned units ($$T_1$$).

• With $$\tau$$ point identified, we can estimate sharp bounds on $$\Gamma$$.

Fuzzy RD:

Note: Table on page 848
Subpopulation Types of units
$$X = c^+, D = 1$$ $$C_0, A_0, T_1$$
$$X = c^-, D = 1$$ $$A_0$$
$$X= c^+, D = 0$$ $$N_0, U_1$$
$$X = c^-, D = 0$$ $$C_0, N_0$$
• Unit Types and Combinations: There are five distinct unit types and four combinations of treatment assignments and decisions relevant to the analysis. These distinctions are important because they affect how potential outcomes are analyzed and bounded.

• Outcome Distributions: The analysis involves estimating the distribution of potential outcomes (both treated and untreated) among potentially-assigned compliers at the cutoff.

• Three-Step Process:

1. Potential Outcomes Under Treatment: Bounds on the distribution of treated outcomes are determined using data from treated units.

2. Potential Outcomes Under Non-Treatment: Bounds on the distribution of untreated outcomes are derived using data from untreated units.

3. Bounds on Parameters of Interest: Using the bounds from the first two steps, sharp upper and lower bounds on the local average treatment effect are derived.

• Extreme Value Consideration: The bounds for treatment effects are based on “extreme” scenarios under worst-case assumptions about the distribution of potential outcomes, making them sharp but empirically relevant within the data constraints.

Extensions:

• Quantile Treatment Effects: alternative to average effects by focusing on different quantiles of the outcome distribution, which are less affected by extreme values.

• Applicability to Discrete Outcomes

• Behavioral Assumptions Impact: Assuming a high likelihood of treatment among always-assigned units can narrow the bounds of treatment effects by refining the analysis of potential outcomes.

• Utilization of Covariates: Incorporating covariates measured prior to treatment can refine the bounds on treatment effects and help target policies by identifying covariate distributions among different unit types.

Notes:

• Quantile Treatment Effects (QTEs): QTE bounds are less sensitive to the tails of the outcome distribution, making them tighter than ATE bounds.

• Inference on ATEs is sensitive to the extent of manipulation, with confidence intervals widening significantly with small degrees of assumed manipulation.

• Inference on QTEs is less affected by manipulation, remaining meaningful even with larger degrees of manipulation.

• Alternative Inference Strategy when manipulation is believed to be unlikely. Try different hypothetical values of $$\tau$$

devtools::install_github("francoisgerard/rdbounds/R")
library(formattable)
library(data.table)
library(rdbounds)
set.seed(123)
df <- rdbounds_sampledata(10000, covs = FALSE)
#> [1] "True tau: 0.117999815082062"
#> [1] "True treatment effect on potentially-assigned: 2"
#> [1] "True treatment effect on right side of cutoff: 2.35399944524618"
#>            x        y treatment
#> 1 -1.2532616 3.489563         0
#> 2 -0.5146925 3.365232         0
#> 3  3.4853777 6.193533         0
#> 4  0.1576616 8.820440         1
#> 5  0.2890962 4.791972         0
#> 6  3.8350019 7.316907         0

rdbounds_est <-
rdbounds(
y = df$y, x = df$x,
# covs = as.factor(df$cov), treatment = df$treatment,
c = 0,
discrete_x = FALSE,
discrete_y = FALSE,
bwsx = c(.2, .5),
bwy = 1,

# for median effect use
# type = "qte",
# percentiles = .5,

kernel = "epanechnikov",
orders = 1,
evaluation_ys = seq(from = 0, to = 15, by = 1),
refinement_A = TRUE,
refinement_B = TRUE,
right_effects = TRUE,
yextremes = c(0, 15),
num_bootstraps = 5
)
#> [1] "The proportion of always-assigned units just to the right of the cutoff is estimated to be 0.04209"
#> [1] "2024-05-13 19:12:33 Estimating CDFs for point estimates"
#> [1] "2024-05-13 19:12:33 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2024-05-13 19:12:35 Estimating CDFs with nudged tau (tau_star)"
#> [1] "2024-05-13 19:12:35 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2024-05-13 19:12:38 Beginning parallelized output by bootstrap.."
#> [1] "2024-05-13 19:12:42 Computing Confidence Intervals"
#> [1] "2024-05-13 19:12:51 Time taken:0.3 minutes"
rdbounds_summary(rdbounds_est, title_prefix = "Sample Data Results")
#> [1] "Time taken: 0.3 minutes"
#> [1] "Sample size: 10000"
#> [1] "Local Average Treatment Effect:"
#> $tau_hat #> [1] 0.04209028 #> #>$tau_hat_CI
#> [1] 0.1671043 0.7765031
#>
#> $takeup_increase #> [1] 0.7521208 #> #>$takeup_increase_CI
#> [1] 0.7065353 0.7977063
#>
#> $TE_SRD_naive #> [1] 1.770963 #> #>$TE_SRD_naive_CI
#> [1] 1.541314 2.000612
#>
#> $TE_SRD_bounds #> [1] 1.569194 1.912681 #> #>$TE_SRD_CI
#> [1] -0.1188634  3.5319468
#>
#> $TE_SRD_covs_bounds #> [1] NA NA #> #>$TE_SRD_covs_CI
#> [1] NA NA
#>
#> $TE_FRD_naive #> [1] 2.356601 #> #>$TE_FRD_naive_CI
#> [1] 1.995430 2.717772
#>
#> $TE_FRD_bounds #> [1] 1.980883 2.362344 #> #>$TE_FRD_CI
#> [1] -0.6950823  4.6112538
#>
#> $TE_FRD_bounds_refinementA #> [1] 1.980883 2.357499 #> #>$TE_FRD_refinementA_CI
#> [1] -0.6950823  4.6112538
#>
#> $TE_FRD_bounds_refinementB #> [1] 1.980883 2.351411 #> #>$TE_FRD_refinementB_CI
#> [1] -0.6152215  4.2390830
#>
#> $TE_FRD_covs_bounds #> [1] NA NA #> #>$TE_FRD_covs_CI
#> [1] NA NA
#>
#> $TE_SRD_CIs_manipulation #> [1] NA NA #> #>$TE_FRD_CIs_manipulation
#> [1] NA NA
#>
#> $TE_SRD_right_bounds #> [1] 1.376392 2.007746 #> #>$TE_SRD_right_CI
#> [1] -5.036752  5.889137
#>
#> $TE_FRD_right_bounds #> [1] 1.721121 2.511504 #> #>$TE_FRD_right_CI
#> [1] -6.663269  7.414185
rdbounds_est_tau <-
rdbounds(
y = df$y, x = df$x,
# covs = as.factor(df$cov), treatment = df$treatment,
c = 0,
discrete_x = FALSE,
discrete_y = FALSE,
bwsx = c(.2, .5),
bwy = 1,
kernel = "epanechnikov",
orders = 1,
evaluation_ys = seq(from = 0, to = 15, by = 1),
refinement_A = TRUE,
refinement_B = TRUE,
right_effects = TRUE,
potential_taus = c(.025, .05, .1, .2),
yextremes = c(0, 15),
num_bootstraps = 5
)
#> [1] "The proportion of always-assigned units just to the right of the cutoff is estimated to be 0.04209"
#> [1] "2024-05-13 19:12:52 Estimating CDFs for point estimates"
#> [1] "2024-05-13 19:12:52 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2024-05-13 19:12:53 Estimating CDFs with nudged tau (tau_star)"
#> [1] "2024-05-13 19:12:53 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2024-05-13 19:12:56 Beginning parallelized output by bootstrap.."
#> [1] "2024-05-13 19:13:02 Estimating CDFs with fixed tau value of: 0.025"
#> [1] "2024-05-13 19:13:02 Estimating CDFs with fixed tau value of: 0.05"
#> [1] "2024-05-13 19:13:02 Estimating CDFs with fixed tau value of: 0.1"
#> [1] "2024-05-13 19:13:02 Estimating CDFs with fixed tau value of: 0.2"
#> [1] "2024-05-13 19:13:03 Beginning parallelized output by bootstrap x fixed tau.."
#> [1] "2024-05-13 19:13:09 Computing Confidence Intervals"
#> [1] "2024-05-13 19:13:19 Time taken:0.46 minutes"
causalverse::plot_rd_aa_share(rdbounds_est_tau) # For SRD (default)

# causalverse::plot_rd_aa_share(rdbounds_est_tau, rd_type = "FRD")  # For FRD

### References

Aronow, Peter M, Jonathon Baron, and Lauren Pinson. 2019. “A Note on Dropping Experimental Subjects Who Fail a Manipulation Check.” Political Analysis 27 (4): 572–89.
Bajari, Patrick, Han Hong, Minjung Park, and Robert Town. 2011. “Regression Discontinuity Designs with an Endogenous Forcing Variable and an Application to Contracting in Health Care.” National Bureau of Economic Research.
Bertanha, Marinho, Andrew H McCallum, and Nathan Seegert. 2021. “Better Bunching, Nicer Notching.” arXiv Preprint arXiv:2101.01170.
Blomquist, Sören, Whitney K Newey, Anil Kumar, and Che-Yuan Liang. 2021. “On Bunching and Identification of the Taxable Income Elasticity.” Journal of Political Economy 129 (8): 2320–43.
Bosch, Nicole, Vincent Dekker, and Kristina Strohmaier. 2020. “A Data-Driven Procedure to Determine the Bunching Window: An Application to the Netherlands.” International Tax and Public Finance 27: 951–79.
Calonico, Sebastian, Matias D Cattaneo, and Max H Farrell. 2020. “Optimal Bandwidth Choice for Robust Bias-Corrected Inference in Regression Discontinuity Designs.” The Econometrics Journal 23 (2): 192–210.
Cattaneo, Matias D, Nicolás Idrobo, and Rocı́o Titiunik. 2019. A Practical Introduction to Regression Discontinuity Designs: Foundations. Cambridge University Press.
Chetty, Raj, Nathaniel Hendren, and Lawrence F Katz. 2016. “The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment.” American Economic Review 106 (4): 855–902.
Gerard, François, Miikka Rokkanen, and Christoph Rothe. 2020. “Bounds on Treatment Effects in Regression Discontinuity Designs with a Manipulated Running Variable.” Quantitative Economics 11 (3): 839–70.
Kleven, Henrik Jacobsen. 2016. “Bunching.” Annual Review of Economics 8: 435–64.
Lee, David S. 2009. “Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects.” The Review of Economic Studies, 1071–1102.
McCrary, Justin. 2008. “Manipulation of the Running Variable in the Regression Discontinuity Design: A Density Test.” Journal of Econometrics 142 (2): 698–714.
Zhang, Junni L, and Donald B Rubin. 2003. “Estimation of Causal Effects via Principal Stratification When Some Outcomes Are Truncated by ‘Death’.” Journal of Educational and Behavioral Statistics 28 (4): 353–68.